Moderate and process user-uploaded images

The day you add an image upload to your product, you become a publisher of whatever people send. Most of it will be fine: avatars, product photos, screenshots. Some of it will not, and the first explicit image that appears on a public profile page will be screenshotted and shared long after you remove it.

This lesson builds a screening step for uploads with four APIs, each answering a different question about an image:

None of them is a judge. They are classifiers that are usually right and sometimes confidently wrong. The design question running through every section is the same: what happens to the user, and to you, when the model is wrong?

01How these APIs take images

Input styles differ across the four, and it matters for how you wire them:

• nsfwimagedetector and imagecaption take a file upload (multipart/form-data, field image), JPG, PNG or GIF.
• facedetect and backgroundremover take either a file upload in image or a public image URL, as url or in image itself. They also accept WebP, and backgroundremover takes BMP.

For screening user uploads you will usually have the bytes in memory already, so uploading is natural. The URL style is convenient for images already in your bucket or returned by another API.

02Screen for explicit content first

NSFW Image Detector returns two booleans on every plan, isNSFW and isSafe, which are always opposites. On plans that include them, confidence (0 to 1, how likely the image is NSFW) and category (sfw or nsfw) let you build a middle band.

Screen first because it is cheap and because it gates everything else: an image that fails here should never reach a captioning model, a public URL or a CDN cache.

The decision is not binary in a well-run system. Use three outcomes:

• Reject clear violations, with a neutral message ("This image can't be used. Please choose another.").
• Hold for review anything uncertain: the image is stored privately, the user sees "processing", and a person decides.
• Publish the rest.

With only the booleans, "uncertain" has to come from context instead of a score: new accounts, accounts with previous rejections, or images going somewhere highly visible (a public profile, a marketplace listing) can be held for review even when isSafe is true, until the account has a track record. With confidence, set two thresholds and send the band between them to review.

03Detect faces, not people

Face Detector answers whether an image contains human faces. hasFaces is on every plan. Premium adds faceCount, a faces array of bounding boxes (x, y, width, height, confidence), imageWidth, imageHeight, averageConfidence and imageCoverage, the percentage of the image covered by faces.

1
2
3
4

curl -X POST "https://api.apiverve.com/v1/facedetect" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"image":"https://images.unsplash.com/photo-1507003211169-0a1dd7228f2d?w=640","confidence":0.5}'

The confidence parameter (0.1 to 1.0, default 0.5) is the minimum score for a detection to count. Raise it to avoid counting faces in posters, crowds in the background or patterns that look like faces; lower it if you are missing small or turned faces.

Practical uses:

• Profile photos must show a face. If hasFaces is false, prompt the user rather than reject outright: they may have uploaded a logo on purpose. With faceCount, a value above one is a hint that the photo is a group shot.
• Anonymise before publishing. Street photos, event pictures and support screenshots of video calls can contain bystanders. With bounding boxes you can blur each face region in your own image pipeline before the image goes public.
• Route identity checks. An ID document upload with no detected face is probably the wrong side of the card.

Be precise about what this is. Face detection finds that a face is present and where. It does not tell you who the person is, and you should keep it that way unless you have a lawful basis and explicit consent. Face recognition, matching a face to an identity, is treated as biometric processing in many places: the EU's GDPR counts biometric data used to identify someone as a special category, and laws such as Illinois's Biometric Information Privacy Act require written consent before collecting face geometry. Do not store face crops or bounding boxes alongside user identities "in case they are useful later". Use the result for the decision in front of you and discard it.

04Draft alt text with captions, then let people edit it

Image Caption returns one field, caption: a sentence describing the image, such as "A golden retriever running through a grassy field on a sunny day".

The obvious use is alt text, and it is a good one, with a constraint. A generic caption describes what is visible; good alt text describes what matters in context. On a product page, "a blue ceramic mug on a wooden table" is a fine start; the alt text the customer needs is "Harbor mug in navy, 350 ml, matte glaze". So:

• Pre-fill, do not publish. Put the caption in the alt text field of the upload form, where the user or editor can correct it. Most people will improve a draft; almost nobody writes alt text from a blank field.
• Never caption before screening. A caption of an explicit image is explicit text, stored and possibly indexed.
• Treat it as user-visible content. Captions can be wrong about people's age, gender or actions. If your product shows the caption publicly, someone will eventually be described inaccurately.

Captions are also useful internally: searching a media library for "whiteboard" or "receipt", or flagging uploads whose caption suggests they are not what the form asked for (a screenshot of text where you expected a product photo).

05Remove backgrounds for product and profile photos

Background Remover cuts the subject out and returns a PNG. Omit bgColor for a transparent background, or pass a hex colour such as #ffffff to fill it, which is what most marketplaces want for product listings.

1
2
3
4

curl -X POST "https://api.apiverve.com/v1/backgroundremover" \
  -H "x-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"image":"https://upload.wikimedia.org/wikipedia/commons/2/26/YellowLabradorLooking_new.jpg","bgColor":"#ffffff"}'

The response gives width and height (same as the input), background (transparent or the colour used), mimeType (always image/png), expires in milliseconds and downloadURL. Premium adds foregroundRatio, the share of the image kept as foreground, and outputSize.

At 40 credits this is the most expensive call in the course, so be deliberate:

• Run it once, when the seller uploads, and store the result. Never on page view.
• Keep the original. Cut-outs fail on fine hair, glass, and objects the same colour as the background; sellers need a way to revert.
• foregroundRatio, where available, is a cheap quality check. A value near 0 means almost everything was removed (the model lost the subject); near 1 means almost nothing was. Either is worth a human look.
• Transparent PNGs are large. If the result will sit on a white page anyway, ask for #ffffff and convert the PNG to JPEG or WebP with the converter from the first lesson.

A background job for product photos, sending the seller's uploaded image by URL and keeping both versions:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

import os
import requests

KEY = os.environ["APIVERVE_API_KEY"]


def make_cutout(source_url, out_path, bg_color="#ffffff"):
    res = requests.post(
        "https://api.apiverve.com/v1/backgroundremover",
        headers={"x-api-key": KEY},
        json={"url": source_url, "bgColor": bg_color},
        timeout=60,
    )
    body = res.json()
    if not res.ok or body.get("status") != "ok":
        raise RuntimeError(f"cutout failed ({res.status_code}): {body.get('error')}")
    data = body["data"]

    ratio = data.get("foregroundRatio")  # premium; None on plans without it
    if ratio is not None and (ratio < 0.05 or ratio > 0.95):
        return {"status": "review", "reason": f"foregroundRatio {ratio}"}

    png = requests.get(data["downloadURL"], timeout=60)
    png.raise_for_status()
    with open(out_path, "wb") as f:
        f.write(png.content)
    # Keep source_url as the original so the seller can revert.
    return {"status": "done", "path": out_path, "width": data["width"], "height": data["height"]}

06Put it together: one screening step for uploads

Order the checks by cost and by consequence: cheap, blocking checks first; expensive, optional work last and only on images that will be kept.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

// Node 18+, ESM. npm install express@4 multer@1
import express from 'express';
import multer from 'multer';

const KEY = process.env.APIVERVE_API_KEY;
const upload = multer({ storage: multer.memoryStorage(), limits: { fileSize: 5 * 1024 * 1024 } });
const app = express();

async function ask(api, file, extra = {}) {
  const form = new FormData();
  form.append('image', new Blob([file.buffer], { type: file.mimetype }), file.originalname);
  for (const [k, v] of Object.entries(extra)) form.append(k, String(v));
  const res = await fetch(`https://api.apiverve.com/v1/${api}`, {
    method: 'POST',
    headers: { 'x-api-key': KEY },
    body: form,
    signal: AbortSignal.timeout(30_000),
  });
  const body = await res.json();
  if (!res.ok || body.status !== 'ok') throw new Error(`${api} ${res.status}: ${body.error}`);
  return body.data;
}

app.post('/avatar', upload.single('photo'), async (req, res) => {
  const file = req.file;
  if (!file || !/^image\/(jpeg|png|gif)$/.test(file.mimetype)) {
    return res.status(400).json({ error: 'Upload a JPG, PNG or GIF.' });
  }

  let nsfw;
  try {
    nsfw = await ask('nsfwimagedetector', file);
  } catch (err) {
    // Screening unavailable: hold, never publish unscreened.
    return res.status(202).json({ status: 'held', reason: 'screening unavailable' });
  }
  if (nsfw.isNSFW) return res.status(422).json({ status: 'rejected', error: "This image can't be used." });

  const faces = await ask('facedetect', file, { confidence: 0.6 }).catch(() => null);
  const caption = await ask('imagecaption', file).then((d) => d.caption).catch(() => '');

  // Store file.buffer privately here, then publish.
  res.json({
    status: 'accepted',
    hint: faces && !faces.hasFaces ? 'We could not find a face. Is this the photo you meant?' : null,
    altDraft: caption,
  });
});

app.listen(3000);

Look at the failure handling, because that is the part that matters in production:

• NSFW screening failing closed. If the classifier is down or times out, the image is held, not published. An upload that waits an hour is an inconvenience; an unscreened explicit image on a public page is an incident.
• Face detection and captions failing open. They are advisory. If either fails, the upload still succeeds without the hint or the alt text draft.
• Background removal is not in the request path at all. For product photos, queue it as a background job after acceptance, so the seller is not waiting on the slowest, most expensive call.

07Consent and transparency

Image analysis touches people who never signed up for your product: the bystander in a photo, the person in someone else's profile picture. A few commitments keep you on the right side of both law and users' expectations.

• Say what you do. Your privacy notice should state that uploads are automatically screened for prohibited content and, if you use it, analysed for faces, and which processors handle that.
• Purpose limitation. Use each result for the decision it was made for. A face check for "is this a profile photo" does not become data for advertising or identity matching later.
• Retention. Keep classifier results as long as you need them to explain a decision (a rejection appeal, a moderation audit), then delete them.
• An appeal path. Any automated rejection should have a human route back. That is good practice everywhere, and in some jurisdictions, decisions made solely by automated processing that significantly affect people come with explicit rights.

You now have the full image toolkit: converting formats, rendering images from HTML and Markdown, reading text out of them, and screening what users send you. The final course of the certification puts documents to work in larger workflows, with QR codes, barcodes, invoices and charts: Automate documents.