The text of a page is what people read. The metadata is what machines read about it: the title in the browser tab, the description a search engine shows, the image and headline that appear when someone shares the link. Most of the time you want the metadata, not the text, and it is a fraction of the cost to get.
This lesson builds a link preview service, the kind that turns a pasted URL into a card, and then covers the two jobs that sit next to it: listing the links on a page and finding out where a short link really goes.
01What a page says about itself
Pages describe themselves in a few overlapping vocabularies:
- HTML basics. The
<title>element,<meta name="description">,<html lang>, the charset and viewport tags. - Open Graph.
og:title,og:description,og:image,og:site_name,og:typeand friends, created for Facebook shares and now read by nearly every chat and social app. - Twitter cards.
twitter:card,twitter:title,twitter:image, used by X and as a fallback by other apps. - Structured data. JSON-LD blocks describing products, articles, events, with prices and authors.
Metadata Extractor returns all of these in one flat object for 2 credits. The basics (title, description, lang, charset, author, keywords, robots, viewport, googlebot) are on every plan. Open Graph, Twitter, jsonld, favicons, canonical, headings and imgTags are premium fields, so design your code to work when they come back null.
curl -X POST "https://api.apiverve.com/v1/metadataextractor" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://github.com"}'requestUrl is what you sent; url is where the page ended up. Open Graph keys contain colons, so read them with bracket syntax.
Four things from a real response matter more than the field list.
Missing means empty string, not null. Calling it on https://example.com, a page with no description or Open Graph tags, returned "description": "", "og:title": "", "og:image": "" and so on. In JavaScript, data['og:title'] ?? data.title therefore returns the empty string and your card shows no title. Use || for fallbacks on this API, or normalise empty strings to null once at the boundary.
Keys have colons in them. data.og:title is a syntax error. Write data['og:title'] in JavaScript and data.get("og:title") in Python.
Sites add their own tags, and they come through. The GitHub response included dozens of keys that are not in the documented list, such as route-pattern, octolytics-url and apple-itunes-app, because the API returns every <meta> it finds. Pick the keys you need. Never spread the whole object into a database row or a response to your own clients, or you will store whatever any site on the internet decides to put in its head.
url is the final address. requestUrl is exactly what you sent; url is where the page landed after redirects (https://github.com became https://github.com/). Use url as the base for resolving relative paths and as the key you deduplicate on.
Your preview code is const title = data['og:title'] ?? data.title; and pages without Open Graph tags show a blank title. Why?
02Building the preview card
A good preview tries the most specific source first and falls back cleanly:
| Card field | Try in order |
|---|---|
| Title | og:title, twitter:title, title, the hostname |
| Description | og:description, twitter:description, description, nothing |
| Image | og:image, twitter:image, image, nothing |
| Site name | og:site_name, the hostname without www. |
Image URLs deserve one more step. Open Graph says the value should be absolute, and plenty of sites ignore that and write /images/share.png. Resolve every image against the final url with the platform URL parser, which handles absolute, root-relative, path-relative and protocol-relative values the same way, and refuse anything that is not http or https. A javascript: or data: value in an <img src> you render is a bug waiting to become an exploit.
// Node 18+, ESM, Express 4. npm install express
import express from 'express';
import { apiverve } from './apiverve.mjs'; // the helper from lesson one
const app = express();
const cache = new Map(); // url -> { card, expires }
const DAY = 24 * 60 * 60 * 1000;
const pick = (...values) => values.find((v) => typeof v === 'string' && v.trim()) || null;
function absolute(value, base) {
if (!value) return null;
try {
const u = new URL(value, base);
return u.protocol === 'http:' || u.protocol === 'https:' ? u.href : null;
} catch {
return null;
}
}
export async function preview(input) {
const hit = cache.get(input);
if (hit && hit.expires > Date.now()) return hit.card;
const m = await apiverve('metadataextractor', { url: input });
const host = new URL(m.url).hostname.replace(/^www\./, '');
const card = {
url: m.url,
title: pick(m['og:title'], m['twitter:title'], m.title, host),
description: pick(m['og:description'], m['twitter:description'], m.description),
image: absolute(pick(m['og:image'], m['twitter:image'], m.image), m.url),
site: pick(m['og:site_name'], host),
};
cache.set(input, { card, expires: Date.now() + DAY });
return card;
}
app.get('/preview', async (req, res) => {
let target;
try {
target = new URL(String(req.query.url));
if (!['http:', 'https:'].includes(target.protocol)) throw new Error();
} catch {
return res.status(400).json({ error: 'url must be an http(s) URL' });
}
try {
res.json(await preview(target.href));
} catch (e) {
res.status(502).json({ error: 'could not read that page' });
}
});
app.listen(3000);A day-long cache is right for previews. Sites rarely change their share image, and a chat app that renders the same viral link ten thousand times should pay for it once.
When the title is all you need
URL Title returns title and, on paid plans, h1: an array of every <h1> on the page. It costs 10 credits. Metadata Extractor also returns title and costs 2. For a plain title, the bigger API is the cheaper one, so do not pick URL Title by its name.
curl -X POST "https://api.apiverve.com/v1/urltitle" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com"}'title is the <title> element. h1 is premium and is an empty array here because example.com uses no h1.
URL Title earns its place when the h1 list is the point: checking that each page of your site has exactly one <h1>, or comparing the visible headline against the <title> to spot pages where the two drifted apart after an edit. When a URL cannot be reached at all, expect an error instead of an empty title, and record it as "unreachable" rather than retrying in a loop.
03Every link on a page
Link Scraper returns the anchors on a page as links[], each with text, href and external, plus linkCount, internalLinkCount, externalLinkCount and maxLinksReached. It costs 25 credits.
curl -X POST "https://api.apiverve.com/v1/linkscraper" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com"}'One external link. maxLinksReached false means this is the complete list.
The API does some cleanup for you: empty anchors and same-page #fragment links are dropped, whitespace is stripped from anchor text, and root-relative hrefs such as /wiki/Main_Page are resolved to absolute URLs on the page's host. Note the wording of that last one: root-relative. Treat every href as untrusted and run it through new URL(href, data.url) before you store it, exactly as with preview images. It costs nothing and turns any path-relative stragglers into absolute URLs.
The limit and the flag
By default you get at most 50 links. Raising it with maxlinks is a premium parameter. Calling the Wikipedia article on web scraping with maxlinks: 5 returned the first five links, all sidebar navigation ("Main page", "Contents", "Current events"), and maxLinksReached: true. Two lessons in that:
- Check
maxLinksReachedevery time. When it istrue, the list is truncated, and any conclusion about "this page does not link to X" is unproven. - Links come in document order. The first links on most sites are navigation. If you want the links inside the content, you need enough headroom to get past the chrome, or you should filter by anchor text and domain afterwards.
Query strings
includequery defaults to false, which strips query strings from hrefs. That is the right default for deduplication, because /pricing?utm_source=footer and /pricing?utm_source=nav are the same page. It is the wrong default for sites where the query string is the identity of the page, such as /product?id=4471 or /search?page=3. Know which kind of site you are crawling and set it deliberately.
Link Scraper returns 50 links for a blog home page and none of them is the post you know is linked there. What should you check first?
04Where does this short link go?
Short links hide their destination, which is fine for printed flyers and a problem for anything that makes decisions based on a URL: moderation, phishing checks, analytics deduplication, or a link preview that should show the real site. URL Unshortener follows the redirects and reports destination, destinationDomain, redirectCount and unshortened, with the full redirectChain on paid plans. It costs 10 credits.
curl -X POST "https://api.apiverve.com/v1/urlunshortener" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://bit.ly/3k3j4"}'Four hops from the short link to the final page. redirectChain lists each one.
That link took four redirects: the bit.ly hop, then an http to https upgrade, a path change, and another upgrade. The chain is useful evidence. A short link that bounces through an ad network or an unrelated domain before landing somewhere legitimate is worth flagging even when the final page is clean.
Sending a URL that does not redirect is safe. https://example.com came back with unshortened: false, redirectCount: 0 and destination equal to the input. That means you can run every URL through the unshortener without first guessing whether it is a short link, and branch on unshortened afterwards.
from urllib.parse import urlparse
from apiverve import apiverve # helper from lesson one
SHORTENERS = {"bit.ly", "t.co", "tinyurl.com", "goo.gl", "ow.ly", "buff.ly", "lnkd.in"}
def canonical_target(url: str) -> dict:
host = (urlparse(url).hostname or "").lower()
if host not in SHORTENERS:
return {"url": url, "domain": host, "hops": 0}
data = apiverve("urlunshortener", {"url": url})
return {
"url": data["destination"],
"domain": data["destinationDomain"],
"hops": data["redirectCount"],
}Checking a known-shortener list first is an optimisation, not a requirement: it saves 10 credits on the majority of URLs that are already direct. Drop the check if you need to catch every redirect, including a site's own /go/ links.
Two judgment calls before you ship this:
- Do not unshorten on behalf of someone who did not ask. Some short links carry per-recipient tracking, and following them registers a click. If you expand every link in incoming email for a security scan, you may be marking messages as "opened" for the sender. That trade-off is usually worth it for security tooling and rarely worth it for convenience features.
- Tracking parameters survive the trip. Destinations often arrive with
utm_*,fbclidorgclidattached. Strip them before you display or store the URL, both for privacy and so that the same page does not appear under twenty different addresses.
Next, the visual side of a page: screenshots and image inventories, and how to handle the temporary file links those APIs return, in Screenshots and images.