Most SEO regressions are not strategy failures. Someone ships a template change and every product page loses its meta description. A staging noindex survives the deploy. A redesign wraps the logo in an <h1> and the real heading becomes the second one. Nobody notices for three weeks, because nobody looks at the <head> of a page they can already see rendered.
Those are exactly the problems a machine is good at catching. You do not need a crawler or a headless browser for them; you need the page's metadata and a handful of structural counts, checked against rules you wrote down. Two catalog APIs return that data, and this lesson turns them into an audit that runs from a script, a CI job or a cron schedule.
01What each API sees
SEO Data Scraper (seoscraper) reads a URL and returns what the page declares about itself: the <title>, the meta description, Open Graph and Twitter card tags, the robots meta tag, the viewport and the charset. It is a POST with a JSON body containing url, and it costs 10 credits.
curl -X POST "https://api.apiverve.com/v1/seoscraper" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com"}'title is set, but description, robots and every og: field come back as empty strings.
Look at what came back for example.com. The title is Example Domain. Everything else that the page does not declare is an empty string, not null and not missing. That matters for your code: if (data.description) works, if (data.description !== null) silently passes a page with no description at all.
Also notice the key names. The scraper returns social tags under their literal attribute names, og:title, twitter:card and so on. In JavaScript that means data['og:title'], not data.og.title. In Python it is an ordinary dict key.
SEO Quick Validator (seovalidator) answers a different question: not "what does the page say" but "what is structurally wrong with it". It is also a POST with url, also 10 credits.
curl -X POST "https://api.apiverve.com/v1/seovalidator" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com"}'checks.h1Count is 0 and hasCanonical is false. The issues array explains each failure in words.
The useful part of this response is the checks object. When we ran it against example.com it reported:
| Check | Value | Reading |
|---|---|---|
hasTitle / titleLength | true / 14 | Title exists but is short |
hasMetaDescription / metaDescriptionLength | false / null | No description; length is null when absent |
h1Count | 0 | No H1 at all |
hasCanonical | false | No canonical link tag |
hasViewport | true | Mobile viewport declared |
imagesTotal / imagesMissingAlt | 0 / 0 | Nothing to check |
externalLinksTotal / externalLinksMissingRel | 1 / 1 | One external link without rel |
Alongside checks you get passed, issueCount and an issues array of plain-English strings. Paid plans also get seoScore (0-100) and a letter grade.
So between them: the scraper gives you the content of the tags, the validator gives you counts and presence flags, including the heading structure the scraper does not report. A useful audit needs both, because "has a title" and "the title is the same on all 400 product pages" are different failures.
Your audit code checks if (seo.description === null) flag('missing description'). A page with no meta description passes. Why?
02Don't let passed make your decisions
The tempting implementation is one line: fail the build if passed is false. Run it against a few real sites first. example.com fails partly because of Missing meta 'keywords' tag in head, and the sample response for apiverve.com lists the same issue. Google has ignored the meta keywords tag for ranking for many years. A gate that fails on it fails on almost every well-built site on the internet, and a gate that is always red gets switched off within a month.
That is the general lesson about audit tooling: the validator reports facts; your policy decides which facts are failures. Some rules are hard errors on any site (no title, two H1s on an article, noindex on a page you want ranked). Some are warnings that depend on the page type (a 30-character title is fine on a login page, thin on a landing page). Some are noise for you specifically (meta keywords; rel on external links if you do not care about it).
Write the policy down as data, not as scattered if statements. When marketing asks why a page failed, you can point to a rule with a name and a reason.
| Rule | Source field | Severity | Why |
|---|---|---|---|
| Title present | checks.hasTitle | error | No title means search engines invent one |
| Title 30-60 chars | checks.titleLength | warn | Longer titles truncate in results |
| Description present | seo.description | error | The snippet is otherwise picked from the body |
| Description 70-160 chars | checks.metaDescriptionLength | warn | Truncation and thin snippets |
| Exactly one H1 | checks.h1Count | error | Zero or several confuses the page topic |
Not noindex | seo.robots | error (on pages meant to rank) | The page is removed from the index |
| Canonical declared | checks.hasCanonical | warn | Duplicate URLs split their ranking signals |
| Images have alt | checks.imagesMissingAlt | warn | Accessibility first, image search second |
| Social preview | seo['og:title'], seo['og:image'] | warn | Shares render as a bare link |
Notice that meta keywords is simply absent from the table. Leaving a rule out is a decision too.
03Build the audit
Here is the whole thing as a Node 18+ ES module (save it as audit.mjs or set "type": "module"). It calls both APIs in parallel, applies the rules above and returns findings you can print or store.
const BASE = 'https://api.apiverve.com/v1';
const KEY = process.env.APIVERVE_API_KEY;
async function post(api, body) {
const res = await fetch(`${BASE}/${api}`, {
method: 'POST',
headers: { 'x-api-key': KEY, 'Content-Type': 'application/json' },
body: JSON.stringify(body),
signal: AbortSignal.timeout(20000),
});
const json = await res.json().catch(() => null);
if (!res.ok || json?.status !== 'ok') {
throw new Error(`${api} ${res.status}: ${json?.error ?? 'no body'}`);
}
return json.data;
}
const RULES = [
{ id: 'title-missing', level: 'error', test: (s, v) => !v.checks.hasTitle },
{ id: 'title-length', level: 'warn', test: (s, v) => v.checks.hasTitle && (v.checks.titleLength < 30 || v.checks.titleLength > 60) },
{ id: 'description-missing', level: 'error', test: (s) => !s.description.trim() },
{ id: 'description-length', level: 'warn', test: (s, v) => v.checks.hasMetaDescription && (v.checks.metaDescriptionLength < 70 || v.checks.metaDescriptionLength > 160) },
{ id: 'h1-count', level: 'error', test: (s, v) => v.checks.h1Count !== 1 },
{ id: 'noindex', level: 'error', test: (s) => /\bnoindex\b/i.test(s.robots) },
{ id: 'canonical-missing', level: 'warn', test: (s, v) => !v.checks.hasCanonical },
{ id: 'images-alt', level: 'warn', test: (s, v) => v.checks.imagesMissingAlt > 0 },
{ id: 'social-preview', level: 'warn', test: (s) => !s['og:title'] || !s['og:image'] },
];
async function audit(url) {
const [seo, val] = await Promise.all([
post('seoscraper', { url }),
post('seovalidator', { url }),
]);
const findings = RULES.filter((r) => r.test(seo, val)).map(({ id, level }) => ({ id, level }));
return { url, title: seo.title, findings, errors: findings.filter((f) => f.level === 'error').length };
}
const result = await audit(process.argv[2] ?? 'https://example.com');
console.log(JSON.stringify(result, null, 2));
process.exitCode = result.errors ? 1 : 0;Run it with node audit.mjs https://example.com. Against that page you should see title-length, description-missing, h1-count, canonical-missing and social-preview, and an exit code of 1. The exit code is what makes it useful in CI.
The Python version is the same shape. Keeping the rules as a list of (id, level, predicate) tuples means the policy stays readable even as it grows.
import os
import re
import sys
from concurrent.futures import ThreadPoolExecutor
import requests
BASE = "https://api.apiverve.com/v1"
KEY = os.environ["APIVERVE_API_KEY"]
def post(api, body):
r = requests.post(f"{BASE}/{api}", json=body, headers={"x-api-key": KEY}, timeout=20)
payload = r.json() if r.content else {}
if not r.ok or payload.get("status") != "ok":
raise RuntimeError(f"{api} {r.status_code}: {payload.get('error')}")
return payload["data"]
RULES = [
("title-missing", "error", lambda s, v: not v["checks"]["hasTitle"]),
("description-missing", "error", lambda s, v: not s["description"].strip()),
("h1-count", "error", lambda s, v: v["checks"]["h1Count"] != 1),
("noindex", "error", lambda s, v: re.search(r"\bnoindex\b", s["robots"], re.I) is not None),
("canonical-missing", "warn", lambda s, v: not v["checks"]["hasCanonical"]),
("images-alt", "warn", lambda s, v: v["checks"]["imagesMissingAlt"] > 0),
("social-preview", "warn", lambda s, v: not s["og:title"] or not s["og:image"]),
]
def audit(url):
with ThreadPoolExecutor(max_workers=2) as pool:
seo_f = pool.submit(post, "seoscraper", {"url": url})
val_f = pool.submit(post, "seovalidator", {"url": url})
seo, val = seo_f.result(), val_f.result()
findings = [{"id": i, "level": lvl} for i, lvl, test in RULES if test(seo, val)]
return {"url": url, "findings": findings, "errors": sum(f["level"] == "error" for f in findings)}
if __name__ == "__main__":
result = audit(sys.argv[1] if len(sys.argv) > 1 else "https://example.com")
print(result)
sys.exit(1 if result["errors"] else 0)A product page returns h1Count: 2. The design team says the logo is wrapped in an H1 on purpose. What should the audit do?
04Robots, noindex and what the scraper does not read
The robots field is the content of the robots meta tag on that page. It is the field that catches the most expensive mistake in this lesson: a staging environment configured with <meta name="robots" content="noindex"> and a deploy that carries the template to production. A regular expression on \bnoindex\b is enough; the tag can also say none, which means noindex, nofollow, so match that too if your stack ever emits it.
Be precise about what this does not cover. A page can be excluded from search in other ways that are not in its HTML: a robots.txt disallow rule, or an X-Robots-Tag response header. Those are not part of this response. If indexing is critical (and for revenue pages it is), check your robots.txt separately in the same job.
05Run it across a site without wasting credits
One URL is 20 credits with both calls. Auditing a 2,000-page site daily is 40,000 credits a day, and almost all of it is re-confirming that nothing changed. Three habits keep the cost in proportion:
- Audit templates, not pages. A site with 2,000 product pages usually has one product template. Pick two or three representative URLs per template, plus the pages that make money. A template bug shows up on every sample.
- Run on change, not on a clock. The audit's natural trigger is a deploy. Run it in CI against the preview URL, then once against production after release.
- Store results and diff them. Keep each run's findings keyed by URL. Alert when a rule newly fails, not every time it fails. A known warning that has been accepted should not page anyone.
Run the URLs with modest concurrency (three or four at a time is plenty) rather than all at once. You are auditing your own site, but the requests still hit your servers, and the per-minute rate limit applies to your key. If you get a 429, read the x-api-remaining-credits header: 0 means the month is used up and retrying will not help; anything else is the rate limit, and retry-after tells you how long to wait.
There is one more failure mode to plan for: the page that was not reachable. A 404, a login wall or a bot challenge produces either an API error or a response that describes the wrong page (a challenge page has a title too). Treat errors as their own outcome, unreachable, never as a pass, and sanity-check that the returned title is not something like "Just a moment..." before trusting the rest.
06Where to go from here
You now have an audit that fails on real problems, ignores noise, and costs a predictable number of credits. The next step in analysing a site is the layer underneath the HTML: what it is built with. Detect a website's tech stack covers how fingerprinting works, how to read confidence scores, and where detection is blind.