Change monitoring sounds trivial. Fetch the page, compare it with last time, alert if different. Build exactly that and point it at a real site, and it will alert on nearly every run. The footer says "© 2026" and will roll over. The sidebar shows "Updated 4 minutes ago". A product page rotates which reviews it shows. An A/B test swaps the headline for half the visitors. None of these are changes anyone cares about, and after the third false alarm the person receiving the alerts mutes the channel. The one real change, a price going up or a clause appearing in the terms, then goes unread.
So the design goal is not "detect changes". It is "detect the changes a human would care about, and nothing else". Everything in this lesson serves that.
01Fetch in a format that diffs well
You have two extraction options in the catalog. Website to Text returns one flat string of body copy at 25 credits. URL to Markdown returns the page as Markdown, with headings, lists and paragraphs on their own lines, at 10 credits.
For monitoring, Markdown wins on both counts. It is cheaper, and line structure is what makes a diff readable: "the third bullet under Pricing changed" is an alert someone can act on, while "character 4,182 of a 9,000-character string changed" is not.
curl -X POST "https://api.apiverve.com/v1/urltomarkdown" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com/","includeLinks":false}'markdown is the page body. wordCount is the number to watch for broken fetches.
Two parameters matter here: includeLinks and includeImages, both off by default. Leave them off unless the links themselves are what you are monitoring. Link URLs are full of tracking parameters, session ids and CDN hashes that change on every deploy, and every one of those would be a false alarm. If you are watching a downloads page for a new release file, turn links on and normalise the URLs instead.
Also notice wordCount. When we fetched example.com, markdown contained the English paragraph followed by the same notice in five other languages, 133 words in all. Website to Text, on the same page and the same day, returned only the English paragraph. Two extractors can disagree about what "the content" is, which is why a monitor must always compare like with like: the same API, the same parameters, every run. Changing either one is a schema change, and the first run after it should re-baseline silently instead of alerting.
02Normalise: the step that decides whether you get muted
Normalisation turns the fetched page into a canonical form where irrelevant differences disappear. Do it in your own code, line by line, because the rules are specific to the pages you watch.
The rules that remove most of the noise:
- Collapse whitespace within each line, and drop empty lines. Re-wrapping and extra blank lines are not changes.
- Remove volatile lines entirely: relative times ("3 minutes ago"), "Last updated" stamps, copyright lines, view and comment counters, "Loading..." placeholders.
- Mask volatile tokens inside otherwise stable lines: full dates, times, long numeric ids and hex hashes become placeholders such as
<date>or<id>. - Keep a per-site ignore list for what the generic rules miss. You will add to it after the first few false alarms; that is normal, not failure.
Be careful not to mask the thing you are watching. A rule that turns every number into <num> also hides the price change you set the monitor up to catch. Mask by shape (dates, times, long ids), not "all digits".
const DROP_LINE = [
/\b\d+\s+(second|minute|hour|day|week)s?\s+ago\b/i,
/^(last\s+)?updated\b/i,
/(©|©|\bcopyright\b)/i,
/\b\d[\d,]*\s+(views|comments|shares|likes)\b/i,
/^loading\b/i,
];
const MASK = [
[/\b\d{4}-\d{2}-\d{2}(t[\d:.]+z?)?\b/gi, '<date>'],
[/\b(jan|feb|mar|apr|may|jun|jul|aug|sep|oct|nov|dec)[a-z]*\.?\s+\d{1,2},?\s+\d{4}\b/gi, '<date>'],
[/\b\d{1,2}:\d{2}(:\d{2})?\s*(am|pm)?\b/gi, '<time>'],
[/\b[0-9a-f]{16,}\b/gi, '<id>'],
];
export function normalise(markdown, ignore = []) {
return markdown
.split('\n')
.map((line) => line.replace(/\s+/g, ' ').trim())
.filter((line) => line && !DROP_LINE.some((re) => re.test(line)) && !ignore.some((re) => re.test(line)))
.map((line) => MASK.reduce((l, [re, to]) => l.replace(re, to), line))
.join('\n');
}You monitor a SaaS pricing page. To kill false alarms from a visitor counter, a teammate proposes replacing every number on the page with <num> before hashing. What is wrong with that?
03Hash for the cheap comparison
Comparing two full pages on every run works, but storing a fingerprint is smaller and makes the common case, nothing changed, a single string comparison. SHA-256 of the normalised text is the standard choice: deterministic, and collisions are not a practical concern.
Hash Generator computes it for 1 credit and can return several algorithms at once.
curl "https://api.apiverve.com/v1/hashgenerator?text=hello+world&algorithms=sha256%2Cmd5" \
-H "x-api-key: YOUR_API_KEY"hashes.sha256 is the fingerprint. Hash the normalised text, never the raw page.
For hello world, the sha256 value we got was b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9. That is the same digest Node's crypto module produces, because a hash is a pure function of its input. So which should you use? In a Node or Python service, hash locally: it is free and adds no latency. The API is the right tool where you have no crypto library, such as a spreadsheet, a no-code automation or a constrained edge runtime, and as an independent check that two systems agree on the fingerprint of the same text.
The one bug to avoid: hash the normalised text, not the raw Markdown. If you hash the raw page, all the normalisation work is wasted, because the hash still changes with every timestamp.
import { createHash } from 'node:crypto';
export const sha256 = (text) => createHash('sha256').update(text, 'utf8').digest('hex');04Diff, confirm, then alert
When the hash differs, work out what changed so the alert can say it. For monitoring, a line-set diff is enough and far simpler than a full edit-distance algorithm: lines present now but not before were added, lines present before but not now were removed. It ignores reordering, which for most pages is noise anyway.
Then, before alerting, confirm. Many false alarms are transient: an A/B variant, a rotated testimonial, a cache that served a stale copy, a CDN edge that briefly returned an error page. Require the same new fingerprint on two consecutive runs before you alert. It costs one extra fetch per real change and removes most of the flapping.
Two more guards belong here:
- A shrinking page is a fetch failure, not a change. If
wordCountdrops by more than about half, you have probably received a bot challenge, a maintenance page or an error template. Log it, keep the old baseline, and do not alert. - The first run is a baseline. Store it silently. An alert that says "everything was added" helps nobody.
A page's fingerprint changes on Monday's run, then returns to the original fingerprint on Tuesday's run. With the confirm-on-two-runs rule, what happens?
05The monitor
Node 18+, ESM, reusing verve.js from lesson 1 and normalise.js above. State lives in state.json, one entry per URL. Alerts go to any webhook that accepts a JSON body with a text field, which covers Slack and many chat tools; set ALERT_WEBHOOK_URL, or leave it unset to print alerts instead.
import { readFile, writeFile, rename } from 'node:fs/promises';
import { createHash } from 'node:crypto';
import { verve } from './verve.js';
import { normalise } from './normalise.js';
const STATE_FILE = './state.json';
const sha256 = (t) => createHash('sha256').update(t, 'utf8').digest('hex');
const WATCH = [
{ url: 'https://example.com/', ignore: [] },
// { url: 'https://vendor.example/pricing', ignore: [/^customers? love/i] },
];
async function loadState() {
try {
return JSON.parse(await readFile(STATE_FILE, 'utf8'));
} catch {
return {};
}
}
async function saveState(state) {
await writeFile(STATE_FILE + '.tmp', JSON.stringify(state, null, 2));
await rename(STATE_FILE + '.tmp', STATE_FILE); // atomic replace: a crash never leaves half a file
}
function lineDiff(before, after) {
const a = new Set(before.split('\n'));
const b = new Set(after.split('\n'));
return {
added: [...b].filter((l) => !a.has(l)),
removed: [...a].filter((l) => !b.has(l)),
};
}
async function alert(url, diff) {
const lines = [
`Change on ${url}`,
...diff.removed.slice(0, 10).map((l) => `- ${l}`),
...diff.added.slice(0, 10).map((l) => `+ ${l}`),
];
const text = lines.join('\n');
if (!process.env.ALERT_WEBHOOK_URL) return console.log(text);
await fetch(process.env.ALERT_WEBHOOK_URL, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ text }),
});
}
async function check(entry, prev) {
const page = await verve('urltomarkdown', { url: entry.url });
const text = normalise(page.markdown ?? '', entry.ignore);
const hash = sha256(text);
const now = new Date().toISOString();
if (!prev) return { hash, text, words: page.wordCount, checkedAt: now }; // baseline, no alert
if (prev.words && page.wordCount < prev.words * 0.5) {
console.warn(`${entry.url}: ${page.wordCount} words vs ${prev.words}; treating as a bad fetch`);
return { ...prev, checkedAt: now };
}
if (hash === prev.hash) return { ...prev, pending: undefined, checkedAt: now };
if (prev.pending?.hash !== hash) {
return { ...prev, pending: { hash, seenAt: now }, checkedAt: now }; // first sighting: wait for confirmation
}
await alert(entry.url, lineDiff(prev.text, text));
return { hash, text, words: page.wordCount, checkedAt: now, changedAt: now };
}
const state = await loadState();
for (const entry of WATCH) {
try {
state[entry.url] = await check(entry, state[entry.url]);
} catch (err) {
console.error(`${entry.url}: ${err.message}`); // a failed fetch leaves the baseline untouched
}
}
await saveState(state);Run it a few times. The first run writes a baseline. Later runs print nothing until the page really changes, and then only after the change has been seen twice.
export APIVERVE_API_KEY=YOUR_API_KEY
node monitor.js
cat state.json06Scheduling and what it costs
Run the monitor from cron, a scheduled CI job or a serverless scheduler. Two habits keep it well behaved. Add jitter, a random delay of a few minutes, so you are not one of thousands of jobs hitting a site at exactly the top of the hour. And keep the per-host gap from lesson 1 if several watched URLs share a host.
# crontab: every 6 hours, with up to 10 minutes of jitter ($RANDOM needs bash, not sh)
SHELL=/bin/bash
0 */6 * * * sleep $((RANDOM % 600)) && cd /opt/monitor && node monitor.js >> monitor.log 2>&1Pick the interval from the cost, not the other way round. At 10 credits per check:
| Pages | Interval | Checks per 30 days | Credits per 30 days |
|---|---|---|---|
| 20 | daily | 600 | 6,000 |
| 20 | every 6 hours | 2,400 | 24,000 |
| 50 | hourly | 36,000 | 360,000 |
Confirmation costs nothing extra in the code above, because it waits for the next scheduled run; the price is that an alert arrives one interval late. If that delay matters, schedule a single re-check a few minutes after a first sighting instead, for 10 credits per real change. Most pages that matter, such as terms of service, pricing, documentation and job boards, change on a scale of days. Hourly checks are rarely worth twenty-four times the daily cost; when they are, it is usually a sign the page has a feed or an API you should be reading instead.
The final lesson combines everything so far into one pipeline that discovers pages, extracts, cleans, enriches and stores them, and can be re-run safely: Build a content pipeline.