Every content pipeline starts with the same unglamorous step: get the words off the page. A search index, a summariser, a retrieval-augmented chatbot and a change monitor all want the same thing, which is the text a human would read, without the navigation bar, the cookie banner, the footer links and the forty <div> wrappers around each paragraph.
There is no single correct answer to "what is the text of this page". A news article's text is the story. A Wikipedia article's text arguably includes the infobox table but not the 300 category links at the bottom. A pricing page's text is mostly in lists and tables that a pure article extractor will mangle. So instead of one tool you get three, and the skill is knowing what each one keeps.
01Three converters, three inputs
| API | You send | You get back | Credits per call |
|---|---|---|---|
| Website to Text | A URL | One flat string of text, plus title, date, language | 25 |
| URL to Markdown | A URL | Markdown with headings, lists and tables preserved, plus counts | 10 |
| HTML to Text | Raw HTML you already have | The main article text, or nothing | 2 |
Two things stand out before you write any code. First, the URL-based APIs fetch the page for you, which is why they cost more: they do the network work, follow redirects and parse whatever comes back. Second, the cheapest one does not fetch anything. If your system already holds the HTML (a crawler you run, an email body, a CMS export, a page a user pasted), sending that HTML to HTML to Text costs a fraction of asking another service to fetch the page again.
All three are POST endpoints that take a JSON body. Here is one helper used for the rest of this lesson.
// Node 18+, ESM. Run with: node yourfile.mjs
const KEY = process.env.APIVERVE_API_KEY;
export async function apiverve(id, body) {
const res = await fetch(`https://api.apiverve.com/v1/${id}`, {
method: 'POST',
headers: { 'x-api-key': KEY, 'content-type': 'application/json' },
body: JSON.stringify(body),
signal: AbortSignal.timeout(30_000),
});
const json = await res.json();
if (!res.ok || json.status !== 'ok') {
throw new Error(`${id} ${res.status}: ${json.error ?? 'unknown error'}`);
}
return json.data;
}import os
import requests
KEY = os.environ["APIVERVE_API_KEY"]
def apiverve(api_id: str, body: dict) -> dict:
res = requests.post(
f"https://api.apiverve.com/v1/{api_id}",
headers={"x-api-key": KEY},
json=body,
timeout=30,
)
payload = res.json()
if not res.ok or payload.get("status") != "ok":
raise RuntimeError(f"{api_id} {res.status_code}: {payload.get('error')}")
return payload["data"]The 30-second timeout is deliberate. These calls wait on someone else's web server, and a slow origin is the most common reason one of them takes longer than a typical lookup API.
02Website to Text: everything, flattened
Website to Text returns the page's readable text as a single string, along with title, title_alt, description, date, language and publisher.
curl -X POST "https://api.apiverve.com/v1/websitetotext" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com"}'Small page, so the text is one sentence. Notice the language code and the null date.
On a tiny page that looks perfect. On a real article it shows its character. Converting the Wikipedia article on RSS returned roughly the whole page as one paragraph-less line: the intro, then every section heading followed by a literal [edit], the example XML from the article body, the full reference list with ↑ markers, the navigation boxes at the bottom, and finally the hidden maintenance categories. Nothing was lost, and nothing was organised.
Three details from that response are worth remembering because they will bite you in production:
- No paragraph breaks. The string has no newlines. You cannot split it into paragraphs afterwards, because the information is gone. If structure matters downstream, this is the wrong tool.
dateis a guess. For that article it returned1999-03-15, which is the date RSS was first released, lifted from the body text. It is not when the page was published or edited. Treatdateas a hint for articles that declare a publication date, and never as a crawl timestamp or a freshness signal.titlemay be the site, not the page. The same call returned"Wikipedia"as the title. When you need the page's own title, cross-check with the Markdown converter'stitleor the metadata API in the next lesson.
Where Website to Text is the right tool: full-text search, keyword and entity extraction, language detection, spam and content classification, and "did this page mention X" checks. All of those treat text as a bag of words and do not care about paragraphs.
You store data.date from Website to Text as the article's publish date and use it to hide stale content. What is the risk?
03URL to Markdown: keep the structure
URL to Markdown returns markdown, title, wordCount, imageCount and linkCount. The same Wikipedia page came back with its ## and ### headings intact, the infobox as a Markdown table, and bullet lists as bullet lists. That structure is the whole reason to prefer it when text is going into a language model.
curl -X POST "https://api.apiverve.com/v1/urltomarkdown" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url":"https://example.com"}'wordCount and linkCount describe the converted content. Links and images are counted even when they are not written into the Markdown.
includeLinks and includeImages are premium parameters, both false by default. With them off, anchor text stays in the Markdown but the URLs do not, and linkCount still tells you how many there were. For most retrieval work you want them off: a paragraph stuffed with [text](https://...) URLs spends tokens on strings the model cannot use and makes embeddings noisier. Turn links on when the links are the content, such as a documentation index or a reading list.
Why Markdown for LLM and RAG pipelines
Retrieval systems split documents into chunks, embed each chunk, and later hand the best-matching chunks to a model. The quality of that system depends heavily on where the chunk boundaries fall. Splitting a flat string every 1,000 characters cuts sentences in half and glues the end of one topic to the start of another. Splitting Markdown at headings gives you chunks that are each about one thing, and the heading itself is a short, dense description of that thing.
The splitter below keeps the heading path with each chunk, so a chunk from "Variants" on the RSS page carries RSS - Wikipedia > Variants as its first line. That small prefix noticeably improves retrieval, because a paragraph that never names its subject still matches queries about it.
import { apiverve } from './apiverve.mjs';
const MAX_CHARS = 2000;
export async function pageToChunks(url) {
const { title, markdown } = await apiverve('urltomarkdown', { url });
const chunks = [];
let path = [title];
let buf = [];
const flush = () => {
const body = buf.join('\n').trim();
buf = [];
if (body.length < 40) return; // skip empty sections and stubs
for (let i = 0; i < body.length; i += MAX_CHARS) {
chunks.push({ url, heading: path.join(' > '), text: body.slice(i, i + MAX_CHARS) });
}
};
for (const line of markdown.split('\n')) {
const m = /^(#{1,3})\s+(.*)/.exec(line);
if (m) {
flush();
path = [title, ...path.slice(1, m[1].length - 1), m[2]];
continue;
}
buf.push(line);
}
flush();
return chunks;
}
const chunks = await pageToChunks('https://en.wikipedia.org/wiki/RSS');
console.log(chunks.length, chunks[3]?.heading);Sections longer than the cap are sliced, which is crude but rarely matters: a heading-scoped section that exceeds 2,000 characters is still a far better chunk than an arbitrary window. If you want overlap between slices, add it there.
What to strip before you embed
Neither converter knows which parts of a page you care about. On reference-style pages the bottom third is often citations, "See also" lists and navigation boxes. Embedding those chunks pollutes retrieval, because a question about RSS history now matches a list of 40 citation titles. A short denylist of heading names (References, External links, See also, Notes, Further reading) dropped before embedding removes most of it. Do this on the Markdown, where the headings still exist, not after flattening.
You are building a support chatbot over 500 help-center pages. Which pairing gives the most useful retrieval chunks?
04HTML to Text: when you already have the HTML
HTML to Text is the odd one out. It takes html, not a URL, and it does not try to keep everything. It looks for the main article body and returns that, reporting how it got there in extractionMethod.
curl -X POST "https://api.apiverve.com/v1/htmltotext" \
-H "x-api-key: YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"html":"<html><body><nav>Home | Docs | Pricing</nav><article><h1>Version 2.4 released</h1><p>Version 2.4 adds batch exports and fixes the timezone bug in scheduled reports.</p><p>Upgrade by running the installer again.</p></article><footer>Copyright 2026 Example Inc.</footer></body></html>"}'The nav and footer are gone, and so is the h1. extractionMethod is article.
That input returns the two paragraphs, separated by a blank line, with wordCount 19 and characterCount 120. The navigation and footer are dropped, which is the point. So is the <h1>, which is the surprise. Article extraction keeps body paragraphs, not headings. If you need the headline, take it from the HTML yourself or from the metadata API.
The three values of extractionMethod are the contract you code against:
extractionMethod | parsed | What happened |
|---|---|---|
article | true | A main body was found; text is its paragraphs |
markup | true | No article body; text is the markup with tags stripped |
none | false | Nothing usable; text is null and the counts are absent |
The markup fallback needs care. Sending <ul><li>Starter</li><li>Pro</li><li>Mega</li></ul> returned the text StarterProMega with a wordCount of 1. Inline-joined list items are fine for a "does this page mention Pro" check and useless for display. If your input is mostly lists and tables (pricing pages, spec sheets, navigation-heavy pages), use URL to Markdown or parse the HTML yourself.
The none case is where real bugs live. An image-only fragment returned "text": null, "parsed": false, and no wordCount field at all. Code that does data.text.length or sums data.wordCount will throw or produce NaN. Branch on parsed first.
from typing import Optional
from apiverve import apiverve
def extract_body(html: str) -> Optional[str]:
data = apiverve("htmltotext", {"html": html})
if not data.get("parsed"):
return None
if data["extractionMethod"] == "markup" and data["wordCount"] < 20:
return None # probably a nav fragment or list soup, not content
return data["text"]
if __name__ == "__main__":
with open("saved-page.html", encoding="utf-8") as f:
print(extract_body(f.read()))05Pages that render in the browser
Some sites send an almost empty HTML document and build the page with JavaScript after it loads. Whether a URL-based converter sees the finished page depends on whether it runs that JavaScript, and you should not assume either way for a given site. Measure instead. Convert the page, then compare wordCount against what you see in a browser. A product page that shows 400 words to you and returns 12, or returns only "Loading..." or "Enable JavaScript to use this site", was rendered client-side and the content never reached the converter.
When that happens you have three options, in order of preference:
- Find the data source. Client-rendered pages fetch their content from somewhere, often a JSON endpoint visible in the browser's network tab, or an RSS feed (lesson four). That source is more stable than any rendering of it.
- Render it yourself and send the HTML. Run a headless browser in your own infrastructure, take
page.content()after the page settles, and pass that HTML to HTML to Text. You pay for the browser, and the extraction costs 2 credits. - Accept the gap. For a monitoring job across thousands of sites, logging "too little text" and moving on is often the right call.
A cheap guard catches this in batch jobs before bad data lands in your index:
export function looksEmpty({ markdown, wordCount }) {
if (wordCount < 50) return true;
return /enable javascript|loading\.\.\.|please wait/i.test(markdown.slice(0, 500));
}06Choosing, and the rules that come with it
The decision usually takes one question: what happens to the text next?
- Search index, classifier, keyword or language detection: Website to Text. Structure is irrelevant and you get
languagefor free. - Anything a language model reads, or anything a human reads later: URL to Markdown. Headings and lists survive, and the call is cheaper.
- You already fetched the HTML, or you only want the article body: HTML to Text. It is the cheapest by a wide margin and refuses to return junk when there is no article.
Cache by URL. Page text changes on the scale of days for most sites, so a 24-hour cache on conversions is reasonable, and re-converting the same URL inside a single batch run is pure waste.
Finally, extracting text does not change who owns it. Respect a site's robots.txt and terms for automated access, do not republish full articles you converted, and be careful with pages that contain personal data: storing the text of a public profile page in your vector database is still storing personal data, with the obligations that come with it. Summaries, quotes with attribution and private indexing for your own users are the normal, defensible uses.
Next, the smaller pieces of a page that matter just as much: titles, Open Graph tags, links and short URLs, in Metadata, titles and links.