Build Data Pipelines
Build web data jobs that run on their own: a crawler that respects robots.txt, text cleaning, page change alerts, and a pipeline that restarts safely and tracks its costs.
Build Data Pipelines badgeComplete all 4 lessons to earn it. About 2.5 hours.
0 of 4 lessonsStart course
What you'll be able to do
- Crawl within robots.txt, crawl-delay and per-host rate limits, with caching and a small concurrency queue
- Turn extracted text into records with a trusted language, keywords, a summary and a search key
- Monitor pages for changes with normalisation, hashing and confirmation that keep false alarms out
- Build an idempotent feed-to-store pipeline that resumes after a crash and stops when credits run out
- Account for every credit a run spends and reconcile it against your balance
This course is one of three skill areas on exam APV-210, APIVerve Certified Developer: Web Data: 10 of its 30 questions. See the full path
Syllabus
- Crawl websites politely and legallyStart hereBuild a polite web crawler in Node.js that obeys robots.txt and crawl-delay, paces each host, caches results and keeps concurrency low with a small queue.
- Clean and structure scraped textTurn scraped text into structured records with language detection, keyword extraction, summaries and punctuation-free search keys, plus the traps in each step.
- Monitor web pages for changesBuild a website change monitor in Node.js: fetch pages as Markdown, normalise away dates and counters, hash, diff and alert only on changes that persist.
- Build a content pipeline from RSS feeds and sitemapsBuild a content pipeline in Node.js: RSS feed or sitemap to extracted, enriched records, with retries, idempotent re-runs and credit accounting.