Web Scraping Best Practices in 2026: A Practitioner's Guide
These are the practices that have mattered most while building and maintaining the production scrapers in my catalog.
Architecture: Think in Pipelines, Not Scripts
The biggest mistake I see is treating scraping as a single-step process. Production scrapers are data pipelines:
- URL Discovery — find what to scrape (sitemaps, category pages, search, APIs)
- Request Execution — fetch the data with proper retry and rotation
- Parsing — extract structured fields from raw responses
- Normalization — clean, validate, and standardize the output
- Storage — push to datasets, databases, or downstream systems
Each step should be independently testable and retryable. When Sephora changes their product page layout, only step 3 needs updating — the rest of the pipeline stays stable.
Always Prefer APIs Over HTML Parsing
Before writing a single CSS selector, check if the site has:
- Public APIs — documented endpoints that return JSON
- Private APIs — XHR/fetch calls visible in browser DevTools
- GraphQL endpoints — increasingly common, often with introspection enabled
- Embedded JSON —
__NEXT_DATA__,window.__INITIAL_STATE__, or JSON-LD in the HTML
API responses are structured, versioned, and far more stable than HTML layouts.
Proxy Strategy: Match the Protection
Not every site needs residential proxies. Here’s my decision framework:
| Protection Level | Proxy Type | Example Sites |
|---|---|---|
| None / Basic | Datacenter | Small sites, many public APIs |
| Rate limiting | Rotating datacenter | Medium e-commerce, content sites |
| Fingerprinting | Residential | Major brands |
| Advanced WAF | Residential + TLS fingerprint | Akamai, Cloudflare Enterprise |
The key insight: proxy cost scales with protection level. Don’t pay for residential proxies on sites that only rate-limit. My Shopify scraper defaults to Apify’s datacenter proxy, but many stores block datacenter ranges, so residential proxies are its recommended setting.
Session Management Is Everything
The difference between a 60% and 99% success rate is usually session management:
- Rotate sessions, not just IPs — a new IP with the same cookies looks suspicious
- Warm up sessions — visit the homepage before hitting product pages
- Respect rate limits — 5 concurrent requests beats 50 that get blocked
- Exponential backoff — 1s, 2s, 4s, 8s retries, not immediate retries
A robust scraper manages session tokens with automatic refresh and exponential backoff, keeping persistent sessions that look like real browsing patterns.
Normalize Your Output
Raw scraped data is messy. Normalize everything:
Prices
Store as integers (cents, not dollars). $29.99 becomes 2999. This avoids floating-point precision errors that corrupt financial data downstream. My e-commerce scrapers follow this convention, with one exception: the Sephora scraper keeps decimal prices (72.0).
URLs
Always store absolute URLs, never relative paths. Resolve them at extraction time.
Dates
ISO 8601 (2026-04-01T00:00:00Z), always with timezone. Never store locale-formatted dates.
Text
Strip excess whitespace, normalize Unicode, and decide on an HTML handling policy (strip tags vs. preserve formatting).
Error Handling: Expect Failure
Production scrapers fail constantly — the question is how gracefully. My approach:
1Request fails (network error, timeout, 4xx/5xx)
2 → Retry with exponential backoff (up to 5 attempts)
3 → Rotate session/proxy on retry
4 → Log failure with full context if all retries exhausted
5 → Continue processing remaining URLs (don't crash the batch)
Track success rates per URL pattern. If /category/* pages suddenly drop below 90% success, the site probably changed something — you’ll catch it before users report it.
Monitor and Alert
A scraper without monitoring is a scraper waiting to silently fail. Track:
- Success rate per run and per URL pattern
- Output count — sudden drops mean something broke
- Data quality — null fields, unexpected values, schema violations
- Cost — proxy usage, compute time, storage
My Apify actors all expose these metrics. When success rates dip, I get notified within hours — often before any user notices.
Start Simple, Add Complexity
Every scraper I build starts as the simplest thing that works:
- HTTP + Cheerio first (fastest, cheapest)
- Add fingerprinting only if blocked
- Add browser rendering only if JavaScript is required
- Add proxy rotation only if rate-limited
A scraper that reads structured data over plain HTTP needs no browser. My Universal Web Printer uses Playwright because it must render JavaScript. Right tool for the job.
These aren’t theoretical principles — they’re extracted from running production scrapers. If you need a custom scraper built with these practices, let’s talk.