get data
en
~/utility/rag-web-crawler

RAG Web Crawler — Web Pages to Markdown & Text Chunks

Crawl public web pages into clean Markdown, plain text, and overlapping text chunks with source URLs and character offsets, ready to embed for RAG.

RAG data TypeScriptMozilla ReadabilityjsdomTurndown Global
run on Apify
proooxy/rag-web-crawler — spec
categoryutility / RAG data
languageTypeScript
stackTypeScript, Mozilla Readability, jsdom, Turndown
marketsGlobal
outputclean, RAG-ready JSON

key features

Clean Markdown and plain text from each page, with headings, lists, links, code blocks, and basic tables

Ready-to-embed chunks that overlap and end at paragraph or sentence breaks, each with a stable id, its source URL and title, and exact character offsets for citations

Main content first: menus, headers, footers, and sidebars are left out, and CSS selectors remove anything else you name

Link following within each site, up to five links deep, with every page fetched once

Search instead of URLs: each result page of a query is crawled like a start URL (needs a paid Apify plan)

robots.txt respected by default, with polite pacing between requests to the same site

Pages kept in their own language — English, Chinese, and others — without translation

A status and an error code for every page, and a summary for every run with counts and a stop reason

use cases

  • Building a RAG chatbot's knowledge base from documentation and help-center pages
  • Grounding AI agents in current web pages they can cite
  • Turning a list of articles into embeddings for semantic search
  • Collecting the pages a search finds on a topic, in their original language
  • Refreshing a vector store on a schedule, with chunk ids that stay the same while a page does not change
  • Giving AI agents web pages as Markdown through Apify's MCP server

input parameters

ParameterTypeRequiredDescription
startUrlsarrayoptionalPublic page URLs to crawl, up to 1,000
querystringoptionalA search query whose result pages are crawled (needs a paid Apify plan)
queriesarrayoptionalMore search queries, merged with query
maxResultsPerQuerynumberoptionalSearch results to crawl per query (default 5, at most 50)
maxPagesnumberoptionalMost distinct pages a run schedules (default 20, at most 1,000)
maxDepthnumberoptionalHow many links deep to follow within each site (default 0 = start URLs only, at most 5)
maxConcurrencynumberoptionalPages processed at the same time (default 3)
maxSearchRequestsnumberoptionalSearch requests per run, retries included (default 10, 0 = no search)
maxFetchRequestsnumberoptionalAll network attempts per run, robots.txt and retries included (default 100)
requestTimeoutSecsnumberoptionalTimeout for each request, in seconds (default 30)
maxResponseBytesnumberoptionalLargest page response accepted (default 2 MiB)
maxContentCharsnumberoptionalLongest text kept per page, in characters (default 50,000)
maxOutputBytesnumberoptionalTotal output per run (default 20 MiB)
respectRobotsTxtbooleanoptionalCheck each site's robots.txt first (default on)
onlyMainContentbooleanoptionalKeep only the page's main content (default on)
removeSelectorsarrayoptionalCSS selectors of elements to remove before extraction
chunkSizenumberoptionalChunk length in characters, not tokens (default 1,000)
chunkOverlapnumberoptionalCharacters shared by consecutive chunks (default 150)

Output Example

One saved page from a real run, with the text shortened and one of its five chunks shown:

 1{
 2  "schemaVersion": "1.0",
 3  "url": "https://www.w3.org/WAI/tutorials/tables/one-header/",
 4  "title": "Tables with One Header",
 5  "description": "Accessibility resources free online from the international standards organization: W3C Web Accessibility Initiative (WAI).",
 6  "languageCode": "en",
 7  "canonicalUrl": "https://www.w3.org/WAI/tutorials/tables/one-header/",
 8  "markdown": "# Tables with One Header\n\nin [Tables Tutorial](https://www.w3.org/WAI/tutorials/tables/)\n\n## Overview\n\n…",
 9  "text": "Tables with One Header\n\nin Tables Tutorial\n\nOverview\n\nThis page covers tables that have one simple header for rows or one simple header for columns. …",
10  "chunks": [
11    {
12      "id": "495cb4e58e13bbf9b5cdbe922f7ea652973ca4dfda61b952aaa021db045a450c",
13      "index": 0,
14      "text": "Tables with One Header\n\nin Tables Tutorial\n\nOverview\n\nThis page covers tables that have one simple header for rows or one simple header for columns. …",
15      "startChar": 0,
16      "endChar": 988,
17      "sourceUrl": "https://www.w3.org/WAI/tutorials/tables/one-header/",
18      "title": "Tables with One Header"
19    }
20  ],
21  "contentTruncated": false,
22  "sources": [],
23  "crawl": {
24    "depth": 0,
25    "httpStatusCode": 200,
26    "loadedAt": "2026-10-19T07:03:53.643Z",
27    "contentType": "text/html",
28    "renderingMode": "http",
29    "extractionMethod": "main"
30  }
31}

Chunk offsets count Unicode code points in text. Each run also writes REQUESTS, an outcome for every page with a status and an error code, and OUTPUT, the run summary with counts, the stop reason, and the events charged.

Tips

  • Use id as the record ID in your vector store. It stays the same across runs while the page and your chunk settings don’t change, so writing chunks by id on a scheduled run replaces them instead of adding copies.
  • Start with a few pages. Check the Markdown of your target pages before raising maxPages or maxDepth, and drop leftovers such as related-link boxes with removeSelectors.
  • Read REQUESTS alongside the Dataset. A run can succeed while some pages were skipped or failed, and each one says why.

faq

How is it different from a general web scraper?
You give it URLs or a search query, not selectors. It keeps each page's main content as Markdown and plain text and cuts the text into chunks that are ready to embed, each with the page's URL, title, and exact character offsets, so an answer can cite where it came from.
Do I need an API key?
No external API key, for crawling or for search. You need an Apify account; calling the Actor through Apify's API uses your Apify token.
Does it render JavaScript?
No. It reads pages over HTTP, so a page that needs JavaScript to show its text can fail with NO_CONTENT or come back incomplete. Try a few of your target pages first.
Which pages can it read?
Public HTML, Markdown, and plain-text pages. Logins, cookies, paywalls, CAPTCHAs, proxies, PDFs, and images are out of scope, and private-network addresses and URLs that carry credentials are rejected.
Is crawling websites allowed?
Crawling publicly available, non-personal data is generally legal, but page content is often copyrighted and some sites restrict crawling in their terms. The crawler follows robots.txt by default; you are responsible for how you use what you collect.

related in ~/utility

Run RAG Web Crawler — Web Pages to Markdown & Text Chunks, or get a custom build

Start extracting on Apify in minutes, or hire me to build a bespoke scraper and RAG pipeline for your exact source and schema.

run on Apify get custom data