Crawl web pages into structured data for AI agents
Give it a URL and it returns markdown text ready for AI processing, without you having to manage browser execution or proxy settings. Crawl4AI is an open-source tool that crawls web pages and converts them into a format that LLMs can easily understand. It offers two ways to use it: a hosted service via a cloud API or running it directly on your own machine.
Cloud and local usage
Crawl4AI Cloud is a hosted API that handles browser execution and proxy management on the service side. You can get a free API key to use 1,000 credits per month without registering a credit card. On the other hand, this repository contains crawler code under the Apache-2.0 license, allowing you to install it as a Python library or run a server with Docker to operate it yourself. The cloud version automatically handles pages with heavy JavaScript or bot blocking, while the local version lets you directly control all settings such as sessions, cookies, and user scripts.
LLM-friendly data extraction
The core of this tool is converting web pages into a format that LLMs can read well. Unnecessary elements like menus and footers are removed by filters, while titles, lists, tables, and code blocks are organized into markdown while preserving their structure. Links within the page are converted into a numbered reference list to make citation easy. You can also quickly extract structured JSON without an LLM using CSS selectors or XPath schemas, and connect an LLM if you need to perform more complex data extraction.
Installation and execution
To run it locally, install it in your Python environment with the pip install -U crawl4ai command, then run crawl4ai-setup to prepare the browser. When running a server with Docker, you must set the CRAWL4AI_API_TOKEN environment variable; without a token, it only responds from inside the container. To use the cloud API, put your API key in the CRAWL4AI_KEY environment variable and send the URL to the https://api.crawl4ai.com/scrape endpoint. You can connect AI agent tools like Claude Code or Cursor via an MCP server with a one-line command.
Criteria for choosing
If you do not want to spend time on browser management or proxy settings and want to use web search functionality as well, the cloud API is suitable. You can test it within the free limit of 1,000 credits per month and switch to a paid plan if needed. On the other hand, if you value data sovereignty or need specific browser settings, you should consider local execution. Local execution is free, but users must bear the infrastructure costs, and for pages with heavy JavaScript or sites with bot blocking, you must set up proxies and stealth mode yourself. Since it is under the Apache-2.0 license, commercial use is possible, but it is recommended to include a badge or citation in your project.
