Collect web data from simple requests to large-scale crawling with a single Python framework
Imagine extracting data from complex web pages without launching a browser, or instructing an AI agent to navigate sites directly. Scrapling is a Python-based web scraping framework that handles static pages, JavaScript-rendered sites, and the underlying API calls in one go. With this tool, you can convert entire websites into clean Markdown files or extract structured data via terminal commands or AI chats, without writing manual code.
Flexible data collection with or without a browser
Scrapling supports a wide range of collection methods, from simple HTTP requests to headless browser automation. The Fetcher class mimics browser TLS fingerprints and headers to send fast HTTP requests, while sites requiring JavaScript are rendered using Playwright-based Chromium or Google Chrome via StealthyFetcher. It includes built-in features to bypass bot-blocking barriers like Cloudflare Turnstile or interstitials, allowing data retrieval from sites that typically block standard scrapers. Session management maintains cookies and login states for consecutive requests, and proxy rotation automatically changes IP addresses to reduce blocking risks.
Interfaces for AI agents and the terminal
A key feature of this repository is its tight integration with AI. Through an MCP server, AI agents like Claude or Cursor can directly call Scrapling to explore web pages, take screenshots, and extract data while maintaining login states. It also supports resolving captchas or bot-blocking issues encountered by AI using Stealthy browsers and Cloudflare solvers. An 'Agent Skill' is provided to help coding agents understand the latest API accurately, ensuring that AI-generated code matches the current library version. You can also run the scrapling extract command in the terminal to extract page content using specific CSS selectors or convert it into LLM-friendly Markdown using the --ai-targeted option, without writing any code.
Structure and templates for crawler development
For collecting large-scale websites, you can utilize the Spider API. Similar in structure to Scrapy, it defines start_urls and asynchronous parse callbacks, allowing you to set domain-specific throttling and concurrency limits. If you need to stop crawling, the crawldir option saves progress, enabling you to restart later in the same directory to resume from where you left off. Pre-made templates like CrawlSpider, SitemapSpider, and ShopifySpider reduce repetitive tasks, allowing you to collect entire catalogs from specific site types (e.g., Shopify stores) with minimal configuration. Collected data can be exported directly in JSON, CSV, or XML formats, or processed in real-time through pipelines.
Design for performance and stability
Scrapling boasts fast processing speeds among Python scraping libraries. This is based on the average of over 100 runs, maintaining minimal memory usage through memory-efficient data structures and lazy loading. It also features 92% test coverage and full type hint coverage, ensuring that IDE code completion and static type checking (PyRight, MyPy) work perfectly. For development convenience, an IPython-based interactive shell is provided, which can convert curl requests into Scrapling code or display results directly in the browser, speeding up script development.
Things to check before use
Scrapling requires Python 3.10 or higher, and the default installation (pip install scrapling) includes only the parser engine. If you want the MCP server or shell features, you must install them with the [ai] or [shell] options, respectively, or install [all] for all features. This library is provided for educational and research purposes, and users must comply with local and international data scraping laws, website terms of service, and robots.txt files. The license is BSD-3-Clause, and you can also use a Docker image to immediately access an environment with all browsers and dependencies included.
