PyScrappy

Open source alternative to Firecrawl, ScraperAPI and Bright Data

263 stars
MIT
Active this month
Last commit 27 days ago

Last verified Oct 20, 2026

Transform websites into LLM-ready structured data with adaptive selectors, stealth mode, and 24 built-in scrapers. Python library or MCP server for AI agents.

Visit website GitHub263

Read the full breakdownWho it is for, how it works, trade-offs, the quick start and the FAQ

who it's for

Who PyScrappy is for#

AI engineers building training datasets or RAG pipelines

PyScrappy converts any public web page to clean markdown or JSON in one API call, removing the HTML noise that degrades embedding quality. The batch crawl API lets you queue thousands of URLs and pull results asynchronously, which fits standard ETL pipelines. Built-in scrapers for Wikipedia, GitHub, news feeds, and documentation sites return pre-structured data. Use the MCP server mode to let AI agents scrape web data on demand during inference.

Skip if:

Your dataset sources are already in structured formats (RSS, sitemaps, public APIs). PyScrappy's value is extracting from unstructured HTML. If your sources do not require it, a lightweight HTTP client is simpler.

Data analysts replacing paid scraping services

Teams currently paying $300+/month for ScraperAPI or Diffbot can self-host PyScrappy and eliminate recurring API costs. The adaptive selectors reduce maintenance overhead compared to custom scripts that break on every site update. Built-in scrapers for stock data, e-commerce (Amazon, IKEA, Newegg), and news feeds handle common use cases without writing custom parsers. DataFrame output integrates directly into pandas-based analysis workflows.

Skip if:

You need entity extraction (people, companies, products) from unstructured text. Diffbot's structured entity API is more advanced than PyScrappy's text extraction. PyScrappy gives you clean markdown and JSON; you handle entity recognition separately.

Developer teams building AI agents with MCP

The MCP server exposes all 24 scrapers as tools for Claude, Cursor, or any MCP-compatible agent. An agent can call scrape_wikipedia, scrape_stock, or scrape_url naturally and receive structured data back without the developer writing scraper code. For local models (Ollama), the built-in pyscrappy chat agent lets models that support tool calling (Llama 3.1, Qwen 2.5, Mistral) use the same scrapers directly.

Skip if:

Your AI agent workflow does not need live web data at inference time. If you pre-fetch and embed all sources offline, the MCP server adds unnecessary runtime complexity. Use the Python library in your data pipeline instead.

Developers who need reliable scrapers with minimal maintenance

The adaptive selector system reduces the maintenance burden for long-lived scrapers. When a site redesigns and breaks a CSS selector, PyScrappy relocates the element by structural and textual similarity instead of returning empty. The confidence score tells you how certain the relocation was. For sites that change markup frequently, this is more reliable than hard-coded selectors that silently break.

Skip if:

You are scraping APIs or structured feeds (JSON, XML, RSS) where selectors are not needed. Adaptive selectors solve a problem that only exists when parsing unstructured HTML.

the problem

The problem it solves#

Web scraping at production scale faces three major pain points. First, brittle selectors: custom scrapers break silently every time a site updates its HTML structure, requiring constant maintenance and monitoring. Second, anti-bot blocking: many sites detect standard HTTP clients by their TLS fingerprints and serve CAPTCHA challenges or error pages before you see any content. Third, cost at scale: commercial scraping APIs like Diffbot, ScraperAPI, and ScrapeOps charge per-page fees that range from $299 to over $1,000 per month for anything beyond toy projects.

The output quality problem compounds these issues. Raw HTML from dynamic JavaScript sites requires extensive cleaning before it is usable for AI training or RAG pipelines. Most scraping libraries return unstructured markup that needs a separate parsing and normalization step. For teams building LLM-powered systems, this preprocessing overhead adds latency and fragility to the data pipeline.

how PyScrappy solves it

How it solves it#

Adaptive self-healing selectors

CSS selectors that remember an element's structural fingerprint and relocate it by similarity when a site changes its markup, so scrapers do not silently break after a redesign. The system uses weighted signals (stable id/data attributes count more than sibling order) and volatility-aware text matching (prices and dates are down-weighted). Confidence scores (0-100) tell you how certain the relocation was. Fingerprints persist in a local JSON store namespaced by site.

TLS-fingerprint impersonation (stealth mode)

Bypass anti-bot systems that block plain HTTP clients by mimicking a real browser's TLS/JA3 fingerprint. Set impersonate="chrome" to pass TLS-level detection without running a headless browser. Supports Chrome, Firefox, and Safari fingerprints via the optional curl_cffi backend. Works on the synchronous path; async clients raise a clear error if you set this option.

LLM-ready output formats

Every scrape result converts to clean markdown (.to_markdown()), structured JSON (.to_json()), or pandas DataFrame (.to_dataframe()) with one method call. Markdown output strips HTML noise and preserves semantic structure (headings, lists, tables, links) for direct embedding or LLM context. JSON output is typed and validated. No separate parsing step needed.

24 built-in scrapers for common platforms

Pre-built scrapers for Wikipedia, GitHub, IMDB, stock quotes (Yahoo Finance), news feeds (RSS/Atom), YouTube, Amazon, IKEA, Newegg, Hacker News, LinkedIn jobs, weather, crypto, currency, dictionaries, and more. Each scraper returns structured data in the same ScrapeResult format. GenericScraper handles any URL with auto-extraction of text, links, images, tables, and metadata.

MCP server for AI agents

Run PyScrappy as a Model Context Protocol server that exposes all 24 scrapers as tools for AI agents (Claude, Cursor, local LLMs via Ollama). Agents call scrape_url, scrape_wikipedia, scrape_stock, or search_github as tools and receive structured markdown or JSON back. For local models that lack MCP support, the built-in pyscrappy chat agent talks to Ollama directly and exposes the same 22 tools.

JavaScript rendering with Playwright

Optional Playwright backend for sites that require a browser to render. Scrapes React, Vue, and Angular SPAs by running headless Chromium server-side, so you get the same DOM state a real visitor would see. No Puppeteer or Playwright setup needed on the caller side. Install with pip install 'pyscrappy[browser]' and playwright install chromium.

strengths · trade-offs

Strengths and trade-offs#

Strengths

MIT license with full self-hostingThe entire toolkit is MIT licensed, so you can run it on your infrastructure, modify scrapers, and use it commercially without restrictions. Unlike Diffbot (API-only, per-page billing) or ScraperAPI (managed proxy service with usage tiers), you pay once for the server and own the pipeline. For teams scraping more than a few thousand pages per month, this eliminates recurring API costs entirely.
Self-healing selectors survive redesignsAdaptive selectors are a genuine innovation over traditional CSS or XPath scrapers. Hard-coded selectors break silently the day a site changes its markup. PyScrappy's fingerprint-based relocation uses weighted structural signals (stable ids and data attributes count more than tag order) and volatility-aware text matching (prices and dates are down-weighted) to find the element again. This reduces maintenance overhead for long-lived scrapers.
Zero external dependencies for basic scrapingThe core library works with Python 3.9+ and standard HTTP clients, with no required external services or databases. Optional extras (browser support, MCP server, stealth mode, DataFrame output) are truly optional. You can install the base package, write a scraper, and run it on any Python environment without configuring a headless browser or proxy service first.
Concurrent scraping with built-in rate limitingscrape_many and scrape_all run scrapes in parallel with exponential backoff retries and per-domain rate limiting built in. You pass a list of URLs and get back results as they complete, with failed requests automatically retried up to max_retries times. This is faster than sequential scraping and safer than naive threading without rate limits.

Trade-offs

-JavaScript rendering requires 2GB+ RAM per workerThe Playwright backend runs headless Chromium, which needs at least 2GB RAM per concurrent worker. Lightweight VPS deployments under 1GB will hit out-of-memory errors on JavaScript-heavy sites. You can run the core library (static HTML scraping) on minimal resources, but dynamic SPAs require a beefier server or cloud deployment. This is not unique to PyScrappy (all browser-based scrapers have this cost), but it is a real constraint.
-No hosted option or managed servicePyScrappy is self-hosted only. There is no official managed API or cloud service, so you provision and maintain your own infrastructure. Commercial services like Diffbot or ScraperAPI handle proxy rotation, anti-bot challenges, and infrastructure scaling for you. With PyScrappy, you handle those yourself (or use a third-party proxy service, which the library supports). This tradeoff makes sense for cost-sensitive or privacy-focused teams, but adds operational overhead.
-Some built-in scrapers need proxies for blocked sitesA few built-in scrapers (Instagram, Twitter, Spotify, eBay) are marked as needing a proxy because those sites aggressively block automated requests. PyScrappy supports proxy configuration and scraping-API services (ScraperAPI, ScrapeOps, ScrapingBee) to route through, but you pay for those separately. The README is honest about which scrapers work without a proxy and which do not.

versus alternatives

PyScrappy vs alternatives#

PyScrappy vs Diffbot

Both tools convert web pages to structured data, but they serve different deployment models and use cases. Diffbot is a managed API with enterprise pricing and advanced entity extraction. PyScrappy is open source with a self-hosted path and LLM-optimized output.

Feature	PyScrappy	Diffbot
License	MIT	Proprietary
Self-hosting	Yes	No
JavaScript rendering	Yes (Playwright)	Yes
Output formats	Markdown, JSON, DataFrame	JSON (entity-structured)
Pricing	Free self-hosted	$299+/month managed API
Entity extraction	Manual (via LLM or NLP)	Built-in (people, companies, products)

PyScrappy is the better choice when you need self-hosting for data privacy, cost control on high-volume scraping (thousands of pages per month), or the ability to modify the extraction pipeline. The markdown and JSON output formats are optimized for LLM ingestion with minimal preprocessing. Diffbot is worth considering when you need its structured entity extraction (people, articles, products) out of the box with no infrastructure management. Diffbot's entity recognition is more advanced than PyScrappy's text extraction, but you pay per API call and cannot self-host.

PyScrappy vs ScraperAPI

ScraperAPI is a managed proxy service that handles anti-bot challenges and returns raw HTML. PyScrappy is a self-hosted toolkit that scrapes and structures data, with optional proxy support.

Feature	PyScrappy	ScraperAPI
License	MIT	Proprietary
Self-hosting	Yes	No
Anti-bot handling	TLS impersonation + proxy support	Managed proxy rotation
Output	Structured (markdown, JSON)	Raw HTML
Pricing	Free self-hosted	$49-$249+/month (usage tiers)
JavaScript rendering	Yes (Playwright)	Yes

PyScrappy is the better choice when you want structured output (not raw HTML) and the ability to run scrapers on your infrastructure without per-request billing. ScraperAPI solves the proxy and anti-bot problem for you, but you still need to parse the returned HTML yourself. With PyScrappy, you handle anti-bot challenges via TLS impersonation or your own proxy (or route through ScraperAPI if you want), but you get clean markdown or JSON back instead of raw markup. For teams building AI pipelines, PyScrappy's LLM-ready output eliminates the HTML-to-text preprocessing step.

PyScrappy vs BeautifulSoup or Scrapy

BeautifulSoup and Scrapy are established Python scraping libraries. PyScrappy builds on similar patterns but adds adaptive selectors, LLM-optimized output, and MCP server support.

Feature	PyScrappy	BeautifulSoup / Scrapy
Adaptive selectors	Yes (self-healing)	No (hard-coded CSS/XPath)
LLM-ready output	Yes (markdown, JSON)	Manual (HTML parsing)
Built-in scrapers	24 (Wikipedia, GitHub, stocks, etc.)	None (you build all scrapers)
MCP server for AI agents	Yes	No
TLS-fingerprint stealth	Yes (optional)	No (requires separate proxy)

PyScrappy is the better choice when you are building AI workflows that need LLM-ready data, when you want pre-built scrapers for common platforms, or when you need adaptive selectors that survive site redesigns. BeautifulSoup and Scrapy are more mature and widely used, but they return raw parsed HTML and require you to write custom cleaning and structuring logic. For traditional scraping workflows where you want full control, Scrapy is still the standard. For AI-native workflows, PyScrappy's markdown output and MCP server integration are purpose-built advantages.

install · quick start

Quick start#

bash

PyScrappy requires Python 3.9 or newer. The core library installs via pip with no external dependencies. Optional extras add browser support (Playwright), MCP server capabilities (Python 3.10+ required), stealth mode (TLS impersonation), and DataFrame output.

```bash
# Core library (static HTML scraping, no browser)
pip install pyscrappy

# With browser support for JavaScript-rendered pages
pip install 'pyscrappy[browser]'
playwright install chromium

# With MCP server for AI agents (Python 3.10+)
pip install 'pyscrappy[mcp]'

# With stealth mode (TLS-fingerprint impersonation)
pip install 'pyscrappy[stealth]'

# Everything
pip install 'pyscrappy[all]'
```

For self-hosting in production, run the core library on any Python 3.9+ environment. If you need JavaScript rendering, provision at least 2GB RAM per concurrent worker for headless Chromium. Deploy the MCP server with pyscrappy-mcp (stdio by default) or pyscrappy-mcp --http for remote HTTP deployments. Register with Claude Code using claude mcp add pyscrappy pyscrappy-mcp. For Claude Desktop, add the server to your claude_desktop_config.json under mcpServers.

tech stack · detected from GitHub

What it's built on#

Languages: Python

frequently asked

FAQ#

Is PyScrappy free to use?

Yes. PyScrappy is MIT licensed and free to run on your own infrastructure. There is no managed cloud API or paid tier. The entire toolkit, including the MCP server and all 24 built-in scrapers, is open source. You pay for your own server or cloud compute, but there are no per-request fees or usage limits beyond what your infrastructure can handle.

Can I use PyScrappy with AI agents like Claude or Cursor?

Yes. PyScrappy ships an optional MCP (Model Context Protocol) server that exposes all scrapers as tools for AI agents. Install with pip install 'pyscrappy[mcp]' (requires Python 3.10+), run pyscrappy-mcp or python -m pyscrappy.mcp, and register it with your agent's MCP config. For Claude Desktop, add the server to claude_desktop_config.json. For Claude Code, run claude mcp add pyscrappy pyscrappy-mcp. The agent can then call scrape_url, scrape_wikipedia, scrape_stock, and 20+ other tools naturally.

How does PyScrappy handle sites that block automated requests?

PyScrappy has three anti-bot strategies. First, TLS-fingerprint impersonation (stealth mode): set impersonate="chrome" to mimic a real browser's TLS/JA3 fingerprint and bypass detection at the TLS handshake layer, before the site serves a CAPTCHA. Requires the optional stealth extra (pip install 'pyscrappy[stealth]'). Second, proxy support: route requests through a proxy or rotating proxy list. Third, scraping-API integration: configure ScraperAPI, ScrapeOps, or ScrapingBee to handle proxies and anti-bot challenges for you. Some built-in scrapers (Instagram, Twitter, Spotify) are marked as needing a proxy because those sites block aggressively.

What is the difference between PyScrappy and Diffbot or ScraperAPI?

Diffbot and ScraperAPI are managed cloud APIs with per-page billing ($299+/month for production workloads). PyScrappy is self-hosted and MIT licensed, so you run it on your infrastructure with zero per-request costs after setup. Diffbot offers advanced entity extraction (people, companies, products) from unstructured text; PyScrappy focuses on clean text/JSON/markdown extraction and leaves entity recognition to your LLM or separate NLP pipeline. ScraperAPI is a proxy service that handles anti-bot challenges; PyScrappy includes TLS impersonation and proxy support, but you manage the infrastructure. PyScrappy is the better choice when you need cost control, data privacy, or the ability to modify the extraction pipeline.

Do I need to run a headless browser for PyScrappy to work?

No, not for most sites. The core library scrapes static HTML with standard HTTP clients and no browser required. You only need the optional Playwright backend (pip install 'pyscrappy[browser]' and playwright install chromium) if you are scraping JavaScript-rendered single-page apps (React, Vue, Angular). Sites that serve full HTML on the initial request work fine without a browser. The README's GenericScraper supports both modes: it auto-detects when JavaScript rendering is needed, or you can force it with render_js=True.

also worth a look

Similar open-source tools#

Agent Reach

Featured

Alternative toFirecrawl

Give any AI agent internet access, free and local

100,165 GitHub stars
Licence MIT

crawl4ai

Alternative toFirecrawl

LLM-ready web crawling without API keys or rate limits

85,984 GitHub stars
Licence Apache-2.0

ego-lite

Alternative toBrowserbase

AI agent browser with shared Chrome state, zero cost

18,122 GitHub stars
Licence MIT

Firecrawl

Alternative toApify

Turn any website into clean markdown or structured JSON for LLMs

193,245 GitHub stars
Licence AGPL-3.0

FckSignups

InCustomer Support

Open-source tools that work instantly, no signup required

4,511 GitHub stars
Licence GPL-3.0

treg

Alternative toComposio

One token for 2,600+ agent tools across 47 providers

5,904 GitHub stars

by category

Open source tools in these categories#

196

Featured

Inbox Zero

Alternative toSuperhuman

AI email assistant that drafts replies and clears your inbox

12,712 GitHub stars
Licence AGPL-3.0
#Email

rakazo

Alternative toGrok Bot

AI teammates you own: your keys, your model, your machine.

4,904 GitHub stars
Licence Apache-2.0
#AI Agents

Omnara

Alternative toClaude Managed Agents

Open-source agent deployment API. Self-host or use Omnara Cloud.

2,957 GitHub stars
Licence Apache-2.0
#AI Agents

Feature your project

The projects we add and every new guide. Free, unsubscribe any time.

Find open source alternatives to paid software

PyScrappy vs Diffbot

Feature	PyScrappy	Diffbot
License	MIT	Proprietary
Self-hosting	Yes	No
JavaScript rendering	Yes (Playwright)	Yes
Output formats	Markdown, JSON, DataFrame	JSON (entity-structured)
Pricing	Free self-hosted	$299+/month managed API
Entity extraction	Manual (via LLM or NLP)	Built-in (people, companies, products)

PyScrappy vs ScraperAPI

ScraperAPI is a managed proxy service that handles anti-bot challenges and returns raw HTML. PyScrappy is a self-hosted toolkit that scrapes and structures data, with optional proxy support.

Feature	PyScrappy	ScraperAPI
License	MIT	Proprietary
Self-hosting	Yes	No
Anti-bot handling	TLS impersonation + proxy support	Managed proxy rotation
Output	Structured (markdown, JSON)	Raw HTML
Pricing	Free self-hosted	$49-$249+/month (usage tiers)
JavaScript rendering	Yes (Playwright)	Yes

PyScrappy vs BeautifulSoup or Scrapy

BeautifulSoup and Scrapy are established Python scraping libraries. PyScrappy builds on similar patterns but adds adaptive selectors, LLM-optimized output, and MCP server support.

Feature	PyScrappy	BeautifulSoup / Scrapy
Adaptive selectors	Yes (self-healing)	No (hard-coded CSS/XPath)
LLM-ready output	Yes (markdown, JSON)	Manual (HTML parsing)
Built-in scrapers	24 (Wikipedia, GitHub, stocks, etc.)	None (you build all scrapers)
MCP server for AI agents	Yes	No
TLS-fingerprint stealth	Yes (optional)	No (requires separate proxy)

Who PyScrappy is for#

AI engineers building training datasets or RAG pipelines

Skip if:

Data analysts replacing paid scraping services

Skip if:

Developer teams building AI agents with MCP

Skip if:

Developers who need reliable scrapers with minimal maintenance

Skip if:

The problem it solves#

How it solves it#

Adaptive self-healing selectors

TLS-fingerprint impersonation (stealth mode)

LLM-ready output formats

24 built-in scrapers for common platforms

MCP server for AI agents

JavaScript rendering with Playwright

Strengths and trade-offs#

Strengths

Trade-offs

PyScrappy vs alternatives#

PyScrappy vs Diffbot

PyScrappy vs ScraperAPI

PyScrappy vs BeautifulSoup or Scrapy

Quick start#

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

Web Development

AI & Machine Learning

Data & Analytics

Find open source alternatives to paid software

Who PyScrappy is for#

AI engineers building training datasets or RAG pipelines

Skip if:

Data analysts replacing paid scraping services

Skip if:

Developer teams building AI agents with MCP

Skip if:

Developers who need reliable scrapers with minimal maintenance

Skip if:

The problem it solves#

How it solves it#

Adaptive self-healing selectors

TLS-fingerprint impersonation (stealth mode)

LLM-ready output formats

24 built-in scrapers for common platforms

MCP server for AI agents

JavaScript rendering with Playwright

Strengths and trade-offs#

Strengths

Trade-offs

PyScrappy vs alternatives#

PyScrappy vs Diffbot

PyScrappy vs ScraperAPI

PyScrappy vs BeautifulSoup or Scrapy

Quick start#

What it's built on#

FAQ#

Similar open-source tools#

Open source tools in these categories#

Web Development

AI & Machine Learning

Data & Analytics

Find open source alternatives to paid software