Best Open Source Scraping Tools in 2026, Compared by Job

For most teams, the best open source scraping tools are Scrapy or Crawlee for crawling many pages, Beautiful Soup or Cheerio for parsing HTML, Playwright for JavaScript-heavy pages, and Crawl4AI or trafilatura when you need clean Markdown for LLMs. Pick by the job, the license, and the infrastructure you are willing to run.

The table is a shortlist grouped by category, not a ranking.

lxml, selectolax, ScrapeGraphAI, and Apache StormCrawler are covered in their sections below.

An open source scraping tool publishes its core code under a license approved by the Open Source Initiative, such as MIT, BSD, Apache 2.0, or AGPL-3.0, so anyone can run, study, change, and share it. Source-available licenses such as SSPL or the Business Source License do not qualify, and neither do cloud-only features.

The reference point is the OSI's Open Source Definition. Public code on GitHub is not enough. The license text decides what you can do.

The SPDX License List flags which licenses the OSI has approved. Scraping tools fall into four groups:

AGPL-3.0 is OSI-approved and open source, but it is copyleft, not permissive. Its network clause matters most. If you modify an AGPL tool and let others use it over a network, for example inside a hosted product, you must offer them the modified source.

Source-available licenses look open because the code is public. The Server Side Public License is the common example. browserless, a Docker service for headless browsers, is licensed "SSPL-1.0 OR Browserless Commercial License". Its LICENSE file says closed-source commercial use needs the commercial license. That is not open source, so it is not on our shortlist.

Crawl4AI's LICENSE file is the Apache 2.0 text followed by an added "Attribution Requirement" section. It asks distributions, publications, and public uses, including blog posts, to show a specific attribution line. We describe it as "Apache 2.0 with an added attribution requirement" and don't rule on how that affects OSI approval.

Always read the LICENSE file itself. The GitHub API reports Heritrix as "NOASSERTION", yet its LICENSE file is the Apache 2.0 notice. This is not legal advice. Check with your legal team before you redistribute a tool or offer it as a hosted service.

Several projects are "open core": the repository is open source, and a company sells a hosted version with more on top.

• Firecrawl: its README says the cloud version includes additional features.
• Crawl4AI: the repository also powers Crawl4AI Cloud.
• ScrapeGraphAI and browser-use: MIT code, plus paid hosted offerings.
• Crawlee: Apache-2.0, maintained by Apify, a commercial platform. The library runs without an Apify account.

This guide evaluates only the open part of each project.

We included tools whose core code is released under an OSI-approved license, or one with added terms that we label, plus a release or commit in the last 12 months, a self-hostable core, and a documented install. We checked licenses and versions on November 6, 2026, tested the Python tools on the same sandbox sites, and ignored GitHub stars.

Every tool on the shortlist had to meet five criteria:

1. License. The core repository uses an OSI-approved license per its LICENSE file or, where noted, an OSI license with added terms. Labeled, not judged: Crawl4AI (added attribution requirement) and Firecrawl (AGPL-3.0 core, some features cloud-only). For open-core projects, only the open part counts. SSPL and BUSL are out.
2. Maintained. The repository is not archived, and it has a release or a commit in the last 12 months.
3. Self-hostable. The main function runs on your own machine without a vendor account. Tools that need an LLM API key are labeled.
4. Does a scraping job. It fits one of five categories (parser, scraping framework, browser automation, LLM-ready extraction, or web-scale crawler) and has a documented install.
5. Primary evidence. Versions and licenses come from repositories and package registries. We ran the Python tools on the practice sites at toscrape.com and label tools described from documentation only.

Some well-known tools did not make the shortlist:

Start from the job. Use a parser when the data is in the HTML, a framework when you need to crawl many pages, a browser automation tool when JavaScript builds the page, and an LLM-ready crawler when the output feeds a model. Then check the license and the infrastructure you must run.

As a flowchart, four questions cover most cases:

1. Is the data in the raw HTML? If yes, start with a parser.
2. More than a handful of pages? If yes, use Scrapy or Crawlee instead of a homemade loop.
3. Does JavaScript build the content? For some pages, add Playwright through your framework. For all, use Playwright or Crawlee's browser crawler.
4. Does the output feed an LLM? If yes, try Crawl4AI or trafilatura.

Then one license check: will you ship a modified version as a hosted service? If so, review AGPL obligations and added terms such as Crawl4AI's attribution clause.

• Python: the widest choice in every category, from Beautiful Soup and Scrapy to Playwright, Crawl4AI, and trafilatura.
• Node.js: Crawlee, Cheerio, Puppeteer, and Playwright. See our guide to Node.js scraping libraries.
• Go: Colly for crawling, goquery (BSD-3-Clause, v1.13.0, released August 27, 2026) for parsing, and Katana for URL discovery.
• Java: jsoup (MIT, 1.24.1, released November 3, 2026) for parsing, plus Apache Nutch, Heritrix, and Apache StormCrawler for large crawls.
• Ruby: Nokogiri (MIT, 1.19.7, released November 2, 2026) for parsing.

Each category brings its own infrastructure:

If Scrapy looks like the right category but the wrong tool for your team, see our list of alternatives if Scrapy is the wrong fit.

For static pages, Beautiful Soup with lxml is the easiest choice in Python, parsel and selectolax are faster selector-first options, and Cheerio is the standard in Node.js. Parsers do not fetch pages, follow links, or run JavaScript, so you pair them with an HTTP client or a framework.

If you are unsure whether you need a library or full framework, start with a parser and move up once you are writing queues and retries by hand.

Beautiful Soup (MIT, 4.15.0, released June 7, 2026) is the most forgiving way to read messy HTML in Python. It wraps an underlying parser, usually lxml (BSD-3-Clause, 6.1.3, released September 2, 2026). lxml is built on C libraries and also works on its own when you want XPath.

parsel (BSD-3-Clause, 1.12.1, released September 28, 2026) is Scrapy's selector library, available on its own, so selectors you write now carry over to Scrapy later.

selectolax (MIT, 1.0.2, released October 22, 2026) is a small CSS-selector parser. Version 1.0 removed the Modest backend, so from selectolax.parser import HTMLParser now raises an ImportError. Import the Lexbor backend instead, as below.

This sample parses the same 20 titles and prices from one books.toscrape.com page with three libraries:

# Sample: three parsers, one static page
import httpx
from bs4 import BeautifulSoup
from parsel import Selector
from selectolax.lexbor import LexborHTMLParser  # selectolax 1.0+

url = "https://books.toscrape.com/catalogue/page-1.html"
html = httpx.get(url, timeout=30).text

soup = [(a.h3.a["title"], a.select_one(".price_color").text)
        for a in BeautifulSoup(html, "lxml").select("article.product_pod")]

lex = [(n.css_first("h3 a").attributes["title"], n.css_first(".price_color").text())
       for n in LexborHTMLParser(html).css("article.product_pod")]

sel = Selector(text=html)
par = list(zip(sel.css("article.product_pod h3 a::attr(title)").getall(),
               sel.css("article.product_pod .price_color::text").getall()))

print(len(soup), len(lex), len(par), soup == lex == par)
print(soup[0])

Sample output:

20 20 20 True
('A Light in the Attic', '£51.77')

All three return the same 20 records, so choose by API style and the rest of your stack.

In Node.js, Cheerio (MIT, 1.2.0, released January 23, 2026) gives you a jQuery-style API over parsed HTML, and Crawlee's HTTP crawler builds on it. jsoup (Java), Nokogiri (Ruby), and goquery (Go) fill the same role. None of them run JavaScript.

Scrapy is the most complete Python framework for crawling many pages, with scheduling, retries, pipelines, and feed exports built in. Crawlee offers similar crawling in Python and Node.js and can switch between HTTP and browser crawlers. Colly is the lightweight choice for Go teams that want a single binary.

A framework handles what scripts get wrong first. That means queues, deduplication, retries, concurrency, politeness, and export.

Scrapy (BSD-3-Clause, 2.20.0, released November 6, 2026) schedules requests, follows links, retries failures, runs item pipelines, and writes feeds in JSON, JSON Lines, CSV, and XML. The Scrapy documentation covers each feature.

Scrapy does not render JavaScript on its own, but it can. The scrapy-playwright plugin (BSD-3-Clause, 0.0.48, released July 10, 2026) hands chosen requests to Playwright. Our tutorial shows how to render JavaScript in Scrapy with Playwright.

This sample spider follows "next" through all 50 catalogue pages of books.toscrape.com, obeys robots.txt, waits between requests, and exports JSON Lines:

# Sample: Scrapy spider with pagination and a JSON Lines feed
import scrapy

class BooksSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://books.toscrape.com/catalogue/page-1.html"]
    custom_settings = {
        "ROBOTSTXT_OBEY": True,
        "DOWNLOAD_DELAY": 0.25,
        "CONCURRENT_REQUESTS": 4,
        "USER_AGENT": "my-catalog-bot/1.0 (+https://example.com/bot)",
        "FEEDS": {"books.jsonl": {"format": "jsonlines", "overwrite": True}},
    }

    def parse(self, response):
        for book in response.css("article.product_pod"):
            yield {
                "title": book.css("h3 a::attr(title)").get(),
                "price": book.css(".price_color::text").get(),
                "url": response.urljoin(book.css("h3 a::attr(href)").get()),
            }
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Run it with scrapy runspider books.py. The crawl visits 50 pages and stores 1,000 items, one JSON object per line:

{"title": "A Light in the Attic", "price": "\u00a351.77", "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"}

Crawlee is maintained by Apify and released under Apache-2.0. The JavaScript and TypeScript version is 3.18.2 (released September 29, 2026), and the Python version is 1.11.1 (released November 5, 2026). HTTP and browser crawlers share one interface, so you can move a crawl from plain requests to Playwright without restructuring it. Results go to local datasets.

This sample runs one pagination crawl twice: BeautifulSoupCrawler on quotes.toscrape.com, and PlaywrightCrawler on its JavaScript version.

# Sample: Crawlee for Python, HTTP crawler vs. browser crawler
import asyncio
from crawlee.crawlers import (BeautifulSoupCrawler, BeautifulSoupCrawlingContext,
                              PlaywrightCrawler, PlaywrightCrawlingContext)

async def main():
    static, rendered = [], []

    http = BeautifulSoupCrawler(max_requests_per_crawl=20)

    @http.router.default_handler
    async def on_page(ctx: BeautifulSoupCrawlingContext):
        static.extend(q.get_text() for q in ctx.soup.select("div.quote span.text"))
        await ctx.enqueue_links(selector="li.next a")

    browser = PlaywrightCrawler(max_requests_per_crawl=20, headless=True)

    @browser.router.default_handler
    async def on_js_page(ctx: PlaywrightCrawlingContext):
        await ctx.page.wait_for_selector("div.quote")
        for q in await ctx.page.query_selector_all("div.quote span.text"):
            rendered.append(await q.inner_text())
        await ctx.enqueue_links(selector="li.next a")

    await http.run(["https://quotes.toscrape.com/"])
    await browser.run(["https://quotes.toscrape.com/js/"])
    print(len(static), len(rendered), set(static) == set(rendered))

asyncio.run(main())

Sample output (last line):

100 100 True

Both collect the same 100 quotes with nearly identical handlers, so you pay for a browser only where you need one.

Colly (Apache-2.0) is the Go option: a single binary with callbacks, rate limits, and caching, but no JavaScript rendering. It is actively developed. Its latest release, v2.2.0, is from March 2025. We describe Colly from its documentation.

Playwright is the most capable open source tool for JavaScript-heavy pages, with Chromium, Firefox, and WebKit support in several languages. Puppeteer suits Node.js teams on Chrome, and Selenium fits teams already invested in WebDriver. Scrapy and Crawlee can hand individual requests to Playwright when only some pages need a browser.

These tools start a real browser, often a headless browser, and let your code load pages, click, scroll, and read the rendered DOM. Crawling logic stays yours, or your framework's.

First check whether you need one. This sample loads quotes.toscrape.com/js/ with an HTTP client and with Playwright:

# Sample: do you need a browser for this page?
import httpx
from bs4 import BeautifulSoup
from playwright.sync_api import sync_playwright

url = "https://quotes.toscrape.com/js/"

raw = BeautifulSoup(httpx.get(url, timeout=30).text, "lxml")
print("HTTP client + parser:", len(raw.select("div.quote")))

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto(url)
    print("Playwright:", len(page.query_selector_all("div.quote")))
    browser.close()

Sample output:

HTTP client + parser: 0
Playwright: 10

A script builds the quotes in the browser. Run this check on your own target. If both numbers match, skip the browser.

Playwright (Apache-2.0) is maintained by Microsoft: npm 1.65.0 (released November 5, 2026) and Python 1.64.1 (released October 22, 2026), plus Java and .NET. It drives Chromium, Firefox, and WebKit through one API, waits for elements automatically, and can intercept network requests. The Playwright documentation covers setup.

Puppeteer (Apache-2.0, 25.16.0, released October 22, 2026) controls Chrome and Firefox from Node.js. Crawlee for Node.js can run on Puppeteer or Playwright.

Selenium (Apache-2.0, 4.52.0, released October 29, 2026) automates browsers through WebDriver in many languages, and Selenium Grid spreads them across machines. Teams that already test with Selenium can reuse those skills. We describe Puppeteer and Selenium from their documentation.

Every page is a full browser session, so memory and CPU grow with concurrency. Two newer AGPL-3.0 projects are worth knowing: nodriver, a Python library that controls Chrome over the DevTools Protocol, and Lightpanda, a young headless browser engine that Playwright and Puppeteer can connect to.

Crawl4AI and a self-hosted Firecrawl turn web pages into Markdown or structured JSON for LLM pipelines, while trafilatura extracts the main text without any model. ScrapeGraphAI uses an LLM to fill a schema from a prompt. Check licenses carefully here: Firecrawl's core is AGPL-3.0, and Crawl4AI adds an attribution requirement.

For hosted options, see our guide to commercial AI web scraping tools.

Crawl4AI (0.9.5, released October 14, 2026) renders pages with Playwright and returns Markdown, cleaned HTML, or JSON, with optional schema or LLM extraction. It runs as a library or a Docker server. Its license is Apache 2.0 with an added attribution requirement.

This sample turns one book page into Markdown without calling any model:

# Sample: Crawl4AI page to Markdown
import asyncio
from crawl4ai import AsyncWebCrawler

url = "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url=url)
        markdown = str(result.markdown or "")
        print(result.success, "A Light in the Attic" in markdown)
        print(markdown[:300])

asyncio.run(main())

Sample output (first lines):

True True
[Books to Scrape](https://books.toscrape.com/index.html) We love being scraped!
  * [Home](https://books.toscrape.com/index.html)
  * [Books](https://books.toscrape.com/catalogue/category/books_1/index.html)

The Markdown contains the book title plus navigation links, so add content filtering before chunking for RAG.

Firecrawl's core is AGPL-3.0. Its SDKs and some UI components are MIT. The latest GitHub release is v2.11.0 (June 19, 2026). Its scrape and crawl endpoints return Markdown, HTML, links, screenshots, or JSON.

Self-hosting is a real deployment. Docker Compose runs the API and workers, Playwright, Redis, RabbitMQ, and a PostgreSQL-backed queue, and you own backups and security for each. The cloud version has additional features, so confirm what you need is in the open code. We describe Firecrawl from its documentation.

ScrapeGraphAI (MIT, 2.3.1, released October 6, 2026) builds pipelines around a language model. You describe the data in a prompt or schema, and the model fills it. Every extraction is a model call, so check its documentation for supported providers. We describe it from its documentation.

trafilatura (Apache-2.0, 2.3.1, released October 6, 2026) extracts main text and metadata without a model, outputs text, Markdown, JSON, XML, CSV, or HTML, and discovers URLs from sitemaps and feeds:

# Sample: trafilatura main-content extraction to Markdown and JSON
import json
import httpx
import trafilatura

url = "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
html = httpx.get(url, timeout=30).text  # any HTTP client works; trafilatura extracts

markdown = trafilatura.extract(html, output_format="markdown",
                               with_metadata=True, favor_recall=True)
record = json.loads(trafilatura.extract(html, output_format="json",
                                        with_metadata=True, favor_recall=True))

print(markdown[:80])
print(record["title"], len(record))

Sample output:

---
title: A Light in the Attic
description: It's hard to imagine a world withou
A Light in the Attic 19

The Markdown opens with a metadata block, and the JSON record has 19 fields. browser-use (MIT) is an agent that lets an LLM drive a browser. It needs an LLM API key and suits tasks more than bulk extraction.

Apache Nutch, Heritrix, and Apache StormCrawler are built for very large crawls rather than single-site scraping. Nutch runs batch crawls on Hadoop, Heritrix writes archival WARC files for web archives, and StormCrawler runs continuous crawls on Apache Storm. All three expect Java skills and dedicated infrastructure.

Apache Nutch (Apache-2.0) runs batch crawls on Hadoop and indexes into Solr or Elasticsearch through plugins. It is actively developed. Its latest release is 1.23, from July 2025. Nutch vs. Scrapy is a different job size. Use Scrapy to extract fields from known sites, and Nutch to crawl and index a very large set of hosts.

Heritrix (Apache-2.0, 3.18.0, released October 6, 2026) is the Internet Archive's crawler. It writes WARC files, defined by the WARC specification, runs on the JVM with a web interface, and handles robots.txt. Choose it for preservation, not structured fields.

Apache StormCrawler (Apache-2.0, 3.7.0, released July 29, 2026) provides components for continuous crawling on Apache Storm 3.1.0 with Java 25 or newer, plus OpenSearch and Solr modules. It suits teams already running stream processing.

We describe all three from their documentation. For URL discovery only, Katana (MIT, security-focused, optional headless mode) and GoSpider are lighter.

The license is free, but the operation is not. You pay for servers, browser memory, storage, proxies, and the engineering time to fix selectors when sites change. Browser-based and LLM-based tools cost the most to run, because each page needs a full browser or a model call.

Budget for these before you commit:

• Hosting and scheduling: run crawls on time, restart failures, alert on stops.
• Browsers: each session needs memory and CPU, multiplied by concurrency.
• Queues and databases: self-hosted Firecrawl brings Redis, RabbitMQ, and PostgreSQL.
• LLM calls: ScrapeGraphAI, browser-use, and Crawl4AI's LLM modes pay per model call.
• Proxies: no tool here includes a proxy network. You supply and rotate them.
• Maintenance: someone must notice and fix broken selectors.
• License compliance: AGPL obligations and Crawl4AI's attribution.

Use a scraping API when your open source stack works on test sites but fails on real targets because of blocking, CAPTCHAs, or JavaScript rendering at volume, or when maintaining proxies costs more than the data is worth. You can keep Scrapy or your parser and replace only the fetching layer.

You don't have to choose one. Selectors, pipelines, and storage work well as open source. The fetching layer, with its proxies, retries, and browsers, is where operating cost sits. Scraper API takes a URL and returns the page body, a JSON envelope, or selected fields. It operates proxy routing, rotation, and retries behind each request, and it renders JavaScript when you set render_js=1. Your spider or parser keeps working on the HTML it gets back. Scraper API is a paid, managed service, not open source. For cooperative or internal targets, the tools above are enough.

robots.txt, standardized in RFC 9309, tells crawlers which paths a site asks them to avoid. It is a request, not a permission system. Following it is good practice, but it does not make every collection lawful. A site's terms of service still apply.

Scrapy obeys robots.txt when ROBOTSTXT_OBEY is on. Heritrix handles it built in. Add delays, cap concurrency per site, and send a User-Agent that identifies you. Avoid personal data you don't need. For background, read our overview of web scraping legality, and involve your legal team before large or sensitive projects.

Start with Beautiful Soup and an HTTP client such as httpx or requests. You can scrape one page in a few lines and learn how selectors work. When you need to crawl many pages with retries and exports, move to Scrapy. If the page only fills in after JavaScript runs, add Playwright.

Usually yes. MIT, BSD, and Apache 2.0 allow commercial use if you keep the license notices. AGPL-3.0 requires you to share your changes if you offer a modified version over a network, and Crawl4AI adds an attribution requirement. Free to license is not free to run, and free web scraping tools with free tiers are not open source.

Yes. Firecrawl's core is released under AGPL-3.0, which is an OSI-approved copyleft license, and its SDKs and some UI components are MIT. You can self-host it with Docker Compose. Its cloud service includes features beyond the self-hosted version, and running a modified copy as a network service brings AGPL source-sharing obligations.

Some can. Parsers such as Beautiful Soup and Cheerio cannot, because they only read the HTML the server sends. Playwright, Puppeteer, and Selenium run a real browser. Scrapy renders JavaScript through the scrapy-playwright plugin, and Crawlee and Crawl4AI have browser rendering built in.

A library handles one step, such as fetching a page or parsing HTML, and your code controls everything else. A framework runs the whole crawl: it schedules requests, follows links, retries failures, applies rate limits, and exports results. Beautiful Soup is a library. Scrapy and Crawlee are frameworks.

No. Frameworks such as Scrapy let you plug in proxies through settings or middleware, and Crawlee has proxy configuration built in, but none of them ships a proxy network. You supply the proxies yourself, along with rotation and health checks, or you send the fetching step to a managed scraping API.

Keep your open-source parser and pipeline, and send the hard URLs to Scraper API. It handles proxy routing, retries, and optional JavaScript rendering, and it returns HTML, JSON, or only the fields you select. Start with 5,000 free requests. Paid plans start at $19/month for 7,000 requests, with JavaScript rendering and country-level targeting included on every plan.

Start with 5,000 free requests

This product includes software developed by UncleCode (https://x.com/unclecode) as part of the Crawl4AI project (https://github.com/unclecode/crawl4ai).