Trafilatura
by adbar
A Python library and CLI that strips a web page down to its main text and metadata, exporting the result as plain text, Markdown, JSON, CSV, or XML: the extraction engine several MCP scraping servers wrap around.
About
Trafilatura isn't itself an MCP tool-call server. It's the extraction library that other people's MCP servers (and a lot of scraping and RAG pipelines) call under the hood. Given a downloaded page or a URL, its `extract()` function separates the actual article or content body from navigation, ads, and boilerplate, and pulls out metadata alongside it: title, author, publication date, site name, and categories or tags, using pattern matching tuned specifically for messy real-world HTML rather than a generic readability heuristic.
For gathering pages in the first place, it handles the crawling side too: parsing sitemaps (TXT and XML) and feeds (RSS, ATOM, JSON), deduplicating and filtering discovered URLs, and downloading in parallel. Output can include or exclude comments, tables, and links depending on what you pass in, and can be written out as TXT, Markdown, CSV, JSON, HTML, XML, or XML-TEI, which is why third-party MCP wrappers built on top of it (rather than trafilatura itself) are what typically get exposed to an agent as callable tools.
Key features
- Main-content extraction that filters out navigation, ads, and boilerplate
- Metadata recovery: title, author, date, site name, categories, tags
- Sitemap (TXT/XML) and feed (RSS/ATOM/JSON) parsing for site-wide crawling
- Parallel download queues for bulk page gathering
- Optional inclusion of comments, tables, and links in extracted output
- Multiple export formats: TXT, Markdown, CSV, JSON, HTML, XML, XML-TEI
Use cases
- Feeding clean article text into a RAG pipeline instead of raw, ad-laden HTML
- Bulk-archiving a site's content by crawling its sitemap and extracting every page to Markdown or JSON
- Building a custom MCP scraping tool where trafilatura does the actual extraction step
- Recovering structured metadata (author, date) for citation or deduplication when aggregating articles from many sources
Available tools
extract()
Extracts the main text and metadata from downloaded HTML into a chosen output format.
fetch_url()
Downloads a page by URL for subsequent extraction.
sitemap/feed discovery
Parses a site's sitemap or feed to enumerate URLs worth crawling.
trafilatura CLI
Command-line interface for extracting from a URL or piped HTML file without writing Python.
Frequently asked questions
Is trafilatura itself an MCP server I can add to Claude?
No. It's a Python library and CLI. If you want it as an MCP tool, you'd add a third-party wrapper (or write a small one) that calls trafilatura's `extract()` function and exposes the result as a tool.
What formats can I export extracted content to?
Plain text, Markdown, CSV, JSON, HTML, XML, and XML-TEI, chosen via the `output_format` argument or CLI flags.