I Stopped Paying for Web Scrapers — Claude Does It in Plain English Now

The library that does it: ScrapeGraphAI

ScrapeGraphAI is an open-source Python library (29k+ stars, MIT, built by Marco Vinciguerra and Lorenzo Padoan) that combines an LLM with graph-based pipeline logic. You hand it a URL and a sentence describing the data you want; it fetches the page — through a Playwright headless browser when the site needs JavaScript — and returns structured JSON. Their tagline is "You Only Scrape Once," which undersells it slightly: the same prompt keeps working after the site changes, which is the part that used to cost money forever.

Six pipelines cover the realistic cases: SmartScraperGraph for a single page, SmartScraperMultiGraph for a list of URLs, SearchGraph to pull from search results, ScriptCreatorGraph when you'd rather it write you a standalone Python scraper, plus multi-page script generation and even audio output. It runs on OpenAI, Gemini, Groq, or Azure keys — or fully local through Ollama, which is where the "zero running cost" claim becomes literal.

Where Claude comes in

The library alone still asks you to write a config dict and remember pipeline names. The trick that removes the last of the friction: wrap it in a skill, so Claude handles the Python and you handle the English. Setup is two commands and one instruction:

pip install scrapegraphai
playwright install

Then, in Claude Code (or Claude with Cowork), one message: "Set this up as a skill — when I ask you to scrape something, use ScrapeGraphAI, extract what I describe, and save the results as a CSV." Claude writes the SKILL.md, wires up the config, and from that point on the interface to all of web scraping is a sentence. If you want the skill format done properly, the official Agent Skills spec is a ten-minute read.

After that, requests look like requests, not code:

scrape every apartment listing with price and location
from propertyfinder.ae

grab all remote AI jobs from remoteok, dedupe them,
drop the CSV in Downloads

extract what this company does, its founders, and their
LinkedIn profiles from this homepage

A real test run

I ran it against two live targets to see how it holds up outside a demo. First, Dubai real estate: one sentence produced a CSV of twenty current listings — villa plots, townhouses, apartments — each with price in AED, property type, bedroom count, and area, from Jebel Ali to Masdar City. Second, a remote-jobs board: 24 postings scraped, deduplicated down to 21 unique openings with title, company, and posting details. Both runs hit real pages with real prices; nothing was mocked or cached.

The run that impressed me most wasn't either of those, though. Pointed at a company homepage with the prompt "extract a description of what the company does, founders, and social media links," it came back with the founder names, their roles, and working LinkedIn URLs — as structured JSON. If you've ever paid a lead-gen tool per enriched contact, you can see where this goes: point it at a directory of companies in your niche and you've built the list yourself, from public pages, for the cost of the tokens.

One honest note from the same test: bot walls exist. One property portal blocked the headless browser outright. The workflow didn't die — Claude fell back to a direct fetch and let the model extract from the raw HTML — but "never gets blocked" is not a claim anyone should make about any scraper, free or paid.

What it actually costs

The library is free. The browser is free. What you pay for is inference on each run — and a single-page extraction is a small prompt, so with an API model you're talking cents, not dollars. Run it through Ollama on your own machine and the marginal cost is genuinely zero, at the price of slower runs and a slightly less clever extractor.

Compare that with the subscription you'd replace: most scraper SaaS pricing starts around $50/month and climbs fast once you want more pages, more frequency, or an API. The one scenario where managed services still earn their fee is industrial scale — thousands of pages a day against aggressively defended sites, where rotating proxies and anti-bot infrastructure matter. ScrapeGraphAI itself sells a cloud API for exactly that case. For the other 95% of scraping — research, lead lists, price checks, content migration, monitoring a handful of pages — the self-hosted library is the whole product.

Scrape like an adult

Because the barrier just dropped to a sentence, the judgment has to come from you now. My rules: public data only, throttled requests, respect robots.txt, and extra care with anything that's personal data — a founder's LinkedIn URL on the company's own about page is public information, but bulk-collecting personal details can still put you on the wrong side of GDPR depending on what you do with it. Terms of service vary per site, and "the tool could" has never been the same as "you may." None of this is legal advice; if you're building a business on scraped data, pay a lawyer before you pay for proxies.

The takeaway

Web scraping used to be a product category. It's becoming a capability — one your coding agent picks up with a pip install and a paragraph of instructions. The pattern is the same one playing out across this whole directory: the moat was never the browser automation or the HTTP requests, it was the understanding layer, and that layer is now a commodity you can run on your laptop.

If you're already in Claude Code, the twenty-minute version of this: install the library, tell Claude to make it a skill, and ask for something you'd actually use — competitor prices, listings in your city, a lead list for your niche. The first time a clean CSV lands in your Downloads folder from a single sentence, the monthly scraper invoice becomes very hard to justify.

Frequently asked questions