recipes article-extraction readability structured-data

How to extract structured data from articles in 2026

Pull clean article bodies, JSON-LD, OpenGraph, Twitter Cards, and reading-time metadata from any news or blog page — the modern alternative to building a Readability fork.

The short answer: call /v1/extract/clean for the body and /v1/extract/structured for the metadata, and you're done

Article extraction in 2026 is a solved problem if you let it be one. The two endpoints together return what a content app, a RAG pipeline, or a research tool needs: a clean body as HTML and plain text, the title, word count and reading time, plus every piece of structured metadata the publisher embedded — which is where the byline, publish date and hero image live. One HTTP call per shape.

The thing you used to build with a Readability fork and a stack of regex is now a JSON response. The rest of this page is the tour.

The problem

Article pages are noisy. A typical news page is 80 percent navigation, ads, related-articles rails, comment widgets, newsletter modals, and tracking scripts. The actual article is somewhere in there, wrapped in three nested divs whose class names change every quarter when the publisher's CMS team ships a redesign.

For years the standard answer was to vendor Mozilla's Readability library into your codebase and hope it kept up with the modern web. It mostly did. When it didn't, you wrote a per-publisher override and added it to a growing pile of selectors that someone on your team had to maintain.

Meanwhile, every publisher embeds schema.org Article JSON-LD in their pages because Google's news ranking rewards it. The structured data you actually need is already on the page. You just have to parse it reliably across thousands of templates, which is its own kind of tedium.

What the top alternatives offer

Mercury Parser, originally from Postlight and now community-maintained, set the bar for hosted article extraction. The original API made content apps possible for a generation of developers, and the open-source successor is still a fine library if you want to run extraction in-process. The Mercury team deserves credit for shaping what a clean extraction response should look like.

Mozilla Readability is the canonical open-source library, and it powers Firefox's Reader View. If your stack is JavaScript and you want zero network hops, Readability is the right call. The maintainers have done years of careful work on edge cases and the code is small enough to read in an afternoon.

Diffbot built a rich extraction product with strong entity-recognition and knowledge-graph layers on top of the basic article shape. For research and intelligence workloads that need entities, sentiment, and topic tagging out of the box, Diffbot is one of the most mature options in the market. Their Article API has been a reference implementation for over a decade.

Firecrawl is also doing excellent work here. Its markdown-first approach is especially well-suited to LLM workflows, and composing extraction with downstream processing is straightforward. These teams ship quality.

Where Ollagraph goes further

Ollagraph treats article extraction as two complementary calls. /v1/extract/clean handles the Readability-style body and the reading metadata. /v1/extract/structured handles every form of embedded metadata the page exposes: JSON-LD, microdata, RDFa, OpenGraph, Twitter Card and standard meta tags. You call one, the other, or both, and you compose the results the way your app needs.

Both endpoints return JSON with stable field names. No HTML to parse for the metadata. When the source page doesn't declare something — no JSON-LD, no Twitter Card — that field comes back empty rather than guessed.

And for teams building LLM pipelines, /v1/scrape with format: "markdown" returns the whole page as one markdown string, and /v1/scrape/llm-ready returns the article text pre-chunked for embedding.

The step-by-step

Step 1 — Clean extraction for the body

Start with /v1/extract/clean. Pass a URL. Get back a structured object with the cleaned article HTML, the plain-text version, the title, the word count, and reading time in minutes.

curl -X POST https://api.ollagraph.com/v1/extract/clean \
  -H "Authorization: Bearer $OLLAGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.example-news.com/2026/05/15/ai-policy-update.html"
  }'

The response looks like this: the body in two forms, and the reading metadata as flat fields.

{
  "status": "success",
  "url": "https://www.example-news.com/2026/05/15/ai-policy-update.html",
  "title": "New AI policy update reshapes enterprise procurement",
  "short_title": "New AI policy update reshapes enterprise procurement",
  "content_html": "<div><p>The announcement on Tuesday marks a turning point...</p></div>",
  "content_text": "The announcement on Tuesday marks a turning point for how large organizations evaluate AI vendors...",
  "word_count": 1420,
  "reading_time_min": 6,
  "fetch_time_ms": 820
}

Need the body as markdown? Send content_html to /v1/convert/html-to-markdown, which returns a markdown string plus word_count and byte_count.

curl -X POST https://api.ollagraph.com/v1/convert/html-to-markdown \
  -H "Authorization: Bearer $OLLAGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "html": "<div><p>The announcement on Tuesday marks a turning point...</p></div>"
  }'

Step 2 — Structured extraction for the metadata

Run /v1/extract/structured in parallel when you need the publisher's embedded metadata. It returns one field per supported format, each parsed and ready to use.

curl -X POST https://api.ollagraph.com/v1/extract/structured \
  -H "Authorization: Bearer $OLLAGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.example-news.com/2026/05/15/ai-policy-update.html"
  }'

The response gives you everything the page exposed.

{
  "status": "success",
  "url": "https://www.example-news.com/2026/05/15/ai-policy-update.html",
  "json_ld": [
    {
      "@context": "https://schema.org",
      "@type": "NewsArticle",
      "headline": "New AI policy update reshapes enterprise procurement",
      "datePublished": "2026-05-15T08:30:00Z",
      "author": { "@type": "Person", "name": "Jordan Reyes" },
      "publisher": { "@type": "Organization", "name": "Example News" }
    }
  ],
  "microdata": [],
  "open_graph": {
    "title": "New AI policy update reshapes enterprise procurement",
    "type": "article",
    "url": "https://www.example-news.com/2026/05/15/ai-policy-update.html",
    "image": "https://www.example-news.com/images/ai-policy-hero.jpg",
    "site_name": "Example News",
    "description": "The announcement on Tuesday marks a turning point..."
  },
  "twitter_card": {
    "card": "summary_large_image",
    "site": "@examplenews",
    "title": "New AI policy update reshapes enterprise procurement",
    "image": "https://www.example-news.com/images/ai-policy-hero.jpg"
  },
  "rdfa": [],
  "meta": {
    "title": "New AI policy update reshapes enterprise procurement",
    "description": "The announcement on Tuesday marks a turning point...",
    "canonical": "https://www.example-news.com/2026/05/15/ai-policy-update.html"
  },
  "products": [],
  "fetch_time_ms": 790
}

Each call costs one credit, so the pair costs two credits and gives you a complete picture of the article.

Step 3 — One markdown document for LLM pipelines

If your downstream is a model, you usually want one markdown document per article with the metadata on top. Compose it in your own code from the two calls above plus the HTML-to-markdown conversion — a few lines of Python:

import os, requests

API = "https://api.ollagraph.com"
H = {"Authorization": f"Bearer {os.environ['OLLAGRAPH_API_KEY']}"}

def article_markdown(url):
    clean = requests.post(f"{API}/v1/extract/clean", json={"url": url}, headers=H).json()
    meta = requests.post(f"{API}/v1/extract/structured", json={"url": url}, headers=H).json()
    md = requests.post(f"{API}/v1/convert/html-to-markdown",
                       json={"html": clean["content_html"]}, headers=H).json()["markdown"]

    article = next((j for j in meta["json_ld"] if j.get("@type") in ("NewsArticle", "Article", "BlogPosting")), {})
    author = article.get("author") or {}
    front = [
        "---",
        f"title: {article.get('headline') or clean['title']}",
        f"author: {author.get('name', '') if isinstance(author, dict) else ''}",
        f"published_at: {article.get('datePublished', '')}",
        f"source: {meta['meta'].get('canonical', url)}",
        "---",
    ]
    return "\n".join(front) + "\n\n" + md

The front matter is YAML so most retrieval frameworks parse it for free. If you would rather skip the composition and just embed, /v1/scrape/llm-ready returns the article's main text as a chunks array with a token_count per chunk and a total_tokens figure for routing decisions.

Step 4 — Batch a list of articles

For ingestion jobs, fan the extract calls out from your own worker pool. When you only need the raw page per URL, /v1/scrape/batch/async takes up to 100 URLs, returns a job ID immediately, and posts the results to your webhook when the batch finishes; run your own Readability pass over the returned HTML, or send each page through extract/clean afterwards.

curl -X POST https://api.ollagraph.com/v1/scrape/batch/async \
  -H "Authorization: Bearer $OLLAGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "format": "html",
    "urls": [
      "https://www.example-news.com/2026/05/15/ai-policy-update.html",
      "https://www.another-publisher.com/2026/05/14/cloud-pricing.html",
      "https://www.blog-site.com/posts/the-state-of-rag.html"
    ],
    "webhook_url": "https://yourapp.com/hooks/extraction"
  }'

The async variant /v1/scrape/async works the same way for single-URL fire-and-forget jobs that you'd rather poll than block on.

Step 5 — Handle multi-page articles

For long-form pieces that span numbered pages, follow the chain yourself. A /v1/scrape call with a js_eval expression reads the next-page link from the rendered page; its value comes back in eval_result.

curl -X POST https://api.ollagraph.com/v1/scrape \
  -H "Authorization: Bearer $OLLAGRAPH_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.longform-site.com/the-essay-part-1.html",
    "format": "text",
    "js_eval": "(document.querySelector(\"link[rel=next], a[rel=next]\") || {}).href || null"
  }'

Call /v1/extract/clean on each page in the chain and concatenate the bodies in order in your own code. Sum the per-page word_count values for the full piece.

A realistic scenario

Take a media-monitoring startup. Call the founder Sam. The product surfaces relevant news to enterprise communications teams every morning. Sam's team was ingesting 40,000 articles a day through a homegrown extractor built on top of Readability and a queue of per-publisher overrides. Three engineers spent a third of their time maintaining selectors.

They moved to Ollagraph over a sprint. The ingestion job now fans out /v1/extract/clean calls from a small worker pool, and the per-publisher overrides are gone. The structured extractor picks up JSON-LD and OpenGraph for the metadata sidebar. Total maintenance time on the extractor in the six months since: zero hours.

The interesting part is what those three engineers are doing now. They built the entity-tagging layer their customers had been asking for. The extractor was never the product. It was a chore that pretended to be the product.

Pricing math

Clean extraction, structured extraction and HTML-to-markdown conversion each cost one credit per call, and a failed call is refunded. A full article record (clean + structured + markdown) is three credits; most pipelines that only need the body and metadata spend two.

A media-monitoring pipeline ingesting 50,000 articles a day lands comfortably with room for growth, and a team running a single content-research app at a few hundred articles a day fits cleanly on a small allocation. The pricing page has the live credit packs.

Pricing across serious vendors in this space clusters in the same range because the underlying work is similar. Pick on output quality and developer experience rather than headline rates.

Composing extraction with scraping

Sometimes the article body only appears after a click on a "load more" or "read full story" button. In that case, use /v1/scrape with an actions array — for example {"action": "click", "selector": "button.read-more"} — and format: "html". Actions run once on the loaded page before it is captured; they don't follow a navigation to a new URL. Then convert the returned HTML with /v1/convert/html-to-markdown or run your own Readability pass over it, since the extraction endpoints take a URL rather than HTML.

For sites that publish article lists on an index page, use the crawl endpoint to walk the index, then fan out to extract on each discovered URL. The pattern is the same one used by most large-scale news ingestion pipelines and it's roughly five lines of code on top of Ollagraph.

When the source requires authentication, note that every Ollagraph call is stateless: each one starts a fresh browser, and there are no persistent sessions, cookies or logins between calls. The companion recipe on pages behind a login covers what that means in practice.

Operational notes

Three things separate good extraction pipelines from fragile ones. First, normalize URLs before you call. Strip tracking parameters, follow obvious shortener domains, and use the canonical URL from the structured response (meta.canonical) when you store the record. This avoids ingesting the same article eight times under eight different query strings.

Second, treat the publish date as best-effort. Publishers are inconsistent about updating it on edits, and JSON-LD sometimes lags the visible page. If freshness matters, cross-check the JSON-LD datePublished against the OpenGraph article:published_time and store both.

Third, store a hash of the extracted body alongside the record. When a publisher updates an article and your downstream consumers want to know about it, the hash gives you a cheap diff signal. Compute it in your own code from content_text, and use /v1/text/diff when you want to see exactly what changed.

When to use which endpoint

Use /v1/extract/clean when you need the body as readable text or HTML, and add /v1/convert/html-to-markdown when you need it as markdown. Use /v1/extract/structured when you need the publisher's metadata. Use both when you need a complete record. Use /v1/scrape/llm-ready when the consumer is a model and you want pre-chunked text.

Use /v1/scrape as a pre-processing step only when the article needs a page action to appear. The extraction endpoints fetch the page themselves and you almost never need to scrape and extract separately.

All of these endpoints share the same authentication, the same rate limits, and the same observability surface. Mixing them in a single pipeline is not a decision; it's how the platform is designed to be used.

Cross-references and further reading

For the underlying scrape API, see the capabilities page. For full reference docs, the documentation covers every parameter and response field. Pricing is on the pricing page.

External background reading: schema.org Article defines the JSON-LD types this guide assumes, and the Wikipedia overview of web scraping covers the broader landscape. Both are useful primers if you're new to the space.

For the upstream pipeline that feeds extraction, see the recipe on building a RAG knowledge base.

The shape of a clean extraction response, in detail

A response is only useful if you know what each field means and when to trust it. The title field is the page title as Readability reads it from the HTML; short_title is the same title with site-name suffixes trimmed where Readability can detect them. If you want the publisher's canonical headline, take the JSON-LD headline from the structured call.

The body comes back twice. content_html is the cleaned article HTML, useful when you need to preserve inline structure for a reader app or convert it to markdown. content_text is the same body with every tag stripped, useful for keyword indexing, language detection and hashing.

The reading_time_min field is calculated from the cleaned body word count, not the raw HTML, which means it reflects time-to-read for the article itself and ignores navigation and ads. The word_count is the same denominator and is exposed separately so you can apply your own reading-speed model if the default isn't right for your audience.

The clean response does not carry a language, author, hero image or publish date. Language comes from /v1/lang/detect on content_text; author, publish date and image come from the structured call's json_ld and open_graph fields.

The shape of a structured extraction response, in detail

The json_ld field is an array because publishers routinely embed multiple JSON-LD blocks per page: one for the article, one for the publisher organization, one for the breadcrumb trail, and sometimes one for a video or audio embed. Each entry is parsed from the page's JSON, so you can iterate the array and dispatch on @type without writing your own JSON-LD parser.

The open_graph field is a flat object keyed by Open Graph property name with the og: prefix dropped: title, type, url, image, site_name, description, and any other properties the page declares. Properties the page doesn't declare are simply absent, so check for presence before you read them.

The twitter_card field follows the same pattern with the Twitter Card meta tag namespace. The microdata and rdfa fields are arrays for the sites that still use those formats instead of JSON-LD. The meta object carries the page title, the canonical URL and standard tags such as description and author, and products is a flattened list of any schema.org Product entries.

One small but useful detail: the structured extractor keeps JSON-LD blocks in the order they appear in the source HTML, which matters for sites that put their primary article block first and supplementary blocks after.

Composing extraction with downstream tasks

The most common downstream after extraction is summarization or embedding. Both work well with the markdown body and benefit from the typed metadata. A summarization prompt that includes the title, the publisher, and the publish date produces noticeably better summaries than one that just receives the body. A retrieval system that indexes the body but stores the metadata as filter columns gives users much better search than one that flattens everything.

The second most common downstream is classification and tagging. Reading time is a surprisingly good prior for content type: very short articles are usually news, medium articles are usually features, and long articles are usually essays or analysis. Language is the trivial first filter for multilingual pipelines. Word count is useful as a quality signal for spam detection.

The third common downstream is republication or syndication, which is where the structured extractor earns its keep. The JSON-LD output gives you exactly what you need to rebuild the article's canonical metadata in your own CMS, including the original author and publisher attribution. This matters for legal compliance and for trust with the source.

Handling weird article pages

Some article pages don't fit the standard mold. Live blogs append new content to the same URL over hours or days. Slideshow articles split each "page" into a click. Q&A interviews use unusual heading structures. Readability-style extraction handles many of these, and the rest are usually a little post-processing in your own code.

For live blogs, re-extract the same URL on a schedule and diff the new content_text against the last one (/v1/text/diff does this) to pick out the new entries. For slideshow articles, treat each slide URL like a page in Step 5 and concatenate the bodies with your own labels. For Q&A formats, keep content_html rather than the plain text so the speaker labels encoded in headings and bold text survive.

The general principle is to start from what the publisher meant, not just what the HTML contained. When publishers use semantic markup, the extractors give you that directly. When they don't, a small amount of post-processing on top of the clean and structured responses covers the common patterns.

A short note on language coverage

The clean extractor works on any language with standard left-to-right or right-to-left text, because the body-detection heuristics are structural rather than lexical. Less-represented languages work equally well from a structural perspective; the only caveat is that language detection becomes less confident at the long tail, which /v1/lang/detect reports in its confidence score.

Reading time uses the same words-per-minute baseline universally, which slightly under-reports for character-based languages like Chinese and Japanese where the equivalent measure is closer to characters per minute. If reading time matters for your product in those languages, divide the character count by 500 as a reasonable approximation.

Closing

Article extraction used to be a research project with a long tail of edge cases. In 2026 it's a few endpoints and a webhook, with JSON responses you can build on directly. Build the thing your users actually care about on top of it.

If you want to try the calls in this guide against your own URLs, grab a key and run the clean extraction example at the top of this page. Signup includes 1,000 free credits while you evaluate.

Common questions

What's the difference between Readability and Mozilla Readability?

Readability is the historical name for the algorithm that extracts the main body of an article from a noisy web page. Mozilla Readability is the open-source library Firefox uses for Reader View, derived from the original Readability project. Both refer to the same family of techniques. Ollagraph's clean extraction sits in this lineage.

How do I get clean markdown from a news article?

Call /v1/extract/clean with the article URL. You get back the title, the cleaned body as HTML (content_html) and plain text (content_text), word count and reading time. Pass content_html to /v1/convert/html-to-markdown to get the body as markdown, ready to feed to a model or render in a reader app.

Does Ollagraph return JSON-LD as structured objects?

Yes. /v1/extract/structured parses every JSON-LD block on the page and returns them as an array of objects. OpenGraph and Twitter Card meta tags come back as flat objects keyed by property name, microdata and RDFa as arrays, and standard meta tags (description, author, canonical and a few others) in a meta object.

What languages does article extraction support?

Clean extraction works on any language with standard left-to-right or right-to-left text. The body extractor doesn't depend on language-specific dictionaries. The extraction response does not include a language field; if you need one for routing, send content_text to /v1/lang/detect, which returns a language code and a confidence score.

How accurate is reading time?

Reading time uses a baseline of roughly 238 words per minute, close to the standard for adult English-language reading on screens. The number is calculated from the cleaned body, not the raw HTML, so it ignores navigation chrome and footers. For other languages the same baseline is used, which slightly under-reports for dense languages.

Can Ollagraph extract paywalled article content?

No, and this is a deliberate choice. Ollagraph is built for legitimate access patterns: open articles, your own content, and pages where you have a license. Paywall circumvention isn't supported. There's plenty of valuable work to do on the open web.

What's the rate limit on extraction endpoints?

Extraction endpoints share the standard per-key rate limit with the rest of the API. For large jobs, fan out calls from your own worker pool, or use /v1/scrape/batch and /v1/scrape/batch/async to fetch many URLs in one request.

How does clean extraction handle multi-page articles?

/v1/extract/clean works on one URL at a time. For articles split across numbered pages, find the next-page link (a /v1/scrape call with a js_eval expression that reads the rel=next href works well), call extract/clean on each page, and concatenate the bodies in your own code. For articles using infinite-scroll patterns, use the scrape endpoint with a scroll_to_bottom action.

Can I get LLM-ready output in one call?

Yes. /v1/scrape with format set to markdown returns the whole page as one markdown string, and /v1/scrape/llm-ready returns the article's main text already split into token-sized chunks for retrieval. For a markdown document with title, author and date front matter, combine extract/clean, extract/structured and /v1/convert/html-to-markdown in your own code — the recipe below shows how.

PRODUCTION PLAYBOOK

Start with 1,000 free credits.

Every endpoint, one bearer token, no credit card required. Build the pipeline above in an afternoon.

Request access Read the docs

Amit Sharma

Technical research, crawler intelligence, and LLM search optimization at OllaGraph.

Share article: Post LinkedIn