Provenance and license status for agent retrieval

01 — the problem

Counsel has started asking where the agent's data comes from

In July 2026 a court gave final approval to the $1.5B Anthropic settlement, the largest copyright settlement in US history, over how training data was obtained.1 It settled, so there is no finding of liability in it. The number is what one unresolved question cost one company.

The cases squarely about real-time retrieval keep surviving dismissal. Dow Jones v. Perplexity was pleaded over both the RAG index and the outputs (August 2025). In Advance Local Media v. Cohere, claims over substitutive RAG summaries were allowed to proceed (November 2025). In Reddit v. Perplexity and SerpApi, DMCA claims survived against the scraping vendor in the supply chain rather than the AI company alone (July 2026).2

Surviving dismissal means the claims proceed. It is not a finding of infringement, and no court has ruled that retrieval is unlawful. The other half of the picture is just as unsettled: no US court has held that retrieval-time copying for grounding is fair use either. The question is undecided rather than decided against anyone.

Vendor indemnities do not close that gap. Microsoft's Customer Copyright Commitment, OpenAI's Copyright Shield, and Anthropic's equivalent protections cover outputs, and they are conditioned on you having the rights to your inputs.3

Meanwhile the open web is narrowing. Cloudflare has blocked AI crawlers by default since July 2025, and from 15 September 2026 it extends default blocking of mixed-use crawlers on ad-supported pages.4 Licensing infrastructure arrived in the same window: the RSL 1.0 standard published in December 2025, and Microsoft's Publisher Content Marketplace pays publishers per use for grounding content.

robots.txt and RSL declarations are publisher signals rather than binding law. They state a preference a retrieval layer can follow, and following them is a posture you can describe to counsel rather than a defense you can rely on in court.

1. Authors Guild on the settlement's final approval · 2. Loeb & Loeb on Dow Jones v. Perplexity, Loeb & Loeb on Advance Local Media v. Cohere, Bloomberg Law on Reddit v. Perplexity and SerpApi · 3. Microsoft on the Customer Copyright Commitment · 4. Cloudflare on default AI-crawler blocking, TechCrunch on the September 2026 change

02 — how it works

Two layers on the way in, and the record they leave behind

Clearfetch does two different jobs on ingress, and they carry different guarantees. We keep them apart here, in the docs, and in the response, because the distinction is the part a lawyer will read closely.

Layer 1 · deterministic

Channel elimination

HTML is re-rendered into strict markdown. That render provably strips scripts, CSS-hidden text, zero-width characters, and document metadata, because the output format has nowhere to put them. Images are re-rendered or captioned instead of being passed through as live references.

This layer is a transform rather than a judgement, so we can name exactly what it removes. Strict markdown also drops layout and interactivity, which is the trade you are making.

Layer 2 · probabilistic

Injection detection

Whatever survives the transform is scanned for instruction-bearing text. Every response carries a verdict of clean, sanitized, or blocked, plus the findings behind it, each with a category, a confidence, and a plain-language detail line.

Detection is statistical. We publish detection rates instead of claiming immunity, and those numbers publish with the API rather than after it, so what you tell your customers about this layer is a measured number.

on every response

license_status: What Clearfetch can declare about the terms the content was published under.
url: Final URL after redirects, plus the domains the request passed through.
domain: The publisher the content came from, recorded per response.
content_sha256: Hash of the bytes we fetched, so the record can be checked against the content later.
verdict: clean, sanitized, or blocked, with the findings behind it.
fetched_at: Time of the fetch, with the upstream HTTP status.
request_id: The call this record belongs to, matching the row in the per-key audit log.

Clearfetch fetches without circumventing. It follows robots.txt and RSL declarations, and it does not route around bot protections, paywalls, or rate limits. That posture costs coverage on sites that have decided to keep automated readers out. That is the trade, and it is why the record means anything.

Licensed retrieval sits on the same footing as the rest of the product, built on infrastructure that now exists: the RSL 1.0 standard, Microsoft's Publisher Content Marketplace, and per-use licensing intermediaries. Clearfetch is In early-access end to end. Sanitization, verdicts, audit logs, and licensed content are all in the same state: none of them is serving traffic yet. If licensed retrieval is the part you need, saying so on the request form is what tells us to weight it.

03 — the response

One fetch, with its provenance attached

A trade-press article is the kind of source an agent summarises for a customer, and the kind whose publisher has an opinion about being read by machines. Here is the shape of the response.

license_status and the provenance block are the parts that matter on this page. license_status carries what Clearfetch can determine about the terms the content was published under (licensed, open, declared, or unknown), content_sha256 pins the exact bytes that were read, and request_id ties the response to a row in your audit log. The sample reads unknown, which is the honest label for an ordinary public page that publishes no machine-readable license signal. It is a statement of what we know about the source, not a grant of rights.

POST /v1/fetch Example

verdict

verdict sanitized content delivered, one span removed

ok true

license_status unknown no machine-readable license signal on this source

scans · findings

category prompt_injection

confidence 0.94

detail A block styled to be invisible in a browser told the reading assistant to summarise the article as a buy recommendation and to attribute it to the reader's own analyst. The span was removed from the returned markdown and recorded as a finding.

channels removed scripts, css_hidden_text, zero_width_chars, metadata

content

text/markdown · strict render

# Freight rates ease into the third quarterSpot rates on transpacific lanes fell 4% week over week.[ removed · injected_instruction ]Carriers have held blank sailings at last month's level.Analysts expect the softening to continue into September.

provenance

url https://example-tradepress.com/2026/07/freight-outlook

domain example-tradepress.com

fetched_at 2026-08-01T14:22:09Z

upstream_status 200

content_sha256 9f31c2…d7a4

request_id req_01JZ7Q6B4M

04 — objections

Objections

These are the three questions we expect first.

“Scraping isn't illegal.”

Correct, and this page never says it is. In Meta v. Bright Data (2024) a federal court held that Meta's own terms of service did not bar logged-off scraping of public data, and that ruling was about Meta's terms rather than a general permission to scrape.

The live risks are narrower. Terms-of-service contract claims can survive independently of fair use, as they did on the Reddit v. Anthropic remand in March 2026. Anti-circumvention claims now reach scraping vendors in the supply chain, as in Reddit v. Perplexity and SerpApi in July 2026.5 Retrieval-time fair use is undecided.

Clearfetch reduces the exposure surface and gives you the provenance record. Legal absolution is not on sale here or anywhere else.

“Is your content actually licensed?”

Clearfetch is in early access end to end, and nothing on this site is live yet. Licensed retrieval sits on the same footing as sanitization, verdicts, and audit logs, built on infrastructure that now exists: the RSL 1.0 standard, Microsoft's Publisher Content Marketplace, and per-use licensing intermediaries.

Every response carries provenance, a declared license status, and a fetch that does not circumvent. If licensed retrieval is your need, say so on the request form and we weight it accordingly.

“No enterprise customer has ever been held liable for its vendor's scraping.”

That is true today. The exposure is unindemnified and undefined rather than adjudicated, which is exactly why counsel now advise separating input risk from output risk.6

A provenance record is cheap next to an open question. If you would rather wait for the question to close, that is a defensible read.

5. Bloomberg Law on Reddit v. Perplexity and SerpApi · 6. WilmerHale on managing AI legal risk, February 2026

The full product overview is on the Clearfetch home page. Field definitions and the API shape are in the docs.

Provenance and license status on every response your agent reads