Methodology

One row per item in one day's scan run; the latest completed day by default, or add ?day=YYYY-MM-DD for a specific day. Discovery is public press feeds plus per-topic web searches; page text is fetched directly with a reader fallback; enrichment is a small model pass whose tags come from a fixed topic taxonomy. relevance is advisory and downstream triage is expected. This is a working corpus, not a redistribution channel. Downloading requires an access key.

Schema

Column	Type	Definition
item_id	text	Stable UUID of the scanned item.
run_day	date	The scan day (YYYY-MM-DD, UTC).
url	text	The item URL as discovered.
normalized_url	text	Canonical dedupe form: host plus path plus sorted query, tracking params stripped.
headline	text	Headline as reported by the feed or search.
source_domain	text	Hostname of the URL, www stripped.
published_on	date	Publication date when known (YYYY-MM-DD).
discovered_via	text	The discovering topic's slug for a feed item, or web_search.
topic_slug	text	The scan topic that discovered the item.
topic_code	text	Taxonomy code of that topic.
summary	longtext	Two to three model-written sentences; empty when enrichment was skipped.
tags	text	Taxonomy codes the enrichment assigned, joined with a semicolon.
entities	text	Companies, agencies, and people named, joined with a semicolon.
relevance	number	Advisory 0 to 1 relevance. Since 2026-09-02 the median of score-only votes from the enrichment panel (see relevance_votes); before that, the enriching model's single read.
enrich_status	enum	done, skipped, error, or pending.
fetch_status	enum	done, failed, skipped, or pending.
fetched_via	enum	direct or jina.
text_chars	number	Character count of the retained page text.
full_text	longtext	The complete retained page text, capped at 24,000 characters.
enriched_by	text	Model that produced the enrichment (summary, tags, relevance); null before model tracking began.
source_tier	number	Source reliability tier, 1 (most reliable: regulators, primary company sources, wire services, research houses) to 4 (least: social platforms, stock tip and promotional sites). Deterministic for a known domain from a curated map and suffix rules; a domain neither covers is rated once by the utility model and cached. Null when the domain has not been rated yet.
source_kind	enum	The kind of source behind the domain: regulator, primary, research, wire, major, trade, tech_press, general, aggregator, pr_wire, blog, social, promo, or unknown. Rated on the same path as source_tier. Null when not yet rated.
content_kind	enum	The kind of content in the piece, assigned during enrichment: news, analysis, data, press_release, marketing, opinion, or other. Discounts promotional or opinion text found inside an otherwise reliable source. Null when enrichment has not classified it.
priority	number	Composed ranking score for the importer: relevance times the source tier weight (tier 1 is 1.0, tier 2 is 0.85, tier 3 is 0.6, tier 4 is 0.25) times the content kind weight (news, analysis, and data are 1.0, opinion and other are 0.85, press_release is 0.7, marketing is 0.4), rounded to two decimals. An unrated source counts as tier 3 and an unclassified piece counts as other. Null only when relevance is null.
relevance_spread	number	Disagreement across the enrichment panel's relevance votes: the highest vote minus the lowest, 0 to 1. Null when only one model has voted so far (a single read, not yet ensembled).
relevance_votes	text	The raw per-model votes behind relevance, as a compact JSON object of model id to 0 to 1 score, for example {"deepseek/deepseek-v4-flash":0.7,"z-ai/glm-5.3-flash":0.4}. Null when no panel votes are recorded yet.

External scan, daily

Request access Unlock with an access key

/api/datasets/external-scan · 26 columns · access key required

One day of the automated external news scan: items across financial services and technology topics discovered from public press feeds and web search, with full text, a model summary, taxonomy tags, and named entities.

Methodology

Schema

Column	Type	Definition
item_id	text	Stable UUID of the scanned item.
run_day	date	The scan day (YYYY-MM-DD, UTC).
url	text	The item URL as discovered.
normalized_url	text	Canonical dedupe form: host plus path plus sorted query, tracking params stripped.
headline	text	Headline as reported by the feed or search.
source_domain	text	Hostname of the URL, www stripped.
published_on	date	Publication date when known (YYYY-MM-DD).
discovered_via	text	The discovering topic's slug for a feed item, or web_search.
topic_slug	text	The scan topic that discovered the item.
topic_code	text	Taxonomy code of that topic.
summary	longtext	Two to three model-written sentences; empty when enrichment was skipped.
tags	text	Taxonomy codes the enrichment assigned, joined with a semicolon.
entities	text	Companies, agencies, and people named, joined with a semicolon.
relevance	number	Advisory 0 to 1 relevance. Since 2026-09-02 the median of score-only votes from the enrichment panel (see relevance_votes); before that, the enriching model's single read.
enrich_status	enum	done, skipped, error, or pending.
fetch_status	enum	done, failed, skipped, or pending.
fetched_via	enum	direct or jina.
text_chars	number	Character count of the retained page text.
full_text	longtext	The complete retained page text, capped at 24,000 characters.
enriched_by	text	Model that produced the enrichment (summary, tags, relevance); null before model tracking began.
source_tier	number	Source reliability tier, 1 (most reliable: regulators, primary company sources, wire services, research houses) to 4 (least: social platforms, stock tip and promotional sites). Deterministic for a known domain from a curated map and suffix rules; a domain neither covers is rated once by the utility model and cached. Null when the domain has not been rated yet.
source_kind	enum	The kind of source behind the domain: regulator, primary, research, wire, major, trade, tech_press, general, aggregator, pr_wire, blog, social, promo, or unknown. Rated on the same path as source_tier. Null when not yet rated.
content_kind	enum	The kind of content in the piece, assigned during enrichment: news, analysis, data, press_release, marketing, opinion, or other. Discounts promotional or opinion text found inside an otherwise reliable source. Null when enrichment has not classified it.
priority	number	Composed ranking score for the importer: relevance times the source tier weight (tier 1 is 1.0, tier 2 is 0.85, tier 3 is 0.6, tier 4 is 0.25) times the content kind weight (news, analysis, and data are 1.0, opinion and other are 0.85, press_release is 0.7, marketing is 0.4), rounded to two decimals. An unrated source counts as tier 3 and an unclassified piece counts as other. Null only when relevance is null.
relevance_spread	number	Disagreement across the enrichment panel's relevance votes: the highest vote minus the lowest, 0 to 1. Null when only one model has voted so far (a single read, not yet ensembled).
relevance_votes	text	The raw per-model votes behind relevance, as a compact JSON object of model id to 0 to 1 score, for example {"deepseek/deepseek-v4-flash":0.7,"z-ai/glm-5.3-flash":0.4}. Null when no panel votes are recorded yet.