On this page

Methods

Last revised 18 November 2026

How a question is turned into a weighed answer: question intake, search iterations, how claims are weighed, outlet assessment, social-media perception, stopping and filing, and how changes spread to everything that depends on them. This page describes the method in use and is revised whenever the rules change.

Summary#

The system answers one question at a time by iterating: plan queries, search, read, record each claim with the exact words that support it, weigh the evidence, resolve contradictions, ask follow-up questions, and repeat until the sources are exhausted or every sub-question is answered or saturated. There is no fixed limit on the number of iterations, so a project can run for days.

Every source is weighed by a profile of its outlet, built by a separate assessment: a pool of the outlet’s own articles fact-checked against primary records and independent references (a random draw plus the articles research itself read and cross-checked, age-weighted), its ownership chain and stakes, its measured leaning, how its wording compares with other outlets, and how social-media audiences see it, each audience read through a profile of that platform’s users. No outlet gets weight from reputation or personal trust. Confidence rests on evidence actually obtained. An inaccessible page is not evidence that an outlet is wrong, but its absence can prevent a conclusion, widen uncertainty or make a sample unrepresentative; retrieval failures and replacements remain in the record.

Questions with an ethical, philosophical or legal component get the agent’s own analysis of it, with judgments given a status instead of a percentage. When the questions run out, synthesis reviews ask follow-up questions about the findings as a whole until nothing more is worth asking. Before a report is filed, a red team argues against the draft answer, every key claim is tested by removing each independent group of sources in turn, and the share of sources in the subject’s own language is checked. Reports are filed as provisional while any outlet behind a claim still lacks a profile, and are updated as profiles arrive.

The pipeline at a glance#

  1. Question intake: a question is written down, scoped, and clarifying questions are asked, each with a default; research starts on the defaults.

  2. Scoping: sub-questions, question kind, the subject’s languages, starting claims from earlier findings, and a check for ethical, philosophical and legal components (sections 3a to 3c).

  3. Search iterations: queries of several kinds per question, sources read and registered, claims recorded with verified quotes, confidence set per claim, contradictions worked through, follow-up questions added.

  4. Outlet assessments, running alongside: every outlet that carries a claim gets a profile (or a reevaluation when its pool of graded articles doubles or its profile ages).

  5. Stopping checks: stop rules, synthesis reviews until nothing more is asked, red team, ethics, philosophy and law checks, base rates for headline claims, sensitivity, language coverage.

  6. The report: the current state is shown in the shape of the final report; filing marks it provisional or final, with a data page and the findings kept for reuse.

  7. Afterwards: re-weighing when profiles change, follow-up questions, and calibration of resolvable claims.

1. Question intake#

A question enters as written text with an optional priority and deadline; anything else written with it is context.

Clarifying questions#

Each new entry gets 2-6 clarifying questions only where the answer would change the research. Every question carries the default the research will use, and research starts at once on those defaults: nothing waits for the asker. Answers are applied when they arrive; answers that come after filing go into the next report update. Unanswered questions are listed in the report’s method section with the default that was used.

Every question is also checked for an ethical component (section 3a), a philosophical component (section 3b) and a legal component (section 3c), whether or not the asker asked for one. When either is only a side component, the single inclusion question asks whether to include that analysis (default yes). The agent chooses analytical frameworks, traditions and applicable instruments, and explains why. Factual clarification remains permitted when it changes the answer, including where an event occurred, the parties’ legal status and the jurisdiction applicable to the asker’s situation; alternatives are analyzed conditionally if facts are unavailable. Normative or contested questions (“who is right”, “is X a terrorist group”, “is this genocide”) are split into parts answered separately, moral judgments are kept separate and labeled, each side gets equal search effort, and each side’s own language is searched.

Scoping#

The root question is restated with its deliverable, limits (period, region, language) and what “answered” means, then classified by kind. The kind decides which sources are searched first and weighted most:

KindPriority sourcesSecondary channels (always searched, with their role)
political, current affairswires, established news, government statements, official records, statements of the partiessocial media and forums: alternative explanations, opinion, early reports, what each side claims
medicalPubMed, Cochrane, trial registries, CDC, WHO, NIH, regulators, guidelinespatient forums: experience in practice, never evidence of efficacy
scientificpeer-reviewed journals, preprints (marked unreviewed), datasets, standardsscience journalism, expert blogs: context, criticism, replication
legalstatutes, regulations, court records, regulator guidancelegal journalism, forums: interpretation and practice
economicstatistical agencies, central banks, filings, IMF, World Bank, OECD datafinancial press, analyst notes: expectations and sentiment
historicalarchives, primary documents, scholarly historiespopular histories, encyclopedias, local accounts
ethical, normativepeer-reviewed ethics and philosophy literature, the Stanford and Internet Encyclopedias of Philosophy, authoritative normative texts (treaties, human-rights instruments, professional codes)advocacy groups, op-eds, religious authorities: each side’s argument in its own terms; surveys of moral attitudes: opinion only
philosophical, conceptualpeer-reviewed philosophy journals, the Stanford, Internet and Routledge encyclopedias of philosophy, primary texts of the philosophers and traditions, PhilPaperspopular philosophy, lectures, philosophy blogs; religious and cultural texts as statements of positions; specialist surveys such as the PhilPapers Survey as opinion only
legallegislation and treaty texts, court and tribunal judgments, findings of official bodies (mandate noted), peer-reviewed law journals, the Max Planck Encyclopedia of Public International Law, the ICRC customary humanitarian law databaselaw blogs (EJIL:Talk!, Just Security, Lawfare, Opinio Juris); the parties’ court submissions as positions; advocacy groups’ legal analyses, discounted for their interest; news reports of rulings only to find the primary text
opinionpolls and surveys with methodssocial media and comments, weighted by sample size and representativeness
factual (default)many sources of mixed kind, weighted by qualityeverything else relevant, at low weight

A question can have two kinds; a question that fits none gets a new kind written into the method before research starts. Earlier findings are read first: known findings become starting claims traced back to their original evidence roots, and known gaps become questions. A summary of earlier findings is not independent corroboration of its own sources. Then 4-10 sub-questions are recorded with the part of the root question each serves and a priority, a first outline is drafted, and the subject’s main languages are set, so that native-language queries run from the first iteration. A coverage map records the relevant positions, competing explanations, channels and languages, what was searched, and remaining gaps.

2. Search iterations#

Search and reading tools#

Searches use general web search and routes to primary sources (agencies, registries, journals, filings, datasets). Every query and every page read is recorded, exact repeats are refused, and each page is saved with a content hash so quotes can be checked against what was read.

What an iteration does#

  1. Open contradictions come first, then due retries of earlier failures.

  2. For 3-5 chosen sub-questions, 2-4 queries each, of different kinds:

Query kindUse
plainthe question in natural language
termskey terms, synonyms, exact phrases and names found so far
primaryroutes to agencies, registries, journals, filings, standards bodies, datasets
dateddate-bounded queries for recent change
local-languagethe subject’s own language
disputecriticism, replication, “versus”, “data on”
resolutionaimed at a contradiction: primary data, corrections, retractions, fact-checks, neutral sources

Each query is recorded before it runs; an exact repeat is refused and must be rephrased. Every iteration includes the kind’s priority sources and at least one secondary channel.

  1. Results are judged for relevance, classed (priority, secondary, social) and registered with their outlet; the best are read, priority sources first, and the page text is saved.
  2. Claims are recorded with verbatim quotes verified against a saved source snapshot and its content hash. A quote’s presence establishes occurrence, not entailment: each evidence use distinguishes direct support, attribution and inference, with an explanation of how it bears on the claim. Inferences from research require appraisal of design, measurement, confounding, selection and applicability. Model knowledge enters only marked as unverified and cannot support a scored claim.
  3. Evidence is grouped per claim by its original observation, document or dataset, not merely by publisher. Syndication, shared reporting and outlets under central or shared editorial control are dependent by default. Distinct same-outlet evidence roots need a recorded justification; separate websites repeating one dataset are not independent confirmation.
  4. Each query’s novelty is recorded. A query adds to the findings only with a new claim, a new independent source for a known claim, a contradiction, a more precise or more recent value, or a new lead; the same claim from a dependent or weaker source adds nothing.
  5. Confidence is set for every new or changed claim, contradictions are opened, follow-up questions are added (repeats are refused), and question statuses are updated.
  6. Outlets are recounted and any outlet that needs a profile is queued (section 4).
  7. The outline is reorganized when findings cluster along another axis (topic, chronology, geography, actors, causal chain, for and against, process stages, comparison), with the reason logged.

Question statuses#

StatusMeaning
answered2+ independent sources, or 1 source of record; at least one priority source; no open contradiction
saturatedthe last 3 successful, result-recorded queries, of 2+ kinds across priority and secondary channels, added nothing; the coverage map shows relevant positions and explanations examined, and no contradiction is open
exhaustedno unseen sources after reformulations including primary routes
blockedsources exist but cannot be read, and all their retries are used up
droppedoutside scope, with the reason recorded

A sub-question more than 3 levels below the root needs a written justification. The system refuses “answered” without verified supporting evidence from two reviewed independent roots or one direct source of record, including priority-source support, and while a contradiction is open. Counts of unsuccessful or unrecorded searches do not establish saturation. A coverage assessment is a recorded judgment, not something a query count proves.

Failures and retries#

A page or search that fails for a technical reason (blocked, rate-limited, timeout, server error) is never dropped and never counted against an outlet. It is retried, by another route when the failure was a block. Deliberate bot walls are not worked around; another copy or the abstract is used and the limit is stated. While retries are pending, results are provisional.

3. Weighing claims#

Per-claim weight#

Weight is set per claim, not per source, from: how well the source fits the question kind, the outlet’s measured reliability for that statement type and subject, bias on this claim (lower when it favors the outlet’s side or owner, higher when it goes against its interest), evidence strength, and recency.

Evidence strength is recorded as record, strong, moderate, weak or assertion, with field scales: medical meta-analysis above trials above cohort studies above case series above expert opinion; science replicated peer review above single peer review above preprint above press release above commentary; political and factual primary record above a named on-record source above reporting citing documents above anonymous above unsourced. Three questions are kept separate: is the record authentic, does the claim accurately represent it, and does the underlying design support the substantive conclusion? A paper is a record of its own reported findings, not automatic proof of its causal interpretation; a statute is a record of its text, not of compliance. The record exception to the single-source confidence cap applies only to direct or attributed claims about the record, not unsupported extrapolation.

Own-interest claims#

A claim touches an outlet’s stake when it concerns a subject in the profile’s interests: its own government’s policy and performance, its owner’s businesses, its funders’ industries, its products. Such claims are weighed with the outlet’s measured accuracy on its own-interest items instead of its overall rate, and the bias discount follows the measured gap between the two: small when no gap is measured, moderate for a gap of 10-25 points, large above 25 points or when under a third of the outlet’s own-interest claims could be checked at all; a claim under a large discount needs independent support to rise above 50. With fewer than 8 checkable own-interest items the discount stays a stated judgment. Admissions against interest are raised, and documents an outlet publishes (law texts, datasets) are weighed by their authenticity.

Confidence anchors#

ValueMeaning
95+several independent, reliable sources with direct evidence agree; nothing credible against
80-94strong support, small gaps
50-79more support than not, but thin, indirect or partly contested
20-49more evidence against than for
below 20credible evidence against; reported as probably false

Confidence is a case-by-case probability judgment about the stated factual proposition, not a score for evidence quantity. Its written reason names the original evidence roots for and against, their reliability and stake, evidence strength, counterevidence and important missing material. Weak support alone does not establish that a proposition is probably false; when no probability is defensible, the assessment is explicitly indeterminate. No numeric confidence is accepted without verified evidence. One independent root cannot take a claim above 90 unless it directly establishes the record claim itself. Technical failures are not counted as falsehoods, but can make evidence insufficient or uncertainty wider. Probabilities remain judgmental until checked against resolved outcomes of the relevant claim type.

Base rates and how confidence is scored#

Before the evidence is weighed, the claims the short answer rests on (headline claims) get a base rate: the kind of claim or event each one is (its reference class) and how often such claims turn out true, preferably from a source (replication or retraction rates for a study design, how often an agency revises a figure), otherwise labeled estimated, or in words when no number is defensible. The record also says how the evidence moves the estimate from that starting point to the claim’s confidence. Claims that only report what a record or a person said get no base rate, with the reason stated. A report cannot be filed until at least one headline claim is marked and each has a current base rate; rewording the claim makes the record stale. The base rate is a starting point and a check, not a formula: the confidence remains the reasoned judgment described above, and the red team looks for headline claims whose confidence sits far from a sourced base rate with a thin explanation, or whose reference class was chosen to flatter the answer.

Language models rate their own answers more confidently than the same answers attributed to someone else (up to 26% in one 2026 study). Each confidence is therefore scored on the claim restated as another researcher’s submission, and the written reason says what the evidence shows rather than restating the claim.

Full numerical Bayesian updating (a likelihood ratio for each piece of evidence, multiplied out) is deliberately not used: the likelihoods for documentary evidence would be invented, multiplying them hides correlation between sources, and the result looks more precise than it is.

When a claim carries an outlet’s characteristic framing (“freak” event, “propaganda”), the factual core is recorded as the claim and the framing goes in the evidence note; the report shows the framing next to other outlets’ wording.

Contradictions#

  1. Is it real? Same definition, period, place, population, unit and measurement. If not, both values are kept and explained.

  2. Each side is traced to its origin; independence groups show how many independent sources each side really has.

  3. Resolution queries look for primary data, corrections, retractions, fact-checks, reviews and sources from neither side.

  4. Both sides are weighed (independent sources, reliability, bias on this claim, evidence strength, recency).

  5. A best estimate is recorded with confidence, the main reason and what would change it. The minority position stays in the report.

A contradiction still open after 2 iterations or 12 resolution queries is closed as unresolved, with a best estimate or an explicitly indeterminate assessment. Unresolved does not mean resolved: a material dispute remains attached to the affected finding, and if it could alter the headline, the short answer states the competing interpretation and its consequence. Time spent or repeated failed retrieval is not evidence for either side.

3a. Ethical analysis#

At scoping, every question is checked for an ethical component, asked for or not. It has one when the answer involves right or wrong, blame, justification or legitimacy, harm to people, rights and duties, fairness, a choice between policies, a contested practice, or a legal category with moral weight (genocide, terrorism, occupation). The finding is recorded with its reason as explicit (the question asks for a moral judgment, so the ethical analysis is the answer), side (the subject has an ethical dimension the question does not ask about), or none. For a side component the asker is asked one thing only, whether to include the analysis, with yes as the default; research goes ahead on the default. The red team later checks whether an ethical component was missed.

When the analysis is included, the agent writes its own ethical sub-questions into the question tree and researches them in the same iterations, with the same query kinds, statuses and quote rules as the factual ones:

  1. Which facts matter morally (taken from the factual research, with their confidence).

  2. Who is affected, and which ethical questions arise: harms, rights, duties, responsibility, proportionality, consent, fairness.

  3. What the main frameworks conclude: consequentialist; rights-based and deontological; contractualist and justice; virtue; care; plus the applied frameworks that fit, such as just-war theory and the humanitarian-law principles of distinction, proportionality and necessity, bioethics principles, or professional codes.

  4. What ethicists and other experts argue, what the affected parties argue in their own terms and languages, and what the religious and cultural traditions relevant to the parties hold (recorded as positions, not as arbiters).

  5. Where the frameworks converge and where they split.

  6. The agent’s own reasoned judgment.

An ethical sub-question is saturated when the main positions of each framework and each affected party have been found and further queries add nothing. “Ethicist X argues Y” and “framework X implies Y” are ordinary claims with verbatim quotes and a confidence. Surveys of moral attitudes count as evidence of what people think, never of what is right.

A moral judgment is not true or false the way a fact is, so it gets no percentage. It gets a status:

StatusMeaning
convergentthe disclosed relevant frameworks agree under the stated premises
predominanta documented body of relevant expert opinion leans one way, with population and evidence stated; not a vote among frameworks
contestedserious positions disagree without a clear balance
framework-dependentthe answer flips with the framework used

Each judgment also records the factual premises it rests on, each with its confidence; a judgment is never stated more firmly than its weakest factual premise allows. It also records the strongest counterargument, what would change the judgment, and what happens when each disputed premise is removed (fragile when the judgment flips). The agent’s own judgment is reasoned and labeled as the agent’s. Frameworks are disclosed with their relevance and selection rationale, not counted as votes: splitting one tradition into several schools cannot manufacture a majority. Where the frameworks converge it says so plainly, without false balance; where they split it gives conditional conclusions and explains the decisive values. Empirical expert consensus is reported separately from logical agreement between frameworks.

Controls:

  • Steelman: each side’s moral case is put in its strongest form, from its own sources and language, before it is assessed.
  • Symmetry: every principle is applied the same way to every party. A judgment that would change if the parties’ roles were swapped is revised.
  • Factual disputes and value disputes are reported separately.
  • The red team also makes the strongest moral case against each of the agent’s judgments.

A report cannot be filed until the steelman and symmetry checks are recorded, every moral judgment has its status and premises, and the red team has covered the ethics.

3b. Philosophical analysis#

Every question is also checked for a philosophical component, asked for or not. It is explicit when the question itself is philosophical: mind and consciousness, knowledge and evidence, what exists, identity, causation and free will, legitimacy, rights or justice as concepts, what morality is (metaethics), meaning, aesthetics, or the philosophy of science, religion or history. It is a side component when the answer depends on a contested concept or assumption the question does not ask about, such as what “communism” means in a question about China, sovereignty and self-determination in a territorial conflict, whether animals are conscious in a welfare question, or what counts as a cause in a history question. Whether something is right or wrong belongs to the ethical analysis; a question can have both. The finding and its reason are recorded, a side component adds one question (“Include a philosophical analysis in the report?”, default yes), and the red team checks for a missed component.

When included, the agent researches its own sub-questions in the same iterations and under the same rules:

  1. Concepts: what the key terms mean, their ambiguities and distinctions, and what the question presupposes.

  2. Facts that bear on it (scientific, historical), with their confidence.

  3. The main positions, each in its strongest form: analytic and continental traditions, non-Western traditions (Chinese, Indian, Islamic, African, Indigenous) searched in their own languages where relevant, and historical figures where they still matter.

  4. The best arguments and objections for each position, including the key thought experiments and what they rely on.

  5. What specialists hold. Surveys such as the PhilPapers Survey are opinion data, never proof.

  6. Where positions agree or split, which disputes are verbal (they dissolve once the terms are fixed) and which are substantive, and which conclusions depend on which premises.

  7. The agent’s own reasoned position, labeled as the agent’s.

A sub-question is saturated when every major position, with its main arguments and objections, has been found and further queries add nothing.

A philosophical judgment gets a status instead of a percentage:

StatusMeaning
consensusbroad agreement among specialists
predominantmost specialists lean one way
contestedserious positions disagree without a clear balance
openno position has decisive support
framework-dependentthe answer flips with the school or framework
verbalthe dispute dissolves once the terms are fixed

Each judgment records its argument as numbered premises and a conclusion with a validity check, its conceptual premises, its factual premises with their confidence (it is never stated more firmly than the weakest allows), the strongest objection, what would change it, and what happens when each disputed premise is removed.

Controls, all required before filing:

  • Charity: every position in its strongest form, from its own texts.
  • Consistency: the same standards of argument for every position, the agent’s own included.
  • Traditions breadth: the relevant non-Western and non-English traditions searched in their own languages, or their absence stated as a limit.
  • The red team argues against the philosophical judgments, and synthesis reviews may add philosophical questions.

Every question is also checked for a legal component, asked for or not. It is explicit when the question asks a legal question (“is X legal”, “is this genocide under the Genocide Convention”). It is a side component when the answer depends on a legal category (terrorism, genocide, occupation, war crimes, crimes against humanity), the legality of an action or policy, constitutional or treaty rights, regulation (ownership caps, antitrust, tax limits), or jurisdiction and enforcement. For a side component the inclusion question is “Include a legal analysis in the report?” (default yes). The agent researches applicable law and selects instruments; necessary factual jurisdiction questions are allowed, rather than silently choosing facts. The red team checks for a missed component.

The legal analysis is kept separate from the ethical and philosophical ones and sits alongside them. The legal one says what the law says and how courts and official bodies ruled, the ethical one whether something is right, the philosophical one what the concept means. For example, the legal definition of terrorism (Security Council resolution 1566, national lists, the UN comprehensive convention that was never agreed) sits next to the ethics of whether terrorism can be justified and what it leads to, with a line noting where they diverge (legal but wrong, illegal but defended).

The agent’s own sub-questions, researched in the same iterations:

  1. Which legal questions arise, under which jurisdictions and instruments (domestic law, treaties, customary international law).

  2. The text of the law and its definitions, quoted.

  3. How courts, tribunals and official bodies have ruled or found, each marked binding, advisory or pending, with the body’s mandate.

  4. What legal scholars argue, majority and minority.

  5. The legal arguments of the parties and of states.

  6. How the law applies to the facts, using the factual premises with their confidence.

  7. The agent’s own legal assessment, labeled as such. When a question concerns the asker’s own situation, it is analysis, not legal advice.

A legal judgment gets a status instead of a percentage:

StatusMeaning
settleda binding ruling, or clear and uncontested text
predominantmost courts, bodies and scholars lean one way
contestedserious positions disagree without a clear balance
untestedno ruling, or a novel question
pendinga case is under way
jurisdiction-dependentthe answer differs between jurisdictions

Each judgment records the jurisdictions and instruments, the provisions and rulings cited (each binding or not), its factual premises with their confidence, the strongest counter-argument, what would change it, and the premise test. A pending judgment is registered for follow-up with the case and an expected date, so the report is rechecked when the ruling comes. Nonnumeric legal follow-ups are tracked separately from numerical probability calibration and do not enter Brier scores or confidence-band tables.

Controls, all required before filing:

  • Primary text: every legal claim quotes the provision or ruling verbatim.
  • Same standard: the same legal test is applied to every party.
  • Currency: the law is stated as of a date, with pending cases and recent amendments checked.
  • The red team argues against the legal judgments too.

4. Outlet assessment#

Every outlet that supports or opposes a claim needs a profile before a report can be final. “Outlet” covers news organizations, wires, government bodies, parties, companies, journals, preprint servers, NGOs, think tanks, fact-checkers, reference works and official accounts; ordinary users and forum posts are never profiled. A host’s document passage can be weighed as the document and its author only after a saved comparison with a genuinely separate copy verifies authenticity; its own URL cannot serve as that comparison. This exemption applies to that document use, never to the host’s commentary or every use of the source. Document authors’ validity and claim-level evidence independence still need assessment. Outlets used only as references inside another assessment are not assessed.

When an outlet is assessed, and in what order#

  • New (no profile) or triggered (a change was found: new owner or controlling funder anywhere in the chain, new editor or agency head, change of government, major retraction or fabrication, a court, regulator or press-council ruling, an announced change of editorial line). Each use of a current profile runs one short search for such changes. A profile older than 180 days (and used since), one whose graded pool has doubled, one whose random share fell short or one whose error rate drifted is reevaluated while it stays in use; these reevaluations never block a report.
  • Outlets are assessed in order of how many questions are waiting on them: new and triggered outlets first, then top-ups, then stale profiles. Reassessments never block a report.

Article pool and sample size#

Outlets that carry claims in many research runs affect many conclusions. A run is a project in which the outlet supported or opposed at least one claim; runs rank outlets in reports but do not set the sample size. The sample is a pool of graded articles in two strata:

  • Random: items drawn at random, with a recorded seed, from the outlet’s archive of the last 24 months and graded by the assessor.
  • Encountered: articles research projects read. Each read, non-social source is graded from its project’s own evidence on the first claim it supports (chosen without regard to the outcome): accurate when a source from another outlet and another independence group, which was read, supports the same claim and no real contradiction concerns it; waiting for an explicit grade when the claim is in a real contradiction (graded false or misleading with a severity when the contradiction closes against it); not checkable otherwise. Not-checkable items are missingness, never errors. One article counts once however many projects read it.

Each checkable item weighs 1 inside the 24-month window and 1 / (1 + (age - 24 months) / 1 year) beyond it: half weight at three years, a third at four, about a tenth at ten, never zero. The estimate is the weighted rate over both strata with a Wilson interval on the weighted mass (the fresh-equivalent count), labeled as an approximation for pooled non-random data. Weights are recomputed whenever the pool is evaluated, which for a profile means at its reassessment.

A first assessment needs 12 fresh-equivalent checkable items. Random mass must stay at least a third of the pooled mass, measured in fresh-equivalents, at all times (capped at 72 of random mass); whenever it falls short, the shortfall is drawn at random. For an empty pool that is 15 draws, at the default 0.8 share of draws that turn out checkable. There is no precision-driven extension: a wide interval lowers the assessment confidence instead. The wording target is a third of the level sample (4 at 12 items, 8 at 24), and a third of the level sample among the random draws is own-interest (4 of 12).

The profile is reevaluated each time its graded pool doubles (12, 24, 48, 96, …): the figures are recomputed with current weights, ownership and the outside record are rechecked, and the extra wording comparisons run, without a new article sample unless the random share has fallen short; everything that used the outlet is re-weighed. A profile older than 180 days is reevaluated too, but only if research has used the outlet since; a stale profile stays usable and never blocks a report. A worse error rate after some date (a one-sided exact test across candidate dates, at least 8 items either side, adjusted for the number of dates tried) flags the outlet for immediate review.

An ownership change, merger or editorial change opens an epoch at its date. The earlier epoch is frozen with its profile figures: reports, claims and news stories that used articles from before the change are left as they were, with a note, and later articles are weighed under the new epoch, which until it holds 12 fresh-equivalent items borrows the earlier figures with confidence capped at low and a post-change mark.

Because encountered articles are selected by relevance, search ranking and how easily a claim can be corroborated, they are not a random sample. The random-share rule bounds their weight in the estimate; the two strata’s rates, their gap and a divergence flag (the random stratum’s interval excludes the encounter rate, in which case the lower rate is the headline) are always shown; subject and statement-type figures use pooled items with at least 8 per subject, which describes the claims research actually uses; a worst-case lower bound treats every not-checkable item as false. The encounter-minus-random gap across outlets is collected for later and not applied until 20 or more outlets allow a defensible correction and the method owner approves it.

The six parts of a profile#

  1. Identity and ownership. The chain from the outlet to its ultimate owner, each link with share, control type (majority, golden share, board control, minority) and source (filing, registry, annual report, regulator); stakes of 10% or more; funding, leadership, ties to governments, parties and companies; editorial and corrections policy. Investment and disposal notes in current accounts are read as well as headline ownership, because selling an outlet can leave an equity stake or board seat that still matters for own-interest claims; group control, immediate legal ownership and beneficial ownership are kept apart, and no link in the chain is inferred from a group schedule. A controlling owner gets its own group profile with its other outlets and relevant businesses (industries, contracts, donations, lobbying) and a grade of editorial independence (central, shared, independent, unknown) from evidence such as shared newsrooms, content deals, must-run segments and near-identical text across sister outlets.

  2. Accuracy from a random sample. The frame is the outlet’s items from the last 24 months, built from its archive, sitemap or feed rather than from search rankings (which favor popular stories), 100+ items where possible. The selection rule, the time and section coverage and known omissions are stated before bulk fetching (a large URL count from one day per month is not a census); fetching runs in small checkpointed batches that slow down or change route after rate limiting, and each selected item’s publication date and article text are checked, since a successful fetch can still return a consent page, navigation or a truncated text. Before any draw the outlet’s stakes are recorded by one rule for every outlet (ownership chain and owner’s businesses; for governments their own policies, officials, disputes and rivals; for companies their products, competitors and regulation; for NGOs and think tanks their funders and advocacy; for journals their own publications), and at least a third of the sample is drawn from items touching them. The draw is seeded and recorded. Each item’s main claim is graded: what the headline or lead is about, the factual claim an opinion rests on, or for an own-interest item the claim that serves the stake. It is checked against primary data first, otherwise against two references independent of the outlet and of each other (different owners; for political claims, different leaning), and graded accurate, mostly accurate, misleading, false or not checkable. Every error gets a severity: minor (the point stands), material (a fact a reader would rely on) or central (the story’s point rests on it). Not-checkable items are split into substantive (nothing exists to check) and technical (a page could not be read, which is retried and, if it never succeeds, replaced by a fresh draw from the same month and subject). The accuracy rate is (accurate + mostly accurate) / checkable with a Wilson 95% interval, reported overall (reweighted so the own-interest oversampling does not distort it), on own-interest items, on other items, and by statement type and subject.

  3. Outside record: retractions, corrections, court, regulator and press-council rulings, fact-checker findings, and media-rating groups as supporting evidence only, each with its method stated. A fact-checker that rates the outlet is one reference, not the verdict.

  4. Orientation from content: story selection, word choice, attribution and framing on the axes that matter for the outlet, each with strength and confidence. Outlets that cover a country’s domestic politics are also placed on that country’s left-right scale (far left, left, center left, center, center right, right, far right), or marked “not measured” when the sample has too few political items. Labels such as “propaganda outlet” appear only as attributed quotes, next to how others describe the same outlet.

  5. Wording compared with other outlets: randomly chosen sampled events are compared with coverage by other outlets, one picked at random from each bucket that covered the event (state domestic, state foreign, private domestic, private foreign, public service, wire, nonprofit, academic), at least 3 buckets and, for political events, at least 2 orientations. The profile describes patterns in the outlet’s terms in context; it does not rule which word is right. An event covered in fewer than 3 buckets is a real limit.

  6. Perception on social media, weighted least of all and never moving the reliability score (section 5).

Uncertainty and missing material#

The reweighted overall interval propagates uncertainty in both stratum accuracy rates and the estimated own-interest share of the frame. It uses a conservative envelope of four Wilson intervals with a Bonferroni allocation (approximate 95% coverage), not a plug-in normal approximation that collapses to zero width at all-success or all-failure boundaries. An unobserved stratum remains uncertain. This is a conditional estimate for the accessible, checkable material and the recorded sampling frame, not a guarantee about unseen material or every outlet output.

Original random draws are retained when replacements are added. Statistics report original and replacement counts, technical and substantive missingness, the replacement ledger, and worst/best outcomes for missing original items with a sampling-interval envelope. These sensitivity diagnostics describe the original unweighted sample, not a newly reweighted population estimate. Same-month or same-subject replacement does not establish random missingness. Material uncertainty in missing outcomes affects assessment confidence and sampling decisions; it does not regrade unread material as false.

Blind second grading#

A quarter of the random draws’ graded items (at least 3, seeded; grades derived from research evidence are rule-based and not double-graded) are graded again by a separate grader working from packets that hide the outlet’s name and the first grader’s verdict words. It may re-open the listed pages but runs no searches and never sees the first grades or the profile. Disagreements get a third blind look and the majority grade stands. Agreement under 80% (judged from 6 double-graded items) lowers the assessment’s confidence one level, and agreement is compared across outlets of different orientation as a check on the “same standards for every outlet” rule.

Reliability and assessment confidence#

ReliabilityMeaning
90+errors rare and corrected; high accuracy across types and subjects
70-89generally accurate; some errors or misleading items, usually in specific types or subjects
50-69mixed; frequent misleading or false items in some areas
below 50unreliable for facts; most checkable items fail or mislead

The score is set case by case from the sample rates and intervals, error kinds and severities, correction behavior, the outside record, and differences between subjects and statement types, with a written reason.

Assessment confidence says how much the profile itself can be trusted:

PartHighMediumLow
Pooled sample (fresh-equivalent checkable items)24 or more, interval within 25 points, random share met12 or morebelow that
Wording comparisons complete75% of the level target37.5%below that
Perception20+ posts on 2+ platforms from 2+ audience sides5+ postsunder 5

A second estimate runs alongside, for comparison only: a hierarchical Bayesian model treats each outlet’s accuracy rate as drawn from a distribution learned from the assessed outlets (outlets with at least 12 items; per kind once a kind has five such outlets), so small samples are pulled toward the group and large samples keep their own rate. The learned prior counts at most as much as a typical assessed sample and never assumes any outlet is perfect. It changes no score, confidence level or claim weight. Its first run (11 assessed outlets) put the prior at 95.6% accuracy and raised the lower ends of the outlets’ ranges by 3 to 14 points against the Wilson intervals; because nearly every sampled outlet scores close to 100%, neither method can yet tell the outlets apart from random samples alone.

Overall confidence is the lower of sample and wording; low perception caps high at medium, and blind agreement under 80% lowers it one level. Limits include a small or unrepresentative accessible sample, items with no independent record, material technical missingness, events nobody else covered, limited audience coverage, mostly unprofiled references, few checkable own-interest items and unstable grades. A failed retrieval is not an outlet error; it can still leave too little evidence to assess the outlet confidently. A profile is provisional while retries, a top-up, missingness review or blind grading are open, and usable with those limits stated. Completion is checked from the sample statistics and the confidence check together, not from a tool’s final label: no ungraded items, no further draw needed, the sample complete, no pending retries or top-ups, blind grades and adjudications done, and the wording comparisons complete or exhausted with their coverage limit recorded. A completed low-confidence assessment can be final; an unfinished high-confidence one cannot.

Identity checks#

Outlets that the daily news digests use only now and then get a quick once-over instead of a full assessment: one to three searches for owner chain, ties, editorial and corrections policy, bylines, and stakes. The score is a provisional base by outlet kind (wire 75, journal 75, fact-checker 70, reference 65, news 60, government 60, think tank 60, company 55, NGO 55, preprint 55, party 45, account 40), moved at most 10 points with a written reason, at low confidence. A full assessment replaces it once the outlet carries news stories on 2+ days, alone carries a story of impact 7+, or any research project relies on it.

Seeding#

So that coverage does not depend on which outlets the digests happen to cite, outlets can be queued on purpose: the local press of a place with a landscape report, or a balanced national list. A national list is chosen by audience reach from at least two independent measurements, banded by the median of at least two published bias ratings (for balance only; the assessment measures leaning itself), with at least 3 outlets per band (2 for the extreme bands, or as many as have national reach) and wires kept regardless of band. The current US list has 28 outlets.

5. Social-media perception and platform audiences#

Perception posts are collected from many platforms (Hacker News, Bluesky, Mastodon, Lemmy, readable right-leaning venues such as Free Republic, GETTR, Rumble and patriots.win, and Reddit and X when readable), aiming for audiences on at least two sides. Each post is recorded with its platform, its community (subreddit, Lemmy or Mastodon server, forum board) and the trust it expresses on a five-step scale (trusts 100, mostly trusts 75, mixed 50, mostly distrusts 25, distrusts 0).

Every platform has its own profile with an overall audience leaning and a breakdown by community, each with leaning, strength and confidence. A post takes its community’s leaning, else its server’s, else the platform’s. Two kinds of evidence are kept apart:

  • External: surveys and studies of the user base (Pew’s party split of each platform’s news users, academic studies), operator positions and community rules. High confidence needs a representative survey of users. Far left or far right needs the venue’s own rules or documented content, never party shares alone.
  • Internal, the trust gradient: for posts on outlets with a full profile and a measured left-right step, the gap between perceived trust and the outlet’s measured reliability, fitted against the outlet’s position on the scale. If an audience trusts outlets less as they move right at equal measured accuracy, it leans left, and the reverse. Comparing against measured accuracy keeps an audience’s leaning apart from real accuracy differences between outlets.

Platform profiles are not reassessed by age. They are made when an assessment first meets a platform (a community once it has 5 posts) and refitted from all posts gathered so far when the count reaches 30, 100, 300, 1,000, 3,000 and 10,000; the internal fit’s confidence rises with those steps.

In an outlet’s profile, each audience’s view is reported with its posts, communities, trust distribution and perception gap, labeled cross-audience (sides agree; the strongest signal), divergent, or aligned with the audience’s leaning. A platform with no posts about the outlet is a result only within the documented search scope; a search with no usable hits does not show the platform has none. Post dates are verified (from public post metadata when the page shows only a relative age), current posts are kept apart from historical context, every record keeps its exact excerpt and post link, and a comment about writing quality or political position that expresses no factual trust is recorded as “not stated”. Comments from one discussion are reported as clustered, self-selected observations, not as independent samples or a consensus across audiences. No view is removed: a reliable outlet that a far-right audience consistently calls lies or propaganda is reported as exactly that, with that audience’s gap. Summaries weigh audience sides equally, and perception is high only with posts from at least two sides.

6. Stopping#

RuleFires when
1sources exhausted
2coverage complete: every sub-question answered, saturated, exhausted, blocked or dropped, and no contradiction open
3two substantive, successfully researched iterations added nothing and raised no new questions, with coverage reviewed; empty iterations do not count
4a narrow factual root question got a decisive answer

A final filing is blocked while required outlets lack profiles, evidence is unverified, a claim lacks an explicit numerical or indeterminate assessment, or retries are pending. Provisional outlet-only filing is possible only after all other gates pass. Finishing rechecks readiness rather than trusting an earlier stop result. For rules 1-4 it also waits for these checks:

  • Synthesis review, first and as many rounds as needed: the agent reads the draft answer as a whole and asks the questions that only appear at that level: the mechanisms and causes behind the findings, connections and tensions between sub-answers, trends across findings, implications and what the findings predict, links the short answer relies on that no question established, and what an informed reader would ask next. They enter the question tree as normal sub-questions and are researched in normal iterations under the same rules. When the stop rules fire again, it reviews again. There is no round limit; it ends when a review after the latest iteration finds nothing more to ask, with the reason recorded. Convergence comes from the existing rules (repeats are refused, and every question must say how it serves the root question), not from a counter. Every round is listed in the report’s method section
  • Red team: a separate reviewer makes the strongest case against the saved draft answer in about 10-20 queries: definitions or framings under which it flips, contradicting evidence searched for directly (criticism, replication, opposing institutions, sources in the subject’s language), and questions the tree missed. With an ethical analysis it also argues against each of the agent’s moral judgments and checks for a missed ethical component. Its findings meet the same evidence rules as everything else; if it found anything, further research runs. Reviews identify the reviewer and draft, validate referenced claim and question IDs and recorded searches, and bind to the substantive evidence state. Changes to conclusions, evidence or decisive premises invalidate prior approvals even within one iteration; synthesis, red team and sensitivity are renewed as relevant, not merely once per project.
  • Ethics: when the report includes an ethical analysis, the steelman and symmetry checks are recorded and every moral judgment has its status, premises, counterargument and premise test.
  • Law: when the report includes a legal analysis, the primary-text, same-standard and currency checks are recorded and every legal judgment has its status, authority, premises, counter-argument and premise test.
  • Philosophy: when the report includes a philosophical analysis, the charity, consistency and traditions checks are recorded and every philosophical judgment has its status, argument, premises, objection and premise test.
  • Sensitivity: for every key claim (confidence 80+ or marked key), the conclusion is judged again with each original evidence root removed in turn; dependent outlets and shared underlying records count together. Numerical assessments report the revised confidence; indeterminate assessments report whether removing the root changes what can be concluded. A claim is fragile when one removal lowers confidence 20+ points or moves it across 80 or 50. Fragile claims go in the caveats, and the short answer says when a headline conclusion rests on one.
  • Base rates: every headline claim has a current reference class and base rate with the update from it, or a stated reason why none applies; at least one headline claim is marked.
  • Language: under 20% of sources or of key-claim support in the subject’s main language is a substantive limit, stated in the caveats.

7. Filing and reports#

  • The report comes with a data page (sources, question tree, claim and evidence tables) and a saved snapshot of every source read, with its link, date and verbatim passages.
  • The report gives a short answer, key findings with confidence, contradictions with best estimates and the minority position, caveats, and a method section. When it includes an ethical analysis, a separate section follows the findings: the ethical questions identified and why, the morally relevant facts with their confidence, a table of what each framework concludes, the positions of ethicists, affected parties and their traditions, where frameworks converge and split, each judgment with its status, premises, strongest counterargument and what would change it, the agent’s own judgment labeled as such, and the symmetry check. A legal analysis gets its own section next to these: the jurisdictions and instruments, the provisions quoted, rulings marked binding, advisory or pending, scholarly views, the parties’ legal arguments, each judgment with its status and authority, the agent’s own assessment, and a line on where the legal, ethical and philosophical answers diverge. A philosophical analysis gets its own section in the same form: concepts clarified, positions and traditions with their best arguments and objections, verbal versus substantive disputes, each judgment with its status and argument, and the agent’s own position. The short answer keeps its factual, legal, conceptual, philosophical and moral parts visibly apart, each analyzed part with its status. Reports list their analyses. The method section covers the question kinds, the organizing scheme and why it was chosen, what the red team tried and changed, unanswered clarifying questions with the defaults used, the language share, and pages that could not be read.
  • The caveats name every fragile conclusion (with and without the group it depends on), every outlet with 10+ runs whose confidence is still low and the limit behind it, and the language share when it is a limit.
  • Provisional reports: when only outlets without a profile remain, the report is filed. Claims resting on unprofiled outlets are marked and weighed by kind, and the pending profiles are listed. Each new profile re-weighs those claims and updates the report, the data page and the filed findings together. A change that moves the short answer or a key claim across 80 or 50 gets a dated line in an “Updates” section. When every outlet has a profile, the report becomes final.
  • Claims that will resolve later (forecasts, routinely revised figures, pending disputes) are registered for follow-up. Numerical factual predictions retain their original probability for calibration; nonnumeric legal judgments are followed up without numerical scoring. With 50+ resolved numerical predictions, a table shows how often each confidence band came true; a band off by more than 10 points (with 10+ claims) prompts a review of the anchors. Results are separated by claim type where the counts support it; completed answers alone are not calibration outcomes.

8. Changes propagate#

Standing rule: when anything a result rests on changes, everything that used it is updated.

  • A filed or changed outlet profile re-weighs its research claims (and the reports and findings built on them) and every news story that used the outlet.
  • A changed owner-group profile flags its member outlets for reassessment.
  • A created or refitted platform profile updates the Perception section of every outlet whose posts came from it. If that changes an outlet’s confidence, its claims and stories are re-weighed in turn.
  • The outlet rankings page is regenerated with every profile change.
  • Daily news digests weigh each story’s confidence with the same outlet profiles and rules. Dependent copies count once, and own-interest reporting is discounted. A story reaches 80+ only with a wire or a profiled outlet among its sources, and a single source stays at 90 or below unless it is the record.

10. Neutrality controls#

  • The same standard applies to every outlet: one stake rule, one pool minimum and random share, one set of grade definitions. A Western government’s statements on its own policy are stake items exactly as any other government’s.
  • Random draws are seeded from archives, not from search rankings, and encountered articles stay in their own stratum.
  • Accuracy and orientation are reported separately; leaning never lowers a reliability score.
  • Blind second grading masks the outlet’s name, and agreement is compared across orientations.
  • Wording comparisons draw outlets from several ownership buckets and orientations at random.
  • Perception is read per audience, with every audience’s view kept and sides weighed equally.
  • Normative questions get equal search effort for each side, in each side’s language, with moral judgments labeled.
  • Ethical analysis puts each side’s moral case in its strongest form first, applies every principle to every party the same way (the roles-swapped test), and keeps value disputes apart from factual ones.
  • Philosophical analysis puts every position in its strongest form from its own texts, holds every position (the agent’s own included) to the same standards of argument, and searches the relevant traditions in their own languages.
  • Legal analysis quotes the primary text for every legal claim, applies the same legal test to every party, and marks each ruling binding, advisory or pending.
  • Seeded outlet lists are balanced across bands, and the least-profiled band goes first.

11. Known limits#

  • Some platforms cannot be read by this system at all (Reddit and X), and others show only snippets or no comments without a login. Perception therefore rests on the platforms that can be read, and right-leaning venues are still few in the samples; perception confidence is capped when posts come from only one audience side.
  • The blind second grader is the same model as the first by default, so the two grades are not fully independent. A different model would be a stronger check and is recorded when used.
  • The trust gradient only works once outlets at several measured left-right steps have full profiles. Until then, platform leanings rest on external surveys.
  • Calibration needs many resolved claims before it says much, and resolvable claims are mostly forecasts and statistics. Each confidence band carries a Bayesian estimate of its true share with a range that is wide while few claims have resolved, and a band is flagged for review when that estimate is at least 90% likely to be more than 10 points off its stated confidence, which can happen before 50 claims resolve. No claim had resolved by 2026-11-18, while 83% of the 261 scored claims across projects stood at 80 or higher.
  • Encountered articles are selected, not random, and so far most are not checkable: the first derivation graded 17 of 107 accurate, left 83 not checkable and 7 waiting for a contradiction to close. The pool therefore fills slowly from research, most outlets still need about 15 random draws for a first assessment, and the age-decay constants, the one-third random share and the 72 cost cap are judgements, not fitted values.
  • A blocked kind of page (for example paywalled investigations) replaced by readable ones may bias a sample. Original draws and replacements stay linked, accessible-sample accuracy is labeled as conditional on retrieval, and missing-item sensitivity is reported rather than assumed away.
  • Deferred evaluation: after several dozen fully assessed outlets and several completed questions, review within-outlet precision by subject, claim type and own-interest; independently double-graded agreement; calibration from independently resolved outcomes by claim type; and synthesis/red-team yield and stopping effectiveness. The amount of relevant evidence, not a profile-count milestone alone, determines whether tuning is justified.

Changes#

Dated list of method changes
  • 2026-11-15: Ownership is read from investment and disposal notes, with control, legal and beneficial ownership kept apart; the sample frame is stated before fetching and fetched dates and texts are checked; perception results are limited to the searched scope, with verified post dates and exact excerpts.
  • 2026-11-15: Evidence sufficiency is separated from the probability of truth; verified passages, original evidence roots and claim-specific warrants are required; reviews are bound to the draft they reviewed; technical missingness and replacement provenance are preserved; uncertainty intervals are boundary-safe; framework conclusions are conditional.
  • 2026-11-15: First version of this page.
  • 2026-11-15: Ethical analysis added (section 3a): moral judgments take a status instead of a percentage, with factual premises, strongest counterargument and a premise-removal test; steelman and symmetry checks are required.
  • 2026-11-15: Synthesis review added before the red team: repeated follow-up rounds on the findings as a whole until a round adds nothing.
  • 2026-11-15: Philosophical analysis added (section 3b), on the same pattern as the ethical analysis.
  • 2026-11-15: Legal analysis added (section 3c), separate from the ethical and philosophical analyses, with its own statuses and checks.
  • 2026-11-17: Outlet accuracy now comes from a pool of graded articles: random draws from the outlet’s archive plus articles read in research, graded from research evidence. Older articles count for less but never drop out, a first assessment needs 12 fresh-equivalent items, reevaluation follows pool doubling and a 180-day review, and an ownership change opens a new epoch that leaves earlier reports untouched.
  • 2026-11-18: Headline claims get a base rate before filing; each confidence is scored on the claim restated as another researcher’s; the calibration table carries a Bayesian estimate per confidence band; and a hierarchical Bayesian estimate of outlet accuracy runs beside the existing intervals for comparison only. Full numerical Bayesian updating of claims was considered and rejected.