We can measure media reliability, and we should

The Free Press is 99.3% reliable, while the Daily Wire, Fox News, and the Washington Post are 99.1%, 98.2% and 82.4% reliable, respectively. These are the percentages of headlines and articles that did not contain a false claim, according to the Media Reliability Project’s systematic study. The Post’s reliability falls to 62% for its fact checks specifically, and 44% for fact checks unfavorable to President Trump — more than a hundred fact checks claiming that Trump said something false were themselves false, while thousands turned out to be invalid, addressing opinions or claims that cannot be fact-checked. Grok 5 and Gemini 4 were used to confirm the findings.

I’m developing a project to systematically assess the reliability of major media outlets. Surprisingly, no one has done this. The above fictional passage is an example of the findings we should be able to make.

We don’t have any data on the reliability of the New York Times, ABC News, the Free Press, CNN, Politico, Fox News, the Washington Post, the Daily Wire, etc. There’s nowhere to go to see a rigorous assessment of their reliability. There’s no report (preferably live/updated). There’s nothing.

Legacy/leftist media in particular make lots of false claims, and while those claims are regularly exposed on social media, no one is comprehensively sampling and assessing their headlines, articles, and TV segments for false claims and tabulating that data. Without such a rigorous approach, there’s no way to hold them to account or to establish which outlets are more reliable.

What do I mean by reliability? I mean the inverse of their false claim rates. This is fundamentally about false claims. Let’s focus on headlines specifically for a moment. An outlet’s reliability with respect to headlines would be the percentage of their headlines that are not false.

The project will also include full coding of article text and TV segments, but headlines are worth assessing separately because headlines are all most people see and coding headlines is a good Phase 1 study before tackling entire articles.

Right now, unreliable legacy/leftist media outlets are favored by search engines, Apple/Google News, Wikipedia, and emerging AIs, while non-leftist outlets — which I predict are more reliable — are marginalized. This erodes our connection to reality — false claims are laundered, legitimized, and heavily repeated while inconvenient facts are suppressed.

This project will yield a treasure trove of scientifically rigorous, unimpeachable data, with which we can address claims that legacy/leftist media are bulwarks against “misinformation” (claims embedded in public school curricula on “media literacy” and academic misinformation research), pressure media to correct false claims and biases, and tackle the problem of unreliable leftist media being used as reliable sources for web search, AIs, Wikipedia, etc. before it’s too late.

To properly make the case that legacy/leftist outlets are unreliable and to upgrade the quality of our information sources, we need hard data, rigorously and transparently collected and coded. We can cite plenty of examples of false reporting, but to prove which outlets are more or less reliable, we need systematic data collection covering leftist and non-leftist outlets.

If you think false reporting from major outlets is exceptionally rare, well let’s just take a look.

Here’s a false headline from the New York Times. It’s still up, uncorrected:

The executive order does not bar “transgender student-athletes” — a category that includes women — from women’s sports. It seeks to bar men from women’s sports, whether they’re “trans” or not. The key passage directs the Secretary of Education to:

prioritize Title IX enforcement actions against educational institutions (including athletic associations composed of or governed by such institutions) that deny female students an equal opportunity to participate in sports and athletic events by requiring them, in the women’s category, to compete with or against or to appear unclothed before males. (bold added)

Therefore, the headline is simply false in two respects:

1. The category “transgender student-athletes” includes women (who “identify as” men, “nonbinary”, or a newer construct) — they would not be barred from women’s sports, since they’re women.

2. Men would be barred regardless of whether they identify as trans, so again “transgender student-athletes” doesn’t map to the EO. For example, the two male boxers who beat up women at the 2024 Paris Olympics aren’t trans — they’re just men. Trans and “gender identity” are new leftist constructs — non-leftists don’t need them for anything and the EO doesn’t need or rely on them.

Virtually all legacy/leftist media outlets published versions of this false headline claiming that the Trump administration was banning trans athletes from women’s sports, or even from sports as such — it was a systemic collapse of journalistic standards, a collapse that extends to coverage of gender ideology more broadly. These headlines would all be captured by the reliability study as false.

---

False impression = false headline

Here’s another example:

I chose this because it’s an interesting, more complex case. Technically, each claim in the headline is accurate, but it’s false in suggesting that she was sentenced for ending her pregnancy. She was sentenced for burning her fetus and pled guilty to concealing or abandoning a dead body. (She aborted her child during the third trimester.) By contrast, NBC News furnished a clean, honest headline:

The principle we could apply here is: When a headline reports an arrest or sentencing in conjunction with an act that is not the act for which the person was arrested or sentenced, the headline is false because it misleads readers to believe that the act in the headline is the basis for the arrest/charges.

I think we can convincingly argue for this principle, and nest it in a broader principle: Does the headline give readers a false impression? From that foundation we can build subsidiary principles like one regarding how arrests are reported. We could even run some studies to establish that readers get a false impression from certain headlines, but in the above NYT example I think any reasonable person would agree that no study is necessary.

The NYT biased it even more by calling an adult a “teen”, which most people will assume is a minor — many readers will not see the sub-headline giving her age, depending on the platform or news app in which they encounter the headline.

Readers who see only the main headline will come away with the impression that “red” states are jailing teens for using abortion pills, presumably very early in their pregnancies (which is when abortion pills are prescribed). That’s wildly different from the reality here of a young woman who aborted her child during her third trimester and burned and concealed the corpse, the act for which she was jailed.

ABC: The principle outlined above can be broadened to capture a lot of biased or false headlines. The core structure here is that A does B because of C. What legacy/leftist media often do is conceal C and replace it with something generic or even irrelevant. Here A is [Nebraska], B is [sentenced this woman to 90 days of jail], and C is [for burning and concealing the corpse of her third-trimester fetus/baby]. Obviously intending to deceive their readers, the Times concealed C and replaced it with the fact that she took abortion pills.

This corruption of the basic ABC structure is a common media tactic and underexposed.

We should also pause here to note that the New York Times is plainly corrupt and should not be cited or treated as a credible news source. And yet, it is treated as credible by Google search, Apple News, Wikipedia, the various AIs, etc. The goal of this project is to fully expose such organizations with scientifically rigorous data so that they can be marginalized and replaced with more reliable sources.

The Denominator

Since I’m operationalizing reliability as the percentage of headlines, articles, or segments that do not make false claims, a key question is the choice of denominator — the set of headlines to include and code as false or not. Since this project is focused on political bias and reporting, only politically relevant headlines will be included. This would include headlines mentioning a politician, party, branch of government, elections, political issues, and so on. The criteria will be refined over time based on what we learn. Leftism is unique in holding that “everything is political”, so we sometimes find leftist ideology and narratives embedded in sports, entertainment, or even cooking articles, but the vast majority of the reporting we’re interested in will be in the normal news and politics beats.

We can also refine the datasets to answer more specific questions. For legacy/leftist outlets, I predict that their reliability will be lowest for headlines and articles that mention Trump or Republicans, or intersect with leftist gender, race, or climate ideologies. For conservative outlets, I predict that their reliability will be lowest for headlines that mention Democrats and leftists (though I think we’ll find exceedingly few false claims at outlets like the Daily Wire and Free Beacon — I’ll be impressed if you can find one in an hour of searching).

For centrist outlets like the Free Press and Quillette, I have no pattern predictions, though I expect them to do well in general. Consistent with the opening passage, I predict that conservative, centrist, and libertarian outlets will prove significantly more reliable than legacy/leftist media, but perhaps I’m wrong.

Other efforts

NewsGuard

No one is systematically assessing media outlet reliability. NewsGuard, a leftist NGO and censorship effort produces “Trust Scores” using a secret subjective rating process by a handful of activists. They claim to apply various objective criteria, but they don’t disclose their actual process, rubrics, sampling methodology, or any details for a given outlet’s scoring, nor do they release their data, so there’s nothing we can do with their “Trust Scores”. (Their process might be so informal that there is no “data” per se.) In fact, they don’t even disclose the scores — you have to pay to see them (though they’ll sometimes disclose a score in their newsletters).

For conservative outlets, they seem to home in on whatever leftists are calling “misinformation” at the time and rate the outlets poorly if they promote the purported misinfo, even when the topic is a contested, complex scientific and policy issue like COVID mitigation strategies. Ironically, NewsGuard is an enforcer of misinformation – they swung into action to harm media outlets that shared or promoted the COVID lab origin theory (which is almost certainly correct) and they assigned an unqualified lay activist to “fact check” the Great Barrington Declaration website. NewsGuard abandoned its purported role as a media watchdog and took partisan political stances in favor of totalitarian COVID policies.

For leftist outlets, they simply ignore their notorious false claims – e.g. the Trump-Russia collusion hoax, the Hunter Biden corruption emails/laptop billed as Russian disinfo, COVID lab origin as a debunked conspiracy theory, denying that children are subjected to mutilating “gender” surgeries (over 6,000 kids so far), COVID vaccine efficacy, claims that all “transgender athletes” are being banned from women’s sports, that atmospheric CO2 is toxic, several hoaxes attributing the slaughter of Gazan civilians to the IDF, unscientific subjective opinions like a state of climate “crisis/emergency” treated as journalistic background facts, etc.

For example, NewsGuard gives the notoriously unreliable leftist outlet PolitiFact a perfect 100 Trust Score. Even on grave matters like the surgical mutilation of children, Politifact exhibits no trace of rigor, truth-seeking, or any effort to mitigate their leftist bias – they claimed there were only two examples of American children receiving such surgeries. (Their “fact check” is still uncorrected, as are hundreds of their other false fact checks.)

NewsGuard’s co-founder even pushed the false claim that the Biden emails/laptop was Russian disinfo. So we see again that NewsGuard is not the impartial watchdog they claim to be — they’re part of the broader leftist/Democratic Party machine and their beliefs about what is “misinformation” at any given time are exactly whatever leftists are calling misinformation at that time, whether it’s COVID lab origin, the Biden laptop, or the view that lockdowns were net harmful, unwise, or unethical (the last being a philosophical and political appraisal, not a falsifiable matter of fact, but that didn’t stop NewsGuard from calling it “misinformation”).

Critically though, it’s not possible to validly assess a media outlet’s reliability without using scientifically valid sampling of the outlet’s reporting – either gather all relevant headlines and articles for a given period and rigorously assess them for false claims or take a large random sample of those headlines/articles. NewsGuard doesn’t do that — they look for whatever leftists are calling misinformation at any given time, focus on that handful of topics/narratives, don’t disclose their detailed methodology, then generate a subjective opinion rating of the outlet. That’s not anything, and there’s nothing we can do with the results of such recreations.

We don’t know the specifics of their methodology or if they even have a formal methodology, but we know they’re not coding thousands of headlines/articles and they don’t look for leftist misinfo. Since they’re a leftist group, they wouldn’t be able to competently rate media reliability. Their staff will have the same false beliefs as the staff at other leftist NGOs and media outlets and the same authoritarian leftist epistemology that would look to a random, two-year-old government webpage to answer a scientific question like whether masks reduce COVID spread instead of looking directly at scientific evidence. Would NewsGuard staff catch the false NYT headlines above? Almost certainly not, which means they cannot rate media reliability.

Media Research Center

The Media Research Center (MRC) is a conservative media watchdog. They do good work assessing a few forms of leftist bias in media and in aggregators like Apple News. For example, they reported that in January 2026, Apple News promoted 440 stories from left-leaning outlets and zero stories from right-leaning outlets.

However, the MRC doesn’t assess the reliability of media outlets. They have no reliability data for the various left-leaning, right-leaning, or centrist outlets. What if the left-leaning outlets are more reliable? Since MRC can’t show that the NYT, Washington Post, and the Guardian, for example, are less reliable than the Daily Wire, Fox News, and the Free Beacon, their complaints about bias from the likes of Apple or Google are somewhat weakened.

The issue is that the decisionmakers at Apple and Google likely believe that their favored legacy/leftist outlets are more reliable than non-leftist outlets. Since no one has stepped up to do the rigorous, comprehensive research it takes to determine the comparative reliability of the various outlets, leftists have been able to get away with simply asserting that their outlets are more reliable. (They’ll also contend that leftist outlets like the NYT or Washington Post are not in fact leftist or biased, another question that can be settled with rigorous research.) They might even cite invalid, unscientific, secretive, opinion-based ratings by leftist NGOs like NewsGuard and Media Matters, or fraudulent research by leftist academics.

Since neither MRC nor anyone else has rolled up their sleeves to rigorously assess media reliability, we can’t refute their claims and institutional leftists have been able to carry on as though the New York Times, Washington Post, Guardian, et al. are reliable and “mainstream” bastions of journalism. It’s long past time to put these beliefs to the scientific test.

Yes, anyone claiming that leftist outlets are more reliable is also unable to support their contention with rigorous data. For example, Wikipedia didn’t commission a study to generate their Reliable Sources list — no, it’s based on a “consensus” of editors, i.e. the opinions of leftists. (And any study they might commission is unlikely to be valid.) Google, Apple, Anthropic, et al. would have the same difficulty objectively measuring media reliability that NewsGuard does — the people involved would likely be leftists and they wouldn’t be able to do the job. (A good test would be whether they’d notice the false NYT headlines above without prompting or clues.)

However, the fact that these companies and Wikipedia haven’t demonstrated that legacy/leftist media is more reliable than non-leftist outlets doesn’t stop them from favoring them. It’s not fair that the burden is on others to prove them wrong, but life isn’t fair and we do what we must.

AllSides

AllSides uses a handful of balanced partisans to rate the left-right bias of various media outlets. They don’t assess reliability or false claims. They seem like a smart, well-intentioned team, but ultimately bias should be objectively measured, not voted on by a handful of people. My project aims at measuring reliability first, but it will also give us a treasure trove of data that we can use to objectively measure an outlet’s bias. For example, see these NYT headlines:

Both men were former US Senators and both died in the same year (2024), so it’s a tightly controlled example of bias at the Times. This specific type of bias — derogation of the dead in headlines depending on their political affiliation — can be straightforwardly assessed by objective methods. We’d gather all headlines on the deaths of political figures, code the political orientation/affiliation of the fallen, and code the headlines for derogatory content. We’ll probably find opportunities to further break down derogatory content into subcategories, and we could also look at praise and positive content (e.g. milestone headlines — “first X to achieve Y” as with Senator Lieberman).

Here’s an example of what I term Front Page Bias, a form of aggregate bias in front page headlines.

Shooting a rare duck? It would be hard to justify putting that story at the top of their homepage, and the whole package of headlines is obviously extremely biased. These outlets know that most people only see the headlines of stories, and they’ve optimized their homepages so that skimmers will see a plethora of derogatory headlines about Republicans/Trump, even a prediction that Trump’s “purge” will cause another 9-11… (This screenshot is from February 5, 2025.)

We can objectively measure Front Page Bias, along with many other forms of bias.

Closing

The bottom line is that American media makes lots of false claims, and this needs to be captured and documented. A civilization like ours should have reliable media. We should also know exactly how reliable our media outlets are. This project will be transparent and open to the public: the methodology, rubrics, AI tuning, and data will all be open source. Every single headline, article, and TV segment included in a rating will be available, along with the coding. If something was coded as false, the explanation and nature of the false claim will be right there for everyone to see.

There will be a page for every outlet, a dynamic report with interactive dashboards. This is critical — we need to be able to point people to the complete report and data for any outlet, make it a click away¹.

Coding headlines and article bodies for false claims is somewhat harder than it might seem. There are tough epistemic issues to work out in adjudicating that something is “false”. I started developing this project on the assumption that it would require well-trained human raters and I’m now exploring how we can automate as much as possible with well-trained AIs. Regardless of how much AIs are able to do, this project will be a fantastic challenge in applied epistemology. We’ll operate at much higher standards of rigor and validity than academic “misinformation” researchers — it will be good to demonstrate the contrast.

I’ll be working on the project methodology and funding for the next few months. Systematically assessing media reliability seems like the most obvious thing in the world to me and it’s confusing that it hasn’t been done. We need to be as well connected to reality as possible, because reality is where we live. This project aims to strengthen that connection.

---

Joe Duarte grew up in small copper mining towns in Southern Arizona, earned his PhD in social psychology, and focuses on political bias in media and academic research. You can find him on X here and contact him at gravity at protonmail.com.

1. One of the limitations of the MRC’s approach is that they don’t gather all their findings for a given outlet in one place. The MRC doesn’t have a page/report for CNN, ABC, Fox, etc. Their findings regarding a specific outlet are scattered across blog posts and social media posts.