Cracking Down On Fake News since 2017

The Rating

Here is how we calculate our ratings for every article. The first section describes the preliminary work we have manually done and the structure of our database. The second part explains the automated rating process our algorithm goes through every time a new article is sent in. This page takes you through the rating process for a sample article. You can read the original article here.

Preliminary Ratings of News Sources

We chose the list of news sources on the right from Alexa's top news websites, using an equal number of right-leaning and left-leaning news sources. For each calendar year from 2012 to 2016, we randomly chose two major United States news stories that were widely covered and rated each organization's article on each story. We gave every article five scores from 0-10 for the:

  1. Factual accuracy and objectivity of the headline

  2. Objectivity of the article itself

  3. Usage and frequency of quotes from relevant and reliable sources

  4. Factual accuracy of the main story

  5. Factual accuracy of the supporting material

We use these ratings to shed light on typical reporting by each organization in the process of assessing individual news articles.

We do not intend to publish these ratings because they are preliminary and subject to change as more articles come in.

We rated:

  • CNN

  • New York Times

  • The Guardian

  • BBC

  • Washington Post

  • Huffington Post

  • Fox News

  • Bloomberg

  • Reuters

  • USA Today

  • Wall Street Journal

  • The Atlantic

  • The Hill

  • Time

  • New York Post

  • NBC

  • Chron

  • LA Times

  • US News

  • CBS News

  • SF Gate

  • The Daily Beast

  • News Busters

  • Twitchy

  • Free Beacon

  • Young Conservatives

  • American Thinker

  • The Economist

  • NPR

  • Slate

  • Newsweek

  • Politico

  • Salon

  • Indy Media

  • The Village Voice

  • Daily Kos

  • New Yorker

  • Alternet

  • Common Dreams

  • Crooks and Liars

  • Talking Points Memo

  • Mother Jones

  • Counterpunch

  • The Blaze

  • Breitbart

  • IJ Review

  • News Max

  • WND

  • Daily Caller 

  • Washington Times

  • National Review

  • Town Hall

  • Hot Air

  • CNS

  • Western Journalism

  • Washington Examiner

What's in the Database

The News Source Ratings

For each news source, we store weighted arithmetic averages of the all the ratings we have for their articles. We take into account whether organizations use material from the Associated Press in each article when calculating our averages. The rating averages for each news source are used to evaluate all of its articles. As more and more articles are scored by our program, the scores will be used to update our assessments of each organization. As a result, the accuracy of our ratings for each news outlet will improve over time, and organizations whose articles we haven't fact-checked manually will still be given credibility ratings. The usage for these ratings in scoring individual articles is explained below.

The Collection of News Stories

We store all article ratings that have been automatically produced by our algorithm (the articles rated in our preliminary step are not stored). We organize these news stories by headline. How we use this rating collection and how it is expanded are explained below.

The Steps

Step 1: The Headline


After the URL of the article is sent to our central server, the server first checks to see if the database already contains articles reporting on the same story. If the number of matching articles stored is none or very few, the server then conducts a web search using the headline as a query. Based on the results, our algorithm takes into account the following:

  1. ​The number of news sources reporting on the same story, determined by date filtering and looking for matching key words between headlines.

  2. The stored ratings we have for the news outlets that are reporting on the same story.

  3. The rating for any relevant article that pops up in the search that the server has previously rated.

These three factors, along with the bias detected in the headline's phrasing, determine the first component of our rating: the headline score. A higher number of credible news sources reporting on the same story obviously makes an article more likely to be true, resulting in a higher headline rating.

Sample Article

We will illustrate this step of the process via a Google search (click image to expand). Note that the top four results are news articles reporting on the same story. While some articles have far more supporting headlines, the fact that four outside news organizations reported on the same story makes it likely that the article's main claim is accurate. However, the words "unhinged" and "rant" indicate substantial bias in the headline, resulting in a headline rating that is mediocre overall.

Step 2: Quotes/References


The server then scans the article's text for quotes and references, along with their possible sources, by searching for quotation marks, key words, and capitalized words. An article will receive a higher score if it contains a significant number of relevant and accurate quotes and references trustworthy sources. When covering a controversial topic, articles are rewarded for quoting speakers with different, and perhaps conflicting, views on the subject at hand.

This article does contain several lengthy quotes from Mr. Olbermann himself. The truth of these quotes is supported by other news articles reporting on the same story, lending credibility to the idea that Mr. Olbermann actually spoke about Mr. Trump.

Step 3: Spin


The server then looks for evidence of possible opinion insertion from the author. Evidence so far takes on three different forms:

  • Certain words and phrases, especially adjectives (e.g. "ridiculous," "stupid," and "brilliant"). The algorithm considers that these keywords are more likely to suggest bias when located outside of quotes.

  • Sentence length, as shorter sentences tend to indicate more prevalence of opinion and lack of nuance.

  • Sentiment analysis - the more clearly positive or negative the author's overall tone is, the more it interferes with an unbiased reading of the article.

  • Sentence structure

Words in this article such as "unhinged" (in the headline), "torched" (in the first line), and "yet" (both in the first sentence and in the headline) suggest that the article is trying to portray Mr. Olbermann in a bad light. This news source makes no indication that this is an opinion article (which we would not rate), so based on our detection of bias we deduct from the article's credibility rating.

Step 4: Supporting Facts


Now that the server has checked the accuracy of the headline and searched for bias, it moves on to verifying the supporting information. First, the server verifies any statistics or quotes provided by scanning for them in similar articles.

Next, by searching for connections between proper nouns and entities, the server finds the five most important fact-based claims within the article. It then searches the text of articles reporting on the same story to verify each claim. If a claim has substantial supporting evidence in other articles, it is deemed fully verified and given a perfect accuracy score. Otherwise, it is labeled either partially verified or completely unverified and accuracy deductions are made accordingly. The accuracy ratings for statistics, quotes, and each of the five claims are combined into a single supporting fact accuracy score. 

The Blaze's article on Mr. Olbermann does not contain many supporting facts to verify; most of the article consists of quotes and there are no numbers or statistics to check. The supporting facts that are included, however, are mostly accurate and can be verified on other websites: the quotes are all real and Olbermann did speak on "The Resistance." Therefore, the article would receive a high supporting facts rating, although the server would recognize and take into account the fact that this judgment was reached by looking at very few pieces of information.

Step 5: Overall Rating


The conclusions (which are done empirically and numerically) of the first four steps are aggregated to produce a final rating from 0-10, which essentially represents the extent to which the article represents the facts of the story as accurately as possible in the most neutral tone possible. Accuracy and spin subscores are also calculated (on a scale from 0-10). Generally, ratings from 8-10 mean that the article is generally unbiased or trustworthy, 6-8 indicates that the article should be read cautiously, and below 6 suggests that there is a deliberate attempt to mislead. These ratings are sent back to the client, tweeted, and stored in our server.

The Blaze's article on Mr. Olbermann would receive a mediocre rating for the headline, a good rating for quotes and references, a poor rating for spin, and a good rating for supporting facts. The overall score would be mediocre, as the high accuracy subscore would be counterbalanced by the low spin subscore. Our final algorithm will conclude and inform the reader that while there may be factual basis to the article, it is quite likely that the author is trying to spin the article and make the reader react a certain way. Thus, if the reader should choose to read the article, he or she should keep an open mind and be wary of such spin.


A few disclaimers about our rating system:

  1. Our rating system does not seek to absolutely determine whether or not every fact presented in an article is factual. We are progressing towards that stage and currently check an article's most important facts, but for now our scores represent merely the algorithm's best estimate of the extent to which an article is truthful and unbiased.

  2. Our system does not check the factual accuracy of quotes. For example, if an article reports that a famous figure made some sort of erroneous claim, the system might give the article a high score if multiple other sources report that the same erroneous claim was made by that figure. Thus, a reader should keep in mind that the rating only supports the making of the claim, not the claim itself.

  3. Our rating system is not perfect; a completely accurate and unbiased article will most likely not receive a perfect 10. Score differences between articles of up to 0.5 are typically not especially significant. However, flawless articles will almost always receive ratings between 8.5 and 10, while biased and inaccurate articles will be rated well below 6. When comparing scores, it helps to think of them in ranges rather than considering them absolute quantities.

  4. The methodology detailed above is not static and is subject to change as we continue to improve our algorithm. We expect news sources to find ways to work around our system and produce inaccurate or biased content that receives a good rating. We will update our system accordingly.