What We Learned About Editors vs. Algorithms from 4,000 Stories in Apple News

Original article was published on Artificial Intelligence on Medium

Background

Editors and algorithms at Apple News choose stories for millions of people every day, and that number is growing quickly. In fact, the app has climbed to over 125 million monthly active users. That number is on par with Flipboard, a similar app with 145 million monthly users, and is about half as many users as Google News (based on estimates).

So what is Apple News showing to these millions of people every day?

Is it showing a diverse set of sources, as it claims, or do most stories come from just a few select places? And what are the stories about? Last year, we tried to address these questions by collecting stories from Apple News for two months. I wrote a program that saved stories from the app every five minutes — here is what it looked like running on February 27th of this year:

In total, we collected over 1,268 “Top Stories,” which are curated by human editors, and over 3,144 “Trending Stories,” which are curated by an algorithm. Our analysis showed that human editors choose sources more evenly, and also prefer a more diverse set of sources compared to the trending algorithm. We also found the trending algorithm featured more “soft news” stories. Here are a few more details about the main findings.

1. Human Editors Chose Sources More Evenly

One of our main research questions was how news sources were concentrated in the app. A previous study in the United Kingdom showed that 75% of stories from Apple News editors were from just six news outlets, so we were interested if this concentrated pattern was happening in the United States.

Can you imagine if 50% of all stories in Apple News were from a single outlet like CNN or Fox? Not only could this skew the news for millions of readers, it could mean disproportionate payouts for some news outlets.

We found concentration patterns in both sections, but the editors’ Top Stories were distributed much more evenly. For example, the four most common sources chosen by the algorithm claimed more than 50% of all Trending Stories. In contrast, the four most common sources chosen by human editors only claimed 32% of all Top Stories (check out the paper for details about statistical significance).

2. Human Editors Chose More Diverse Sources

The four most common sources in the algorithmic Trending Stories — CNN, Fox News, People, and BuzzFeed — were chosen less frequently by the human editors at Apple News. Here is a comparison of sources:

Common sources in Apple News. More than half of the algorithmic Trending Stories were from CNN, Fox, People, or BuzzFeed. Local and regional sources are uncommon in both sections.

What exactly does this mean? The algorithm is choosing more stories from BuzzFeed and People, while the editors are choosing more stories from The Washington Post and the New York Times. On the surface, this suggests that the algorithm could be choosing more stories about celebrities and other “soft news.” To find out if this was true, we analyzed every headline that appeared in the data.

3. Human Editors Chose Less “Soft News”

Our data showed that, as The New York Times summarized in their report about Apple News, the algorithmic Trending Stories “tend to focus on Mr. Trump or celebrities.” Here are some common phrases that appeared in the data:

Common phrases in headlines that appeared in Apple News. Editors chose stories about public health, national politics, and international events, while the algorithm chose many stories about celebrities and other soft news.

Apple’s human editors actually never chose a story with Mr. Trump’s full name in the headline. Phrases such as “Trump Tax Returns” were common, but no headline used his first name. Common phrases in Top Stories headlines included international subjects (Brexit Deal, Kim Jong Un) and U.S. policies related to health (Affordable Care Act) and immigration (sanctuary cities).

Still, the editors balanced more substantive topics with some entertainment news (ex. Avengers: Endgame) and one story about a Northwestern University alumna having a baby — specifically, Meghan Markle giving birth to Archie. They also regularly curated collections of “The Week’s Good News.”

Takeaways

Our results highlight the trade-offs between human curation and algorithmic curation. While our study only looked at one platform, it shows that human editors were “much more subtly following the news cycle and what’s important,” as Lauren Kern (editor in chief at Apple News) put it.

For many readers and publishers, this is good news. The data shows that editors choose stories about important topics from a diverse set of sources, and choose those sources quite evenly. This is less true of the algorithmic Trending Stories, where readers will see more “soft news,” and just a few major publishers tend to make the cut.

Here is another anecdotal piece of evidence on the value of human curation. At the beginning of this post, we saw how human editors were valuable in the context of the U.S. protests. They were also valuable in the context of the COVID-19 pandemic. If you checked the Google News home page at the end of February, this is what you would have seen (based on the WayBack machine):

All of the stories mention two things: coronavirus and Donald Trump. If you read them, you may glimpse some information about the impending pandemic — a “severity warning” from the CDC, for example. The headlines probaly grab your attention, but they do not provide meaningful information.

Apple’s editors had a different approach that day, featuring an article with the headline “Coronavirus’s spread in U.S. is ‘inevitable,’ CDC warns.” It was a formal, descriptive piece from the Washington Post that quoted several officials at the Center for Disease Control. I scrolled back in the Apple News desktop app to find it:

During my commute on the CTA purple line, I remember one quote from the article that changed my expectations for the coming months:

“Disruptions to everyday life may be severe, but people might want to start thinking about that now”

I am grateful that humans chose my news that day.