May 24, 2009
The future of RSS: smart search engines

Over at Mashable, Ben Parr wonders, "What is the future of RSS? Is social media a better alternative?" I believe the world is groping towards a better solution than either.

In two years I have switched several times between feedreaders like Google Reader and social news sites like Reddit, Twitter and Friendfeed. Feedreaders amplify the volume of my reading. Social news helps me find the highest quality stories. The tension between quality and quantity keeps me switching. With growing volumes of news in an increasingly-online world, feed readers and social news are each incomplete.

But I am an extreme case; most people read far less. Do power users matter? They can help highlight a trend. Every reader online wants relevant news; some just go to greater lengths than others. Perhaps the migrations of power users between feedreaders and social news sites can teach us how to serve all readers.

Feedreaders assume you want to read everything by everyone you subscribe to, and nothing by anyone else. Subscribe to too many, and those assumptions start to break down. Chronological ordering starts to suck. Frequent writers drown out the rest, regardless of who you care about. Sifting through the noise becomes a challenge.

The popularity of social news is largely explained by this challenge. 10% of the users on a social news site vote on stories, and only 1% comment. The rest of us are all using social news purely to find interesting stories, often because the feedreader didn't work out.

But using a social news site has its own drawbacks. You can't find as many stories. Stay at the front page or in a small community, and filtering works. Lower down the list, quality drops. A larger community provides faster turnover, but it's also susceptible to lowest-common-denominator effects - think pictures of lolcats or youtube videos.

Even when filtering works, you only find stories your friends find interesting. Over time, you start to ignore interests that don't overlap with your network. You risk spending time reading low-quality comments or flame wars. Echo chamber effects suppress dissenting voices, though those are often the most interesting.

In a healthy community people do their reading in private, and come together to discuss what they read.

If neither works, where does that leave us? Competing incomplete interfaces create false dichotomies. Asking readers to provide favorite sources is a good start, and so is voting on stories for your friends. Accepting recommendations from friends is a part of the puzzle, but a small part, lest you risk endlessly regurgitating each other's recommendations. There's no reason these signals can't be combined.

The essential property of both feedreaders and social news sites: they aggregate content from many sources before presenting it to the reader. We need a better aggregator, a feedreader that can handle firehoses. One that can rank stories smarter than just chronologically or alphabetically, perhaps even adapt to our changing interests.

What would such an aggregator look like? It would have scale, to discover feeds quickly, and to crawl all the feeds out there. It would have smarts, to connect you up with only the stories you find interesting, and to prioritize them. These are big changes; the new tool looks nothing like its forebears. What it resembles most is a search engine. It crawls and indexes everything it can find. Rather than responding to queries, it knows you and your tastes and alerts you to interesting pages. It can be consumed in multiple layouts wherever you go - facebook, twitter, your feedreader.

This is our vision at MeeHive.

(Thanks to Tracy Lou and Jonathan Nelson for reading drafts of this.)
Making the big picture easy to see, in software and in society at large.
Code (contributions)
Prose (shorter; favorites)
favorite insights
Social Dynamics
Social Software