Search is Dead, Long Live Gourmet Content

Three years ago I founded Dulcinea Media, a human-powered search alternative, on the premise that most Internet users cannot find the information they need online, and that search engines will never match the ability of a human curator to find the best content. At the time, this was a contrarian view. A  report by Pew Internet in January 2005 had shown that 75% of Internet users were satisfied with search engine results, and most pundits believed Google and Wikipedia would dominate the online information landscape forever.

In the past three years, the market has warmed to my view that uncurated, general search engines are a less-than-perfect tool for finding information online. More recent studies from the USC's Center for the Digital Future showed that user satisfaction with search results declined to 62% in 2006, and again to 51% in 2008. A survey commissioned by Yahoo! curiously issued a damning indictment of search engine results: 85% of initial search queries fail to return the information users were seeking, causing the users to try and try again, resulting in "search engine fatigue." A study from the UK exposed as a myth the notion of a “Google Generation” of young people with native ability to find information online.

Next, Nicholas Carr, who famously asked “Is Google Making Us Stupid?", and a number of other columnists bemoaned the reality that most users today read an Internet that is a mile-wide and an inch-deep. The center of their media world is a technology driven algorithm and “the wisdom of crowds” that simply uncover the same recycled headlines and updates from a slew of news sources. Google dipped a toe in the “human-powered” waters to tweak some of its search rankings, although it still accords technology most of the weight in the equation. Roger Schank, an artificial intelligence expert from Yale University, reversed his 30-year-old prediction that we would create machines as smart as humans in his lifetime. Schank came to recognize that "[h]umans are constantly learning. ... [e]very new experience changes what we know and how we see the world." Schank attributed this to "an unconscious indexing method that all people learn to do without quite realizing they are learning it."

Now a growing chorus of observers is acknowledging that search engines often fail the user. The impetus is the rise of “content farms,” which assure that search engines are only going to get worse at delivering quality results on the first search results page. Demand Media, Associated Content, Mahalo, Bukisa, eHow, HubPages and a voracious pack of others are paying freelance writers a modest per-article fee to create tens of thousands of articles each day. These companies excel at getting their content to rank high in search engines, regardless of quality.

The biggest challenge with these sites, paradoxically, is that some of the content is actually good, and most of it reads well. But, as Richard MacManus of ReadWriteWeb wrote after analyzing Demand Media’s content, it “lacks passion and often also lacks knowledge of the topic at hand.” Worse, the quality varies greatly from article to article – these brands stand for nothing other than “we create lots of content cheaply, SEO it superbly, and monetize it well.” Since no flashing neon lights warn “keyword-ridden trash” for weak submissions, the average Internet user does not know that the article was written in 20 minutes by someone with no expertise on the topic.

Many pundits agree that this spells trouble for search engines, but differ on further implications. Michael Arrington of TechCrunch sees the “end of hand crafted content.” In his view, the “fast food content” created by content farms has produced a “race to the bottom situation, where anyone who spends time and effort on their content is pushed out of business.” Fred Wilson of Union Square Ventures is more hopeful. He sees search engines being displaced by social media tools, where, “with the help of machines, our friends and trusted sources” will tell users what content matters.

What I see is that this avalanche of fast food content will lead to the “clarion call” that I predicted, in a July 2008 article  would cause a “flight to safety.” Internet users will turn to the "new portal" - trusted sources that consistently deliver “gourmet content” - important, relevant, reliable and comprehensive information, from a wide variety of resources across the Internet. The only sites that will succeed at this are those that rely on a human touch. For all the dismissive talk about Yahoo, its audience is massive - because it gathers content from around the Web, albeit of inconsistent quality across the verticals.  Companies that aggregate and organize content in an elegant way, and combine that with their own proprietary, high-quality content, will inherit the position at the top of the Web food chain that search engines abdicate.

So where is Dulcinea Media in all this ? Naturally, planning to be one of those trusted sources, or perhaps an engine that powers the new portals to which users flock. We are still executing  on the business plan we created three years ago. findingDulcinea now offers Web Guides to the best information alone about more than 700 broad topics, and we’ve created thousands of Beyond the Headlines and Features articles that provide a full context view of news stories. A Spanish-language sister site, encontrandoDulcinea, replicates much of this content in Spanish.

To make all this content easier to access, we’ve introduced SweetSearch, a custom search engine that harnesses Google’s technology and the 100,000+ hours of Web site evaluation that is the bedrock of findingDulcinea. SweetSearch returns results only from a “whitelist” of 35,000 sites that we’ve evaluated and approved. And we are tweaking SweetSearch to ensure that it remains the best search engine for students, and indeed, the only one they can use effectively. We've also introduced SweetSearch4Me, which is the only search engine that displays prominently high quality sites created for younger learners.

As our audience continues to grow steadily, we’ve also found that our “best customers” – those who visit our site the longest and consume the most pageviews, and thus are most likely to return – are college, high school and middle school students. And thus we focus our content on subjects that would be of interest to teachers, librarians, and students.  We presented at two national conferences in the fall - the AASL conference for school librarians, and the NCSS conference for social studies teachers - and we received an overwhelmingly positive response to our products.  We learned there is a critical need in the marketplace for free products that promotes effective, efficient, safe and responsible use of the Internet, and that ours fit the bill magnificently. And we've also had some very encouraging discussions with forward-thinking media companies about partnering with us to help make them trusted sources for content from around the Web.

We remain steadfast in our principles that (i) we will not use technology to aggregate links for Web Guides or articles; everything will pass through the prism of human judgment; and (ii) we will never compromise on the quality of our product, which will all continue to be created by our full-time editorial staff or subject-matter expert freelancers, and edited by a full-time editor.

To address scaling issues while holding form to these principles, we plan to introduce a program early next year in which we invite librarians and educators to submit content. Practitioners of these professions are trained to find, evaluate and recommend outstanding information resources, and library Web sites have always been the closest comparable to our Web Guides. We envision findingDulcinea and SweetSearch becoming a repository of the knowledge and insight of tens of thousands of librarians and teachers.

And we’ll stick with that vision, for as long as it takes to make it a reality.