Music Lover? This Startup Knows What Tunes You Want to Hear

If you listen to music online, chances are that Brian Whitman knows what songs you like, and the sounds you crave—be they Kanye West’s tribal drums, Michael Bolton’s dulcet croon, or Taylor Swift’s bubblegum chirp. In fact, Whitman can probably predict which rising song-stars you’ll love even before you’ve heard of them. Such prognostication isn’t inborn, of course—Whitman is no musical mystic. His divining is data-driven. Whitman is the co-founder of a tech startup called The Echo Nest, what he refers to as a “music intelligence” company. By monitoring our Internet listening habits, and by analyzing the way we critique and discuss music online, The Echo Nest helps music fans find tunes they can groove to among the millions available on the web.

While you may not have heard of Whitman or The Echo Nest, you might have enjoyed the fruits of their labor. The startup works behind the scenes, collecting trails of digital information that online listeners leave behind each time they search for a band, skip a song, or “like” a track using a streaming service such as Vevo, Rdio, or MTV.com—all Echo Nest clients. Whitman’s team also keeps tabs on musical discourse: When fans blog about an album on Tumblr, for instance, or a critic reviews an album for an online magazine, they create data points—bits of information ripe for analysis.

The Echo Nest uses e-intel like this to suggest songs that are fine-tuned to individual listeners’ tastes. The more people talk about and listen to music on the Internet, the richer the data—and the better The Echo Nest can serve its clients, and their patrons, on the web. This digital terrain is part of what Whitman refers to as “the music app ecosystem,” and he’s bent on mining its farthest reaches.

Tall, bearded, and barrel-chested, Whitman is quiet and reserved with an air of confidence belying shy self-awareness. He sports the standard indie rock uniform—sneakers, close-fitting jeans, and a casual collared shirt. While discussing his work, he speaks measuredly, as if sifting through a mental database of scientific jargon for words that a layperson can understand.

Whitman has always loved music; it runs in his family. His brother Keith is a veteran electronic musician who has recorded more than a dozen albums. After graduating from college at Worcester Polytechnic Institute with a degree in English, Whitman “took a couple years off” to pursue his own musical aspirations full-time, recording computer-based “intelligent dance music,” and playing shows under the moniker Blitter.

Although he poured himself into his craft, success was hard to come by. It was the late 90s, “I was living in New York, I was touring, I was playing a lot of shows,” he says. “A couple thousand people would buy my records. It was not a money-making operation.” Frustrated, Whitman started wondering how an anonymous artist such as himself could reach an audience of acolytes—especially ones that he didn’t know existed, wherever they may be. The key, he knew, was the Internet, home to a chorus of ongoing musical discussions and browsing. If only one lonely blogger was writing about Blitter in some dark corner of the web, Whitman wanted his computer to know about it.

With renewed determination, Whitman enrolled at Columbia University in New York, where he earned a master’s in computer science with the Natural Language Processing Group, which studies the relationship between linguistics and machine learning, among other concentrations. After that, he pursued a doctorate with MIT’s Machine Listening group, where he investigated the relationship between music and the language that we use to critique it.

During the course of his research, Whitman found the bass clef to his treble in Tristan Jehan, a fellow Ph.D. candidate who was also studying music intelligence. Instead of using computers to scour the Internet for words about music, however, Jehan concentrated on programming them to understand the sonic elements of songs. Their research was complementary, their goals the same: Teach computers to understand music in the nuanced way that humans do. The two men joined forces, and The Echo Nest was born at MIT’s Media Lab. Before long, it spun off as a startup.

To explain how The Echo Nest works, media outlets often draw a comparison to the Internet radio service Pandora. Both companies recommend music that eventually reaches online listeners, but they do so in fundamentally different ways. Pandora licenses songs to include in a digital library, marketed directly to consumers. Music analysts listen to the songs selected for that library, completing a survey that describes the lyrics, rhythm, instrumentation, melody, and other qualities. Using those surveys and Pandora users’ account preferences, the company generates personalized playlists of similar songs. Type “80s pop” into the service’s search box, for example, and songs by The Human League, Simple Minds, and other comparable—and familiar—bands or artists might start streaming.

In contrast, The Echo Nest doesn’t license any music, and it doesn’t have a library. Instead, it analyzes digital information about songs—including what people are saying about them and how they interact with them via streaming online music services—to provide recommendations for use in custom Internet radio stations and tailor-made playlists. A proprietary computer algorithm crunches the data. (The Echo Nest’s clients work out their own music licenses.)

Here’s one way the system works: Say that someone is chatting about Blitter’s self-titled EP in an online forum. The Echo Nest’s computers will most likely discover that discussion, locate the album under scrutiny, and incorporate information about the record into its database. (Pandora, meanwhile, relies on a submission process for new songs.) “We can know about all music, no matter how new or unpopular it is, because our stuff is [partially] automated,” Whitman says.

As another example, say you’re listening to music on an app like Rdio, and a tune you can’t stand comes on. If you skip it, The Echo Nest will know and work that insight into its algorithm. “We have a tremendous amount of data on you as a user—what you’ve played, what you’ve ‘Thumbs Up’d,’ what you’ve added to a collection,” says Chris Becherer, Rdio’s vice president of Product. “We send that to The Echo Nest, and they combine that with their data.” The result is what The Echo Nest calls a “taste profile”—a portrait of what listeners want to hear.

The taste profile, which lives on The Echo Nest’s servers, is essentially the brain that powers products like Rdio’s new You FM and Friend FM stations, which automatically generate tracks for subscribers. What’s more, as those subscribers increasingly engage with Echo Nest clients, The Echo Nest’s system learns even more about listener habits—in other words, it gets smarter.

For quality control, Whitman employs humans, too. For example, an editorial team of data curators assigns a hierarchy to The Echo Nest’s data based on various criteria, such as the media source that certain information came from—a song mentioned in a Rolling Stone review gets more weight than, say, your cousin’s off-handed tweet. Humans also review the data for anomalies that could mar the end user’s experience.

The combination of machine-curated content and human supervision has allowed the startup to build a staggering database, with information on more than 35 million recorded compositions—plenty of options for potential playlists. (Pandora has around a million compositions.)

These days, Whitman is singing a happy tune. After several successful grants from the National Science Foundation and several rounds of funding from investors, he’s delegated a lot of the day-to-day programming work to staff in order to focus on R&D. Facing competition from Google and Apple, The Echo Nest continues to expand—you can see its success in a running tab on its homepage, which charts the size of its database and the number of apps that use it. At press time, Whitman’s startup had more than a trillion data points on more than 2.5 million artists being served to more than 400 music apps. When it comes to music intelligence, The Echo Nest’s numbers are off the charts.

*This article was amended on October 8, 2013 to reflect the fact that “The Echo Nest” is the full name of the startup, not the “Echo Nest,” as previously written.