Howard Lindzon was nice enough to have me on his Stocktwits.tv show recently. For those who don’t know Howard, he writes a fantastic blog. He writes in such an irreverent way it’s easy to overlook the wisdom behind what he says. My favorite recent Howard-ism was, talking about investing, “I like to look outside and see my [investments].” I take this to mean he likes to invest in things he understands, can touch, go visit, etc. This is probably the single best piece of advice in order to have survived the recent financial crisis. Fancy things like CDOs, Auction-Rate Securities, etc turned out to function much differently than advertised. Diversification across asset classes (CAPM etc) turned out to be useless: when things got bad, correlations went to 1. One reason I like investing in startups is you can go visit them – they are something tangible and understandable.
Howard is also the founder of Stocktwits. Stocktwits is potentially genuinely disruptive in that it dis-intermediates Wall Street. It is one of those things that some people think is a toy now but could end up being the next big thing.
If you sell stuff: Companies like BazaarVoice have proven that displaying user reviews on ecommerce sites increases conversion rates. You can pay BazaarVoice for this or get it free from Hunch. Here’s an example widget for Mario Kart Wii:
Readers can click through the widget and rate your blog on Hunch. This in turn can drive traffic back to your blog (Hunch had 1.2M uniques last month).
You can go here to make a badge. If your blog isn’t in Hunch’s database you can add it here
I think you could make a strong argument that the most important technologies developed over the last decade are a set of systems that are sometimes called “collective knowledge systems”.
The most successful collective knowledge system is the combination of Google plus the web. Of course Google was originally intended to be just a search engine, and the web just a collection of interlinked documents. But together they provide a very efficient system for surfacing the smartest thoughts on almost any topic from almost any person.
The second most successful collective knowledge system is Wikipedia. Back in 2001, most people thought Wikipedia was a wacky project that would at best end up being a quirky “toy” encyclopedia. Instead it has become a remarkably comprehensive and accurate resource that most internet users access every day.
Other well-known and mostly successful collective knowledge systems include “answer” sites like Yahoo Answers, review sites like Yelp, and link sharing sites like Delicious. My own company Hunch is a collective knowledge system for recommendations, building on ideas originally developed by “collaborative filtering” pioneer Firefly and the recommendation systems built into Amazon and Netflix.
Dealing with information overload
It has been widely noted that the amount of information in the world and in digital form has been growing exponentially. One way to make sense of all this information is to try to structure it after it is created. This method has proven to be, at best, partially effective (for a state-of-the-art attempt at doing simple information classification, try Google Squared).
It turns out that imposing even minimal structure on information, especially as it is being created, goes a long way. This is what successful collective knowledge systems do. Google would be vastly less effective if the web didn’t have tags and links. Wikipedia is highly structured, with an extensive organizational hierarchy and set of rules and norms. Yahoo Answers has a reputation and voting system that allows good answers to bubble up. Flickr and Delicious encourage user to explicitly tag items instead of trying to infer tags later via image recognition and text classification.
Importance of collective knowledge systems
There are very practical, pressing needs for better collective knowledge systems. For example, noted security researcher Bruce Schneier argues that the United States’ biggest anti-terrorism intelligence challenge is to build a collective knowledge system across disconnected agencies:
What we need is an intelligence community that shares ideas and hunches and facts on their versions of Facebook, Twitter and wikis. We need the bottom-up organization that has made the Internet the greatest collection of human knowledge and ideas ever assembled.
The same could be said of every organization, large and small, formal and and informal, that wants to get maximum value from the knowledge of its members.
Collective knowledge systems also have pure academic value. When Artificial Intelligence was first being seriously developed in the 1950’s, experts optimistically predicted they’d create machines that were as intelligent as humans in the near future. In 1965, AI expert Herbert Simon predicted that “machines will be capable, within twenty years, of doing any work a man can do.”
While AI has had notable victories (e.g. chess), and produced an excellent set of tools that laid the groundwork for things like web search, it is nowhere close to achieving its goal of matching – let alone surpassing – human intelligence. If machines will ever be smart (and eventually try to destroy humanity?), collective knowledge systems are the best bet.
Design principles
Should the US government just try putting up a wiki or micro-messaging service and see what happens? How should such a system be structured? Should users be assigned reputations and tagged by expertise? What is the unit of a “contribution”? How much structure should those contributions be required to have? Should there be incentives to contribute? How can the system be structured to “learn” most efficiently? How do you balance requiring up front structure with ease of use?
These are the kind of questions you might think are being researched by academic computer scientists. Unfortunately, academic computer scientists still seem to model their field after the “hard sciences” instead of what they should modeling it after — social sciences like economics or sociology. As a result, computer scientists spend a lot of time dreaming up new programming languages, operating system architectures, and encryption schemes that, for the most part, sadly, nobody will every use.
Meanwhile the really important questions related to information and computer science are mostly being ignored (there are notable exceptions, such as MIT’s Center for Collective Intelligence). Instead most of the work is being done informally and unsystematically by startups, research groups at large companies like Google, and a small group of multi-disciplinary academics like Clay Shirky and Duncan Watts.
We are very excited to announce today that Jimmy Wales, founder of Wikipedia, has joined Hunch’s board of directors. He’ll also be hanging out with us some in NYC, which I think could be great for the NYC startup ecosystem. Jimmy blogs about joining us here, and we have some Hunch milestone stats on the Hunch blog.
Business-wise, Jimmy is the perfect fit — we’ve always talked about Hunch aspiring to be “Wikipedia for decision-making.” Personally, it’s also really thrilling: I think of Wikipedia as an incredible and extremely significant human endeavor.
Today we are releasing a new version of the Hunch question-and-answer interface. The main idea behind the new interface is that you now see Hunch’s top recommendations updated in real time as you answer questions.
Besides giving instant feedback, this also lets you see how each answer you give affects the recommendations. We think it does a better job exposing Hunch’s intelligence, particularly the statistical intelligence the system has acquired from the tens of millions of user feedback clicks collected over the past few months.
Here’s an example. Suppose you are looking for a video game and you’re a brand new Hunch user – Hunch knows nothing about you (you can try this yourself by logging out and going to our video games topic). You start by seeing a list of video games ranked by overall popularity among users:
Now suppose I answer the question on the left by clicking on XBox 360. The list on the right then updates, showing only the most popular Xbox 360 games:
Since at this point I haven’t told Hunch anything about myself, this list is still just an un-personalized list of popular XBox 360 video games. Now let’s try some basic personalization. Suppose I start over and instead of answering which video game console I prefer, I click on “About me” tab:
and click on Male. The video game list then changes to show games that tend to be preferred by males:
This list is determined statistically from Hunch user responses. For example, Fallout 3 rose to the #1 spot after I said I was male because Hunch user responses correlate liking Fallout 3 with being male:
This is obviously a very basic example of personalization. Things get more interesting when you answer a series of questions and Hunch combines filters and statistical data on the fly, thereby giving you highly personalized and relevant results.
Another new feature is that if you click Yes or No next to a result:
Hunch will not only learn from your click – thereby getting smarter – it will also recompute the result list on the screen instantly.
For example, let’s start video games from the beginning again and click “No” next to the top result Brain Age. Hunch then changes my result list to show more “hardcore” games that are statistically anti-correlated with Brain Age (a game hardcore gamers tend to think is “kiddie”):
Please feel free to give us any feedback on the new interface.
If you look at the right sidebar on this blog you’ll see a new Hunch widget. It’s meant to be both fun and informative for the blogger and also the readers.
1. For the blogger, you can learn a lot of interesting things about your readership (for example, here are stats on cdixon.org readers). Soon, we’ll be adding more features for the blogger, such as inferred stats about your readers, derived by cross referencing their answers against our data set of 40M answers.
2. Blog readers get to learn about how they compare to other readers of the blog, and how readers of the blog compare to the larger population. They can also play what we call the “prediction game” where Hunch tries to guess how you’d answer new questions you haven’t answered. In our tests Hunch does a really good job. It’s meant to be fun and also, frankly, a way for us to show off the power of Hunch’s predictive abilities. If you want to try it, first answer 25 questions in the widget and then you’ll be be given the option to play the game or look at how you compare to other cdixon.org readers and Hunch users overall.
If you want to embed this widget on your own blog, go to http://www.hunch.com/blogger/ (you’ll need to have a Hunch account and be logged in).
One of the hardest things to do as a startup is to create a new category. Bloggers and press have a natural tendency to “pigeonhole” – to group startups into cleanly delineated categories, and then do side-by-side comparisons, comment on the “horserace” between them, and so forth.
At my last startup, SiteAdvisor, we were at first consistently pigeonholed as an anti-phishing toolbar, even though what we did was help search engine users avoid spyware, spam, and scams, which (for various technical reasons) had almost no functional overlap with anti-phishing toolbars. My co-founder at Hunch, Caterina Fake, had a similar experience at Flickr. Early on, people compared Flickr to existing photo sharing websites – Shutterfly, Ofoto, SnapFish - and found Flickr lacking in features around buying prints, sending greeting cards, etc.
Pigeonholing is one reason startups should actually welcome direct competitors. It was only once a direct competitor to SiteAdvisor appeared that people started treating “web safety” as its own category (Walt Mossberg was the first one to legitimize the category with this article).
At my current startup, Hunch, being pigeonholed as a so-called Answers site is one of our main marketing challenges. Hunch is a user-generated website similar to Wikipedia except, instead of creating encyclopedia entries, contributors create decision trees that help other users make choices and decisions. For example, about 50 computer enthusiasts came together to create this decision tree about computer laptops that helps users with less expertise find the right laptop. Hunch gets smarter over time as more people contribute to it. So far, about 10,000 users have made 115,000 contributions to the site. Last month, our third month after launch, over 600,000 unique visitors used those contributions to make decisions.
Manyoftheinitialreviews of Hunch accurately reflected that Hunch is trying to create a new category of website. Nevertheless, the tendency to pigeonhole Hunch as an Answers site remains. Answers sites allow users to ask a question and get back direct answers from other people. There are many Answer sites including Yahoo Answers, Mahalo Answers, Vark, Answerbag, and ChaCha. These are all excellent and useful services – but have as much to do with Hunch as Ofoto had to do with Flickr.
There is no easy solution to avoid being pigeonholed. All you can do is consistently, straightforwardly describe what you do, and then keep beating that drum over and over until the message gets through.
A friend asked me the other day “Which VC firms should I pitch?” and I started to respond to him, but then realized that most of my knowledge of VC firms is already available online in the Which VC firm should I pitch? Hunch decision topic. That is the idea behind Hunch: to crowdsource the creation of decision trees, so that a group of knowledgeable people can get together and create a “virtual expert” that can be accessed by anyone.
Here is the VC chooser topic in embedded widget form (anything you create on Hunch can be embedded anywhere):
Like everything on Hunch, this topic is completely user generated (“topic” is our word for what some people would call a “decision tree”). Users have full control over the questions it asks, the results (in this case VC firms), the descriptions, and a lot of more advanced functionality for “sculpting” the decision tree. If you go to the VC topic’s About page you can see that so far 7 people have contributed 86 firms and 5 questions to this topic (other topics have a much wider range of contributers, this one for example). The VC topic has been played (used by non contributors) 506 times, many of those users coming in via Google organic results for phrases related to pitching VC firms.
In addition, the results are all “trained” to be associated with responses to questions – meaning users have taught Hunch what to “believe” about each of the firms. For example, in red is what Hunch believes about Union Square Ventures:
Users who find mistakes can just click and fix them, similar to how you fix things on Wikipedia.
So if you see anything missing or that you’d like to change, feel free to do so. I was one main people who worked on this particular topic so it is biased toward my tastes (e.g. Hunch’s own VCs – Bessemer and General Catalyst – rank extremely high).
If you don’t like Hunch’s Q&A process you can jump directly to the See All page, and then using the filters on the left to drill down.
If you are not logged into Hunch, the VC firms you see will be ranked by their popularity amongst all Hunch users. Hunch personalizes the rankings specifically for you if you create an account and answer what we call “Teach Hunch About You” questions. For example, when I am logged in and go to the Hunch page for Bessemer I see this on the right sidebar:
Meaning that Hunch has learned to statistically correlate the questions I’ve answered about myself with liking Bessemer. At this point Hunch has statistically significant data (over 40M user feedbacks total) in most of our ~5000 topics so it usually works really well.
