A few successful websites were built almost entirely through viral growth. The vast majority, however, started off by partnering with other, already successful websites. Even Google began by partnering with Yahoo. As superior as Google’s search algorithm was, it was very hard to get the masses to switch to a new search engine.
In the web 1.0 world (approximately pre-2004), integrating two web services involved lots of manual work, such as negotiating legal contracts and custom technical integration. Creating these kinds of partnerships is usually referred to as “business development” or “BizDev” (personally, I usually just call it “BD”). In the web 2.0 world, it became common for websites to create fully functional, self-service API’s with standardized legal terms. This made it possible to drastically reduce the friction of integrating services. My Hunch cofounder Caterina Fake coined the term “BizDev 2.0″ to refer to this idea (and of course Flickr was a pioneer of super robust APIs).
There is no question that removing legal and technical hurdles is a win for everyone (except lawyers). However, unless your service is extremely high profile and its value is easily understood, it still needs to be marketed to potential partners. Many websites won’t consider using a self-service API until they’ve seen it working on other sites with measurable results. So how do you overcome this particular kind of chicken-and-egg problem?
During his interview process, Hunch’s Shaival Shah, said something that struck a chord with me: he didn’t want to be called “VP BizDev” because, he said, a good BizDev person makes BizDev irrelevant. The idea is to create a number of BizDev 1.0 partnerships while simultaneously building and marketing a full service API. If you can do BizDev 1.0 with some number of (ideally high profile) websites and demonstrate that it is valuable to them (ideally quantitatively), you can then scale your service BizDev 2.0 style. Maybe this could be called BizDev 1.5.
It has become customary to use “graph” to refer to the underlying data structures at social networks like Facebook. (Computer scientists call the study of graphs “network theory,” but on the web the word “network” is used to refer to the websites themselves).
A graph consists of a set of nodes connected by edges. The original internet graph is the web itself, where webpages are nodes and links are edges. In social graphs, the nodes are people and the edges friendship. Edges are what mathematicians call relations. Two important properties that relations can either have or not have are symmetry (if A ~ B then B ~ A) and transitivity (if A ~ B and B ~ C then A ~ C).
Facebook’s social graph is symmetric (if I am friends with you then you are friends with me) but not transitive (I can be friends with you without being friends with your friend). You could say friendship is probabilistically transitive in the sense that I am more likely to like someone who is a friend’s friend then I am a user chosen at random. This is basis of Facebook’s friend recommendations.
Twitter’s graph is probably best thought of as an interest graph. One of Twitter’s central innovations was to discard symmetry: you can follow someone without them following you. This allowed Twitter to evolve into an extremely useful publishing platform, replacing RSS for many people. The Twitter graph isn’t transitive but one of its most powerful uses is retweeting, which gives the Twitter graph what might be called curated transitivity.
Graphs can be implicitly or explicitly created by users. Facebook and Twitter’s graphs were explicitly created by users (although Twitter’s Suggested User List made much of the graph de facto implicit). Google Buzz attempted to create a social graph implicitly from users’ emailing patterns, which didn’t seem to work very well.
Over the next few years we’ll see the rising importance of other types of graphs. Some examples:
Taste: At Hunch we’ve created what we call the taste graph. We created this implicitly from questions answered by users and other data sources. Our thesis is that for many activities – for example deciding what movie to see or blouse to buy – it’s more useful to have the neighbors on your graph be people with similar tastes versus people who are your friends.
Financial Trust: Social payment startups like Square and Venmo are creating financial graphs – the nodes are people and institutions and the relations are financial trust. These graphs are useful for preventing fraud, streamlining transactions, and lowering the barrier to accepting non-cash payments.
Endorsement: An endorsement graph is one in which people endorse institutions, products, services or other people for a particular skill or activity. LinkedIn created a successful professional graph and a less successful endorsement graph. Facebook seems to be trying to layer an endorsement graph on its social graph with its Like feature. A general endorsement graph could be useful for purchasing decisions and hence highly monetizable.
Local: Location-based startups like Foursquare let users create social graphs (which might evolve into better social graphs than what Facebook has since users seem to be more selective friending people in local apps). But probably more interesting are the people and venue graphs created by the check-in patterns. These local graphs could be useful for, among other things, recommendations, coupons, and advertising.
Besides creating graphs, Facebook and Twitter (via Facebook Connect and OAuth) created identity systems that are extremely useful for the creation of 3rd party graphs. I expect we’ll look back on the next few years as the golden age of graph innovation.
Howard Lindzon was nice enough to have me on his Stocktwits.tv show recently. For those who don’t know Howard, he writes a fantastic blog. He writes in such an irreverent way it’s easy to overlook the wisdom behind what he says. My favorite recent Howard-ism was, talking about investing, “I like to look outside and see my [investments].” I take this to mean he likes to invest in things he understands, can touch, go visit, etc. This is probably the single best piece of advice in order to have survived the recent financial crisis. Fancy things like CDOs, Auction-Rate Securities, etc turned out to function much differently than advertised. Diversification across asset classes (CAPM etc) turned out to be useless: when things got bad, correlations went to 1. One reason I like investing in startups is you can go visit them – they are something tangible and understandable.
Howard is also the founder of Stocktwits. Stocktwits is potentially genuinely disruptive in that it dis-intermediates Wall Street. It is one of those things that some people think is a toy now but could end up being the next big thing.
If you sell stuff: Companies like BazaarVoice have proven that displaying user reviews on ecommerce sites increases conversion rates. You can pay BazaarVoice for this or get it free from Hunch. Here’s an example widget for Mario Kart Wii:
Readers can click through the widget and rate your blog on Hunch. This in turn can drive traffic back to your blog (Hunch had 1.2M uniques last month).
You can go here to make a badge. If your blog isn’t in Hunch’s database you can add it here
I think you could make a strong argument that the most important technologies developed over the last decade are a set of systems that are sometimes called “collective knowledge systems”.
The most successful collective knowledge system is the combination of Google plus the web. Of course Google was originally intended to be just a search engine, and the web just a collection of interlinked documents. But together they provide a very efficient system for surfacing the smartest thoughts on almost any topic from almost any person.
The second most successful collective knowledge system is Wikipedia. Back in 2001, most people thought Wikipedia was a wacky project that would at best end up being a quirky “toy” encyclopedia. Instead it has become a remarkably comprehensive and accurate resource that most internet users access every day.
Other well-known and mostly successful collective knowledge systems include “answer” sites like Yahoo Answers, review sites like Yelp, and link sharing sites like Delicious. My own company Hunch is a collective knowledge system for recommendations, building on ideas originally developed by “collaborative filtering” pioneer Firefly and the recommendation systems built into Amazon and Netflix.
Dealing with information overload
It has been widely noted that the amount of information in the world and in digital form has been growing exponentially. One way to make sense of all this information is to try to structure it after it is created. This method has proven to be, at best, partially effective (for a state-of-the-art attempt at doing simple information classification, try Google Squared).
It turns out that imposing even minimal structure on information, especially as it is being created, goes a long way. This is what successful collective knowledge systems do. Google would be vastly less effective if the web didn’t have tags and links. Wikipedia is highly structured, with an extensive organizational hierarchy and set of rules and norms. Yahoo Answers has a reputation and voting system that allows good answers to bubble up. Flickr and Delicious encourage user to explicitly tag items instead of trying to infer tags later via image recognition and text classification.
Importance of collective knowledge systems
There are very practical, pressing needs for better collective knowledge systems. For example, noted security researcher Bruce Schneier argues that the United States’ biggest anti-terrorism intelligence challenge is to build a collective knowledge system across disconnected agencies:
What we need is an intelligence community that shares ideas and hunches and facts on their versions of Facebook, Twitter and wikis. We need the bottom-up organization that has made the Internet the greatest collection of human knowledge and ideas ever assembled.
The same could be said of every organization, large and small, formal and and informal, that wants to get maximum value from the knowledge of its members.
Collective knowledge systems also have pure academic value. When Artificial Intelligence was first being seriously developed in the 1950’s, experts optimistically predicted they’d create machines that were as intelligent as humans in the near future. In 1965, AI expert Herbert Simon predicted that “machines will be capable, within twenty years, of doing any work a man can do.”
While AI has had notable victories (e.g. chess), and produced an excellent set of tools that laid the groundwork for things like web search, it is nowhere close to achieving its goal of matching – let alone surpassing – human intelligence. If machines will ever be smart (and eventually try to destroy humanity?), collective knowledge systems are the best bet.
Design principles
Should the US government just try putting up a wiki or micro-messaging service and see what happens? How should such a system be structured? Should users be assigned reputations and tagged by expertise? What is the unit of a “contribution”? How much structure should those contributions be required to have? Should there be incentives to contribute? How can the system be structured to “learn” most efficiently? How do you balance requiring up front structure with ease of use?
These are the kind of questions you might think are being researched by academic computer scientists. Unfortunately, academic computer scientists still seem to model their field after the “hard sciences” instead of what they should modeling it after — social sciences like economics or sociology. As a result, computer scientists spend a lot of time dreaming up new programming languages, operating system architectures, and encryption schemes that, for the most part, sadly, nobody will every use.
Meanwhile the really important questions related to information and computer science are mostly being ignored (there are notable exceptions, such as MIT’s Center for Collective Intelligence). Instead most of the work is being done informally and unsystematically by startups, research groups at large companies like Google, and a small group of multi-disciplinary academics like Clay Shirky and Duncan Watts.
We are very excited to announce today that Jimmy Wales, founder of Wikipedia, has joined Hunch’s board of directors. He’ll also be hanging out with us some in NYC, which I think could be great for the NYC startup ecosystem. Jimmy blogs about joining us here, and we have some Hunch milestone stats on the Hunch blog.
Business-wise, Jimmy is the perfect fit — we’ve always talked about Hunch aspiring to be “Wikipedia for decision-making.” Personally, it’s also really thrilling: I think of Wikipedia as an incredible and extremely significant human endeavor.
Today we are releasing a new version of the Hunch question-and-answer interface. The main idea behind the new interface is that you now see Hunch’s top recommendations updated in real time as you answer questions.
Besides giving instant feedback, this also lets you see how each answer you give affects the recommendations. We think it does a better job exposing Hunch’s intelligence, particularly the statistical intelligence the system has acquired from the tens of millions of user feedback clicks collected over the past few months.
Here’s an example. Suppose you are looking for a video game and you’re a brand new Hunch user – Hunch knows nothing about you (you can try this yourself by logging out and going to our video games topic). You start by seeing a list of video games ranked by overall popularity among users:
Now suppose I answer the question on the left by clicking on XBox 360. The list on the right then updates, showing only the most popular Xbox 360 games:
Since at this point I haven’t told Hunch anything about myself, this list is still just an un-personalized list of popular XBox 360 video games. Now let’s try some basic personalization. Suppose I start over and instead of answering which video game console I prefer, I click on “About me” tab:
and click on Male. The video game list then changes to show games that tend to be preferred by males:
This list is determined statistically from Hunch user responses. For example, Fallout 3 rose to the #1 spot after I said I was male because Hunch user responses correlate liking Fallout 3 with being male:
This is obviously a very basic example of personalization. Things get more interesting when you answer a series of questions and Hunch combines filters and statistical data on the fly, thereby giving you highly personalized and relevant results.
Another new feature is that if you click Yes or No next to a result:
Hunch will not only learn from your click – thereby getting smarter – it will also recompute the result list on the screen instantly.
For example, let’s start video games from the beginning again and click “No” next to the top result Brain Age. Hunch then changes my result list to show more “hardcore” games that are statistically anti-correlated with Brain Age (a game hardcore gamers tend to think is “kiddie”):
Please feel free to give us any feedback on the new interface.
If you look at the right sidebar on this blog you’ll see a new Hunch widget. It’s meant to be both fun and informative for the blogger and also the readers.
1. For the blogger, you can learn a lot of interesting things about your readership (for example, here are stats on cdixon.org readers). Soon, we’ll be adding more features for the blogger, such as inferred stats about your readers, derived by cross referencing their answers against our data set of 40M answers.
2. Blog readers get to learn about how they compare to other readers of the blog, and how readers of the blog compare to the larger population. They can also play what we call the “prediction game” where Hunch tries to guess how you’d answer new questions you haven’t answered. In our tests Hunch does a really good job. It’s meant to be fun and also, frankly, a way for us to show off the power of Hunch’s predictive abilities. If you want to try it, first answer 25 questions in the widget and then you’ll be be given the option to play the game or look at how you compare to other cdixon.org readers and Hunch users overall.
If you want to embed this widget on your own blog, go to http://www.hunch.com/blogger/ (you’ll need to have a Hunch account and be logged in).
One of the hardest things to do as a startup is to create a new category. Bloggers and press have a natural tendency to “pigeonhole” – to group startups into cleanly delineated categories, and then do side-by-side comparisons, comment on the “horserace” between them, and so forth.
At my last startup, SiteAdvisor, we were at first consistently pigeonholed as an anti-phishing toolbar, even though what we did was help search engine users avoid spyware, spam, and scams, which (for various technical reasons) had almost no functional overlap with anti-phishing toolbars. My co-founder at Hunch, Caterina Fake, had a similar experience at Flickr. Early on, people compared Flickr to existing photo sharing websites – Shutterfly, Ofoto, SnapFish - and found Flickr lacking in features around buying prints, sending greeting cards, etc.
Pigeonholing is one reason startups should actually welcome direct competitors. It was only once a direct competitor to SiteAdvisor appeared that people started treating “web safety” as its own category (Walt Mossberg was the first one to legitimize the category with this article).
At my current startup, Hunch, being pigeonholed as a so-called Answers site is one of our main marketing challenges. Hunch is a user-generated website similar to Wikipedia except, instead of creating encyclopedia entries, contributors create decision trees that help other users make choices and decisions. For example, about 50 computer enthusiasts came together to create this decision tree about computer laptops that helps users with less expertise find the right laptop. Hunch gets smarter over time as more people contribute to it. So far, about 10,000 users have made 115,000 contributions to the site. Last month, our third month after launch, over 600,000 unique visitors used those contributions to make decisions.
Manyoftheinitialreviews of Hunch accurately reflected that Hunch is trying to create a new category of website. Nevertheless, the tendency to pigeonhole Hunch as an Answers site remains. Answers sites allow users to ask a question and get back direct answers from other people. There are many Answer sites including Yahoo Answers, Mahalo Answers, Vark, Answerbag, and ChaCha. These are all excellent and useful services – but have as much to do with Hunch as Ofoto had to do with Flickr.
There is no easy solution to avoid being pigeonholed. All you can do is consistently, straightforwardly describe what you do, and then keep beating that drum over and over until the message gets through.
