Chris Dixon

In the last decade there have been major advances in storing, analyzing, and acting upon extremely large data sets.  Data sets that were previously left dormant are now being put to (mostly) constructive use. But the vast majority of information in the world isn’t available for analysis because it isn’t being electronically collected.

This is changing rapidly as new data collection mechanisms are implemented – what engineers refer to as instrumentation. Common examples of instrumentation include thermometers, public safety cameras, and heart rate monitors.

Smart phones are one obvious new source of potential instrumentation.  A person’s location, activities, audio and visual environment – and probably many more things that haven’t been thought of yet – can now be monitored.  This of course raises privacy issues.  Hopefully these privacy issues will be solved by requiring explicit user opt-in.  If so, this will require creating incentives for people to do so.

Foursquare instruments location in an opt-in way through the check in. The incentives are social and game-like, but the data produced could be useful for many more “serious” purposes.  Fitbit instruments a person’s health-related activity. The immediate incentive is to measure and improve your own health, but the aggregate data could be analyzed by medical researchers to benefit others.

In manufacturing, there has been a lot of interesting innovation around monitoring machinery, for example by using loosely joined, inexpensive mesh networks.  In homes, protocols like ZigBee allow devices to communicate which allows, for example, automation of tedious tasks and improved energy efficiency.

In the next decade, there will be a massive amount of innovation and opportunity around the big data stack. Instrumentation will be the foundational layer of that stack.

I don’t know if Malcolm Gladwell is right when he claims “the revolution will not be tweeted,” but I can say with certainty that the Twitter he describes is not the Twitter I know. Gladwell’s central argument is that Twitter creates weak ties but social movements require strong ties. I’ve made more strong ties through Twitter (and blogging) than I have through any communications medium I’ve ever used before. The relationships start off weak – a retweet, @ reply, or blog comment – but often strengthen through further discussions and eventually become new friendships and business relationships.

I can see why Gladwell gets this wrong – he doesn’t seem to really use Twitter (he does blog occasionally). I barely tweeted or blogged for a long time too. I read blogs basically since their advent, but social services are fundamentally participatory: reading blogs/tweets is to social services as watching TV is to a real life conversations. I finally relented at the insistence of Caterina, who had the foresight to insist that everyone at Hunch blog, tweet, contribute to open source projects, etc. I now get some of my best ideas from responses to tweets and blog posts, and have developed dozens of strong relationships through the experience.

I made some jokes on Twitter the past few days about Kleiner Perkins’ new social fund.  These were meant to be lighthearted: I only know one person at KP and from everything I’ve seen they seem to be smart, friendly people. But underneath the jokes lies a real issue: the partners there don’t seem to really participate in social services (something they only underscored by announcing their new fund at a press conference that targeted traditional media outlets).

I’d love to engage in a debate with smart people like Gladwell about the impact of the social web on culture, politics, activism and so on. I also think it’s great to see savvy investors like KP allocate significant resources to the next wave of social web innovation. But it’s hard for me to take them seriously when they don’t seem to take their subject matter seriously.

It is widely believed that a flourishing democracy requires an independent, diverse, and financially solvent press.  With print newspapers set to disappear in the next few years, the future of quality journalism is highly uncertain. This year, the online version of the New York Times will generate about $200M in revenue, a number that will need to approximately triple to support the current Times newsroom.

Most people who understand Internet economics believe that the best hope for online journalism is online advertising. Luckily, online advertising has significant room for improvement. Most of the revenue of the Times’ online business is generated through display ads. The main metric used to price display ads is derived from the rate at which users click on the ads, a rate which today is dismally low.  Thus the Times could continue to support its current newsroom staff if display ads became even moderately effective.

Lots of smart people are working on improving the efficacy of display advertising. Large companies like Google and Microsoft are investing billions in the problem. As usual, though, the best ideas are coming from startups. Companies like Blue Kai and Magnetic are bringing search intent (particularly purchasing intent – the core of Google’s profits) to display ads.  Companies like Media6Degrees are using social relations to target ads based on the principle that “birds of a feather flock together” (Facebook will likely start doing this soon as well).  Solve Media turns the hassle of registration into an engaging marketing event.  Convertro is working on properly attributing online purchases “up the funnel” from sites that harvest intent (search, coupon sites) to sites that generate intent (media, commerce guides). All told, there are a few hundred well-funded ad tech startups developing clever methods to improve display advertising.

Many of these targeting technologies rely on gathering information about users, something that inevitably raises concerns about privacy. Until recently, online privacy depended mostly on anonymity. There is a big difference between advertisers knowing, say, users’ sexual preferences and knowing users’ sexual preferences plus personally identifiable information like their names.  Like most people, I don’t mind if it’s easy to find my real name along with my job history, but I do mind if it’s easy to discover other personal details about me. When I’m not anonymous (e.g. on Facebook) I want to control what is disclosed – to have some privacy – but when I’m anonymous I’m far less concerned about information gathered for marketing purposes.

Before the rise of social networks, online ad targeting services (mostly) tracked people anonymously, through cookies that weren’t linked to personally identifiable information.  Social networks have provided the means to de-anonymize information that was previously anonymous. Apparently, the wall has been breached between 1) my real identity plus my self-moderated public information, and 2) my anonymous, non-self-moderated private information.

The good news is that the things users want to keep secret are almost always the least important things to online advertisers. It turns out that knowing people are trying to buy new washing machines or plane tickets to Hawaii is vastly more monetizeable than their names, who they were dating, or the dumb things they did in college. Thus, there are probably a set of policies that allow ad targeting to succeed while also letting users control what is associated with their real identities.  Hopefully, we can have an informed and nuanced debate about what these policies might be. The stakes are high.

Note:  As with almost everything I write on this blog, I have a ton of conflicts of interest.  Among them: I’m an investor, directly or indirectly, in a bunch of technology startups.  Some of these – including some companies mentioned above – are trying to create new advertising technologies. I am currently the CEO & Cofounder of Hunch, which among other things is trying to personalize the internet through an explicit user opt-in mechanism.

Many singles bars have “ladies’ night” where women are offered price discounts. Singles bars do this for women but not for men because (heterosexually-focused) bars are what economists call two-sided markets – platforms that have two distinct user groups and that get more valuable to each group the more the other group joins the platform - and women are apparently harder to attract to singles bars than men.

Businesses that target two-sided markets are extremely hard to build but also extremely hard to compete against once they reach scale. Tech businesses that have created successful two-sided markets include Ebay (sellers and buyers), Google (advertisers and publishers), Paypal (buyers and merchants), and Microsoft (Windows users and developers). In some cases individuals/institutions are consistently on one side (buyers and merchants) while in other cases they fluctuate between sides (Ebay sellers are also often buyers).

In almost every two-sided market, one side is harder to acquire than the other. The most common way to attract the hard side is the ladies’ night strategy: reduce prices for the hard side, even to zero (e.g. Adobe Flash & PDF for end-users), or below zero (e.g. party promotors paying celebrities to attend). Rarer ways to attract the hard side is 1) getting them to invest the platform itself (e.g. Visa & Mastercard), and 2) interoperating with existing hard sides (e.g. Playstation 3 running Playstation 2 games).

If you are starting a company that targets a two-sided market you need to figure out which side is the hard side and then focus your efforts on marketing to that side. Generally, the more asymmetric your market the better, as it allows you to market to each side more in serial than in parallel.

Reposted from Oct 7, 2010 from cdixon.posterous.com

By “incumbents” I mean the big companies that are loosely competitive to your startup.

- The first thing to do is try to understand the incumbent’s strategy.  For example, see my analysis of Google’s strategy.

- Being on an incumbent’s strategic roadmap is a double-edged sword.  On the one hand, they might copy what you build or acquire a competitor.  On the other hand, if you build a valuable asset you could sell your company the acquirer at a “strategic premium.”

- Incumbents that don’t yet have a successful business model (e.g. Twitter) might think they have a strategy, but expect it to change as they figure out their business model.  An incumbent without a successful business model is like a drunk person firing an Uzi around the room.

- Understand the incumbent’s acquisition philosophy. More mature companies like Cisco barely try to do R&D and are happy to acquire startups at high prices.  Incumbents that are immature like Facebook only do “talent acquisitions” which are generally bad outcomes for VC-backed startups (but good for bootstrapped or lightly funded startups). Google is semi-mature, and does a combination of talent and strategic acquisitions.

- Understand the incumbent’s partnership philosophy.  Yahoo and Microsoft are currently very open to partnerships with startups.  Google and Facebook like to either acquire or build internally. If you don’t intend to sell your company, don’t talk seriously about partnerships to incumbents that don’t seriously consider them.

- Every incumbent has M&A people who spend a lot of their time collecting market intelligence. Just because they call you and hint at acquisition doesn’t mean they want to buy you – they are likely just fishing for info. If they really want to buy you, they will aggressively pursue you and make an offer.  As VCs like to say, startups are bought, not sold.

- Try to focus on features/technologies that the incumbents aren’t good at.  Facebook is good at social and social-related (hard-core) technology.  Thus far they’ve kept their features at the “utility level” an haven’t built non-utility features (e.g. games, virtual goods, game mechanics).  Google thus far has been weak at social and Apple has been weak at web services.

- Try to focus on business arrangements that the incumbents aren’t good at.  Facebook and Google only do outbound deals with large companies.  With small companies (e.g. local venues, small publishers) they try to generate business via inbound/self service. Building business relationships that the incumbents don’t have can be a very valuable asset.

- Be careful building on platforms where the incumbent has demonstrated an inconsistent attitude toward developers. Apple rejects apps somewhat arbitrarily and takes a healthy share of revenues, but is generally consistent with app developers.  You can pretty safely predict what they will will allow to flourish. Twitter has been wildly inconsistent and shouldn’t be trusted as a platform.  Facebook has been mostly consistent although recently changed the rules on companies like Zynga with their new payment platform (that said, they generally seem to understand the importance of partners thriving and seem to encourage it).

- Take advantage of incumbents’ entrenched marketing positioning.  The masses think of Twitter as a place to share trivial things like what you had for lunch (even if most power users don’t use it this way) and Facebook as a place to talk to friends.  They are probably stuck with this positioning.  Normals generally think of each website as having one primary use case so if you can carve out a new use case you can distinguish yourself.

- Consider the judo strategy.  When pushed, don’t push back.  When Facebook adds features like check-ins, groups, or likes, consider interoperating with those features and building layers on top of them.