A massive misallocation of online advertising dollars

In an earlier blog post, I talked about how sites that generate purchasing intent (mainly “content” sites) are being under-allocated advertising dollars versus sites that harvest purchasing intent (search engines, coupon sites, comparison shopping sites, etc).  As a result, most content sites are left haggling over CPM-based brand advertising instead of sponsored links for the bulk of their revenue.

But there is an additional problem:  even among sites that monetize via sponsored links there is a large overallocation of advertising spending on links that are near the “end of the purchasing process” (or “end of the funnel”). For example, an average camera buyer takes 30 days and clicks on approximately 3 sponsored links from the beginning of researching cameras to actually purchasing one.   Yet in most cases only the last click gets credit, by which I mean:  1) if it’s an affiliate (CPA) deal, it is literally usually the case that only the last affiliate (the site that drops the last cookie) gets paid, 2) if it’s a CPC or CPM deal, most advertisers don’t properly track the users across multiple site visits so simply attribute conversion to the most recent click, causing them to over-allocate to end-of-funnel links 3) if it’s a non-sponsored link (like Google natural search links) the advertiser might over-credit SEO when in fact the natural search click was just the final navigational step in a long process that involved sponsored links along the way.

What this means is there are two huge misallocations of advertising dollars online: the first from intent generators to intent harvesters; the second from intent harvesters that are at the beginning or middle of the purchasing process to those at the end of the purchasing process.  This is not just a problem for internet advertisers and businesses – it affects all internet users.  Where advertising dollars flow, money gets invested. It is well known that content sites are suffering, many are even on their way to dying. Additionally, product/service sites that started off focusing on research are forced to move more and more toward end-of-funnel activities.  Take a look at how sites like TripAdvisor and CNET have devoted increasing real estate to the final purchasing click instead of research.  For the most part, you don’t get paid for the actual research since it’s too high in the funnel.

As with all large problems, this misallocation of advertising dollars also presents a number of opportunities.  One opportunity is for advertisers to correctly attribute their spending by tracking users through the entire purchasing process (in the case of cameras, the full 30 days and multiple sponsored clicks).  Very likely, these sites are currently overpaying end-of-funnel sites (e.g. coupon sites) and underpaying top-of-funnel sites (e.g. research sites). There is also an opportunity for companies that provide technology to help track this better. Finally, if over time advertising dollars do indeed shift to being correctly allocated, this will allow research sites to be pure research sites, content sites to be pure content sites, etc instead of everyone trying to clutter their sites with repetitive, “last click” functionality.


Every time an engineer joins Google, a startup dies

VC returns over the last decade have been poor. The cause is widely agreed to be an excess of venture capital dollars to worthy startups. Observers seem to universally assume that the solution is for the VC industry to downsize.

For example, Fred Wilson says about VC:

You cannot invest $25bn per year and generate the kinds of returns investors seek from the asset class. If $100bn per year in exits is a steady state number, then we need to work back from that and determine how much the asset class can manage…. I think “back to the future” is the answer to most of the venture capital asset class problems. Less capital in the asset class, smaller fund sizes, smaller partnerships, smaller deals, and smaller exits

Similarly, Bill Gurley writes:

There are many reasons to believe that a reduction in the size of the VC industry will be healthy for the industry overall and should lead to above average returns in the future.

All of these analyses start with the assumption that aggregate venture-backed exits (acquisition and IPOs) will remain roughly constant. I don’t see why we need to accept that assumption. The aggregate value of venture-backed startups, like all valuations, is a function of profits generated (or predicted to be generated). In technology, profits are driven by innovation. I don’t see any reason we should assume venture-backed innovation can’t be dramatically increased.

For example, innovation has varied widely across times and places – the most innovative region in the world for the last 50 years being Silicon Valley. What if, say, Steve Jobs hadn’t grown up in Silicon Valley? What if he had gone to work for another company? Does anyone really think Apple – and all the innovation and wealth it created – would exist if Jobs hadn’t happened to grow up in a culture that was so startup friendly? Jobs is obviously a remarkable person, but there are probably 100 Steve Jobs born every year. The vast majority just never have a chance or give a thought to starting a revolutionary new company.

Some people blame our education system, or assume that there is some fixed number of entrepreneurs born every year. I think the problem is cultural. As much as we like to think of our culture as being entrepreneurial, the reality is 99% of our top talent doesn’t seriously contemplate starting companies. Colleges crank out tons of extremely smart and well-educated kids every year. The vast majority go into “administrative” careers that don’t really produce anything – law, banking and consulting. Most of the rest join big companies. As I’ve argued many times before, big companies (with a few notable exceptions) aren’t nearly as successful as startups at creating new products.  The bigger the company, the more likely it suffers from agency issues, strategy taxes, and myopia. But most of all: nothing is more motivating and inspiring than the sense of ownership and self-direction only a startup can provide.

Whenever I see a brilliant kid decide to join Goldman Sachs, McKinsey, or Google, I think to myself: a startup just died, and as a result our world is a little less wealthy, innovative, and interesting.

Collective knowledge systems

I think you could make a strong argument that the most important technologies developed over the last decade are a set of systems that are sometimes called “collective knowledge systems”.

The most successful collective knowledge system is the combination of Google plus the web. Of course Google was originally intended to be just a search engine, and the web just a collection of interlinked documents. But together they provide a very efficient system for surfacing the smartest thoughts on almost any topic from almost any person.

The second most successful collective knowledge system is Wikipedia. Back in 2001, most people thought Wikipedia was a wacky project that would at best end up being a quirky “toy” encyclopedia. Instead it has become a remarkably comprehensive and accurate resource that most internet users access every day.

Other well-known and mostly successful collective knowledge systems include “answer” sites like Yahoo Answers, review sites like Yelp, and link sharing sites like Delicious.  My own company Hunch is a collective knowledge system for recommendations, building on ideas originally developed by “collaborative filtering” pioneer Firefly and the recommendation systems built into Amazon and Netflix.

Dealing with information overload

It has been widely noted that the amount of information in the world and in digital form has been growing exponentially. One way to make sense of all this information is to try to structure it after it is created. This method has proven to be, at best, partially effective (for a state-of-the-art attempt at doing simple information classification, try Google Squared).

It turns out that imposing even minimal structure on information, especially as it is being created, goes a long way. This is what successful collective knowledge systems do. Google would be vastly less effective if the web didn’t have tags and links. Wikipedia is highly structured, with an extensive organizational hierarchy and set of rules and norms. Yahoo Answers has a reputation and voting system that allows good answers to bubble up. Flickr and Delicious encourage user to explicitly tag items instead of trying to infer tags later via image recognition and text classification.

Importance of collective knowledge systems

There are very practical, pressing needs for better collective knowledge systems. For example, noted security researcher Bruce Schneier argues that the United States’ biggest anti-terrorism intelligence challenge is to build a collective knowledge system across disconnected agencies:

What we need is an intelligence community that shares ideas and hunches and facts on their versions of Facebook, Twitter and wikis. We need the bottom-up organization that has made the Internet the greatest collection of human knowledge and ideas ever assembled.

The same could be said of every organization, large and small, formal and and informal, that wants to get maximum value from the knowledge of its members.

Collective knowledge systems also have pure academic value. When Artificial Intelligence was first being seriously developed in the 1950′s, experts optimistically predicted they’d create machines that were as intelligent as humans in the near future.  In 1965, AI expert Herbert Simon predicted that “machines will be capable, within twenty years, of doing any work a man can do.”

While AI has had notable victories (e.g. chess), and produced an excellent set of tools that laid the groundwork for things like web search, it is nowhere close to achieving its goal of matching – let alone surpassing – human intelligence. If machines will ever be smart (and eventually try to destroy humanity?), collective knowledge systems are the best bet.

Design principles

Should the US government just try putting up a wiki or micro-messaging service and see what happens? How should such a system be structured? Should users be assigned reputations and tagged by expertise? What is the unit of a “contribution”? How much structure should those contributions be required to have? Should there be incentives to contribute? How can the system be structured to “learn” most efficiently? How do you balance requiring up front structure with ease of use?

These are the kind of questions you might think are being researched by academic computer scientists. Unfortunately, academic computer scientists still seem to model their field after the “hard sciences” instead of what they should modeling it after — social sciences like economics or sociology. As a result, computer scientists spend a lot of time dreaming up new programming languages, operating system architectures, and encryption schemes that, for the most part, sadly, nobody will every use.

Meanwhile the really important questions related to information and computer science are mostly being ignored (there are notable exceptions, such as MIT’s Center for Collective Intelligence). Instead most of the work is being done informally and unsystematically by startups, research groups at large companies like Google, and a small group of multi-disciplinary academics like Clay Shirky and Duncan Watts.

What’s strategic for Google?

Google seems to be releasing or acquiring new products almost daily. It’s one thing for a couple of programmers to hack together a side project. It’s another thing for Google to put gobs of time and money behind it. The best way to predict how committed Google will be to a given project is to figure out whether it is “strategic” or not.

Google makes 99% of their revenue selling text ads for things like airplane tickets, dvd players, and malpractice lawyers. A project is strategic for Google if it affects what sits between the person clicking on an ad and the company paying for the ad. Here is my rough breakdown of the “layers in the stack” between humans and the money:

Human – device – OS – browser – bandwidth –  websites – ads – ad tech – relationship to advertiser – $$$

At each layer, Google either wants to dominate it or commoditize it. (For more on the strategic move known as commoditizing the complement, see here, here and here). Here’s my a brief analysis of the more interesting layers:

Device: Desktop hardware already commoditized. Mobile hardware is not, hence Google Phone (Nexus One).

OS: Not commoditized, and dominated by archenemy (Microsoft)!!   Hence Android/Google Chrome OS is very strategic. Google also needs to remove main reasons people choose Windows. Main reasons (rational ones – ignoring sociological reasons, organizational momentum etc) are Office (hence Google Apps), Outlook (hence Gmail etc), gaming (look for Google to support cross-OS gaming frameworks), and the long tail of Windows-only apps (these are moving to the web anyways but Google is trying to accelerate the trend with programming tools).

Browser: Not commoditized, and dominated by arch enemy! Hence Chrome is strategic, as is alliance with Mozilla, as are strong cross-browser standards that maintain low switching costs.

Bandwidth:  Dominated by wireless carriers, cable operators and telcos. Very hard for Google to dominate without massive infrastructure investment, hence Google is currently trying to commoditize/weaken via 1) more competition (WiMAX via Clearwire, free public Wi-Fi) 2) regulation (net neutrality).

Websites/search (“ad inventory”): Search is obviously dominated by Google. Google’s syndicated ads (AdSense) are dominant because Google has the highest payouts since they have the most advertisers bidding. This in turn is due largely to their hugely valuable anchor property, Google.com. Acquired Youtube to be their anchor property for video/display ads, and DoubleClick to increase their publisher display footprint. On the emerging but fast growing mobile side, presumably they bought AdMob for their publisher relationships (versus advertiser relationships where Google is already dominant). The key risks on this layer are 1) people skip the ads altogether and go straight to, say, Amazon to buy things, 2) someone like Facebook or MS uses anchor property to aggressively compete in syndicated display market.

Relationships to advertisers:  Google is dominant in non-local direct-response ads, both SMB self serve and big company serviced accounts.  They are much weaker in display. Local advertisers (which historically is half of the total ad market) is still a very underdeveloped channel – hence (I presume) the interest in acquiring Yelp.

This doesn’t mean Google will always act strategically. Obviously the company is run by humans who are fallible, emotional, subject to whims, etc. But smart business should be practiced like smart chess: you should make moves that assume your opponents will respond by optimizing their interests.

Why the web economy will continue growing rapidly

Here’s the really good news for the web economy over the next decade.  Consumers are spending more and more time online, yet only about 10% of all advertising dollars are spent there.

Let’s assume that, over time, ad spending on a medium becomes roughly proportional to the time consumers spend using that medium. I doubt there are any technologists reading this blog who doubt that in five years most people in industrialized countries will spend 50% or more of their “media time” on the web.  This means there are hundreds of billions of ad revenues waiting to move to the web.

Advertising is usually divided into two categories: direct-response and brand advertising. Direct-response advertising tries to get users to take immediate action. Brand advertising tries to build up positive associations over time in people’s minds. In the past decade, we saw a massive shift of direct response advertising to the web. The main beneficiary of this shift has been Google. We saw far less of a shift of brand advertising to the web.

It is therefore very likely that most of this new ad spending will be brand advertising.  This is why Google, Yahoo and Microsoft are all so intensely focused on display advertising. It is why they paid huge premiums to acquire Doubleclick, Right Media, and Avenue A.

Right now there are lots of inhibitors to brand advertising dollars flowing onto the web. Among them 1) most of the brand dollars are controlled by ad agencies, who seem far more comfortable with traditional media channels, 2) it is hard to know where your online advertising is appearing and whether it is effective, 3) banner ads seem extremely ineffective and are often poorly targeted, 4) big brand advertisers seem scared of user-generated content, today’s major source of ad inventory growth.

But economic logic suggests these problems will be figured out, because advertisers have no choice but to go where the consumers are.