What’s not evil: ranking content fairly *and* letting public content get indexed

Please see update at bottom

Most websites spend massive amounts of time and money to get any of their pages index and ranked by Google’s search engine. Indeed, there is a entire billion dollar industry (SEO) devoted to helping companies get their content indexed and ranked.

Twitter and Facebook have decided to disallow Google from indexing 99.9% of their content. Twitter won’t let Google index tweets and Facebook won’t let Google index status updates and most other user and brand generated content. In Facebook’s case this makes sense for content that users have designated as non-public. In Twitter’s case, the vast majority of the blocked content is designated by users as public. Furthermore, Twitter’s own search function rarely works for tweets older than a week (from Twitter’s search documentation, they return “6-9 days of Tweets”).

There is a debate going today in the tech world: Facebook and Twitter are upset that Google won’t highly rank the 0.1% of their content they make indexable. Facebook and Twitter even created something they call the “Don’t be evil” toolbar that reranks Google search results the way they’d like them to be ranked. The clear implication is that Google is violating their famous credo and acting “evil”.

The vast majority of websites would dream of having the problem of being able to block Google from 99.9% of their content and have the remaining 0.1% rank at the top of results. What would be best for users – and least “evil” – would be to let all public content get indexed and have Google rank that content “fairly” without favoring their own content. Facebook and Twitter are right about Google’s rankings, but Google is right about Facebook and Twitter blocking public content from being indexed.

Update: after posting this I got a bunch of emails, tweets and comments telling me that Twitter does in fact allow Google to index all their tweets, and that any missing tweets are the fault of Google, not Twitter. A few people suggested that without firehose access Google can’t be expected to index all tweets. At any rate, I think the “Why aren’t all tweets indexed?” issue is more nuanced than I argued above.

Forces that affect whether a large company will buy your product (according to Marc Andreessen)

From Marc Andreessen’s “Moby Dick Theory of Big Companies“:

You can count on there being a whole host of impinging forces that will affect the dynamic of decision-making on any issue at a big company.

The consensus building process, trade-offs, quids pro quo, politics, rivalries, arguments, mentorships, revenge for past wrongs, turf-building, engineering groups, product managers, product marketers, sales, corporate marketing, finance, HR, legal, channels, business development, the strategy team, the international divisions, investors, Wall Street analysts, industry analysts, good press, bad press, press articles being written that you don’t know about, customers, prospects, lost sales, prospects on the fence, partners, this quarter’s sales numbers, this quarter’s margins, the bond rating, the planning meeting that happened last week, the planning meeting that got cancelled this week, bonus programs, people joining the company, people leaving the company, people getting fired by the company, people getting promoted, people getting sidelined, people getting demoted, who’s sleeping with whom, which dinner party the CEO went to last night, the guy who prepares the Powerpoint presentation for the staff meeting accidentally putting your startup’s name in too small a font to be read from the back of the conference room…

Man, I wish Marc still blogged.  (ht saul lieberman)

The lodsys case

Lodsys filed a complaint last week against a number of online retailers.  Lodsys is a so-called “non-practicing entity” – a patent holding company that doesn’t build products.  They previously filed lawsuits against Apple developers among others.  I am not a lawyer but I wanted to try to understand the case so read some of the documents.  This is my best understanding of the situation.

Here is an example of Lodsys’ accusations, in this case against bestbuy.com:

The reference to ’908 is referring to patent 5,999,908, which was filed in 1997.  It is an invention that appears to allow users to give feedback to websites:

Here is one example diagram from the patent about how the invention could be embodied:

Here is the specific claim (claim 37) that bestbuy.com and other retailers are allegedly infringing:

As far as I can tell, the first clause says that it the claim is referring to a computer product, which in the case of bestbuy.com seems to be their website, where the user and server can send information back and forth.  The second clause seems to say that this system includes computer code. The third clause seems to say this system includes a database.  The fourth clause seems to say there is a information transmitted between the user, web server and database.

Accurate contrarian theories

When Google released its search engine in 1998, its search results were significantly better than its competitors’. Many people attribute Google’s success to this breakthrough technology. But there was another key reason:  a stubborn refusal to accept the orthodox view at the time that “stickiness” was crucial to a website’s success. Here’s what happened when they tried to sell their technology to Excite (a leading portal/search engine in the late 90s):

[Google] was too good. If Excite were to host a search engine that instantly gave people information they sought, [Excite’s CEO] explained, the users would leave the site instantly. Since his ad revenue came from people staying on the site—“stickiness” was the most desired metric in websites at the time—using Google’s technology would be counterproductive. “He told us he wanted Excite’s search engine to be 80 percent as good as the other search engines,” … and we were like, “Wow, these guys don’t know what they’re talking about.” – Steven Levy, In The Plex (p. 30)

Famed investor/entrepreneur Reid Hoffman says world-changing startups need to be premised on “accurate contrarian theories.”  In Google’s case, it was true but non-contrarian to think users would prefer a better search engine. What was true and contrarian was to think it made business sense to get users off their site as quickly as possible. The business model to support this contrarian theory wouldn’t emerge until years later, and by then Google would already have become the world’s most popular search engine.

Apple and the TV industry

The TV industry is a major segment of the consumer electronics industry and Apple is the leading consumer electronics company in the world. Thus far Apple has entered the TV market with a stand-alone device, Apple TV. There has been speculation about whether Apple might enter the TV market by creating an actual TV. The most convincing objections to that idea cite the unfavorable industry structure: the power of the cable operators, the low margins on TVs, the infrequency of people buying new TVs, etc.

I thought it would be interesting to go back and look at the reasoning analysts used to predict the failure of the iPhone before its launch in 2007. Some predicted it would fail because the other handset makers would successfully compete with Apple:

The iPod also conquered the problem of small screens and cheesy navigation. With its newfound popularity, the company was also able to get music publishers to agree to its terms. Unfortunately for Apple, problems like that don’t exist in the handset business. Cell phones aren’t clunky, inadequate devices. Instead, they are pretty good. Really good. Why do you think they call it a Crackberry? Because the lumpy design and confusing interface of the device is causing people to break into cars? No, it’s because people are addicted to it. Samsung has scoured the world’s design schools and hired artists on three continents to keep its phones looking good. Motorola has revived its fortunes with design. KDDI, a Japanese carrier, has a design showcase in the teen shopping area of Tokyo just to be close to trends. And Sharp doesn’t skimp when it comes to putting LCD TVs on its phones. Apple, in other words, won’t be competing against rather doltish, unstylish companies like the old Compaq. The handset companies move pretty quick and put out new models every few weeks. [emphasis added]

Other analysts predicted Apple’s phone was doomed because of the mobile phone industry structure – mobile operators commanded so much power via subsidies, retail distribution etc:

Apple will launch a mobile phone in January, and it will become available during 2007. It will be a lovely bit of kit, a pleasure to behold, and its limited functionality will be easy to access and use. The Apple phone will be exclusive to one of the major networks in each territory and some customers will switch networks just to get it, but not as many as had been hoped. As customers start to realise that the competition offers better functionality at a lower price, by negotiating a better subsidy, sales will stagnate. After a year a new version will be launched, but it will lack the innovation of the first and quickly vanish. The only question remaining is if, when the iPod phone fails, it will take the iPod with it.  [emphasis added]

I am not citing these analysts to mock them. Hindsight is 20/20 and it was quite reasonable at the time to assume that a new phone from Apple would confront the same issues that new phones from other companies confronted. What Apple ended up doing, however, was creating a phone that was so incredibly desirable to consumers that it completely restructured the industry, causing a massive shift of power away from the carriers.

Regarding the TV industry, here is what Steve Jobs said last year at AllThingsD:

Q: Is it time to throw out the interface for TV? Does television need a new human interface.

A: The problem with innovation in the TV industry is the go-to-market strategy. The TV industry has a subsidized model that gives everyone a set top box for free. So no one wants to buy a box. Ask TiVo, ask Roku, ask us… ask Google in a few months. The television industry fundamentally has a subsidized business model that gives everyone a set-top box, and that pretty much undermines innovation in the sector. The only way this is going to change is if you start from scratch, tear up the box, redesign and get it to the consumer in a way that they want to buy it. But right now, there’s no way to do that….The TV is going to lose until there’s a viable go-to-market strategy. That’s the fundamental problem with the industry. It’s not a problem with the technology, it’s a problem with the go-to-market strategy….I’m sure smarter people than us will figure this out, but that’s why we say Apple TV is a hobby.

So Jobs doesn’t believe an “additional box” is a viable strategy for seriously entering the TV industry. This leaves three places to enter: 1) integrating into set top boxes, 2) integrating into other TVs, or 3) Apple creating its own TV. Regarding #1, the last thing the cable operators want is for internet-delivered programming that bypasses their cable channels to become widespread – they see that as the fast track to become a dumb pipe. Re #2: This just seems very unlike Apple – the most vertically integrated company in tech, and famous for wanting to control every aspect of the product and user experience.

Re #3, let’s imagine Apple develops a TV that is as groundbreaking as the iPhone was. The biggest problem “smart TVs” have today is that they need clunky IR transmitters to control set top boxes because the cable operators won’t willingly interoperate. So a new Apple TV would have to drum up such incredible consumer demand that the operators would feel compelled to support it. This does indeed seem harder in the TV than in the mobile industry. At least in the US you had 4 nationwide mobile operators at the time of the iPhone launch. In TV, consumers normally have at most two real choices for traditional cable programming – cable and satellite – and two real choices for two-way internet – cable and DSL/FIOS.

Perhaps Apple won’t enter the market due to its structure. But that didn’t stop them in mobile phones where the structure was similarly difficult. The mistake analysts made about the iPhone was to assume the current industry structure would be sustained after Apple’s entry. I’d be wary of making the same assumption about the TV industry.