←

→

Search and the social graph

December 14th, 2009 | online advertising, search, tech companies

Google has created a multibillion-dollar economy based on keywords.  We use keywords to find things and advertisers use keywords to find customers.  As Michael Arrington points out, this is leading to increasing amounts of low quality, keyword-stuffed content. The end result is a very spammy internet. (It was depressing to see Tim Armstrong cite Demand Media, a giant domain-name owner and robotic content factory, as a model for the new AOL.)

Some people hope the social web — link sharing via Twitter, Facebook etc — will save us.  Fred Wilson argues that “social beats search” because it’s harder to game people’s social graph.  Cody Brown tweeted:

On Twitter you have to ‘game’ people, not algorithms. Look how many followers @demandmedia has. A lot less then you guys: @arrington @jason

These are both sound points. Lost amid this discussion, however, is that the links people tend to share on social networks – news, blog posts, videos – are in categories Google barely makes money on. (The same point also seems lost on Rupert Murdoch and news organizations who accuse Google of profiting off their misery).

Searches related to news, blog posts, funny videos, etc. are mostly a loss leaders for Google. Google’s real business is selling ads for plane tickets, dvd players, and malpractice lawyers. (I realize this might be depressing to some internet idealists, but it’s a reality). Online advertising revenue is directly correlated with finding users who have purchasing intent. Google’s true primary competitive threats are product-related sites, especially Amazon. As it gets harder to find a washing machine on Google, people will skip search and go directly to Amazon and other product-related sites.

This is not to say that the links shared on social networks can’t be extremely valuable.  But most likely they will be valuable as critical inputs to better search-ranking algorithms. Cody’s point that it’s harder to game humans than machines is very true, but remember that Google’s algorithm was always meant to be based on human-created links. As the spammers have become more sophisticated, the good guys have come to need new mechanisms to determine which links are from trustworthy humans. Social networks might be those new mechanisms, but that doesn’t mean they’ll displace search as the primary method for navigating the web.

Share:
  • Digg
  • del.icio.us
  • Facebook
  • Google Bookmarks
  • Reddit
  • Slashdot
  • Suggest to Techmeme via Twitter
  • Tumblr
  • Twitter

Related posts:

  1. Anatomy of a bad search result
  2. Google should open source what actually matters: their search ranking algorithm
  3. A massive misallocation of online advertising dollars
  4. News is a lousy business for Google too
  5. Speculation on Apple’s purchase of Quattro Wireless
  • Chris, i would challenge your point about social networks shared link do not have an enough of an impact on purchase intent: i think it is all related to the way brands will manage their digital life and timely distribute for-purchase links. When i see how dell is using Twitter to generate nearly 7m USD a year from this channel only, i guess that altogether when Brands really learn how to use SNS right, this might come to scale and therefore will change the way ad budget will be allocated. Probably not before 24 months from today
  • Whether it's a search on google, or on twitter, or on hunch.. what people do boils down to the algorithm used to power search.

    The algorithm itself defines the behaviour of the people that create the content. Every marketer focuses on SEO, and these content farms are no different. They aren't gaming anything. They are simply behaving in the way that the algorithm(s) value.

    As you point out Chris, the incentives are primarily economic, so the behaviour is to create content that, on average, will drive increased economic value to someone like demandmedia.

    A question we should ask ourselves is whether or not these behaviours are the right ones to incentivise?

    Should we be looking at other incentives that emphasise different positive behaviours -> socially, culturally, creatively or even personally?

    Should every page rank consider the social reputation of the people posting the link? How do you measure social reputation?

    Can we create algorithms that do this? Or is it always a double edged sword? i.e., do you always enable both positive and negative behaviours no matter what?

    P.S. I'm glad companies like demand media are around. If it wasn't for them, we wouldn't have to worry or innovate around this stuff for many more years. The more low quality content they create, the greater the demand for the algorithms to change and new solutions like hunch to be adopted. If Google open sourced their page rank algorithms... with visibility, these negative behaviours would have happened sooner and they would have been addressed sooner.
  • Enter in Bing Tweets & soon to be launched Google with real time top of the page results = the aggregation of conversation & site indexing. Plus google, bing and others are also indexing the social dialog and counting "term focus" into site link weight and organic ranking.

    Sure stuffing happens on site (can get sites sandboxed) but that is also a occurrence within the social conversation as well.

    So it all boils down to preference in search platform for finding information for the user.
  • Cody Brown
    I think for most people Google Search is a solid last resort.

    Given a choice I think most would rather have links/products recommended to them by people in their network as this comes with an extra layer of security. For example, If I buy a digital camera off a friends rec, I have someone to call if something goes wrong. If I buy it because it was on the first page of Google and it turns out to be crap, I don't think I'd get a response from Larry Page if I sent him an email.

    When it comes down to it Social Recommendations are the most powerful but they will always be limited by your individual network, everyone will have holes.

    This is why I think the most interesting company to come out in this space is Vark. They go after your 2nd 3rd degree connections, exactly the people you aren't aware of but are in a close enough social sphere -- very difficult to game.
  • "that the links people tend to share on social networks – news, blog posts, videos – are in categories Google barely makes money on"

    True, but part of Google's strength comes from the perception that they rank and filter all of the content on the web. From a Brand perspective I'd think they'd want to index this content even if it wasn't profitable, if only to maintain this illusion of comprehensiveness.
  • Ha, illusion! Love that description.
  • AndreaF
    Chris, totally agree with your view. I believe that we are heading towards a very different way of searching/consuming. Bing is moving the first steps in that direction as it's starting to help us more in taking action than a simple aggregation of content a la Google. I also think that Hunch is doing something right although the challenge is massive. I will only start using Hunch regularly when I will stop needing to go through the entire set of questions and answers and it (the system) will simply recognise me and have my recommendations ready for me. In order to do that you need massive scale but also peer-to-peer filtering; i.e. my results ought to be different from yours because of the influence people I know/admire/hang out with have provided more weigh to those results than other random people.
    On another note, I see an issue with Twitter, FourSquare and other similar services due to our own fault. We want to be seen and heard and connect with many more people than we need to; we follow many people and try to be followed by many others we don't have much in common with; this causes noise and clutter. I believe it's a temporary issue until the excitement of these new tools wear off, but we need better filters wether we are trying to stay up to date with tech news or are trying to buy a washing machine. We need real relevance (we are trying to solve that problem) and less un-useful content.
  • ronald
    Google makes money from:
    Business Value = Eyeballs * Intent
    while the old media was using (and still doesn't get over it)
    Business Value = Eyeballs * Demographics

    But the future of search is not reduction, since:
    Meaning = Data augmentation with [my] data
    Learning = Self Organization of Data (which build context, which leads to Information, which leads to knowledge ...)
    In other words we can teach a machine to understand, thereby reducing links to pointers of addresses only and eliminating them as an information organizational crutch.

    In other words the filter works by expanding and then definiteness. The expansion is out of the controll of content provider, very different from map reduce or any crutch algorithm currently used. The typical spam content will just fall out since it doesn't provide anything new/useful when put into context.
    Which will turn Google's intent upside down, since it can be extremely precise for a machine to find what you look for. Google needs fuzziness to grow.
    Which brings us back to the Demographics, who has the best demographics? Social networks.
    In other words the future is a combination of monetizing the new demographics with search.
  • I think social networks and SEO will continue to vie for supremacy in people's minds. That being said, the combination of the two is where the real power could lie. Could you imagine restaurant reviews or rankings based only on your FB friends reviews? That's something I'd love to see, it would be helpful for me today!
  • (Would comment above w/ Fred & Chris but it'd get squeezed.)

    Here is my take on social search math:

    _x_ Size of social graph (what is average?)

    __% of graph accessible on given platform

    __% of people with familiarity of a topic (some)
    __% of people with expertise on a topic (few)

    __% of people who care enough to review topic

    __% of people who have already reviewed, or
    __% of people who can review quickly for you

    _?_ ability to triangulate all of the opinions

    Fred's math <> Chris' math <> my math <> my Mom's math etc.

    IMO, if we had to rely on social search for most topics, we'd spend a lot of time waiting for results that were inconclusive. Ideally, I'd like to see a blend of hard data, expert reviews, user reviews & social search.
  • And a neural network to help tie the density functions together to best weight and create a conditional/weighted solution.
  • Or, more concisely, x=a^b(c-log d)
  • here's mine: many parallel
    f = w'x + b
  • Agree. You want a system that collects knowledge and learns. (Bias alert!)
  • aa
    i say use hunch.com for all product search!
  • for instance :)
  • Chris, great post. A few comments: first, re: combining social and search for high-value products/services, it seems Yelp has an effective model for this. I have used it recently to find a good auto body shop and a dentist in the L.A. area. Of course, Amazon is another good example. Interestingly, in these cases, I do not need to know the recommenders to find their input useful.

    Also, in this discussion of social vs. search, the assumption often seems to be that Google is limited to search and will never be competitive in social. However, Google seems to be making strides in social and with the ongoing rise of Gmail as a base of engagement, in 2-3 years Google could be more competitive in social as well.
  • You are quite right in relation to Google, for example, Google makes it very difficult to find a small business web sites in Finland. Google will first list the various contact information services webpages. After these search results you will find at page 2 or 3 the link for the company website, what you are looking for!
  • totally agree chris, but twitter and facebook need to work with merchants to get their customers sharing links of value. that's where the big win is.
  • I can see "Here's a song I love," but hard to see "here's a malpractice lawyer I love."
  • I also think it depends on how you are looking at social networks right now...maybe I wouldn't tweet about a malpractice lawyer I love, but I would certainly add them to my LinkedIn network (if I could) and I might talk him up to friends who are in need (probably sharing a link here and there as warranted).

    So I think as more data streams get added to our social networks (so beyond "what are you doing?" and into "where are you", "who are you with", "what do you think about X" [which btw is where I think hunch leads people], etc.)...the more that gets added, the more likely you are going to see solid niche data go through social network streams...

    I also think the subtle point lost here is very much what people initially didn't 'get' (myself included) about long form blogging...it's not always about sharing the data with other people, sometimes it's just about sharing the data with a future version of yourself (I can't tell you how many times I refer back to my own blog for 'how' or 'why' I did some bit of code...and more and more I find myself tweeting interesting links just as a casual way for me to bookmark something and be able to forget about it until a future version of me has time to go back and digest the actual content)...
  • Oh yeah Kevin I link to blog posts in comments whenever it's relevant. I could spend hours thinking over a concept and writing it up, and compress it into a link for those interested.

    I certainly do at least a couple dozen searches on Google or my blog a day "Victus spiritus economy of minds" etc.

    I get to enjoy shifts in perspective as well. Early hunches transform into more polished perspectives.
  • why not? i love marty schwimmer who is a copyright/trademark attorney. i'd tweet him up anytime.

    http://www.schwimmerlegal.com/
  • Btw Marty didn't like this, I did Fred
  • True Fred, but we're not normal. :)

    What percentage of searchers know a great copyright lawyer? Or have a friend that publishes quality content on the web? 'Search within your social graph' plays great with us that influence and report on the web, but I've gotta imagine we're in the top .001% of users.
  • Good reality check Marshall. It frames a market of super users that could really benefit from advanced social search tools.
  • Fred, what are the odds of you tweeting up marty schwimmer around the same time that someone in your SN is looking for an IP attorney?

    That's the fundamental difference between passive laid-back social discovery and active (I want it NOW!) search.

    But I think there's a lot of scope for leveraging passive SN shares into real-time search results (even though it does come with quite a lot of semantic baggage).
  • it doesn't have to be real time

    these systems can be searched
  • I would even argue that too many people are getting distracted by the 'real time' aspect of things these days...the short-term potential power to me is in the aggregation of our history more so than the awareness of what's happening right now (ie. learn from the past before we focus on the now or worry too much about the future).
  • I agree Kevin. Real-time is just the stream-of-consciousness layer of the web. Interesting, but only so as far as it informs the long-term memory systems of blog posts, comments, and position papers.
  • Yeah, based on the comments here and back at AVC, it looks like the discussion is gravitating towards a single solution - which is itself interesting - that social recommendations be integrated (with higher authority) into general RT search results.
  • jpmarcum
    I think anything that diverts usage from Google's omnibus is a big problem for Google whether it's against $1 CPCs or $.10 CPCs. Would also argue that the problem is greatly exacerbated if the activity is captured by either the owner of the public conversation and the influence hierarchy or the owner of private conversation, attention, the social graph and brand affinities (who is desperately trying to flush more private conversation out into the open).

    And that doesn't even include potential innovation: if Twitter or Facebook search parsed on relevance and source proximity over time you could see it being a highly valuable machine for plane tickets, dishwashers, etc.
  • Isn't the owner of a public conversation, the public?

    I'm interested because of the gears turning in my ratchedy brain that dream of a complete personal user profile that iteratively improves based on social media usage, blogging, purchasing habits, social graph, beliefs, passion...

    I basically want a machine readable version of my web pursuits, a digital ghost, and I'm willing to create a business to do just that.
  • jpmarcum
    It may be owned by the public but it lives at twitter and I'd argue that that is what matters most. While facebook is being aggressive in bringing public voices in (through product/celebrity fan pages) and making private conversations public (through dodgy privacy policy updates), it's largely a destination for private conversation.

    Is friendfeed enough of a ghost for you?
  • I have used friendfeed, and met some great folks there. But I'm thinking of something a little more active. A tool that helps take all the bits of information I publicly share, and generates great finds/leads/etc for me to review while I have time.
  • Woah! High comment density alert! I think you're talking about who owns the data, right?

    That's the million dollar question. At the moment, the answer is whoever has ultimate access to the DB on which it's stored.

    But I see that changing over time...
  • jpmarcum
    ha! guilty as charged - i have powerpointed and tweeted my brain to the point where any reversion to prose results in a gummy tar.

    yes, am talking about who owns the data but also user expectations of the data they'll find at each destination.
  • i think you point out a really interesting distinction about "twitter search." everyone talks about "real time" search as if its about getting stuff happening now. i find this interesting only in certain cases and i don't think its very monetizable. Much more monetizable is searching older stuff that is endorsed by your social graph.
  • Absolutely Chris! Digging up a recommendation from a trusted firend 3 years later can give us a better starting point when we as consumers seek out a business.

    Side note: have you considered passively aggregating users social data to "automatically" answer some of the hunch questions. I'm starting with a short tag memory derived from semantic extraction, but have plans to build up a better data profile on users in the future by slowly filling in a machine based image of their likes/dislikes. The long term goal being creating personalized virtual assistants to aid in search.
  • i agree. there is still a big opportunity in getting people to share information regarding mundane purchase decisions (whether about lawyers or furniture)...so that someone in their social graph can leverage that knowledge years down the line.
  • perhaps. as you know I'm also a big believer in having a good "read/write" ratio - the internet having a good memory a la wikipedia. so people don't have to ask you the same question over and over, you can collect the best wisdom from a bunch of different people, etc. obviously I am hugely biased here re hunch. .better than google for finding a washing machine btw ;)
  • Exactamundo
  • Smart personal twist on this important topic. Chris, thnx.

    I'm certain that you are correct that social will drive inputs for better search algorithms. I'm a believer that it will work the other way as well and that search will drive me to find more social contacts in niche interest areas.

    If that is true Google still prospers under the current model. I don't want to break their model, just spend less time finding what I want.
  • But the problem with human generated/rated content is that you CAN game humans to an extent. (Slightly harder than you can game humans), but so often it all comes down to a question of taste. What is right for the majority is not always right for you. http://thebln.com/2009/12/machines-like-empty-c...
blog comments powered by Disqus