Take for instance the coupling of a dozen software services that all independently produce improved customer purchase action. The combination of these independent sources in a novel way, can give a multiplicative value ad of 1.1^12 or a factor of 3 improvement. Now expand this to hundreds or even thousands of independent techniques or connections within a network and you can reveal massive improvements in quality.

A brain with only a couple of nodes is pretty weak. A mind with billions of nodes and hundreds of billions of connections is capable of advanced conscious connections, creativity, and unpredictable value advancement. The potent software applications of the future will exist and thrive by utilizing the network of APIs optimally in the construction of their databases and decision architectures.

If your curiosity is piqued but you're still not convinced, check out some of Kevin Kelly's swarm and emergence concepts. I really enjoy some of his far out predictions and their closer than most folks would guess (in the 10-20 year horizon).

10th popular post, and I give it a 7/10 only because you didn't tie in network effects.

Take for instance the coupling of a dozen software services that all independently produce improved customer purchase action. The combination of these independent sources in a novel way, can give a multiplicative value ad of 1.1^12 or a factor of 3 improvement. Now expand this to hundreds or even thousands of independent techniques or connections within a network and you can reveal massive improvements in quality.

A brain with only a couple of nodes is pretty weak. A mind with billions of nodes and hundreds of billions of connections is capable of advanced conscious connections, creativity, and unpredictable value advancement. The potent software applications of the future will exist and thrive by utilizing the network of APIs optimally in the construction of their databases and decision architectures.

If your curiosity is piqued but you're still not convinced, check out some of Kevin Kelly's swarm and emergence concepts. I really enjoy some of his far out predictions and their closer than most folks would guess (in the 10-20 year horizon).

I agree. You might like to check out FluidDB, which is all about using a better data representation to change how we work with information. See http://fluidinfo.com and http://blogs.fuidinfo.com/fluidDB

I've also written about this exact subject a few times. A starting link:
http://blogs.fluidinfo.com/terry/2007/03/19/why…

And I talk a little about Alex Wright at http://blogs.fluidinfo.com/terry/2008/01/04/tag…

Please feel free to get in touch, I'm terry fluidinfo com and would be happy to go into more depth / hear more from you, etc.

For example, I had some work a few years ago that used HMMs to label chord information on a set of Beatles tunes. However, instead of using massive amounts of training data on the HMMs, I used zero training data. Zero. Instead, I initialized the HMM using musicologically-sensible initial conditions, and then I adjusted the standard HMM E-M algorithm so that the [B] output probability matrix did NOT get updated; it stayed stable.

All of the intelligence was in the algorithm. There was no human labeled data. And the algorithm performed best in class — better than solutions that had been trained on lots of human-labeled data.

So to say we can't make smarter algorithms, or that breakthroughs will only come via data, simply doesn't sit right with me.

I think what people tend to mean when they say “it's all about the data” is that there is no purpose coming up with better general purpose machine learning algorithms, i.e. SVMs vs. gaussian mixture models vs. Markov random fields vs. whatever. If that's your main point, then I agree.

But we can also create more intelligent, specialized, intelligent algorithms by building our own smarts into a general purpose ML algorithm, thereby making the algorithm smarter. And we can do it without the need for massive amounts of data. Again, imho.

So I just don't buy that

But isn't that the point? At the end of the day, no matter what the reason, you resorted to clever algorithms, rather than large data. So the thing about data being the only good source of future AI breakthroughs just ain't true. Relying on large data isn't “wrong”. It's just not the universal panacea that you make it out to be.

The way I see it, there is a small head of large-scale, breadth-loving domains (e.g. web) and/or tasks (e.g. known-item finding.. rather than exploratory search..) in which large data is very appropriate.

At the same time, there is a long tail of medium and small-scale, depth-loving domains (e.g. content-based music search) and tasks (e.g. exploratory search) in which large data does not give you as much as an intelligently-constructed algorithm.

So what if the only reason you can construct those algorithms is because the domain is well-enough constrained? We know from power-law distributions that the volume (usage, whatever) of tasks and problems in the tail sums up to be equal in magnitude to the head.

So at the most, you can say that large data will help you make AI breakthroughs in *half* of the open problems. Intelligent algorithms will still be necessary for the other half. imho.

So it seems to me that the only way out of the constrains imposed upon Bing search is for Bing to come up with clever-er algorithms that do something different and better, despite the gradient which the data is pointing it toward.

If one relies on the data alone, one will not solve a very large range of AI problems. Intelligent algorithms are needed to make those breakthroughs.

Re domain specific – I agree, but I think the reason what you say works is precisely because the domain becomes small enough that you can do all sorts of things to fill in the gaps in the data. I think of my last company, SiteAdvisor, as a data company. The way we got from 80% to near 100% was all sorts of techniques, from hacks to manual processes to integrating other data sets. We couldn't have used those techniques in a horizontal setting.

While I agree with the thrust of your post, you've taken a very horizontal view of the problem. I do not spend time on Google-scale problems, but on much more targeted, vertical solutions to the “big data” problem. By layering domain specificity onto the problem of semantic analysis, many of the pitfalls of NLP and AI become far more manageable. I'm not saying they're a panacea, and certainly not when trying to solve problems in real-time, but they can take you a lot farther than when applying them to horizontal data sets.

And yes, tagging rich data and creating additional metadata for analysis holds many of the keys to extracting true meaning from unstructured data sets. I could write on this topic for hours. Thanks for the post.

Roger

I have seen entities make big mistakes in trying to solve major problems with machine learning (and resting on their laurels) without considering the fact that not all data needed that influences the outcomes are being sourced or even though about.

http://calebelston.com/are-algorithms-the-magic…

Re semantic tagging – If publishers were to ubiquitously start doing so, that would qualify as a massive new data source in my way of thinking. People have been talking about that for years but right now their is no real incentive for publishers to do it. Maybe if Google made it help you SEO or something people would start to care.

Starting with the Web 1.0 where it's onerous of publishers to generate the data sources, Web 2.0 made it possible to catapult the rate of data generation possible. The big outstanding question is that the user-generated data cannot really be used to structure due to 1) unstructuredness, 2)lack of cleanliness, 3) lack of irrelevance, at least to systematically digest what human beings really try to convey, etc. Many user contents are indeed useful, but not much money floating around to buy them.

The semantic web never delivered its promise because people keep banging their heads against structured vs. unstructured data when the real problem is structuring inconsistent, meaningless, and sometime garbage data wouldn't really lead to a value creation for the ultimate data consumer.

If I were to aggregate all the world's information (cost aside) and structure the data somehow, would I be able to answer all questions in the universe? I have doubts. Who knows?

It is good to hark back to Google in 1998 or to the nascent WWW ten years earlier. To be thinking of what would it take to make another radical improvement in information management. My view is that the next generational shift will be ubiquitous semantic tagging of public data by the publisher. These tags will be interpreted using consistent, open algorithms but they will be interpreted subjectively by each subscriber, according to private data unevenly distributed across the system.
The high cost of creating tags is an empirical observation: true in respect of the Semantic Web and no doubt other systems, but not a universal law. When the requirement for objectivity is dropped, semantic tags with good-enough efficacy can be created at very low marginal cost.

http://irgupf.com/2009/04/09/retrievability/
http://irgupf.com/2009/04/23/retrievability-and…
http://irgupf.com/2009/04/09/large-data-versus-…

In a nutshell, large data allows you to solve certain types of problems well, but may end up making other types of problems much more difficult, if all you have is naive Bayes on top of that data making your inferences.

AI exists for a purpose, a specific function or task. Just like a basic lifeform needs relevant environmental information to increase it's chances of success, AI functions best with access to the richest, most rapidly computable, system/task-relevant data. e.g. robots that can navigate the Darpa Road Challenge need maps, real-time road/environment sensors, ability to sense and determine the meaning of signs, etc. — Circling back to the original point, the algorithm is just part of the AI – the other part is an environment of structured data. Intelligence arises from the interplay of the two, depends on the system context. So we can expect the algorithms that most effectively draw on the best data available to them for given tasks to be most successful – that means ongoing rise of AI-ish bots tailored for / carefully tuned to new data environments increasingly capable of performing more complex tasks. Clearly there's an expanding market for these (search being a huge part of that), as Norvig and company have realized.

When considering complementary data sources and the drive to increase intelligence in the system, it's occurred to me that we generally appear to be trending toward the super-structuring of all data (the everything graph), or total system quantification. By cross-referencing different rich data sets, we can interpolate value, push toward quantification / state closure, generating much value and “intelligence” along the way. If it becomes understood that this process is making our system smarter, then data may continue to centralize, be drawn together for certain higher uses, thus commoditizing current data structures, algorithms, and combinations thereof.

Related articles that explore these thoughts:
http://www.memebox.com/futureblogger/show/1591-…
http://memebox.com/futureblogger/show/1518-inte…
http://socialnode.blogspot.com/2009/06/simulati…

there's only one thing that really bothers me about the Netflix competition (which I would imagine must not be unique)… what was it about the 10% threshold that so many people and teams ended up getting to 8 and 9% relatively quickly, but the last year or two of the competition ended up being dominated by incremental progress to the 10% mark.

Do you think if they had set the threshold to 15% the results would have been different? Do you think the Netflix team had some unique insight that there was some magic boundary around 10%…

Improved solutions to existing problems might not have nearly as large an impact as figuring out “new” problems that can be solved using ML.

Hi Chris, the companies are indeed very aware of the centrality of data. You and your readers may want to check out this recent article by three of Google's research scientists:

“The Unreasonable Effectiveness of Data”
http://googleresearch.blogspot.com/2009/03/unre…
http://www.computer.org/portal/cms_docs_intelli…

1. Axiom Structure Unstructured Data:
2. Corollary: Collect new pools of data to create competitive advantage (Chris maybe you can improve)

Chris, good move to get disqus rolling

Right now, the interface for using machine learning is to go talk to a scientist. BAAD interface.
Business people are not sure what technologies are applicable to their problem, or how machine learning can empower them. Users are confused by the interface to the artificial intelligence, which is either daunting or oversimplified.

This is likely to remain the case in the foreseeable future. Until major improvements in natural language processing occur that natural language communication is a viable interface, the real gains will be seen by companies that can creatively and naturally interface machine learning for everyday users.

For example, look at the WSJ story about My TiVo Thinks I'm Gay: (http://online.wsj.com/article_email/SB103826193…)
The problem is that the workings of the system are opaque.
This example underscores the importance of having an interface that communicates with the user, and attempts to convey its interpretation of the user's query.

The companies with the real competitive advantages are not those that can eke out 5% more accuracy in topic models, it's the companies that can figure out how to make topic model useful and simple for everyday users.