The idea maze for AI startups


An “idea maze” is a map of all the key decisions and tradeoffs that startups in a given space need to make:

A good founder is capable of anticipating which turns lead to treasure and which lead to certain death. A bad founder is just running to the entrance of (say) the “movies/music/filesharing/P2P” maze or the “photosharing” maze without any sense for the history of the industry, the players in the maze, the casualties of the past, and the technologies that are likely to move walls and change assumptions.

- Balaji Srinivasan, “Market Research, Wireframing and Design

I thought it would be interesting to show an example of an idea maze for an area that I’m interested in: AI startups. Here’s a sketch of the maze. I explain each step in detail below.

screen shot 2015 02 01 at 11 51 53 am

“MVP with 80–90% accuracy.” The old saying in the machine learning community is that “machine learning is really good at partially solving just about any problem.” For most problems, it’s relatively easy to build a model that is accurate 80–90% of the time. After that, the returns on time, money, brainpower, data etc. rapidly diminish. As a rule of thumb, you’ll spend a few months getting to 80% and something between a few years and eternity getting the last 20%. (Incidentally, this is why when you see partial demos like Watson and self-driving cars, the demo itself doesn’t tell you much — what you need to see is how they handle the 10–20% of “edge cases” — the dog jumping out in front of the car in unusual lighting conditions, etc).

At this point in the maze you have a choice. You can either 1) try to get the accuracy up to near 100%, or 2) build a product that is useful even though it is only partially accurate. You do this by building what I like to call a “fault tolerant UX.”

“Create a fault tolerant UX.” Good examples of fault-tolerant UXs are iOS autocorrect and Google search’s “did you mean X?” feature. You could also argue Google search itself is a fault tolerant UX: showing 10 links instead of going straight to the top result lets the human override the machine when the machine gets the ranking wrong. Building a fault tolerant UX isn’t capitulation, but it does mean a very different set of product requirements. (In particular, latency is very important when you want the human and machine to work together—this generally affects your technical architecture).

Ok so let’s suppose you decide to go for 100% accuracy. How do you get there? You won’t get the 10–20% through algorithms. You’ll only get there with lots more data for training your models. Data is the key to AI because 1) it’s the missing ingredient — we have great algorithms and virtually endless computational resources now, and 2) it’s the proprietary ingredient—algorithms are mostly a shared resource created by the research community. Public data sets, on the other hands, are generally not very good. The good data sets either don’t exist or are privately owned.

“Narrow the domain.” The amount of data you need is relative to the breadth of the problem you are trying to solve. So before you start collecting data you might want to narrow your domain. Instead of trying to build a virtual bot that can do anything (which would basically mean passing the Turing Test—good luck with that), build a bot that can just help someone with scheduling meetings. Instead of building a cloud service that predicts anything, build one that can predict when a transaction is fraudulent. Etc.

“Narrow domain even more.” After you are done narrowing the domain, try narrowing it even more! Even if your goal is to build X, sometimes building an MVP that is part of X is the best way to eventually get to X. My advice would be to keep narrowing your domain until you can’t narrow it anymore without making the product so narrow that no one wants to use it. You can always expand the scope later.

“How do you get the data?” Broadly speaking, there are two ways: build it yourself or crowdsource it. A good analogy here is Google Maps vs Waze. Google employs thousands of people driving around to map out roads, buildings, and traffic. Waze figured out how to get millions of people to do that for them. To do what Google does, you need far more capital (hundreds of millions, if not billions of dollars) than is generally available to pre-launch startups.

Startups are left with two choices to get the data. 1) Try to mine it from publicly available sources. 2) Try to crowdsource it.

The most common example of 1) is crawling the web, or big websites like Wikipedia. You could argue this is what the original Google search did by using links as ranking signals. Many startups have tried mining Wikipedia, an approach that hasn’t led to much success, as far as I know.

The most viable approach for startups is crowdsourcing the data. This boils down to designing a service that provides the right incentives for users to give data back to the system to make it better. Building a crowdsourced product is its own topic (which is why that part of the idea maze points to another, nested idea maze), but I’ll give an example of one approach to doing this, which was tried by company called that we invested in last year. Wit’s idea was to provide a service for developers for doing speech-to-text and natural language processing. The v1.0 system gave the right answer most but not all of the time. But it also provided a dashboard and API where developers could correct errors to improve their results. For developers using the free version of the service, the training they performed would get fed back to make the overall system smarter. Facebook acquired Wit so their future will unfold now as part of a larger company. The approach they took was very clever and could apply to many other AI domains.

This is a rough sketch of how I see the AI startup idea maze. A few caveats: 1) I could very well be mistaken or have overlooked other paths through the maze — idea mazes are meant to aid discussion, not serve as gospel, and 2) As Balaji says, new technological developments can “move walls and change assumptions.” Look out especially for new infrastructure technologies (internet, smartphones, cloud computing, bitcoin, etc) that can unlock new pathways in many different idea mazes, even ones that at first seem unrelated.

Next post: "It all blossomed out of this tiny little seed"
Previous post: Come for the tool, stay for the network

Views expressed in “content” (including posts, podcasts, videos) linked on this website or posted in social media and other platforms (collectively, “content distribution outlets”) are my own and are not the views of AH Capital Management, L.L.C. (“a16z”) or its respective affiliates. AH Capital Management is an investment adviser registered with the Securities and Exchange Commission. Registration as an investment adviser does not imply any special skill or training. The posts are not directed to any investors or potential investors, and do not constitute an offer to sell -- or a solicitation of an offer to buy -- any securities, and may not be used or relied upon in evaluating the merits of any investment.

The content should not be construed as or relied upon in any manner as investment, legal, tax, or other advice. You should consult your own advisers as to legal, business, tax, and other related matters concerning any investment. Any projections, estimates, forecasts, targets, prospects and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Any charts provided here are for informational purposes only, and should not be relied upon when making any investment decision. Certain information contained in here has been obtained from third-party sources. While taken from sources believed to be reliable, I have not independently verified such information and makes no representations about the enduring accuracy of the information or its appropriateness for a given situation. The content speaks only as of the date indicated.

Under no circumstances should any posts or other information provided on this website -- or on associated content distribution outlets -- be construed as an offer soliciting the purchase or sale of any security or interest in any pooled investment vehicle sponsored, discussed, or mentioned by a16z personnel. Nor should it be construed as an offer to provide investment advisory services; an offer to invest in an a16z-managed pooled investment vehicle will be made separately and only by means of the confidential offering documents of the specific pooled investment vehicles -- which should be read in their entirety, and only to those who, among other requirements, meet certain qualifications under federal securities laws. Such investors, defined as accredited investors and qualified purchasers, are generally deemed capable of evaluating the merits and risks of prospective investments and financial matters. There can be no assurances that a16z’s investment objectives will be achieved or investment strategies will be successful. Any investment in a vehicle managed by a16z involves a high degree of risk including the risk that the entire amount invested is lost. Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by a16z is available at Excluded from this list are investments for which the issuer has not provided permission for a16z to disclose publicly as well as unannounced investments in publicly traded digital assets. Past results of Andreessen Horowitz’s investments, pooled investment vehicles, or investment strategies are not necessarily indicative of future results. Please see for additional important information.