In the last decade there have been major advances in storing, analyzing, and acting upon extremely large data sets. Data sets that were previously left dormant are now being put to (mostly) constructive use. But the vast majority of information in the world isn’t available for analysis because it isn’t being electronically collected.
This is changing rapidly as new data collection mechanisms are implemented – what engineers refer to as instrumentation. Common examples of instrumentation include thermometers, public safety cameras, and heart rate monitors.
Smart phones are one obvious new source of potential instrumentation. A person’s location, activities, audio and visual environment – and probably many more things that haven’t been thought of yet – can now be monitored. This of course raises privacy issues. Hopefully these privacy issues will be solved by requiring explicit user opt-in. If so, this will require creating incentives for people to do so.
Foursquare instruments location in an opt-in way through the check in. The incentives are social and game-like, but the data produced could be useful for many more “serious” purposes. Fitbit instruments a person’s health-related activity. The immediate incentive is to measure and improve your own health, but the aggregate data could be analyzed by medical researchers to benefit others.
In manufacturing, there has been a lot of interesting innovation around monitoring machinery, for example by using loosely joined, inexpensive mesh networks. In homes, protocols like ZigBee allow devices to communicate which allows, for example, automation of tedious tasks and improved energy efficiency.
In the next decade, there will be a massive amount of innovation and opportunity around the big data stack. Instrumentation will be the foundational layer of that stack.