Collective knowledge systems

I think you could make a strong argument that the most important technologies developed over the last decade are a set of systems that are sometimes called “collective knowledge systems”.

The most successful collective knowledge system is the combination of Google plus the web. Of course Google was originally intended to be just a search engine, and the web just a collection of interlinked documents. But together they provide a very efficient system for surfacing the smartest thoughts on almost any topic from almost any person.

The second most successful collective knowledge system is Wikipedia. Back in 2001, most people thought Wikipedia was a wacky project that would at best end up being a quirky “toy” encyclopedia. Instead it has become a remarkably comprehensive and accurate resource that most internet users access every day.

Other well-known and mostly successful collective knowledge systems include “answer” sites like Yahoo Answers, review sites like Yelp, and link sharing sites like Delicious.  My own company Hunch is a collective knowledge system for recommendations, building on ideas originally developed by “collaborative filtering” pioneer Firefly and the recommendation systems built into Amazon and Netflix.

Dealing with information overload

It has been widely noted that the amount of information in the world and in digital form has been growing exponentially. One way to make sense of all this information is to try to structure it after it is created. This method has proven to be, at best, partially effective (for a state-of-the-art attempt at doing simple information classification, try Google Squared).

It turns out that imposing even minimal structure on information, especially as it is being created, goes a long way. This is what successful collective knowledge systems do. Google would be vastly less effective if the web didn’t have tags and links. Wikipedia is highly structured, with an extensive organizational hierarchy and set of rules and norms. Yahoo Answers has a reputation and voting system that allows good answers to bubble up. Flickr and Delicious encourage user to explicitly tag items instead of trying to infer tags later via image recognition and text classification.

Importance of collective knowledge systems

There are very practical, pressing needs for better collective knowledge systems. For example, noted security researcher Bruce Schneier argues that the United States’ biggest anti-terrorism intelligence challenge is to build a collective knowledge system across disconnected agencies:

What we need is an intelligence community that shares ideas and hunches and facts on their versions of Facebook, Twitter and wikis. We need the bottom-up organization that has made the Internet the greatest collection of human knowledge and ideas ever assembled.

The same could be said of every organization, large and small, formal and and informal, that wants to get maximum value from the knowledge of its members.

Collective knowledge systems also have pure academic value. When Artificial Intelligence was first being seriously developed in the 1950′s, experts optimistically predicted they’d create machines that were as intelligent as humans in the near future.  In 1965, AI expert Herbert Simon predicted that “machines will be capable, within twenty years, of doing any work a man can do.”

While AI has had notable victories (e.g. chess), and produced an excellent set of tools that laid the groundwork for things like web search, it is nowhere close to achieving its goal of matching – let alone surpassing – human intelligence. If machines will ever be smart (and eventually try to destroy humanity?), collective knowledge systems are the best bet.

Design principles

Should the US government just try putting up a wiki or micro-messaging service and see what happens? How should such a system be structured? Should users be assigned reputations and tagged by expertise? What is the unit of a “contribution”? How much structure should those contributions be required to have? Should there be incentives to contribute? How can the system be structured to “learn” most efficiently? How do you balance requiring up front structure with ease of use?

These are the kind of questions you might think are being researched by academic computer scientists. Unfortunately, academic computer scientists still seem to model their field after the “hard sciences” instead of what they should modeling it after — social sciences like economics or sociology. As a result, computer scientists spend a lot of time dreaming up new programming languages, operating system architectures, and encryption schemes that, for the most part, sadly, nobody will every use.

Meanwhile the really important questions related to information and computer science are mostly being ignored (there are notable exceptions, such as MIT’s Center for Collective Intelligence). Instead most of the work is being done informally and unsystematically by startups, research groups at large companies like Google, and a small group of multi-disciplinary academics like Clay Shirky and Duncan Watts.

12 thoughts on “Collective knowledge systems

  1. The rising acceptance of Nintendo Wii is storming the gaming environment
    with just about every minute that goes by. A lot of of these web sites are totally free so users must not be pushed into providing dollars to be
    a part of groups or betting cash during the recreation. A
    lot of sport web-sites are recognised, each individual working day, highlighting the games that are
    reviewed each and every month.

  2. In order to cope in a very fast-paced market, new applications are needed.
    You can run your online business perfectly in the
    event you become cordial towards the customers. ll hold the practical guidance
    you may need on how to find a concierge business”.

  3. Hello there! This is kind of off topic but I need some guidance from an established blog.
    Is it tough to set up your own blog? I’m not very techincal but I
    can figure things out pretty fast. I’m thinking about
    setting up my own but I’m not sure where to start. Do
    you have any tips or suggestions? Many thanks

  4. Your website must give you a clear message with a goal-oriented direction, setting you apart from
    your competition. She invites one to visit her site
    where she’ll share a proven approach to start an business
    online. it also can be used for headhunting and employment.

  5. And inside the course for these analysis, it emerges that many dollar spent inside the pursuit of your online degree is likely to increase one’s lifetime
    earnings more than once fold. You can run your home business perfectly in case you become cordial for the customers.
    If you’re still with a loss, you are able to contact the buyer care team either by email, live chat, or phone during standard west coast business hours.

  6. Undeniably believe that which you said. Your favorite justificaton seemed to
    be on tthe net the simplest thing to be aware of. I say to you, I
    certainly get annoyed while people consider worries that they just do
    nott know about. You managed to hiit the nail upon the top and also defined out the
    whole thing without having side effect , people could take a signal.
    Will likely be back to get more. Thanks

  7. Bulks in the advertisers are primarily private homeowners, letting
    managers and property agents. So unless service repair shop completes the formalities and find the necessary certifications they
    could’t start their business. ” The company sees innovation as considered one of its core organizational competencies.

  8. Using ingenious technology and innovation the Romans made an Empire that withstood the test of energy.
    The facts are, people join MLM opportunities as a consequence of who introduced them.
    I have observed my share of scams, and have in reality done a great job avoiding being taken to get
    a sucker and I’m here to share with you, Ameriplan is just not a scam.

  9. The dedicated team of AVG professionals is available here
    all round time, whom you’ll be able to reach by calling on the AVG tech support number.
    Investors are there to help you the entrepreneur and
    to realize a make money from their investment through the entrepreneurs company.
    A well-developed business plan is like your blueprint for victory.

  10. In order to cope inside a fast-paced market, new applications are
    needed. Businesses are to heavily dependant upon customers
    for his or her survival, without customers an enterprise
    would cease to exist. ” Employees at Nike are encouraged to get curious and open to new ideas, whatever their source.

  11. In a similar sense, businesses would be wise to provide their workers something to
    unite under. You can run your online business perfectly
    if you become cordial towards the customers.
    Donnie Jonston will be the author of this article about the best way to make cash on Ebay Donnie
    has many years of work experience as being a writer plus working
    with drop shippers in a very variety of entrepreneurial ventures.

  12. Furthermore, my experience also may include identifying the competitive advantage for a few
    different companies and crafting that advantage into a web marketing strategy to develop start up business.
    So unless any organization completes the formalities and find the necessary certifications they could’t start
    their business. it can also be useful for headhunting and employment.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s