Every system built by a single institution has points of failure that can bring the entire system down. Even in organizations that have tried hard for internal redundancy – for example, Google and Amazon have extremely distributed infrastructures – there will always be system-wide shared components, architectures, or assumptions that are flawed. The only way to guarantee there aren’t is to set up completely separate, competing organizations – in other words, new institutions.
This insight has practical implications when building internet services. One thing I learned from my Hunch co-founder Tom Pinckney is, if you really care about having a reliable website, always host your servers at two data centers, owned by different companies, on networks owned by different companies, on separate power grids, and so forth. Our last company, SiteAdvisor, handled billions of requests per hour but never went down when the institutions we depended on went down – which was surprisingly often. (We did have downtime, but it was due to our own flawed components, assumptions etc.).
The importance of institutional redundancy is profoundly more important when applied to the internet at large. The US government originally designed the internet to be fully decentralized so as to withstand large-scale nuclear attack. The core services built on top of the internet – the web (HTTP), email (SMTP), subscription messaging (RSS) – were made similarly open and therefore distributible across institutions. This explains their remarkable system-wide reliability. It also explains why we should be worried about reliability when core internet services are owned by a single company.
The principle of not depending on single institutions applies beyond technology. Every institution is opaque to outsiders, with single points of failure, human and otherwise. For example, one of the primary lessons of the recent financial crisis is that the most important form of diversification is across institutions, not, as the experts have told us for decades, across asset classes. The Madoff fraud was one extreme, but there were plenty of cases of lesser fraud and countless cases of poor financial management, most of which would have been almost impossible to anticipate by outsiders.
Related posts:
View Comments ↓
I think this applies to us more today as most of us are moving our applications/data to the cloud. Somebody stumbling over a power cord somewhere could just wipe out your precious information.
It applies to twitter also. If they had made the twitter protocol something similar to SMTP or like google wave and opened it up for others to host it as well, it would have saved a lot of failwhales and money to them.
I wonder if it could be applied to our governments as well
Great point about natural clumping caused by financial gravity. Big and great companies get more resources and can easily provide better services for a wide variety of areas. But this centralization efficacy comes at a cost of durability.
As a society we are challenged to spur on competition, innovation, for a variety of competing services so that overall, our services strengthen in reliability. Services you can trust builds long term user retention.
Have you seen businesses like Google, Apple, and MSFT become too big as to open up potential vulnerabilities? What in particular do you suggest to remedy the situation. Should large businesses have some financial motovation to spin off fully independent subsidiaries?
[...] This post was mentioned on Twitter by chris dixon, hrjn_rss. hrjn_rss said: The importance of institutional redundancy: Comments http://url4.eu/nLgY [...]
Sure, loosely coupled systems (like the web itself) are more fault-tolerant than highly-concentrated ones.
But here's a thought: for redundancy to be really efficient the key challenge is synchronization. This goes for the Cloud, SaaS applications, caches on low-bandwidth devices (mobile phones), etc, etc.
IMO, effective synchronization solutions will represent the backbone of the web of the future.
I have been heading business operations (for those who don't get what ops people do – we do what Roy Disney did for Walt Disney – Wikipedia it) for early and larger late stage startups my entire career. The #1 thing I do every time I join an organization is to put in plan B for everything. You have a T1, get wireless T as a backup. Using Amazon cloud, better have VM ready to go on RackSpace cloud. Genius graphic artist doing projects for you, get another one pronto. Have a real awesome coder on staff, better start looking for replacement. I never rely on a single supplier, vendor, etc.
You simply can't, because things will always go wrong and you have to have the plan B. Unfortunately many execs don't like to think about the doomsday and end up getting caught with their pants down.
Wise words. Ultimately, it's always about cost vs risk. The more you spend/invest, the less you risk. Redundancy, whether we are talking about an extra server in a separate location or an institution, has a cost. If you can afford it, de-risk it.
def agree it applies to twitter, see http://cdixon.org/?p=1296
agree about synchronization. already see this as a huge challenge e.g. with db replication
i'm not one to suggest the govt break up non-monopolistic companies, but i do think a world of smaller companies is probably a good thing.
..and not only. Think about browser based apps – you want to work offline and then synchronize with the cloud. This can get incredibly tricky when multiple users have shared write permission on the same assets.
I see this as a key tech/investment space going forward.
Heck ya, synchronization of asynchronous information is a BIG hurdle.
Developers have had to deal with this for a while already: merging simultaneously modified files. It requires some communication and intelligent collaborative effort beyond some fantastic tools (github, svn, dropbox).
In a similar vein, if you see some opportunity why not start redundant companies to pursue it? That way, you have a stake in both yourself and the competition.
Definitely agree. Would be interesting to hear how the same principle applies to personal life.
[...] From cdixon.org [...]
This is exactly why I am currently working on a clone of myself
[...] This post was mentioned on Twitter by Ivan Kirigin, chris dixon. chris dixon said: Rackspace down… Institutional redundancy. Institutional redundancy. Institutional redundancy. http://bit.ly/5z1nim [...]
This is exactly why I am currently working on a clone of myself