Cryptolawyer, decentralizing the things.

peter (at) coincenter (dot) org

We Are All Hayekians Now: The Internet Generation and Knowledge Problems

Economist F.A. Hayek

Economist F.A. Hayek

The much-appropriated phrase “we are all Xs now,” apparently got its start in the late 19th century when William Vernon Harcourt somewhat prematurely declared “We’re all socialists now.” Since then, we’ve all been Bolsheviks, according to Shaw, Keynesians, according to either Nixon or Friedman, Thatcherites, according to Peter Mandelson, and Austrians according to Ron Paul. The dismal history of this particularly draconian rhetorical flourish–effectively meaning: regardless of your ideology you and everyone else actually agree with me in practice–should probably dissuade me from piling on the “we’re all” bandwagon, but being as we’re all doing it now, here goes.

We are all Hayekians now. Specifically, the “we all” is not quite everyone. The “all” to which I’m referring is people of the internet–people who’ve grown up with the net and use it for a majority of their day-to-day activities. And, the “Hayekian” to which I’m referring is not his theories on capital, or the rule of law, but, specifically his vision of knowledge.

Hayek’s Two Problems

Primarily in his The Use of Knowledge in Society but also in his other contributions to the socialist calculation debate, Hayek crafted a brilliant statement of a perennial problem. In the world of human endeavor, we have two types of problems: economic and technological.

Technological problems involve effectively allocating given resources to accomplish a single valuable goal. Deciding how to build a bridge once you’ve a bundle of bridge-making resources is a quintessential technical problem. It’s like a question in an engineering exam: “You have x tonnes of steel, arrange them in a way that spans the river, lets heavy vehicles travel over, and resists wind or other natural forces.”

Deciding whether to build a bridge in the first place and where to build said bridge is a quintessential economic problem. The choice to build the bridge necessarily means that the resources that were headed to some other project (say a skyscraper) will now go to the bridge instead. The choice to build the bridge is a choice between this bridge or that skyscraper as well as any other alternative use of those resources. Each alternative use would have different benefits (and unseen costs). This is not a mere question of engineering the strongest or even the most cost-effective structure to get across the Hudson, this is a question of what is the strongest or most cost-effective possible future version of New York City. And it’s a question that no one person can ever hope to answer. Just imagine finding that question on, what would have to be, an exam from hell.

Hayek Gets Googled.

Hayek as Google's logo compliments of the excellent Google logo generator here.

Hayek as Google’s logo, compliments of the excellent Google logo generator here.

Flash forward from Hayek and bridges to Google and search. From a recent Quora question on most absurd but ultimately successful start-up ideas comes this bleak portrait of Google’s prospects at its inception: “We are building the world’s 20th search engine at a time when most of the others have been abandoned as being commoditized money losers. We’ll strip out all of the ad-supported news and portal features so you won’t be distracted from using the free search stuff.”

But, of course, Google survived, prospered, and continues towards its apparent goal of eating the entire internet (while also making cars drive themselves, putting cameras on everyone’s heads, and generally making Steve Ballmer very very angry). So, why did Google win? The answer is, perhaps surprisingly, in Hayek’s theory.

To see this, let’s carefully describe what Google set out to do. From their blog: “Our goal always has been to index all the world’s data.” Talk about anemic goals, come on Google, show some ambition! So, is this one of Hayek’s technical problems or is this an economic one? Our gut might first tell us that it is technical. Unlike the scarce resources needed to build a bridge or a skyscraper or a submarine, the world’s data is all in one place (roughly), the internet, and it’s not rivalrous. If I use the data to make my search engine it doesn’t limit Steve Ballmer’s ability to use the same data. The Internet comes closest as anything yet in history to being what Hayek calls, “given” data.

“The ‘data’ from which the economic calculus starts are never for the whole society ‘given’ to a single mind which could work out the implications.” Hayek’s Use of Knowledge in Society


But Hayek is still correct, even with Google and the internet. Sure, all this data is now hanging out in one place for free, but to make a useful index you need to determine how much people value different data. We need data about the data. So, Google’s position is not unlike the position of the Soviet Politburo in the middle of the 20th century.

In Soviet Russia, “We have all the country’s resources! How should we arrange them?”

In Mountainview: “We have all the world’s digitalized data! How should we arrange them?”

In Soviet Russia, failed attempts at arranging resources destroyed the information about the resources. The free market is the best way to figure out how individual people value individual resources. When left to trade voluntarily, people reveal their preferences with their willingness to pay. By arranging resources through coercion you’ve blinded yourself to the emergent value of the resources because you’ve forbidden voluntary arrangement in the economy.

This is different on the internet. The data resources are not rivalrous, so a company like Altavista can gather them all up and start arranging them through central planning (or any mechanism they want) without disturbing the use-data about those resources. But, this doesn’t mean that you will be able to figure out through sheer intellectual sorting ability what order is best. Central planning still fails, even when resources are non-rivalrous, if you ignore revealed preferences. But, unlike the Soviet situation, you don’t destroy the internet; you just produce a search product of absolutely no value.


“Schwarm” Flock behavior as photographed by Axel Hartmann ; I like having photos in blog posts. Visual metaphors can be very handy at communicating complicated concepts. Spontaneous order, however, is one of those concepts that is tough to show in pictures. There’s a worthy, as-of-yet un-assumed project in being the graphic designer for these ideas.

The Bad Old Days of Search.

Search used to be really bad. Why? Because search companies were using either (a) content-producer willingness to pay for indexing, (b) mere keyword search or (c) some combination of editorial centralized decision-making to organize lists of sites. These methods only work if you think that the best site about ducks is either (a) the site that has the most money to pay Altavista for prime “duck” listing, (b) the site that has the most “ducks” in its text, or (c) the site that was most appealing to your employees tasked with finding duck sites.

Google’s brilliance came in the form of PageRank™, an algorithm that relied heavily on the truest decentralized revealed preferences the internet has to offer, links. If 999 other websites linked to one website about ducks, you can bet that most people think that this site is better at explaining ducks than a site with only one link to it (even if that link was horse-sized). As Google says (in #4 of the 10 things they know to be true):

“Democracy on the web works.Google search works because it relies on the millions of individuals posting links on websites to help determine which other sites offer content of value. We assess the importance of every web page using more than 200 signals and a variety of techniques, including our patented PageRank™ algorithm, which analyzes which sites have been “voted” to be the best sources of information by other pages across the web. As the web gets bigger, this approach actually improves, as each new site is another point of information and another vote to be counted.”

Talkin’ ’bout My Generation

So Google uses the decentralized Hayekian knowledge of the masses to function. Why does this mean we’re all Hayekians? I could just say, check your Google search history and see how often you, yourself, have relied on that Hayekian knowledge to explore your world (I’m up to about 50 today and 40,000 lifetime). But that’s a little too neat. We still do plenty of things without Google.

The point I want to make is broader. All of the questions of organizing activity on the internet are solved (when they are, in fact, solved successfully) using Hayekian decentralized knowledge. Yelp, Google+, Foursquare and Seamless are how we now find great restaurants. These sites put the curated, editorializing Zagats-of-the-world out of business (or bought Zagats and made it crowd-sourced). Amazon customer reviews are how we find good products. Ebay feedback is how we find good individual sellers. And, moreover, whole brick and mortar services are moving to a crowd-sourced model, with sites like AirBnB for lodging and RelayRides for car rental.

The fascinating thing to notice here is that the knowledge critique of central planning in governments works on firms. The giant firms of tomorrow won’t be those that carefully curate internal knowledge about their industry hoping to break through to a novel and innovative invention; the giant firms of tomorrow will be those that empower people to freely share their knowledge and resources in a vibrant marketplace.

Today, the central challenge for a firm is not to develop careful internal management but rather the non-trivial task of building marketplaces and forums to encourage decentralized knowledge production and cooperation.

Our generation already understands this on a gut level. We Google everything.  We defend freedom on the internet as if it was our own personal real-world liberty at stake. We mock the antiquated central planners of the early web, looking at you AOL, Prodigy, for their ineffectual obviousness and denial of crowd-sourced knowledge. We spend our precious but unpaid time writing blogs, or wiki articles to amplify our voice in the chorus. We even aspire to solve the complicated problems once reserved for the state like criminal investigation or patent-screening using our communal abilities. We all know where the best economic knowledge lies, in the many and never the few.

We are all Hayekians now.