SmoothSpan Blog

For Executives, Entrepreneurs, and other Digerati who need to know about SaaS and Web 2.0.

If You Want to Favor Diversisty on the Internet, Increase Friction (+ Insights from Genetic Algorithms)

Posted by Bob Warfield on October 15, 2007

Tim O’Reilly writes about similarities between Facebook, Quant Funds, and the Techmeme Leaderboard.  He says there is a danger of self-fulfilling prophecies when feedback mechanisms emphasize herd behaviour, and that this can ultimately interfere with the wisdom of crowds, which requires a diverse set of independently acting individuals to achieve this effect.

This is exactly what I’ve been writing about in my series on punctuated equilibrium.  It is ironic that much of the promise held for the Internet has revolved around its ability to empower those who were previously unable to make a difference.  The dark side is that the Internet is one of the most ruthlessly efficient means of allocating almost all the wealth to a few yet discovered.  Why?  Because it eliminates friction and in so doing reinforces herdlike behaviour.

We’ve all seen the articles about various long tail phenomena:

The list goes on.  And it is normal in the course of human affairs for consolidation to happen around just a few areas.  That’s always been the case, the Internet is just radically more efficient at running the Darwinian processes that lead to the extinction of diversity. 

How does this happen?

Think of competition.  Initially, there are many different approaches to a market.  Some are better than others, and everyone in the market starts copying the better features and trying to be more like the better players.  A couple of things happen.  First, features start getting attributed to the better players no matter who has them, which only favors the better player more.  The ultimate example is if a company name gets turned into a verb, “I’ll Google her company and see what they do.”  Second, there are a set of flaws in the early competitors.  Sometimes these flaws are very hard to weed out.  Enter the Fast Follower.  These folks take the good and leave the bad.  Pretty soon, all the good features and almost none of the bad are concentrated in just a few of the players.  Others that try to copy that formula are pointless–unless they bring something new to the table, they are non-starters. 

This all fits perfectly well into the evolutionary model.  “Punctuated Equilibrium” is when the initial set of brand new features come to a market, perhaps even creating a new market.  That’s the Big Bang that brings that market into existence.  The rest is just competitive forces refining the original ideas until a few emerge. 

I’ve seen this firsthand playing with genetic algorithms on a couple of projects.  These algorithms mimic the evolutionary process in order to solve a problem of some sort.  They work well for problems that are too complex to be solved directly.  Genetic Algorithms start out with a randomly generated population that has “features”.  Let’s say we wanted to create on that evolves a stock trading system.  Each individual has a set of rules (“the features”) that determine the trading behaviour.  Perhaps there are 16 rules that vote to buy, sell, or hold.  We randomly generate a set of “traders” that have different random sets of rules.  We evaluate how well those organisms perform against historical stock market data and assign a score to each one. 

Next, we evolve the next generation.  To do so, we randomly generate some organisms.  It works best if there aren’t too many randoms.  Perhaps just 1/4 to 1/3.  Then we take the top 1/4 or 1/3 best scoring organisms from the last generation and copy them over unchanged.  Now we fill up the middle in a very clever way that is analagous to sexual reproduction.  We pick pairs of organisms randomly, but skewed by score.  Organisms that scored higher are more likely to be chosen–they’re the beautiful people.  The ugly ducks may still be chosen, but much less likely.  From the pair, we randomly pick rules until we have another 16 rules to make a new organism.  This is the comingling of chromosomes.  Research shows this is the most important part of the process.

Can you see why this is so much like what happens on the Internet as sites compete in a market for “fitness”.  Maybe fitness is as simple as a slot on a leaderboard.  You can see the feature comingling at work, although it tends not to be as random because we can measure and sense why some features are better.  We keep the original market leaders in play until the new guys do better.  What I can tell you is that if you run such genetic algorithm experiments, they lead to the same long tails.  Just a few organisms own the shop and it happens amazingly fast.

What does it take to help diversity succeed?  In the real world, it has taken dinosaur-killing events that were very violent.  But there are some less violent alternatives to learn from.  Islands have been repositories for independent evolution that has bred diversity.  Witness the unique animals on each continent.  You can do the same with genetic algorithms by creating multiple islands that don’t inter-breed, and it works well as a way to keep diversity up over longer periods.

This happens on the Internet too.  The “islands” on the net are those areas outside leaderboards or other measures of fitness and scrutiny.  There are interesting memes and products brewing out there.  Their question is whether they have to move back into the mainstream to thrive.  Look in these areas of increased friction for diversity.  If there is a bubble brewing as some say, the survivors will be the ones at the top of the long tail graphs and the ones off the edge of the map on islands in the Internet.

VC’s and entrepreneurs focused on “new model” for Internet startups take heed: you’re herding together in a way that will make you vulnerable to these effects.

Related Articles:

Widget makers are unhappy, while Performers, such as bloggers, prosper.  Why?  Because Performers are well equipped to create punctuated equilibrium over and over again, while Widgets are flashes in the pan that rapidly consolidate at the top.

7 Responses to “If You Want to Favor Diversisty on the Internet, Increase Friction (+ Insights from Genetic Algorithms)”

  1. […] strategy, the utility of that strategy quickly declines towards zero. As Bob Warfield puts it at Smoothspan, the Web adds to this problem because it “eliminates friction and encourages herdlike […]

  2. […] it works sounds right.  Scoble describes something very similar to a genetic algorithm, which I’ve talked about in this context before.  There is a set of sites that are the “fabric” of TechMeme.  […]

  3. […] strategy, the utility of that strategy quickly declines towards zero. As Bob Warfield puts it at Smoothspan, the Web adds to this problem because it “eliminates friction and encourages herdlike […]

  4. […] I’ve been saying this for a long time.  If you’re seeking out what the likes of Digg, Reddit, and Techmeme have identified as the “hot” items, you’re just making yourself more average.  You’re opting into group think, and I don’t mean the wisdom of crowds.  One of the key requirements for crowds to be wise is that they be independent.  A whole bunch of people on a service like reddit or Techmeme are just oscillating around the same meme feedback.  Very little new is being created there. […]

  5. […] If you want diversity then you need to increase friction.  LeMeur’s original issue that there were 7000 Tweets from LeWeb and he didn’t want to read all of them is a manifestation of friction.  If you’re done talking about anything new and interesting, and just want to spread the word of a few, then knock out all the friction.  The Echo Chamber will be alive and well.  I wish there was a knob for all of my different Internet feeds that let me smoothly vary the friction to go from looking at the Echo Chamber to looking at the most diverse fringe elements of the Internet.   I’d like the Anti-Techmeme that finds for me those things off the edge of the radar but within my sphere of interest.  Shiny new things I haven’t seen before.  People I haven’t heard from.  I’m guessing Scoble is on the same kind of mission, given the potent news network he has built up. […]

  6. […] Content memes proceed along these typically evolutionary cycles.  As conventional wisdom settles in, the original meme is widely imitated and only mildly improved, if at all.  This goes on for quite some while.  The dinosaurs lasted over 160 million years and it took a catastrophic event to shake their hold enough that something new could take their place.  Those events create the punctuated equilibrium where evolution suddenly moves ahead a quantum leap.  So it is with punctuated equilibriums and the social evolution of memes of content on the Internet. […]

  7. […] Here’s the problem for everyone that isn’t Scoble:  if you really want to get the good stuff, it isn’t on Techmeme.  It isn’t in the most widely read blogs, whether you prefer Techcrunch, RWWeb, or some other.  And it definitely isn’t in the latest walled garden Social Network.  Those places are emphasizing the short head, not the long tail.  They are the Echo Chambers.  If you want the good stuff, you have to hike up the river, until it becomes a stream, and then still further to the very sources high in the mountains.  That’s where the cool clean glacial waters of the Long Tail originate.  That’s where the good stuff is.  You’ll never see most of it if you spend all your time in the Echo Chamber, and it’s a real shame because the promise of the Internet is that it unlocks the Long Tail.  Get off the Bell Curve and move not just one, but several stand deviations away from the mean.  That’s where the real juice is.  Sure it’s hard work.  But if you want to maximize your information diversity, there’s going to be a little friction. […]

%d bloggers like this: