Software monocultures, imperialism, and weapons of math destruction

This past Friday, Facebook reported that they suffered a security breach that affected at least 50 million users. ‘Security breach’ is a bit of newspeak that is meant to hint at active malice and attribute fault outside the company. But as far as I understand it — and I am no expert on this — it was just a series of three bugs in Facebook’s “View As” feature that together allowed people to get the access tokens of whoever they searched for. This is, of course, bad for your Facebook account. The part of this story that really fascinated me, however, is how this affected other sites. Because that access token would let somebody access not only your Facebook account but also any other website where you use Facebook’s Single Sign On feature.

This means that a bug that some engineers missed at Facebook compromised the security of users on completely unrelated sites like, say, StackExchange (SE) or Disqus — or any site that you can log into using your Facebook account.

A case of software monoculture — a nice metaphor I was introduced to by Jonathan Zittrain.

This could easily have knock-on effects for security. For example, I am one of the moderators for the Theoretical Computer Science SE and also the Psychology and Neuroscience SE. Due to this, I have the potential to access certain non-public information of SE users like their IP addresses and hidden contact details. I can also send communications that look much more official, along-side expected abilities like bans, suspensions, etc. Obviously, part of my responsibility as a moderator is to only use these abilities for proper reasons. But if I had used Facebook — disclosure: I don’t use Facebook — for my SE login then a potential hacker could get access to these abilities and then attempt phishing or other attacks even on SE users that don’t use Facebook.

In other words, the people in charge of security at SE have to worry not only about their own code but also Facebook (and Google, Yahoo!, and other OpenIDs).

Of course, Facebook is not necessarily the worst case of software monoculture or knock-on effects that security experts have to worry about. Exploits in operating systems, browsers, serves, and standard software packages (especially security ones) can be even more devastating to the software ecology.

And exploits of aspects of social media other that login can have more subtle effects than security.

The underlying issue is a lack of diversity in tools and platforms. A case of having all our eggs in one basket. Of minimizing individual risk — by using the best available or most convenient system — at the cost of increasing systemic risk — because everyone else uses the same system.

We see the same issues in human projects outside of software. Compare this to the explanations of the 2008 financial crises that focused on individual vs systemic risk.

But my favourite example is the banana.

In this post, I’ll to sketch the analogy between software monoculture and agricultural monoculture. In particular, I want to focus on a common element between the two domains: the scale of imperial corporations. It is this scale that turns mathematical models into weapons of math destructions. Finally, I’ll close with some questions on if this analogy can be turned into tool transfer: can ecology and evolution help us understand and manage software monoculture?

Today if you go to your local store and pick up some bananas, they will most likely be Cavendish bananas. These bananas were first mass produced in 1903, but were extremely unpopular until the 1960s. During this time, the dominant banana was the Gros Michel. The Cavendish was overall an inferior fruit: it bruised more easily, was harder to ship, and was much less creamy and delicious than the Gros Michel. Thus, the various realms of the European colonial empires primarily grew the Gros Michel. In fact, given the imperial urge to standardize, they grew almost identical strains of Gros Michel. As you might notice when you eat a banana, there are no seeds in this fruit; it’s flower also produces no viable pollen. Commercial bananas are reproduced asexually, making whole plantations into monocultures of genetically (nearly) identical bananas. This was good for consumers and easy for imperial corportations, but also good for the fungus Fusarium oxysporum.

When this fungus colonizes the roots of a banana plant, it causes Panama disease. This disease will kill the plant and thus stop the production of bananas. It is resistant to fungicides. And it usually appear asymptomatically in the shoots of banana plants. Since these shoots are used by farmers to reproduce the banana, they end up planting diseased plants. Finally, the fungus can persist asymptomatically in certain common weeds and the soil itself, so even if the whole field is replanted from a healthy individual, they will likely still get sick.

By the 1960, Panama diseases wiped out almost all production of the Gros Michel. It was one of the most destructive plant diseases of modern times. This systemic failure led us to the inferior Cavendish banana, which seemed to not be affected by Fusarium oxysporum. Unfortunately, the Cavendish is now starting to succumb to Panama disease as well. Let’s hope that the agriculture industry has learned something in the last 70 years.

Hopefully the analogy I’m making with software is clear. The banana is our favourite piece of software and the fungus is an unexpected bug or exploit. But the most important part is not an analogy, but the same: imperial corporations.

The monoculture of bananas was not an inherent, biological property of the banana itself. The rampant monoculture was an effect of the socio-biological system of imperial agriculture. The biological details of asexual reproduction of the banana plant and the aggressive, treatment-resistant fungus were important. But they only made half the story. The other half was the social and economic systems in place that reproduced the Gros Michel banana plant and spread it all over the world.

We have a similar thing to worry about with software. Especially the social algorithms that underlay weapons of math destruction.

For me, a social algorithm is a mathematical model, usually embedded in computer code, that often takes the form of a person scoring system. The scores are used to determine access to or denial of certain resources to that person. Usually, these algorithms are not (completely) hard-coded but trained on population (and individual) level data. Examples can be anything from recommendation engines, social network feeds, to credit scores and predictive policing.

There isn’t anything inherently wrong with a social algorithm (at least not with all social algorithms). Just like there isn’t anything inherently wrong with an asexually reproducing banana plant or a banana-killing fungus. The issue comes when this technology is combined with an imperial social organization. In that case, a social algorithm can become a weapon of math destruction. For Cathy O’Neil, the three main characteristics of a weapon of math destruction are: (1) scale, (2) opacity, and (3) vicious self-reinforcement. And here, I want to focus on scale.

One of the main mantras of Silicon Valley seems to be scale. The goal isn’t just to build a new technology, but to build a technology that will scale to the whole world. Build a single system that will reach everyone. It is this imperial ambition that pushes us towards software monoculture.

Once a monoculture is established, even a small bug or bias in the system can have huge repercussions. The reason that exploits on Facebook, YouTube, or Twitter matter isn’t because they are particularly powerful — but because they reach so many. Combine this with opacity: I usually don’t know how Facebook or Twitter order my timeline, or how YouTube recommends my next video. Finally, throw in some vicious self-reinforcement: I stay longer on the social networking sites and engage with a more homogeneous type of content.

Suddenly, even a small bias can start to have big effects.

To make things even more difficult: for the most important systems and processes, the repercussions can be difficult to spot. In the case of Gros Michel the effect was obvious: crop yield dropped until there was no more commercially viable Gros Michel bannanas. The farmer could directly see how the technological bug — a fungus attacking an asexually reproduced plant — was destroying their system. In the case of the Facebook security bug, the repercussions are relatively obvious: potential unauthorized access to 50 million accounts. The link to technology takes a little digging: finding the three bugs. But how do you approach measuring the repercussions of misinformation on social media? Or echo chambers?

Solutions aren’t always obvious either. Simply having more companies, for example, doesn’t help if everyone follows the same methods. That was the point of systemic failure in banking networks. In social media, you can see this with all three of Facebook, Twitter, and YouTube providing equally vicious reinforcement schedules for their users.

I don’t know how to approach many of these questions, but I wonder if there are useful insights that I could offer.

If we stick to the metaphor of ‘software monoculture’ then it suggests that evolution and ecology might offer a useful lens. Do you, dear reader, think that there is more than a metaphorical connection? Can evolution and ecology help us understand software monoculture? I’ve encountered the work of Stephanie Forrest on the evolution of software, and Russell Dinnage just pointed me to a study of biodiversity in the Linux ecosystem. I’m also familiar with the import of ecological models into finance for studying the ecology of banks. But what else should I be reading?