What if the truth is too complex to fit inside one human mind?
A poker player sees the world as a hand of Texas holdem. A playwright sees the world as a stage. I started my career managing a few hundred machines as they talked to each other about the stock market, and so I see the difficulties of our world reflected in the challenges of distributed computing. I think there’s something deep to this metaphor, and so I’ll try to share that image here.
What is distributed computing, you ask, and what does it have to do with politics? Let me explain it to you simply, with people.
Suppose you have a list of 10,000 numbers you need to sort. You have, at your disposal, 100 students. These students are able and willing to perform whatever task you give them. The only catch is that each student has one white board to work with, and that whiteboard has limited space. They can draw pictures, work out equations, or store numbers on their whiteboard, but each whiteboard is only so big.
If each whiteboard can store exactly 150 numbers, then you have more than enough total storage space to represent all 10,000 numbers. 10,000 numbers is way too many for any single student to deal with. But with 100 students, this problem is solvable.
Your first step might be to divide the 10,000 numbers up into 100 chunks of 100, and give each student a whiteboard with their 100 numbers. Then, you’d tell each student, “sort the numbers on your whiteboard.” They happily oblige — and then what?
You need some way for the students to communicate to each other, about who has the largest number, and who has the second largest. You’ll have to merge all of those sorted lists together.
Remember, though — no student has a white board big enough to represent the entire list. You’ll have to design a way for the students to communicate and exchange numbers with each other, as they gradually sort the entire list.
Maybe you have each student partner up with the student next to them. They trade numbers back and forth, until one partner’s numbers are all higher than the others.
There are many different ways to solve this problem, but the key thing to understand here is that no one student can see the whole picture — it’s just too big to fit on one whiteboard.
Some solve the problem, others help coordinate.
In addition to using their whiteboard to store numbers, the students may also need to use that whiteboard to keep a list of who they’ve talked to, or to figure out who they should talk to next. Perhaps each student keeps a list of students near them, and they periodically ask each student near them, “What’s the biggest number you have? What’s the smallest?”
The students will have to write that information on the whiteboard as well. Each student has some extra space, so they can store numbers, in addition to details about who they’ve talked to and what answers they heard.
If we want to solve this problem as quickly as possible, you can imagine it might be more efficient to have some students use their whiteboards entirely for communication. Imagine one “manager” student who walks around with their whiteboard, talking to a group of 10 students. This manager student talks to each student in her group of 10, and figures out who has the highest numbers, who has the lowest numbers, and helps them decide which others student to pair off with and exchange numbers:
How is this relevant? Why does this matter? How is sorting numbers relevant to anything?
The entire human race processes information about its environment in order to survive and prosper.
Each of us encounters problems in our daily life. Sometimes, we can draw on known answers to solve those situations. Other times, we have to work out a solution on our own.
Imagine a self-driving car approaching an intersection. It needs to figure out which route to take — and it has to solve this problem before it reaches the intersection. A car can slow down to stop and compute , aka “think.” The car can spend the time it takes to reach the correct conclusion. Humans don’t always have this luxury.
Historically, if you saw a strange adult male you didn’t know, you had to decide, quickly, whether or not this male was a threat to you. If you decided wrong, you’d risk dying.
Culture is one solution to that problem. “If the adult male has a certain emblem, then he’s most likely safe. If he doesn’t have that emblem, then he’s probably dangerous.”
States are another solution to that problem. “If I am within the boundaries of my state, then I’m probably safe. If i’m outside the boundaries of my state, then I might be in danger.”
“Is this stranger safe?” is just one problem we have to solve in order to survive. “What will I eat today” is a simpler example of a problem that each person must solve on a daily basis, or else, eventually, they’ll die.
Other problems might include:
- My house is on fire. How can I make that not be true?
- My child is sick. What’s wrong with her?
- I feel terrible and don’t feel like being alive any more. Why should I bother to continue living?
The human species solves, or attempts to solve these problems, all day long.
Imagine that instead of “sorting a list of 10,000 numbers”, the students are working on the problem of “How can we allocate and consume resources to maximize human welfare?”
And instead of a fixed set of numbers, the students are given “the observed conditions of physical reality”, which changes over time. Having a solution eventually, with infinite time, just isn’t good enough. You need a solution in time to actually use it.
This “resource allocation” problem is a problem that the human species works on constantly, and “corporate capitalism via sovereign states” is the heuristic we came up with. The system of private ownership, trade and exchange, with violence coordinated via geographically-bound entities known as “states” is the dominant protocol we’re speaking to each other with when we humans attempt to navigate our reality.
Labor is the people actually sorting the individual numbers on the whiteboards. Capital facilitates the communication between labor nodes, and makes decisions about how to allocate the scarce resources. The state consists of enforcers and politicians. Enforcers are guys walking around with a whiteboard and a gun, shooting anyone they see as causing trouble, or perhaps anyone with names from their whiteboard. Politicians manage the enforcers.
What each of the agents do depend upon what’s written on their whiteboards. That’s true of all of us. Much of our culture and history can be seen as other information on the whiteboard each of us carries in our minds.
Things get much harder when you don’t have trust.
Now for a small digression on trust: Imagine how hard it would be to sort those numbers if some of the students were actively trying to sabotage the sorting process.
Now, in addition to having students sorting numbers, and students facilitating communication, you have students trying to figure out who the bad students are.
That’s a much harder problem to solve. In distributed computing, the added difficulty of “bad guys trying to sabotage you” are known as Byzantine agents. We say that a system has “byzantine fault tolerance” if it still works fine, even when there are some number of bad agents.
I have faith that “a human politics that actually works” is a solvable problem, largely because there is a solution to a problem called “Byzantine generals”, which is the problem of “how can a group of agents that don’t trust each other agree on the result of some computation?” That solution is called bitcoin, by the way.
Even if there were no evil in the world, if people incorrectly perceived each other as evil, they might act in ways that confirmed those suspicions. When there is actual evil, I think having a functioning system which determines “who is trying to sabotage us, and who just got a bad message or made a mistake” is essential to have things functioning correctly. Even in the absence of evil, we need to understand why people do things that seem evil.
I think the problems of today will seem, to people in the future, like obvious consequences of a simple fact few are aware of: we only solved byzantine generals a few years ago.
Ok! How’s that for a diversion? Let’s get back to the main topic here:
Nobody’s Whiteboard is Correct. Nobody’s can be.
Remember that each of us, personally, has to survive. We use our whiteboards both to coordinate and communicate with other nodes, but also to ensure our own survival and manage our own lives.
These contents of these whiteboards — our internal narratives to ourselves, in both mundane details (the contents of my fridge), to more abstract details (the balance of my checking account) to the most spiritual of details (the purpose of my being) — these represent reality to each of us.
And yet reality itself, is far, far too large to find it one person’s mind.
When I walk on the floor, I can’t be thinking about what the floor is made of and whether it will support my weight. I have to trust that it will. If each step required me to inspect the floorboards to make sure they were made of proper material and installed correctly and not rotted, I couldn’t do anything all day.
And yet someone has to know these things. Someone has to do it. That someone’s whiteboard will be much more focused on, and defined in terms of, structural integrity.
Which brings me back to the start. Each of us sees reality in terms of what we use to survive and navigate our daily lives. The poker player sees life as a game of poker. To the playwright, it’s a stage. To the entrepreneur, their life is a business and they are the CEO.
We have to constantly remind ourselves that we’re not seeing all of it, and maybe the people who we think are “uncaring, selfish assholes” (to use one set of moral values), or else “tradition-blind, disloyal perverts” (to use another set of moral values) — maybe these people are trying to prevent the problems they’re seeing, which it simply isn’t our job to deal with.
Maybe we actually need people we think are obsessed with trivial nonsense, because they are aware of a real danger that we’re partially blind to. A house divided against itself cannot stand. A social network partitioned against itself will never reach the stars.
Learning about distributed computing actually helped me come to understand empathy. I think that’s a lesson our whole world needs to learn.
Mark P Xu Neyer 