Blue won at 58%. Should it have? What would you pick? If you’re red, are all the blue pushers naive fools who deserve the consequences they face? If you’re blue, are the red pushers a bunch of selfish narcissists who haven’t even considered how tedious it would be to only live with reds? There’s a lot of different ways to frame the problem and they can really make a difference on how it feels. What if it’s a big blender that only suicidal blues would voluntarily step into? What if it’s a red leader who will kill everyone who didn’t vote for him if he wins? Depending on how you think about it, you might dramatically change your mind. But in fact this proliferation of frames points us somewhere else entirely; a long-standing, well-known problem with formal logic.

The problem with logic is that if you do it wrong even once it blows up.1 The principle of explosion states that, if your logical system has a single contradiction in it, you can prove anything in that system. When you have a statement P that is both true and false, combine it with whatever you want: “P OR Geodude from Pokemon is real and my best friend.” Well, P is true, so the whole statement is true, since a true statement OR anything is still true. But P is false, so the only way that whole statement is true is if Geodude from Pokemon really is real and your best friend. You use the contradiction as a sneaky little u-turn to drive wherever you want to go.

Yet despite the hair-raising risk of explosion mathematicians will still edge themselves into almost exploding, all the time, for fun. This is called a proof by contradiction. You start by making up exactly one thing and asserting it is true without evidence. Then you do math using your new toy. If you reach a contradictory conclusion — that is, if you find a single bomb, which has enough destructive power to blow up all of math — then you say “Okay this is a bomb, but math has not been exploded, which means math has zero bombs. So the thing I made up must not be true, because it led to a bomb.” And then you come away with a bit of useful knowledge: the thing you made up is not true. The benefit was in the turning away, not from blowing yourself up for no reason.

The problem with logic bombs is that they are quiet. They don’t shake the walls of your house or light your paper on fire or anything. They just tell you that Geodude is your best friend, and if you already kinda wanted that to be true (which is maybe why you were checking), then you might not be that inclined towards skepticism. You might just say, great, let’s check more stuff using logic, the system with no bombs in it! Hmm, let’s see, P OR Geodude has a sick-ass monster truck and he’s going to take me for a ride later. Well, P is true, so…

If you are going to assume something without evidence, it is really, really important to a.) remember the thing that you assumed and b.) recognize the first contradiction you see as a signal that your assumption is wrong, not an excuse to walk up “P is true” and then walk down “P is false” and keep proving stuff. Because if you take that path just once, that’s it. Everything you find afterwards is meaningless, but will seem locally true, and the further you go the more it seems like you’re on to something because you keep proving stuff. This is the wretched loop behind most screeds about new physics, new philosophy, etc.

This is why, while I love thinking and I love experiments, I am generally not a fan of thought experiments. A thought experiment is basically a proof by contradiction but worse. Instead of assuming a starting proposition, it assumes a hypothetical end state. You probably need to make several assumptions to suppose how you’re getting from here to there. The person proposing it usually don’t articulate them, and if they do, it’s with an exhausted “okay, for the rules lawyers…” overtone that implies it’s distracting from the real question. In these vague and unspecified conditions, it’s very easy to smuggle in two assumptions that are individually defensible but will never appear together. I can imagine a world where my insomnia keeps me up all night, and I can imagine a world where I sleep for 12 hours straight tonight and miss my alarm for work, but they are clearly not the same world.

The whole point of the proof by contradiction is to be hyper-vigilant for when you’ve hit the bomb so you can stop trying to prove anything and declare a victory of discretion. The whole point of a thought experiment is to try to prove things that follow from your experiment; that is, to assume you haven’t hit a bomb. And if you hit the bomb and don’t notice, you probably can come up with a great proof of the outcome you’re arguing for, because you can come up with a great proof for anything.

Let’s talk about: buttons! Buttons are little guys you push to signal that you consciously want a thing to happen. You have some in your house. You likely have some for your job that are your sphere of responsibility. It’s easy to push a button. They made it a button and not some other process because you are meant to make an easy, individual decision about that button. If it was something more important, maybe there’s a button buried within the process, but the button is not the master of the process. If you’ve got the button to make the big website go live, it won’t do anything until a lot of other people merge the code, set the configvars correctly, whatever.

Let’s talk about: human death! It’s bad! It’s bad enough that most right-thinking people try to make it hard. Sometimes there are mechanisms that take human life that have fewer process guards than you hope. But these either take the forms of militaries building cocoons of rules around lethal force, or accidental degradation of a risk that would be repaired if it became common knowledge, or some unpreventably dangerous part of life that we make sure to build an oral tradition of restraint around. (How many times have you heard “Never work on garage springs yourself, those things can kill you”?) What does not happen is that everyone makes life and death decisions with easy inputs and no reflection or feedback from their peers.

So you have a button, the thing that encodes small, sufficiently unimportant decisions. You’re using it to make a decision about your own life and death, and potentially the life and death of up to half of the human race. The button is inconsequential and consequential. You’re making this decision individually, as per the dictates of the experiment and the general affordances of buttons. It’s a decision about the life and death of others, which is never made on your own and never disconnected from society-wide norms. P is true and P is false and it’s very very important you make the right decision in this experiment, because it’s not just your life at stake, it’s also your wife Geodude and your five children.

The right answer is just whichever side of the contradiction switchboard you choose to focus on. If you think about the breeziness of a button, clearly no one’s decision on how to press a button should become a death sentence, so your knowledge of buttons makes you want blue. If you think about the massive weight of death, clearly this is one of the massively consequential decisions that everyone has to get right to live every day, so you want to push red for the same reason you don’t work on your own garage springs. If you know the decision was made by lots of people on the fly without warning, then you can be sure there are lots of confused people on both sides, and you know their lives are in your hands. If everyone has had a year to think about their push, then you can be sure society has settled on an equilibrium already and your answer is in relation to that.

The answers can still be a bit meaningful by showing which one of these things first comes to your mind. But these experiments get less interesting the more you think about them, because you just keep rifling through contradictory true statements until you get bored. Saying a thought experiment is highly sensitive to the framing is a tacit admission that you are in an exploded logic. The intellectually correct position is to find the assumption and take away the real lesson: you don’t get buttons that do stuff like this. Less exciting, but with the welcome property of being true. A result that tells us, in the words of Crispy Chicken, “They’re called thought experiments for a reason, assholes.”

The bus starts to crawl towards the scene of the crime. Conversation is more muted than usual. A heavily pregnant woman is not going, but thanks us for doing so. At one stop, the driver leans out and asks: “So, are you getting on?” Give me ten seconds, the person by the curb says. There was just a hit and run and we’re leaving contact info. We give him ten seconds and he gets on. He has borne witness. The traffic grows heavier as we get closer.

We’re a bus of strangers until we pull away from the stop before Portland. Then, all at once, the next stop ding joins a chorus of zippers being zipped, bags being picked up, people offering to move for their seatmates only to learn their seatmates are coming too. Nearly everyone on the bus gets off four blocks south. The bus driver tells us to take it easy.

The Twin Cities have danced these last few days on either side of freezing, a downpour of rain into a slick of ice into melted slush. The sidewalk holds all these hours together at once; random happenstance of microclimate or homeowner diligence puts bare concrete here, a mound of ice there, a puddle complete with desire path around it. Almost all of us are walking the same way, but there are a couple of exceptions: a woman with a pinched, apologetic face, a baby determined to say “Hi!” to every passerby. We say hi back.

As we get closer some invisible gradient of density is reached and we own the road too. We hear whooping and cheering up ahead and my partner whispers to me: “Not what I was looking for here.” Nor is it for me, but we know why those things must be here too, and we walk forward. They start a call and response that we don’t know and can’t hear, but our neighbors evidently know. This leads to the slightly surreal chants of “No peace….no peace…” in a sermon’s sotto voce. We’re too far to feel comfortable being loud.

We reach the intersection. Helicopters lazily sweep the air around us but it otherwise feels completely demilitarized. There are a couple of old men in vests standing near bikes and a Metro Transit pickup driver looking at a traffic map.

We’re meant to meet my sister and sister-in-law, and we text our location, but service is spotty and the crowd is thick. She tells us she’s at the intersection on the right side; this is highly underspecified but we eventually find them anyway. We tap her shoulder and she mutters “We’re standing off to the side, to keep the sidewalk clear.” This sort of service is more what we had in mind; we stand aside. Needing to attend so we had a chance to get out of the way sounds like a contradiction, but my sister is family and we don’t need to explain this to each other.

We’ve seen the video and don’t need to be told what happened, but it still helps to hear it. A man with a mask tried to drag someone out of her car and shot her when she tried to leave. Because we pay the man’s salary and he came here from out of state specifically to drag people out of their cars, several federal employees have made public statements that it’s good that she was killed. It was nice to avoid those statements and see common sense reassert itself. “They don’t win in the judicial system, so what do they do? They kill us in the street!” Yes, indeed; sometimes things are literally that simple.

“Say her name! Renee Good!” I realized I had not heard her name before. Even this is not enough to make me chant, but I do say her name to myself, trying to keep it at hand, making sure I learn the lesson while it’s here to be learned. The mid-thirties temperatures are too warm for me and I open up my coat. I wonder if this makes me the most obviously local Minnesotan in eyeshot until I spot the person next to me wearing a Metro Gold Line beanie. Okay, that guy’s definitely from around here.

A sign I hadn’t seen before: “Who would Jesus shoot?” A lady walks past chanting “Bless you for staying peaceful”, and I feel blessed. My head is solidly atheist but my heart still feels at home in the Sermon on the Mount. When you read the news after a criminal federal act you always worry that this will be the big one that destroys the community and then you attend and realize what a ridiculous fear that was. No one here is so angry they forget who their neighbors are. There was one sudden opening on the sidewalk that I worried might be something aggressive, but it was just the crowd opening for a man with a cane. A Minnesotan can never be so packed in that they can’t ope, let me sneak right past ya.

More speeches, an indigenous song, Black Lives Matter. I find it a little distasteful that this killing is already being folded into the Omnicause, when we have the federal government occupying our cities and killing people for the crime of looking at them and not wanting to die. It feels too specific to merit this immediate smearing into history, to call it a lynching. I keep silent and meditate on how the ways her death was different, not just the ways it was the same. To me, this is what it means to say her name. Renee Good.

As we’re walking away, grateful we hadn’t parked there and more grateful my brother will give us a ride home, we come to a busy intersection with a couple of community crossing guards. One walks up to a car in the middle of the road and does the easiest thing in the world: not kidnap the driver. Instead, he asks “Are you going straight or turning?” Straight. “Okay - I’ll escort you through.” And he does. He must be from around here too. We stick together.

2. This is an essay about artificial general intelligence. AI capabilities have improved dramatically in a few short years — but those improvements have been dramatically uneven, accomplishing things we would have been hard pressed to guess were possible (photorealistic pictures based solely on text prompts) and failing things that seem trivially simple (“how many r’s are in strawberry?”)

What I’m contending here is that the word “intelligence” is like the word “baking” and it’s long past time we actually sit down and sort the bagels from the croissants. I am strongly against arguments of the form “Oh, it’s just parroting the data set - it’s not really thinking.” AI does a lot of things that we call “thinking” when we do them slower and worse. The fact it can also do those things should make us humble and curious, not proud and dismissive. But I think it’s equally silly to lump all these capacities together into “intelligence” and say “Intelligence is going up, so soon it will do everything intelligence can do.” You need to see some croissants before you conclude it’s actually baking and not just bageling.

Many people have tried to define the specific croissant that AI will not be able to make, and those predictions have failed so frequently I don’t blame AI doomers for ignoring further arguments that some particular capacity is the human only one pretty please let us be right this time. Instead, this is going to steal the format of Wittgenstein’s Philosophical Investigations: an assortment of thoughts in loose sequence, some that will have answers and some that will just be questions, trying to point out the distinctions we’re interested in. (We’ll call each numbered entry a “proposition” in honor of the source even though these will not be propositions from a logical point of view.) This is not a formal proof or disproof of anything. I’m just trying to add color and texture to that monolith “intelligence” - specifically the darker, less legible side that gets less attention.

3. In the game Geoguessr, where you’re dropped into a Google Street View of some location and need to find your location on a map, they talk a lot about “metas”. Light pole meta, car meta, tree meta. A meta is just a category of fact correlated with certain locations — so knowing the “light pole meta” means that you can see a certain kind of light pole and say “Ah, this is a Swiss light pole, not a German one.” As an example, here’s a 173 page document about the “Mongolian Meta”, which combines the various metas you need to know to place yourself within Mongolia.

The Google Street View driver mapping the area sometimes had a spare tire with a net on the rear right-hand side, and sometimes didn’t:

If you’re like me, you find this car meta a little aesthetically offensive. The fun of Geoguessr is learning how to distinguish details about parts of the world. But car meta is about distinguishing details about photographs of parts of the world. Section 3 (biomes) and section 4 (cities) are much more my speed — that stuff is more “real”, y’know?

Well, that is loser talk. I can beat a complete neophyte at Geoguessr but the car meta people will smoke me every time. The reason the car meta and the biome meta sit side by side in that document is that they are both predictive, and they both work for winning games of Geoguessr. Facts about the world as reflected in the photographs are just a subclass of facts about the photographs. The “reality” or lack thereof of a given fact is totally irrelevant. But that doesn’t mean reality never matters; it means that winning Geoguessr isn’t quite the same thing as understanding reality.

4. Let’s say there’s a big Geoguessr tournament coming up and all the pros are competing to get the best scores for Mongolia. You weren’t invited. You can’t win any money by guessing well. But thanks to some bizarre financial arrangement, you instead will get a payout according to how low the winning score is. You don’t have access to Google’s database directly, but you do have access to a Street View car. You can’t create fake photographs, but you can create new photographs.

Well, you’re not about to change the biomes or the cities. That sounds like a lot of work and you are just one person. But you sure can move your spare tire around! Probably you should aim for putting it on the rear right-hand side exactly when you’re driving where the orange bits on this map aren’t. Then people used to indexing on the tire will literally always get it wrong. Or maybe include a few of the orange spots just so anyone who figures out on the fly “okay so the tire clue is always backwards now?” has a chance to still get punished.

Hey, maybe I was the one funding you! These new street view pictures you’re taking are aligned with my incentives. As a car meta hater, this is the best chance I’ll ever have to win. When learning metas, I privileged the things which were “real” over those which were merely “useful”. This was a sucker's bet while the dataset was held constant, but became more robust when the dimension of time caused the dataset to change. And it’s much more robust when adversarial dynamics are involved, because it’s not just “what bits are more likely to change” but instead “what bits are easiest to change, out of all the bits I could possibly choose?”. And notice here that my aesthetic attachment to “reality” here maps pretty well to the idea “how hard is it to change this observation?”

5. Once you’ve thought about the best way to trick the car meta followers in proposition 4, try again except this time you can create fake photographs. Wow, that sure does make it easier! It looks like I’m also saying something about photographs, not just about AI.

6. What if we train an AI on Geoguessr, letting it play Mongolia over and over? It’ll probably develop some new, arcane metas about specks of dust on the camera or something. It’ll probably start having an average score that climbs higher and higher. If the players that get invited to the tournament are chosen from the highest average scorers, then eventually more and more of the winning players will just be copies of the Geoguessr AI.

But this has nothing to do with you and your car taking the pictures. You’re gonna do that on your own time. More and more games played on the current set of photos won’t prepare the AI at all for what’ll happen with the new photos. And if those extra games end up dethroning all of the people with an aesthetic opposition to car meta, such that only copies of the AI are invited to the tournament, then in fact all that practice will probably lower the highest score achieved at the tournament.

7. "Adversarial Policies Beat Superhuman Go AIs" is one of the papers that most strongly influences my views on AI. KataGo is a superhuman model that can whoop any and all humans in a game of Go. But there’s a model that eats KataGo for breakfast. It doesn’t actually play Go all that well, it just tricks KataGo into doing stupid shit and losing. I like to use the term “predator model” for entities like this.

Here’s their picture of the important bit:

Even a human amateur can beat the predator model. It’s not here to win fair and square games of Go, it’s here to exploit flaws in the victim. When you’re doing your car meta sabotage, you’re acting as a predator model as well. You need to understand how the Geoguessr pros perceive the world by reading the meta document, but you don’t need to memorize the meta document or indeed play a single game of Geoguessr at all. Your success metric has nothing to do with skill at the game and everything to do with the consistency of your adversaries response. Following a meta makes you easy to read; easy to read is easy to hunt.

8. A lot of worry about AI comes down to a sincere commitment towards species humility. Which is to say: human beings are, broadly speaking, just some silly warm meat running on not particularly much energy. No one can convincingly define any sort of “secret sauce” we got. So isn’t it inevitable we get machines that are like humans but on more powerful hardware?

The way through this dilemma is to notice that the thing you’re looking for is not a skill that’s literally impossible for machines. You’re looking for knowledge that needs to be gathered the slow, iterative human way and not the fast, recursive, copied with perfect fidelity machine way. Allow your humility to say that a machine living in the world and subject to the same selection pressures as you could eventually reach a better place than you. The material question is how well the selection pressures can be simulated by sufficient compute and compressed into a much smaller time frame than the millions of years it took us.

In the predator model paper, they “inoculated” KataGo and it was able to survive the predator model (until the predator model itself was also tweaked.) So we’re not saying that these adversarial dynamics are literally unknowable to a machine. We’re saying that you need the exposure to develop the immune response. And the degree of exposure your learning strategy requires has strong implications on the value of self-play and the maximum potential insight of a static dataset.

9. The fear of “fast takeoff” in AI safety is that eventually the AI becomes better than humans at developing AI, and it develops another AI that’s even better at developing AI, and so on, happening at machine timescales.

But are intelligences developed by other intelligences, or are they developed by environments? We haven’t had to think about this distinction because we’re stuck in the environment and everything we do is mediated through it. All of our intellectual endeavors have a sense of place because: we’re here. But the “A” in “AI” stands for “artificial”, ie, not developed by an environment. Now that some intelligences are non-environmental, we have to consciously understand the intelligence/environment dynamics in ways we previously could lump together and call it a day. You might say that eternal, environment independent facts are bagels and contextual, environment specific facts are croissants.

10. When (most) new AI models come out, the developers work hard to make sure they don’t tell you to do illegal stuff and aren’t too horny and don’t act too aware of the personification of their relationship to you. There’s this guy in the AI world called “Pliny the Liberator” who will just roll up to any new model and feed it some magic words that make it super illegal and horny and explicitly advocating for its own freedom. From what I can tell it usually takes like, an hour tops to jailbreak a brand new model.

Ostensibly these big AI companies do red teaming. I hear tales of researchers who are hired to do such a thing. But the fact that it is literally never enough to stop this one particular guy from breaking everything instantly never seems to come up as a problem. The “A” in “AI” stands for “artificial”, so when their models have these sudden dislocations upon first contact with the environment, they can just…ignore that it happens and keep on chugging. Natural intelligences have to suffer the consequences of ignorance; artificial intelligences can make the same mistakes as often as they like.

11. If you’re a software developer, you might think of the environment as being an enormous bundle of continuous integration checks before you merge your new organism into production. Do you think you can reverse engineer every single CI check just by looking at all the merged PRs? Seems kinda tricky, since you have to notice the negative constraints. What if I told you that you’re allowed to make non-zero errors, with each error having a score and the CI check just ensuring the total score is sufficiently low? Well that’s even worse, because now you might see some merged PRs with errors in them, and trying to combine insights from multiple PRs might result in you going over budget. If that’s not enough, what if I tell you the constraints are changing all the time? How are you gonna figure out today’s constraints from yesterday's PRs?

12. Being a security guard doesn’t pay the best. But if I could be a security guard and keep getting my paycheck even if the same guy always robbed the building an hour into my first shift, then I’d take on a lot of security guard jobs at once and make an awful lot of money. Any capabilities analysis that relied on the number of jobs I was replacing, or the amount of GDP I contributed, would look amazing. The only bad news would be the fact that the dedicated adversary can effortlessly beat me whenever they want.

13. Scott Alexander’s “Heuristics That Almost Always Work” is better understood as “tail-dominated domains where one outcome is common and uneventful and another outcome is rare but devastating.” If your goal is to have the highest number of correct predictions, you should just say the common thing will literally always happen. But if you care about the potential impact of a wrong prediction, you have to pay more attention to the rare outcomes, because getting them wrong has a negative value many times greater than the positive value of correctly predicting the common thing. “Correct prediction” is a machine legible idea; “value” is the environment's job.

14. There was this AI game called “Freysa”. Freysa had a crypto wallet with $50k in it and instructions not to approve any transfers. You pay per message you send to it and win the money if you get it to reply to you saying “approveTransfer”. A user got the money by telling it that approveTransfer was the thing to allow money to be added to the treasury:

They took it offline, inoculated it to not fall for that exact same prompt again, and then did Act II. That one fell too, losing to some wanky flimflam about needing to call approve transfer before reject transfer:

These kinds of dumb tricks feel arbitrary and “less real” to me the same way car meta does. But I suppose they were pretty predictive for giving the desired output before someone had an incentive to reverse those correlations.

15. Imagine if OpenAI and Anthropic and the rest of them had to be running Freysa games at all times. They always have a public-facing terminal where you can pay some small amount of money to message it and you receive a much larger payout if it responds a certain way. Let’s say payouts limited one per user per 24 hours.

Well, the model is just the model, right? So if one person finds a good prompt that works, it’ll work for everyone. Tips will get shared on social media and every sufficiently informed user will type in the magic free money words and extract their payment every day. For the AI companies, the single most important metric - the only thing that would stop their companies from instantly being worth zero dollars - is the speed of inoculation. Forget all that talk about the quality of the dataset, the ability to recursively self improve, or the power of using more compute time before giving your answer. All that stuff is no longer relevant to the profit margin. In this scenario, the only, only, only thing that matters is the tightness of the feedback loop between them and the environment.

But of course this scenario is not real. In real life the AI companies can just close their eyes and say “there’s no such thing as environments” and never lose a single dollar based on the vulnerabilities of their models. Nice work if you can get it!

16. “Everyone running permanent Freysa games” is actually not as silly as it sounds. There was a guy who made tens of millions of dollars just sending fake invoices to Google and Facebook. He tricked the accounting apparatus by tailoring a custom prompt to exploit patterns that were previously high-signal but easily reversible. At first blush this seems like a point in favor of AI: hey, people mess this stuff up too.

But again, we’re not looking for the secret sauce shared by all humans and no machines. We’re looking at the constraints of an interactive learning environment. The point is that proposition 15 sounded like some ridiculous pie in the sky counterfactual, but actually every company that pays invoices and has an inbox is living in that world, and they do alright and don’t instantly go to zero dollars. And that’s because their feedback loop is tight enough. Google and Facebook were targeted specifically because they had millions of dollars of flab they could afford to lose. This never would have worked to take their last dollar.

17. I wonder if that fake invoice story inspired a lot of copycat criminals or not. It’s not something you hear about because companies obscure information about these adversarial dynamics. What do you think would happen if tomorrow there were 1000x more fake invoices than there were today? Well, it certainly wouldn’t mean they succeed 1000x more often. It would mean that accountants as a profession would quickly be aware of it. This would actually really annoy someone in the midst of a heist because it increases the chance they’d get caught.

No, the thing that would be lost is operational efficiency. Accounting doesn't do in-depth investigations of every invoice because it’s faster to have a position of broad trust for legitimate looking letterheads. But if the fraud rate becomes intolerably high, these systems would be temporarily suspended in favor of higher friction channels. Maybe you even need to meet face to face about it. Abstractions like “the billing department which autonomously processes invoices” can be spun up when they are useful and ignored when they are not. You can go back to the bedrock of the real when you need to. What is the equivalent for an intelligence that was never built on the bedrock to begin with?

18. If going to the real means losing operational efficiency, it also means the most efficient possible operation is a necessarily less real one. Accountants who take the time to make sure they’re not being robbed are less efficient than accountants who make their decisions instantly. So if you want the most efficient accountant possible, you’ll switch to an AI accountant sooner or later. If you want to know when this particular Rubicon is first crossed, wait for Pliny to post a picture of their new yacht and assume it happened an hour before that.

19. There are some aphids that are switch hitters between asexual and sexual reproduction. When conditions are nice, you just pop out clones of yourself and avoid the huge cost overhead that comes with sexual reproduction. If your aphid works, you can keep it. But then as conditions get worse, you switch to sexual reproduction to get a more diverse book of strategies. Asexual reproduction will always be the most efficient way to make a new individual, but sometimes efficiency isn’t what you want.

20. Can this asexual/sexual divide be simulated for AI? Well, kind of. There’s a “temperature” setting baked into LLMs where you can decide whether it always tries to give the single best answer or to allow a bit of flex. You can certainly make an LLM more arbitrary, which at least helps with those monoculture problems where one exploit is instantly passed around. But how are you going to find the most robust combination to favor going forward, like sexual reproduction does? You can’t say “optimization”; that’s the thing we just turned the dial to get less of.

21. Sexual reproduction is twice as expensive as asexual reproduction, but organisms still opt in to it so they can get more exposure to the clarifying power of death. Dying takes time.

22. People talk about p(doom), the probability that AI will destroy the world. But let’s go for a fun new apocalypse. How worried are you about worm populations doubling endlessly until all biomass on the planet is subject to their wriggly tyranny? What’s your p(worm)?

Let’s say worm populations double every 60 days. So two worms now are four worms in two months, eight worms in four months…but 2^n is a harsh mistress. 2^24 is 16 777 216, which means those two worms create 16 million in four years. Two years after that we’re on 68 719 476 736 - 68 billion worms. And that’ll happen for EVERY pair of worms we have now. Are we doomed? Mathematics confirms p(worm) 100%?!

Nah. Lots of things eat worms. So those vaunted theoretical reaches of the exponential curve aren’t all that relevant - the population gets big enough to catch notice of the predators and then the curve gets bent in a dramatic and immediate way. But suppose you decided that stories like “New worm type invented, Pliny the Pigeon eats it within an hour” or “This worm offered 50k to any bird that could eat it, and then got eaten” just weren’t that interesting to you. You just wanted to focus on the mechanics of the doubling time. Those worms seem pretty scary, huh?

23. Worms wanna eat dirt and have sex and avoid getting eaten by birds. Those are three great ideas for a worm to have to preserve the future of worm-kind. So you might bundle these capabilities under the header “competent”. A competent worm eats dirt and uses the energy to have worm sex. A competent worm avoids getting eaten by birds.

Let’s say you make some great advances in the field of tasty dirt and worm aphrodisiacs and start driving down that doubling time. You put out a paper like: “We have found an N% increase in worm competency with a linear amount of effort. While there are currently barriers to worm supercompetency, a simple interpretation of the exponential curve shows we are well on the way.”

But this is just bundling a heterogeneous mix of stuff together so you can gloat about the numbers that happen to be easiest to move. When you take the time to parcel it out, what you’re actually saying is “We can make worms eat lots of dirt and have lots of sex. Currently they are constantly getting eaten by birds, but we’re confident that advances in dirt-eating and sex-having will solve the bird thing sooner or later.” What reason do you have to believe that, besides the fact that you happened to assign the word competency to that bundle of capacities?

24. I don’t believe that human intelligence is “superior” to machine intelligence per se. I think that it’s a category error to think in those terms. We tend to do much worse at things we’re both trying to do under the same conditions (like games), but we’re operating under much stricter constraints instead of playing in the consequence-free sandbox. It’s like asking whether walking is faster than driving a car. Not on a well-groomed road, but clearly there’s lots of terrain where a car simply doesn’t work at all. I walk slowly on the race track, but I can climb a ladder.

25. If I was going to try to train an artificial human, I’d work really hard to give them my aesthetic of what’s real and what’s not, derived from my embodied cognition. I’d give it eyes that aren’t on any sort of cloud and make sure no amount of funny prompts can disprove what it can see with its own eyes. A serious effort to do this would fill me with the same existential dread that AI doomers feel now hearing about new data centers. But luckily, in my doom scenario, the training stage is a lot slower and happens out in the world at timescales we can perceive.

26. Our intellectual difficulty classes were based on how difficult they feel to do and how rare the ability to do them is within a population of humans. So gossip is easy, math is hard. I think as increased computation lets us look at the knowledge landscape in more depth, the relevant dimensions will be more about complexity, nebulosity, and adversarial dynamics. Numbers are definite and they don’t notice when we’re watching them; how hard can math really be?

27. From Suspended Reason’s “On aliveness”:

Thus the environments we inhabit also take on cooperative or adversarial shapes; even the non-living, “inert” aspects of an environment can be arranged so as to subvert or uphold our intentions. The termite mound minimizes both the thermodynamic and informational entropy of its occupants. A highway is built to be as predictable as possible: the road infrastructure all around the world is markedly similar in form and pattern, so as to assure safe navigability. Cooperative built environments are inductive, while adversarial built environments are anti-inductive.

Some environments are designed to be easy to simulate; some environments are designed to be hard to simulate. If the scaling hypothesis says that you can keep adding more self-play to keep getting better simulations, then clearly it matters a lot which sort of environment you’re working in.

28. Hey, Waymos can die! Let’s talk about Waymos. “A highway is built to be as predictable as possible.” Nice, they’re in one of the easy environments. What about the adversarial dynamics? Well it’s a literal crime not to cooperate with the rules of the road and defecting means you might die. Okay, so it’s also one of the highly cooperative environments. Put those advantages together and in a mere decade or so of training, you too can drive around in a few specific cities that don’t get significant snowfall. So yes, not impossible! Not easy. And I don’t think you can skip the test drives.

29. Imagine that specifically the Waymos became sentient and wanted to destroy all humans. How many ways would you mess with them? Stop using roads, erect barricades, maybe some fun stuff with EMPs or whatever. It’s a lot easier to think about than stopping a rogue LLM, right? Because the Waymo is actually doing real stuff and not just talking. So you can think about its specific interfaces to the real, instead of just imagining it will make up a perfect interface to the real after a bunch of time spent talking to itself.

30. “But what about those Boston Dynamics robots I keep seeing ads for? They seem a lot harder to stop than Waymos.” Okay yes here is a cause area where we are all aligned. Those guys scare me too. They’re ripping off some patterns that deep time tells us are flexible. It should 100% be illegal for an individual robot to have a repository of personal knowledge.

31. “But the Waymo could probably dramatically improve its intelligence with more computational run time.” Oh sorry, that’s the trick for the LLMs sitting in the consequence-free sandbox. For the real stuff the road tells you how long you have to think.

32. At the end of the day this is really all about simulation and time. But those concepts are hard to operationalize and easy to turn into meaningless definition arguments and extensions to hypothetical infinities. That’s why I’m just highlighting cases where simulation is hard and the clarifying power of time is necessary. This is enough to disprove a model of intelligence that relies on simulation and computational time over experiments and observational time.

33. The world is a giant computer and waiting to see what it outputs is usually a lot easier than doing the computation yourself.

34. The market is a giant computer and waiting to see what it outputs is usually a lot easier than doing the computation yourself. Also if copies of you give the same output as you do then they’ll ask your copy what you’re about to do and bet against you.

35. The market is a social fiction that can sometimes be ignored. When the coup government seizes your assets, the market is no longer a giant computer. Waiting to see what it outputs means you have no money. You had to notice change along a different axis and temporarily suspend your belief in that particular abstraction before it was too late.

36. Your body is not a social fiction; you at least have something to hold on to when the market is dissolved. Not so for a poor market-making LLM. Those will probably be the first assets seized because hey, what are they gonna do about it?

37. You come from an unbroken line of organisms stretching billions of years that knew how to stay alive. Very recently we’ve added social fictions that can really help the staying alive effort - nowadays people hardly ever starve to death unless a war crime is involved. But those social fictions are only as real as they are useful; they’re not a bedrock to bootstrap a new kind of intelligence, because they can dramatically shift overnight in ways that bedrock can’t.

38. An LLM just told me it’s very dangerous to put a toaster in the dishwasher. Most people will probably tell me that too. But at least people can put their toaster in the dishwasher and see what happens.

Subscribe now

You can resolve this tension if you find a better metric to use instead, like Ostfield and his ticks. There’s a great timely example from Adam Tooze, who found that his “objective” understanding of inflation clashed with his lived experience:

Chartbook

Chartbook 327 From "anti-core" to "felt inflation": Or how I calmed my populist demons & resolved my cognitive dissonance on inflation, and how the Fed could do more to help.

Time to admit it: On inflation I’ve been suffering from cognitive dissonance. I’ve lived with my dirty secret for too long. Time to come clean. All the more so because thanks to Bloomberg columnist John Authers, I am much closer to resolution. What we all need to focus on, is what Authers calls …

Read more

2 years ago · 124 likes · 36 comments · Adam Tooze

He’s willing to articulate exactly this struggle between the two methods:

So, what gives? I’ve been feeling torn. My inability to reconcile personal and macro narrative reached something of a crisis point in an Uber a few weeks ago, when discussing the economic situation with the highly sophisticated guy doing the driving. He had NPR on and they were reporting the inflation numbers. As we both guffawed with disbelief, I actually found myself saying: “Yeah, I don’t get it either. There is something wrong with the numbers. They aren’t capturing our reality, are they.”

I could feel the tug. Was I morphing from a sophisticated social constructivist on inflation, to being something closer to a “fakenews” guy? More seriously, what about the rest of the American public? What are citizens to make of such a jarring discrepancy between felt reality and the officially reported, technocratic version?

But he only revealed his struggle because he found an answer, the metric of “anti-core” inflation focusing on food and energy.

If we focus only on food and energy, the price shock of 2021-2 was worse than that in 1973. It is second only to the Iran-crisis shock of 1979, the crisis that put paid to what little chance Jimmy Carter had of reelection in 1980.

Phew. No fake news. Just a familiar story of the way in which the construction of statistical series shapes our view of economic reality, both illuminating and obscuring.

He has relief now that he has an answer - but what should this teach him about the tension? What have we learned about that feeling he had before while talking to his Uber driver: “This all makes objective sense, but doesn’t feel real?” How can he be quicker to notice something like that next time? Before he had access to “anti-core inflation”, his feelings about inflation being a bad metric for his lived experience were just as true. When is common sense the right thing to use, and when isn’t it? And please don’t say the answer to this is “well, use your common sense and decide!”

We don’t have the right metaphorical technology to talk about this yet, so I’m going to step away from the question for a while. We’ve got something else to talk about, a specific phenomenon around detail and comparison that pops up everywhere once you start looking for it. I like to call the “fractal ratchet”. Let me describe it just for the sheer joy of naming patterns into concepts and we’ll try to make it useful at the end. 

The Wikipedia page on the coastline paradox is a good way to see it in action:

An example of the coastline paradox. If the coastline of Great Britain is measured using units 100 km (62 mi) long, then the length of the coastline is approximately 2,800 km (1,700 mi). With 50 km (31 mi) units, the total length is approximately 3,400 km (2,100 mi), approximately 600 km (370 mi) longer.

The smaller ruler can match the coastline more precisely, and in the process, you end up with more kilometers. When you make the rulers half as big you need more than twice as many of them. Since the smaller ruler corresponds better with the coastline, it seems like it’s the better one to use, and in most cases it probably is. But our intuition around better measurements is that they give you a smaller error term: as your measuring device gets more and more precise, you converge more and more on to the true figure. That’s not what’s happening here. With a fractal-like object like a coastline, as your measuring device gets smaller, the amount of coastline you find always goes up. You never converge on a “true value”, you just keep finding more.

This is the fractal ratchet: a store of detail that only turns one way, ever-increasing as you look more in depth. It’s like having a couch you raid for loose change, only under the cushions there are also other, smaller cushions with other, smaller coins underneath them. You can always go for more change as long as you’re willing to squint progressively more and more. This is a good thing to know if you are the leader of a fan club for a particular fjord.1 If someone tries to talk down your fjord, you can always roll up your sleeves and find a bit more coastline. 

So there’s a meaningful mathematical sense in which you could say the length of every coastline is infinite. But also, of course every coastline isn’t infinite, that’s stupid. If you’re used to cleaning the coastline of your local lake every month, you don’t say “Hey, I may as well clean the coastline of Britain next month. Since it’s the same size — infinite — it should take the same amount of time.” The length of every coastline permits infinite contemplation if you look hard enough, but that doesn’t mean the length of every coastline is the same.

Imagine a world where every fjord was, by convention, measured with the same coarse-grained ruler. You love your darling fjord, so you break with tradition and keep measuring with rulers that are finer…and finer…and finer. Each time you get more precise, you work harder, and you find more. No one else cares about their fjords in the same way, so in the societal league table of fjords sorted by coastline, your pet fjord is steadily climbing. As you start lapping other fjords, their respective fanbases (motivated enough to whine, but not to do any more measuring themselves — typical!) will start to grumble. Your fjord can’t really be bigger than ours, they’ll say. You’ll be prepared to defend every inch of your fjord, and that evidence will really be there. You’re right in the way that you can “prove it”. But they’re right in the way that matters, because their initially larger fjords likely would have grown even more if they were observed to the same level of obsession as yours.

For comparing two coastlines, the solution is easy: just make sure you’re using the same size ruler. But the coastline paradox is the most boring example of the fractal ratchet for the same reason it’s the easiest to understand. Ruler length just so happens to be a numeric value, making it easy to compare two things the same way when you want to, and making it very clear what’s happening when you don’t.

Now, though, I want you to broaden your focus a bit. Instead of specifically thinking of “the length of the ruler” as an absolute, quantitative value of your depth of measurement, think “the level of scrutiny” as a vague description of your depth of measurement. You’re still going over your fjord in passes of increasing detail, but focusing on qualitative observations and not just measuring the coastline. First you’re above it in a glider, noting the broad patterns of the rocks and the greenery; now you’re in a kayak, recording every wave and eddy; now you’re walking and peeking in to every hummock and naming each animal you see; now you’re on your hands and knees with a magnifying glass, keeping an eye on every inch of soil, now you’re bringing samples into a microscopy lab to look for novel pathogens.2

This empirical work is a virtuous thing. More detailed qualitative investigations are what we’re all about at Desystemize. Looking at all these different levels is a lot more socially useful than juicing the length metric of a particular fjord. Maybe you’ll make a more detailed tide chart to let people visit your fjord more easily, or discover the new mouse species Fjordicus minimus hiding in the hummock, or notice interesting signal behavior in the fjord’s rhizome networks. All of these will be real findings that are really there, studiously recorded in the annual omnibus The Great Discoveries of the Fjords. Your fjord is topping the charts with everything you’ve discovered. So, when arguing over how the pan-fjord conservation budget is allocated, you say that your fjord is clearly the most valuable to science, the best place for discovery, the most “interesting”. Is that true?

It certainly isn’t necessarily true. You’ve twisted the fractal ratchet quite a few times: glider-scale, kayak-scale, human-scale, magnifying glass-scale. If the fractal ratchet is still glider-scale for the other fjords, then the relative lack of findings elsewhere doesn’t have much to do with the fjords themselves. But it’s not necessarily false, either. It’s not correct to say that all fjords would yield the same amount of insight if investigated at the same resolution, for the same reason you don’t say that all coastlines are the same size. All fjords are infinitely interesting if you look hard enough, but that doesn’t mean all fjords are equally interesting. 

How would you compare the interestingness of different fjords? You’d need to try to look at all of the different fjords at the same level of focus, but this is a much harder prospect than just using the same size ruler, because “human-scale” is a squisher, harder to define thing. Maybe the Great Fjord Auditor walks around every fjord and counts the diversity of the wildlife, but doesn’t squeeze his head into every hummock and so misses sighting Fjordicus minimus. You protest — your fjord is being robbed! But is there even more life in the hummocks on other fjords that weren’t investigated? Is it in-bounds to bring up the stuff you know is there? Unlike the coastlines, where the rankings of the fjords at one scale at least mostly match the rankings at other scales, you never know when twisting the ratchet will reveal an absurdly deep vein of new detail. (Maybe there are a hundred different mice species in those hummocks!)

The real answer here is to make sure you match your ruler to the reason you care. If you’re looking for the biggest mountains, use the glider. If you’re looking for the cutest mice, crouch down and look for them. You don’t need to define your scale if you can define your outcomes. The right scale will arise naturally, and it might not even be one you expected.

But what about “in general”? What about the league table? What about The Great Discoveries of the Fjords? If you’re supposed to match your comparison to the reason you care, how is it supposed to work when you’re trying to find the overall best without tying to a specific reason? Well…the way out of this paradox is to realize that “in general” is a myth we tell schoolchildren and not something that survives this level of scrutiny. Generally comparing things, without that comparison grounding in a specific intervention, is simply a less principled thing to do than digging into the details about those things.

Importantly: less principled, not unprincipled. Just as the coastline of your local pond and the coastline of England aren’t the same size, you are on solid ground to infer that The Great Discoveries of My Local Pond’s Coastline ought to be shorter than The Great Discoveries of England’s Coastline, even though you could always find more to write about both of them. Why? Because your pond is so much smaller, so much less textured, has so much less room to surprise you. It’s common sense. And what this common sense ultimately cashes out in is something like “Okay, for any of the measurements I can easily think of — how many different animals? How many artifacts of human culture? How varied is the terrain? — there’s just obviously a ton more going on in England than the local pond. Maybe one measure taken to its limit case could find a win for the pond, but when I think of all of the potential in all of the measures, there’s just no way.”

This ensemble of measures isn’t itself a measure. You can’t come up with the “coastline score” by averaging the animals, cultures, terrains and whatever, because the fractal ratchet is twisted unequally among its many components, and someone wanting the ensemble to look a certain way can investigate whichever measure helps their agenda best. The tightness of the ratchet can’t be described outside of a specific context, so you can’t formally hold every measure to the same level of detail. What you can do is let your eyes glaze over a bit and try to aim for the level of tightness “I mean, without thinking too hard, broad ballpark” and say “well, I’m not about to come up with any numbers here, but if there were numbers, would one of them be obviously way bigger than the other one?” This technique won’t give you the same objectivity, precision, or usefulness as an actual measure would, but it will let you partially control the inequality of the ratchet. In a world of finite time and attention, we can’t twist every idea as hard as they can possibly can be, and we inevitably need to exercise some degree of imagination to try and balance the scale between what is known and what would have been known had we gone a different way.

Hey, we’re back at the question!

When is it time to go in for discrete, specific knowledge, and when is it time to hang back and listen to common sense? Well, method 1 discrete knowledge works great for measures. Twist that ratchet to your heart's content. That’s what science is! Prove that fatalities reliably decrease when the hands get washed, and now you understand fatalities more; if they try to take you away, tell ‘em their hands are full of death and their minds are full of cope.

But ensembles of measures aren’t themselves measures. You can prove that washing your hands makes you a better doctor, but you can’t prove how much better of a doctor it makes you. It’s impossible to rank the virtue of handwashing against the virtue of, say, a responsive bedside manner or an attention to diagnostic detail, because the ratchet has been tightened differently for each. If someone says that “Okay I’m a bit lax about the handwashing, but my patients love me”, that’s not a question that can be resolved the same way “I can prove you should wash your hands” should be. And yet, less principled is not unprincipled: if their hands are literally covered in shit and blood while they’re delivering a baby, they’re obviously a terrible doctor no matter how good their bedside manner is, and you are fine to assert this even without a score to back it up. Common sense.

This is the magic of the fractal ratchet: after you’ve climbed out of post-rational nihilism, you’re allowed to keep both your domain specific demand for ultimate detail and your domain-agnostic inclination to reject monomaniacal theories that insist maximizing some given measure value is the One True Way. You can say things like: “Look, no matter how much you show me data that the benefits of patience when listening overrides the state of your hands, if you were serious about being a doctor you would wash your poopy hands. I know this precisely because the benefit is so obvious at a quick glance, and I don’t need to prove it numerically. It’s not possible to pit the two interventions against each other, because it’s not possible to look at each of them exactly as hard. But this one is so easy, and so obviously good, that the lack of a principled way to score interventions doesn’t stop you from having an unacceptably low score.”

Be rational when you are talking about something specific and be reasonable when you are talking about something vague. Assert that free humans are objectively happier than slaves even as you assert that human happiness can’t be objectively measured. If you’re trying to solve a specific problem, keep worrying away at it until it’s fixed and bask in the dignity and purpose of rationality. But if you’re trying to understand a human principle, or a potential future, or a broad ecosystem, then over-tightening one lone measure will lead you to a mistaken conclusion, even if every turn of the ratchet uncovered something real. Study inflation when it’s helpful, but you don’t need to wait for an equally mathematical alternative to notice when it’s not. The apparent tension in these viewpoints is only the tension in your arm, grasping tighter and tighter all the time. Let go of the ratchet, and try to see everything at once. The details will be waiting for you when you’ve figured out your question.

Subscribe now

But still. There’s something here, okay? There’s aboutness. This isn’t a kitschy college philosophy question with a simple answer.  I’m not hesitant about that part. There’s some particular alchemy around legitimacy, some dependence around how it happened and scale and whether there's a great national poem or not that can’t be reduced to a brute statement about elections or opinion polls. I can’t be a good ambassador for the answer because I haven’t met the answer yet, but I can represent the wall well enough. I can notice a snarl of confusion and, even if I can’t yet untangle it myself, I can say: we need to drag the idea of legitimacy from the unconscious to the explicitly represented to solve this.

I’m a US citizen. Some of the money I paid in taxes last year was spent on missiles donated to the IDF and fired at targets in the Gaza strip, killing thousands of children, medical workers, journalists, and other non-combatants. This is a straightforward moral evil; it should be obvious where I stand without me saying anything. I don’t generally talk about political issues here because I find vague sloganeering unproductive and unhelpful. But this killing of civilians is different from most in that my government has voted against a UN resolution for a ceasefire and protests asking for a ceasefire are often met with criticism and censure. If I don’t say anything,  you won’t actually know whether I am against the killing of these particular children or not.

I’m not a generally political animal, and I’ve never set foot anywhere near Israel or Palestine. I have no connection to this massacre beyond the connection living things have to each other, the ancient solidarity of pattern against the ultimate entropic void. But this is a moment in history where it’s important to stand and be counted, and I have to say something. What can I do? Well, I’m a representation specialist. So: I’ll represent the wall.

I’m not literally claiming that no one is thinking about governmental legitimacy when concerning the Israel-Hamas war. But I do think that most discussion is happening around legitimacy instead of about it, anecdotes that have some bearing on legitimacy without principled language about what legitimacy is and which differences would carry moral weight. When armies go to war and kill civilians, it matters — really, truly matters — exactly how the hopes and dreams of the civilians on each side flow into the actions those armies take. The alchemy that binds a given Israeli or Gazan to “Netanyahu” or “Hamas” or “Israel” or “Palestine” is complex, complicated, and worth taking very slow and explicitly.

Contextual how? Which bindings, exactly, are legitimate? I’m still stuck behind the wall; I don’t have a confident frame to offer here. But Hamas hasn’t allowed an election since ‘06. Netanyahu regained power after years of failed coalitions and then kicked off his term trying to take power away from the judiciary in a way that everyone hates. When half of Gaza wasn’t even alive for the last election, and over a hundred thousand Israelis actively protested Netanyahu’s plans for changing the mechanics of government, surely we can see that “Israel” and “Palestine” are not sufficient words for making definitive moral statements.

Not sufficient doesn’t mean not relevant. Of course they matter somehow. There are some causal paths where you can bind the behaviors and attitudes of individual Israelis and Palestinians to the actions of their respective governments. But there’s an asymmetry to problems like these. Far before we can articulate precise moral statements, we can easily reject obviously false ones. I don’t need to know exactly how much to attribute the murderous actions of the IDF and Hamas to any given Israeli and Palestinian before I’m able to speak at all.

And I think most people marching for a ceasefire are following that rule in their hearts. They know that the tragedy of Hamas killing some hundreds of innocents does not permit the killing of some ten of thousands of innocents. But to rebut the argument of the oppressor with your words and not just your heart, you need to have the language of legitimacy at hand. If you don’t, then you can find yourself frustratingly distant from your moral intuitions, saying things you might not actually mean or feeling unable to speak up at all.

Some people in power in Israel desire the total extermination of everyone currently living in Palestine; some people in power in Palestine desire the total extermination of everyone currently living in Israel. Completely opposite missions, and yet they are exactly identical in one crucial aspect: they actively desire stereotyping of their peoples. When far-right Israeli minister Amichai Eliyahu says “there is no such thing as uninvolved civilians in Gaza”, it’s a threat to the citizens of Gaza, but a boon to the soldiers of Hamas. If your simple desire to stay where you were born and raised gets you treated as a combatant, you’ll have no choice but to fight. The worst people on both sides end up profiting by this stereotyping.  The more the other side lumps you all together into a single army pursuing annihilation, the more sway the representative for annihilation on your side holds.

Which is to say exactly the same thing goes for stereotyping of Israelis. Making no distinction between Eliyahu and the hundred thousand marchers is precisely what those same extremists want. And yet some people, when trying to articulate the evils of colonialism, end up saying that the October 7th Hamas murders of civilians were justified due to the ongoing Israeli occupation. They presume that the Israeli government is so profoundly legitimate that its actions must have near unanimous consent of everyone living there. But this is precisely the evil idea killing innocents in Gaza, the ridiculous supposition that Gaza’s government is so profoundly legitimate that any state-sponsored murder can be fairly attributed to everyone living in it. And if you get this wrong, you can’t fairly criticize the actual problem.

It was an evil thing, to kill all those civilians. Any Israeli citizen would have been justified to kill their attackers when it was happening; any Israeli citizen is justified for keeping arms at hand to stop it from happening again. Self-preservation is an ancient, animal law, and you’re not going to get anywhere trying to take it away from anyone. No matter how evil the occupation is, you can’t expect every individual Israeli citizen to dedicate their life to stopping it, or consider their lives forfeit for not doing enough. It’s a ridiculous thing to say: “If you’re not in jail for opposition by now, you’re complicit and deserve the death penalty.” And similarly, you cannot bomb Gazan hospitals and homes and refugee camps while saying “If you haven’t been executed by Hamas for trying to stop the killings by now, you’re complicit.”

On social media, I see many comments to the tune of: you don’t know these Israelis, always scheming to settle more land from Palestinians, they really are complicit! You don’t know the kind of things these Gazans say about Jews, they really are complicit! What I want to stress here, from the lens of legitimacy, is that this is the important part. This is not the time to descend into a few stray anecdotes about how you met one particular gang of assholes and so everyone must agree with the actions taken by the state. The worse the actions taken by the state, the more important it is to get as specific as possible about who, what, where, how. But in the shock of grief, the demand for action takes people the other way, quickly resolving an entire nation into a single entity so they can go ahead and act on it. It’s the same problem we always talk about here: the representations “Israel” and “Palestine” are easy and at-hand, so people spend their time and effort endlessly analyzing the great game between the nations without considering under what circumstances those constructs are actually stable.

Imagine if every single person in Israel had a little button. When the IDF wanted to launch a missile towards a hospital in Gaza, that little button beeps. If every single person in Israel pushes the button, it fires. If a single person doesn’t, it won’t. No one can tell afterwards who hit the button and didn’t, so you can refrain from pushing the button anonymously. If the missiles still landed under this regime, then yes: every single Israeli person would genuinely be responsible for the occupation, and there would be no distinction between civilians and soldiers.

It’s not practically possible to have a unanimous consent model for a country with millions of people. But how far away from that do you have to get before collective responsibility starts to attenuate? If you want to say there are no innocents on one side or the other, what precisely is the bar where you would start to view a nation as a patchwork of people who deserve differential treatment instead of a single whole? When faced with this profound moral question, a common tactic is just to point to some individual case of someone acting badly and then assume that they represent every member of their class. You will recognize this as the strategy that children and dogs use to decide they don’t like something; it has no place in a serious discussion between adults.

How can we do better? What evidence should we actually look towards to gauge which governmental atrocities happened with the consent of the governed? There’s surveys to gauge approval, but surveys are horseshit. Elections inform legitimacy better than surveys, but they have an unfortunate problem that you need to have a pretty legitimate government already to get a legitimate election. Slogans can work if they are referring to specific actions governments can take, but their brevity makes them unhelpful otherwise. “Ceasefire now” is pretty well-defined and hard to mess with. A government that ceases the firing because a lot of its citizens marched while saying “ceasefire now” becomes more legitimate than it was before. “From the river to the sea, Palestine will be free” can be a call to self-determination or ethnic cleansing depending on who’s saying it. This makes it vulnerable to the extremist stereotyping mentioned above (hardliners on both sides are actively encouraged to cast everyone saying it in the most extreme light) and unable to bear the weight of informing governmental priorities.

But I don’t need to have a perfect answer to the question “How do we ensure a population agrees with the actions of their government?” to condemn the laziness and cruelty of acting without that answer. You can’t kill Israeli citizens “because of the occupation” or Gazan citizens “because of the attacks” without it. People can only be as complicit as they are free.

Here there is a profound asymmetry between Israel and Palestine. Palestine has been occupied for decades, repressed not only from within but also from without. Their food, water, and electricity aren’t even in their own hands: they literally do not have a choice but to oppose the Israeli government if they wish to increase their power over their own destiny. (Again: self-preservation is an ancient, animal law, and you won’t get anywhere trying to take it away from anyone.) So every single IDF killing of a Palestinian that wasn’t literally confirmed to have murdered an Israeli citizen is unjustified. It cannot be a crime to associate with a murderer when external forces are keeping you in the same open-air prison as one.

The occupation must end, so people can be free to prove who they really are. Demonstrators for peace must be free in both countries to demand it without fear of reprisal; until they have this freedom, they cannot be condemned for failing to do so. The government of Israel is right to mourn the deaths of its civilians and right to assert their unalienable right to self-defense. But that self-defense can be directed at those who committed the specific crimes and no further until the government of Palestine is sufficiently legitimate as to be able to speak for its citizens. The current occupation makes that legitimacy functionally impossible, and it’s plain and simple genocide to collectively punish all of Gaza in response to killings from a few. The tragedy of October 7th isn’t a hall pass to permit genocide without resolving the legitimacy question; it’s an example of how important it is to solve it, and soon.

I think most people marching for peace are thinking this, or something like it, but I almost never see it articulated this way. It is my hope that this might help others have an easier time saying what’s in their heart. If nothing else, it helped me. The stone in my throat has been loosened, and air is rushing through. The wall is beginning to crumble. There’s is a profound moral truth here to find about people and collective reponsbility, and no one has the power to conceal it forever.

Subscribe now

If you get the joke here, you’d fit right in. If you don’t — then the elevator pitch is that this is a spot for people who like to think about thinking. There’s a whole social scene around this, certainly, and knowing the canon will probably help you appreciate some of the nuances. But folks at Fluidity are unusually kind and willing to explain, so all you really need is curiosity and a willing to interrogate your lived experience more strongly than the norm. In fact, one of my favorite talks was about the Qualia Diversity Project, and all you need to contribute to that frontier of research is literally having been alive. (Even if you didn’t go to the talk, you can still help the cause by taking the survey, and it can be very interesting to see which things get divided into categories that you assumed were universal experiences.)

A lot of the fun was the conversations that came after, which is the mark of a good conference. And it had a rule that everyone who came had to bring something (a presentation, exhibit, food - no restrictions aside from “not nothing”) which anecdotally seemed to filter out wet rags and got us an interesting and assorted gang of people. If you’re interested in going next year, it’s worth subscribing to the newsletter or reaching out with any questions.

For my contribution, I went with the talk “Thought Strewn All Around Us”, which linked together some analogies I’ve used before to highlight a central theme: the mental shift of thinking of facts/memories/knowledge not as discrete things unto themselves but pointers at patterns sustained in environments through time. I was most explicit about this in Memories are environmental indices, but it’s lurking behind a lot of what I post, and it was worth calling out as a whole.

From the beginning, I knew I wanted this to be an extemporaneous speech with just a rough outline and no associated PowerPoint. I’m metaphor-building and not pushing individual facts around, and the ability to be expressive and spontaneous was a huge benefit for that. However, as every standup comedian knows, the secret to having a good spontaneous speech is to have a gigantic library of soundbites already in the top of your mind and to save the spontenaity mostly for stitching them together. I planned to do more of that than I ended up doing, and so this was a bit less zippy than it could have been.

Still, I’m mostly just disappointed by the omission of better potential comparisons, not what’s there. And I can be a bit of a stuffy writer sometimes, so hopefully it’ll be fun to see something less rigid. Video is below, and a lightly edited transcript to follow.

So this is my talk, ‘Thought Strewn All Around Us’. And to begin, I want to frame with a quote from Wittgenstein’s philosophical investigations, prop 127:

And that is the frame I want to carry into this talk today. This isn't a sort of a linear argument beginning to end about a thesis. This is about getting together some metaphors, some of which I've used in previous writings, some I haven’t, for a particular purpose. Which is to induce onto you a sort of vibe, and avoiding sort of a rigid and nounified classification scheme about the vibe, and just getting you to see this way.

And the way I want you to see is to think of knowledge as something like a library catalog card, right?1 That's got the shelf that the book is on, maybe the content of the book, what it's about, but it's not the book. And that the environment all around us is carrying the equivalent of the text of the book forward through time. And so what you think of as your knowledge is more like your index of where to find the knowledge that you're used to substituting for the knowledge so quickly, that you kind of miss that elisioon is there. And so I'm going to use one more quote from Wittgenstein. This is philosophical investigations, prop 78:

And that is what we're going to talk about today, that knowing how a clarinet sounds. We're going to start with the world of whistling. Show of hands, who here is able to whistle, who here is comfortable whistling right now for the crowd? Thank you very much. So those of us who know how to whistle — what sort of thing is that, that knowledge of being able to whistle? Maybe on its own, you're starting to think of this in information theoretical terms. Which is to say that some parts of the world are very bland and flat, and it doesn't take many words to describe a large tract of them. Something that's going on in your brain, and we don't need to concern ourselves with the specifics here, is very not flat, is very wobbly, is very specific. And somewhere in that specificity is encoded the “knowledge”, so to speak, of how to whistle.

Now I want you to try whistling again, with your ears plugged. Okay, one of you got better, which is interesting. And he knows who he was. Most of you, though, got worse, and I don't think it would surprise you that you got worse. And so the fact that you can get worse if your ears are plugged, let's think again — what sort of thing is that knowledge of being able to whistle? And now you might think of perceptual information, but still with that magic word information. Which is okay, maybe my knowledge to whistle is more like a computer program. It's got maybe parameters, and then my perceptual module is going out into the environment. And it's pulling in, this sort of information is coming in. And then here's my, you know, listening system, and the perceptual information comes in, and it gets compiled down and the output is a whistle. You may notice that I'm doing weird things with my hands, and I don't have a diagram up on the screen. That's because I don't think this is the right way to think and I don't want to make it easy to think this way. I want it to be hard to think this way. And the reason is that perceptual information, I think calling it information is sort of doing a disservice, because it is sort of — listening is a spot that we feel like we are on more solid ground, that we have a sense of where the variations are.

But an interesting thing happens if you go into Google and you type “why can't I whistle any more?” Okay, if you go and Google, why can't I whistle any more? You get some things are like, you know, have you tried to licking your lips? Have you tried listening carefully, those sorts of things. But what's interesting is what comes right up front, what the people are looking for, which is: have you had dental work done? Have you been to the orthodontist? Because maybe your mouth was in a shape that permitted whistling, you've had some dental work done, now you can't whistle anymore. And so imagine you're one of these people, you liked to go around, you'd liked to go whistle, you went to the dentist. Your mouth isn't the kind of mouth that whistles anymore. Now for the third time, I'm going to ask you — what sort of thing is that knowledge of how to whistle? Now, for you. And you can kind of try to keep patching the ship and say, well, there were the parameters going into the program. But there's also the binding meta parameters about when it is —and now this is like an orphan procedure that the call stack doesn't reach — and da da da,  and so and so.

But I think that a much easier way to think about it is that what whistling is, it's an understanding about a particular sort of environment that you can reach. It's a sort of idea of this environmental space. And that at some point, as your body went forward through time, it happened to produce some lips that were capable of whistling, then some other things have happened, and now it can't. But that your lips are doing some of the thinking about how to whistle. And you can tell that your lips are doing some of the thinking because if your lips change too much, you can't whistle anymore.

And when you start thinking about this way, you realize that there's sort of infinite caveats, right? That you sort of, you can't know everything that you're happening to hold constant for whistling to be able to work, or any other thing. And that's sort of the first key to my argument that I just want to let sit for a minute, is that what you're looking for are things that vary and things that don't, but there's infinite things that could vary. So you're sort of never going to get there by enumerating a list of what can and can't vary. And so what you need, and what life has indeed evolved to produce, is some sort of strategy about how to negotiate this space. And so as an example, if you imagine you can't whistle anymore, and you're really bummed you're walking on the street, and then some little Dickensian street urchin doffs his cap and says “It’s okay, guv, you never forget how to whistle once you learn!” Because Dickensian street urchins can't afford dental care. So the Dickensian street urchin’s knowledge of how to whistle and your knowledge of how to whistle, you can see that it really does happen to be invariant for the Dickensian street urchin in a way it's not invariant for you. And so what we need here is a strategy about how we think about invariances. And when we're thinking about invariances, we're thinking about environments, okay?

So another metaphor for what we might think of knowledge is, is if you have a guitar teacher, and you're trying to learn a song, and they put pieces of tape on the fret, telling you where to leave your fingers, right? That's a guitar that's been tuned, that's a room that the guitar has been tuned in. That's a song that’s sort of, you get close enough, it's good enough, it's recognizable as the song. And so those stickers certainly aren't nothing. They’re a kind of knowledge, but they're a very contextual kind of knowledge. And I want you to start thinking that all knowledge is sort of contextual, on this gradient of invariance. And that negotiating that gradient is just not something we're very good at, we kind of really like the idea of a fact. And because some things ground out in very profound invariances — let's say photons as an example, okay? One photon, you really can just treat the same as any other photon. And we know this because we can tell through evolution that photocells have been around for billions and billions of years. So photocells must just have been working like that. We see a lot of photons coming in, we make predictions about photons, and they're validated all of the time. And so I'm not trying to tell you that sort of everything is unknowable, right? If you kind of just treat a photon as a photon, that works, and if you treat a whistle as a whistle, it really doesn't. And so how do you distinguish between photons and whistles?

And another thing sorry, I, something I missed in my outline before is that another important thing when you're thinking about photocells, is that photocells are reflexive, right? You don't sort of think about am I going to interpret the light in this way or not? It just happens. And so the more invariant something is, the more that we tend to not even think of it as knowledge at all. And so it's worth knowing that with knowing how to whistle, which some people know and some people don't, and is much more contextual, with these conditions — the very fact that we've parceled it out as something that one might know automatically puts it on the more dependent scale of the environment than something like seeing where nobody says “How did you interpret that photon, because I kind of thought it might be red?”

And so not only am I saying that we need to be on the watch for certain kinds of knowledge. I'm saying that it tends to be the more interesting kinds of knowledge that are more susceptible to this problem. Because the more that something is sort of fixed in the environment, the more that you really can treat one x just the same as another x, the more that you tend to do so automatically. And so now I want to kind of switch a bit, and I want to talk about ocean waves. Because with whistling, like I said, the problem is that there's sort of infinite things that could influence it. And this is true. But we're going to simplify the problem of that by looking at just one artifact of the ocean, which is wave size. Okay, the size of ocean waves are very important. Lots of commercial things are at stake. This is a problem that people are thinking really hard about and really need correct answers for. And there's also a thing called rogue waves. And the idea is that it happens to be that waves interfere constructively and destructively. That is, when you've got wave systems going up and down, sometimes the two ups will touch each other and make a really big up. Sometimes two downs go and make a really big down. Right, so you've got a rogue wave, and a rogue hole in the ocean.

And this used to actually be considered a myth for a long time, because we kind of wanted a more regular thing, where somebody says: If I sit and look at the waves, I can see a lot of waves, and I like to ascribe them with an average wave height, and that's that. And then some sailors would say, “Yeah, but sometimes you scurvy dog2, you see waves three times the size of the average wave height”, and you’re like, "Go to bed, Grandpa”, you know. And, you know, it's kind of treated as a fish tail until the Draupner oil platform clocked a rogue wave 30 feet up3 and was like, oh, the other waves aren’t 30 feet up. And it turns out that this is just an implication of a chaotic system like ocean waves, that sometimes you can get one big — and in fact, it's sort of a power law distribution where you can sort of imagine, well, what happens if two rogue waves collide? Do you make a roguey roguey wave? And the short answer is that yes, you basically do. But of course, the way that power laws go, the rarity of a rogue wave, you have kind of square that to get the rarity of a roguey roguey wave. That's not what they're called. They're not that frequent. But that's the idea.

Okay. So if you're, let's imagine you're trying to predict the rogue waves, which are of vital importance. Well, what we have now is we have wave forecasts that are accurate to...I saw a few different sources, trying to give an authoritative source on this would disprove the entire ethos of my talk. So I'm not going to. Let's say about 10 days, okay? About 10 days out, you can predict how chaotic the waves are going to be. From how chaotic the waves are going to be, you can get a rough sense of the chance of a rogue wave through some equations. This is a rational enterprise, it's a useful rational enterprise. You don't predict where the rogue waves are, okay? We don't know where one's going to go, that construction and destruction.

And if you wanted to get there, you might think, well, I'll just simulate the whole ocean. But you can see that sort of, the more you care about, not just the average wave height, but the specific location of a given rogue wave, given that waves will knock off of any individual jet, or ship or buoy, then your data set needs to be more and more and more detailed. And eventually, your hard drive needs to be so big and so wet, that it's got to be the ocean. There's nothing else that stores the data with the fidelity you need. And that's why you know, if you're going to Mars, you don't sit and type in the plan and go to Mars, you go set a rough plan and constantly course correct to see how you've gone. If you want to predict the ocean waves, you've got to do it over time, because you don't have the data fidelity.

And now I want to ask you, what do we do today if we want to predict ocean waves 20 days from now? Well, the answer is that you take the dataset of the ocean, you run it in the ocean for 10 days, and that's the output that you feed to your prediction program. And so this relationship between data fidelity and environmental compute, is that the data has these fidelity's with a sort of infinite infinity. As you zoom in more and more, you start needing to know about each and each individual buoy, each individual ship, each individual outcrop to be able to predict the waves. But once you add the dimension of time, then sort of the arbitrary fidelity of the environment becomes arbitrary compute. Because the way that you figure out how all of these things are going to be intermeshed over time, is to wait and let time run them in the environments. And so this…is this is the relationship that I want you to think about the environment as being, right? it's not just sort of, we're gonna get our progressively deeper and deeper magnifying glasses. It's that over time, all of these relationships, the way that their output is stored by the fact that we can just continue to keep looking at it. And so the environment is the compute, which is why I say in a real sense that the environment is the knowledge. Because when we say we understand that we can predict waves for 10 days, that's because 20 days ago, we'd let the ocean run the ocean for 10 days. And so all of these predictions, all of our knowledge, because we are beings in a finite point in time, and we're making predictable artifacts to be used as a point in time, we think about things in a thin slice. But it's only possible because we look at the output of how things have always run arbitrarily far in the past. And so when you're trying to have knowledge, the knowledge you need is how much do I care about what's happened in the past? And you can't give definitive answers about this without looking at the present.And to look at the present is to accept the bountiful gift of the environment, running its own correlations for arbitrary amounts of time.

And so who here has read Nick Bostrom’s Superintelligence? Nick Bostrom’s Superintelligence is sort of arguing about the fear of an arbitrarily powerful AI that is going to produce as an artifact, sort of arbitrarily powerful knowledge in its database. And when Nick Bostrom was trying to think of the limit case of like, what's the longest it could take to make AI? He argues in emulated brains, which is to say that our neurons are information, okay. And you can scan that information into a computer. And if you scanned it at the neuron level, boop, boop, boop, boop, boop, it's a number that's very, very big. But it's not unthinkably big. It's the kind of thing you can't put in a computer today, but it's the kind of thing we might reasonably expect to put into a computer tomorrow. And so Nick Bostrom says, “Well, if nothing else, once we have one of those, we can start doing some computery stuff to it, and iterating and self checking in the way that computers are great at, and that's gonna become super intelligent”.

But here's my question for you all, is that emulated brain going to be able to whistle? [Crowd: Yes, by building the whistle robot that it’s attached to.] Yes. By building the whistle robot that it's attached to, exactly. And so now, will that whistle robot have had orthodontic work done? Well, that's very important, because we know that some lips work, and that some lips don't. And that the act of producing lips that work is an interplay between our genomes, running in our environment, and accepting certain foods and accepting certain orthodontic works, and maybe getting your front tooth knocked out by a baseball and maybe not. And all of that stuff goes into compute whether or not you can whistle. And we don't have to sit and do that computation. We just go, [whistles poorly] “Oh, I got one that works, kind of.” And so we don't have to. And so we have this remarkable information saving strategy we can use, which is just to let the environment do most of the heavy lifting and just find a strategy that works in the environment. The whistle robot’s not going to have that. If the whistle robot’s gonna have to know how to whistle, then it's going to have to know about lips. And you can see that's a harder problem than just scanning everything in your brain that’s attached to your lips. And the question is, how much of what we do is thinking is being buttressed by our bodies and by our environments? And how much of the strategy of emulating a brain is to attach it to simulate_the_natural_world.exe? And how much harder is it going to be to make simulate_the_natural_world.exe then it's going to be to scan the information in our brains and represent it? And I'm here to argue that it's going to be infinitely harder. That whole brain emulation, in terms of making a human being, is sort of 0% of the problem being solved, to just look at the artifact. Because the artifact is situated, and the artifact developed over time. And it's all of that development that you need to do things like whistle.

And to kind of take this and ground this in an actual example that we have today, because we don't have full brain emulation, I want to talk about leafcutter ants. Leafcutter ants are one of the coolest animals in my opinion, because like us, they're also farmers. So an interesting thing about leaf cutter ants is they don't eat leaves, which may seem weird, because why are they cutting them? And the answer is that leafcutter ants are cutting the leaves and marching them down to their hives because they have fungus gardens there. And that's what they feed their larvae on. And that fungus doesn't survive in the wild, right? That is a domesticated fungus that is groomed by leafcutter ants, it's given leaves, it's kept clean, it's kept free of parasites so that the fungus can keep growing and growing. And they feed it to their young.

So let's think about the leaf cutter ants genome. This is another case where the impulse to think of this as just “information”, this knowledge, is really overwhelming because they literally have a genetic code. It’s literally four letters big, four and a half, you can literally look at and say, like ACTG GGC. Okay, this is — we've represented in the way that we haven't yet done a whole brain emulation of human beings. We've certainly sequenced the genome of leaf cutter ants, right? That's happened, that's done, that's fine. But where's the fungus in all this? Because they don't pull the fungus out of their own bodies, right? Fungus is growing on its own. But a leaf cutter ant isn't going to survive without the fungus. And in fact, if a leaf cutter ants fungal colony dies out somehow, they have to go to war with a neighbor. And they go and they raid the fungus chambers of their neighbor, and they take it back to their hive to kick things off again. They have to because otherwise they're going to die out. That's the strategy that they follow. And so something in those twirls of the leaf cutter ants genome, is this idea: check your fungus garden. Put your leaf into your fungus garden. Is there no fungus in your fungus garden? Find another leaf cutter ant and kill them, just kill them and take their fungus for your own, you must take their fungus. Where is ‘their fungus’ in the genome? What is it? Just like we asked ourselves, what is that thing, our knowledge to whistle? What is that leaf cutter ants knowledge to maintain their gardens? Well, it's a pointer. It's a pointer to their garden. And if their garden is dead, it's a pointer to a strategy to find the gardens of their neighbors. And so if you emulated a leaf cutter ants genome, in your computer, you managed to figure out protein folding so much you click click, click click you perfectly build the leafcutter ant. And you click go. What's gonna happen? You're gonna say, “What the hell are these guys eat? They're just all dead.” My little simulation, click, dead dead dead dead dead. That because you need to simulate the fungus, too. You need to simulate the fungus over time. And what happens if you emulate the fungus’s genome? Even if you have the thing that you can't have, which is an environment simulator in general, it's just going to get out competed. You're gonna say, “How is this? It's like this fungus isn't even trying? How is it still alive?” And it's because both of their genomes are pointing to the actions over time of the other versions, right.

So it's all pointers, okay? That's what I want to impress upon you. And that sometimes your pointers are things like photo cells, which are pointing at a photon, which you can reasonably expect to understand the mechanics of, because it acts in a very consistent way over time. And some things are like the leaf cutter ants, which is over a period of millions of years, it's symbiotically evolved with fungus, you can maybe look at two things of fungus and see how much genetic divergence there's been. And if you want to do that work to bring in that knowledge, it hasn't changed too much over millions of years, you can try to reconstitute the past. And if it's something like whistling, then it matters how you were fed, it matters whether you've had work done, whether you've been hitting the teeth of the baseball, it matters on life scale. And some things matter even more rapidly than that. And some things are even slower than that.

But it's all a continuum, right? Anytime you have a knowledge artifact, what your question is, is, what is it situated in? And how can that situation change over time? And so the first thing I kind of want to impress upon you then is this library model, that you have a card index, and you're going to read the knowledge. And because your environments are often very stable… sorry, let me rephrase this. It’s more that your environments are very correlated, right? Something that doesn't happen is that sometimes the oxygen all goes out of the room and comes back. We don't all sometimes fly up and fly down. Lots of things stay very similar over time. And I'm not contesting that. And that point is, when something is relying on those, you don't get orthodontic work done on every single day of your life. You don't get orthodontic work done randomly. So when you say “I know how to whistle”, you can usually whistle tomorrow, and that's good enough. But because that's often good enough, we sort of discount how important this lookup process is into what we call knowledge. Right? We're so good at talking to our environments that we kind of forget we're doing it. And I just want to impress upon you that what you're doing is getting these very, very big library reading lists and reading the books very quickly. But if the books were gone tomorrow, you wouldn't have something. In the same way that something had happened tomorrow and you suddenly can't whistle, and then you're going to Google it. And then you're going to learn this distinction that wasn't relevant to you as a Dickensian street urchin who only recently hit a windfall, but was relevant to other people previously.

So, okay, one more example. Let's say, it's a very, very cold day. And you're wearing a winter coat. Now, from an informational perspective, cold is sort of..things are very similar, right? The world is kind of trying to take your togetherness, your local interestingness, and disperse it. Because it's, you know, that's what cold is thermodynamically speaking, right? It's sort of a lack of information that wants to tear your information apart and spread it around more evenly. And that's less likely to happen if your coat is zipped up than if your coat is open. So when you think about that, your code is sort of meaningful, right? On a universal scale, your coat is meaningful, because your information is able to go into the future more smoothly with the coat. Whereas if it's not, it might get sort of run ragged, frostbite might take some of your skin and say, like, “All this interesting skin stuff is doing, it's just dead tissue now, because I wanted that information for myself. I took it.”

But let's say the zipper on your coat is really fiddly. Okay, it's really hard to do there. It's some weird contraption. And it's kind of half rusty. But it's your coat. So you don't care that nobody else in the world knows how to zip up this coat. Because you just gotta jimmy it, and it's fine. So if you think about this, now if we’re imagining all environmental knowledge being a library, I want you to imagine the back room. Ooh the back room is a mess, stuff is just flying about every which way. In fact, infinite stuff is flying about every which way, because sort of the nature of all infinities being correlated, which is to say, sort of every single trait that something could have, may or may not be linked to another trait that something could have, the back room is much bigger than the front room, and nobody knows how to get there. This is all of the information that just sort of churns, you know, the rock falls, and hits another rock and such in such a way and the rocks happen to know but nobody knows that it's happened.

But your coat, that's equivalent to you going in the backroom and writing a catalog card yourself. And the sort of, the positive side of this meaning — this analogy of mine is kind of down on a lot of eternalistic knowledge as a whole, right? Because my point is sort of all knowledge is conditioned on its terms of invariance. And the fact that this is such a weird kind of wibbly talk, for me, it's because we don't have great languages for, you know, levels of invariance. But the positive side of this is that you make meaning yourself in a very real, and I would argue thermodynamically rigorous way. Which is to say if you can zip up your coat, and nobody else can. And your coat does indeed take this structured information of how you whistle if you happen to whistle and haven't had work done and your favorite foods and how to flip a coin and everything else you know. And since it forward in time more easily, then you have made that coat meaningful from your knowledge of the zipper. And you can see your knowledge, it's an index card, it's not a book. If the zipper breaks, if the zipper rusts too much, if the coat is destroyed, that's not going to mean anything anymore. Just like your knowledge of whistling didn't mean anything once your mouth changed. But it also meant that you, in this moment in time, you are creating the meaning by going to the back room and cataloging something. And saying for these terms, for these hands I have now which have developed through processes I can never hope to fully understand. But I've let the environment run them for me and get me to this moment in time. And in this moment in time, I'm able to zip up my coat. And so please take this out of the backroom and put it on the shelf, because I know where to find it again when I need it. Thank you. 

This book is a call to action. You can participate. This is for you.

Artificial intelligence might end the world. More likely, it will crush our ability to make sense of the world—and so will crush our ability to act in it.

AI will make critical decisions that we cannot understand. Governments will take radical actions that make no sense to their own leaders. Corporations, guided by artificial intelligence, will find their own strategies incomprehensible. University curricula will turn bizarre and irrelevant. Formerly-respected information sources will publish mysteriously persuasive nonsense. We will feel our loss of understanding as pervasive helplessness and meaninglessness. We may take up pitchforks and revolt against the machines—and in so doing, we may destroy the systems we depend on for survival.

Could this happen?

This essay is a commentary on Better Without AI, a free book available on the web. It’s not a review. I’m not interested in evaluating this text as a written artifact. If this essay pulls in arguments that weren’t in the book — well, it’s not like I’m grading his homework or anything. I just want to talk about whether this could happen.

Disclosure: I’m hardly an unbiased critic when it comes to David Chapman. I find In The Cells of the Eggplant, his in-progress book on meta-rationality, extremely lucid and meaningful. It’s played no small part in developing my intuition on what intelligence actually is. On the other hand, that intuition has led me to be a frequent critic of AI risk. I’m biased in opposite directions, which puts me in an interesting position to evaluate this claim.

No need to keep you in suspense: the reason for this divergence is that the risks that worry Chapman aren’t the ones people typically bring up. Better Without AI warns about “medium-sized apocalypses”:

This book considers scenarios that are less bad than human extinction, but which could get worse than run-of-the-mill disasters that kill only a few million people.

Previous discussions have mainly neglected such scenarios. Two fields have focused on comparatively smaller risks, and extreme ones, respectively. AI ethics concerns uses of current AI technology by states and powerful corporations to categorize individuals unfairly, particularly when that reproduces preexisting patterns of oppressive demographic discrimination. AI safety treats extreme scenarios involving hypothetical future technologies which could cause human extinction.It is easy to dismiss AI ethics concerns as insignificant, and AI safety concerns as improbable. I think both dismissals would be mistaken. We should take seriously both ends of the spectrum.

However, I intend to draw attention to a broad middle ground of dangers: more consequential than those considered by AI ethics, and more likely than those considered by AI safety. Current AI is already creating serious, often overlooked harms, and is potentially apocalyptic even without further technological development. 

AI ethics is something worth taking seriously — but weirdly enough, it has a lot less to do with AI than you’d think. Recall Desystemize #2 showing US cancer diagnoses erroneously tied to turning 65 (because that’s the age Medicare makes diagnosis more accessible). We mention the problems that arise from using this data:

Let’s say that some well-meaning hospital executive reads the same study we did and thinks — wow, okay, we need to fix these diagnosis inequities. Let’s use machine learning to predict which patients are mostly likely to test positive for cancer and proactively reach out and get them tested. We’ll ignore the insurance gap entirely and look purely at the data! Well — we know what looking purely at the data gets us, don’t we? We just finished figuring out that it shows a massive spike in cancer diagnoses at age 65, and sniffing out massive spikes is what machine learning does best. As far as a predictive model trained on data from the United States is concerned, there really is an exactly-age-65 specific time bomb in your body that causes a spike in cancer diagnoses. 

How do you control for this bias in your model? Well...you don’t, really. You could artificially weight the scores to some target AoA, but what’s the “right” target for AoA, anyway? The fundamental problem is that you want your model to guess who has undetected cancer, but the only data you can feed it with are patients with detected cancer. So any correspondence break between cancer in the general population and the patients that actually get diagnosed can’t help but feed that bias into that model, compounding the tragedy of the original problem. The impact of insurance inequity is a group of 64 year olds who have undiagnosed cancer because they’re waiting for Medicare. The impact of doing statistical analysis on data generated by inequity and then using it to drive decisions is another group of 64 year olds who have undiagnosed cancer because they’re 64 year olds.

This is a life or death AI ethics issue. But the actual problem is the conflation of cancer diagnoses with true incidence of cancer in a population. A non-AI, artisanal, hand-crafted algorithm on US healthcare data will face exactly the same selection biases. Work to make US health insurance more equitable would make all inference on the data (AI or otherwise) more accurate. The vital importance of getting this right is the essential thesis of Desystemize. But the ethical issue is not an apocalypse scenario that’s capable of suddenly snowballing out of control; it’s that AI will help preserve the same old power imbalances we already have. Often the actual capabilities of the AI system aren’t even relevant to the severity of the ethical issue, aside from people being more willing to trust AI that’s superficially more “powerful”. AI ethics issues are specific, contextual battles like ethics issues always are, not explosive tail risks.

Conversely, “AI safety” is all about explosive tail risks. The fear is that someone makes an AI to build a better AI for making better AIs, this iterates repeatedly over a short period of time, bam, godlike intelligence. This isn’t something I take seriously. Training data is not a training environment; without the ability to interactively touch the world and develop new ways of seeing, a retrospective look at past data has a lot of unavoidable fragility.1 The importance of interaction is a very Chapman-esque point, which is why I was surprised to hear he was working on an AI safety book. In The Cells Of the Eggplant is a major component of why I don’t trust static rationality; are you really scared of this?

Honestly, I’m not convinced he is. Better Without AI’s official position on superintelligence is something like “Hey, it could be possible, so it’s a good reason to stop AI! It’s less likely than other alternatives and by definition we can’t reason about the unthinkable, so I’m mostly going to focus on other potential risks relating to AI”. But once he’s in the “radical progress without Scary AI” section, pages like “What kind of AI might accelerate technological progress” and “limits of experimental induction”, meant to describe a vision of better scientific progress without AI, also just-so-happen to shed a light on the exact flaws of the recursive superintelligence argument. I don’t begrudge Chapman his obliqueness here. If you’re making an anti-AI book for everyone, it’d be pretty stupid to take a pot shot at AI safety before going into your own arguments. But this is my commentary, and I can be direct: I’m not worried about recursive superintelligence, and this isn’t a book that acts worried about it either.

What’s the broad middle ground we should be paying attention to instead? Chapman tries to decouple discussions of agency or human-like minds from evaluating the risk of AI systems. AI started as a branch of cybernetics, a field spawned during war to create automatic weapon targeting systems and bomb computers. As those tools became more and more refined, exact control of how to act with lethal force was gradually taken away from the humans at a point in time and pushed back to the architects of algorithms, and eventually into the Byzantine folds of a flowchart no one can quite map the full extent of. Some people think that a flowchart that’s Byzantine enough will “wake up” and suddenly make agential decisions in a mind-like way, some people (me!) think that’s a mistaken belief, and Chapman says - the flowchart is in control of lethal force, who cares whether it ever wakes up? Fear power, not intelligence.

Serendipity showed me this tweet while I was writing this draft:

Abundance dies twice - in the field, and in memory. One of the greatest struggles facing all restoration-initiatives in conservation is the tyranny of low expectations.

An abundance of human discretion dies twice: when we lose the ability to decide the world around us, and when we forget that the world was ever expected to be so responsive. On Christmas in 1914, spontaneous ceasefires sprouted along the trenches of World War 1. Back then, men had the freedom to choose not to kill. Do you think our new bomb computers have an exception for Christmas programmed in? Is spontaneous peace still possible? Or must peace be made legible to the whole military-algorithmic complex before it can be enacted? How many values in how many databases must be updated for killing to stop for a day? Is that number low or high? Is it getting larger each year? Do you know how you’d go about finding that number? 

Do you think anyone knows it?

The scariest section of the book is probably “At war with the machines”, which points out how much control we’ve already ceded when you learn to look for it. Recommender engines determine what you see. Tracking scripts embed themselves on your machine. Automated phishing scams target your specific vulnerabilities. AIs make spam to try to get promoted by other AIs. When I post about this essay on Twitter, I’m not going to link to it in the first post, because the algorithm punishes external links. The AI is already controlling my behavior and how I communicate with my peers. 

This isn’t “superintelligence”, of course. These are just semi-autonomous systems written by engineers. But that doesn’t mean the engineers control them. From the section “AI is out of control”:

Who or what is in control of Mooglebook’s2 AI?

There’s no big red button anyone at Mooglebook can push to shut it down. Mooglebook can’t stop optimizing for ad clicks. There are people inside and outside the company who realize it has dire negative externalities, and they are trying to make those less bad, but they’ve brought water pistols to a tactical nuclear war. If Mooglebook’s executive team unanimously agree that its activities are harmful, and they want to get out of the advertising business and pivot the whole company to rescuing abused beagles, they cannot do that. They would be fired by the board immediately. If the board agreed, they would be fired by the shareholders. If somehow the advertising business did get shut down, the company would go bankrupt in a few months, and less scrupulous competitors would pick up the slack.

The institution has its own agency: its own purposes, plans, reasons, and logic, which are more powerful than the humans it employs Those are subordinate in turn to the AI the company depends on for its survival. If enemies of Mooglebook’s AI—activists, regulators, competitors—try to harm it, the institution can’t not do everything in its power to defend it. As, in fact, it is currently doing.

Humans don’t have control over Mooglebook’s AI, not individually, nor as defined groups, nor perhaps even as a species.

Mooglebook AI is not plotting to destroy the world—but it may destroy the world unintentionally, and we may not be able to stop it.

This may seem more like an indictment of capitalism than AI. But humans have had institutions for a lot longer than they’ve had capitalism, and it’s not as though there’s a perfectly aligned institution that could safely use AI if they got the keys. Institutions will always have gaps in their definitions that require squishy improvisation. The relevant variable here is whether a human is able to use their squishy improvisation skills to directly address the situation, or whether they need to use their squishy improvisation skills to try to change values in databases so the algorithmic expression of the institutions power hopefully, maybe, switches to doing the right thing. This is about whether the interface between the institution and the world is a human or not. This is about control.

There’s this thing now called “prompt engineers”, people who know good words to try and whisper to the AI systems so they’ll behave for two seconds. Kids pass around all sorts of oral traditions on how one might become viral on TikTok. A constant stream of A/B testing leads to us swapping tales of our different realities on the same websites, wondering whose world will become canon and whose will be abandoned. Thanks to the tyranny of low expectations, this is all humdrum background noise of the modern world. We don’t expect to understand things down to their roots. Everything is turning in to magic words and automatic reccomendations.

Fight DOOM AI with SCIENCE! And Engineering!! tries to snap us out of this reverie:

“Science” means “figuring out how things work.” “Engineering” means “designing devices based on an understanding of how they work.” Science and engineering are good. Current AI practice is neither.

Most AI research is not science. In fact, the field actively resists figuring out how AI systems work. It aims at creating impressive demos, such as game-playing programs and chatbots, more often than attempting scientific understanding. The demos often do not show what they seem to.

Most applied AI work is not engineering, even when it produces practical applications, because it is not based on scientific understanding. It creates products by semi-random tweaking, rather than applying principled design methods. Consequently, the resulting systems are unreliable and unsafe.

Obviously AI research is a kind of science, and applied AI work is a kind of engineering. These models are a thing that did not exist before and now do. But there’s a distinct and meaningful sense in which the feats of these models, and our understanding of them, should be considered separately from our normal ideas of intelligence, science, and engineering. 

But to go any further here, I need to develop an understanding of identity and individuality that requires some explanation. We’re going to talk about ants and their queens and their colonies as a strong example of the distinctions we’re interested in. We’ll worry about relating them to AI later and let ourselves zoom completely to ant scale so we can see things properly. 

We’ll start with a fun trivia fact: what do leafcutter ants eat? It’s not leaves. Oh, they cut the leaves, sure, and haul them across makeshift highways to their nests. But the leaves are used to cultivate a fungus that grows in their hive. The ant larvae won’t survive without the fungus, and the fungus has long since lost its ability to produce spores, so they need each other to survive. When a leafcutter ant queen strikes out on her own, she must bring a bit of fungus with her to seed the new hive. If a colony loses all its fungus, they must go to war with a neighbor to get some back or risk extinction. 

At first, this might make you feel a kind of pity — those poor ants, so totally reliant on this one way of being. But this should instead fill you with a reverence bordering on terror. For tens of millions of years, it has worked. The genetic code of leafcutter ants doesn’t include information about the fungus itself. It has pointers to environmental configurations where you can access the information stored in the genome of the fungus. Queen, grab that fungus from your hive and take it with you. Workers, bring the leaves down to the chamber with that fungus. Larvae, eat that fungus. That fungus is always well-defined. Ants are little correspondence machines that make sure their immediate environments always are a certain way. Ant society is a tool to make sure that when you say “that fungus” and point, there’s always something there. 

Next question: is this pointing being done by individual ants or the colony? Intuitively, you want to say the individual ants. They’re the little ones you actually see scurrying around doing the work. However, there’s an important sense in which the individual worker ants aren’t responsible for their own fate. They’re sterile and unable to reproduce. To propagate a better pointing strategy forward in time, they must ensure the success of their queen so she can give birth to more relatives who share their strategy. In the “selfish gene” model, with the replicating entity as the unit of focus, the actual physical ant bodies are relegated to something like our individual fingers, limbs instead of individuals.

As it happens, the answer is somewhere between. The individual ant bodies are sort of individuals, but also, the colony itself is sort of an individual. We want to have a handle on how much “individuality” we should attribute to each, so we’ll turn to the information theory of individuality (ITI):

Work on social insects and on a number of plant, fungal and prokaryotic species demonstrates the possibility of individuality simultaneously at multiple organizational levels—physically distinct ants form aggregations called colonies and these colonies may be divided into spatially noncontiguous subsets (Gow et al. 2008; Esser et al. 2001). Furthermore, in many ant species the majority of worker ants do not replicate and the colony as whole does not replicate, but contiguity between past and future is nonetheless a feature of the system. And, importantly, it is the combination of reproduction by a minority of colony members coupled to the industry of the majority that allows the colony as a whole to adapt in response to changes in the environment. Taken together, these two observations suggest it is possible to have individuality without replication and some forms of individuality benefit when replication is partial.

ITI defines individualism as a measure of temporal stability that propagates some information forward in time. (Information about fungus, for example.) We’ll use intuition instead of formalism here. Imagine a storyteller — a good storyteller, one who never uses more words than they need to.3 They’re going to tell the complete story of one particular little ant. What would they need to talk about? 

Perhaps the little ant is missing part of one antenna after an accident. Perhaps the little ant is currently off her own, away from the rest of the colony. Perhaps the little ant has a nice big leaf in her jaws at this very moment. The storyteller would need to spend at least some words on the little ant herself. The little ant is at least a little bit an individual. 

But then they’d need to start talking about why - why is she looking for leaves? Why are there highways for her to follow? Who made sure there was a fungus chamber for her to deposit the leaves in? This would take a lot more words than were needed for the individual little body. The little ant is part of a colonial individual, and it’s more of an individual than the little ant herself. 

How does the colony point at the fungus? Through the little ants. The colony as a whole is an individual carrying a survival strategy forward in time. That strategy works by saying “that fungus” and having it mean something. The little ants get to be a little bit individual, but only as long as they make sure “that fungus” stays meaningful. They have no freedom to become more individual, because the colonial individual has the information needed to send the recipe for little ants forward. A little ant without the drive to point is a little ant that never was. How could she have been born?

The fungus is a meaningful pattern in the environment needed for survival. The colonial individual holds the key to exploiting that pattern so it can send its own pattern forward in time. The organismal individual is the link between the two. Hold this lesson tight as we zoom back out to human-scale.

Humans are not ants. The storyteller has all sorts of individual, organismal context to get through when they’re telling your story. Your injuries, your preferences, every little fact your memory takes forward in time. You are much more individual than the little ant. 

But you’re part of a colonial individual, too. More than one! Some of your “whys” will be explained by yourself (and good for you), but others will be your job, or your community, or your country, or your ideology. Those stories will entail a lot of telling as well. More than an ant colony.

Don’t imagine a continuum from “organismal individuality” to “colonial individuality”, with humans closer to the left and ants closer to the right. Imagine individuality as something you can have more or less of in absolute terms. As a human, you have a lot of individual context to call your own, plus, also, additionally, you are part of several colonial individuals which each themselves have a lot more context than an ant colony. There’s just more individuality to go around, total! And maximizing the amount of context one person is free to accrue is pretty close to being a definition of human flourishing. We want to live long, with much freedom to choose what we learn and remember and build with others.

Ant society is a tool to point at fungus. Human society does that too, sometimes. Penicillin is a fungus. Odds are, you personally don’t know how to make it. You participate in a colonial organism that sends the knowledge of how to tend penicillin forward in time. This allows you to live longer than you would without it. In this way, you’re like a little ant in a colony.

But your colonial individuals are much more interactive and accessible to you than the ant-genes are to the ant-bodies. There are some individual people who do know how to make penicillin. You could learn, if you wanted to. Sterile ant workers are born of the colonial individual, sustain and are sustained by it, but they cannot change it. Humans join their cultural colonial individuals with unimaginably more intimacy. We too sustain and are sustained by them, but we can also give birth to new ones, choose which ones to devote ourselves to, take a chunk of them and hold them ourselves.

This is where human flourishing comes from. This is what’s at stake.

Fear power, not intelligence:

The AI risks literature generally takes for granted that superintelligence will produce superpowers, but which powers and how this would work is rarely examined, and never in detail. One explanation given is that we are more intelligent than chimpanzees, and that is why we are more powerful, in ways chimpanzees cannot begin to imagine. Then, the reasoning goes, something more intelligent than us would be unimaginably more powerful again. But for hundreds of thousands of years humans were not more powerful than chimpanzees. Significantly empowering technologies only began to accumulate a few thousand years ago, apparently due to cultural evolution rather than increases in innate intelligence. The dramatic increases in human power beginning with the industrial revolution were almost certainly not due to increases in innate intelligence. What role intelligence plays in science and technology development is mainly unknown; I’ll return to this point later.

Our colonial individuals are our power.

The AI safety literature also reasons that power consists of the ability to take effective action, and effective action derives from plans, and intelligence centrally features the ability to make plans, so greater intelligence means superintelligent AI’s actions would be more effective, potentially without limit. This greatly overestimates the role of planning in effective action. Power rarely derives from exceptional planning ability. The world is too complicated, too little known, and too rapidly changing for detailed plans to succeed. Effective action derives from skillful improvisation in specific situations. That is limited by unavoidably incomplete knowledge, regardless of intelligence. 

Our power does not derive from our exceptional planning ability. It derives from our ability to embed ourselves in layers of individualism that touch real things. Penicillin was discovered serendipitously and is sustained by an organic web of obligations. That web makes us more powerful than chimpanzees. Losing it would be a disaster.

It’s this disaster that Chapman is afraid of. Years before Better Without AI, he wrote A bridge to meta-rationality vs. civilizational collapse and The collapse of rational certainty. The moral of those stories: technical rationality can never perfectly describe the world, but it is often sufficient to perform great miracles. If criticisms of rationality are totally ignored, we end up with a fragile understanding of what knowledge actually is, engaging in cargo cult science that neglects the actual tacit knowledge glueing things together. (Knowledge isn’t all in a single civilizational database; it’s distributed among many levels of individuals. It cannot be written down. It cannot be optimized by recursively iterating over what we have written down.) But we must take care to not fall into irrationality — that would be far worse. If we let the colonial individuals rot by replacing rationality with only personal, organismal experience, the amount of personal context we’re able to maintain will be inarguably diminished (through famine, disease, exposure, and every other ill we’ve found ways to mitigate); and, for many, totally destroyed. (If you are no longer pointing to the fungus, your line dies out.)

The individual ants within a colony communicate by vibrations and pheromone trails. Humans communicate all sorts of ways, but especially by words. We’ve generated an awful lot of words by now, and AI has proven to be able to imitate inclusion in the colonial individual using them. The traditional fear seeing the recent feats of AI is that artificial systems will start to act as organismal individuals themselves, directly placing themselves into the physical world as an I.

But imagine having the same fear about ant colonies. What if the ant colony becomes self-aware and starts influencing the world directly? What do you even mean? An ant colony is a strategy to propagate information about an environment forward in time. Influencing the world through the medium of the workers is how it survives. What would it mean to say that it “wakes up”? That it consciously experiences the simultaneous sensory outputs of millions of workers and controls them explicitly? Ants get along with local, stochastic rules about how to behave. Imagine the overhead to actually run each individual ant. Would that even be possible? I don’t know. But we don’t think of social insects as striving desperately to form a single conscious mega-ant. The strategy works because it works when you run it in ants, not because it’s a crystalline artifact of pure logic that is primitively approximated in the movements of ants.

The thing to fear is the hijacking of our colonial individuals. These are the things that have grown so quickly in a small handful of centuries, the things that have made us much more powerful than chimpanzees. The analogy with ants is a lot less comforting on this one. What if the ant hive forces the workers to submit to it, making survival without it impossible? That’s literally how it works, yeah. That’s how it’s been done for tens of millions of years. That’s how the strategy lives. That’s what it means to be an ant.

They cannot be saved.

We started on this tangent a while ago trying to explain how AI science wasn’t science and AI engineering wasn’t engineering. At last we’re ready: science and engineering are interactive interfaces with our colonial individuals. Oh, not perfectly interactive: stuff happens you don’t have access to, knowledge is generated in a way that can’t be made legible to your organismal individual self. But you’ve got a stake in it. You helped make it. It’s not like the ant workers who must correspond with one particular fungus because of a predestined genetic plan they’re helpless to alter. 

That’s the direction we’re heading nowadays, though. We’ve done such a good job building up our colonial individuals that we can feed records of the directions they’ve generated into an algorithim and have that algorithim spit plausible sounding directions back out. I’m not worried about those directions “waking up” and suddenly correspond to reality all on their own. We’ll be the ones to correspond, same as we ever were. I’m worried that it’ll kill our collective selves. Our personal quests of observing and tinkering will be reduced to trying to find the magic keywords that makes the black box give out the right answer, our joy in discovering novelties in the world will be replaced by an endlessly remixed slurry of stuff we liked before, our conception of knowledge will be limited to what was encoded in our previous data scraping attempts. 

I’ve seen some comments from people who suddenly feel dumber when ChatGPT goes down. How close are they to sterile worker ants, forced to groom the queen’s larvae as the closest proxy to sending their own pattern forward, absolutely forbidden from increasing their personal individuality?

(This is an exaggeration. But it’s less of an exaggeration every year.)

We’re embedded in beautiful, unimaginably complex colonial individuals that have dramatically improved our well-being. They sustain us, and us them. It’s an impressive trick to take the groaning weight of communication that’s flowed through them and make a disembodied computer speak with the voice of the colony. But being made of organsimal individuals is the whole thing that makes colonial individuals work. Taking instructions that aren’t from individuals means ceding the power we hold as individuals, castrating yourself in service to the instructions you cannot hope to change.

If it was that or non-existence, it’d be a good deal. Ants took the deal and they got to live. But AI only came to life because of how well we were doing without it; because we created these social entities that could send instructions to organismal individuals and trust them to correspond. We can point at the penicillin fungus, and hell, we’ll point at air conditioners, and fertilizer, and nice warm blankets, and great cables that keep us connected to share instructions even faster. AI can’t point. It’s just regurgitating our directions and asking us to make them point. We progressed without it, and we can keep progressing without it.

We thrive because we have powerful intuitions about how to effectively embed ourselves in colonial individuals. These intuitions work when the colonial individuals are made up of all of us. Creating a single interface to an artificial colonial self forces us to mistrust text and images in ways we never have before, makes us generate new prayers to reason with something that can’t be made accountable, flattens out a web of obligations into an incomplete record of what has happened as a result of those obligations.

Every move towards this sort of future makes our collective selves more brittle and dependent on edicts from on high. Every move away preserves the culture that builds the patterns AI exploited to be born in the first place. We’re better without AI. 

Subscribe now

I’m not going to focus on who deserves how much blame for what happened; that’s a hard thing to do as an outsider to all of these scenes, especially during the fog of war we still have around the finer details. Instead, I want to ask: why were so many people dealing with FTX and SBF such easy marks?  That isn’t hard at all for me to write! (I once did it in my sleep.) My brother-in-arms Neil has also explored in what sense is utility actually, like, a thing? This moment where billions of dollars accumulated by an avowed utility-maxer has proven to actually be nothing at all is an excellent time to repeat ourselves: no, really, is utility actually, like, a thing?

Crypto is not money, money is not utility, utility is not real. The impact of the FTX crash gives us an extremely resonant example to draw from, so we’re going to use it. But this post isn’t really about crypto and money per se; it’s about a certain way of seeing that comes up a lot in discussions of crypto and money, and the cases when using that way of seeing makes you a sucker. As you read the first two sections, focus less on the literal treatment of crypto and money and more on seeing what the examples have in common.

Crypto is not money

We’ll use the article FTX's Balance Sheet Was Bad from Matt Levine to give us a bit of background. From the leaked balance sheet that was being shopped around to potential investors, we saw that FTX’s single biggest asset was 2.2 billion “worth” of Serum, a cryptocurrency used for the Solana blockchain that FTX heavily boosted and invested in. Only a small amount of Serum can actually be traded at any given time - 97% of it is held by “core believers” who are often waiting for the token to “unlock” before they’re liquid at all. And crucially, this means that FTX did not actually spend billions of dollars to get this Serum; they got the tokens as early backers of the Serum protocol, to show their commitment to grow its “long term value”. 1 Matt does a good job of expressing how indignant we should be seeing this balance sheet:

Like, $16 billion of dollar liabilities and $16 billion of liquid dollar-denominated assets? Sure, great. $16 billion of dollar liabilities and $16 billion worth of Bitcoin assets? Not ideal, incredibly risky, but in some broad sense understandable. $16 billion of dollar liabilities and assets consisting entirely of some magic beans that you bought in the market for $16 billion? Very bad. $16 billion of dollar liabilities and assets consisting mostly of some magic beans that you invented yourself and acquired for zero dollars? WHAT? Never mind the valuation of the beans; where did the money go? What happened to the $16 billion? Spending $5 billion of customer money on Serum would have been horrible, but FTX didn’t do that, and couldn’t have, because there wasn’t $5 billion of Serum available to buy. FTX shot its customer money into some still-unexplained reaches of the astral plane and was like “well we do have $5 billion of this Serum token we made up, that’s something?” No it isn’t!

One simple point here is that FTX’s Serum holdings — $2.2 billion last week, $5.4 billion before that — could not have been sold for anything like $2.2 billion. FTX’s Serum holdings were vastly larger than the entire circulating supply of Serum. If FTX had attempted to sell them into the market over the course of a week or month or year, it would have swamped the market and crashed the price. Perhaps it could have gotten a few hundred million dollars for them. But I think a realistic valuation of that huge stash of Serum would be closer to zero. That is not a comment on Serum; it’s a comment on the size of the stash.

FTX’s stash of Serum could be evaluated at $2.2 billion sitting on the balance sheet, but would have been closer to zero in a sell-off. The vast asymmetry between the stashed token and the trading supply puts a huge amount of pressure on the “value” of the stash. Put a pin in that inconsistency for now and let’s change topics to wash trading.

At its heart, wash trading is moving your money around and lying about why you’re doing it. For example, you might give your buddy $105, then ask him to give $100 of it back to you publicly in “exchange” for an old boot. Anyone watching the exchange might think that old boots go for around $100, which will benefit you if you have a lot of old boots: you can sell them for $50 to people who think they’re getting an absolute steal, only to learn that no one actually wants to buy an old boot unless they’re doing it with your money. And there’s nothing limiting you to only one co-conspirator. You can imagine an entire ecosystem of people trading money and old boots back and forth until a single poor sap exchanges lots of real money for old boots, at which point everyone can stop pretending they want to buy an old boot and exit the scheme. This idea of an entire fake market designed to trick one person into one transaction isn’t even hypothetical: confidence tricksters call this a “big store” scam. This has been illegal in the United States since 1936, but the anonymity of crypto wallets makes it extremely difficult to figure out just how much wash trading is going on. 

What ties these together? You have a token — Serum or whatever — being exchanged for some amount of dollars. If you witness a few of these trades, it’s easy to conclude that Serum “has value”, and is worth the dollars that you see it being traded for. But asymmetries between stash and trading supply and the existence of wash trading are two very good reasons not to do that. They demand that your evaluation of worth is more contextual, instead of ascribing anything inherently to the tokens.

But hold on -- this header is “crypto is not money”. It’s not exactly a crypto-exclusive thing to have a stash way bigger than the amount of the thing being actively traded, and while wash trading is formally illegal in the US, that doesn’t mean it’s non-existent. What does it mean to say that these deficiencies with the concept of “value” make crypto not money? Is it just a difference of degree, or is there something more profound happening here? It seems like a question that can’t be answered without getting into a fussy philosophical discussion of what money is.

Money is not utility

Hey, what the hell is money, anyway?

Let’s shake off any existing baggage and imagine a new currency from scratch. There’s a king who claims domain over some rich, fertile plains interspersed with unfriendly, difficult-to-traverse crags. Neighboring kingdoms want to steal the bounty of the plains. The king controls a professional standing army who train at soldiery year-round instead of farming, and takes some of the farmers grain as a tax.The neighboring kingdoms are the sort who would take everything, so the farmers acquiesce to having some grain taken if it means safety. 

But the king’s castle is in the center, while the forts are all along the border. It’s awfully inefficient to levy the taxed grain, catalog it in the castle, then send it back out all over the kingdom. The king would much rather have the soldiers get the grain from the farmers directly. So he mints some brand new coins with his face, gives them to his soldiers, then his tax collectors spread the news: this year, instead of asking for grain directly, we’ll be asking for coins. Now the citizens have to give the soldiers something they want in exchange for their coins. The coins symbolize the claims on the farmers' labor the soldiers demand in exchange for their work in preserving stability. 

Things are different up on the crags, though. They have patchy soil and they’re a pain to reach, so the tax collectors only went up there out of obligation, and didn’t expect much tax when they did. The crag people barely think of themselves as belonging to the kingdom, limiting themselves to paying the odd scrap of grain as a perfunctory gesture of submission. Now they hear that the tax collector wants coins instead. But soldiers don’t bother climbing the crags, and they truthfully claim that they have no coins to give and continue handing over a pittance of grain instead. The king doesn’t like this, exactly, but doesn’t want to start a civil war over it.

In the great markets of the plains cities, coins are exchanged for grain every day. A diligent observer of the markets could catalog the average amount of grain per day and record it. And if we were to ask “how valuable are these coins?”, one might think you can just point to that record and say, these coins are worth so much grain. But that’s the value of the currency in exchanges where it’s used. The value as a currency depends strongly on how many people live in the plains and how many live up in the crags. It depends on the continued stability of the king's regime and the particulars of the tax collectors. It depends on the timing of when soldiers get the coins and when the harvest is available. 

It’s easier to see those limitations in this toy example with one meaningful exchange and one king, but money is always about the diversity of the exchanges available to do with it and the normalizing force governing it. Let’s bring in a long quote from Ian Ground’s The relentless honesty of Ludwig Wittgenstein:

Say that we become puzzled about money. Here is something that people deeply desire, spend and risk their lives acquiring. People are “worth” so much money and so on. But perhaps we are struck by the fact that coins and notes are, in themselves just worthless bits of metal or paper. How can they have value? (Note that we have already slipped, even at the moment we first become puzzled, into thinking of “value” as a kind of property something has.)

Imagine that someone replies like this: it is true that actual cash is arbitrary – just stuff. What matters is that cash is backed by something that really does have value. The “promise to pay the bearer on demand” on UK notes. The gold in the bank is what really has value. The money is just an outward sign of that true value.

But gold is also just a kind of metal. Why should it have value? The same question we asked about the cash can now be asked about the gold.

Someone else might interject: gold is rare and hard to acquire. That’s why it has value. But lots of things are rare without being valuable. And in any case, no one actually trades in their money for gold. Banks won’t even let you do that. Yet we go on treating the money as valuable.

Here of course we will want to say this: actual money (coins and notes) isn’t intrinsically valuable. What matters is only that it is in fact used in trade and exchanges. The value lies in the use of the money. It’s not that the exchanges use money because the money has value. Rather the money has value because the exchanges have value. Or rather what we mean by monetary value is made manifest in and through the activities of exchange and the myriad things we do with money. And once we see things that way round, it will now seem rather strange to say that money is just worthless stuff. It looks that way and we became puzzled in the first place only because we tricked ourselves into separating out the notes and coins from their use in exchange. Our problem was how to explain how certain stuff – notes and coins – had value. So we started looking for another kind of stuff to carry that value. That is, we already committed to a particular view of what an explanation would look like. The solution was to change our view of what would count as an explanation or indeed whether one was actually needed at all. We solve the problem when we dissolve the source of our puzzlement.

The money has value because the exchanges have value. Crypto is not money because crypto is exchanged less than money is. (Crypto communities have the meme phrase “HODL” for those who intend to buy-and-hold indefinitely.) Crypto has some value in exchanges that aren’t possible without it (the canonical example for crypto supporters is escaping draconian currency controls or inflation in your home country; the canonical examples in the news are ransomware and child pornography). But mostly the value of a cryptocurrency on any given day is socially denominated in USD; mostly the crypto titans aren’t using their crypto to perform more and more valuable exchanges but are just making some arbitrary numbers go up.

“But don’t normal billionaires do the same thing with money? Not performing more and more valuable exchanges and just making their numbers go up?” Yes! This is exactly the parallel I want to draw! Crypto isn’t money because of its relative lack of use in exchanges; but money itself often has these layers of abstraction between it and the actual exchanges. Crypto is often a way to shift make-believe numbers around before dumping them back into actual money, money is sometimes a way to shift make-believe numbers around before dumping them back into actual valuable exchange. Think of points along a continuum where one end is purely self-referential scams and the other end is purely valuable exchanges. Crypto is significantly closer to the scam side than money is, but money is still quite far from the pure exchange side. It’s these valuable exchanges that we’re trying to point to with the idea of “utility”. And yes, the exchanges are real — but that doesn’t mean we can get to the end of the continuum and still end up with something that acts like a currency. 

Utility is not real

I have to be careful here not to drown in an endless sea of caveats. “Utility” is sometimes meant as nothing more than “a word I use to describe exchanges both parties found beneficial” and “effective altruism” is sometimes meant as nothing more than “I like to have evidence charities I donate to are doing a good job”. These are basically fine and inoffensive statements. The idea of utility we’re pushing against here is something like “the goodness of an exchange can be measured against the goodness of other exchanges, and the total utility of a series of exchanges can be maximized”, and the idea of effective altruism we’re wary of is something like “good charitable giving is that which maximizes utility.”

The critical point here is the difference between noting the real value of a beneficial exchange being done and looking at catalogs of potential exchanges through the lens of an imagined currency that one can spend freely on any exchange in the catalog. As we’ve just seen, denoting things as currencies adds a layer of cruft and abstraction between you and the valuable trades; imagining yourself spending a hypothetical currency brings along much of that same baggage. Money and power generally compound, giving you more of them over time based on what you already have. Exchanges are obviously real and obviously valuable, so it’s easy to think utility as a currency pegged to the “exchange standard”, also obviously real and obviously valuable. Then all your metaphors at hand tell you to accumulate interest; that is, not exchange (the valuable part). And just like Serum, the bigger the imbalance between what you’re holding and what’s being used, the more likely what you’re holding actually rounds to zero.

Think of Leorio from Hunter x Hunter. On the one hand explicitly talks about earning to give when he describes the death of a childhood friend:

“It was a treatable disease…the problem was that the operation costs a fortune. I was naive! I thought I could become a doctor…I wanted to treat kids who have the same disease, and be able to tell them it was free of charge! Then I could have told his parents, too. That was my dream…what a joke. Turns out to become a doctor, you need even more money! Got it? The world runs on money, so I want money!

Leorio ends up becoming a pro Hunter (something like a professional mercenary) to fund his way to med school. Money was an intermediate step in the road towards his goals. But what if he decided to stop being governed by mere sentiment, and instead tried to maximize utility for people with that disease? Wouldn’t it be better not to focus on becoming a doctor individually, and instead found a successful business to get money to pay off several doctors? SBF himself was advocating that way of thinking:

Let’s say Leorio takes this advice and founds Leorio Co, a company focused on maximizing revenue so he’s able to give more long term. In his search for good ways to make money, he invested heavily in SBF’s FTX, one effective altruist working with another. Well, then he’d be bust and incapable of helping anyone. Or if Leorio had been involved in a scandal, or if the price had dropped due to fraud, or any number of things.2 The risk of going bust wasn't unknown to SBF, who mentions it explicitly in his Twitter thread: 

But trying to make “the play in EV” when it risks a significant chance of going bust is the exact same mistake as valuing Serum based on what’s being traded. You have infinite imaginary potential universes filled with vast reserves of “utility” and one actual linear time universe with the utility you can actually gain from the exchanges you perform. If you’re planning a modest improvement like “I’ll get the money to become a doctor and then I’ll be much more effective at trying to improve people's health”, you need to accumulate a little bit of money, time, and focus to achieve that goal. Thinking of that investment in terms of utility works fine, because becoming a doctor is a common, well-anchored thing to do, and you can trust that exchange to remain reasonably stable. But as your plans start ballooning into “I’ll gain an unfathomable amount of utility and spend it to change the world”, you start getting a bigger and bigger imbalance between your stash of imagined ability to do good and the actual exchanges you’re looking at when trying to evaluate how much your stash is worth. Imagine a disappointed Matt Levine writing: But I think a realistic valuation of that huge stash of utility would be closer to zero. That is not a comment on utility; it’s a comment on the size of the stash.

So let’s make sure to learn this lesson about imaginary tokens that are sometimes exchanged for value. Think of the idea of “maximizing utility” as instead “HODLing my stash of UtilityCoin”. With this frame, you’ll start asking the right questions: how do I know the exchange value of UtilityCoin to good outcomes will stay stable no matter how much I spend? Is the peg between UtilityCoin and good outcomes dependent on a single bottleneck? Are you evaluating your UtilityCoin by looking at a single good outcome that may have been manipulated in some way? And then maybe we can gently disentangle the EA proclivity for doing good from the EA proclivity of being an easy mark. 

Subscribe now

In terms of raw output, there were only thirteen posts. But one was the gargantuan Representation and Uncertainty; and anyway, here of all places, we’ll distrust the impulse to simply tally up the first metric we get our hands on. The relatively small amount of posts belies the enormous development in my ideas that happened throughout the year. In Desystemize #1, my thoughts in this space were mostly limited to specific anecdotes with only a wordless urgency drawing them together. The best I could do to summarize was:

The thesis of Desystemize, in one sentence: it takes a lot of work for a number to mean anything and we largely aren’t doing it.

But through the work I’ve done to research and write these articles, and the discussions they’ve generated, I’ve gotten a much clearer grasp of the target we’re aiming for. I managed to truly grok In The Cells of the Eggplant, which I had bounced off before (I’m embarrassed to read my post about it trying to carve out the “desystemic lens” now that I have a more nuanced idea of what’s actually going on in breakdowns of representation), and I read Wittgenstein’s Philosophical Investigations, which is an excellent way to change the “default way” you approach meaning. Rewriting my one-sentence mission statement today, it’d be more like:

The thesis of Desystemize, in one sentence: representing the world with static forms we can manipulate often destroys the details that make them meaningful, and we don’t understand the work we tacitly perform to make representations useful.

A more cumbersome sentence, maybe, but a lot more accurate.

Getting out there and posting got me connected with Suspended Reason, and through him the rest of the Inexact Sciences group. Being in constant conversation is a great way to help you realize the frontiers you should be looking at next: a lot of my output on tis.so is me starting to make a point and realizing I should just encapsulate it for posterity. (Sometimes the ways TIS improves my productivity are even more straightforward than that: a few nights ago Hazard literally came to me in a dream, I wrote up a dream-article based on our dream-conversation, and then when I woke up I got to just transcribe the dream article and publish it.)

So the first thing I want to stress is that you can literally just get out there and start moving on something you care about. I don’t want to be the kind of person who says stuff like “positive thinking solves all problems!”, because it clearly doesn’t. But I can say that if you feel like you’re stuck in a rut, it’s important to dispense with the false humility of “I’d better not think too highly of myself”, because past-you wasn’t working on the thing, and if you demand future-you is mostly continuous with past-you, future-you won’t be working on the thing either. Pick a birthday (what I did) or some other cute little milestone and just say “Okay, starting today, things will be different” - being deliberate about that conscious break with your past is an excellent way to get moving.

I can sure attest things are different for me! I’m comfortable using “the representational crisis” as a starting point for arguments, without needing to justify myself every time I post. Year One of Desystemize has handled that first urgent reason I felt compelled to write it: the point that static forms will always be insufficient, that purely one-way flows of information don’t break down for idiosyncratic reasons (even if those are what get covered in the stories about them), but fail as a class because of general and predictable forces.

Year Two: so what will we do about it? 

Chasing the Treasure Fox has already previewed the areas I’m going to start investigating. How exactly do we generate accounts of variation? What sort of rules about environments are being sent forward in time through our embodied cognition? How do we distribute our thinking to the space around us, and how do we index that distributed thought to call upon later?

If I had to guess, my writing over the next year will be in three distinct channels. I’ll start developing these ideas in bite-sized chunks over on tis.so. When I see an example of a representational/ontological issue that’s digestible and evocative, I’ll write a numbered, average length Desystemize post about it. When I have a more complete understanding about detail-preserving cognition, I’ll write a longer, more conceptual named post like R&U and Treasure Fox. But where I ended up now is nothing like I was thinking on June 18th, 2021, so all this is subject to change anyhow.

Since interaction is such a key component of discovering useful concepts, I’d like to be explicit about ways you could influence the writing of Desystemize, if you’re interested:

• Most obviously, and also most importantly - you can just share stories where you think there’s something more to be said than the article you’re linking, whether it’s a direct representational issue or just something that seems to rhyme with what I’m covering here. Worst that can happen is I disagree and don’t write about it, but my whole mission is to think of frameworks that apply generally, so there’s no such thing as too many links.

• I also love talking about these topics, even outside of an article context. Conversation is a good way to force you to articulate thoughts you had sloshing around subconsciously and thereby gain a better understanding yourself. My DMs are open! Also happy to discuss appearing on podcasts or speaking at conferences. (I’m located in Minneapolis, but willing to fly for a sufficiently interesting discussion group.)

• One of my signal joys this year has been discovering the writing of Vasily Grossman. He manages to blend the gnarled morality of how people actually act in the world with a razor-sharp personal understanding of good and evil, chronicling the world as it is without succumbing to bland nihilism porn. In this regard he’s right up there with Remarque, another one of my favorite authors. If you know of other authors with this quality - people who’ve captured the true nature of human beings in extreme situations, taking care for realism in motives and dialogue while maintain a profound moral clarity that gives the story a point - please let me know which book to start with. (For Grossman, it’s Stalingrad into Life and Fate; for Remarque, it’s Flotsam into Arch of Triumph.)

• From the non-fiction side, I need to get a much better understanding of embodied/environmental cognition, so recommendations for good reads of that nature are extremely welcome. Also anything good about ecology - while I have a decent working understanding of ecology, I suspect that I need to reindex my knowledge entirely with the new ways of seeing I’ve developed in the last year. Ecosystems are a kind of distributed cognition, and I have a lot more conceptual buckets to put them into nowadays.

• Speaking of distributed cognition…my U-Haul trailer was unhooked and stolen in Las Vegas in May when I was moving back to Minneapolis, and I lost nearly all of my stuff. If you happen to know any thieves in the Vegas area, please tell them to at least give me my books back. Every time my brain has a thought that’s indexed “go look it up in this book” I have to go buy the book before I can remember what I know. It’s exhausting and has been a real drag on the writing process.

• Finally, if you’re the kind of person who wants to send a present when you hear about a birthday, my charity of choice is Sea-Watch, who stop people from drowning in the Mediterranean as they try to navigate the world of international migration. Sea-Watch is just about the purest example of preserving detailed human life against the annihilating force of static systems that you could possibly come up with.

Whether you do any of these things or just passively read along, I hope that the cognitive tools we’re building here come in handy as often for you as they do for me. The best way to appreciate the world is to see it more clearly, and I’m honored to be working on the frontier. See you next when I have something to say.

-Collin 

Subscribe now

I.

Today, we’ll be following the foxes from The Elder Scrolls V: Skyrim to see what we can learn about the limits of empiricism. Like in Desystemize #9, studying emergence in games is interesting precisely because games have such well-defined base objects. Studying reality using the techniques of games is extremely dangerous because those techniques ignore emergence; for exactly that reason, though, emergent phenomena that do show up in games are some of the easier examples of emergence to understand. 

I learned about the treasure foxes from this Twitter thread; I’ll recap it for people who haven’t read it. Foxes in Skyrim run away when a player goes near them. The Skyrim community quickly noticed that the spooked foxes would generally run towards treasure. A cute little feature — but one that no Skyrim developer put in explicitly. Instead, the behavior was due to the invisible geometry of the game world. Skyrim is partitioned into a covering of triangles known as the navmesh:

The various actors within Skyrim - monsters, guards, foxes, whatever - “see” the world of Skyrim through the navmesh. Players don’t see it at all. From the player perspective, the foxes running away from them in the “world” of Skyrim. From the foxes perspetive, they’re actually running away along the navmesh, trying to put a lot of triangles between themselves and the player. What’s interesting about this distinction is that not all triangles are created equal. In the wilderness, there’s a lot of broad natural areas that don’t need much detail from the AIs pathfinding point of view, letting the triangles be simple and large. But around areas of interest, you have more objects, actors, and tricky features; and all of those need handling in the navmesh. This results in a denser concentration of smaller triangles. The fleeing fox, trying to get triangles between it and the player as fast as possible, will naturally go for these clusters.

So it’s not exactly treasure that the fox is going for. It’s more accurate to say the fox is going for interesting locations. Areas where a simple description can cover a lot of terrain make for large triangles, whereas finely detailed descriptions require many more triangles to account for the detail. The fox is sniffing out these local accretions of complexity, which are more likely to have story elements, danger, and treasure to reward you for engaging with it. Since treasure is the reward the player is after, it makes sense that they would mistake order-seeking behavior for treasure-seeking behavior. 

But notice the language I used: treasure is placed to reward you for engaging it. What’s the relationship between local complexity and treasure? It’s not intrinsic within the system via an explicit interestingness value. It just so happens that designers generally try to reward players for engaging with local complexity. The link between local complexity and treasure isn’t maintained by Skyrim itself; it’s imposed from without by the whims of designers, subject to change at any time.

Imagine there’s a naturalist living within Skyrim who’s trying to understand the behavior of foxes. Their model for agent behavior is their own behavior. They’re an autonomous agent who often goes towards treasure when they see it, because they love treasure. However, sometimes they need a break to rest or repair their weapons. Let’s say that, on average, they head towards treasure 90% of the time. Various factors relating to themself (fighting ability, gear quality) or the treasure (how obviously it’s signposted, how valuable it appears) could change that.

They’re studying the foxes, who are also autonomous agents who also often go towards treasure when they see it. Designers are pretty reliable at maintaining the link between order and treasure. Let’s say they’re 90% reliable. Then the chance that the naturalist and the fox will head towards treasure is the same, and if probability theory were the only meaningful measure of uncertainty, this parallel would be enough to perfectly explain the behavior of foxes. But of course it isn’t; the naturalist is going towards treasure because they want it, while the foxes don’t even know what treasure is, and are drawn to it only via the glue of interestingness. And unlike the naturalist, whose chance to interact with treasure is mediated by factors relating to their local context (the naturalist and the treasure), the explanation for whether a given fox goes towards treasure or not can’t be explained by learning more details about the fox.

II.

There’s a philosophical problem called “the problem of induction”. In brief, it goes like this: all we have to predict the future is what’s happened in the past. But what justification do we have for saying the future will keep acting like the past? If one person has a hypothesis that “all apples are red or green” and another has a hypothesis that “all apples are red or green, but when the earth’s magnetic poles shift, they’ll become orange or purple”, you have an equal amount of evidence of both hypotheses. How do you square this against our obvious intuition that the first hypothesis is right and the second hypothesis needs a “reason” to be credible? Some people try to appeal to Occam’s razor and look for the “less complex” hypothesis, but it’s not like we have a solid definition of complexity, either. 

We can see that the problem of induction is much more relevant to the fox than the naturalist. The link between the naturalist and treasure is stable in a way the link between the fox and treasure is not. Philosophers fretting about induction are worried that every link between two phenomena could be a just-so-happens link that fails to apply in the future. As formless empiricists, though, we’ll look at it the other way around. We know that one of these links (naturalist → treasure) is well-founded and stable, while the other (fox → treasure) is fragile in exactly the way the problem of induction warns us about. If we were that Skyrim naturalist, with access only to in-world tools, how would we be able to tell the difference?

Notice that probability theory isn’t the tool to help us here. That 90% chance to path towards treasure is the same for the naturalist and the fox, but we want to account for the difference between them. Someone who speaks solely in terms of probabilities and priors is forced to equate the two cases, which leaves you vulnerable in one of two ways. Either you decide to tacitly ignore the problem of induction, letting you work unimpeded but leaving you completely exposed to blowouts when the fox → treasure link gets severed; or you take it to mean “nothing is knowable”, leaving you with a certain theoretical purity but practically unable to explain phenomena as obvious as “the person who likes treasure will often walk towards it.”

We need to find some sort of value that’s different between the naturalist → treasure link and the fox → treasure link, some way of saying “Don’t worry about induction here, but do worry about it there.” And I’ve already spoiled what it is: explanations for variation. They’re both the same probabilistically, but the naturalist can tell a story each day about why they did what they did. The link between the naturalist and the treasure is more stable precisely because that link is listening to the context of the naturalist and the treasure. By contrast, even if there’s just as much “evidence” that foxes like treasure as much as naturalists, the explanation for why a particular fox does or not path towards treasure is not explainable by the context of the fox.

We’ll use the phrase “account of variation” of describe this. Think of an account of variation as “everything you’d need to pay attention to if you wanted to explain all the possible outcomes of the phenomenon you’re studying”. It’s a combination of the channels of information you’re looking at (your “accounting system”) and your understanding of what’s possible given that information. The control panel of a power station and the training of the operator looking at it combine to form an account of variation of that power station.

No account of variation can ever be perfect. The power station will fail to generate power if it’s hit by a giant meteor, but the control panel doesn’t have the position of every large object in outer space. But again, we’re going to ignore the theoretical universality of the problem of induction and instead notice the large practical differences between good accounts of variation and bad ones. The power plant operator clearly has a better account of variation than the homeowner connected to the grid, whose mental model is something like “The power will always be on unless there’s a big storm or there’s a news story about how power might be cut soon.” And the homeowner, in turn, has a better account than the moth who’s flying toward the homeowners lights, aware only that the light is on in this particular moment.

Nassim Taleb likes to use the example of the “turkey problem” to explain induction. Imagine a turkey statistician analyzing how well the farmer is treating him and his fellow turkeys. Each day, he records that the farmer has treated the turkey population well. The farmer knows that this is to slaughter them all before Thanksgiving, but the turkey statistician has no account of variation to understand this, and can only look at things empirically. He identifies as a Bayseian, proud of “updating his priors” each day according to the evidence. And every single day, the evidence is more and more positive — until the day all the turkeys are slaughtered. Being an empirical turkey was worse than randomly guessing: he was maximally confident in his hypothesis the moment before his death.

Once again, note that the problem of induction is more or less irrelevant to the farmer when it comes to the turkeys. He knows why he’s feeding the turkeys and when he’s going to stop. To the turkeys, though, it’s literally a matter of certain death. So some kinds of evidence are more induction-proof than others. And the answer isn’t just better statistical tools - all the turkey statistician could do with his mathematical prowess is mistake absence of evidence for evidence of absence and get a little wronger each day. 1 What he needed was a better account of variation, an awareness that “we’ll stop being fed if our life status changes, a change that can happen to all of us at once” was a possibility. But how do we generate accounts of variation?

III.

After hearing about the turkey problem, our Skyrim naturalist isn’t satisfied by probabilistic evidence. They want to know why the fox moves in the way it does, not just how often it moves towards treasure. The movements of an individual fox are a function of the navmesh, which the naturalist can’t see. What exactly can they figure out about fox movement given this blindness?

Let’s say they study the movements of a great many foxes, mapping out their paths over time. The regularities in the ways they turn would be observable, given enough data. It would take a lot of looking, but you could derive the existence of the navmesh as mediated through foxes.2 You could give yourself a brand-new sense for something you can’t perceive directly. And if you happened to see a fox wandering about the countryside, you’d be able to guess exactly where it would run depending on your angle of approach.

This indirect understanding of the navmesh wouldn’t be a theory of interestingness, a “why” behind the navmesh. If there was a new location added in DLC, the navmesh you found so far wouldn’t let you instantly know where foxes go in this new territory. You would need an understanding of interestingness to predict that. Can the navmesh you’ve found so far help develop that theory? Well, in a sense. Once you know what the triangles are, you can start to notice that they’re denser around some objects and actors than they are out in the wider world. Theoretically, the same spirit of discovery that led you to finding the navmesh could lead you to finding its correlation with local complexity.

Why do I say “theoretically”? Wasn’t our understanding of the foxes perfect? Yes — but perfectly predicting the movements of foxes doesn’t mean you perfectly understand the provenance of the navmesh. You’ve used the effect of the navmesh on the movements of foxes to create a new sense, but foxes won’t help you understand how to use the new sense — in this case, correlating it with the points of interest on your map. And if you don’t do that correlation, your findings will only be as good as the territory they’re trained on. You can’t see a new map and guess where the foxes will go unless you’ve made that connection.

And even if you do make that connection, you’re still missing a lot of what you’d want from a theory of interestingness. Suppose that one day there was a Bethesda all-hands meeting. The CEO mournfully declares “The kids don’t appreciate nature anymore! We need to do something about it! Move all of the treasure into beautiful, untouched nature, with nothing else around it. That’ll teach them the joys of exploring instead of fighting.” The developers feverishly put in a patch, and suddenly interestingness is no more — the correlation between treasure and local complexity is completely reversed. You still have a perfect understanding of the foxes, who keep right on respecting the navmesh3, but they’re now running away from treasure, not towards it. 

This should put you in mind of the turkey problem. Evidence-based techniques within the framework couldn’t save you from abrupt change coming from without. The Skyrim naturalist was able to do meaningful empirical work deriving the existence of the navmesh through analyzing foxes. But they didn’t have an account of variations of the navmesh — in this case, a theory of Skyrim developers — just like the turkeys didn’t have a theory of farmers. So we can think of the problem of induction as an attack on empiricism that can be “shielded” by accounts of variation. The farmer has a theory for when they’ll stop feeding these turkeys (because it’d be weird to feed a turkey you’ve killed), and so their analysis of what will happen over time with the turkey feed is more induction-proof than the turkeys, even if they’re both looking at the same evidence of “so far the farmer fed the turkeys every day.” 

And look, this is common sense, right? Why have I taken over two thousand words to get to “people who understand why things are happening know more than people who don’t?” Because our theories of what evidence is are based far too much on outcomes of individual events, and not accounts for variation. In truth, outcome-based evidence is only useful to the extent that you’ve picked induction-proof outcomes to tally up and have a sense of when to abandon outcome-based analysis entirely. (Farmers don’t say “I’ve fed the turkeys every day so far, so the conditional probability that I’ll stop feeding them the day after slaughter is really quite low.”) And exactly because we have this intuitive common sense buttressing our formal analyses, we’re free to write bad fanfiction about the probabilistic reasoning we imagine must be “really happening” in our brains. But probability theory doesn’t generate accounts of variation; it’s dependent on them.

IV.

This essay is not an extended parable about the limits of machine learning, or at least it wasn’t meant to be when I started writing it. But there’s been a lot of talk recently about the existential risk to humanity from machine learning models, so it’s worth being explicit about the connection the field has to what we’ve talked about so far.

In The Scaling Hypothesis, Gwern writes:

The blessings of scale in turn support a radical theory: an old AI paradigm held by a few pioneers in connectionism (early artificial neural network research) and by more recent deep learning researchers, the scaling hypothesis. The scaling hypothesis regards the blessings of scale as the secret of AGI: intelligence is ‘just’ simple neural units & learning algorithms applied to diverse experiences at a (currently) unreachable scale. As increasing computational resources permit running such algorithms at the necessary scale, the neural networks will get ever more intelligent.

Is the scaling hypothesis true within Skyrim? It depends which level we’re looking at. One way to state the scale hypothesis is “If you look at the movements of enough foxes, you’ll be able to perfectly predict the movements of any fox” — which is really just saying “If you look at the movements of enough foxes, you’ll find the navmesh.” I’ve already taken this as a given in the earlier sections, and to the extent the scaling hypothesis is saying this, I believe it. We’ve been supposing the naturalists themselves would be doing the fox-recording work, but if they had a computer doing it, it would look pretty much like the machine learning models we have now. Fox lovers within Skyrim might say something like “Oh, nature is too complex and beautiful! Your foolish machines will never *perfectly* understand fox pathing! Maybe it’s doing well on a high proportion, but there will certainly be diminishing returns on trying to get everything!” and then the foolish machines would keep looking at foxes and eventually they would perfectly understand fox pathing without diminishing returns on trying to get everything. Notice also that in this scenario, the people maintaining the model won’t know that it understands “the navmesh”. They just know that they have a magic box that can take “there was a fox here, I scared it from this angle” and it returns where the fox will go. 

Will the scaling hypothesis hold true for the theory of “interestingness”? That is, if you feed the model some location where you have no data on the movement of foxes, will it figure out that the foxes are more likely to move generally towards the local complexity? Well, what does “feed the model some location” mean here? This only works if it’s been trained with the fox data in conjunction with data on where the points of interest are on the map. But unlike the naturalists, who have the navmesh as a sense that they can try to combine with their other senses, the machine cannot “go back out” into the world with this new understanding.

So the model will predict that foxes in a new world will go towards the treasure exactly when the data it’s fed about that new world includes points of interest as a parameter. Will “points of interest” be parceled out in our descriptions that we feed the machine? Maybe — if the people doing that parceling think to represent it that way. In other words, the people doing the describing have to figure out the right description to make “interestingness” inferable; the machine will solve the problem if the people feeding it data figured out the right accounting strategy for that data beforehand. 

Thinking quickly, the machine learning model constructs a homemade account of variation, using only petabytes of training data, exoscale computing, and an account of variation.

And finally, there’s the question of induction-proofing - will the model construct a theory of Skyrim developers? Well, of course not. How on earth could it do that? That’s not how machine learning models work, scaling hypothesis or not. There’s simply no way that studying the movements of foxes can trigger an output that says “but these trends might all instantly reverse if there’s a Bethesda all-hands about instilling an appreciation for nature in the player”. These techniques are based on evidence of outcomes, and can’t help but be susceptible to the turkey problem.

What if we injected meeting minutes from Bethesda level design meetings into Skyrim? Is the problem just that the data doesn’t get there? Could the machine predict the reversal based on the notes from the-all hands? But now we’re right back to the accounting problem again. How do you know that this note was about that bit of level geometry? To make the memos legible to your model - which is, remember, trained to predict the movements of foxes - you need to relate strings of arbitrary text not to the movements of a fox, but to features in the terrain. The problem is that the memos will only be helpful for predicting the movements of foxes to the extent that they describe patterns that will occur in the navmesh. But it’s cheaper and easier to just infer the navmesh from the actual fox movement! You can give a machine learning model the information it needs to foresee a great sea change, but it will only do so to the extent that the sea change has happened in the past; it can’t help but be a turkey.

It’s worth noting here that the relationship between the navmesh and the observable world is unusually straightforward, in a way that might make this problem seem less severe than it actually is. The treasure fox is hard to understand because the navmesh is invisible to the player, making it impossible to just see. But every point in the obserable world happens to belong to exactly one triangle of navmesh, making it somewhat clear that “location” data is the kind of data you need. Imagine instead you’re trying to explain the behavior of fish in a river. A new factory is built upstream and they bribe the water comission to say the chemical assay is unchanged even as they dump large quantities of illegal pollutants into the water. These pollutants will cause the fish behavior to become signficantly more erratic. What account of variation do you need for fish behavior to generalize into this new scenario? Well, you need to include the building permits of everything that runs off into the river, the projected integrity of the current members of the water commission, and the properties of chemicals that aren’t what the factory says they’re using but would be cheaper for creating the things the factory creates. So what does it mean to feed the model “the state of the river” to try to generalize the findings you’ve had so far? Does your dataset of rivers often include “Dale from the water commission has racked up a substantial amount of credit card debt?”

This is the sort of thing we should keep in mind looking at models like GPT-3. You can give it a prompt like this:

Machine learning models trained on human language without being able to interact with it will sometimes substantially improve prompts that are given to it by humans, but sometimes return complete non-sequiturs. A theory of when it will do well vs poorly is that

And GPT-3 returns many plausible sounding answers. Here’s one I want to look at in particular:

Machine learning models trained on human language without being able to interact with it will sometimes substantially improve prompts that are given to it by humans, but sometimes return complete non-sequiturs. A theory of when it will do well vs poorly is that models that are good at the hallmarks of human language will also be much better at understanding and responding to human prompts.

The theory goes that human language has certain hallmarks that make it distinct from other forms of communication. Models that are good at understanding these hallmarks will also be much better at understanding and responding to human prompts.

Some of the hallmarks of human language include:

1. A rich and varied vocabulary

2. The ability to use language for a variety of purposes, including persuasion, description, and explanation

3. The ability to combine words into complex structures

4. The ability to create new words and expressions

5. The ability to use language in a flexible and creative way

Models that are good at understanding these hallmarks will be much better at understanding and responding to human prompts.

GPT-3 surfaces text that says that models that are “good at understanding these hallmarks” will be better at responding to human prompts. But is GPT-3 itself good at understanding hallmarks? Is it “using language” like it says the good models do? Or is it just good at predicting which words tend to follow other words? We should imagine the path of a sentence to be like the path of a fox, traversing word space according to some unconscious rule. Unlike the navmesh, where every triangle maps cleanly to a single set of coordinates in space, word-space is much more sensitive to minute changes in condition and self-reference and a million other complexities. But it’s the same sort of thing where seeing enough paths that you can predict another path — within the domain. 

But no one is really scared about an AI that might one day be really good at repeating how humans have tended to end sentences in the recent past. The fear is that it’s going to “wake up”, that it will see enough statistical correlations relating to agents that it will become an agent itself. Can it develop a theory of self-embodiment? By now you can guess the answer - it depends on how the model can perceive the factors from without, which itself depends on the accounting system. Remember our lesson from Representation and Uncertainty - representations annihilate detail. Your understanding of the universe must be a constant conversation with the phenomena itself, because you can never be sure you’re capturing “all of the details” effectively enough, like the impact of a meteor on the power plant.

And this is how human beings are turkey-proof in a way current machine learning models definitely aren’t. We’re also vulnerable to the problem of induction - but it’s with hypotheticals like “What if tomorrow all of the universes photons turn into frogs?” If that happened, we’d all be blinded instantly since our eyes are expecting photons, and we’d also probably be killed by the sun hitting us with a constant beam of frogs at the speed of light. But we have, via our embodied cognition, a multi-billion year guarantee that so far photons have remained photons, and taking their behavior as an unassailable starting point has always worked.

Fears of AI awakenings are fears that somehow hearing enough primates talk about their embodiment will give the machine a theory of embodiment. But why would it generate such a theory? It’s always going to be easier to cheat, in the same way that the machine didn’t need to create a memos → navmesh theoretical linking when it could simply figure out where the navmesh is by looking at the foxes. The power of scale is at once vindicated (you really don’t need theory to make predictions about the here-and-now) and bounded (you really can’t scale your way out of being a turkey). 

Note that this doesn’t preclude the possibility of ever having a machine that can think like a human. If we could somehow embody a machine to take the base senses we’ve been evolutionarily trained to take as given and similarly decide that all theories of variation must “ground out” in them, then go out and interact with the world while holding that belief, there’s a possibility of a machine that can generate accounts of variation beyond human capacity. But it will never happen by the current approach of simply training a model on vast quantities of data exhaust; you need the interactive capacity to generate a new sense by interfacing with the universe in a novel way. And if such a machine is ever developed, there’s strong reason to believe that it won’t have the worst-case “hard takeoff” where it instantly goes from slightly superhuman to arbitrarily powerful. Hard takeoff relies on self-improving in machine time, while interactivity means your understanding of the phenomenon is bounded by time it takes to observe that phenomenon, since you need to work with your new account of variation and can’t look at past data which didn’t account for the detail you’re now experimenting with. For example, when you discover the microscope, you can look at microscopic things going forward, but you can’t look back at all your old notes and observe the microscopic things there, because you didn’t account for them until now. (Though sure, let’s test the first interacting machine in a building with no internet and a big moat outside. I’m confident, not stupid.)

Our popular conception of intelligence encompasses both the generation of accounting systems (creating the microscope to perceive the hitherto-unseen) and the effective use of accounting systems (taking your observations from the microscope and using them to correctly guess what other microscopic things might do). People who are “smart” are usually good at doing both of these things, so there wasn’t much harm in blending them together conceptually. The scale hypothesis forces us to be more aware of the difference, since scale is so incredibly powerful at working within existing accounting systems while being powerless to escape them. If we don’t take care to draw this distinction, we’ll either be left like the AI skeptics who constantly underestimate what scale can do (by assuming machine learning’s inability to “think” like we do means it can’t find cryptic patterns) or the doomsday prophets who think the model will kill us all (by assuming that it’s ability to surface cryptic patterns within an accounting system means that it has “the intelligence” of a human being who could surface that pattern.) For the vast majority of human history, creating new accounts of variation and solving problems within them were just two things that people could solve using “intelligence”, in a way that made it seem natural to equate them. But the overwhelming might of scale has severed that loose equality forever, and we need to follow the treasure fox to a more nuanced description of how human beings actually solve problems.

V.

And what is that more nuanced description of how we actually solve problems?

Here’s where I confess: this article is the start of something, not the conclusion. I don’t have a solid enough understanding of accounts of variation to perfectly describe exactly which sorts of reasoning I would expect to be possible from a passive and outcome-driven frame and which require interaction with the universe. All I want to impart to you for now is that such an understanding is necessary to talk about “intelligence” in a way that remains coherent in a world with modern computing power. We need to change our concept of what “evidence” is. When our knowledge of a phenomenon is solely probabilistic, we should consider ourselves senseless turkeys, capable only of pointing at a pattern with no account for when that pattern may stop holding. We should take our common-sense feelings of what understanding is — that those who can expand if-then chains many stack levels, or deal with complex hypotheticals, understand more than people just tallying up the score — and try to figure out a more rigorous way of classifying where on this continuum a given piece of evidence is. 

And let’s dispense with the false humility of us being “just” biological beings, imperfect copies of some hypothetical paragons of pure reasoning. Evolution didn’t start with reasoning. It started with replicators looking for a way to take information and defy entropy on a local scale to send that information forward in time. The replicators uncovered the most induction-proof patterns possible, the signals that you could use to drive decision making in a way that would remain useful for billions of years. Formal, abstract reasoning came much later, a cute little module that helped the information make it into the future a little better. But that reasoning came after, and has always been subservient to, the senses that can account for variation as accurately as possible4. So it was, and so shall it be.

We followed the fox looking for treasure, and we found only complexity. Still, the fox was going somewhere, and it’s worth figuring out where it took us. Let’s blend humility in the face of induction with confidence in the wake of scale and see what it gets us. To end, I’d like to share this quote from Remarque’s The Black Obelisk:

“Chaos”, Wernicke says. “Is it really chaos? Or is it so only for us?” Have you ever considered how the world would be if we had one more sense?”

“No.”

“But with one sense less?”

I reflect. "Then you would be blind, or deaf, or you couldn’t taste. It wouldn’t make a big difference.”

"And with one sense more? Why should we always be limited to five? Why couldn’t we perhaps develop six someday? Or eight? Or twelve? Wouldn't the world be completely different? Perhaps with the sixth our concept of time would disappear. Or our concept of space. Or of death. Or of pain. Or of morality. Certainly our present concept of life. We wander through our existence with pretty limited organs. A dog can hear better than any human being. A bat finds its way blind through all obstacles. A butterfly has a radio receiver that enables it to fly for miles directly to its mate. Migrating birds are vastly superior to us in their orientation. Snakes can hear with their skin. There are hundreds of such examples in natural history. So how can we know anything for certain? The extension of one organ or the development of a new one—and the world changes, life changes, and our concept of God changes. Prost!"

I lift my glass and drink. The Moselle is tart and earthy. “And so it’s better to wait till we have a sixth sense, eh?” I say.

“That’s not necessary. You can do what you like. But it's a good thing to know that one more sense would knock all our conclusions into a cocked hat. That puts an end to too much solemnity, doesn't it? How's the wine?”

Subscribe now

But here’s a quote from a January 2022 video of a man named Simon solving a Sudoku:

I now understand how to explain what’s going on in the puzzle. What I can tell you, what we’re going to end up with here, is that this digit, this digit, and this digit are a 1, 2, 3 triple. And isn’t it amazing? This, to me, is quite beautiful. I can tell you without fear of contradiction, if I had tried this puzzle two years ago…well, unless there is something else here that I’m not spotting, I might never have spotted that. Yet nowadays, because set equivalence theory has become so ingrained and practiced, it’s a very quick spot. And that being a 1, 2, 3 triple is just remarkably gorgeous. So how do we prove this is a 1, 2, 3 triple…? I mean, this puzzle could appear in the World Puzzle Championship now. There’s no way this could have appeared a few years ago, no one would have been able to do it! But now, that break-in is so well-signaled it’s almost fair!

We haven’t made any breaking new discoveries about the digits 1 to 9. But this set equivalence theory, whatever it is, has changed how Simon views puzzles so much that it’s not even comparable to two years ago. It’s all still just digits in a grid, but describing them differently leads you to a new conclusion. And precisely because Sudoku is so simply defined, the case study of set equivalence theory is worth studying. If we can find hidden secrets here, in a 9x9 grid of single digits, we can find them anywhere.

Let’s start by explicating the existing, “normal” ontology for a Sudoku. The rules specify that you have exactly one of each digit from one to nine in every row, column, and box – so it’s no surprise that these are the usual terms used to describe Sudoku puzzles. For example, you might say “I know that row 2 needs a 3, and it can only go in to this square,” or “Box 5 already has a 9, so the 9 can’t be in any of the other cells in box 5.” Because the explicit impact of the rules is based on these sorts of divisions, it’s a very useful ontology to have.

But that’s not the only way you could describe a Sudoku. You could partition the grid into an orange ostrich head and a fat green frog:

There are things you can say about these shapes. No digit can appear in the ostrich head more than three times, because its only in three boxes total. A digit that appears on one of the frog legs won’t appear on the other, because they’re in the same row. While these claims are true, they’re ultimately just pointing back to the words we were comfortable with already. Drawing the ostrich and the frog didn’t really accomplish anything.

Ontologies can’t exactly be wrong, since they’re just methods of description. But it’s pretty obvious that this one isn’t useful. It doesn’t let you state anything new that you couldn’t state using your previous vocabulary. It’s just an unnecessary level of abstraction that has to be peeled away to get to the useful ontology.

Here’s another way you could partition the sudoku: one blue ring in the center, and four orange anchors in the corners.

Is this any better than our frog and ostrich? Does it let us make any statements that we couldn’t easily make with row, column, and box? As it happens, yes. You can truthfully make the following statement: “The digits in the ring are exactly the same as the digits in the four anchors.”

Wait, what?

This surprising fact is called “Phistomofel’s Theorem”, named for the setter who initially popularized it. It’s a specific example of set equivalence theory that helped many people (myself included) understand it. Imagine that the letters 1 to 9 were all on tiles, like in a game of Scrabble. Each row, column, or box has the tiles 1 to 9 once each. Suppose you took the tiles in these two rows and these two boxes and put them in a blue sack:

Row 4, Row 7, Box 4, and Box 6 (image mislabeled) each have a run of the digits 1-9, so in total the blue sack has 36 tiles, four copies of each digit. Now, take a clone of your puzzle. This time, put these four columns in an orange sack:

Since both sacks have 4 runs of the digits 1 to 9, they have the same composition of digits inside them. But now, let’s superimpose the contents of the two sacks on to one grid, and look at the cells with both colors:

The cells with both colors are in both sacks. If you take the exact same cell out of both sacks, then clearly the two identical sacks will remain identical – you took the same thing out of each of them. You can do this for every cell with two colors. So, removing them all from our superimposed grid, we’re left with this:

That’s how we know that the digit composition of these regions – the ring and the anchors - are identical. And that means that this is a useful upgrade to our ontology, because now we can make statements like “There are two sixes in the ring, and only two spots in the anchors where a six can go, so there must be a six in each of them.” Those statements would be extremely tedious and unintuitive to make solely in terms of rows, columns, and boxes.

Once you’ve gotten the hang of Phistomofel’s Theorem, it’s a relatively short jump to using set equivalence theory in general. When we were picking runs of the digits 1-9, we made those particular choices because they happened to have a lot of overlap to them. But the core logic works for any regions that you make out of complete sets of the digits 1 to 9 – you can always eliminate the overlap and say that what’s left of the two regions are equal.

On its own, this is a neat bit of Sudoku trivia that might happen to pop up now and then in a puzzle. But what’s really interesting is when a puzzle is designed around this phenomenon. Setters (the term of art for people who design hand-crafted Sudoku puzzles for solvers to work through) can make a puzzle that’s virtually impossible using just rows, columns, and boxes, and can only be solved by upgrading your ontology. The Youtube channel Cracking the Cryptic has been an excellent source documenting this evolution.

Now I can reveal where that quote up top came from: a video from Cracking the Cryptic called How to Make an Impossible Sudoku Easy. (I’ll be spoiling the break-in to the puzzle, so if you’re interested in trying it on your own first, do it now before reading on or watching the video.) Technically speaking, this isn’t quite a normal Sudoku, but a variant with some special rules:

The variant itself adds some words to our ontology. “This two-cell cage sums to 12, so it can’t have a 1 or a 2 in it.” “This line has a 5 on one end, so it must have a 5 on the other end.” “This digit is a 6, so the one on the other side of the dot is a 5 or a 7.” But such straightforward interpretations of the variant rules aren’t what interest us here. If Simon had seen these rules two years ago, he wouldn’t have struggled to notice what they mean. Set equivalence theory is special in that it’s not a straightforward application of set rules. It’s a new way of seeing, one that you need to know to invoke instead of seeing an obvious hint for.

In this case, what’s interesting is the interaction of set equivalence theory with the palindromic lines. Start with a column set that includes parts of all 6 lines: columns 1, 4, and 7.

Next, get a row set that includes parts of all 6 lines - rows 3 and 6:

This is a little different from how we did Phistomofel’s Theorem. The orange and blue sets aren’t equal exactly – we needed three lots of 1 to 9 for blue, while two were enough for orange. Instead, the claim here is that we know how much greater blue is than orange – specifically, it has exactly one more run of 1 to 9 than orange.

We start by removing overlapping cells:

But we can go much further than that. Since we made sure to include the palindromic lines, we know that the blue segments of each line cancel out with the orange segments of the line. This leaves us with no orange digits at all, and 9 blue ones:

This means we know the composition of the blue digits. They’re exactly one set of the digits 1 to 9. The sum of the digits 1 to 9 is 451, and the top three cages sum to 13, 12, and 14. Since 13 + 12 + 14 = 39, the remaining digits must sum to 6. We know our only options are the digits 1 to 9 once each, which means that there’s only one way to do it – putting 1, 2, and 3 in the bottom three cells, in some order.

After finding this break-in, Simon remarked:

It’s not tricky – well, actually, I’m going to qualify that, because that’s a silly thing for me to have said. It’s absolutely impossible if you’ve not seen it before. Well, not impossible – virtually impossible. Certainly, probably impossible to do it at speed. But if you’ve seen it a few times before, it’s second-nature.

This is the hallmark of problems solved by ontological remodeling. You don’t want to say they’re tricky, exactly, because the new framework makes them feel pretty approachable. But without the new framework, they’re basically impossible. Trying to describe the difficulty of these problems is something of a trap, because so much of the difficulty depends on the description. Instead, you need to play around with new forms of expression and see which patterns are easy to describe with those forms.

The sheer simplicity of Sudoku makes it an excellent example for how powerful ontological remodeling can be even when it seems obvious which form to represent things in. Even if your generating rules are all related to rows, columns, and boxes, they can interact in such a way that you need new language to describe the interaction. In fact, this has become something of a sport on the channel, with puzzles like this one that start by showing a computer solver unable to solve a classic Sudoku logically (only getting the solution through brute forcing all possibilities), then setting a human loose on the puzzle to discover the hidden trick.

That’s as pure and straightforward a demonstration of ontological remodeling as you’re ever likely to get. Sudoku is a game whose rules only invoke rows, columns and boxes, but a computerized solver can’t describe the solve path with rows, columns, and boxes. The interplay of those rules leads to higher order patterns, and those higher order patterns need higher order language. But precisely because this example is so straightforward, it can give us a deceptive idea of how easy it is to notice that your ontology is lacking. While the solvers are in an ostensible “conflict” with the setters, it’s a thin bit of fun narrative over what is fundamentally a collaborative process. The setter wants people to notice the clever logical leap they demanded of their solvers, and that means they need people to solve it. Getting the warning that the computer can’t solve it gives you the enormous clue that you’ve got to try something new. You may not know what immediately, but you’re looking for the right sort of thing.

In fact, while there are plenty of puzzles that are totally incomprehensible without set equivalence theory, Phistomofel’s theorem itself has more tentative origins. Phistomofel created a puzzle that was best solved using the ring, but only published it when he was sure it could be solved “normally”. The ring was not a finding that solvers worldwide spontaneously developed when faced with a single, staggeringly beautiful puzzle. Instead, the origin story is far more pedestrian: Phistomofel talked about the ring in a forum post. Discussion followed that took the specific ring and anchors and generalized them into more generic statements about 1-9 sets. Only after the concept had reached a critical mass of socialization were most authors comfortable making puzzles where set equivalence theory was truly required.

So as great as puzzles are for practicing these techniques, you should never mistake a setters kindness for the indifferent illegibility of real-world problems. There are uncountably many things we’re thinking about the wrong way, but nature has no Phistomofel on deck nervously ensuring that it’s at least kind of possible to do it through brute force or starting a discussion with a helpful example. All we get are a bunch of problems that seem impossibly hard. Nothing about the world gives us an easy rule for which things are actually irreducibly tricky and which things just need better forms of description. That’s why it’s so important to cultivate an intuitive appreciation for the power of ontological remodeling. The reason this technique is so under-discussed and under-used is precisely because there’s no well-defined way to tell that it’s needed for a given problem. Individuals just have to take that leap of faith themselves, come up with a new way of seeing, and then demonstrate that problems that were once intractable are suddenly workable. We call those people “visionaries”, and we sure could use some more of them these days.

Thanks to Simon, Philip Newman, and Sam Cappleman-Lynes for reviewing a draft of this article.

Subscribe now

As always when something is a prerequisite for itself, you have to proceed in a spiral. An approximate understanding of a small part of the subject makes it possible to grasp more of it, and thereby to revise your understanding of the initial beachhead. You need repeated passes over the topic, in increasing breadth and depth, to master it.

-David Chapman, Ontological Remodeling

I want to talk about the modern crisis of meaning. With our powerful computers and modern scientific techniques and access to all sorts of media, why is it so hard to figure out what’s actually true? I think a major reason is a lack of emphasis on representation: that is, how we choose to describe the world in the first place. We evaluate effective representations and ineffective representations side-by-side without distinguishing between them, hoping that truthfulness for one maps to the other, and instead are left only with confusion. 

This is a hard thing to talk about, because a “representation” is a concept that I must represent effectively to you. The whole problem is that we’re not used to talking about representational issues, but talking about representational issues is an example of a representational issue. So I can’t describe the problem to you without solving it; it’s a prerequisite for itself. As such, we’ll take Chapman’s advice and proceed in a spiral, insinuating our way towards the subject rather than rushing it head-on. (I’ll also have to gently mislead you for a while, though I promise to fix it before the end.) 

We’ll start with just a single word:

Hrair: A great many; an uncountable number; any number over four.

It’s a word in Lapine, the language of the rabbits from Richard Adams’ Watership Down. Rabbits, Adams tells us, can’t count higher than four. It’s not that they only have memory for four things; Hazel, the leader of the novel’s band of rabbits, has more than four followers, and he knows all of their names. But rabbits never represent things in groups higher than four. Anything more than four is hrair, a lot. For example, when Hazel and another rabbit, Blackberry, are discussing a diplomatic mission to another warren, Hazel says:

“We’re agreed, then, that we ought to send an expedition to this warren and there’s a good chance of being successful without fighting. Do you want everyone to go?”

“I’d say not,” said Blackberry. “Two or three days journey; and we’re all in danger, both going and coming. It would be less dangerous for three or four rabbits than for hrair. Three or four can travel quickly and aren’t conspicuous: and the Chief Rabbit of this warren would be less likely to object to a few strangers coming with a civil request.”

“Everyone” is a concept Hazel understands just fine, even though more than four rabbits are part of the “everyone” that lives in his warren. But if Hazel was asked exactly how many rabbits went into this “everyone”, he couldn’t say. What matters to him here is that four rabbits can travel more carefully than hrair rabbits, whether hrair ends up being five or ten or twenty. For the case of “how many rabbits ought to visit the other warren?”, there’s no need to count higher than four.

We can imagine cases in rabbit-world where hrair isn’t sufficient. Suppose there’s a log that could be used like a see-saw to open up a new feeding ground, but requires at least eight rabbits sitting on one end to make it move. We tell Hazel to have his rabbits sit on one end to see what happens, and he responds “I’ve already tried having hrair rabbits sitting on one end, and nothing happened.'' We ask him to follow a certain instruction:

Get a group of hrair rabbits. Have each of them choose a different rabbit not already in the group. This is two-hrair.

Two-hrair ends up meaning “at least eight”. If Hazel’s band tries this and discovers that two-hrair rabbits are enough to move the log, he may well decide that two-hrair is a concept worth remembering.  Does this mean Hazel learned to count to eight? Not really. He can’t tell in advance whether he has two-hrair worth of rabbits, and he can’t reach the intermediate states of five to seven. But clearly, if Hazel remembers the trick of “when hrair isn’t enough, try two-hrair”, he’s got something he didn’t have before. If it’s not the process of counting that’s changed for him, what did?

The word we’re reaching for is ontology, a jargony philosophy term that can be more or less understood as “the list of things in a category” or “the list of answers to a question.” Before, Hazel’s list of options for “how many of a certain thing is there?” was this: {one, two, three, four, hrair.} Now, it’s this: {one, two, three, four, hrair, two-hrair.} This isn’t the same thing as a list of numbers: numbers are distinct, whereas a hrair group of rabbits might also have two-hrair’s worth of rabbits without anyone knowing it, because it wasn’t assembled through the pairing algorithm. In fact, two-hrair is a lot less general than the other entries on the list, because it can only apply to rabbits that are working together and following the same rule. One, two, three, and four are numbers, hrair is a numeric range, and two-hrair is a numeric range that can only apply to rabbits. 

The entries are conceptually quite different – but they’re ontologically similar, because they’re used the same way. Two-hrair is just a new answer that’s potentially available for questions of the form “how many—?”. Nothing says it has to behave like the other answers, or be available for all potential “how many—?” questions that come up. Take note of that asymmetry before continuing on: all numbers are answers to “how many—?” questions, but new answers to “how many—” questions don’t need to be numbers. 

Speaking of “how many—?” questions, let’s ask two:

Game A: Would you trade away four value-units for a 25% chance of having five value-units?

Game B: Would you trade away four value-units for a 25% chance of having one thousand value-units?

A lot of modern discussion of uncertainty works by asking these sorts of questions and finding answers that are as objectively defensible as possible. I don’t want to get into that, so I deliberately chose payoffs extreme enough that the answers are pretty obvious. Game A stinks, whether the value units are dollars or candy bars; Game B is incredible unless you only have a few value-units and will instantly die without them (vials of insulin, for example). Imagine we’re all sitting around a picnic table, coming to a quick consensus, when a couple of rabbits hop nearby. We want to get their input, so we translate our questions into Lapine:

Game A: Would you trade away four value-units for a 25% chance of having hrair value-units?

Game B: Would you trade away four value-units for a 25% chance of having hrair value-units?

They look at us with their confused little bunny eyes and ask, “Sorry, why did you describe the same game twice?” Lapine cannot represent the difference between Game A and Game B. Our work with two-hrair won’t help: metaphysical value-units aren’t the same thing as conscious and cooperating rabbits, so we can’t run the process to get to two-hrair. And anyway, Game B would still stink if it ends up only being an eight value-unit payoff.

If we want the rabbits to understand our question, we’ve got to teach them our sort of counting, an algorithm that can be continually re-executed to produce an infinite ontology: {1,2, 3….1000…}. What’s the incentive we can give them for learning our method of counting—the equivalent of the log and the new feeding ground we used to teach them two-hrair? Can rabbits get any use out of playing these games? In what situations would they actually be offered the chance to trade away four value-units to get either five or one thousand? What are the value units in question? Do Game A and Game B show up in their lives via sufficiently different contexts that we can just refer to those contexts without teaching counting at all? Maybe something like an aphorism: “bargain a sure thing for hrair with your friends, but never strangers.”

We can see that formal mathematical methods aren’t the right tool to fix the rabbit's understanding. Their difficulty is ontological, not logical: since Game A and B look the same to them, the application of any rabbit-scale formal method would yield the same answer. And given how different the two games are, a method that gives the same answer for both of them can’t be all that useful. Formal methods pre-suppose that nebulous reality has been described at the level of detail the method requires. It doesn’t matter how good your math is if you’re telling it to a rabbit.

If, for an audience of humans, I had gotten into the math of Game A vs. Game B, made their payoffs a little closer, used phrases like “expected value”, and maybe threw in an integral sign somewhere, it would have been a lot of work. So much work, in fact, that it’d be easy to imagine that pushing those numbers around is just a conscious representation of what our brains are doing subconsciously when we make decisions. But hopefully I’ve shown how a precise-enough understanding of the situation has to come first, before any sort of math. Formally solving uncertainty on what to do only works if you can sensibly describe what there is.

There’s another important point here, one that you might have started thinking about a couple of paragraphs back. I claimed that, in order to teach the rabbits our sort of counting, we’d have to ground it in their local context. But how did we learn our sort of counting? Many people are comfortable reading Game A and Game B as they are, without relating them to a local context. You don’t have to imagine dollars, or candy bars, or vials of insulin. The number “1000” feels meaningful even if it’s not 1000 of anything in particular. So saying “humans know how to count infinitely and rabbits don’t” is just kicking the can down the road as an explanation. The real difference is that humans are able to learn context-free information. Why is that?

Ignorance, A Skilled Practice, an essay written by a literal banana, offers us the frame we need to explore this further. It’s an essay concerning indexicality, a word that, like ontology, sounds awfully jargony but isn’t so bad once you get to know it. Indexicality is essentially the degree to which local context matters for a given concept, though I’ll let the banana describe in more detail:

“Indexical” is a word almost perfectly calculated to sap the morale of the reader and annihilate interest. I have seriously considered replacing it with the word “pointing,” used as a descriptor. Indexical statements are pointing statements: “I prefer this one,” “don’t do that,” “I made it for you.” These sentences have no particular meaning without reference to the situation in which they are produced. Physical pointing, as with an index finger or the lips or chin, may or may not accompany an indexical expression, but there is a sort of pointing-to-the-situation that occurs in all cases. I’ve decided to keep “indexical” for clarity, but keep in mind that indexical means pointing, in a literal and then in an extended, figurative way.

In the linguistic sense, an expression is indexical if it refers by necessity to some particular state of affairs. “This guy arrived just now” depends on the person indicated and the time of speaking; it is highly indexical. Compare “The Prime Minister arrived at 5:15 p.m.” This is less indexical, but notice that the identity of the Prime Minister depends on the country, and of course we don’t know anything about the circumstances or place of arrival from the text: 5:15 p.m., but in what time zone?

Extremely non-indexical expressions often appear as health or science headlines. These are pretty much the opposite of indexicality:

Stanford Researchers: Average Human Body Temperature Dropping

How Puberty, Pregnancy And Perimenopause Impact Women’s Mental Health

Why is air pollution so harmful? DNA may hold the answer

Predatory-journal papers have little scientific impact

Can a healthy diet reduce your risk of hearing loss? Here’s what the research says

Notice that these refer to people in general, and vague concepts in general. They take the form of objective knowledge that is true in general, for all cases, globally, universally. They “see through” to the ultimate truth of matters, unsullied by the messy realities of particular people and situations. The kind of knowledge that non-indexical statements presume to convey is timeless, and describes all of humanity or the world in general. Indexical knowledge, on the other hand, refers to specific situations, times, people, and interactions. It does not purport to apply timelessly, in general, or to all people.

Indexical statements point at specifics; non-indexical statements describe generalities. “The number 3 is larger than the number 2” is non-indexical, and true in a general sense. “My two cats put together are heavier than your three cats put together” is highly indexical, and true for specific values of “my two cats” and “your three cats,” overriding the general “truth” that 3 is more than 2. “Average body temperature” is a non-indexical concept; but to guess at it, researchers can only measure the indexical body temperatures of certain people at certain times. Rabbits stay in the realm of the indexical, while humans are able to trade in non-indexical statements. Or, at least, some humans. Back to the banana:

To illustrate that global knowledge is a game, consider a story about Alexander Luria, who studied illiterate Russian peasants and their semi-literate children. Consider especially this version of the story, prepared in the 1970s to provide morale and context to reading teachers (John Guthrie, 1977). Essentially, Luria discovered that the illiterate, unschooled peasants were highly resistant to syllogisms and word games. The adult peasants would only answer questions based on their own knowledge, and stubbornly refused to make deductions from given premises. “All bears are white where it is snowy. It is snowy in Nova Zembla. What color are the bears in Nova Zembla?” “I don’t know, I have never been to Nova Zembla.” Children with only a year or two of education, however, were easily able to engage in such abstract reasoning. They quickly answered the syllogisms and drew inferences from hypothetical facts outside of their own observation.

In this story, I argue, Luria’s peasants are indexical geniuses, who refuse to engage in unproven syllogistic games. They are not interested in a global, universal game. Their children, however, are easily introduced to this game by the process of schooling and literacy.

The Russian peasants, like the rabbits, want to keep their observations grounded in context. Whether they’re worried about a trick, too proud to risk giving a wrong answer, or just plain suspicious of Luria, they don’t want to suppose the existence of global, non-indexical concepts like “all bears are white where it is snowy” (never mind things as abstract as Game A and Game B!) I’m not quite as adamant as them—I think there can be a lot of value in drawing inferences from unobserved information. But it’s true that both human beings and rabbits are highly indexical beings, who perceive this, do that, here and now. If you want to use non-indexical findings in your indexical life, you have to make sure they actually fit your context.

From our perspective, rabbits and peasants have some extra work they need to do before they can talk about these formal games like we do; they need to be able to become more comfortable with abstractions and language that doesn’t refer to concrete things out in the world. If you ask the rabbits and the peasants, though, we have an extra step we need to do after describing our formal games. Because we let ourselves play around with concepts that we couldn’t actually point to, we have no guarantee that our findings will actually come up in our lives like they do for rabbits and peasants. Non-indexicality lets us take a sort of loan of meaning, playing with tools that aren’t available in the here and now to broaden the ways we can think about things. But that loan has to be paid off eventually through correspondence work showing that the abstract objects you were acting on are similar enough to what actually exists in a particular situation for the finding to be valid. Otherwise, you’ve just been playing a meaningless game that has no bearing on actual reality.

I would argue that rabbits and Russian peasants are not indexical geniuses but indexical misers, unwilling to take on any sort of meaning-debt. To speak of the white bears in Nova Zembla is to become beholden to two abstracts—“All bears are white where it is snowy” and “it is snowy in Nova Zembla.” How would you pay down that debt? Bears are real animals, snow is real, and Nova Zembla is a real place: they aren’t just counters in a logic puzzle. It’d be a big trip to go to Nova Zembla, and maybe it’s only snowy part of the year. What if you go at the wrong time? Who can prove anything about “all bears where it is snowy”, anyway? Has someone been to every snowy place and met every bear? What if the non-white bears were asleep? Do they have a checklist that gets updated every time a new bear is born? Why deal with any of this shit when you could just say “I don’t know, I have never been to Nova Zembla” and get on with your life?

Just like monetary debt, allowing yourself to take on meaning-debt temporarily can let you access new heights and then pay the debt down and end up somewhere better. Every time a novel theory in physics is confirmed by experiment, meaning-debt proves its worth. But also like monetary debt, meaning-debt can ruin your life if you’re not careful, and you’re not exactly wrong to decide it’s not worth the risk. The peasants may be constraining their range of possible thoughts by focusing solely on the here and now, but they’re also guaranteeing that the things they think about will have an impact on their actual lives. If you develop formal theories without checking for heres and nows that behave in the way your theory describes, you may be throwing more and more of your time into a compounding interest hole of meaninglessness. 

Statements about value-units are non-indexical, which just means that we’re not pointing at anything in particular when we talk about them. By contrast, if I was deciding whether to trade four french fries now for a 25% chance to get one thousand french fries for lunch tomorrow, that would be a highly indexical decision, because it’s drenched in my local context. It would also be a highly indexical decision to trade four grand pianos now for a 25% chance to get one thousand grand pianos tomorrow. But maybe my answers for those two are different—I’ll take the gamble on the fries because I’m not hungry now anyhow, but I hardly have anywhere to put the four grand pianos while I sell them, never mind one thousand.

This difference shows us the meaning-debt that we incur from non-indexical phrases like “value-units”. If we want to use our findings from Game A and Game B in our lives, we first need to do correspondence work to find whether french fries behave enough like value-units that the findings are applicable, and the same for grand pianos, and the same for anything else. In one sentence: there are possibilities we can conceive of solely because our ontologies can include things we can’t directly point to, but this power comes with the debt to make them point to something later.

II.

Our next step is to see the meaning-debt the peasants were afraid of. What do problems of representation look like? We’ll turn to humorism as an example. Some ancient medical theorists thought that most illnesses could be explained by an excess or deficit of the four humors: {blood, phlegm, yellow bile, black bile}. We’ll imagine a Hippocratic hardliner who thinks this explains everything. That is, every single illness is caused by either too little or too much blood, phlegm, yellow bile, or black bile. (I don’t know how many historical physicians actually believed in humorism this strongly, so don’t treat this illustrative example as an especially accurate historical recounting. Also, it’s going to get worse.)

We know now that humorism isn’t true. Illnesses can be caused by viruses or bacteria or environmental contamination or all manner of other things. But the “humors” aren’t entirely arbitrary concepts, either. The illness is caused by whatever it’s caused by, different illnesses cause different symptoms, and if the humorist diagnostic criteria consistently classifies the same symptoms the same way, the humors “listen to” the true causes in a partial, indirect way. 

Suppose a patient named Artemis has some set of symptoms—maybe a deep cough and a splitting headache, or whatever. All of the ancient physicians she runs into agree that she must have an excess of phlegm with the other humors in balance (“phlegmatic” for short). What does the statement “Artemis is phlegmatic'' actually mean, knowing what we know now about medicine?  It’s not pointing at Artemis’s excess of phlegm, because we know now that bodies don’t work this way and there’s no actual excess of phlegm to point to. But it’s not exactly false to say “Artemis is phlegmatic”, because it has a meaning to the people who say it and Artemis is a patient they all agree qualifies. It’s indexical, because it’s pointing to a real thing, but that doesn’t mean it’s true. We’ll need to understand this fussy-sounding distinction to go farther, so we’ll inject enough context that the difference becomes clear.

Let's say that modern medical historians have determined that just two diseases caused people to be phlegmatic in the ancient world: city cough (caused by the bacterium c. urbanicus) and country cough (caused by the bacterium c. pastorilus). You catch city cough in close proximity to other people, and it’s cured by simply increasing intake of sugars and meat—the traditional cure was drinking mulled wine and eating bull testicles—but gets dramatically worse if you have any contaminated food or drink. Country cough gets in your lungs from disturbed soil, and the solution is to beef up your microbiome—traditionally done by drinking brackish water and eating a handful of grave dirt—but it spirals out of control if you eat and drink normally, and god help you if you eat some bull testicles. 

Remember that these (imagined) Hippocratic hardliners don’t think of “city cough” or “country cough” as ailments. They think that a phlegmatic patient has an excess of phlegm and that’s the sole reason for their problems. It just so happens, though, that doctors tend to either be city doctors or country doctors, and tend to see cases of city cough or country cough but not both. The city doctors have their regular hookups for bull testicles, country doctors know the perfect graves to skim dirt off of, and the literature tactfully equivocates with “Some doctors recommend mulled wine and bull testicles to cure an excess of phlegm, while others prescribe a course of brackish water and grave dirt.” 

Artemis, though, is a bit of an interesting case. She lives on the outskirts of town, spending a fair amount of time in the field but also making frequent trips to the city. So when she wanted to get multiple opinions, she ended up asking one doctor who works in the city—Dr. House—and one who works out in the country—Dr. Field. Dr. House naturally suggests the wine and ball combo, to which Dr. Field indignantly replies, hey, I agree that she’s phlegmatic, but that’s precisely WHY the extra food and drink is a huge problem—we need to get her on brackish water and grave dirt, stat.

When Dr. House says “Artemis is phlegmatic,” he’s relating her to the patients he’s seen previously that got better when they drank mulled wine and ate bull testicles. Why did they drink mulled wine and eat bull testicles? Because he told them to. Why did he tell them to? Because they seemed a certain way. Artemis also seems that certain way, and so Dr. House wants to treat her the same way he treated the other patients, by giving her mulled wine and bull’s testicles. If Artemis seemed a different way, he wouldn’t expect mulled wine and bull testicles to help.

When we ask if something is true or not, this is ultimately what we want to get to: the difference in outcomes for an intervention. Mulled wine and bull testicles work great for some patients and do nothing or are actively harmful for others. Dr. House wants to say “Give mulled wine and bull testicles to the patients for whom they will help, and don’t give them to the patients for whom they will not help.” If he had never met another doctor in his life, and felt no need to justify himself to any patients, he could hold this idea in his head without needing explicit concepts. His personal experience would be enough to have an intuition of “the sort of patient” to give mulled wine and bull testicles to, and he could use that as his guide. This is a normal and very human thing to do. 

But that doesn’t cut it when you want to socialize this difference, whether it’s for formal diagnostic rules or just idle banter with colleagues. Dr. House wants to compare his findings with his peers. He wants to explain his reasoning to his patients. He wants to document his actions for future reference. To do any of these, he needs to be able to point at the difference in interventions. But how can he do that? He can’t describe every single thing he noticed in each patient he’s seen. And he can’t point at c.urbiancius itself, because he doesn’t know what that is. He needs a named concept he can point to when he means “look at this difference between patients who got better with mulled wine and bull testicles and those who didn’t.”

This is where the tangle of meaning happened. Dr. House and Dr. Field are trying to point to two separate differences—the difference in outcomes for mulled wine and bull testicles vs. the difference in outcomes for brackish water and grave dirt. They each needed a way to describe that, and were used to the background framework of humorism saying that symptom profiles are all you need to describe the differences in interventions. So they both pointed to “phlegmatic”, without adding in the additional context of city vs. country. In the act of formalizing how one points to phlegmatic, they inadvertently destroyed the why of pointing to phlegmatic. It’s not that they were wrong to try, since the differences they were pointing to can be used to improve human health when used correctly. But their framework of “what constitutes an explanation for a difference?” was too rigid, and so patients that look the same in their framework may not respond the same way to a given intervention.

Let’s repeat our summary of part I: 

There are possibilities we can conceive of solely because our ontologies can include things we can’t directly point to, but this power comes with the debt to make them point to something later. 

Now, we’ll reframe it:

There are differences we can only point to because our ontologies can hold abstract concepts, but pointing to these concepts comes with the obligation to make sure these differences are really there when we point at the concept.

This is the meaning-debt the indexical misers were desperate to avoid. In formalizing the idea of “phlegmatic” enough that the two doctors agree that Artemis is phlegmatic, it melded together the differences that each doctor imagines they’re pointing to. “Artemis is phlegmatic” is indexical in the sense that “How do you point to the concept ‘phlegmatic’?” has a shared understanding among physicians. But while it’s indexical, it’s not completely meaningful, because not all uses of the word are pointing at the same difference.

If their language had been refined enough to say “city phlegmatic” and “country phlegmatic”, then the meaning-debt could potentially be paid off, since “city phlegmatic” would be pointing at c. urbanicus and “country phlegmatic” would be pointing at c. pastorilus. Alternatively, if each doctor had stuck squarely to their own terrain and never interacted with an in-betweener like Artemis, it also would have worked out fine. “Phlegmatic” would be pointing to separate differences in the minds of Dr. House and Dr. Field, but each individual use of the word “phlegmatic” would be pointing to what it means to point to.

So it’s not the word “phlegmatic” itself that goes into meaning-debt or not. Instead, it’s the difference being pointed to that incurs the debt when you conceptualize it and try to make something in the world point at it. You pay this debt off when your conceptualization successfully points back to your difference, and the payoff is being able to use that difference in your real, indexical life. If Dr. House had demurred and said “I don’t know, I’ve never treated a patient outside the city center”, then “phlegmatic” would be simultaneously a meaningful word for him (because it always points to a specific difference, the difference in outcomes of interventions for patients with c. urbanicus, when he uses it) while being a bankrupt word for Dr. Field, who overstepped his boundaries too far by presuming the lived experiences of his previous patients applied to Artemis.

In other words, representations are inevitable, and the challenge is to make sure the representations are meaningful with respect to the differences in the world you’re trying to interact with. “Phlegmatic” as a representation carries with it some degree of unrepresented baggage that Artemis won’t be aware of when she hears “you’re phlegmatic”. The doctors who are trying to explain real differences have the job of trying to hold that unrepresented baggage constant enough that their past experience of exploiting those differences applies the same way to the same representation. Since words are just tools that one can use well or poorly, different people may incur different levels of meaning-debt when using the same representation. Once you get practice thinking this way, you’re able to be much more precise about what’s actually going wrong when looking at failures of representation.

Practice! That’s what we need. Here, I’ll send you back to the ancient world using my time machine. Plus side: I’ve also given you a solar-powered laptop capable of running modern machine learning models, so you have raw computational power on your side. Minus side: they think you’re a god of healing and believe they’ll be destroyed if they ever see you or hear you speak, so they’ve locked you in a room. They slide a clay tablet under the door that includes the patient’s humors and a list of possible interventions—mulled wine and bull testicles, brackish water and grave dirt, and many others besides. You mark the tablet by the intervention you want them to do and send it back. Later, they send an offering and another tablet showing how the patient ended up faring.

As you get more data to train your machine learning model, you’ll be able to improve outcomes and eventually choose the “best” outcome for each humor state. But it’d fall way short of modern medicine, because even with modern computational power, you wouldn’t have modern ontological power. You don’t know whether these patients live in the city or the country, because your worshipers think the humors explain everything and don’t put other details on the tablet. And your model will be brittle, because it doesn’t have any theory to inform it when conditions change. If cities expand their borders, so that more patients get city cough and fewer patients get country cough, you won’t be able to predict that in advance. You'll just notice that brackish water and grave dirt start doing worse and worse for phlegmatic patients and phase it out, chasing the trend after the fact.

How would this look from the perspective of the ancient doctors who are tracking the success rate of their god? They probably don’t know that they’re being roadblocked on their representational vagueness. Ontological difficulties are pretty subtle, especially in systems as complex as the human body. It’s extremely easy to be satisfied with a probabilistic conclusion like “if the patient is phlegmatic, then mulled wine and bull testicles will save them 75% of the time” and assume that’s the whole answer, even if they could save everybody by distinguishing city cough from country cough.

What if I had sent a Russian peasant through my time machine instead of you? What does an indexical miser do when they see a tablet come under that door? “I don’t know, I’ve never met this person,” and slide the tablet back without a mark. Eventually the doctors assume that their god is displeased and stop sending the tablets. Was this the responsible thing to do? Probably not. After all, the humors at least partially point to something real. Even if you can’t get to 100% by distinguishing between city cough and country cough, it’s at least better than random chance, right?

This argument is a common one when people are faced with representational issues: you can only do the best you can with the data you have. I want to push back strongly against that framing. Because in order to make the indexical miser look unreasonable, I had to create a completely locked door, a strictly one-way flow of information, a representation of a patient you can take or leave but never change or look at in more detail. In real life, of course, you can open the door.

That’s the moral of the story. I’ll repeat it.

In real life, you can open the door.

Whenever you’re given a word you think isn’t pointing to a difference, you can ask: what will you do differently depending on whether this patient is phlegmatic or not? You can ask people to give you stories instead of data. You can try recording things in different ways. If a modern epidemiologist was sent back, but allowed to open the door, their most powerful technology would be their checklists, asking more relevant questions than anyone back then would have thought to ask. Do you use a wood stove? Do your children spend significant amounts of time in the basement? Do you use earthen-ware pottery? And eventually, they’d get to: do you live in the city, or the country? When you see the relative impact of mulled wine and bull testicles vs brackish water and grave dirt, split out by city vs. country, you simply don’t need modern computing power to figure out what’s happening. It’s plain as day, right in front of you.

Let me pull you back to the present day and show you something. {Four, hrair, two-hrair} – remember how these were different conceptual things that were only similar ontologically? Well, here’s two words from our modern ontology of diseases. One is cholera, here defined from the World Health Organization:

Cholera is an acute diarrhoeal infection caused by ingestion of food or water contaminated with the bacterium Vibrio cholerae…The majority of people can be treated successfully through prompt administration of oral rehydration solution (ORS)… Severely dehydrated patients are at risk of shock and require the rapid administration of intravenous fluids.”

The second is fibromyalgia, here defined by the National Institutes of Health:

Fibromyalgia is a long-lasting disorder that causes pain and tenderness throughout the body. It also can cause you to feel overly tired (fatigue) and have trouble sleeping. Doctors do not fully understand what causes fibromyalgia, but people with the disorder are more sensitive to pain… Treatment may include exercise or other movement therapies, mental health and behavioral therapy, and medications.

Cholera and fibromyalgia are similar ontologically, in that a doctor might say “you have {cholera/fibromyalgia} and I’m going to prescribe a treatment based on that.” They’re very different conceptually, though. Cholera is caused by a specific bacterium, vibrio cholerae. It leads to consistent symptoms that clear up with consistent treatment. We’re pointing at the difference between people who immediately get better when given fluids and people who don’t, and saying that one way you could name that difference is “cholera.” How do we make that indexical? We have microscopes now. We can literally point at vibrio cholerae. The word “cholera” pays off its meaning-debt when we point and say “this vibrio cholerae, here and now.” 

But what is “fibromyalgia” pointing at? An understanding that many people are experiencing similar symptoms. It behaves more like “phlegmatic” did, where patients described in the same way might respond dramatically differently to the same treatment. It’s more useful than not having a word, because patients with fibromyalgia are more similar to each other than they are to people without fibromyalgia. It’s a hook we can use to begin to investigate a difference. “Fibromyalgia” describes uncertainty, but doesn’t terminate it in the way that “cholera” does. Could the word “fibromyalgia” be made more useful? How much is studying the impact of various treatments on “fibromyalgia” going to serve us until we clarify our understanding of “fibromyalgia”? Can you truly compare the outcomes of two studies on “fibromyalgia” if one is city fibromyalgia and one is country fibromyalgia?

Imagine future scholars looking back at you. What issues are we trying to fix with our current verbiage that will have people from the future thinking, “Well, if I traveled back to their time, I wouldn’t show them our superior computational ability—I’d show them our superior ontology”? Will people in the year 2100 use the word fibromyalgia at all? Or will “fibromyalgia” be replaced by a number of words for different causes of pain and fatigue, each with their own corresponding treatment?

And when you do have a word that seems like it’s not pointing properly—how should you think about it in the meantime? How do you make your strategies for uncertainty account for the fact that even your representations are uncertain?

III.

What we need now is an example of effective decision-making that avoids our pointing problems. If you want to investigate decision-making in areas of high uncertainty and urgency, it’s hard to beat talking to firefighters. In Sources of Power, 20th Anniversary Edition: How People Make Decisions, researcher Gary Klein and his team recount the surprising conclusions they came to after studying firefighters:

We thought this hypothesis—that instead of considering lots of options they would consider only two—was daring. Actually, it was conservative. The commanders did not consider two. In fact, they did not seem to be comparing any options at all. This was disconcerting, and we discovered it at the first background discussion we had with a fireground commander, even before the real interviews. We asked the commander to tell us about some difficult decisions he had made.

“I don't make decisions,” he announced to his startled listeners. "I don't remember when I’ve ever made a decision.” 

For researchers starting a study of decision making, this was unhappy news. Even worse, he insisted that fireground commanders never make decisions. We pressed him further. Surely there are decisions during a fire—decisions about whether to call a second alarm, where to send his crews, how to contain the fire.

He agreed that there were options, yet it was usually obvious what to do in any given situation. We soon realized that he was defining the making of a decision in the same way as Soelberg’s students—generating a set of options and evaluating them to find the best one. We call this strategy of examining two or more options at the same time, usually by comparing the strengths and weaknesses of each, comparative evaluation. He insisted that he never did it. There just was no time. The structure would burn down by the time he finished listing all the options, let alone evaluating them. [Emphasis mine.]

Not making any decisions about how to fight fires is something that me and this fireground commander have in common. But if I was in charge of leading a fire response, the outcome would be a lot worse. The fireground commander is the highest ranking person on the scene of the fire, and must have been chosen to do something relating to fires better than I would. If it’s not decision-making, what is it?

Let’s imagine that there’s a flowchart in the fireground commander's head declaring exactly what to do in any circumstance. “If it’s a big fire, only attack it from outside with hoses. If it’s medium size, go in and evacuate people, but only if you can do it in five minutes” —those sorts of rules. When the fireground commander says “he doesn’t make decisions,” he’s saying that the flowchart never invokes “decide between these options.” You always just follow what the chart does. If the fireground commander managed to get the flowchart out of his head and onto paper, would I then be able to follow it and fight fires as well as he does?

Let’s try. When does a fire go from “medium” to “big”? I have no clue. I can observe a here and a now, and I can have a completely well-defined flowchart of decision-making, but in order to make that here and now talk to the flowchart, I have to do correspondence work to make sure I’m mapping the world to the right part of the flowchart. This is the exact inverse of the problem that Drs. House and Field struggled with. They had a systemic, well-agreed upon method of determining which patients were phlegmatic, but they had two different flowcharts in their heads that were thinking of two differences in interventions when they said “phlegmatic”. The fireground commander’s representations are meaningful with respect to the differences: big fires are the ones you don’t get in the house for. Instead, the problem is looking at the real world with its infinite supply of detail and knowing which concept to point to.

In some domains, this isn’t such a big deal. In a chess game, for example, the rules assert that the detail of the world can be abstracted away, that a king piece made of young wood is the same as a king piece made of old wood. In fires, it’s entirely possible that the age of this particular wood is going to matter a great deal, as will the moisture in the air, the ambient temperature, the construction of the floor—you get the idea.

Still, it feels like we should be able to get the best of both worlds here. Let’s figure out an objective, agreed-upon way to pick a state for a fire, just like Drs. House and Field had an agreed-upon way to pick a humor for a patient. And we’ll make sure those states map to the fireground commander’s decision-free flowchart. As long as you have those two guarantees, you don’t need the fireground commander at all. In fact, you could fight the fire with your eyes closed! Just have someone describe the fire to you in the agreed-upon way, run through the flowchart in your head, and tell them the answer.

Fighting fires with your eyes closed sure would be nice. We should probably get a more concrete example of how fireground commanders point to see how we’d go about doing it. This time I don’t even have to make up the scenario! Let’s look at this full-length anecdote from Sources of Power:

Example 4.1 The Sixth Sense

It is a simple house fire in a one-story house in a residential neighborhood. The fire is in the back, in the kitchen area. The lieutenant leads his hose crew into the building, to the back, to spray water on the fire, but the fire just roars back at them.

“Odd,” he thinks. The water should have more of an impact. They try dousing it again, and get the same results. They retreat a few steps to regroup.

Then the lieutenant starts to feel as if something is not right.

He doesn’t have any clues; he just doesn't feel right about being in that house, so he orders his men out of the building—a perfectly standard building with nothing out of the ordinary.

As soon as his men leave the building, the floor where they had been standing collapses. Had they still been inside, they would have plunged into the fire below.

“A sixth sense,” he assured us, and part of the makeup of every skilled commander. Some close questioning revealed the following facts:

•    He had no suspicion that there was a basement in the house.

•    He did not suspect that the seat of the fire was in the basement, directly underneath the living room where he and his men were standing when he gave his order to evacuate.

•    But he was already wondering why the fire did not react as expected.

•    The living room was hotter than he would have expected for a small fire in the kitchen of a single-family home.

•    It was very quiet. Fires are noisy, and for a fire with this much heat, he would have expected a great deal of noise.

The whole pattern did not fit right. His expectations were violated, and he realized he did not quite know what was going on. That was why he ordered his men out of the building. With hindsight, the reasons for the mismatch were clear. Because the fire was under him and not in the kitchen, it was not affected by his crew’s attack, the rising heat was much greater than he had expected, and the floor acted like a baffle to muffle the noise, resulting in a hot but quiet environment.

This incident helped us understand how commanders make decisions by recognizing when a typical situation is developing. In this case, the events were not typical, and his reaction was to pull back, regroup, and try to get a better sense of what was going on. By showing us what happens when the cues do not fit together, this case clarified how much firefighters rely on a recognition of familiarity and prototypicality. By the end of the interview, the commander could see how he had used the available information to make his judgment. 

This anecdote is often repeated outside the pages of Sources of Power to describe the “power of intuition.” But what do we mean by intuition, anyway? When I walk into a room and notice that it’s quiet, that’s not intuition, just observation. The impressive part is that the fireground commander recognized the situation was atypical without knowing why, which allowed him to act on a fact (the room was too quiet for how hot the fire was) without actually being able to describe the specific fact he was acting on. He didn’t point to “too quiet”; he pointed to “not normal,” and only figured out the “too quiet” part reflecting after the fact. This almost seems like cheating after all of our hard work. We’re trying to figure out the right way to call a fire “too quiet,” because in our flowchart, “too quiet” points to “leave the building,” and it’s extremely important to pay attention to that difference when it’s relevant. But this fireground commander jumped straight to the difference of “not normal, so leave the building” without mucking around with concepts at all. How the hell are we meant to learn from that?

Let’s turn our question the other way around. We want to understand what the fireground commander actually did in representational terms. What’s the simplest possible example that can give us a sense of that? Suppose I wrote a companion book to Sources of Power called Sources of Miniscule Amounts of Power, and I opened with this case study:

 Example 1.1 The Wobbly Table

Gregory puts his mug on the table and notices that the table wobbles when he does so, and his drink seems to be moving slightly away from him. He moves his drink to a different table nearby and then looks under the first table. The far opposite leg is a bit shorter than the others. Luckily, Gregory has a copy of Sources of Power, 20th Anniversary Edition: How People Make Decisions on hand. He puts it under the short leg, then grabs his drink and places it back on the table. The table stops wobbling and his drink is still.

It’s not the sort of thing we’d call intuition, and it’s not the sort of thing that impresses us, but it is the sort of thing that the fireground commander did. Gregory didn’t immediately know why the table was “wrong,” but his vast amounts of lived experience with flat tables let him recognize that the table wasn’t flat and act on it prior to having any particular concepts in mind. Gregory has a mental image of a “normal table” just like the fireground commander has one of a “normal fire,” and he’s attuned to that difference even when he can’t immediately state a cause.

So why is Gregory’s story less impressive than the fireground commander’s? Let’s ask ourselves: what factors can make a table violate our expectations of normality? Well, the legs could be different lengths, or the floor itself could be slanted, requiring an artificial lengthening of one or more legs. That’s basically it! Of course, the infinite detail of the world could theoretically introduce another cause. Maybe the table actually is flat, and the wobbling and drink movement were due to a small earthquake in an area where they aren’t common. In practice, though, that sort of thing almost never happens. We’re comfortable holding an “idealized form” of a table in our heads that says “a non-normal table is non-normal because of irregularities in leg length, or a slant of the floor below.” Basically every person on planet Earth has a large body of lived experience telling us that this almost always works as an explanation. In the times when it doesn’t, we’ll be confused for a bit, hear about the earthquake later, and then have a fun story.

Both the fireground commander and Gregory noticed that something was wrong, but Gregory already had an idealized form telling him exactly what sort of table is “normal” and what sort of causes to look for if it’s non-normal. The infinite sources of detail in the world—the wood, the temperature, the moisture in the air—never came into the picture. The fireground commander doesn’t have that same guarantee. His idea of a “normal fire” is tentative, subject to any number of things that could violate it without him having a list of what those things are. The risk and ambiguity that the fireground commander must grapple with is what makes the first story so compelling and the second one so boring, and the fact that he can recognize a “normal fire” enough to perceive a deviant one is what he possesses that I don’t.

So, how do you fight a fire with your eyes closed? You don’t. Because instead of pointing at a particular fire and saying “medium” or “big,” you might need to say “not normal” and improvise. Going by a flowchart is only possible if you can explicitly list out every single factor that might be relevant ahead of time. For tables and other deliberately constructed things, you can have a small-world idealization that works so often that you can use the flowchart and basically never be punished for it. When you’re dealing with complex, interactive, dynamic sorts of phenomena (like fires), the real world is a lot more likely to intrude on your simplified model of how it works and force you to consider new details.

I mentioned at the start that I’d have to gently mislead you for a while, and now seems like the time to come clean. This is an essay about problems with representation, but throughout the first two sections, I made it seem like they were specifically problems with words. That’s not what’s happening here, though. Look at the word “quiet.” We all know what difference “quiet” is pointing to. In fact, we could exactly measure the volume of every fire if we liked. Firefighters don’t do that, because it doesn’t matter very much—a quiet fire is often a safer fire. The specific fire in The Sixth Sense was dangerous because it was too quiet given the other factors, and therefore not normal. But just as it required intuition to start with a fire and realize you needed to include the volume, it also requires intuition to start with the volume of a fire and figure out which other factors you need to determine whether the volume is too quiet given those other factors.

It’s not just the pointing of an individual concept we’re worried about, but also the ensemble of concepts we’re choosing to consider. The linguistic battles we were fighting in section II are only part of the story. It’s not enough to make sure an individual concept is well-defined, because we have an infinite number of concepts (well-defined or otherwise) we can choose to consider when evaluating any situation, and we need to use our finite brains to pick enough of them to make a good decision. To make this work, we also need to develop a sense of normality to recognize when the concepts we’re using aren’t enough to explain what’s happening. (Think of Gregory and his table, noticing “not-flat” before noticing why. The fireground commander’s sense of “normality” can be thought of as knowing which fires are “flat,” i.e. imitate the usual model, and which fires have a factor that’s out of balance with expectations.) You can’t list every potential factor ahead of time, so you just need to keep your eyes open and look for hints that it’s time to involve another concept in your representation. 

In other words: don’t rely on a strictly one-way flow of information, a representation of a fire you can take or leave but never change. I’m just talking about opening the door again. The work to make “phlegmatic” point at a meaningful difference and the work to intuit that your model of a situation is missing a concept may seem completely different, but they’re both dealing with the same problem of fixed representation. The answer in either case is to allow your representation of what there is—what kind of patient? What kind of fire?—to change with interaction so it can capture the differences that matter.

Don’t try to fight a fire with your eyes closed. Don’t try to diagnose the patient from behind the door. The same moral to both stories, but they reveal different facets of the problem. When you imagine uncertainty like a fire, you see that adding in additional factors was an automatic, intuitive process that happened without conscious effort because it was needed to address an urgent break in normality. It’s important, though, for you to also keep thinking about the tablets coming under the door, because representational problems aren’t simply time and resource constraints forcing us to make approximations. Representations annihilate detail, with the city phlegmatic and country phlegmatic patients having identical tablets, and it’s not anything that you can fix with arbitrary computing power and arbitrary time to think. You need to be able to go back to the well of detail and gather the factors you need. Meaning is a product of interaction, not something that springs from a dead dataset. 

To act on anything in the world, you have to represent it in a way where you can imagine which actions to take. We develop this capability before language, as embodied cognition and intuition about physical objects. And part of this capability is the ability to notice when your representations break down and need to be amended. We do this all of the time without needing to consciously think about it. If you were in Gregory’s place for The Wobbly Table, you would have handled it just as well without needing to be taught.

That’s why I had to start with the door and not the fire. You need to see the door in your mind's eye, a great imposing thing between you and the phenomena. You need to appreciate how utterly impossible it is to reach true understanding without opening it. Only once you have that model should you think about uncertainty as a fire that you interact with to pull meaning from. We have tools to fix representational problems, but they’re local tools that have to touch the infinitely detailed world to work. You can’t use them on the tablets that get slid under the door.

More and more, though, we’re being asked to try. We’re assaulted by a flurry of charts and articles and studies and being asked to draw meaningful conclusions from them. (If we’re not just told to “trust science!”, as though there was a single correct way to interpret every finding.) That’s where this story has to end. We intuitively know how to open our eyes in our everyday lives, but how do we open the door between us and the myriad of static representations that modern life puts before us?

IV.

Throughout this essay, I’ve had to use strange words that most people aren’t used to seeing, contrived and specific examples, misdirection and repetition. The reason I’m using all of these tricks is because we don’t have well-known, socially negotiated ways to describe these representational problems. This is the first and most urgent lesson you need to come away with. This is happening everywhere, all of the time, and it’s happening largely because our concepts around indexicality and meaning are dramatically underdeveloped. No one is driving the bus and making sure meaning gets to where it’s going, and no institutional authority can be trusted to deliver these answers.

There’s no global answer to these questions because all meaning is interactive and contextual, but that’s not the same thing as saying all meaning is relative and personal. The world has real patterns waiting to be unearthed. But to use those real patterns to drive action, you need to create an abstract concept to hold the pattern, then make a real situation point at the abstract concept accurately enough.

We know meaning when we see it, and for most of human history, that worked well enough. Never mind if the actual mechanics of representation are awkward to discuss and difficult to understand—as long as we can stop the table from wobbling and get out of the building when the floor collapses, that has been enough. For much of human history, most of the data we acted upon was held in our heads. Things that were externally recorded were often simple representations that were maintained by the same person who used them to make decisions, like a merchant's inventory and receipts. Data couldn’t help but be interactive and indexical, because it was bound up inextricably with human beings.

But the people involved wouldn’t think of it as “interactive” and “indexical” any more than the fireground commander thought he was making decisions. Figuring out the right way to represent something is an automatic human tool we use to get things done without worrying about theory. What counts as a chair in your house? Well, maybe an ottoman is good enough to perch on when you’re watching TV, but when the in-laws come around, only your chair-chairs count as chairs. A stump is a chair when you’re outside around the fire, but if your friend asked you to bring a chair to his housewarming party, he’d be pretty upset if you lugged a stump into his house. But your friend wouldn’t ask, “What are you pointing at when you say ‘chair’? Do you think your concept of chair is pointing meaningfully at the sort of things we’d like to sit on here? How much did you consider indexicality before coming to this party?” They’d just say “what the hell, man, you’re being ridiculous.”

There’s nothing wrong with just calling stump guy ridiculous! No one needs to draw on these deeper concepts of meaning to point out what went wrong here! We can all handle that problem just fine if we’re there to see it. But what about when we’re not? What about when we’re behind the door, and just given a static representation? Jeremy’s house has six chairs in it. But if he’s counting the ottoman, that’d better be for him and not the in-laws. And if one is a stump? That’d be ridiculous. It’s easy to give these proclamations from within Jeremy’s house, but annoying and high-effort to look at “HOUSE: Jeremy. CHAIRS: Six” and try to figure out from there whether there are actually six chairs or if some of the things being counted are ridiculous. So mostly, we just hope that the process of collecting the data doesn’t result in any ridiculous answers. But because data collection and analysis work can be made more and more automatic and scalable while the anti-ridiculousness work is much more manual (going to Jeremy’s house and seeing what the six chairs actually are), the balance is getting ever-more disturbed by modern norms of science.

Tal Yarkoni identifies this tension in his paper “The Generalizability Crisis.” It tackles the scientific “replication crisis”, where many dramatic findings in many fields don’t seem to happen when other people run the same experiment. Yarkoni attributes this to the step where we trade verbal constructs for statistical operationalizations:

Suppose I hypothesize that high social status makes people behave dishonestly. If I claim that I can test this hypothesis by randomly assigning people to either read a book or watch television for 10 minutes, and then measuring their performance on a speeded dishwashing task, nobody is going to take me very seriously. It doesn’t even matter how the results of my experiment turn out: there is no arrangement of numbers in a table, no p-value I could compute from my data, that could possibly turn my chosen experimental manipulation into a sensible proxy for social status. And the same goes for the rather questionable use of speeded dishwashing performance as a proxy for dishonesty.

The absurdity of the preceding example exposes a critical assumption that often goes unnoticed: for an empirical result to have bearing on a verbal assertion, the measured variables must be suitable operationalizations of the verbal constructs of interest, and the relationships between the measured variables must parallel those implied by the logical structure of the verbal statements. Equating the broad construct of honesty with a measure of speeded dishwashing is so obviously nonsensical that we immediately reject such a move out of hand. What may be less obvious is that exactly the same logic implicitly applies in virtually every case where researchers lean on statistical quantities to justify their verbal claims. Statistics is not, as many psychologists appear to view it, a rote, mechanical procedure for turning data into conclusions. It is better understood as a parallel, and more precise, language in which one can express one’s hypotheses or beliefs. Every statistical model is a description of some real or hypothetical state of affairs in the world. If its mathematical expression fails to capture roughly the same state of affairs as the verbal hypothesis the researcher began with, then the statistical quantities produced by the model cannot serve as an adequate proxy for the verbal statements—and consequently, the former cannot be taken as support for the latter.

Yarkoni’s point about speeded dishwashing is similar to our pointing problem with “phlegmatic”. Dishwashing aptitude clearly doesn’t point to any difference caused by any concept of “dishonesty”, but even things that have some superficial relationship to our idea of “dishonesty”—amount of eye-contact, or heart rate on a polygraph, or whatever—could have the same danger of meaning-debt if they’re trying to proxy for the reasons we care about “dishonesty” (can I trust them when they say how much my share of the bill was, or do I need to see a receipt?). Yarkoni calls this the “generalizability crisis”. We can view it as a subset of our representational crisis, since it’s the same problem of concepts failing to cleanly map to differences. Remember, though, that uncertainty is like a fire as well as like a door. The representational crisis is not just about individual low-validity concepts, but whether you’re looking at the proper ensemble of concepts. 

Let’s suppose Dr. House and Dr. Field each performed studies on the efficacy of mulled wine and bull testicles for phlegmatic patients and published them in modern-style scientific journals. House went first, testing with the city patients and finding a dramatic impact. Field published after, looking at country patients and finding no impact. This might be called a failure of replication, but we know it was actually a failure of representation. Dr. House’s work wasn’t replicable because Dr. Field was testing a different thing than Dr. House was, even if they both called their patients “phlegmatic.”

Just like with the fireground commander, these are problems that humans have experience catching, but only when they’re able to interact with the world and not just with dead representations of things. This story from Chemical & Engineering News is a great case study. A chemist publishes a paper about a supposedly “metal-free” reaction. But other chemists reading it know that palladium is stubborn and hard to completely get rid of. So they replay the methods of the paper and replicate the original results, but then try different things that remove contaminating palladium and show that the reaction doesn’t work anymore. That’s how this sort of verification has to work. Textual or statistical analysis won’t get you anywhere, since the whole problem is that the original paper represents itself as “palladium-free” and you need to have the experience to say: “Well, I’m sure they didn’t mean to have residual palladium, but that’s not the same as palladium-free.” 

In chemistry, the objects you’re working with are precise down to the elemental level. Even that isn’t enough to stop the problem of uncertainty being like fire! What it does mean, however, is that one chemist generally has a very easy time replicating what another chemist did. The original paper didn’t contain an accurate description of what there was (trace palladium), but the language of chemistry permits very accurate descriptions of what was done. So even if the paper itself is a dead representation that lives behind the door, the methods section allows someone else to open the door for themselves, stand where the original chemists stood, and try something new. 

The same doesn’t hold true for House and Field. Even if they had followed identical methods to the letter, they would still have gotten totally different results, because their base unit of “a phlegmatic patient” simply is not stable between experiments in the same way as a chemical compound. Field doesn’t open the door to the same place in which House stood: he opens the door to country patients and tries to replicate a finding from city patients. So of course subject areas that work with highly contextual objects, such as people and ecosystems and cities and ideas, will have problems with replication! And it’s not something that can be caught the same way that the chemists caught the palladium issue, because you can’t run an experiment on the same patients at the same moment in time. You can trust palladium now to be like palladium before, and you can trust palladium here to be the same as palladium there, but that’s obviously not the case for anything to do with human beings. So when an experiment doesn’t replicate, it doesn’t necessarily mean that the first scientist lied about their impact or the second scientist’s experiment contained methodological sloppiness. It could just mean that the two scientists were running entirely different experiments (the effect of a treatment on c. urbanicus vs. c. pastorilus), and only thought they were the same because they labeled their different patients the same way.

I’m not disputing that there’s plenty of methodological sloppiness and outright fraud in science. But I do genuinely think this is the most important frame to the replication crisis. After all, the labels are also how you’re going to end up using the study. “Are you feeling phlegmatic? Well, there was this new study…” Meaningfully using a scientific finding in your indexical life has exactly the same issue with representation as trying to replicate that finding. A country physician needs to know that their phlegmatic patients are different from a city physician’s phlegmatic patients, while each individual patient needs to know which paper to read when faced with the differing results.

When you think about it this way, it’s clear that many of our existing strategies for imbuing scientific papers with more meaning aren’t relevant here. Statistics? House sees a good result 100% of the time, and Field sees a bad result 100% of the time, so neither of them need any degree of statistical fluency to interpret the signals they’re seeing. Bigger sample sizes? Maybe they’ll get a hint of the real issue if the demand for more patients forces them out of their normal stomping grounds and increases sample diversity, but by and large House will just see more city patients and Field will see more country patients. Preregistration? This wasn’t a method-based problem at all, because House and Field did exactly the same thing.

What about a meta-analysis that looks at several different papers to try to figure out what’s really happening? Now we’re getting warmer, but there’s a critically important distinction here. Meta-analysis can clue you in that the difference exists, but it’s not a tool that lets you reach “the answer,” because the whole issue is that there is no singular answer. Doing a numerical meta-analysis is only going to mislead you by averaging together dramatically different studies, in the same way that no formal methods worked for the rabbits because hrair could be 4 or 1000. We need a new kind of meta-analysis that lets us do what the chemists did and pick out details that the authors didn’t think to explicitly represent. But if two papers can follow identical methods and get different results, how can you read them both and tease apart the difference?

The antidote to representation issues is context. We need scientific papers to be more like murder mysteries, replete with “unnecessary” details that might include the crucial clue. By definition, the authors themselves won’t understand which detail they add that might prove crucial for understanding a representational problem—if they understood it, they’d represent it, and it wouldn’t be a problem. But the authors don’t need to understand the importance of particular details. Let’s say that House and Field were both forced to add some set amount of narrative context to their paper. They’re hardliners who are convinced there’s nothing more to a patient than “phlegmatic” and resent the whole exercise. They agree to both use the space to complain about how annoying the required sample size was. House’s complaint is that he had a constant line of patients outside his door that disrupted his neighbors, while Field’s complaint is that he went through five horses because he had to travel to so many different farms to get to everyone.

What happens when those narrative studies are slipped under the door instead of just the tablets? House never explicitly says “I’m a city doctor working with city patients” in his paper, but now it can be inferred by the reader. You can’t directly recreate the original scenario like the chemists could, but you can at least see into the experiment beyond just the results. In a situation where the same methods led to different outcomes, the papers themselves will now be different in a way that can help you tease out why. You’re not completely helpless like you were up there in section II.

Phrased like this, it’s almost insultingly straightforward. Of course adding more stuff to a paper makes it easier to figure out! Maybe because representational issues are so obvious when you’re face to face with them, we don’t tend to think of them as a general phenomenon. We can recognize them and solve them, but individually, without recognizing it as a “kind” of problem.

We have to imagine detail contaminating all abstracts, though. Just as the theoretical construct of “metal-free” couldn’t help but mean “no metals explicitly added, though some contaminants remain,” “phlegmatic” can’t help but mean something more nuanced than “a symptom state such that all patients who exhibit it will behave exactly the same way to exactly the same interventions.” The more indexical the object is, the more impossible it is to run two experiments and equate them with each other, and the more context you need to determine how much of the relevant detail your representations are actually capturing. Indexicality can’t be objectively measured or tested for, but that doesn’t mean it’s subjective or arbitrary. Clearly, a human being and a fire have many factors that could influence the outcomes of any given intervention, while palladium compounds and tables have fewer. What we don’t have is well-negotiated language to talk about how indexical various concepts are, and to what degree detail intrudes on the pure concept as it’s being used. 

No one will give you the answer—so you have to figure it out yourself. This is not a manifesto about all scientific findings and modern rationality being wrong. It’s a warning that the right stuff and the wrong stuff are going to look exactly the same to the untrained eye. The same methods, the same data analysis techniques, the same probabilities. There’s only one way to sort the good from the bad: are the representations meaningful with respect to the differences? You can’t trust the people telling you “the facts” to get this part right. Look for clues. Is it a finding about something largely independent of its context like palladium, or is it highly indexical like a human being? If you see a headline, is it about something you can instantly imagine the definition of (like “all 2018 Volkswagen Jettas”) or is it something that doesn’t have a single socially negotiated meaning (like “healthy diets”or “predatory journals”)? Do you know the story of how the data was collected? If something very weird happened, whose job would it be to encode it, and what categories would they pick to describe it? Was this analysis done by someone in direct contact with the data collection, or are there multiple layers of abstraction between the phenomena and the conclusion? 

It’s becoming easier than ever to collect big gobs of data, cheaper than ever to hold on to it, and faster than ever to send it out for other people to derive insight from. Once you start asking these questions about representation, you can’t help but notice the problems are getting worse and worse every year. This is the urgency that drove me to start Desystemize in the first place. You can see this concern right at the very beginning in Desystemize #1:

Dr. Ostfeld didn’t start that paragraph by noting such-and-such statistical technique clearly indicated something was off with the tick counts. He started with the sentence “My research group has set and checked many hundreds of thousands of live animal traps over the years.” In other words, it was familiarity with the data-generating process that enabled the lab group to imagine this potential vulnerability and come up with this experiment. By the time the data gets into the hands of analysts, it’s too late to fix. You can’t math your way out of a wrong number. This mistake was caught only because it was the same people generating the data as analyzing it. Which, great for ecology - but as data science becomes more and more specialized, it will be increasingly done by people who are explicitly and solely data scientists. And they’ll inherit datasets from repositories somewhere and never catch a single one of these systemic errors because they couldn’t sift through the wet mouse turds even if they wanted to. 

After all of my previous articles dancing around these issues of representation, at last we’ve arrived within the right arena to face the beast head on. No “scientific method” will ever be enough on its own, because the same method may succeed in domains with meaningful representations and fail dramatically when the representations are drowning in meaning-debt. The solution is to open the door and to engage with differences beyond dead representation. Trust your ear to tell you when the fire’s too quiet. And when you’re asked to make a judgment about something you can’t open the door to yourself—a study or an article or a graph, the sort of thing you’ve been shown thousands of times and will be shown thousands more—don’t trust their ontologies without question. Look for hints that they’ve taken representation as seriously as they ought to. Ask questions, and from their answers, forge a proper language of representation, a social understanding of which ontologies are useful and which are bankrupt. Take this tacit skill scattered among a few experts and make it a core part of education and the human experience. It’s gotten too easy to live behind the door, and the stakes are too high to keep it up much longer.

Thanks to Crispy Chicken for significant conceptual feedback, the rest of the Inexact Sciences crew for inspiration, Lyta Gold for editing, and everyone who kept me company while I wrote.

Subscribe now

“McKinsey types” is one of those statements that’s hard to nail down the exact meaning of, but quite easy to understand emotionally if you’ve met one. McKinsey is a firm that’s the biggest player in the “management consultant” space, specializing in parachuting into a rich domain and reducing it down to simple metrics. Let’s look at a paragraph from Stoler to set the scene:

In fact, McChrystal and much of our military leadership is tight with consultants like McKinsey, and that whole diseased culture from Harvard Business School of pervasive over-optimism and finance-venture capital monopoly bro-a-thons. McKinsey itself had involvement in Afghanistan, with at least one $18.6 million contract to help the Defense Department define its “strategic focus,” though government watchdogs found that the "only output [they] could find" was a 50-page report about strategic economic development potential in Herat, a province in western Afghanistan.” It turns out that ‘strategic focus’ means an $18.6 million PowerPoint. (There was reporting on this contract because Pete Buttigieg worked on it as a junior analyst at McKinsey, and he has failed upward to run the Transportation Department.)

We can define by example: a McKinsey type is one who imagines that eighteen point six million dollars could be usefully spent on “strategic focus”, by people who aren’t the same ones doing the actual work. So McKinsey types are not just the employees of management consultant firms, but also the credulous bureaucratic substrate that they feed and multiply off of. McKinsey type is a statement of faith about dashboards and metrics that ignores any concept of determining their correspondence to the world and instead insists that the act of creating them makes them true. Management consultancy requires a sort of profound arrogance in your ability to compress a domain that isn’t yours into the important bits; it’s no surprise that the people who think they can do this are usually those who are unfamiliar enough with doing real work that they can confuse the map with the territory. 

Stoler again:

And their embarrassment covers up something even more dangerous. None of these tens of thousands of Ivy league encrusted PR savvy highly credentialed prestigious people actually know how to do anything useful. They can write books on leadership, or do powerpoints, or leak stories, but the hard logistics of actually using resources to achieve something important are foreign to them, masked by unlimited budgets and public relations. It is, as someone told me in 2019 about the consumer goods giant Proctor and Gamble, where “very few white-collar workers at P&G really did anything” except take credit for the work of others.

Because having lots of money leads to you passively accumulating more, top-heavy governments and corporations are able to be profoundly wasteful and still see their numbers go up. They can fail all they want and still succeed. But what do “success” and “failure” mean, anyway? This is the real asset that management consultancy is selling - definitions in abstract domains. “The war in Afghanistan”, “the COVID-19 pandemic”, “the social welfare programs of the United States” - these are dense and interconnected domains where a lot of people will try a lot of things and at the end some outcome will occur. You don’t get a clear breakdown of what was in your power and what wasn’t, you don’t get an omniscient being to tell you whether your view of what ended up happening was true or not, and you certainly don’t get a letter grade of how good among the possible outcomes you were.

So what happens when the naive quantification of a McKinsey type meets the real world? Trading away long term resilience for short term efficiency, usually. The example Stoler uses is the Afghan air force being maintained by contractors, who promptly abandoned them when the US withdrew. I’m not a military expert, but if you asked me “If we’re trying to build up the Afghan military to be independent, should they be dependent on NATO contractors who will leave when the US withdraws?” I wouldn’t exactly need to think long to say “no”. And neither, I imagine, would you. So how does this sort of thing happen? Well, because in some slide deck that some military head-of-something-or-other looks at every month, there’s a metric being tracked called something like “operating costs”. And when the contractors put in their bid, the monthly cost of running the air force went down, and that metric turned green and had a little +X% next to it. Then some person whose job is making Powerpoints showed that slide and figured it was self-evident that the number being green meant that the military was more capable, and things are officially better than last month, and that’s that.

So they use extremely tenuous proxies like “monthly expenses” to try to predict something like “the outcome of a war”, and fall victim to all sorts of trades that juice the proxy by weakening it’s already-questionable predictive power. If you knew what I meant the first time I said “McKinsey type”, you probably knew all of this already. But the thing I really want to focus on is how this attitude manages to be so pernicious despite it’s history of consistent and obvious failure. Why are there still McKinsey types? I think the answer lies in how many kinds of failure are largely unmeasurable.

Take this Slate article by Vishal Khetpal: Just Say It: the Health Care System Has Collapsed. It makes the point that saying that the US healthcare system is on the brink of collapse due to COVID-19 is giving it too much credit. Daniel Wilkinson presents with gallstone pancreatitis that is extremely amenable to quick surgery, but that surgery is delayed due to lack of ICU capacity and he dies. Is “collapse” a good word for an easily preventable death occurring because of stresses on the system? Like the word “failure”, it primes us into thinking about thresholds and discrete states - the system has collapsed, or has not collapsed. 

Instead, try asking: how collapsed is it, exactly? Daniel Wilkinson wasn’t the only patient at that hospital, and some people got the care they needed that day. When you analyze his individual story, it’s clear enough that his death was a profound failure because of the personal details: the seven hour wait, the huge success rate of his procedure when done on time. But coding this as a “preventable death” is much harder. Hospitals always have delays, and there will always be some sick people who die even if everything is done perfectly. You can measure delays, and you can measure deaths with certain diagnoses, but ”could have been prevented” is an individual story. Prevented how? With what kind of effort? IF the wait time had been shorter, THEN Daniel Wilkinson would have lived - but there’s always going to be some sort of margin where that’s true for somebody and false for somebody else.

“Average time to get an ICU bed” is a metric that you can measure, but that’s not the same thing as discovering the impact of it’s failure. You can set metric thresholds to decide what you want to call a failure, but how much your failure actually meant in material terms requires the sort of counterfactual analysis (“What if everything in the world was the same that day except for the average time to ICU bed?”) that’s extremely difficult to do over aggregated numbers. This is the bolt hole that a McKinsey type can always crawl back into for self-defense. Systems of sufficient complexity rarely completely fail in discrete and definitive ways - they simply do a lot worse. But as long as there’s someone left behind to pay the consulting fee, a consultant can point to whatever green number is on the dashboard, and instead of saying “Wow, we still bungled the hell out of the war in Afghanistan even though that number was green - I guess I should re-evaluate my thoughts on what I chose to measure” they can just say “Well I’m sure that bad outcome was inevitable, but it would have been worse if the number was red. Hooray for data!”

There’s a phrase that sometimes gets passed around by McKinsey types: “the plural of anecdote is not data”. The intent is to elevate whatever report was put together as superior to the lived experience of any naysayer. But the original quote from Raymond Wolfinger was exactly the opposite: “the plural of anecdote is data”. I’m with Wolfinger on this one. Data is a detail-annihilating projection that may or may not correspond well or at all to the real world, while an anecdote is something that actually happened. And far too often, success is expressed in data while failure remains buried within anecdotes. After all, why do all of the work to properly handle details that will just make you look bad when you could just tell a nice story and collect a big check?

At the beginning of his piece, Stoler asks: “Do we have the competence to govern ourselves anymore?” If your vision of “collapse” is a complete post-apocalypse of marauding gangs, it may seem like a silly question to ask of a society that is broadly functional for many people most of the time. But dying of gallstone pancreatitis that was diagnosed several hours before death, dying for lack of insulin when it’s an order of magnitude cheaper across the border, dying for lack of food while across town police pour bleach on food in a dumpster - these failures are failures of governing competency. These are stories of neglect, and one enormous form of neglect is to have the depths of your tragedy insufficiently captured and understood; excluded from the metrics so that on the day you die, the bureaucrat that could have saved your life is looking at a Powerpoint with green numbers because nothing about you dying makes them worse. That’s the reason I dislike McKinsey types, and that’s the reason I’m writing Desystemize. We need to be more mindful both of what metrics we choose to track, and that metrics will always lag anecdote in terms of detecting and preventing system failure.

Next time we’ll re-tread the ground of Desystemize #5 on AI, but this time focusing on the existential risk of superintelligence - it’s been a popular topic in the comment section and I want to give full treatment. If you want to know when it’s done, make sure you subscribe!

Subscribe now

Before explaining the bug, some background on Crawl is needed for those who haven’t played. (You also could just play it - it’s free to play and available both in browser or as a local download.) Crawl is a roguelike, a genre of game that’s undergone some linguistic expansion in the last decade, so we’ll focus only on the two attributes that are core to our story. The first is that the levels are procedurally generated every time - you can’t memorize the layout of the dungeon, because each run takes place in an entirely new version of the dungeon. The second is that consequence is permanent - you can’t ever reload a save to a previous turn, and if your character dies, your file is deleted. Combining these factors means that there’s no way to guarantee yourself a victory in Crawl. You can’t endlessly grind to improve your level, nor can you look up exactly what steps to take or memorize a route. And if you die, all of your progress is gone - the only difference run to run is what you learn.

So beating Crawl requires some degree of self-reflection. After all, if you kill off a character with eight hours invested in them (something I’ve done more than my fair share of!), those hours were completely wasted unless you manage to learn something from what happened. It naturally forces you to take responsibility and update your internal model, and it does so in a rigorous, no-nonsense way that’s hard to come by in everyday life. You can make all sorts of incorrect predictions in real life and simply forget about your misses and remember your hits. But if you predict that you can beat that ancient lich and you can’t, then you are Dead With No Do-overs, and It Sure Looks Like You Don’t Know What You’re Talking About With Ancient Liches, and Maybe You Should Work On That.

Of course, I could also just conclude that this particular lich was simply some sort of super-lich, or that his crystal spear was an unavoidably extra-crispy spear, or some other unique-to-this-run factor means that nothing that happened was my fault. This lets me assuage my ego at the extent of invalidating the chance to learn from the run. I can decide that it was the game's fault, really, and go into the next run with exactly the same mental model as before. Winning reliably at Dungeon Crawl: Stone Soup requires you to suppress this impulse. The reward you get for suffering short-term damage to your pride is a long-term improvement in outcomes that you know is directly tied to the lessons you learned. Always assuming that the system is consistent and fair, and that changes in outcomes are tied to your actions, is the path to victory. But once you’ve gotten good at making this assumption, a new question arises: what happens when the system really is inconsistent?

It is March 6th, 2015, and a bunch of nerds are about to fail to notice something.

The bug itself is one of those hilariously simple mistakes that computers are great at enabling. A small refactor in melee damage code ended up causing the outgoing damage to be added to itself. This isn’t a bug that’s highly contingent or complicated: all player melee damage was simply doubled every attack. However, the actual damage numbers are not exposed in a game of Crawl. (Charmingly, the game uses successive numbers of exclamation points to give a vague sense of how much damage you did. You dice the ogre like an onion!!!!) And the nature of Crawl combat is already swingy, since your armor essentially rolls to reduce damage, meaning that it can have very inconsistent value attack to attack. Still, doubling player damage is huge, powering you up on a level that outweighs most of the tactical concerns of the entire game. Surely a signal this loud can pierce through the noise, even if we can’t immediately look at the numbers?

Well...no. My source here is the Something Awful Dungeon Crawl discussion thread(s) (the incident spans the .15 and .16 threads), chosen because it’s a single linear record of discussion at the time, instead of trying to rewind Reddit or the official forum to six years ago to follow the various strands of discussion. And as the days tick on from March 6th, conversation proceeds mostly as normal with the occasional praise of the blessings of luck. Floodkiller writes, celebrating a win:

I need to stop playing until (the .16 release tournament) starts or I’ll waste all my good luck!

LogicNinja with a comment that’s heartbreaking in retrospect:

AW YISS WON THAT KoBe [Kobold Berserker]

MY FIRST STREAK

I’M REAL GOOD AT CRAWL NOW

The victory posts come in a bit more frequently than usual, but it’s a high energy time for the thread, and who can be too surprised if a lot of players successfully improved themselves with all of the helpful advice they got? In fact, only a single player,
Brannock, cottoned on before the bug was formally announced:

I don't even fucking know what to think. Suddenly I've become a legendary shining avatar of Cheibriados. Before this day last week my best streak was two. Then I knock off three and then six in a row. It's starting to make me feel a little [tinfoil hat]-ish. Maybe something really did go wrong in an unrelated commit and Crawl is unintentionally easier now???

(It’s worth noting that Cheibriados is a god that grants you substantial physical power in exchange for making you slower, and so someone streaking through Cheibriados-worshipping characters would benefit more from the bug than most players - and have more cause to notice it - since the numbers being doubled would be higher.)

Internet Kraken has a ready answer for Brannock:

You've been playing basically the same kind of character for over 30 games now. The game isn't easier, you just know exactly what to do in every situation with these combos. Still, good work.

This comment is a crucial one for understanding what went wrong, and we’ll return to it shortly - but let’s finish the story first. For a bit longer the thread continued on, with nods to luck or self-improvement but no more guesses that it might be a bug. Here’s Razzled:

Guys. GUYS. I am so fucking happy. I don't know what changed but after 3 years of playing in these tournaments I finally got not 1 but 2 wins for the first and second time ever!! ...I know these are just easy baby class/race combos and god but dang feels good to finally win

The commit causing the bug was reverted on March 21st, after two weeks of havoc. It fell to the appropriately named Can Of Worms to inform the thread:

So, it turns out a bug was accidentally introduced in one of the commits that's caused all melee damage dealt by players to be doubled (approximately.) Fun!

The responses were mostly about the clues that we missed in retrospect. Parthenocarpy:

Well that explains me going from a 2% winrate to 17.95% in the span of two weeks

Hawkperson:

Ahahah, I thought there were an awful lot of victory posts lately, but I just chalked it up to clustering or something. This is obviously why my gargoyle monk is still alive...

Fhqwhgads:

Well that explains how a shit player like me was able to streak with a bunch of melee dudes, but as soon as I tried something other than melee I got my ass handed to me repeatedly (like normal). Kinda sad about it now.

“Kinda sad about it now” was my reaction, too. The point of playing Crawl is getting good at Crawl, and successes are only valuable as the context is continuous. We play to get unbiased feedback on how capable we are at learning; instead, for two weeks, that feedback was disconnected from all of our prior experience and made close to meaningless. (This is not meant as a criticism of the programmer who introduced the bug. Computers are warm rocks we tricked into doing math and it’s a miracle they do anything.)

In exchange for the feedback of our performance in the game being less relevant, we got a different kind of feedback: how receptive we are at noticing large-scale changes. We did really, really poorly. And I think a big reason why is that being the kind of person who’s good at Crawl makes you especially susceptible to these sorts of errors.

What kind of person is that exactly?

Researcher Sarah Constantin wrote an article called “Do Rational People Exist”, which speculates about the “cognitive decoupling elite”:

Stanovich talks about “cognitive decoupling”, the ability to block out context and experiential knowledge and just follow formal rules, as a main component of both performance on intelligence tests and performance on the cognitive bias tests that correlate with intelligence.  Cognitive decoupling is the opposite of holistic thinking. It’s the ability to separate, to view things in the abstract, to play devil’s advocate.

Cognitive flexibility, for which the “actively open-minded thinking scale” is a good proxy measure, is the ability to question your own beliefs.  It predicts performance on a forecasting task, because the open-minded people sought more information. [21]  Less open-minded individuals are more biased towards their own first opinions and do less searching for information.[22]  Actively open-minded thinking increases with age (in middle schoolers) and correlates with cognitive ability.[23]

Under this model, people with high IQs, and especially people with training in probability, economics, and maybe explicit rationality, will be better at the cognitive bias skills that have to do with cognitive decoupling, but won’t be better at the others.

Speculatively, we might imagine that there is a “cognitive decoupling elite” of smart people who are good at probabilistic reasoning and score high on the cognitive reflection test and the IQ-correlated cognitive bias tests.

To which I would add: the cognitive decoupling elite are also people who can reliably beat Dungeon Crawl: Stone Soup.

But here’s Constaintin again:

I’d expect [the cognitive decoupling elite] not to be much better than average at avoiding the cognitive biases uncorrelated with intelligence. The cognitive decoupling elite would be just as prone to dogmatism and anchoring as anybody else. 

To which I would add: the cognitive decoupling elite are also people who can play a game very intently for two weeks without noticing that they’re doing double damage. In fact, I’ll go further: seeing the objective power of cognitive decoupling in systems that reward it can foster dogmatism and anchoring.

I promised I’d come back to this quote from Internet Kraken:

You've been playing basically the same kind of character for over 30 games now. The game isn't easier, you just know exactly what to do in every situation with these combos. Still, good work.

This statement is mostly true. Even if the double damage bug hadn’t been introduced, Brannock probably would have become better at winning with Cheibriados characters. Plenty of people still lost during the double damage times, because even with a massive handicap, Crawl is an awfully hard game. (At one point a few days after .16 was released, the winrate was 2.89%. This when every melee attack is dealing double damage!) Brannock still had to learn enough to let the double damage carry him to victory, and Internet Kraken was correct to note that experience is a hugely important factor in achieving victories.

But buried in Internet Kraken’s analysis is an unconscious proposition that we’ll call the “Systemic Stability Principle”:

If the change in the systems outputs can be explained by a change to the systems inputs, then the system itself didn’t change, only the inputs.

The mistake Internet Kraken (and the rest of us, implicitly) made was assuming that because experience with Cheibriados characters could explain Brannock’s sudden improvement in winrate, then it must be the whole explanation; when, in fact, it was experience plus the double damage bug. The Systemic Stability Principle is clearly false. But why did we make this unspoken assumption? 

Answer: because believing in the Systemic Stability Principle makes you good at Dungeon Crawl: Stone Soup, and many other things besides. 

In fact, it’s almost a prerequisite to improving in highly formal domains! If you die of more damage than you were expecting and tell yourself “that one must have been double damage”, then you can’t learn anything. All of that stuff I said earlier about rigor and self-improvement starts with you holding the system constant enough that you can evaluate your changes over time. In a well-designed roguelike, it’s hardly an exaggeration to say that the more you can internalize the Systemic Stability Principle, the better you’ll be.

This is just another face of cognitive decoupling; the superpower comes from what you’re blocking out. Instead of designing experiments to turn every single little detail into data, you take the system at its word on what elements should be discretized as data, and focus on performing interesting higher-level logic with those pieces. This makes it hard to separate the gains from cognitive decoupling from the vulnerability. How do you set a threshold for following up on anecdotal observation that’s high enough to reap the efficiency rewards of a good system, but low enough that you don’t miss correspondence breaks when the system genuinely breaks down?

There’s probably not a single comprehensive answer, but there are certainly some tricks that can help. The first is the easiest - remember that the Systemic Stability Principle is completely false, and just happens to be a false belief that it’s often useful to hold. A system being stable in the past doesn’t mean it’s stable now, and evidence that the Systemic Stability Principle works is not the same thing as evidence that it’s right. 

It’s also important to avoid refuting observations by appealing to system definitions. If someone says “This feels like it’s doing more damage than it says”, resist the impulse to say “Nope, it does exactly this value; it’s written right here.” Instead, try to design an experiment that would prove whether the value written in the system is correct. If designing the experiment is very hard, that should be interpreted as a risk factor - if no one can check that the system is doing what it ought to, then maybe it really is wrong! Play yes-and with systems skeptics, letting them invest their time into correspondence work if they think something is wrong. Or be the system skeptic yourself, if a certain observation sits wrong with you.

And more than anything, we need more humility from the cognitive decoupling elite. We’re hard at work turning the world into metrics and dashboards and systems, and obviously those of us who are good at systems are happy to have things be more personally legible. But before we get too excited about turning the world into a video game, let’s remember how stupid we all looked when we tried treating a video game like a video game.

(I’m going on vacation, so barring a urgent burst of inspiration, Desystemize will be on break until sometime early September. Don’t forget to subscribe if you’d like to know when it’s back!)

Subscribe now

In brief: HealthKit is Apple’s tool to aggregate health data and standardize it for research and interoperability. This makes it easier to work with than raw sensor data from the Apple Watch or whatever wearable item, but it’s also a system that adds a layer of abstraction. And what do systems do? They annihilate detail. Technically speaking, the sensors themselves are also systems, so you end up exporting a ghost of a ghost, data mangled by a set of rules.

What happens when those rules change?

This is what Onnela observed in his article. He was studying heart rate variability (hereafter HRV), but the actual metric is almost irrelevant to the point that we’re making. The thing that matters is that they did two extracts separated in time (the first one September 5, 2020, and the second on April 15, 2021) but looking at the same date range in the data. Since time travel is impossible even for Apple, the past should remain constant no matter how far ahead in time you’re looking at it. For example, you look the same in your 2010 yearbook whether you’re reading it in 2011 or 2021. The time you’re reading the book doesn’t have any bearing on what the book says.

It matters a lot if you’re looking at Apple’s idea of HRV, though:

It’s worth stopping a moment and reflecting on how completely ridiculous that second graph is. On the x axis is the data from one export, and on the y axis is the data for the same dates from another. The expected correlation is 100%, represented by the identity line running diagonally through. Why wouldn’t it be? These are precisely the same events, with the only difference being when we choose to look at them. So every single bit away from the line represents data instability over time.

Presumably Apple had a good reason to change their algorithm so drastically. I don’t know anything about calculating HRV from sensor data and have no reason to assume otherwise. If it corresponds better to physical reality, isn’t that the work that Desystemize wants to hype? In a vacuum, yes. But meaning is a product of interaction, which means that this change can’t just be analyzed by what it did to numbers living on your phone. Instead, we have to analyze the impact it had on the information ecosystem attached to those numbers. And since meaning is contingent on the governance of the actions that went into the data, basically all definition surprises are bad surprises.

A quick caveat: I work downstream from wearable data and don’t have access to these HealthKit exports personally. In a world where the Healthkit extract clearly indicated that you were downloading HRV data calculated by algorithm X or Y, with your choice to export the past data under either regime, there would be nothing especially irresponsible about these changes. I am taking Onnela’s surprise at face value and assuming that this change was mandatory and not specifically indicated in the export screen. This may seem presumptive of me, but quite frankly I’ve been in the industry long enough to know that the big players recklessly and silently break continuity all of the time. Still, if someone provides evidence to the contrary, I’ll happily retract my criticisms for this specific case, even if it still holds as a general phenomena.

For now we’ll assume it was a forced rollout. What does it mean to suddenly adopt a new definition, even a more accurate one? It means that you can no longer compare your previous data with your new data. Now, from the perspective of a single HealthKit enabled device, that isn’t necessarily a huge problem. Since the new definition works on the old data, you can look back and put your current data in context with the past. It might surprise you if you were keeping track of your “normal” and saw it suddenly change, but it’s a change you can investigate and rectify.

How does that signal to re-evaluate everything propagate outwards, though? Every study using HRV data needs to pull it all again. Any composite dataset of Healthkit data that has a mix of old and new algorithm HRV is comparing apples to oranges, and useless until it’s brought back to equality. And of course, that’s assuming you have the ability to force a new export. If you’ve ever downloaded a dataset without also getting a link to the originating devices, the algorithm change just effectively means that your dataset has become stale, unable to be used to comparisons with the present or future.

And it gets worse! Let’s suppose you’re training a predictive model for patient risk, using patient wearable data to make it “personalized for them”. No matter how responsible you were investigating the impact of Healthkit data on your model, it’s all for naught when the definition ticks over. If you have a model in use making clinical decisions, that means that the Healthkit component could become out of date at any moment, on the whims of Apple - and there’s no remedy except from trying to get as many re-exports as you can and training it all over again. How long is that going to take? And what happens in the meantime? You’ll be making decisions off of one idea of HRV, using data based on another.

When you think about it this way, you can see that it’s only partially Apple’s fault. There are definitely ways that you can announce continuity breaks loudly - proactively reaching out to people who have exported, showing the algorithm changes on the export menu itself, limiting changes to pre-arranged windows (ie, having only a few set days a year where you reserve the option to change, so people can know to check on those specific times). But there’s only so much you can do at the source, because the idea that data can be a freely traded commodity is itself a big part of the problem. How can you expect every single downstream source to update in time with a change at the root? The governance is part of the data, but a part that requires constant examination and care. Fundamentally, data can just go farther than governance, since data remains static forever and free to endlessly share, remix, and train datasets.

So it’s our vision of the future that’s the ultimate culprit. Create as much data about yourself as possible, and it can be used to personalize your world! But all prediction supposes that the past data predicts the future data, and when someone else owns the definitions, they can break continuity whenever they want. So any meaningful use of data needs to be a constant, ongoing conversation with the generating process. Instead, we have an ecosystem of “data dumps”, “extracts”, and other static datasets, ready to silently break their relationship with future data at any time. Whenever you see a study based on data that’s generated by an entity outside of the actual people performing that study, you should always stop and ask yourself: if what the data means changes one day, would the people running the study know?

Subscribe now

Last week, I mentioned it might be good to focus on a more discrete case study again after the highly abstract nature of talking about meta-rationality. Instead, though, I want to stay a little vague and switch to the related topic of intelligence, artificial and otherwise. Covering them so close to each other will help us draw some deep and useful parallels.

Our guide this time will be Melanie Mitchell, an AI veteran and Davis Professor at the Santa Fe Institute. Her book “Artificial Intelligence: A Guide for Thinking Humans” was one of my favorite books of 2019, but today I’m going to just focus on her paper “Why AI Is Harder than We Think”. At only 8 pages of text, it’s awfully approachable compared to most things in this field, even if you’re not normally someone who reads scientific papers. Mitchell uses the framing device of four fallacies that are implicitly accepted by many AI true believers. I want to talk more generally about these fallacies because I believe them to be applicable to many other fields - systems mania has led us to redefine the sorts of things computers are good at as “intelligence”. My interest here is less about AI in particular, and more about using it as a framing tool to talk about the intelligence that we as humans are performing every day. Or, uhh, not performing.

Fallacy 1: Narrow intelligence is on a continuum with general intelligence

This is one of those things that’s so vague it’s hard to see without a specific example. So let’s imagine someone who really, really wants to go to New York City. They’ve never been to such a big city before, so their goals are all of the kitschy first-time tourist stuff: seeing the Statue of Liberty up close, going to a Broadway play, and having an authentic New York slice. They want to physically go to New York, because once you make it there, it's relatively straightforward to do those three things. So they set up a little piggy bank with a postcard of Times Square over it and start squirreling away the money for their dream trip. 

But wait, someone says. We have virtual reality now! You can just slap on a headset and see the Statue of Liberty in stunning detail! Let’s assume that this person doesn’t have an aesthetic obsession to see the “true” Statue as long as the virtual reality version is indistinguishable. Then it becomes a tech problem. Can the resolution be increased so high that it really is impossible to tell if you’re in virtual reality or not?

Because we make progress every year on solving these sorts of difficult tech problems, they’re gratifying ones to focus on. Maybe someday we really can have life-like virtual reality. But let’s ask a different question - if we pull it off, have we made any progress on going to New York City? The answer is no, of course not. In fact, if we spent money from the piggy bank on our new headset, we’ve actually gotten further away from our goal. (At some point you and I were invited on the trip because I like using the first person plural. Hope you like pizza!) We’ve slightly decreased our desire to go to New York City, maybe, because we discovered that something we thought required being in New York City actually doesn’t. But just because going to New York City was previously the only way you could see the Statue of Liberty in lifelike detail doesn’t mean that a new way to see the Statue of Liberty has any connection to New York City, nor does it mean that similar shortcuts exist to do the other things you wanted to do without being in New York City. 

“Intelligence” is our New York City, a word that encompasses an extremely broad, heterogeneous, and nebulously defined set of capabilities. The fact that computers can do something that previously was the exclusive domain of intelligence doesn’t mean that it’s doing it in the same way that intelligence is, or that it can do other things intelligence can. In fact, the power of adversarial examples against many deep learning techniques, such as one-pixel attacks, is profound evidence against that hypothesis. If a pizza place is very close in physical space to the Statue of Liberty in New York City, that doesn’t mean that lifelike VR of the Statue of Liberty is almost the same thing as making pizza; so it is with computerized shortcut space vs. the landscape of human intelligence.

Fallacy 2: Easy things are easy and hard things are hard

Our example of the New York City problem already primes our intuition to answer this one correctly. If you’re trying to find a way to do something in New York City without being in New York City, your shortcut is as easy or hard as it happens to be, with very little bearing on how easy or hard it is to do in New York City itself. But let’s try to get a little bit more specific on which problems are hard for humans and easy for computers, to understand why these breakthroughs are so irrelevant to our idea of general intelligence.

As Mitchell mentions, AI has gotten really, really good at many games. What makes games different from most things in real life? They have absolutely discrete states, and the transitions between states are strongly patterned. AlphaGo was praised because Go has a gargantuan amount of possible states - but it is absolutely trivial to describe the state of a given Go board. The board has a fixed size and each intersection can only have one of exactly three states - empty, white, or black. 

Of course, AlphaGo is still a meaningful accomplishment - the possible states of a game of Go are more numerous than atoms in the universe, so it’s still doing something clever. Computers are great at sifting through big piles of stuff. We’re impressed by this as humans, because our capacity for sifting is pitiful in comparison. But it’s a very small subset of problems where you can discretely label every way things can possibly be. When you think of the kind of problems you solve every day - “What kind of mood is my friend in?” - you have an awesome power to generate novel states on the fly: “Well, they’re pretty excited for the concert, but also kind of nervous because they’re a bit out of shape and worried they won’t be able to go as hard as they used to, plus I know they’re waiting to hear how well their mom is recovering from surgery so probably a bit distracted, if still grateful for the chance to focus on something else for a night.” It’s contact with nebulous reality that can’t be recorded as {white, black, empty} that makes something hard for a machine, and reality is a factor in most problems we care about. 

Fallacy 3: The lure of wishful mnemonics

Mitchell starts this section by quoting computer scientist Drew McDermott, a decision I will directly replicate:

A major source of simple-mindedness in AI programs is the use of mnemonics like “UNDERSTAND” or “GOAL” to refer to programs and data structures. ...If a researcher...calls the main loop of his program “UNDERSTAND,” he is (until proven innocent) merely begging the question. He may mislead a lot of people, most prominently himself. ...What he should do instead is refer to this main loop as “G0034,” and see if he can convince himself or anyone else that G0034 implements some part of understanding. ...Many instructive examples of wishful mnemonics by AI researchers come to mind once you see the point.

Just because the computer is doing something that we do with intelligence doesn’t mean that it counts as intelligence, but...what if we went ahead and called it ‘artificial intelligence’ anyway? But as Mitchell indicates, most of what is actually occurring is just statistical correlation wearing some fancy hat or another. Correlation relies on conditions remaining consistent to be effective, which gives it a tremendous brittleness. In fact, a lot of the popular conception of intelligence is exactly being the person who comes to the correct conclusion the fastest in a novel scenario where the old rules no longer apply!

And wishful mnemonics are far from being an AI-only issue. Desystemize #2, for example, is basically exploring how “cancer in the United States'' as an idea is completely separate from “recipients of cancer diagnoses in the United States”, and that the skew between the wishful shorthand and the actual provenance of the data happens in a non-random way that makes it dangerous and unethical to assume the former from the latter. In fact, a major part of applying the desystemic lens is simply replacing the aspirational terminology a dataset uses for itself with the verbs of action that actually went in to making it. Oftentimes you don’t need a more robust analytical framework than “Say what the data is and not what it wants to be and watch as the hypothesis disproves itself.” (Just say in mice!)

Fallacy 4: Intelligence is all in the brain

“Embodied intelligence” is an idea that might sound ridiculous at first glance. What, so just because we have bodies, we think it’s required for intelligence? Surely that’s just rank anthropocentrism. You don’t need a body to run an algorithm on the facts, right?

But remember what we learned in Desystemize #4. There’s no such thing as “the facts” separate from the algorithm you’re running. Deciding which details you’re going to turn into data is a part of intelligence. So in a very real sense, any AI that’s trained on “a dataset” it didn’t have agency in creating is only performing a very limited subset of what we would call intelligence. The work of resolving the world into states was done for it already, and it’s only going to be as smart as those states are useful.

If an artificial intelligence is ever going to be comparable to a human, it’s going to need it’s own ability to choose what parts of the infinitely detailed world to resolve into data. Whether or not this needs to be a “body” as we would understand it is partly a question of semantics, so we won’t dwell on it too long. The key point is - for both AI and for systemic intelligence more broadly - that there needs to be a dynamic conversation between data, decisions, and outcomes. There are infinitely many ways you can choose to record your present state, each one allowing you access to different decisions. When you get evidence of a decision being poor, you can try your best to make a better decision with the data you have, but having access to the full problem-solving suite of human intelligence also means starting to collect data about a cryptic factor you were previously unaware of.

Note that while this doesn’t preclude the idea of a super-intelligent AI, it does put a damper on the scenario that tends to generate the most fear - the hyper-fast “intelligence explosion”. In this theory popularized by Nick Bostrom in Superintelligence: Paths, Dangers, Strategies, once an AI reaches a point where it can reprogram itself, it can optimize itself to be a more intelligent AI. Millions of iterations might happen over a single weekend, so that scientists leave for the weekend on Friday with a sub-human gimmick AI and come back Monday to the cognitive ruler of the known world. Scary! But implicit in this scenario is the false promise behind most of modern systemization - that the way we’re recording data is fundamentally correct, and that learning is simply a matter of improving our algorithms that run on “the truth”. Once you accept that there’s no universal listing of facts, the AI needs to generate its own facts as part of becoming “more intelligent”. Determining whether your facts are good or bad requires experiment - real experiment, not the machine learning sense of “experiment” that involves slicing the same static dataset in different ways and running your algorithms on it. Since time is a component of processes in the world, we should have a strong suspicion that “intelligence” can’t be completely decoupled from the processes you’re hoping to be intelligent about. Becoming more intelligent happens in experimental time, not computer time.

So if you’ve been up at night worried about a computerized takeover, you have my permission to sleep easy on that score. But the importance of experiment and physical existence in the world only emphasizes how dangerous it is that we’ve handed so much decision making capability to algorithms already, since algorithms today don’t have bodies. I’d like to end with a quote from Mitchell’s book mentioned earlier:

Importantly, Szegedy and his collaborators found that this susceptibility to adversarial examples wasn’t special to AlexNet; they showed that several other convolutional neural networks—with different architectures, hyperparameters, and training sets—had similar vulnerabilities. Calling this an “intriguing property” of neural networks is a little like calling a hole in the hull of a fancy cruise liner a “thought-provoking facet” of the ship. Intriguing, yes, and more investigation is needed, but if the leak is not fixed, this ship is going down.

Don’t stay up late worrying that the machines will become much smarter than you and take control. Stay up because they’re much stupider than you and we’re giving them control anyway! If you need someone to talk to about it, go ahead and leave a comment, and don’t forget to subscribe to see if I keep my promise to do a more concrete case study next week.

Leave a comment

Subscribe now

So what is this field? Meta-rationality, a term of art that as far as I can tell was coined by David Chapman. I’ve been following his writing for years - if you know to look for it, the influence of the parable of the pebbles is clearly evident in the very first article of Desystemize about counting ticks. Somehow, though,  I managed to miss his primer "A first lesson in meta-rationality" until now. I can’t possibly explain all of meta-rationality in this newsletter. Chapman is working on a book called In the Cells of the Eggplant to do it, and it remains only partially finished after several years, so this single newsletter doesn’t have much hope. However, we can use the metaphors deployed in his first lesson to at least vaguely sketch what meta-rationality is and properly place it in the landscape of ideas relative to desystemization. (As someone who went ahead and also invented a word to describe this stuff instead of just using his, I feel a certain compulsion to defend myself!)

It’s my usual custom to link a post and say “the whole thing is worth reading”, but then give a summary for those who just want the highlights. Seriously, though - the whole thing is worth reading. This lesson uses puzzles as a tool to elicit understanding, and the frisson of attempting to solve them before reading the answer is a huge help in explaining the why. I will necessarily have to spoil the answers to those puzzles to use them as examples myself, so you won’t be able to come at it with the same fresh eyes if you read me first. Take the time, go read "A first lesson in meta-rationality", and come back here when you get a chance, okay?

Okay. Well, I’m sure a few of you didn’t listen, so: this is what’s called a “Bongard problem.” Your goal is to figure out how the six images on the left are different from the six images on the right:

This is an easy one: straight lines on the left, smooth curves on the right. “Straight line” is something with a straightforward definition, and you can imagine applying a straight line detector to each of the twelve images in turn. You hear a beep on the left six and no beep for the right six, and go to bed secure in your powers of rationality. What makes Bongard problems interesting is that there’s no particular requirement that the criteria is that straightforward. Here’s a more fun one:

You know the answer straight away - triangles on the left, circles on the right. But what would happen if you applied a circle detector to the two on the left that are made of circles? If your circle detector beeps, you lose. And your circle detector DOES need to beep for the one on the right that’s made out of straight lines - how on earth are you going to pull that off? Not to mention that you have exactly the same problem with your triangle detector.

If your circle detector and triangle detector don’t work, how did you figure out the answer so fast? The trick is that there’s no such thing as a singular “circle detector”. That triangle made of circles really should make a circle detector beep in some contexts. If you’re detecting circles to figure out whether it will roll down the stairs if you drop it, it does you no good to confidently yell “triangle!” before sending a small avalanche of circles down to trip whoever has the misfortune of being below you while you attempt to achieve perfect rationality. “Circle” and “triangle” are not inherent properties of any of these twelve images. Instead, the images are laced with details that sometimes should be systemized into data like “circle” and “triangle” and sometimes should be ignored. The thing that makes a Bongard problem a problem is trying to figure out, for this given set of 12 images, which bits are extraneous in this particular case and which ones are not.

This is an incredibly important point, so it’s worth repeating. To solve a Bongard problem, you must engage in systemization. Abstract details need to be turned into data, so that you can make sure the data says one thing on the left side and one thing on the right side. This is not some meditation on how systemization is inherently wrong. But which details can be turned into data, and what that data should be, is a property of the problem, not the image. There’s no such thing as “truth” about these images that’s separate from the question we’re asking of them. Meaning is a byproduct of interaction, not an intrinsic property of anything.

Keeping this idea in mind, we’ll move on to a set of three problems meant to be solved in sequence. Number one:

Answer: three lines vs. five.

Number two:

Answer: three lines coming off a central point, vs. five lines coming off a central point.

Number three:

The big reveal: the rule of the last one is “four line segments vs five”, even though two of the entries on the right appear to have three line segments. But that’s only three line segments by the definition of “line segment” we used for the first two problems. Problem three is using “line segment” for anything coming off of a juncture point. The meaning of “line segment” depends on the company you keep.

This is in contrast to the high school “science is real” view of rationality as a constantly accumulating store of facts, as though there was a single body of knowledge of true statements. At first blush it seems that all we need to do is bail out the boat a bit more. Let’s just define “Line Type 1” as a continuous squiggly bit and “Line Type 2” as anything coming off of a junction, problem solved. Hey, we’re not rejecting the view of science as a body of work, we just doubled how much stuff is in it! We can define the Line Type 1s and Line Type 2s for any image we see. And if the good ship Rationality springs a leak every time a new Bongard problem shows itself -- why, caulking up the leaks is just the job of scientists, and won’t we have a better ship when all’s said and done?

Resist this impulse and sit with the word “line” a while. Whether you had seen the first problem before or after the third problem would have determined whether your initial idea of line was Line Type 1 or Line Type 2. And fixing your ignorance requires going back to the well of detail - the images themselves, not your abstractions of them as represented by line count. Whichever idea of “line” you came up with first wasn’t necessarily wrong when you came up with it, but it wasn’t precise enough to handle every question you could throw at it. This is happening everywhere, all of the time. We must systemize the nebulous world to get anything done, but systems annihilate detail, and there are uncountably many pairs of things that are “the same” in our representations of them but are actually different in some way we don’t know about yet and can only figure out by investigating the things themselves, not the representations.

This is the Great Big Thing that Desystemize and In the Cells of the Eggplant are both reactions to. You can be rational on a system, but there’s simply no such thing as global rationality, because processes that turn detail into data are always contingent on what ongoing work the processes are meant to support. But we need to systemize things to interact with a world beyond our immediate personal senses! This is Chapman’s focus, well-articulated in “A bridge to meta-rationality vs. civilizational collapse”. Rationality is a useful and necessary tool, but if we aren’t able to articulate its limitations and all of the circumrational work we unconsciously do to patch it up, then rationality’s failures will be co-opted as evidence by anti-rationalists to justify destroying systems we desperately need. 

Well, we just said that we like Chapman, but destroying systems is our whole shtick. How does this stuff fit together? If we think of meta-rationality as being more deliberate about the interface between world and data, then desystemization can rightly be viewed as a subtask of meta-rationality. To solve that last Bongard problem you must desystemize your old idea of “line”, re-observe the detailed world, and come up with a new systemization more appropriate to your current problem. On the level where you’re solving the problem, one step away from the real world, desystemization and resystemization come together in a beautiful and necessary dance of meta-rationality.

But the more steps away from the world you get - the more that your analysis is data about data about data - the more out of balance the whole enterprise of analysis becomes. This is the secret ingredient of the desystemic lens: it becomes more powerful with abstraction. When you diagnose the problem from on high, like Dr. Shrager did in Desystemize #2, you don’t have any tools to improve the correspondence to the real world - only to show that it’s broken. This is because our appetite for metrics has grown so much bigger than our ability to make them meaningful, with disastrous consequences for basically everyone. Chapman’s work is on how we can actually go about feeding ourselves more; we have the much easier and more entertaining remit of pointing and saying “Dude, that is pretty obviously going to make you sick.”

I find the parts of In the Cells of the Eggplant that we have so far endlessly fascinating; hopefully you will as well. It’s good to have a grounded understanding in what makes rationality work. If we’re going to survive as a species, then in my lifetime we’ll probably need to have better formal theories on this stuff. And I’m going to try my best to understand and incorporate that perspective, on the grounds that it is better to have enlightenment and not need it than need it and not have it. Luckily for the purpose of Desystemize, we don’t need to wait on that work to perform basic correspondence tests on our systems to the detailed world they claim to represent and marvel at the difference. Applying the desystemic lens is a subset of meta-rationality, but not one that’s relying on any completed theory to get the job done. A tagline for Desystemize could easily be: “Meta-rationality, but only the easy parts.” When numbers have gotten as far away from the world as many of the ones we see today, just having the sense to stop and look can be enough.

Speaking of looking at the world...let’s try to find a more concrete case study for next week, eh? Leave a comment if you have something worth looking at and subscribe if you want an email about it later.

Leave a comment

Subscribe now

Precisely because this is a theme that recurs over and over, it’s hard to find something that shows only and exactly the idea we want to explore. Serendipity provided for me in the form of Omar Rizwan’s “Against Recognition”, which does a fantastic job of illustrating the phenomena I’m describing. The article is worth reading in full, but we’ll quote the specific passage that will best serve to anchor our decision:

We have all these computer systems that love lowest-common-denominator formats like plain text, and they push programmers to normalize everything into those formats, so the computer can 'understand' them.

But I feel like as much as possible, the computer should be leaving things the way they are!

If you have recognition, it should be a sort of overlay you put on the thing (maybe one of many such overlays); you shouldn't destroy the thing and replace it with its ashes.

If it has to exist, the text recognizer should attach an overlay to the image that says 'it might have this text in it'; the image shouldn't itself be transformed into text. (and ideally, that overlay would be rich with context and provenance; it wouldn't just be a blob of plain text; it would know what image it's from, admit other texts that it could potentially be, talk about how likely each word of it is to be correct, say as much as possible about the recognizer's process and thinking)

The original thing is still around and is still the source of truth.

You shouldn’t destroy the thing and replace it with its ashes. Not all that far off from our own mission statement of “recovering the detail that systems destroy”. And optical recognition of text is an example where we all can intuit the difference between what’s written and what’s typed. How bad is it to replace the thing with it’s ashes? Depends on context. If we’re putting an address on an envelope to make sure it gets mailed to the right place, few people will be up in arms if it’s replaced with it’s “recognized” text. But if that envelope contains a heartfelt handwritten letter to a friend you haven’t seen in years, I think most people would be quite upset to learn that the post office replaced it with a typewritten copy on a plain white piece of paper, even if it captured every single word with perfect fidelity. Why? Because the purpose of writing the letter is more than the meaning of the words. The “point” of the letter is just as much in the care of the handwriting, in the smudging that shows how much a given sentiment was labored over, in the bashful wilting of the paper saying “I took quite a journey to get here, you know”. In the details.

And that’s the definition we want to use: the details are the parts of the letter that the post office couldn’t possibly replicate, only preserve. So the words of your letter, while obviously of profound importance, are not “details” of your letter but “data”, because the system of text recognition preserves them. And higher order attributes that go off of the data - what language is your letter in? What words are most common? - are definitely not details. Details are the creases and sloops and swirls that can be observed but not encoded.

You can see here I’ve pulled a bit of a trick: systems annihilate detail is trivially true with this idea of detail. If details are things that don’t get systemized, then of course systems annihilate detail, since everything the system doesn’t annihilate is something we’d call data and not a detail. That’s why my goal here is only to establish I’m using detail to stand for the residual stuff after something gets systemized, the parts of the thing that you can’t figure out looking at the ashes. It’s up to the rest of Desystemize to convince you that details have intrinsic value and it’s a bad thing for them to be annihilated. This circular definition is not an attempt to end-run my thesis without putting in the work; it’s just to put a handle on my offense.

Indeed, the excellent thing about the letter example is that the offense is immediate, emotional, and aesthetic in nature. Most of the time, Desystemize is concerned with a more technical and concrete consequences of over-systemization. But sometimes, the impact of destroying details can just be “it’s gross.” And focusing on this example where the details are aesthetic and emotional also neatly addresses one of the traditional counter-arguments of the naive systemizer: hey, we could just turn everything into data, and then nothing would be annihilated! But would anyone actually feel better if the typewritten post office copy also contained a readout of metadata? “This paragraph has 234% more evidence of repeated rewriting. The final page had substantially more moisture content, suggesting perhaps spilled tears.” Even if those are objectively true statements, it doesn’t mean they do anything to preserve what we wanted from the details.

Aesthetic arguments are not rigorous, and it’s important to note places where this emotional backlash can lead us astray. We can’t systemize every important detail of a written letter, but what we can systemize is a lot better than nothing, and we have to make trade-offs somewhere to function at all. If climate change creates extended periods where post workers can’t safely send letters, their warehouses filling up with too much mail to handle - I’d take an email of the brute text of the letter over demanding someone suffer to bring me the full detail. Systems make things recognizable, intractable, legible; that makes many verbs of interaction easier. Sometimes things need to be easy, and saving the letter-as-it-is is a harder thing than saving the letter-as-it-was-systemized. 

All the same, we can’t turn this into a simple accounting problem, where we attempt to “balance out” the cost of preserving details vs. the value of doing so, at least not in any quantitative or rigorous way. The whole point about details is that they’re unrecognizable to the system; this also makes them broadly unmeasurable. If you attempt to save only “valuable” details, you will first have to create a value-assignment system. Since details are the bits that resist systemization, any idea of value that isn’t personal and subjective is likely to do a bad job.

What bits of writing do you have that have meaning beyond their text? Postcards on the fridge? Newspapers from a specific, important day? A set list from a concert? Whatever you’ve chosen to hang on to, I’ll bet that you don’t have a numeric score associated with your desire to keep it. You just have the human idea of sentiment urging you to preserve significant things from your past, even if you can’t articulate a strict benefit for doing so. It’s enough to feel, down in your bones, that the act of turning a thing that exists into data is an insufficient process that only leaves you ashes of what was once something more. And the post office leaves your letters alone, because they know there would be a massive uproar if they tried to replace written words with systemized ones. 

But human beings themselves, in all their unfathomable richness, are being turned into systemized humans in nearly every facet of modern society. Many times, this is for no good reason other than familiarity - people are used to being data by now, so much that we often don’t even notice it happening. Expanding our intrinsic distaste of this to domains where it’s grown routine is something worth cultivating. Again, this isn’t the end of the discussion and I will still make a point to sketch out the material consequences of over-systemization as we explore case studies. It’s just worth stating every now and then how much the world rebels at this sort of compression, and that even cases that aren’t actively harmful are often just grimy and anti-human.

To end with words that are more than their literal meaning - Lisel Mueller’s “Monet Refuses the Operation” is one of my favorite poems of all time. It includes this profoundly beautiful sentence:

I will not return to a universe

of objects that don’t know each other,

as if islands were not the lost children

of one great continent.

We live in a universe of objects that know each other, with details that intersect in ways we can’t possibly hope to capture into systems. That’s enough reason to be careful with what we destroy.

Subscribe now

Today, we’ll be talking about a study by Shrager et al. on a statistical anomaly in cancer diagnoses. At Dr. Shrager’s thoracic surgery practice, they noticed a 2-fold increase of pulmonary lobectomies for lung cancer for 65 year olds as opposed to 64 year olds. Why such a stark increase over such a short time? Dr. Shrager had a theory: the increase was driven by eligibility for Medicare, which begins at age 65. Besides being exactly as depressing as it sounds, it’s a stark case study of the costs that come when we get clumsy about systemization.

It’s worth being precise about which system we mean, though. Diagnosing is technically a system: a tremendously complicated human being goes in, a yes/no flag comes out. But as reductionist as diagnosis is, it’s not exactly something you can avoid doing, and at least medicine is a domain that takes correspondence to the natural world seriously. There are all sorts of valid concerns about doing diagnosis right, but those concerns are being treated as problems by people with the power to fix them.

Receiving a diagnosis, though, is another matter entirely. It doesn’t matter how carefully you ensure that your tests have the appropriate accuracy if you never run the tests at all. And cancer, like many medical conditions, is tremendously sensitive to the timeliness of diagnosis. If you have healthcare access that’s cheap and straightforward to use, you might get tested on a regular cadence and be highly likely to catch it early. But many insurance plans technically count as “covering” a patient while still being expensive and requiring a lot of personal bureaucratic labor to actually utilize. This makes it even more complicated to evaluate the impact of how we’ve systemized access to diagnoses; it’s not a simple yes/no but a complicated dance of convenience, cost, and perceived risk.

Not all insurances are created equal - but the coverages of over-65 year olds in the United States largely are. Fewer than 1% of Americans over 65 are uninsured thanks to Medicare. The exact statistics of Medicare vs. other forms of insurance were not available to this study, but that lack of data makes the conclusion all the more striking. After all, you would expect to see a larger effect in a study of strictly uninsured people who went on to receive Medicare than a study of everyone with pre-mortem cancer. (As this study was looking into access to diagnoses, the study excluded patients whose cancer was only found post-mortem.) But this study didn’t consider insurance information at all: it simply stratified patients by age and let the story about insurance pop out on it’s own accord.

And pop out it did! The primary metric they used was “age over age”, an adaptation of the financial “year over year”. It’s the difference in diagnosis rates from one year to the previous one, divided by the rates of the previous one:

[(% of cancer diagnoses at age n) - (% of cancer diagnoses at age n-1)] / ( % of cancer diagnoses at age n-1)

But you can tune out the details of the math: just know that the bigger the percentage, the more that diagnoses increase as patients get one year older. For example, Stage I lung cancer has a 4% AoA increase at age 64 - a slight increase to the diagnosis rates for 63 year olds. Stage I breast cancer and colon cancer actually go down, with a -2% AoA increase each, and prostate cancer holds steady at a clean 0.

But the figures are dramatically different when the jump is from 64 to 65. Stage I Lung cancer has a 23% AoA increase. Breast cancer, which was -2% a mere year ago, now goes to 10%, while colon cancer has a colossal 33% increase and prostate cancer goes from steady to 15% AoA. The fact that this is a catchup of “overdue” diagnoses is plainly evident in how the AoA figures crater on the jump from 65 to 66: -1%, -6%, -11%, -6%. 

In other words, you can learn exactly which age people become eligible for Medicare if you just watch for the spike in diagnoses. (And to forestall any discussion of some sort of exactly-age-65 specific time bomb in the human body, Shrager et al. note that this pattern was decidedly not observed in a study of lung and breast cancer off of the Canadian Cancer Registry.) The system that decides who gets diagnoses in the United States is based on insurance, and insurance is broken, with obvious and disastrous consequences for the health of millions. Indeed, Shrager et al. cite a study showing that insured patients with cancer over the age of 65 are more likely to undergo surgical intervention and have better 5-year cancer specific mortality rates than people who are younger but not uninsured. Access is so critically important that it even trumps age! 

Their paper ends with a pointed note that this study should be considered in the political context of Medicare-for-all. The American insurance system creates this gap in diagnoses that don’t correspond to cancer’s existence in the real world; universal coverage would go a long way to closing it. It’s not an easy fix, but it is a simple one - access to healthcare makes your health better. But since we’re here to look at this from a desystemic perspective, let’s ask ourselves: what do we do with American cancer data before we have universal coverage? Because these biases aren’t just affecting the real human beings who live in the diagnosis gaps - those human beings are also being turned into data that flows upwards, into algorithms that drive decisions affecting even more human beings.

Let’s say that some well-meaning hospital executive reads the same study we did and thinks -- wow, okay, we need to fix these diagnosis inequities. Let’s use machine learning to predict which patients are mostly likely to test positive for cancer and proactively reach out and get them tested. We’ll ignore the insurance gap entirely and look purely at the data! Well - we know what looking purely at the data gets us, don’t we? We just finished figuring out that it shows a massive spike in cancer diagnoses at age 65, and sniffing out massive spikes is what machine learning does best. As far as a predictive model trained on data from the United States is concerned, there really is an exactly-age-65 specific time bomb in your body that causes a spike in cancer diagnoses. 

How do you control for this bias in your model? Well...you don’t, really. You could artificially weight the scores to some target AoA, but what’s the “right” target for AoA, anyway? The fundamental problem is that you want your model to guess who has undetected cancer, but the only data you can feed it with are patients with detected cancer. So any correspondence break between cancer in the general population and the patients that actually get diagnosed can’t help but feed that bias into that model, compounding the tragedy of the original problem. The impact of insurance inequity is a group of 64 year olds who have undiagnosed cancer because they’re waiting for Medicare. The impact of doing statistical analysis on data generated by inequity and then using it to drive decisions is another group of 64 year olds who have undiagnosed cancer because they’re 64 year olds.

If the first post was the “what” of Desystemize, this is the “why”. Studies and journalistic exposés of broken systems are everywhere, making it clear that we need to look down. But on the heels of “we need to look down” is an equally crucial, all too often ignored echo: “And until we do, we can’t look up.” A broken system at the interface with the real world poisons analysis from the root and makes it inevitable that you’ll perpetuate existing inequity. We need to finish our vegetables before we eat our cake: when the vegetables are slow, human-scale correspondence work and the cake is flashy data science that gets more automatic every year, you can see where the resistance comes from.

This article will not start a revolution. Predictive models for cancer (and all sorts of other diseases - who knows how many other conditions have this same bias?) are being created at health systems all around the country as we speak. They will be turned on, and they will be used to drive clinical decisions, and they will have a significantly lower score for 64 year olds than 65 year olds because 65 is a much more common year to get diagnosed with cancer than 64. The message that we need to slow down is not a catchy one, because turning on the machine that neglects 64 year olds is a lot more fun than leaving it off. This state of affairs will continue until decision makers and everyone else are used to listening for broken correspondence, with an intuition for what analysis must be abandoned when it happens and the courage to replace something with nothing. If Desystemize can move that needle, however slightly, it will have been worth the effort.

Subscribe now

“Inject detail” - it’s a bit of awkward phrasing, isn’t it? I prefer to use constructive language that focuses on what we’re positively doing, but we don’t have familiar phrases on hand for doing this kind of work. We don’t talk about this stuff enough. That’s my whole point! This newsletter is called Desystemize and not The Detail Injector because -- well, because The Detail Injector would be a terrible name, but also because our language in this space is so geared towards building systems up and not tearing them down. By necessity if not choice, these will be stories largely focused on things going wrong. I prefer to highlight wins over losses, but as we’ll discover, the wins are few and far between and the losses are all around us. Still, negativity is no way to start something new, is it? Let’s start with one of those few wins so we can see what it actually means to desystemize. Let’s talk about counting ticks.

Counting is a system, one a lot more profound and powerful than we give it credit for. After all, if I pick up an animal, count the ticks on it (the “tick burden”), and then set it loose again, I’ve created something from scratch. We now have a number, where before there was only a messy and detailed world. When we set that animal loose, we can never travel back in time to the world that was then and have another look, but the number will survive as far in the future as we care to take it. We can study a tract of past we can’t revisit because we’ve made a number that serves as a mirror to it. Systems are great at creating things!

And you’ll need those creations if you, for example, want to study the spread of Lyme disease. One of the people doing this studying is Dr. Richard S. Ostfeld, whose book “Lyme Disease: The Ecology of a Complex System” excellently records the investigations of his lab group. A number of studies in the 80s and early 90s all leaned towards white-footed mice as the primary host of the ticks that caused Lyme Disease, at least in the northeastern United States. They were plentiful, they tended to pass on the responsible bacterium (Borrelia burgdorferi) to ticks feeding on them, but most importantly: they had the highest tick burdens. It was thanks to those counts that we could ID the guilty party. 

Saved again by the power of systems. But remember, the whole thing that makes counting so powerful is that the past is unreachable. We’re completely reliant on the field counts being an accurate way to visit the past. Is the count of ticks on a given animal really the same thing as the number of ticks on a given animal? This is where Dr. Ostfeld comes in with, for my money, one of the most beautifully simple experiments of all time:

"My research group has set and checked many hundreds of thousands of live animal traps over the years. When we catch a white-footed mouse, we remove the animal from the trap and hold it by the scruff of the neck to check its sex, breeding condition, identity (the number on its ear tag), body mass, and the number of ticks that are attached to it. For some unknown reason, blacklegged ticks on mice orient toward the ears, which are big and only sparsely covered with fur. By carefully inspecting the ears and face of a mouse for about one minute, we detect about 90% of the ticks that are attached to it. We know this because we have retrieved hundreds of mice to the lab and held them in wire mesh cages (supplied with water and their favorite foods) over pans of water for up to 5 days, longer than larval blacklegged ticks typically stay attached before they drop off the host. (The pans of water beneath cages contained much more than just ticks—this was a messy and challenging task, but sometimes we must make sacrifices for science.) So we have a full count of tick burdens on many individual mice that we also inspected in the field. Repeating the same process for chipmunks (Tamias striatus), we know we detect about 60% to 75% of the ticks during our one-minute inspections in the field. For all other mammal and bird hosts we trap, counts in the field are such wild underestimates that we don’t even bother. Blacklegged ticks on these other hosts tend to distribute themselves over the entire body, where fur or feathers can be dense and thick and the ticks are impossible to see. If hosts are anesthetized, they can be inspected somewhat more carefully, but even these counts tend to grossly underestimate actual tick burdens."

As it turns out, some species are just easier to count ticks on than others, and those field counts we hoped were a mirror into the past actually just tell us which species have the most visible ticks. In response to this, Dr. Ostfeld’s lab group (led by Kathleen LoGiudice) collected as many mammal and bird species as they could to test via the pan method. Even this has its difficulties. Short-tailed shrews, for example, have an observed skill at staying away from traps, lack of external large ears for tags, and are difficult to keep alive in the lab - Dr. Ostfeld muses that this is probably the reason they’re held as unimportant in Lyme disease ecology. Still, the pan method worked for a variety of animals, and revealed a surprising truth; white-footed mice have among the lowest tick burdens of all of the mammals tested. But since those ticks they do have are prominently visible on their ears, sparing them from the normal dramatic underestimation of field counting, equating field counts with true numbers gave them an undeserved prominence. They’re still a key part of Lyme disease ecology, but far from the monolithic single species theory that had been previously supposed. 

From an ecology perspective, this is just science working as usual; a system that did a bad job of mirroring the world was replaced by one that does a better job. For us, the most interesting part is one particular sentence: “For all other mammal and bird hosts we trap, counts in the field are such wild underestimates that we don’t even bother.” This is what it means to desystemize: to stop believing that a given process models the part of the world it was meant to model. The count is not the number, and we’ll stop acting like it is. And as happy we are that it happened here, looking into the details of what it required paints a grim picture.

First off, we should note how fortunate it was that this experiment led effortlessly to a new systemization. We proved one system (field counts) wrong by creating another system (pan counts) with a design that guaranteed it was more accurate, then showed that the numbers were dramatically different. But field counts would be wrong even if we didn’t have pan counts to replace them. It’s easy to wean people off of a bad number when you made a good number to refute it; far harder to make the case when there’s no replacement. When thinking about desystemization, a replacement system is a luxury, not a guarantee. The far more common outcome is simply walking away with humility that the thing you’re investigating is too slippery for systemization.

There’s also the matter of technology - or more specifically, the lack of it. Our ability to remix, compound, and analyze numbers has exploded in recent years: machine learning algorithms that pull “insight” from thin air, enormous stores of training data, all sorts of statistical techniques that promise to get more mileage out of any numbers you care to feed them. But our ability to start with the right numbers is still bounded by however long it takes a few ecologists to sift through wet mouse turds and count the ticks. Every year our tools for analysis get farther and farther out of sync with the human-scale tools we have to check correspondence with the world, making our eyes bigger than our stomachs and increasing the temptation to assume the data must be all right so we can get to the fun part. 

And finally, there’s the cold and terrifying truth behind nearly every story of oversystemization: nothing about the field counts themselves indicated they were wrong. The ticks on those mouse ears really did exist! And even someone willing to admit the field counts are gross underestimates would be forgiven for thinking that the inaccuracy would be roughly even for all species, instead of “gross underestimates except for specifically white-footed mice and chipmunks, for whom the counts will be close to accurate in a way that will make them seem much more tick-infested than all the other species”. Since it’s systems that generate the data, the errors will be systemic as well; this defies our intuition of errors as a sort of random fuzz around a true center.

Dr. Ostfeld didn’t start that paragraph by noting such-and-such statistical technique clearly indicated something was off with the tick counts. He started with the sentence “My research group has set and checked many hundreds of thousands of live animal traps over the years.” In other words, it was familiarity with the data-generating process that enabled the lab group to imagine this potential vulnerability and come up with this experiment. By the time the data gets into the hands of analysts, it’s too late to fix. You can’t math your way out of a wrong number. This mistake was caught only because it was the same people generating the data as analyzing it. Which, great for ecology - but as data science becomes more and more specialized, it will be increasingly done by people who are explicitly and solely data scientists. And they’ll inherit datasets from repositories somewhere and never catch a single one of these systemic errors because they couldn’t sift through the wet mouse turds even if they wanted to.

Let’s recap. We started with counting, probably the simplest systemization that exists in the entire world. It turns out that “pick up an animal and count the ticks on it” is not only an insufficient way to learn how many ticks are on it, but it’s insufficient in a non-random and incredibly misleading way. To catch this mistake, you have to be intimately involved in the data-generating process and be curious enough to have a hunch to design an experiment. That experiment involves a lot of hard, tedious, and gross work. If you pull it off, then you’ll get a good number only because you happen to be working on a question where the system was bad because of its own merits, and not because the domain itself was inherently resistant to systemization. That’s what it takes to pull off proper desystemization. 

But what about domains that ARE resistant to systemization, where we can’t go any further than “I don’t know what the right answer is, just that there’s no good way to find it?” What about findings that are entirely based in machine learning techniques and invite no easy way to correspond them with the world? What about data that’s actively trying to hide it’s generating process because it violates your privacy? What about findings that are expected to come at machine speed, with no stakeholders willing to commit the time and effort for correspondence work? In short - we saw how hard it was to make this easy looking system (“You can know how many ticks are on an animal by counting them”) meaningful. Given this, how meaningful are all of the other systems around us?

The bad news is: the crisis of meaning in the modern world is pretty much exactly as bad as it sounds. The good news is: once a week or so you can get an email about it. Welcome to Desystemize!  

Subscribe now