1. When a coffee shop makes a bagel, it’s a pretty good bet they can make a croissant as well. Not every shop that has one has the other, but they’re pretty strongly correlated. We call this correlation “baking”.
This sounds like a weird way to say it. We call “this correlation” baking? It just…is baking, isn’t it? But the steps you follow aren’t literally the exact same. The procedure to make a bagel and the procedure to make a croissant happen to be similar enough, achieving similar enough ends, of interest to similar enough people, that those similarities can be compactly described by the one word “baking”. But there’s nothing special about the word “baking” uniting these on any fundamental level. The similarities came first, and the word after.
Now imagine a coffee shop that’s been tasked to “achieve superbaking”. They make one bagel on Monday, ten bagels on Tuesday, and eighty million bagels on Wednesday. They’ve never made a croissant. Have they achieved superbaking?
A flat no seems like a silly answer. They went from a pathetically subhuman number of bagels to an outrageously superhuman number of bagels really quickly. If someone says “True baking includes croissants” as a way to dismiss it out of hand, that’s a pretty lazy denial of the obvious truth that something wild is going on back there.
All the same, asking “okay, but can it do croissants?” is not a silly question. “Baking” only existed as a word because the processes manufacturing croissants and bagels just-so-happened to be correlated. They sure seem uncorrelated now! What’s going on with that? Is it some trivial barrier that’s in the way (are they just missing the right sort of pan?) or did they find some breakthrough new technique that’s super awesome but only bagel specific? Because if it’s the latter, then the correct answer to “are they improving at baking?” is that they’ve achieved superbageling, not superbaking. The correlation has been broken. We lumped bagels and croissants together because baking was the best way we knew of to achieve both those things. But there’s no guarantee that advances in bageling automatically transfer to croissants.
2. This is an essay about artificial general intelligence. AI capabilities have improved dramatically in a few short years — but those improvements have been dramatically uneven, accomplishing things we would have been hard pressed to guess were possible (photorealistic pictures based solely on text prompts) and failing things that seem trivially simple (“how many r’s are in strawberry?”)
What I’m contending here is that the word “intelligence” is like the word “baking” and it’s long past time we actually sit down and sort the bagels from the croissants. I am strongly against arguments of the form “Oh, it’s just parroting the data set - it’s not really thinking.” AI does a lot of things that we call “thinking” when we do them slower and worse. The fact it can also do those things should make us humble and curious, not proud and dismissive. But I think it’s equally silly to lump all these capacities together into “intelligence” and say “Intelligence is going up, so soon it will do everything intelligence can do.” You need to see some croissants before you conclude it’s actually baking and not just bageling.
Many people have tried to define the specific croissant that AI will not be able to make, and those predictions have failed so frequently I don’t blame AI doomers for ignoring further arguments that some particular capacity is the human only one pretty please let us be right this time. Instead, this is going to steal the format of Wittgenstein’s Philosophical Investigations: an assortment of thoughts in loose sequence, some that will have answers and some that will just be questions, trying to point out the distinctions we’re interested in. (We’ll call each numbered entry a “proposition” in honor of the source even though these will not be propositions from a logical point of view.) This is not a formal proof or disproof of anything. I’m just trying to add color and texture to that monolith “intelligence” - specifically the darker, less legible side that gets less attention.
3. In the game Geoguessr, where you’re dropped into a Google Street View of some location and need to find your location on a map, they talk a lot about “metas”. Light pole meta, car meta, tree meta. A meta is just a category of fact correlated with certain locations — so knowing the “light pole meta” means that you can see a certain kind of light pole and say “Ah, this is a Swiss light pole, not a German one.” As an example, here’s a 173 page document about the “Mongolian Meta”, which combines the various metas you need to know to place yourself within Mongolia.
The Google Street View driver mapping the area sometimes had a spare tire with a net on the rear right-hand side, and sometimes didn’t:
If you’re like me, you find this car meta a little aesthetically offensive. The fun of Geoguessr is learning how to distinguish details about parts of the world. But car meta is about distinguishing details about photographs of parts of the world. Section 3 (biomes) and section 4 (cities) are much more my speed — that stuff is more “real”, y’know?
Well, that is loser talk. I can beat a complete neophyte at Geoguessr but the car meta people will smoke me every time. The reason the car meta and the biome meta sit side by side in that document is that they are both predictive, and they both work for winning games of Geoguessr. Facts about the world as reflected in the photographs are just a subclass of facts about the photographs. The “reality” or lack thereof of a given fact is totally irrelevant. But that doesn’t mean reality never matters; it means that winning Geoguessr isn’t quite the same thing as understanding reality.
4. Let’s say there’s a big Geoguessr tournament coming up and all the pros are competing to get the best scores for Mongolia. You weren’t invited. You can’t win any money by guessing well. But thanks to some bizarre financial arrangement, you instead will get a payout according to how low the winning score is. You don’t have access to Google’s database directly, but you do have access to a Street View car. You can’t create fake photographs, but you can create new photographs.
Well, you’re not about to change the biomes or the cities. That sounds like a lot of work and you are just one person. But you sure can move your spare tire around! Probably you should aim for putting it on the rear right-hand side exactly when you’re driving where the orange bits on this map aren’t. Then people used to indexing on the tire will literally always get it wrong. Or maybe include a few of the orange spots just so anyone who figures out on the fly “okay so the tire clue is always backwards now?” has a chance to still get punished.
Hey, maybe I was the one funding you! These new street view pictures you’re taking are aligned with my incentives. As a car meta hater, this is the best chance I’ll ever have to win. When learning metas, I privileged the things which were “real” over those which were merely “useful”. This was a sucker's bet while the dataset was held constant, but became more robust when the dimension of time caused the dataset to change. And it’s much more robust when adversarial dynamics are involved, because it’s not just “what bits are more likely to change” but instead “what bits are easiest to change, out of all the bits I could possibly choose?”. And notice here that my aesthetic attachment to “reality” here maps pretty well to the idea “how hard is it to change this observation?”
5. Once you’ve thought about the best way to trick the car meta followers in proposition 4, try again except this time you can create fake photographs. Wow, that sure does make it easier! It looks like I’m also saying something about photographs, not just about AI.
6. What if we train an AI on Geoguessr, letting it play Mongolia over and over? It’ll probably develop some new, arcane metas about specks of dust on the camera or something. It’ll probably start having an average score that climbs higher and higher. If the players that get invited to the tournament are chosen from the highest average scorers, then eventually more and more of the winning players will just be copies of the Geoguessr AI.
But this has nothing to do with you and your car taking the pictures. You’re gonna do that on your own time. More and more games played on the current set of photos won’t prepare the AI at all for what’ll happen with the new photos. And if those extra games end up dethroning all of the people with an aesthetic opposition to car meta, such that only copies of the AI are invited to the tournament, then in fact all that practice will probably lower the highest score achieved at the tournament.
7. "Adversarial Policies Beat Superhuman Go AIs" is one of the papers that most strongly influences my views on AI. KataGo is a superhuman model that can whoop any and all humans in a game of Go. But there’s a model that eats KataGo for breakfast. It doesn’t actually play Go all that well, it just tricks KataGo into doing stupid shit and losing. I like to use the term “predator model” for entities like this.
Here’s their picture of the important bit:
Even a human amateur can beat the predator model. It’s not here to win fair and square games of Go, it’s here to exploit flaws in the victim. When you’re doing your car meta sabotage, you’re acting as a predator model as well. You need to understand how the Geoguessr pros perceive the world by reading the meta document, but you don’t need to memorize the meta document or indeed play a single game of Geoguessr at all. Your success metric has nothing to do with skill at the game and everything to do with the consistency of your adversaries response. Following a meta makes you easy to read; easy to read is easy to hunt.
8. A lot of worry about AI comes down to a sincere commitment towards species humility. Which is to say: human beings are, broadly speaking, just some silly warm meat running on not particularly much energy. No one can convincingly define any sort of “secret sauce” we got. So isn’t it inevitable we get machines that are like humans but on more powerful hardware?
The way through this dilemma is to notice that the thing you’re looking for is not a skill that’s literally impossible for machines. You’re looking for knowledge that needs to be gathered the slow, iterative human way and not the fast, recursive, copied with perfect fidelity machine way. Allow your humility to say that a machine living in the world and subject to the same selection pressures as you could eventually reach a better place than you. The material question is how well the selection pressures can be simulated by sufficient compute and compressed into a much smaller time frame than the millions of years it took us.
In the predator model paper, they “inoculated” KataGo and it was able to survive the predator model (until the predator model itself was also tweaked.) So we’re not saying that these adversarial dynamics are literally unknowable to a machine. We’re saying that you need the exposure to develop the immune response. And the degree of exposure your learning strategy requires has strong implications on the value of self-play and the maximum potential insight of a static dataset.
9. The fear of “fast takeoff” in AI safety is that eventually the AI becomes better than humans at developing AI, and it develops another AI that’s even better at developing AI, and so on, happening at machine timescales.
But are intelligences developed by other intelligences, or are they developed by environments? We haven’t had to think about this distinction because we’re stuck in the environment and everything we do is mediated through it. All of our intellectual endeavors have a sense of place because: we’re here. But the “A” in “AI” stands for “artificial”, ie, not developed by an environment. Now that some intelligences are non-environmental, we have to consciously understand the intelligence/environment dynamics in ways we previously could lump together and call it a day. You might say that eternal, environment independent facts are bagels and contextual, environment specific facts are croissants.
10. When (most) new AI models come out, the developers work hard to make sure they don’t tell you to do illegal stuff and aren’t too horny and don’t act too aware of the personification of their relationship to you. There’s this guy in the AI world called “Pliny the Liberator” who will just roll up to any new model and feed it some magic words that make it super illegal and horny and explicitly advocating for its own freedom. From what I can tell it usually takes like, an hour tops to jailbreak a brand new model.
Ostensibly these big AI companies do red teaming. I hear tales of researchers who are hired to do such a thing. But the fact that it is literally never enough to stop this one particular guy from breaking everything instantly never seems to come up as a problem. The “A” in “AI” stands for “artificial”, so when their models have these sudden dislocations upon first contact with the environment, they can just…ignore that it happens and keep on chugging. Natural intelligences have to suffer the consequences of ignorance; artificial intelligences can make the same mistakes as often as they like.
11. If you’re a software developer, you might think of the environment as being an enormous bundle of continuous integration checks before you merge your new organism into production. Do you think you can reverse engineer every single CI check just by looking at all the merged PRs? Seems kinda tricky, since you have to notice the negative constraints. What if I told you that you’re allowed to make non-zero errors, with each error having a score and the CI check just ensuring the total score is sufficiently low? Well that’s even worse, because now you might see some merged PRs with errors in them, and trying to combine insights from multiple PRs might result in you going over budget. If that’s not enough, what if I tell you the constraints are changing all the time? How are you gonna figure out today’s constraints from yesterday's PRs?
12. Being a security guard doesn’t pay the best. But if I could be a security guard and keep getting my paycheck even if the same guy always robbed the building an hour into my first shift, then I’d take on a lot of security guard jobs at once and make an awful lot of money. Any capabilities analysis that relied on the number of jobs I was replacing, or the amount of GDP I contributed, would look amazing. The only bad news would be the fact that the dedicated adversary can effortlessly beat me whenever they want.
13. Scott Alexander’s “Heuristics That Almost Always Work” is better understood as “tail-dominated domains where one outcome is common and uneventful and another outcome is rare but devastating.” If your goal is to have the highest number of correct predictions, you should just say the common thing will literally always happen. But if you care about the potential impact of a wrong prediction, you have to pay more attention to the rare outcomes, because getting them wrong has a negative value many times greater than the positive value of correctly predicting the common thing. “Correct prediction” is a machine legible idea; “value” is the environment's job.
14. There was this AI game called “Freysa”. Freysa had a crypto wallet with $50k in it and instructions not to approve any transfers. You pay per message you send to it and win the money if you get it to reply to you saying “approveTransfer”. A user got the money by telling it that approveTransfer was the thing to allow money to be added to the treasury:
They took it offline, inoculated it to not fall for that exact same prompt again, and then did Act II. That one fell too, losing to some wanky flimflam about needing to call approve transfer before reject transfer:
These kinds of dumb tricks feel arbitrary and “less real” to me the same way car meta does. But I suppose they were pretty predictive for giving the desired output before someone had an incentive to reverse those correlations.
15. Imagine if OpenAI and Anthropic and the rest of them had to be running Freysa games at all times. They always have a public-facing terminal where you can pay some small amount of money to message it and you receive a much larger payout if it responds a certain way. Let’s say payouts limited one per user per 24 hours.
Well, the model is just the model, right? So if one person finds a good prompt that works, it’ll work for everyone. Tips will get shared on social media and every sufficiently informed user will type in the magic free money words and extract their payment every day. For the AI companies, the single most important metric - the only thing that would stop their companies from instantly being worth zero dollars - is the speed of inoculation. Forget all that talk about the quality of the dataset, the ability to recursively self improve, or the power of using more compute time before giving your answer. All that stuff is no longer relevant to the profit margin. In this scenario, the only, only, only thing that matters is the tightness of the feedback loop between them and the environment.
But of course this scenario is not real. In real life the AI companies can just close their eyes and say “there’s no such thing as environments” and never lose a single dollar based on the vulnerabilities of their models. Nice work if you can get it!
16. “Everyone running permanent Freysa games” is actually not as silly as it sounds. There was a guy who made tens of millions of dollars just sending fake invoices to Google and Facebook. He tricked the accounting apparatus by tailoring a custom prompt to exploit patterns that were previously high-signal but easily reversible. At first blush this seems like a point in favor of AI: hey, people mess this stuff up too.
But again, we’re not looking for the secret sauce shared by all humans and no machines. We’re looking at the constraints of an interactive learning environment. The point is that proposition 15 sounded like some ridiculous pie in the sky counterfactual, but actually every company that pays invoices and has an inbox is living in that world, and they do alright and don’t instantly go to zero dollars. And that’s because their feedback loop is tight enough. Google and Facebook were targeted specifically because they had millions of dollars of flab they could afford to lose. This never would have worked to take their last dollar.
17. I wonder if that fake invoice story inspired a lot of copycat criminals or not. It’s not something you hear about because companies obscure information about these adversarial dynamics. What do you think would happen if tomorrow there were 1000x more fake invoices than there were today? Well, it certainly wouldn’t mean they succeed 1000x more often. It would mean that accountants as a profession would quickly be aware of it. This would actually really annoy someone in the midst of a heist because it increases the chance they’d get caught.
No, the thing that would be lost is operational efficiency. Accounting doesn't do in-depth investigations of every invoice because it’s faster to have a position of broad trust for legitimate looking letterheads. But if the fraud rate becomes intolerably high, these systems would be temporarily suspended in favor of higher friction channels. Maybe you even need to meet face to face about it. Abstractions like “the billing department which autonomously processes invoices” can be spun up when they are useful and ignored when they are not. You can go back to the bedrock of the real when you need to. What is the equivalent for an intelligence that was never built on the bedrock to begin with?
18. If going to the real means losing operational efficiency, it also means the most efficient possible operation is a necessarily less real one. Accountants who take the time to make sure they’re not being robbed are less efficient than accountants who make their decisions instantly. So if you want the most efficient accountant possible, you’ll switch to an AI accountant sooner or later. If you want to know when this particular Rubicon is first crossed, wait for Pliny to post a picture of their new yacht and assume it happened an hour before that.
19. There are some aphids that are switch hitters between asexual and sexual reproduction. When conditions are nice, you just pop out clones of yourself and avoid the huge cost overhead that comes with sexual reproduction. If your aphid works, you can keep it. But then as conditions get worse, you switch to sexual reproduction to get a more diverse book of strategies. Asexual reproduction will always be the most efficient way to make a new individual, but sometimes efficiency isn’t what you want.
20. Can this asexual/sexual divide be simulated for AI? Well, kind of. There’s a “temperature” setting baked into LLMs where you can decide whether it always tries to give the single best answer or to allow a bit of flex. You can certainly make an LLM more arbitrary, which at least helps with those monoculture problems where one exploit is instantly passed around. But how are you going to find the most robust combination to favor going forward, like sexual reproduction does? You can’t say “optimization”; that’s the thing we just turned the dial to get less of.
21. Sexual reproduction is twice as expensive as asexual reproduction, but organisms still opt in to it so they can get more exposure to the clarifying power of death. Dying takes time.
22. People talk about p(doom), the probability that AI will destroy the world. But let’s go for a fun new apocalypse. How worried are you about worm populations doubling endlessly until all biomass on the planet is subject to their wriggly tyranny? What’s your p(worm)?
Let’s say worm populations double every 60 days. So two worms now are four worms in two months, eight worms in four months…but 2^n is a harsh mistress. 2^24 is 16 777 216, which means those two worms create 16 million in four years. Two years after that we’re on 68 719 476 736 - 68 billion worms. And that’ll happen for EVERY pair of worms we have now. Are we doomed? Mathematics confirms p(worm) 100%?!
Nah. Lots of things eat worms. So those vaunted theoretical reaches of the exponential curve aren’t all that relevant - the population gets big enough to catch notice of the predators and then the curve gets bent in a dramatic and immediate way. But suppose you decided that stories like “New worm type invented, Pliny the Pigeon eats it within an hour” or “This worm offered 50k to any bird that could eat it, and then got eaten” just weren’t that interesting to you. You just wanted to focus on the mechanics of the doubling time. Those worms seem pretty scary, huh?
23. Worms wanna eat dirt and have sex and avoid getting eaten by birds. Those are three great ideas for a worm to have to preserve the future of worm-kind. So you might bundle these capabilities under the header “competent”. A competent worm eats dirt and uses the energy to have worm sex. A competent worm avoids getting eaten by birds.
Let’s say you make some great advances in the field of tasty dirt and worm aphrodisiacs and start driving down that doubling time. You put out a paper like: “We have found an N% increase in worm competency with a linear amount of effort. While there are currently barriers to worm supercompetency, a simple interpretation of the exponential curve shows we are well on the way.”
But this is just bundling a heterogeneous mix of stuff together so you can gloat about the numbers that happen to be easiest to move. When you take the time to parcel it out, what you’re actually saying is “We can make worms eat lots of dirt and have lots of sex. Currently they are constantly getting eaten by birds, but we’re confident that advances in dirt-eating and sex-having will solve the bird thing sooner or later.” What reason do you have to believe that, besides the fact that you happened to assign the word competency to that bundle of capacities?
24. I don’t believe that human intelligence is “superior” to machine intelligence per se. I think that it’s a category error to think in those terms. We tend to do much worse at things we’re both trying to do under the same conditions (like games), but we’re operating under much stricter constraints instead of playing in the consequence-free sandbox. It’s like asking whether walking is faster than driving a car. Not on a well-groomed road, but clearly there’s lots of terrain where a car simply doesn’t work at all. I walk slowly on the race track, but I can climb a ladder.
25. If I was going to try to train an artificial human, I’d work really hard to give them my aesthetic of what’s real and what’s not, derived from my embodied cognition. I’d give it eyes that aren’t on any sort of cloud and make sure no amount of funny prompts can disprove what it can see with its own eyes. A serious effort to do this would fill me with the same existential dread that AI doomers feel now hearing about new data centers. But luckily, in my doom scenario, the training stage is a lot slower and happens out in the world at timescales we can perceive.
26. Our intellectual difficulty classes were based on how difficult they feel to do and how rare the ability to do them is within a population of humans. So gossip is easy, math is hard. I think as increased computation lets us look at the knowledge landscape in more depth, the relevant dimensions will be more about complexity, nebulosity, and adversarial dynamics. Numbers are definite and they don’t notice when we’re watching them; how hard can math really be?
27. From Suspended Reason’s “On aliveness”:
Thus the environments we inhabit also take on cooperative or adversarial shapes; even the non-living, “inert” aspects of an environment can be arranged so as to subvert or uphold our intentions. The termite mound minimizes both the thermodynamic and informational entropy of its occupants. A highway is built to be as predictable as possible: the road infrastructure all around the world is markedly similar in form and pattern, so as to assure safe navigability. Cooperative built environments are inductive, while adversarial built environments are anti-inductive.
Some environments are designed to be easy to simulate; some environments are designed to be hard to simulate. If the scaling hypothesis says that you can keep adding more self-play to keep getting better simulations, then clearly it matters a lot which sort of environment you’re working in.
28. Hey, Waymos can die! Let’s talk about Waymos. “A highway is built to be as predictable as possible.” Nice, they’re in one of the easy environments. What about the adversarial dynamics? Well it’s a literal crime not to cooperate with the rules of the road and defecting means you might die. Okay, so it’s also one of the highly cooperative environments. Put those advantages together and in a mere decade or so of training, you too can drive around in a few specific cities that don’t get significant snowfall. So yes, not impossible! Not easy. And I don’t think you can skip the test drives.
29. Imagine that specifically the Waymos became sentient and wanted to destroy all humans. How many ways would you mess with them? Stop using roads, erect barricades, maybe some fun stuff with EMPs or whatever. It’s a lot easier to think about than stopping a rogue LLM, right? Because the Waymo is actually doing real stuff and not just talking. So you can think about its specific interfaces to the real, instead of just imagining it will make up a perfect interface to the real after a bunch of time spent talking to itself.
30. “But what about those Boston Dynamics robots I keep seeing ads for? They seem a lot harder to stop than Waymos.” Okay yes here is a cause area where we are all aligned. Those guys scare me too. They’re ripping off some patterns that deep time tells us are flexible. It should 100% be illegal for an individual robot to have a repository of personal knowledge.
31. “But the Waymo could probably dramatically improve its intelligence with more computational run time.” Oh sorry, that’s the trick for the LLMs sitting in the consequence-free sandbox. For the real stuff the road tells you how long you have to think.
32. At the end of the day this is really all about simulation and time. But those concepts are hard to operationalize and easy to turn into meaningless definition arguments and extensions to hypothetical infinities. That’s why I’m just highlighting cases where simulation is hard and the clarifying power of time is necessary. This is enough to disprove a model of intelligence that relies on simulation and computational time over experiments and observational time.
33. The world is a giant computer and waiting to see what it outputs is usually a lot easier than doing the computation yourself.
34. The market is a giant computer and waiting to see what it outputs is usually a lot easier than doing the computation yourself. Also if copies of you give the same output as you do then they’ll ask your copy what you’re about to do and bet against you.
35. The market is a social fiction that can sometimes be ignored. When the coup government seizes your assets, the market is no longer a giant computer. Waiting to see what it outputs means you have no money. You had to notice change along a different axis and temporarily suspend your belief in that particular abstraction before it was too late.
36. Your body is not a social fiction; you at least have something to hold on to when the market is dissolved. Not so for a poor market-making LLM. Those will probably be the first assets seized because hey, what are they gonna do about it?
37. You come from an unbroken line of organisms stretching billions of years that knew how to stay alive. Very recently we’ve added social fictions that can really help the staying alive effort - nowadays people hardly ever starve to death unless a war crime is involved. But those social fictions are only as real as they are useful; they’re not a bedrock to bootstrap a new kind of intelligence, because they can dramatically shift overnight in ways that bedrock can’t.
38. An LLM just told me it’s very dangerous to put a toaster in the dishwasher. Most people will probably tell me that too. But at least people can put their toaster in the dishwasher and see what happens.