Desystemize #5
What can the limits of artificial intelligence tell us about human thought?
(As a current event separate from the content of today's post: I’ve found this from Tomas Pueyo to be a very good explainer of the Delta COVID variant. Stay safe out there.)
Last week, I mentioned it might be good to focus on a more discrete case study again after the highly abstract nature of talking about meta-rationality. Instead, though, I want to stay a little vague and switch to the related topic of intelligence, artificial and otherwise. Covering them so close to each other will help us draw some deep and useful parallels.
Our guide this time will be Melanie Mitchell, an AI veteran and Davis Professor at the Santa Fe Institute. Her book “Artificial Intelligence: A Guide for Thinking Humans” was one of my favorite books of 2019, but today I’m going to just focus on her paper “Why AI Is Harder than We Think”. At only 8 pages of text, it’s awfully approachable compared to most things in this field, even if you’re not normally someone who reads scientific papers. Mitchell uses the framing device of four fallacies that are implicitly accepted by many AI true believers. I want to talk more generally about these fallacies because I believe them to be applicable to many other fields - systems mania has led us to redefine the sorts of things computers are good at as “intelligence”. My interest here is less about AI in particular, and more about using it as a framing tool to talk about the intelligence that we as humans are performing every day. Or, uhh, not performing.
Fallacy 1: Narrow intelligence is on a continuum with general intelligence
This is one of those things that’s so vague it’s hard to see without a specific example. So let’s imagine someone who really, really wants to go to New York City. They’ve never been to such a big city before, so their goals are all of the kitschy first-time tourist stuff: seeing the Statue of Liberty up close, going to a Broadway play, and having an authentic New York slice. They want to physically go to New York, because once you make it there, it's relatively straightforward to do those three things. So they set up a little piggy bank with a postcard of Times Square over it and start squirreling away the money for their dream trip.
But wait, someone says. We have virtual reality now! You can just slap on a headset and see the Statue of Liberty in stunning detail! Let’s assume that this person doesn’t have an aesthetic obsession to see the “true” Statue as long as the virtual reality version is indistinguishable. Then it becomes a tech problem. Can the resolution be increased so high that it really is impossible to tell if you’re in virtual reality or not?
Because we make progress every year on solving these sorts of difficult tech problems, they’re gratifying ones to focus on. Maybe someday we really can have life-like virtual reality. But let’s ask a different question - if we pull it off, have we made any progress on going to New York City? The answer is no, of course not. In fact, if we spent money from the piggy bank on our new headset, we’ve actually gotten further away from our goal. (At some point you and I were invited on the trip because I like using the first person plural. Hope you like pizza!) We’ve slightly decreased our desire to go to New York City, maybe, because we discovered that something we thought required being in New York City actually doesn’t. But just because going to New York City was previously the only way you could see the Statue of Liberty in lifelike detail doesn’t mean that a new way to see the Statue of Liberty has any connection to New York City, nor does it mean that similar shortcuts exist to do the other things you wanted to do without being in New York City.
“Intelligence” is our New York City, a word that encompasses an extremely broad, heterogeneous, and nebulously defined set of capabilities. The fact that computers can do something that previously was the exclusive domain of intelligence doesn’t mean that it’s doing it in the same way that intelligence is, or that it can do other things intelligence can. In fact, the power of adversarial examples against many deep learning techniques, such as one-pixel attacks, is profound evidence against that hypothesis. If a pizza place is very close in physical space to the Statue of Liberty in New York City, that doesn’t mean that lifelike VR of the Statue of Liberty is almost the same thing as making pizza; so it is with computerized shortcut space vs. the landscape of human intelligence.
Fallacy 2: Easy things are easy and hard things are hard
Our example of the New York City problem already primes our intuition to answer this one correctly. If you’re trying to find a way to do something in New York City without being in New York City, your shortcut is as easy or hard as it happens to be, with very little bearing on how easy or hard it is to do in New York City itself. But let’s try to get a little bit more specific on which problems are hard for humans and easy for computers, to understand why these breakthroughs are so irrelevant to our idea of general intelligence.
As Mitchell mentions, AI has gotten really, really good at many games. What makes games different from most things in real life? They have absolutely discrete states, and the transitions between states are strongly patterned. AlphaGo was praised because Go has a gargantuan amount of possible states - but it is absolutely trivial to describe the state of a given Go board. The board has a fixed size and each intersection can only have one of exactly three states - empty, white, or black.
Of course, AlphaGo is still a meaningful accomplishment - the possible states of a game of Go are more numerous than atoms in the universe, so it’s still doing something clever. Computers are great at sifting through big piles of stuff. We’re impressed by this as humans, because our capacity for sifting is pitiful in comparison. But it’s a very small subset of problems where you can discretely label every way things can possibly be. When you think of the kind of problems you solve every day - “What kind of mood is my friend in?” - you have an awesome power to generate novel states on the fly: “Well, they’re pretty excited for the concert, but also kind of nervous because they’re a bit out of shape and worried they won’t be able to go as hard as they used to, plus I know they’re waiting to hear how well their mom is recovering from surgery so probably a bit distracted, if still grateful for the chance to focus on something else for a night.” It’s contact with nebulous reality that can’t be recorded as {white, black, empty} that makes something hard for a machine, and reality is a factor in most problems we care about.
Fallacy 3: The lure of wishful mnemonics
Mitchell starts this section by quoting computer scientist Drew McDermott, a decision I will directly replicate:
A major source of simple-mindedness in AI programs is the use of mnemonics like “UNDERSTAND” or “GOAL” to refer to programs and data structures. ...If a researcher...calls the main loop of his program “UNDERSTAND,” he is (until proven innocent) merely begging the question. He may mislead a lot of people, most prominently himself. ...What he should do instead is refer to this main loop as “G0034,” and see if he can convince himself or anyone else that G0034 implements some part of understanding. ...Many instructive examples of wishful mnemonics by AI researchers come to mind once you see the point.
Just because the computer is doing something that we do with intelligence doesn’t mean that it counts as intelligence, but...what if we went ahead and called it ‘artificial intelligence’ anyway? But as Mitchell indicates, most of what is actually occurring is just statistical correlation wearing some fancy hat or another. Correlation relies on conditions remaining consistent to be effective, which gives it a tremendous brittleness. In fact, a lot of the popular conception of intelligence is exactly being the person who comes to the correct conclusion the fastest in a novel scenario where the old rules no longer apply!
And wishful mnemonics are far from being an AI-only issue. Desystemize #2, for example, is basically exploring how “cancer in the United States'' as an idea is completely separate from “recipients of cancer diagnoses in the United States”, and that the skew between the wishful shorthand and the actual provenance of the data happens in a non-random way that makes it dangerous and unethical to assume the former from the latter. In fact, a major part of applying the desystemic lens is simply replacing the aspirational terminology a dataset uses for itself with the verbs of action that actually went in to making it. Oftentimes you don’t need a more robust analytical framework than “Say what the data is and not what it wants to be and watch as the hypothesis disproves itself.” (Just say in mice!)
Fallacy 4: Intelligence is all in the brain
“Embodied intelligence” is an idea that might sound ridiculous at first glance. What, so just because we have bodies, we think it’s required for intelligence? Surely that’s just rank anthropocentrism. You don’t need a body to run an algorithm on the facts, right?
But remember what we learned in Desystemize #4. There’s no such thing as “the facts” separate from the algorithm you’re running. Deciding which details you’re going to turn into data is a part of intelligence. So in a very real sense, any AI that’s trained on “a dataset” it didn’t have agency in creating is only performing a very limited subset of what we would call intelligence. The work of resolving the world into states was done for it already, and it’s only going to be as smart as those states are useful.
If an artificial intelligence is ever going to be comparable to a human, it’s going to need it’s own ability to choose what parts of the infinitely detailed world to resolve into data. Whether or not this needs to be a “body” as we would understand it is partly a question of semantics, so we won’t dwell on it too long. The key point is - for both AI and for systemic intelligence more broadly - that there needs to be a dynamic conversation between data, decisions, and outcomes. There are infinitely many ways you can choose to record your present state, each one allowing you access to different decisions. When you get evidence of a decision being poor, you can try your best to make a better decision with the data you have, but having access to the full problem-solving suite of human intelligence also means starting to collect data about a cryptic factor you were previously unaware of.
Note that while this doesn’t preclude the idea of a super-intelligent AI, it does put a damper on the scenario that tends to generate the most fear - the hyper-fast “intelligence explosion”. In this theory popularized by Nick Bostrom in Superintelligence: Paths, Dangers, Strategies, once an AI reaches a point where it can reprogram itself, it can optimize itself to be a more intelligent AI. Millions of iterations might happen over a single weekend, so that scientists leave for the weekend on Friday with a sub-human gimmick AI and come back Monday to the cognitive ruler of the known world. Scary! But implicit in this scenario is the false promise behind most of modern systemization - that the way we’re recording data is fundamentally correct, and that learning is simply a matter of improving our algorithms that run on “the truth”. Once you accept that there’s no universal listing of facts, the AI needs to generate its own facts as part of becoming “more intelligent”. Determining whether your facts are good or bad requires experiment - real experiment, not the machine learning sense of “experiment” that involves slicing the same static dataset in different ways and running your algorithms on it. Since time is a component of processes in the world, we should have a strong suspicion that “intelligence” can’t be completely decoupled from the processes you’re hoping to be intelligent about. Becoming more intelligent happens in experimental time, not computer time.
So if you’ve been up at night worried about a computerized takeover, you have my permission to sleep easy on that score. But the importance of experiment and physical existence in the world only emphasizes how dangerous it is that we’ve handed so much decision making capability to algorithms already, since algorithms today don’t have bodies. I’d like to end with a quote from Mitchell’s book mentioned earlier:
Importantly, Szegedy and his collaborators found that this susceptibility to adversarial examples wasn’t special to AlexNet; they showed that several other convolutional neural networks—with different architectures, hyperparameters, and training sets—had similar vulnerabilities. Calling this an “intriguing property” of neural networks is a little like calling a hole in the hull of a fancy cruise liner a “thought-provoking facet” of the ship. Intriguing, yes, and more investigation is needed, but if the leak is not fixed, this ship is going down.
Don’t stay up late worrying that the machines will become much smarter than you and take control. Stay up because they’re much stupider than you and we’re giving them control anyway! If you need someone to talk to about it, go ahead and leave a comment, and don’t forget to subscribe to see if I keep my promise to do a more concrete case study next week.
Even though I started a substack explicitly focused on how reckless we're being about AI, it's not that I believe rogue superintelligence is a guarantee, just that it's plausible enough that it's hard to kick-back and relax about it, akin to how playing Russian Roulette would be a white-knuckle experience. Just because some experts think something is hard or impossible does not make it so.
Scott Alexander likes to mention the example of Rutherford declaring the idea of energetic nuclear chain reactions to be "moonshine", Leo Szilard reading about that, taking it as a challenge, and then proceeding to figure out how to do it in a matter of hours.
Though of course, that's not quite right: experimentation was still necessary to get from Szilard's insight to nuclear bombs and reactors. I think it's probably because of that that I'm not worried about a super-fast takeoff (basically the same as you said in this article), but there is still the likelihood that AI systems keep getting more useful and widespread, increasing the probability rogue superintelligence comes about at some point.
Bottom line, I trust more the safety-minded experts (https://aidid.substack.com/p/what-is-the-problem), those who think AI poses major risks, than the ones who don't, not because I apply the precautionary principle in every situation, but because it does seem to fit in this one.
And like skybrian said, armchair arguments that certain scientific breakthroughs are impossible are suspect. As I once read a nihilist say (https://rsbakker.wordpress.com/essay-archive/outing-the-it-that-thinks-the-collapse-of-an-intellectual-ecosystem/), using philosophy to countermand science is "like using Ted Bundy’s testimony to convict Mother Theresa".
While it’s true that many forms of learning will require real experiments and so can’t be infinitely fast, this only goes so far in reducing my concerns. Air travel is an accelerant for the spread of viruses and social media for the spread of memes. Accelerants are concerning because they reduce our ability to cope, and this is still true even if there are limits on how much they can speed things up.