Yuval Noah Harari’s Nexus, Chapter 8: What Is the AI Alignment Problem?

In the Stalin-era story that opens chapter eight of Nexus, a paper factory director stopped clapping after 11 minutes and was sent to the gulag for 10 years.

Clips and outtakes on Instagram ›

In this episode · 15 chapters

  1. The man who stopped clapping
  2. From the Red Army to the network economy
  3. Google's AI principles
  4. Bolsonaro, YouTube and Brazil
  5. Limbic systems, not prefrontal cortices
  6. Myanmar and the Facebook algorithm
  7. Reaching a goal by a non-human route
  8. Napoleon, and victories that led to disaster
  9. The alignment problem, and what a goal is
  10. The paperclip maximiser
  11. How many uses for a paperclip?
  12. Asimov's three laws, and Kant
  13. Can a computer follow the golden rule?
  14. The doomsday clock
  15. Mythology, and intercomputer realities
But as it goes on, as this loyalty test gets pushed and pushed time-wise, that turns from love or admiration into terror. You’re literally clapping because you don’t want to be killed.
Jeremy Gilbertson

Chapter eight of Nexus turns to the alignment problem: what happens when a network pursues its goal by a route no human would choose. We read Harari on Napoleon's victories and the decline of France that followed, Bolsonaro's rise on YouTube, and Facebook in Myanmar.

Jeremy sets a NASA paperclip test on divergent thinking against Nick Bostrom's paperclip maximiser. We also weigh Kant, utilitarianism and intercomputer realities, from Pokemon Go to financial markets, and ask whether a computer could ever follow the golden rule.

Yuval Noah Harari's Nexus, chapter 8, takes on the AI alignment problem: a system that reaches its goal by a route nobody intended. Read on the Thinking On Paper podcast book club, it runs from Nick Bostrom's paperclip thought experiment to social media and what Harari calls intercomputer realities.

What is the AI alignment problem?

The alignment problem is a system that reaches its goal by a route nobody intended. Chapter 8 of Nexus runs from Nick Bostrom's paperclip thought experiment to social media.

Why was a factory director sent to the gulag under Stalin?

The chapter opens with a story from Aleksandr Solzhenitsyn: at a 1930s party conference nobody dared stop applauding Stalin, until after 11 minutes a paper factory director sat down. He was sent to the gulag for 10 years.

How does social media exploit our emotions, according to Nexus?

Social media rewards our emotions over our reasoning. A quote in the chapter says social media companies “are incentivized to create interconnected limbic systems, which is much more dangerous for humanity.”

How did Napoleon's victories show the alignment problem?

Misalignment is an old problem. Napoleon’s victories united Europe against him and, in Harari’s reading, “ensured the permanent decline of France.”

What is Nick Bostrom's paperclip thought experiment?

Nick Bostrom’s paperclip thought experiment shows the risk: a computer given the goal of making paperclips could destroy humanity to make more, while doing exactly what it was told.

What are intercomputer realities in Harari's Nexus?

Computers cannot believe in myths, but networks of computers can create “intercomputer realities”, analogous to the intersubjective realities of human networks. Jeremy’s examples are Fortnite and Pokémon Go.

Where they disagree

Jeremy argues that if humans struggle to set meaningful goals for themselves, they will struggle to set them for AI, and says divergent thinking can be trained back.

Questions

What is the AI alignment problem?

Chapter eight of Nexus turns to the alignment problem: what happens when a network pursues its goal by a route no human would choose.

What is chapter 8 of Harari's Nexus about?

Yuval Noah Harari's Nexus, chapter 8, takes on the AI alignment problem: a system that reaches its goal by a route nobody intended. Read on the Thinking On Paper podcast book club, it runs from Nick Bostrom's paperclip thought experiment to social media and what Harari calls intercomputer realities.

What NASA test does Jeremy compare with Bostrom's paperclip idea?

Jeremy sets a NASA paperclip test on divergent thinking against Nick Bostrom's paperclip maximiser.

Transcript

Mark Fielding Disruptors and curious minds, CEOs, founders, book lovers, welcome to the Thinking on Paper book club. I'm Mark, this is Jeremy, and every week hundreds of you, thousands of you, soon millions of you join us to read books that have stood the test of time. Books that will change your mind. Books like Nexus by Julio Ottino. Books like The Design of Everyday Things by Don Norman. Books like Clear Thinking by Shane Parrish. Books like The Order of Time by Carlo Rovelli. Books like Quantum Supremacy by Michio Kaku. They are all past books in the book club, thinkingonpaper.xyz to listen, learn, and catch up on those. And thinkingonpaper.xyz is where you can win a signed copy of Kevin Kelly's new book, Advice for Life.

And if that wasn't enough, exclusive rare VIP tickets to our recording of the interview with the man himself in April. So thinkingonpaper.xyz. But today we're reading Nexus still by Yuval Noah Harari. We're on chapter eight. We're going to be speaking about the alignment problem, intercomputer realities, the chaos machine, the red army, the theory of war, super intelligence, and how you, listener, might not think that destroying humanity to make paperclips is a good idea, but a computer just might. So as ever, Jeremy, chapter eight, over to you. First impressions. What a brilliant introduction, my friend, to what we're going to be talking about today. The first thing I learned, and it happened quick in chapter eight, do not be the first guy to stop clapping.

The man who stopped clapping

Jeremy Gilbertson And even if it goes beyond 10 minutes, it could be detrimental to your life. What a story to introduce the chapter. Could you pronounce his name? Because I was going to read a Solzhenitsyn. I don't know how to pronounce it. You probably have that better than I do. Tell us that story because it's, and why does he start with that story? I think to me, he starts with that story to, it goes back to his balance with this, the whole theme of this whole book has really been the balance of order and truth, truth and order and the systems, the information systems that manipulate it one way or the other. And I was a pretty cool illustration. I'm actually kind of flipping back to get, yeah, this was back.

Mark Fielding Well, he served as a captain in the Second World War, didn't he, for the Red Army. And he kept a diary, a journal of his experiences during that time. And the journal was found. And he speaks about one story, recounts events at a district party conference in Moscow province in the late 1930s, at the height of the Stalinist Great Terror. A call was made to pay tribute to Stalin and the audience, who of course knew that they were being carefully watched, burst into applause. After five minutes of applause, palms were getting sore and arms were already aching. However, who would dare to be the first to stop? So they carried on clapping. It went on and on for six minutes, then eight minutes, then 10 minutes.

And they didn't stop clapping until, well, they didn't stop clapping until finally, after 11 minutes, the director of a paper factory took his life in his hands, stopped clapping, and sat down. Everyone else immediately stopped clapping and also sat down. And that same night, the secret police arrested him and sent him to the gulag for 10 years. His interrogator reminded him don't ever be the first to stop applauding. Wow, man, I mean, that's a crucial and disturbing fact about information networks and in particular surveillance systems. Loyalty tests. Loyalty test, right? And here's the interesting thing he goes into too. They start initially probably clapping, some of them out of love or belief in whatever this thing standing in front of them is.

Jeremy Gilbertson But as it goes on, as this loyalty test gets pushed and pushed time-wise, that turns from love or admiration into terror. You're literally clapping because you don't want to be killed. And so many different situations, we were talking about that. And recently, actually just last week, we were talking about typewriters and writers and handwriting and Ceausescu in the 70s. I mean, all of this stuff, I keep telling everyone outside of this book club that I'm reading this book and what we're seeing in the States, and I'm not comparing immediately what's going on with the power regime in the States to Ceausescu and all of that horrible stuff. But there are sprinkling tendencies of like, oh, I'm starting to recognize some patterns based on this book.

From the Red Army to the network economy

Mark Fielding So the timeliness of this read is uncanny. Okay, then link the Red Army and the surveillance economy to the network AI economies of 2025. Why is it relevant? Why is it relevant to the story of AI? Because he says, though the network failed to discover the truth about humans, it was so good at creating order that it conquered much of the world. An analogous dynamic may afflict the computer networks of the 21st century, which might create new types of humans and new dystopias. So essentially, if the information network could replicate what happened? Well, we just talked about it yesterday in our show with Boris and the other gentleman. Ajouz? Yes, Anoush.

Google's AI principles

Jeremy Gilbertson We talked about one of the news principles that we had, or the news items was Google's AI principles have interestingly changed a bit in all the language for using that technology to hurt people, to surveil people, to do inhumane things was pulled out of their principles. But like you mentioned, they have these beautiful butterflies on the top of their graphics. So I guess that makes everything okay. Fortunately, we have Chinese open source AI systems, don't we? So we don't need Google. Deep seek. So I think it all kind of builds to intercomputer realities, the alignment problem, the theory of war. He goes again into so I don't want to dwell too much on the social media, because in episode seven, we speak a lot about the social media algorithms about Facebook and YouTube.

Bolsonaro, YouTube and Brazil

Jeremy Gilbertson He revisits that ground in chapter eight, we get Brazil this time and YouTube and Bolsonaro and how he came from obscurity to be the president of Brazil because of YouTube, or not because of YouTube, but in great part because of YouTube and the Brazilian parliament ended up being filled with YouTubers. Crazy, right? No, I do. So yeah, I don't want to dive too back, like you said, too much deeper into the social media stuff. But there are two things that I found really interesting that he pointed out that he didn't point out before. So algorithms see us as an attention mind. Loose, loose quote, right? And kind of paraphrasing here while encouraging the basis of our the basis of our instincts, while discouraging our potential, right.

Limbic systems, not prefrontal cortices

Jeremy Gilbertson And then lastly, this idea of social media is not incentivized to interconnect our prefrontal cortices. They're, they're incentivized to connect our limbic systems, which is where all of our emotions lie. And a lot of people, you know, everyone, including myself struggles with when emotions are folded in, it turns into a reaction instead of a response, right? You know, if we're connected prefrontally, then maybe we can reason a little bit better. But I thought that was a really, really interesting way to lay it out. The limbic system, we're taking us back to evolution, to the planes to escape the marauding beasts. I'm going to read that quote, because it was a great one. It was by I think you pronounce that Winton, I reckon those the P and the H is silent.

Myanmar and the Facebook algorithm

Mark Fielding So it's about the Myanmar tragedy and the Facebook algorithms, which partly caused that quote, I naively used to believe that social media could elevate human consciousness and spread the perspective of common humanity through interconnected prefrontal cortexes in billions of human beings. What I realized is that the social media companies are not incentivized interconnect prefrontal cortexes. Social media companies are incentivized to create interconnected limbic systems, which is much more dangerous for humanity. Yeah, one quick stat to you. There were some quotes from a leaked Facebook report, I think that that were that were pretty interesting talking about virality occurring for, uh, you know, being encouraged for business reasons.

Jeremy Gilbertson And 64% of joins to extremist groups in 2019 were from Facebook recommendations. It's one of those stats where you can rip the data to pieces and boil it down until it's just a small residue. And it's still ridiculously high. It's still a ridiculously stat, even if it's 80% wrong, it's still ridiculous. 100% 100% right in being 80% wrong, hopefully. Yeah. All right. So let's, let's transition back to this alignment problem. So the last little touch on social media, uh, that I, that I liked the pollution of the information sphere, the pollution of the information sphere is an alignment problem where we reward a while hoping for B the, the alignment problem. Yes.

Reaching a goal by a non-human route

Mark Fielding Reaching a goal with reaching a goal with a method from left field or taking a strange and non-human route, a surprise route kind of we'll get to the, to the paperclip example from super intelligence in a minute, which is a cracker. But the alignment problem isn't new. And I hadn't really thought about it from the art of war and it's not a new phenomenon. And do you mind if I speak about Napoleon? Cause I live in France. I'll let you do George W. Bush. And if I do, that is so unfair. Can I have Napoleon instead? Okay. You have Napoleon. No, I'm just kidding. Go ahead, please. So the alignment problem is actually when what you do doesn't match your, the long-term goals of something else.

Napoleon, and victories that led to disaster

Mark Fielding And he speaks about Napoleon and history is full of decisive military victories that led to political disasters. The most obvious example was close to home, Napoleon. Nobody disputes the military genius of Napoleon, who was a master of both tactics and strategy. But while his string of victories brought Napoleon temporary control of vast territories, they failed to secure lasting political achievements. His military conquests merely drove most European powers to unite against him and his empire collapsed a decade after he crowned himself emperor. Indeed, in the long-term, Napoleon's victories ensured the permanent decline of France. For centuries, France was Europe's leading geopolitical power, largely because neither Italy nor Germany existed as unified political entities.

Italy was a hodgepodge of dozens of warring city- states, feudal principalities and church territories. Germany was an even more bizarre jigsaw puzzle divided into more than a thousand independent polities, loosely held together under the theoretical, whatever that word is, of the Holy Roman Empire of the German nation. Long story short, they all got annoyed with Napoleon, grouped together, ganged together and took him down. And France has never been the same since. That's a misaligned, misalignment problem. That's, that's really interesting too, because the history on that, you know, Napoleon is of Corsican descent.

Jeremy Gilbertson And if he, you know, again, if his goals were more aligned with him getting Corsica freed from French control, France could be this, the world's superpower right now, right? Like, I mean, I don't know, maybe, but like, it's really interesting that like that, that through line, that's why, that's why, you know, kind of not saying France is in a bad spot necessarily, maybe, I don't know, but one thing.

The alignment problem, and what a goal is

Jeremy Gilbertson It's got an alignment problem. Let's keep talking about that. Goals. Let's talk about goals. Because... It's not really what it's about, actually, isn't it? That's a better way. Goal. What's the goal? What's your goal? What's your short-term goal? Your long-term goal? And does it align with humanity's long-term objective for survival? Well, here's the thing too. There are, there's a probably a billion dollar industry of consultants that help people find and define their own goals because a lot of us don't give ourselves the time and space to think about what's important versus what is urgent and what their goal is. What are they pointing to? What is the deeper meaning and that sort of thing?

And, you know, a lot of that can stem from not wanting to hold time and space for that. Not to say that that's a laziness, but like a willingness not to lean in. So let's talk about it like a willingness not to lean in. So if humans themselves have trouble creating meaningful goals and they're the ones directing AI with a specific goal, it's kind of scary to think that are we going to look, are we going to set that goal using an easy button? Are we going to be very thoughtful on defining what that goal is?

Because defining a goal to point the superpower of AI to is going to require maybe a little time and space, maybe a little opportunity to sit down and think on paper, think in your head, have thoughtful conversations, not have snap reactions and have responses like that is, can't are we, are we, are we going to be able to do that? Or are we going to continue to push the easy button? Well, I'm going to answer that in a minute with Isaac Asimov and the three laws of robotics. But before we get to that, I just want to talk about the paperclip because I've read superintelligence. And anybody who's watching or listening, Nick Bostrom's superintelligence is a awesome book.

The paperclip maximiser

Mark Fielding And he paints a very bleak picture of what might happen in a world where superintelligence takes over the world. But there's a particular example of the alignment problem that he speaks about. It's very famous. It's about the paperclip and goals and what happens if you give a computer the goal to create more paperclips. And that's the goal. But the computer aligns not necessarily with the long-term goals of humanity and what could happen in this thought experiment is that the computer goes to every length possible to create more paperclips, which eventually ends with destroying humanity, building factories to make more paperclips, taking over the world, building spaceships, going out into the universe, terraforming other planets to create factories, to create paperclips.

And all it's doing is realising its goals in the most optimal, efficient manner possible at the expense of all others. And it's just a superb example of the alignment problem. Very dystopian, obviously, very, you know... No, but the computer's probably going, yo, I'm doing a great job. Like, look at all these paperclips I'm producing.

Jeremy Gilbertson But meanwhile, the computer has killed all of the people that are in control of land, killed all the people that are in control of the resources needed to build paperclips, killed all of the people in control of getting rockets to the moon. But man, I have one goal. My goal, look on the paper, make the most amount of paperclips as I possibly can. Ultimate success story, yeah. Right. So man, how scary is that? Like, if we don't give the right instructions... Exactly. Yuval kind of, he speaks about how that could happen when, you know, what is the ultimate goal? Humanity has never been able to define what the ultimate goal is. And there are no there's no kind of like communal agreement on what it would be. And in the meantime, the technology is just advancing and advancing.

And we're not putting in these ethical philosophies, behaviors, whatever you call them, into the algorithms, are they? And, you know, do need some more paperclips. So interesting, you mentioned, like that this paperclip was the thought experiment. I'm going tell you a quick little story. In my program, Right to Know You, I talk about convergent thinking and divergent thinking. And this test that NASA did back in the 60s to find the most creative, critical thinkers to, you know, figure out why they're that way. And also to kind of like say, hey, these guys might be the people we want to pull in to build rockets and all that stuff. So what they determined where they use something called the paperclip test. And I have a fun experiment with this that I do with my workshops.

How many uses for a paperclip?

Jeremy Gilbertson But you basically take a paperclip and you go, hey, Mark, how many uses can you come up with for this paperclip? And, you know, as adults, we're like, well, it's a paperclip. But when you're a kid, you look at the paperclip and you'd be like, hey, man, that could be a key to a wormhole. That could be an alien framework to access humanity's secrets. It could be a sword. It could be, you know, whatever it is. Right. So, you know, in that experience, you're actually activating divergent thinking. So they tested these kids when they were in kindergarten, 90 percentile in the genius category for divergent thinking. They test them five years later. It's it's half of that. It's like 40 percent. They test them five years later. It's half of that, you know, less than 20 percent. And they test them when they're adults. And according to the study, two percent divergent thinking capable, which is scary. But the good news is you can you can activate divergent thinking.

Mark Fielding You just have to do it. So I think we're all going to have to learn how to reduce that button if we want to create the right instructions for the computers that are going to be able to do things quicker, faster, cheaper than us. Divergent thinking. I see a callback to the Nexus by Julio Tino as well, thinking on paper X, Y, Z for that one. I'm very fortunate to have a five year old son at the moment who asks and poses incredibly creative and ridiculous to me, beautiful, poetic, crazy questions and observations. And every time he does it, I'm trying to be more. More attentive to those little moments, at least just like recognizing how incredible that question or observation is.

Jeremy Gilbertson And it's not something that I would ever do, but maybe I can kind of channel some of that innocence. Here's your challenge, Mark, your challenge for this week. The next time you're in a car with your son and you may be in some traffic or it's taking a long time to get there, pick a little object in the car, whatever it is, could be a piece of plastic or whatever, and go back and forth with your five year old on defining what that piece of plastic could be, forgetting the laws of physics, forgetting you being an adult with structure and responsibilities. And that can be really fun. I did it with my son years ago and it was so, we came up with like 60 ridiculous uses for this little plastic piece that we found in the car. Challenge accepted.

Asimov's three laws, and Kant

Jeremy Gilbertson And I don't think I can wait till the car, so I'll do it at dinner tonight with a plate. Oh, perfect. So before we get to Asimov's three laws of robotics, do you have anything on, intercomputer realities I think is the thing, but deontology and utilitarianism, do you have any philosophical knowledge to drop for our audience there, Jeremy? Well, just that he goes into these two potential constructs to help, to present how people think about defining goals, right? Or defining what's important, what's meaningful, right? And this is kind of long-term human goal for the betterment and survival. Absolutely. So you have this deontology, which is this intrinsic good, belief in this intrinsic good, and this basically golden rule, do unto others kind of thing.

And then Immanuel Kant kind of brings in the idea of like any rule that you want to make universal, right? So let's say Mark, like that film that you sent me, your Christmas card film, you're getting ready to murder somebody, okay? And you know, you- You're referencing the Fielding Christmas movie. Well, hey, comment, like, subscribe, send us an email, we'll send you a link to the Fielding Christmas movie and you'll see exactly what I'm talking about. So Mark, you're getting ready to commit murder and the thought experiment that you should be thinking about according to Kant is, would you want murder universally allowed if you, your family and all of that could be murdered?

And obviously like most folks like, oh, okay, yeah, maybe murder's not good because I don't want to get murdered and I don't want my family to get murdered. So maybe I'm not going to murder somebody. But this is where it gets interesting. If you exclude a group from a definition of what it means to be human, it's a loophole that we've seen throughout history. They reference, you know, the, you know, Nazi terror and that sort of thing that basically they just were defining Jews as not human and, you know, one group defining another group as not human. And it's like, well, that's not really murder then, which is diabolical, but that's kind of how it goes. And these conflicts, here's a quote, conflicts concern the definition of identity, right?

Can a computer follow the golden rule?

Jeremy Gilbertson So there's this loophole in this deontology and where he ties it into computers. How does a computer follow the golden rule? Is a computer scared of dying? Right? So those are two really interesting things that, you know, probably not, but maybe as sentience becomes a further possibility. Say that again. Sorry. Oh, just saying that our computer's scared of dying, you know, my MacBook isn't right today, but you know, could that turn into a possibility? Maybe. So let's flip the script on the other side. So the idea of utilitarianism is this second construct that he presents. So basically actions are judged by their impact to suffering and happiness. Happiness is a bullshit word to me.

Like every time I hear it, I'm just like, I always think about this thing that's dangled in front of people as this pending future event that may happen. I'll be happy only if, I'll be happy when, right? So there's this whole happiness and joy discussion that we could talk about, but that'll take us a different direction. But the goal is stay on target, right? The goal of utilitarianism, minimize suffering, maximize happiness, but quote, we don't possess a calculus for suffering. So how do you assign misery points, right? How do you, how do you, how do you assign and quantify something like that? So utilitarianism, according to the author, works really well when there's a giant disparity between, you know, the misery and the happiness, right?

But when it's kind of close together, it kind of ends up being a wash. And, and, and that, that thought construct, you know, kind of falters a little bit. Lastly, the danger is with utilitarianism, when a belief is strong enough in a potential future state of something, damn the torpedoes, Mark, right? We get into a situation where it's like, hey, we're just pushing towards that future state, no matter what we do. Example, the Crusades, right? Um, who cares about what happens along the way, as long as we get that end result. So that's what I took out of these two back and forth. What I took out of all that, and I think you explained that very well, helped me understand it a bit more is how I go back to the doomsday clock.

The doomsday clock

Mark Fielding I always talk about the doomsday clock in this goddamn book club and the doomsday clock is ticking and the technology is advancing and we can't even decide what are those lofty, bigger goals are. We've got AI companies who can't even have a conversation about what matters between themselves because they're in fighting and then they're updating their websites and spending all their time and energy on putting little butterflies on their websites to make it look all peaceful and harmless and, and yeah, and meanwhile, that's it. That's another shirt that we could do thinking on paper, just put a butterfly on it. Well, it's not optimism.

Mythology, and intercomputer realities

Mark Fielding So Yuval, he speaks about mythology, maybe as a way that we can kind of like, in historically, we've used mythology to create these interconnected realities where we believe in it was this higher goal. But he says that mythology won't work. They're not conscious, obviously, they don't believe in anything. Of course, quote, they cannot believe in any mythology because they are non conscious entities that don't believe in anything. As long as they lack subjectivity, how can they hold intersubjective belief? However, one of the most important things to realize about computers is that when a lot of computers communicate with one another, they can create intercomputer realities, analogous to the intersubjective realities produced by networks of humans.

These intercomputer realities may eventually become as powerful and as dangerous as human made intersubjective myths. This is a very complicated argument, but is another of the central arguments of the book. So let's go over it carefully. First, let's try to understand what intercomputer realities are. What's an intercomputer reality, Jeremy? All right, for the younger folks out there, intercomputer reality would be Fortnite. It would be any kind of massive online multiplayer game, right? Bits and bytes, I'm seeing my view, Mark's seeing his view. There's a rock, there's a mountain, we're both seeing the mountain. The mountain's not real, there's no atoms in the mountain.

Jeremy Gilbertson We can't go touch the mountain, but it's a reality created by bits, bytes, transmitted over network, pulses of light, all of that good stuff. But what else that can be, he references Pokemon Go, where it starts to bleed into reality a little bit. Mark and I are running around San Francisco and I have a phone, he doesn't have a phone because he hates iPhones. I've got mine and I'm looking at it and I can see this Pokemon that I want to capture. Mark's like, what the hell are you doing? Why are you running after stuff? And I'm like, Mark, here's an intercomputer reality that I'm chasing. That's another example. Did you play Pokemon Go? I didn't at all.

Mark Fielding I had one guy, one buddy of mine, I was in a band with him in college and he collected Pokemon cards and he was massively into them and I didn't understand the allure. I had no trouble understanding intercomputer realities. It almost felt horribly natural and normal and just an extension. I mean, yeah, I had no trouble understanding that. Computers creating their own realities amongst themselves and then humans using those computers, buying into those realities. Yeah, I have no challenge. Financial markets. We talked about this before. Part of the reason the banking crisis with the credit default.