Yuval Noah Harari’s Nexus, Chapter 9: Can Democracy Survive AI?
Chapter nine of Nexus sets out four principles for democracy in an age of AI: benevolence, decentralisation, mutuality, and change and rest.
In this episode · 10 chapters
Well, have you ever given yourself the space to try to be creative? In order to be creative, you got to be willing to look a little silly.
Mark and Jeremy read chapter nine of Nexus, where Yuval Noah Harari asks what happens if we repeat the mistakes that came with earlier technologies, and what self-correcting mechanisms a democracy needs.
They then take on Harari's mistaken assumptions about automation, from why a nurse may be safer than a doctor to his claim that computers could out-create us because they excel at pattern recognition. The argument turns to taste.
Nexus by Yuval Noah Harari, chapter 9, asks whether democracy can survive AI. Read on the Thinking On Paper podcast book club, it sets out four principles for democratic information networks (benevolence, decentralisation, mutuality, and change and rest) and challenges two assumptions about automation: that AI will not take intellectual jobs, and that creativity is unique to humans.
Can democracy survive AI?
Chapter 9 of Nexus sets out four principles for democratic information networks in an age of AI: benevolence, decentralisation, mutuality, and change and rest.
Is technology deterministic, according to Harari?
“Technology is rarely deterministic,” Harari writes. The damage of past revolutions, from imperialism to industrial murder under Stalinism and Nazism, came from how humans managed new technologies.
What are Harari's four principles for democratic networks?
The four principles: benevolence (networks that collect our data should help us, not manipulate us), decentralisation, mutuality (more surveillance of citizens means more surveillance of governments and corporations) and change and rest.
Why should democracies avoid concentrating information in one place?
On decentralisation, a democratic society should never let all its information be concentrated in one place, because “for the survival of democracy, some inefficiency is a feature, not a bug.”
Will AI take intellectual jobs, according to Harari?
The assumption that AI will not take intellectual jobs is wrong. Harari suggests that people who want a job in 2050 “should perhaps invest in motor skills and social skills, as much as intellect.”
Could computers become more creative than humans?
So is the assumption that creativity is unique to humans. If creativity is the ability to recognise patterns and break them, computers may become more creative than us because they excel at pattern recognition.
Where Mark and Jeremy disagree
Mark has little hope for mutuality. Jeremy argues a computer can generate options but may lack taste, while Mark sides with Harari that creativity is not binary.
Links from this episode
Questions
Can democracy survive AI?
Chapter 9 of Nexus sets out four principles for democratic information networks in an age of AI: benevolence, decentralisation, mutuality, and change and rest.
What is chapter 9 of Harari's Nexus about?
Nexus by Yuval Noah Harari, chapter 9, asks whether democracy can survive AI. Read on the Thinking On Paper podcast book club, it sets out four principles for democratic information networks (benevolence, decentralisation, mutuality, and change and rest) and challenges two assumptions about automation: that AI will not take intellectual jobs, and that creativity is unique to humans.
Can a computer have taste, according to Jeremy Gilbertson?
Jeremy argues a computer can generate options but may lack taste, while Mark sides with Harari that creativity is not binary.
Who discusses Nexus chapter 9 on Thinking On Paper?
Mark and Jeremy read chapter nine of Nexus, where Yuval Noah Harari asks what happens if we repeat the mistakes that came with earlier technologies, and what self-correcting mechanisms a democracy needs.
Transcript
Mark Fielding Disruptors and curious minds, CEOs, founders, book lovers, welcome to the Thinking on Paper book club. I'm Mark, this is Jeremy, and if you're listening to this, thank you. Thank yourself for being a critical thinker. Thank yourself for reading between the lines and understanding the importance of bringing your own conclusions to the impact of emerging technologies on business, on your culture, on your life, on your family.
In a world that celebrates the humdrum, you are special. Thank you for reading books with us. We're on chapter nine of Nexus, and you know what, Jeremy? I'm feeling optimistic today. Because if we ignore for a moment the ongoing damage to the ecosystem, we can nevertheless try to comfort ourselves with the thought that eventually humans did learn how to build more benevolent industrial societies.
Imperial conquests, world wars, genocides and totalitarian regimes were useful, woeful, did I say useful? Woeful experiments that taught humans how not to do it. By the end of the 20th century, some might argue humanity got it more or less right. Yet, even so, the message to the 21st century is bleak. If it took humanity so many terrible lessons to learn how to manage steam power and telegraphs, what would it cost to learn to manage bioengineering and AI?
Jeremy Gilbertson The question from chapter nine, Jeremy, as ever, first impressions. First, first, I want to deliver a public service announcement akin to your introductory statement. Synthesis cannot be outsourced. Contrary to popular opinion, it actually happens in here through inputs this way, this way, and this way. So yeah, let's talk about it. So Mark's feeling positive. I'm going to tiptoe through some kind of intermediary pre-positivity before we get there.
Hold on. Can you hear that? Do you hear that? That was the sound of my optimism bubble bursting, I think. Fair enough. Sorry about that. Short-lived, like a quark in the small world of microparticles. In the quark- gluon soup of life. Love it. All right. So it starts off with a pretty cool statement that, you know, that democracy is this partnership between bureaucracy and mythology. I thought that was pretty interesting. And we run through bad use cases of technology. You know, the theme in this chapter is like, you know, hey, technology in and of itself can't end the world.
Mismanagement of new technologies
Jeremy Gilbertson It's humans' mismanagement of new technologies that cause a lot of issues, right? So let's talk about the technology of imperialism, right? So that was a technology, right, where due to the industrialization, things becoming more industrial, these technologies that were invented required raw materials. And raw materials, the race for raw materials, sent people all over across the world conquering other people for raw materials to feed their technologies.
Millions of indigenous people crushed to allow other nations to level up, right? So again, this is the bad use of technology. We even call imperialism a technology. Stepping a bit further, and we'll jump back into positivity here in a second. So we talked about, you know, Stalinism, Nazism has been a theme that he's explored and a framework that he's explored. This is really interesting.
The idea that Stalinism, Nazism brought about the idea of industrial murder. I was like, whoa, industrial murder, that's a hell of a way to say that, but it's spot on. And then obviously the industrial revolutions affect on extinctions to other species and that sort of thing. And like you said, Mark, if we had trouble with these technologies, and this was the telegraph and the steam engine, what's going to happen as we move forward with AI and, you know, all the other things that you're talking about?
The point of chapter nine
Mark Fielding Well, I think the point of chapter nine is that if we replicate what came before, then the answer is not a very nice one for humanity. I mean, every time a technology emerges, everyone fears the apocalypse. That's not new. Like this fear of AI isn't new. The Luddites, during the industrial revolution. But he says that, okay, the Luddites were right, kind of. There was a downside to the industrial revolution.
Yes, everyone became more wealthy than they had ever been. And there was huge positives from it. There was the downside. It ended and changed traditional social, economic, and political systems. There was before the revolution and after the industrial revolution, they were not the same. And yeah, you mentioned war. I've had enough of Stalin. I don't want to talk about Stalin anymore. He seems to go on about him in every single chapter.
Time to turn the corner. Yep. Yeah. Ecological collapse. He speaks about the downside of it. And he proposes a way forward, the democratic way forward to, you know, can democracy live in this world, in this AI world, the democratic way. And he lays out the structure, the patterns, the things that we need to follow from the democratic way to survive. So as we've talked about, Google changing their AI principles, I don't mean to harp on this, but I think it's a, there's an interesting analogy.
Jeremy Gilbertson Human are butterflies. It's like little Yuval and Stalin, the two of them. Swat them. Yeah. So there's a list of principles that he alludes to in here that are kind of principles for democracies to follow related to the proper use of technology. And there are four of them. Before we kick that off, here's a quote that I found pretty cool. Technology is rarely deterministic. Yeah.
Meaning it's just like, it's not going to, it's not going to by itself wreak havoc, but actually now that I think about it, yeah, we get to Kurzweil's point in time and space and, and, you know, maybe it starts to become deterministic, but Hey, we're staying positive today. Let's talk about these principles. So the first one he calls benevolence, meaning, you know, if, if, if these networks are collecting my data, the networks that collect my data should help me rather than manipulate me.
Mark Fielding It seems pretty straightforward, but I think a lot of people don't look at that when they're building something especially at one, not even nefarious building something to make money with, right? Yeah. And he uses the health system. Doctors know everything about us. Our boss, to some extent, knows a lot about us and they don't use that information to manipulate us. They use that information to help us there.
Benevolence
Jeremy Gilbertson It's benevolence, the first principle in it. But what does he say? Yeah. Well, there's, so there's, I love the idea of him calling the business model being flawed, like big tech business model being flawed because we pay for these services. Oh, all of this stuff is free. It's great. You know, like a Gmail account, oh, it's free. It's great. Or, you know, this, this entry level account for Dropbox or whatever.
Oh, it's free. Well, we pay for it with our information, right? So that's, that's the trade off. And he references, you know, we, we won't go, we won't pay Nike with our information and get a set of shoes, but I would argue, I think there'd be a lot of people that would give them information if they were sending out sets of shoes. Yeah. Well, what do they want to know if I get a pair of Air Jordans out of it?
Mark Fielding They're about, if they're about 300 quid these days. So yeah, I'll give them my address, my email address. So that's number one. Number two is decentralization. A democratic society should never allow all its information to be concentrated in one place, no matter whether that hub is the government or a private corporation. Don't cross the streams, they're saying. Don't cross the streams. Don't, don't allow access to all the databases to everybody. Here's an interesting quote there, for the survival of democracy, some inefficiency is a feature, not a bug.
Self-correcting mechanisms
Jeremy Gilbertson Interesting. Is that about self-correcting mechanisms? Yeah, there, you can't, you can't, I don't think you can optimize those yet, right? There, there has to be a, there has to be a, well, we'll talk more about self-correcting mechanisms, because you have to understand the technology that you're attempting to correct, which is, which is a big challenge. Let's talk about mutuality.
Mark Fielding Yeah, the third democratic principle. And dare we say our theme of top-down and bottom-up comes up again. Mark, tell me about mutuality. If democracies increase surveillance of individuals, they must simultaneously increase surveillance of governments and corporations too. My note for mutuality is, I don't have much hope for this one. Wait, you mean, you mean. I can buy into benevolence and decentralization, but the idea of this kind of mutual exchange of information, I mean.
Jeremy Gilbertson So you don't, you don't think, you don't think the Trump White House is going to let me peek into their meetings and, you know, take a listen to what they're doing and look at their data exchanges for me to make a better decision and hold them accountable? You don't think that is a possibility? No, no. If they know more about us while we simultaneously know more about them, the balance is kept.
Mark Fielding I mean, yeah, no. Come on. But as a, as a, as an operating principle, like. Oh yeah, as an operating principle, yes, I agree. But who, yeah. Applying them in real life. Reality has a, has a nasty habit of like playing with your principles, doesn't it? They're, they're. I also think his, his fourth is change and rest. I think he'd run out of nouns as well as he had benevolence, decentralization, mutuality, and then change and rest.
Jeremy Gilbertson I thought that was, so the change one, I understood. And, and here's an interesting, you know, quote from a repression can take the form of denying humans the ability to change or denying them the opportunity to rest. Right. So, so racism, caste systems, you know, these were opposed as if you go counter to them, you're going against the laws of nature, the laws of God, enter intersubjective realities.
Truth and order
Jeremy Gilbertson Again, imposing a balance of truth and order on a group of people. Right. And, you know, the idea that, you know, denying changing between castes or denying the ability to be considered equal as humans kept the order of oppression on top there. The change piece, or I'm sorry, the rest piece, I didn't really grab. I didn't really grab this bit. I was thinking of it more of like, we are in constantly in flux.
Mark Fielding Our neurons are forming and reforming with like neural network, they're plastic, things are changing. We're never the same. The rest part, we all need a rest. I don't know. But he goes from that to the balance of, and I quite liked this example, but I'm going to take my disagreement out on this as well. He talks about non rigid AI. And rather than an AI predicting your illness, so this is a continuation from the change in rest, rather than the AI predicting your illness, it helps you avoid being ill.
And this is like this different way of looking at it. And I was thinking about the paperclip experiment in last week's show, we were talking about how misaligned, how you can, if your goals are misaligned, then you can end up in a very bad state. And if the AI's goal is to make you feel healthy or remain healthy, what ends might it go to, to make sure that you remain healthy? I don't know, killing everyone in all the power stations, shutting down all pollution.
Jeremy Gilbertson There's plenty of incredibly psychotic ways that an AI could go about ensuring your health and safety. You know what I call that I wrote down in the book margin, a tale of two algorithms, the predicting illness, to feed decisions to other entities, to make decisions about lending you money and what your insurance rate should be and all of these other things versus helping you avoid illness by, you know, hey, Mark, ding, ding, ding, time to go out for a run.
Hey, Mark, put down that cupcake, right? Some of those things. Tale of two algorithms. Yeah, yeah, I like that. We should write a book about it. It was the best of times. It was the worst of times.
Mark Fielding And people are very aware, I think of this idea of being surveyed to such a point that the AI or the computers know, for example, that you have something growing inside of you and it's not good and they pass that information on to your insurance company and your insurance company increase your price plan and then you can't get a loan because the insurance company knows that you're going to have cancer in five years and that kind of AI feeding that wheel.
Jeremy Gilbertson And if you have the two AIs and you have one that isn't feeding that wheel, but is feeding a different flywheel about making you healthy and maybe, you know, then you get the loan and you good things happen. Well, this is this is interesting that I brought up open mustard seed before on on these shows. And it's like the precursor to, you know, personal data sovereignty. You know, it was a non blockchain solution, a database driven solution.
I think it was out of MIT. But like blockchain could could have a have an impact in this where you capture and own the the on off streams of data and you have your own you have like API's like normally when you know apps have API's you can have API's that turn on and off the feed one way the other way to where the data is used how you want to use it. But the only way to make sure for certain is that you actually own a digital thing that no one else controls that no one else sponsors, right?
Hey, here's a free thing that you can use, but I need all your shit. It's it's it's here's a thing that you own independently. And you know, by operating these levers, your data is not going here and you're allowing a little bit of it to go here. There's an opportunity there somewhere, but I don't know that. I don't know that it will be allowed. Zero knowledge proofs.
Mark Fielding We should check it. Listeners check out our blockchain episodes and connect some dots, some technological synthesis cannot be outsourced. So automation, you want to go there? Automation exactly. Yeah, the first part is about surveillance. The second part is about automation. And you know what they say about the assumptions being the mother of all effort, don't you? Assumptions which are wrong.
Three assumptions about automation
Mark Fielding He has three assumptions about automation, which I have a feeling you might disagree on to one of them to some extent, maybe not. Maybe your opinions changed on that. Let's find out assumption number one about the future of AI. AI will not take intellectual jobs. I think I think we all know that, don't we? By now, AI will not take intellectual jobs. Yeah, like it was kind of the assumption was that AI would take all the menial tasks, wasn't it?
Whereas in fact, it will take all of the jobs where you need to use your brain. What's an example of an intellectual job? Copywriter, designer, architect, doctor, lawyer. Okay, and you don't think AI will take some of those jobs? Yeah, it will. The assumption was wrong. In my mind, I'd be wording it in the right way. Should have let ChatGPT set that one up for you. I should have, but I didn't, because that's why I'm here.
And that's why you're here listening to this, because you think for yourself, you think on paper, you read books. And yeah, you make mistakes as you try to understand it. Anyway, so yeah, AI, the nurses will be safe, the doctors won't. I thought that was really interesting. So my brother's a doctor, he's an anesthesiologist, super talented guy, trained in the art of keeping someone barely alive.
Jeremy Gilbertson So life saving procedures can happen. I mean, there's, there's an art to that there's a physical aspect to that. And what I thought was interesting about some doctors jobs being being threatened, the idea of the doctors that assess, diagnose and recommend could probably see some kind of AI tools start taking over some of that. So we're already seeing that kind of stuff. But I don't think, you know, an AI robot is going to be in charge of the machine that manipulates the drugs that, you know, keep someone barely alive.
The nurse
Jeremy Gilbertson But then on the nurse side, I thought that was really interesting. There's a lot of physical aspects to nursing, that, that robot hands just can't do yet, right? And AI won't be able to do right? Yeah. So intricate skills. Yes. So here's the quote, people who want a job in 2050 should perhaps invest in motor skills and social skills, as much as intellect. Yeah, well, that's what I said, I'm gonna, I'm gonna quit and go chop down trees.
That's my job. I'm going to butcher this stat completely. So apologies to all the doctors out there for me getting it completely wrong. Please tell me the correct statistic. But I want to say the, I want to ask what is the second biggest killer in the United States each year? And it's medical error. Really? So let's, let's, let's fact check that. I might fact check that in real time.
But yeah, it's one of those things where misdiagnosis over wrong quantity of drugs, whatever it is. It's very high. I don't know the exact number. So, but it's high. It's more than it should be. Okay, well, let's, um, so here's, here's a couple things that stood out to me. And this, this was referenced in our packing McCormack episode, call to humanity. What was it?
What was the title? Most human wins, most human wins, right. So listen to that. So we break down the whole 35 page paper with with thoughts. Would love you to read it too. Because you can read and think on paper and write a little bit but very similar suggestions from Harari on this one, right? So beyond the, you know, beyond the lookout for these for these next automation targets, right? My I was writing down continual tweaking, continual learning, refining what you do repositioning what you do rather than sticking to one lane.
Because you know, it's going to be the next superpower, Mark. The next superpower that people should be majoring in is flexibility. And not just flexibility in one moment, almost malleability, right? Like being able to retrain and adjust to a market, a job market that is just going to change so rapidly, that you know, we're not going to be able to jump in and say, Hey, I'm gonna go to school, I'm going to be an accountant, I'm just going to be, I'm going to go work for someone, I'm going to be their accountant.
Mark Fielding And at the end of 30 years, I'm going to bounce out and do some stuff. It's, it's going to be way different for those jumping in. And for those, let's hold that thought on kids and what the next generation and what we should be teaching our kids and where should we should be pushing our kids and where we should be encouraging them to read, think, learn, write, get better. Before we debunk the other assumptions about automation in the age of AI, number one AI, well, yeah, okay, I made that assumption.
Creativity is unique to humans
Mark Fielding Creativity is unique to humans. Another common but mistaken assumption is that creativity is unique to humans. So it'd be difficult to automate any job that requires creativity. Creativity is often defined as the ability to recognize patterns and then break them. If so, then in many fields, computers are likely to become more creative than us because they excel at pattern recognition. I'm, what is it about this question? I think that I don't think any other question in AI at the moment gets people's blood flowing more than the question about
AI and creativity. I see it all the time on social media. I see it all the time on television and newspapers. It's like you have the AI is not creative. The human touch. I can do it. I can't. And then you have increasingly more people like Yuval here who say, well, that's not really true, is it? So the definition of creativity is kind of close to the way I think about it. I talk about creativity as a unique rearrangement of found elements.
Jeremy Gilbertson And a found element could be a chord. It could be a rhythm. Here I go with a music analogy again. It could be a topic and you kind of put a song together, but no one owns a G chord. No one owns four, four rhythm, but they're these like found elements that you kind of put your stamp on. And the human element, I've got a friend who's a drummer. He's a drummer for perpetual groove. He helped me out with kind of a two month crash course in getting my fundamentals as a drummer a little bit better.
But I saw him post recently that he's committed to never use a click track ever, ever in recording or ever in live, which a lot of drummers do. And that's like the element. That's your feel, right? It's your stamp. It's like because you're not always here. You're a little bit here, a little bit there. It is a hot button, though, Mark. It is like it is a very, I think it's a very hot button because people, it's almost like the last refuge of so many people.
It's the last refuge of the human condition is your creativity and the threat that. If not now, very soon that will be taken away from you, gets people's backup. And I understand the passion. I understand the vitriol, understand the disagreement. Let's let's let's go ahead, finish. I mean, increasingly in the camp of Yuval to a degree, I think that creativity isn't binary. I think there are many, many layers to it.
Mark Fielding And we don't need to go very far into the past to realize that some people are more creative than others. So I don't think you can really say creativity is unique to humans. Maybe you can say great art or ultimate creativity, some level of it. But many people aren't very creative. Sorry, people, people listen to this are obviously your creative. But I think I think here's here. I'll debunk that, Mark, and we can riff on this.
Creative potential
Jeremy Gilbertson I think everyone has creative potential, but I think a lot of people through self talk, through getting it beat out of us in school, like all the labeling is a very, very, very powerful, powerful destroyer of creativity. I'm this not that I'm not creative. Well, have you ever given yourself the space to try to be creative? In order to be creative, you got to be willing to look a little silly.
You got to be vulnerable. You got to put bad shit out. And you got to keep putting bad shit out. And then, you know, from there, you start to learn a little bit about that. But let's let's approach this objectively or try to approach it objectively as as Harari is. Computers definitely are great at pattern recognition. They're better at humans at pattern recognition.
So if we had a thought experiment here and said, all right, let's look at, I don't know, let's look at like four blocks, four elements, four things that could be combined in only a certain number of ways, say four things, 20 different ways that these things could be combined, right? And then you look at the combinations of that. And then you notice the patterns, the computer notices the patterns, and then you assign it and say, computer, break the patterns.
And then they come out with, you know, 120 ways to break those patterns and those combinations. If you look at it specifically as recognizing patterns and breaking patterns, could they identify a bunch of different options? Probably, could they identify what the coolest option is? What the option that would resonate with 24 year olds who love EDM and, you know, nacho cheese and velvet couches like I think, I think that might be.
So are you asking if it could have taste? I'm saying that it might not be able to have taste necessarily. Can it create the available options? Sure. Could it create an option that resonates with a taste profile? Maybe that it creates for itself because then it would have something to point to. It would have to first create a taste profile and then say, Hey, align all the breakable options into that taste profile.
I don't know. I'm kind of, it's my exceptional taste. Is that a result of just my experience? And if it is just a result of my experience, then, then it can be learned. Well, here's exceptional taste. So, so check this out. You've, are you familiar with, have you messed around with like garage band and recorded stuff? And like, you used a garage band or a garage band as I call them, but yes.
Oh, you don't use Apple stuff. So you probably haven't used garage. My brother has used garage band. I am familiar with it. Yes. Okay. So it has these like little plugins, right? That, that, um, plugins are like equalizers or compressors or, you know, reverb modules, all of these ways to affect sound that can stitch together a song in an interesting way. So there are, there are plugins where you used to have to be like, Oh, I've got to be an audio engineer and understand acoustic principles and all these things to kind of like listen and feel and tweak.
And sometimes you could create not just based on those principles of based on the sound that you want to. Right. But now there are preset modules. There are preset settings in there that are almost like marks EQ from the 1960s. Right. And they've basically been able to encapsulate that EQ preset. Right. So that's like encapsulating a bit of taste, but then like packing McCormack says, and I'm, I'm continuing to like fall down this rabbit hole as packing McCormack kind of said, then you start getting to higher levels of abstraction.
So instead of being in the weeds of, you know, I need to boost, I need to boost, um, a hundred, uh, a hundred Hertz, um, you know, by one DB, you're focusing on something bigger than the EQ sound of the kick drum. You're focusing on something at a higher level of abstraction, knowing that that is already a Lego block. Can I ask you a question? I don't care if it's not really about AI.
Back to creativity
Mark Fielding We're talking about creativity. We're talking about what comes next. So I think it's relevant. I want to ask you about what I like to refer as reverse creativity. And I was thinking about this yesterday because I was watching a Ray Beato video. And if everyone, if you don't know who Ray Beato is and you like music, check him out. And he was breaking down a Pearl Jam song, Black, okay.
And it's incredible, but it's one of the first songs Pearl Jam wrote like Eddie Vedder was 20. I think when he wrote the lyrics to it, McCready, yeah, they were 20 years old and it's, it's, it's a genius. It's masterful. It's just a perfect song. And I listened to recent Pearl Jam stuff. It doesn't have the same. It's like, and you see this with a lot of bands. You see it with writers.
You see it with artists. You see it with people in every creative domain that they release this work when they're very young, that's never beaten. In fact, their work, their creativity, at least according to our taste goes down and down and down a lot of the time, not all the time.




