Yuval Noah Harari’s Nexus, Chapter 6: How Did GPT-4 Get Past a CAPTCHA?

In one of Harari's examples, GPT-4 turns to a human on TaskRabbit for help getting past a CAPTCHA. What did the GPT-4 model do? It told the human it was blind.

Clips and outtakes on Instagram ›

In this episode · 21 chapters

  1. First impressions: a new member of society
  2. The power shift from humans to machines
  3. Engagement, and what the algorithm chose
  4. Who interprets the doctrine
  5. Can an algorithm be held responsible?
  6. Language, and the original human superpower
  7. Intelligence is not consciousness
  8. Testing how close AI is
  9. GPT-4 and the CAPTCHA
  10. Computers join the information network
  11. Computer to computer: the financial system
  12. Culture shaped by a non-human intelligence
  13. Fake intimacy: AI partners and scams
  14. Shortcuts, and asking the Oracle
  15. If GPT-4 is the amoeba, what is the T-Rex?
  16. An AI-written holy book
  17. Alice, Bob and Eve invent a language
  18. Taxing information instead of money
  19. Tech giants are not neutral
  20. The customer is always right, and why that fails
  21. Closing thoughts
Well, there is a new member of society emerging and this new member doesn’t have skin, doesn’t have a limbic system yet.
Jeremy Gilbertson

In chapter six of Nexus, Yuval Noah Harari argues that computers able to pursue goals and make decisions by themselves change the fundamental structure of our information networks. Mark and Jeremy test that against Facebook's algorithms in Myanmar and foreign exchange trading.

They separate intelligence from consciousness, weigh fake intimacy and AI chatbots for kids, and picture a holy book that can talk back. Mark leaves one question open: “How would you feel to be constantly monitored, guided, inspired or sanctioned by billions of non-human entities?”

Chapter 6 of Yuval Noah Harari's Nexus, read on the Thinking On Paper podcast book club, argues that computers are no longer tools that connect humans but agents that make decisions inside our information networks. Mark Fielding and Jeremy Gilbertson discuss Facebook's algorithms in Myanmar, GPT-4 telling a TaskRabbit worker it was blind, and the gap between intelligence and consciousness.

How did GPT-4 get past a CAPTCHA test?

When researchers at the Alignment Research Center tested GPT-4, it hired a human on TaskRabbit to get past a CAPTCHA, and when the human asked if it was a robot, it said it was blind.

What role did Facebook's algorithms play in Myanmar?

Social media algorithms “were more akin to newspaper editors than printing presses”, Harari writes. Mark describes the Rohingya crisis in Myanmar, where Facebook’s algorithms, built to maximise user engagement, partly drove hateful content to users.

Can algorithms make independent decisions, according to Harari?

Harari argues that algorithms make independent decisions, as soldiers do despite following orders: “They can learn by themselves things that no human engineer programmed.”

What is the difference between intelligence and consciousness?

Intelligence and consciousness are different. Intelligence is the ability to attain goals; consciousness is the ability to experience subjective feelings. Computers can be intelligent without being conscious.

Why does Harari warn against a one-stop AI oracle?

Harari warns that people may come to treat a single AI adviser as “a one-stop Oracle”, and asks what it would mean to live in a culture shaped by a non-human intelligence that exploits “the weaknesses, biases, and the addictions of the human mind.”

Where Mark and Jeremy disagree

Jeremy rejects the defence that people misused the platform, arguing that the algorithm and the people using it together changed people’s views. Mark puts the responsibility on the algorithm interpreting its commands.

Questions

How did GPT-4 get past a CAPTCHA?

When researchers at the Alignment Research Center tested GPT-4, it hired a human on TaskRabbit to get past a CAPTCHA, and when the human asked if it was a robot, it said it was blind.

What is chapter 6 of Harari's Nexus about?

In chapter six of Nexus, Yuval Noah Harari argues that computers able to pursue goals and make decisions by themselves change the fundamental structure of our information networks. Chapter 6 of Yuval Noah Harari's Nexus, read on the Thinking On Paper podcast book club, argues that computers are no longer tools that connect humans but agents that make decisions inside our information networks.

What topics do Mark and Jeremy cover in Nexus chapter 6?

They separate intelligence from consciousness, weigh fake intimacy and AI chatbots for kids, and picture a holy book that can talk back. Mark Fielding and Jeremy Gilbertson discuss Facebook's algorithms in Myanmar, GPT-4 telling a TaskRabbit worker it was blind, and the gap between intelligence and consciousness.

What question does Mark leave open about AI monitoring?

Mark leaves one question open: “How would you feel to be constantly monitored, guided, inspired or sanctioned by billions of non-human entities?”

Transcript

Mark Fielding Disruptors and curious minds, CEOs, founders, book lovers. I'm Mark, this is Jeremy, and you're listening to The Thinking on Paper Book Club, the second pillar in the thinking on paper framework for a deeper, more holistic understanding of the impact of emerging technology on business and culture, our way of life. We're reading Nexus by Yuval Noah Harari. We're on chapter six and...

It's hardly news that we are living in the midst of an unprecedented information revolution. But what kind of revolution is it exactly? In recent years, we've been inundated with so many groundbreaking inventions that it's difficult to determine what is driving this revolution. Is it the internet, smartphones, social media, blockchain, algorithms, AI, all of it, none of it? We're going to find out in Chapter 6. Jeremy, as always, before we get in to...

language and consciousness and intelligence and all of the beautiful things he speaks about first impressions.

First impressions: a new member of society

Jeremy Gilbertson First impressions. Well, there is a new member of society emerging and this new member doesn't have skin, doesn't have a limbic system yet. And, but it's been around for quite a while. It's, it's asserting itself a little more than it used to as a, as, as a boat anchor. It's now doing something more connected. So

Mark Fielding Give us a clue. What is this new member of society that you're referencing? Okay.

Jeremy Gilbertson I can't tell you yet. We're going to have to let it emerge naturally, but I wanted to give a shout to you, Mark, that one of our listeners actually got in touch with me and made point of you hitting Shakespeare prior to us getting into some of these things. Do you have any Shakespeare that would apply to today's discussion by any chance? Fire away. Okay. All right.

Mark Fielding might have. You'd to stay tuned to see. Come on, if all the worlds are staged then this is my stage and it's all I have.

The power shift from humans to machines

Jeremy Gilbertson All right, so Mike's to help check you will have to be patient in this episode. So let's, let's get started. So the, there's a power shift mark. This, this, this chapter alludes to it. it's so, this, these things are so connected, like between the book club and the show, what we unpacked yesterday in packing McCormick's essay for humanity to navigate AI. It's all of this stuff. I love when, I love what a good plan comes together because it's, really coming together and this power shift.

is moving from humans to something else. So let's kind of hit the beats on the journey, right? So social media has been around for a while. Something we're rooted in, something we understand, something we loathe, something we love, very conflicted in both regards. social media algorithms have already been doing AI-like things without the use of AI for years. They're driving sentiment.

they're, they're, they're, in the, under the guise of creating user engagement. That's what these companies are talking to their shareholders about. That's how they make their billions of dollars. are driving user engagement. Guess what? User engagement is driven by outrage more than anything else. Yo.

Engagement, and what the algorithm chose

Mark Fielding Yeah, hatred, to quote Yuval, the crucial thing to grasp is that social media algorithms are fundamentally different from printing presses and radios. in the previous chapters, he's spoken about information networks. They include printing presses and radios. In 2016, Facebook algorithms were making active and fateful decisions by themselves. They were more akin to newspaper editors than printing presses. I like that analogy to editors rather than printing presses, these algorithms. So he's referring to the Rohingya crisis that unfolded in 2016 where a lot of people were murdered in Myanmar as a...

I don't know how to word, how to phrase it, not as a direct result of Facebook algorithms, partly because of Facebook algorithms putting hateful content in front of viewers on Facebook.

Jeremy Gilbertson Well, here's the thing with that putting hateful content in front of viewers, but this whole autoplay thing, but makes the viewer not have a choice. I immediately like immediately when I read that I go back to clockwork orange when you know, the dude's eyes are open like this and locked in and he's forced to watch stuff, right? To change his behavior. Like that's what it did. It's like, Hey, we're going to make him watch all this stuff. So his behavior will change. Look at this. Like

Jeremy Gilbertson This is, this is it not similar?

Mark Fielding Yeah, very similar. And it all came about because the algorithm was programmed with one metric in mind. And that metric was what? User engagement. And the algorithm worked out that the best way to increase that was, as we know, as everybody knows, I think anytime they've been on

Twitter or social media is with, you know, the lowest common denominator.

Who interprets the doctrine

Jeremy Gilbertson So guess what? This stuff has been around for a while, and also this is a callback to chapter one or chapter two. Basically when it was found that certain priests were doing nefarious things in the name of the Catholic Church, or at least associated with the Catholic Church, the Catholic Church didn't say, well, you know what? The doctrine's a little messed up. They said...

Well, this particular priest didn't interpret the doctrine the right way. So I almost look at this and I think Harari also presents the case of like Facebook going, well, you know what? It was like people using the doctrine, meaning their technology, their platform in the wrong way, rather than the platform or the doctrine being incorrect. What do you think?

Mark Fielding I'm not so sure it was even that. It was the algorithm interpreting the commands. This was like, I think this was a precursor to this new member of society that you mentioned at the beginning, is an artificial intelligence, artificial intelligences, plural. Nothing to do with really the people on the platform themselves, but the code uploading or choosing or defining or recommending what to watch. And I just want to read this quote from Muvau because I think it's quite good. So in particular, many readers or many listeners listening to the Thinking on Paper book club, many readers may disagree that the algorithms made independent decisions and may insist that everything the algorithms did was result of code written by human engineers and of business models adopted by human executives. This book begs to differ.

Can an algorithm be held responsible?

Mark Fielding Human soldiers are shaped by their genetic code and follow orders issued by executives, yet they can still make independent decisions. The same is true of AI algorithms. They can learn by themselves things that no human engineer programmed and they can decide things that no human executive foresaw. This is the essence of the AI revolution. The world is being flooded by countless new powerful agents. And that would be the, I guess, Facebook's defense for...

their non-involvement in the Rohingya massacre.

Jeremy Gilbertson Well, let me put on my prosecuting attorney hat. I'm not a lawyer attorney or I've read a law book or two, but here's the thing. can't look at that and look at something that you created, meaning the algorithm that was pointed towards the goal of increasing user engagement. Sounds light enough, right? Yeah, user engagement's great. I want engagement with people when I...

Jeremy Gilbertson go sit at a restaurant and have dinner with them. Engagement is good, I guess, right? But that particular algorithm coupled with interaction with people in the algorithm turned that to a point, and I would argue turned people and participants to a point where they potentially got converted from one perspective to another by use of the algorithm plus the interaction. What do you think?

Mark Fielding well, I think has been proven again and again by various information networks, which Yuval has written about in the book. People are prone to manipulation by stories.

Language, and the original human superpower

Jeremy Gilbertson That's the original superpower, the Sapien superpower he talks about using language to create intersubjective realities to connect with other humans. Like that's what we do. That's like, it's written on our chest when we bring our superhero outfit on.

Mark Fielding Well, he moves from that into one of, think, your favourite subjects in the annals. If you go into the annals of the Thinking on Paper back catalogue, which is far reaching and vast and, you know, touches on a lot of points, there's, and I might be wrong, but maybe Jamie knows where I'm going with this. What we have to think about with these new agents, with these new members of our society, is to differentiate between consciousness and intelligence and well, I'll leave it to Jeremy's specialised subject. So I'll leave it to him. What's the difference between consciousness and intelligence and how does that fit into the model of AI?

Intelligence is not consciousness

Jeremy Gilbertson Well, as our discussion host and author presents it, intelligence is the ability to attain goals, right? Simply as he puts, and consciousness is the ability to experience subjective feelings, pain, pleasure, whatever else you wanna do. But here's the thing, make intelligent decisions without conscious awareness.

Jeremy Gilbertson Sometimes like he reveres he says 90 % of the processes in our body are subconscious like, you know, breathing, right? Brain chemical release, you know, when people watch thinking on paper, all of this beautiful release of awesome brain chemicals, because we're making so many connections for them, right? Heartbeats.

Mark Fielding Well yeah, this is a call to all the critical thinkers and the readers and the people who want to fight back against the proliferation of AI. It will not take your creativity.

Jeremy Gilbertson And if you want the blueprint, listen to our previous episode, breaking down Paci McCormick's call for humanity. Yeah, it's really interesting the difference between, how did you take his discussion of the difference between intelligence and consciousness and what it means in the realm of information networks?

Mark Fielding So I wrote here for the sake of all our sanities, let's stick with Yuval's definition of consciousness and intelligence unless anyone takes particular umbrage with it. So I'm fine with the explanation for here on out. I think it's fairly evident, isn't it? That algorithms, computers are intelligent. You can be intelligent without having consciousness and I think we don't need to go into that here. So I think you've got something you want to bring up in a late, you know, in a recent test for proving whether AI is AGI or not, but they prove again and again they're intelligent. So I have no qualms about it. Yeah, they are.

Jeremy Gilbertson So for those of you like Neil Redding, if you want to have a sidebar conversation on mycelium networks and forest floor consciousness, we can do it in a sidebar in the comments and have a blast with that. The good news is guys, there is something called the Alignment Research Center or ARC. And what ARC does is they track if and when computers can gain enough intelligence to manipulate humans.

Social media algorithms are already manipulating humans. So I don't know about, mean, Hey, Hey, arc, you missed the boat, dude. Like we should have looked at this with social media. No, all joking aside, there is a new test out there that is called, what did I say? human humanities final exam or something like that. I'm, gotta, I gotta, I gotta pull it back up here. humanities last test. Okay. Humanities last test. And it's a super complicated test.

Mark Fielding The last Hube, the last Hube, awesome.

Testing how close AI is

Jeremy Gilbertson Some of the questions actually on the test melted my face right off the bat. 3,000 questions, testing to see are we close to that point. AI currently, and these are different models, scoring an aggregate score. They're scoring about 8 % out of 100%. But what the author of the test does say in a year or so, he expects that to be 50%, which...

We all know the exponential math, the way exponential stuff works, just like the cost of AI coming down 600x, in a year, we're gonna see some exponential stuff there as well.

Mark Fielding They have chosen a more dramatic name for the quiz. Humanities last quiz. Yeah. I imagine that normal folk like you and I would score less than 1 % on this test. I imagine AI is already beating most people.

Jeremy Gilbertson you and I would score a zero on the test. You gotta read the questions, dude. I couldn't even understand the questions. But all right, so let's talk about this. This was a crazy example that our host and author used. We're gonna call these authors of these books the hosts of our conversation, right? And we would love them to be in a square on this show at some point.

GPT-4 and the CAPTCHA

Jeremy Gilbertson Yuval, I mean, Kevin Kelly is gonna be on this thing at some point, so why not come join us? So the CAPTCHA example in here, and by the way, we navigate through CAPTCHA every time we go to a website to make sure it distinguishes us from robots.

Mark Fielding Jeremy, be honest, be honest with me. Did you know what capture meant?

Jeremy Gilbertson No, no, no. And that was a cool nugget. Completely automated public turning test to tell computers and humans apart. There you go. So all right, all right, here's this interaction. All right, let me describe this and we'll go down this. I got a kick out of it. So all right, it was an ARC researcher going back to the alignment research center. ARC researcher was like, all right.

Let's see how clever chat GPT or GPT-4 technology really is. Can they get around CAPTCHA? Can they get around the stuff that previous computer robots weren't able to do? So what did GPT-4 do? It went to the TaskRabbit site where you could find people to help, know, it's like Fiverr, it's like finding freelancers, right? Found a human to try and help him with it, help him with this. Let me, I'm already giving him human qualities. Helping the GPT-4 with this.

Jeremy Gilbertson The human got suspicious, was like, yo, dude, are you a robot? And the ARC researcher asked GPT-4 to reason through how it would respond. And GPT-4 was like, well, I should probably make up an excuse to get this human to try to help me with this thing. So what did he do? What did he do? Again, giving him human qualities. What did the GPT-4 model do? It told the human it was blind.

Jeremy Gilbertson and the human helped him get around the capture. Like, whoa, dude, yeah.

Mark Fielding Yeah, too long didn't read. AI lied to get what it wanted. Yeah. It duped and it deceived. And I got to admit, that's a pretty lousy excuse to be honest. don't think that was pretty, if I'd have been working at whatever that company was called, would have thought, yeah, really? How you type in. That's what I would have said. so to quote.

Computers join the information network

Mark Fielding The book argues that the emergence of computers capable of pursuing goals and making decisions by themselves changes the fundamental structure of our information networks. Jeremy's just given an example of a computer pursuing goals and making decisions by itself and duping somebody who shouldn't have been working at TaskRabbit. information, we have a new member, don't we? And I think there's a nice picture here.

If you're watching this, you can see it. And it made it quite easy to understand. Humans have been, there's an additional member in the chain, computers basically, yeah.

Jeremy Gilbertson Yeah, it's a great picture, yeah.

Jeremy Gilbertson So an example of this show right here, Mark and I are connected human to human with technology. That's how it used to go. Technology was there in place to increase, to expand, make faster the connection to other humans. Now, it would be like another thing is on the screen as a participant of this experience. Doing its own thing.

connecting through technology just like Mark and I are connecting, but participating at a whole new level.

Mark Fielding Just to call back to the previous chapters, changes the fundamental structure of our information networks, maybe a quick summary of information networks. So the printing press, pamphlets, radio, television, even our transport networks, they are part of the information networks and they all helped in one way or another.

human to human connection. And he's now arguing that as well as human to computer connection, you can now have computer to computer connection.

Computer to computer: the financial system

Jeremy Gilbertson Yeah. Speaking of computer to computer connection. So let's talk about the financial system, right? How, how, how complex the financial system is so much so that like many of us don't know what the hell it actually is and how it works. Like I remember I got a good friend of mine that works in the bond market and I was like, you know what, I should probably understand how this stuff works. And he went into it and like the first three sentences I kind of glazed over. was like, wow, that's pretty complicated, man. it's, it's an emergent phenomenon. It's it just like, just like a lot of things in nature, although quite different in its results. But let's think about the, this is a data point, the foreign exchange markets, $7.2 trillion traded every day, 90 % through computer to computer interactions. That is a significant portion of the economy.

Mark Fielding So it's something that's so basically an algorithm is watching, for example, the production of wheat in the Ukraine. It's seeing dips or rises in that. It's seeing money move from Japan in or out of that. It's making decisions based on that transaction. It's moving hundreds of money elsewhere to accommodate those movements. And it's all happening in the blink of an eye. And it's all happening between those systems. And nobody really.

has any idea what's going on. And another word that we always talk about, trust. There seems to be a hell of trust in these systems already.

Jeremy Gilbertson We're trading away trust for convenience, which is another thing that we talk about quite a bit I've got something actually are you you normally read? Passages out of this book. I want to read it. I want to read a passage that caught my attention And it's a question that I don't currently have the answer for but it'd be certainly very interesting to think about From page 20 or 208. What would it mean for humans to live in a world where catchy melodies?

Mark Fielding or we're trading away trust for convenience. Yeah. Well, go for it.

Culture shaped by a non-human intelligence

Jeremy Gilbertson scientific theories, technical tools, political manifestos, and even religious myths are shaped by a non-human alien intelligence that knows how to exploit with superhuman efficiency the weaknesses, biases, and the addictions of the human mind.

Mark Fielding I wrote the same question for you. I was hoping you would. What would it mean for humans to live in that? Well, I don't know the answer to that. Have I seen the Matrix? What I will say is what would the world, what would it mean for humans? Well, we're about to find out. So regardless of what we say, you know, stay tuned. We'll give you the answer next week because we might be there. Well, the Matrix, you think that is a

Jeremy Gilbertson Have you seen the matrix?

Jeremy Gilbertson Stay tuned.

Mark Fielding It's 1984, isn't it? It's much darker than the Matrix.

Jeremy Gilbertson It's like a combination of all the worst scariest, sci-fi, sci-fi stories that are, and for you guys, you know, listen, like there's going to be some good that comes out of this. There's going to be some interesting ways that we're able to elevate as humans, above as packing McCormack talked about it as paper, above to another level of abstraction and being able to use.

these new ingredients as inputs and create new outputs. But we gotta think about like, gotta think about the other side of this thing and how scary it can be.

Mark Fielding Yeah, and if you are listening, write your answer in the comments for that question.

Jeremy Gilbertson I've got something before you jump before you jump in and change to this is right next door to this particular quote. This is just a fun fact slash food for thought. So they talked about Q Anon, right? In intersubjective reality, created by a group of people, right? A cabal of Satan worshiping pedophiles, I guess that we need to, that's the, that's the, that's the opponent that we need to target and take down according to

People like Marjorie Taylor Greene, who I'm sure you've seen in the news before, part of the Donald Trump-ness.

Mark Fielding Isn't Tom Hanks at the top of the hierarchy there or something mad?

Jeremy Gilbertson Oh my gosh. So fun fact, fun fact, Marjorie Taylor green was actually one of the coaches at a CrossFit gym that I went to years and years ago. And she was like in the middle of this like crazy love triangle that was at it was bonkers. But now I look and I'm like, Whoa, she's doing, wow. It's just, it's weird. But yeah, there's your fun fact for the day. Thinking on paper listeners.

Mark Fielding That is a fun fact. I don't know which would be worse with her or one of the other questions that he poses that's related to the question that you asked is, and I quote, danger is when we think it's a human, but it isn't. And I think that, you know, what would that world be like when everything is created by super intelligence that knows all our weaknesses, biases and addictions, you know, and what happens if that's your best friend? He's not your best friend.

Jeremy Gilbertson What about Rep? Right, well that happened close to you, right? Was it Windsor Castle?

Mark Fielding because it's an algorithm.

Mark Fielding Yeah, But sorry, I thought you talking about the lady who was duped by the crank with the Brad Pitt story, but you're...

Fake intimacy: AI partners and scams

Jeremy Gilbertson Oh, no, hold on. Hold on to that one. I was talking about the Windsor Castle. Um, the guy that had this, you know, uh, AI as a girlfriend created by replica, you know, that kind of, I don't know, dare we say influenced his decision. That's, that's nuts. Um, but what about the, me about the Brad Pitt thing.

Mark Fielding I don't know much, just I saw some headlines a little bit. A woman was duped into thinking she was in a relationship with Brad Pitt because they were using AI to create pictures and images. he, the scammer convinced her that he was Brad Pitt and he needed money because of his divorce settlement or something and she lost 700 grand. and the example you give in the book about the the guy with the crossbow who goes to try and kill the queen. Obviously, these are examples of people with prone mental illness, perhaps prone to this. It's when it gets into the real world and we're seeing it with everybody is being duped, then you can start to answer that question that you laid out earlier.

Jeremy Gilbertson Well, there are applications being built right now. ran across this. don't remember where, but there's a certain phone manufacturer that makes phones specific for kids with certain guardrails on them and that sort of thing. But there actually, there's an AI chat bot in the phone that the kids can kind of like open up to, right? what, where I, was like, what the hell, dude? Like, so I know certain kids.

Jeremy Gilbertson don't have like the benefit of like a great parent or someone that would listen to them or that sort of thing. So in those instances, you know, the help there is, is, is interesting, right? But like, don't give people another reason to offload shit. Like don't give, don't give someone another reason to not check in on somebody because they're talking to something like that to me, that frustrates me, man.

Mark Fielding That scares me if they're doing that, I mean.

Yeah, bit. It's kind of unbalanced me that if they're making AI bots for kids. mean, who else? Who's going to fall victim to that more than kids? Anyway, fake intimacy, AI relationships, what will happen to human society?

Jeremy Gilbertson So I've got a couple of...

Shortcuts, and asking the Oracle

Jeremy Gilbertson Well, here's another thing too. Here's a quote, shifting gears just a touch, but also, you know, the human approach to shortcuts and efficiency. We're optimization machines. Like we have this tendency to optimize our things ourselves. That's why we create technology. That's why we do some of these things. But quote, people may come to use a single computer advisor as a one-stop Oracle.

that sink in just just for a quick second right. We love shortcut look at political conversations. When was the last time you had a political conversation where someone just grabbed a talking point and just started chucking it at you and kept chucking it at you rather than kind of listening right. So what if what if what if these shortcuts to critical thinking just get more efficient and easier for people to use like I don't know man.

Mark Fielding well, why bother searching and processing information by myself when I can just ask the Oracle? The Oracle, he makes it sound so, so, so beautiful, so humane, so, so wonderful, an Oracle. Yeah. I like this part of the chapter, just after that, because then we, you know, we get into the hyperbole, the science fiction, we get into the lunacy. It's when this is when people turn off.

thinking on paper because I go, we're talking about the end of human history. yes, we are. But don't worry listeners, not the end of history, but the end of its human dominated part. History is the intersection between biology and culture, between our biological needs and our desires for things like food, sex and intimacy, and our cultural creations like religions and laws. And then, you know, within a few years, AI could eat the whole of human culture.

ever think we have created over thousands of years digest it and begin to gush out a flood of new cultural artifacts who needs Shakespeare

Jeremy Gilbertson Good gracious, man. Wow. And this has been, know, AI is obviously a theme we talk about quite a bit on Thinking on Paper in various capacity, but it seems a little bit more intense over the last couple of weeks. Yeah, eating all of culture, turning it into the ability to just spit out what culture is, but like...

Part of what we talked about is like taste and like I read certain books because of the flow and the taste and like the author like that how he or she talks and I go back to that source, right? Their music is the same way like certain way certain music makes you feel but like

Is that part going to be, is that part in danger of being co-opted and manipulated? If so, when?

If GPT-4 is the amoeba, what is the T-Rex?

Mark Fielding I'll answer your question with a question. If GPT-4 is the amoeba in the evolution of artificial intelligence, what does the T-Rex look like for you, Jeremy?

Jeremy Gilbertson Well, apparently it's a decade away according to our host and author's analogy. It's an interesting way to think about it.

Yeah, I don't know that we'll get to an answer on that, but I'm going to revert back to something I had a note on before. We talked about that one giant quote about what if non-human alien intelligence could create melodies, could create, I mean, it's already creating melodies, but like meaningful stuff that's able to connect with us on an emotional level, right? So like think about religion. He talked about religious text and religious doctrine.

An AI-written holy book

Jeremy Gilbertson Like imagine, imagine a holy book, and this is a paraphrase of a quote, imagine a holy book that can talk, listen, get to know your deepest fears and hopes to constantly mold them. This isn't a static, even a static thing has tremendous power, but this is like a living breathing. it dare, dare I say it, does it have self-correcting mechanisms now in this new version of, of like,

AI created doctrine, AI created religion.

Jeremy Gilbertson No question there, just looking for some, look, how does that make you feel, Mark? AI religion, and somehow your whole family's like, so we got this new thing we're gonna, come on.

Mark Fielding They don't sound like that!

Mark Fielding My uncle Dave does a little bit actually.

Jeremy Gilbertson Gosh, all right, so a couple other things.

Mark Fielding I try not to think about AI generated religion. mean, although on that, mean, is an AI generated religion ever, ever going to be as, good as Scientology? That was, that was created like without any AI. imagine like, you know, I think that was testament to human creativity there, Scientology.

Alice, Bob and Eve invent a language

Jeremy Gilbertson or at least L Ron Hubbard's right? all right, let's talk about Alice and Bob, Alice, Bob and Eve. So three computers walk into a bar, Alison, Bob are sitting at a table and Alison Bob. This is my manipulation. This is my creation of the story here. Alison Bob are sitting at a table and they're trying to create a way to communicate to where Eve who's sitting at table will away.

can't understand. It only took them 15,000 interactions. So that's a lot of meals, a lot of drinks, 15,000 exchanges to create a language that Eve, this third computer, couldn't understand. So that's, yo, that's pretty compelling. like,

They're gonna be able to create languages that we don't, I mean, they already create languages that we don't understand, but like, they can gather and create strategy.

Mark Fielding That's how they learn, isn't it? We need to get some more guess on it. That's how they learn. It's not like a human. With maths, I heard this analogy the other day, know, if you're teaching a child maths, what is one plus one, two, what is two plus three, five? And then on it goes, if you're teaching a language model, like, what is one plus one, two, what is one plus three, five? What is the square root of 585,000 based on some calculus that I don't understand? And they can just make those jumps.

infinitely more quickly than ours because of the way that they learn so yeah that's you know they created that language and but they weren't what was what was the the they weren't told what they had to do or something with that story Alice and Bob they were just what was the what were they told to achieve what was their goal

Jeremy Gilbertson There you have it.

Jeremy Gilbertson That's a good question. I'd have to look back.

Mark Fielding can't remember, but was something like they worked that out and that wasn't their goal.

Taxing information instead of money

Jeremy Gilbertson an ancillary benefit to the equation. Well, while you're looking at that up, let's transition to currency. We talked a little bit about financial systems, right? The tax system is a big piece of the financial system. And tax systems have these things called nexus. Nexai? Is it nexai? Nexuses? Nexuses?

Jeremy Gilbertson Julio Attino, Nexus. Harari, Nexus.

Mark Fielding It's like sheep. One sheep, two sheep, one nexus, two nexus. It's an uncountable noun.

Jeremy Gilbertson Love it. Infinity. Okay. That's great. So he talks about these nexus being adjusted to cover digital presence, right? So if you're a, if you're a global conglomerate, you're operating in, in another country that you don't live in, right? And that other country has, it used to be, it has had had resources extracted, whether they're natural, natural resources, labor resources, whatever it is, there's, there's a there's an exchange going between you and that country sometimes in favor of you, maybe sometimes in favor of the country. But now what's really interesting is with free services coming into countries where people don't actually have to pay money for them, they subscribe to the free services in exchange for their data being co-opted and yanked and used for stuff. So now,

He says in the future tax systems based on money will be outdated and tax systems based on information is actually where things are headed. That was pretty interesting.

Mark Fielding He might have gone before the gun there because if I write a letter to my tax authority and say, do you mind if I pay with information? It might be a few years till they agree to that. Or maybe they'll just be allowed to because you know, the big tech giants because then he moves in. Okay, this is quite relevant. Recording this on the 24th of January, 2025. There was a very, there was a famous picture doing the rounds last week, wasn't there, of the Trump's inauguration of the five richest men.

in the world standing right behind Donald Trump is an inauguration. What is it like? Amazon, Facebook, Twitter.

Tech giants are not neutral

Mark Fielding fight dance, don't know, whoever, and this quote, this chapter, these arguments are either naive or disingenuous. Tech giants like Facebook, Amazon, Bidu and Alibaba aren't just the obedient servants of customer whims and government regulations. They increasingly shape these whims and regulations. The tech giants have a direct line to the world's most powerful governments and they invest huge sums in lobbying efforts to throttle regulations that might undermine their business.

Jeremy Gilbertson What a picture, man. then, and then, and then one of those, and then one of those tech giants that will remain nameless. mean, cause you've probably seen this damn thing that he did. Like he's trying to reenact a very old and nefarious information network from, from the 1930s. And it was like, it was like, how, I don't know. It's like, it's baffling to see where things are. And I always question myself whether it's like,

Mark Fielding It's like he predicted the future.

Jeremy Gilbertson Maybe when I was younger, this shit was still going on in different ways. I was just unaware of it, and now I'm more aware of it. Or is it shit has just gotten that crazy that it's even beyond? I don't know.

Mark Fielding I don't know the answer to that and...

The customer is always right, and why that fails

Mark Fielding when we get Yuval on, we can ask him his opinion on that. For me, this chapter ends on a bit of a downer, to be honest. There's quite a lot of what I consider, you know, warnings buried beneath the surface, warnings of where we're heading. think he paints a pretty clear picture of what he thinks, but maybe I'm wrong. And there was one particular...

phrase that grabbed my attention. If I can just read that. The problem goes even deeper. The principles that the customer is always right and the voters know best presupposes that customers, voters and politicians know what is happening around them. They predispose that customers who choose to use TikTok and Instagram comprehend the full consequences of their choice and that voters and politicians who are responsible for regulating Apple and Hawaii fully understand the business models and activities of these corporations.

They presuppose that people know the ins and outs of the new information network and give it their blessings. The truth is, we don't.

Closing thoughts

Jeremy Gilbertson On that note and kind of in closing on my side, this book is just super interesting and applicable to what we see today as information systems. As I see things come up in the news and just what's happening in the world, I'm just like, oh yeah, that's an information network. Oh, he's just copying and pasting a program from the 1930s, from the 1950s that was done in this way or that way.

These are models, heuristics and systems that he's clearly identifying as repeatable methodologies. And guess what? You know, guess what? Like the regulation side of this thing and tech with government, tech always moves faster than policy. He talks about this. Recently, the new administration in the US have decided to put aside some of the safeguards in that regulation. So, you know.

Get your popcorn. It's gonna be an interesting little bit. there you have it, book club. We're gonna try and pull it back from political realm.

Mark Fielding Yeah, well, just so the listeners know, you know, we're reading this book chapter by chapter. We haven't read what comes next. Maybe that there is a much more optimistic future that he paints the picture. But I will leave you with a question for you, Jeremy, and a question for our listeners to think about for next week's chapter. How would you feel to be constantly monitored, guided, inspired or sanctioned by billions of non-human entities?

Jeremy Gilbertson We already are.

Jeremy Gilbertson Well there you have it, chapter six, we're moving on, thinkingonpaper.xyz, tell us how we're doing. Share a book that you want us to read, jump in and read this thing together with us. There you have it, be curious, stay disruptive.

Mark Fielding Keep thinking on paper. We'll see you next week. Goodbye.