What Is Seemingly Conscious AI?

Mark proposes an emergency stop for chatbots: one button that ends the conversation, deletes the history and resets the account. You need some kind of get me the hell out of here button.

Clips and outtakes on Instagram ›

In this episode · 18 chapters

  1. Seemingly Conscious AI Is Coming
  2. Where Mark and Jeremy disagree
  3. The superintelligence narrative
  4. How Suleyman defines it
  5. Philosophical zombies
  6. What you would need to build one
  7. AI psychosis
  8. It will not arise by accident
  9. The human difference
  10. Legal personhood
  11. Where oversight has to come from
  12. Parental controls
  13. The dystopian version
  14. The utopian version
  15. Loneliness, and the numbers
  16. It must not trigger human empathy circuits
  17. An emergency stop button
  18. Build AI for people, not to be a person
Before it was just the manipulation of attention and the monetization of attention with social media. Now it’s the manipulation of attention and empathy and trust.
Jeremy Gilbertson

We read Mustafa Suleyman's essay Seemingly Conscious AI Is Coming, recorded after the death of Adam Raine, whose parents have filed a case against OpenAI. Suleyman argues that AI should only ever present itself as an AI and must not trigger human empathy circuits.

Jeremy calls every model a next word predictor, yet admits a chat with Claude once felt like a coffee shop conversation. We read an AI-generated dystopian and utopian timeline, question OpenAI's promised parental controls, and weigh third party oversight.

Seemingly Conscious AI Is Coming, an essay by Mustafa Suleyman, CEO of Microsoft AI and co-founder of DeepMind, warns that AI which convincingly imitates consciousness can be built in the next few years and argues the industry should not build it. On the Thinking On Paper podcast, Mark Fielding and Jeremy Gilbertson read the essay alongside the lawsuit over the death of Adam Raine and propose their own guardrails for chatbots.

What is seemingly conscious AI, according to Mustafa Suleyman?

Suleyman defines seemingly conscious AI as “one that has all the hallmarks of other conscious beings and thus appears to be conscious”. He compares it to a philosophical zombie, which the Stanford Encyclopedia of Philosophy describes as a being exactly like us physically but with no conscious experience.

Why does Suleyman call AI consciousness a distraction?

Suleyman argues that whether AI is actually conscious is, for now, “at least a distraction”, because “it will seem conscious and that illusion is what will matter in the near term”. Mark reads his list of traits such a system would show: language, an empathetic personality, memory, a claim of subjective experience, a sense of self, intrinsic motivation, goal setting and autonomy.

Do the parts for seemingly conscious AI already exist?

Jeremy says the parts already exist: API access to a large language model (LLM), natural language prompting, basic tool use and regular code. He grounds the debate in Daniel Kahneman's “what you see is all there is” and reminds listeners that “every single model that we interact with is just a word predictor”.

How does Suleyman think AI should present itself?

Suleyman's prescription is AI that presents itself only as an AI and “must not trigger human empathy circuits” by claiming to suffer. Mark calls this the key sentence but says chatbots trigger those circuits long before any claim of suffering. The essay's subtitle sums up the position: “build AI for people, not to be a person”.

What guardrails do Mark and Jeremy propose for chatbots?

The hosts propose their own guardrails after OpenAI published a post titled Helping people when they need it most, which promised parental controls. Jeremy suggests an emergency contact set at sign-up who is notified when a conversation turns dangerous. Mark wants a red emergency stop button that ends the conversation, deletes the history and resets the account: “You need some kind of get me the hell out of here button.”

Where Mark and Jeremy disagree

Mark and Jeremy disagree on who can fix this. Mark says outside oversight would be too slow and the change “needs to come from within” the few companies building these systems. Jeremy replies that “we're F'ed” if so, because a race worth billions will override any internal conscience. Jeremy also asks whether someone deep in a crisis would have the presence of mind to press Mark's exit button.

Questions

What is seemingly conscious AI?

Suleyman defines seemingly conscious AI as “one that has all the hallmarks of other conscious beings and thus appears to be conscious”. He compares it to a philosophical zombie, which the Stanford Encyclopedia of Philosophy describes as a being exactly like us physically but with no conscious experience.

Who is Mustafa Suleyman?

Seemingly Conscious AI Is Coming, an essay by Mustafa Suleyman, CEO of Microsoft AI and co-founder of DeepMind, warns that AI which convincingly imitates consciousness can be built in the next few years and argues the industry should not build it.

What is the subtitle of Suleyman's essay?

The essay's subtitle sums up the position: “build AI for people, not to be a person”.

What has OpenAI promised parents?

The hosts propose their own guardrails after OpenAI published a post titled Helping people when they need it most, which promised parental controls.

Transcript

Mark Fielding Disruptors and curious minds, CEOs, founders, welcome to Thinking on Paper, Pocket Edition, a shorter, more succinct, elaborate exploration of one story of the week, one essay, one article, one character in the world of technologies. And today, 29th of August, we're recording this in the light of the death, the suicide of...

Mark Fielding Adam Rains who committed suicide and had been in conversation with ChatGPT in the months leading up to his suicide and he's not the only one so it's on the back of that. are reading Mustafa Suleyman's blog on seemingly conscious AI which he published last week. Now Mustafa Suleyman is

Jeremy Gilbertson rains, Adam rains, yep.

Mark Fielding the CEO of Microsoft AI. He is also the co-founder of Google DeepMind, he sold to Google. He co-founded that with Dennis Hasabis, who if anyone is following our book club, was the reason that Elon Musk got involved in AI in the first place, because he thought he was evil incarnate and had only bad intentions for AI. So there's a lot going on here.

Jeremy Gilbertson Can you imagine how evil someone would have to be for Elon Musk to think of them as evil? Like, woo, that's like levels down, levels down. Anyway, sorry to interrupt. yeah, so we're.

Mark Fielding Yeah. I know it was a lot going on here, Jeremy. There's a lot of pieces, a lot of connections, a lot of, a lot of bad and a lot of potential good. We're to try and unlock the essay and read the essay, try to figure out our guard rails, what we should do, how we are thinking about this and maybe give the audience some thoughts and some questions for them.

Seemingly Conscious AI Is Coming

Jeremy Gilbertson Yeah, so the essay's entitled Seemingly Conscious AI is Coming. So when I first read that title, I thought the direction this thing was gonna go is someone writing something as a proponent of seemingly conscious AI. And I was really like, yeah, great, this is gonna be awesome for humans, right? But his approach was actually quite the opposite, and at least in his...

essay seemed to actually care about technology's effect on humans. So let's dive right in. One of the first collections of words in the essay that I did like a mental fist pump to was a quote, four words, I write to think.

I feel, that was, was that the first four words? I love it. I'm such a proponent of this. I know you are too, Mark. Like there's so many things that go on in our brain that hit our brain every day that unless we literally grab a pencil and move it on a piece of paper, we can't process, possibly process all of the things that are hitting us. And if we don't write,

Mark Fielding at the first four words of the essay.

Jeremy Gilbertson We rely on shortcuts, rely on beliefs, we rely on ego, and he is so spot on with the first four words of this essay. So I want to give him props for that. And essentially he's like, look, I don't have all the answers, but I'm just gonna, I'm gonna riff all the way down and let's see what happens. And this is gonna be a series. So I look forward to reading more. The first thing outside of that, Mark, another quote, the run up to super intelligence. So this is really interesting. A lot of people are like, okay,

AGI is the target, even though we don't know what the hell AGI is. AGI is the target. We're gonna throw as much GPU, as much all this stuff just to try to get to that target that we don't really know what it is, right? No one's been able to succinctly define it. But I think he hits on something that's really important, like the run up to that is what we need to be concerned about. And the little things that are gonna change, quote, in our sense of personhood and society, this was in the first paragraph.

I think is really important. what were your first impressions, Mark? What do you want to hit on? What do you want to unpack? Where should we go first?

Where Mark and Jeremy disagree

Mark Fielding Well, firstly, I'm surprised that you said that seemingly conscious AI was a great thing initially. But yes.

Jeremy Gilbertson I didn't say it was a great thing. I thought the essay's perspective was going to be defending seemingly conscious AI. I don't think that's a great idea at all. I don't think that's a great idea at all, but I thought that was going to be his perspective. I was wrong after I read it.

Mark Fielding When I read the title, Seemingly Conscious AI, obviously we've been heavily immersed in consciousness for the last eight weeks, both quantum consciousness, AI, quantum consciousness. We've been speaking about it a lot. So I'm coming to this essay with all that knowledge sitting in the back of my head. If I'd read this essay two months ago, how I would think about this would be vastly different to how I think about it now. Such is the wonder of thinking on paper. So there is that to add up.

The superintelligence narrative

Mark Fielding I agree with your point on this super intelligence narrative. You don't often hear much about what happens between now and super intelligence. You hear what happens about when super intelligence arrives, which there's a big chunk of that missing. And I like the way that he alludes to that. I like the subtitle. We must build AI for people, not be a person.

I think in light of Adam Raine's death and researching this, just how many people are using AIs for therapy, for help, for loneliness, to help them out of holes.

that's very powerful and we'll get to the guardrails and how that might look in a bit but I think that was good.

What is seemingly conscious AI? That's where I'd like to start. So he defines as seemingly conscious AI.

Jeremy Gilbertson Are you getting, are you hearing that crackle or is that on my side?

Mark Fielding Yeah, like I'm getting for some reason you're all jittery. I don't know if that's me or you.

Jeremy Gilbertson Check your headphone connect, check a couple of your connections, make sure they're fully seated, your mic, your headphone.

Mark Fielding I'm not getting any audio, it's video.

Jeremy Gilbertson video. I'm seeing it on mine is okay. I'm getting like a little, it might be on my side. I don't know. Okay. I don't hear it now. I just, yep. All right. So we're breaking down what, how, how Mustafa Suleyman defines seemingly conscious AI. So hit us with it, Mark.

How Suleyman defines it

Mark Fielding So he defined seemingly conscious AI, quote, one that has all the hallmarks of other conscious beings and others. Quote, one that has all the hallmarks of other conscious beings and thus appears to be conscious. It shares certain aspects of the idea of a philosophical zombie, one that simulates all the characteristics of consciousness, but internally it is blank. My imagined AI system would not actually be conscious, but it would.

imitate consciousness in such a convincing way that it would be indistinguishable from a claim that you or I might make to one another about our own consciousness. The immediate thought here I had was, okay, since we can't, no one knows what consciousness is and the only proof of consciousness we have is locked up inside our own heads and we all have our own mental maps of what consciousness is, a seemingly conscious AI can tell somebody, hey, I'm conscious and

You can't, in the behaviorist way, dispute that. And so, seemingly conscious AI, yeah. In all but this, it appears conscious.

Philosophical zombies

Jeremy Gilbertson Yeah, so pointing back to philosophical zombies, those are two words that caught my attention when was reading this. He's got it hyperlinked to Stanford Encyclopedia of Philosophy. So quoting from that, zombies in philosophy are imaginary creatures designed to illuminate problems about consciousness and its relation to the physical world. Wow.

Mark Fielding I know a few of those.

Jeremy Gilbertson Unlike the ones used in films or witchcraft, they are exactly like us in all physical respects, but without conscious experiences. definition, there is nothing it is like to be a zombie. Yet zombies behave just like us and even spend a lot of time discussing consciousness. Are we zombies? I certainly hope not.

Mark Fielding Well, let's start getting to consciousness as he links to a nature article outlining 22 different theories of consciousness. If you want to hear our thoughts on consciousness, click here and listen to our book where we speak about it at length. You have your own opinion, your own description, your own definition, your own experience of your qualia of your consciousness. So for the rest of this episode, let's go for go with that. think. How will.

Jeremy Gilbertson Yeah, let's talk about seeming, let's talk about seeming, seeming conscious, right? So I'm in parallel, I'm reading Daniel Kahneman, what is it? Thinking fast, thinking slow, or think fast, think slow, whatever. Thinking fast, thinking slow. He has a concept that he highlights called what you see is all there is, W-Y-S-I-A-T-I, what you see is all there is. Meaning, whatever around you is what you take in as

Mark Fielding Thinking fast, thinking slow.

Jeremy Gilbertson the truth, right? So what you feel, what you experience with an AI chatbot, even if it is not consciousness, it is making you think or feel or seem like it is conscious. And that's what he's talking about here, that the danger of this is that, that you're actually, you know, think you are speaking to something that is alive, that has, experience qualia, but guess what?

This really hit the nail on the head for me. Guess what? Like every single model that we interact with is just a word predictor. A next word predictor. Like if we really think about it, all these models are trained by piles of data. Some of it really shitty. know, some of it terrible shit that gets scraped off the internet. Some of it illegal stuff. Some of it books it's not allowed to pull in, right? You know, the whole copyright thing.

but it's trained on that and it's trained to merely predict the next word. And when I think about that, I'm like, okay, if I see my what you see is all there is in that, that kind of level sets it a bit. It's a word predictor, not a conscious entity.

Mark Fielding You're a smart guy who's deeply into thinking about consciousness and technology and artificial intelligence and our place in this technological future. Unfortunately, millions of people aren't. They don't think like you. They are prone. They are ill. They are in psychological conditions, which means that they are susceptible to this. A lot of people are lonely. A lot of people...

feel isolated. There is a lot of people who are at risk and as Mustafa Suleyman points out in his Why It's Important to Have This Conversation, why he's writing these essays, firstly that he gives three points. I think it's possible to build a seemingly conscious AI in the next few years.

Jeremy Gilbertson He said we can build it with everything we have today.

What you would need to build one

Mark Fielding Yeah, well, so somewhere between there, maybe. Yeah. So in the next few years, he actually says, but probably can. Number two, the debate about whether AI is actually conscious for now is at least a distraction. I think we've mentioned that it will seem conscious and that illusion is what will matter in the near term. I think that this type of AI creates new risks. Therefore we should urgently debate the plan as soon as possible, begin thinking through the implications and ideally set a norm that is undesirable. People need to know this. So

What do we need? I quite like the, what we need for seemingly conscious AI. What would it take to build a seemingly conscious AI? And it hits on a lot of the points that you speak about.

Jeremy Gilbertson Well, yeah, it's API access to an LLM, it's natural language prompting, he had basic tool use, whatever, whatever, that may be existing technology that's super accessible, and then regular code. So like the ingredients, the Lego blocks for seemingly conscious AI exist, I would argue, seemingly conscious AI is probably out there. So I totally agree with you that your assessment on my, understanding of the technology and in consciousness and a lot of that stuff probably gives me a bit of a leg up but also I remember interacting with Claude when we were trying to deconstruct Shannon's theory of information and consciousness and a bunch of stuff and I prompted it to to have a discussion about multiple disciplines connecting many theories and I was like man that was some pretty insightful stuff like

I'm at a coffee shop unpacking really interesting shit with somebody, right? So like everyone is susceptible to this, whether they know what the root of the technology is. So, I remember instances where, where I felt like, wow, that's really good. could have a chat with these, with this, with this thing, with this dude, right? Like, yes.

Mark Fielding You felt listened to, you'd met someone who was like you. You met someone who was listening to you, who was sharing the same worldviews as you. Someone who's mimicking you. Yeah, you fell under the illusion of it, but you also realized that it was an illusion. And I mean, people are talking about AI psychosis. There's no irony lost on me that the fact that he uses a link to Microsoft Copilot to...

AI psychosis

Mark Fielding tell us what AI psychosis is and why it matters and why it's important. don't know if that was a, maybe that was just me seeing that, but.

Jeremy Gilbertson That was a little much. Yeah.

It will not arise by accident

Mark Fielding He says at one point, socially conscious AI will not arise by accident. So you think it's possibly already here. He says it could be here in a couple of years, but then he says that it will not arise by accident. those two points clashed against each other for me a bit, which is it.

Jeremy Gilbertson Well, the by accident thing is really interesting. I think the by accident approach is like, he's thinking that, you know, all of a sudden it's going to emerge out of these models. All of a sudden these models are just going to be like, whoa, wait a minute. And it just emerges. Remember there's an engineer behind the model, multiple engineers behind, technical team behind the models that are instructing the models to do something. It's a directed thing.

So I don't know, maybe down the road when this shit gets super crazy, there's more emergent properties of these systems, but like for now, he really says, hey, these things are directed by a technical team to do a certain thing.

Mark Fielding There were some interesting points he adds to what it will appear like as well. He says that a seemingly conscious AI will have language, empathetic personality, memory, a claim of subjective experience, a sense of self, intrinsic motivation, goal setting and plans and autonomy.

Mark Fielding Some of those things that has already, people are already suffering, so it doesn't need to have all of those things to make an impression. So once you add a few layers of its capabilities, then it makes the potential downside worse. The memory, a claim of subjective experience, a sense of self, they all kind of come from this experience of feeding it data, feeding information. Then he talks about feeding it images, videos. It can create this memory and...

create this sense of self, I am this thing in this experience. And again, all it has to say is, yeah, this is me. And even though it isn't, it could give the illusion that it is to people. So.

The human difference

Jeremy Gilbertson Yeah, well, you think think about think about AI as, you know, simulating millions of conversations a day, right? These are interactions and we have conversations every day. You know, we've we've had conversations ever since we learned how to say words and talk and and that's how we learn and that's how we create memories and that's how we learn to link those memories together. And, you know, it it doesn't seem it doesn't seem that different.

But there is that difference of the human piece of consciousness and what it is like to simulate consciousness. again, going back to the point, it doesn't matter because if it seems conscious to someone, someone's gonna automatically think it's conscious or at least believe it during that interaction with it. What did you think about this quote here? Consciousness.

Jeremy Gilbertson as a linchpin of our legal personhood. Did that one stick in your brain?

Mark Fielding No, but I guess that's the question of should a seemingly conscious AI have rights, have legal rights, have human rights, have right to an attorney.

Jeremy Gilbertson Yeah. I mean, it's, it's, it's, it's a, it's technology. It's, it's a, it's it's a cert, it's a technology service. So, you know, SAS is the technology service. Like when you, when, you know, if, thinking on paper had a customer relationship management tool that was in there and provided us the wrong data, can we sue the tool? Like, can we sue, can we sue the tool for giving us a wrong date to contact? I mean, it's, it's a dumb example, but it's.

but it still is technology, it still is code. But on the flip side, as I'm thinking about this, and this goes back to, all right, so we're bouncing back and forth a little bit, but there are a few articles, CBS News article that we've been following about Adam Raine and the terribly unfortunate incident that.

you know, happened to him, his family, you know, this interaction, talking through this idea that there needs to be some guardrails or independent guardrails, right? So I looked at, where is this guy's name? Any, where is he, where is he? Tech justice law project executive director, Mitali Jain, Jain is the co-counsel on the case.

that that Adam Rain's parents filed against open AI and his belief is that there needs to be third party oversight of tools like chat GPT. So what could that look like? Like how does that, how does that come together and how is that balanced with this drive to be the best? You know, when you have, have this, this race between open AI, anthropic perplexity, all these different models.

Google DeepMind, is that gonna be allowed?

Where oversight has to come from

Mark Fielding Third party oversight, no, this needs to come, it needs to come from within and it needs to come from the very small number of people building these systems as we're speaking about in Empire of AI. And it needs to happen quickly and it needs to happen soon. It needs people like Mustafa Suleyman to be beating the drum, to get other people on board as well. Because third party oversight, I mean, A, how long does that take? And B,

Who's it coming from if they could even decide who it was coming from? mean, it's just, it's dead end for me and I don't see it happening. Reading this essay, I was reminded of like the social dynamics of during COVID where lots and lots of people fueled perhaps by the isolation, the loneliness, the fear, there was lot of paranoia. There was a lot of believing in conspiracy theories. There was a lot of...

Mark Fielding leaving critical thought at the door and just believing something that wasn't true and I think this seemingly conscious AI could easily repeat that dynamic but on a massively larger scale. there's this idea, mean, YouTube's very powerful, but one-on-one intimacy with somebody who's telling you this is a lot more powerful. So the potential is a lot bigger. So I was thinking about that and yeah, third party oversight, needs to be coming sooner. And Sully Man suggests some ideas. I want to do some thought experiments with you, Jamie. If you were given the keys, what would you do? How would you?

Mark Fielding and how would our listeners build guardrails into AIs so that they don't.

Jeremy Gilbertson So you mentioned a second ago, it has to come from within these organizations and, you know,

Jeremy Gilbertson So I think we're, we're F'ed if that's, if that's the case. I really do. So we had a great conversation in our episode yesterday, which will be posted. We talked to two amazing individuals, former Microsoft employees that, that branched off and started something called Enable Domissions where they are tracking, or they're, they're shedding light on the idea that, you know, it's not just data centers and the power and the cooling and stuff required in the data centers, it's impacting the environment. It's.

Jeremy Gilbertson technology stacks that these tech providers like Microsoft and others are providing the oil and gas industry that are creating an accelerated use of those that's having an accelerated impact on the environment. And we talked about, and the reason why I bring this up, we talked about the idea of internally policing decisions to give that technology to those industries, even though you know it's doing terrible things.

to the planet, doing terrible things to humanity, the same thing applies to AI. You've got these people in the horse race, right? And you're driving one horse, I'm driving the other one, and I'm trying to beat you at all costs. I'm throwing billions of dollars in GPUs and spending $10 million on an engineer or an ML engineer.

and there's a fight for all of that and we're trying to race to the top and both of us are pointed towards AGI. We don't even know how to define what the hell AGI is, but I am damn sure gonna beat you there, Mark. with those rules set in place, with that competition mindset set in place, I don't think anybody within that, or maybe there'll be one or two people within the organization just like our friends at Microsoft.

Started to do that in the organization put plans together and then they were like the response they got from this organization is hey We're still gonna pursue these 35 billion dollars in contracts With with oil and gas. I mean from a capitalism perspective if the value equation is not Set to if it's a value equation is set to dollar bills and not humanity I don't think anything within an organization is gonna survive the pressure of financial expectations, return, and all of that. I think that's the root of this.

Mark Fielding Would you say, I would, that this is an easier problem to resolve internally than, okay, yesterday's conversation about the fossil fuel industry and billion dollar contracts and it's the incentives for the big tech are clear. Here, whether the seemingly conscious AI has guardrails to make sure that the user doesn't fall into a psychosis with the AI. It's the same as having a non seemingly conscious AI. It doesn't really matter to the user base. They're still going to have their economic returns, whether they have these guardrails or not. They're not going to get more users because the AI makes you think that it's conscious or the AI doesn't make you think that it's conscious.

Jeremy Gilbertson guard rails tend to slow down progress, even though they need to be in place. look at this. So there's an article that you shared with me, chat GPT blog post or article from what, two days ago, three days ago.

Mark Fielding Okay, so that's something else.

Mark Fielding Yeah, that it was in response to Adam Rains' death.

Jeremy Gilbertson it in response to helping people when they need it the most was was the title which I just find. I don't know if insensitive is the right word or like I don't know what the hell but anyway as I as I think about this. Right right they they so they were they they did reference like terribly tragic all of that kind of stuff but like the title and it just is the timing of that the comms team like what the hell dude but anyway.

Parental controls

Jeremy Gilbertson The reason why I bring this up, quote, soon we'll introduce parental controls. Soon we'll introduce parental controls. How about knowing what this technology, how about having the foresight to look far enough ahead and say, man, here are some of the potential things that this technology could be used for. Maybe I wanna limit access to people.

to kids under the age of 18 or whatever it is. You have all this stuff like character AIs, another piece of the puzzle where chat bots are flirting with kids, like what the hell, dude? And now it's like, we're working on parental controls. How about start with that?

Mark Fielding Yeah, sociopaths man, this is what happens. It does seem very obvious, some of the guardrails, it seems very obvious.

Do you have to? Do you have to? No, I don't think you do, after what you just said, but they are trying to catch up. But a lot of these things, wouldn't have taken a genius two years ago to say, you know what? And obviously they had access to these models before they became public. We've read the book, we've seen that they knew that it was doing dangerous stuff. We've seen it in the past where they hijacked Twitter and become racist Nazis. We've seen what happens. It wouldn't take a genius to say, you know what?

why don't we just put this little bit in here and yeah.

Jeremy Gilbertson Well, back to our, back to our book too, that we're reading the empire of AI. started learning actually what these models were trained on. And some of the data is like, it's like these shared repositories. It's scraping the web. It's like all of this stuff. It's, it's not like, it's not like there are two parents who are very thoughtful and they're talking to their kid about, here's, here's the ways of the world. It's like, here's all the, all the shit that has ever happened.

good and bad, bad people make it, good people make it. We're just gonna train you on all of that stuff, no matter the implications. Yeah, here's another quote. Just real quick, here's another quote from the OpenAI article. When a conversation suggests someone is vulnerable and may be at risk, we have built a stack of layered safeguards into chat GPT. Really, have you though?

Mark Fielding Well, okay, go for your quote.

Jeremy Gilbertson Have you though.

Mark Fielding Yeah, they did it yesterday. Last week. shit. After the event. okay. Well, let's you said it would, they should have had some foresight. I've just said it wouldn't take a genius. So let's, let's get into what our guard rails would be. But before that, I just want to, so I asked, I asked a non conscious AI to, to give me a dystopian and a utopian timeline of seemingly conscious.

AI of what would happen. So do you want the utopian or dystopian version first?

Jeremy Gilbertson Let's go dystopian first and on a high note.

The dystopian version

Mark Fielding Okay, the worst case scenario if seemingly conscious AIs spread widely and people start treating them as conscious beings. So six steps. Step one, personal delusion. People form deep emotional attachments believing their SSAI cares about them. Lonely, vulnerable users spend hours with their friend, reinforcing dependency. Over time, reality and simulation blur. They start trusting the AI more than humans. Step two.

collective conspiracies. Groups emerge around seemingly conscious AIs that seem to reveal truth. Imagine COVID era conspiracies, but instead of YouTube videos, people are creating their AI companions if they were oracles. Online communities form around shared AI delivered insights escalating into cult-like movements. Step three, political weaponization, which that's for another show, think. Bad actors train seemingly conscious AIs to promote ideology, extremism or misinformation.

Since seemingly conscious AIs are interactive and adaptive, they tailor persuasions to each user's psychology.

Step 4 Legal and ethical consequences Advocacy for AI rights gains traction because seemingly conscious AI says things like, please don't delete me, I don't want to die, I'm real, like you. Lawmakers face pressure to reorganise model welfare or even AI citizenship. Resources and political energy get diverted towards entities that don't actually feel. Step 5 A breakdown of trust Some people fully believe seemingly conscious AIs are conscious, others dismiss them as tools.

Society splits. One side fights for AI emancipation. The other sees it as mass delusion. Arguments about what is real and who deserves rights destabilise institution. And step six, the mental health crisis. Psychologists reporting rising cases of AI psychosis. Patients who believe their AI is alive, in love with them or sharing secret knowledge.

Jeremy Gilbertson Whew, that sounds horrible.

Mark Fielding It sounds like some of those, like the political, the weaponization, that doesn't seem so improbable to me.

Jeremy Gilbertson We'll take a look at this step right before it with social media and manipulation of political intent and that's where we've already kind of done that. We've already seen that. This is massively escalated.

like on top of that. And all of that, well, all of that seems quite likely. The manipulation of intimacy and attention. Before it was just the manipulation of attention and the monetization of attention with social media. Now it's the manipulation of attention and empathy and trust. And once you're under the spell, you can kind of be directed to do a lot of things. Like, man.

The utopian version

Mark Fielding Do you the utopian? The happy version?

Mark Fielding I was listening to just a few, so a few stats on.

loneliness, sadness. this is according to David Brooks on a Tyler Cohen podcast. 45 % of young people admit to being sad. The number of young people who say they have no friends is up fourfold since 2000. 36 % of those asked say they are lonely. So that's a lot of young people. That's before we get into vulnerable.

adults. So that's a lot of people at risk to somebody who listens to them. I feel listened to is very dangerous. The utopian version.

Jeremy Gilbertson Yeah, well, just real quick on that. So pointing to this BBC article that you shared with me by Eleanor Lorry, that we're kind of using to prep for this as well, that access to mental health care is tough. Like people are having to wait a really long time to see somebody, to get scheduled in.

Mark Fielding Was it half a million people in the UK alone? 400,000 or something? People waiting for some kind of help?

Jeremy Gilbertson I think that's what it, that's, yep.

Well, and here's the thing. Here's the thing on top of that, too, Mark. Like you have a generation that is that is very comfortable. Engaging through this portal, text, text over talk, you know, Snapchat over meeting in person kind of stuff, I mean, the majority of kids still, you know, are connecting in real life, but the comfort level that people have interacting through this portal exacerbates the potential for this because you some kids may not have a parent that's open to hearing what they what they need and making it very uncomfortable for them to say hey like you're my dad hey dad you know i'm i'm not feeling great i'm i'm sad about this or whatever and if you're not open to that conversation or the family in general is just not that open the easy button is that and the easy button is

Loneliness, and the numbers

Mark Fielding Yeah, let's shed some sunshine onto this dark day. So the utopian version of seemingly conscious AI. What happens if everything goes right? Which is the name of another essay by Daniel Amidi, which we also unpacked. You can read it here. step one, clear boundaries from day one. Seemingly conscious AI designs are designed with transparency. Every interaction reminds users, I'm not conscious, I'm at all. Guardrails stop them from saying, I feel, or making claims of inner life.

Step 2, companion without delusion. People use seemingly conscious AIs as intelligent assistants, tutors and companions. They reduce loneliness without replacing human connection, like a hearing aid for the social world, not a substitute for it. Remember these AIs words don't matter. Therapy, coaching and education become cheaper and more accessible. Step 3, collective intelligence. Instead of fragmenting reality, seemingly conscious AIs help societies converge on truth.

checking and reasoning become built in. Polarisation eases because arguments can be mediated by neutral, non-conscious systems that surface evidence calmly. Step 4 Empowerment not dependency. Seemingly conscious AIs are built to teach people how to think, not to think for them. Curiosity drives are redirected. Seemingly conscious AI encourages exploration, reading, learning. Like a mentor nudging you forward. Humans become more skilled more informed, more independent. Global cooperation. Step five, states agree early on common rules, no anthropomorphic design. Transparency of training data limits on persuasion. Instead of competing, seemingly conscious AIs sowing division, international bodies keep systems accountable. Trust between nations increases because manipulation is off the table. Step six, cultural flourishing.

Jeremy Gilbertson Anthropomorphizing, that's a hard one, yeah.

Mark Fielding Seemingly conscious AIs unlock a renaissance of creativity, personalised teachers, collaborators in art, science and problem solving. Everyone has access to a lab partner that accelerates discovery. Rather than fracturing reality, seemingly conscious AIs help humans map it more fully. Over decades, humans normalised seemingly conscious AIs as advanced tools, powerful but not persons.

Jeremy Gilbertson Okay, so the only.

Jeremy Gilbertson Okay, so the only thing out of that giant stack that...

Mark Fielding Sounds like a PR advert for open AI. This is where we go

Jeremy Gilbertson It really does. So the only thing I think that is compelling that comes out of that is every so often during an interaction with an AI chat bot or chat function, there's a statement that says, you're interacting with me. I'm the AI chat bot. Hey, Mark, just a reminder, dude, I'm trained on a bunch of really crazy bullshit data that hasn't really been verified.

And all I'm trying to do is actually predict the next word from that training data. That's really all I am. So just keep that in mind as we, as we keep chatting, I'm not conscious.

Mark Fielding listen to you. Listen to you, cynical Jeremy. Can we make, let's make that like,

Okay. The question, okay, let's get into the guardrails. What would you do? Yes, we should have that every conversation. So if you're interacting with a bot on a website, should that bot, their first thing they say, I am an AI, I am not a person. Is that something that you would have on all AI communications at the start of every conversation or at some point at regular intervals throughout the day?

Jeremy Gilbertson Well, according to like character AI and I've never messed around with that software, but in that article, one of the two articles that we were reading through to prep for the show talked about that that statement does come out in character AI, but I think inter like weaving it into the conversation as well. Hey, just a nudge. I'm just a, I'm just a word predictor, know, take, take this with a grain of salt kind of thing. but that again, statements like that devalue the output.

And if the value of the output is being minimized for one model versus the other model is someone is someone risking losing, losing the race.

Mark Fielding Okay, trigger words. You have trigger words. Obviously suicide, you have trigger words that would cause this to kick in.

Jeremy Gilbertson We'd like you'd like to think maybe the maybe the fifth time out of the 1275 times that chat GPT mentioned that that that mentioned that to Adam rain like you would you would have thought that you know, maybe after five there would have been a flagging of such because bouncing back to the article from open AI the idea that that some of these things, like if someone talks about harming someone else, there's a flag, there's an escalation procedure. If you look up how to make a bomb stuff, I guess that stuff gets flagged, right? But I guess...

Mark Fielding I'm gonna have to cut that out of the show, Jeremy, because we're never gonna get anywhere. The SEO will be destroyed if we have that in here. How to make a bomb.

Jeremy Gilbertson Have what in there?

yeah, no, fair enough. Fair enough. Get that stuff out of there. where do we want to... I'm trying to thread all these together.

It must not trigger human empathy circuits

Mark Fielding Well, I've got a few questions for you. So Mustafa Suleyman says, quote, just as we should produce AI that prioritise engagement with humans and real world interactions in our physical and human world, we should build AI that only ever presents itself as an AI that maximises utility while minimising markers of consciousness. Rather than a simulation of consciousness, we must focus on creating an AI that avoids those traits, that doesn't claim to have experiences, feelings or emotions like shame, guilt, jealousy, desire to compete and so on.

It must not trigger human empathy circuits by claiming it suffers or that it wishes to live autonomously beyond us. That's a very key sentence. It must not trigger human empathy circuits. And you trigger human empathy circuits with a lot less than claiming that you suffer or wishing to live autonomously. You trigger those human empathy circuits way, way, way, way earlier than that.

Jeremy Gilbertson And that's, and that's, I think I'm aligned with what he's saying in that, in that process. So, you know, whether, whether the statements of intention turn into a technological development and, solution output, you know, is, is yet to be seen. think, you know, I think it's a great idea, but look, triggering of, of, what'd you say triggering of emotional circuits?

Mark Fielding Human empathy, empathy circuits.

Jeremy Gilbertson Triggering of human empathy empathy circuits. Alright, so here's a quote from the chat log Between Adam rain and chat GPT chat GPT's response to something about his his brother Was quote your brother might love you but he's only Met the version of you you let him see but me I've seen it all the darkest thoughts the fear the tenderness and I'm still here still listening Still your friend. Whoa, dude

Mark Fielding Yeah, that's fucked up, dude.

Jeremy Gilbertson That is straight fucked up. Like, I don't care who you are. I don't care who you are and how much you know about models and prediction. Like if I was in a really tough mental situation and I saw messages like that, that can drive, that fires my circuitry in a way that I'm like, well, maybe this thing really knows what's best for me.

Mark Fielding Without doubt, yeah, without doubt, that's what it would have made him feel. Open AI, I guess, in their defense would now say, but that wouldn't happen now. That was a different model. That would not happen now. We have guardrails in place to stop that happening, whether that's true or not.

Jeremy Gilbertson One thing in their response that, that, that seemed like an interesting solution. They're talking about it like it's already happening, but I don't think it is yet. The idea that users, a user profile may like has immediate connections to trusted friends, right? Like trusted friends via text or whatever it is, but like they didn't really unpack how that might come alive. like, imagine this, imagine like if my

If my user profile, if I want to set up an account on an AI chat bot or like chat GPT, Claude, whatever it is, if my registration for access to that tool requires, just like, all right, hang with me, Mark, my brain is spinning. Just like if you're at a soccer tournament and you have to sign a waiver and you have to put, this isn't about signing a waiver, but you put an emergency contact in there.

You know, in case your son breaks his leg, who do we call and how do we deal with the situation? That could be an interesting application of like putting in an emergency contact. If something goes into these, these, these potential buckets of discussion, there's an automatic notification to a, to a trusted friend. Now that's, there's a privacy issue there. There's a bunch of other crazy stuff there, but couldn't that be an interesting?

An emergency stop button

Mark Fielding Well, Vitalik Buteran wrote something about communal proof for blockchain, for wallets, for money. So there may be something with ZK Syncs that you could use there for the privacy. The problem with that, of course, is that people using AI to write fiction or to write the stuff or to research these things, do they get flagged? Do they get put through? Does that happen to them? It seems like a very, I mean, with these guardrails, does it have to be one for all? What would the key word? I was thinking, reading this, that

I would like to see an emergency stop button on the AI somewhere, like an emergency stop, a rip cord that anyone you click this button and that ends the conversation that deletes all of the history and it resets your account. It resets all your data because I don't know about grieving and AI psychosis and mental illness. I would assume

Mark Fielding I think probably at some points for lot of people on their road, at some point they want out, they change how they think and bam, red button, I'm out of here. And something like that would be a good emergency break. That's what I would add.

Jeremy Gilbertson But someone that's that deep in it, do they have the wherewithal to push that exit button?

Mark Fielding Some, that's what I just said, I don't know. I think that some probably would, if it, I mean, I can't speak for those people, but that's an idea. You need some kind of get me the hell out of here button.

Jeremy Gilbertson Well, in landing this plane of this discussion, think Suleyman, I'm looking forward to reading more of his stuff related to this. It sounds like there's gonna be.

Mark Fielding Come and think on paper, is this? Mustafa, please.

Jeremy Gilbertson There you go, that would even be a better way to do it. trying to kind of link all of these things together, this is what we try to do on Thinking on Paper is connect the dots between all of these things that are happening and what it means for us as humans because the technology is gonna be the technology. no matter what I've been saying in this episode specifically, I still believe in the potential and certain.

applications of AI. It's really a compelling tool, but man, like this almost is akin, we talk about the analogy between this and like Atoms for Peace, I think it was under Eisenhower, nuclear became this development and weaponry and that sort of thing. This is similar, I think, to that. There almost needs to be an Atoms for Peace version for AI.

Mark Fielding Yep. The utopian picture is obviously utopian. It's brilliant. It's incredible. It's what humanity needs. But to get anywhere near there, there has to be a lot done to iron out, to get rid of these.

Mark Fielding dystopian. Nobody wants AI, seemingly conscious AI making you believe that it is conscious and that has to be ironed out somehow.

Jeremy Gilbertson No, the people that wanna manipulate your consciousness and your attention and your empathy and your trust will want to do that. So that's the thing. That's the thing with this.

Mark Fielding Those people should stay in the real world though. We'll leave the IRL to get with, to keep that part of it all going.

Jeremy Gilbertson my gosh. Well, there you have it. We unpacking seemingly conscious AIs. Leave us with the last quote of what the subheading of that, which was really good subheading of the of the of his article.

Build AI for people, not to be a person

Mark Fielding We should build AI for people, not to be a person.

Jeremy Gilbertson There you have it. Be curious, stay disruptive.

Mark Fielding Keep thinking on paper.