Karen Hao’s Empire of AI: Who Really Benefits From the Race to AGI?

Part one of Empire of AI follows OpenAI from a nonprofit set up in opposition to Google to a company chasing compute.

Clips and outtakes on Instagram ›

In this episode · 17 chapters

  1. Empires crumble
  2. The names behind him
  3. Trust
  4. By chapter five
  5. Bill Gates
  6. The power struggles
  7. Was it really about safety?
  8. The Brockman interview
  9. The current AI paradigm
  10. Pushing Elon out of the nest
  11. Who makes him feel pressure?
  12. Why the board moved
  13. Six black researchers
  14. Chapter three, the nerve centre
  15. Concentration of power
  16. Cade Metz
  17. The most important book of the decade
We’re building something that’s going to get so out of our realm to control that at some point it’s just going to be there.
Jeremy Gilbertson

Mark and Jeremy read part one of Karen Hao's Empire of AI, chapters one to four. They follow OpenAI from a nonprofit set up in opposition to Google to a company chasing compute, and ask whether its founders ever meant the mission.

The discussion covers the names behind Sam Altman, Elon Musk's exit, the board that removed Altman and the staff who brought him back, and Greg Brockman's answers to Hao. It ends on why describing AI as learning like humans can help companies avoid responsibility.

Empire of AI: Dreams and Nightmares in Sam Altman's OpenAI by Karen Hao, part 1, chapters 1 to 4, read on the Thinking On Paper podcast book club, follows OpenAI from a nonprofit set up for the good of humanity to a company racing to scale towards AGI. Mark Fielding and Jeremy Gilbertson test Hao's frame of AI as an empire.

How does Karen Hao define an empire?

Hao’s frame for the book, read on the show: empires “seized and extracted resources that were not their own and exploited the labor of people”.

How did Karen Hao source the emails and documents in Empire of AI?

Hao spent time inside OpenAI with its leaders, and says that any quoted emails, documents or Slack messages come from copies or screenshots, or appear exactly as they do in lawsuits.

How did OpenAI define AGI?

OpenAI defined AGI as software that can match or exceed human performance on most “economically valuable tasks”. Jeremy singles out the qualifier: not everything people do can or should be monetised.

How does Hao say the current AI paradigm limits other paths?

Hao argues the current AI paradigm “is also choking off alternative paths to AI development”. Jeremy cites the book’s figure of six Black researchers among 8,500 attendees at the NeurIPS conference in 2016.

How does describing AI as humanlike help companies avoid liability?

In a passage drawing on Cade Metz, a longtime chronicler of AI, the book argues that describing AI as learning or reading like humans “has become a rhetorical tool for companies to avoid legal responsibility.”

Where Mark and Jeremy disagree

Jeremy asks whether OpenAI’s nonprofit ideals were sincere or a way to beat Google; Mark says he is not in their heads. He also stops short of calling it the most important book of the decade.

Questions

What is Empire of AI by Karen Hao about?

Empire of AI: Dreams and Nightmares in Sam Altman's OpenAI by Karen Hao, part 1, chapters 1 to 4, read on the Thinking On Paper podcast book club, follows OpenAI from a nonprofit set up for the good of humanity to a company racing to scale towards AGI.

What do chapters one to four of Empire of AI cover?

Mark and Jeremy read part one of Karen Hao's Empire of AI, chapters one to four. They follow OpenAI from a nonprofit set up in opposition to Google to a company chasing compute, and ask whether its founders ever meant the mission.

Who discusses Empire of AI on the Thinking On Paper book club?

Mark Fielding and Jeremy Gilbertson test Hao's frame of AI as an empire.

Transcript

Mark Fielding Disruptors and curious minds, CEOs, founders, book lovers. Welcome to Thinking on Paper Book Club, the rebellion to the 15 minute book summary. There will be no AI chat summaries. We will read every sentence. We're reading Empire of AI Some, some, Jeremy, are calling this the most important book of the decade.

I mean, if you've got any interest in AI technology and where we're heading as a species, I'd say it's pretty important. Let's set the scene. Perhaps you are a professional who has sped up your administrative works in ways that have boosted your productivity. Maybe instead you are a company leader who has been able to trim your workforce while increasing your margins to stay competitive in the market. But like the cotton gin in the 1790s, the education technology startup in Massachusetts, the facial recognition companies in South Africa, and the many more examples detailed in the coming pages. The costs of this vision are pressing down on vast swathes of the global population who are vulnerable. This is the empire's logic. The perpetuation of the empire rests as much on rewarding those with power and privilege as it does on exploiting and depriving those often far away and hidden from view. Without them. Jeremy, Empire of AI.

Karen Hao dreams and nightmares in Sam Altman's Open AI. First impressions. Off we go.

Jeremy Gilbertson Off we go. First impressions.

First impressions. What's up guys? drop, drop some questions in the chat. Good to see everybody. Glad we're doing this live again. this is kind of back, back to where it all began. Right, Mark? starting off just the word empire, I think is pretty arresting, just that word in general, it just puts you, puts you on a sense of awareness. It's like, yeah, we're gonna understand some of the deeper, more important connections of this thing, but it's a dark word. Empire is kind of a dark word. The empire strikes back. That empire is a dark word. Empires in general, colonialism, all that stuff that we'll get into, pretty dark stuff. But I think it's important. What's happening with AI right now, we're just seeing things running and gunning and building and...

more compute, more power, more compute, more power, models get better that way. And, before we know it, it's like down the road. I'm excited to dive into this kind of call for call for pause to at least understand what it is that we're building and how it's going to affect us, our minds, society. the, thing that scares me the most about a lot of this stuff is, you know, we're already, humans have a proclivity to shortcuts, And this is like the ultimate shortcut. And are we gonna shortcut things like creative thinking? Are we gonna see that atrophy? Are we gonna see creativity atrophy in light of an easy button?

Empires crumble

Mark Fielding Well, empires crumble and if you're gonna, the word I think is obviously it's a very, we speak a lot about language on this show. to build an empire, to create an empire.

the damage, the collateral damage, the side effects, the doom, the gloom, what has to disappear to create that empire is monumental.

Obviously my first thought of empire is I just resort to cinema. So the Godfather and I see the Sam Altman as Michael Corleone. then during this book, we'll get into the book, but I've drawn a mind map. I'm starting to draw a mind map of all the characters that we meet in Empire of AI because although

Sam Altman is the face of open AI. took a lot of egomaniacal psychopaths behind him with him to get... We're talking about empires, come on. You don't build an empire with a ⁓ passive, lethargic, empathetic personality, do you? These are all the billionaires and the trillionaires and the VCs and it's the same names.

Jeremy Gilbertson Wow, a strong language. Hot take, hot take.

The names behind him

Mark Fielding Paul Graham, Kevin Scott, Emmett Shear, Brett Taylor, Larry Summers, Reid Hoffman, Peter Thiel, Greg Brockman, Elon Musk. I mean, it's the same names cropping up everywhere, isn't it? LinkedIn, Airbnb, Nvidia, Facebook, Google.

Jeremy Gilbertson talk about the name ⁓ on the binding right here, Karen Hao. the reason why, and she sets this in the first chapter and kind of in through chapters, you know, one through four. And we're dealing with one through four today, part one in the book. She essentially was embedded in the OpenAI organization. she's been researching a lot of this stuff for a long, time.

And her perspective is really interesting because it catches it, what the intent was at the start and how we'll notice through this building of empire methodology that that intent has morphed nefariously perhaps.

Mark Fielding She spent the day with Brockman and Sutskever She spent a lot of time with the people in this book. Any quoted emails, documents or Slack messages come from copies or screenshots of those documents and correspondences or are exactly as they appear in lawsuits. In other words, anything which is quoted, she says she has read in official documents.

Trust

Jeremy Gilbertson Everything's about trust, man. You know, we talk about trust a lot on thinking on paper, but so let's dive right in. So a lot of the original intent for some of this stuff that, know, Sutskever and a bunch of these other leaders are talking about, there's a quote from him, early commitment to say, hey, you know, as we ramp this thing up, from an open AI perspective, as we ramp this thing up, if someone else gets ahead of us and someone else does a better job than us, we are just gonna stop and join forces and we're gonna make it all happen together for the benefit of humanity. But guess what? When capitalism comes in, guess what? When fundraising and expectations from that fundraising come in, the mission changes one of the questions that ran through my mind, how do we govern AI, What does that look like? And there's governing from the building of AI. So there's a lot of focus on, all right, how many chips are getting built? Who gets the chips?

Who has the money for the chips? Who can build the facilities to house the chips to let the chips do the things? And when are we hitting limits from that perspective? But then you have like, how do you govern the use of AI? And how do you deal with things like right now, you Meta's AI, this is in the news presently, but Meta's AI actually like flirting with kids or like talking about the idea, which is like, holy crap, like that's nuts, you know, and how do you manage that? How do you govern that? What do you think about when you say, how do you govern AI?

Mark Fielding I'm very skeptical about how, is it country-based? how do you create a interactive, interoperable system of governance or guardrails?

you have all of these competing teams, competing teams called nations who want to be the best at what they do. And that'd be like asking two of the best premier league soccer teams to cooperate. Right. Like as an analogy.

Do we just trust, trust, trust, trust, trust? we trust them? Reading this book, I don't trust them she slams at home how often at the beginning they speak about for the good of humanity, AGI, for the good of humanity, for the good of humanity and how that message just gets diluted and weaker and weaker and weaker as the opening chapters pass by and to the point where, okay, it becomes an afterthought. it isn't what's driving them.

By chapter five

Mark Fielding By chapter five, that isn't what's driving them. isn't for the good of humanity. It's for the good of them. It's for their egos, their, they want to be first. So how do you govern? and when they paying off Congress, when they're in Congress, when they are more powerful than the political bodies that might put in any global regulation, how do you do that? as the time passes and more and more models and

There's no data left, there's no human information left. When does it become too late? When does the question have to change?

Jeremy Gilbertson Mm. Yeah, yeah,

We're building something that's going to get so out of our realm to control that at some point it's just going to be there. wrote this earlier in a reference to kind of like the big bank situation where the big banks, we got to throw money at these big banks because they're too big to fail. I mean, we're going to get to a point where whoever's holding the reins of all of this compute and capability

Are we gonna get to a situation where we're like, shit, it's too big to fail, man. We gotta support it or help it. So let me read something here that I found really interesting in the prologue and on page 16 related to empires that is a great little level set. So empire seized and extracted resources that were not their own and exploited the labor of people. They subjugated to mine, cultivate, refine.

those resources for the empire's enrichment. They projected racist dehumanizing ideas of their own superiority and modernity to justify and even entice the conquered into accepting the invasion of sovereignty, theft and subjugation. she continues, I mean, very, very powerfully written stuff to kind of say, well, yeah, this.

there are some parallels to this. You when you can look back at history and go, man, how different is this exactly? And it puts your mind in a very, thoughtful space to kind of consider instead of like, man, look at all the cool shit I can do with this technology. It's kind of like, well, there's that. Don't get me wrong. There's really interesting stuff being done, but there's also like, whoa,

We definitely have to keep our eyes on stuff and what rules can be made and also who's making the rules and for what benefit.

Mark Fielding they didn't know they did it like we now, cause we parrot what big tech has told us about the advantages of AI, what it's going to do. They're not our thoughts for the most part. We've been fed those by the marketing, by the tech giants, by these voices who say, AI is going to do this, this and this. When they were building it, they had...

They had no idea. GPT that we've all become familiar with wasn't part of it. And they had all this company set up. They had all this money coming in and they didn't know what they were going to do with it. They didn't know what AI was for. And

Bill Gates

Mark Fielding Bill Gates was the man responsible for what happened because OpenAI needed more investment. After Elon Musk left, he took his investment with him. They needed more money. they were flirting with Microsoft. And in the end, Microsoft invested a billion dollars, but that rested on a useful tool and they didn't have one. And it was...

Bill Gates who said, want an AI that can summarise books for me in short. And they had this GPT model that was a bit very basic. They used that. They demonstrated that to Bill Gates and it was enough for him to sign off on the investment. And that was kind of the domino to where we are. when everyone's on social media, AI is great for this, this, this and this. That's not what it was for originally. They didn't know what it was for.

Jeremy Gilbertson kind of still, we still haven't figured out the, is it for in a lot of ways. And just a quick comment on the scale of this, everybody. This was listed in the prologue as well. OpenAI GPT-5 was recently released, but GPT-4 was 15,000 times larger than GPT-1.

And guess what? That was five years ago. You think things are moving a little faster than they were five years ago? Probably so.

The power struggles

Mark Fielding So one of the big power struggles in the book, cause every empire has a power struggle, doesn't it? Every, every empire is built on power struggles and this is no different. So the big power struggle between Elon Musk and Sam Altman et al is this idea of being an open source for the good of humanity, nonprofit organization. And that's how

It all started because they were fearful of what Google doing. They wanted to build the opposite to what Google doing, something good, something humane. What happened?

Jeremy Gilbertson Well, know, money got thrown into the mix and the drive to be the first and the best and the fastest and not even knowing what that end result target is. You they talked about being the first to get to AGI and PS and distributing the benefits to humanity. That's like a little subtitle or a subset to that first goal. But, without even understanding the full definition of AGI,

Mark Fielding GPU as well, like eventually they figured out that in order to stay ahead, to be the first, needed, well, they talk about Moore's Law and then OpenAI's Laws, don't they? And it was GPU and that insanely expensive and just as a nonprofit, Sam Altman and they all believed it just wasn't possible.

Was it really about safety?

Jeremy Gilbertson Well, let me ask you this. Do you think it was really truly the intent of these guys let's get like a, let's get a nonprofit like board together to do what's best for humanity. was that really the intent or was that like just a bit of a spin to, know, cause they're looking at Google, they're, they're seeing Google and they're like, man,

Was it we wanna protect everybody from them or we wanna beat them?

Mark Fielding Fortunately, I'm not in their heads, so I can't answer that question.

Jeremy Gilbertson We got Namsa Michelle commenting in the chat. She thinks their intentions were not good, but naive. And then once they saw what they could do and what they could get away with, off we go. Insert Homelander meme.

Mark Fielding He was very young, wasn't he? Maybe they had that childlike naivety of, we're going to, we're going to do this. And then the, that reality breaks down and people change.

Jeremy Gilbertson the way things scale, like the way technology scales, the way the world rewards the building of something is not tied to what's the best for humanity. It's tied to what kind of return can come back to an investment on building something that people are seeing as novel enough to spend money on.

Jeremy Gilbertson that's the internal mechanic of how this stuff works. And I think until you redefine the evaluation mechanic, don't think things will change in that regard.

The Brockman interview

Mark Fielding there's an interesting question in there that she poses about, has a technology or the benefits of a technology ever been successfully distributed? think she asks that to

Greg Brockman when she's actually in OpenAI and he, you said I'm very wishy washy. He was very wishy washy and he's the bloody.

Jeremy Gilbertson man, that interview was like, yeah, that interview was just like, you're saying, you're saying words, but there's no substance to the words. You're dancing. There's a, there's a comms person in your ear for sure. ⁓ so, so, NOMSA, ⁓ also had another comment about, governments can open doors that, that no dollar amount can. So that, jumps me to an interesting, interesting little discussion here

Mark Fielding Yeah, yeah, yeah, it was embarrassing.

The current AI paradigm

Jeremy Gilbertson here's a quote from the book, the current AI paradigm is also choking off alternative paths to AI development, So we're seeing the number of researchers not funded by big technology decreasing. And if these ideas aren't tied to this quick, short-term commercial benefit that Bill Gates was like, yeah, let's push a bunch of money into it. You they might not get the resources and guess what happens? The diversity of those ideas comes down and we end up with AI defined by systems that are...

Mark Fielding Yeah, there's some very stark numbers, isn't there, about how many researchers, have turned their back on AI for good for the money, because there's such vast amounts of money being thrown at it. I've got a question for you on that, she was speaking about Geoffrey Hinton and the other godfathers of AI, and part of their deal was that they could continue teaching and be professors at their respective universities. Should they be allowed to have these jobs at the tops of these companies and still be professors at universities? Should they be allowed to do that?

Jeremy Gilbertson That's a bit of cake and eat it too kind of thing, it definitely, it definitely tightens up the parameters of what research is

We need to figure out the things like social cohesion and global cooperation, because guess what? AI is jet fuel on whatever the hell we're gonna be doing. And if we continue to do the things that aren't tied to the benefit of our future, the benefit of us as humans, we're gonna go quickly. We're gonna move quickly to the spot where we would have ended up anyway. So chapter one, divine right. Talked about super intelligence, the paperclip factory.

example pops up Nick Bostrom I think is always thrown into these these kind of discussions AI alignment, Y Combinator, YC, the talks a lot about Sam Altman's political background, his upbringing, his mentors, Peter Thiel, Paul Graham, about how he saw economic value in Silicon Valley. But this was where the beginnings of the Michael Corleone race of power started because he was creating, curating part of his life to get to the top of the Silicon Valley network and his decisions and his success or lack thereof with Lopt his former company helped him work up and he seemed to be a bit of a genius. I mean, when you could.

positioning and navigating and inserting himself into all of these systems and connecting them to power opportunity for him and his business as an organization, pretty incredible navigation of that system out there. is, yeah.

Pushing Elon out of the nest

Mark Fielding He pushed Elon Musk out of the nest. That's how good he is. it, was it the quote from Paul Graham? If Sam Altman was dropped onto an island full of cannibals, you'd go back in three years and he'd be the king.

Jeremy Gilbertson Yeah, that was pretty crazy quote. ⁓ so back to the YC stuff, the, the idea was, told in the book that, YC start a Manhattan project for AI. Holy crap. Those are pretty crazy words to throw in together, Because we know what, what spun out of that, but then coupled with create the first general AI and use it for individual empowerment. Yeah. Again, we're talking individual empowerment. Everyone's hugging. It's great. We're going to do the best for the world until until the dollar signs and incentives start rolling in. But Sam's dad, the little story about Sam's dad in here I thought was really interesting. And I don't know if he is still around or not, but Sam's dad had a brilliant quote that, know, some of the things that I tell my kids, and I'm sure you tell your kids that, you know, Sam's dad told Sam, you always help people, even if you don't think you have time, figure it out. That's a pretty awesome quote from dad to kid, right? And I wonder,

I wonder how that's translating to kind of what he is doing and building and the scale of it and that sort of thing. Maybe we, I don't know, is Sam, is?

Mark Fielding was, he is very generous. There's a lot of evidence that he was generous. helped a lot of people in Silicon Valley was a great listener. He is a great listener, they say, but it all seems to fuel his meteoric rise to the top. In chapter two, she talks about he changes his image. Doesn't he? He got bigger. He got...

nicer clothes, he curated his personal brand a lot so he's yeah which he was too young

Jeremy Gilbertson Well, there was a potential run for a governor at some point too, right? Yeah, his manifesto, the United Slate, I actually tried to find it. It's no longer on the website. I don't know if anyone has ever read that or has a PDF of that hiding anywhere, but post it in the chat.

Mark Fielding a language model that might have read it.

Jeremy Gilbertson Ooh, fair enough. Actually, you know, that's really, man, that's really interesting. So all the all the stuff that gets yanked off the internet, could we could we find it in chat GPT or Claude or that's a really interesting piece of the puzzle.

Who makes him feel pressure?

Mark Fielding I'd like to know who makes him feel pressure. which people in his life is he accountable to? Which people in his life say no to him? how many of his entourages are yes men? Cause they're mostly men.

Jeremy Gilbertson I think that the societal pressure, not that society is calling him on the phone and be like, yo, hey, this is society. Yeah, please don't put us in a deep dark hole for the remainder of human existence, as a human, have to somehow feel the tinge of the impact of

Jeremy Gilbertson of what's cooking and how it could go.

so chapter one, me your quick, the best takeaway from chapter one that you have.

Mark Fielding really I learned that behind Sam Altman is a lot of other people. He might be the face, but there were other people behind him. They had no idea what they were doing. They just knew they had to do something. So they're pushing this technology forward without an end game in mind.

where we are is a result of circumstance rather than design. I learned I don't really like any of these people. They're not painted as nice people on the whole in the context of this.

Jeremy Gilbertson In the context of this, in the context of this story. Yeah. Yeah. Cause Mark, Mark has never broken bread with Sam Altman. I'm sure he's a charming individual.

Mark Fielding What I did notice as well was all of the dots that we're connecting. So there's so many callbacks to other books in the book club. Obviously Nexus, Yuval Noah Harari is called back a lot. There are callbacks to quantum supremacy. There are callbacks to numerous episodes of the show that were covering the same subjects. So that I enjoyed that. My favorite in chapter two was ImageNet, obviously, and the cats. Cats for the win.

Jeremy Gilbertson Cats for the win. for the win.

Mark Fielding I don't have my cat cup but yeah.

Jeremy Gilbertson Namsa. Namsa, thank you for all the engagement. This is, this is awesome. are we going to talk about the part with the board shenanigans? think Mark jumped into that a little bit before just the whole board shuffle and then Sam's out and then, you know, the big weekend on, everyone's at Sam's house trying to figure out how to get them back in. mean, there's a, there is an episodic similar to you. You reference white Lotus, very white Lotus-y kind of.

Mark Fielding stuff that went down there and it's, and it's very important as well because there's a one of the things that Elon Musk brings up is there's a lack of talent back in the early days and it would have seemed that all of the talent back in 2020 was at Sam Altman's house and they were Sam Altman's house because they believed in him and so he literally had an army of believers and any other AI there was maybe little pockets of AI talent elsewhere at Google, most notably, but they all ended up anyway with him and just the sheer power of his personality, his dream, his vision, whatever it was is they were all at his house. And essentially to the listeners who haven't read the book, Sam Altman was kicked out of open AI Yeah, he was kicked out.

Jeremy Gilbertson Jobsian, you know, with his boot from Apple at one point.

Why the board moved

Mark Fielding because the board didn't think that he was open enough. They thought he was, know, he's a little bit dodgy and all the employees rallied around him and eventually the board had to reinstate him as the head of open AI. It is incredible.

Jeremy Gilbertson That's such a great word.

did you think about the communication coming down from the, but what if you were an employee at open AI and you got this series of bullshit communication from the top about the future of the company and what's heading basically words without answers, like words without answers. And it's just like, I thought that was mind blowing how they handled the communication. Yeah.

Mark Fielding It was mind blowing, yeah.

Yeah. And as you know, Jeremy I don't do Slack, so I wouldn't be, I wouldn't have been employed anyway, but yeah, just almost infantile.

Jeremy Gilbertson Yeah, we're still trying to figure out communications as a species. Chapter two rolls through the societal impacts really, and everyone knows, listen, discrimination, whether racial, gender, class, whatever it is, is embedded in the internet.

And like you said, how did these models get trained? That's how they got trained. And what was really eye-opening, I'd heard about Timnit and her work and, you know, I'm actually going to be following that work a lot more closely, just kind of got reminded and highlighted about it. But ⁓ NeurIPS, the big AI conference,

Six black researchers

Jeremy Gilbertson The stat in 2016, six black researchers among 8,500 attendees, I don't care what you say. Like with that kind of representation and the models being trained by the internet with these inherent biases, that's not a model for something that's gonna serve humanity at scale, is it?

Mark Fielding No, it isn't. How do you benefit all of humanity when everything lacks global representation?

Jeremy Gilbertson what's that quote a long time ago? The future's here, it's not just evenly distributed. power and water are not evenly distributed

Power and water aren't evenly distributed. And if you start thinking about like the Maslow's, right, you think about Maslow's hierarchy, When you don't have water and you don't have power and maybe don't have access to a fuel source, you're in that level. You're not in the level of the top thinking about, well, how do I educate myself? How do I develop new products? How do I help my society grow? You're figuring out.

Mark Fielding when nothing is evenly distributed.

Jeremy Gilbertson how to have a sanitary experience, How to be fed and you know, it's a different thing. until like that level set gets bumped up a little bit, I think it's very difficult to evenly distribute something that is aimed to solve all of humanity's challenges.

Mark Fielding I'm just letting that rest just to people can, I think we should do that more thinking on paper when we should let thoughts and ideas and questions just sit for a moment. Somebody's on a walk, maybe give them a little bit of time to think about or to answer the question.

Jeremy Gilbertson to let that shake out of my brain too.

Chapter three, the nerve centre

Jeremy Gilbertson Let's fire into chapter three, the nerve center. So one quote I wanna throw out here, there's a definition of AGI that was put out.

So as open AI defined it, there's a quote from book page 77 a piece of software that has just as much sophistication, agility and creativity as the human mind to match or exceed its performance on most parentheses, economically valuable tasks. And I think that's a that's a key key piece economically valuable, because not all things that we do can or should be monetized,

Mark Fielding What is the purpose of technology, Jeremy, listeners? What is the purpose of technology? So another one we'll let sit And then I'll read what Brockman says. What do you think, Jeremy What is the purpose of technology for you?

Jeremy Gilbertson Extension of ourselves right technology is an extension of of us. It's it's like a microphone, And I was back Hey, guess what guys Jeremy's gonna use in a musical analogy, right? So I've got a brilliant microphone. I've got an awesome microphone preamp. That's technology I've got a computer that all of that stuff connects into the computer has plugins that makes my voice sound good bad weird strange But guess what if the melody and the lyric that I put together going into the microphone?

You know, that's, that's an extension of me. That's an extension of, of us, but the, but where technology is going, is it going to take away even that piece of the puzzle where now we have easy buttons for, Hey, make a melody. Hey, give me a lyric. Hey, generate this. Hey, generate that. ⁓ man, that's a big question mark.

Great comment here, technology is a tool like anything else. I completely agree with you. Augment what we're trying to do, maybe speed up some of the stuff that we'd rather not do and open our eyes to things that we would rather do.

Concentration of power

Mark Fielding Greg Brockman replied quickly, this is one thing that's really important. What is the purpose of technology? Why is it here? Why do we build it? We've been building technologies for thousands of years now, right? We do it because they serve people. AGI is not going to be different. Not the way that we envision it, not the way we want to build it, not the way we, they're my italics by the way, that we think it should play out.

Talk to me about the concentration of power, of the concentration of benefits, the concentration of resources and the concentration of rewards because AI technology, yeah, okay, it's a tool, but who's designing that tool and what is the purpose of that tool? Because a tool is all encompassing.

Jeremy Gilbertson I don't remember if it was Brockman or Sutskever talked about other technologies like fire, like, like, like cars, but fire, fire wasn't, wasn't coaxed into existence or built from zero. was something activated in nature, it's not like we.

Mark Fielding fire reference was a Freudian slip of the God complex in these people. what they're doing to the creation of fire? Come on. God complex.

Jeremy Gilbertson Ooh, wow, maybe so.

again, fire, that'd be like you and I in a room, we figure out how to make fire. We control the parameters of how fire is made, the ingredients needed to make fire happen, the access to turn on fire, you know, all built by the you and I, we And someone funded us to make fire. And now we're limiting the access of fire. we're actually, there are now laws in place,

Jeremy Gilbertson based on all of our work and lobbying that certain people have to come to us for a permit to light a fire. Yeah, all right, I'm off my soapbox. See, that point that the veil, that the shutters are drawn down. this is when OpenAI close things down, don't they? So we're not allowing journalists in, we're not gonna say what we're doing and they're gonna shut up shop telling.

Jeremy Gilbertson let's move into chapter four and try to land the plane here.

Cade Metz

Mark Fielding For me, the most interesting part was the part from Cade Metz, a longtime chronicler of AI, and he's speaking about the tradition of anthropop... I can't say this word. Anthropop, yeah. Well, yeah, and he continues, aided by Hollywood tales combining the idea of AI with age-old descriptions of human-made creation suddenly waking up.

Jeremy Gilbertson Namsa just talked about it, anthropomorphizing of text. calls it irksome, which I agree, but continue, Mark.

Mark Fielding AI developers speak often about how their software learns, reads or creates, just like humans. Not only has this fed into a sense that current AI technologies are far more capable than they are, it has become a rhetorical tool for companies to avoid legal responsibility.

what they've done to anthropomorphize it as they have is so smart because it puts us all on an equal footing it is like us, we are like it, therefore it's not a technology, it's an extension. Yeah. And it's very, very dangerous.

Jeremy Gilbertson We should accept what it says and yeah.

This has been a blast, Mark. I'm so glad we're doing this live again. A little bit more energy, a little more fun.

Thanks for listening. Mark, any closing thoughts?

The most important book of the decade

Mark Fielding So I'm calling it the most important book of the decade. I don't know if I go that far just yet, but it is very enlightening. I'm learning a lot. It's asking a lot of questions and it's removing some of the veneer, some of the public marketing veneer that this industry has. And I'm grateful for that. And I'm looking forward to reading part two and getting into the weeds of the empire of AI. So yeah, thank you. I think that's all there is to say.

Jeremy Gilbertson stay disruptive.

Mark Fielding Keep thinking on paper. Boom. Bye.