Karen Hao’s Empire of AI: The Human Cost of ChatGPT
By part two of Empire of AI, Jeremy asks Mark how his Godfather character map of the book's cast is looking. It's got a lot of blood on it.
In this episode · 21 chapters
- The exploitation of poverty is not new
- Ascension
- Can you scale and still do the right thing?
- Safety and ethics as a product blocker
- Hyman Rickover
- The minus to plus mistake
- Can Microsoft just pay them afterwards?
- Scale at all costs
- Not whistleblowers
- Timnit Gebru
- Where was the foresight?
- Classic Godfather
- Three steps
- Reinforcement learning, and quality of training
- Is that clickbait?
- Data annotation
- The case study from Kenya
- Fifteen thousand pieces a day
- Disaster capitalism
- Will this make you think twice?
- Cialdini, and pre-suasion
So I would almost argue that code is more valuable than text maybe in a way because code teaches something how to move through a logical process to make a decision.
Mark and Jeremy read part two of Karen Hao's Empire of AI. They follow OpenAI's push to scale, from GPT-2's sign-flip error to an API released without content moderation, and ask whether safety can survive when it is treated as a product blocker.
The final chapter turns to the data annotation workers in Kenya, Venezuela and Colombia who labelled disturbing content for low pay. The hosts discuss what that work did to them, and whether knowing it changes how anyone uses these tools.
Part 2 of Karen Hao's Empire of AI, read on the Thinking On Paper podcast book club, follows Sam Altman's OpenAI in its push to scale and commercialise, and the data annotation workers in Kenya and Venezuela behind models such as ChatGPT, some of whom labelled violent and sexual content to make them safer to use. Mark Fielding and Jeremy Gilbertson also cover Timnit Gebru's Stochastic Parrots paper.
What did Sam Altman write about building AGI?
Sam Altman wrote under a quote from Admiral Hyman Rickover that “building AGI that benefits humanity is perhaps the most important project in the world.” Mark reads it against what followed.
What did the Stochastic Parrots paper warn about?
On the Dangers of Stochastic Parrots, co-written by Timnit Gebru while at Google, set out four warnings: energy use, toxic scraped data, datasets too big to audit, and outputs people mistake for real meaning. Jeremy describes Google pulling the paper.
What did OpenAI admit about bias in GPT-3?
OpenAI admitted in its GPT-3 paper that the model entrenched stereotypes of gender, race and religion, and left fixing them to “the subject of future research”.
What did data annotation workers in Kenya have to review?
Data annotation meant labelling text, including the most violent and sexual content people produce. One worker in Kenya reviewed 15,000 pieces of explicit content a month.
How did Venezuela's economic collapse affect annotation workers?
During Venezuela’s economic collapse, workers on annotation platforms competed for tasks that vanished in “days to hours to seconds”. Hao writes: “Poverty doesn't just manifest as a lack of money or material wealth.”
Where Mark disagrees with the book
Mark criticises the book’s timeline for jumping between years and models, and doubts a claim that a calculator gets more testing than OpenAI’s models: “It's clickbait, isn't it?”
Links from this episode
Questions
What does part two of Empire of AI cover?
Part 2 of Karen Hao's Empire of AI, read on the Thinking On Paper podcast book club, follows Sam Altman's OpenAI in its push to scale and commercialise, and the data annotation workers in Kenya and Venezuela behind models such as ChatGPT, some of whom labelled violent and sexual content to make them safer to use.
Who are the data workers in Empire of AI?
The final chapter turns to the data annotation workers in Kenya, Venezuela and Colombia who labelled disturbing content for low pay.
What happened to safety as OpenAI pushed to scale?
They follow OpenAI's push to scale, from GPT-2's sign-flip error to an API released without content moderation, and ask whether safety can survive when it is treated as a product blocker.
Transcript
Mark Fielding Disruptors and curious minds book lovers. Welcome to the thinking on paper book club. I am Mark. This is Jeremy and we are reading Empire of AI by Katherine. Katherine, I said that last week. Karen, how we're on part two and it's about exploitation. It's about power dynamics. It's about employing hundreds of thousands of people for pennies to sift through some of the most grotesque, ugly, violent sexual content that's ever been created by humanity to feed the language models, to make them safe, to make them child-friendly, to make them what you use every day today. Jeremy, how do you feel reading some of the chapters in here, Venezuela?
Jeremy Gilbertson that, let's just make sure we kind of stay organized a little bit, but I definitely want to jump to that because it's the one thing that stuck out to me and stayed in my brain, man. It was like I read that chapter, was that chapter eight? Chapter nine, disaster capitalism, right? Chapter nine, I read that over the weekend. I was sitting on my porch normally, know, great setting, reading, enjoying the outside and...
Mark Fielding Yeah. Chapter nine.
Jeremy Gilbertson I was very troubled, very disturbed by... It's almost like we were talking a little bit. It's almost like when you watch a gnarly horror movie and your brain's scrambled and you feel very weird and uncomfortable and like unsettled and ungrounded. ⁓ That's the way I walked away from that chapter. took... I mean, I still haven't really shaken it off, but I I shook off like what I had earlier. But man, like this has been happening for years, though. This is not AI specific. This is like...
The exploitation of poverty is not new
Mark Fielding Yeah, the exploitation of poverty stricken peoples is not new. And yeah, you're right, it's chapter nine, we get ahead. But I think that the big part of part two of this book is that. I mean, in chapters six, seven, eight, we get more of the same really the ascension that the rise to power of Sam Altman and open AI, the scale, scale, scale we get.
the commercial AI applications, get ChatGPT 3, that API, how that worked, we get this, the paranoia of open AI, the lockdown mentality, they were fearful of their model weights being stolen, there's a lot of...
paranoia on the the rise up so we can go through that first for sure
Jeremy Gilbertson Yeah, but the mechanics of the rise, the rule of the game of the rise are to have the most compute, do it the fastest, push product out the quickest, matter the cost. That's the race and everyone's...
Mark Fielding Competitive advantage, competitive advantage, stay ahead, stay ahead, stay ahead at all costs,
Mark Fielding I'm at Walter Hell. Bloody hell.
Jeremy Gilbertson Mark is feeling very spicy today. We're ⁓ rolling into this, but I think there's a reason for the spice. So what chapter are we starting with? Mark, what do you want to roll with?
Ascension
Mark Fielding Let's start with Ascension. Let's just go through the kind of the philosophy to 10x everything and how... I've got a criticism just before we start with the book. For the book, which... Are you ready to hear this criticism? If you're in the comments, maybe you agree with this critique, maybe you don't. It's not a very important critique. It just makes life at book club more difficult to plan these discussions because...
Jeremy Gilbertson For me or the book? Okay, I'm open to it too. How you doing in the comments guys? Hit us up.
Mark Fielding the timeline jumps all over the place. It doesn't follow a coherent timeline from 2016, doesn't year by year, month by month. It jumps around. It jumps around not only from the year, so sometimes 2021, 2022, it goes back to the beginning, then it goes to lockdown, then it goes to 2024. And it also jumps between models of open AI, it's GPT-3, GPT-4, GPT-1. And so it's quite difficult to keep track of a logical...
timeline for the show. That's my critique. I don't know if you agree with that or not.
Jeremy Gilbertson I definitely sensed it bouncing back and forth and when I'm in one chapter it bounces back to stuff from a previous chapter to, and I'll actually add my notes to that previous chapter and there's a little bit of jumping back and forth. I didn't find it terribly jarring, but it definitely wasn't.
Mark Fielding not jarring for the reading experience and I'm all for Hollywood movies that jump around but just for outlining the show, like what are going to talk about? It makes it... Yeah. mean you've got to read 120 pages and...
Jeremy Gilbertson Oh, oh yeah, pure logistically planning. Yeah. Yeah. I can't believe, I can't believe, I can't believe Karen Howe didn't structure her book based on how we were going to run through it in book club. That's very disappointing. Um, all right. So sentence that jumped out at me, chapter six, Ascension quote from Sam Altman. We really built an empire. All right. So, so that's a, that kind of seeds the thing in 2019, his goal is to be number one.
Jeremy Gilbertson at a few things, technical results, compute with.
Mark Fielding speaking about YC Combinator when he said that.
Jeremy Gilbertson was he? Let me go back.
Mark Fielding Like, because that was the first mention like with YC Combinator, he had created an empire. I'm pretty sure that that quote was referring to that. Anyone in the chat if you can confirm or deny please.
Mark Fielding Don't correct yourself too soon.
Jeremy Gilbertson Yeah, yeah, yeah, let's see.
already pretty okay yeah he's definitely yeah he's definitely referring to ⁓ to YC in that YC in that but definitely an empire mindset dare we say
Mark Fielding Well, yeah, maybe it's even more telling that he was referring to empires for YC Combinator. And we'll get to that in a minute. There's lots of links, always links. So YC Combinator, when we get to the exploitation of the workers in Venezuela and Kenya, that is all around ⁓ a third party. I don't know, how would you call those companies like Outsourcing of Work, Scale AI, which was founded by Alexander Wang.
who was a YC Combinator startup and lived with Sam Altman during lockdown for a couple of months, but he ended up.
Can you scale and still do the right thing?
Jeremy Gilbertson Yep. Well, let's so in, in chapter, in ascension, let me, let me, I wrote some, I wrote some, a bunch of notes here in the, in the side as we, as we dove into this. And I think it was after I read it. So in the race to the top powered by a fear of losing the race makes, makes someone more susceptible to compromising values. If you stick to your guns, someone beats you becomes a validated shortcut and then gets more money. It's more GPUs.
gets more favors, can you do the right thing in scale wars? Is it possible? And I kind of had thought experiment there, right? And the flip side of that, you have safety being viewed as product prevention, My years in sales, we used to always call the financial department or the credit department sales prevention, right? Because you'd sell something and then, hey, can they finance it, right?
It's kind of the thing and anything that prevents 10x in this case is kind of bad. Anything that gets in the way of 10x, 10x or bust, at all costs kind of thing. So that's kind of the root of this and it keeps coming back to the rules of this game that they're playing are not for humanity in a way.
Safety and ethics as a product blocker
Mark Fielding Safety and ethics being a product blocker is just...
sad state of affairs but yeah you're right
Jeremy Gilbertson Well, there's a reference in, and I think we'll get to this in the later chapters, but there's a reference into when they were restructuring kind of, when, when, the Omidy siblings, divorced out and they were restructuring their research arm. One of their goals stated in the research paper was to understand neural networks and deep learning so we could figure out how to use the tools we have or figure out what the tools we have can do. So you're building and building and building and building. shit. What can these things do? Right.
Mark Fielding your question, you said, can you scale indefinitely and have the good of humanity at heart at the same time? No, of course you can't. There's a very interesting quote that Sam Altman, paints the, you know, get the reindeer waterfall quotes on your wall. You'd put these stupid, redundant, motivational quotes on your wall. And he's got this quote by
Hyman Rickover
Mark Fielding Hyman Rickover, an admiral in the US Navy, he's always an admiral in the US Navy with these people, they think they're doing something protecting, I don't know, the people. And the quote, blah, blah, blah, blah, the future of the human race, blah, blah, and Sam Altman wrote below this, building AGI that benefits humanity is perhaps the most important project in the world. We must put the mission ahead of any individual preferences. Low stakes things should be low drama.
so can save our high drama capacity for high stakes things. And it stinks because it's just a massive juxtaposition with what the reality turns out to be. It's like he's trying to convince himself.
with redundant quotes that what he is doing is for the good of humanity and yet we see it that it's just simply not the case is it? If it was for the good of humanity they would have guardrails in place before they wouldn't react after somebody kills themselves because they use that technology we got that one we have to sort that out after the event after the event after the event wait, know, so when GPT, I think it was GPT-2 first came out, they only released the 8 % version. So that was their safety protocol. They only released the 8 % version. Hey, friends and neighbors in the chat, I want you to rate Mark's spiciness today from zero, is bland as hell, 10 is fire shooting out of your face. So.
week, Jeremy, you said I was wishy washy and I thought, right.
Jeremy Gilbertson Way to take a stance, Mark, way to take a stance. ⁓ So a couple other things ⁓ noted in here, just kind of confirming what I call the rules of the game in this race. You know, the idea is, you know, make money to do more research, not do more research to make more money. I think that was a paraphrase of a Walt Disney quote in some form or fashion. I'm like, okay, yeah, you you bring, you bring steamboat Mickey into the house and maybe it seems more palatable, I guess, or I don't know. ⁓
The minus to plus mistake
Jeremy Gilbertson What did you think of the simple little minus to plus mistake? I think it was GPT, was that GPT-2 maybe? Yeah, it was GPT-2. So, know, so ⁓ in one of the processes, one of the update processes that the researcher put in place, and remember guys, like these are mathematical machines predicting the next word in the sentence.
Mark Fielding Remind me remind me what?
Jeremy Gilbertson Let's just keep coming back to that. So in the programming of this mathematical machine designed to predict the next word in a sentence, the researcher or someone pushing the update actually had a minus sign flipped to a plus sign in whatever code he put in. And basically all the output was like lewd and offensive and all of that. And imagine just like a flip of a certain switch or missing of it. Talk about not having your I's dotted and your T's crossed. ⁓
Jeremy Gilbertson That turned into a bit of a shit show, didn't it?
Mark Fielding Did that person get a pay reward or did they lose their job?
Jeremy Gilbertson That's a fair question. could be looked at both, especially if you're looking at it coming from the product prevention department or the grow at all cost department. Ben, what's up dude? It's been a long time. Great to see you in the chat. Ben gives you an eight spicy. So I feel that's good. I that's good. If you need to throttle up, we can.
Mark Fielding I was saying, room for more, room for more.
No, wait till we get to chapter nine, Ben. That's all I say.
Jeremy Gilbertson Alright, so what else? We're still in chapter six. We need to kind of keep this thing moving and grooving. Again, more references to the Manhattan Project sprinkled in here. ⁓ There's a lot of folks ⁓ that have different positions on this as AI is akin to the proliferation of nuclear weapons, right? The growth of nuclear weapons. We talked about the atoms for peace thing.
back in the Eisenhower days and having that awareness of this is a tool. Other people view this as a, this is gonna be a next version of how people interact with the internet, simplifying it. But yeah, Manhattan Project.
Mark Fielding after a few guests we've had, kind of leaned to the latter of those at the moment. And just for the listeners, there was a brief explanation of what was actually happening at OpenAI. had the GPT, GP3 API. There was a bit of the team, Emodi, weren't ready to release it, but Microsoft were pushing on them to get this, get it out there, start commercializing it. That was the start of the Emodi siblings kind of like, okay, we're going to go off.
start anthropic, that's what happened, they left. You also had the, again another serendipitous or accident was the scraping of Reddit which gave them all the coding capability that they hadn't really been planning on. was an afterthought, was an ⁓ accident because of the data they were scraping off Reddit and GitHub where they had all these coding. that became, there was actually some interesting emails.
from people in Open Eyes. maybe we should, maybe we should pay some of these people for all of this information that we're stealing from Reddit and GitHub and maybe Microsoft.
Jeremy Gilbertson Is it plagiarism? Is it plagiarism? I don't
Can Microsoft just pay them afterwards?
Mark Fielding What was the thing about maybe Microsoft can pay them after the event and Microsoft gave like, I don't know, 10 pence to charity or something?
Jeremy Gilbertson Yeah, check the box. Well, here listen, so let's put our corporate hats on for a minute here. ⁓ It's easy to be the independent guys on the outside going, man, you guys are doing the world right, right? But you also, you're scaling a company and you have the resources, you tied your boat to Microsoft, a bunch of money, a bunch of compute, hey, if the rules of the road,
Scale at all costs
Jeremy Gilbertson If the rules of this game we're playing are scale at all costs, you got Microsoft, got access to the biggest amount of chips, compute the whole nine yards. But then with that billion dollar investment... There you go.
Mark Fielding Well yeah, just don't tell us that humanity is at the centre. totally, with my corporate
Jeremy Gilbertson Because Microsoft's like, yo, I gave you all this stuff. What are you giving us? And hey, tapping my foot, yo, when's this return coming? So, but again, like you said, outward facing, we're saying, hey, we're designing something that's gonna benefit all of humanity. Yeah, are you really? Or are you trying to, you you can't do both. Like you said, it's very difficult. If you do both, something's gonna fail. ⁓
Mark Fielding But then of course, if Microsoft are dangling another two billion in front of them, which they were for the API to be a success. So of course, yeah, from a corporate perspective, onwards scale.
Jeremy Gilbertson a couple of other things in six and then we'll move on to seven. ⁓ This is where they start talking about the API for GPT-3, which is like, all right, let's get this thing out the door. Let's get as many people using it. The more people that use it, it creates this flywheel benefit for us. So let's push that product out there. ⁓ PS, the initial live version with this API didn't have any content moderation filtering. So.
⁓ Hey, we're still in the race. We're, we're abiding by the rules of the race, not the rules for benefit of humanity, but the rules of scale. So you mentioned code generation. All right. So this is really interesting. think I pulled this, there was, this is where the chapters bounce back and forth, but, ⁓ so code, this is really interesting. So code in and of itself. And there was a quote from the book is, is the most obvious and largest source that, that's, that presents encoded.
structured patterns of logic. So I would almost argue that code is more valuable than text maybe in a way because code teaches something how to move through a logical process to make a decision. Right? So that that's this big aha moment and Microsoft's like, yo, hey Sam, we've got GitHub. There's like billions of lines of code. Why don't you grab this stuff? ⁓ wait, wait, but
Jeremy Gilbertson But isn't that a free online community or an online community that people are doing it for the benefit of humanity and sharing their code to help other people code and create model. Aw, damn it, just grab it ⁓ and we can get really smart really fast. Scale at all costs, right? ⁓ Let's see, what'd you think of page 155, how to destabilize an organization? That felt.
Mark Fielding And so I'll tell you a little, let me just get the right window open. So that was Peter Willender. He was originally in robotics for OpenAI. Eventually he became the VP of product and he was talking about the internal discussions that were going on. I think this was maybe around summer 2020. I'm not sure again, it jumps around, but he says it was like the US intelligence manual on non-violent sabotage where
They were just using, you know, talk as much as you can, ask questions that don't make any sense, refuse to understand any of the logic, refuse to understand any question. It was just kind of this. Again, we spoke about last week, the petulance, the immature nature of this, the like the school ground one-upmanship. But there's actually a Jocko podcast where Jocko, if you're familiar with the Jocko podcast, he's the ex-military guy. He speaks to a lot of veterans, but he has one episode. It's like four hours long.
and he goes through that manual point by point and it's fascinating because he uses it. The outset is a military manual but he goes, okay, you're in your job, you don't like your job, do this and he goes through it point by point and it's really, really fascinating. So I recommend you and listeners find that out if you can because it's a very interesting manual.
Jeremy Gilbertson Wow, that's a good tie-in, man. That's a good nugget. That's a good nugget for book club. I never would have thought of that. Never would have thought Jocko Willink and the art of, or the empire of AI. ⁓
Mark Fielding Yeah, yeah, that's my job.
okay, just on that note, we're talking about an empire, military. A lot of these founders, a lot of these very famous CEOs and businessmen that we label as saints, they quote from military leaders. I mean, obviously the art of war plays a big role in everything, which is a military book that people use for business, but they've always quoted these, and I'm sorry, Jeremy, mostly American.
a carte blanche to do what they want. So there is a tie in I think between big business and the military which is why the Jocko podcast can be pretty interesting.
Jeremy Gilbertson Well, what does, what's, what, you know, one aspect of the military is there's a mission at the top and there are people in between the top and the very bottom guy that's participating in a very small aspect of that mission to cascade down and influence and rally everybody in lockstep towards that mission. So what does a company do? Very similar, very similar kind of thing. I mean, the, the outputs of a military campaign versus growing a company. I'm not saying, you know, the importance of one over another, but I'm saying the mechanics are likely pretty similar, which is why they're pulling from that stuff. ⁓ Okay, so we got through the... Yeah, so we go into chapter seven, science and captivity. This is where we start talking about carbon footprints of training models and Emma Strobel. ⁓
Mark Fielding Let's get serious. Let's get serious, Jeremy.
Jeremy Gilbertson ⁓ kind of came out with a, with a, with a kind of a reference that, you know, one model's training impact is the equivalent of nearly 5,000 flights from New York to San Francisco. Kind of the first time maybe someone started putting their hands up and going, yo, like what's, what's this stuff doing to the environment? I know it talks about a lot of compute PS. If you guys want to hear a really good episode about what AI is doing to the environment, Mark, tell them about enabled emissions.
Mark Fielding I can't go there, Jeremy. If my spice is eight, if I go there, my spice will be 25. Yeah, okay, I will. Day before yesterday, we spoke to two whistleblowers from Microsoft who had been spent... Well, we'll use it for the sake of this. And after 10 years in the sustainability departments of Microsoft, they realized that in fact, big tech was in bed, is in bed with big oil, using their technology to...
Not whistleblowers
Jeremy Gilbertson Not whistleblowers. They didn't like that term. They didn't like that term.
Mark Fielding exacerbate the search and exploitation, no it's not the right word, extraction of fossil fuels to a very very scary degree. It puts the data center carbon footprint in the shade. Check it out.
Jeremy Gilbertson It really does, I didn't know it, and I'm pretty deep in data center land. ⁓ Ben, nice seeing a couple of these great comments, you know, talking about ⁓ this concept of mission and individual and, you know, coupled with extreme pressure, there's a bit of God complex that comes out in that. You're absolutely right. ⁓ Yeah, it's.
like some God complex going on for sure. Hero, hero complex.
Jeremy Gilbertson All right. ⁓ Ooh, what you got on one six five.
Mark Fielding So if anyone in the chat has read on the dangers of stochastic parrots, can language models be too big? This was kind of the paper that set the ball rolling in terms of what could be the harmful effects of all of this. In short, it outlines four key warnings. As you mentioned, the energy impact, number one.
⁓ Second, the demand for data was growing so vast that companies were scraping whatever they could find on the internet, inadvertently capturing more toxic and abusive language as well as subtler racist and sexist references. Third, because such vast datasets were difficult to audit and scrutinise, it was extremely challenging to verify what was actually in them, making it harder to eradicate toxicity. So it's kind of this flywheel of toxicity, making it worse and worse. And we're also starting to...
screw up the environment. And finally, the model outputs were getting so good that people could easily mistake its statistically calculated outputs as language with real meaning and intent. This would make people prone not only to believing the text to be factual information, but also to consider the model a competent advisor, a trustworthy, confident and perhaps even something sentient.
Jeremy Gilbertson And that's been a theme we've been talking about for months and maybe years. ⁓ Maybe we ought to do a, I thought about it'd be interesting to do a pocket edition on the stochastic parrots, but the breakdown in the book was so great on the points of it. And how do you pronounce, Timnit, how do you pronounce the?
Timnit Gebru
Jeremy Gilbertson ⁓ of Timnit, one of the authors. ⁓
Mark Fielding Yeah, I've written that down.
Jeremy Gilbertson Anyway, these, you know, the, two authors of this one, one came, one's more of a linguistics gibbler. Yeah. So she was within, within, within Google, and assembling it research team to put this paper out, partnered with a linguistics professor, outlining all of these different things, getting ready as part of the part of what companies do release research, put it out there, take stances and.
Mark Fielding Jibru, yeah, Jibru. We spoke about her last week.
Jeremy Gilbertson Google ended up pulling the paper and not allowing her to publish it, ⁓ leading to a lot of discord. ⁓ yeah, that's kind of, ⁓ know, silencing the things that need to be said. ⁓ her ground didn't she and she fought and she fought and eventually...
Jeremy Gilbertson Yes, not, not with, ⁓ not tied in with, with Google though outside, right? A version of it. ⁓
Mark Fielding And just on that, there's just a quote here, OpenAO had simply admitted in its research paper describing the model that GPT-3 did indeed entrench stereotypes related to gender, race and religion, but the measures for mitigating them would have to be the subject of future research. And again and again we see this after the event, after the fuck up, after the toxicity, after everything that we're trying to eradicate, they go, we'll sort that out later or God, we should have sorted that out before.
Jeremy Gilbertson Well, hey, there's no, there's Mark rules of the game. There's no time for that. We're, buying GPUs. We're throwing shit. We're throwing, we're throwing as much data, no matter how shitty, how toxic or whatever it's data. We're chucking in, in this model. Cause more data, more chips gets us where we want to get us where we want to go. And you know what?
Where was the foresight?
Mark Fielding Where's the foresight? Where was the foresight? Where was it?
a military collateral damage, Jeremy. This is maybe their thought pattern. If they read enough military... Not propaganda, but whatever. Then eventually, collateral damage. Yes, some people have to suffer because we're saving humanity.
Jeremy Gilbertson Yep. Yep. So the tail end of this chapter, there's some reference of like, I thought this was pretty interesting. ⁓ Karen, how references, you know, big AI or big AI going big tobacco, you know, so, Hey, Hey, this, this, these cigarettes are great for you, dude. It's all good. We're not going to publish anything about that. They'll kill you in the next eight to 10 years. ⁓ but now it's all good because it looks really cool. Check out our ads. We're great. ⁓ so is the like the Aaron Brockovich, the beginnings of the Aaron Brockovich movement against big tobacco then? Yeah, against big AI.
Jeremy Gilbertson Yep. So conveniently after, ⁓ and this is where it bumps around cause we're going from GPT three to now they're referencing chat GPT, ⁓ right around chat GPT timeframe. Guess what open AI did according to the book, according to Karen's research, they stopped openly publishing research. So. Remember that member a long time ago, we heard that yeah, lock it down boys. Lock it down. Listen, listen, here's, ⁓
Mark Fielding What did they do, Jeremy?
Classic Godfather
Jeremy Gilbertson Remember earlier on it was like, if someone else gets where we all want to go before we do, hey, we'll all hug up. We'll all make this thing happen. Let's all do it together. But no, like you said, the drawbridge is, is closing, but guess what?
Mark Fielding classic, classic Godfather. The Corleones, right, lock it down. I don't want the family speaking to anybody. I don't want anybody outside the family coming into this room. I want no discussions beyond the Corleones. That is it. Lock it down.
Jeremy Gilbertson How's your Godfather character map looking these days?
Mark Fielding It's got a lot of blood on it.
Three steps
Jeremy Gilbertson Here's it. Here's it. So Ben great comments you're throwing in here. So he says that's what happened to Elon Tesla was saving humanity. So everything else is is excused. There's a piece of this puzzle that we could like maybe even sprinkle back to and and I don't want to get too too political and I'm just stating kind of what What what the history of this stuff lays out? So the the war on terror right back in the day at all costs. We got to go hunt people down
And we need access to all of your information and when there's a big issue we trade You know, we trade security for ⁓ the fear balance or the lack of fear balance, right? So there's a lot of parallels been great insight. ⁓ keep it coming dude And if you have any questions That you're if you're reading the book with us, ⁓ Questions that that are sparking your mind that you want to riff on pop them in the chat. All right, so
While they stopped openly publishing research, Stanford has a transparency tracker, which I think that's pretty cool. I didn't know about that. So that's a good little tool to see who's doing what. But the dawn of commerce, Mark, this is post-Omodei divorce, new research direction. Guess what? We want to have an aligned AI system with vastly more capabilities or vastly more capable. How do you do that?
Mark Fielding say there's three steps to that, Jeremy. I would say there's three steps. First, scaling GPT-3 by another 10x using a new supercomputer from Microsoft arriving in the third quarter with 18,000 Nvidia A100s, the newest, most powerful GPUs then existed. I think you need that first, then maybe doing more research to increase by 25. ⁓
Mark Fielding X OpenAI's compute efficiency. And third, would say would be improving the quantity and quality of training data. Quality of training data in part by tapping into user data and shifting the model towards the best parts of the data distribution with reinforcement learning from human feedback.
Jeremy Gilbertson and let's deploy as many models as possible as products, flywheel, flywheel, flywheel. ⁓ So yeah, so the one thing.
Mark Fielding getting the hang of it by this point, weren't they? They were like, yeah, we've got this, we've got this. There was a bit at the beginning, 2016, 2017, they weren't sure, know, Microsoft may be gonna invest a billion, but by this point, they're flying.
Jeremy Gilbertson So, aligned AI system, I mean, we talk a lot about AI alignment, you know, with human and humanity kind of stuff. It's hard to do when you're still operating in the scale, in the race to scale. Like, they're diametrically opposed, right? ⁓ I do agree from a technical standpoint that, you know, after investing billions in compute, maybe you can look at, you know, how to get them a little bit more efficient, right? Use what you got a little bit better, but... ⁓
Reinforcement learning, and quality of training
Jeremy Gilbertson The quality of training, this is when we start bringing in the RLHF, Reinforced Learning, human feedback stuff that we'll continue to talk about. ⁓ Quote, one of the things in the research ⁓ initiative, the new, we're gonna organize now that Dari Amadeus left, we're gonna reorganize. One of the goals, Mark, of my new research organization, we're gonna study the science of deep learning a little bit better.
so we can understand how what we're building works. We're gonna, let's do that. Does that sound like a good initiative? Can we catch up to the train?
Mark Fielding sounds like something I might have done a few years before that particular moment in time, but hey, I'm not them. I did like the quote, I use your thing, quote, scaling multimodal models meanwhile could potentially quicken the pace of improvements even further with enough breakthroughs the document said with remarkable definiteness we might actually reach AGI.
models feed into models, it quickens the process. So from a business perspective, again, makes sense.
Jeremy Gilbertson I have a question.
I have a question. have a question. What's AGI?
Mark Fielding That depends doesn't it Jeremy on whose definition of AGI it's like moving the goalposts It's this one week. It's this the next week. It seems to be becoming more attainable the more the definition is weakened. So, I don't know. AI, can do essentially everything that we do to a higher level.
Jeremy Gilbertson Yeah, I asked that a little sarcastically because the definition, like you said, is all over the place. there was a, so we talked about this a little bit of this research initiative, the scale, that sort of thing. ⁓ The secret sauce, this is a loosely captured quote, the secret sauce of open AI can be written on a grain of rice. One open AI researcher said a long time ago, and then there was like a reference to scale. So again, back to the rules of the road.
Is that clickbait?
Jeremy Gilbertson Let's see, I thought this was interesting. ⁓ Klopf, I think, had a reference in here about there's more testing that goes into calculators. Like when you make a calculator and you put it out, there's more software testing and a calculator than what OpenAI was doing. That caught me as like, yo.
Mark Fielding Yeah, that caught me as well, but I think that's for dramatic. It's clickbait, isn't it? It's not true. They're using that. It's just to, yeah, clickbait. It's not true. Surely it's not true. I didn't believe it.
Jeremy Gilbertson Is that clickbait? Is that clickbait?
Data annotation
Jeremy Gilbertson All right, so let's dive into the muck of this last chapter, I, like how disturbed I felt reading through this and what is kind of quantified or qualified rather by this industry as, you know, two words that seem very non-threatening. Oh, data annotation.
Data, and we're doing data annotation. Hey, I'm gonna hire a bunch of people. We're gonna do some data annotation. sounds kind of official. Database management sounds kind of similar, right? But what this actually is is looking at content, you know, whether it's text, video, pictures, whatever it is, and labeling that content. So, ⁓
Mark Fielding Most of this is written though, they don't get into the video yet, which is mostly written content for this.
Jeremy Gilbertson for this, so this was written content? All of it was written content?
Mark Fielding because then, was it scale AI that they mentioned? Basically, everybody in Venezuela and Colombia and Kenya was so disturbed by what they'd done with the written content that the idea of then doing that with video. But I guess you're right, I guess they did eventually as well.
The case study from Kenya
Jeremy Gilbertson Yeah, so imagine like, so one guy, there was a case study of a ⁓ guy in here. I definitely want to reference his name because I think it's important. ⁓ a few guys, there? There's that ⁓ Moffat O'Connay and Oscarina Veronica Fuentes Anaya from Colombia as well.
Jeremy Gilbertson Where there's one.
Yeah, so the one from Kenya, what was that? What was his name?
Mark Fielding M-m-mothat o-okin ye?
Fifteen thousand pieces a day
Jeremy Gilbertson Okay, so what he had to do on a daily basis is review 15,000 pieces of explicit, no, I'm sorry, per month. 15,000 pieces of explicit content per month. I'm let that sink in for a bit. What does that do to your mind? That was the most disturbing thing to me was what reviewing that kind of thing, even if it's text-based.
does to your mind over time and for him, it pulled him away from his family, it pulled him away from his wife, it pulled him away from a lot of things because I don't think many people could do that and come out on the other side unscathed. And by the way, they were getting paid pennies on the dollar to do some of this really dark work that...
Mark Fielding It wasn't just explicit content, it was the most explicit content that a human being could create, the most nefarious, evil human beings. It's talking about parents raping their children, it's about children being raped by animals, it's about the most grotesque.
things the human imagination caught with and on it every day thousands upon thousands and they mentioned it like he his his marriage breaks down and his wife leaves him because he becomes so inside himself that he just can't function in a relationship properly.
Disaster capitalism
Jeremy Gilbertson So I wrote a note underneath disaster capitalism and I put, I'll read it to because my handwriting's shit, but the opposite of clockwork orange. It's a conditioning the opposite of clockwork orange. So remember the character that had his eyes wide open and he had to watch a certain content to change his conditioning? Like this is changing mental conditioning. This is changing your your identity, your personhood, how you think when you got this spike just shit fired at you consistently, like over months and months and months months and months. And again, nobody, I would argue not many people can come out of that. Maybe nobody can come out of that unscathed.
Mark Fielding And what makes it worse is that the exploitation of it, they had no choice. Many of them thought they had no choice. They wanted to keep doing it at beginning because they wanted the money. They couldn't live without it. that's the, when you employ thousands of people in the poverty stricken global south, as they call it, you're literally holding them to ransom. Like you create this feeling of need.
And it's almost like the like, summer, like the first company was doing this named after this had the same name as summer's Twitter Twitter, bloody handle, and then scale AI. It's almost like they're saying they should be grateful.
Jeremy Gilbertson Yeah, and well, there's a whole she Karen how runs down a whole ⁓ kind of theory, I guess that that finding these crashing economies, these countries with crashing economies and going, man, we need to go here because these guys will do just about anything. And unfortunately, you get situations like Venezuela when where the economy is like crumbled, like the the the inflation was like, I mean,
10,000 million whatever the heck it was it was like it wasn't working. They couldn't get food They couldn't provide for people and it's like on one side these companies are like well, hey, we're doing them a solid but are you like do if you're do them a solid like Let's let's pay them for for what they're doing or I don't know man. It's it's it's praying it's praying on folks and that that was exploitation. Again, it's for you got to scale and quote for all the work that the workers did to help each other. The platform pitted them in competition. Projects were first come first serve. A task stuck around in queue only as long as it took for enough workers to claim it. This window between a task's arrival and its disappearance shrank over time from days to hours to seconds as more and more workers, including many Venezuelans in crisis, joined and vied for scraps of work. And then you have Fuente saying, despite how much stress and hair pulling it was causing, she couldn't imagine leaving the platform. like this ⁓ quote, Fuentes told me two truths I would see reflected again and again among other workers who would similarly come to this work amid economic devastation. The first was that even if she wanted to abandon the platform, there was little chance she could. Her story as a refugee, as a child of intergenerational instability, as someone suffering chronic illness was tragically ordinary among these workers. Poverty doesn't just manifest as a lack of money or material wealth. The workers taught me. It seeps into every dimension of a worker's life and accrues debt across it. It's horrifying.
Jeremy Gilbertson Wasn't there something in here, I may not have this actually correct, but I'm gonna talk through it and keep me in line and those in the chat keep me in line if you're reading along with us. When some of these workers started using automated tools to help them respond to things quicker, to help them automate the tasks in their job, they were identified and booted from the platform.
But the platform, the platform is like, what they're doing is building a version of that for people more fortunate to use. Which is like, that to me was like, dude, that's the whole, like, you can't win for losing kind of argument. It's like, you're gonna tell them they can't use tools like that, but they are helping build a tool that does essentially that. The, right?
Will this make you think twice?
Jeremy Gilbertson Man, what else in this last one? I know, this is pretty dark on this last little chapter, but it's important to highlight this piece of the puzzle. When you're using ChatGPT or you're using some of these tools, there are people behind the use of these tools that are getting paid nothing to do mind-altering stuff.
Jeremy Gilbertson in a bad way. So, wow, yeah. Will that get you thinking twice on using some of these tools? I mean, I don't think anyone's gonna not use an easy button because of that, but like, I don't know, can there be an accountability? Can there be a, I don't know.
Mark Fielding Well, you said recently that you'd been using Claude Moore. Will this change? I mean, if the last week of thinking on paper hasn't caused pause for concern.
I mean, if we don't do something, we can't really expect most of the population to do something like this.
It's pretty... Eye-opening is not the right word.
And it transcends that, it's transcendental, it's...
Cialdini, and pre-suasion
Jeremy Gilbertson Well, it goes back to, uh, I don't know if you've ever read any of Robert CLD needs books, um, pre pre-suasion. And there's a lot of psychology behind, behind helping someone. think there was a study in that book where there was an apartment complex in a, in a rough side of town where a woman was outside, literally getting stabbed and murdered and screaming for her life. And everyone's sitting in their apartments thinking someone else.
Mark Fielding Yeah, pre-suasion, is that him?
Jeremy Gilbertson someone else has got to be helping. Like I don't need to go out there. Someone else has got to be helping. And it's one of those things where we, we have to scale the help, you know, the, so on the flip side, we're let's balance, let's balance the scales. Right? So the scale on one side is just, you know, AGI or bust damn the torpedoes. And then, you know, on the other side, it's kind of like, how do we scale the, like what we're talking about? And the way you do that is like by people coming together in, and talking and then more people coming together and talking and more people coming together and talking. But it's a process like we were talking with the Enabled Admissions folks the other day. You know, these ex-Microsoft ⁓ super talented, you know, technical, but also ⁓ environmentally focused. It's a lonely road for them. Like they're coming out and they're doing the hard work of saying, yo, forget about data center footprint. You know, look at what, you know, tech firms are using AI to like basically do the opposite of all the stuff we're trying to do to save the environment and they're doing it at scale. So it's like, it's a lonely road, but like, hopefully as you walk through these towns, maybe another person walks with you and then another person walks with you. And hopefully we're creating some awareness in that because I, man, it's not a popular message from a business perspective, but it's an important one.
Mark Fielding No, it's not a popular message. Like
⁓ AI has a very happy smiley face, but behind that face there is a lot more that meets the eye. Do you think that this is the end of reinforcement learning from human feedback? We've had a couple of guests who've spoken about that AI will increasingly take on that role itself. Are these the darkest days?
of this? Or I don't know if you know the answer to that, maybe someone in the chat can shed some light on that. But yeah, have we seen the end of the darkest days or is the worst yet to come?
Jeremy Gilbertson Man, that's a question to ponder if anyone has any ideas. Just looking at some of the comments in the chat, Ben talking about his AI use, ⁓ had some preliminary conversations with, I'm not familiar with, Liberty AI. Liberty AI, it's a decentralized project, which could be pretty interesting. ⁓ Projects that are kind of walled off and built outside of things can set their own goals and do different things. So that's pretty cool.
⁓ Joe re, ⁓ talked about CLD, saw him live years ago and recently in a zoom meeting, man, I would love to be in a zoom meeting with Robert CLD. There's a lot, man. What, I wonder what his take on all this stuff would be. That would be. Mark. That's your, that's your mission. Mission. Go get Dr. CLD.
Mark Fielding I choose to accept it. Well, if Jory has been in a Zoom meeting with him, maybe he has his contact details and he can link us.
Jeremy Gilbertson That'd be powerful. Rich Tella popping in. Thanks for joining us too, man. Good to see you. Hope all is well. All right. So how do we let...
Mark Fielding I can't see his comment. What does he say?
Jeremy Gilbertson ⁓ He he says love you fellas we feel the love too man ⁓ He's yeah, he's got a bounce He's me he's meeting with it he's he's he's has a meeting and he's pretty sure it's with a real human So ⁓ let us know how that turns out. Let us know what your how you determine ⁓ carbon and silicon Mark, how do we land this plane, dude?
Mark Fielding Let's... he's got a bounce, okay, but rich.
Check out Rich Teller's work. Anybody who's in the chat looking at the chat, check out Rich Teller's work. He is the real deal.
Jeremy Gilbertson There you go, there you go.
Mark Fielding and a former guest.
Jeremy Gilbertson An early guest and he's almost an OG. It's almost an OG. All right. So we move into part three for our next, our next session. ⁓ those of you not listening live, if you have a question and you're reading this book, ⁓ you know, comments in, in the, in the, in the chat, as you rewatch this or send us a note, hello at thinking on paper. X, Y, Z, and we'll respond and have a, have a chat. We're, we're, we're learning as well. This is, this is basically open source.
Mark Fielding One of the first, yeah.
Jeremy Gilbertson Mark and I learning. So you're seeing all the weird stuff, all the silly questions, all the wrong comments, but we're trying to get to a process of synthesis.
Mark Fielding learning, Jeremy, real time ⁓ putting our reputations on the line, putting our thought patterns on the line. I like it.
Jeremy Gilbertson same yeah it feels good hope you guys are enjoying it as well ⁓ we have all of our stuff up at thinking on paper.xyz ⁓ Instagram LinkedIn YouTube all that stuff mark any any closing thoughts as we land the plane here
Mark Fielding I really, really, really can't recommend our latest show enough. I think it's really, really, really important. We spoke to Microsoft, ex-Microsoft employees about the story of big tech and big oil and how big tech is aiding and abetting the extraction of fossil fuels. And when you hear about net zero by 2030, when you hear about we've got to keep it under 1.5 degrees, when we hear about this shift to sustainable green energy, it is going in the wrong direction. awareness is the key. Yeah, we're digging ourselves a very, very, very big and dark hole, which if we don't start filling that hole very, very soon, it's gonna be too deep to get out of.
Jeremy Gilbertson Exponentially, exponentially in the wrong direction.
There you have it. Well, food for thought guys. Hope everyone has a great weekend. Thinking on paper.xyz. I'm Jeremy. This is Mark. Be curious. Stay disruptive.
Mark Fielding Don't have nightmares.




