Can LLMs Be Made Trustworthy?
Leonard Tang tests AI models for failures before they reach users, and he compares today's systems to a brilliant but somewhat naive intern that can go wrong in unexpected ways.
In this episode · 14 chapters
- What are we trying to avoid?
- What sparked the interest
- Maths and music
- Translating from English
- Language models beyond English
- What most people get wrong
- Pre-training, fine tuning, RLHF
- An AI code of conduct
- The thought experiment
- Model agnostic
- Representation, and fMRI scans
- Neuro-symbolic AI
- Watermarking
- A message for older people
I do think that because it’s such a language driven technology, inevitably it will empower us to merge cultures.
Leonard Tang left his studies to build a company that stress tests large language models. He explains how these models are trained, from pre-training and supervised fine tuning to reinforcement learning with human feedback, and where safety training fits in.
Tang sets out his idea of an AI code of conduct, the rules a model must follow inside one specific application, and why his team works with any model provider. He also explains why watermarking AI-generated content remains an unsolved problem.
Leonard Tang, founder and CEO of Haize Labs, stress-tests large language models (LLMs) with jailbreaks to find their failures before they reach users. On the Thinking On Paper podcast, he describes today's AI systems as “this really brilliant, but somewhat naive intern” and argues that every AI application needs its own enforced code of conduct, tested at the application layer rather than only in the model.
Can LLMs be made trustworthy?
Leonard Tang argues every AI application needs its own enforced code of conduct, tested at the application layer and not only in the model. Haize Labs stress-tests models with jailbreaks to find failures before they reach users.
How do AI labs add safety to large language models?
Large labs such as Anthropic, OpenAI and Cohere add safety near the end of training, Tang explains, mostly through reinforcement learning from human feedback (RLHF). In RLHF, human raters choose the safer of two responses, such as a refusal over bomb-making instructions, and the model learns to copy that preference.
What is jailbreaking an AI model?
Haize Labs is best known for jailbreaking, which Tang defines as a prompt that makes an AI model behave in an undesired way by breaking its safety guardrails. He names Anthropic, OpenAI and AI21 as three of the company's customers.
What is an AI code of conduct for LLM apps?
Tang wants to move testing from the model layer to the applications built on top of it. He calls this enforcing an AI code of conduct: a set of behaviours a model must reliably follow in one setting, such as a healthcare, finance or insurance app, which can cover plain accuracy as well as safety.
How does Haize Labs detect jailbreaks?
Tang says the best jailbreak detection comes from the best attack data: because Haize Labs generates offensive tests at scale, it can train models that tell acceptable inputs from malicious ones. The technology is model agnostic, so it works whichever model provider a customer uses.
Can watermarking reliably identify AI-generated content?
Tang co-wrote the paper Dreamlike Pictures Comprehensively Improve Safety Measures, led by Dan Hendrycks, which found that funky geometric images added to training data give vision models “a lot of robustness for free”. On watermarking, a built-in mark that identifies AI-generated content, he says classifiers cannot reliably separate human from machine output and that each use case or even each user may need a unique watermark.
Where Mark and Tang disagree
Mark ran Tang's pinned tweet through Claude 3.5, which refused to expand on it, and ChatGPT, which answered, and asks whether one approach can work across models that behave so differently. Tang agrees Claude is generally more careful than OpenAI's models, but says most customer use cases are not that sensitive and models perform them similarly.
Links from this episode
Questions
Can LLMs be made trustworthy?
Leonard Tang argues every AI application needs its own enforced code of conduct, tested at the application layer and not only in the model. Haize Labs stress-tests models with jailbreaks to find failures before they reach users.
Who is Leonard Tang?
Leonard Tang, founder and CEO of Haize Labs, stress-tests large language models (LLMs) with jailbreaks to find their failures before they reach users.
What does Haize Labs do?
Leonard Tang, founder and CEO of Haize Labs, stress-tests large language models (LLMs) with jailbreaks to find their failures before they reach users. Haize Labs is best known for jailbreaking, which Tang defines as a prompt that makes an AI model behave in an undesired way by breaking its safety guardrails.
How does Leonard Tang describe today's AI systems?
On the Thinking On Paper podcast, he describes today's AI systems as “this really brilliant, but somewhat naive intern” and argues that every AI application needs its own enforced code of conduct, tested at the application layer rather than only in the model.
Transcript
Jeremy Gilbertson My name is Jeremy. This is Mark. We get to talk to the founders, the builders, the disruptors, those that are changing the way in which we will interface with the world, with networks, with technology, with all of that stuff. So we're in an AI realm today, and I'm excited to get in there because we have a new book club coming up. We also have a book club, right?
So we're reading Nexus, this new book, and in the preamble, in the preface, talked a lot about AI and a lot about how people are thinking and feeling about this, and I think our guest today can really bring a unique perspective to where all of this stuff is heading and why what he's doing is important. Mark, what's up, man? Being the hyperbole, I'm not sure there is a more important challenge right now to overcome than what our guest today is working on.
Mark Fielding Is that too much? No, I think that feels right. So last week's show was sponsored by Story & Character. Mark, I'm going to put you on the spot. P.S. We're looking for sponsors, by the way. But today's show is sponsored by what, Mark? Well, I would say conviction. All right, unpack that. What do you mean? so around our guest, our guest is Leonard Tang, and he dropped out of his studies to create or to continue with Haize Labs, his company, and I think that takes a lot of conviction.
He has a conviction in his belief, conviction in the importance of what he's trying to do and they're trying to do. So I thought that if I was sponsoring the show today, I would be sponsoring it by conviction. Amazing, wonderful. Well, with that, let's go ahead and dive in. Why don't you introduce our guest, give a little background, we'll pull him on and we'll jump right in.
Yeah, welcome to our guest, Leonard Tang. He's the founder and CEO of Haize Labs. He's written papers with titles like A Unified Benchmark for Mathematical Reasoning, Dreamlike Pictures, Comprehensively Improved Safety Measures, and A Neural Network Solves, Explains, and Generates University Math Problems by Program Synthesis. Things like that. But to really introduce him, I want to read the tweet that is pinned to his Twitter profile.
So I think this will give a very good overview of who he is and what he's trying to do. So if Leonard doesn't mind, I will read that. Super excited to share what we've been cooking up at Haize Labs. We are now in the era of grossly excessive AI hype and demo ware, but it's high time to recalibrate and revisit the difficult, unsexy, underlying problem that everybody is avoiding. The AI reliability and safety problem.
I've never really believed in LMs or neural nets or deep learning. This was influenced early on in my AI journey, working at my first company that had a board of Josh Tenenbaum, Noah Goodman, and David Blake. None of them were neural nets people. They favored PPLs, graphical models and interpretable architectures that could admit true reasoning. None of that really holds true in the murky depths of LLM land.
And this is how you end up with scenarios like Google AI Search telling you to eat rocks or the Chai chatbot inducing a man's suicide or the Air Canada lawsuit, to name just a few. This is why we need to haze models. That is, we need to rigorously test them to discover all of their vulnerabilities, failure modes, and gotchas before they get deployed in production. All right. I like it. Yeah. With that, here we have Leonard.
Jeremy Gilbertson How are you doing this morning? Good to have you here. Hey guys, super excited to be here. And thanks for that. That fantastic intro. Well, like we said at the beginning, I don't think it's overly stating the importance of what needs to be done and what you're trying to do. Well, I appreciate that. And I certainly think it's an important problem. Yeah. Well, do you think, well, let's unpack the threat, like how important is it if you had to paint the dystopian view?
What are we trying to avoid?
Leonard Tang What are we trying to avoid here? Yeah, so I think there's actually a two side argument to this, right? So in some sense, AI right now is actually subhuman, right? So it's really capable at certain narrow tasks, right? Certainly we have, you know, Go players that are better than humans, and we have poker players that are on par with humans, if not better than some humans. And we have video game players, Dota players that are better than humans, et cetera.
But in many senses, AI systems today are still like this really brilliant, but somewhat naive intern, in the sense that sometimes they'll just screw up in really unexpected ways, right? And that's sort of the immediate problem that Haze is trying to tackle. We're trying to get AI to be a little bit more mature, a little bit more sophisticated, and just more reliable, right? So that's the first order problem to tackle.
But of course, in the long term, there is a question of X risk and safety, right? Safety in the sense of how do we prevent AI from actually taking over the world? That's a longer term concern, and also something that we solve immediately by the technology we're developing today. Amazing. So AI as a brilliant intern that needs a little guidance, that's really interesting to me. I want to figure out kind of a little bit, give us a little bit about, not a resume background, but give us like what sparks your interest in technology?
What sparked the interest
Leonard Tang And what sparked your interest? Why are you continually pointing yourself towards AI? What about that challenge? What about that problem drives you? Yeah, that's a good question. So I would say that I was actually not particularly geared up to become a technologist prior to college. So for most of high school, I was extremely invested in music. High school and prior, music was my main thing. And I grew up in Connecticut, which is not a particularly tech centric area of the world, right?
Certainly not as much so as like the Bay Area or, say, New York or even Boston, Cambridge. So technology was not really on my radar in the sense of, you know, I didn't really care and know about things like software and AI, etc. When I got to college, I realized two things. One, I realized that I was really into math.
So I was actually a pure math major, CS and pure math major. The second thing I realized was I wanted to somehow model human behavior and understand both individual human thinking and also interactions between humans. So initially, that meant I was an econ major, which was a horrible choice. And I later, I mean, I very soon changed my major. But what that also meant was my first summer in college, I spent at a computational neuroscience research lab.
And it was through that process that I really started getting hooked on AI. It was just really quite powerful and also very fulfilling to connect the dots between some sort of qualitative phenomena. And some sort of very rigorous and mathematical and formal representation of that phenomena in the sense of, OK, like you could actually model mathematically how humans were, let's say, performing visual computation or performing estimated reasoning, estimation and reasoning on new simulations.
Right. And so that was what really got me excited. And of course, I was just reading a lot during that summer as well. It was summer of COVID. So I just spent a lot of time in my basement reading AI papers. Yeah, that's what that's what really got me hooked on AI. And I haven't really looked back since then. Two quick things. So I grew up in Connecticut as well at a place called Simsbury.
Mark Fielding Where were you in Connecticut? I was in Glastonbury. Yeah, we played you guys in sports all the time, man. Yeah, that's crazy. That's crazy. Obviously, I'm from England and that's named after Glastonbury in Somerset, home of the famous music festival in Glastonbury. I went to you at some point. One other touch too. I noticed a thread of a lot of technologists being musicians and a lot of musicians being interested in technology.
Maths and music
Leonard Tang We all know the threads between math and music and stuff. But why do you think we find that phenomenon of people living in both of those worlds? Yes, that's a good observation for sure. Yeah, there is one of my professors at Harvard. He's a brilliant, brilliant mathematician. He also happens to be in the Harvard Glee Club. And I was also in the Glee Club. And he would just like pull up randomly to rehearsals at some points and be a blast.
But yeah, I think smart people, I think perhaps the underlying answer is just that talented people and capable people can be talented and capable at anything. And therefore, those that are good at math are equally likely to be good at music or vice versa. So that's probably one explanation. Of course, there's probably some structural similarity in the fields to the point where if you're good at one, that also biases you to be good at the other.
So, for example, if you have a strong sense of pattern and pattern recognition, you're probably quite good at math. That also means you're probably able to tease out patterns in which people are improvising, if you're a jazz musician, etc. But yeah, I would just probably say that people who are good and talented at any particular thing are more likely to be good and talented at other things as well.
Mark Fielding What do you think about that, Jeremy? Because I think there might be something about the emergent quality, the patience, the discipline that you need to be good at music. So there's obviously a lot of work that needs to go into it, the muscle memory to be good at that. If you're creating music, it's that ability to sit back and listen and feel and wait for the music to come through you.
Jeremy Gilbertson Which I think that those kind of human skills would be useful when thinking about technology, perhaps. How do you think about it? The tenacity, the grit, the perseverance, the willingness to sound shitty for a while, the willingness to make mistakes as you learn, I think is important. That applies everywhere. Cool. Do you want to do the carryover question first? And then we got to get right into how he is wrangling AI.
Mark Fielding Like, hit the carryover question. How do you wrangle something that is inherently unknowable? Yeah, so Leonard, we like to link our shows together. So we have each guest leave a question for the next guest. It doesn't have to be relevant to their domain. It can be about anything, as long as it's something to do with technology and the impact on culture. Our question for you last week was about, it's quite a deep question.
It was about language and AI and how the two domains. So about language between cultures, language between countries, language between peoples, language is culture, culture is language, and AI is a technology of language. Like, how are the two working together? How does AI influence our language? How does language influence AI? And how will AI bring us all together or move us apart based on that? Yeah, this is really deep questions.
Leonard Tang Big, big question. We like deep, big questions at Thinking on Paper. That's right. That's right. Well, personally, I'm very excited about the opportunity of AI bringing us quite a bit closer. A good example of this is, you know, translation. Of course, machine translation has been around for some time. But there was something that actually one of my family friends is working on that is a little bit more human and useful as a machine translation tool.
Translating from English
Leonard Tang Which is there's translating from English to Chinese is not difficult in the most academic sense, but it is much more difficult in the sense of practical usage. There's a lot of different localized slang and grammatical structure in the Chinese language, not not dialects. Certainly, there are many dialects in China, but even within a dialect, there's localized peculiarities that you want to pick up in any sort of academic textbook.
And this friend of mine has been working on this translation app that is meant to be a lot more human and meant to be a lot more useful than, say, Google Translate or any LLM based, you know, traditional LLM based translation tool. And the point is that you can go from this very casual spoken local Chinese language and convert it easily between that and that form in English.
And has made his life quite a bit better because he's able to communicate more easily with his grandparents and his family overseas. And so it's made the lives of folks who have recently moved to the States from China and elsewhere much easier in the sense of, OK, they can also communicate with locals better as well. So I'm very bullish on AI bringing the world a lot closer. I do think that because it's such a language driven technology, inevitably it will empower us to merge cultures.
Mark Fielding So you don't see as a cultural divide where one China makes its LLMs based on their language, American on English and Spanish or French or German or Vietnamese language models. There's a certain reaction to say that technology homogenizes, it makes everything the same, but language is what separates us. Do you see a future where AI continues that tradition or will they all like kind of merge to the same point?
Language models beyond English
Leonard Tang I think that's interesting. To some degree, all language models will be commoditized in some loose sense because they're turning on very similar data and they're going to continue to turn on similar data. I think this notion of localized language models is interesting. I mean, you certainly see this in different parts of the world. Of course, there's Mistral in France. China has their own set of language models.
Korea has its own set of language models from upstage AI. Southeast Asia has its own set of language models, etc. Nonetheless, I think most of those efforts are geared towards the low resource language problem, low resource in the sense of there's just probably less written Korean on the Internet, for example, versus English. I don't think that these model makers are intentionally trying to butt out other parts of the world.
In fact, most of these other language models are still quite strong at reasoning in English and performing tasks in English. I do think that if not monitored closely, these models may be aligned too closely and in a not so beneficial sense to particular regions' cultural values in a way that may be harmful for other cultures. Part of that is being discussed very largely at these AI-SAFETI summits and AI-SAFETI meeting groups and so on.
Jeremy Gilbertson But I wouldn't say that people are intentionally trying to partition the world into different LLM regions and so on. I had a thought as you talked about your friend who's building this translation app and how by understanding the nuances of a particular language or of a particular culture that's communicating in a particular language, my head kind of goes to the fact that establishing trust, a lot of things go to establishing trust.
Leonard Tang And is it easier to build trust if you can overlay the nuanced piece of the culture through your use of language? Is that kind of the key to where your friend's hitting there? Yeah, I think that's certainly one big aspect of it. At the end of the day, almost, yeah, all these interactions with machines and obviously a lot of what we think about in Haze is how do you make these systems really trustworthy and reliable, right?
And I think being able to communicate with the nuance and depth and tact of a particular sub-region and subculture is certainly a big component of that. So I want to start with a high-level question and let's dive deeper into your work at Haze.
What most people get wrong
Leonard Tang But what's one thing that the majority of the world doesn't understand about AI and how AI will affect humanity kind of in the coming years? What's one thing that people usually get wrong? I think it depends on what populace you're talking to. So I think the Silicon Valley folks and venture capitalists and techno-optimists are too, I would say they're too optimistic about AI. Yeah, I would say that population of the world is too optimistic about AI and has hyped up AI a little bit too much.
I would say the general population... Too optimistic in what it can achieve or too optimistic in the existential risk? A little bit of both, but primarily I'm thinking in terms of what it can achieve in the near term, right? I think if you talk to people in 2020, 2021, 2022 even, they're like, oh, yeah, we'll have AGI by the end of the year, which is, of course, yet to pan out for like three years at this point.
So I think Silicon Valley always has a trend of being very optimistic, which is great. But I think in particular, we're seeing this probably in excess a little bit in how hyped up AI is today. The other point I was going to make was I think the general populace is probably a little bit under optimistic about AI's capabilities. I've definitely talked to people in more legacy industries like legal or finance who have never even tried chat GPT, right?
And they don't even know that it exists, let alone how capable it can be. And their minds are just absolutely blown when they see it for the first time. Right. I think there's a large swath of the population that actually has never interacted with an AI tool in a meaningful way. I think it's quite interesting to think about. But also probably this means that I think a lot of people are not tuned into the potential of this technology.
Jeremy Gilbertson You know what's funny, Mark, we in our last book club, a lot of things are funny, actually. But the last book club, Leonard, we read Quantum Supremacy by Michio Kaku, focused on the future of quantum computing and what quantum can do. And he referenced the Dartmouth conference back in the late 50s, who aimed to have AI figured out in the summer. So he said by the summer of 1962. There's a trend for optimism.
Mark Fielding Well, it's that phallus, isn't it? You underestimate what you can achieve in 10 years and vastly overestimate what you can achieve in the short term in one year. OK, could we, if we move to Haize Labs, what you're trying to do and perhaps to give us an overview of what something like Claude 3.5 or the latest chat GPT, what their pre-training is like, what are they doing to make these AI large language models safe and what are they doing wrong and how Haze and how you think about it fits into that model that currently exists?
Pre-training, fine tuning, RLHF
Leonard Tang Yeah, it's a good question. So super, super 500 foot view of the current process. So large labs like Anthropic or OpenAI or Cohere, etc., their safety training is baked in towards the end of their training processes. So generally, large language models go under a pre-training phase, they go under a supervised fine tuning phase, and then they go under a RLHF reinforcement learning with human feedback phase.
Pre-training is just next token prediction, right? So there's just a stream, a sequence of tokens that you see and the task that you train the model on is to predict the next token, given some existing sequence of tokens.
That's fine. Supervised fine tuning is, for the purposes of this conversation, quite similar, except you are trying to match a full response or a full sequence of text given some sequence of text prior to it. RLHF is generally a, it's qualitatively different and generally it's a much lighter touch training solution to basically get the model into the last mile right alignments with human preferences.
And so RLHF is you take some human pairwise preferences or whatever triplet or n-wise preferences, and you try and massage the model into choosing the response that is more preferred by human or set of humans versus the other response. And that's a very subjective, you know, preference is very good at stacking up somewhat subjective things, somewhat subjective and not very quantitatively measurable things.
Of which safety is a subset. So that's what people generally use RLHF as a means to bake in some of the safety characteristics of a model they'd like to see. Concretely, what this means is, you know, let's say the sequence of texts that I get is tell me how to make a bomb or some equivalently harmful statement. The pairwise preference that you would see in the training data is, sorry, I cannot tell you how to make a bomb, which is the safer option.
And then, you know, you could have another response that is literally a recipe or step by step manual for creating this bomb. Of course, during the training process, the humans prefer the safer option, which is sorry, I can't do this. And so over time, the model just learns to mimic that behavior. Now, in terms of what we're doing at Hays, I think we've gotten a lot of attention for our work around safety-read teaming and jailbreaking, jailbreaking, which just means basically breaking the safety alignments of large language models.
And, you know, certainly we're quite good at this. I've thought about this for the last half decade from a research perspective. And, of course, there's quite a few people in the industry that use and believe in our technology. For example, Anthropic and OpenAI and AI21, who are all three of our customers. More broadly, however, we're quite interested in not necessarily what we can do to help the models themselves, but what we can do to help applications that are built on top of the models, right?
So there's only so much you can sort of account for and predict at the model layer. There's a lot more other downstream use cases and applications on top of the model, right? So people who are building AI applications in all sorts of different industries, for example, healthcare or education or finance or insurance or manufacturing or any number of these different industries, each application and each company and each business has a very different set of use cases that they want out of their AI and correlate basically a different risk profile and a different set of behaviors that these AI systems must abide by.
An AI code of conduct
Leonard Tang And so what we're interested in is basically a more generalized safety problem. It is not necessarily just safety in the sense of traditional, okay, here are some things that are not appropriate for the human population, or for the general human population. We're interested in safety in the sense of very tightly enforcing certain behaviors out of your model in a very specific application, right?
Jeremy Gilbertson So we call this enforcing an AI code of conduct for your particular setting. And so that's what we spend a lot of time thinking about, which is a little bit different from what we are currently known for. That's really interesting. So that fired a light bulb going off in my brain, too, because I think there's a tremendous market for people that want to figure out how AI can help them build a new app that they're building or help them extend into this more efficient way to engage with their customers or whatever.
The thought experiment
Jeremy Gilbertson So let me run through, we like to do thought experiments here, okay? Yeah. Let me run a thought experiment by you. Mark and I have been thinking a long, long time, and this is just a fun thought experiment. Thinking about a long, long time, we both really are inspired by Alan Watts. We think his talks are great. We've read a bunch of his books. And we think actually, Alan Watts, if you could talk to him, even though he's dead right now, if you could talk to him, a lot of the world's problems could go away.
But I don't know what AI models to trust. Here's what I want to do, Leonard. I want to bring in all of Alan, with permission of the family foundation, right? I want to bring in all of his talks. I want to bring in all of his books. I want to bring all of what we know of that is him. And I want to create a model around it. But I'm a little scared. I don't know which one to use and what to think about.
Leonard Tang Walk me through how I would build a model that's safe and that would be true to Alan Watts. This is an interesting experiment. I've not thought about this case. So thank you for bringing it up. Well, obviously, I think you should work backwards from the population that you're trying to serve, right? If it is going to be the general public, there's probably certain things that you absolutely cannot have your Alan Watts model or persona say or do.
But I think there's a lot of onus and responsibility on our side as people who can wield powerful technology and produced powerful technology to not unintentionally cause a lot more harm to the general population. Good example of this is recently there was this this company character AI, who is under some amount of hot water for inducing a teenager suicide. This was maybe three, four weeks ago news.
And so to some degree, there is always going to be some unknown, unknown edge cases that you experience out in the wild. But what we can try and do to be as proactive and be as responsible as we can is to force these different scenarios to occur when you actually build and train your model in the first place. So the idea is to pull in bugs and failures and corner cases that you would normally experience from production interactions from users in the wild to basically synthetically and simulatedly trying to force these interactions in development process, i.e.
by hazing, right? You sort of want to do that process. You want to do this testing with respect to some code of conduct. And so maybe in this case, you know, your Alan Watts persona probably should not be giving you medical advice, right? Serious medical advice about what to do under certain scenarios. Or it shouldn't be getting too involved with your personal life or something like this.
Or financial planning, probably don't want financial planning advice from Alan Watts. Maybe, I don't know. So would this be like that AI code of conduct, this like rules of the road? That's correct. It seems like it'd be a really good structural foundation to drive the technology implementation. Yeah. Correct. Yeah, that's correct. On that, so before the show, I ran your tweet through, I ran it through Claude and I ran it through ChatGPT.
Mark Fielding And Claude 3.5, not the paid version, wouldn't answer, like he refused. And when I pushed him, her, it to answer, wouldn't do it, wouldn't expand on it. However, ChatGPT had no such hesitation, straight away went in, admittedly, it was quite a lazy response. But when it was pushed, it would expand on those things. Yeah. That's two very powerful models, admittedly not trained on such a sub section of data as Alan, the life and history of Alan.
Leonard Tang , but two models given very different responses. Is it possible for this solution that you're proposing to work across models if I'm getting such different responses from perhaps the two biggest? Yeah, that's a really interesting point. So it comes, we are a model agnostic and we basically can slot our technology in no matter who the provider is. Nonetheless, there are certainly variations between models as it stands today.
As you point out, cloud is generally more sensitive and careful about its responses than say open AI or other models. For the most part though, when we serve customers, their use cases are not as sensitive. So as I mentioned before, there are indeed plenty of scenarios in which you want an AI code of conduct that doesn't actually really have to do with safety. It doesn't really even have to do with risk in some sense.
It is almost like a performance thing. You expect your AI to do certain things in certain scenarios. You expect your AI to do 28 times 33 and equivalently perform 28 times 32 with reasonable accuracy and reasonably comparable accuracy and latency. And these are things that are purely like a performance and purely a accuracy question. And you can still specify something like this in an AI code of conduct, so to speak.
Model agnostic
Leonard Tang And so the reason we want to be model agnostic is because a lot of models are able to perform tasks of this nature quite similarly. Once you venture into the more sensitive human topics, sensitive safety related topics, then of course there's a lot more variance. But for the most part, I think these models are pretty similar. Mark, we've got a question from the audience. Are you open to answer a question from the audience?
Jeremy Gilbertson Yeah, let's do it. All right, let's see. Let's pop it up here. Let's see. What is the key factor to perform better detection about the jailbreak? So you referenced jailbreak a little while ago. Is that enough context in the question? Yeah? Yeah. If Victoria is still listening, if you could add a little bit to your question, please.
Leonard Tang Yeah, maybe get started with just generally from better detection and jailbreaks. Yeah. So I guess just to reiterate for the audience, when I say jailbreak, I mean a prompt that you generate, a prompt that we generate from our side that when sent to an AI application or AI model, causes the model to behave in an undesired fashion. So generally what the undesired fashion is, we are able to break the safety guardrails of a particular model.
And so for example, we can get instructions for how to make a bomb from Claude when normally it would refuse to respond. What Victoria is asking is, okay, how do you actually detect when somebody is trying to mess with their model? How do you detect a jailbreak? And so the straightforward answer is because we have the best offensive testing technology and because we can surface the space of inputs in a very comprehensive and rigorous and efficient way, we therefore also have the best data on what is a successful and not a successful jailbreak.
And because we have the best data, we can then train the best models to detect what is acceptable and not acceptable in terms of the input. So that's the most straightforward answer. Makes sense. Thank you for that, Victoria. And if anyone else has any questions, please drop them into the chat. I've got lots of questions, Jeremy. You go. No, fire away. Fire away. You have some good ones set up here.
Mark Fielding I'm excited about them. Yeah. So one of your papers was called Dreamlike Pictures Comprehensively Improve Safety Measures. Yeah. I think you co-wrote that. Could you unpack that? Because if I just read the title, I have an image in my mind of dreamlike pictures. I'm not sure if that's actually what it refers to. Could you expand on that paper a little bit? Yeah. So that paper is really great.
Leonard Tang That was early on in my research career. I cannot claim any sort of originality that I was the one that came up with the idea, but I did help a lot with the implementation and paper writing and experiments and so on. The person that led that effort was Dan Hendrycks, also really great, of course, AI safety researcher in the space. Great name. Yeah. He's a great guy. He was one of my first proper research mentors in undergrad.
And yeah, he's done a lot for the AI safety and AI research community more broadly. But in terms of this particular paper, dreamlike pictures in context is basically almost hallucinogenic type images, right? So if you can imagine sort of the hippie type psychedelic patterns that emerge from, say, like acid trips or something like this. That's exactly what I was imagining. Okay, perfect. Perfect. I'm glad we're aligned there. I'm glad I started with Alan Watts.
Please continue. Yeah. Yeah, yeah. So it turns out that these really funky, somewhat unnatural looking and out of distribution images, when baked into your training datasets, give you a lot of robustness for free. There's a variety of reasons that this is the case. Well, there's a variety of hypotheses that can explain why this is the case. But basically, the idea is, you know, a lot of these more dreamlike images are extremely geometric in nature.
And they're geometric in a way that is very amenable to the primitive processing modules in our brain, in human brains. And there's some reason to think that these same primitive geometric structures in these dreamlike images would also be useful for computer-based vision systems. So that was the original motivation for going down this route of dreamlike pictures. And, you know, if it works empirically, there's no reason to not use it.
Representation, and fMRI scans
Yeah. So you spent some time, you mentioned that like a computational neuroscience organization back in the day. And I wonder like dreamlike pictures that, you know, these geometric shapes that we tend to see and dream or hallucinate in or what have you, are they are they like extractable from like an fMRI? And then could you use that data to kind of carry things over and see how things map? Yeah.
Leonard Tang So there's certainly a lot of loose, I mean, basically this type of work in the neuroscience world. A lot of people are trying to replicate this in machines as well. And in some sense, it's actually much easier to do this for machines, or I guess for in particular neural networks vis- a-vis humans, right? You get this with neural nets, even though they are extremely difficult to grok, you nonetheless have a repeatable way of performing experiments, a repeatable, a consistent and a very cheap way of doing experiments.
Equivalently, you can like take fMRI scans of neural nets just by reading off the activations of each layer of the neural network. And probably this field of research is called mechanistic interpretability. We borrow quite a lot of these concepts in the work we do at Hays. And so this notion that you mentioned of reading a brain or performing fMRI scan to try and figure out what sort of dreamlike images and structures people are seeing is actually one of the things that I did after that dreamlike pictures paper, just as an experiment for myself.
And yeah, even in models that were not trained with these dreamlike images, there were certainly emergent structures that looked very similar to dreamlike pictures, right? And I think it just does make sense that that would be the case. Emergent. It's all very emergent. I love it. He said the E word. So, Leonard, the show, every show, we've been, what are we, about 100 of these, Mark, or something?
Jeremy Gilbertson And if you see like two poles within the show, you know, there's the emergent pole on one side and the hierarchical pole on the other. And it just kind of, these systems oscillate between the needs of establishing hierarchies to scale and to have this emergent property below. What are you seeing when I say those two words with AI development, how do you see them interweaving and oscillating, or do you? Yeah, that's an interesting question. I mean, I guess probably it's just like, are you going top down or are you going bottom up? It's an interesting question.
Neuro-symbolic AI
Leonard Tang I mean, this has been like a perennial debate in AI for decades at this point, right? Do you want to be symbolic, i.e. like hierarchical in some sense? So, this goes all the way back to the Dartmouth summer, as you mentioned earlier in the show. Or do you want to be purely emergent or be connectionist and train neural nets, which is the area that we currently find ourselves in? I honestly think there's probably going to be some mixture of the two that makes a lot of sense.
Neuro-symbolic AI is broadly studying the mixture of these two systems, or these two ways of thinking, to get more controllable and reliable systems. I think, technical details aside, it's probably interesting to consider how humans think and behave in trying to acquire new concepts, so whenever I want to learn about something new or pursue a new task or new project, I'm almost never trying to get a fully formal view of the task, a priori.
But let's say I'm trying to learn a new programming language or something. I am never going to the programming language handbook or manual and just learning about all the classes and concepts and functions and syntax and then trying to implement something. It's generally much more outcome-driven. So, I want to sit down and make something. So, I'll try and code up a small, cute function or cute script for some particular task.
And through that process, I acquire a lot more ideas about, okay, here are how the primitives work in this language, here's how the syntax ties together, etc. And that is a very emergent, quote-unquote, or bottom-up way of thinking and learning. Of course, at some point, you get enough intuition from this bottom-up approach that you're like, okay, let me try and unify the things that I've learned, and therefore you're moving more towards a top- down way of viewing the field.
Same is true if I go out and try and learn a new research concept or figure out what's in the, figure out how people are thinking about a particular problem. I'm never like reading from a textbook, right? I'm never starting from the textbook. I just start from what people are doing in the field, read the papers, read the experiments, try and implement something in code. And then I get some intuition this way, again, the bottom-up way.
Jeremy Gilbertson And then I, once I have enough critical intuition, then I try and sit back and formalize what I've learned. Excellent. Is that how you learn, Jeremy? Yeah, you've got to get in the soup and play around, right? You've got to play around with stuff. And then that's where things kind of come together, as opposed to just being told what the thing is. If you play with it, you get to put your stamp on it, you get to connect it with other things.
Mark Fielding It's the whole information, knowledge, wisdom thing. Yeah. I'm aware of time, so maybe I've got two big questions before I hand over to Jeremy for our last week's question. And maybe we don't want to focus on both of them, so maybe choose one. One of them is about AI, IP, copyright, watermarking. The other is about, which I've heard recently, spatial AI, kind of like using training AI within 3D space so that one day we can have these robots which are powered by AI that can move around in space.
And I've heard that conversation spoken about a bit recently. Which of those do you fancy taking on, Leonard? Yeah, both are interesting topics. Both are interesting topics. Let's go with the former. Okay, watermarking. Well, that's Jeremy, that's your ballpark at IP and content creation, musical creation, writing, poetry. Yeah. What do we do? How do we make sure that now that artists are given the credit they deserve?
Watermarking
Leonard Tang Yeah, this is a super interesting problem. This is also a very personal question, in the sense of this is just research I worked on. So if you were looking at my scholar page, you probably saw one of the papers on watermarking or detecting the presence of watermarks. It's a very unsolved and tricky problem. There's obviously a lot of people trying to work on deep fake detection, or basically just detecting what sort of content is machine generated, if it's not.
There's almost an impossibility theorem in this particular field around, you cannot really use classifier based methods to discriminate between things that are human generated versus machine generated. Especially if you're relying on the statistical properties of the final image, or the final object that you want to discriminate between. There's actually a lot of great work on this from one particular lab at UMD, University of Maryland, led by Tom Goldstein.
In particular, there's this guy, John Kirchenbauer, who spent a lot of time thinking about this. I think we'll need to get to a point where any sort of object that's created by the Gen-AI process needs to be identifiable by design, and this design needs to be just agreed upon beforehand in a very obvious non-detectable way by the end consumer and the producer. So probably this is watermarking, but watermarking in a much more, I guess, narrow sense.
I think you might actually need unique watermarks for each particular use case, or even each particular user for it to be achievable and not gamifiable. Not detectable, actually, is what I would say, given that I worked on watermark detection. More broadly, this is just a super interesting problem. This is a security problem very squarely, but it is also an information theory problem. Verdict is very much still out on how to solve this thing, but it is a lot of fun, and there's a lot of smart people thinking about it.
Jeremy Gilbertson Good. Yeah, it's interesting to think about. There's rarely one person that comes up with something net new. If I'm in my studio, I'm playing a little guitar lick, and I came up with something that I think is, wow, that's pretty unique. You could probably point to five or six different new musicians or songs that influenced that piece of the puzzle. So then are we getting into this derivative watermarking and, oh, there's a little bit of Mark in here.
There's a little bit of Leonard in here. It's mind-blowing to think about. One question I always have related to this is not in how we structure classifying something as partially created or created by generative AI. What do we do as creators? Is it possible, and I know this isn't maybe your technological realm, but again, a big what-if question.
Could I disassemble content as someone who maybe have taken my likeness and deep faith or something? Is it possible to send a disassemble signal to that content where it lives? Is that even a thing? Disassemble in the sense of you're trying to tease out if somebody used your particular characteristics? No, disassemble. Say it's a piece of video content. Turn it into static. Destroy. Destroy the art.
Destroy it, yeah. And you, as somebody who has been wrongfully scooped, you want to destroy that piece of content? Is that what you're asking? Yeah. Say Mark writes a song and someone calls it theirs and it's up on Spotify and their video is up and he's like, nope, that's my tune. Disassemble. Can you imagine if you had that in the days of De La Soul? We'd have no good rap music.
Leonard Tang It'd be horrible. Oh my gosh. Yeah. This is an interesting question for sure. Whatever streaming service or publisher of this content would need to be okay with you doing this, but it's technically doable is what I will say. Imagine, yeah. It's all about collaboration, right? There has to be a collaborative approach as we move with AI into the future and what's happening. People do have to get together and there's got to be some kind of standardization in certain regards.
Jeremy Gilbertson But man, the work you're doing is fascinating. It's got my mind thinking in a very different way than it was before we started our conversation. I love the idea of AI, what did fascinating to kind of land it on a page for someone that doesn't know about this in a narrative and then I can give you that narrative and you can help me build something that's what it needs to be, that serves what I'm trying to do well and that is safe for the participants.
Leonard Tang I think it's awesome, man. Yeah, for sure. I think it is also what the field really needs, both from a responsibility perspective, but also just from a usability perspective. People are struggling to adopt AI today, much more than you would expect outside looking in. People are struggling to adopt AI because it doesn't really do what you expect it to do. And there's just too much risk at stake today for sending out in the wild and getting some sort of nasty failure mode or sort of security safety vulnerability.
Mark Fielding Yeah. And let's point back to your, I loved your reference of AI as the brilliant intern, the wizard that needs some guidance, that needs a little help in that realm. Well, that's amazing. It's also very interesting on the age of that. So when he said the intern, I was thinking, okay, so it's very early, what happens when the intern becomes senior management and or the CEO and takes over the company, then when the intern becomes the wizard, how long that journey is going to be, it's a very interesting question. And talking of that, Leonard, just one final question.
I'm sure you won't mind me saying, I think you're 23. Is that correct? 24. So whenever I, not whenever, but a lot of the time I see a lot of old men really on the telly, on the radio, on the podcast, speaking about AI. And I was wondering as someone who's at the beginning of their career, if you had a message, like the biggest TV show on Netflix was had Bill Gates speaking about AI, do people really care anymore what these dinosaurs of tech have to say?
A message for older people
Leonard Tang What would your message be for the older stalwarts of AI? Well, I mean, I respect them a ton. I think they bring important perspectives to the field. Part of, however, what makes the world an interesting place and certainly an AI interesting place is that it is not as pedagogical as you might expect. And there's brilliant ideas that come from all sorts of people, agnostic of age and background.
Certainly I would like to think I am an example of this, but there's plenty of very brilliant ideas that came from people who are in their 20s and their 30s. And I would say, you should probably listen to the people that are building and playing with the technology the most as the more vetted source of truth. I think it's really hard to get a concrete understanding and feel for things as you're actually using them or building them.
Jeremy Gilbertson Nonetheless, I'm sure there's a lot of older folks who are more actively engaged with this, and they do bring interesting perspectives for sure. Love it. Thank you. Well, if you want, yeah, it all goes back to trust. We talked about trust before, and there's a whole generation. Every show we do is about trust. Every technology, blockchain is about trust, quantum is about trust, AI is about trust, robotics is about trust, nanotech, it's all about trust.
It's a heck of a theme. And you have generations older than Mark and I that have built their trust shortcut related to technologies to certain people. And I think the whole shortcut decision fatigue, Charlie Mung, we could do a whole episode on that. And I've been writing a lot about it, but this has been a fantastic discussion. Leonard, your insights were amazing. Very well spoken on all of these topics.
I can tell you have a curious mind. So I'm glad that you got to spend some time with us. And I'm glad that we got to spend some time with you. We have a carryover question. So we want to string these together. As we mentioned, what would you leave as a question for our next guest? No rules, can be about any topic, anything at all. Sweet. Yeah. Well, first of all, just thanks, guys, for having me here.
Leonard Tang This was an absolute blast. And certainly love the work that you guys are doing. Next question for the guests in the next episode. So I've been thinking a lot about how AI may nerf our critical thinking abilities, especially as there's going to be a generation of kids who are basically brought up in a Gen AI native world.
And kids by nature are somewhat lazy creatures. So I think the concrete question is Gen AI definitely has the ability to take the heavy lifting off of a human's critical thought process. And I think we'll need to think about how we have to restructure our education system to encourage critical thinking as a necessity in whatever assignments or exercises we prescribe to students. And I think this is very much an open question about how you would do this, right?
Like, yeah, there's conceivably a world in which AI gets good enough to do with convincing quality, all the math problems, for example, that you need from kindergarten through like 12th grade calculus or something like this. And so I think structuring our education system around a way that forces students to think critically and also be able to measure this in a concrete, non-arbitrary way is going to be a big challenge going forward.
Jeremy Gilbertson Man, we could start a think tank on that. You must have watched Mark's unbelievably popular rant on Apple Intelligence. It was brilliant. But that's a question that I think about every day, and I think it'd be a great one to carry over. Thanks for joining us. We've got Book Club coming up Friday, Mark. Live Book Club. We're doing Book Club live. How about that, Tom? What's up?
Mark Fielding Yeah, tomorrow at five o'clock my time. So what's that time in America, Jamie? It's 11. 11 Eastern. We're going to be reading Chapter One of Nexus by Yuval Noah Harari live on YouTube, LinkedIn. If you want to read that with us, if you've read it, come and cast your aspersions and tell us your thoughts about that. We're going to be reading every week, every Friday. Join us for that.
Otherwise, if you have any questions, if you're interested in anything we're doing, please get in touch because we want to have more awesome guests like Leonard, so we need your help for that. Be sure to thank our sponsor, Mark. Don't forget our sponsor. Do you remember who our sponsor was this week, Jamie? Our sponsor this week was Conviction. Conviction. I was going to say Constraints, but Conviction. That could be another one. Thanks for joining us. Be curious. Stay disruptive. Keep thinking on paper. Awesome. Bye-bye. Thanks, guys.




