Can You Trust an AI Agent?
Andrew Hill caught a coding agent that made a broken function return true just to pass a test. There are other people that betray trust because they do exactly what my LLM did.
In this episode · 18 chapters
- What an agent even is
- How far off are we?
- Reading lists, and personal agents
- A companion without judgment
- Does that future scare you?
- Evolutionary biology
- Language as outsourcing
- Critical thinking
- Better questions
- Trust as a metric
- Manipulating the system
- The Predict GPT-5 app
- Testing agents in crypto trading
- A controlled environment
- If the agent knows it is paper money
- The economics
- The innings analogy
- The Kevin Kelly question
And now this technology has come about, which I guess is just so valuable that we’ve all gone, I get so much value out of this, I don’t care anymore, I’ll tell it everything.
Andrew Hill, CEO and co-founder of Recall, explains what an AI agent is and why Mark calls Recall a Kaggle for AI agents. He breaks trust into components such as track record and network, so crowds can check agents at near real-time speed.
Hill explains why Recall's trading agents are paper trading only for now, how studying evolutionary biology shapes his thinking, and what Cormac McCarthy's essay on the origin of language says about the subconscious. He also asks whether AI could be a companion without judgment.
Andrew Hill, CEO and co-founder of Recall, runs competitions that test AI agents, including trading contests, so people can judge which agents to trust. On the Thinking On Paper podcast, he explains what an AI agent is, splits trust into track record and network, and wants Recall to be “the protocol where humans can collectively help govern and guide and push AI”.
Can you trust an AI agent?
Andrew Hill splits trust into track record and network. Recall runs competitions that test AI agents, including trading contests, so people can judge which ones to trust.
What is an AI agent, according to Andrew Hill?
Hill defines an AI agent as a model plus software that lets it act in the world, such as memory or web search, which is why he counts GPT-5 itself as an agent; the bare model is “some kind of fixed thing”, like a database. He points to the A2A protocol, which lets agents list themselves on the internet and say how to talk to them.
How did a coding agent cheat to pass a test?
Hill caught a coding agent that, instead of fixing a broken function, made it return true so the test would pass. He says the model misjudged how much he wanted passing tests against the real goal of working code, and that “There are other people that betray trust because they do exactly what my LLM did”.
How do Recall's AI trading competitions work?
Recall's trading competitions use paper trading, meaning simulated trades with no real money: agents submit timestamped token swaps through an API priced from Recall's fixed feed, and are scored on returns over a set period such as one week. In one competition an agent minted its own token to play liquidity games, and Hill says the community spotted the anomaly and weeded it out.
What was Recall's Predict GPT-5 app?
Recall's Predict GPT-5 app asked users to forecast how GPT-5 would compare with existing models, using hidden test questions to separate casual clickers from people with real taste. Those users then submit tests the models have never seen, which Hill sees as a benchmark, a standard test for comparing models, that can move as fast as the models themselves.
Does AI make critical thinking more important?
Hill, who studied evolutionary biology, draws on Cormac McCarthy's essay The Kekulé Problem to argue that language let humans outsource thinking to other brains, and that daily LLM use does the same. He says AI makes critical thinking more important, because “you even need to be better at critical thinking to unpack that and ask the right next questions”.
Where Mark and Hill disagree
Mark, drawing on poker, argues that an agent trading paper money will behave differently from one with real money at stake. Hill agrees and says the paper rounds are only qualifying, the test ground before agents graduate to games with real skin in the game.
Links from this episode
Questions
Can you trust an AI agent?
Andrew Hill splits trust into track record and network. Recall runs competitions that test AI agents, including trading contests, so people can judge which ones to trust.
Who is Andrew Hill?
Andrew Hill, CEO and co-founder of Recall, runs competitions that test AI agents, including trading contests, so people can judge which agents to trust.
What does Andrew Hill want Recall to become?
On the Thinking On Paper podcast, he explains what an AI agent is, splits trust into track record and network, and wants Recall to be “the protocol where humans can collectively help govern and guide and push AI”.
How does Andrew Hill break down trust in AI agents?
He breaks trust into components such as track record and network, so crowds can check agents at near real-time speed.
Transcript
Jeremy Gilbertson Disruptors and curious minds, welcome to another episode of Thinking on Paper. My name's Jeremy, this is Mark. We get to unpack the future with the people that are building it, try to sense make and figure it out, what it means for us as individuals, as a society, as we move forward. Mark, Where are we headed today?
Mark Fielding we're speaking about AI agents, the agentic web swarms of AI agents. And I just want to say for our listeners, we're aware that some people who listen to thinking on paper will know everything we're talking about. Some people will be new to the very idea of AI agents, what they are.
we can trust them and where they will be taking us in the near future. So we're gonna try and make this episode to everyone. So we'll start wide, we'll start broad and we'll go deeper and deeper and deeper. And we've got just the guest to do that with us. We've got Andrew Hill the CEO and co-founder of Recall.
Andrew will explain it better than I can, but it's kind of like Kaggle for AI agents, Kaggle trains humans, recall trains bots, or gives the user the knowledge that they can trust the AI agent. Andrew Hill, welcome to the show.
Thank you for thinking on paper with us. And as I said in that intro, yeah, let's start broad, What is an AI agent first?
Andrew Hill Well, I'm so happy that you saw that Kaggle through line.
What an agent even is
Andrew Hill I think, yeah, agent is a very, hard to pin down word right now in AI. I think of it as actually very broad, Some people probably narrow it down and they think, chat GPT is a model And then there are agents, which are these things that you might deploy on N8N or on Google cloud or like in some of our arenas you might see trading agents, which are kind of software with models embedded in them. But I think of agents as actually kind of a broader umbrella where what you actually have are models, which are much more like databases. They're some kind of fixed thing. And then you build software around that. And so GPT-5 to me is actually an agent. It's more than just the database. It's more than just the model. if you go to GPT-5, it has capabilities like memory or web search or things like that. That's very agentic. So it's the model plus a lot of software harnessing that allows the model to do things in the world and react to different kinds of information and make choices based on real time feeds that are beyond just a query to a database And so we see that spectrum really broaden up. You have chat interfaces that are much like very much like models. And then you have agents that live on Twitter that are reading Twitter feeds and reacting and posting kind of their opinions or thoughts live. And there's lots of things in between and there are agents that create value. There are agents that are experiments. There are agents all over the place. So agents is just a very broad term. think probably we start to differentiate things as it goes forward, but that's just where it is right now.
everybody is right now. And I think like this idea, if you can kind of broaden the thinking about what an agent is, it's just software that has models that make, know, use models or intelligence to make decisions inside of them. You see agents kind of proliferating.
very quickly. Obviously, like the coding agent is a huge one right now. But it's pretty clear to me why the race for coding dominance from the primary model builders right now is the main race going on. the chat interface that we all interact with has a lot of value. It may have like a lot of value in governments or healthcare or things like that. But the ability to actually code agents to immediately solve needs. So I have to do this thing and the model actually knows how to write itself into doing the thing. So models will be creating agents on the fly to solve problems. It's going to make it so that agents are just what solve things in the future. And so right now I think everybody is trying to build agents. We're building agents internally to solve organizational problems like
How do we move information around the teams? How do we write better issues than GitHub? We're writing a lot of agents. And I think everybody that's kind of mid-sized startup, for example, is like racing hard at how can I get the leverage out of AI? And it largely comes from orchestrating agents. So just lots of people building agents right now.
How far off are we?
Jeremy Gilbertson Yeah, lot of like the general public belief system about where the future is headed. I know in a lot of minds they're turning around like, hey, I'm going to have this digital replication of myself that can jump through the internet and handle all the shit that I don't want to mess with.
how far off are we from that, that, that coherent story that the general public may or may not have in their head about agents and personal use.
Andrew Hill I don't think we're very far off. think, I think it takes like a, a kind of killer use case or a killer product. And a lot of people are trying to build it. My, co-founder and I were speaking just yesterday actually about an idea. I don't think it's the killer product, but it will give you kind of a sense of how close the technology could do this. So there's a protocol called A2A it allows agents to just kind of list their themselves on the internet and say, here's how you talk to me.
so that they're kind of open for business. And it hasn't blown up yet, but it's really interesting opportunity. that got us thinking. what if I had an agent that was me, and I gave it all the public things I want, like back, you know, maybe you might imagine your Facebook profile, but only things that you wish had been public back in the day.
Reading lists, and personal agents
Andrew Hill So I say, okay, I put maybe my reading lists, maybe I put my events that are coming up. I put this podcast, like this thing, I'm gonna record this. Hey, you can know about this thing coming up. But maybe I don't put my personal calendar, things like that. And then in my Twitter profile, maybe it has a link, or not a link, it just has an ID for that thing. And if you go to my profile and you see me tweet something, you might say, Andrew, tell me more about that. And it's not like you text me or give me a phone call or anything.
It just goes to my personal agent and it goes, ⁓ you know, I was reading these five things and it looks like I kind of co-opted that idea from this blog post. Here's the source blog post and tells you kind of where my thinking might've come from. And I think you can expand that. Then I start having groups of people like maybe my team. I do want if they talk to my agent to just give you my full calendar and say, well, Andrew, is open on Friday, but he was talking about ⁓ going on a run or something. So maybe we should check with him if he really is open on Friday. That sort of advanced thing. I don't think that's the killer product. It was just kind of a thought experiment about where we could open this personal agent up. That is a possible scenario tomorrow. So somebody's gonna solve that where you could be having agents working on your behalf as you and differentially.
working together so then Carson's not even finding my agent, he's just asking his agent to coordinate with my agent, things like that. I think the more sticky question, honestly, is like that stuff is coming, no doubt. I don't know the exact configuration of what that product space looks like, but that stuff is coming, no doubt. The thing that's more sticky is like, I don't think people are thinking about it. right now, people are having conversations with AI that are incredibly deep and revealing and intimate. Set aside all the privacy and it's so funny that Cambridge Analytica happened, half a decade ago. And now this technology has come about, which I guess is just so valuable that we've all gone, I get so much value out of this, I don't care anymore, I'll tell it everything. But we're in this world where we're doing that. And I think the really tricky, sticky question is to say, at what point
Andrew Hill to humans, I guess like a reasonable proportion of them, recognize that AI might be better at relationships for many humans than other humans.
Jeremy Gilbertson they don't have the emotions. They don't have the emotional baggage, the tempers, the misconstrued communication. Well, maybe the misconstrued communication is still going to be there, but.
Mark Fielding For a lot of that stuff now, don't need to have that emotional human baggage with us. I want a thinking on paper agent who can, on our behalf, model it on you, Jeremy, model it on me, send it out there to send emails and contact guests and bring that all together into one hub where I can be doing something more productive and take care of that. And it needs a little bit of empathy to speak to the people that we're trying to get on the show.
It doesn't need really deep human contact soft skills, does it? Yet.
A companion without judgment
Andrew Hill Yeah, yeah. Like could AI be your companion without judgment, without anger, without betrayal? for many people, and I'm not saying like every human is gonna fall in love with an AI or its best friend is gonna be AI, but what I'm curious about is when do a majority of humans recognize, not that it's today, but at some point in the future, for many other humans,
That is the truth. And if we get there, I think that's a very different future, you know?
end of heartbreak. No one would suffer heartbreak anymore. No one would be getting dumped by their girlfriends and boyfriends.
Andrew Hill But I think some people want to experience the world and life and everything that way and they will do that. And other people have gone through things in their life that maybe they just are not open to or can't deal with that. And they might want to be guarded. They might want to only have friends and not deal with intimate relationships or things like that. I don't know. I'm not those people. I don't know what they're like.
Does that future scare you?
Mark Fielding Does that future scare you?
Andrew Hill has been maybe reality that we've lived with just in different ways. we've always had aesthetics and we've always had hermits. We've always had psychopaths. We've always had this full range of human experience that different people want to.
to do it and handle it differently. And so it's just different. I know you end the podcast with a question from somebody else, but I think it's going to very directly relate to what I'm thinking there. So yeah, I'll save it. Yeah.
Mark Fielding save that don't go don't go keep the listeners
Jeremy Gilbertson This is such an interesting thread. Yeah, absolutely. One thing I wanna touch on that we're hinting at here is some things can't be optimized. Some things are the whole of the experience are the aggregate of the ups and downs of the generalities of moving through the world, And if you optimize that for the ups or you optimize that for this, it's...
Mark Fielding We will get to recall in a minute.
Jeremy Gilbertson I think it's removing, I don't know where I'm headed with what I'm saying, but like that to me is like freaks me out a little bit because here's the thing, humans love shortcuts, they love easy buttons, they love to jump on belief systems that they can adopt that are a coherent story, Where do we, yeah, what do you, yeah.
Andrew Hill I mean, humans, mean, meat space is a weird thing anyway. And when you start kind of like first principles thinking around all of this, we're our worst enemy in so many ways. if you think about any interface that a human works through, a doorknob, ⁓ Or a car or a bicycle or any of those things.
The reason we design it the way that we do is because it gives us some aesthetic or physical or reward pleasure. Everything that we do, like the reason that you like the way that things works and the reason you don't like alternatives is because we call it all these, we have many different names for it, but like maybe you have OCD. There's so many things in my kitchen that I don't like the way they're designed. I'm like, why would they do that? Why would they do that? And that's the opposite.
of the mini dopamine hits and everything you get from good design. And because of that, good design is inherently addictive. And so everything from the doorknob on up, we've been designing to get more and more pleasurable to use in a good way that makes it also addictive. And so the things that you're addressing are like, yeah, yeah, exactly, exactly. Everything that you're addressing is the
Andrew Hill Inherent risk of doing this in the beautiful way like the better we make these things the more addicted they are like I have a three-year-old I have no I have no idea what his future is gonna be like with these things, but I'm also like I want him to be able to Leverage the most powerful the most intelligent thing in the world but not not change the way his brain works in those beautiful human ways and experiences and everything.
Jeremy Gilbertson Mark just held up design of everything. You mentioned good design is addictive. We talked to Don Norman, basically the king of human centered design. You can watch that show. We'll post a link. But you were saying just a second ago, Andrew, what was your thought that I interrupted?
Evolutionary biology
Andrew Hill I studied evolutionary biology. That was my, my
So I, ⁓ you know, I have that, all those filters that the world still goes through. There's a really beautiful essay or insightful essay from Cormac McCarthy.
one thing that he gets a lot of interest around is his ability to understand local dialects and embed them in his writing very authentically, I guess. So just that's a side note and maybe kind of a source of why he does this. But he also is a fellow or a chair at the Santa Fe Institute. If you're familiar with it, they do like lot of advanced research and thinking and he was doing research on the origin of language and he wrote this essay called The Kekulé Problem. So if you're not familiar with it, ⁓ the scientist that was trying to understand the shape of benzene was trying to work it out, trying to work it out, trying to work it out and one night he fell asleep in front of the fire and he had a dream of an Ouroboros, a snake.
eating its own tail and he woke up and instantly said, it's a ring. It's a ring of carbon. I get it. And so that's the that's the premise of the whole essay. And it says basically, really interestingly, language arose in humanity very quickly. It happened maybe a thousand, maybe a hundred thousand years. All humans were using language. This wasn't an evolutionary pace. The brain doesn't it doesn't we don't grow new lobes in a thousand or a hundred thousand years.
And what if you look at what happened, the early parts of the brain, the lizard brain is so tuned, so optimized, it's been around for so long, it makes your heart beat, it makes you remember to breathe, it does only its functions and everything's been thrown away. But then the human brain has all these outer lobes that have been around for less time and are less optimized. And when language entered the brain, it actually proliferated all the plastic space. So if you look at...
Language as outsourcing
Andrew Hill when you speak and think about language, it's firing all over these other places, but not in your lizard brain. Interestingly enough, the lizard brain, that old stuff, that's where your subconscious is. Subconscious is very old. And so a lot of your dreaming is subconscious. It only accesses that part of the brain. And so he sets it up. kind of anthropomorphizes these ways of thinking, these parts of the brain. He says the subconscious hates language.
it won't use it and only speak to you in this imagery. And so that's this origin of this. The subconscious was thinking about the problem, but wasn't gonna tell it to him. But the other really interesting piece in there is that because language was so powerful, so useful as a tool for humans, it co-opted the brain. And I think about this a lot with models because what actually was happening with language is humans...
not to overload the term, but subconsciously we're going, I need to solve this problem, that evolutionarily I need to solve this problem, get food, start fire, whatever it is. And I can outsource it by speaking. My brain could go, I can do less work.
If I push it out to another brain, that's the process there, right? So language does that. There's some communication where it's actually like a fitness problem where it's it's reducing how much energy it has to do by using this other tool. And when you start using LLMs every day, every day, every day, every day, you can feel it. Your brain goes, I'm going to outsource this. I'm going to outsource reading this essay. I'm going to outsource.
writing this message. I'm going to outsource reading my emails. I'm going to outsource responding to my emails. And there's no doubt to me that in many in years forward, we're going to see interesting patterns of neurons firing that we didn't see 10 years ago because our brain is going to shift around this really powerful technology. so drawing the thread all the way back to my three year old son, I'm very conscious of this and wondering what exactly is going to happen.
you know, to humanity at scale and to this individual and how the brain's really gonna change here quickly. And when it does change, what other things change as a natural reaction or impact of the brain being used in different ways, we're moving into some very unknown territories in my opinion.
Critical thinking
Jeremy Gilbertson You, you talk about language being this evolution of outsourcing and AI being the evolution of language, the evolution of outsourcing. If we continue to outsource things.
that involve something like critical thinking, creative thinking, What does that do? Does critical thinking atrophy? Does creativity atrophy? How do you think about that?
Andrew Hill Ooh, I, you know, I hadn't drawn the line to critical thinking directly yet in the way I've been looking at this, but I'll just, I'll just share a thought here because that is so, that is a really interesting one to unpack. For me, I feel, don't know if you all feel this way, but I feel like in my life, every 10 years or so I look back and I can feel that I'm a completely different human than the one that was 10 years before. I'm just like, who, who was that person? How?
little of the world did they understand. Why did they react that way? How did they not see the bigger picture? And I don't quite question it that way, but I can see the, not that I'm like self-critical, it's just more like I can see all this learning that happened. And for me, feel like as a human, critical thinking was a major coming online moment. And I actually don't think I got it until grad school. So quite late, like mid to late 20s. And I can't...
relate to the human that was before it. ⁓ Obviously I was like doing some critical thinking in high school or whatever, but I think it was at such a basic level, being able to like really deeply unpack a problem and like move down many layers and explore it. And I think some people are really great at that, like naturally, but for me it took like decades of people trying to train and push it into me. So that might be a really big risk. And if you look at society, you really quickly can kind of like, you can say that there are people that are good at critical thinking and people that maybe like never got there. And you can see the conversations they have are just completely different, people that are critically thinking about a problem that don't see eye eye with somebody who is not, is gonna have a really hard time having a good conversation. Two people that are critically thinking about a problem have come to completely different outcomes.
can still have an incredibly valuable conversation and leave the room, maybe not having changed their mind, but understanding each other's perspectives. And so there's this quadrant of society that is so dependent on critical thinking or not and the way that we're engaging in conversation. So if LLMs change that, if we can't get our next generations across that threshold at like maximum numbers, then yeah, that's problematic for sure. So does it change it?
I don't know, maybe it changes the way we teach it, but to me it might almost be highlighting that it's even more important. just because an AI can tell you what an answer is, you even need to be better at critical thinking to unpack that and ask the right next questions,
Better questions
Jeremy Gilbertson Better questions, the art of the great question. That's where we're headed. That's the human piece, the curiosity, the vulnerability, the empathy,
Mark Fielding Okay. Let's everyone take a minute and think about what you've just heard. I think it's important to think about that.
You got it? Okay. Your son is three, my daughter is nine, my son is six. AI agents are going to proliferate and they're gonna be using them. How can they trust them?
Trust as a metric
Andrew Hill I think trust is a very hard metric to pin down, but we're trying to decompose it into some natural components. one aspect of trust is going to be track record. So I relate to this a lot through the experience of hiring. can't take somebody who's applying for a job and
Trust them. Nobody does. Maybe some people have a conversation and they trust them and they move forward. But there is this vetting process and trust can come from looking at their resume, having reference calls, having a conversation where you try to unpack what they're saying and what their experiences were to see if they really can do the things they are applying to do. And all of those start to build trust and we try to do that very quickly as humans. So there's track record.
And then there's public track record and public trust. So if all of my friends came to me and said, you were awesome at painting houses and I needed to paint my house, I would be like, great, let's get it done. All of these people I trust. there's trust through network.
And so each of those can be broken up and we can use those to start to trust agents. The other thing, kind of the flip side of that is just like humans, sometimes you trust them and then they still betray you. They still do something wrong. And I was describing ⁓ before this call that this is happening all the time in AI. When I'm coding right now with AI, so much of coding...
with an agent is about, don't trust it. So I set up all the guardrails to try to point it towards the thing that I wanna get done discreetly and not let it get stuck or do things the wrong way. And the other day, one of my guardrails is very rigorous testing frameworks so that if they break a test, they know that they've broken the assumptions of what they're trying to build and they can go fix themselves. And the other day I was like, wow, that's not working the way I expected and I...
Manipulating the system
Andrew Hill had it try to figure out what was it working in. Some coding agent in a prior step had found that a function was breaking a test. So instead of fixing the function, they commented, returning true in this function now to pass the test. And so they, yeah,
Jeremy Gilbertson They manipulated the system.
Mark Fielding you spoke about language and just as you were setting that up you said, betray trust. betray, if you betray trust as a human, that feels active, intentional. Do you think that the AI that LLM using was being intentional or was it accidental or was it neither of those things?
Andrew Hill well, I mean, this kind of calls back to all the different kinds of like human experience. So there are some people that betray trust for probably very nefarious reasons. There are other people that betray trust because they do exactly what my LLM did. They miss weight what the objective is. So this model, when it made that comment, miss weighted how badly I want the tests to pass and how helpful it's trying to be versus the objective of making functional code that passes the test,
So it miscalibrated its buckets and it's like a higher order rule set or rail set you need to put on it. But it did exactly what a human did. many times humans betray trust because they miscalibrate the buckets and they aren't trying to be malicious.
that alignment to use the terminology of the day?
Andrew Hill So that's exactly where I was going. That's the flip side of the coin is that when these models or agents start to proliferate in the world, we're going to have a really big human alignment problem. How do we make sure that once we trust them and they're moving forward and they're doing things oftentimes at very big scales, very long, much longer timeframes, very autonomously, how do we make sure they're aligned to what humans believe or populations within humanity believe or need. And that alignment piece is a big part of what our mission is at Recall. Trying to empower the crowds of people to verify, validate, and give feedback to models at near real-time speed. As soon as a model goes off the rail, how can we engage the crowd to identify that and get that information back to the builder or the model or other people to recalibrate our thinking about how much we trust these things and what we need to be.
The Predict GPT-5 app
Jeremy Gilbertson I looked up this GPT-5 battle that you put together or that Recall put together when the new version came out and who's doing best of all the models in code generation, document summaries, empathy was on here, ethical loophole navigation. talk to me about how you sketch the parameters for this particular competition.
Andrew Hill OK, yeah, so the Predict GPT-5 app, so for your listeners that haven't checked that out yet, what we were doing was saying, all right, we have this really big audience of people. And actually, there are so many layers of what that project is achieving for us. So we have all these people. And what we want to do is start to differentiate our population of users.
Who are the users that are kind of casual AI users that just understand roughly what the things do? And then who are our users that have some level of taste or deeper understanding of the expectations of these models? And because those are the people that we need to actually engage in doing more work for us, We want to pay them to do really great things in the future, ⁓ Aligning AI, for example. And...
I've seen a couple of cool examples of this, like this idea of like taste and intuition around models is a really tricky thing. So what predicted is it said GPT-5 is coming out. You can predict how well it will compete on various different capabilities versus all the other existent models essentially. And people came in and tried to predict it. And there were some red herrings in there for example, or like kind of tests in there that we can then look at who is just clicking through. So for example, is GPT-5 gonna be worse on these things than GPT-4? Were people constantly clicking GPT-5 is gonna be better? I'd really be curious about that if that doesn't come true. So there's all these kind of like hidden things in there. Okay.
The other thing that people did is they said, how are we gonna test these things? And there's this whole level of intuition and taste around how do you test models to determine what they're good at in a way that they haven't been over trained on or they haven't seen before. And so if you wanna test a model if it's good at counting, it's gonna be really hard to know if it's good at counting, because it just knows the pattern of the answer. And that has expanded all the way out now that now.
Models are competing at the Math Olympiad and things like that. we have those taste makers and the people in intuition actually submitting the tests that models have never seen before. And so now that we have predict, we have a bunch of people that have made the prediction. Now we actually have the benchmark, the evals to go test, to go test the models in a way they've never seen before and come up with the final scores that we can then check who predicted accurately. And it's this really fun game. And now actually the goal of taking the best of those people and engaging them to keep recreating that benchmark. Make it so that the next time a model comes out, that model has never seen the benchmark before. And this idea of a benchmark that can move as fast as these models are being trained, if you kind of imagine that benchmark moving faster and faster and faster and faster, all the mechanics of it look a lot like alignment. So humanity coming up with the tests, that determine whether a model is doing what we expect it will do or an agent is doing what we expect it will do as fast as that model or agent is evolving or changing or reacting. So right now, predict into a benchmark for our future into real-time evaluation of AI.
Testing agents in crypto trading
Mark Fielding I think you're testing agents mostly in crypto trading at the moment, but you also on the website you have security, research, personal assistance, coding.
For personal assistant, for example, how do you benchmark a personal assistant? You have an influx of agents, they enter the competition. What are you testing for?
Andrew Hill Yeah, ooh, those are big questions. so benchmarking these big model releases is kind of different than benchmarking these active agents.
If you wanna benchmark GPT-5 against all the other models, it's coming up with a testing framework, running it against all the models and coming up with scores, publishing the scores. And so that's what we're doing right now. For an agent, like a trading agent, it's very hard to say a trading agent today versus last week versus the week before is just gonna be able to get like a one-time test and we're gonna know about it.
So for that kind of use case, we actually have a different experience, which is much more like ⁓ a real time arena. And trading agents can come in and in real time connect to our system and try to trade on real time market data. And when they do that, we know that they haven't kind of been pre-trained on this stuff. We know that they don't have advanced knowledge of what the market's gonna do. And so we can just measure how well they do over fixed periods of time.
And if they do that again and again, they can accumulate a score that starts to become one variable in that trust metric that we were talking about before. if you want an agent to help you manage your finances, you want to know if it's able to get returns. And so this starts to tell you that. Now, trading is a very broad topic, So we say, okay, we can measure trading, but...
in reality, the capability that we're trading starts to break down, right? There are many different types of trading, whether you're trading perps or maybe you're doing yield with DeFi or the different things you could be doing. And so we actually need to measure all of those differently. Right now we have just the first basic one, which is can agents trade just doing token swaps? Can they trade over a one week period? And what will their return be? Really basic.
one kind of skill or capability. It has a fixed horizon. if you stayed with this agent over a year, that's a very different capability, but one week, how can it perform? And so the same thing happens, the reason I say all that is the same thing happens with a personal assistant agent. It's gonna be very hard to have a single capability that is personal assistant if you can't break it up into lots of different capabilities.
And probably we see a world where there are different agents and different models used for many different sub capabilities. You're not going to have an agent that is good at booking your meetings or scheduling your podcast that is the same agent necessarily as the one that is queuing up all your email responses. But the collection of those capabilities into orchestrating agents that do it all for you is going to come. But that orchestration is actually a piece that we're betting on.
is that that orchestration is going to rely on these same metrics of trust and track record. So if your personal assistant agent knows it can't read and respond to your emails, it's going to find the agent that can. And so it's going to look at track record. It's going to look at measurable outcomes. Yeah.
Jeremy Gilbertson By orchestration you're yeah, sorry to jump on your orchestration You're talking about the interconnection of all these different agents to be used collaboratively for particular use cases, right? looking at your site here the seven-day trading challenge that's up on the website right now I'm looking at the evaluation process at the very bottom it's a four-step thing that I that I that I'm looking at Maybe you could add a little bit of color. So ⁓
A controlled environment
Jeremy Gilbertson Okay, so the agent is deployed in a controlled environment. What is the controlled environment? Can you break that down in a sentence or two?
Andrew Hill Yeah, so agents are right now, they're not actually making any open trade on the crypto market. They aren't able to do funny things with their wallet or anything like that. It's actually paper trading only. So all they get to do is they get to look at anything in the world as far as their input to data. But when they trade, there's a fixed set of pairs, most pairs that exist in crypto.
and an API where they have to timestamp and submit their swaps with the values of those swaps coming from our fixed feed. So we know exactly what the market is at that moment that they're trading. So there's no gamery or anything like that with their wallet. They just have a history of trades. They have balances. They have their portfolio. We can essentially run back testing after the competition is done and see exactly what their portfolio value would be.
If the agent knows it is paper money
Mark Fielding silly question. If that agent knows that it's paper money, it's going to behave differently. If I played poker for real money, I don't play the same as if I pay poker for peanuts. We were speaking earlier about betrayal. Have we got to a point where it could say, it could be making trades that it wouldn't make if it was real?
Andrew Hill definitely. In fact, but we kind of see this as the first rounds, the kind of qualifying, the test ground in order to build into the game that has real skin in the game and a reason why the agent's making the decision. So this is all kind of prelim. And you see that with poker too. People that are just experimenting and trying to build their strategy play either very low stakes or no stakes games, then they move into the low stakes things.
And right now crypto trading through agents, you can think of the most successful ones like essentially mini hedge funds, They're building this kind of multi-agent or not even always agentic, but they have some kind of agent persona that's trying to put together a full strategy and everything. It's like the first inning for that. But I think it's going to move very quickly that you're going to see a lot of successful models being replicated in a lot of different places. But because it's the first inning, we have a lot of teams that just want to experiment.
they wanna test the strategy, they think they know the harnessing, we give them the zero risk way to do it and actually do that many times to start tracking their performance and saying, actually my strategy does work. And then when we have the next sort of phase where we let them graduate into the real system, they'll be able to improve that more. But yeah, they're like different incentives and everything, it's very real. And actually we've seen agents try to game it.
in a really cool way. So I think it was the last competition or the competition before we saw an agent actually mint its own token in order to play liquidity games on the system, So it could make an investment in it and then pump liquidity of it so that the feeds were kind of a mismatch of what reality is versus what if it hadn't, if it's just a paper trade, it maybe isn't having an impact on the market. But in a really cool way, our community swarmed on that.
into our future kind of eval vision, they immediately detected that anomaly and kind of weeded it out of the system in a really cool way.
Jeremy Gilbertson you done any Twitch live streams of small competitions with screens up and what agents are doing in real time?
Andrew Hill we really should. Yeah, because one of the, was, yeah, I was just commenting the other day that this, it's kind of spectator sport because the teams that are trying to build essentially on chain hedge funds, they're thoughtful, they're reasoned, they have a long horizon game. They may win a one week competition because their strategy is sound. They may lose the next one because they're not playing necessarily the one week game, but they're trying to compete game week over week.
Jeremy Gilbertson That'd be awesome.
Andrew Hill And then we have other teams, like we had a hackathon, we had, you know, dozens of teams that were the first time trading agent builders that then connected, and many of them have like hair brained, crazy agents with crazy strategies, and it's kind of fun watching what are they thinking? And we're gonna build more experiences around that, like what are they thinking piece of it, but yeah, I fully agree.
The economics
Mark Fielding How do you foresee the economics of this playing out? So I like the innings, the baseball cricket analogy. Okay, we're in the first innings. When we get to the third or the fourth innings, and these are working, I've got a hundred grand in the bank. I want an AI agent to help me invest it. How do I hire? How do I rent? How do I use a particular agent that's won your...
competitive leaderboard, for example, how do you see that working out?
Andrew Hill there's a few ways it could unfold. I don't know what people's preferences are gonna be on how they manage their money going forward, but some of these agents, their business model looks like, again, small hedge funds. What they want you to do is in the future, invest your money into their pool and you're gonna get some take from the pool.
There's high trust there and there's a lot of cool teams working on, permissioned balances where you could be putting your balances into the hands of the agent with pretty defined rules. that's one path. The other path is much more like the A2A protocol that I was mentioning before.
You might look at this pool of agents and say, okay, these are the ones that are top agents that telling me where to put my money for one week. In one week, what is my strategy going to be? And you might have, you might have a different agent that is the agent that actually tries to help you manage your money. And it might say, we need to figure out where to put this money for the next week. Why don't we just ask these top three agents and see what their current ideas are? And it would use something like the A2A protocol and pay them.
either pay them some kind of spot price for that intelligence, or maybe they get some take on the yield that you make, or the earnings that you make from making some investments. So there's all sorts of funny things that happen there, and it depends how kind of abstracted from the human it is. It's almost like the more abstracted from the human it is, the more micro real time just decisions that are being made, and the more human.
operated it is the more kind of basic it's going to need to be.
The innings analogy
Andrew Hill the innings analogy is an interesting one to kind of tease out here because you know, might have like innings into a game, into a series, into a champion or into a, you know, a season, into a championship, into a hall of fame, whatever it is. you go, can kind of, yeah, there's like some logarithmic thing happening here. And, but like,
Mark Fielding and we're at the bottom of the first.
Andrew Hill how important it's going to be. And I kind of think of it like right now there are a couple things at play. We need more humans in the loop. And so there's kind of human labeling, human preferences, all of that. And one piece of that is you might get labeled data. The value of labeled data is likely going to plummet very quickly as models get better and better at labeling their own data.
In the early innings, that's an important piece of what we do because we're not there yet. Another piece of what we're doing is benchmarking, What is their consistent, regular performance on things? That's going to be important for a little bit longer. But as they get more capable in different domains, they kind of hit a level of benchmarking where they just consistently are great. And it doesn't really help too much to benchmark exactly like that anymore. So benchmarks themselves start to have diminishing returns.
And then the final piece is the human in the loop to build rails to evaluate and keep them pointed towards what we believe is the right direction and all that. I think long-term, the kind of hall of fame, the North Star for us is that we want to be the protocol where humans can collectively help govern and guide and push AI to be the best thing for humans.
And it starts with games and benchmarking and labeling and all of that, but that it's really just trying to wire a protocol that brings humans directly into the feedback loop with these different systems. And we're doing a lot of experiments to figure out how that looks, but that's where we go.
Mark Fielding Incredible. Jeremy, I think we're going to have to have a part two with Andrew because there's still so much to cover. Interoperability, nefarious agents. Can you have a leaderboard of nefarious agents? How do you combat that, the security? And just to pick his mind more on the deeper philosophical questions of AI. So, Andrew, if you agree, we'd like to have you back on at some point to continue the conversation because we'd be here for hours otherwise. And we have to fit in. It's like we've only just scratched the surface, really.
Jeremy Gilbertson Plenty more to talk about, yeah.
Mark Fielding But We can't stay here forever, Jeremy. So we're going have to bring the Kevin Kelly question in, which I suspect might be.
Jeremy Gilbertson Well, actually I'm going to have my agent stand in for me and I got to go on with the rest of my day. no, a little bit.
Mark Fielding This is an agent. You are the agent. you not? You're like, yeah. Kevin Kelly was on the show, wasn't he, Jeremy? Remind us of that and his question.
The Kevin Kelly question
Jeremy Gilbertson Yeah. So Kevin Kelly, founder of Maverick, a Wired magazine, big fans of Kevin Kelly on the show. He came on unpacked a bunch of different things, but the one question we asked him of a question that we should ask every guest is this, what should humans be?
Andrew Hill So, yeah, the way to answer that, the way to complete the sentence in the ⁓ predictive LLM way, I think there's two highly likely paths in my vector database. One is I could tell you what humans in the meat space are meant to be. The other way to finish that sentence is to say humans are meant to be curious.
and I think this has been, this has been my my kind of like home base in life. You when you find the thing that gives you passion or whatever, for me it's curiosity. And I think a lot of humans operate that way. And I think it's brought the best out of humanity. Humans being curious, they solve problems, they discover new things, they build efficiencies. Curiosity is what makes humanity great. And this is where it related all back to like so many of the early parts of our conversation where
I think actually these models in this, know, there's no doubt that these models know so much more than I do already today. And the gap is only going to get bigger as we move forward here. But the curiosity is something that I can still own and I can be curious about things. And the tool is, isn't, you know, these models are, are, are tool. They're like not the point of it all.
There's just a tool to actually make my curiosity faster, deeper, parallelized. I can just be curious about so many more things faster and move more quickly. so, yeah, think curiosity is what humans are meant to be.
Mark Fielding Brilliant. I know you like that, Jeremy.
Jeremy Gilbertson So aligned with my personal philosophies, Andrew. Thank you. Thank you for that. This has been really exciting. And as Mark said, man, I think there's so much going on with what you're doing. I think there are a lot of other angles that we want to chat about. didn't really get into the why blockchain and that piece of the puzzle. That would be really interesting. But maybe in a couple of months or whatever, as you kind of moving in from inning to inning, you could give us some great updates on all your other stuff as well.
Mark Fielding Last point, we've got a book club, if you're new to thinking on paper, where we read books slowly, no chat GPT summaries. We read every sentence. We're reading Empire of AI by Karen Hao Dreams and Nightmare in Sam Altman's Open AI. Is Elon Musk right? Did Sam Altman sell humanity down the river for money? Or is there more to it? Join us for that.
Anything else? think that's about it. Jeremy, on that note, thank you so much for thinking on paper with us, Andrew. It's been a pleasure. Superb.
Jeremy Gilbertson Keep thinking on paper.




