How to Take Down the World’s Most Advanced AI Data Center

Shapol, CEO of Entangl, says about two thirds of data centre outages come from someone turning off the wrong switch. And we essentially direct the engineers to do the right work.

Clips and outtakes on Instagram ›

In this episode · 14 chapters

  1. Optimising manufacturing
  2. The main problem
  3. Real time monitoring
  4. What if it is already built?
  5. A major constraint
  6. Human error
  7. How it used to be done
  8. The Dunning-Kruger effect
  9. Why this is different
  10. We do not want creative autopilots
  11. Trust
  12. O-rings
  13. Space-based solar power
  14. Kevin Kelly's question

Shapol and his co-founder Antanas used to lead a reusable rocket programme. Their company, Entangl, now reads a data centre's designs and tells engineers which work to do, step by step, so fewer people press the wrong switch.

He explains why his system is 100 percent deterministic, closer to an aircraft autopilot than to ChatGPT, and how human error and generator lead times shape the work. We also ask about Challenger's O-rings and data centres in space.

Shapol, CEO of Entangl, a Y Combinator company building AI for data centre operations, tells the Thinking On Paper podcast that about two thirds of data centre outages come from someone turning off the wrong switch. Entangl's software reads a site's design drawings and writes the procedures engineers follow: “And we essentially direct the engineers to do the right work.”

What causes most data centre outages?

Human error causes most data centre outages, Shapol says: about two thirds come from someone pressing the wrong switch, because engineers face thousands of switches and lack a full grasp of the design. He puts cyber attacks, flooding and other causes in the remaining third.

How does Entangl turn designs into operating procedures?

Entangl turns uploaded design documents into operating procedures, the step-by-step instructions engineers follow for each maintenance task or failure. Shapol says the software detects every piece of equipment on site, writes a procedure for every possible job, and updates the drawings when new assets such as generators are installed.

Why is Entangl's system deterministic, unlike ChatGPT?

Entangl's system is deterministic, meaning the same question always produces the same answer, unlike ChatGPT, Shapol says. He compares it to an aircraft autopilot and argues that in fields “where you really can't afford mistakes”, variability is not acceptable.

Why do new AI data centres open without backup generators?

Many new AI data centres open without backup generators because lead times run to 18 or 19 months, Shapol says, which forces operators to update designs and procedures later. He estimates maintenance at about 50% of the cost of running a data centre.

How do hyperscalers test tools like Entangl?

Hyperscalers adopt tools like Entangl slowly, over months or years of testing at lab sites, Shapol says, using blind tests where the system must find which past procedure caused an incident. Customers have also started using it to train junior engineers before they touch live equipment, a use Entangl did not design for.

Where Mark and Shapol disagree

Mark asks how a system can be trusted when AI is prone to hallucination, and later, citing Challenger, whether humans keep the final say. Shapol answers that for now “they do”, and that whatever comes next will be constrained by operators wanting work done more reliably and more cheaply.

Questions

What is Entangl?

Shapol, CEO of Entangl, a Y Combinator company building AI for data centre operations, tells the Thinking On Paper podcast that about two thirds of data centre outages come from someone turning off the wrong switch.

Who founded Entangl?

Shapol and his co-founder Antanas used to lead a reusable rocket programme. Their company, Entangl, now reads a data centre's designs and tells engineers which work to do, step by step, so fewer people press the wrong switch.

What share of data centre costs goes on maintenance?

He estimates maintenance at about 50% of the cost of running a data centre.

Transcript

Jeremy Gilbertson disruptors and curious minds. Welcome to another episode of thinking on paper where we unpack the future with the people building it. My name's Jeremy. This is Mark. Today we have a really special guest that, is going to help us unpack some things beneath the layers of the way we operate with AI. For instance, you users as you're busting around, messing around with nano banana or chat GPT or Claude or whatever you're doing. You don't tend to think about the layer beneath that supports the systems, right? Not just the IT systems, but the systems that support the IT systems. We're talking about mechanical electrical infrastructure that helps keep all that stuff up and running.

So we want to get a couple layers deep, but hang with us. We'll walk with you. We'll help you understand this. Cause I think it's beneficial to understand what the heck is going on with AI in general, why people are spending billions upon billions to build these facilities and what it means.

to the user operating the tools that are supported by them. So Mark, who are we talking to today?

Mark Fielding We are talking to Shapol the CEO of Entangl. And I just want to read a message which I read, which really, when I read this, I thought we have to speak to the people at Entangl. Quote, Hey folks, Shapol and Antanas here, his co-founder. We used to lead a reusable rocket program and oversaw the launch of four missions. We hated engineering design software so much that we created our own, which is why we're here today. And it's why Shapol.

is here today with us. Welcome to the show, Shapol Thank you for thinking on paper with us.

Thanks for having me. Great to be here.

Optimising manufacturing

Jeremy Gilbertson There's a lot of experimentation that happens trying to build rockets, right? You build rockets, blow them up initially, they fail. You learn from the failure. There's a lot of experimentation. There's a lot of refining, a lot of iteration. Data centers don't like experimentation. They don't like to be messed right? So.

What we're doing is trying to optimize manufacturing, trying to make it as cheap as possible. And that's where a lot of experimentation With data centers, you're obviously trying to make them...

as cheap as possible, in terms of actually building a data center, there's not much room for experimentation there. Obviously you have a lot of experimentation on the chip side, on the actual electrical hardware side of things, and you see companies like Nvidia do a lot of this, and they probably fail all the time in their labs, but you just don't see it as much. But in terms of the actual building of four walls of a data center, there's not that much.

room for optimizing the manufacturing of or construction of that data center. So that's one key difference.

The main problem

Jeremy Gilbertson let's talk about specifically how your company, how your, how your platform helps on those two phases. If we look at design and engineering upfront, how these facilities are built, and then we look at operations on the backend, what you guys are doing to automate or create an autonomous means of engineering and operations.

start with the latter. And the main problem that we're solving today is, as you pointed out at very start, one of the biggest risks in data center operations are the people that operate them.

about two thirds of the time, whenever there's an outage at a data center, it's someone turning off the wrong switch. It's as simple as that. And the reason why they turn off the wrong switch is because they don't have that thorough understanding of a data center design. You've got thousands and thousands of switches in a data center. And when you've got to do maintenance or you've got to do some sort of work, let's say Nvidia releases a new GPU, and you're tasked with upgrading existing hardware, you've got to figure out, how do I do this work? Which switches do I et cetera, et cetera? And for a normal engineer, that's quite hard to figure out just because of the level of complexity that there is. So what happens is, you know, they press the wrong switch, the site goes down, you can't access Zoom or EWS or planes don't take hospitals don't work, et cetera, et So it's a very big problem, especially given that our society pretty much runs on data centers. You're not aware of it, but I mean, you'll be aware of it when they stop. But other than that, you're not really interacting with them on a day-to-day basis. Now, where do we come in? We can take in all the data centers design. So we have...

will understand how all the circuits work, how every server is linked with power plant, etc. And we essentially direct the engineers to do the right work.

So the engineer says, okay, I've got these new racks that I want to install, these new servers that I want to install. They AI it just breaks that down. I'm okay, you've got to go to this data hole first, you've got to go to this rack, you've got to press this thing, do that, X, Y, Z, what we do today. It's still very much human driven in terms of what's happening on site, but...

A lot of these companies also build in it to be lights out. And by lights out, mean it's autonomous. You have robots in these data centers that run the show. And we are one of the very few companies that actually have that understanding of, how do you operate data center based on the designs? So that's where we come in. But today, predominantly, it's all very much human driven.

Real time monitoring

Jeremy Gilbertson is there any real time monitoring? Do you tie into like monitoring system and talk me through how that all works? Yeah.

on site, you typically have lots of sensors, lots of things to track.

power of things that are, that's linked to something called BMS, Building Monitoring and we integrate into that. So we can connect to that, understand how everything's doing, if something goes down or if something's about to go down, we can tell them, hey, this is gonna go down, this is the work that you gotta do in order to keep things running. So that's one part of our integrations on site.

Jeremy Gilbertson And there's another piece too of the operational side that is pretty traditional, right? You have these operational plans that include what are called standard operating procedures, methods of procedures. So if this happens, the human response should be this, And sometimes they're complicated. Sometimes they're easy. Sometimes the training programs are great. Do you import those into your, into, is that part of the, the input side to help them?

We actually output that. let's say you've built in site.

you can just upload all your designs and then it detects every single piece of equipment that you have, every single possible work that you could possibly do it on site. And then it creates every single one of those documents you mentioned. And each one of those contains instructions for how to do that work when let's say a server goes down or generator goes down, So that's the output mainly for us.

What if it is already built?

Jeremy Gilbertson What do do if someone already has that developed?

It's a really good question. you still need to like update your procedures. It's never Maintenance is still one of the big costs of running the data centers. It's about 50 % of running data centers is maintenance. every company wants to have the perfect documentation.

And rather than you don't really get to that promised land, you always have things changing to upgrade your system hardware. For example, a lot of these AI data centers today, they don't have generators. So you typically have power from your mains that you bring from the power station or substation directly to the data center. And then in addition to that, you have a redundant power supply in case the mains goes down, which is generators. In lot of these cases you don't have generators because the lead times are so crazy, like 18 months, 19 months in most cases. they do is they continue without that second power source, they build their data center, but you're still going to need that generator in 18, 19 months down the line. And that means you will update your designs, will update your procedures,

A major constraint

Jeremy Gilbertson you mentioned something very interesting one of the major constraints in the design time is lead time for switch gear, lead time for generators and that sort of thing.

Is that something you guys help with on the design side?

We help with how you make that design as reliable and redundant as possible. So we take in the learning from the operations of your existing If a particular incident keeps reoccurring and that's...

that can be prevented on design side, that's something that we help. An example would be, let's assume a data center has got only one power source, your AI data center, on site keep turning off that main power supply and things go down because, I don't know, you've got to interact a component in between your racks and your power supply.

we would basically take that learning and tell the design team, hey, in order to prevent these same things from occurring, you've got to add a second power supply here, So those are the kinds of things that we help with. We take the learning from the operations side and back to the design. We do help with also taking the designs and cross checking them against.

local policies. a lot of time, a data center is very quickly cut out. You've got your four walls, you've got your power equipment, your cooling equipment, and your service. It's pretty quick. You can go to any place you can put down, could cut out and build data center. What really changes is the policies that you're building around. So when you're building a data center in California, it's very different to building a data center in Texas. And a lot of the time you have these

You know, you go through these long approval processes just to prove that your design matches the local regulations. And sometimes it conflicts and you've got to fix that issue. That's something we help with. So we check your designs and be like, these are all the conflicts that you have and this is how you solve it for easier regulatory approval.

Mark Fielding jump in as like, feel like a gooseberry on a date with two people who know everything about data centers, having a conversation about the intricacies of a data center. I'm reminded of the episode of the Simpsons where Homer goes to work and he presses the wrong button in the nuclear power plant and Springfield goes into an emergency because there's a nuclear meltdown at the nuclear power station, which was Homer's error, human error.

Human error

Mark Fielding A lot of what you're talking about is human error. For the layman out there, essentially data centers go down because people press the wrong button or there isn't a secondary power supply. If those are the two main, would you say those are the two main causes of downtime first? Human error and lack of a power. Is there any technical reasons they would go down?

The main one is human error. You obviously have lots of other causes that take up the last third, know, cyber attacks, flooding, et cetera, et that are very much edge cases. But about two thirds of time, it's literally, as you described it, someone pressing the wrong thing.

How it used to be done

Mark Fielding So, so on thinking on paper, we like to connect the dots of all these technologies. A long time ago now, back in the early days of thinking on paper, we had Don Norman on the show, the godfather of design. And he spoke about the error of design that essentially leads to humans making mistakes and whole systems going down. And he was speaking from very, very simple things like a light switch or revolving doors in a cinema, like these very simple everyday occurrences. And it's both quite reassuring and also ultimately terrifying that those same problems go into, as you said earlier, perhaps the most important technological infrastructure propping up humanity right now. Before you guys and Entangl, how were the hyperscalers, how were the Metas and the open AIs and the X, working to resolve these problems before you came along?

Before, what you used to do was you'd write these procedures by hand. So you go into your Word or Excel, you would say, is the work that I'm trying to do. I'm gonna write instructions for how to do this so I don't make a mistake. You bring out your design drawings, you bring out manuals maybe. So if you have a particular equipment that you're working on, it probably has a manual where you check against that. And then you write each step. Okay, step one based on the design document I go and go to this place. Then once you've drafted that you'll send it for review and different companies have different ways of reviewing their work, but typically you would go into a meeting where you have a few people reading that procedure, they'd verify, okay, this makes sense or it doesn't make sense. And then they do the work. And then once they do the work, then it would either be correct or it would be incorrect.

Mark Fielding Sounds so slow and so outdated when you speak like that. Word documents and instruction manuals.

Jeremy Gilbertson where you'd actually, you'd actually, actually do rehearsals. You'd actually do mop rehearsals where you would sit around the table and read it like a script and be like, okay, now Shapol is going to turn this breaker off. Mark, your next step is to make sure your team is positioned in XYZ locations to verify this happened. It's yeah.

Yeah, no, no, it definitely is very slow. It can take, in some cases where you have to write those documents months ahead of time, ⁓ scheduled ahead of time, be like, okay, this is when I'm gonna do this in a month's time. ⁓ But the issue still is there, you you still make mistakes and that's because you still have that human error in...

checking those design documents, checking those manuals, bringing out the right items from that data.

an ideal world, they would check all those design documents, but most of the time they don't as well. In all honesty, you can have some complacency, human complacency in there, where someone thinks they know what to do, especially as you gain more experience, like, I know my way around the data center, and then you write something, and then it's actually the wrong thing. That can happen as well. The other thing design data is not...

always up to date, in fact rarely up to date.

You have the design documents submitted once you've built your site. And then you've done lots of work over time. This data center that's 10 years old, you've installed new generators, you have new racks, you may have a different structure, you may have different components within that data center. So things change. And those design documents are not updated over time. Whereas our software basically tracks every single piece of work is that when someone installs a new generator it detects that it says new there's a new asset in the data center and then updates your drawings and that keeps the system up to date whereas with the current process you have error in understanding those that design document you have error updated those drawings are and you also have that human complacency in writing and also reviewing that procedure.

The Dunning-Kruger effect

Mark Fielding You got me Dunning-Kruger effect, a lot of these errors come from people over evaluating their strength and their knowledge in a particular area. We're all guilty of that, but sometimes the stakes are a lot higher. Hallucinations, almost like AI is a victim sometimes of the Dunning-Kruger effect. It over evaluates its...

capacity to understand what's going on. Some of these designs will be incredibly complex and complicated. They will be bringing sources from all over the place. AI is prone to hallucination if less, perhaps less than it was. First question, how are you dealing with that? Second question, how do you convince the chiseled old guard that they should be trusting this

Why this is different

I'm glad you asked that because we get asked that a When you go to ChatGPT you're asked something that That's most people's experience with AI.

The difference between our system is that, know, ours is 100 % deterministic. That means every single time you ask us to do this thing, every single time we'll output the exact same thing. What do I mean by that? When you go to ChatGPT if you ask it, I don't know, how you doing on a Monday, it'll come back with you with one answer.

If you ask it the same question immediately after, it'll come back with another answer. There's variability in that answer. In ours, there is no variability. And one analogy I can make is an aircraft autopilot. An aircraft autopilot is 100 % deterministic.

it piles the pain itself, can predict what's going to happen, etc, etc. It's autonomous. And you may even say it's intelligent to a certain extent, but it's 100 % deterministic. And in those kinds fields where you really can't afford mistakes, you really need to follow that alternative where you don't have any variability. Our system is based on 100 % human designs and every single time you say, how do I do X work, it will come back with that same exact answer because that's what the fact is.

We do not want creative autopilots

Mark Fielding So we don't want autopilots getting creative. And is that determinism, is that it always gives the same response? Why chat GPT and these LLMs, ⁓ they want to be creative, they're programmed and designed to be creative and think outside the box. And whereas in these systems, yeah, you don't want any creativity.

Yeah, no, no, I think that's a very good way to actually put it.

With these LLMs, you want them to be general, want them to have that creativity. In some ways, hallucinations is actually a feature, not a bug in some of these LLMs where I can think outside the box, as you mentioned. ⁓ But in stuff you want done 100 % accurately, you don't want that.

Trust

Jeremy Gilbertson stay on the trust piece of the puzzle. Cause I know you're having conversations with these, providers and hyperscalers and that sort of thing that have a lot on the line. They have a lot of tribal knowledge about how to operate these systems.

How do you convince them that the model has what it needs to take them where they want to go?

the process to actually convince them is a very long process. It's never like...

single meeting or two meetings that's months long or years long where they test it out, they bring all these different people to review what the tech is and verify that it's to actually work. But typically, you would either test this in these lab environments or at one single site. So most of these hyperspace are built like labs where they test out new technology. It's less critical than data centers that actually serve the public and once you verify that all this is this actually works as it's supposed to, then you can roll it out a lot of, there's various tests that you can do. One test you can do is, okay, you get one of your engineers and you say, okay, write the instructions for this work. Then you get the AI to do it. And then you compare, okay, how long did it take with the AI? long did take with human? The AI is instant. The human takes a few hours or a day or a week, depending on the work. And then you look at the issues. Well, our system is 100 % deterministic. It's based on the designs. It's got 100 % accuracy.

like how an autopilot needs to have 100 % accuracy, our system has that. A human has got that variability. And then the second test you can do is if you've had a particular incident in the past, which you almost certainly have, you can do a blind test where you put in hundreds of procedures in the system and you're like, okay, find out which one caused the issue. And then the system finds out, oh, it's this one. If you give 100 of these documents to a human and you tell them,

Okay, figure out which one of these caused an issue. I can guarantee that they will be able to find it. Whereas with AI, because it knows as a matter of fact what happened and what will happen, it can very easily figure out, okay, this is the wrong one. And that also proves that the system works as intended. Once you've proven that value, then you can roll it out across its sides and take advantage of it.

Jeremy Gilbertson What are the coolest use cases that you've seen that come back that you never would have thought of? Like your client, your customers are coming back, hey, we use it for this and helped us do this?

a lot of different ways I guess people use training new engineers is one thing where we didn't really design the system for that, but we've seen people tell their new engineers, their junior engineers, hey, before you actually touch anything, we want you to just use the system and get your work up to a level of standard.

and then we can unleash you into the actual data center. That's something that we never designed before, we didn't think about it, but people are using it for

O-rings

Mark Fielding want to take it out into space in a minute, but not much documentation out there about you and Entangle But what I did find one of the leading pictures Challenger crash 86 and that was caused by the.

faulty o-rings because it was too cold and then they launched and it was too cold and they were damaged by that. But some people in NASA control were worried and flagged the temperature that day and their decisions were overridden and the launch went ahead. So even though the problem gets flagged sometimes in your systems do the humans have the final say?

Will they always have the final say? Do you envisage a time when they won't?

Right now they do. What happens to the future, I mean many things could happen. It could be that humans always stay in the loop. It could be that we have AI systems that are far better decision makers than any human would be. And in all this I don't know what the future will look like, but what I will know...

fact is that these companies will always want things to be done more reliably and they'll always want things to be done more cheaply and I don't know what the future will hold but it will definitely be constrained by those two things.

Space-based solar power

Mark Fielding market forces will dictate. We can't speak about data centers without two threads which have been coming up more and more and more and more on thinking on paper. One is data centers in space and the other is space-based solar power. How do they link to what you're talking about? At beginning of the show, you mentioned secondary power supplies and lead times and how problematic that was. We have

On reliable information with space-based solar, you can connect, you can have your own sovereign power source stuck to the top of your data center, being powered down from space without connecting to the grid.

What's your thoughts on space-based solar power?

if you get to a certain scale, will be cheaper to build in space than on Earth? be interesting to see what the regulations are with beaming energy down.

see a world where people complain about this kind of stuff, they're like, do we want energy being down from space? I mean, you have 5G towers and building your homes and people go crazy. So if you have energy being down from space, think people will be a bit more, they'll complain a little bit more. But that said, they may not even know that energy is beamed down from space.

And by the they do find out that it is being done from space, maybe like years down the line, they're like, it's coming from space. I didn't know that. We've been fine for years. mean, space-based power, Will be needed for some applications. don't know. It may be for on Earth applications. It may be for outer space applications.

Mark Fielding so you're not totally convinced by space based solar power. Okay. Star cloud data centers in space.

data centers in space, I am bullish on. So I don't know if you know this, but Philip and I are in the same YC group, so Philip and Star Cloud. And I actually refer them to YC. I not a matter of if it's going to happen, it's just a matter of when.

in my mind, like eventually you are going to have all these heavy industries out in space. People don't want this stuff on earth, you you would much rather have it not on earth. And that will happen. And I think there's many reasons, many, benefits for that on.

even on the design side. Obviously, it's not an easy thing. People think that these guys that are doing space-based data centers think it's easy and it's going be like an easy ride. It's not. It's going to be very, very hard. There's going to be lots of design challenges. But I think it's going to happen, and I'm very excited for that future.

Kevin Kelly's question

Jeremy Gilbertson let's get back to the human side. So Kevin Kelly's question that we like to ask all of our guests is what do we want humans to be and how does technology help us get there if it does?

The vast majority of people are doing jobs that they probably don't want to be doing. What they do enjoy about their jobs is socializing with people, seeing other people on a daily basis. But if you actually ask them deep down, do they actually enjoy their work on a daily basis, they probably won't say yes. It'd be a very small minority of people that truly do love their job.

You know, if you go back 200 years and you were to ask people the same question where they were doing heavy manual labor, whether it was in agriculture, et cetera, you truly ask them, do they love what they're doing, they'd probably have said no. And it's probably for the better that people are doing those jobs less today than what they were doing previously.

I'm very pro a world where people are no longer doing these jobs that they don't like and it sounds very scary at first but I think it'd be better. So really the question is how do we shape society so that people still get fulfillment and you get fulfillment by...

speaking to others, having a laugh, helping out other people, etc. etc. And that's where I think we should really focus on. And I think the vast majority of work that's being done today will be done by some AI in the future.

And I don't necessarily see that as bad thing. It's scary because a lot of things change. The other scary thing is how that technology is controlled. If it's only 10 companies in the world that control this thing, That would be a very, very disturbing world. That should be avoided at all that would be the thing that I would fight against the most, not against the tech itself that helps people live their lives, but actually the...

the power being concentrated in the hands of very, very few people. to answer your question, what will humans be?

There will no longer be slaves to jobs they no longer want to do. That's how I describe it.

Mark Fielding I love how everybody answers this question differently, Jeremy. Always everybody takes it in a different direction, but it's so interesting to hear. I think I agree with the most part of that answer.

Jeremy Gilbertson Disruptors and curious minds you can check out all of this stuff out Thinkingonpaper.xyz got a lot of amazing guests coming up. We got a book club space to grow. We just started that this past week. We're in we're in space world right now And it's very exciting. We got a Philip Mesker coming on the show Speaking speaking of putting things in space and more industries going in space. He had a great paper that we broke down we're excited buckle up. It's gonna be a heck of a year

Jeremy Gilbertson Stay curious, be disruptive.

Mark Fielding Keep thinking on paper.