The Hidden Dangers of AI in Modern Warfare

BIANNA GOLODRYGA: Now, the U.S. is reportedly deploying artificial intelligence to help fight its war with Iran. As the Pentagon pushes for less human oversight over the use of this technology, our next guest is sounding the alarm around the safety and reliability of these tools, particularly in facilitating what is called a, quote, “kill chain.” Heidy Khlaaf is the chief A.I. scientist at the A.I. Now, Institute, and she shares her concerns on the growing use of A.I. systems in the military with Hari Sreenivasan.

 

HARI SREENIVASAN: Bianna, thanks. Heidy Khlaaf, thanks so much for joining us. You are someone who helped pioneer the field of AI safety. And as an engineer, what does it mean and what does it look like in practice?

 

HEIDY KHLAAF: So, AI safety has a lot of different definitions to different types of people. But I come from the traditional safety engineering discipline, which is about making sure that systems are safety critical. So, so things like airplanes, nuclear plants, our infrastructure: if it fails, human lives are at risk here. And that’s a very different type of discipline than what people think about in terms of AI safety. And over the years, AI safety has really become about, you know, existential risks or this fear that they will, these AI models, will become super intelligent and, you know, then become a risk, to society at large. But the differences here – the risks that these AI companies talk about when they talk about AI safety – are really hypothetical. They’re not concerned with the everyday risk that AI models can pose to human lives. That’s very different from safety engineering, which is my discipline, which thinks about the human lives that are affected from sort of like the failures that could occur from our infrastructure. 

 

And so I often view AI safety as sort of safety revisionism — or that that term has been co-opted — because we’ve moved very far away from trying to make sure that our systems are accurate and reliable, towards this idea that we’re going to build some super-intelligent being that’s going to solve all of our world problems. And I think it’s very important that we always focus on the science and how these systems actually fail, rather than hypothetical sci-fi situations that actually don’t help us make these systems reliable at all.

 

SREENIVASAN: There is this kind of life threatening scenario that people are getting familiar with, which is how is AI being used in warfare? How do you see AI contributing to the way that militaries are carrying out their actions?

 

KHLAAF: When you’re using things like generative AI or large language models for writing an email, these models getting something wrong is very low risk, right? Nothing, no one dies, nothing changes. But then when you move to trying to implement them — in safety critical systems, like in defense, you’re literally, you know, determining the lives of people, right? You, this is, this is very much high stakes. 

 

And, you know, when you’re looking at the accuracy of these systems, it, they shouldn’t be near any sort of targeting at all. So, for example, Maven, which is currently being used by the U.S. in Iran, has low accuracy rates. You know, two years ago an investigation came out that showed their accuracy rate are as low as 30% in some situations. And overall, when you’re looking at the averages of these models, their accuracy rate is as low as 50%. And you know, that’s really not far from flipping a coin, is it, right? The sort of of 50/50 random chance. And I think that should make us question: why are these systems even near targeting at all if they’re this inaccurate? And, you know, again, there could be other uses of AI where you, you know, there aren’t life or death consequences, but in the case of military, that very much is what’s at stake.

 

SREENIVASAN: So help us kind of explain the differences in how the military uses it. I mean, right now we think of, you know, this the phrase of autonomous sort of killing machines and we’re ascribing this power to AI and we’re having this kind of debate about whether or not companies should be doing that. But there’s, you know, another layer of just like intelligence and intelligence gathering. So how is AI involved in that?

 

KHLAAF: So first I wanna preface with the fact that AI has been used in the military since the 1960s. But it’s a very different type of AI than what we’re seeing today. Back then, and up until a few years ago, they were using what we call sort of purpose-built military AI models where they were very task specific and they were trained on specific tasks with specific data for, you know, some mission. And that’s very different from what we’re seeing today in the use of generative AI or things like large language models where they’re being implemented in what we call decision support systems, which are essentially tools that bring together a lot of data like satellite images, social media feeds, intercepted communications. And that model then uses all this information to make military recommendations, including targeting recommendations. 

 

And I think a lot of people are probably confused about this type of term because we’re also hearing a lot about autonomous weapon systems. And the difference between decision support systems and autonomous weapon systems, is that autonomous weapon systems are allowed to select and engage with targets without oversight from a human being, versus decision support systems that do have the so-called oversight, right — and it’s questionable how much oversight there really is — that tend to provide a game-like, or a chat bot interface that a military operator then uses to approve AI target recommendations. But overall, AI is being used in every part of what we call the “kill chain.” So things like intelligence, surveillance, and now we’re looking at the selection and then the strike of the targets as well.

 

SREENIVASAN: You’re talking about taking something that wasn’t designed for the military, the large language models, and we’re kind of putting that into the military’s needs. How do we measure how accurate those systems are in the type of tasks that we’re asking in the middle of war?

 

KHLAAF: I mean, that’s a very good point. You know, if you have vision models, things that have been trained to detect a tank — they already had low enough accuracy rates before. You know, we had the Air Force that had a targeting model, which they thought had 90% accuracy, and actually in practice only had 25% accuracy. So we were already dealing with these issues long before, large language models were being implemented within sort of military decision making. 

 

Unfortunately, it is a case and has been shown by a lot of research that commercial general models are much less accurate than military purpose-built models. And so we have an issue where we’re actually going towards models that have reduced accuracy in terms of military context. 

 

And they also have security issues. And I think we’re not talking about this enough. Because they are built on a commercial supply chain, the supply chain is not vetted as we typically would see with a military system. So there’s actually security issues as well. It’s not just a safety issue. They can build back doors into these models. We have seen operations from Russia and China that put out a lot of different types of, you know, propaganda to try to skew the outputs of large language models. And Anthropic themselves have admitted you only need to change about 250 documents or data points for a model to be able to change its behavior. 

 

So we have multiple issues here. And so it’s very unfortunate that instead of trying to improve on these task-specific models that we’ve had before — which again had their own accuracy issues — we’re moving towards something that’s much less deterministic, much less predictable, and unfortunately not accurate or reliable either.

 

SREENIVASAN: There’s a video message from the head of U.S. CENTCOM last week and, and it’s, he said, partly “Humans will always make final decisions on what to shoot and what not to shoot and when to shoot, but advanced AI tools can turn processes that used to take hours and sometimes even days into seconds.”

 

So I’m trying to figure out here, if you’re saying that these models are inherently not as accurate and reliable as we think, and if these decisions are made so fast, even when a human gets that information in front of them. Is there sort of a bias where I might say, this is probably good?

 

KHLAAF: Absolutely. There’s definitely a bias here, and that’s why human in the loop is typically not a very meaningful solution. In our field, we have what we call automation bias, which is this idea based on decades of research showing that humans often trust the recommendations of algorithms without corroborating with other sources to check if those recommendations were correct or not, even if they’re required to by law, in the case of, you know, military decision making. And this is especially the case in military context, when operators typically only have a few seconds to make determinations on whether or not to act on algorithmic output. 

 

For example, with Maven, the military is hoping to reach the point where it can select a thousand targets in a single hour, and then they claim that a target excel of 20 people, can replace previous operations that had 2,000 personnel instead. So this creates the very conditions where automation bias would thrive, especially when you have things like Palantir’s platform Maven, that kind of obscure where the AI output really is or doesn’t really make it easy for you to trace or verify that decision. And either way, a lot of these models are, you know, have enormous scale, so they’re black boxes. 

 

So we’re kind of at the point where sometimes you do wonder if the distinction between decision support systems, as I was talking about earlier, or autonomous weapon systems, you know, are, is superficial in practice. Because if really we’re, the operators are defaulting to the recommendations that the AI algorithm is making really, it shows that, you know, the human in the loop is really not the solution here, especially when you pair it with the lack of reliability of these systems.

 

SREENIVASAN: You know, what’s interesting to me right now is that there’s this back and forth between Anthropic and the Pentagon. And the core of the argument seems to be at least publicly reduced to the idea that Anthropic is saying, We don’t want these models used for autonomous weapon systems. We don’t actually think they’re accurate enough, and we don’t also want them used in the mass surveillance of U.S. citizens. 

 

My question is, are they reliable enough for the decision support systems that you’re mentioning in the surveillance and intelligence gathering in the first place?

 

KHLAAF: I mean, that’s a fantastic point. You know, when you consider automation bias with their lack of accuracy and the CEO of Anthropic himself admitting that these systems are not reliable, then it’s very much the case that if they believe their models aren’t reliable enough for autonomous weapons systems, they’re also not reliable enough for decision support systems. And we should be questioning altogether whether or not these systems can be successfully used in military settings, especially targeting. 

 

SREENIVASAN: So there was a horrible, horrible mistake on February 28th when a missile hit an Iranian girls school in southern Iran. It killed more than 170 people. In the preliminary investigations right now show that the U.S. was responsible. And I wonder if — was this an intelligence failure or was this an artificial intelligence failure? And how will I know?

 

KHLAAF: Well, the lack of clarity surrounding the situation of whether or not AI was used in the school case actually touches on a very important point that shows how AI models make it really easy to obscure accountability. Because the use of these system makes it difficult to distinguish if these civilian attacks were in fact deliberate, or due to intelligence failures, or due to the lack of AI accuracy itself, as you said, or could be a combination of all three. For example, the AI could have been used to determine this intelligence based on the data it was given, and then that intelligence was then used for targeting. But the black box and inaccurate nature of AI makes that really, really difficult to determine. And a recent investigation actually showed that a strike on a civilian Iraq in 2024, the US Central Command admitted to not knowing whether some strikes were in fact AI recommendations or not. And if the Department of War is in fact deliberately not recording when AI-based decisions are being used, then it shows that AI is really being used to muddy the accountability here, especially for the liability of the decision makers in the chain of command.

 

SREENIVASAN: Wow. Because if a human being was directly found responsible that they intentionally — we would have, there would be a consequence, there would be somebody or some chain of command to hold accountable. But you’re saying that right now, all the people in that chain of command could be well-intentioned, not intending to, of course, strike a girls’ school, but say, this is the intelligence we were presented and based on this intelligence, this is the action that I’m supposed to take.

 

KHLAAF: Exactly. And I think there’s a larger question about the involvement of these companies as well. Because they’re the ones that are taking military data and fine tuning their models towards that, right? So who’s ultimately responsible here? Is it the people who provided intelligence data? Is it the intelligence data that could have been, that AI could have been used to essentially create? Is it the people on the ground who then approved a recommendation but for example, weren’t given enough time to check if that recommendation was in fact accurate? And so this is really the core issue that we have with AI and the lack of accountability. And it could have been very much the case that it was deliberate, but we still wouldn’t know that. 

 

And, you know, I think it’s very concerning that we have the US CENTCOM essentially admitting that they’re not recording that. And it’s quite a trivial engineering feature to implement, you know, if an AI recommendation was being made or not, right? This isn’t a difficult engineering problem.

So I think it should give us pause and make us question how are militaries using AI to also evade accountability, right? And even if they’re not trying to evade accountability, what if something goes wrong? Who is responsible here? Especially when you have a lot of the operators themselves not understanding the failures and the lack of accuracy of these models, I think that puts responsibility on them that they’re probably not prepared for.

 

SREENIVASAN: You know I’ve gotta imagine that part of their pitch to the Departments of War in any country that they might be working in would be, listen, I can help save lives, right? I can help you prosecute this without putting boots on the ground. I have now intelligence systems that will help you target, that will help you find exactly the right targets that only the military invest — kind of installations. And I can minimize civilian harm. What’s wrong with that?

 

KHLAAF: Well, I think that the angle that they’re actually selling, you know, in combination with what you just said is speed, right? They’re saying that you don’t have to put boots on the ground. It’s because speed gives you an advantage in these types of military operations. And I think it’s actually very dangerous that speed is somehow being sold to us as strategic here by these companies. Because large language models, you know, can just become a cover for indiscriminate targeting when you consider how inaccurate they are, right? And so you’re not only just muddying that accountability, you’re using AI to legitimize the speed in combination with their low accuracy rate, and it might just become a high tech version of carpet bombing. And so I think militaries need to be very careful in assessing the claims, you know, that these AI companies are putting forward.

 

For example, I actually believe that defense standards are some of the most strict and rigorous standards that there are, right? They require very high reliability rates for a reason, right? Again, lives are at stake. And also if military equipment fails or you’re overusing your missiles for civilian targets, that’s not an advantage for you in warfare. And yet here we are, right? Being told by these companies that this is an advantage and we are signing away these contracts where we’re no longer having that rigorous defense oversight. These companies are often grading their own homework, right? And so they’re saying, we will implement this new system for you and we – ’cause we’re the only people who understand the system – evaluate it for you. So we’re actually moving away from these rigorous independent verification that defense used to carry out during procurement process and just believing what these AI companies are saying. 

 

SREENIVASAN: You can already see that there are these competitive forces that are also affected by speed, right? I mean, there was a recent statement from the chief science officer of Anthropic who said, you know, they basically decided to drop their flagship kind of safety pledge. They said, “We felt that it wouldn’t actually help anyone for us to stop training AI models. We didn’t really feel with the rapid advance of AI that it made sense for us to make unilateral commitments…if competitors are blazing ahead.”

 

KHLAAF: Well, I think, you know, just like many other tech companies that have come before them, OpenAI for example, or Google, they always end up sort of dropping their safety pledges. And Anthropic themselves or justifying their rollback, you know, by claiming that their rivals didn’t adopt similar measures which forces positions. And this really implies that they believe they’re sort of the rightful developers of capabilities that they themselves admit will accelerate the arrival of the very risks that they feared. And I think, you know, it shows that these voluntary policy, again, co-op the safety terms to give a veneer of safety, but ultimately it was never sufficient to guarantee any meaningful safety guardrails. And that is exactly why we’re meant to have independence and oversights over what these companies are doing. because for them, they can just look at the term safety and change it to mean whatever they think it’s suitable at the time. 

 

So for example, in the case of Anthropic, they overemphasize on what they call CBRN, which is the AI having capabilities to develop chemical, biological, nuclear and radiological weapons. And their entire safety framework was sort of based on that, when you should be much more concerned with, you know, that the targeting accuracy if you’re putting these models in sort of military decision making. 

 

And so I think we need to be careful when they’re putting forward this idea of safety. 

 

SREENIVASAN: Chief AI scientist at the AI Now Institute, Heidy Khlaaf, thanks so much for joining us.

 

KHLAAF: Thank you for having me.