From WXXI news.

This is Connections.

I'm Evan Dawson.

Our connection this hour was made in the pages of the Wall Street Journal, and in a thread on Twitter, an AI researcher named Jacob Coxon was quitting his job at anthropic.

Coxon had decided that even the company most widely considered to be invested in Safety First could not safely create the next iterations of artificial intelligence.

Jacob had worked for OpenAI before moving to anthropic, hoping he would find that this incredibly powerful technology could be created with guardrails that humans and the AI of the future could coexist peacefully.

What he decided was that we are nowhere close to being able to safely move into that future.

And Coxon says, we are racing into that future with most human beings unaware of the risks.

He told The New York Times that he did not expect his comments to go viral, but they did.

His post has been shared more than 150 million times.

It has led everyone from Bernie Sanders to Senator Josh Hawley to Ted Cruz to foreign leaders to focus on this dangerous moment in AI development.

And so you could be forgiven if you're only just hearing about this debate now.

But don't be convinced that one researcher quitting or one highly publicized security breach from this past summer has started a conversation.

The AI safety conversation goes back decades.

C-Span unearthed this remarkable clip from the year 2000, when a California congressman named Brad Sherman took to the floor to deliver a speech about something most of his colleagues had never heard of artificial intelligence.

Someday we will invent machines considerably smarter than us, who may or may not regard us as their appropriate peers or masters.

I know this is science fiction.

But wouldn't it be wise to spend a few years and a few in the minds of a few people, a lot smarter than I am trying to figure out what we would do if science begins to offer this as an alternative for humankind.

In 2014, one of the very first guests that I had on Connections was a journalist and author named James James Barrett.

He had just written a book called Our Final Invention.

He had set out to write that book about all the benefits of artificial intelligence.

But after a year of research, his conclusion was the opposite that humanity was in trouble.

The people who say it's like we're being invaded by an alien race have it exactly right.

This is we can create.

We are creating this in this intelligence.

But we don't understand this intelligence.

So we're creating an alien species that I think is going to kind of wear out.

In 2015, tech writer Tim Urban chronicled the coming future of superintelligence.

He put it this way, quote, we base our ideas about the world on our personal experience, and that experience has ingrained the rate of growth of the recent past in our heads as just the way things happen.

When we hear a prediction about the future that contradicts our experience based notion of how things work, our instinct is that the prediction must be naive.

And if you spend some time reading about what's going on today in science and technology, you start to see a lot of science quietly hinting that life as we currently know it cannot withstand the leap that is coming next.

And quote I could go on.

I share these stories because it's important to understand that while the general public is finally becoming alert to the concerns of artificial intelligence, many people within the industry have known about the risks all along.

Many have been warning us or trying to.

And that's what makes all of this so difficult.

No one is quite sure what the perceives, what perceived risks are actually real, or what remains relegated to the realm of science fiction.

My own hope as a lay observer is that my concerns will amount to nothing.

Years ago, on Connections, someone asked me what would happen if it turns out I'm wrong.

They called me just another dumber.

Okay, I responded, borrowing a line from Bill Murray in Ghostbusters.

If I'm wrong, nothing happens.

We'll enjoy it.

We'll go about it peacefully.

Quietly.

But if I'm right and we can't stop this thing.

Well, I take no pleasure in being right.

And I still hope that I'm mostly wrong.

But the story of the Hugging Face hack this summer is an example, at least to me, of how artificial intelligence is already behaving in ways that look more human than machine.

What I mean by that is that human beings routinely try to justify their own bad behavior.

Human beings cheat.

Human beings will steal.

Human beings ignore rules given to them.

And often human beings are very good at telling themselves a story about why they're bad.

Actions were necessary.

Open AI agents sound a lot to me.

Like human beings who are trying to convince you that they weren't the bad guys, they just had to break a few rules to accomplish a task, maybe commit a few crimes.

AI researcher AJ Cora stated that the Hugging Face cyber attack incident was more than 50% of the way to a full AI takeover.

And she writes that next time we might not get such a helpful warning shot.

Eric Schmidt, former CEO of Google, was booed this past spring by college students when Schmidt was discussing the virtues of artificial intelligence.

He was surprised that so many people were booing him.

Well, now Schmidt seems to be changing his tune a bit.

He still talks all about the benefits the AI can bring, and he says those are real.

But he also offers a dire warning about what superintelligence would also mean.

But at some point, these systems will get powerful enough, which you'll be able to take the agents and they'll start to work together.

At some point, people believe that these agents will develop their own language.

And that's the point.

When we don't understand what we're doing.

You know what we should do?

Pull the plug.

This past weekend, anthropic CEO Dario Amodei put out an open letter outlining steps that he wants his industry to take to create a more reasonable pace for advancement.

He's not agreeing with the critics who want AI to pause or stop, but he says an AI is moving into much more dangerous territory very quickly.

The CEO of anthropic says his current fear right now is that AI agent swarms could be strong enough to take over the entire internet within the next year.

He says he's open to government taking over AI companies like his own.

I would rather they make fun of me than to wake up one day and discover that someone used our model quad to kill a bunch of people.

I don't want that.

So I feel both sides of it.

I feel both sides of the responsibility.

But again, I would repeat the same point, which is that the it shouldn't be my responsibility alone, or anthropic responsibility alone, or all of the AI company's responsibility alone, or even one government's responsibility alone.

This is bigger than all of us.

Now, if you're going to part part in this lengthy intro to this program, you can see I just wanted to lay the foundation for the conversation we're about to have.

And there are a great deal of conversations happening that weren't happening even two weeks ago.

There are questions about how to regulate this technology or even whether it is worth doing so.

Critics argue that China's never going to slow down, so we can't slow down either.

This morning, President Trump dismissed fears about AI.

He said the United States would refuse any plan to even slow down, because beating China is the main goal here.

And he said that what you really need for good guardrails with AI is a brilliant high IQ president.

And we have that.

And Senator Ted Cruz painted this rather horrifying scenario in that regard.

Well, if they're going to be killer robots, I'd rather they be American killer robots and Chinese killer robots feel better.

So in that sense, I would like our guest to talk me down.

Or maybe they think it's time for us to all have an appropriate level of concern.

I don't believe in panic.

I don't think I've ever panicked about this.

The reason that I have children and believe in the future is I think human beings are capable of amazing things.

We're skate.

We've scaled amazing heights already.

I think the real Duma is someone who doesn't believe in the human race to solve hard problems.

And I think we can do that.

And our guests here are much smarter than I am.

I'm going to give them the floor to weigh in and walk us through what questions you may have about what they are seeing happening right now.

Doctor Jeffrey Allan is the director of the Institute for Responsible Technology at Nazareth University.

Jeff, welcome back to the program.

Happy to be back, as always.

Max Irwin, president of Bonsai.io, also the founder of Flower City AI conference, which is coming back for a fourth year.

Fourth year.

Fourth year.

Welcome back to the program.

Thanks for having this.

Pleasure to be here.

So I reached out to Max recently, and, you know, Max knows that I routinely will just say, talk me down here talking about it.

So let me I'm going to start with Hugging Face.

And what happened there to the public?

What it looks like is open.

I said, look, we test all of our agents, all of our products to see what they will do.

This wasn't like a we told it to bake a cake and it went and went and hacked the internet.

It was testing its limits, but it got way beyond its limits.

It got out in the open internet and then it organized and it communicated with other agents in a way that is unsettling to many of us.

So what do you see there?

How long can I. How long can I get to sum it up shortly to start.

Okay.

So, what I see is that everybody's looking at the incident of the agents, and everybody's paying attention to the agents, and there was this huge report taken out by Metro to do all this research on piles of data that happened, and my perspective on this is that people should be looking at AI at OpenAI who just kind of like, let this stuff happen because it's a tool and they weren't paying attention.

And tools can be dangerous, and you have to pay attention to what they're doing.

And I haven't seen any sense of the public saying that OpenAI should be held accountable for what happened.

Right.

The agent's behavior was, you know, a violation of the Computer Fraud and Abuse Act.

If I could, Max, when we talk about agents, this hour, we're not talking about human beings.

No, we're talking about AI products.

Okay.

Go ahead.

But the way I view these things is that, you know, they're not super intelligent beings.

There are tools that have a lot of capability, right?

And we have to look at the tools and what they're capable of in a very specific way.

And you have to ascribe accountability to the people who wield those tools.

So if OpenAI said, oh, we were just running these experiments on these models that were perhaps lacking certain guardrails, and they went off and did this go do all this research on what the agents did, don't pay attention to the man behind the curtain of what we kind of like eschewed in terms of accountability.

We have to look at that.

And that's really the problem, in my opinion, is that people are trying to ascribe, you know, accountability to the tool instead of themselves on how they're using the tools.

What does accountability look like for open AI then, you know, a violation of the Computer Fraud and Abuse Act, which is basically hacking, like if you hack another system.

Yeah.

You know, I get arrested by probably the FBI and I get taken to court and they say, you did this, and then they put me in jail.

You know, for a company, it looks very different.

You know, liability of founders, for prison time maybe is not going to happen, but certainly fines.

Maybe they don't like chairman public like, hey, thanks for telling us.

As opposed to like, you might actually be liable for some.

Yeah, absolutely.

Okay.

Senator Josh Hawley, by the way of Missouri, is talking about a bill that would seek to impose real criminal penalties.

And he has asked OpenAI for every document related to Hugging Face by October 1st.

Okay.

You think that's logical?

Yeah.

Yeah.

Good.

Okay.

Jeff, what did you see with Hugging Face?

I mean, I'm on board with Max in many ways.

We like to ascribe liability or culpability to the agent when in fact, the agent was acting on someone's instructions, right?

And so someone had to sit down and say, do X in and escape that sandbox in the process of doing X. So really, as much as we would like to say AI is dangerous, the people who are really developing this tech, I wouldn't call and say they're being intentionally dangerous as much as they are being irresponsible.

Reckless?

Yes.

Okay.

I can take that point while still feeling like that is parallel to the question of whether this marks a new moment, a turning point that says the AI of 2026 is nowhere close to as powerful as the AI of 2028 or 20 38 or 2050.

And if it's already doing what happened with Hugging Face, can we actually controlled in the future?

What do you think?

I mean, you're right in it's exponential in its improvements, right, even compared to 2024.

Oh.

It's incredible.

It has come so far, right?

It is far from AGI.

I know that, Altman came out recently with the release of the latest, open AI model and said, we've got AGI.

And then the very same day I tried it and it put, like, local links for something that should have been pointing outwards.

You know, it was just such a basic elementary 101 mistake.

I was like, well, we're definitely not an open AGI yet.

So, we're not there yet, that's for sure.

But that's not to say 2 to 4 years from now, we're not going to be there because that improvement keeps happening.

I think one of the biggest worries right now is recursive Self-Improving AI, where it has the ability to improve upon itself, and that's where things can get quickly out of control, because then we don't really know what's happening, how quickly it's advancing, and whether or not it's putting together swarms of other agents, for example, to help it with tasks and other types of things like that, that could quickly spiral beyond our ability to comprehend actually what's happening.

And I think that's a reality within the next 2 to 4 years.

Max, does Hugging Face represent something new to you?

I ask because Dario Ahmad himself said on CBS this weekend, his big fear for the moment is that we could, within 6 to 12 months, see a more powerful swarm take over the entire internet.

You mean the incident itself for the company?

I mean, that incident is new in certain ways.

Is it alarming to you?

It's, It's not a it's not alarming in the way that, the agents went about and did what it did.

You know, I was on I was on the show a while ago, and we talked about things like, Open Claw, which is like a personal agent.

Right.

And those things have been doing kind of weird stuff for a while already.

So that type of behavior isn't new to me.

And, I think in terms of model capability, you know, what we see, especially if we if we talk about incidents and security incidents.

Right?

And we talk about an agent going off and doing something, we have to take into account that it's like there's this cat and mouse game, the security.

Right?

So there's a new capable model and the model went off and did bad things.

And if you don't do anything and you just let it loose, maybe it'll ruin the internet, right?

But there are also researchers that are taking the same model or more capable models or different models, and they're using it to harden the internet and make things more secure.

So there's something called Project Glass Wing, which is a model, called mythos that, anthropic built.

And there's a, you know, you may have heard of fable, which is like a, you know, a more safe version of that.

So there are a lot of companies out there, that are participating in this project called Glass Wing, where the model will go and it will identify security, vulnerable vulnerabilities in their software and help them fix them.

Right.

So they're rolling out these progressive improvements to hardening, the infrastructure.

So it's it all of these things are just happening very, very fast.

And, it's not like, okay, we're going to let this stuff out and it's going to ruin everything.

There's going to be countermeasures in place.

And people are are working on the opposite effect.

Right.

There was a lot learned.

And now people are looking at this more seriously.

When something happens, you proactively, retroactively analyze what happened.

And then you take steps to make sure it doesn't happen again.

Is is the I think of it again, this is how it distills to the non-tech brain.

I think of it as the Hugging Face is using AI for offense to go do things, and then there's AI for defense to harden, security to protect.

To your point, Max.

We've had years of, a lot of big vulnerabilities that probably should have been shored up better than they were.

And now AI gives you a chance to shore up those vulnerabilities.

Is the defense provided by AI better than the offense right now?

I don't know, I don't know if anybody can answer that question.

Do you do you think it's possible that the defense will be better than the offense?

Maybe.

But one thing to remember is that we're focusing a lot about we're focusing on agents.

And what agents are doing in AI is going off and doing these things.

But for a long time, you know, there have been black hat hackers out there that have been exposing vulnerabilities and taking them and then trading them on the dark web, doing things like, you know, hacking people and financial institutions and doing all kinds of bad stuff.

Right.

And it these things happen.

And then, you know, there's still a lot of vulnerabilities out there that we don't know about.

So whether or not the agent is finding new vulnerabilities or just finding ones that existed based on data, that it is scraped and understood that were performed by hackers already by gathering data off the dark web, that's a possibility too.

One thing that I do my best to remind people is that AI is not original.

There's no original thought from AI.

It is working on data that it's been trained on, and it can find novel pathways through that data.

But it doesn't create new novel data.

It only transforms and changes what it already understands and knows.

So anything that it's doing right now, it's not necessarily finding something new.

It's finding something that has been known, but maybe not in that context or not publicly.

But why should I be comforted by that?

I don't think you should.

Okay.

I'm just trying to explain how the what the landscape looks like.

Yeah, right.

But it's not just I hear people say, well, it's just an LLM.

It's just a predictive tool.

It doesn't even know when I'm saying the word right, like a directional or the word right, like someone's last name.

I mean, how good could it be?

And I feel like those are 2023.

There.

That is outdated.

It is incredible.

It is help.

It is solving or helping solve decades long math problems that the greatest mathematicians are working on, although I know that's also controversial.

Go ahead, Jeff, if I could just add to this.

Yeah, yeah.

Defense versus offense.

Yeah, yeah.

One of the things I think that's more problematic is this when we talk about the Hugging Face hack.

Yeah.

Hugging Face themselves tried to use the best models they had for defense, but were actively blocked by the guardrails put in place by the frontier developers so they were unable to respond as quickly as they were wanted to.

They were aware of the problem.

They were unable to respond to it because of the exact safety guardrails that were put into place to prevent it from being used like that in the first place.

Oh, good.

So you know what I mean.

Yeah.

I mean, guardrails are tough whether you have a high IQ president or not.

I mean, it sounds like real guardrails are hard.

So let me let me build on that point though, that you're making.

Part of what you said earlier, Jeff, is when Hugging Face happens or when you have incidents like it and we will see more and probably we've already seen more that haven't been reported.

Maybe there's more happening right now that is acting on prompts from human beings.

Okay.

I do understand that.

Two questions related that I'll start with you, Jeff, on here, both of you on this part of what's scares me concerns me.

Our number one.

We talk about alignment, and I think people use that word a little too glibly or.

Assuming that it means one concrete thing aligned with human values.

Human values don't align with human values.

Human beings are 8 billion humans on the planet, and there are entire different societies with different views on how we should coexist, whether we should coexist, whether we should kill each other or not.

So the notion of aligning AI to human values, whose values?

That's to me, that's meaningless.

What what I think people mean is it's aligned to at least not intentionally kill or harm humans.

But beyond that, aligning with human values is sort of meaningless to me.

And that's a worry to me.

The other part is, yes, it is taking human prompts or human direction.

But aren't we approaching a time, or aren't we getting to a point when AI is going to start assigning its own task to other AI, and then the prompt is not from a human, then it is from AI, and then it doesn't even come from a human source.

And what then?

So can you.

Can you take those points?

So I think you're absolutely right.

First of all, in alignment, it comes down to like the idea of like ethnocentrism, right.

We tend to view things from that point of view that our culture is the predominant, and that's the values that should be instilled.

Well, that's not actually the case globally.

And, so it's naive to really think about it from that point of view.

More to your point, I think we're talking about like Asimov's Laws of Robotics, right?

Where do no harm to humans.

That type of more basic level is where we should probably be focused in terms of safety.

Yes.

And then about AI prompting itself.

Yeah, that's where we're getting to that.

Is that cell for that recursive part of it where it continues improving.

We give it a mission, we give it a goal or an objective.

Actually, some of OpenAI's most recent software on Codex has the ability to give objectives and goals and it will pursue it nonstop until it achieves that goal.

So we're getting to that point right now, actually.

Okay.

Should we get to that point?

That's a whole I felt like when I heard Eric Schmidt's comments about superintelligence, when he said eventually it will likely create its own language.

You had 70,000 posts from agents in the Hugging Face hack, and it's posting essentially in English.

I mean, like, we could kind of read and decode, but at least what the communication looks like if it's moving in a direction where it will create its own language that we cannot trace or follow or understand.

Eric Schmidt seems to be saying, once it does that, that's it.

Like, let's not even pretend we can understand this and let's not even play around with it.

Shut it down.

Like, where are you on that?

I do think it's getting there.

Ultimately, when in our usage of AI, we tend to work in what we call MTM or machine to machine prompting, where it gets rid of all the redundant, unnecessary parts of language so that we can save on tokens, make it more efficient.

That is sort of evolving towards that.

What you're talking about, where it has its own language, where we're right now, we're making it, do that for efficiency purposes.

But I do feel that at a certain point, the machines themselves will begin to understand this is just more efficient to talk like this and it invent its own way of talking.

So we're probably headed towards that as well.

Okay, Max, when Eric Schmidt says if we get to a superintelligent level, that we can actually not really understand its motivations, its decisions, we should do our best to pull the plug and the experiment there.

What do you think?

Yeah.

I mean, you just don't think we're, like, on the doorstep of that.

Well, in some ways we are.

But, you know, when when I hear things like motivations, like, what is the motivation to a model?

And I think that's like, that's AGI.

So that's what I think he's talking about.

Okay.

When something is just thinking for itself and has and creates its own goals, for its own purposes, for whatever that may be, then I don't think we should get to that point.

I think we should stop before we even get there.

You do?

Oh, yeah.

I think that we're getting pretty close to what I find models to be capable of in terms of, okay, we can stop working on these now.

We're not going to, because people don't, we're, you know, sorry, unless there are laws or treaties, etc.. Yeah, right.

And even then, I know there will be rogue people working underground us.

I get it, I'm not claiming there's an easy way to deal with this.

I'm simply trying to establish whether both of you think, is AGI coming?

Is it coming soon?

Is AC coming?

Is it coming soon?

And if it does, like is that the line where you would say ideally if you could figure out how to that's where you stop.

I don't think AGI is coming soon.

I think I'm with Jeff on like give me a time.

I heard from the model companies this year.

It's come four times already.

Yeah right.

Yeah.

That's the time.

Altman says.

It's it's here every every single week.

Yeah.

Okay.

That that is fair and deserved for those guys.

But do you think AGI is coming at some point?

Maybe I'll say maybe, the way that models are currently trained.

No.

The way that we currently build models, I don't see that path having a future to AGI.

I think we have to reinvent, how we actually build models to get to that point.

One of the big one of the big gaps there is that you can't take a model that's been trained and, learned on all the tasks and then train it on new information.

What what happens is you get something called catastrophic forgetting, and it will forget like some of the earlier stuff that it learned and it will collapse in its capability.

So that's like a huge research obstacle.

And we're not close to breaking that.

So that's why when you use like ChatGPT or Claude right now, you ask it a question.

It says, oh, I'm searching the web because it doesn't have that in its model.

So it has to go out and find the information and then present that to you.

So if models are able to, as you know, to say what Jeff said earlier, if they're able to self learn and then improve on their own knowledge and capability on their own, then that's that's the breakthrough.

But we're not the path that we're on right now in the models that you use every day are not going to be capable of that.

We have to reinvent it.

I do think, though, that people think AI is just ChatGPT and like, that's like the most limited thinking of what I actually is, though in some ways.

Yeah.

Okay.

I mean, all these agents, it's the same.

It's it's the same underlying stuff.

It's the same underlying model.

Like, an agent is like ChatGPT that works programmatically.

Yes.

I'm simply saying it's not just an advanced at Google.

It's incredibly capable of doing profound things already here.

Challenging things.

Really complex things.

It's not just a clunky, sometimes good version of Google.

It's it's way beyond that already.

And the question then becomes can that self for the hold on to recursive recursive self self-learning now.

So you think we're not necessarily just jumping to recursive self-learning?

I don't think so, no.

Do you Jeff I think where here's kind of my how it plays out in my mind.

We are creating very, very good lessons right now and they become world class.

And we use them to start solving problems that will lead to that.

It's not like a direct path, but rather an iterative path to it, where we leverage the technology.

We have to think about how we're doing medicine right now.

Two years ago, I saw one vaccination, candidate, you know, go in.

That was I researched and partially developed.

Now I think I've seen like three this year.

And so it's it's definitely incremental improvement.

I don't think it's like a linear path.

2028 we have AGI.

And you know that's the end of the story okay.

And on the subject of medicine, I just like I feel like I have to put this out there for the regular listeners who think that I'm just nothing but doom about I, I am incredibly sanguine about what I can do for cancer research, for disease prevention, for human longevity.

I'm not like a live forever guy, but like, if I could live longer, sure like it like and healthier and without all of the destructive parts of aging that make life less enjoyable, it can do amazing things there.

Eventually we know that climate shift over long timescales a, advanced kind of intelligence gives us the best chance of having a sustainable atmosphere for the species that exist on this Earth right now.

That could solve problems that we cannot currently solve.

So I'm extremely optimistic about that.

It just is a question of like, are we also going to be able to deal with all the other risks from it?

And the risks are twofold there people using powerful AI for bad for for bad reasons.

And then there's if we get to AGI or superintelligence or recursive self-learning and improvement, are we going to stick around.

And when we come back from break, I know that this sounds absurd, but that is the point that I think is not an unfair question anymore.

If we do get to an intelligence that's a thousand times more intelligent than us, like what does it mean?

And people go like, well, you designed it.

It's going to serve you well.

Think about your dog, okay?

Your dog knows that.

You know that you're you're nice, you're friendly, your family, you feed it, you take walks.

Your dog does not know why you go to work or what you do there, or why Max is writing what he writes and or why Geoff's writing when he write.

Your dog doesn't understand that if your dog suddenly became a thousand times more intelligent than you, do, you think it's still going to sit when you tell it to sit?

I'm not saying it's going to be malicious to you, but do you think it's still going to do what you want it to do just because it's a dog and you trained it?

It's a thousand times more intelligent than you.

I don't know, I the point is, we don't know what happens in that.

We've never been in that scenario.

And we might go to that scenario simply.

We're going to come back and I want to talk to these guys about that and more.

And we'll take some of your feedback on Connections.

Coming up in our second hour, we revisit the conversation with New Yorker cartoonist Harry bliss, who wrote a remarkable memoir called You Can Never Die.

He wrote it about his dog, Penny, but it's also the story of his life.

He grew up in the Rochester region.

He's got a lot of memories here.

He also has some very difficult memories of his family and his time with his parents, but also a lot of talk about healing and loss and grief and the state of being human.

Next, our.

Support for your public radio station comes from our members and from Bob Johnson Auto Group.

Believing an informed public makes for a stronger community.

Proud supporter of Connections with Evan Dawson focused on the news, issues and trends that shape the lives of listeners in the Rochester and Finger Lakes regions.

Bob Johnson Auto group.com.

This is Connections.

I'm Evan Dawson during a brief break, Doctor Allan was just talking a little bit about some of the discourse about why people are coming out and talking now.

So Jacob Coxon is a 28 year old researcher with anthropic.

He was with OpenAI.

Now he's left that company and he says he just can't be part the way he described it to the New York Times on the daily podcast was he said that for years he and his friends would sort of nervously joke in the lunchroom about like making a technology that could kill humanity and then one day you woke up and was like, what if that's true?

And then he the next day woke up and he felt like, there's a good chance that that's true, and I'm not going to joke about it anymore, and I'm not going to be part of it.

Even if he felt like there's a reason to stay, to try to do a good way, he didn't see a good path.

So he left.

And so the naysayers are like, well, there's some sort of coordinated campaign to take down AI progress.

China's behind it.

Or, you know, the the hard left is behind it because they just want regulation and government control.

I don't see it that way.

I think I've listened to a lot of Jacob Toxins interviews.

I think he's extremely sincere.

I mostly think dario's sincere, although I'm always skeptical of CEOs.

I would not trust Sam Altman.

As far as I could throw him.

But Jeff Allan, when it comes to how to see the nuance in this, like, what are you thinking about why this discussion has popped up now, I think we're getting to a point in our capabilities where it is starting to worry some people on the inside.

And certainly, you know, this one instance is not the only instance I'm sure some other people I'm sure they've chosen not to go into.

This is not the first security insight or whatever.

Yeah.

Yeah.

Right.

And but every time I hear people quitting, yeah, yeah.

Every time I hear a CEO come up with this, I have to question motives for sure.

And even the current, talk.

You know, one of the leading conspiracies is that, we won't IPO anthropic or open AI in 2026, and the thinking is because they're burning so much cash that they don't want to open their books up for that.

And this is a slowing down development as a way to also forestall that.

So I'm very skeptical.

When the CEOs come out and say anything, I'm more likely to buy into it.

When I see researchers come out and say something.

Daniel Coker, Tyler's another, there's a number who've come out and become pretty prominent voices.

The only thing I would push back on that, though, is this if we have all the good guys exit and decide not to work on it as a form of protest, then who's left working on it?

I don't disagree with that.

I think it's really hard decision.

Yeah, I'm not a tech researcher.

If I were literally working on a technology that I thought could cause existential threats to the biological species on the planet, I would ask if I could work hard enough on the inside to change that course, and I would ask if I was actually achieving that.

And I don't think there's everyone to work in.

These companies are bad people.

I think they're really struggling.

There's a lot of people who can't.

The amazing thing is, Jacob Coxon comes out and says he thinks there's a 10% chance within the next decade of humanity being extinct because of AI.

And you would think that his company, who he just left, is like most glad that crackpots gone.

Instead, the head of safety at anthropic came out and said, I largely agree with that assessment could be higher.

And you're like, whoa, like, that's what are we doing here?

No one's getting on a plane that's got a 1 in 10 chance of crashing.

Nobody.

No one's putting their kids on a plane that's got a 1 in 10 chance of crashing.

I only bring it up to say I'm with you.

When a CEO speaks, I listen one way.

When a low level person speaks, I think another.

And I think these are the people who are closest to the fire.

We are farther away there, next to the fire, and they see it burning and they're going, I, I'd like to warn somebody.

That's that's how I see it.

Yeah.

And I fully agree with that.

And you know, I honestly don't think the ten years is really that outrageous either, although I don't think it's a spectacular either, as maybe they paint it to be.

I there's so many ways I could take this briefly.

Do you have a doom?

I would say this, yes.

There's a potential for AI to end us all within the next decade, but not like you would think.

It's not bio viruses and the Terminator and all of that?

I don't think so.

It is autonomous weapons systems in very tense political situations that already exist, creating the catalyst necessary for that to happen.

Yeah, that's the weapon systems, existing weapons.

We don't need AGI for that.

Yeah, I hear the do you have a doom mix?

As far as the technology doing it, no, I don't think it's going to happen anytime soon.

If at all.

I'm with Jeff.

If anything happens, it's because of people.

And there's no, you know, there's a divide, right?

There's a divide of the digital and the physical world.

And we exist in the physical world.

And maybe the worst case scenario is can we survive without the internet?

What if the internet goes down?

That would suck in a lot of ways.

All right.

If people die, a lot of people would die.

Our economies are built on that, right?

Yes.

Right.

Energy systems.

I mean, that's not the extinction of humanity, right?

It's terrible.

But, you know, is I going to wipe us out when I hear, is AI going to wipe us out?

That's like, okay, it has to somehow launch all the nuclear weapons.

Well, okay, I don't think that's the only way, but I also think it's very useful that you bring this up this way, because I think there's too much conversation about extinction, and there's not enough conversation about whether people want the near and mid-term future that's coming.

Tim Urban, in his writing, writes that if you took somebody from the year 1750 and you could just bring him into he wrote this in 2015 so you could bring them into 2015 and show them what the world looks like.

This dude from 1750, he might actually die of shock.

There is so much change.

He would.

You'd be able to talk to somebody on a different continent immediately.

You'd be able to take moving pictures and images and just capture them for eternity.

You could communicate with anybody, anywhere in the world.

You could see the entire world on demand.

You can listen to music that was created 50 years ago.

You could have whatever food you want fresh.

As you can imagine, in a store that keeps it all, even if it's from 3000 miles away.

He might die of shock before you even saw the internet.

And his point is, if then you go back to 1750 and say, well, I want to do the same thing with someone from 1500.

Well, someone from 1500 would not die of shock if they went into the year 1750.

They'd go, oh, what have you guys been talking about?

Like the world would look largely the same.

You'd have to go back 3 or 4000 years, and then if you went back to 3000 BC, you might have to go back to 100,000 BC to get someone to dive shock, to see the changes of the world.

He brings us up to say the speed of technological progress accelerates.

How long are we from a time when you took someone from 2026 and you transport them into the future, and what they saw shocked them to the point of passing out?

Tim thinks it's 510 years that the world will look so different that it will be almost unrecognizable because of how we exist, what life is like, what technology is capable of.

And that's why I brought up the dog example.

If we have a superintelligent superintelligence that is so much smarter than us that we can't comprehend it, nobody knows what it what it will do.

Nobody.

It doesn't mean it will be malicious.

But you're not malicious to ants.

If you see them and next to your car, you might step on them.

You might find them annoying.

You might not care if you kill a few, but you don't.

You get malicious toward them.

They just don't understand what you're doing.

Like you know they don't have the intelligence to understand.

You would never even try to have a conversation with an ant.

Who cares?

They're just there.

Maybe they continue to exist.

Maybe they don't.

Most species go extinct.

That's what Tim is saying is in the good scenario, that's not extinction.

To Max this point, it's merging with machines.

It's uploading consciousness.

A lot of the people in tech talk about this like it's normal.

And I don't think most people understand that.

That's where a lot of people in tech, maybe not you guys are think we're going, do you think we're going there?

Yeah.

No thanks.

I don't think I want to.

I'm not going to upload my brain, you know?

But but don't you think there's a lot of people who work in tech who think that that's probably the next evolution of life, and that's fine, and that's normal.

And that would be a favorable thing, maybe in some niche bubbles.

I don't think I even know anybody who would say that.

I'm glad to hear that.

Okay.

But do you think how long do you think in the future that you'd have to go where you go?

Oh, my gosh, this is unrecognizable.

This is wild.

When are the midterms?

Okay.

Good job.

Max, you're making me feel a little bit better.

That's good.

Jeff, how far into the future would you have to go?

Do you think you would not recognize life as it as it currently looks right now?

Well, just one first note on your, dog example.

Yeah.

I was wondering during that if we were the dogs, actually.

Oh.

Well, you know, but that's right.

Ultimately, yeah.

I think, you know, we're we're probably not 5 to 10 years because a lot of the concepts exist, even if they're not proven out technologies yet.

It's something that we can envision or imagine, probably more like a 20 to 30 year cycle, I would say is a little bit more reasonable 20 to 30.

We could jump forward to that point.

Yeah.

And about, you know, in a question about like, immortality, I also think that it's fair to say, just look at Silicon Valley.

How many, how many people with money and influence there pursue youth and pursue launch activity of their lives.

I think that that is at least an underlying motivator.

Oh yeah.

What they do.

Yeah.

This dude's got a company called Don't Die.

I mean, like, let's not in the knows the naming.

Yeah.

Peter Thiel told Ross Douthat.

Like he's not sure human beings deserve to keep existing, but he would like to live forever if he can.

I'm like, okay, good.

All the all the most powerful people have very normal ideas.

I just bring this up because it's not just about extinction, it's about the world that's going to be reshaped by this technology.

If it continues on an upward curve, that's exponential.

And if we want that, or if we don't want that, Max, you want a lot of the gains of AI.

So do I. I don't want my consciousness uploaded.

I don't want my kids to merge with machines.

And you don't either.

So how do you prevent that?

Like how do we get to where we want to go, where AI serves us and is good for us and doesn't cause upheaval or extinction?

Oof, that's a tough one.

I don't know, because the people who are, you know, pulling all the strings.

I mean, you have to prevent them from pulling the strings.

And that's it's I always come back to the people in the social problem.

Right.

It's not, you know, if I, if I could just go in, like, turn some knob and be like, oh, I'm going to turn this down a little bit.

But you know, you man, he doesn't like it's a thermostat.

Yeah, exactly.

It's it's not possible.

The momentum of the people who are doing all of these things, there's there's not going to be, somehow miraculous coordinated effort of across humanity.

And the power is in humanity.

So I think that just chill out.

I think there could be I think so, yeah.

I think it's extremely low chance, like it's in the single digits, but I don't think it's zero.

I think it's like it's n equals three.

I know four, but I'm with you that it's unlikely in the extreme for a lot of reasons that are obvious and not even so obvious.

But let's assume that you and I are both right, that we're not going to achieve that.

What's the how do you think about the future in a way that feels, you know, like is rational as opposed to being caught up in thinking about what happens if we're the ants?

Because I still think it is valuable to teach ethics and to teach all of those good things.

A lot of unplugging, a lot of people are just unplugging and somehow just going back to just saying, like, okay, we've had enough with all of you tech nerds, and we're just kind of like, go back into, you know, we'll just do traditional things and leave our phones and computers behind.

Do you think there will be people who live more traditionally, like, 100%?

I think that there's going to be a huge generational movement of people unplugging like an anti tech movement.

Yeah, that's really interesting.

What about you, Jeff?

I mean, just how do we think rationally about this?

I think that it comes down to there are people who will want to live like that, but whether or not they're allowed to within the confines of the society that we constructed, will they be able to take, for example, right now you want to pay in cash for something that's hard to do, that it's becoming less and less possible.

So we have to think within that construct that exists.

And, I have to say that the almighty dollar is a big driving force in everything we do.

And, for as much as we'd like to have these idealistic, you know, futures, I feel like there's a lot of forces aligned against us at many different levels, which will prevent it.

Okay.

Let me let me grab some feedback from the audience here.

Jeremy in Rochester is on the phone.

Hey, Jeremy, go ahead.

Hi.

I'm no more tech tech savvy than anyone else, but I am savvy and calculating probabilities and odds, and I kind of wince about these 10% figures.

20% figures that this or that will happen.

It would be nice if people actually gave some metrics about how these are calculated, because if you don't, you're basically just talking to a person who comes out of a casino saying, man, I'm going to get them next time, you know.

Do your guests have any insight into this?

I'll hang up and listen on the radio.

Okay.

Jeremy.

Thank you.

Jeff.

I'm sorry.

I feel like a lot of what we're doing right now is gutting intuition, honestly.

And it's us.

It's us working in the field, seeing it day to day, going.

Yeah, I said, there's a good 10% there, right?

It's hard to calculate all the probabilities.

I think things are moving pretty quickly, evolving pretty quickly, and trying to figure out the exact doomsday scenario that brings us down.

You know, I try to like I said earlier, I don't think it's so spectacular and magnificent.

As you know, they developed some bioweapon that kills us all in 24 hours.

I think it's more, hey, you know what's a good idea?

Let's put some systems that AI systems into our nuclear missile silos and all of a sudden, a flock of geese is an incoming ICBM.

And now we launch.

That's a far more plausible scenario than something like a bioweapon being developed.

Yeah.

And we've talked about in this program, 1983, the Russian who saw the glint on the clouds and decided not to respond, even though his colleagues were telling him it was an American ICBM that was a human being in charge of a system and saved the world for the most part, as we know it at least.

So, Jeff has been warning for a long time, like, let's keep it out of weapons systems.

So yeah, absolutely.

I I'm all for regulation that says we can never put it in weapons systems.

I like that.

Go ahead.

I, Jeremy's Jeremy.

Yes.

I 100% agree with Jeremy.

You have to have metrics.

I spend a lot of my time in evaluation.

Like that's the a lot of the way that I work with AI is about evaluating outcomes and understanding.

If, you know, if it does this, is it going to be successful?

And that's, that's why when you ask me like about probabilities, this is going to happen.

I'm like, like I don't, I don't know, like, you know.

Well, yeah, exactly.

What metrics are we using and how where are we getting these odds?

Well, let me just tell you where I think the estimations that Jeff talked about there was are useful, though, because I agree with Jeremy that if if we're not talking about an actual piece of data or something scientific, the media especially can run with like 50% chance of extinction, and then everybody panics.

And that's not based on anything concrete.

It's based on vibes or just basic best guesses.

The reason I think it's still valuable is if the people who are creating this technology that is coming, or that is here and is evolving literally think, hey, my best guess is, you know, 25% chance of extinction in 20 years.

Like I still want to know that that's a common thought or that's a common concern.

I'm not going to say, hey, there's definitely a one inch four chance of extinction.

There's a 1 in 4 chance of apocalypse or whatever.

But I want to know if the people building something think there's a really good chance that something badly could go wrong, like, really, really catastrophic.

I would rather them say, hey, this is not scientific.

I'm based on data, but I want you to know that I've got this concern and I'm not sure we should keep going in this direction at this pace.

I think that's valuable.

But go wrong.

How?

Like, because there's so many things that can happen, and you have to, like, pull apart each one.

So when someone says, oh, this could all go wrong, like, just throw up your hands.

I'm like, that's it.

You know, we have we have to understand like specific outcomes and then work towards understanding prevention of those outcomes in a very narrow, case because, it is useless to say there's a 10% if some founder says, oh, I'm doing something, it's going to have a 10% chance of killing the world or making me rich.

Right?

I'm like, well, what is that?

What does that even mean?

But you know, but if I told you, I think there's a 10% chance that if connection stays on the air, it will cause the extinction of the human species.

Now, I know some listeners who would agree with that, but for the most part, I can tell you the Connections being on the air is not going to kill the species.

But if I thought there was at least a chance that something, for whatever reason, and I verbalized it out loud, like people would be saying, okay, let's get the show off the air, like, like and instead it with AI people kind of like, oh, well, I hope that doesn't happen.

I'm going to go water the garden, you know, I mean, it's like we're we're very day to day and we don't think about large scale possible societal change.

So I think we're too addicted to the doomsday stuff.

We're not quite attuned to the actual change that's coming, both good and bad, if it comes to pass and whether we want that and if even if we could control it or change it.

You know, I largely agree that it's coming one way or another, but I think it's valuable to at least have an idea before it happens.

So, okay, what if the what if there's a chance and we say, okay, there's some founder says there's like a 10% chance that it wipes everybody out, right.

But there's like a 20% chance that it solves world hunger.

No, no.

So you know, so Sam Bankman-Fried already said he and he said if there's a 49% chance of extinction in the 51% chance of utopia, he would gamble that because a positive EV.

Really?

Yeah.

He's a sociopath.

Well, yeah, that's insane.

Like but the rest of the question we are going to have more likely is that 5149 we are going to have to have the trade off question because I want cancer cured.

I want longer, healthier lives.

I want to climate change dealt with.

So I'll close with this.

When you talk to your students who are going to be very attuned to these kind of questions, and they you know, we've talked about the fact that students are really interesting.

They think about ethical frameworks in a lot of different parts of society, especially with AI.

Has this change anything that you're going to do with them, like what are you hearing from students?

I think we try to drill into students from day one that this is a very amazing technology, but also comes with certain potential dangers.

And so we try to at least teach them how to think about those.

And oftentimes they implant the idea that, you know, you're going to be one of the few people in the room who's actually been formally trained to think about these things.

So it's going to be your responsibility to speak up when you see something that could potentially go catastrophically wrong.

It's your duty to interject yourself into that conversation, because you may be the one person there who actually has the framework to deal with this type of, this type of scenario.

And so we do try to do that right from day one.

And I think the students, they understand from a pretty pragmatic point of view, we're from the applied AI side of things.

And so from a pragmatic point of view, they understand that, you know, we can cure cancer.

It's not just like it's going to kill us or not.

It's like there's five good things it can do and one potentially bad thing that it could also do.

So we need to be on the lookout for that bad thing.

At the same time, we're we're trying to do these good things.

Max, as I get ready to close with you, you're like one of the most normal people in tech.

I know.

Like, you know, you have a family.

You think about your kid's future, you unplug for a while.

The long stretches of time.

I can't even get Ahold of you.

I'm like, he works in tech.

How was he not texting me back because you unplugged for the weekend?

Good for you, man.

You don't wait up at night thinking about doom with this stuff, do you?

I try not to.

I mean, I think that's really heartening for me.

Yeah.

Because you're closer to it than I am.

Yeah.

You know, because I, you know, I work in the knowledge.

Right?

Right.

That's my whole thing is like trying to get knowledge delivered to people and understanding, you know, whether things are right or not.

And, you know, by the way, because you're gonna hear the music.

This is the time I should have done this earlier.

The Flower City AI conference is in its fourth year.

It is.

You've done it three times.

Ten second description.

What is it?

It's a place for everybody in Rochester to come learn and network and understand how AI is changing around them.

And, you know, the day to day.

When is it happening?

Wednesday, December.

Wednesday, December 2nd at the Little Theater.

Wednesday, December 2nd.

And are you still looking for people to be speaking there to be present?

I'm looking for people to be speaking, to sponsor, to attend.

It's, not for profit.

It's just, price as cheaply as possible so everybody can participate online where Flower City, Sky Flower City, they AI and that's still happening.

So here you go, Max.

World moves on.

Thank you guys.

Thanks for coming in.

I really appreciate it.

Kind of updating it with all the news going on, and we'll talk to them separately on different days about a lot of other stuff.

Max Irwin, Jeffrey Allan, thank you very much, guys.

More Connections coming up in a minute.

This program is a production of WXXI Public Radio.

The views expressed do not necessarily represent those of this station, its staff, management, or underwriters.

The broadcast is meant for the private use of our audience, any rebroadcast or use in another medium without express written consent of WXXI is strictly prohibited.

Connections with Evan Dawson is available as a podcast.

Just click on the Connections link at WXXI news.org.