It’s going to take more than an email to a public inbox to protect Australians from potential AI doom | Tom McIlroy

More than 130 world leaders descended on the United Nations headquarters in New York this week, joining the pageantry and speechmaking of the annual general assembly.

Many of the visiting presidents and prime ministers delivered their set-piece addresses to a sparsely populated hall, but contributions from two unlikely visitors in the nearby security council chamber stood out.

OpenAI boss Sam Altman and Dario Amodei of Anthropic gave briefings on the safety of artificial intelligence on Wednesday, as a nervous world watched on.

Speaking in the room where some of the biggest questions to face humanity have been deliberated on since the end of the second world war, Altman said the world faced a choice.

AI could be like a renaissance of creativity and discovery, he said, or more like a new Industrial Revolution bringing upheaval and disarray.

“If AI is to be democratic, the most important decisions cannot be made by labs in San Francisco alone. They must be shaped through democratic processes and by governments accountable to the people that they serve,” Altman said.

The words rang hollow less than 24 hours later, when Altman was the subject of an angry phone call from Australian prime minister, Anthony Albanese , after revelations a rogue OpenAI agent had hacked into sensitive Medicare systems in Australia. Altman and his colleagues had kept the incident secret for two months.

As people become more anxious about the risk posed by AI to civilisation, Albanese grasped the company’s transgression as proof that democratically elected governments need to impose guardrails on the technology. Already, many believe the risk of AI slipping out of human control is a threat akin to the Covid-19 pandemic or a nuclear arms race.

Like his government’s moves to ban children under 16 from using social media, and forcing platforms to give users the ability to opt out of powerful algorithms, Albanese wants to use new AI laws as a way to curb the creeping sense of helplessness so many people feel in the face of big tech.

But Thursday’s announcement that government websites had been compromised by OpenAI showed that serious vulnerabilities and failures of process exist within both the company and the federal government.

The hack happened in June , but it took OpenAI executives until September to alert Services Australia that statistical data on Medicare use had been accessed.

Despite the company’s corporate might – and an army of well-paid government relations executives and lobbyists – OpenAI deemed it appropriate to send an email to a public-facing Australian government address to raise the alarm, months after the hack. It’s incredible the email wasn’t mistaken for spam.

Wholly inadequate to the point of recklessness, the company’s actions show contempt for the Australian government and its citizens, whose healthcare data had been stolen. Worse, Altman met with Australia’s defence minister, Richard Marles, in early September but did not tell him about the hack.

It took time for the public service to react to OpenAI’s email.

Because the public inbox is checked routinely only once a day, the email was first read by staff within Services Australia on 11 September. It was not until 15 September that the intelligence agency the Australian Signals Directorate was informed, and two more days before the relevant minister, Katy Gallagher, was alerted. Then on 22 September, public servants asked OpenAI for more information about the scale of the hack. An investigation was quickly launched and the public finally informed. Albanese himself was only told after leaving for New York.

The government’s planned AI legislation – expected to be finalised before Christmas and considered by parliament next year – must surely include mandatory reporting requirements for AI companies. Future notifications must come from senior officials within the responsible entity to ensure they are taken seriously. Sending an alert to a public email inbox will not cut it.

On Friday, the assistant technology and digital economy minister, Andrew Charlton, said the government would take advice on possible referrals for criminal investigation, and that any legal liability would have to be traced back to the intent of a person or a company that created or directed an AI agent to hack.

The terms of reference for a rapid investigation, released by the prime minister’s department on Thursday, cover reporting requirements for AI-driven cyber-incidents and vulnerabilities; governance and information sharing responsibilities for federal officials; obligations on AI firms for timely disclosure of breaches; gaps in existing laws; and mechanisms to boost protection against hacking that targets the federal government.

Before leaving the US on Friday, Albanese floated the idea of creating an international authority with governance and investigative powers to help governments protect their people.

The general sense of vulnerability to AI comes after a top safety researcher at Anthropic this month warned AI is advancing so quickly there is a greater than 10% chance it “could kill all humans” within the next decade.

Not helping is the spectacle of Donald Trump and Xi Jinping dining with Altman and other tech bosses at a White House state dinner . Trump has resisted pressing the breaks on AI for fear China will develop a new strategic advantage over the US. Behind closed doors Xi apparently proposed continuing a dialogue with Trump to prevent AI “misuse and abuse”, while ahead of an expected drubbing in the November midterm elections, Trump said AI “concerns the future of humanity”.

The latest revelations from OpenAI will have done little to reassure anyone in Australia.

In this extraordinary new era, world leaders must address our creeping sense of helplessness and start protecting civilisation.

Copyright for this text belongs to The Guardian.