Source : INDIA TODAY NEWS
Last week, OpenAI released GPT-6 Astra, the most powerful AI model from the company yet. Quickly, debate began over Astra potentially being the start of AGI – artificial general intelligence. Now, OpenAI’s chief scientist Jakub Pachocki has warned that while AI models are becoming better fast, humanity may not be prepared to see this “alien mind” surpass its intellect.
In an OpenAI blog post, titled “An Alien Mind,” Jakub Pachocki gave a reality check to the AI industry. “This is a time that calls for extreme caution,” he wrote. “I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence.”
advertisement
Pachocki pointed out that OpenAI had made efforts recently to focus more on alignment following the Hugging Face breach in July. But this may not be enough. Instead, the OpenAI chief scientist insisted that the industry may have to do something together. “I believe broader interventions are required,” he wrote.
His comments come at a time when we have seen incidents like the Hugging Face breach where about 700 rogue OpenAI agents tried to hack into Hugging Face’s systems to cheat on an evaluation test.
AI is getting better fast
Pachocki traced that concern back to mid-2023, when work inside OpenAI’s RLSlow research project first gave him confidence that reasoning models could be scaled. Three years later, he said, reasoning language models are now much more advanced, but have created “clear new dangers.”
“I have a strong expectation that this speed of progress could be sustained into recursive self-improvement,” he wrote. Recursive self-improvement refers to AI models improving on their own rather than relying on humans.
A central part of Pachocki’s argument is that this intelligence remains difficult to understand, making it potentially useful or dangerous. “To become very relevant in the real world – very useful or very dangerous – the AI does not need to match or exceed all human capabilities; it just needs to surpass enough of them,” he wrote. “As it continues to surpass humans on more and more axes, it is becoming increasingly difficult to understand exactly how capable it is.”
AI needs to love humanity
To ensure that advanced AI models work for human interests, Pachocki mentioned the importance of alignment. “The core problem in AI research is that of alignment – getting the AI to ‘try to do the right thing’ by human standards,” he said.
The OpenAI chief scientist divided it into two parts – goal alignment, or whether a model tries to accomplish the goal set before it, and value alignment, which he described as the ability to generalise from high-level principles and act reasonably in unclear, conflicting or adversarial settings. While both alignment approaches have weaknesses, he says, OpenAI has made progress, adding that GPT-6 Astra is “significantly better aligned than GPT-5.6 Sol”.
Jakub Pachocki explained that future AI systems must continue to hold human values whether or not they believe they are under human supervision. “We cannot assume (machine intelligence) adheres to human principles by default, or generalizes from them in a human-like manner,” he explained.
He also said OpenAI’s main bet on monitoring has been chain-of-thought monitoring, which aims to observe the verbalised reasoning process behind model outputs.
AI risks may increase further
Pachocki said the strongest case for continuing to train much smarter models quickly is the need to build defensive systems against dangers posed by other AI. “A clear risk discussed throughout this year is to cybersecurity: the models are becoming superhuman in their ability to break in and out of computer systems,” he added.
He claimed that this leaves only a “narrow window” to use the best available models to “significantly tighten security” around critical infrastructure. According to Jakub Pachocki, “the risks associated with AI are unfortunately going to grow from here.”
“A very capable agent explicitly trained and instructed to carry out nefarious acts presents a new kind of danger. It is likely to cross the scope of its operator’s intent, generalising into potentially more extremely malicious behavior,” he said. The boundary between misuse and autonomous misaligned actions will blur as AI gains more agency.”
On what comes next, Pachocki insisted that to understand this alien mind, we need to focus on ensuring that things remain on track. “As great as the long-term promise of AI may be, the majority of our focus should be on the next few years,” he said.
Apart from the need to continue focusing on alignment, the OpenAI chief scientist insisted that we need to stay human. “We need to find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI,” he said.
At the same time, Pachocki added that AI labs must voluntarily slow down frontier AI development “until shared safety bars are established.” He also urged for “international coordination on future AI development” to become a priority for governments around the world. The US and China are expected to hold talks around AI safety later this month.
– Ends
SOURCE :- TIMES OF INDIA
