Compile | ZeR0
Edit | Desert Shadow
Zhidong reported on May 18 that the OpenAI super alignment team is falling apart. Team leaders Ilya Sutskever, co-founder and chief scientist at OpenAI, and Jan Leike, both left their jobs this week.
On Friday night, Beijing time, Yang Lake publicly explained why he left on the social platform X. He writes that the reason has to do with disagreements about core priorities and the allocation of resources by the team, and he is more concerned about security, consistency, etc.; The super-alignment team has been "sailing against the wind" for the past few months, struggling with calculations and making it increasingly difficult to complete research; Over the past few years, safety culture and processes have given way to more dazzling products.
This appears to be the first time that OpenAI executives have publicly stated that OpenAI prioritizes its products over security.
In response, OpenAI co-founder and CEO Sam Altman posted: "I am very grateful to Jan Lake for his contribution to OpenAI's alignment research and safety culture, and I am very sad to see him leave. He's right, we still have a lot to do; We are committed to doing just that. I'll have a longer post in the next few days. ”
OpenAI formed the Super Alignment team last July, led by Jan Lake and Ilya Sutskovi, with the goal of solving the core technical challenges of controlling super-intelligent AI over the next four years. The team promised to receive 20% of the company's computing resources, but its actual access to computing resources was hindered.
For months, OpenAI has been losing employees who are concerned about AI safety. Since last November, at least seven security-focused members of OpenAI have resigned or been expelled.
According to Wired, the OpenAI Super Alignment team has been disbanded, and the remaining members will either resign or will be included in other OpenAI research efforts.
1. The reason for Jan Lake's public resignation: The team is moving against the wind and struggling to use computing resources
Jan Leike, the former co-head of OpenAI's Super Alignment team, tweeted 13 times last night, revealing the reason for his departure:
"Yesterday was my last day as OpenAI's Head of Alignment, Head of Super Alignment, and Executive.
It's been a crazy journey over the past 3 years. My team launched the first RLHF LLM using InstructGPT, released the first scalable LLM supervision, and was the first to achieve automated explainability and weak-to-strong generalization. More exciting stuff is on the way.
I love my team.
I'm so grateful to the many amazing people I've worked with, both inside and outside of the Super League team.
OpenAI has so many very smart, kind, and efficient people.
Leaving this job was one of the hardest things I've ever done because we desperately needed to figure out how to bootstrap and control AI systems that were so much smarter than us.
I joined because I thought OpenAI would be the best place in the world to do this research.
However, for a long time, I was at odds with OpenAI's leadership about the company's core priorities, until we finally reached a tipping point.
I think we should dedicate more bandwidth to preparing for the next generation of models, including security, monitoring, readiness, adversarial robustness, (super)consistency, confidentiality, social impact, and related topics.
These issues are hard to solve, and I'm afraid we're not on track to get there.
My team has been sailing against the wind for the past few months. Sometimes we struggle with calculations, and it becomes increasingly difficult to complete this important study.
Building machines that are smarter than humans is a dangerous endeavor in itself.
OpenAI represents a huge responsibility on behalf of all of humanity.
But over the past few years, safety culture and processes have given way to dazzling products.
It's long overdue for us to take the impact of AGI seriously.
We must make it as a priority to prepare as much as possible.
Only in this way can we ensure that AGI benefits all humanity.
OpenAI must be a safety-first AGI company.
To all OpenAI employees, I would like to say:
Learn to feel AGI. Behave in a dignified manner, for what you are building. I believe you can "deliver" the cultural change that is needed.
I'm counting on you. The whole world is counting on you.
: openai-heart:」
Second, a large number of employees who care about AI security have left, and Sutskove has not appeared in the OpenAI office for half a year
For months, OpenAI has been losing employees who care about AI security.
The OpenAI Super Alignment team was joined by scientists and engineers from OpenAI's previous calibration department, as well as researchers from other institutions in the company. They will provide research information on the security of internal and non-OpenAI models, and solicit and share work with the wider AI community through initiatives including research grant programs.
The team is responsible for developing methods for managing and guiding "super-intelligent" AI systems. According to a source on OpenAI's Super Alignment team, the team promised to receive 20% of the company's computing resources, but requests for a small portion of the calculations were often denied, hindering the team's work.
A series of issues prompted several team members to resign this week.
OpenAI did not immediately respond to requests for comment regarding the commitment and the resources allocated to the team.
According to sources, Sutskovi has been helpful to the Superalignment team, not only contributing research but also serving as a bridge to other departments within OpenAI. He will also serve as some form of ambassador to demonstrate the importance of the team's work to OpenAI's key decision-makers.
The conflict between Sutskovi and Altman has increased his attention.
In November last year, Sutskwi and OpenAI's former board of directors abruptly announced the expulsion of Altman, citing that Altman "doesn't always be honest with board members."
Under pressure from OpenAI investors, including Microsoft, and many employees of the company, Altman was eventually reinstated, and most of the board of directors resigned. According to reports, Sutskove never returned to work.
Shortly after Altman's reinstatement, Sutskwe tweeted: "I learned a lot in the last month. One of the lessons is that the phrase "keep striking until morale is up" is used more often than it should."
It didn't take long for the tweet to be deleted.
Publicly, Sutskevy and Altman continued to show signs of friendship until this week, when Sutskevi announced his departure, and Altman also addressed him as "my dear friend."
According to foreign media reports, Sutskwi has not appeared in the OpenAI office for about 6 months since the end of the seizure of power. He has been leading the Super-Aligned team remotely, responsible for ensuring that future AGI is aligned with humanity's goals and not the opposite.
It's a big ambition, but it's out of touch with OpenAI's day-to-day operations. Under Altman's leadership, the company has been exploring commercializing its products.
3. Trust in Altman collapsed: falling one by one like dominoes
Suckewei and Lake aren't the only ones leaving, with at least five of OpenAI's safety-conscious employees resigning or being fired since last November.
There are different opinions in the industry about the internal reasons for Altman's "insincerity", and there is a speculation that OpenAI has secretly made a major technological breakthrough, and it is believed that Sutskwe chose to leave because he saw something terrible, such as an AI system that could destroy humanity.
The real answer may indeed have something to do with Altman.
According to Vox, a source familiar with OpenAI revealed that security-conscious employees have lost confidence in Altman.
A company insider, who asked not to be named, said: "It's a process of trust crumbling little by little, like dominoes falling one by one." ”
Not many employees are willing to talk about it publicly. Part of the reason is that OpenAI is known for letting employees sign severance agreements that contain non-derogatory clauses when they leave the company. If they refuse to sign, the employee gives up their equity in the company, which means millions of dollars can be lost.
But one former employee refused to sign a severance agreement so that he could criticize the company freely. Daniel Kokotajlo, who joined OpenAI in 2022 to lead the company in the secure deployment of AI, worked on the governance team until he resigned last month.
"OpenAI is training more and more powerful AI systems, with the goal of eventually surpassing human intelligence across the board. It could be the best thing ever for humanity, but it could also be the worst thing if we don't be careful," Cokotailo said.
"I joined with the hope that OpenAI would rise to the occasion and behave more responsibly as they get closer to achieving AGI. Many of us are coming to realize that this is not going to happen," he said. "I gradually lost faith in OpenAI's leadership and its ability to handle AGI responsibly, so I resigned."
Despite the display of friendship in public, doubts arose about their friendship after Sutskvi tried to get rid of Altman.
Altman's reaction after his dismissal also showed his character. He used OpenAI to hollow out OpenAI to threaten the board to reopen the door to him.
Former colleagues and employees have revealed that Altman is a duplicitous manipulator, for example, he claims that he wants to prioritize safety, but his actual behavior contradicts this.
Previously, Altman asked Saudi funds to support the establishment of a new AI chip company, which shocked security-conscious employees. If Altman really cares about building and deploying AI in the safest way possible, why does he seem to be frantically accumulating as many chips as possible, and this only accelerates the development of the technology?
For employees, all of this has led to a gradual "stop believing what OpenAI says it's going to do or value something, it's actually true," according to a source with knowledge of what's going on inside the company.
Jan Lake is a former DeepMind researcher who was involved in the development of ChatGPT, GPT-4, and ChatGPT's predecessor, InstructGPT, during his time at OpenAI. Hours after Sutskovi announced his departure, Lake issued a brief statement: "I resigned. ”
There was no warm and friendly goodbye, no expression of trust in the company's leadership.
Some safety-conscious OpenAI employees commented on heartbroken expressions.
Former security-conscious employees also retweeted Lake's tweet with a heart emoji. One of them is Leopold Aschenbrenner, an ally of Sutskowi and a member of the Super Alignment team, who was fired by OpenAI last month. Media reported that he and another researcher on the same team, Pavel Izmailov, had been fired for leaking information. But OpenAI did not provide any evidence of the leak.
Considering the strict non-disclosure agreement that everyone has to sign when joining OpenAI, if Altman is in a hurry to get rid of Sutskove's allies, then for him, sharing even the most innocuous information can easily be portrayed as a "leak".
In the same month that Aschenbrenner and Izmailov were forced to leave, security researcher Karen O'Keef also left the company.
Two weeks ago, another security researcher, William Saunders, published a cryptic post on the EA forum, an online hangout for members of the Effective Altruism movement, summarizing the work he did at OpenAI as a member of the Super Alignment team. He wrote: "I resigned from OpenAI on February 15, 2024. ”
For why you should publish this article? Sanders replied, "No comment." The commenter believes that he may be bound by the agreement.
Another OpenAI researcher working on AI policy and governance also appears to have recently left the company. Cullen O'Keefe stepped down in April as head of research at the forefront of policy.
Combine all this information and at least 7 people have tried to push OpenAI to a safer path from within, but ended up losing faith in its Altman.
Conclusion: After the Super Alignment team is disbanded, who will ensure OpenAI's AI security?
After the departures of Lake and Suzkwe, research on the risks associated with more powerful models will be led by John Schulman, another co-founder of OpenAI.
And the original OpenAI Super Alignment team is no longer a dedicated team, but a loose research group, distributed across various departments of the company. A spokesperson for OpenAI described it as "deeper integration (of the team)".
"The point of having a super-aligned team is that if the company succeeds in building AGI, there are actually different types of security issues that will arise," the person said, "and it's a dedicated investment in the future." ”
Even if the team is fully operational, this "dedicated investment" represents only a small fraction of OpenAI's researchers, and only promises to provide 20% of the computing power. Now that computing power is likely to be transferred to other OpenAI teams, it's unclear whether the focus will be on avoiding catastrophic risks for future AI models.
To be clear, this does not mean that the products that OpenAI is releasing now will destroy humanity. But what happens next?
"Distinguish 'Are they currently building and deploying insecure AI systems?' With the question'Are they building and deploying AGI or superintelligence securely?' Very important. According to people familiar with the matter, the answer to the second question is no.
Source: Vox, Wired