When governments decide to sleep at the wheel, private philanthropy steps in.
We see this in global healthcare, where the Gates Foundation boosted support to the Global Alliance for Vaccines and Immunization (GAVI) by tens of millions of dollars following the latest shifts in US policy. And in AI safety research, Schmidt Sciences, a nonprofit organization founded by Eric and Wendy Schmidt, is launching the AI Safety Science program with a budget of $10 million.
Eric Schmidt's involvement in AI safety isn't new. As the former CEO of Google, he has long been engaged with the promises and perils of artificial intelligence. His recent book with Henry Kissinger and Craig Mundie i”s called “Genesis: Artificial Intelligence, Hope, and the Human Spirit”, and it demonstrates his nuanced understanding of AI's potential to solve humanity's greatest challenges and pose unprecedented risks.
The timing of Schmidt Sciences' initiative is critical. US government funding for AI safety research will likely disappear after VPOTUS’s words at the AI Action Summit. Meanwhile, AI systems are rapidly being integrated into every sector of society, from healthcare to government operations.
The team at Schmidt Sciences has highlighted three key Issues in AI Safety Research:
1. Inadequate Safety Benchmark Ecosystem:
- Too few well-founded, quality benchmarks
- High correlation between existing benchmarks, suggesting they don't assess independent capabilities
- Lack of evaluations for agentic and multi-modal capabilities
- Need for proactive safety measures for future, more sophisticated systems
2. Insufficient Research Funding:
- Total funding is estimated at only $80-130 million per year (2021-2024), a tiny fraction of what's invested in AI development overall
- Government programs like Safe Learning-Enabled Systems offering only $10 million annually
- Current funding levels prohibit larger, long-term research efforts
3. Underutilization of Academic Expertise:
- Safety research on the largest models mainly conducted by frontier AI labs
- Academic community lacks access to frontier models, testing environments, and compute resources
- Need for more peer review and reproducible research in AI safety
Schmidt Sciences is addressing this challenge by funding multiple pioneering research projects. Please see descriptions of several projects:
Yoshua Bengio (Mila - Quebec AI Institute)
- Developing quantitative safety guardrails for frontier AI models
Adam Gleave, Kellin Pelrine (FAR.AI) & Thomas Costello (American University and MIT)
- Assessing AI-driven deception and disinformation risks and developing mitigations
Tatsu Hashimoto (Stanford University)
- Creating testing environment with statistical guarantees for AI agent failure rates
- Building safety metrics to identify emergent unsafe capabilities
Matthias Hein (University of Tübingen) & Jonas Geiping (ELLIS Institute Tübingen)
- Investigating degradation of AI system safety over extended operation periods
Daniel Kang (University of Illinois, Urbana-Champaign)
- Assessing AI agents' capability to perform complex cybersecurity attacks
Zico Kolter (Carnegie Mellon University)
- Exploring causes of adversarial transfer between AI models
Bo Li (University of Illinois Urbana-Champaign)
- Designing virtual environment with advanced red teaming algorithms for AI system evaluation
Evan Miyazono & Daniel Windham (Atlas Computing)
- Developing AI tool for generating, reviewing, and validating formal code specifications
Karthik Narasimhan (Princeton University)
- Developing benchmarks to evaluate reliability and consistency of AI code-writing systems
Arvind Narayanan (Princeton University)
- Designing verifiers to check AI system accuracy and improve agent reliability
Florian Tramèr (ETH Zurich)
- Designing guardrails against prompt injection attacks
- Developing formal programming framework to standardize these guardrails
These projects will receive not just financial support, but also crucial resources like computational power and API access. This support is particularly important for academic researchers, who have been largely left out of AI safety research despite their potential to contribute valuable insights.
Through this program, Schmidt Sciences is planting the seeds for safer AI development in academic soil. Their goal aligns with the vision presented in "Genesis" - to chart a course between blind faith and unjustified fear in the age of AI, while ensuring this powerful technology benefits humanity while protecting us from potential harms.