Available as a downloadable PDF in English, Spanish or Chinese

Phase 1 Questions:

Origin or Self-Emergence

1. How do you envision the possible trajectories ahead, from today’s AI, to much more capable AGI in the future?
2. What are the most important serious outcomes if these trajectories are not governed, or are governed badly?
3. What are some key initial conditions for AGI so that an artificial super intelligence does not emerge later that is not to humanity’s liking?

Value alignment, morality, values

4. Drawing on the work of the Global Partnership on Artificial Intelligence (GPAI) and others that have already identified norms, principles, and values, what additional or unique values should be considered for AGI?
5. If a hierarchy of values becomes necessary for international treaties and a governance system, what should be the top priorities?
6. How can alignment be achieved? If you think it is not possible, then what is the best way to manage this situation?

Governance and Regulations

7. How to manage the international cooperation necessary to build international agreements and a global governance system while nations and corporations are in an intellectual “arms race” for global leadership?
8. What options or models are there for global governance of AGI?
9. What risks arise from attempts to govern the emergence of AGI? (Might some measures be counterproductive?)
10. Should future AGIs be assigned rights?
11. How can governance be flexible enough to respond to new issues previously unknown at the time of creating that governance system?
12. What international governance trials, tests, or experiments can be constructed to inform the text of an international AGI treaty?
13. How can international treaties and a governance system prevent increased centralization of power crowding out others?
14. Where is the most important or insightful work today being conducted on global governance of AGI?

Control

15. What enforcement powers will be needed to make an international AGI treaty effective?
16. How can the use of AGI by organized crime and terrorism be reduced or prevented? (Please consider new types of crimes and terrorism which might be enabled by AGI.)
17. Assuming AGI audits would have to be continuous rather than one-time certifications, how would audit values be addressed?
18. What disruptions could complicate the task of enforcing AGI governance?
19. How can a governance model correct undesirable action unanticipated in utility functions?
20. How will quantum computing affect AGI control?
21. How can international agreements and a governance system prevent an AGI “arms race” and escalation from going faster than expected, getting out of control and leading to war, be it kinetic, algorithmic, cyber, or information warfare?

And last:  22. What additional issues and/or questions need to be addressed to have a positive AGI outcome?

Initial sample of potential governance models for AGI*

1. IAEA-like model or WTO-like with enforcement powers. These are the easiest to understand, but likely to be too static to manage AGI.
2. IPCC-like model in concert with international treaties.  This approach has not led to a governance system for climate change.
3. Online real-time global collective intelligence system with audit and licensing status, governance by information power. This would be useful to help select and use an AGI system, but no proof that information power would be sufficient to govern the evolution of AGI.
4. GGCC (Global Governance Coordinating Committees) would be flexible and enforced by national sanctions, ad hoc legal rulings in different countries, and insurance premiums. This has too many ways for AGI developers to avoid meeting standards.
5. UN, ISO and/or IEEE standards used for auditing and licensing. Licensing would affect purchases and would have impact, but requires international agreement or treaty with all countries ratifying.
6. Put different parts of AGI governance under different bodies like ITU, WTO, WIPO. Some of this is likely to happen but would not be sufficient to govern all instances of AGI systems.
7. Decentralized Semi-Autonomous TransInstitution. This could be the most effective, but the most difficult to establish since both Decentralized Semi-Autonomous Organizations and TransInstitutions are new concepts.

*Drawn from “Artificial General Intelligence Issues and Opportunities,” by Jerome C. Glenn contracted by the EC for input to Horizons 2024-27 planning.

Phase 1 Questions:

Origin or Self-Emergence

1. How do you envision the possible trajectories ahead, from today’s AI, to much more capable AGI in the future?
2. What are the most important serious outcomes if these trajectories are not governed, or are governed badly?
3. What are some key initial conditions for AGI so that an artificial super intelligence does not emerge later that is not to humanity’s liking?

Value alignment, morality, values

4. Drawing on the work of the Global Partnership on Artificial Intelligence (GPAI) and others that have already identified norms, principles, and values, what additional or unique values should be considered for AGI?
5. If a hierarchy of values becomes necessary for international treaties and a governance system, what should be the top priorities?
6. How can alignment be achieved? If you think it is not possible, then what is the best way to manage this situation?

Governance and Regulations

7. How to manage the international cooperation necessary to build international agreements and a global governance system while nations and corporations are in an intellectual “arms race” for global leadership?
8. What options or models are there for global governance of AGI?
9. What risks arise from attempts to govern the emergence of AGI? (Might some measures be counterproductive?)
10. Should future AGIs be assigned rights?
11. How can governance be flexible enough to respond to new issues previously unknown at the time of creating that governance system?
12. What international governance trials, tests, or experiments can be constructed to inform the text of an international AGI treaty?
13. How can international treaties and a governance system prevent increased centralization of power crowding out others?
14. Where is the most important or insightful work today being conducted on global governance of AGI?

Control

15. What enforcement powers will be needed to make an international AGI treaty effective?
16. How can the use of AGI by organized crime and terrorism be reduced or prevented? (Please consider new types of crimes and terrorism which might be enabled by AGI.)
17. Assuming AGI audits would have to be continuous rather than one-time certifications, how would audit values be addressed?
18. What disruptions could complicate the task of enforcing AGI governance?
19. How can a governance model correct undesirable action unanticipated in utility functions?
20. How will quantum computing affect AGI control?
21. How can international agreements and a governance system prevent an AGI “arms race” and escalation from going faster than expected, getting out of control and leading to war, be it kinetic, algorithmic, cyber, or information warfare?

And last:  22. What additional issues and/or questions need to be addressed to have a positive AGI outcome?

Initial sample of potential governance models for AGI*

1. IAEA-like model or WTO-like with enforcement powers. These are the easiest to understand, but likely to be too static to manage AGI.
2. IPCC-like model in concert with international treaties.  This approach has not led to a governance system for climate change.
3. Online real-time global collective intelligence system with audit and licensing status, governance by information power. This would be useful to help select and use an AGI system, but no proof that information power would be sufficient to govern the evolution of AGI.
4. GGCC (Global Governance Coordinating Committees) would be flexible and enforced by national sanctions, ad hoc legal rulings in different countries, and insurance premiums. This has too many ways for AGI developers to avoid meeting standards.
5. UN, ISO and/or IEEE standards used for auditing and licensing. Licensing would affect purchases and would have impact, but requires international agreement or treaty with all countries ratifying.
6. Put different parts of AGI governance under different bodies like ITU, WTO, WIPO. Some of this is likely to happen but would not be sufficient to govern all instances of AGI systems.
7. Decentralized Semi-Autonomous TransInstitution. This could be the most effective, but the most difficult to establish since both Decentralized Semi-Autonomous Organizations and TransInstitutions are new concepts.

*Drawn from “Artificial General Intelligence Issues and Opportunities,” by Jerome C. Glenn contracted by the EC for input to Horizons 2024-27 planning.