Toby Ord. The Precipice: Existential Risk and the Future of Humanity (2020)

The Precipice:

Existential Risk and the Future of Humanity

Toby Ord (2020)

In a paragraph

Humanity is on a precipice where it risks destroying itself and losing its vast future potential. We have immense leverage at this time, so reducing existential risks should be our top moral priority.

Key points

• Dedication: To the hundred billion people before us, who fashioned our civilisation; To the seven billion now alive, whose actions may determine its fate; To the trillions to come, whose existence lies in the balance.

• A species precariously close to self-destruction, with a future of immense promise hanging in the balance. We threaten, in the name of our transient aims and fallible convictions, to foreclose it all.

• The Precipice gives our time immense meaning. Safeguarding humanity’s future is the defining challenge of our time.

• An existential risk is a risk that threatens the destruction of humanity’s long-term potential

• Concern about existential risk could also spring from considerations of our past, our character and our cosmic significance. From a variety of moral approaches.

• Longtermism looks at humanity itself as a group agent over deep time.

• If we fail, that upwards force, that capacity to push towards what is best or what is just, will vanish from the world.

• If we are the only moral agents that will ever arise in our universe — the only beings capable of making choices on the grounds of what is right and wrong — then responsibility for the history of the universe is entirely on us. Humanity would be the only form of life capable of stewarding life itself.

• We would do well to be humble, to leave our options open, and to ensure our descendants have a chance to see more clearly, and choose more wisely, than we can today. Protecting our future has immense option value

• Protection from existential risk is an intergenerational global public good

• Estimates of existential risks in next 100 years:

Total natural risk                               ~ 1 in 10,000

Nuclear war                                         ~ 1 in 1,000

Climate change                                   ~ 1 in 1,000

Other environmental damage         ~ 1 in 1,000

‘Naturally’ arising pandemics          ~ 1 in 10,000

Engineered pandemics                      ~ 1 in 30

Unaligned artificial intelligence      ~ 1 in 10

Unforeseen anthropogenic risks     ~ 1 in 30

Other anthropogenic risks                ~ 1 in 50

Total anthropogenic risk                   ~ 1 in 6

Total existential risk                          ~ 1 in 6

• Risk factors and security factors

• Cost-Effectiveness = Importance × Tractability × Neglect 

• Most existential risk comes from human action: from activities which we can choose to stop, or to govern effectively.

• Grand Strategy for Humanity: Reaching Existential Security, The Long Reflection, Achieving Our Potential

• Our long-term survival requires a deliberate choice to survive. 

• Our current predicament stems from the rapid growth of humanity’s power outstripping the slow and unsteady growth of our wisdom.

• Humanity is akin to an adolescent, with rapidly developing physical abilities, lagging wisdom and self – control, little thought for its long-term future and an unhealthy appetite for risk.

• Discount rate ρ = mg + δ. Diminishing marginal utility from growth plus catastrophe risk. mg nil as not a monetary comparison.  Catastrophe risk 0.1 would imply future worth 1,000 x next year.

Comments

An impressively powerful, well written and thoroughly researched statement of the case for making reducing existential risk our top moral priority. 

Notes

Dedication Page

To the hundred billion people before us, who fashioned our civilisation; To the seven billion now alive, whose actions may determine its fate; To the trillions to come, whose existence lies in the balance.

Part One: The Stakes

1. Standing at the Precipice

A species precariously close to self-destruction, with a future of immense promise hanging in the balance.

Safeguarding humanity’s future is the defining challenge of our time.

To allow this book to reach a diverse readership, I’ve been ruthless in stripping out the jargon, needless technical detail and defensive qualifications typical of academic writing.

In ecological terms, it is not a human that is remarkable, but humanity.

Either humanity takes control of its destiny and reduces the risk to a sustainable level, or we destroy ourselves.

The Precipice gives our time immense meaning. Our actions have uniquely high stakes.

I think safeguarding humanity through these times is among the most noble purposes you could pursue.

2. Existential Risk

An existential risk is a risk that threatens the destruction of humanity’s long-term potential.

If we fail, that upwards force, that capacity to push towards what is best or what is just, will vanish from the world.

We threaten, in the name of our transient aims and fallible convictions, to foreclose it all.

The immense value of humanity’s potential.

To risk destroying this future, for the sake of some advantage limited only to the present.  It privileges this particular century over the millions, or maybe billions, yet to come.

People matter equally regardless of their temporal location.

While Epicurus’s argument may provide consolation in times of grief or fear, it is not fit to be a guide for action, and no one treats it so.

Concern about existential risk could also spring from considerations of our past, our character and our cosmic significance.

One doesn’t repay one’s parents. One passes it on. If we drop the baton, succumbing to an existential catastrophe, we would fail our ancestors

Look at humanity itself as a group agent.

Civilizational virtues and vices.

Whether we are alone in the universe is one of the greatest remaining mysteries of science.

If we are alone, our survival and our actions might take on a cosmic significance. We may yet be one of the most rare and precious parts of the cosmos.

Ethics from the perspective of humanity.

Humanity over deep time: It changes the way we see the world and our role in it, shifting our attention from things that affect the fleeting present, to those that could make fundamental alterations to the shape of the long-term future. What matters most for humanity? And what part in this plan should our generation play? What part should I play?

If we are the only moral agents that will ever arise in our universe — the only beings capable of making choices on the grounds of what is right and wrong — then responsibility for the history of the universe is entirely on us.

This is the only chance ever to shape the universe towards what is right, what is just, what is best for all.  For it would only be through us that a part of the universe could come to fully understand the laws that govern the whole. Humanity would be the only form of life capable of stewarding life itself,

We could understand the importance of existential risk in terms of our present, our future, our past, our character or our cosmic significance.  Doesn’t rely on any single school of moral thought, but springs naturally from a great many.

Failing forever in what could well be our most important duty.

There would be immense value of information in finding out more about whether our future will be positive or negative. Protect humanity until we have a much more informed position on this crucial question.

Do well to be humble, to leave our options open, and to ensure our descendants have a chance to see more clearly, and choose more wisely, than we can today.

Protection from existential risk is an intergenerational global public good.

Bertrand Russell, Albert Einstein, Jonathan Schell, Carl Sagan, Derek Parfit, John Leslie, Nick Bostrom.

Existential risk still seems new and strange, but I am hopeful that it will soon find its way into our common moral traditions.  To establish the pivotal importance of safeguarding humanity, and to place this among the pantheon of causes to which the world devotes substantial attention and resources.

Part Two: The Risks

3. Natural Risks

Think seriously about imprecise probabilities of unprecedented events.

4. Anthropogenic Risks

Nuclear war, global warming, environmental degradation.  Unlikely on own to end humanity.

Green Revolution, Norman Borlaug, who may be responsible for saving more lives than anyone else in history.

5. Future Risks

We would cede our status as the most intelligent entities on Earth.

We might hope to retain control.

Reward function.

The AI system could gain access to the internet and hide thousands of backup copies,

evolution optimising us towards the spreading of our genes, regardless of the effects on what we value.

A locked – in dystopia.

Preserving our options.  Preserving our long-term potential.

Nanotechnology. Soil samples. Alien civilisation. Radical scientific experiments.

Since humanity’s power is still rapidly growing, we shouldn’t be surprised if some of these novel threats pose a substantial amount of risk.

Unforeseen risks are thus important to understanding the relative value of broad versus narrowly targeted efforts.

We may have a chance of stumbling across something that offers the destructive power of the atomic bomb.

Part Three: The Path Forward

6. The Risk Landscape

The 20 percent risk would actually be 2.25 times as important as the 10 percent risk,

The threat of great-power war may (indirectly) pose a significant amount of existential risk.

‘Risk factor’. Consider stressors for humanity or for our ability to make good decisions.  These include global economic stagnation, environmental collapse and breakdown in the international order.

We can call the difference between PR ( X | f = fsq ) and Pr ( X | f = fmin ) the contribution that F makes to existential risk .

Security factors. Things such as education, peace or prosperity may help protect us.

The more a problem is important, tractable or neglected, the more cost-effective it is to work on it. Cost-Effectiveness = Importance × Tractability × Neglect.  Choose for fit and leverage

Soon, sudden and sharp.

There will probably be more resources devoted to risks that occur later on, as humanity becomes more powerful and more people wake up to humanity’s predicament.

Warning shot. Early action is higher leverage, but more easily wasted. A focus on knowledge and capacity building. Targeted and broad interventions.

7. Safeguarding Humanity

If we behave rationally and humanely; if we concentrate coolly on the problems that face all of humanity, rather than emotionally on such nineteenth century matters as national security and local pride.

Most existential risk comes from human action: from activities which we can choose to stop, or to govern effectively.

Build the communities, norms and institutions that will safeguard our future.

Whether we are remembered as the generation who turned the corner to a bright and secure future, or not remembered at all, comes down to whether we rise to meet these challenges.

When exploring these issues, I find it useful to consider our predicament from humanity’s point of view: casting humanity as a coherent agent and considering the strategic choices it would make where it sufficiently rational and wise. Or in other words, what all humans would do if we were sufficiently coordinated and had humanity’s long-term interests at heart. This frame is highly idealised.

I paint an ambitious vision of humanity getting its house in order that I hope can guide us over the coming decades, even if the reality is messier and more fraught.

Grand Strategy for Humanity. Reaching Existential Security, The Long Reflection, Achieving Our Potential

But being of humanity’s own making, they are also within our control.

Our long-term survival requires a deliberate choice to survive.

Most work in moral philosophy so far has focused on negatives — on avoiding wrong action and bad outcomes. The study of the positive is at a much earlier stage of development.

The ultimate aim of the Long Reflection would be to achieve a final answer to the question of which is the best kind of future for humanity.

We could think of these first two steps of existential security and the Long Reflection as designing a constitution for humanity.

We can’t rely on our current intuitions and institutions that have evolved to deal with small- or medium-scale risks. We cannot afford to fail even once.

This reactive trial and error approach don’t work at all when it comes to existential risk. We will need to take proactive measures: sometimes long in advance, sometimes with large costs, sometimes when it is still unclear whether the risk is real or whether the measures will address it.

This puts us at risk of chasing phantoms — being asked (or forced) to make substantial sacrifices on the basis of little evidence. Iraq war.

If a disaster does occur, it is much more likely to be because there was an estimation mistake and the real risk was higher, rather than because a one in a trillion-event occurred.

The existing global order splits humanity into a large number of sovereign states, each of which has considerable internal coherence, but only loose coordination with the others. This structure has some advantages, even from the perspective of existential risk, for it has allowed us to minimise the risk that a single bad government could lock humanity into a terrible stable outcome.

195 countries may mean 195 chances that poor governance precipitates the destruction of humanity.

Moscow–Washington Hotline

Explicitly prohibit and punish the deliberate or reckless imposition of unnecessary extinction risk. 1997, UNESCO passed a Declaration on the Responsibilities of the Present Generations Towards Future Generations.

Our current predicament stems from the rapid growth of humanity’s power outstripping the slow and unsteady growth of our wisdom.

Slowing technological progress should help to give us some breathing space,

Humanity is akin to an adolescent, with rapidly developing physical abilities, lagging wisdom and self-control, little thought for its long-term future and an unhealthy appetite for risk.

A good example of successful governance is the Montreal Protocol, which set a timetable to phase out the chemicals that were depleting the ozone layer.

Speeding up the development of protective technologies relative.

There is substantially more transition risk than state risk.

The study of existential risk is in its infancy. We are only starting to understand the risks we face and the best ways to address them. And we are at an even earlier stage when it comes to the conceptual and moral foundations, or grand strategy for humanity.

Alongside these many strands of research on concrete topics, we also need research on more abstract matters. We need to better understand long termism, humanity’s potential and existential risk: to refine the ideas, developing the strongest versions of each; to understand what ethical foundations they depend upon, and what ethical commitments they imply; and to better understand the major strategic questions facing humanity.

Small crisp insights. For example: that a catastrophe killing 100 percent of people could be much worse than one killing 99 percent because you lose the whole future; that the length of human survival so far puts a tight bound on the natural risk; that existential risk reduction will tend to be undersupplied since it is an intergenerational global public good; or the distinction between state risks and transition risks .

Open Philanthropy Project

What not to do. Don’t regulate prematurely. Don’t take irreversible actions unilaterally. Don’t spread dangerous information. Don’t exaggerate the risks. Don’t be fanatical. Don’t be tribal. Don’t act without integrity. Don’t despair. Don’t ignore the positive.

The contraceptive pill, one of the most revolutionary inventions of the twentieth century, was made possible by a single philanthropist. Katharine McCormick.

8. Our Potential

This chapter is about potential, not prophecy. Not what we will achieve, but what is open for us to achieve if we play our cards right; if we are patient, prudent, compassionate, ambitious and wise.

Human history so far has seen 200,000 years of Homo sapiens and 10,000 years of civilisation.

Such a timescale is enough to repair the damage that we, in our immaturity, have inflicted upon the Earth.

Only we could save the biosphere from the effects of the brightening Sun.

Humanity’s instrumental value may yet be profound. For if we can last long enough, we will have a chance to literally save our world.

Perhaps, with ingenuity and commitment, we could extend the time allotted to complex life on Earth by billions of years, and, in doing so, more than redeem ourselves for the foolishness of our civilisation’s youth.

When I contemplate the expected timespan that Earth-based life may survive and flourish, the greatest contribution comes from the possibility of humanity turning from its destroyer, to its saviour.

The sunlight hitting Earth’s surface each day carries 5,000 times more energy than modern civilisation requires. It gives in two hours what we use in a year. Earth intercepts less than one part in a billion of sun’s energy.

If we were to find other life — especially intelligent life — it could profoundly change our future direction.

If we could travel just six light years at a time, then almost all the stars of our galaxy would be reachable.

A sphere around us extending out 46 billion light years in all directions, known as the observable universe.

Frank Ramsey:  I don’t feel the least humble before the vastness of the heavens. The stars may be large, but they cannot think or love; and these are qualities which impress me far more than size does. I take no credit for weighing nearly seventeen stone. My picture of the world is drawn in perspective, and not to scale. The foreground is occupied by human beings and the stars are all small as threepenny bits. The matter of which we are comprised has been so delicately arranged as to allow us to think and love and create and dream. The rest of the universe appears to lack such qualities. In terms of value, the stars are as small as threepenny bits.

The potential quality of our future is also grand beyond imagining.  Human life, for all its joys, could be dramatically better than it is today. Peak experiences. What beauties are we blind to?

The space of possible experiences and modes of life, and the degree of flourishing they make available, may be similarly vast, and that everyday life may acquaint us with a similarly parochial proportion.

Our investigations of flourishing thus far in history may be like astronomy before telescopes — with such limited vision, it is easy to think the universe small, and human-centred.

Implants granting digital extensions to our minds, or developments in artificial intelligence allowing us to craft entirely new kinds of beings to join us or replace us.

We stand before something extraordinarily vast and valuable

Moral philosophy has been focused on the more pressing issues of treating each other decently in a world of scarce resources. But there may come a time, not too far away, when we mostly have our house in order and can look in earnest at where we might go from here. Where we might address this vast question about our ultimate values. This is the Long Reflection.

Appendices

Appendix A: Discounting the Long-term Future

ρ = mg + δ

Nicholas Stern sets pure time preference to zero and set δ to a catastrophe rate of 0.1 percent per annum. This values humanity’s future at about 1,000 times the value of the next year. If annual risks become low in the long term, then the expected value of the future is very great indeed.

Appendix B: Population Ethics and Existential Risk

Population ethics. Total View. Repugnant conclusion. Person-affecting views.

Notes

There is effectively another book-worth of content tucked away in the notes for readers who are eager to know more.  I’ve tried to be disciplined in keeping the main text on a straight path to its destination, so the scenic detours are all hidden in the notes.

Maps of Time by David Christian

Foraging typically required about ten square miles of land to support each person

Francis Bacon’s Novum Organum (1620) is the canonical exposition of the scientific method

The $2 a day poverty line

100 billion people have lived

Potential moral achievements.  Humanity contains the potential to forge a truly just world and realising this dream would be a profound achievement.

‘There are many other possible measures of the potential loss — including culture and science, the evolutionary history of the planet, and the significance of the lives of all of our ancestors who contributed to the future of their descendants. Extinction is the undoing of the human enterprise.’

Custodial duties to preserve the inheritance of humanity passed on to us by our ancestors and convey it safely to our descendants.

‘If we are the only rational beings in the Universe, as some recent evidence suggests, it matters even more whether we shall have descendants or successors during the billions of years in which that would be possible. Some of our successors might live lives and create worlds that, though failing to justify past suffering, would have given us all, including those who suffered most, reasons to be glad that the Universe exists.’

Humans are the only moral agents.

The theory of how to make decisions when we are uncertain about the moral value of outcomes was almost completely neglected in moral philosophy until very recently — despite the fact that it is precisely our uncertainty about moral matters that leads people to ask for moral advice and, indeed, to do research on moral philosophy at all. Remedying this situation has been one of the major themes of my work so far.

Our present understanding of axiology might well be confused.

Protecting our future has immense option value. The path that preserves our ability to choose whatever turns out to be best when new information comes in.

Williams (2015), who generalises this idea: ‘. . . we should regard intellectual progress, of the sort that will allow us to find and correct our moral mistakes as soon as possible, as an urgent moral priority rather than as a mere luxury.  Non-excludability. Non-rivalry.

Russell ‘As I write, I learn that a second bomb has been dropped on Nagasaki. The prospect for the human race is sombre beyond all precedent. Mankind are faced with a clear-cut alternative: either we shall all perish, or we shall have to acquire some slight degree of common sense. A great deal of new political thinking will be necessary if utter disaster is to be averted. Mankind must put an end to war — or war will put an end to mankind . . . Today, every inhabitant of this planet must contemplate the day when this planet may no longer be habitable. Every man, woman and child live under a nuclear sword of Damocles, hanging by the slenderest of threads, capable of being cut at any moment by accident, or miscalculation, or by madness. The weapons of war must be abolished before they abolish us.’

Environmental Kuznets curve.

A balancing act between getting sustainable long-term protections in place and fighting fires

Our leverage on the future is high just now.

A discount rate that depends on the interest rate rather than the growth rate.