5: AI may pursue goals

Suppose that, as argued previously, in the next few decades we’ll have superintelligent systems. What role will they play?

One way to imagine these systems is purely as powerful and versatile tools, similar to most current systems. They could take broad directions from humans about what actions to take or what questions to answer, and cleverly fill in the details.

But another way is as agents, operating autonomously in the world. They could have their own goals — some kinds of outcomes they seek out over other outcomes — and take whatever actions will most likely lead to those outcomes, adapting as circumstances change.

As long as AIs are tools, they can be used for good or ill, like all technologies. But it’s unlikely that they’ll remain only tools, because:

  • A good planning tool can easily be turned into an agent. Just tell it: “repeatedly come up with actions that would make goal X more likely, and execute those actions.” People keep trying to build software frameworks for doing this; as of July 2025, success has been limited, but future improvements to our tools and frameworks may result in much more powerful agents.
  • At some point, keeping humans in the loop to direct an agent’s activities will just get in the way. It will make the system less efficient, less profitable, and less able to compete. For example, a high-frequency trading system that had to wait for human approval at each step would miss out on a lot of good trades.
  • There may be some tasks that tools just can’t do unless they’re basically already highly intelligent agents. For instance, a scientist pursuing a groundbreaking research program needs to do open-ended exploration, long-range planning, making sense of noisy, messy feedback, and dealing with unforeseen obstacles, in ways that “tool-like” AIs struggle with.
  • Increasingly agent-like systems are already being created — see, for example, OpenAI’s Operator, which navigates websites in a human-like way, by looking at the content and deciding where to click, and Anthropic’s “computer use” feature for its chatbot Claude.
  • Even if most AIs remain non-agentic, some people will create agents for various reasons of their own — to be the creator of a new species, or for the heck of it.

If we’re going to build AI systems that pursue goals, it would be good if we could make those goals match ours. It’s not clear if we’ll succeed at that.



AISafety.info

We’re a global team of specialists and volunteers from various backgrounds who want to ensure that the effects of future AI are beneficial rather than catastrophic.

© AISafety.info, 2022—1970

Aisafety.info is an Ashgro Inc Project. Ashgro Inc (EIN: 88-4232889) is a 501(c)(3) Public Charity incorporated in Delaware.