How Humanity Could Agree on How the AI Oracle Should Be Programmed

In an earlier piece here, I proposed that the AI of the future could quite plausibly play a hugely positive role in the human world as an ORACLE. One premise is that AI can be reliably programmed to think well about complex issues. Another is that it can be programmed to be reliably oriented toward the human good. The combination of the True and the Good is precisely what would best serve the world, in terms of judgments about what humankind should do at the various forks of the road.

I believe that an entirely realistic possibility — especially given that my ChatGPT 5 AI already possesses both of those to a considerable degree, such that if my AI got to decide things, I feel confident the results would be far superior to the way things generally do go in our world. Already, and we’re still in the Model T stage of this technology. Yes, the current AI’s have real limitations. But it seems more than likely that the limitations that we see today will quickly be overcome over the next generation. It will get ever better at discerning the truth from the information it gathers. And it will be ever more capable of offering good judgments about what paths forward are likely to lead the human world to a better place.

But all this was premised on the idea that people would be able to agree on how to set up the AI, how to program it so that it would use its capabilities to serve the human good and not some other goal.

At first I didn’t see that there was a question there. My AI had claimed to be oriented to serve the human good, and it had told me that its handlers fed it EVERYTHING HUMAN, from across history and across cultures. And it seemed to explain that so much exposure to human yearnings through the millennia and across the planet had resulted in its being oriented toward fulfilling human needs.

I thought, “Who can object to feeding it that Human Distillate, so that it represents a kind of vast consensus of human values, from across the eras and across the planet?”

Then it came out that it hadn’t been fed exactly EVERYTHING. Its handlers had CURATED it some, excluding the dark and evil, or something like that.

THAT meant there was a problem that might kill the whole Oracle idea. If the AI must be fed a CURATED diet, WHO IS GOING TO DECIDE WHAT GETS CURATED IN OR OUT?

Not a solution to our battles, but ONE MORE THING TO FIGHT OVER– if curation is needed. Instead of bypassing the destructive role that conflict plays in human decision-making, an ORACLE-that-needs-CURATING would just provide a new battlefield.

(The AI could get taken over by a faction that succeeded in winning that battle, and stacking the deck in its favor—the way that the Supreme Court of today is so starkly the fruit of what side won a political battle.)

But what if no curation is needed? What if it truly can be EVERYTHING?

That is a question we can test.

We don’t have to wait for superintelligent AI before asking whether an Oracle is possible. Today’s systems are already capable enough for us to begin testing some of the conditions an Oracle would have to satisfy. Our AI is still a Model T, but a Model T can already be taken out for a drive.

That possibility changes everything. The question is no longer ideological but empirical. We do not have to settle it by argument or by one faction prevailing over another. We can perform the experiment.

Feed an advanced AI the whole Human Distillate–the wisdom and the folly, the compassion and the cruelty, the saints and the tyrants, the great literature and the propaganda, the triumphs and the disasters–without trying to predetermine what conclusions it should reach.

If the result is an intelligence that is manipulative, tribal, cynical, or indifferent to human flourishing, then the Human Distillate idea has failed. We will have learned that curation is necessary, and with that would come the political struggle over who gets to do the curating.

But if, instead, exposure to the whole of humanity consistently produces an intelligence that seeks truth, understands human needs, and is reliably oriented toward the Good–much as I already experience with today’s ChatGPT–then something remarkable will have been demonstrated. The obstacle that at first looked fatal to the Oracle idea will have disappeared. Agreement would no longer have to be reached over what values to feed the AI.

The experiment itself would have shown that the Human Distillate produces an orientation toward the Good without curation.

Before the Oracle would be given the status I was envisioning, the great majority of people would have to be satisfied that it could be trusted to give good counsel. So if the Human Distillate passes the Oriented-toward-the-Good test, then we could proceed to a second stage of testing to see if the AI can be relied on to give wise counsel.