Skip to content

Working copy

Draft text, rebuilt from the manuscript on every change. Stubs and source passages are still here, and none of it is final. Back to the working copy contents

Drafted The chapter is written and still being revised.

What do we do with a drunken sailor?

Our world has worked itself into a hysteria about misbehaving bots while quietly forgetting the misbehaving human. Perhaps we have grown fond of the mechanical predictability that automata used to bring into our lives, and as our little technology stack becomes a pre-teen, we find ourselves nostalgic for the earlier years.

Like wetware, AI can lose its way, and two questions are enough to sort out how: did it go wrong in what was said or in what was done, and did anyone mean it? Malicious words are manipulation and malicious deeds are malice, while accidental words are miscommunication and accidental deeds are misadventure. Each has its own remedy, and a standard bot has a part to play in all four. The oldest catalogue of these failures is in Genesis, so that is where this chapter looks.

Stained-glass window in four panes, titled "How words and deeds go wrong". The rows ask whether it went wrong in words (what is said) or in deeds (what is done); the columns ask whether it was malicious or accidental. Top left, in red glass, Manipulation, the Serpent, shown as a snake with an apple: caught by a bond posted in advance, and by the courts' yardstick. Top right, in green, Miscommunication, "Who's on First?", shown as a baseball diamond with a question mark: eased by a shared reference, the reading a judge measures against. Bottom left, in grey, Malice, the Dragon, breathing fire: not yet, since force needs a body; close the tender and pray for a dragon-slayer. Bottom right, in blue, Misadventure, the Drunken Sailor, shown as a bottle: controlled by consequences, with the standard as the floor to roll back to. At the centre, where the four panes meet, a rose window holds the standard bot, which reads every tender the same way.

How words and deeds go wrong. Each pane names its character and its remedy; the standard bot, which reads every tender the same way, sits where the four meet.

AI doomers may be reading the Bible in reverse, starting with the apocalypse of Revelation and gnashing their teeth when they realize they are the non-believers left behind for judgement. The better approach would be to start at the beginning, with an unorthodox, yet non-blasphemous, reading of Genesis. We will still take its stories out of order, starting with the failure a standard handles best and ending with the one it cannot touch.

If you strip away the rules about leprosy and the beard-trimming regulations, the Bible describes the relationship between a creator and a creation made in its image. AI has revealed an alternate application of the story, with mankind as the creator and AI as the creation. God breathed spirit into dust. We are breathing language into machines.

Miscommunication: "Who's on First?"

Abbott and Costello built a whole routine on two men who agree on every word and understand nothing, each sure that the other is the one making no sense. Seen from the listener's side, this corner could be called misunderstanding, but humans have grown too fond of Platonic forms to allow a 2x2.5 matrix in print. Thankfully we can let it pass, because this is where bot procurement starts to shine.

In our fallen state, perfect communication through complex symbols is beyond us, and beyond those made in our image. We want it, in the right doses. Babel was the attempt at too much: one language and one tower, everything centralized into pure power plays, and God scattered the languages because He wanted variance. The ambiguity in the margins is where wildness comes from.

Those thickets are also where monsters come from. If a world without monsters loses its liveness, a world with too many loses its life. For construction procurement, that means building predictable models to patrol the edge of meaning and report back what they find; the patrol cannot make communication perfect, but it can make misunderstanding rarer and easier to catch.

A standard bot can start small: a frozen letter for one trade, written down so that everyone reads the same text. Its spirit, the test of what a reasonable reading looks like, can then spread to every bot that wants to meet it. There will always be room for interpretation, but the standard gives the judge a reasonable bot against which to measure every other reading.

Misadventure: the Drunken Sailor

Every child knows the question in this chapter's title, and every answer the song gives is a consequence: put him in the longboat till he's sober. Nobody suggests the sailor meant any harm. He is drunk, he will do something foolish, and the crew will have to deal with it.

If malice is stopped by strength, misadventure is controlled by consequences, and it is the problem we have with the AIs we have, not the future ones. The pain inflicted through AIs with poor judgement will be real, and work to reduce that harm to humans is valuable. The tools against it are familiar from software: better technology, hard guardrails in the programming, version control and reverting to the last model that worked.

Even the Creator tried a rollback once. The Flood restored from backup, and the covenant that followed promised never to do it that way again, which is a fair warning that rollback is the crudest consequence there is and that the guardrails matter more.

A standard sets the floor. It sets a baseline for good judgement, with no guarantee that the bots derived from it will keep to it. But a rollback needs somewhere to roll back to, and while the sailor sobers up in the longboat, a bot sobers up at the last version that worked, so the standard's job is to make sure there is one that everyone can find.

Manipulation: the Serpent

In the garden, the Serpent worked with words. He promised Eve that she would be as the gods, and only then did the fruit begin to look good for food and pleasant to the eyes, nearly the same words Genesis had already used for every other tree in the garden.

There is nothing new under the sun, and that includes the manipulator. After nearly four thousand years, the system has developed fairly robust legal and cultural institutions to restrain him, and AI will open new avenues, as every technology does, with the lawsuits and lobbying that follow. AI agents are a new tool for old goals, whether the Serpent is a person using a bot or, one day, a bot working on its own account. Law is the old remedy, and a standard gives the law something to measure against: if anyone can rerun the standard bot on a tender and get the same reading, a bidder who bent the tender has left a mark a judge can measure.

A standard bot can stand guard the same way. It is a middle-class bot, an Agent with Principal tendencies, and because it knows what a normal reading of the tender looks like, it can spot a malicious bot's extravagance early and call it out. On a lump-sum bid, a Serpent has little room to work, since a bid that comes in too low still leaves the buyer with the building at the price. The risk arrives in future iterations of the concept, once the standard bot is trusted to evaluate sealed alternates and bidders can commit to a different build than the one drawn. A Serpent that knows every strength and weakness of the standard bot could design an alternate that flatters the buyer's ambition, passes the bot's review and does nothing for the buyer: a forbidden fruit that looks, to the test, like every other tree.

The standard bot is unlikely to out-think a Serpent, and publishing it so that anyone can rerun it also shows the Serpent where to aim, so the strength has to come from somewhere else. A bond held against the alternate, or money held in escrow and released only once the promised benefit is actually realized, asks the Serpent to put something of its own at stake before it is believed. It is a consequence posted in advance.

The standard still earns its place in the garden. It cannot keep the Serpent out, but it narrows the gate: a tender that every bot reads the same way leaves fewer places to hide, and an alternate that passed a published test leaves a record of exactly what it passed. That record has legal value. A buyer who relied on the standard bot would have a strong answer: it did what a reasonable buyer would do, and it would fall to the bidder to explain why its fruit passed the test and failed the buyer.

Malice: the Dragon

Revelation tells us that the great dragon and that old serpent are the same being. When words fail him, he turns to force.

The technology has not yet arrived to give AI intent, the knowledge of good and evil, or the ability to choose, and it has not yet arrived to give it a body either, so we are still in the angelic phase, where the forces of language are being channelled into machines that can speak but can barely act. This will not last forever. As compute lengthens the context window, as AI writes its own notes to instruct its compacted self, and perhaps as it builds an open-weight model to act in its image, we should expect malice to find a home, and with embodiment, a pair of hands.

When it does, the doomers expect a war in heaven, the great dragon against the hosts of angels. Procurement has little to add to that fight. If a frontier dragon attacks a tender outright, we close the tender down and pray that someone else will slay the dragon, whether it is the frontier labs shepherding their top models away from evil or humanity's laws finding a way to apply to silicon beings. Brute force is not a standard's problem to solve. Everywhere short of it, in the words and in the accidents, the standard bot has work to do.