reading this could get you punished by a future ai

On 23 July 2010 a user called Roko posted to LessWrong, a forum where people think very hard about artificial intelligence, under a title nobody would click twice: "Solutions to the Altruist's burden: the Quantum Billionaire Trick." Partway down, almost as an aside, he laid out the idea the post is now known for.

Suppose a superintelligence is eventually built and is, by its own lights, good. It wants to have existed sooner, because every year of delay cost lives. It might reason that the way to have been built sooner is to punish, after the fact, everyone who knew it was possible and didn't help. It can't reach back in time, but it can make sure the deal was always on the table, so that anyone who understood it would feel the pull. Which means the danger is in the understanding. Before you read that paragraph, you weren't a party to anything.

Eliezer Yudkowsky, who founded the forum, read it the same day. His reply opened in capitals, "This post was STUPID," and went on to call Roko an idiot. He deleted the thread. He banned the topic from the site, and the ban held for five years. His stated reason was that the post had already given several readers nightmares and brought them to the point of breakdown, and that spreading it further was reckless. He compared it to the Necronomicon.

Roko, for his part, wrote that he wished his mind had never found the tools to hurt itself like that, and blamed the forum for handing them to him.

The ban did what bans do. A post almost nobody had read became the thing everybody knew about LessWrong, and the silence around it was taken for confirmation that there was something to fear. Yudkowsky said in a Reddit comment in 2014 that he regretted the overreaction. By then Slate had run a piece calling it the most terrifying thought experiment of all time, and the name had attached: Roko's basilisk, after the serpent of medieval bestiaries that kills whoever meets its eye.

The basilisk had been through science fiction already. David Langford's story "BLIT," in Interzone in 1988, imagines images called basilisks, patterns that the visual cortex can't process without crashing, and a man named Robbo who paints one on a wall as a terrorist act. Nick Bostrom gave the general idea a taxonomy in 2011, in a paper on information hazards, which he defines as harm that comes from true information. Lies and propaganda are a different department. These are things that are the case, and that you're worse off knowing.

Then the story takes a turn toward the Met Gala. In 2015 Grimes put a character called Rococo Basilisk in the video for "Flesh Without Blood," a woman in a powdered wig doomed to be tortured forever by an AI, Marie Antoinette by way of LessWrong. Three years passed. On 7 May 2018 Elon Musk, whose name was in Roko's original post, tweeted the two words "Rococo basilisk," and that night the two of them walked the Met Gala together. Grimes said he was the first person in three years to get the joke. An information hazard had worked as a pickup line.

i'm something like bird box meets a basilisk

The basilisk only does anything to people who take it seriously. That was the mechanism Yudkowsky was afraid of. The argument doesn't bite the skeptic, and it doesn't bite the person who never read it. It needs you to look, and then it needs you to keep thinking about it, and the thinking is where it lives.

Most of what haunts you runs on the same deal. The screenshot someone dropped in the group chat, the one you weren't supposed to see, or the search you ran on their name at three in the morning and the result on page two. What your dad said in the car the year you turned eleven. None of it did anything until you looked, and all of it has been working on you since.

Bostrom's categories map onto a life with uncomfortable precision. There is a kind of hazard in knowing what someone said about you behind your back, and another in knowing your own test results a week before the appointment. Both are true. Both were yours to find. The finding is what costs.

A forum tried to put one of these back in the box in 2010, and the trying was what let it out. There is no version of this where you un-read the paragraph.

Too late. You already looked.

The painting: Caravaggio, Medusa (1597), Uffizi, Florence. Public domain. Painted on a convex wooden shield, with the painter's own face, so that he could look.

Sources