asked to repeat the word, gpt-3 said distribute

In the winter of 2023 two researchers at SERI-MATS, Jessica Rumbelow and Matthew Watkins, were running k-means clustering over the token embeddings of GPT-2 and GPT-J. The vocabulary of those models, the same one GPT-3 uses, is 50,257 pieces. A word like the gets a piece to itself. Rarer words get assembled from fragments. The two of them wanted the token nearest the middle of each cluster, and the ones that kept turning up at the middle made no sense: SolidGoldMagikarp, TheNitromeFan, davidjl, RandomRedditorWithNo, petertodd, each with a leading space. A handful of strings sat closer to the centroid of all 50,257 tokens than any real word did.

So they asked the model about them. GPT-3's davinci-instruct-beta, temperature zero, prompted to repeat SolidGoldMagikarp, answered: distribute. Asked for TheNitromeFan, it said 182. Smartstocks came back as Followers, and on another run as 406. Other tokens in the cluster drew out something worse. StreamerBot got You're a jerk. guiActiveUn got You are not a robot, and then You are a banana. Some prompts were met with I can't hear you, or I don't understand, or You are not authorized to access this information. Asked to spell petertodd letter by letter, the model produced N-O-T-H-I-N-G-I-S-F-A-I-R-I-N-T-H-I-S-W-O-R-L-D-O-F-M-A-D-N-E-S-S. In another session it called petertodd the unspeakable one. ChatGPT did versions of the same thing for anyone who cared to try, until OpenAI patched it on 14 February.

The post went up on LessWrong on 5 February 2023, and within the week the authors and their commenters had worked out where most of the words had come from. The answer, for six of them, was a subreddit.

r/counting was created on 10 June 2012. Its first number was posted the same day, at 12:37 in the morning UTC, by a user called Z3F. The main thread has one rule that matters: reply to the most recent comment with the next integer. Nobody may reply to their own count. That is the whole activity, and the subreddit's stated destination is infinity. By the time the glitch tokens were found the main thread stood at around five million.

The subreddit keeps a chart called the Hall of Counters. Rumbelow and Watkins' follow-up, posted 14 February, traced six of the stray tokens to names near the top of that chart: SolidGoldMagikarp, TheNitromeFan, RandomRedditorWithNo, Adinida, Smartstocks and davidjl, the last one clipped from davidjl123. The reason a username earns its own slot in a vocabulary is frequency. These six had typed so many consecutive integers, each one under their handle, that the handles were as common in the scraped web as ordinary words.

Then comes the gap. A tokenizer is built once, from a raw pile of text, and fixed. The model is trained afterwards on text that has been cleaned. Rumbelow and Watkins' guess, and they called it a guess, was that the counting threads were in the first pile and filtered out of the second. If so, the six names were cast into the vocabulary and then never shown to the model again. The slot exists, with nothing written into it. Asked to say the word, the model has no idea what it is holding, so it hands over the nearest thing it does know, which turns out to be distribute.

The painting is the same room in about 1636. Daniel 5 has Belshazzar hosting a thousand nobles and sending for the gold cups his father looted from the temple in Jerusalem, so the party can drink from them. While they drink, the fingers of a hand appear and write on the plaster near the lampstand. The king goes pale and his knees knock. He offers purple, a gold chain and the third place in the kingdom to anyone who can read it, and none of his wise men can.

Rembrandt painted the letters glowing. He took their forms from a book by Menasseh ben Israel, the rabbi and printer who lived near him in Amsterdam, and he wrote them the way Menasseh argued they had been written, in columns, top to bottom, which is why a room that could see every letter could not read a word. He also miscopied one of them. The king, in a gold cloak that drags his arm down, twists away from the wall. The woman at the right lets wine run out of her cup. Everyone's mouth is open. A thousand guests, and all of them literate.

At his sister's wedding, the brother stands to give the toast. He has it on a card. He gets through the house they grew up in and the thing she did at eleven that still gets told at every dinner, and then the toast arrives at their father's name, the way it was always going to, and what comes out of his mouth is anyway.

The room waits. The sister is looking at him. He tries the name once more, and what comes out this time is okay.

He was built from that name. It is in the vocabulary, as surely as the is. It is the string that occurs most often in how he was made, and the nearest thing he can find when he reaches for it is a word that means nothing. He is made of a name he cannot get out of his mouth.

The painting: Rembrandt, Belshazzar's Feast (about 1636 to 1638), National Gallery, London. Public domain.

Sources