a program nine years dead stood up at 9:30

In the last week of July 2012, a technician at Knight Capital in Jersey City copied a new piece of software onto the company's order router, a few servers a day. The router was called SMARS. It took the orders Knight's customers sent in, split each one into smaller child orders, and fired those out to the exchanges until the parent order was filled. There were eight SMARS servers. The technician did the copy by hand, and nobody was required to check the work afterward, because no written procedure at Knight said so. The new code reached seven of the eight.

Knight was not a small shop. Through 2011 and 2012, roughly one share in every ten traded on American exchanges passed through its systems.

The new code existed because the New York Stock Exchange was opening something called the Retail Liquidity Program on 1 August, and Knight wanted its customers in on day one. Inside SMARS the code was written to take over a slot occupied by an older routine named Power Peg. Knight had stopped using Power Peg around 2003. It had never been deleted. It sat on the servers, present and callable, through nine years of upgrades.

Power Peg had once known when to stop. As child orders were filled, a counter tallied the shares executed against the parent, and when the tally matched, the routine quit. In 2005 someone moved that counter to an earlier point in the SMARS sequence, where the live code needed it. Nobody tested whether Power Peg still worked without it, because nobody expected to call Power Peg again.

The new program was switched on by a flag carried in each incoming order. Rather than add a new flag, Knight's engineers reused the old one, the flag that had once meant run Power Peg. On seven servers it now meant the new program. On the eighth it still meant what it had meant in 2003.

At 8:01 on the morning of 1 August an internal system began generating emails. They were reject messages from a component called BNET, and each carried the phrase Power Peg disabled. Ninety-seven of them went to a group of Knight staff before the market opened. The SEC's order on the matter notes that Knight had not designed these messages to be alerts, and that staff generally did not read them. The messages were the eighth server announcing itself, in real time, for ninety minutes, to people who had filed that sender under noise years before.

At 9:30 the bell rang and the retail orders arrived. The ones with the flag set that reached the eighth server triggered Power Peg, and Power Peg did what it was built to do, which was to send child orders for each parent until the counter said stop. The counter was no longer there. So it sent child orders continuously, in rapid sequence, while another part of Knight's system, which could see that the parents had been filled, had no way to tell SMARS.

For 212 parent orders in all, Power Peg sent millions of child orders, and over about forty-five minutes Knight obtained four million executions in 154 stocks, more than 397 million shares. In 37 of those stocks Knight was more than half the trading volume and moved the price by more than ten percent. The positions piled into an internal holding account called the 33 Account, which had a two-million-dollar limit that was not wired to anything able to refuse an order. The risk screen Knight's senior staff watched, PMON, showed the account swelling, but it was a display rather than a brake. It did not show the limits, and it fell behind under heavy volume, so the numbers on it were wrong anyway.

Knight's technology team went looking for the fault in a live market. One of their attempts was the obvious one: pull the new code off the seven servers where it had been installed correctly. That removed the only thing standing between the flag and Power Peg on those servers too. Now all eight were running the routine from 2003.

okay that's a dead subject you bring it up that's a zombie

By the time the orders stopped, Knight was net long about $3.5 billion in 80 stocks and net short about $3.15 billion in 74. Closing the position out cost a realized pre-tax loss of approximately $440 million, by the company's own count the next day. The shares, which had traded above $12 three weeks earlier, were at $2.85 before the open the following Monday. That morning Knight announced $400 million in new money from six firms led by Jefferies, in preferred stock that converted into a little more than seventy percent of the company. Eleven months later Knight stopped existing as an independent firm, folded with Getco into KCG Holdings. In October 2013 the SEC fined Knight $12 million for lacking controls that could have stopped it, and noted along the way that in October 2011 one of Knight's desks had kept quoting from weekend test data on a Monday morning, which cost $7.5 million and produced a checklist item.

The eighth server is the part that stays. It had no bug. Its software was nine years out of date and its counter had been moved in 2005, and when an order came in with the old flag set to yes it did exactly what it had been told to do by people who no longer worked that way, if they worked there at all. The signal that woke it was the one piece of the old arrangement anyone had kept.

nobody deleted it. they just stopped calling.

The painting: Sebastiano del Piombo, The Raising of Lazarus (1517-1519), National Gallery, London. Public domain.

Sources