Nasdaq OMX recently agreed to pay $10 million to settle civil charges resulting from a software error that ultimately triggered millions of dollars in losses during Facebook’s initial public offering in 2012, according to U.S. securities regulators. The settlement constitutes the largest penalty ever carried out on a stock exchange, Reuters reported.
In administrative proceedings and documents published by the U.S. Securities and Exchange Commission, the agency described a series of “ill-fated decisions” and technical errors that led to more than 30,000 orders for Facebook stock becoming stuck in Nasdaq’s system, Reuters reported. Investors and market makers lost an estimated $500 million as the market continued trading despite observing the errors. The exchange agreed to pay $10 million to settle SEC charges without a dispute.
“This action against Nasdaq tells the tale of how poorly designed systems and hasty decision-making not only disrupted one of the largest IPOs in history, but produced serious and pervasive violations of fundamental rules governing our markets,” said George Canellos, co-director of the SEC’s enforcement division, according to Reuters.
The exchange has already separately agreed to pay out as much as $62 million in compensation to those who lost money in the glitch, Reuters reported. Nasdaq is also is in arbitration with UBS, which is trying to recoup additional compensation after losing $300 million.
New visibility on the glitch
An SEC document detailing the failed IPO offered additional insight into the events of May 18, 2012, that led to the losses and frustration. The problem arose during the “Display Only Period,” a 15-minute window before the stock begins trading that firms can use to commit to orders and name their price. During this period, firms can also decide to cancel orders. As the document explained, at the end of the DOP Nasdaq’s “IPO Cross Application” runs a check of all the orders to determine the price at which the largest number of shares will trade. This check takes around one to two milliseconds.
At the same time, the Nasdaq runs another validation check that confirms the orders in the IPO Cross Application matched those in the exchange’s matching engine. However, one reason the orders may not match is that Nasdaq allows orders to be cancelled at any time up until the end of the DOP, including the time it takes to run the IPO Cross Application calculations. Any mismatch would cause a validation failure and force the IPO Cross Application to run again. Additional orders and cancelations could then be filed in the time it takes to rerun the application.
Normally, this process would only run once or twice before successfully producing a calculation that satisfies the validation check. However, if orders continued to come through, the process could potentially produce a loop that would prevent the completion of the cross. The massive volume of orders during the Facebook IPO exacerbated this problem. The IPO cross calculation on the day of the Facebook IPO took 20 milliseconds, instead of the normal one or two, enabling a cancelation. Further cancelations occurred as additional validation checks were made.
When failing over fails
Nasdaq engineers determined that the problem was a validation check error but did not realize the specific cause (i.e. the loop from cancelations). They recommended using a failover system that would remove the validation code and enable trading to go forward. However, all cancelations and orders made during this validation and troubleshooting period – which lasted 19 minutes – were caught in limbo. Ultimately, around 30,000 orders were “stuck.”
The result, ultimately, was a series of massive losses and regulatory violations, ending in the $10 million settlement Nasdaq was forced to pay this year. Given the high stakes of such errors, ensuring that issues such as validation loops are eliminated from software is essential for developers. Using tools such as static analysis software, organizations can catch potential glitches and avoid costly fallout.
Software news brought to you by Klocwork Inc., dedicated to helping software developers create better code with every keystroke.