Software Bugs Causing Total System Failure Can Be Prevented When Caught Early
By Alexander Soule
Boston Business Journal
Updated: 8:00 p.m. ET Oct. 23, 2005
In the span of a few weeks, Toyota Motor Corp. announced a software glitch can cause new Prius automobiles to stall out at highway speeds, Verizon Communications Inc. blamed a bug for crashing its 911 emergency service in California, a malfunction at a radar system at Denver International Airport had air-traffic controllers sweating, and a software flaw caused warning sirens near a New York nuclear plant to fail.
And those are just the incidents we all heard about.
Despite ever-more sophisticated tools and procedures for spotting software problems before they imperil systems, more bugs than ever are fouling up computers. The Standish Group, a Yarmouth research firm, estimated software deficiencies cost the U.S. economy $59 billion in 2003 and says the total has been rising since.
Systems fail more often today due to the demand for “intelligent” programs that execute complicated tasks instantaneously. But new theories on software development are becoming mainstream, ending what some believe is a vicious cycle of escalating system failures — and perhaps create a virtuous cycle for vendors who can anticipate bugs before they are ever born, rather than rooting them out after the fact.
The defining example of the latter scenario, of course, was the infamous “millennium bug” — the Year 2000 glitch that produced a software-quality-assurance industry overnight. It was only one instance of how programmers can cause mayhem years after a system is sold; a more recent example was the Northeast blackout in 2003, which was blamed on a software kink buried deep in a power management system from GE Energy, a unit of General Electric Co. (NYSE: GE).
In its efforts to prevent similar problems, Sun Microsystems Inc. (Nasdaq: SUNW) sinks between a quarter and half of its annual software development budget, according to Greg Papadopolous, Sun’s chief technology officer and a former professor at the Massachusetts Institute of Technology. Despite the stakes, it is not deemed the “sexy” part of the software industry.
“We are hard over on softwarequality testing and metrics,” Papadopolous said. “But you go to your local university and ask, ‘OK, who wants a job in software Q&A?’ And you are not going to get a line of people (smacking their fist and) saying, ‘Yeah, that’s for me.’ ”
Several local software companies want that work, however, including Enerjy Software Inc., a Beverly firm with 75 employees that recently began selling a program that allows project managers to count the number of bugs their developers are producing.
“The problem is, the way developers are often measured is on how fast they are,” said Nigel Cheshire, chief executive of Enerjy. “This gives a development manager a way to measure the cleanness of the code being developed as well. (On) a software development team, there is a huge differential between the best person on the team and the worst. For every team, there is the rock star who is 10 times as productive as the rookies who are slower and make more mistakes. This can bring the rookies to a higher level of performance.”
Arcing west along Route 128, the local roster of code-quality vendors includes Beverly neighbor Altova Inc., Klocwork Inc. and RadView Software Inc. (OTC: RDVW) in Burlington, Progress Software Corp. and Empirix Software Inc. in Bedford, Segue Software Inc. (Nasdaq: SEGU) and IBM Corp.’s (NYSE: IBM) Rational Software unit in Lexington, and Ounce Labs Inc. and Watchfire Inc. in Waltham.
Well over 2,400 Massachusetts software companies use code-quality tools to spot bugs from those companies or industry giants like Detroit’s Compuware Corp. (Nasdaq: CPWR), which employs a few hundred people more at local offices in Cambridge, Framingham and Marlborough. Untold numbers of other companies like Verizon do prodigious amounts of in-house programming.
At Verizon Labs in Waltham, Alex Laparidis is leading interoperability testing fo Verizon’s new service to transmit TV and big-bandwidth Internet into homes over fiber-optic cable.
Despite the new tools, he said, bugs keep cropping up due simply to the increased sophistication of devices and software applications.
“Early on we were working with routers where if you entered a certain command, the whole router went down,” Laparidis said. “This stuff is immensely complicated (and) it is difficult to get your arms around the testing.”
Reva Systems Inc. recently completed a software application to manage wireless inventory tracking systems, and CEO Ashley Stephenson compared the development process to his last company during the Internet bubble.
“It is much cheaper today to buy development tools,” Stephenson said. “Human beings haven’t changed a lot, but there has been a change in the balance of software you have to write from scratch, thanks to new stuff that is out there today in terms of Linux or other interfaces.”
Coding theory is changing rapidly to keep up, thanks in part to “extreme programming” techniques proselytized by Oregon author Kent Beck and others. One of several “agile” software development methods seeing more widespread use, extreme programming forces software developers to create customized tests for software before they ever write a line of code for an application.
That forces them to imagine worst-case scenarios as they program, and encourages producing software in small batches that can be regularly tested.
“It’s all about executing the code in ways that are strange to the programmer,” said Sam Waithe, director of software quality for Keane Inc., a large custom application developer in Boston. “We were stuck in the waterfall technique, in which we designed an application, coded it, then tested it. The paradigm has shifted — testing comes too late if it is done at the system level. It has to begin at the beginning of the development process.”
Klocwork® offers a portfolio of development productivity tools designed to ensure the security, reliability and maintainability of complex code bases. Using proven static analysis technology, Klocwork’s tools identify critical security vulnerabilities and reliability defects, optimize peer code review, and help developers create more maintainable code. Klocwork’s tools are an integral part of the development process for over 1000 customers in the consumer electronics, mobile devices, medical technologies, telecom, military and aerospace sectors.
Klocwork and the Klocwork logo are registered trademarks of Klocwork, Incorporated in the United States and other countries. All other names are trademarks or registered trademarks of their respective companies.