Putting It To The Test

Eighteen months from now it will be easy to spot the executives who have labored mightily to ensure that computer systems operate beyond the stroke of midnight. Theyll be among the first to hoist a second glass of champagne -- not to celebrate their achievements, but to ease the dread that their work has been in vain. If experts are correct, the year 2000 (Y2K) will dawn with many computer systems having been modified but not tested. Given the high error rate in programs that have been tested, that will mean that systems failures will be widespread, affecting virtually all companies. Even those companies that have gotten their own houses in order may be blindsided by business partners who will have run out of time and money before fully checking their systems -- thats the dark side of supply-chain integration. "About half the companies we survey dont plan to test the code theyve fixed for Y2K," says Lou Marcoccio, Y2K research director for the Gartner Group Inc., a consulting and research organization in Stamford, Conn. "Theyre just going to put it in production, and thats scary." Marcoccios figure isnt just a snapshot or a rough estimate: Gartner monitors 15,000 companies in 23 vertical industries and 87 countries. Time and money are the biggest reasons so many companies seem likely to ignore this crucial phase of software development. Experts say that testing can consume anywhere from 40% to 80% of the total resources needed for a successful Y2K conversion. Another reason is poor planning. "Companies wont be able to simply buy themselves out of this problem," says Richard Bergeon, executive vice president-product management with Systemic Solutions Inc., Seattle. Even if the money is there, the time wont be. "If youre still changing the code as 1999 begins, youll reach the millennium before youve been able to test everything." There is also a corporate-culture issue. Many companies do little or no testing of the software they write. They put it into production and swat the bugs when they appear. As a result, there are no procedures -- and no staffers -- to tap for critical testing. And critical it will be, because while the fixes may be easy, testing them is not. In fact, to some degree it is precisely because the fixes are easy to make that errors creep in. "Going through programs looking for dates that need to be changed is very tedious," says Jim Woodward, a senior vice president at consulting group Cap Gemini America, New York. "About 80% of programmers say they find the work very boring, and that leads to careless mistakes that can sit there, like cancer in remission, waiting to cause problems later." Programmers each tend to tackle their piece of the pie a little differently, and that inconsistency can lead to trouble later on. The situation may be worse for companies that have farmed out the Y2K fixes; often the companies that have sprouted up overnight to tackle such work make no guarantees about how well their fixes will work in a fully integrated production environment. "Companies are much more on their own when it comes to testing," says Bergeon. "Outsourcers take no responsibility for the final result. You as the client are on your own, with ultimate responsibility for testing." Third-party companies may test the changes theyve made to any one program, but thats a far cry, for example, from running a payroll program and a benefits program against the human-resources database to see if all systems will be "go" in the next century. Its not that outsourcers intentionally flee from such work, but integrated testing requires company involvement -- and in most cases total project ownership -- in order to be done well. Analysts say that, in general, the larger the company, the better the situation. "Most large companies at least have a process in place and are moving forward," says Lynn Holloway, vice president and director of quality assurance for RCG Information Technology Inc., Edison, N.J. "But small and midsized companies are just beginning to realize the magnitude of the testing issue." Even large companies can encounter unpleasant surprises. Fina Inc.s Fina Oil & Chemical Co., Dallas, for example, has been plugging away at its Y2K conversion effort for almost three years. And the company has known for some time that testing is a major part of the project. As early as 1997 Fina had a plan in place to lease mainframe computer space from a third party in order to run through the full range of testing it believed was necessary. "We were just about ready to start," says Jack Sanders, group leader and project manager for Y2K applications conversion, "when we ran into licensing issues. Our software contracts often stipulated that the software run in one data center only, not two." So Fina carved out some space on its own mainframe to do the requisite testing. The company was able to change its strategy largely because it began tackling the Y2K issue early and took the time to develop clear strategies. In fact, Fina decided to move to a new version of Cobol, the language many of its programs are written in, at the same time that it sorted through 16 million lines of code looking for dates that needed changing. "With this new version, MVS Cobol," Sanders says, "we can accommodate a four-digit year, so well be in good shape by years end when all our changes are made and were focused purely on testing." Fina will be in good shape in terms of its software, anyway. Sanders says that the company did not get as early a jump on Y2K issues that might affect its embedded systems and its supply-chain operations. "In late 95 we thought Y2K was just a software issue," he says. "Now we see that embedded chips and the supply chain are big issues as well. Weve only been addressing those for about eight months." Fina may have an advantage in that while its big enough to have the resources to meet the Y2K challenge, it is also small by petroleum-industry standards. "We dont have as many locations or as many partners," says Gary Bussell, senior group leader at Fina, "so were confident that we can catch up." Fina has another advantage: executive leadership. Analysts say that those companies in the best shape when it comes to Y2K are those that have sent a strong message from the top that Y2K compliance must be a priority. "The CEO of Petrofina has sent the word out globally on this topic," Bussell says. Global companies such as Fina worry that their own Y2K fixes wont mean much if their supply-chain partners arent up to the task. At Mason Co., Chippewa Falls, Wis., Y2K project manager Bruce Fineseth says his company is sending questionnaires to its supply-chain partners as a way of having "a basic compliance statement and a way to know if well have supply-chain issues. If we get a sense that someone isnt dealing well with Y2K, well get a new supplier." A relatively small mail-order footwear manufacturer, Mason knows that testing its changes will be vital. "Were working to make the changes by years end," Fineseth says, "so that we have all of 99 to test." Unlike Fina, which plans to take its code through a relatively complex series of tests before putting it to actual use, Fineseth doesnt rule out putting some of Masons altered programs directly into production as a way to give them real-world shakedown. "But with a year to test," he says, "we should be O.K., even if we do encounter some gotchas." Although Fineseth is confident that the fix-now, test-later approach will work, analysts such as Bergeon caution that some companies may be setting themselves up for a rude surprise. "Some companies are waiting until they have a compliant test environment," Bergeon explains, referring to the Catch-22 of trying to test applications on computers that have inherent Y2K problems. "But while theyre waiting for vendors to supply fixes or waiting for new equipment to be installed, they are losing a lot of time." And then there is the critical question of just what sorts of testing to do. Many companies are familiar with whats known as "regression" or "unit" testing, fairly simple procedures that try to ascertain if what has been written works and if fixing old problems has caused new ones. But Y2K testing, if done well, takes many companies into new territory. "Were talking about full-scale testing across all interfaces, and this is something most companies havent done before," says Leon Kappelman, associate professor for business computer information systems in the College of Business Administration at the University of North Texas, Denton. "When it comes to Y2K testing, the bad news is that our applications and our enterprises are very connected." "Many companies are planning only to do date-simulation testing," says analyst Stephanie Moore of Giga Information Group Inc.s Westport, Conn., office. "But mission-critical applications need date-warped testing, which uses a portion of the mainframe or a completely separate machine to essentially replicate a future environment." That involves creation of a computerized world in which multiple applications are run together as though it really were Dec. 31, 1999, or Jan. 1, 2000. Thats what Fina is doing by creating a separate area of its mainframe computer devoted to Y2K testing. And even here analysts disagree about how much testing is enough. Fina, for example, is testing five dates: 12/31/99, 1/1/00, 2/29/00 (in the past leap years have caused computer problems), 3/1/00 (the post-leap-year date), and 1/1/01. But Bergeon and others believe that, depending on the application at issue, two dozen or more dates should be run. "I think companies should go forward a quarter and back a quarter," he says, suggesting that multiple dates from the last three months of 99 and the first three months of 00 be used. Others say the entire year 2000 should also get a thorough testing. Companies can buy tools that help test code fixes, but theres a catch: Many of them have a long learning curve, which means that time may run out before programmers are adept at running them. The good news is that because testing is iterative and repetitive, once the tools have been mastered and processes are in place, the same scenarios can be run over and over. All of which means that any Y2K plan that doesnt allow for significant time and resources for testing is in big trouble. "Its fascinating to me how little hysteria there is out there," says Moore. "There is not enough awareness or enough quality resources to do the proper testing. We are running out time."