Faced with the challenge of testing and debugging increasingly complex software, embedded developers are turning to the advanced capabilities of virtualized software development. By enabling the software development phase to begin earlier in the production cycle and delivering features above and beyond those of traditional approaches, virtualized software development is establishing itself as a powerful tool for developing complex systems like multicore.

When the Department of Defense developed the F-4 fighter in the 1960s, less than 10 percent of its flight avionics functionality was tied to software¹. In the case of the modern F/A-22 stealth fighter, which contains more than 1.7 million lines of code, more than 80 percent of its functionality is software-based¹.

These numbers illustrate the dramatic shift that has taken place in the structure of military systems over the past few decades and hint at the consequential impact on software developers. Design of an electronic system is in two parts – hardware completion and software development. Historically, software comprised a much smaller, less significant portion of a typical electronic system. Then, the primary focus centered on hardware design, and project managers could simply treat the software as a secondary concern. However, as Moore’s Law dictates, the speed of semiconductors has increased, making it more convenient and cost-effective to implement a greater portion of a system as software running on a chip (see Figure 1). As a result, the size of the software component in a given electronic system has exploded, particularly in recent years, without any change to the deficient capabilities of traditional software development practices.

Recent changes in chip architecture are making their way into military embedded systems that will further increase the size and complexity of the software content. When progress in single-core processor performance began to slow considerably, chip designers started to look towards parallelism and multicore design to meet the ever-present demand for elevated computing performance. Now, with the seemingly inevitable migration toward multicore architectures and the accompanying increase in both the amount and complexity of software code, the daunting task of testing and debugging software has become even more complex and time-consuming. The critical path to shipping systems has shifted from hardware to software, and the key to maintaining competitive systems production schedules now lies in developing and debugging the software earlier, before hardware is available. This demand has prompted military software developers to investigate new approaches and tools for testing and debugging.

Virtualized software development is garnering increasing attention because of its innovative approach to software development and its advanced testing capabilities. By utilizing a highly-accurate, simulated model of the hardware, developers can begin to test and debug software well before the availability of actual production hardware, enabling them to discover problems much earlier in the development cycle where they can easily be addressed. In addition to enabling software development to commence much earlier, virtualized software development offers a level of flexibility and determinism that is impossible under the constraints of real hardware. Any computer system, ranging from single boards to multicore and multiprocessor computers, can be easily controlled, frozen, and analyzed in order to locate and resolve bugs. With the capabilities offered by virtualized software development, developers will find themselves well-armed to test and debug software systems that promise to only grow in complexity.

Traditional approaches to testing and debugging Software developers have employed a number of methods to test and debug code, utilizing some approximation or model of the final hardware. Traditional development platforms range from host-based to hardware-based to co-verification-based, but each of these exhibits limitations that result in low-quality software, schedule slippage, or cost overruns.

One traditional approach to developing and testing code involves running the software on a specially built test scaffold on a PC. However, the accuracy in this approach is severely limited, especially by its inability to replicate any real-time performance. Developers using this approach run the risk that the software written on a PC and running on test scaffolding will not behave like the production code, passed through a different compiler, running on a different processor. If the developer discovers that the code requires reworking during the system integration phase, what results is a delayed schedule and a consequent increase in project cost. The host-based development approach oftentimes proves to be too inaccurate to generate reliable code on schedule.

Due to the inaccuracies presented by PC-based testing, companies often adopt a second approach – to employ real hardware in one of three ways: buying real hardware, if it exists; using an approximation of the real hardware (for example, the processor and some peripherals); and making a mockup with FPGAs. This hardware-based approach, however, brings with it major drawbacks in terms of schedule and cost. Hardware is not only often costly to purchase but difficult to maintain, upgrade, and supply for each software developer working on a team. Waiting for the availability of test hardware also delays the start of the software development phase, introducing a bottleneck that results in missed schedules. In reality, many projects go through an early phase where the software developers have their hands tied because they are waiting for hardware availability.

A third approach involves the use of hardware-software co-verification tools. In the past, co-verification tools like Mentor’s Seamless, for both ICs and board-based systems, have proven to be somewhat effective for developing very low-level code but have suffered from the inability to run a full load of application software in practice. Developers require the ability to loop quickly through their edit-compile-debug cycle several times throughout the day, but it can take an entire day to boot the operating system using co-verification. With traditional co-verification tools, developers achieve the desired level of accuracy, but the slow speed of the tools has rendered the approach impractical.

Now that software is the critical path to shipping the product, the inefficiencies and impracticality of these traditional development methods are being exposed (see Figure 2). Adding to the difficulty of testing is the increasing interconnectedness of systems. Complex interactions within the system can result in problems that show up after production and development. This problem has only increased as the system architecture has dramatically evolved in recent years.

Modern approaches to virtualized software development In the modern virtual software development platform, programmers are finding a robust tool that can tackle development challenges even in the most complex systems. Using a simulation-based development approach, developers can model hardware with pinpoint accuracy, allowing the actual production binaries, operating system kernels, and device drivers to be run on the simulated model. The software being run on the virtual platform cannot detect the difference and behaves just as it would on the actual production hardware.

Due to past experience with low-level simulators, people tend to assume that virtualized software development approaches are not fast enough to be used as part of the a programmer’s edit-compile-debug cycle (see Figure 3). Recent advancements in virtualization overturn that assumption. With approaches such as just-in-time compilation and the availability of faster and less expensive workstations, virtualized software development can simulate complex systems at speeds measured in billions of simulated instructions per second, closely approximating the speed of the actual hardware.

Developing a simulated model of the hardware, whether it is a chip or a board level system, is much simpler and quicker than developing the hardware itself because a model can be built immediately after the key architectural decisions have been made about the hardware (see Figure 4). Eliminating the need to wait for hardware leads to faster time-to-market, higher software quality, and reduced risk in the project schedule. Additionally, software development can proceed concurrently with hardware development, which means software errors can be discovered at an earlier stage of the game, where they are generally cheaper to resolve. Otherwise, risk continues to build up until the system integration phase when everything comes together for the first time. When inevitable problems surface at such a late stage in the development cycle, the only option is to slip (or miss) the schedule. Performing integration and running all tests in the virtual world means that when the hardware is finally available, the final bring up should go smoothly. As an added benefit, less hardware is required for R&D, and there is a further reduction in the amount of hardware required for the test lab, which can result in big savings since such labs must be maintained for many years for a typical military program.

Advanced capabilities of virtualized software development With virtualized software development, developers can approach testing and multicore challenges with powerful new techniques that are not possible in the world of real hardware (see Figure 5). The simulation-based platform is cycle-approximate and fully deterministic, enabling developers to eliminate the main challenge in real-time embedded system debugging: the difficulty of reproducing failing conditions. Notorious “heisenbugs,” which change their behavior when they are observed but disappear when you try to take a closer look at them, are the most difficult to test with real hardware. These bugs depend on subtle timing interactions, and repeatedly running the full system is still often inadequate to track down the problem. The simulation infrastructure underneath a virtualization platform manages time, which allows a developer to single-step or stop the system to examine internals. Developers can reproduce an error many times over, even when simulating multiple processor cores and multiple processes on each core, making debugging a multiprocessor as easy as debugging a single program on a single processor.

Additional techniques, such as varying the relative clock speeds of the processors or adding additional cores, can be used to further stress that thread locking is handled correctly and that the software will behave correctly. Many applications are single-threaded and are only now being converted to thread-based approaches to take advantage of multiple cores, potentially generating a new class of errors. It is also possible to perform regression testing and fault injection, where it is typically impossible to inject faults in hardware, even at the board level, without destroying it.

Simulation also offers increased temporal control, allowing developers to set a breakpoint that deterministically stops all the processors at the same place each time. Virtualization can accelerate time for unimportant code such as idle loops by examining the event queue to determine when the next hardware event that could affect an outcome might occur. One benefit is that virtualization can run the code faster than real time in lightly loaded systems, meaning that the software developer can test how the system will react after weeks or months of light use in a shorter amount of time.

More surprisingly, virtualization rewrites the traditional rule that code can only run forward. Capable of executing code in reverse, virtualization makes it possible to simply wait for an error to occur and then run backward to determine the root cause. Running backwards depends on two criteria: the capability to checkpoint an entire system inexpensively and the underlying speed of simulation. Running backward one instruction is accomplished by restoring a checkpoint and running forward all but one instruction. This process appears almost instantaneous to the user due to the simulation’s speed, even though checkpoints are not recorded all that frequently. This reversible debugging environment works with multiprocessors and multicore processors, providing the capability for tracking down bugs such as lock conflicts, deadlock and priority starvation, all of which will be more common in multicore architectures.

The Navigation Systems Division of the Northrop Grumman Corporation has recently chosen to deploy a virtualized development environment for a pilot project involving software development and testing for selected navigation systems. Northrop Grumman plans to simulate its navigation systems and customize their software for a variety of applications, providing an opportunity to examine and test how newly created software will operate. This move toward simulation will help validate software modifications and speed completion of the project.

Virtualized software development: Closing the gap With military embedded systems moving toward multicore processing in order to achieve the computing performance required in the near future, virtualized software development is the only approach that allows developers to begin earlier in the production cycle, prior to hardware availability, providing both cost and time-saving advantages that keep companies competitive. Virtualization offers advanced capabilities not found in hardware-based development, including the system-wide controllability, insight, and determinism necessary to resolve the development challenges presented by multicore and truly concurrent environments. While it is undeniable that software development has not kept pace with advances in hardware development in the past, virtualized software development has enormous potential to help close the gap.

Peter S. Magnusson is founder and Chief Technology Officer at Virtutech, Inc. He served as CEO of Virtutech from its founding in 1998 to 2005. After concluding studies at the Stockholm School of Economics and the Royal Institute of Technology (Computer Science), Peter conducted research in simulation technology at the Swedish Institute of Computer Science since 1991, with a number of international publications in the field. For several years he was a columnist for the computer magazine Datateknik and an advisor for the SalusAnsvarÖhman IT-Fond, a mutual fund.

To learn more, contact Peter at:

Virtutech Inc. 1740 Technology Dr. Suite 460 San Jose, CA 95110 Tel: 408-392-9150 E-Mail: psm@virtutech.com Website: www.virtutech.com

Reference 1. Patience Wait, “Weapons projects misfire on software,” Government Computer News, July 3, 2006.