Do-it-yourself Linux ñ a money pit: Commercial Linux saves time and money

Military Embedded Systems — January 13, 2008

Companies that haven’t clearly understood the high life-cycle costs of in-house RTOS development typically find themselves victims of the “software money pit.” But in the buy-versus-make RTOS decision, commercial Linux clearly provides cost and complexity advantages versus the do-it-yourself Linux alternative.

For the past 25 years, engineering organizations developing software for embedded devices have become embroiled in the buy-versus-build decision for (RTOSs). At the core of the debate is the emergence of commercial RTOSs, providing a complete system off-the-shelf and eliminating the need to write an system in-house. However, these COTS RTOSs carried a tidy price, and some entrants into the embedded market view them as more expensive than writing their own RTOS in-house.

However, most companies who choose to develop their own RTOS in-house have learned that there are many cost considerations that lie beneath the surface. Many organizations haven't fully understood the life-cycle costs encountered in developing an RTOS in-house, and most have learned the hard way about the development process costs involved. Thus, the in-house alternative often results in a "software money pit" where engineers start down the in-house development road, then pick up cost after cost until they realize it has become very expensive. By then, though, it is difficult to stop since so much has been invested.

, however, is one RTOS that has established itself as the preferred Operating System (OS) for a wide variety of embedded devices. It too comes in two forms: Commercial and in-house. As time has proven, commercial has many advantages that developers should consider over building an in-house version of embedded Linux. An understanding of such considerations removes most of the debate as to whether or not to build in-house or purchase a commercial Linux package. Important factors developers need to consider include "obvious" and "hidden" costs, along with the rising open-source complexity issues that affect the in-house versus COTS Linux equation.

The explosion of software content in devices

There are two dimensions of rapid expansion in the embedded system world that underlie this move to Linux over traditional RTOSs:

  • The growth of new and complex product categories
  • The growth of requirements for the system software in the embedded device itself

A case in point: A military manufacturer that previously dealt with small amounts of firmware needs to develop a whole new range of products based on a complete OS environment with multi-programming, TCP/IP connectivity, built-in security, and more. Consequently, this manufacturer suddenly finds its product teams forced to make decisions about the development and adoption of millions of lines of OS software into their product lines. With such an explosion in internal costs and in the complexity of product design and development, there is the risk of a software crisis throughout the technology product industries (See Figure 1). This is especially true in the embedded military market, where there is an increasing amount of scrutiny over cost, complexity, and time-to-market issues. Other considerations include:

21
Figure 1

  1. The need for such system software cuts across a large number of different product organizations and teams. The external competitive situation doesn't tolerate a high cost structure for system software. Current budget constraints will not support the in-house teams formerly maintained; a company's engineering resources must focus on value-added technology to remain competitive.
  2. The royalty component of current off-the-shelf system software in the cost of goods is large and impacts corporate margins.
  3. The selection of a common, strategic system software platform will help a company avoid getting mired in a multiplicity of alternatives, each with a high cost structure and no overall leverage. Industry experience shows that disparate platforms inflate costs and increase product development cycles.

The growing popularity of embedded Linux as an alternative to traditional RTOSs is evident from the tens of millions of electronic devices that have already shipped with Linux as their OS. Ironically, however, even within organizations facing increasing pressure to produce software-intensive products with high functionality and quality requirements, as well as strong schedule pressure, the temptation to implement Do-It-Yourself (DIY) Linux remains alive and well, as opposed to off-the-shelf Linux, and development costs are a key factor – both obvious and hidden.

The "obvious" development costs

There are many processes and costs to understand in embedded Linux that are native to both in-house and commercial implementations; however, the difference is that these factors are typically included in the cost of commercial packages:

  • It is not unusual for an embedded Linux or Linux distribution in general to comprise more than 30 million lines of source code.
  • The code base can comprise 19 or more different, unsynchronized and non-integrated code repositories.
  • The code base often changes daily.
  • Many large companies use a wide range of embedded processor architectures requiring support for as many as 24 microprocessor architectures and their variants, and more than 100 hardware platforms.
  • Multiple host computing environments and various revisions must be supported (such as Windows and Solaris, among others).
  • The time to construct, test, and deliver an initial distribution can easily reach more than 30 development years (not including the ongoing maintenance, minor enhancements, or support time), which easily amounts to a cost in the seven-figure range. The ongoing maintenance, addition of minor enhancements, support, and the building of a development infrastructure also add significant ongoing cost.

Most commercial Linux distributions not only take these factors into account, but also have tools, resources, and expertise to help address them, making it an even more viable option for developers than downloading from kernel.org and building their own flavor. Considering that the typical cost of outfitting a mid-sized development team (~10 engineers) with a commercial distribution can be well under $100,000, the cost savings over a seven-figure in-house development effort reflect a real bargain.

The "hidden" development costs

In addition to the more "obvious" development activities, there are numerous "hidden" or often overlooked efforts that are essential to a successful development effort. Efforts involved in in-house development include:

  • The development of a comprehensive Testing/QA capability. Test suites need to be developed for the OS itself and the wide range of I/O devices present, especially in SoC devices. (A typical SoC for a mobile device has 20-30 complex I/O devices built in ‚Äì all these need to be tested.) The physical infrastructure to host all the reference boards for each architecture needs to be developed and deployed.
  • The creation of a build computing environment that is efficient and capable of turning the build around as quickly as possible (hours not days); otherwise, the build process itself becomes a bottleneck in the overall project development flow.
  • Tool development: cross debuggers, memory leak detection tools, performance-tuning tools, kernel-aware debuggers, and the list continues. Rarely are developmental dollars allocated to tool development, since most of the money had to be invested in developing the kernel itself.
  • Training course material and course delivery. The Linux system is quite large and functional. Developers need training on its programming model, device driver structure, and development tools in order to be productive quickly.

These additional "hidden" costs can also affect delays, which can be quite costly. In fact, a December 2007 Embedded Market Forecasters report states that the average cost for each month of design delay is $386,250 assuming that the average cost per engineer is $150,000 per year and an industry average of 30.9 developers per project www.embeddedforecast.com.

Process development, the "new complexity," and costs

The process development costs that Linux DIY projects incur are another cost consideration in the make-versus-build equation. It's clear that without the proper "Linux/open source-aware" processes in place, the development schedule and product quality will both suffer the potential for skyrocketing costs and/or outright failure.

Unlike a traditional in-house development process where the overall procedure could be strictly controlled from start to finish, in-house embedded Linux development begins with a largely uncontrolled process: open source. While an individual company might actually contribute to a particular open source project, by no means can that company totally control the process – it can only hope to influence and participate. For example, the rate of change of Linux from release to release can vary, from minor bug fixes and modest new feature enhancements to wholesale changes in underlying subsystems, introducing new capabilities as well as new instabilities and bugs.

The sheer volume of these changes can be daunting. Consider tracking the activities around an individual architecture for Linux, MIPS for example. To keep abreast of the changes occurring on a daily basis, a developer needs to monitor the email traffic of 11 different and unsynchronized open source projects: kernel.org, the core home of the Linux kernel; the gcc and glibc projects (the core tool chain and libraries from FSF at fsf.org); and at least nine other components that would typically comprise a useable Linux development environment.

Kernel.org itself may have up to 5,000 messages a day with 1,000 of these being patches that need to be evaluated and possibly applied to the source base. Simply ignoring the traffic, figuring that the system in use seems to be working well enough, can lead to disastrous consequences later. For example, a recent security patch that took all of 13 lines of code to implement against an embedded Linux system would have taken more than 800k lines of source patches to implement if the previous trail of patches had been ignored. It's a classic case of pay now or really pay later.

Adding it all up

All of the aforementioned factors combine to make developing a DIY embedded Linux distribution a significant investment (read "big bucks") in time and money. If there ever were a situation where the "software money pit" could really take hold, it's in owning 30 million lines of constantly changing source code. Even in the simplest case, the development costs are typically in the millions of dollars, with the cost for the more comprehensive implementations substantially higher (Figure 2).

22
Figure 2

Commercial Linux is the answer

Where do we stand today on this debate? On the one hand, one could argue that the build-versus-buy Linux debate will take some time to decide because on one hand, engineers can just download Linux and go. On the other hand, the industry has matured overall, and there is certainly a better recognition of the real costs of developing software.

Accordingly, complexity and development costs have grown over the past several years and are significant factors when selecting an embedded operating system. We can safely conclude that if it made economic sense years ago to buy instead of create, which has played out in the RTOS environment, it overwhelmingly makes sense now when the move is being made to a far more capable but far more complex embedded Linux-based environment.

Jim Ready is a recognized authority in the embedded systems and real-time software industry. He founded MontaVista in 1999 to provide the Linux operating system to the embedded systems market and to offer embedded-systems expertise to the open source Linux community. He received his BA in Physiological Psychology from the University of Illinois at Urbana-Champaign and his MA in Physiological Psychology from the University of California, Berkeley. For more information, email info@mvista.com.

MontaVista Software 408-572-8000 www.mvista.com