In 2005, Ken Klein, CEO of Wind River Systems, delivered the Embedded Systems Conference keynote speech, where he said:

“I’m one of those guys who sits in a board room and obsesses everyday over the bottom line. You’re an engineer or engineering manager in the developer trenches. Everyday you contend with big technology challenges, growing complexity, and daunting deadlines.

Believe it or not, we are joined at the hip.”

It’s 2009, and the very fact that I still remember his speech enough to actually look up his quote prior to writing this post is significant. It was actually the very start of a journey for me. I remember, at the time, thinking that Ken Klein was dead wrong. Today, I know he was dead wrong.

I am a Canadian software engineer, and a few days ago, an icon in the Canadian high tech industry, Nortel Networks, was delisted from the Toronto Stock Exchange. Nortel announced that, after several months of bankruptcy protection, the company’s assets would be sold and the company dissolved. This was after many years of executives, who were paid millions of dollars a year in salary, drove the company into the ground, and walked away wealthy. Many engineers who have worked for the company their whole life have lost half of their pensions. Once again, this story made me think, what executive is really worth the salary they get paid? What kind of system rewards incompetence so handsomely? In my opinion, nobody, not a single person on the face of this earth, can produce enough value in a single year to justify a million dollar salary. The real heroes of our industry, the engineers who pour their lives into organizations like Nortel, only to have their pensions stolen by the CEO, deserve better career options.

The business that Nortel Networks was in does require a classic corporate structure. Designing and manufacturing carrier grade telecom equipment is capital intensive, and therefore requires investment from stock holders, who will insist on classical organizational design. The business that Wind River is in, is not capital intensive at all, and does not require stock holders. Given the world we live in today, there is an opportunity to embark on a new business model. A model where we do not need to pay incompetent people huge salaries.

I originally thought this new business model would work for the embedded software industry about three years ago. Back then, I was just a software engineer, and had no idea on how to put it together. So, for the last two years, I have been working on an MBA, and I am almost finished. During my studies, I learned how Michael Dell invented a new business model that eliminated the middle man and dropped shipped PCs directly from the factory to the consumer after an online order was made. I also learned how Robert Young, founder of Red Hat, stumbled across the new economic model of open source, and used it to improve the IT software industry. In Open Sources: Voices from the open source revolution, Robert Young says:

“You can’t compete with a monopoly by playing the game by the monopolist’s rules. The monopoly has the resources, the distribution channels, the R&D resources; in short, they just have too many strengths. You compete with a monopoly by changing the rules of the game into a set that favors your strengths.

At the end of the 19th century, the big American monopoly concern was not operating systems, but railroads. The major railroads held effective monopolies on transportation between major cities. Indeed, major American cities, like Chicago, had grown up around the central railway terminals owned by the railroad companies.

These monopolies were not overcome by building new railroads and charging several fewer dollars. They were overcome with the building of the interstate highway system and the benefit of door-to-door delivery that the trucking companies could offer over the more limited point-to-point delivery that the railroad model previously offered.”

Robert Young’s message is about how Red Hat competed with Microsoft by offering something Microsoft was not prepared to offer – free software. Red Hat changed the rules of the game and generates over $600 million a year in revenue.

With the rise of open source in the embedded software industry and the ease that individuals can collaborate using social networking tools, we are standing at a unique point in time. A time where Wind River has lost its ability to control access to software, and where collaborating with a developer in China is as easy as popping your head over the cubicle partition. A time where we can eliminate the pay cheque for middle men like Ken Klein. A time where our future security will depend on ourselves, and the value that we create along with our peers. A time where we can change the rules of the game to favor our strengths, and compete with Wind River, even when it is backed by the resources, the distribution channels, and the R&D resources of Intel.

In 2006, IBM Global Business Services published a report called The power of many. ABCs of collaborative innovation throughout the extended enterprise. This report says “The IBM Global CEO Study 2006 found that, to drive innovation, many top CEOs are collaborating beyond their organizations – with their extended networks of suppliers, customers, business partners and others.” A case study in the report documents how Eli Lily, in order to combat the financial loss of several patents, launched “research without walls”, that lead to an innovative new business model described as:

“Lilly created a novel business model for collaborative innovation when it started InnoCentive. This company, which Lilly has subsequently spun off but still retains some ownership of, leverages the global reach of the Internet to bring together “seeker” companies with “solver” scientists in order to identify innovative solutions to diverse problems. To date, InnoCentive has registered over 110,000 “solvers” in more than 175 countries.”

Embedded device software development has reached a point where the industry is ripe for some serious business model innovation. We are in a position to offer the industry something that Wind River is not prepared to offer. We can offer an organizational design that has absolutely no executives sucking the life out of it. The ultimate virtual corporation. We are going to model this organizational design after the film industry, we are going to drive it with drop ship cost reductions, and we are going to create business agility that companies like Wind River can only dream of.

Many films are produced by independent film producers. The independents are very small companies that do not have employees, they contract out everything, on a project basis. They get funding for a film project, hire on a team, produce the film, and then dissolve the team. The producers are not associated with any particular film company. Producers will have pools of favored professionals that they work with on a regular basis, they work as a high performance team. The model works well because the financial rewards are very closely connected to the value creators.

With open source becoming the dominant choice in the embedded software industry, companies like Wind River have lost their ability to lock out independent software producers, making the film industry model very attractive for embedded device development. We are going to change the rules of the game and drive the embedded software industry to a model where independent embedded device producers get funding for a device, then they contract out the development to their favored team of hardware/software contractors, and the team is dissolved at the end. We are going to eliminate the middle man, and drop ship our software directly from our home office to giants like Motorola to run their phones. This model will be executed on a global scale. An independent device producer in the US will engage with freelance marketing professionals in the UK, freelance sales worldwide,  system modeling architects in Canada, hardware developers in Taiwan, and software developers in India, all made possible by Web 2.0, and the Internet savvy of the modern technology worker. I could see Motorola developing phones this way, Dell developing consumer devices this way, Telecom companies developing new soft switches this way, auto manufacturers developing in-car systems this way, factories automating their lines this way, and on and on.

Not only is this a collaborative innovation model, it will also be a very new business model for embedded devices which, according to IBM, will add up to financial success for whoever drives it. We, the hardware/software engineers that actually create the value that goes into embedded devices, collaborating with the whole world, are going to drive it.

You may be wondering what this post has to do with Model Driven Architecture. Well, we are on a journey. A journey inspired by a technical vision, but empowered by good business sense. What we are doing is hard work, and hard work should be rewarded with significant wealth creation. I just don’t plan on sharing the value of my engineering intellect with an overpaid CEO. So, Ken Klein, its been fun, but, its time to remove you from my hip.

Chris Stone.

In 1999, James Ready founded MontaVista Software to create a Linux distribution suitable for embedded devices. Initially much of the development work involved getting the Linux kernel and a minimal root file system to work on embedded single board computers with MIPS and PowerPC CPUs. In doing so, MontaVista invented the embedded Linux market. They created a pre-built distribution that one could use to bootstrap an embedded device development project. Today, MontaVista ships the sixth version of that distribution in what they call market specific distributions. Each MSD contains hundreds of megabytes of compiled source code for ones target architecture.

These distributions are quite useful. One can build a device from them faster than if one had to compile all the source oneself. Nonetheless, they have the following problems:

1. The learning curve is long. Learning basic embedded Linux, i.e. getting a command prompt up on a running kernel, is not too tough. Try learning GTK, or the Hildon Application Framework, or even modifying a major GTK application like Evolution. I have done these things and our estimates ended up very low, pushing the project many months passed the original delivery date.
2. Development is at the code level. Open source is a code centric culture. The top contributors are called hackers, because they don’t really engineer anything, they just jump into code, hacking it into a new purpose. The top contributors are just programmers. This is exactly opposite of the kind of culture we want to encourage in a professional environment trying to build sustainable competitive advantage.
3. Tools reinforce a code level culture. It is interesting to note that the biggest advances the top embedded Linux suppliers have announced over the last few years are around code debugging tools, and the next hot thing is static analysis. When one is working with millions of lines of code, that one doesn’t understand, and one is focused at the code level, a debugger and static analyzer seem like good ideas.
4. Development is labor intensive. Due to code level development, and the number of lines of code in the average embedded Linux distribution, development is a long labor intensive process. This factor is accelerating the offshoring of jobs, as the BRIC countries are able to reduce the labor costs significantly.
5. Development is difficult to distribute. I will probably do a whole other post on this topic, but, for the purposes of this argument, it should be noted that code level engineering is difficult to distribute amongst many global teams. Open source veterans will choke on this statement, as they point out how the Linux kernel is developed by volunteers across the globe. Open source is globally developed, but the fact is that this development, from a business process point of view, is highly inefficient. It works when volunteers do it for no money, but, when one is trying to optimize their supply chain, code level engineering is a difficult monster to globalize.

Over time, all of these problems are getting worse, because we just keep adding lines of code to the problem. We need to consider a different paradigm. If we were able to catalog all of the open source code into a database; then write a tool that would decode a set of requirements; reach into the database and extract code that has almost implemented those requirements; then generate 85% of our embedded device code; these problems would go away. This is what Model Driven Engineering does. With an MDE tool we can reverse engineer an application or library into a platform specific model and store that model in a database. We can then abstract out a level and write transforms to convert the platform specific model into a platform independent model and vice-versa. We can then use the platform independent model and transforms, perhaps combined with other models, to create new applications.

An MDE process allows us to standardize code generation with the following benefits:

1. Learning curve is short. When I try to modify an open source application, I am constantly looking for a way to make my modifications without delving too deeply into the code. I almost use a trial and error approach, where I try a modification without fully understanding whether it will work. I am trying to find the shortest path to making the needed modifications. MDE processes do this by default. With models, details of implementation are abstracted away, and one can make modifications without knowing too much of the implementation details.
2. Engineering is at the architecture level. With MDE, we are actually engineering. Instead of programming we are software engineering. This means we use defined processes to produce repeatable, reusable, and robust results.
3. Tools reinforce an engineering paradigm.
4. Development is efficient with human resources. By promoting reuse of code and providing automatic code generation we only develop code that has not been written before.
5. Development processes integrate globally. An MDE development process combined with Agile project management is designed to be efficiently distributed between global resources. Communication is efficient due to the use of common engineering languages such as UML or domain specific languages. Just like civil engineers around the world unambiguously understand the language of mathematics they use to build bridges, software engineers around the world unambiguously understand the UML and DSLs they will use to build software.

Open source today is a vast digital natural resource that is there to be mined. This is the true value of what has been created. MDE technologies will allow us to code mine this database when we develop new embedded applications. Instead of understanding several million lines of code, we just need to understand the application we are developing. By combining components from platform independent models into our new application, then generating the platform specific model, and then the code, we will essentially be doing a great big query on the open source code database, where that query returns an application that is 85% complete.

As we create models, we build up more and more components that bootstrap the rapid development of more complex applications.  For instance, the source code for Kmail is a very detailed description of how to make a mail application. After extracting the essence of that description into a higher level of abstraction than the code, one can use this description to build other slightly different mail applications, let’s say for a small screen device.  The old saying that “one can’t see the forest for the trees” is very true when you look at a large open source project at the code level. Trying to map a forest while walking amongst the trees will take a lot longer than mapping it while flying over it.

Chris Stone.

Those who are trained in the art of success, will always begin with the end in mind. Thus, in this post, the first post of this blog, I will lay out my vision for the world of embedded software development. I will define the end I have in mind. Then we can begin the journey of getting there.

BTW, this will not be a short journey as I am sure it will take many years to realize these dreams. Additionally, along the way, I hope to have my vision shaped by many others, so where we actually end up may be quite different from what I envision today. One thing I am certain of, the time for some dramatic change in the way we develop embedded software is here right now.

Almost all embedded device software development seems to be stuck in the 1980’s. I have listened while the most experienced and seasoned professionals have explained to me that C coding using vi is the best way to write software. As I listen, I can’t help thinking that they are so cute at that age, they are adorable, they say the funniest things. They would have us all believe that no significant advances in the art of Software Engineering have been made for the last 20 years, since they graduated. These people try to amaze me as they show me how fast they can develop a new feature by cloning code and kludging it into a new purpose. “You can’t do that with your fancy MDA they explain”, not realizing that the only thing amazing about their expertise is it’s ignorance.

Now let’s define software engineering because understanding the difference between a software engineer and a programmer is key to our journey. A good definition of software engineering can be found on Wikipedia: (http://en.wikipedia.org/wiki/Software_engineering)

“Software engineering is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software. It encompasses techniques and procedures, often regulated by a software development process, with the purpose of improving the reliability and maintainability of software systems.”

I would add that software engineering today also includes the purpose of reducing cost, increasing business agility,  retaining jobs, enabling innovation, and building competitive advantage.

On average, software engineering does not exist in embedded software development today. In the past five years, I have worked for the top two embedded Linux corporations in the US. In all of that time, I did not meet a single software engineer. I met many hackers and programmers, but not a single software engineer. These companies simply don’t do software engineering. They hire the brightest minds, tell them to forget everything they learned in University, and have them writing C code on their first day of work. With the hundreds of millions of dollars in their annual budgets, these companies are completely unaware of the business case for software engineering. Given that they sell an operating system, I found this fact difficult to reconcile with their marketing messages, which usually involve claims of quality.

To lead our industry into the 21st century, and in the interest of beginning with the end in mind, the embedded software world I would like to see is a world where:

• Software is developed from a Model Driven Engineering process. Code is largely generated from models. The choice of implementation language is late-bound thus code can be generated in whatever language you want.
• From a Model Driven Architecture we create reusable, synthesizable, reliable, secure, and inter-operable components. The life-cycle of software is significantly enhanced. The ability to incrementally build systems of increasing complexity is enabled because we stop starting over and over again.
• Vendors deliver product as models, not code. Full code generation is enabled in the models. Proprietary products are delivered as binaries plus component interface models.
• Product development is the process of assembling software components using component synthesis. Economies of scale are hard at work.
• Product verification is started early. Product verification begins with verifying the model using simulators. Test points are built in to every component so that all code can be tested during the traditional product verification cycle. Round-trip engineering brings code changes back into the model.
• The OS doesn’t matter, the OS is late-bound. You don’t choose your OS at the beginning, you choose OS components at the end of a project. Operating systems may be swapped one for the other with no impact on schedule.

Chris Stone.