The Perfect Process Storm

Integration of CMMI, Agile, and Lean Six Sigma

Abstract. Many organizations have struggled over the past few decades with a blizzard of process improvement methodologies such as Total Quality Management (TQM), Kaizen, JIT Production, and Re-Engineering. These operations are understandably leery of adopting new methodologies given their past experience, especially with a focus on return on investments and leveraging existing practices. This article examines the relationship of Agile, CMMI®, Lean Production and the Six Sigma Define, Measure, Analyze, Improve, and Control (DMAIC) roadmap.  The intent is to explain how these methodologies might be synergistically combined for a cohesive approach to enhance continuous process improvement.

Introduction

CMMI, Lean Six Sigma (LSS) and Agile development are arguably the most commonly used methods of process improvement in today’s technical workplace. Many operations are unique in that they employ all three methods in their project portfolio. This article proposes combining these seemingly disparate methods into a cohesive approach to enhance project process improvement.

• CMMI helps integrate traditionally separate organizational functions, sets process improvement goals and priorities, provides guidance for quality processes, and establishes a point of reference for appraising current methods and procedures.

• Six Sigma’s implicit goal is to improve all processes to produce long-term defect levels below 3.4 defects per million opportunities [1]. In recent years, some practitioners have combined Six Sigma ideas with Lean Production manufacturing to yield the LSS methodology that incorporates the elimination of waste; including process waste.

• Agile development is characterized by frequent rapid delivery of useable software by self-organizing teams with regular adaptation to change [2]. Working software is the principal measure of progress; and increased throughput (velocity), by reduction of bottlenecks, is the primary measure of efficiency.

A Brief History of Process Improvement

“I can say, without the slightest hesitation, that the science of handling pig-iron is so great that the man who is …  physically able to handle pig-iron, and is sufficiently phlegmatic and stupid to choose this for his occupation, is rarely able to comprehend the science of handling pig-iron.”

-Frederick Winslow Taylor, father of Time and Motion Studies

“The old days is just 15 years ago.”

– Billy Corgan, The Smashing Pumpkins

Frederick Taylor, regarded as the father of scientific management, was a mechanical engineer in the late 19^th century who sought to improve industrial efficiency. Taylor thought that by analyzing work, the “one best way” to do it would be found. He is most remembered for developing scientific management and time and motion studies, wherein a job was broken down into its component parts and measured to the hundredth of a minute.

In my University of South Florida college days, one of our

classes delved into Taylor’s work. During an exercise where

we practiced measuring a worker’s activities, I remember the

instructor noting, “Make no mistake about it. While you are standing there with your stopwatch scribbling timed activities on your clipboard, that worker hates your guts.”

It was at that moment I decided to avoid this profession altogether. Nonetheless, as I went on to be an engineer and project manager most of my life, it seems clear I came to embrace measures, metrics, and process enhancement. I have now spent the last seven years as a full-time process improvement consultant. Go figure.

Modern process improvement began around 1948 with the Japanese Kaizen system, targeting quality, effort, employee involvement, willingness to change, communication, and elimination of waste in business processes. This led in the 1980s to the popular but short-lived TQM concept, meant to improve quality by ensuring conformance to internal requirements (stifling yawn). Then in 1986 the marketing people at Motorola invented Six Sigma, an exciting quality improvement initiative promising to reduce the number of defects and impurities to zero. No one knows quite why they selected six instead of five or four sigma, but it was the new wildfire once Jack Welch at GE went nuts over it and became its leading advocate [3]. Since any project manager can see that a team laser-focused on defects will neglect all their milestones in pursuit of such perfection, this opened the gate in 1990 to Lean Production, based on the Toyota Production System (sometimes called JIT Production), which had fallen in popularity by 1975 in favour of the more generic Lean Production system. In Lean Production, everyone involved in making a product—design and manufacturing engineers, suppliers, labourers, even marketing and salespeople— works together from concept through production. And because the team is focused on one product, there is a cycle of continuous improvement, resulting in cost savings [4].  In the late 1990s AlliedSignal and Maytag decided to combine increased production and reduced defects with the introduction of LSS. Any CEO leery of a process keyed to a single parameter had to love the sound of LSS. In 1996, paired programming and iterative development began when Kent Beck invented Extreme Programming to rescue a Chrysler project that had been scrapped. This first Agile project was subsequently followed by projects using similar iterative methodologies including Scrum, Crystal, and Feature-driven Development leading to the meeting of the Agile Alliance in 2001 where a dozen or so guys (most visibly were Alistair Cockburn, Kent Beck, and Jim Highsmith) generated the Agile Manifesto, promising working software every 30 to 60 days.  Software teams worldwide dumped the Waterfall methodology best known for its phased approach where code was not developed until the full set of requirements were identified, documented, and designed (often taking years) for not just rapid development, but rapid delivery. In the software field, LSS concepts have been influential in the formulation of the “Agile methodology.”

(Editor’s Note: The original provides Figure 1 showing the Timeline of Modern Process Methodologies and Figure 2 showing Relationships between Lean Six Sigma Define Measure Analyse Improve & Control (LSS DMIAC) and CMMI Measurement & Analysis (MA) and Associated Processes.)

The confluence of Agile, LSS and CMMI created a potential perfect process storm that in large part has yet to be realized.  Many organizations employ at least one of these process methods, but few if any have deployed all three in tandem despite the benefits of doing so.

While LSS has connections to multiple CMMI Process Areas (PA), this discussion is primarily limited to interrelationships between Measurement and Analysis (MA) and LSS. Hence, the DMAIC aspect of LSS is considered in the process improvement context of CMMI MA, whereas other relationships of LSS that align more closely with the project execution aspects of CMMI are addressed later under “DfLSS and CMMI Compatibilities.”

As measurement is critical to both LSS and CMMI, an understanding of how they relate in the context of the MA PA is central to envisioning how they might be used by an organization in combination. The following subsections show how the four general MA areas align with the DMAIC roadmap.

CMMI Measurement and Analysis and LSS

Years ago I began working as a program manager for a software firm that automated various financial functions for about half of the world’s 100 largest commercial banks. In my first week I was asked to solve a problem for one of our New York-based banks. Originally estimated as a one-year project, we were already over a year into the implementation, only 50% complete, and we were charging them more than double the original budget.

I defined the problem using existing data from which I created measures that allowed me to analyze the inconsistencies. I then improved and controlled the situation through a report to senior management regarding inadequate estimates, doublebilling, unacceptable bug rates and other pertinent facts. We then negotiated our billing at 50 cents on the dollar and rebaselined the schedule. I had inadvertently employed an LSS DMAIC solution for an emergency one-time fix. The ROI on this was that we did not get sued and the client remained a loyal customer.

Impressed with my solution, the company asked me to fix the remaining projects that were experiencing similar problems.  On average our projects’ time-to-market was about 200% of estimate and our defect rate was through the roof.  Development blamed Quality Assurance (QA) for testing beyond the requirements and QA blamed development for not coding to requirements. Fortunately we had already collected data on estimates-to-actuals and defect rates by lifecycle phase.  I specified measures on this existing data and verified the productivity and quality issues. I then informed the entire company that I would be measuring actuals against estimates by phase along with defect rates, and would be issuing a report after the next billing period. To my surprise, the next period actuals averaged 90% of estimates and defects were virtually non-existent. By simply measuring the problem I had changed it for good. Using causal analysis and resolution techniques I discovered that once project personnel realized they were a team and would be held accountable individually, they began communicating. Business analysts wrote less ambiguous requirements and developers sat down with testers to explain why they coded a function a certain way based on those requirements. Here, I had accidentally used MA techniques suggested by CMMI to solve a problem for the long-term.  (Editor’s Note: The original provides Figure 2, showing the relationship between LSS DMAIC and CMMI processes, which are then detailed in the following sections.)

Define Phase and Establish Measurement Objectives

This first step in the CMMI MA process area aligns very closely with the define phase in a LSS DMAIC project, as indicated in Figure 2. The first important distinction and added value that comes from the conjunction of LSS and CMMI is that LSS places primary emphasis on understanding and managing performance (outcomes) while CMMI (often in practice if not in principle) places more emphasis on maturity level.

Whereas maturity level is important for government organizations in particular, it may not be sufficient in and of itself to quantitatively demonstrate improved outcomes in terms of cost, quality, or cycle time.

Using DMAIC, the LSS roadmap provides a very specific approach to establishing the overall objectives and identifying potential measures for an improvement project. Similarly, when properly structured, measurements established under the CMMI MA process should trace directly to business and/or project objectives. In either case, project metrics that fail to support such objectives were likely established as a “good idea” initially, but provide no benefit to the project. They should be discarded as a waste of time. One of the greatest impediments to a successful measurement program is the perception that data collection and analysis are being performed for no apparent reason, or because “we have always done this.”

Whether in LSS, CMMI or Agile, individual metrics, as well as the overall metrics program, should be evaluated periodically for usefulness. Notably, in many cases the metrics program itself has been discovered to be the true roadblock to productivity.

Back in my programming days for a defense contractor we were required to count Lines Of Code (LOC) generated each quarter by project. Eventually we developed a macro that differentiated between code, comments, and spaces and spit out LOC. We then stored the data like squirrels hoarding nuts for winter. Winter never came and at project end we just disposed of the data. I guess someone once thought this would be a useful exercise, but all it did was create bottlenecks and reduce velocity. When you collect measures, be sure to follow the Agile concept of avoiding activities that do not contribute to the final product.

Measure Phase and Specify Measures and Data Collection

“Data is like garbage. You had better know what you are going to do with it before you collect it.”

-Mark Twain

These second and third general steps in the CMMI MA process area very closely align with the Measure Phase of DMAIC. Again, the LSS roadmap provides detailed guidance for how to conduct these activities. The Measure Phase in LSS is prescriptive while the CMMI MA process area is proscriptive. SEI is unconcerned with the method, as long as the process is defined and repeatable. Use the measure phase of DMAIC to accomplish the goals outlined in CMMI. For example, use the guidance in the plan to measure results and plan to collect data steps from the DMAIC measure phase to accomplish the establishing objectives and specifying measures specific practice in the CMMI MA process area.  Similarly, use the guidance in the collect and qualify the data step of the measure phase in DMAIC to collect measures and place them in a measurement repository to satisfy the CMMI MA specific practice of specify data collection and storage procedures.

As a project manager I found that senior managers were always extremely impressed with huge amounts of measures being collected, analyzed and processed. More seemed to be better. Especially in an Agile environment, the development staff will take the opposite approach: keep it simple. Two or three measures, probably dealing with velocity and defect rates, should keep an Agile team busy and informed.

Analyze Phase and Specify and Conduct Analysis Procedures

The analyze phase of DMAIC encompasses the activities envisioned by the MA requirement to specify and conduct analysis procedures. LSS training includes instruction in selection and application of appropriate statistical tools, including criteria for determining which tools and methods are most applicable to a particular situation. While the argument prevails that the DMAIC roadmap provides detailed guidance on how to proceed, and the CMMI MA processes leave such decisions up to the practitioner, organizations usually provide this direction within project measurement plans. The SEI promotes the use of operational definitions for each specified metric. This is typically a table that defines the supported goal, collection/storage criteria, and review techniques spanning simple trend/variation analysis to complex statistical process control.

Consequently, any specific instructions and criteria demanded by an LSS application can easily be incorporated into the CMMI MA operational definition framework.

(Editor’s Note: The original provides Figure 3 showing an

example of an operational definition for an automated metric

generated through the Agile scheduling and issue-tracking tool

JIRA, noting “relax, it is freeware”).

Improve and Control Phase and Using the Measures and Analyses

“You will miss 100 percent of the shots you never take.”

-Wayne Gretzky

The final steps in DMAIC (Improve and Control) parallel CMMI Level 4 and 5 Support Process Areas. The structure of the CMMI separates MA, where data is collected and analyzed, from the other process areas (causal analysis and resolution, organizational innovation and deployment and quantitative project management) that use the measures and analyses to define and implement improvements. In this respect DMAIC is structurally, although not substantively, different from CMMI. DMAIC envisions a continuous flow of activities from problem definition through solution and implementation performed by the same team, illustrating a distinction between the CMMI “what” (defined by a series of PAs) and LSS “how” (defined by a project roadmap such as DMAIC as shown in Figure 2). Again, the combination of CMMI and use of organizational measurement processes currently provides the “how” and “when” aspects used by LSS practitioners. Additionally, while the requirements of MA are limited to analysis and communication of results, SEI encourages expanding MA efforts to include aspects of higher-maturity to take advantage of benefits associated with causal analysis resolution and quantitative project management. In fact, leveraging these benefits tends to improve MA results and provides the organization with tools to achieve the project’s established quality and process performance objectives.

If the primary goal of an improvement initiative is to create organization infrastructure and institutional capability (as SEI intended in government organizations for which CMMI was originally designed), then the separation of MA from various types of improvement activities clearly makes sense. MA focuses on the creation of measurement infrastructure, while DMAIC is typically more narrowly focused on time-limited resolution of a specific problem. Although different in approach, the result over time is essentially the same.  Therefore, the integration of LSS and CMMI provides the opportunity to institutionalize a measurement infrastructure that supports quick response to problems that require immediate attention and a process to closure, the very definition of issue resolution in an Agile-based environment.  The Control phase of a LSS DMAIC project most closely aligns with the following CMMI Generic Practices 2.8 and 3.2:

• GP 2.8 – Monitor and Control the Process against the plan for performing the process and take appropriate corrective action.

• GP 3.2 – Collect Improvement Information. Collect work products, measures, measurement results, and improvement information derived from planning and performing the process to support the future use and improvement of the organization’s processes and process assets.

DfLSS and CMMI Compatibilities

The description given in this article applies only to the LSS DMAIC roadmap and the CMMI MA process area. In order to implement the full connection between LSS and CMMI, organizations need to consider Design for LSS (DfLSS—

Define, Measure, Analyze, Design/Build, Verify), generally used to develop new products or processes, as well. While DfLSS is beyond the scope of this discussion, it should be noted that DfLSS can have important implications for all process categories. For instance, CMMI Requirements Management, a project-management process area, entails five specific practices, several of which have direct connections to DfLSS. The most obvious and significant impacts, however, are on the CMMI Engineering category.

• Requirements Development: Developing, analyzing, and validating customer/product requirements.

• Technical Solution: Goal 1, selecting product-component solutions, aligns most directly with the analyze phase, while Goal 3, implement the product design, aligns with the design/build phase of the DfLSS roadmap.

• Verification and Validation directly align with the verify phase of the DfLSS roadmap. Note that certain validation activities are ongoing throughout the lifecycle during define, measure, and analyze [5, 6].

Agile, CMMI, and LSS

“Truth is incontrovertible, malice may attack it and ignorance may deride it, but in the end, there it is.”

-Sir Winston Churchill

In “Good to Great” [7] Jim Collins explained it is vitally important for an organization to understand the brutal facts of its environment and its problems, but to never lose faith in the organization’s ability to win out in the long term. As he noted, Winston Churchill never failed to confront the most brutal facts. During WWII he created an entirely separate department outside the normal chain of command, the Statistical Office, with the principal function of feeding him—continuously and unfiltered—the most brutal facts of reality. He slept soundly knowing these facts. Recent research defining best organizational practices for project management similarly suggests the optimum way to improve project management is to have the difficult conversations necessary to keep projects healthy. When we maintain a steady culture of discipline, we are able to give our employees more freedom to experiment and find their own best path to results while stimulating change, improvement, innovation, and renewal.  Consideration of best practices associated with the integration of Agile, CMMI and LSS concepts within a single project, as opposed to deploying them separately, may well lead to that important culture of discipline.

When viewed holistically, CMMI’s ultimate goal (i.e., continuous process improvement) is to cause an organization to become less wasteful, leaner, and more in touch with their actual development progress. Ultimately, both Agile and CMMI, especially in high-trust environments, expect organizations to see gains in productivity by eliminating unnecessary effort. It is true that implementing Agile methods will often eliminate many nonproductive efforts and behaviors at the project level. However, even with Agile retrospectives, what CMMI offers beyond Agile is an infrastructure of organizational learning and improvement that benefits the projects even before they begin [8].

The DMAIC methodology is commonly used to identify problems in a process, measure key data issues of concern, analyse the resulting data, improve the process, and control the future state process to reduce defects. One of the standard tasks in this methodology is the assessment of process waste, also a core principle of Agile software development. In identifying and eliminating waste in a process, the disciplines of LSS DMAIC and Agile development share many attributes.  While Agile practices focus narrowly on improving the software development process, the broad discipline of LSS DMAIC is often used to improve manufacturing and business processes. By highlighting these similarities, the integration of LSS and Agile development, in combination with CMMI continuous process improvement, can lead to that culture of discipline that will allow teams to operate more efficiently while increasing morale, productivity and quality.

Summary

“My greatest strength as a consultant is to be ignorant and ask a few questions.”

-Peter Drucker

As organizations truly interested in process improvement mature in CMMI measurement and analysis performance, the relationships between LSS, Agile, and CMMI should be understood and leveraged. While a primary focus of LSS is cycle-time reduction and elimination of delays, and Six Sigma targets prevention and remediation of “defects” (in the broadest sense, including cost overruns, schedule delays, etc.), they are in fact highly synergistic and have come to be fully integrated within the LSS framework. Similarly, there are many ways Agile, LSS, and CMMI can be synergistically combined, such as follows.

• Defining Objectives. The LSS roadmap approach to establishing overall objectives and identifying potential measures for an improvement project is very similar to the initial CMMI MA practice of tracing business and project objectives to specific measures. The common question of “why” data is collected and analyzed is easily answered in both cases by defining the links to organizational needs.

• Measure. The measure phase of DMAIC provides detailed guidance for measurement results planning, data collection, and data integrity. CMMI MA specifies practices for measurement specification, data collection, and storage procedures that include activities designed to ensure data integrity. While LSS designates how these actions should take place, and CMMI leaves the method up to the practitioner (as long as the process is defined and repeatable), the two approaches can be synergistic. Methods such as the SEI Goal-Question-Indicator-Measure (GQIM) process can be used to satisfy both CMMI and LSS approaches to measurement specification, collection, and storage.

(Editor’s Note: The original provides Figure 4 Process Integration Benefits and Figure 5 JIRA Measurement & Analysis Process.)

The combination of CMMI and use of organizational measurement processes currently provides the “how” and “when” aspects that advocates of LSS infer. Expansion of MA efforts to include the benefits associated with causal analysis resolution and quantitative project management will further this connection.

• Analyze. The analyze phase of DMAIC encompasses the activities envisioned by the MA requirement to specify and conduct analysis procedures. While DMAIC provides detailed analysis guidance, and CMMI processes leave such decisions up to the practitioner, relative CMMI MA direction is given within project measurement plans. The CMMI practice of using Operational Definitions for each specified metric helps define the supported goal, collection/storage criteria and simple to complex review techniques. Therefore, any specific instructions and criteria demanded by an LSS application can be easily incorporated into the CMMI MA operational definition framework, and efficiencies inherent to each method will only strengthen project analysis procedures.

• Improve. In general, leveraging both the managed and repeatable benefits associated with MA, and the laser-targeted results of LSS, will provide the organization with tools to achieve the project’s established quality and process performance objectives.

• Control. Although MA is limited to analysis and communication of results, the higher-maturity CMMI PAs of L4 and L5 can be leveraged to take advantage of benefits associated with causal analysis resolution and quantitative project management. In fact, leveraging these benefits improves MA results—further enhancing organizational tools for achieving established project quality and process performance objectives. The integration of LSS and CMMI provides the opportunity to institutionalize a measurement infrastructure that supports timely response to problems requiring immediate attention and a process to closure—again, the essence of issue resolution in an Agile-based environment.

• Synergy. Important connections between Agile and LSS are clear. Both target short lifecycles. What Agile calls velocity, LSS calls throughput, and therefore both attempt to reduce bottlenecks to increase productivity. Both methods are adverse to any activities that do not directly contribute to the final product, such as paperwork (although countless projects that have gone the nuclear option of “no documentation” have lived to regret it).

While not so obvious, there are numerous ways CMMI and LSS can be synergistically combined. Where a CMMI implementation might target the creation of a comprehensive MA infrastructure, an LSS approach would more likely focus on achieving a specific improvement to a particular problem that has a quantifiable (normally currency) near-term benefit—ultimately leading to an infrastructure quite similar to that resulting from a CMMI initiative. While the emphasis is different, with LSS placing greater significance on smaller, shorter (typically 4 months or less) projects with measurable benefits, in the end, aggregate outcomes may be very similar [9].

Agile provides software development methodologies, purposely absent from CMMI. CMMI provides the systems engineering practices (including the process management and support practices) that help deploy and continuously improve Agile methods in a project or an organization, regardless of its size. Unfortunately, project personnel are frequently left out of process design activities and are disinclined or openly sceptical toward the adoption of process improvement activities [8]. This situation is typical of some LSS-style approaches to process improvement as well. Using Agile principles and project personnel input when designing and selecting process activities can create more acceptable and efficient implementations of CMMI, LSS or even Agile itself.

Conclusion

“Faced with the choice between changing one’s mind and proving that there is no need to do so, almost everyone gets busy on the proof.”

-John Kenneth Galbraith

To this point I have offered very little in the way of unique thought. Just as I believe that nothing has actually been invented (from the wheel to the iPod), I have simply conducted research, organized and referenced the thoughts of others, and added my opinion derived from my own experience with dozens of projects. But based on my research, I will leave you with one suggestion.

W. Edwards Deming offered 14 key management principles for transforming business effectiveness [10] that were adopted by many American companies hoping to emulate Japanese business success. A number of Japanese manufacturers had applied his techniques widely and experienced theretofore unheard-of levels of quality and productivity. The improved quality combined with the lowered cost created new international demand for Japanese products. Most of these American experiments failed because a framework and corporate culture for integrating the principles did not exist.  One prime example is Deming’s insistence on all individual performance appraisals being abolished, in order to “drive out fear.” This only served to cause fear in U.S. corporate boardrooms.

There are many advocates of LSS who believe that once LSS is in place, projects can simplify CMMI implementation because much of the CMMI work (processes and artifacts) is already done. I would argue the opposite, however, that once a non-prescriptive process improvement framework such as CMMI is deployed, Agile and LSS project methodologies can be easily integrated.

Think of CMMI as an empty vessel with bins for continuous process improvement. Fill the bins with Agile user stories, daily meetings, short lifecycles, and frequent releases. Then apply the LSS roadmap—establishing overall objectives, performance measurement, issue analysis, progress monitoring, and targeted progress goals. The synergy realized, then, enables projects to select the best of Agile, LSS, and CMMI practices, for a cohesive approach to enhance continuous process improvement.

(Editor’s Note: The original provides ABOUT THE AUTHOR and REFERENCES details.)

Disclaimer:

CMMI® is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.

Peter D. Morris, PMP, Process Engineering Consultant

(Editor’s Note: The following is a brief extract from a Nov/Dec 2012 CrossTalk article. The original is available at http://www.crosstalkonline.org/storage/issuearchives/2012/201211/201211-… )