It’s the Product, Stupid!

Too often when I meet with executives I get confronted with,  “Hey, you’re the CMM guy. How come my outsourcer is Level  5 and the software they sent me is full of bugs?” CMM and its successor, CMMI, were never meant to be  certifications for defect-free software products. Rather, they  were roadmaps to help organizations adopt best software  engineering practices that have proven over many decades to  produce high quality software products.  There are many factors beyond process that affect the quality  of a software product, such as:

.The skill of the developers and their knowledge of the domain of application,

. The novelty of the technology,

. The accuracy and volatility of the requirements, and

. The effectiveness with which a rapidly growing organization  can sustain its high maturity development practices.

A high maturity process can mitigate the detrimental effects of  these factors, but it cannot eliminate them. Ultimately, process  only provides capability-the potential to produce high quality  products. The achieving the full potential of an organization’s  capability only occurs with sustained discipline and a  dedication to managing factors that degrade it.  Most of us involved in the early development of CMM-Watts  Humphrey, Mark Paulk, Charles Weber, and myself among  others-believed that we did not even need to use the term  ‘effective’ when describing the processes to be implemented.  We believed that if developers would just adopt a practice or  process, even if its results were less than expected, they would  improve it over time to achieve optimal results. Unfortunately  this did not happen as often as we had hoped.

Too many organizations were transfixed on getting their ‘Level 5’ for marketing reasons. They were more worried about compliance with ‘the CMM/CMMI standard’ than they were about implementing a process that produced excellent results.

Robotic compliance does not ensure continual improvement.  Process appraisals compounded this problem. Appraisers were  trained to evaluate compliance with CMM/CMMI practices,  but did not assess their result, that is, the quality of the product  produced. For instance, in high maturity appraisals appraisers  would inspect charts of defect data to see if the appraised  organization had gotten defect rates under ‘statistical control’.  Yet, there was no penalty for high defect rates based on the  assumption that actions would be taken continually to reduce  them. Although compliance to high maturity practices should  help reduce defect rates, proof of significant reductions or  even the achievement of a minimum threshold was not  required to be appraised at Levels 4 or 5. The only guarantee  of continually declining defect rates is a relentless devotion to  product excellence.

This was the lesson of the Space Shuttle Primary Avionics  Software System (SS-PASS), arguably the most process- disciplined software project in history. The SS-PASS team  produced a product that was near-defect-free for most of its 30  year life. Before every shuttle launch the SS-PASS managers  had to sign an affidavit affirming to NASA that to the best of  their knowledge there were no safety-related defects in the SS- PASS. Consequently, the entire SS-PASS organization was  laser-focused on the quality of their product.  The rigor of SS-PASS’s development process was only a  means to gain the confidence needed to sign NASA’s affidavit  in good faith. They were the living model we used in defining  Levels 4 and 5 of the original CMM, but in our interactions  they never claimed to be Level 4 or 5. However, they had no  trepidation in signing NASA’s affidavit. Would your suppliers  be willing to sign such an affidavit about the risks their product  poses to your customers or your business?  The risk to the business is in the functionality, structure, and  execution of the software product, period. A high maturity  development process should reduce the risk, but many factors  conspire to ensure that this is not always guaranteed. To  paraphrase a famous quote about misplaced focus in past U.S.  presidential elections (“It’s the economy, stupid”), good  governance of IT investments must focus on the end result that  is delivered to the business or customer-“IT’S THE PRODUCT, STUPID!”

Convenient governance is not necessarily good governance.  Convenient governance accepts a process appraisal result  delivered by someone else as ‘certifying’ that a supplier has a  mature, perhaps even high maturity, development process.  However, good governance requires that you perform the due  diligence required to ensure that these processes are  consistently applied by the department or location where your  software will be developed.

Since a high maturity process enables but does not guarantee a  high quality result, good governance requires more. You  should evaluate a supplier’s historical data to verify they are  achieving the product results that a high maturity process  should deliver. If they do not have such data, run.

Most important, every organization that receives a software product from internal or external sources should proactively determine that its quality is sufficient to meet the needs of customers or the business without exposing them to unacceptable risks. There are at least four attributes that should  be evaluated in exercising proper governance over the quality  of a software product:

1. The functionality is correct and sufficiently complete to  meet user needs,

2. The structure of the software does not exhibit weaknesses  that expose the customer or business to unacceptable risks of  outages, security breaches, data corruption, or other damaging  operational problems,

3. The structure of the software does not expose IT to  excessive costs of ownership, and

4. The software will behave as expected when loaded into the  operational environment.

Good governance requires a Quality Gate where these  attributes of the delivered software can be evaluated and  approved before it is put into operation. Even if a Quality Gate  is operated by contracted staff, the organization acquiring the  software must oversee its operation and results. Among the  attributes of a good Quality Gate are the following features.  . A senior member of the acquiring company oversees and  reviews all results.

. This person has the authority to stop a software delivery from  entering operations.

. The evaluation is conducted against quality thresholds that  are written into the requirements, or for external suppliers, into  a contract or documented agreement.

. Data is retained across deliveries for evaluating the capability  of different sources.

The technologies for functional testing, static structural  analysis, and dynamic behavioral analysis have matured  sufficiently to provide strong, reliable feedback on the various  quality characteristics of a software product. The 1990s was  about trying to get software suppliers to implement a  development process that was capable of delivering high  quality products. The 2000s were about defining the  architectural and structural rules for a reliable, secure, scalable  software product. The 2010s must be about creating the  governance mechanisms that ensure the increasing automation  of business decreases rather than increases risks to customers  and operations.

In the face of serious operational problems, governance is only  defensible if IT can affirm that it used state of the practice  techniques to try and detect product flaws before going  operational. However, in the absence of governing the quality  of software products, IT stands fully exposed in front of  customers and the CEO. CEO’s don’t care to hear about a  developer’s processes or maturity. Once a software product  goes operational, you own whatever happens next. Ultimately,  it’s the product!

(Editor’s Note:  The following is a reprint of an October 2012   CISQ blog.  The original  is available at http://it-cisq.org/its- the-product-stupid/ )