Non-Functional Requirements Be Here

(Editor’s Note: The following is an extract of a December 2012 CISQ blog. The original is available at http://itcisq.org/non-functional-requirements-be-here/)

Non-Functional App Killers

Non-functional requirements are the killers of most modern multi-language, multi-layer applications. The continuing stream of business system disasters being reported in the press are rarely the result of a functional defect. Rather they are structural flaws in the system that we cluster under the catchall phrase, ‘non-functional requirements’. Okay, fine, but shouldn’t these structural, non-functional requirements have been stated clearly from the beginning? After all, they are requirements!

The brilliant Greek computer scientist Diomidis Spinellis warned us about this conundrum in his book Code Quality (2006, Addison Wesley):

“…a failure to satisfy non-functional requirements can be critical, even catastrophic …an insecure web-server or unreliable anti-lock brake system are worse than useless…non-functional requirements are sometimes difficult to verify. We cannot write a test case to verify a system’s reliability or the absence of security vulnerabilities.”

 

The Challenge of Verifying Non-Functional Requirements

Quality assurance has become reasonably good at testing for the correctness of functional requirements. Such requirements can be stated clearly if the system analysis and customer interaction processes are performed properly. But as Spinellis points out, non-functional requirements are a different phenomenon. Not only will we have difficulty writing test cases to verify them, we cannot expect customers or analysts to even state them with any detail beyond simple descriptions of system behavior.

Non-functional requirements are usually stated initially as system behaviors, so from here on I will refer to them as nonfunctional behavioral requirements. Such requirements are usually presented in one of two forms, quantitative or restrictive. Examples of these forms include:

Quantitative: The system shall respond to user commands within 500 milliseconds. The system shall be available during 99.98% of business hours.

Restrictive: Unauthorized users shall not be able to access financial data. Data shall only be accessed through approved data access methods.

At the behavioral level we understand these requirements.  However, how do we ensure they have been achieved in the delivered software? How do you test that the system will be available during 99.98% of business hours?  The standard answer is that non-functional behavioral requirements are verified through dynamic, load, stress, or penetration testing. When properly performed these verification techniques are critical constituents of a comprehensive quality assurance program. However, raise your hand if you have the resources to fully simulate your application’s operational environment. I thought so, and the guy at the back of the internet with his hand up should go into sales or politics.

The fact is that few QA organizations have sufficient resources to simulate the loads and stresses that their applications will experience when they go live in the operational environment.  Consequently we rarely have the confidence to state with conviction that we can satisfy non-functional behavioral requirements such as those stated in the examples above. So, what to do?

 

Non-Functional Requirements = Structural Quality

First, take a step back. Each of the system attributes described in non-functional behavioral requirements is controlled to a large extent by structures in the code. Consequently, the first step in satisfying non-functional requirements is to ensure that the software does not contain violations of good architectural and coding practice that would cause the system’s behavior to fail the requirement.

If we could detect architectural or component-level defects that affect outcomes such as 99.98% availability or unauthorized access to data, we could take targeted remedial actions toward satisfying an application’s non-functional behavioral requirements. In essence we would be decomposing the nonfunctional behavioral requirements that are difficult to verify, into non-functional structural requirements about specific attributes of the system’s architecture or code that can be verified. We will call these non-functional structural requirements, since they provide the foundation on which nonfunctional behavioral requirements can be satisfied.

 

Many Non-Functional Structural Requirements Are Already Defined

Isn’t decomposition of non-functional behavioral requirements into non-functional structural requirements a lot of work for analysts, architects, and developers? Yes, but fortunately it has already been accomplished. For instance, a behavioral security-related requirement restricting data access to authorized users could be decomposed into specific nonfunctional structural requirements using the Common Weakness Enumeration (CWE) repository hosted by Mitre Corporation with support the US Department of Homeland Security (cwe.mitre.org). The software assurance community has already catalogued security-related structural flaws that should be eliminated from software systems to protect confidential information.

If you just take the CWE Top 25 weaknesses, you have a list of non-functional structural requirements for the software that cover most of the popular vulnerabilities hackers exploit to penetrate confidential data. Using static analysis, code reviews, or other methods, you would look for and eliminate instances of cross-site scripting, SQL injection, potential buffer overflows, and related security weaknesses identified in the CWE. The CWE Top 25 should be listed as non-functional structural requirements for every IT system with a security related non-functional behavioral requirement related to security.

Penetration testing to assess compliance with the nonfunctional behavioral security requirements should still be conducted. Nevertheless, static analysis and peer reviews of the software system will have already detected many, perhaps most of the structural defects that threaten satisfaction of the non-functional behavioral requirements for security. The cost of removing security-related weaknesses in the system is 10X cheaper when detected as violations of non-functional structural requirements during development, compared to being tragically discovered as violations of non-functional behavioral requirements during operations when confidential data is already out the door.

 

CISQ Measures Assess Non-Functional Structural Requirements Risk

The Consortium for IT Software Quality (CISQ) has recently completed an enumeration of the non-functional structural defects in four critical areas—Reliability, Performance Efficiency, Security, and Maintainability. An international team of computer scientists defined between 15 and 28 rules of good architectural and coding practice that ensure software exhibits high levels of each of these system characteristics. In that sense, each of these rules constitutes a non-functional structural requirement for software that can be directly related to non-functional behavioral requirements for Reliability, Performance Efficiency, Security, or Maintainability. For instance, a rule instructing developers to avoid placing computationally expensive operations inside loops would constitute a non-functional structural requirement related to Performance Efficiency.

 

(Editor’s Note: The original provides further information on detecting and counting violations of the rules of good architectural and coding practice and references the rules and associated measures underlying each of the CISQ Software Quality Characteristic Measures. These are consistent with the new ISO 25000 series standards that govern software and system product quality. )