Software quality

Software is the collection of computer programs, related data, and associated documentation developed for a particular customer or for a general market. It is an essential part of the modern world, and it has become pervasive in telecommunications, utilities, commerce, culture, entertainment, and so on. The question What is software quality? can generate different answers, depending on the involved practitioner's role in a software system. There are two main groups of people involved in a software product or service:

• Consumers: are people who use software. In this group, we can differentiate between customers (that is, people responsible for the acquisition of software products or services) and users (that is, people who use the software products or services for various purposes). Nevertheless, the dual roles of customers and users are quite common.
• Producers: are people involved with the development, management, maintenance, marketing, and service of software products.

The quality expectations of consumers are that a software system performs useful functions as specified. For software producers, the fundamental quality question is fulfilling their contractual obligations by producing software products that conform to the Service Level Agreement (SLA). The definition of software quality by the well-known software engineer Roger Pressman comprises both points of view:

An effective software process applied in a manner that creates a useful product that provides measurable value for those who produce it and those who use it.

Quality engineering

Quality engineering (also known as quality management) is a process that evaluates, assesses, and improves the quality of software. There are three major groups of activities in the quality engineering process:

1. Quality planning: This stage establishes the overall quality goal by managing customer's expectations under the project cost and budgetary constraints. This quality plan also includes the strategy, that is, the selection of activities to perform and the appropriate quality measurements to provide feedback and assessment.

2. Quality Assurance (QA): This guarantees that software products and processes in the project life cycle meet their specified requirements by planning and performing a set of activities to provide adequate confidence that quality is being built into the software. The main QA activity is Verification & Validation, but there are others, such as software quality metrics, the use of quality standards, configuration management, documentation management, or an expert's opinion.
3. Post-QA: These stage includes activities for quality quantification and improvement measurement, analysis, feedback, and follow-up activities. The aim of these activities is to provide quantitative assessment of product quality and identification of improvement opportunities.

These phases are represented in the following chart:

Software Quality Engineering Process

Requirements and specification

Requirements are a key topic in the quality engineering domain. A requirement is a statement identifying a capability, physical characteristic, or quality factor that bounds a product or process need for which a solution will be pursued. The requirement development (also known as requirements engineering) is the process of producing and analyzing customer, product, and product-component requirements. The set of procedures that support the development of requirements, including planning, traceability, impact analysis, change management, and so on, is known as requirements management. There are two kinds of software requirements:

• Functional requirements are actions that the product must do to be useful to its users. They arise from the work that stakeholders need to do. Almost any action such as, inspecting, publishing, or most other active verbs can be a functional requirement.
• Non-functional requirements are properties, or qualities, that the product must have. For example, they can describe properties such as performance, usability, or security. They are often called quality attributes.

Another important topic strongly linked with the requirements is the specification, which is a document that specifies in a complete, precise, verifiable manner, the requirements, design, behavior, or other characteristics of a system, and often the procedures for determining whether these provisions have been satisfied.

Quality Assurance

Quality Assurance (QA) is primarily concerned with defining or selecting standards that should be applied to the software development process or software product. Daniel Galin, the author of the book Software Quality Assurance (2004) defined QA as:

Systematic, planned set of actions necessary to provide adequate confidence that the software development and maintenance process of a software system product conforms to established specification as well as with the managerial requirements of keeping the schedule and operating within the budgetary confines.

The QA process selects the V&V activities, tools, and methods to support the selected quality standards. V&V is a set of activities carried out with the main objective of withholding products from shipment if they do not qualify. In contrast, QA is meant to minimize the costs of quality by introducing a variety of activities throughout the development and maintenance process in order to prevent the causes of errors, detect them, and correct them in the early stages of development. As a result, QA substantially reduces the rates of non-qualifying products. All in all, V&V activities are only a part of the total range of QA activities.

ISO/IEC-25000

Various quality standards have been proposed to accommodate these different quality views and expectations. The standard ISO/IEC-9126 was one of the most influential in the software engineering community. Nevertheless, researchers and practitioners detected several problems and weaknesses in this standard. For that reason, the ISO/IEC-9126 international standard is superseded by the ISO/IEC-25000 series of international standards on Software product Quality Requirements and Evaluation (SQuaRE). This section provides a high-level overview of this standard.

The ISO/IEC-2500 quality reference model distinguishes different views on software quality:

• Internal quality: This concerns the properties of the system, that can be measured without executing it.
• External quality: This concerns the properties of the system, that can be observed during its execution.
• Quality in use: This concerns the properties experienced by its consumer during operation and maintenance of the system.

Ideally, the development (process quality) influences the internal quality; then, the internal quality determines the external quality. Finally, external quality determines quality in use. This chain is depicted in the following picture:

ISO/IEC-2500 Product Quality Reference Model

The quality model of ISO/IEC-25000 divides the product quality model (that is, the internal and external attributes) into eight top-level quality features: functional suitability, performance efficiency, compatibility, usability, reliability, security, maintainability, and portability. The following definitions have been extracted directly from the standard:

• Functional suitability: This represents the degree to which a product or system provides functions that meet stated and implied needs when used under specified conditions.
• Performance efficiency: This represents the performance relative to the amount of resources used under stated conditions.
• Compatibility: This is the degree to which a product, system or component can exchange information with other products, systems or components, and/or perform its required functions, while sharing the same hardware or software environment.
• Usability: This is the degree to which a product or system can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.
• Reliability: This is the degree to which a system, product, or component performs specified functions under specified conditions for a specified period of time.
• Security: This is the degree to which a product or system protects information and data so that persons or other products or systems have the degree of data access appropriate to their types and levels of authorization
• Maintainability: This represents the degree of effectiveness and efficiency with which a product or system can be modified to improve it, correct it, or adapt it to changes in environment and in requirements
• Portability: This is the degree of effectiveness and efficiency with which a system, product, or component can be transferred from one hardware, software, or other operational or usage environment to another

On the other hand, the attributes of quality in use can be categorized into the following five characteristics:

• Effectiveness: This is the accuracy and completeness with which users achieve specified goals.
• Efficiency: These are the resources expended in relation to the accuracy and completeness with which users achieve goals.
• Satisfaction: This is the degree to which user needs are satisfied when a product or system is used in a specified context of use.
• Freedom from risk: This is the degree to which a product or system mitigates the potential risk to economic status, human life, health, or the environment.
• Context coverage: This is the degree to which a product or system can be used with effectiveness, efficiency, freedom from risk, and satisfaction in both specified contexts of use and in contexts beyond those initially explicitly identified.

Verification and Validation

Verification and Validation -also known as Software Quality Control- is concerned with evaluating that the software being developed meets its specifications and delivers the functionality expected by the consumers. These checking processes start as soon as requirements become available, and continue through all stages of the development process. Verification is different to validation, although they are often confused.

The distinguished professor of computer science Barry Boehm expressed the difference between them back in 1979:

• Verification: are we building the product right? The aim of verification is to check that the software meets its stated functional and non-functional requirements (that is, the specification).
• Validation: are we building the right product? The aim of validation is to ensure that the software meets consumer's expectations. It is a more general process than verification, due to the fact that specifications do not always reflect the real wishes or needs of consumers.

V&V activities include a wide array of QA activities. Although software testing plays an extremely important role in V&V, other activities are also necessary. Within the V&V process, two big groups of techniques of system checking and analysis may be used:

• Software testing: This is the most commonly performed activity within QA. Given a piece of code, software testing (or simply testing) consists of observing a sample of executions (test cases), and giving a verdict on them. Hence, testing is an execution-based QA activity, so a prerequisite is the existence of the implemented software units, components, or system to be tested. For this reason, it is sometimes called dynamic analysis.
• Static analysis: This is a form of V&V that does not require execution of the software. Static analysis works on a source representation of the software: either a model of the specification of design or the source or the program. Perhaps, the most commonly used are inspections and reviews, where a specification, design, or program is checked by a group of people. Additional static analysis techniques may be used, such as automated software analysis (the source code of a program is checked for patterns that are known to be potentially erroneous).

It should be noted that there is a strong divergence of opinion about what types of testing constitute validation or verification. Some authors believe that all testing is verification and that validation is conducted when requirements are reviewed and approved. Other authors view unit and integration testing as verification and higher-order testing (for example, system or user testing) as validation. To solve this divergence, V&V can be treated as a single topic rather than as two separate topics.

Software defects

Key to the correctness aspect of V&V is the concept of software defects. The term defect (also known as bug) refers to a generic software problem. The IEEE Standard 610.12 propose the following taxonomy related to software defects:

• Error: A human action that produces an incorrect result. Errors can be classified into two categories:
  1. Syntax error (program statement that violates one or more rules of the language in which it is written).
  2. Logic error (incorrect data fields, out-of-range terms, or invalid combinations).
• Fault: The manifestation of an error in the software system is known as a fault. For example, an incorrect step, process, or data definition.
• Failure: The inability of the software system to perform its required functions is known as (system) failure.

The term bug was first coined in 1946 by the software pioneer Grace Hooper, when a moth trapped in rely of an electromechanical computer caused a system malfunction. In this decade, the term debug was also introduced, as the process of detecting and correcting defects in a system.

In addition to this level of granularity for defects, it is also interesting to contemplate incidences as symptoms associated with a failure perceived by the software consumer. All in all, error, faults, failures, and incidences are different aspects of software defects. A causal relation exists between these four aspects of defects. Errors may cause faults to be injected into the software, and faults may cause failures when the software is executed. Finally, incidences happen when failures are experienced by the final user or costumer. Different QA activities can be carried out to try to minimize the number of defects within a software system. As defined by Jeff Tian in his book Software Quality Engineering (2005), the alternatives can be grouped into the following three generic categories:

• Defect prevention through error removal: For example, the use of certain processes and product standards can help to minimize the injection certain kinds of faults into the software.
• Defect reduction through fault detection and removal: The traditional testing and static analysis activities are examples of this category. We discover the specific types of these mechanisms in the body of this chapter.
• Defect containment through failure prevention: These activities are typically out of the scope of the software system. The objective of containment is to minimize the damage caused by software system failures (for example, walls to contain radioactive material in case of reactor failures).

Software defect chain and associated QA activities

Static analysis

Static analysis of a software piece is performed without executing the code. There are several advantages to software analysis over testing:

1. During testing, errors can hide other errors. This situation does not happen with static analysis, because it is not concerned with interactions between errors.
2. Incomplete versions of a system can be statically analyzed without additional cost. In testing, if a program is incomplete, test harnesses have to be developed.
3. Static analysis can consider broader quality attributes of a software system, such as compliance with standards, portability, and maintainability.

There are different methods that can be identified as static analysis:

• Inspection (first proposed by Michael Fagan in 1976) are examinations of software artifacts by human inspectors aimed at discovering and fixing faults in the software systems. All kinds of software assets are subject to be inspected, for example the specification, design models, and so on. The primary reason for the existence of inspection is not waiting for the availability of executable programs (such as in testing) before starting performing inspection.
• Review is the process in which a group of people examine the software and its associated documentation, looking for potential problems and non-conformance with standards, and other potential problems or omissions. Nowadays, reviews are frequently carried out for new code before being merged in a shared source code repository. Typically, the review is done by a different person to the code author within the same team (peer review). This process is quite expensive in terms of time and effort, but on the other side, when correctly performed, it helps to ensure a high internal code quality reducing potential risks.

A walkthrough is a special form of review. According to IEEE Standard for Software Reviews, a walkthrough is a form of software peer review in which a designer or programmer leads members of the development team and other interested parties through a software product, and the participants ask questions and make comments about possible errors, violation of development standards, and other problems.

• Automated software analysis assesses the source code using patterns that are known to be potentially dangerous. This technique is usually delivered as commercial or open source tools and services, commonly known as lint or linter. These tools can locate many common programming faults, analyze the source code before it is tested, and identify potential problems in order to re-code them before they manifest themselves as failures. The intention of this linting process is to draw a code reader’s attention to faults in the program, such as:
  1. Data faults: This may include variables declared but never used, variables assigned twice but never used between assignments, and so on.
  2. Control faults: This may include unreachable code or unconditional branches into loops.
  3. Input/output faults: This may include variables output twice with no intervening assignment.
  4. Interface faults: This may include parameter-type mismatches, parameter under mismatches, non-usage of the results of functions, uncalled functions and procedures, and so on.
  5. Storage management faults: This may include unassigned pointers, pointers arithmetic, and so on.

Halfway between static analysis and dynamic testing we find an especial way of software evaluation, called formal verification. This kind of assessment provides mechanisms to check that a system operates according to its formal specification. To that aim, software is treated as a mathematical entity whose correctness can be proved using logical operations, combining different types of static and dynamic evaluation. Nowadays, formal methods are not widely adopted mainly due to scalability problems. Projects using these techniques are mostly relatively small, such as critical kernel systems. As systems grow, the effort required to develop a formal specification and verification grow excessively.