System Defects: Do we trust our methods for handling them?

On June 29, GMV held an internal webinar in which speakers from different sectors and with different outlooks and needs discussed the topic “System Defects: Do we trust our methods for handling them?”.

The origin of this event was an article published in the digital media that indicated that, in the latest battery of tests, the Pentagon had found nearly 900 software defects in the F-35 fighter, some highly critical and striking, such as the 25 mm cannon the aircraft is armed with not working properly, with “unacceptable” precision with regard to hitting ground targets.

This news as the subject of internal discussion in GMV’s Department of Quality and RAMS (Reliability, Availability, Maintainability and Safety). And we wondered: Are 900 defects a lot? A few? The number of defects seemed very high, but how many of them could actually have severe or catastrophic consequences? All of that led us to consider whether any of the systems we have developed or are currently developing at GMV could have that number of defects “released to the client”, and what would be the hypothetical consequences if, in addition, any of those defects could result in a severe or catastrophic incident. All of these questions motivated us to organize this webinar.

During the event, an important aspect was pointed out that should be taken into consideration when assessing the quality of a development, in addition to the number of defects that might be released to the client, which was their critical level, determined by the consequences that the failures could have. This is an aspect that GMV’s Quality Management System considers in the sense that it is adaptable based on the critical nature of the system to be developed. To do that, the development projects are categorized as “Operational Development Projects” and “Critical Operational Development Projects”, adjusting the base processes, procedures and practices of the Quality Management System based on whether the project belongs to one category or the other. This has been a key element when implementing an efficient development methodology.

In the case of non-critical “Operational Development Projects”, a minimal set of base practices have been established that enable us to develop high-quality products at an optimal cost.

The elements that have enabled us to reach that objective include having a productive model for software defects that helps us estimate in the initial stages of the projects, based on the defects detected in the different verification and validation activities, the quality of the software to be delivered, also making it possible to identify corrective actions when the data could indicate that the quality might be deficient. This model also makes it possible to control the cost of the defects, which combines with the objective of being more efficient when handling the defects.

It is a very powerful tool that other companies in the aeronautics sector do not generally have and only companies like GMV, which have reached maturity level 5 (the highest level) according to the CMMI improvement model, have developed.

Likewise, it should be mentioned that all GMV projects have institutionalized, very generally and rigorously, a set of efficient practices when detecting defects in the stages prior to testing, such as formal code inspections (which can detect up to 66% of the total defects found throughout the life of the product) and formal requirements and design reviews.

All of these practices and tools enable GMV to deliver products that, while never entirely free of defects (it is important to remember here that all products delivered by any company have defects), they are of excellent quality, with a low failure rate when in use by the client, all without extra charges to achieve that level of quality. In the case of software developments, the rate of defects released to the client is less than 1 defect per thousand lines of code.

With regard to the “Critical Operational Development Projects” it is not only a matter of delivering products with a low failure rate, but also to avoid releasing a failure that could have severe or catastrophic consequences. To achieve that objective, on the one hand, more rigorous verification activities are applied and, on the other hand, exhaustive RAMS analyses are conducted. GMV has a consolidated RAMS methodology that it applies to the “Critical Operational Development Projects” to identify and implement additional requirements and recommendations aimed at avoiding failures of a severe nature. Considering, according to some state-of-the-art studies related to SW quality, that high severity defects are primarily related to design/requirements, the RAMS engineering activities based on high-level analyses of those elements constitute the best guarantee in avoiding severe defects that could lead to incidents with severe or catastrophic consequences.

In light of the arguments presented above and others expressed during the webinar, the general impression among participants was that GMV has a solid development methodology that enables efficient defect handling.

Author: Santiago Ledesma