The desire for COTS products and some
The desire for COTS products––and some potential problems
There are also technological reasons in support of the use of OTS products. For example, Dawkins and Riddle (2000) point out that the relatively small safety-related systems market cannot sustain the rate of technological advancement stimulated by the huge commercial market, and that it would suffer technological retardation if it did not use OTS products.
A further potential problem is that in the absence of source code and design documentation from the supplier, there is a need to verify OTS software through black-box testing. Yet there are severe technical limitations on the confidence that can be derived from this, some of which are reviewed by Armstrong (2001). It is therefore not surprising that a great deal of research is currently enquiring into ways in which OTS software may be justified (e.g. Dawkins and Riddle, 2000; Bishop et al., 2001; Jones et al., 2001).
The issue of evidence Thus, gaining sufficient evidence about an OTS product to have confidence in its use in a safety-related application is, at best, difficult. Frankis and Armstrong (2001) suggest that to assess OTS software adequately it must be validated against thirteen ‘evidential requirements’, and they list five WM-1119 of examination of the requirements: black-box methods, previous usage analysis, design-intention assessment, code assessment, and open-box assessment. Such extensive validation, and the analysis to facilitate it, are not usual aspects of implementation programmes. Although the proposals are extensive, they have not been shown in practice to be sufficient. But they represent an early attempt to address the evidential problem, for the COTS debate is concerned with whether sufficient product evidence can be derived to allow safety assessment, and whether such evidence can replace the process evidence demanded by the standards. These issues recur in Section 5.
The issue of risk Thus, in the absence of evidence to provide confidence in the reliability of the OTS product, autotrophs should be assumed that if it could cause a hazardous event it would––i.e. that its probability of dangerous failure is unity. To do otherwise, certainly if it is software, would be contrary to safe practice. In their report on the failure of Ariane 5 Flight 501, the Inquiry Board (1996) stated, ‘software should be assumed to be faulty until applying the currently accepted best practice methods can demonstrate that it is correct.’
The challenge to the standards––the other side of the debate Further, the relationships between safety targets and particular development techniques differ between standards and therefore cannot be said to represent a co-ordinated view. Whereas IEC 61508 provides two extensive annexes of tables that explicitly define the appropriateness of specific techniques to the various SILs, the Motor Industry Software Research Association guidelines (MISRA, 1994) provide a single table in which only outline guidance is offered. The Ministry of Defence standard, MOD 00-56 (MOD, 1996), calls for ‘design rules and techniques appropriate to each safety integrity level’ to be ‘determined prior to implementation of the functions of the system’. Only IEC 61508 lays down strict relationships between techniques and SILs, and they are based on expert judgement. In MOD 00-56 the relationships between techniques and safety targets are defined by project personnel, may be different from those of IEC 61508, and may differ from project to project. If a system constructed according to MOD 00-56 turned out to be ‘safer’ than one built according to IEC 61508, it could be claimed that the latter’s rigorous requirements are unnecessary. Further, if a safety-related system not developed according to any of the standards is found in practice to meet a given safety target, it might be used as a basis for refuting the standards’ call for development-process evidence.