| 4.4.12-4-1 Ensure plans for validating integration and verification test environments are complete and correct |
Phase B |
Phase C |
Phase D1 |
Phase D2 |
Phase D3 |
|
Ensure assessment of the contractor's plans for validating the elements of their integration and verification test environments to ensure they meet their requirements and perform as needed before use in verifying software-related requirements at any level. The elements of the test environments include test beds, test facilities, simulators, emulators, test drivers, test data generators, automated test tools, and other test tools as necessary. Ensure the integration and test environments are maintained after launch to support anomaly resolution and testing of changes to flight software and operational flight constants.
|
NA
|
Test and Evaluation Master Plan (TEMP)
|
TOR-2006(8506)-5749, Mission Assurance Tasks for Software; Mission Assurance Guide, TOR-2007(8546)-6018, Rev B, Software Mission Assurance Chapter
|
| 4.4.12-4-2 Ensure the subsystem, element, segment, and system test plans are assessed |
Phase B |
Phase C |
Phase D1 |
Phase D2 |
Phase D3 |
|
Ensure the contractor's higher level test plans at the subsystem, element, segment, and system (as appropriate for this program) for correctness, completeness, consistency, feasibility, clarity, and lack of ambiguity. Ensure that all software-related requirements are traced to planned tests, and that the plans address all requirements verification methods (I, A, D, T). Ensure that the software-related requirements will be verified under conditions as close to the operational environment as possible (including target processors, hardware configuration, workloads, data rates, etc.).
|
NA
|
System Test Plan; Integration and Test Plans; Test and Evaluation Master Plan (TEMP)
|
TOR-2006(8506)-5749, Mission Assurance Tasks for Software; Mission Assurance Guide, TOR-2007(8546)-6018, Rev B, Software Mission Assurance Chapter
|
| 4.4.12-4-3 Ensure assessment of contractor's software unit test planning |
Phase B |
Phase C |
Phase D1 |
Phase D2 |
Phase D3 |
|
Ensure the contractor has established adequate unit testing criteria and test cases (input, expected results, evaluation criteria), test procedures and test data for testing the software of each software unit. Ensure that the test cases cover, as a minimum, the correct execution of all branches and statements in the unit, all errors and exception handling, all software interfaces including limits and boundary conditions, start-up, termination and restart (where applicable), and all algorithms. Ensure that any legacy software that has been modified undergoes unit testing. Where reuse software units have not been modified, ensure retesting of reuse units where track record indicates potential problems and for all critical reuse software units. Verify that the contractor records information from software unit testing as stipulated in compliance documents.
|
NA
|
Integration and Test Plans
|
TOR-2006(8506)-5749, Mission Assurance Tasks for Software, Mission Assurance Guide, TOR-2007(8546)-6018, Rev B , Software Mission Assurance Chapter, TOR-2004(3909)-3537, Rev B, Software Development Standard for Space Systems, section 5.7
|
| 4.4.12-4-4 Ensure assessment of contractor's software integration test planning |
Phase B |
Phase C |
Phase D1 |
Phase D2 |
Phase D3 |
|
Ensure the contractor has established adequate software integration testing criteria and test cases (input, expected results, evaluation criteria), test procedures and test data for testing the software as new units are integrated. Ensure that the test cases cover, as a minimum, the correct execution of all interfaces; integration error and exception handling across the software units under test; all end-to-end functional capabilities for the software units under test; performance testing; stress testing, including worst-case scenario(s); start-up, termination and restart (where applicable); fault detection, isolation and recovery handling; and resource utilization measurement (e.g., CPU, memory, storage and data bus bandwidth). Ensure that all reuse software, including modified and unmodified legacy reuse, Open Source Software (OSS) and COTS software undergoes integration testing. Verify that the contractor records information from software integration testing as stipulated in compliance documents. Ensure that, wherever possible, software integration testing is to be performed on target hardware configured as close to the operational configuration as possible
|
NA
|
Integration and Test Plans
|
TOR-2006(8506)-5749, Mission Assurance Tasks for Software, Mission Assurance Guide, TOR-2007(8546)-6018, Rev B , Software Mission Assurance Chapter, TOR-2004(3909)-3537, Rev B, Software Development Standard for Space Systems
|
| 4.4.12-4-5 Ensure assessment of contractor's software qualification test planning |
Phase B |
Phase C |
Phase D1 |
Phase D2 |
Phase D3 |
|
Ensure that the contractor's software testers have developed adequate test preparations, test cases, test procedures and test data to be used for software qualification testing, as well as traceability between test cases, test procedure steps and the software item and software interface requirements. Ensure that the software tests covers, as a minimum, verification of all software requirements under conditions that are as close as possible to actual operating conditions; verification of all software interface requirements using the actual interfaces wherever possible or high-fidelity simulation where not; verification of software reliability requirements including fault detection, isolation and recovery requirements; stress testing including worst-case scenario(s); and resource utilization measurement (e.g., CPU, memory, storage and data bus bandwidth). Ensure that the test developer has dry run all software qualification test procedures to ensure they are complete and accurate and the software is ready for witnessed testing, and that software qualification testing is performed on target hardware in a configuration as close to the operational configuration as possible.
|
NA
|
Integration and Test Plans
|
TOR-2006(8506)-5749, Mission Assurance Tasks for Software, Mission Assurance Guide, TOR-2007(8546)-6018, Rev B, Software Mission Assurance Chapter, TOR-2004(3909)-3537, Rev B, Software Development Standard for Space Systems, section 5.9
|
| 4.4.12-4-6 Ensure assessment of contractor's software test environments |
Phase B |
Phase C |
Phase D1 |
Phase D2 |
Phase D3 |
|
Ensure assessment of the adequacy of the contractor's test environments (e.g., test beds, test facilities, simulators, emulators, test drivers, test tools) for performing software unit testing, integration testing and qualification testing to ensure the environments are suitable for a robust software testing program. Assess the contractor's plans for procuring the test environments to ensure the plans will enable the elements of the test environments to be ready when needed. Assess the contractor's plans for validating the test environments, especially simulators and emulators, to ensure they satisfy their requirements and function as needed. Ensure the test environments are in place and validation is complete before they are used.
|
NA
|
NA
|
TOR-2006(8506)-5749, Mission Assurance Tasks for Software, Mission Assurance Guide, TOR-2007(8546)-6018, Rev B, Software Mission Assurance Chapter, TOR-2004(3909)-3537, Rev B, Software Development Standard for Space Systems, TR-2004(8583)-1, Rev. A, Test Requirements for Launch, Upper Stage, and Space Vehicles
|
| 4.4.12-4-7 Ensure an independent verification and validation of contractor's plans |
Phase B |
Phase C |
Phase D1 |
|
Ensure assessment of the Independent Verification and Validation (IV&V) contractor's plan is correct, complete, consistent, clear, unambiguous, and feasible. Ensure that the IV&V Plan includes verification of requirements, design and code. Ensure that schedule is sufficient to allow problems found to be addressed. Ensure that the IV&V Plan includes validation of software. Ensure the IV&V Plan identifies resources needed for effort (simulators/emulators, flight (or flight-like) hardware, and personnel).
|
NA
|
NA
|
Mission Assurance Tasks for Software TOR, IEEE Std 1012(TM)-2004, IEEE Standard for Software Verification and Validation, TOR-2001(1465)-0934e, Software Independent Verification and Validation
|