| 5.4-6-1 Ensure higher level test cases, procedures, execution, data analysis and test result reports for the software-related portions of the system, segment, element, and subsystem are adequate for Qualification of requirements and risk reduction for integration at higher levels |
Phase D1 |
|
Ensure the higher level Qualification procedures captures appropriate results data for software requirements Qualification. Ensure data analysis is robust, e.g., includes content, format, range, units, type, accuracy, and precision checks. Ensure software interface test procedures between systems, segments, elements, and subsystems include limit, boundary, and range testing. Ensure entities under test in higher level testing have passed lower level tests and are integrated and tested prior to execution of higher level testing. Ensure the integration of entities in higher level test is done in a logical, incremental method prior to entry into higher level testing to reduce integration risk to the program, i.e., no big bang integration and test.
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|
| 5.4-6-2 Ensure robust contractor processes exist that result in software integration and qualification environments integrated across all software team members and that the environments support the programs' processes and tools |
Phase D1 |
|
Assess the contractors' software integration and qualification testing environment development and maintenance processes to ensure they provide an environment that supports the program's software integration and qualification testing requirements. Ensure any gaps in the processes are addressed well in advance of the need dates. Ensure the environment development and maintenance processes handle resolution of competing needs for facility use and support. Ensure the processes are updated and maintained to reflect changes that impact the integration environment. Ensure the processes are followed to make needed updates to the environments.
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|
| 5.4-6-3 Ensure the contractors' software qualification testing processes, methods, and tools are appropriate for the software qualification testing |
Phase D1 |
|
Review the qualification test processes to Ensure they meet the requirements of the Software Development Standard, including: preparation of test plans, test cases, test procedures, regression test cases, and test reports, and conducting testing and retesting according to those preparations. Ensure the software test processes address initial test, retest, and regression test; incorporation of failure, detection, and recovery scenarios, test-like-you-fly, test-like-you-operate, day-in-the-life, duration, and other operational tests required by the Software Development Standard. Ensure the processes require qualification testing to be performed by personnel independent of the software developers and addresses involvement of external stakeholders, (e.g., customers, quality assurance, users and readiness assessors). Ensure the processes require updates to test cases and procedures due to changes that impact software and hardware, and require rerunning tests when software, hardware, or environment changes are made. Ensure the software qualification test processes are followed. Ensure verification and test program defined, including failure and recovery testing. Ensure that, where necessary, test labs and special test equipment to enable failure and recovery testing have been designed/implemented. Ensure procedures for gathering necessary test data defined (e.g., recording operating time and unplanned failures) are complete.
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|
| 5.4-6-4 Ensure critical software is identified and documented for consideration for integration and qualification test planning |
Phase D1 |
|
Ensure critical software items or portions thereof whose failure could lead to violations of critical requirements are considered in the integration and qualification test planning. Critical software requirements should be documented in the software development plan and includes: safety; security; privacy protection; dependability, reliability, maintainability, and availability; and other mission -critical requirements (e.g. derived from Key Performance Parameters as agreed to by the acquirer and developer).
|
NA
|
Integration and Test Plans
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems
|
| 5.4-6-5 Ensure independent testing of critical ground software is performed, as appropriate |
Phase D1 |
|
Ensure the facilities, equipment, including target computer system boards, and source and executable software are available to independently execute the critical ground software. Ensure validated simulations of interfacing software, hardware, and data are available in place of unavailable interfacing entities. Ensure the critical ground software is independently analyzed and executed. Ensure the independent testing is performed with simulations of interfacing entities if the actual entities are not available.
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|
| 5.4-6-6 Ensure the contractors' software integration and testing processes, methods, and tools are appropriate for the software integration and testing |
Phase D1 |
|
Review the integration and test processes to ensure they meet the requirements of the Software Development Standard, including: preparation of test plans, test cases, test procedures, regression test cases, and test reports, and conducting testing and retesting according to those preparations. Ensure the software test processes address initial test, retest, and regression test; incorporation of test-like-you-operate, day-in-the-life, duration, and other tests required by the Software Development Standard and by the contract. Ensure the processes require updates to test cases and procedures due to changes that impact software and hardware, and require rerunning tests when software, hardware, environment, or operational concept changes are made. Ensure the processes are followed.
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|
| 5.4-6-7 Ensure the software integration products: test cases, procedures, execution, data analyses, regression testing, and test result reports; are adequate for integration of the item under test |
Phase D1 |
|
Ensure software integration test procedures and cases have complete coverage of functionality and interfaces. Ensure interface testing includes execution of values within, at, and out of bounds and are demonstrated to meet software architecture and design, and non-functional (e.g., dependability, maintainability, usability, portability) requirements. Ensure interface data analysis is robust, e.g., includes content, format, range, units, type, accuracy, and precision checks. Ensure software threads within the architecture are successfully executed for both nominal and off-nominal conditions from end-to-end through the entities under test and their interfacing neighbors. Ensure the threads use the actual interfaces, (or if not possible, then validated realistically modeled or simulated interfaces).
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|
| 5.4-6-8 Ensure the software qualification testing products: test cases, procedures, execution, data analysis, and test result reports; for the item under test are adequate for verifying software requirements |
Phase D1 |
|
Ensure software qualification test procedures and cases have complete coverage of functional and non-functional requirements, including interface requirements. Ensure the qualification methods (I, A, D, T) completely cover the requirements Qualification, especially for those requirements that are partially verified at lower levels or higher levels. Ensure software qualification procedures capture appropriate data to verify software requirements. Ensure data analysis is robust e.g., includes content, format, range, units, type, accuracy, and precision checks. Ensure test report results accurately and completely reflect the execution of the as-run test cases and procedures and the records include QA signatures, captured files, and procedure annotations. Perform a requirements trace analysis to ensure bi-directional traceability among requirements and test products. Ensure any requirements qualified in lower level tests have adequate evidence to qualify the requirements and the evidence is captured for rolling up to the software item qualification test, or possibly to a higher level if the requirement cannot be verified at the software item level.
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|
| 5.4-6-9 Ensure Program Office software personnel participate in peer reviews and other contractor reviews of software test products and results, including software build reviews |
Phase D1 |
|
Ensure Program Office software personnel receive peer review notifications and access to test products in time to perform a complete review and provide comments. Ensure comments on peer reviewed products, a summary of mission impacting issues and risks, and review metrics are captured and kept up to date. Ensure substantive comments are provided to the contractor early enough to correct the detected defects and implement the suggestions for improvement.
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|
| 5.4-6-10 Ensure the contractor's software qualification test environments are adequate for robust qualification testing |
Phase D1 |
|
Review the contractor's software qualification test environments (e.g., testbeds, test facilities, simulators, emulators, test drivers, test tools, databases) to Ensure they are sufficiently representative of the operational environment. Ensure any test facility functionality gaps are addressed and corrected well in advance of the time the resources are needed. Review the qualification test environments to Ensure they have sufficient capability and capacity to verify requirements (e.g., support all of the coverage specified in the Software Development Standard, and support Test-Like-You-Operate and end-to-end scenarios across the various levels including software items, hardware items, subsystems, elements, segments, and the system). Ensure any capability and capacity gaps are addressed and corrected well in advance of the time the resources are needed. Ensure the qualification test environments are under configuration management. Ensure the test and simulation resources available are sufficient to maintain the environments throughout the qualification testing. Ensure the testing tools support the testing methods to be used. Ensure the testing personnel are trained in advance on the methods and tools.
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|
| 5.4-6-11 Ensure the contractors' software integration and test environments for performing integration are adequate for robust integration and testing |
Phase D1 |
|
Review the contractors' software integration and test environments (e.g., testbeds, test facilities, simulators, emulators, test drivers, test tools, databases) to ensure they are sufficiently representative of the needs for integration and testing at the various levels from software unit integration, software item-to-software item, software item-to-hardware item, and higher levels of integration and testing . Ensure any test facility functionality gaps are addressed and corrected well in advance of the time the resources are needed. Review the integration and test environment to ensure it has sufficient capability and capacity to verify the architecture, design, and interfaces. Ensure any capability and capacity gaps are addressed and corrected well in advance of the time the resources are needed. Ensure early testing environments support, e.g., Test-Like-You-Operate and end-to-end scenarios through larger sets of integrated entities, using simulators when necessary. Ensure the integration and test environments are under configuration management. Ensure the test and integration environment resources available are sufficient to maintain the environments throughout the integration and testing. Ensure the testing tools support the testing methods to be used. Ensure the testing personnel are trained in advance on the methods and tools.
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|
| 5.4-6-12 Ensure the contractor's test databases are adequate for robust qualification testing |
Phase D1 |
|
Ensure nominal and off-nominal values are included over the range and edges of possible values. Ensure the test cases exercise a range of nominal and off-nominal data. Ensure the test databases use invalid data as well as data rates that exceed the capacity of the system. Ensure malformed data streams with broken or missing headers, packets, etc. are used to stress the software. Ensure data is missing for the software to detect the gaps in data.
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|
| 5.4-6-13 Ensure Program Office software personnel assess that contractors are following robust processes for their software qualification testing |
Phase D1 |
|
Ensure Program Office software personnel witness at least portions of software item qualification testing dry runs and formal testing. Ensure the other stakeholders (e.g., customers, quality assurance, users, and readiness assessors) are invited to software item qualification testing dry runs and formal software item qualification testing . Ensure the contractors are following their test execution processes and the test procedures. Ensure data analysis and requirements Qualification decisions are supported by the test data results.
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|
| 5.4-6-14 Ensure the status of the software requirements qualification is maintained, correct and complete and kept up to date |
Phase D1 |
|
Ensure the contractor maintains status of the software requirements qualification and that the status is correct, complete and up to date. Ensure the qualification status record is updated as software qualification tests are completed. Ensure the software qualification test results that support the qualification status are correct and based on results from tests, demonstrations, inspections, analyses, or a combination thereof performed on the item(s) under test in the appropriate environment and test level per the software item qualification plan. As the development proceeds, analyze the test results of requirements scheduled for qualification in lower level tests and ensure there is adequate testing, analysis, demonstration, inspection, or a combination thereof to prove that the requirement is verified prior to roll up to the higher level test.
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|
| 5.4-6-15 Ensure analyses of discrepancy report data are performed |
Phase D1 |
|
Ensure complete discrepancy report (DR) data is obtained from the contractor. Perform independent analysis of DRs to determine process or systemic issues, root causes, and injection phases. Perform software reliability modeling using DR data. Analyze DR open and closure rates to estimate date of software testing completion (e.g., using Rayleigh curves). Perform system dynamic modeling of the test and fix cycle to determine time to complete testing. Ensure anomalies observed during any period of testing that are counter to assumptions and predictions are specially flagged and analyzed for their impact on the system design.
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|
| 5.4-6-16 Ensure the contractor team's analyses of discrepancy reports for anomalies found in software integration and qualification testing are adequate |
Phase D1 |
|
Review the contractor team's discrepancy report (DR) analyses to ensure trending, reliability, time remaining to test are done adequately. Review the contractor team's DR analyses to ensure root cause and injection phase are determined and appropriate process improvements are established to reduce root causes due to systemic or process issues. Perform independent analysis of discrepancy reports to determine process or systemic issues, root cause, and injection phase. Perform software reliability modeling using discrepancy reports. Analyze DR open and closure rates to estimate date of software testing completion, (e.g., using Rayleigh curves).
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|
| 5.4-6-17 Ensure Program Office software personnel assess readiness for existing software qualification test and verification of software requirements |
Phase D1 |
|
Ensure the materials for the test exit review and the test results are available to Program Office software personnel sufficiently before the TER to allow for adequate review and analysis. Ensure all liens, waivers, deviations, discrepancy reports, and work around are documented. Ensure no show-stopper discrepancies (Severity 1) exist and that all known discrepancies and work around are known to the user and operator community. Ensure test results are available to Program Office software personnel sufficiently ahead of the TER to allow for review. Ensure Program Office software personnel analyze the test results and provide comments prior to the test closure, nominally 30 calendar days after test execution.
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|
| 5.4-6-18 Ensure Program Office Software personnel assess readiness for Software Qualification tests |
Phase D1 |
|
Ensure the materials for the test readiness review and the qualification tests are available to Program Office software personnel sufficiently before the TRR to allow for adequate review and analysis. Ensure Program Office software personnel analyze the test products and provide comments prior to the qualification testing. Ensure Program Office software personnel analyze the test dry run materials and provide comments, nominally 30 calendar days before test execution.
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|
| 5.4-6-19 Ensure independent models and simulations of critical ground software are executed using actual measurements of code execution characteristics to analyze whether the performance requirements will be met |
Phase D1 |
|
Ensure the independent models and simulations of critical ground software (developed to model the software architecture and design) are updated to reflect the as-built software and executed using actual measurements of code execution characteristics. Ensure the models/simulations continue to demonstrate the performance requirements can be met by the actual software executing on the target hardware.
|
NA
|
NA
|
TOR-2004(3909)-3537, SW Development Standard for Space Systems; TOR-2011(8591)-20, Space Segment SW Readiness Assessment; ATR-2011(8404)-3 An Increase in SW Testing Robustness: Enhancing the SW Development Standard for Space Systems
|