| 2.2.5-3-1 Ensure the Test and Evaluation Master Plan is complete, updated, coordinated and approved prior to major milestones and after program changes |
Phase A |
Phase B |
Phase C |
Phase D1 |
|
Ensure the Test and Evaluation Master Plan (TEMP) includes applicable test scenarios, appropriate data collection, and performance evaluation over the life cycle of the system. Ensure links are available to reliability growth and planning, scientific test and analysis techniques, allocation of reliability among components, anticipated development and test problem areas, resolution of past deficiencies as applicable, cyber strategy and resources, and cyber test events. Ensure the T&E strategy ties the evaluation framework and knowledge to decisions, requirements, etc. Ensure the T&E management and overall T&E activities are identified for the government and the contractor. Ensure program changes are effectively driven down to the TEMP and incorporated. Ensure changes are coordinated with enough margin to not affect OT&E. Ensure appropriate level of detail for each milestone. Ensure the TEMP accurately reflects the current DBRD, STAR, Air Force Concepts or equivalent, and AoA. Ensure shortfalls in funding and resources are identified in the TEMP.
|
AFMAN 63-119 A10.1; AFMAN 63-119 A10.1.1; AFMAN 63-119 A10.1.2; AFMAN 63-119 A10.2; AFMAN 63-119 A10.7.1; AFMAN 63-119 A10.7.2; AFMAN 63-119 A10.7.3; AFMAN 63-119 A10.7.4; AFMAN 63-119 A10.7.5; AFMAN 63-119 A10.7.6; AFMAN 63-119 A10.7.6.1; AFMAN 63-119 A10.9; AFMAN 63-119 A11.1.8; AFMAN 63-119 A13.2.8; AFMAN 63-119 A14.3; AFMAN 63-119 A32.1; AFMAN 63-119 A32.2.1
|
NA
|
DHS Acquisition Instruction/Guidebook #102-01-001, 1 Oct 2011; AFMAN 63-119 or equivalent
|
| 2.2.5-3-2 Ensure operational performance requirements are defined and captured as appropriate in the program's requirements documents, system threat documents, and Analysis of Alternatives |
Phase 0 |
Phase A |
|
Ensure the operational performance requirements are traceable to the ORD and the Key Performance Parameters listed in the ORD. Design thresholds should represent the level of system performance acceptable to the user to successfully provide desired capability. Thresholds, against which each of the effectiveness and suitability parameters will be measured, are normally quantitative. Ensure the TEMP accurately reflects the current requirements, STAR or equivalent, Air Force Concepts or equivalent, and AoA.
|
AFMAN 63-119 A10.3
|
NA
|
DHS Acquisition Instruction/Guidebook #102-01-001, 1 Oct 2011; AFMAN 63-119 or equivalent
|
| 2.2.5-3-3 Ensure the master plan captures strategy, an integrated schedule including sequencing of test activities, resources, requirement drivers and evaluation criteria |
Phase A |
Phase B |
Phase C |
|
Ensure an executable integrated time sequencing of test and evaluation (T&E) phases and events is detailed within the integrated master schedule over the acquisition life cycle, including incremental deliveries. Ensure test planning is of acceptable risk and credible schedule with reasonable margins. Ensure funding is in place for all testing, and that the cost capability analysis covers T&E. Ensure integrated test planning starts as early as practical to make T&E schedules more efficient and eliminate duplication of effort. Ensure modeling and simulation (M&S) assets are verified, validated and included in the master plan, and reflected in requirements. T&E events should be traceable to applicable test requirements (Key Performance Parameters, Measures of Effectiveness, and Critical Operational Issues). Ensure operational acceptance reviews and fielding decisions are supported by an operational test and evaluation, scheduled as appropriate between development stages. Incrementally deployed releases address specific capabilities and testable performance requirements. Each release must undergo dedicated OT&E. Software builds or increments that are not deployed individually must still support full deployment operational testing.
|
AFMAN 63-119 A10.4; AFMAN 63-119 A10.4.1; AFMAN 63-119 A10.4.4; AFMAN 63-119 A10.4.5; AFMAN 63-119 A10.4.8; AFMAN 63-119 A11.1; AFMAN 63-119 A13.2.5; AFMAN 63-119 A15.8.1; AFMAN 63-119 A22.3.2; AFMAN 63-119 A22.3.6
|
NA
|
DHS Acquisition Instruction/Guidebook #102-01-001, 1 Oct 2011; Mission Assurance Guide, TOR-2007(8546)-6018, Rev B; SDR exit criteria per TOR-2004(3909)-3360, Systems Engineer's Major Reviews for National Security Space System Programs; AFMAN 63-119 or equivalent
|
| 2.2.5-3-4 Ensure required test and evaluation resources are defined, funded, and integrated into the Integrated Master Schedule (IMS) |
Phase A |
Phase B |
Phase C |
Phase D1 |
|
Ensure resource descriptions include required test articles needed for each event: facilities, test equipment, special equipment and instrument costs; threat representation costs, items under test (including interactive assets), other systems and external interfaces including interoperability, models, simulations, test software, procedures, support tools, and personnel/ manpower needs. Ensure test articles have achieved stabilized performance. Ensure all test articles are assessed for any potential impact on validity of dedicated operational tests, including system performance and test execution.
|
AFMAN 63-119 A10.4.3; AFMAN 63-119 A20.1; AFMAN 63-119 A20.1.1; AFMAN 63-119 A20.1.2; AFMAN 63-119 A20.1.3; AFMAN 63-119 A20.2; AFMAN 63-119 A20.3; AFMAN 63-119 A20.3.1; AFMAN 63-119 A20.3.2; AFMAN 63-119 A20.3.3; AFMAN 63-119 A22.4; AFMAN 63-119 A32.1.1; AFMAN 63-119 A32.2; AFMAN 63-119 A32.3
|
NA
|
DHS Acquisition Instruction/Guidebook #102-01-001, 1 Oct 2011; Systems Engineering Requirements and Products TOR-2005(8583)-3, Rev. B; AFMAN 63-119 or equivalent
|
| 2.2.5-3-5 Ensure all organizations participating in the test and evaluation program are identified |
Phase A |
Phase B |
Phase C |
Phase D1 |
|
Ensure organizations include the program manager, the component, the independent evaluator or operational test authority or any organization conducting actual testing, including contractors and operational units. Ensure operational test plans are coordinated as required with other services, and integrated.
|
AFMAN 63-119 A22.3.7
|
NA
|
DHS Acquisition Instruction/Guidebook #102-01-001, 1 Oct 2011; AFMAN 63-119 or equivalent
|
| 2.2.5-3-6 Ensure Technical Performance Measures (TPM) are verifiable |
Phase A |
Phase B |
Phase C |
|
Ensure the System of System (SoS) TPMs and Space Segment TPMs can be sufficiently verified and/or validated using objective techniques such as tests, analysis etc. This activity ensures that TPMs are identified, that TPMs are appropriate for both nominal and off-nominal conditions. Ensure the technical performance measurements are provided for required review, audits, and risk assessment/management at appropriate milestones.
|
NA
|
System TPMs
|
Mission Assurance Guide, TOR-2007(8546)-6018, Rev B; DHS Acquisition Instruction/Guidebook #102-01-001, 1 Oct 2011
|
| 2.2.5-3-7 Ensure assessment of the contractors development, integration and certification strategy |
Phase A |
Phase B |
Phase C |
Phase D1 |
|
Ensure all DT&E or integrated DT/OT conducted by both the contractors and the government is included, and clearly distinguished. Ensure all planned DT&E is discussed per acquisition phase. Each segment of testing should include configuration descriptions (functional capability of system configuration); DT&E objectives; DT&E events; and any test limitations that may affect results or conclusions related to the critical technical parameters. Ensure acceptable test processes are used.
|
AFMAN 63-119 A10.5.2
|
NA
|
Mission Assurance Guide, TOR-2007(8546)-6018, Rev B; DHS Acquisition Instruction/Guidebook #102-01-001, 1 Oct 2011; AFMAN 63-119 or equivalent
|
| 2.2.5-3-8 Ensure Developmental and Operational Test and Evaluation Certification Requirements are defined |
Phase B |
Phase C |
|
Ensure the overall goals and objectives of the developmental and operational test and evaluation programs are described. Ensure specific DT&E exit criteria, system acceptance tests, qualification tests, and test readiness reviews are specified. Ensure the complete test program has the potential to meet operational requirements. Ensure developmental and operational test plans are structured so that operational tests can capture and apply developmental test data to reduce operational test timelines and requirements. Ensure all aspects of test and evaluation are integrated in overall test design. Ensure rationale and provisions are made for operational testing beyond OT&E into Final Operational Test and Evaluation (FOT&E) or incremental phases.
|
AFMAN 63-119 A10.5; AFMAN 63-119 A10.6; AFMAN 63-119 A11.1.2; AFMAN 63-119 A11.1.3; AFMAN 63-119 A11.1.5; AFMAN 63-119 A14.3.1
|
NA
|
Mission Assurance Guide, TOR-2007(8546)-6018, Rev B; DHS Acquisition Instruction/Guidebook #102-01-001, 1 Oct 2011; AFMAN 63-119 or equivalent
|
| 2.2.5-3-9 Ensure the contractor defined design verification strategy complies with System specification requirements |
Phase A |
Phase B |
Phase C |
Phase D1 |
|
Ensure verification method of analysis, test, demonstration, inspection and similarity is detailed in the TEMP. Ensure that preliminary integration and test plan, from unit to system level, is of acceptable risk and credible schedule with reasonable margins. Ensure risks to schedule are identified and have mitigation plans.
|
SFWC 2.1.4.1, SFWC 2.1.4.1.1
|
Integration and Test Plans; System/Segment Specification (SSS); Test and Evaluation Master Plan (TEMP)
|
NA
|
| 2.2.5-3-10 Ensure process in place to capture deficiencies and track to closure |
Phase B |
Phase C |
Phase D1 |
|
Ensure the Test and Evaluation Master Plan (TEMP) describes the program's entire discrepancy report (DR) process and plan which should include formulation and execution of a DR review board, prioritization of DRs, release and review of all DR data to the Government, analyses of DRs in relation to overall system performance and acceptance, and closure plans agreed upon by Government and contractor. Ensure the contractor-operated DR process augments the Government process. Ensure the contractor's deficiency resolution process is compatible with the government process. Ensure critical issues, Measures of Suitability (MOSs), and Measures of Effectiveness (MOEs) are addressed in operational tests. Ensure all critical issues are tracked and resolved before entry into operational testing.
|
AFMAN 63-119 A10.5.1; AFMAN 63-119 A13.5; AFMAN 63-119 A13.5.1; AFMAN 63-119 A14.4; AFMAN 63-119 A19.1; AFMAN 63-119 A19.2; AFMAN 63-119 A19.4; AFMAN 63-119 A22.3.4
|
NA
|
DHS Acquisition Instruction/Guidebook #102-01-001, 1 Oct 2011; AFMAN 63-119 or equivalent
|
| 2.2.5-3-11 Ensure operational test concepts and plans are complete and approved |
Phase A |
Phase B |
Phase C |
Phase D1 |
|
Ensure the operational test concept is developed and briefed as early as feasible, containing characteristics of operations and support environments as well as test scenarios to be used in operational testing. Ensure rigor and realism are planned in operational tests. Testing realism must include all modes and defense condition environments.
|
AFMAN 63-119 A22.1; AFMAN 63-119 A22.1.1; AFMAN 63-119 A22.2; AFMAN 63-119 A22.3; AFMAN 63-119 A22.3.1
|
NA
|
AFMAN 63-119 or equivalent; Mission Assurance Guide, TOR-2007(8546)-6018, Rev B, Integration, Test, and Evaluation Chapter
|