APPENDIX E - Formal Reviews




This Appendix describes activities that shall be performed at formal reviews and output products from these reviews.



This Appendix provides information on the review phase outputs products, activities, and things to look for.




No matter what type of development process a Suppler uses (mil or international standard); there are basic processes that shall be performed in order to have stable and successful software engineering development process.  The supplier could be using US 12207, which provides more flexibility due to the types of life cycle processes (Primary, Supporting, and Organizational).  Additionally, some Supplier’s still use J-STD-016, which is the “demilitarized” version of Mil-STD-498, or using DoD-STD-2167A as a guide.


All development processes specify activities that shall occur before proceeded to the next phase. So when System Integration & Test is performed, the software works with the system in the environment that is was designed for.


Types of Formal Reviews


The following are the types of formal review activity that is normally performed in a software and hardware system.

System Requirements Review

The System Requirements Review (SRR) is conducted after the systems requirements analysis is performed.  This review provides insight into the Supplier’s direction, progress, and status on system configuration. The System Requirements Specification (SRS) is an output of the SRR.


Typical Outputs:

Preliminary System Specification.


Systems Design Review

The System Design Review (SDR) is a review of the Supplier’s overall systems requirements in order to establish the functional baseline documented by the system specification.  Upon successful completion of the SDR, the Functional Baseline is established and the development of hardware and software can start.


Typical Outputs:

a.      Functional Baseline;

­ System Specification.

b.    System Segment Design Document;

c.    Preliminary Software Requirements Specification;

d.     Preliminary Interface Requirements Specification;

e.      Software Development Plan.


Software Specification Review

The Software Specification Review (SSR) is conducted prior to Preliminary Design. This phase focuses on the software requirement analysis.  Upon successful completion of the SSR, the allocated baseline is established and the Preliminary and Detailed Design phase can start.


Typical Outputs:

Allocated baseline;

­        Software Requirement Specification;

­        Interface Requirement Specification.


Things to watch for:

a.     Is the supplier ready?  What is the overall status of:

·      Architectural Design;

·      Requirements allocation;

·      Requirements analysis.

b.    Stability of requirements?

c.     Is all the appropriate documentation complete?  Is the required documentation complete?

d.    Has the review package been provided in advance?

e.     Have all the risk areas and known problems been reported to the Program Office?


Preliminary Design Review

The Preliminary Design is the process for analyzing design alternatives and defining the architecture, components, and interfaces.  The Preliminary Design Review (PDR) is conducted to determine if the Supplier is ready to process to the next phase – Detail Design. 


The PDR may address many CSCI and it may also be held incrementally.  The review will assess the adequacy of the Software Design Document (SDD), Interface Design Document (IDD) and Software Test Plan (STP).


Typical Outputs:

a.     Software Design Document (Preliminary Design);

b.    Interface Design Document (Preliminary);

c.     Software Test Plan;

d.    Developmental Configuration.


Critical Design Review

The Critical Design is the process or refining and expanding the preliminary design of a system or component to the extent that the design is sufficiently complete to be implemented.  The Critical Design Review (CDR) is conducted prior to the Coding & CSU Testing Phase.


The PDR may address many CSCI and it may also be held incrementally.  The review will assess the adequacy of the Software Test Description (STD). The review will also assess the updated Software Design Document (SDD) and Interface Design Document (IDD).


Typical Outputs:

a.     Software Design Document (Detail Design);

b.    Interface Design Document;

c.     Software Test Description (Cases);

d.    Developmental Configuration.


Things to watch for during Design:

a.     Is the supplier ready for the review?

b.    Supplier’s overall performance in:

·      Defining CSCI architecture;

·      Establishing design of each CSU;

·      Documentation (current & consistent);

·      Preparation of SDD, IDD, STP, STD.

c.     Supplier Peer Reviews process – sample peer reviews

d.    Is all the appropriate documentation complete and has the review packages been provided in advance?

e.     Have all design, requirements, deficiencies and other possible risk been reported to the Program Office?


Test Readiness Review

The Coding & CSU Test phase translates the detail design into programming language and test the software to eliminate errors that may exist in the units as the are coded.


Typical Outputs:

a.     Source code;

b.    Source code listings;

c.     Developmental configuration update.


The CSC Integration and Test phase combines the software units and components that have been independently tested into the total software product and to demonstrate this combination meets the system design.


Typical Outputs:

Software Test Description (Procedures).



Once the Coding & CSU Testing and CSC Integration & Testing phase is complete, the Test Readiness Review (TRR) can be conducted.  The TRR shall be successful in order for the supplier to proceed to the formal CSCI Testing phase.


Things to watch for during the TRR:


The TRR is usually conducted by the Government to determine if the Supplier is ready to begin formal CSCI testing.

a.     The following document should be reviewed at this time:

·      CSU test results;

·      Integration test results;

·      Updated Software Test Descriptions;

·      Updated source code and design documents.

b.    Does the Supplier have a technical understanding on the:

·      Informal test results;

·      Test tool selections;

·      Support documentation.


CSCI Testing

A formal software test is required to demonstrate compliance with the contract requirements and this is the first step in determining readiness for software integration into the system.  A successful CSCI Test is a prerequisite for software acceptance.


There are stages of formal software Test.  Note: some of these tests may be combined together.

a.     Formal Qualification Test (FQT);

b.    CSCI/Subsystem integration test;

c.     System integration & test;

d.    Field level system test.


Typical Outputs:

a.     Operation and support documents;

b.    Updated source code;

c.     Product Baseline (Software Product Specification);

d.    Software Test Reports.



Things to watch for during CSCI Testing:

a.     Understand the test requirements and Suppliers interpretation of test requirements;

b.    Are requirements complete and traceable:

·      SSS to SRS;

·      IRS to STP to STD.

c.     Review supplier test planning documents (SDP, STP, etc);

d.    Review test preparations, test cases, procedures;

e.     Review the supplier’s process for recording and reporting software test errors;

f.      Review deficiency reports;

g.    Monitor deficiency correction including impacts on CSCIs;

h.    Monitor re-testing.

i.      Monitor supplier updating of test and design documents as changes occur;

j.      Review final documentation that support final delivery.


Physical Configuration Audit (PCA)


Functional Configuration Audit (FCA)

The Software Product Specification is the basis for the Software Product Baseline that is established at the PCA. 



The PCA may follow or be conducted concurrently with the FCA for software only development.  Most of the time, the PCA occurs after the software is released for integration and testing with the system following an FCA


During the software FCA, the Government verifies the CSCIs perform IAW the requirements and interface specifications by analyzing test results.  The PCA is the formal examination technique of the as-built software product against its design


Formal Review Summary


Depending on the Supplier’s developmental processes, what is required from the Supplier by contract, and your level of involvement as describes in the MOA, some of the activities in this appendix may not be performed or some activities may be combined.


DCMA Software Acquisition Management personnel will need to fully understand their requirements for all these formal reviews.