| 5. |
Oak Ridge MLS Testbed Testing and Test Documentation Requirements
|
 |
5.1 |
Testing Policy |
|
5.7 |
Conduct of the Test |
|
5.2 |
Configuration Management of Testing |
|
5.8 |
Documentation of Test Results |
|
5.3 |
Testing Activities |
|
5.9 |
Analysis of Test Results |
|
5.4 |
Testing Responsibilities |
|
5.10 |
Documentation of Test Conclusions and Recommendations |
|
5.5 |
Test Plan Development |
|
5.11 |
Feedback of Test Results |
|
5.6 |
Test Plan Review |
|
5.12 |
Determination of Next Action |
All security hardware and software are tested for functional completeness and compliance with appropriate security requirements in the simulated DOE environment. Product acceptance testing and covert channel analysis conducted by NSA will not be repeated in the Oak Ridge MLS Testbed. If independent validation and verification (IV&V) testing has been performed by an independent evaluator other than the Oak Ridge MLS Testbed staff, then the available test results will be reviewed for accuracy and completeness. Because performance of the network, which is a combination of numerous COTS products, is a serious concern, all network components will also be performance tested in the Oak Ridge MLS Testbed prior to implementation in NN-50 to determine the amount of performance degradation associated with each security measure. Selected products may also be subjected to penetration testing.
 |
to Sect. 5 menu |
The establishment of configuration management for the testing activities helps ensure that the integrity and continuity of software and hardware testing, as well as the timely development of supporting documentation, is maintained. Configuration management is crucial to the testing activities because the testbed must simulate the sponsor's (NN-50's) environment to ensure successful future integration of a product into that environment. It is possible that deficiencies may prevent the hardware or software product from being readily introduced into the NN-50 environment for a particular phase of operation. Test results are discussed with the vendor providing the product, and an indication of the willingness or ability of the vendor to correct the deficiency is determined. If corrective actions are performed, then the product is scheduled for a retest. A retest will also be scheduled if it is determined that the model to be ordered for implementation in NN-50 differs significantly from the version tested, or if the environment in which the product is to be implemented has been changed. If the vendor is providing proprietary documentation to support the testing of a product, staff associated with the testing of that product will be asked to sign the appropriate nondisclosure statements and all documentation will be treated as sensitive.
|
to Sect. 5 menu |
Activities associated with testing include: (1) identification of the security requirement to be satisfied; (2) identification of proposed product security mechanism; (3) determination of the test objectives (i.e., what must occur, under what conditions, and to what level of success); (4) determination of the test methodology/technique; (5) determination of the test steps and scripts; (6) determination of expected test results; (7) conduct of the test; (8) documentation and analysis of test results; (9) documentation of test conclusions and recommendations, including security issues and performance impact on the network with the security mechanism in place; (10) feedback of test results to appropriate individuals/organizations; and (11) determination of the next action to be taken (e.g., additional testing, corrective action by the vendor, or inclusion in the next phase-specific implementation plan for installation in NN-50).
|
to Sect. 5 menu |
The Lead Technical Analyst works in conjunction with designated project staff to design, implement, and analyze item testing. The System Hardware Administrator coordinates the installation of hardware to be tested. The System Software Administrator loads the software to be tested. The Configuration Manager maintains records indicating the latest versions of hardware, software, and documentation and tracks changes to test plans, configurations, and results. The Project Manager is responsible for delineation and clarification of all project testing activities and deliverables in support of NN-50.
|
to Sect. 5 menu |
Test plans are prepared in the format shown in Table 1. The test plans are numbered sequentially for a given fiscal year for easier accountability. Test plans include the following information.
Table 1. Test plan format.
Test Plan Number:
Security Requirement Number(s):
Security Mechanism Number(s):
Item Name:
Model/Version Number:
Serial Number:
Date of Manufacture:
Vendor Name:
Vendor Address:
Vendor Telephone Number:
Point of Contract:
Test Technical Lead:
Test Team Members:
Test Objectives:
Test Methodology/Technique:
Test Steps and Test Scripts (Attached):
Expected Test Results:
Test Results:
Test Conclusions:
Date of Test Results Review by Configuration Control Board:
Next Action to be Taken:
Expected Date of Completion:
Individual Responsible:
 |
5.5.1 |
Security Requirements and Proposed Security Mechanisms |
|
5.5.2 |
Item(s) to be Tested |
|
5.5.3 |
Test Objectives |
|
5.5.4 |
Test Methodology/Technique |
|
5.5.5 |
Listing of Test Steps and Test Scripts |
|
5.5.6 |
Expected Test Results |
5.5.1 Security Requirements and Proposed Security Mechanisms
The security requirements which the item is supposed to address and the proposed security mechanism (which may be composed of more than one item) are identified on the test plan.
|
to Sect. 5.5 menu |
5.5.2 Item(s) to be Tested
The item(s) to be tested are identified on the test plan. The information includes the item name, model or version number, serial number, and date of manufacture; and the vendor name, address, telephone number, and point of contact.
|
to Sect. 5.5 menu |
5.5.3 Test Objectives
The test objectives are based on the functionality of the item being evaluated. Depending on the items being tested, testing may include both functional testing and penetration testing (to circumvent the intended functions of the mechanism). As a minimum, the following three test objectives are addressed..
- The most important objective is to test and evaluate the effectiveness of the security features and measures implemented against the specified requirements.
- The second objective is to identify and document any vulnerabilities that were detected during testing, in spite of the security measures already implemented.
- The third objective is to provide an initial or updated baseline for future testing.
Generally for security testing, any failure in a security mechanism (i.e., the item does not function as intended when properly installed) is considered as an unsuccessful test.
|
to Sect. 5.5 menu |
5.5.4 Test Methodology/Technique
The test methodology and techniques are based on the functionality of the item being evaluated. Associated vendor product information is considered in the development of testing methods. A combination of measurable test methods is used, depending on the particular feature tested. These test methods include:
- testing for a measurable condition or state;
- demonstration of a predetermined response;
- analysis using engineering judgment or mathematical proofs;
- inspection of physical items such as output (display or hard copy), locks, or labeling of cables;
- simulation of actual events; and
- testing of security monitoring features that generate audit reports, trigger alarms, or send messages to the security staff; and
- evaluation of the response of the product to specific stimuli or a set of events.
|
to Sect. 5.5 menu |
5.5.5 Listing of Test Steps and Test Scripts
To increase the likelihood that like items are tested under the same conditions and procedures, test steps are noted. The steps indicate the order in which test activities are to be performed and the extent of the testing at each step (e.g., number of times test step is to be performed). Test steps/scripts are developed as required, in accordance with the test plan for a particular item. Prior to implementation, the test steps/scripts are peer-reviewed for completeness and accuracy by either another team member of the project or a member of LMES technical staff.
|
to Sect. 5.5 menu |
5.5.6 Expected Test Results
Based on the item functions and security requirements to be met, expected test results are prepared. These expected results are compared after the testing with the documented actual test results.
|
to Sect. 5 menu |
|
to Sect. 5.5 menu |
The test plan is reviewed by the MLS Testbed Configuration Control Board.
|
to Sect. 5 menu |
The Lead Technical Analyst assigns a team member to function as the test technical lead. This individual ensures that the installation of the item is complete and reflects the environment in which the item will be functioning in NN-50. The test is conducted in the order designated in the plan. If the testing is performed in a different order, the change is documented, as is the reason for the change. Issues not tested are noted for later follow-up. Concerns that are evaluated but were not included in the test plan are added to the test results and considered in the analysis.
|
to Sect. 5 menu |
The test results are detailed so that the Test Technical Lead and the Lead Technical Analyst can discuss the results with the rest of the project team and/or with the vendor for the resolution of any problems. If discrepancies are identified between expected and actual functions, the discrepancies are noted. If the need for additional instructions with regard to installation or operation are identified during the testing that could be beneficial in the implementation of the item in NN-50, these are also noted. Copies of the test documentation are maintained in the Testbed Notebook by the Configuration Manager. An index of test results is prepared such that tests can be located based on the date, item, security requirement, and test technical lead for easy reference.
|
to Sect. 5 menu |
Test results are analyzed in consideration of the NN-50 environment and the next implementation phase to be initiated. Considerations include the security requirements and security mechanisms which are addressed by the item and the item's impact on the network's performance and on the users.
|
to Sect. 5 menu |
Recommendations are made to our sponsor regarding the potential use of an item. If the sponsor decides to proceed with this recommendation, documentation is prepared for the NN-50 Configuration Control Board for review and implementation.
|
to Sect. 5 menu |
Depending on the item being reviewed, the test results are made available to the vendor for information or action. Copies of the test results are also made available to the MISSI product leader and the sponsor. Because of the sensitivity of the test reports, distribution of the results is limited to those who have a need to know. Based on past experience in the evaluations of products, and those of other organizations who evaluate products, every effort is made to ensure that products do not inadvertently receive negative publicity as a result of Oak Ridge MLS Testbed activities.
|
to Sect. 5 menu |
Depending on the test results for a given item, additional testing may be conducted to verify or clarify the test results, the vendor may be consulted regarding possible corrective actions, or the item may be considered for implementation in the next phase of NN-50 operation. If the vendor is providing technical support for the correction of item malfunctions, then additional tests will be scheduled at a later date.
If follow-up test activities are required, this requirement is also documented in the Testbed Notebook, along with an indication of the expected date of completion and the individuals responsible for the follow-up.
 |
for Article |
 |
Section |
|
to Conference Proceedings Page |