By 
From IC Insider Epsilon C5I
Testing is a critical aspect of the software development process and can take up as much as 60% of software development resources. Testing ensures programs are running quickly and effectively and are prepared to handle different user scenarios and applications. Within the DoD, testing is extremely crucial – software must be reliable, functional, and able to perform in all situations. A latent defect appearing post-test completion could be significant, both in safety and mission success, and the associated elements of cost and schedule. According to this year’s Capgemini World Quality Report, more than 60% of software businesses are investing in test automation, but 42% lack test data and environments that are available at the right time in development. Only 59% of respondents are confident that they are utilizing the best automation strategy.
Manual testing of software applications is time-consuming, prone to errors, and inefficient. It requires a significant portion of the project’s budget and robust manual test programs may not be feasible for complex programs. Automated testing solutions offer a more efficient approach to test planning and conduct. Those which can be leveraged across multiple programs have a higher return on investment (ROI) and increased efficiency in performance.
Our Solution – TAF
Epsilon Systems Solutions, C5I (Epsilon C5I) designed The Automation Framework (TAF) to provide test automation focused on verification and orchestration stages of the DevOps pipeline. TAF ensures realistic and repeatable data can be integrated into a continuous development pipeline to repeatedly test systems and their capabilities at the volume, variety, and velocity of real-world data. Using TAF, a development team can be confident a system will perform even when presented with uncertainty associated with unexpected scenarios and sub element failures. TAF uses a plugin-based framework to build and execute automated tests based on Keywords which can be reused across Test Cases, Test Suites, and different TAF plugins.
How TAF Works
As depicted in Figure 1 above, TAF consists of a scenario engine and an extensible library of plug-and-play data adapters and simulators that are configured to interface with new and existing products. These adapters are configured to precisely replicate the format and transmission of real data in a way that is transparent to the system under test (SUT). In this manner, TAF enables real-world testing of intelligence systems using injection of repeatable test data and configurable, automation-friendly test scenarios.
TAF provides a dashboard and visualizations which are automatically captured in data logs, screenshots, test reports, trend analyses, and metrics, suitable for submission to DoD test authorities. Figure 2 below shows a Validation and Verification (V&V) snapshot, identifying tests that passed or failed.
Another featured artifact is the Test Log, which includes information on test execution such as the Test Suite’s settings, Test Cases, screenshots, Keyword logging, debug, and trace output. The HTML version of the test log can be expanded and compressed by Test Case or Keyword to expand or reduce the amount of information displayed. Figure 3 displays an example of a test log header and breaks down the results.
Implementation
TAF is presently utilized by the following U.S. Navy programs and projects:
| Agile Core Services (ACS) | Distributed Common Ground System Navy Inc 2 (DGS-N Inc 2) |
| Automated Digital Network System (ADNS) | Maritime Tactical Command and Control (MTC2) |
| Afloat Integrated Learning Environment (AILE) | Overmatch Software Armory (OSA) |
| Consolidated Afloat Networks Enterprise (CANES) | Ship’s Signal Exploitation Equipment (SSEE) |
| Collaborative Software Armory (CSA) | Tactical Shore Platform (TSP) |
TAF is currently used on these programs during the planning, execution and report phases of the following government test events: Developmental Testing (DT), Operational Testing (OT), Integration Testing, Regression Testing, Factory Acceptance Testing (FAT), and Systems Operations Verification Testing (SOVT). Additionally, new capabilities in development include a System of Systems-Design of Experiments (SoS-DOE) capability, Cyber TAF to automate much of the Risk Management Framework (RMF), and Test Automation as a Service (TAaaS), which will provide the TAF toolset as a service configured within a new or existing DevOps pipeline.
Impact
TAF helps keep the test team from executing rudimentary, repetitive testing and refocuses their capabilities in areas where manual testers excel such as exploratory testing, failure reproduction, and advanced troubleshooting.
Per test results provided by DCGS-N Inc. 2, TAF achieved the following:
- 94% reduction in test hours
- Greater than 50% reduction in testing manpower required
- Greater than 80% increase in error detection
- 33% reduction in software build time to delivery
In the Capgemini survey previously cited, only 37% of businesses stated that they received ROI from their automation efforts. TAF, however, consistently improves testing performance while reducing cost and schedule, and our metrics prove a solid ROI for our customers.
Expediting Development with TAF
As the move to automated testing accelerates, Epsilon C5I anticipates TAF will be an important differentiator for the Navy and other DoD programs. Our team is improving and developing the framework and continuing to increase its relevance and applicability, and Epsilon C5I is eager to partner with like-minded technology companies. With Project Overmatch moving the Navy toward greater integration and network commonality, we expect TAF will become critical in creating test environments that are as realistic as possible, while incorporating test capabilities tailored to formal test event conditions. The time and cost saved by TAF will enable new capabilities to be deployed to the Fleet in a fraction of the time that it currently takes.
For more information on TAF and partnering, contact:
Kevin Visalli, Director of Software Products, Epsilon C5I: kvisalli@epsilonsystems.com
Steve Boraz, Vice President and General Manager, Epsilon C5I: sboraz@epsilonsystems.com
About Epsilon Systems Solutions, Inc.
Epsilon Systems is a 100% employee-owned company founded in 1998 and headquartered in San Diego, California. With a presence in more than 20 locations, Epsilon Systems’ customers include the Department of Defense, Department of Energy, Department of Homeland Security and non-profit and commercial customers. Epsilon Systems provides total life cycle support to defense systems, such as basic research, concept development, system architecture, requirements definition and analysis, software development, integration and test, operational support, training, and maintenance and logistics.
About IC Insiders
IC Insiders is a special sponsored feature that provides deep-dive analysis, interviews with IC leaders, perspective from industry experts, and more. Learn how your company can become an IC Insider.
