CTC wins DARPA contract for military AI assurance

Concurrent Technologies Corporation (CTC) has secured a nearly $2 million, two-year contract from the US Defense Advanced Research Projects Agency to assess the behaviour of artificial intelligence in military autonomous systems. The Pennsylvania-based nonprofit research organisation will work with its technology transition affiliate, Enterprise Ventures Corporation (EVC), to develop and apply methods for evaluating drones, robots and related autonomous technologies under representative mission conditions. The competitively awarded programme is CTC’s first contract as a DARPA prime contractor since 2012. The partners have not identified a particular aircraft, ground vehicle or military platform to be modified under the work.

Military autonomy combines sensing, software decisions and physical control, so a system that performs accurately in a controlled trial may respond differently when its surroundings, communications or sensor inputs change. The new research addresses how those differences can be exposed and assessed before an autonomous capability is relied upon in an operational environment. CTC and EVC will examine how to test and validate AI and machine-learning systems across different mission settings. Their commitment is to develop assessment approaches; the award does not itself establish that any existing military robot has achieved a new certification or entered service.

The programme targets a gap in repeatable adversarial testing. A system can be tested against ordinary mission cases while remaining vulnerable to unexpected combinations of conditions or deliberately misleading inputs. For learning-enabled equipment, the relationship between training data, current observations and subsequent decisions may be difficult to verify from conventional pass-or-fail tests alone. Red-team assessment uses structured challenges to expose such weaknesses rather than assuming normal operation represents the full range of possible behaviour. DARPA has funded earlier research into assurance of learning-enabled cyber-physical systems, but that historical programme is separate from the contract awarded to CTC.

An uncrewed aircraft processing camera or radar information during a mission relies on its perception system, which must differentiate relevant objects from clutter, while navigation and control functions act on information that may contain gaps or errors. Varying visibility, unusual object appearances, obstructed sensors or degraded positioning data can expose failures that a standard demonstration route does not encounter. Comparable issues affect ground robots manoeuvring around obstacles and other vehicles. These are examples of the testing challenges facing autonomous systems generally, not a list of specific trials already contracted or completed by the CTC team.

A credible evaluation must also separate a model’s classification performance from the behaviour of the complete machine. Detection accuracy is one measure, but an erroneous classification can propagate through route planning, command logic and actuators before affecting a physical movement. Engineers therefore need to observe both the model’s outputs and the system response within defined operating limits. The award covers methods for assessing and validating autonomous behaviour in representative environments; CTC has not specified the simulation platforms, hardware-in-the-loop rigs, adversarial scenarios or acceptance thresholds it will ultimately use.

The research builds on CTC’s work in AI assurance dating to 2018. In a separate collaboration with the Robotics Research Center at the US Military Academy at West Point, CTC has been developing a machine-learning security assessment capability involving model-stealing techniques. That project examines weaknesses in the protection of AI models and data, including the potential for adversaries to infer information about a model’s architecture or training. The DARPA award addresses broader reliability and security of autonomy in mission conditions. The two lines of research share an interest in measurable assurance without being interchangeable contracts or proof of a single integrated product.

As CTC’s technology transition affiliate, EVC works on moving research and engineering outcomes towards practical applications, including prototype development and manufacturing activity where appropriate. CTC itself conducts applied research, design, development, testing and evaluation. Their joint position on this contract connects assessment research with the practical constraints of deployable equipment, although no production quantity, manufacturing facility allocation or subsequent commercialisation obligation has been identified for the programme. The financial award also should not be confused with procurement of a fleet of drones or robots.

Adversarial testing is particularly difficult when autonomous systems must operate with people or other machines. Testing one vehicle in isolation cannot automatically demonstrate safe behaviour in a larger formation, near friendly aircraft or around changing obstacles. Interfaces between sensors, mission software and human supervision introduce additional conditions that require their own evidence. Where operators retain authority to interrupt or redirect a machine, the response to an intervention is part of the system behaviour requiring evaluation. The current contract does not publish a detailed test matrix, so no claim can yet be made about coverage of those particular operating cases.

Evaluation evidence has to remain traceable to a defined software configuration. Changes to trained models, input-processing software or sensor hardware can alter a system’s response, and test results from an earlier build may not fully describe a later one. This creates a practical need to record configurations and reproduce relevant conditions when assessing AI behaviour. Defence users also have to distinguish resilience against adversarial manipulation from ordinary reliability under harsh environmental conditions. The CTC and EVC work concerns developing assessment methods for those kinds of assurance problems, with the exact deliverables and measures to be determined during the two-year project.

CTC chief executive Chris Hamilton linked the award to the organisation’s existing AI assurance research, while data science executive Dave Saranchak identified predictable behaviour in complex environments as the project’s central engineering challenge. The programme gives DARPA a research route for evaluating learning-enabled autonomy without declaring an operational system safe in advance of the testing. No fielding date, platform acceptance decision or follow-on production award has been disclosed. The next verifiable milestones will be the assessment methods produced and the evidence they generate under the conditions selected for testing.