IARPA posts RFI for credibility assessment datasets

By Loren Blinde February 28, 2018

0 Comments

On February 27, the Intelligence Advanced Research Projects Activity posted a request for information on credibility assessment datasets (CAD). Responses are due no later than 5 p.m., Eastern Standard Time, on May 31.

The Intelligence Advanced Research Projects Activity (IARPA) is seeking information on data sources for evaluation of credibility assessment techniques and technologies. This request for information (RFI) is issued solely for information gathering and planning purposes; this RFI does not constitute a formal solicitation for proposals. The following sections of this announcement contain details of the scope of technical efforts of interest, along with instructions for the submission of responses.

Background & Scope

The assessment of a source’s credibility, or their information, is a core challenge for a broad range of intelligence, defense, homeland security, and law enforcement applications. There are a variety of techniques and technologies that are purported to be useful for establishing the credibility of a source, such as an individual, and/or their information, but it is difficult to evaluate and compare the accuracy, sensitivity, specificity, and practical utility of such solutions. Effective evaluations and comparisons of different credibility assessment methods are hampered by a diversity of testing and demonstration environments, the use of experimental methods that lack construct validity, and other testing conditions that impact the relevance to national security applications, such as poor ecological validity, low stakes or low consequence conditions, and unrepresentative participant cohorts. These concerns are further exacerbated by the potential financial gain for developers of such approaches, which can introduce biased methods and potentially limited reporting about the performance of such systems. Together, these limitations highlight the need for an independent body to motivate and evaluate different credibility assessment techniques and technologies.

To enable such an open and fair evaluation, IARPA seeks data sources that could be compiled and used to independently evaluate and verify the state of the art in credibility assessment techniques and technologies. For this RFI, credibility refers to the veracity of information and/or the person or source providing that information. Assessments of credibility are often complex and may involve an evaluation of many factors of a source and/or the information, to include, but not limited to, veracity, trustworthiness, motivation, considerations about what may be withheld or concealed, and if credibility of information and/or a source has evolved across environments or over time, such as through transmission (e.g. second or third hand information). This RFI seeks data sources that include an evaluation of one, some, or all of these factors, as well as others that are not listed here, as long as they can be theoretically or empirically linked to an evaluation of source and/or information credibility.

Of critical interest are datasets that can be made publically accessible, or accessible via a data use agreement, and which have objective ground truth about the credibility of the source and/or the information. Data sources of interest include, but are not limited to:

Experimental data from real or mock/simulated scenarios;
Data from real world events;
Information that is exchanged in physical or virtual spaces, either face to face or via other channels (e.g. exclusively text based);
Information that is delivered directly by the individual whose credibility is being evaluated, or through a proxy or intermediate (e.g. messenger, translator, avatar);
The credibility of the information and/or source may evolve over time, through transmission, and/or across environments/contexts;
Video, audio, behavioral, text, physiological, or other data types.