CASE CHALLENGE
This IARPA-sponsored challenge asked teams to develop tools and methodology to evaluate credibility assessment techniques. Our solution is an adaptation of a study design from the field of epidemiology. To compare the effectiveness of credibility assessment techniques, we propose a novel framework for an organization-level block-randomized trial, where credibility assessment methodologies are evaluated by their performance in real-world organizations. Using this framework, meaningful outcome measures can be adapted to the needs of the evaluator and the nature of the tool under evaluation. We also propose several example metrics that would be applicable to current CA technology.
Table 1: Example Studies
Figure 1: Sample Factorial Study
Table 1: Example Studies