Assessing Risks and Impacts of AI (ARIA): ARIA 0.1: Pilot Evaluation Report

Details

• Alternative Title:
  NIST Trustworthy and Responsible AI 700-2
• Creators:
  Amironesei, Razvan ; Godil, Afzal ; Greenberg, Craig ; Greene, Kristen ; Hall, Patrick ; Jensen, Theodore ; Fiscus, Jonathan ; Schulman,Noah
• Corporate Creators:
  United States. Department of Commerce. National Institute of Standards and Technology
• Subject/TRT Terms:
  • Artificial intelligence
  • Risk assessment
  • Measurement
  • Testing
  • Human factors
  • Technology assessment
  • Trustworthiness
  • Large language models
  • Generative AI
  • Sociotechnical
• Publication/ Report Number:
  NIST AI 700-2
• DOI:
  https://doi.org/10.6028/NIST.AI.700-2
• Resource Type:
  Tech Report
• Geographical Coverage:
  United States
• Corporate Publisher:
  United States. Department of Commerce. National Institute of Standards and Technology
• Abstract:
  This document describes the procedure used for a pilot of NIST's Assessing Risks and Impacts of AI (ARIA) evaluation: ARIA 0.1. Five organizations participated, submitting a total of 7 AI applications to be evaluated. In this document, we first describe the design of the three evaluation scenarios (TV Spoilers, Meal Planner, Pathfinder) and the three testing levels (model testing, red teaming, field testing). We then discuss the methods used for assessment via dialogue annotation and tester questionnaires. Finally, we describe our approach to measuring validity of AI applications using measurement trees.
• Format:
  PDF
• Download URL:
  https://rosap.ntl.bts.gov/view/dot/93267/dot_93267_DS1.pdf
• File Type:
  [PDF - 7.67 MB]
• Collection(s):
  Artificial Intelligence Deployment Center of Excellence (AI COE)
• Main Document Checksum:
  urn:sha-512:9ce6680e3123505862d3f2b966a590f947599fe20e31ace9a61a2ded31f8b35173c6e73d83bad57524ad2e639bae37b247a65a18dc048364662520e92f7f915c