Artificial Intelligence (AI) for Integrated Safety Assessment Model (ISAM)

Details

• Creators:
  Bhanpato, Jirat ; Chandra, Chetan ; Jing, Xiao ; Sawant, Kshitij ; Oderinde, Timilehin ; Felix, Barbara Sampaio ; Payan, Alexia P. ; Kirby, Michelle R. ; Mavris, Dimitri N.
• Corporate Creators:
  Georgia Institute of Technology. Aerospace Systems Design Laboratory
• Corporate Contributors:
  United States. Department of Transportation. Federal Aviation Administration. Office of Accident Investigation and Prevention
• Subject/TRT Terms:
  • Aviation safety
  • Regulations
  • Graphs
  • Artificial intelligence
  • Machine learning
  • Document markup languages
  • Evaluation and assessment
  • Integrated Safety Assessment Model (ISAM)
  • Knowledge Graph
  • Large Language Model
• Publication/ Report Number:
  DOT/FAA/TC-26/14
• DOI:
  https://doi.org/10.21949/r52z-r306
• Resource Type:
  Tech Report
• Geographical Coverage:
  United States
• Corporate Publisher:
  United States. Department of Transportation. Federal Aviation Administration. William J. Hughes Technical Center
• Abstract:
  This project aims to advance risk-informed decision-making at the FAA by developing machine learning techniques for extracting safety-relevant information from incidents and accident reports and enhancing large language model (LLM) capabilities for navigating aviation regulatory text. The work addresses two primary challenges: the labor-intensive process of manually extracting safety data from Aviation Safety Reporting System (ASRS) and National Transportation Safety Board (NTSB) narratives for use in the Integrated Safety Assessment Model (ISAM), and the inability of general-purpose language models to accurately interpret the hierarchical structure and cross-references within Title 14 of the Code of Federal Regulations (CFR). To improve safety data extraction, the project developed two datasets. The first is a question answering (QA) dataset designed to enable precise supervision for fine-tuning LLMs to extract events, contributing factors, and operational conditions from narrative text. The second consists of synthetic safety reports generated using a fine-tuned LLM informed by a domain-specific knowledge graph (KG) built from FAA aircraft data to eliminate aircraft specification errors and improve multilabel classification performance. To enhance regulatory reasoning, the project built a regulation-aware knowledge graph and developed a retrieval-augmented generation (RAG) system with improved retrieval precision for semi-structured regulatory content. An evaluation framework was also implemented by adapting retrieval-augmented generation assessment (RAGAS) metrics for regulatory contexts and introducing a custom answer completeness metric to address the "missing zone problem" of incomplete retrieval. Finally, the FAA-provided Daedalus multi-agent RAG system was migrated to local infrastructure using the Model Context Protocol (MCP) and extended with a completeness-evaluation agent that provides actionable diagnostics by distinguishing between retrieval and generation incompleteness.
• Format:
  PDF
• Funding:
  692M15-23-F-00178
• Download URL:
  https://rosap.ntl.bts.gov/view/dot/93350/dot_93350_DS1.pdf
• File Type:
  [PDF - 5.62 MB]
• Collection(s):
  FAA William J. Hughes Technical Center for Advanced Aerospace
• Main Document Checksum:
  urn:sha-512:ceac1abe7029fd69db25390d411d69142a5d46690130017e2eeaa4859229f8443e513dac4415d9650e54467a22d20de0461a03a7ef20ba9aae3a4549d04e4d12