Development of a Machine Learning and Large Language Model-Based Classification Tool for Automated Analysis of Transportation Public Comments

Details

• Creators:
  Knezevic, Milan ; Neece, Trevor ; Stevanovic, Aleksandar ; Khazanovich, Lev
• Corporate Creators:
  Morgan State University. Sustainable Mobility and Accessibility Regional Transportation Equity Research Center (SMARTER) Region 3 University Transportation Center (UTC) ; University of Pittsburgh
• Corporate Contributors:
  United States. Department of Transportation. University Transportation Centers (UTC) Program ; United States. Department of Transportation. Office of the Assistant Secretary for Research and Technology
• Subject/TRT Terms:
  Classification Data Processing Operations Decision Making Infrastructure Large Language Models Literature Reviews Machine Learning Performance Public Transit Public Transportation Comments Transportation Safety

  More +
• Publication/ Report Number:
  SM45
• Resource Type:
  Tech Report
• Geographical Coverage:
  United States
• Edition:
  Final Report
• Corporate Publisher:
  Morgan State University. Sustainable Mobility and Accessibility Regional Transportation Equity Research Center (SMARTER) Region 3 University Transportation Center (UTC)
• Abstract:
  Transportation agencies receive large volumes of free-form public comments describing infrastructure conditions, safety concerns, and service issues. Processing and classifying these comments manually is time-consuming, inconsistent, and difficult to scale, limiting their usefulness for operational decision-making. This study presents a machine learning and Large Language Model (LLM)-based framework for automated classification of transportation public comments across a three-level hierarchical taxonomy: Category, Subcategory, and Final Decision (Dismiss, Refer, Potential Project). Comments are encoded using MPNet sentence embeddings and indexed in a train-only vector database. Three classification strategies are evaluated: (1) an embedding-based k-nearest neighbors (kNN) baseline using cosine similarity, (2) a zero-shot/few-shot LLM with schema-constrained outputs, and (3) a retrieval-augmented LLM that incorporates semantically similar historical examples and concise label guidance during inference. Results indicate that while high-level categories are generally well separated, many public comments contain overlapping or multi-modal semantic content that spans multiple infrastructure types. To address this ambiguity, the retrieval-augmented LLM is evaluated using ordered predictions, allowing multiple plausible labels to be returned. This approach improves overall classification performance and provides richer context for agency review. Overall, the proposed framework demonstrates the potential of retrieval-augmented LLMs to support scalable, transparent, and decision-oriented processing of public transportation comments, enabling faster and higher-quality agency workflows.
• Format:
  PDF
• Funding:
  69A3552348303
• Collection(s):
  University Transportation Centers
• Main Document Checksum:
  urn:sha-512:26a991bd27adde5506b52782d5b6cfa737bb0e1df3b1058a718bc6440492782ad504df650960cbe4617fe2a6d2e8bddc6aee64a950c7f5b0c25e2bdfd2ee614b
• Download URL:
  https://rosap.ntl.bts.gov/view/dot/92776/dot_92776_DS1.pdf
• File Type:
  [PDF - 3.56 MB ]