What You Need To Know About The IRT Model Replacing Current CSA Scores

The Federal Motor Carrier Safety Administration (FMCSA) is currently working on an Item Response Theory (IRT) Model that will replace the current Safety Measurement System (SMS) used for CSA Scores.

SMS identifies commercial motor vehicle (CMV) carriers at high risk for future crashes. The IRT Model will focus on “safety culture” rather than trying to predict crash risk.

Currently, the implementation date for the new model isn’t stated because it is still in the development phase.

Why IRT?

After many criticisms, Congress ordered a review of CSA’s Safety Measurement System.

2015 – CSA was reviewed by the National Academy of Sciences (NAS) as part of Congress’ FAST Act.

June 2017 – NAS critiqued many variables of CSA in their 132 page report, “Improving Motor Carrier Safety Measurement.”

SMS Shortcomings:

    • Using highly variable assessments
    • Not accepting for crashes where the motor carrier is not at fault
    • Including carriers that have very different tasks in the same peer groups
    • Using measures that are sensitive to effects from one or more individual states
    • Using measures that are not predictive of a carrier’s future crash frequency
    • Using measures that are no reflective of a carrier’s efforts to improve its safety performance over time

The National Academies reported that the SMS implementations is “fair and not overtly biased;” however, some features could be improved, “if the algorithm currently used were replaced by a statistical model that is natural to this sort of discrimination problem.”

The NAS panel’s report recommended using a more scientific approach to safety scores based on and IRT model with greater data input. 

Item Response Theory

IRT has been used for policy decisions in hospital rankings and SAT scoring in the education field. The IRT Model will use improved data quality taking into account driver compensation, vehicle miles traveled by state, police narratives, carrier characteristics, cargo, driver turnover rate and any other factors that influence the IRT safety calculation.

There will no longer be violation weights, CSA points, BASIC measures or safety event groups; however, the same BASIC data points used to determine SAFER scores will continue to be the foundation. Looking forward there will be only 1 score that represents a company’s “safety culture score.”

CSA scores are improving to address unforeseen problems. Public CSA scores have been used by insurers and attorneys in ways the FMCSA never intended.

It has now been 2 years since the NAS panel recommended the IRT model and it is still under development.

Looking Forward

The full timeline for the CSA score improvements have not been released by the FMCSA but there is a tentative plan for the future:

Summer 2019 – Meeting (Date TBD): Share IRT Model in small scale.

September 2019 : Complete full scale IRT Model.

September 2019-2020/2021 – Meeting (Date TBD): Implementation Period with lengthy Comment Period.

So how can you improve your Safety Culture?

Gorilla Safety cares about helping your fleet stay compliant and improve safety scores. We will keep you updated as the FMCSA continues to develop the IRT Model.

In the meantime, avoiding safety violations will help your company to remain compliant. 

Leave a Reply