Here’s what happened when the IRS’s electronic filing system crashed on Tax Day

The great IRS electronic filing system outage of 2018 could have been avoided, a new Treasury Inspector General report has found.

The report, released Sept. 19, provides a step-by-step look at what went wrong and how, had decisions been handled differently, it all could have been prevented.

Recall: At around 3 a.m. EDT on April 17, 2018 — Tax Day — the IRS  ran into some tech trouble. The ensuing 11-hour outage was embarrassing and prompted the tax agency to extend the filing deadline by one day. And all this drama, it turns out, was caused by a firmware bug that IBM initially discovered in June 2017.

According to the IG report, IBM created a fix for the bug and made it part of the IBM microcode bundle that was released to the public in November 2017. In a December 2017 meeting between the IRS IT team and the agency’s enterprise storage services contractor Unisys Corp., the parties discussed whether to update the microcode bundle to the new version. In the end, the IRS decided, on the advice of Unisys, not to update, the report says.

This wasn’t the IRS’s only opportunity to avoid the eventual outage, either — in January 2018, the IG found, another IBM client experienced a similar outage caused by the firmware bug. In response to this, IBM developed a script to correct the error. IBM did not, however, tell the IRS about this fix option.

The report does commend the IRS for its response to the crash, given the circumstances. “Established contingency tools, processes, and procedures for mainframe outages served the IRS well during the Tax Day outage,” the report states. “The IRS detected, assessed, repaired, and restored mainframe operations by the afternoon of the day the outage occurred and processed almost 4 million tax returns before midnight within acceptable performance time frames for acknowledgement and receipt.”

However, it also makes a number of suggestions for how the agency can learn from this experience and avoid similar issues in the future. The IRS should formalize its monthly microcode bundle meetings, the IG says, and make sure that any future decisions not to update the microcode are both documented and approved by the system’s authorizing official.

The IG also recommends that the IRS seek damages from Unisys for the contractor’s failure to meet contract requirements. The IRS agreed with all of the report’s recommendations.