SolarWinds’ federal footprint is large, and compromise is a ‘nightmare scenario’ for affected agencies

Federal agencies faced the most urgent kind of deadline Monday: They were given until noon, Washington time, to respond to a compromise by foreign hackers in a sensitive piece network management software.

The emergency directive from the Cybersecurity and Infrastructure Security Agency (CISA) ordered all agencies using SolarWinds products to review their networks and disconnect or power down the company’s Orion software. Although many of the details of the hack remained unclear as of Monday afternoon, a few things helped explain CISA’s urgency: Orion has been available to the government for years through a complicated array of contracts, and the software operates at the heart of some crucial federal systems.

SolarWinds has been supplying agencies for a long time, first developing tools to help them understand how their servers were doing, and then branching out into network and infrastructure monitoring. Now IT teams can use those tools to manage virtualization and even security features such as privileged accounts and patching.

Orion is the framework tying all of those things together and helping system and network managers understand what’s going on. At least 32 federal agencies bought SolarWinds Orion software since 2006, according to a preliminary search of the Federal Procurement Data System – Next Generation (FPDS-NG) conducted by The Pulse of GovCon, a boutique market intelligence firm.

That footprint made Orion an attractive target to foreign spies, who used the company’s updating system to push out malware that allowed them to break into the departments of Commerce, Homeland Security and the Treasury. The attacks on the federal software supply chain are part of a campaign staged by Russian hacking group APT29, or Cozy Bear, on behalf of the SVR intelligence agency, The Washington Post first reported.

Many more agencies could be affected, but those were three confirmed by multiple media organizations as of Monday afternoon.

“It’s almost a nightmare scenario, when you think about it, because these are tools that people put into the most sensitive parts of their network, the network management centers, to help them understand what’s going on with everything from the Wi-Fi switch in the conference room to the server that might have the most sensitive data at the agency,” said a retired senior government official, who asked not to be identified to speak freely about the compromise.

“And the adversary has essentially had a conduit to push malware to bypass the firewalls and all the other normal security checks and could potentially have moved anywhere in the infrastructure from there,” the former official said.

A public assessment of SolarWinds’ full federal footprint remains difficult, however, in part because 48 different resellers were awarded some of the 204 known federal contracts for Orion products since 2006. The likelihood FPDS-NG has not recorded all such transactions is high.

Adding to the challenge CISA has of assessing the damage is the fact the Orion vulnerability has been used to deploy malware inside agency networks since March, according to federal officials.

“The compromise of SolarWinds’ Orion Network Management Products poses unacceptable risks to the security of federal networks,” said Brandon Wales, acting director of CISA, in a statement. “Tonight’s directive is intended to mitigate potential compromises within federal civilian networks, and we urge all our partners — in the public and private sectors — to assess their exposure to this compromise and to secure their networks against any exploitation.”

CISA’s emergency directive locks down federal networks, but the next steps will be to look for signs of penetration, identify malware, eradicate it, and potentially fix security configurations that were compromised.

U.S. Cyber Command — which has more mature processes and more experienced cyber protection teams in place for doing such things — is likely coordinating with CISA, the retired official said.

Assessing Orion’s reach

Any agencies that choose to cut ties with SolarWinds over this incident will need to replace the infrastructure they’re losing.

“The cleanup from this could be going on for months and could cost millions of dollars,” the retired official said. “A bunch of days have been ruined here that’s for sure; there’s going to be a lot of IT guys who are going to be working overtime for months to deal with this.”

That number is extensive if contract awards for SolarWinds Orion products on FPDS-NG are any indicator.

“Due to the limitations of these procurement systems and their classification procedures, we can assume that this is the floor — not the ceiling,” a Pulse spokesperson wrote FedScoop.

Since 2006, contracts for SolarWinds Orion products have been awarded by the:

• Bureaus of Land Management, Ocean Energy Management, and Safety and Environmental Enforcement, as well as the National Park Service and Office of Policy, Budget and Administration within the Department of the Interior
• Air Force, Army, Defense Logistics Agency, Defense Threat Reduction Agency, and Navy within the Department of Defense
• Department of Energy
• Departmental Administration and Farm Service Agency within the U.S. Department of Agriculture
• Federal Acquisition Service within the General Services Administration
• FBI within the Department of Justice
• Federal Highway Administration and Immediate Office of the Secretary within the Department of Transportation
• Federal Law Enforcement Training Center, Transportation Security Administration, Immigration and Customs Enforcement, and Office of Procurement Operations within the Department of Homeland Security
• Food and Drug Administration, National Institutes of Health, and Office of the Assistant Secretary for Administration within the Department of Health and Human Services
• IRS and Office of the Comptroller of the Currency within the Department of the Treasury
• NASA
• National Oceanic and Atmospheric Administration within the Department of Commerce
• National Science Foundation
• Peace Corps
• State Department
• Department of Veterans Affairs

While all of these agencies bought SolarWinds Orion products, that doesn’t necessarily mean they were still using them between March and June, when the company suspects the vulnerability was introduced during updates. Agencies that have ongoing contracts for SolarWinds Orion products include the Army, DOE, FLETC, ICE, IRS, and VA.

SolarWinds estimated that, of the 33,000 Orion customers in active maintenance during the relevant period, fewer than 18,000 installed products with the vulnerability, in a report it made to the Securities and Exchange Commission about the cyberattack.