Evidence-based decision making: What DOE, USDA and others are learning

Evidence-based decision making is gaining traction at agencies, a new report says, and there are some clear lessons about how to increase federal use of the practice.

“Agencies have new opportunities to tackle the tough issues and get answers to their big questions using high quality data from multiple sources,” said Nancy Potok, chief statistician of the United States, in a new report released June 27 by the Partnership for Public Service and  Grant Thornton.

The new report, Seize the Data: Using Evidence to Transform How Federal Agencies Do Business, acknowledges that the use of rigorous evaluation and data analysis to inform decision-making is still nascent and not widely adopted in government. But the tactic has taken root in some programs, the researchers say, and those examples point to some recommended practices.

The report urges agencies to use both a “top-down and a bottom-up approach” to build evidence-based organizations, calling on leaders to set expectations, empower staff and develop bureau-level learning agendas. Those opportunities are likely to get greater attention in the wake of recent legislation — the Foundation for Evidence-Based Policymaking Act signed into law by President Trump in January — and the latest White House Federal Data Strategy, released June 4.

A ‘key to our success’ at DOE

Creating a culture for evidence-based decision making starts in part by promoting the benefits, the experts say. The report points to efforts at the Department of Energy, where senior officials realized that some employees were reluctant to report safety issues at DOE facilities for fear of repercussions. The key was to get employees to see such reports as important data, said Josh Silverman, who oversees reporting activities within DOE’s Office of Environment, Health, Safety and Security.

“We’ve gotten people more interested in reporting issues at their sites so we can avoid incidents in the future. They’re doing it for the benefit of others who might learn from it and they’re doing it without fear that they will be judged harshly,” he said. “Reporting and analyzing thousands of minor events is key to our success in preventing the major incidents from occurring.”

The office publishes short reports illustrating how safety issues occurred, such as the improper handling of hazardous material.  “We’ve focused on becoming more of a learning organization,” Silverman said.

Outside help at USDA

The Partnership for Public Service report also emphasizes the need to strengthen internal capacity and external research-practitioner partnerships to build up data-gathering and evaluation skills.

Mark Denbaly, deputy director for Food Economics Data at the U.S. Department of Agriculture’s Economic Research Service made that clearer during a Partnership for Public Service forum held in conjunction with the report’s release.

Trying to leverage data to make better decisions at USDA is a daunting task, given the scale of some of its programs, Denbaly said. He cited a USDA nutrition program for women, infants and children (WIC) that provides assistance through 60,000 store operators across the country and involves more than a million products.

The need for deeper data analytics expertise led USDA to partner with New York University’s Administrative Data Research Facility (ADRF) to develop an intensive one-week training program for USDA and other federal government staff, set to begin this October.

“The training is going to teach them to use the tools to help them answer really useful questions at the local level, such as, ’How much does a WIC household spend on food and what percent of that is spent on WIC-related food products?’ Or ‘What is the shopping pattern of households on WIC compared to households which qualify for WIC but decide not to participate?’” he explained.

The program is on a fixed budget, he said, “so the better you manage the program, the more people who can benefit from that.”

‘Connectivity is really important’

Other recommendations in the report called for driving longer-term and shorter-term learning — relying on continual feedback, rapid experimentation and customer feedback — and to use existing data as well as new data to generate added insights.

“Leaders at all levels are interested in … how you connect spending and cost information with the benefits, “ said Amy Edwards, deputy assistant secretary at the Treasury Department for Accounting Policy and Financial Transparency, during the forum. But she added, more needs to be done to ”utilize the administrative data we have at Treasury and combine it with other data” to get the value out of it. That connectivity is really important.”

Other panelists at the forum also stressed the importance of transparency and asking the right questions.

“It’s really important to bring transparency to the process as well as the products,” said Christina Yancey, the Department of Labor’s acting chief evaluation officer. She’s recommends posting evaluation plans online, for instance, and also posting what active studies are underway.

She also recommends starting with people “who are interested and invested in working with you,” who can become champions for evidence-based decision making. “Don’t try to convert those that are resistant because you’ll just waste way too much time.”

Former Office of Management and Budget associate director Robert Shea, now principal for Grant Thornton Public Sector, summed up the essence of the report, saying: “The important thing is having the information available in its most rigorous form at the point at which it can have the biggest impact on a policy.”