The Five Phases of Successful Data Analytics
TANF Data Collaborative Pilot Resources Toolkit
This toolkit provides individuals and organizations with guidance, drawn from learning and experience, on how to use administrative and other data to inform program improvements. It collects concrete strategies and practitioner-tested tools designed to advance these efforts. These materials were developed in pilot projects with local Temporary Assistance for Needy Families (TANF) agencies as part of the TANF Data Collaborative (TDC).
It is organized according to the five phases of completing a data analytics project used in the TDC’s framework:
- Laying the Groundwork
- Accessing the Data
- Preparing the Data
- Analyzing the Data
- Communicating the Results
Laying the Groundwork
The first phase of a data analytics project is about preparing to develop feasible analyses that will make an impact—using available data—and that align with your organization’s priorities. In this phase you should identify the research question(s) that the analysis will attempt to answer.
Guide to Prioritizing Research Questions
Use the guide to help you set priorities among your research questions. You may not have answers to all the questions in the tool, and you may find some topics more salient than others.
Project Scope: Instructions and Template
Use the instructions and template to complete categories of information that will help you define what are the project’s objectives, who is involved, which types of analytical methods and data will be used, and how you anticipate using the results. The template also facilitates the identification of any risks the project could face and anything the project team might need to complete the analytics on schedule.
Project Scope: Sample
Use the sample project scope as a guide for drafting your own. This scope was developed to help the team understand (1) the families served in a state TANF cash assistance program, (2) the services that they use before and after leaving the program, and (3) how those dynamics relate to outcomes, if at all.
Ongoing Use of Data: Strengthening Analytics in Government Agencies Toolkit
Laying the foundation for a particular project is also a good time to take stock of your program or agency’s ongoing use of data. Consider how this project contributes to a broader goal of moving from knowledge of data to action routinely, creating a reinforcing cycle of evidence building and program improvement. This tool was created to help agencies build the culture and infrastructure needed to apply data analysis routinely, effectively, and accurately—referred to in this publication as “sustainable data use.”
Accessing the Data
The second phase of data analytics work includes activities to address legal, ethical, and cross-agency coordination considerations to ensure data access.
The TDC pilot initiative expected all pilot agency teams to have access to administrative data from the TANF program and data about earnings and employment outcomes from the unemployment insurance system from the beginning of the pilot project. Therefore, pilot agency teams were able to spend more time on preparing and analyzing the data than accessing the data.
Sample Toolkit for Accessing, Linking, and Analyzing Data
The TDI team developed Expanding TANF Program Insights: A Toolkit for State and Local Agencies on How to Access, Link, and Analyze Unemployment Insurance Wage Data to help TANF professionals develop more robust practices using administrative data on earnings for program improvement purposes.[1] It may also be useful to other state human services agencies (for example, the Supplemental Nutrition Assistance Program and child support) that want to expand their data use, as well as policymakers interested in supporting improved workforce outcomes. State labor agencies may also gain useful insights from the data preparation section in Part 4, as well as from the broader discussion of ways to use employment data to improve human services programs.
Further guidance can be found in the companion GitHub repository, which offers open-source code and documentation for staff members preparing employment data for analysis.[2] It also includes resources with related supplemental materials that have emerged from this project. Resources cover a variety of topics, including equity, data security, and programming and data quality control tips.
Preparing the Data
Flawed input data produce flawed outputs. Carefully preparing data for analysis is an important step toward ensuring that it meets certain quality standards. Preparing data for analysis can include various procedures related to linking, deidentifying, and restructuring the data so that it is ready to be analyzed. See the Unemployment Insurance Wage Toolkit for guidance on those procedures. The Applied Data Analytics training program from Coleridge Analytics provides resources and instruction on activities related to preparing and analyzing data. [3] The tools in this toolkit focus on the quality-checking and cleaning activities.
Quality control checklists and memos are useful for documenting features of the data, including any limitations of the data for analysis. They can help the team reach a shared understanding of the data and can help future staff get up to speed quickly in preparation for working with the data.
Data Quality Control Checklist
This tool organizes questions related to data quality checking at five points in the process: (1) before obtaining the data (relating to its nature and source), (2) before processing, (3) before writing code, (4) while processing, and (5) after processing.
Template for Unemployment Insurance Wage Data Quality Control Memo
This tool is a template for documenting information about a file including that includes unemployment insurance data and checking the data’s quality. It covers: (1) project background, (2) file locations, (3) key decisions, (4) checking the raw data file, (5) checking data updates, and (6) checking a person-level file. Although this template is based on using unemployment insurance wage data, many of these topics are relevant for checking the quality of any kind of data.
Analyzing the Data
Once the data have been prepared and cleaned, your team chooses an analytic method suitable for both your data and research question. Selecting the most appropriate method is crucial to effectively answering your research question. Some methods will be more effective than others at answering your particular question.
Analyzing Data: Research Questions and Methods
This tool offers examples of questions and approaches that can answer them.
Did It Work? Interpreting Study Findings
This tool is designed to help you interpret findings from three common, nonexperimental research designs. These designs are ‘nonexperimental’ because they do not involve random assignment. These designs can be used when you can observe how outcomes change over time.
Program Evaluation Resources and Evaluation Glossary
These tools are designed to support staff members who are learning concepts, vocabulary and ways to design and conduct program evaluations and different types of evaluations.
Communicating the Results
This phase includes the final steps for formatting and sharing findings from the analysis, offering the opportunity to explain what you learned and why it matters to different audiences.
Project Summary Report Template
This tool outlines a table of contents in a typical research report that can serve as the final output of a data analytics project. It can draw from a variety of other outputs created during the project itself such as the project scope, analysis plan, or interim reports. The tool offers six categories of report content with prompting questions for you to consider along with possible appendixes such as code used during the analyses.
Briefing Instructions and Template
A final report is one type of dissemination product that can be used to share the lessons and insights learned from a data analytics project. Another type of product is a verbal presentation or briefing for project sponsors, managers, executives, and external partners. This tool offers a five-part table of contents to guide your preparation for a verbal presentation or briefing.
This work was funded by the Office of Family Assistance and the Office of Planning, Research, and Evaluation within the Administration for Children and Families of the U.S. Department of Health and Human Services.
Footnotes
[1] Edith Yang, Sharon Zanti, T.C. Burnett, Richard Hendra, Dennis Culhane, Zarni Htet, Della Jenkins, Camille Preel-Dumas, and Electra Small, Expanding TANF Program Insights: A Toolkit for State and Local Agencies on How to Access, Link, and Analyze Unemployment Insurance Wage Data, OPRE Report 2022-226 (Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2022).
[2] MDRC, “TDC-UI,” website: https://github.com/MDRCNY/TDC-UI, 2023.
[3] Coleridge Initiative, “Applied Data Analytics Training Programs,” website: https://coleridgeinitiative.org/applied-data-analytics-training-programs, n.d.
Document Details
Wavelet, Melissa and Sufiyan Syed. 2024. The Five Phases of Successful Data Analytics: TANF Data Collaborative Pilot Resources Toolkit. OPRE Report 2024-066. Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
