Leigh Parise: Policymakers talk about solutions, but which ones really work? Welcome to Evidence First, a podcast from MDRC that explores the best evidence available on what works to improve the lives of people with low incomes. I’m your host, Leigh Parise.

Over the past decade, there has been renewed interest in career and technical education, or CTE, as a way to expose students to different careers and help them build the skills necessary to succeed in the workforce. One common strategy is to offer opportunities for high school students to earn industry-recognized credentials, or IRCs. These are credentials or certificates that demonstrate that students have learned the skills or competencies in a specific industry or occupation. But what do we know about IRCs? What motivates a student to pursue one, who tends to earn them, and what are the benefits of earning one? In this episode, I talk with Matt Giani, a Research Associate Professor in the Department of Sociology and Faculty Affiliate in the Texas Behavioral Science and Policy Institute at the University of Texas at Austin. Matt recently conducted a study for the Thomas B. Fordham Institute that looks at the employment and postsecondary education outcomes of Texas students who earned industry-recognized credentials in high school. It’s the first statewide study to look at the effects of earning an IRC in high school.  We discuss the interesting, and at times, surprising findings from the study and their implications for CTE policy and practice.

 All right. I'm really excited for this conversation. Thanks so much for joining me today.

Matt Giani: Yeah, thanks for the invitation. 

Leigh Parise: Your recent report for the Fordham Institute looks at industry-recognized credentials in Texas high schools and their impact on students’ education and employment outcomes, a super important topic that I know there's a lot of interest in. Can you first define for us what industry-recognized credentials or IRCs are for anybody who's listening?

Matt Giani: Sure. IRCs are certifications or credentials that students or employees can earn through demonstrating knowledge and skills in a particular domain. They're different than certificates that are often issued by schools or colleges, in that there's usually an outside entity that manages the assessment process and provides the certification. For example, it can be an industry group, like the manufacturing association. It can be a business like Microsoft or Cisco, or it can even be a state entity that authorizes certification. In Texas we have the Texas Department of Licensing and Regulation. So, if you want to be a cosmetologist, that's the state agency that oversees the licensing process.

Leigh Parise: Great. And can you say a little bit about why schools, either in Texas or much more broadly, why is this something that they are offering for students?

Matt Giani: Yeah, so I think a few reasons, one of which is that sometimes we swing the pendulum back and forth between preparing students for college or preparing students for career. And for a while it seemed like the field had moved away from career preparation, vocational education, career academies. And just in the past decade or so, there's really been this resurgence of interest in helping students prepare for jobs and employment after high school. We've seen a resurgence in interest in career and technical education and the latest iteration of the Perkins legislation that oversees and influences career and technical education policy across the country. ESSA, the Every Student Succeeds Act, the reiteration of the Elementary and Secondary Education Act also placed a big emphasis on career readiness. And it seems like many states are using industry certifications as one of the primary indicators of students' career readiness. We're seeing state after state adopt industry recognized certifications to measure whether students are ready for life after high school.

However, I think we're also in a place where there's kind of rhetoric and theory about why these certifications should matter, but there's not a lot of empirical evidence, right? States are adopting these things without a lot of research really looking at the relationship between the certification students earn in high school and their post-high school outcomes. So that was really the impetus of this research. Texas adopted industry-recognized certifications and started factoring them into school accountability as a measure of students' college and career readiness beginning in 2017, which was also the first time that they started collecting data on certifications. Before that time, we couldn't even do the type of study that we did. So really for the first time, we can ask the question, "Well, do these certifications matter? Are they actually helping students in the transition to high school or college?"

Leigh Parise: That seems really important. And I love the way that you described that is really, “This was kind of what was happening in the policy landscape and discussion about what we really needed to do for students, but then we had this opportunity.” You had this opportunity to say, "Okay, maybe, but let's actually see if we can look at the data to see what that tells us and see what we can learn and what questions remain open."

You talked about how the state of Texas has done this. How common is that? Are there other states that are really taking this on in a broad way, where there's policies or practices that are happening at the state level?

Matt Giani: Yeah, I don't know the number of states, but if I had to guess, I'd say somewhere between about a dozen states and half of the states at this point have adopted industry recognized certifications within K-12 accountability policy. More states than that provide opportunities for students to earn certifications, but they might not factor them into accountability. The Education Commission of the States has a report looking at the adoption of industry-recognized certifications. It's not quite every state across the nation, but somewhere between a quarter and a half of states have really promoted IRCs within K-12 state policy.

Leigh Parise: Great. Say a little bit more about what it really means for them to adapt them within their statewide accountability policy.

Matt Giani: Yeah, so there's a couple different ways that can work. I mean, sometimes you see that there's just a greater emphasis on the certifications within career and technical education pathways and programs of study. So, when the states are designing their programs of study of CTE they say, "Oh, here's the associated industry-recognized certifications that we think students should earn along the way." Sometimes states are also providing funding to reimburse students or schools, so that they can earn the certifications. So, one of the mechanisms is that the state's providing funding as an incentive.

And then another option is that the certifications get factored into school accountability ratings. I'll use Texas as an example. Accountability in Texas has a number of domains, one of which is college, career, or military readiness. So, the question is, how is that measured? Well, broadly, students can earn a dual credit course, they can score above certain thresholds on standardized tests—either the K-12 tests used for accountability purposes or tests like the SAT or the ACT, which have different college readiness benchmarks. And then earning an industry-recognized certification is one of the indicators that determine students are career ready, not that they're college ready. But it all gets combined into the broad umbrella of college, career, or military readiness. The school gets a point basically for students earning that certification.

Leigh Parise: Tell us what specifically were you able to study in Texas? And let's jump into some of what you found or what you think the implications are.

Matt Giani: Sure. Texas is one of many states that has something called a Statewide Longitudinal Data System. This is a data system that has student-level data that allows you to link students' K-12 educational records with their college enrollment and attainment records, and with their employment records that are collected through the Texas Workforce Commission or other kind of workforce agencies in other states through unemployment insurance wage data collection.

So, that allowed me to look at three cohorts of high school graduates—those who graduated in 2017, 2018, or 2019—and then examine the relationship between the certifications they earned and their likelihood of going to college, the type of college they enrolled in, whether they were employed their first year after high school, and their earnings their first year after high school. And I was able to control for lots of other things about students, so it's not just a correlation. We control for the high school that they graduated from, their demographic characteristics, their standardized test scores, and all the other courses that they took in high school, to try to isolate this relationship between the certifications they earned and their post-high school outcomes. The total sample was about a million students. Texas has about 350,000 students who graduate from public high schools every year, so the total was a bit more than a million students. So that was all a bunch of quantitative analysis with the Statewide Longitudinal Data System.

But then, I was also really fortunate to get the opportunity to talk to students. Because, oftentimes, you learn way more in hour-long conversations with students than by analyzing a million records over the span of three cohorts. I got to do some interviews and focus groups with students who were in career and technical education programs, not all of which earned certifications. And in fact, one of the interesting findings is that awareness of the certifications varied a lot. Some students knew exactly what the certification was, how they got the certification, and what the process was for going through the exam. They could explain it in excruciating detail, right? They knew every single piece and step in order to get the certification. And other students really didn't know what the certification was. They said, "Oh, I think my teacher mentioned that last year. We might have had a book that helped prepare us, but I don't really know. I think you take the test on the computer, right?" They just couldn't provide a lot of details even about the certification itself. So that was really, really informative to be able to get to speak to students as well.

Leigh Parise: Yeah, that's great. So, first, the opportunity to look across over a million students I'm sure was something that was really exciting. And to look across a whole state is a really unique opportunity. But then I do appreciate really having the opportunity to hear from the students themselves about their experiences, so that's a really nice pairing of what sounds like this huge set of quantitative data with a really rich set of qualitative information.

All right, so if people are going to take just a few things away from what you learned, what are the main things that you would want to drive home that you would want them to walk away from this podcast knowing?

Matt Giani: Great question. I think I'd highlight a few things. The first is that, on average, certifications don't seem to harm students too much. And in the past, there was some research that vocational education might actually harm students. It might divert students from higher education or divert them from four-year colleges to two-year colleges. And overall, that's not really what I found. In contrast, on average, there tended to be a positive association between students earning certifications and their likelihood of going to college and their first-year earnings. Although, I didn't find much relationship with one's likelihood of employment. So, it didn't seem like the certifications were moving the needle on students who would not have gotten a job, and now with the certification they're able to get a job. It's more about kind of a modest, positive relationship on first-year earnings. So, I think one takeaway is that, on average, the certifications seemed to help more than they harm, right? And that is a good thing.

I think the second key thing is that CTE generally had a pretty nasty reputation for a long time and was perceived as a mechanism for reinforcing social stratification, where low-income students, and students of color, and students with disabilities, and immigrant students, and non-native English speakers were funneled into these low-quality vocational pathways in ways that likely exacerbated inequality. But that's actually not the pattern that I'm finding in the data, at least in Texas. So, in fact, Black students were the least likely to earn certifications. White students and Hispanic students had pretty close to equivalent rates, and they were the most likely to earn certifications. Low-income students were a little bit more likely to earn certifications than non-low-income students, but not by much. It was maybe half a percentage point different, so we don't see the socioeconomic stratification.

And in fact, students who earn certifications are higher achieving than students who don't earn certifications. So, a lot of the historical patterns about stratification in CTE are kind of the opposite from what I'm finding in the current iteration of CTE and within industry-recognized certifications.

But I think one of the most fascinating findings is that, although those student characteristics mattered a little bit, what mattered the most by far was schools. The school a student attends exerts a huge influence on their likelihood of earning a certification. It sometimes explained three quarters of the variation in a student's likelihood of earning a certification. And, just for context, schools explain maybe 15%, maybe at most 25%, of the variation in student test scores, when we look at nationally representative samples of students. So, the fact that schools explained three quarters of the variation in students earning certifications was huge, right?

But I honestly don't know why. I've looked at lots of school characteristics that should relate to students' likelihood of earning certifications—the demographic characteristics of the school, whether they adopted different school reform models that should promote CTE or certifications—and I haven't come up with much. I don't really have any explanations for why schools matter as much as they do. So, that's definitely an important area for research. What are schools doing that actually shapes whether students earn certifications or not?

Leigh Parise: All right. I'm curious to go back a little bit. Which, you know, you might hear and think, "Hmm, okay, sure. That's an odd finding for the first thing that you say." But the history of CTE and vocational coursework is important context to know about. And it is interesting to me that the historical patterns seem to have flipped in some ways. It might actually be that there are fewer students of color, or students who are more well-off, who are actually the ones gaining access to this opportunity. And I wonder if you feel like, first, if that's right, and then if there should be a bigger push to really ensure that students are getting more access to these opportunities. And if there's anything that you learned when you actually had the chance to talk with students about who they think seems to be getting access or seems to be offered the opportunity to get an IRC.

Matt Giani: Yeah. Well, I'd love to hear your thoughts on this as well. But I feel like there's a certain irony in career and technical education currently and, I guess, industry-recognized certifications specifically. So, once again, when you look at the historic data, our researchers would often characterize vocational education as tracking. And I think what was meant by that was two things. One, that the programs were lower quality and resulted in worse outcomes for students who participated compared to students who didn't participate. And second, that the students who enrolled in those programs tended to be historically marginalized student groups, right? Low-income students, students with disabilities, students of color, and so forth.

What we see now, to an extent, is that both of those things have changed. Not only in this study, but in other research I've done on career and technical education, I'm finding a stronger positive relationship between CTE and college-going, between CTE and first-year earnings, between industry-recognized certifications and those outcomes. And at the same time, we're seeing a greater diversity of students enroll in certifications, including what appears to be higher achieving students and more privileged students. So, it’s almost like, as the programs are becoming higher quality, we’re seeing a diversity of enrollment.

And I think the irony is that, although I believe the implementation of vocational education was poor and likely resulted in exacerbating inequality, I think the intent was good, which was to focus on historically marginalized populations and those for whom going to college may have felt inaccessible, right? So the question was, how do we still provide students with high-quality programing and opportunities to get good jobs, to get mentorship, and to get experience in the labor market?

So, although in some respects it’s good that the population of students has diversified, I think it’s an open question of who are these programs for, right? Are they for the high achieving kids? Are they for the really privileged kids? Maybe, perhaps, it's for everybody, right? Perhaps every student should participate in CTE or every student should earn certifications. But that wasn't the original intent of vocational education. And I think you could make the argument that we should invest disproportionately in historically marginalized communities and in historically underfunded schools to develop really high-quality programing in those contexts.

So, it's kind of a, a puzzle, right? And I think there's a lack of clarity in policy about what are we trying to accomplish with these programs. Should they prepare students for college, or career, or both? Should historically marginalized students participate in them or all students? There seems to be some messiness in the focus on CTE and certification policy at the moment.

Leigh Parise: Yeah. We’re doing a project that I think actually gets at some of those questions because the idea behind that work is that all students within a given district are supposed to be participating in one of these kinds of opportunities by the time they graduate from high school. So that could be an industry-recognized credential, it could be an internship, it could be an apprenticeship, it could be dual enrollment coursework. It's sort of thinking about, you know, how do you ensure that everybody gets something by the time they leave high school?  We'll learn more about how that's actually playing out on the ground, but I’m curious to hear your thoughts.

Matt Giani: Well, you know, if someone were to ask me, "Should districts use a model like that, where all students are benefiting from these experiences?" I would say, "Yes, that sounds like a great idea." But if someone then asked me, "Well, should we focus our resources on students from historically marginalized or underrepresented backgrounds?" I would say, "Yes, that also sounds like a good idea." But those things are kind of in conflict and we kind of need to decide what we're trying to do, right? And I think we're experimenting as a field with the different ways to approach CTE, and apprenticeships, and internships. And there's a lot to learn about the best ways to approach it, for sure.

Leigh Parise: When you talked about the context for the study, you started by saying that if we're really thinking about preparing students for careers and what comes after high school, this might be one of the things that we could consider. Tell us a little bit about what you found when it came to looking specifically at postsecondary and employment outcomes of students who did earn an IRC.

Matt Giani: Yeah. For post-secondary outcomes, on average, there was a positive but fairly minimal relationship between earning a certification and college-going. Generally, there was about a one percentage point to two percentage point increase in a student's likelihood of going to college if they earned a certification. There was a slightly larger influence on college persistence, which is actually interesting because I might have thought the opposite. If a student earns a certification, maybe they're more likely to say, "Well, I can go into the labor market. I don't need to finish my program." So, I actually thought there might be an inverse relationship with persistence, but we actually found a positive relationship with college persistence as well.

In terms of earnings, as I mentioned, there wasn't much influence on employment. So, we didn't find any evidence that earning a certification increased a student's likelihood of being employed their first year after high school. But we did find about an eight to ten percent increase in students' earnings if they earned a certification. That's on average for students who earn certifications.

I think another key piece of the story is the variation across certifications in these relationships. What I did is, rather than looking at individual certifications because there's dozens and dozens of them and it gets very messy looking at all the individual certifications, I grouped them into clusters or subject areas. So, did a student earn a health science certification? Did they earn a manufacturing one? A transportation one? One in agriculture? And so forth. And then I looked at the relationship between those certifications and different outcomes.

And I think that the findings are not shocking. They're probably what many people would expect. But, for example, the traditional vocational programs, like transportation, manufacturing, construction, and cosmetology, in particular, tended to have a larger positive influence on earnings, but were negatively associated with college-going. Which isn't surprising. If you get a cosmetology license, there's not a ton of added value of going to a four-year institution if you want to continue in cosmetology. If you want to switch gears and study something else, of course. But cosmetology was actually the field that had the greatest positive increase on earnings, but the largest inverse relationship with college enrollment, right? So, I think it's really important to look at not just whether students earned a certification, but also what specific certifications students are earning. And what are their relationships with college and career outcomes?

And finally, I’ll just say that there were some fields that actually had positive relationships with both outcomes. I think health science was the clearest example of this. If students earned health science certifications, they were more likely to go to college, and they had better first-year earnings compared to other students. So, there are some examples of certifications that seem to provide benefits for both types of outcomes, that are genuinely preparing students for college and career. But those fields and those certifications tended to be the exception, rather than the rule.

Leigh Parise: Great. I want to pick up on one of the things that's sort of running through some of what you just shared. Would you say that getting a certification actually means that students are career-ready? Is that the signal that they're ready?

Matt Giani: Well, that's what state policy says. But that's not what I say. I don't think I would consider students to be career-ready if they earned a certification. It's a philosophical question what we mean by career readiness. And I think we can have lots of healthy debate about what is reasonable to expect for 18-year-olds or 19-year-olds who are just coming out of high school and entering the labor market.

But, although I found a positive relationship between certifications and first-year earnings on average, I also found that students tended to be earning about $10,000 a year after their first year after high school. Now, some important caveats. The employment data, the UI wage data, doesn't have things like your occupation or the number of hours you're working. So that’s a key question, are students working full-time? Are they working part-time? Perhaps they're earning a reasonable hourly wage, but they're only working 10 to 15 hours a week because that's all they need to work. In which case, maybe they're doing great, right? But on average, students aren't earning much.

And one way that I defined postsecondary readiness was if students either enrolled in college or they earned 200% of the poverty line, which was about $25,000 a year. Almost no students were earning that much, less than 5% of the sample. It didn’t matter if they earned certifications or not. Almost no 18- to 19-year-olds in the sample were earning $25,000 a year.

Now, if you're living in a place like Austin, where I live, $25,000 a year is not very much, right? So even if a student was earning that and trying to live independently, they'd struggle. It would be very difficult for them to get by, in my view. And once again, those students earning $25,000 a year are by far the exception. That's the top five percent of earners in a place like Texas. So, can we reasonably conclude that students are career-ready just because they earned a certification? I would say unambiguously no. I would not consider these students to be career-ready. And it's an open question, I think, what does career readiness mean? How do we measure it? And how do we ensure students are career-ready? Particularly when the economy has been in shambles the past few years and students are struggling even more than they were in the past. And we have a larger percentage of students not enrolling in college and trying to find jobs after high school. I think it's critical to ensure that we're actually preparing students for the opportunities and the challenges that they'll face in the labor market after high school.

Leigh Parise: I think that is really important context. That, when you say, "There were impacts on earnings," but then, "Actually, almost no one is earning enough to really live independently” is an important thing for people to take away. One of the things that I hear you saying is that one of the takeaways is that high school IRCs might be important, but they're not transformational when it comes to longer-term outcomes for students' lives. Given that you've focused on this work and CTE for a long time, are there things that you think might make CTE experiences more impactful for students? Or do you have advice or key things to think about for policymakers or practitioners who are thinking about how to do this really well?

Matt Giani: Yeah, it's a good question. I think one thing I'm reflecting on, once again, is the purpose of CTE and who should participate because I think that really informs how we should design the programs. One of the other findings that came out in the report is that there's often a pretty weak relationship between the certifications students earn and the jobs that they get out of high school, and specifically the industries that they're working in out of high school. And when I spoke to students about this, they really provided such fascinating insights on their thinking about these programs.

For example, one student was in a construction program and he really enjoyed it. He loved building things, working with his hands. But that's what his dad did, and his dad said, "I don't want you working outside for your career. I don't want you being exposed to the elements." So, he was very ambivalent about continuing in the construction program. Another student, a female student, was in the auto tech program, and I asked her what she wanted to do, and she said, "Oh, I want to be a vet. I'm going to go to veterinary school eventually." So I ask, "Well, why are you doing auto tech?" And she said, "Well, my dad's a mechanic. My uncle's a mechanic. I don't ever want to pay someone to fix my car, right?" That totally makes sense. I hate paying people to fix my car. I do it all the time, and I'm terrible at it because they just rip me off and I don't know anything about cars. I wish I had developed the skills that she has when I was going through high school. But it was clear she wasn't in the program to prepare for her career, right? And another student who was in the construction program said, "Well, eventually I want to go into real estate and flip houses." He doesn't want to be in construction. He doesn't want to build houses necessarily, but he thought it was a useful skill in combination with his desired career path.

So, I think one question that we have to ask is, how much are we trying to align student behavior and motivation with our pre-designed programs versus how much are we conceiving of CTE and certifications as ways for students to explore their interests while they're still in high school? And if they figure out they really don't want to do something because of their experience, that seems like a good outcome to me. But in the data, it seems as if there's a weak alignment between the CTE programs that students are completing and their outcomes after high school.

You asked, "What should we do about it?" Well, I think one option would be to create stronger guided pathways, to give students less flexibility, to really steer them very strongly into a particular pathway while they're still in high school. But I don't know if that's the right path, right? I would still probably fall on the side of, well, let's use CTE as a form of career exploration. And for students who are really dedicated and really want to pursue a career that's aligned with that program, then we need to add on the CTE coursework with the certifications, with internship opportunities, with apprenticeship opportunities, with mentorship from people who are actually working in the field, right? Then, you think of this kind of wrap-around programming. But my perception is that that's a smaller percentage of students who actually know for sure when they're 15 years old or 16 years old that, "This is the field for me and I want to prepare for this for the next two years and then get a job in this field right after high school."

So, I think just one suggestion is to be really clear about, who are the students we want to serve and how do we want to serve them? And then design programs that are responsive to student needs and the population you're serving, rather than just us as researchers and policymakers saying, "Here's what our ideal CTE system look like. Now, let's try to force students to align their behavior and interests with our pre-designed programs."

Leigh Parise: Yeah. I have to say, I think that makes a lot of sense when you say it, right? Let's not have 15-year-olds have to be making decisions for the rest of their lives, but rather giving them exposure to a whole variety of things. And maybe it's not a bad thing if they have the chance to explore something and think, "Oh, maybe that's not for me," before they go off and, you know, spend money on college tuition and things like that that might take them down that path, when they have the chance to do some initial exploration. All right, Matt, are there any final thoughts for anybody listening that you want to add?

Matt Giani: I guess just one thing I'd say is that it's also kind of ironic I've been doing CTE research for a while now, because I was actually very skeptical of CTE research. Which maybe was why when you asked, "What are the key findings?" I started with, "Oh, they don't do harm to students." But it's because I've honestly been pretty skeptical about CTE in the past. And really, it was by doing the research that it changed my perspective because the findings were not what I expected, right? We're seeing different relationships than what we used to in the past.

So, I would first just say that, for those who may be skeptical of CTE or who maybe think it's not as important, I mean, it's increasingly important. It's a growing part of the lives of the majority of students. I mean, something like 80 or 90 percent of students in Texas, at least, participate in at least one CTE course while they're in high school. It's almost omnipresent, right, despite not receiving that much attention. So, I would first just say that CTE is important and industry-recognized certifications are increasingly important. The evidence we have so far is actually quite promising, that we do see positive relationships between these certifications and student outcomes. But I think the key question is, where are the ones with the greatest value? And where are the ones with less value? And it's really up to the field, and it's up to policymakers, and it's up to educational practitioners to take a critical look at those programs and opportunities that aren't providing students with value and to make some tough decisions.

And I think my Fordham colleagues would say, "All certifications are not created equal." And we need to have tough conversations about those that aren't providing benefits. Even if there's interest groups that really support them, or there's a lot of attachment to these programs in school communities, if they're not benefiting students, we need to be honest with ourselves about that and invest in the programs that matter, and that provide return on investment, and that help students hopefully experience social mobility and find good jobs and career paths where they can earn a living wage and sustain themselves and their families.

Leigh Parise: Thank you so much for joining me today. I really enjoyed this conversation.

Matt Giani: Yeah, thanks to you and MDRC for the opportunity. And I look forward to reading more of your research on the subject as well, so thanks.

Leigh Parise: To learn more, check out Matt’s report on industry-recognized credentials at Fordham institute dot org. 

In an upcoming episode, we will look at how another region--Kansas City—is helping high school students earn IRCs and participate in other activities designed to prepare them for postsecondary education and the workforce.

Did you enjoy this episode? Subscribe to the Evidence First podcast for more.