7: Metrics and Evaluation

Why does Evaluation Matter?

Know Where You’re Going

Each sector initiative, as early as possible, should select a set of measures relevant to its specific goals and activities. These measures articulate where the initiative is headed and how it will know when it has gotten there. As you design a benchmarking process that captures the continuous improvement of your sector initiative, consider that your measures should: 

  • Be realistic and utilize data that currently exists, so that you minimize the data collection burden on public and private partners.
  • Balance optimal long-term impact measurement against current capacity to collect and produce such measures.
  • Include both quantitative and qualitative indicators.

 A useful framework (but not the only one) for benchmarking is:

Evidence of Progress
Outputs, Products, Outcomes Longer Term Impact

Such as:
  Action Plan
  Employer support

Such as:
  Skills Standards
  Awareness Campaign
Such as:
  Reduced vacancy rates
  Increased rates of
  advancement

Each of these phases of evaluation can be considered across at least four categories of beneficiaries: employers; jobseekers and workers; public institutions and systems; and the partnership itself. Frameworks similar to this example from Washington State are being piloted by some states and local sector initiatives, each with variation.

Share Your Success

Simply stated, all sector initiatives should work toward being able to tell what happened, to put it into context, to tell the story of that context to multiple audiences, and to make decisions that improve the success of the initiative. Sharing success stories leads to on-going support, interest, and investment. It is the key to long-term sustainability. For example, the ability to provide employers with context and comparisons could lead to industry investment into the partnership and its activities. Some sector initiatives are completely funded by employer member contributions.

Recent History of Sector Evaluation

For the past decade two leading organizations for the evaluation of sector initiatives - The Workforce Strategies Initiative at the Aspen Institute and Public-Private Ventures - have studied dozens, if not hundreds, of sector initiatives across the country. Their reports range from evaluation of specific sector initiatives (such as the garment industry in New York City) or industry-focused strategies in a particular region (such as Central Indiana) to cross-sector studies that evaluate industry-specific approaches to workforce development or demand side outcomes. A number of reports look at worker and employer outcomes across many regional sector initiatives across the country. A recent evaluation of a sample of sector initiatives is Targeting Industries, Training Workers and Improving Opportunities: The Final Report from the Sectoral Employment Initiative (Public-Private Ventures, December 2008). 

In addition to actual evaluations of sector approaches, much recent work is focusing on helping local areas and states to develop frameworks or templates for evaluation that can be standardized to some degree; but also left flexible enough to accommodate the variation in design, approach, and outcomes critical to sector model success. An early attempt at such a framework is: An Evaluation Framework for State Sector Strategies (National Governors Association, Corporation for a Skilled Workforce, and National Network of Sector Partners, May 2008).

Here are some other templates that have been used:

  • A simple, one-page flexible model that is completely qualitative;
  • A highly-specific model for partnerships involving worker training; or
  • A broad matrix that incorporates impact on employer, worker, and public institutions, as well as effectiveness of the partnership.
  • A one-page "scorecard" that assesses impact on workers, employers, financial viability, and other indicators of success.

Assessing Impact

For an overview of the role of evaluation in sector partnerships, view this video.

Impact on Employers

Sector initiatives need to know how they are or are not effectively serving their customers, both employers and jobseekers. Measuring the value of a sector initiative to business can be challenging. Value may vary depending on the industry, the size of employers involved (e.g. are they small manufacturers or enormous hospitals?), and the types of challenges they are experiencing. Because of this variation, each sector initiative must carefully define what success for the industry will look like, how to measure it, and how to collect the data. Some long-term employer benefits might include reduced turnover, improved operations, upgraded workforce skills, a higher quality applicant pool, realized cost savings, increased workforce diversity, or creation of new networking opportunities for employers. Most sector initiatives consider both qualitative measures that can be gathered by survey or testimony, and quantitative measures. The Aspen Institute Workforce Strategies Institute offers one template, the Business Value Assessment that some states and local areas are using as a guide for measuring impact on employers. The Center for Energy Workforce Development has published a toolkit to help energy companies measure the impact of their workforce development activities, and much of this content is applicable to companies in other industries as well.

Consider that your evaluation methods can be as simple or complicated as you want them to be, and that sometimes the best way to assess value to participating employers is to ask them directly: Is this working for you? Is this valuable? 

Impact on Workers and Jobseekers

Sector initiatives have proven to have substantial benefits to workers and job seekers. These include increased rates of employment, higher salaries, better benefits, increased hours, increased job satisfaction, improved career advancement prospects, and increased access to education. To realize these types of outcomes, your sector initiative will need to define success for workers and jobseekers early in your efforts, and design a mechanism to track and collect relevant data. 

Role of and Impact on Education and Training Systems

Part of the long-term outcomes of a sector initiative is how education and training systems participate in and are impacted by the activities of the effort. A useful question to ask is: Do educators agree that the partnership is or will help them to meet employers’ and students’ needs better? If so, consider that the partnership has impacted them, and that their role is critical to the initiative's success. Also consider the strength and flexibility of the services provided by the training institutions - including new or updated skills standards, the development of career pathways, new coursework or credentials, changes to student support policies, increased enrollment in industry-relevant curricula, modularization of curricula, increased retention and graduation rates, delivery of training off campus, enhanced e-learning opportunities, etc. 

Effectiveness of the Partnership

Every sector initiative must self-assess its effectiveness as a partnership. This might include: 

  • Evidence of appropriate partners convening, and identifying industry challenges and solutions.
  • Self-identified short and long term outcomes measures.
  • Evidence of a plan of action, or a roadmap with direct line of sight to goals and outcomes.
  • All partners agreeing that the Skill Panel will collectively address a wide range of workforce / talent development challenges in the region.
  • New and leveraged funding (sources and $ amounts).

Systems Change

As previously discussed, systems change factors can be part of the long term change that results from your sector iniatiative. These types of changes occur when public systems (such as workforce systems, education institutions or economic development entities) or industries alter their long-term and on-going behavior as a result of the initiative. Some changes might be: 

  • New, re-structured education, training, social, and business supports
  • Changes in public policy
  • Changes in employer practices

Chapter 7 Review Questions

  1. Do you know where you’re going? How will you know when you’re there?
  2. Do you have impacts envisioned for employers? For workers? For public systems?
  3. How are you measuring evidence of progress toward long term goals or impacts? Do you know the types of products or outcomes that will lead to long term goals or impacts?
  4. Do all partners understand where you’re going and how you’ll know you’re there? How can you share this with them effectively?