Summary of Key Elements in Using Theory and Research to Improve Evidence-Based Practice
Patrick J. Rottinghaus
University of Missouri-Columbia
This chapter appears in J. P. Sampson, E. Bullock-Yowell, V. C. Dozier, D. S. Osborn, & J. G. Lenz (Eds.), Integrating theory, research, and practice in vocational psychology: Current status and future directions (pp. 130-136). Tallahassee, FL: Florida State University. doi:10.17125/svp2016.ch12
The chapter is based on a paper presented at the conference of the Society of Vocational Psychology, May 16–17, 2016, Florida State University, Tallahassee, FL. A video presentation of a preliminary version of this paper may be retrieved from https://www.youtube.com/watch?v=ZKFnjNzhYa0&index=17&list=PLCsXiACK-LKrVRKb1z_Nw3METZg-SoRFA
Correspondence should be addressed to Patrick J. Rottinghaus, Educational, School, and Counseling Psychology, 16 Hill Hall, University of Missouri, Columbia, MO 65211. E-mail: email@example.com
Keywords: evidence-based practice, meta-analysis, outcomes, career interventions, effectiveness, career development, career counseling
This section covered a broad range of concerns related to the integration of theory, research, and practice, including implications for collaboration and policy. Theory-driven work is paramount in vocational psychology and should guide research and practice initiatives. This set of chapters features several ongoing challenges and opportunities that can facilitate the translation of scholarship to inform the development of evidence-based interventions that matter to practice settings and policy decisions. First, Nadya Fouad (2017) examines the overall role of theory and how it can guide efforts to minimize the gap between science and practice. Next, chapters by Steve Brown (2017), Susan Whiston (2017), and Betsy Becker (2017) offer various perspectives on the state of the art in conducting meta-analyses on the outcomes of career development interventions. Finally, Scott Solberg (2017) offers a personal reflection on efforts to link vocational psychology scholarship to specific policy initiatives that require interdisciplinary collaborations. This chapter summarizes important highlights and themes from these chapters, and concludes with five strategies for purposefully connecting theory and research to strengthen evidence-based practice.
Fouad (2017) highlights personal reflections on her experiences bridging gaps between theory, research, and practice. As an important reminder to students, even distinguished leaders in our field once struggled to know how to apply theories to working with actual clients. Fortunately, theories can work as guiding maps to assist counselors in conceptualizing clients’ concerns. Swanson and Fouad’s (2015) career text focuses on applying career theories to case studies beyond presenting theories as separate from application with clients. Additionally, Fouad (2017) noted that ongoing attempts to bring together researchers and practitioners can be difficult and requires collaborations involving teams of experts from practice and science settings. Drawing on her experiences editing leading applied psychology journals, Fouad shared examples of creative attempts to bridge the science-practice gap. For example, she initiated the Practitioner Forum during her time as the editor of The Counseling Psychologist. Again, hearing a frank discussion that normalizes the challenges our leading scholars face can help future professionals have the courage to build on these efforts by modifying approaches to transferring knowledge between science and practice, guided by theory.
A crucial line of research on career-related outcomes has been the series of initial meta-analyses that demonstrated the effectiveness of career interventions (Oliver & Spokane, 1988; Spokane & Oliver, 1983). Brown (2017) elaborated on three recent meta-analyses that have built upon Oliver and Spokane’s fundamental work, including Whiston, Sexton, and Lasoff’s (1998) study that found individual counseling was more effective than group modalities. Brown and Ryan Krane (2000) identified common critical ingredients [End Chapter, Page 130] (i.e., workbooks and written exercises, individualized interpretations and feedback, in-session exploration of occupational information, modeling, and building support). Liu, Huang, and Wang (2014) examined the effectiveness of job search interventions. From these recent investigations, Brown (2017) highlighted three main findings:
“Career interventions are modestly more effective than doing nothing in promoting outcomes associated with both choice-making and job finding” (p. 83). Based on results from Whiston et al. (1998) and Brown and Ryan Krane (2000), the 95% confidence interval for true effect sizes range from .26 to .36 for choice making difficulty interventions. Liu et al. (2014) used a random effects model, showing that participants in job search interventions yielded 2.67 times greater odds of obtaining jobs than control groups.
“There are several intervention components that seem to be important to outcomes for both choice-making and job finding interventions” (p. 83). Brown highlighted his influential work with Nancy Ryan Krane, demonstrating the five critical intervention components. In addition, Liu et al. (2014) found that the following six components enhance the likelihood of obtaining a job: teaching job search skills, improving self-presentation, increasing self-efficacy, encouraging proactivity, promoting goal setting, and enlisting social support. The focus on job search counseling is not sufficiently attended to in vocational psychology. This is a common reason people seek career counseling services and is a focus of practice. Collaborations between researchers and practitioners could strengthen the literature investigating job seeking strategies and could draw connections to ongoing scholarship of vocational psychology, including vocational interests (Dik & Rottinghaus, 2013), career decision-making (Gati, 2013), and so forth. Moreover, the proactivity aspect struck me as a keystone and as connected to Lent’s (2013) related term, career-life preparedness, given its importance for career planning and adjustment to a changing world of work. According to Lent (2013), preparedness involves “a healthy state of vigilance regarding threats to one’s career well-being as well as alertness to resources and opportunities on which one can capitalize” (p. 7). A whole set of emerging terms, including proactivity, preparedness, agency, and adaptability, represent a nexus of ideas related to intentionality and the core value of preventative perspectives in counseling psychology. Ultimately, professionals need to embolden clients to capitalize on opportunities and overcome setbacks to build resources of KSAOs through education, experience, and reflection on opportunities in order to best navigate the educational and occupational worlds.
“Goal setting, support building, and efficacy enhancement strategies are critical components in both choice-related and job-finding interventions and may represent today the clearest, meta-analytically derived evidence-based practice guidelines that we have for career practitioners” (p. 84). These components are notable in that they emerged as critical across job search and career counseling interventions.
Brown (2017) then highlights three key areas that our field still needs to examine.
“We need to identify core outcome constructs” (p. 84). Brown points out a number of measurement concerns related to outcome measures, underscoring that “measures of constructs are not the constructs themselves” (p. 84). Whiston’s (2001) call for a core battery to address outcomes is related to this, but Brown reminds us to critically examine shared variance. Fundamental research that clarifies nuances among career assessments must receive significant attention while concurrent outcomes research proceeds to move this literature forward. In addition to traditional outcomes, Solberg (2017) underscores the need to attend to other outcomes beyond career exploration, vocational identity, career maturity, decisional status, and satisfaction, including those relates to college and career readiness.
“We need to know whether career interventions actually make a difference in the lives of our clients” (p. 85). Beyond statistical significance, [End Page 131] researchers must attend to the clinical significance of interventions (Brown, 2014). Routine use of alternative methods such as those presented by Jacobson and Truax (1991) can afford opportunities for future meta-analytic investigations of clinical significance. Ultimately, the field needs to increase evidence for the effectiveness across career interventions that is relevant to clients, organizations, policymakers, parents, and other stakeholders.
“We need to conduct outcome research with more diverse samples” (p. 87). This is a frequent call among experts that remains a top priority for the field (Fouad & Kantamneni, 2008; Whiston & Rose, 2015). In the context of discussing meta-analytic research, this is a reminder that the primary studies are the foundation to the entire career development outcomes literature. It is paramount to investigate the effectiveness of interventions for an array of outcomes across diverse samples (e.g., ethnicity, age) and settings (e.g., type of school, region, economic context).
Next, Whiston (2017) expands on these concerns by examining the relative merits of conducting general meta-analyses versus more narrowly defined studies focused on specific populations or settings. Beyond the ongoing need for more studies with racially and ethnically diverse clients, Whiston calls for more studies examining gender and SES. Drawing on the psychotherapy research literature, she also recommends specific studies examining common career-related presenting concerns such as career decision making and work-family conflict, analogous to manualized treatment interventions for anxiety disorders. This more specific approach could mirror the empirically-supported treatment investigations to identify specific intervention components that are more pivotal to achieving successful outcomes that may differ depending on the client’s specific career problem and level of readiness for counseling. Although noting significant challenges to conducting setting-specific meta-analyses, she underscored the potential benefits of such important work, including the capability to identify key interventions that have the largest effect-sizes, which also support accountability to funding sources (e.g., university administrators).
General meta-analyses can help researchers cast a wide net to enhance the robustness of the study. However, this overall approach may mask important conclusions. Coding for specific study characteristics can allow for a more refined attention to testing possible moderators. Whiston (2017) concludes with a call for more systematic research on career interventions. Both general and specific meta-analyses are needed, with particular attention to moderators in order to examine specialized needs across various populations and settings.
Becker (2017) calls for greater attention to using theory to build an evidence base for practice and offers a wealth of information on technical aspects of meta-analyses. She highlights fundamental steps for conducting meta-analyses, including problem formulation and attending to the degree of variability across analyzed studies. By incorporating client-attribute interaction and other moderators into studies, researchers can gain a more refined understanding of treatment effects. She encourages the use of theory-driven meta-analytic path models that can be supplemented with process and treatment variables to identify gaps in the literature and to inform future studies.
To aid in formulating meta-analytic studies, she encourages a strategic organizing framework, MUTOS (Method, Units, Treatments, Observing Operations, and Setting), that was modified from Cronbach’s (1982) UTOS framework. MUTOS demarcates inclusion rules and features of studies that will be analyzed to identify their potential relevance to the particular outcome of interest. The MUTOS framework affords critical attention to possible moderators that are central to differences in results for career intervention outcomes investigations. The M component incorporates the range of methods used across studies included in meta-analyses, enabling a focused analysis of variability regarding client sampling, condition assignment, treatment of attrition, and other key aspects analyzed. Becker (2017) provides more details related to the statistical complexity. The upshot is MUTOS enables researchers to gauge core features related to the outcome(s) of interest, take stock of potential moderators, and strategically examine the generalizability of results for particular groups, presenting concerns, and settings.
Becker (2017) concluded with a call for a shared database (cf. Baldwin & Del Re, 2016) that provides [End Page 132] effect sizes of various career interventions. This effort could help organize what is known (and unknown) about the degrees of effectiveness of intervention components to inform theory-building as well as guide practice and future research.
Finally, Scott Solberg (2017) offers personal reflections on decades of experience conducting community-based participatory research in schools. His passion for applying career development science to school settings highlights the impact of vocational psychology scholarship. Solberg’s practical strategies for building credibility and establishing egalitarian partnerships with school administrators offer useful strategies to achieve results that matter to communities. In particular, he highlights the need to clarify roles, responsibilities, and objectives throughout the stages of conducting research that can be facilitated by project management systems. This overall process takes courage and patience to establish trust between collaborators, identify needs, and nurture relationships over time to achieve results that inform policy efforts.
Solberg (2017) focuses on relevance and the need for conducting research that connects with larger state and national level concerns. His reflections on several creative collaborations demonstrate how to establish scalable programs that map onto broader initiatives. For example, his ongoing Success Highways project focuses on interventions to build social-emotional learning in support of academic success. Recently, his work on the Massachusetts Institute for College and Career Readiness, in partnership with several agencies, brings together researcher-practitioner partnerships in 15 school districts throughout Massachusetts to address challenges districts face in preparing students for postsecondary transitions. Solberg shares examples of how he has learned to design, implement, and evaluate efforts for schools to facilitate pathways to successful outcomes, including academic remediation, whole school reform, workforce readiness, and so forth. This is a reminder of the importance of career development to outcomes within a national context, such as systemic concerns related to graduation rates, college and career readiness, reducing college student debt, employability, and so forth. We need to increase the evidence base for the effectiveness of career interventions that support positive outcomes for clients and key stakeholders, including schools, policymakers, and parents.
Conclusions and Future Recommendations
It has been a privilege to summarize this set of quality chapters from leading scholars on conducting evidence-based outcomes research in vocational psychology. Since meta-analytic investigations are critical to discovering what works with clients, these papers focus on landmark meta-analyses that demonstrate the overall effectiveness and specific critical components of effective career interventions. Pulling back to the big picture at this meta-meta level provides an opportunity to reflect on broad-based recommendations to push the field forward. The following are five take home points for enhancing evidence-based practice and capturing the dynamic interaction between theory, research, and practice.
Nurture ongoing collaborations with researchers and practitioners. Fouad (2017) and Solberg (2017) elaborated on the importance of building meaningful relationships between experts from diverse professional backgrounds. The challenges faced by professionals in schools and other practice settings inherently require interdisciplinary perspectives. Instead of attempting to carve out particular facets within our own separate subdisciplines, which inherently draws on limited perspectives and methods, it is wise to incorporate expertise from those who have faced similar methodological and practical challenges. We need not expect to go it alone! Each of us has particular strengths and perspectives that can inform the broader literature. Organizations such as SVP and NCDA can facilitate connections between professionals with complementary skills who can work together to explore grants that fund research and create strategic dialogs for advancing evidence-based practice.
Conduct more intentional studies that examine moderators and applications of theory for diverse groups. Building on the knowledge from prominent meta-analyses of career outcomes concluding that career interventions are effective, there is a critical need to conduct more primary intervention research (Whiston, 2017), especially across diverse samples (Brown, 2017) involving a range of presenting concerns and [End Page 133] settings. Becker (2017) featured the MUTOS method that enables researchers to examine sources of variability in outcomes across studies, including potential moderators involving client characteristics and features of interventions. Indeed, coordinated efforts to understand what works for whom (Frank & Frank, 1993) under what conditions should be applied more vigorously to career development concerns. As Gordon Paul (1967) astutely asked: “What treatment, by whom, is most effective for this individual with that specific problem, and under which set of circumstances?” (p. 111). Related to Point 1, interdisciplinary teams can offer insights for studying necessary components that support more individualized treatments and their potential scalability. These complex endeavors require an intentional focus on key problems and how they might differ based on the particular needs experienced by diverse clientele across various settings.
Create a database of constructs, measures, and research studies as part of an interactive diagnostic system that informs empirically-supported interventions for particular presenting concerns, populations, and settings. This ambitious goal relates to a broader concern: Our field needs to create an organized set of specialized career assessments that are readily available for research and practice purposes. Progress toward this goal would help identify critical factors involved in detecting meaningful change for diverse career counseling objectives. Vocational psychologists can collaborate and bring together pertinent knowledge that is empirically supported and informed by interdisciplinary perspectives on a wide array of career concerns across fields. A community of researchers and practitioners can benefit from a centralized system that not only organizes the existing research, but also brings professionals together to address common goals related to advancing scientifically informed career development interventions. Such a large, collaborative effort can be facilitated by strategic leadership from SVP, NCDA, and funding from state and federal agencies.
Attend to a broader array of outcomes from career development interventions. There are numerous specific outcomes of career interventions that are important to clients beyond traditional outcomes (e.g., decidedness, satisfaction), and these papers highlight the importance of clinical significance and achieving broader life goals such as college and career readiness, subjective well-being, and decent work. In an effort to understand the practical utility of career interventions beyond statistical significance, the field can further enhance connections between research and practice.
Consider implications for the education and training of students and professional development needs for professionals. The connections between science and practice lie at the heart of training programs and are reflected in the emphasis on evidence-based practice. Similar to the general psychotherapy literature’s focus on addressing the effectiveness and efficacy of treatments for particular disorders (e.g., Nathan & Gorman, 2015; Roth & Fonagy, 2005), vocational psychology and career development training programs attend to the scientific base of our interventions. The chapters in this section highlight current knowledge in the vocational psychology and career development fields, while acknowledging key gaps for particular career concerns (e.g., job seeking and life satisfaction) and the efficacy of career interventions for diverse populations. Overall, these efforts must be expanded to create a more specialized set of recommendations informed by rigorous science. In addition to conveying knowledge of career counseling effectiveness studies and research skills to our students, we must establish coordinated systems and resources to translate ideas from the ongoing scientific literature to practice and advocacy efforts. This book serves as a stimulus for vocational psychology and career development professionals that ultimately can advance our efforts to establish quality interventions that meet the diverse needs of individuals in our schools and communities.
Baldwin, S. A., & Del Re, A. C. (2016). Open access meta-analysis for psychotherapy research. Journal of Counseling Psychology, 63(3), 249–260. http://doi.org/10.1037/cou0000091
Becker, B. J. (2017). Improving the design and use of meta-analyses of career interventions. In J. P. Sampson, E. Bullock-Yowell, V. C. Dozier, D. S. Osborn, & J. G. Lenz (Eds.), Integrating theory, research, and practice in vocational psychology: Current status and future directions (pp. 95). Tallahassee, FL: Florida State University. Retrieved from https://doi.org/10.17125/svp2016.ch10
Brown, S. D. (2014). Career intervention efficacy. In P. J. Hartung, M. L. Savickas, & W. B. Walsh (Eds.), APA handbook of career intervention (Vol. 1, pp. 61–77). Washington, DC: American Psychological Association.
Brown, S. D. (2017). Meta-analysis and evidence-based career practice: Current status and future directions. In J. P. Sampson, E. Bullock-Yowell, V. C. Dozier, D. S. Osborn, & J. G. Lenz (Eds.), Integrating theory, research, and practice in vocational psychology: Current status and future directions (pp. 82). Tallahassee, FL: Florida State University. Retrieved from https://doi.org/10.17125/svp2016.ch8
Brown, S. D., & Ryan Krane, N. E. (2000). Four (or five) sessions and a cloud of dust: Old assumptions and new observations about career counseling. In S. D. Brown & R. W. Lent (Eds.), Handbook of counseling psychology (3rd ed., pp. 740–766). New York: Wiley.
Cronbach, L. J. (1982). Designing evaluations of educational and social programs. San Francisco: Jossey-Bass.
Dik, B. J., & Rottinghaus, P. J. (2013). Assessments of interests. In K. F. Geisinger (Ed.), American Psychological Association handbook of testing and assessment in psychology (vol. 2, pp. 325–348). Washington, DC: American Psychological Association.
Fouad, N. A. (2017). The role of theory in improving evidence-based career interventions. In J. P. Sampson, E. Bullock-Yowell, V. C. Dozier, D. S. Osborn, & J. G. Lenz (Eds.), Integrating theory, research, and practice in vocational psychology: Current status and future directions (pp. 77). Tallahassee, FL: Florida State University. Retrieved from https://doi.org/10.17125/svp2016.ch7
Fouad, N. A., & Kantamneni, N. (2008). Contextual factors in vocational psychology: Intersections of individual, group, and societal dimensions. In S. D. Brown & R. W. Lent (Eds.), Handbook of counseling psychology (4th ed., pp. 408–425) New York: Wiley.
Frank, J. D., & Frank, J. B. (1993). Persuasion and healing: A comparative study of psychotherapy. JHU Press.
Gati, I. (2013). Advances in career decision making. In W. B. Walsh, M. L. Savickas, & P. J. Hartung (Eds.), Handbook of vocational psychology (4th ed., pp. 183–215). New York: Routledge Press.
Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59, 12–19. http://doi.org/10.1037/10109-042
Lent, R. W. (2013). Career-life preparedness: Revisiting career planning and adjustment in the new workplace. The Career Development Quarterly, 61, 2–14. http://doi.org/10.1002/j.2161-0045.2013.00031.x
Liu, S., Huang, J. L., & Wang, M. (2014). Effectiveness of job search interventions: A meta-analytic review. Psychological Bulletin, 140, 1009–1041. http://doi.org/10.1037/a0035923
Nathan P. E., & Gorman J. M. (2015). A guide to treatments that work (4th ed.). New York: Oxford University Press.
Oliver, L. W., & Spokane, A. R. (1988). Career-intervention outcome: What contributes to client gain? Journal of Counseling Psychology, 35, 447–462. http://doi.org/10.1037//0022-0188.8.131.527
Paul, G. L. (1967). Strategy of outcome research in psychotherapy. Journal of Consulting Psychology, 31, 109–118. http://doi.org/10.1016/b978-0-08-019506-3.50019-6
Roth A. D., & Fonagy P. (2005). What works for whom: A critical review of psychotherapy research (2nd ed.). New York: Guilford.
Solberg, V. S. H. (2017). Personal reflections on a career spent creating and sustaining researcher-practitioner collaborations. In J. P. Sampson, E. Bullock-Yowell, V. C. Dozier, D. S. Osborn, & J. G. Lenz (Eds.), Integrating theory, research, and practice in vocational psychology: Current status and future directions (pp. 108). Tallahassee, FL: Florida State University. Retrieved from https://doi.org/10.17125/svp2016.ch11
Spokane, A. R., & Oliver, L. W. (1983). Outcomes of vocational intervention. In S. H. Osipow & W. B. Walsh (Eds.), Handbook of vocational psychology (pp. 99–136). Hillsdale, NJ: Erlbaum.
Swanson, J. L., & Fouad, N. A. (2015) Career theory and practice: Learning through cases. Thousand Oaks, CA: Sage (3rd ed.). Thousand Oaks, CA: Sage.
Whiston, S. C. (2001). Selecting career outcome assessments: An organizational scheme. Journal of Career Assessment, 9, 215–228. http://doi.org/10.1177/106907270100900301
Whiston, S. C. (2017). Conducting general versus population and setting-specific meta analyses. In J. P. Sampson, E. Bullock-Yowell, V. C. Dozier, D. S. Osborn, & J. G. Lenz (Eds.), Integrating theory, research, and practice in vocational psychology: Current status and future directions (pp. 90). Tallahassee, FL: Florida State University. Retrieved from http://doi.org/10.1037//0022-0184.108.40.206
Whiston, S. C., & Rose, C. S. (2015). Career counseling process and outcome. In P. J. Hartung, M. L. Savickas, & W. B. Walsh (Eds.), American Psychological Association handbook of career intervention (Vol. 1, pp. 43–60). Washington, DC: American Psychological Association.
Whiston, S. C., Sexton, T. L., & Lasoff, D. L. (1998). Career-intervention outcome: A replication and extension of Oliver and Spokane (1988). Journal of Counseling Psychology, 45, 150–165. http://doi.org/10.1037//0022-0220.127.116.11