Program evaluation : ǂb pragmatic methods for social work and human service agencies

Be prepared for your future role in a service-oriented agency. This textbook provides practical guidance on program evaluation while avoiding replicating other course material. Drawing on over 40 years of subject knowledge, Allen Rubin describes outcome designs that are feasible for service-oriented agencies and that match the degree of certainty needed by key users of outcome evaluations. The utility and easy calculation of within-group effect sizes are outlined, which enhance the value of evaluations that lack control groups. Instructions are also given on how to write and disseminate an evaluation report in a way maximizes its chances of being used. Conducting focus group interviews and capitalising on the value of non-probabilitysamples will become second nature after following the effective and pragmatic advice mapped out chapter-by-chapter. Half title Title page Imprints page Contents in Brief Contents in Detail Figures Tables Preface Key Challenges for Program Evaluation Instructors and Students with Which This Book Aims to Help Organization Pedagogical Features Supplemental Resources Acknowledgments Part I Introduction Chapter 1 Introduction and Overview Chapter Outline 1.1 Introduction 1.2 Why Evaluate? 1.3 Some Programs are Ineffective or Harmful Critical Incident Stress Debriefing. Scared Straight Programs. 1.4 Historical Overview of Program Evaluation 1.5 Evidence-Informed Practice 1.6 Philosophical Issues: What Makes Some Types of Evidence Better Than Other Types? Contemporary Positivism. Interpretivism. Empowerment. Constructivism. 1.7 Qualitative versus Quantitative Evaluations: A False Dichotomy 1.8 Definitions 1.9 Different Evaluation Purposes 1.10 Types of Evaluation Summative Evaluation. Formative Evaluation. Process Evaluation. Performance Measurement Systems. Evaluating One’s Own Practice. Accreditation. 1.11 Chapter Main Points 1.12 Exercises 1.13 Additional Reading Chapter 2 Ethical and Cultural Issues in Program Evaluation Chapter Outline 2.1 Introduction 2.2 Ethical Issues 2.3 Institutional Review Boards (IRBs) 2.4 Culturally Sensitive Program Evaluation 2.4.1 Recruitment 2.4.2 Retention 2.4.3 Data Collection 2.4.4 Analyzing and Interpreting Evaluation Findings 2.4.5 Measurement Equivalence 2.5 Developing Cultural Competence 2.5.1 Acculturation and Immigration 2.5.2 Subgroup Differences 2.5.3 Culturally Sensitive Data Analysis and Interpretation 2.6 Chapter Main Points 2.7 Exercises 2.8 Additional Reading Part II Quantitative and Qualitative Methods for Formative and Process Evaluations Chapter 3 Needs Assessment Chapter Outline 3.1 Introduction 3.2 Defining Needs: Normative Need versus Felt Need 3.3 Felt Need versus Service Utilization 3.4 Needs Assessment Approaches 3.4.1 Social Indicators Advantages/Disadvantages. 3.4.2 Rates under Treatment Advantages/Disadvantages. 3.4.3 Key Informants Advantages/Disadvantages. 3.4.4 Community Forums Advantages/Disadvantages. 3.4.5 Focus Groups How to Conduct a Focus Group. Types and Sequence of Focus Group Questions. Advantages/Disadvantages. 3.4.6 Community Surveys Advantages/Disadvantages. 3.5 Triangulation 3.6 Chapter Main Points 3.7 Exercises 3.8 Additional Reading Chapter 4 Survey Methods for Program Planning and Monitoring Chapter Outline 4.1 Introduction 4.2 Samples, Populations, and Representativeness Probability Sampling. Non-probability Samples. 4.2.1 Non-response Bias 4.2.2 Sample Size Maximizing Response Rates. Follow-ups. 4.3 Recruiting Hard-to-Reach Populations Tactics for Reaching and Recruiting Millennials. 4.4 Survey Modalities 4.5 Interviews Be Prepared. Professional Demeanor. Be Punctual. Starting the Interview. Note Taking. Use Neutral Probes. 4.6 Interview Guides 4.7 Client Satisfaction Surveys Limitations. 4.8 Survey Questionnaire Construction 4.8.1 Guidelines for Item Wording 4.8.2 Guidelines for Questionnaire Format 4.9 Online Survey Questionnaire Preparation 4.10 Chapter Main Points 4.11 Exercises 4.12 Additional Reading Part III Evaluating Outcome in Service-Oriented Agencies Chapter 5 Selecting and Measuring Outcome Objectives Chapter Outline 5.1 Introduction 5.2 Mission and Vision Statements 5.3 Logic Models 5.4 Stakeholder Goals 5.5 Triangulation 5.6 How to Write Good Program Outcome Objectives 5.7 Operationally Defining Objectives 5.7.1 Direct Observation 5.7.2 Self-Report 5.7.3 Available Records 5.8 How to Find and Select the Best Self-Report Outcome Measures 5.9 Criteria for Selecting a Self-Report Outcome Measure 5.9.1 Relevance 5.9.2 Feasibility 5.9.3 Reliability 5.9.4 Validity 5.9.5 Sensitivity 5.10 Chapter Main Points 5.11 Exercises 5.12 Additional Reading Chapter 6 Inference and Logic in Pragmatic Outcome Evaluation: Don’t Let the Perfect Become the Enemy of the Good Chapter Outline 6.1 Introduction 6.2 Causality Criteria Revisited 6.2.1 Correlation 6.2.2 Time Sequence 6.2.3 Ruling Out Alternative Explanations 6.3 Implications of Evidence-Informed Practice and Critical Thinking 6.4 A Caveat 6.5 A Successful Evaluator Is a Pragmatic Evaluator 6.6 Degree of Certainty Needed 6.7 Chapter Main Points 6.8 Exercises 6.9 Additional Reading Chapter 7 Feasible Outcome Evaluation Designs Chapter Outline 7.1 Introduction 7.2 Descriptive Outcome Evaluations 7.3 One-Group Pretest–Posttest Designs 7.4 Effect Sizes 7.4.1 Between-Group Effect Sizes 7.4.2 Within-Group Effect Sizes 7.5 Non-equivalent Comparison Groups Designs 7.6 Selectivity Biases 7.7 Switching Replication Design 7.8 Switching Replication Design Compared with Waitlist Quasi-experimental Design 7.9 Time-Series Designs 7.10 Choosing the Most Appropriate Design 7.11 Chapter Main Points 7.12 Exercises 7.13 Additional Reading Chapter 8 Single-Case Designs for Evaluating Programs and Practice Chapter Outline 8.1 Introduction 8.2 What Is a Single Case? 8.3 Overview of Single-Case Design Logic for Making Causal Inferences Clinical Significance. 8.4 What to Measure and by Whom? 8.4.1 Obtrusive and Unobtrusive Observation 8.4.2 Quantification Options 8.5 Baselines Are Baselines Ethical? 8.6 Alternative Single-Case Designs 8.6.1 The AB Design 8.6.2 The ABAB Design 8.6.3 The Multiple-Baseline Design 8.6.4 Multiple-Component Designs 8.7 B Designs to Evaluate the Implementation of Evidence-Supported Interventions 8.8 Using Single-Case Designs as Part of the Evidence-Informed Practice Process 8.9 Aggregating Single-Case Design Outcomes to Evaluate an Agency 8.10 Chapter Main Points 8.11 Exercises 8.12 Additional Reading Chapter 9 Practical and Political Pitfalls in Outcome Evaluations Chapter Outline 9.1 Introduction 9.2 Practical Pitfalls 9.2.1 Intervention Fidelity 9.2.2 Contamination of the Case Assignment Protocol 9.2.3 Recruiting Participants 9.2.4 Retaining Participants 9.3 Engage Agency Staff Meaningfully in Planning the Evaluation 9.4 Fostering Staff Compliance with the Evaluation Protocol Goes On and On 9.5 Political Pitfalls 9.5.1 In-House versus External Evaluators 9.6 Conclusion 9.7 Chapter Main Points 9.8 Exercises 9.9 Additional Reading Part IV Analyzing and Presenting Data Chapter 10 Analyzing and Presenting Data from Formative and Process Evaluations Chapter Outline 10.1 Introduction 10.2 Quantitative and Qualitative Data Analyses: Distinctions and Compatibility 10.3 Descriptive Statistics 10.3.1 Frequency Distribution Tables and Charts 10.3.2 Central Tendency 10.3.3 Dispersion 10.3.4 The Influence of Outliers 10.4 Analyzing Qualitative Data 10.4.1 Coding 10.5 Chapter Main Points 10.6 Exercises 10.7 Additional Reading Chapter 11 Analyzing Data from Outcome Evaluations Chapter Outline 11.1 Introduction 11.2 Inferential Statistics 11.2.1 p Values and Significance Levels 11.2.2 Type II Errors 11.3 Mistakes to Avoid When Interpreting Inferential Statistics 11.3.1 Overreliance on Statistical Significance 11.3.2 Disregarding Sample Size (Statistical Power) 11.3.3 Disregarding Effect Sizes 11.4 Calculating and Interpreting Effect Sizes 11.4.1 Within-Group Effect Sizes Step 1. Step 2. Step 3. Step 4. Step 5. 11.4.2 Between-Group Effect Sizes 11.4.3 Why Divide by the Standard Deviation? 11.4.4 A Caution 11.4.5 Odds Ratios and Risk Ratios Odds Ratios. Risk Ratios. 11.5 Overlooking Substantive (Practical) Significance 11.6 Cost-Effectiveness and Cost–Benefit Analyses: Evaluating Efficiency 11.7 Qualitative Data Analysis 11.8 Chapter Main Points 11.9 Exercises 11.10 Additional Reading Chapter 12 Writing and Disseminating Evaluation Reports Chapter Outline 12.1 Introduction 12.2 Tailor to Your Audience 12.3 Writing Style and Format 12.4 Involve Key Stakeholders 12.5 Ethical Issues 12.6 Report Components 12.6.1 Executive Summary 12.6.2 Introduction and Literature Review 12.6.3 Methodology 12.6.4 Results (Findings) Infographics. 12.6.5 Discussion Discussing Negative Findings. What If Parts of the Evaluation Could Not Be Completed? 12.6.6 References 12.6.7 Appendices 12.7 Summary of Mistakes to Avoid 12.8 Dissemination 12.9 Chapter Main Points 12.10 Exercises 12.11 Additional Reading Epilogue More Tips for Becoming a Successful Evaluator Planning the Evaluation Levels of Stakeholder Participation. Obtain Feedback to a Written Draft of the Evaluation Protocol. During Implementation of the Evaluation At the Conclusion of the Evaluation People Skills Show Genuine Interest in Others. Try to Be Humorous. Be Self-Assured. Show Genuine Empathy. Active Listening. References Index "During the nearly half a century that I have been teaching courses on program evaluation I have not found a textbook that fit my course as well as I preferred. The main problem has been that some books cover too much research methods content that duplicates what students learn in their research methods courses, while other books don't have enough of that content. The problem is understandable, because program evaluation is carried out using research methods. Another reason why I have been dissatisfied with the books with heavy doses of research content is that so much of that content is pitched to wellfunded evaluations of national and international projects for which program evaluation is a key component of the program from its outset. Very sophisticated research designs are not only feasible in such projects, they are expected. Program evaluation in local social and human service agencies - where most of my students begin their careers - is unlike program evaluation in those projects in several important ways"-- Provided by publisher