Small-Scale Evaluation in Health : A Practical Guide

Small-Scale Evaluation in Health is an easy-to-use guide that sets out the basics of designing, conducting, and analyzing an evaluation study in healthcare settings from start to finish. The authors take a straight-forward, practical approach, assuming no previous knowledge or experience of evaluation. All the basics are covered, including how to plan an evaluation; research Governance and ethics; understanding data; interpreting findings; and writing a report. Cover 1 Title Page 4 Copyright 5 Contents 6 List of figures 7 List of case studies 8 Acknowledgements 10 About this book 11 Author biographies 12 1. What is an evaluation? 14 What is an evaluation? 14 What do we mean by small-scale evaluation? 14 Why evaluate? 15 What shapes an evaluation? 18 What evaluation can and cannot do 19 Who should do the evaluation? 20 How do we do an evaluation? 21 Summary 21 Further reading 23 2. Searching and reviewing the literature 24 Why do a search of the literature? 24 What does searching the literature involve? 26 Thinking about the topic 26 Places to look 28 Doing the search 28 Examining the results of the search 30 Feeding the search findings into the evaluation plan 31 How to reference the results of the search 33 Harvard (author-date) style 33 Vancouver (author-number) style 33 Summary 34 Further reading 36 3. Evaluation design 37 Purpose of the evaluation 37 Evaluating from within? 38 Types of evaluation – what are we trying to find out? 39 Evaluation methods 45 Is something effective? 45 Issues to consider when designing the evaluation 46 Making sure the data are reliable: repeatability 47 Making sure the data are valid: bias 47 Making sure the data are representative and generalisable: sampling 48 Summary 51 Further reading 53 4. How to plan an evaluation 54 Why plan? 54 What does a plan do? 55 How do we go about planning? 55 Walking through our plan 57 What should go into our evaluation protocol? 58 Title 58 Evaluation team 59 Acknowledgement of funding 60 Background 60 Evaluation aim and objectives 61 Evaluation design/methods 62 Timescales 66 Resources 68 Project management 68 Outputs 68 References 68 Using our protocol to guide the evaluation 69 Common pitfalls 69 Summary 70 Further reading 72 5. Research governance and ethics 73 Research governance 73 Ethics 73 Science 74 Information 75 Health and safety 75 Finance 75 The ethics process 76 Basic requirements for ethics 76 Types of studies needing approval 77 Reasons why applications to ethics may be rejected or require amendments 81 The patient information sheet 81 Patients’ rights are not being protected 82 The scientific value or methods are poor 82 Data confidentiality 82 Consent for children and adults unable to give consent 82 Conflicts of interest 83 R&D approval 83 Applying for approval 83 Summary 84 6. Collecting data 87 Using existing information 87 Types of existing data and how to access them 90 Electronic databases 90 Clinical notes 90 Meeting minutes, existing studies, policies and protocols 91 Video/audio recordings 91 How to use existing data 92 Collecting new data 92 Interviews 93 Dos and dont’s of interview schedule design 95 Observing practice 96 Questionnaires 98 Other methods 106 Data collection specific to evaluation of costs 107 Validity of the data 109 Summary 109 Further reading 113 7. Understanding numerical data 114 The basic ideas of working with numbers 114 Variation 114 Samples 115 Frequency curves 117 Normal distribution 120 Measures of average 120 Measures of variation 121 Sampling variation 122 The standard error of the mean 123 95 per cent confidence interval 123 Sampling 126 Sample size 126 Summary 130 Further reading 132 8. Storing data 133 Recording and keeping qualitative data 133 Recording and keeping quantitative data 136 What is a spreadsheet and what is a database? 136 Setting up a spreadsheet/database 138 Confidentiality and protecting your database/spreadsheet 140 Summary 140 Further reading 143 9. Analysing data 144 The first step in analysing data is to describe the sample 144 How to analyse qualitative data 145 Structured interviews 145 Semi-structured interviews 146 Focus groups 148 Understanding observation data 149 How to analyse quantitative data 149 The planning stage of the evaluation 149 The basics 151 Parametric tests 153 T-test 153 Correlation 156 Non-parametric tests 161 Chi-squared test 161 Spearman’s rank order correlation 163 Analysis of costs 167 In summary 168 Further reading 170 10. Interpreting the findings 172 Putting findings in context 172 Involving the evaluation team in interpreting findings 172 Qualitative findings 173 Quantitative Research 174 Cause and association 174 Observation without a control group 176 The Hawthorne effect 177 Interpretation based on evidence 177 Summarising and presenting data 178 Tables 178 Graphs 179 Practise your tables and graphs 181 Further reading 183 11. Writing a report 184 What is a report? 184 Some types of reports 184 Basic things to think about before writing the report 184 Writing up 185 Body of the report 186 Presentation and style 188 Dissemination: writing for publication 189 Summary 190 Further reading 192 Appendix I: Example of an evaluation report 195 Appendix II: Answers to case study questions 203 Glossary 208 Index 211 Cover......Page 1 Title Page......Page 4 Copyright......Page 5 Contents......Page 6 List of figures......Page 7 List of case studies......Page 8 Acknowledgements......Page 10 About this book......Page 11 Author biographies......Page 12 What do we mean by small-scale evaluation?......Page 14 Why evaluate?......Page 15 What shapes an evaluation?......Page 18 What evaluation can and cannot do......Page 19 Who should do the evaluation?......Page 20 Summary......Page 21 Further reading......Page 23 Why do a search of the literature?......Page 24 Thinking about the topic......Page 26 Doing the search......Page 28 Examining the results of the search......Page 30 Feeding the search findings into the evaluation plan......Page 31 Vancouver (author-number) style......Page 33 Summary......Page 34 Further reading......Page 36 Purpose of the evaluation......Page 37 Evaluating from within?......Page 38 Types of evaluation – what are we trying to find out?......Page 39 Is something effective?......Page 45 Issues to consider when designing the evaluation......Page 46 Making sure the data are valid: bias......Page 47 Making sure the data are representative and generalisable: sampling......Page 48 Summary......Page 51 Further reading......Page 53 Why plan?......Page 54 How do we go about planning?......Page 55 Walking through our plan......Page 57 Title......Page 58 Evaluation team......Page 59 Background......Page 60 Evaluation aim and objectives......Page 61 Evaluation design/methods......Page 62 Timescales......Page 66 References......Page 68 Common pitfalls......Page 69 Summary......Page 70 Further reading......Page 72 Ethics......Page 73 Science......Page 74 Finance......Page 75 Basic requirements for ethics......Page 76 Types of studies needing approval......Page 77 The patient information sheet......Page 81 Consent for children and adults unable to give consent......Page 82 Applying for approval......Page 83 Summary......Page 84 Using existing information......Page 87 Clinical notes......Page 90 Video/audio recordings......Page 91 Collecting new data......Page 92 Interviews......Page 93 Dos and dont’s of interview schedule design......Page 95 Observing practice......Page 96 Questionnaires......Page 98 Other methods......Page 106 Data collection specific to evaluation of costs......Page 107 Summary......Page 109 Further reading......Page 113 Variation......Page 114 Samples......Page 115 Frequency curves......Page 117 Measures of average......Page 120 Measures of variation......Page 121 Sampling variation......Page 122 95 per cent confidence interval......Page 123 Sample size......Page 126 Summary......Page 130 Further reading......Page 132 Recording and keeping qualitative data......Page 133 What is a spreadsheet and what is a database?......Page 136 Setting up a spreadsheet/database......Page 138 Summary......Page 140 Further reading......Page 143 The first step in analysing data is to describe the sample......Page 144 Structured interviews......Page 145 Semi-structured interviews......Page 146 Focus groups......Page 148 The planning stage of the evaluation......Page 149 The basics......Page 151 T-test......Page 153 Correlation......Page 156 Chi-squared test......Page 161 Spearman’s rank order correlation......Page 163 Analysis of costs......Page 167 In summary......Page 168 Further reading......Page 170 Involving the evaluation team in interpreting findings......Page 172 Qualitative findings......Page 173 Cause and association......Page 174 Observation without a control group......Page 176 Interpretation based on evidence......Page 177 Tables......Page 178 Graphs......Page 179 Practise your tables and graphs......Page 181 Further reading......Page 183 Basic things to think about before writing the report......Page 184 Writing up......Page 185 Body of the report......Page 186 Presentation and style......Page 188 Dissemination: writing for publication......Page 189 Summary......Page 190 Further reading......Page 192 Appendix I: Example of an evaluation report......Page 195 Appendix II: Answers to case study questions......Page 203 Glossary......Page 208 Index......Page 211 Setting out the basics of designing, conducting and analysing an evaluation study in healthcare, the authors take a practical approach, assuming no previous knowledge or experience of evaluation. All the basics are covered, including: - How to plan an evaluation - Research governance and ethics - Understanding data - Interpreting findings - Writing a report Cases included throughout to demonstrate evaluation in action, and self learning courses give the reader an opportunity to develop their skills further in the methods and analysis involved in evaluation. . Setting out the basics of designing, conducting and analysing an evaluation study in healthcare, the authors take a practical approach, assuming no previous knowledge or experience of evaluation. All the basics are covered, including: -How to plan an evaluation -Research governance and ethics -Understanding data -Interpreting findings -Writing a report. Cases included throughout to demonstrate evaluation in action, and self learning courses give the reader an opportunity to develop their skills further in the methods and analysis involved in evaluation Setting out the basics of designing, conducting and analysing an evaluation study in healthcare, the authors of this book take a practical approach, assuming no previous knowledge or experience of evaluation