Virtual standard setting : setting cut scores

"Virtual standard setting became more popular since the global outbreak of Covid-19 in 2020. Standard setting practitioners needed to conduct either cut score studies and/or linking studies online. The research presented in this book predates Covid-19 and explores virtual standard setting in two e-communication media (audio and video) and then compares them to the face-to-face environment. The interplay of quantitative methods [i.e., classical test theory (CTT) and Rasch measurement theory (RMT)] and qualitative methods [(i.e., constant comparative method (CCM), and media naturalness theory (MNT)] unravel Ariadne's thread into the labyrinth of virtual standard setting. Illustrative examples of how to conduct and evaluate a virtual workshop are offered to stimulate standard setting practitioners to embrace the opportunities of the virtual environment"-- Provided by publisher Cover 1 Table of contents 13 List of figures 21 List of tables 23 List of acronyms 25 Chapter 1: Introduction 27 1.1 Overview of the study 27 1.2 Scope of the study 29 1.3 Outline of the chapters 30 Chapter 2: Literature review 33 2.1 Background to standard setting 33 2.2 The importance of setting valid cut scores 35 2.2.1 Standard setting methods 36 2.2.1.1 Examples of test-centred methods 36 Variants of the Angoff method 36 The Bookmark method 38 The Objective Standard Setting (OSS) method 39 2.2.1.2 Examples of examinee-centred methods 41 The Borderline Group (BG) method and the Contrasting Group (CG) method 41 The Body of Work (BoW) method 41 2.2.2 Evaluating and validating standard setting methods 42 2.3 Standard setting in language assessment 44 2.3.1 Current LTA standard setting research 45 2.3.1.1 The first publicly available CEFR alignment studies 45 2.3.1.2 Studies investigating understanding of method or CEFR 46 2.3.1.3 Studies investigating external validity evidence 48 2.3.1.4 Studies proposing new methods/modifications 50 2.4 Challenges associated with standard setting 51 2.4.1 Theoretical and practical challenges 51 2.4.2 Logistics 52 2.5 Virtual standard setting 53 2.5.1 Virtual standard setting: Empirical studies 53 2.5.2 Challenges associated with virtual standard setting 57 2.6 Media naturalness theory 60 2.6.1 Re-evaluating virtual standard setting studies through MNT 61 2.7 Summary 62 Chapter 3: Methodology 65 3.1 Research aim and questions 65 3.2 Methods 66 3.2.1 Embedded MMR design 67 3.2.2 Counterbalanced workshop design 68 3.2.3 Instruments 69 3.2.3.1 Web-conferencing platform and data collection platform 69 3.2.3.2 Test instrument 72 3.2.3.3 CEFR familiarisation verification activities 73 3.2.3.4 Recruiting participants 75 3.2.3.5 Workshop surveys 79 3.2.3.6 Focus group interviews 82 3.2.3.7 Ethical considerations 85 3.3 Standard setting methodology 85 3.3.1 Rationale for the Yes/No Angoff method 85 3.3.2 Pre-workshop platform training 85 3.3.3 In preparation for the virtual workshop 88 3.3.4 Description of the workshop stages 88 3.3.4.1 Introduction stage 89 3.3.4.2 Orientation stage 90 3.3.4.2.1 CEFR familiarisation verification activity A 90 3.3.4.2.2 CEFR familiarisation verification activity B 91 3.3.4.2.3 Familiarisation with the test instrument 93 3.3.4.3 Method training stage 94 3.3.4.4 Judgement stage 95 Round 1 Stage 95 Round 2 Stage 96 Round 3 Stage 98 3.4 Data analysis methods and frameworks 100 3.4.1 CEFR verification activities analysis 101 3.4.2 Internal validity of cut scores 101 Classical test theory (CTT) 101 Rasch measurement theory (RMT) 102 The many-facet Rasch measurement (MFRM) model 104 3.4.3 Comparability of virtual cut score measures 105 3.4.4 Differential severity 106 3.4.5 Survey analysis 107 3.4.6 Focus group interview analysis 107 3.6 Summary 109 Chapter 4: Cut score data analysis 113 4.1 Cut score internal validation: MFRM analysis 113 4.1.1 Rasch group level indices 115 4.1.2 Judge level indices 120 4.2 Cut score internal validation: CTT analysis 125 4.2.1 Consistency within the method 126 4.2.2 Intraparticipant consistency 127 4.2.3 Interparticipant consistency 130 4.2.4 Decision consistency and accuracy 131 The Livingston and Lewis method 131 The Standard Error method 134 4.3 Comparability of cut scores between media and environments 135 4.3.1 Comparability of virtual cut score measures 136 4.3.2 Comparability of virtual and F2F cut score measures 140 4.4 Differential severity between medium, judges, and panels 142 4.4.1 Differential judge functioning (DJF) 142 4.4.2 Differential medium functioning (DMF) 143 4.4.3 Differential group functioning (DGF) 144 4.5 Summary 147 Chapter 5: Survey data analysis 149 5.1 Survey instruments 149 5.2 Perception survey instrument 149 5.2.1 Evaluating the perception survey instruments 150 5.2.2 Analysis of perception survey items 151 Qualitative comments for communication item 1 154 Audio medium 154 Video medium 155 Qualitative comments for communication item 2 158 Audio medium 158 Video medium 159 Qualitative comments for communication item 3 162 Audio medium 162 Video medium 163 Qualitative comments for communication item 4 165 Qualitative comments for communication item 5 167 Audio 167 Video medium 168 Qualitative comments for communication item 6 171 Audio medium 171 Video medium 172 Qualitative comments for communication item 7 175 Audio medium 175 Video medium 175 Qualitative comments for communication item 8 178 Audio medium 178 Video medium 179 Qualitative comments for communication item 9 182 Audio medium 182 The video medium 183 5.3 Procedural survey items 191 5.3.1 Evaluating the procedural survey instruments 191 5.4 Summary 192 Chapter 6: Focus group interview data analysis 195 6.1 Analysis of transcripts 195 6.2 Findings 198 6.2.1 Psychological aspects 198 Distraction in the video medium 198 Self-consciousness in the video medium 199 Lack of non-verbal feedback in the audio medium 199 Inability to distinguish speaker in the audio medium 200 Inability to discern who was paying attention in audio medium 201 Cognitive strain in the audio medium 201 6.2.2 Interaction 202 Lack of small talk in virtual environments 202 No digression from the topic in virtual environments 203 Differences in amounts of discussion between virtual and F2F settings 203 6.2.3 Technical aspects 204 Technical problems in virtual environments 205 Turn-taking system 207 6.2.4 Convenience 208 Time saved in virtual environments 208 Freedom to multi-task in virtual environments 208 Less fatigue in virtual environments 209 6.2.5 Decision-making in virtual environments 210 6.3 Summary 211 Chapter 7: Integration and discussion of findings 213 7.1 Research questions 213 7.1.1 Research questions 1, 2, and 3 213 7.1.2 Research question 4 214 7.1.3 Research question 5 221 7.2 Limitations 222 7.3 Summary 222 Chapter 8: Implications, future research, and conclusion 225 8.1 Significance and contribution to the field 225 8.2 Guidance for conducting synchronous virtual cut score studies 226 Demands for facilitators and/or co-facilitators 227 Establishing a virtual standard setting netiquette 227 Selecting a suitable virtual platform 227 Selecting an appropriate medium for the workshop 228 Recruiting online participants 230 Training in the virtual platform 230 Uploading materials 230 Monitoring progress and engaging judges 231 8.3 Recommendations for future research 231 8.4 Concluding remarks 232 Appendices 235 Appendix A CEFR verification activity A (Key) 235 Appendix B Electronic consent form 239 Appendix C Judge background questionnaire 240 Appendix D Focus group protocol 243 Introductory statement 243 Focus group interview questions 244 Appendix E Facilitator’s virtual standard setting protocol 245 Appendix F CEFR familiarisation verification activity results 248 Appendix G: Facets specification file 250 Appendix H: Intraparticipant consistency indices 254 Appendix I: Group 5 group level and individual level Rasch indices 257 Appendix J: Form A & Form B score tables 258 Appendix K: DJF pairwise interactions 259 Appendix L: DGF pairwise interactions 265 Appendix M: Wright maps 268 References 273 Author index 291 Subject index 295 Series Index 301 The study investigated whether cut scores could be set in two synchronous e-communication media.The findings revealed that virtual cut scores set in both media were valid, reliable, and comparable to ones set in face-to-face environment and provides a framework for conducting and evaluating virtual cut score studies.