Applying Item Response Theory in Language Test Item Bank Building (Language Testing and Evaluation)

Item Response Theory, though it has become a widely recognized tool in language testing research, is still not used frequently in practical language assessment projects. This book intends to provide a theoretical overview as well as to give practical guidance concerning the application of IRT in item bank building in a language testing context by presenting a particular project in a higher education setting. Applying Item Response Theory in Language Test Item Bank Building 4 Table of contents 8 Acknowledgements 10 Introduction 12 1. Measurement Theory 14 1.1. General Concerns 14 1.2. Educational Measurement 16 1.3. Classical Test Theory 18 1.3.1. The True Score Model 18 1.3.2 Reliability 20 1.3.3 Validity 24 1.3.4 Traditional Item and Person Statistics 30 1.3.5 Summary 40 1.4. Item Response Theory (IRT) 41 1.4.1. Central Concepts 41 1.4.2 IRT Assumptions 45 1.4.3. IRT Models 47 1.4.3.1. The One-Parameter Logistic Model—The Rasch Model 48 1.4.3.2 The Two-Parameter Logistic Model 49 1.4.3.3 The Three-Parameter Logistic Model 50 1.4.4. Estimation of Item Difficulty and Person Ability 53 1.4.5 Model-Data Fit Statistics 54 1.4.6 Practical Applications of IRT Models 57 1.4.7 Problems with IRT 61 1.4.8 Summary 64 1.5. Applications of Measurement Theory in Language Testing 65 1.5.1. Classical Test Theory in Language Testing 66 1.5.2. Item Response Theory in Language Testing 68 2. Building a Language Testing Item Bank at UP 74 2.1. Background 74 2.2. Research Questions 75 2.3. Stages of Development 76 2.3.1. The Beginnings 76 2.3.1.1. Specifications 77 2.3.1.2. Structural Buildup 78 2.3.2. Modifications 85 2.3.3 Evaluation 87 2.4. Stages of Moderation 90 2.4.1. Piloting 90 2.4.2 Applying Classical Test Analysis 92 2.5. Building an Item Bank 100 2.5.1. Applying the Rasch Model 101 2.5.2 Anchoring and Item Calibration 105 2.5.3 Item and Person Fit 118 2.5.4 Data Management 133 2.5.5 Limitations 136 2.6. Taking Stock and Looking Ahead 140 2.6.1. Lessons to Learn 141 2.6.2. Into the Future 142 2.7. Using the UP Data in Further Empirical Research 145 2.7.1. Stating the problem 146 2.7.2. Research design 148 2.7.3 Results 150 2.7.4 Implications 155 2.7.5 Conclusion 157 Conclusions 160 References 162 Index 174 List of figures and tables 180 Appendices 182 Appendix A: Rasch statistics of the 1994 items in entry order 183 Appendix B: Rasch statistics of the 1995 items in entry order 186 Appendix C: Rasch statistics of the 1996 items in entry order 188 Appendix D: Rasch statistics of the 1997 items in entry order 190 Appendix E: Rasch statistics of the 1998 items in entry order 192 Appendix F: Rasch statistics of the 1999 items in entry order 194 Back Matter 196