Advanced Methods in Automatic Item Generation: Theoretical Foundations and Practical Applications

"Advanced Methods in Automatic Item Generation is an up-to-date survey of the growing research on automatic item generation (AIG) in today's technology-enhanced educational measurement sector. As test administration procedures increasingly integrate digital media and internet use, assessment stakeholders-from graduate students to scholars to industry professionals-have numerous opportunities to study and create different types of tests and test items. This comprehensive analysis offers thorough coverage of the theoretical foundations and concepts that define AIG as well as the practical considerations required to produce and apply large numbers of useful test items"-- Provided by publisher Advanced Methods in Automatic Item Generation is an up-to-date survey of the growing research on automatic item generation (AIG) in today’s technology-enhanced educational measurement sector. Cover 1 Half Title 2 Title Page 4 Copyright Page 5 Table of Contents 6 Preface 8 A Word of Thanks 11 Chapter 1: Introduction: The Changing Context of Educational Testing 14 The Problem of Scaling Item Development 17 Automatic Item Generation: An Augmented Intelligence Approach to Item Development 19 Benefits of Using AIG for Item Development 24 Purpose of This Book 27 References 29 Section 1: Basic Concepts Required for Generating Constructed- and Selected-Response Items 34 Chapter 2: Cognitive Model Development: Cognitive Models and Item Generation 36 Benefits of Using Cognitive Models For AIG 38 Developing Cognitive Models for AIG 39 A Word of Caution When Creating Cognitive Models 43 Two Types of Cognitive Models for AIG 43 Logical Structures Cognitive Model 43 Key Features Cognitive Model 49 References 52 Chapter 3: Item Model Development: Template-Based AIG Using Item Modelling 55 Layers in Item Models 56 Item Generation With 1-Layer Models 59 n-Layer Item Models 62 Item Generation With n-Layer Models 63 Two Important Insights Involving Cognitive and Item Modelling in AIG 68 Non-template AIG: A Review of the State of the Art 70 Is It Preferable to Use a Template for AIG? 74 Note 76 References 76 Chapter 4: Item Generation: Approaches for Generating Test Items 79 The Importance of Constraint Coding 81 Logical Constraint Coding Using Bitmasking 83 Demonstration of Item Assembly Using the Logical Constraints Approach 87 Logical Structures Cognitive Model 87 Key Features Cognitive Model 90 Item Assembly Using the Logical Constraints Approach 91 References 92 Chapter 5: Distractor Generation: The Importance of the Selected-Response Item in Educational Testing 94 The Contribution of Distractors in the Selected-Response Item Format 96 Traditional Approach for Writing Distractors 97 Methods for Distractor Generation 99 Distractor Generation With Rationales 100 Distractor Pool Method With Random Selection 104 Systematic Distractor Generation 106 Note 111 References 111 Chapter 6: Putting It All Together to Generate Test Items: Overview 114 Mathematics Example Using the Logical Structures Model 114 Cognitive Model Development 114 Item Model Development 116 Item Generation Using Constraint Coding 117 Systematic Distractor Generation 120 A Sample of Generated Math Items 121 Medical Example Using Key Features 122 Cognitive Model Development 122 Item Model Development 125 Item Generation Using Constraint Coding 127 Systematic Distractor Generation 129 A Sample of Generated Medical Items 131 Chapter 7: Methods for Validating Generated Items: A Focus on Model-Level Outcomes 133 Substantive Methods for Evaluating AIG Models 135 Cognitive and Item Model Review Using a Validation Table 135 Distractor Model Review Using a Validation Table 139 Substantive Model Review Using a Rating Scale 142 Substantive Methods for Evaluating AIG Items 144 AIG versus Traditional Item Review: Item Quality 144 AIG versus Traditional Item Review: Predictive Accuracy 146 Statistical Methods for Evaluating AIG Items 147 Statistical Analyses of the Correct Option 147 Statistical Analyses of the Incorrect Options 148 Cosine Similarity Index (CSI) 151 The Key to Validating Generated Items 154 References 155 Section 2: Advanced Topics in AIG 158 Chapter 8: Content Coding: Challenges Inherent to Managing Generated Items in a Bank 160 Managing Generated Items With Metadata 162 Content Coding for Item Generation 164 Assembling Content Codes in Item Generation 166 Content Coding Examples 167 Logical Structures Mathematics Model 167 Key Features Medical Model 169 References 171 Chapter 9: Generating Alternative Item Types Using Auxiliary Information: Expanding the Expression of Generated Items 172 Generating Items With Symbols 173 Generating Items With Images 176 Generating Items With Shapes 182 Challenges With Generating Items Using Auxiliary Information 186 References 187 Chapter 10: Rationale Generation: Creating Rationales as Part of the Generation Process 188 Methods for Generating Rationales 189 Correct Option 190 Correct Option With Rationale 191 Correct Option With Distractor Rationale 195 A Cautionary Note on Generating Solutions and Rationales 196 Benefits and Drawbacks of Rationale Generation 197 References 197 Chapter 11: Multilingual Item Generation: Beyond Monolingual Item Development 199 Challenges With Writing Items in Different Languages 201 Methods for Generating Multilingual Test Items 202 Language-Dependent Item Modelling 202 Successive-Language Item Modelling 203 Simultaneous-Language Item Modelling 204 Example of Multilingual Item Generation 206 Validation of Generated Multilingual Test Items 213 References 217 Chapter 12: Conclusions and Future Directions 219 Is AIG an Art or Science? 219 Is It “Automatic” or “Automated” Item Generation? 221 How Do We Define the Word “Item” in AIG? 222 How Do You Generate Items? 224 What Is an Item Model? 225 How Do You Ensure That the Generated Items Are Diverse? 225 How Should Generated Items Be Scored? 226 How Do You Organize Large Numbers of Generated Items? 227 What Does the Future Hold for Item Development? 230 References 236 Author Index 238 Subject Index 243 automatic,item,generation;,banking,system;,cognitive,model,development;,constructed-response,item;,content,coding;,distractor,generation;,educational,testing;,formative,feedback,systems;,item,generation;,item,model,development;,monolingual,item,development;,multilingual,item,generation;,rationale,generation;,selected-response,item,formats;,technology-enhanced,educational,measurement,sector automatic item generation,banking system,cognitive model development,constructed-response item,content coding,distractor generation,educational testing,formative feedback systems,item generation,item model development,monolingual item development,multilingual item generation,rationale generation,selected-response item formats,technology-enhanced educational measurement sector