Research for Practical Issues and Solutions in Computerized Multistage Testing

This volume presents a comprehensive collection of the latest research findings supporting the current and future implementations and applications of computerized multistage testing (MST). As a sequel to the widely acclaimed Computerized Multistage Testing: Theory and Applications (2014) by Yan, von Davier, and Lewis, this volume delves into the experiences, considerations, challenges, and lessons learned over the past years. It also offers practical approaches and solutions to the issues encountered. The topics covered include purposeful MST designs, practical approaches for optimal design, assembly strategies for accuracy and efficiency, hybrid designs, MST with natural language processing, practical routing considerations and methodologies, item calibration and proficiency estimation methods, routing and classification accuracy, added value of process data, prediction and evaluation of MST performance, cognitive diagnostic MST, differential item functioning, robustness of statistical methods, simulations, test security, the new digital large-scale Scholastic Aptitude Test, software for practical assessment and simulations, artificial intelligence impact, and the future of adaptive MST. This volume is intended for students, faculty, researchers, practitioners, and education officers in the fields of educational measurement and evaluation in the United States and internationally. Cover Half Title Title Page Copyright Page Table of Contents Foreword Why Multistage Testing? Why this Book? What’s Next? Preface 1 A Brief History of Computerized Adaptive and Multistage Testing Clinical IQ Testing Educational Testing Military Testing Adaptive MSTs in Practice Practical Issues and Research Questions This Volume’s Road Map References Part I Multistage Test Design and Assembly 2 Purposeful Design for Useful Tests: Considering Choices in Multistage-Adaptive Testing Introduction Testing Purposes and Measurement Aims With MST Design Considerations in MST What Should My MST “Look Like”? How Should My MST Work? How Should I Build and Maintain My MST Now and in the Future? Examples of Operational Multistage-Adaptive Test Designs Example 1: The Graduate Records Exam Example 2: The Certified Public Accountant Licensure Exam Example 3: Programme for International Student Assessment and Programme for the International Assessment of Adult Competencies Example 4: The Massachusetts Adult Proficiency Test Summary of Exemplar MST Designs Summary and Conclusions References 3 MST Strategic Design Issues and Implementation Introduction Panel Configurations, Modules, and Routes Assembling Modules and Routes Assembling and Packaging Panels Operational MST Challenges Item Bank Limitations Strategically Dealing With Constrained Test Assembly Attributes Conclusions Notes References 4 Designing Multistage Tests to Meet Accuracy and Efficiency Goals Introduction MST Design Considerations MST Design Features Considered Simulation Study for MST Designs With Specific Measurement Goals Item Pool MST Design Configuration Simulation Results Discussion and Conclusion References 5 Investigating Hybrid Test Designs in Testlet-Based Adaptive Tests Hybrid Multistage Testing and Item-Level Computerized Adaptive Testing Design (HMCAT) Characteristics of Testlet-Based Items Research Studies On Testlet-Based Adaptive Tests Research Studies On Innovative MST Designs HMCAT Design in Passage-Based Adaptive Testing Passage Selection in HMCAT Within-passage Item Selection in HMCAT MST Configuration HMCAT Designs and Ability Estimation Accuracy: Simulation 1 Item Selection Strategies of HMCAT Designs AllStage_CAT (CC) RoutingStage_CAT (CP) LastStage_CAT (PC) NoStage_CAT (PP) Simulation Study Results MST Configuration Under Each HMCAT Design HMCAT Designs Under Each MST Configuration HMCAT Designs and Pool Characteristics: Simulation 2 Conclusion Implications for Practitioners References 6 A Practical Approach to Finding an Optimal Multistage Test Design Introduction An Algorithm for Discovering an Optimal MST Design Defining the Test Circumstances Item Pool Test Assembly Requirements Test-taker Population The Measurement Objective Function Defining the Design Space Systematically Searching the MST Design Space Evaluating MST Performance Practical Application of the Optimization Methods Item Pool Content Requirements Design Space Objective Function Optimization Strategy Results Conclusions Note References 7 Improving Literacy By Integrating Advances in the Learning Sciences and Natural Language Processing in Multistage Testing Introduction: Using Multistage Testing and the Learning Sciences to Help Improve U.S. Literacy Adaptive Testing – Leveraging MST Leveraging the Learning Sciences to Improve Test Design Thresholds – Are Some Skills Holding Students Back? Personalization Based On Student Ability Level Developing Multistage Designs for Reading Multistage Assessment of Writing Tasks Multistage Assessment of Combined Reading/writing Tasks Summary Acknowledgments References Part II MST Routing, Scoring, and Estimation 8 Multistage Testing With Intersectional Routing for Short-Length Tests Multistage Testing With Intersectional Routing for Short-Length Tests MST With Intersectional Routing for Short-Length Tests Illustrative Simulations Studies Study 1: MST With 2-3 Structure MST Design and Data Results and Discussion Study 2: MST With 3-4 Structure MST Design and Data Results and Discussion Study 3: ISR Scenarios With Other Regression Models ISR With Polynomial Regression Real-world Implementation of MST With ISR Discussion and Conclusion References 9 Effect of Routing Errors On the Psychometric Properties of Multistage Tests Introduction Overview and Introduction of Methods Sources of Routing Errors in MSTs and Their Impact On Ability Measurement Paths, Structure, and Routing Strategy Module Assembly and Item Allocation Item Bank and Ability Estimation Method Simulee Population Multistage Tests Conditions Test Structure Item Allocation Assembly Priority Routing Strategies Testing Procedures Item Bank Test Assembly Test Administration Evaluation Criteria Routing Errors Measurement Precision Results Routing Errors Overall Mean Bias, RMSE, and SEM By Condition and Number of Path Errors Overall Mean Bias, RMSE, and SEM By Path Mean Bias Conditional On θ RMSE Conditional On θ Mean SEM Conditional On θ Discussion and Conclusions Misrouting MST Design Limitations and Future Research Directions Conclusions References 10 Item Calibration in Multistage Tests Introduction The Need for Item Calibration With MST Data Item Calibration With Missing Data Basic MSTs Deterministic Routing Probabilistic Routing Routing Rule Complex MSTs Entangled Multidimensional MSTs Non-entangled Multidimensional MSTs Contextual Information Used in Routing Nonrandom Assignment of Routing Modules MST Assigned Only to a Subset of the Sample Simulation Fitted Models Data Simulation Results Discussion References 11 Item Response Theory Proficiency Estimation Methods Under Multistage Testing Item Response Theory Proficiency Estimation Methods Non-Bayesian Versus Bayesian Number-correct Versus Item-Pattern Scoring Proficiency Estimation Methods and the MST Form Specification Proficiency Estimation and Aberrant Response Behaviors Under MST Caveats in IRT Proficiency Estimator Comparison Conclusions and Suggestions Notes References 12 Multistage Tests Under D-Scoring Approach Introduction Multistage Testing With D-Scoring Method Prerequisite to Conducting MST Simulation With D-Scoring Method Simulation Study 1 With 1-3 MST Structure Method Results Simulation Study 2 With 1-2-3 MST Structure Discussion and Conclusion Note References 13 Development and Application of Probability-Weighted Classification for Multistage Testing Introduction An Illustration: A Binary Classification Testing Problem Uncertainty in Classification Probability-weighted Classification Simulation Study and Discussions Conclusions and Future Work Acknowledgments References 14 Creating Value From Process Data: Implications for Multistage Testing Introduction Process Data in Digital Assessments Benefits of Using Process Data Challenges of Using Process Data Process Data in Adaptive Testing Practical Applications of Process Data in MST Incorporating Test-Taking Engagement Into MST Enhancing Ability Estimation in MST With Action Sequences Summary and Conclusion Note References Part III MST Evaluations 15 Evaluating Multistage Testing Performance Introduction MST Evaluation Criteria Item and Test Characteristics Content Conformance Item Block Information and Conformance to Assembly Targets Item Exposure Rates Test Score Characteristics Score Distributions Conditional Standard Errors of Measurement Marginal Reliability Routing Rates and Routing Errors Methods for Computing Score Distributions and Related Statistics for MSTs Simulation-based Computational Approach Core Distributions CSEM Values Marginal Reliability Routing and Routing Error Rates Analytic Computational Approaches Asymptotic Theory Score Distributions CSEM Values Marginal Reliability Routing and Routing Error Rates Recursive Algorithm Score Distributions CSEM Values Marginal Reliability Routing and Routing Error Rates Discussion Notes References 16 Module Assembly and Routing of Cognitive Diagnostic Multistage Adaptive Test Introduction Theoretical Background Cognitive Diagnosis Models KL-Based Indices for CD-CAT Item Selection Module Assembly in CD-MST The Proposed CD-MST Procedure Overview Grouping Knowledge States By the Number of Mastered Attributes Assemble Modules CD-MST Assembly Method 1 CD-MST Assembly Method 2 Routing Methods Static Routing Dynamic Routing Simulation Study Procedure Outcome Measure Results Discussions Appendix A Chapter 17 Differential Item Functioning in Multistage Tests Introduction Generalizability of DIF Methods to MST Data Missing Data Conditional Dependencies in Observed Data Nonparallel Test Forms DIF Methods for MST Data Observed-score-based Methods Measurement Invariance Methods The MH DIF Statistics IRT and MH DIF The MH DIF Statistics With Missing Data Collection Designs The Missing Data of MST Example Simulation Manipulated Factors DIF Item Position MST Design Test Length: Short and Long Ability Difference: No Or Large Difference Simulation Procedure Step 1: Generation of Item Difficulty Parameters for the MST Blocks Step 2: Generation of MST Response Data Step 3: Conduct the IRT True Score Equating Step 4: Calculate the MH DIF Statistics Using Different Criterion Scores Evaluation Criteria Results DIF Items in the Target Blocks DIF Items in the Routing Block Discussion References 18 Navigating Statistical Challenges in the Transition From Linear to Multistage Adaptive Testing Introduction MST Design Statistical Methods Item Analysis Item Calibration Tests of Conditional Independence Numerical Examples Results for Item Analysis Results for Item Calibration Results for Model Fit Discussion References 19 Conducting Simulation Studies for Computerized MST Research Introduction Selected Literature Review On Computerized MST Simulations Typical Fixed Factors in Selected Computerized MST Simulations Typical Varied Factors in Selected Computerized MST Simulations Using Computerized MST Simulation Studies to Construct Tests Example of Planning a Computerized MST Simulation Study A Priori Study With Simulated Data A Priori Study With Observed Data Post Hoc Study With Simulated Data Post Hoc Study With Observed Data Concluding Remarks and Future Directions References 20 Test Security Considerations for CAT and MST Introduction Background Test Security and MST Applications of ML to Address Test Security Issues in CAT and MST An ML Algorithm for Detecting Item Preknowledge in CAT An Item Selection Algorithm Based On Collaborative Filtering Employing Innovative Measures to Detect Emerging Aberrant Test-Taking Behavior in Home-Based CAT and MST Settings Illustrations of Applying the Introduced Methods in Practical Scenarios Example 1. Test Security Considerations in Practice for a Professional Licensure and Certification Exam Example 2: Identifying Item Preknowledge in CAT and MST Example 3: Applying ML Techniques to Detect Item Preknowledge in CAT Example 4: Applying Hierarchical Mixture Model to Detect Item Preknowledge Using Innovative Measures Discussion Note References Part IV Applications and Technologies 21 Considerations in the MST Design for the New Digital SAT Suite of Assessments Introduction SAT From Linear to MST Considerations in MST Development and Planned Operation Content and Item Pool Development Item Pretesting Prior to and After the Launch Automated Test Assembly Choices of IRT Model and Scoring Methodology Implications for SAT Score Scales Fairness of Adaptive MSTs Routing Accuracy and Consistency Test Administration and Score Reporting Conclusion Note References 22 Build High-Quality MST Panels With Mixed-Integer Programming in R Introduction Mixed-integer Programming Objectives Constraints Solvers The Rmst Package Examples Example 1: Top-Down 1-2 MST Example 2: Bottom-Up 1-3 MST Example 3: Hybrid 1-2-3 MST Conclusions References 23 Bayesian Inference for Multistage and Other Incomplete Designs Introduction The Marginal and the Extended NRM Bayesian Inference for the ENRM Operating Characteristics Multistage Testing Real Data Application Discussion References 24 An Overview of Computerized Adaptive and Multistage Testing Software Introduction Overview of CAT Packages Higher-level CAT Simulation Functions Response Generation Scoring Or Ability Estimation Methods Item Selection Methods Exposure Control and Content Constraints Stopping Rules Overview of MST Packages Higher-level MST Simulation Functions MST Test Assembly Response Generation Ability Estimation Methods and Module Selection/routing Strategy Extra Notes for DexterMST Simulations and Comparisons CAT Simulation MST Simulation Sample Codes for CAT and MST Simulations Sample Codes for CAT Simulation Sample Code for MST Simulation Conclusion Note References 25 How Will AI Change Adaptive Testing? Introduction Machine Learning, Generative AI, and Assessment AI Advances That Support Adaptive Testing An Example From the DET Test Delivery Platforms Item Development Item Calibration Item Routing Item Scoring Multilingual Tests: Translation and Generation Test Security Score Reporting Concerns and Challenges Conclusion Acknowledgments Notes References 26 Afterword: The Emergence of Personalized Ensemble Testing Introduction Rationale and Related Work The PET How Does a PET Fit Within this Volume? Conclusion Acknowledgments Note References Index