Adaptive testlets

ADAPTIVE TESTLETS 1 Running head: ADAPTIVE TESTLETS

A Comparison of Item and Testlet Selection Procedures in Computerized Adaptive Testing

Leslie Keng Pearson

Tsung-Han Ho The University of Texas at Austin

Tzu-An Ann Chen The University of Texas at Austin

Barbara G. Dodd The University of Texas at Austin

Correspondence concerning this article should be addressed to Leslie Keng,Pearson, 400 Center Ridge Drive, Austin, TX 78753, E-mail: Leslie.Keng@Pearson.com .

ADAPTIVE TESTLETS 2 Abstract Testlet response theory (TRT) is a measurement model that can capture local dependency in testlet-based tests. One of the purported advantages of TRT over the more commonly-used polytomous IRT approach to modeling testlet-based tests is that it allows for ad hoc testletconstruction in a testlet-based computer adaptive test (CAT). The goal of this study was to investigate the merits of such a CAT design. Specifically, it examined the use of testlet-based CATs that not only chose each testlet adaptively, but they also adaptively selected each item within the testlet, based on the estimated examinee proficiency. This design was termed a CAT with adaptive testlets, and it wascompared against a CAT whose within-testlet items were all pre-determined and fixed (termed a CAT with fixed testlets). Real data from a large-scale assessment were calibrated using the 3PL-TRT model and used in this simulation study, which compared these testlet-based CAT designs on their measurement and exposure control properties. The study found that the use of adaptive testlets improvedmeasurement precision while achieving better pool utilization rates. The use of an item-level exposure control procedure within a CAT with adaptive testlets resulted in similar measurement precision, but only a modest gain in pool utilization rates, when compared to a CAT with adaptive testlets and no item-level exposure control. This study represented an initial examination of the properties of CATswith adaptive testlets. As such, suggestions for future research are also provided.

ADAPTIVE TESTLETS 3 A Comparison of Item and Testlet Selection Procedures in Computerized Adaptive Testing Computer-based testing (CBT) has become a popular alternative mode of test administration to traditional paper-and-pencil (P&P) testing. CBT leverages the benefits of computer technology, leading to severaladvantages for examinees and test administrators, such as flexibility in scheduling, increased testing opportunities, automated data collection, and prompt score reporting (Bergstorm & Lunz, 1999). It also makes possible the administration of assessments in ways other than the traditional linear fixed format, where every examinee receives the same set of items. One such alternative CBT design isa computer adaptive test. Computer Adaptive Testing The basic logic behind a computer adaptive test (CAT) is to emulate the test-giving approach of an intelligent human test administrator that adaptively gives test items to each examinee based on his or her evaluation of the examinee’s proficiency. This evaluation is typically based on how the examinee has performed so far on the items given bythe administrator. If an examinee struggles, then the administrator would give the examinee a low proficiency estimate and administer easier items; if an examinee excels, then the proficiency estimate for the examinee would be high and the administrator would give harder items. Similarly, a CAT algorithm is designed so that examinees receive a tailored test with a set of items that is mostaccessible to them, based on their estimated proficiency level. In other words, the goals of a CAT is to administer, at each point in the test, items provide maximum information at the examinee’s estimated proficiency. Consequently, a CAT is typically shorter than a traditional linear fixed format test while still achieving equivalent or better measurement precision of the examinee’s ability (Weiss,...