Document Type : Research Paper
Authors
1 Faculty member at faculty of psychology and education, Kharazmi university
2 MA of educational research, Kharazmi university
Abstract
Math skills include different content and cognitive processes domains which indicate the complexity of math skill and its latent traits. Until now, Study of these complexities has been done through traditional data analysis or subjective methods. Therefore, present research studied cognitive dimensions and latent variables of the math test of national university entrance examination through the application of Latent Class Multidimensional Item Response Theory (LCMIRT). In doing so, the math test data of national examinations in 2008, 2011 and 2015 were studied. Results showed that the math test, as a high stakes test in the national university entrance examination, encompasses a set of multidimensional cognitive traits. Results of the unified parallel analysis method showed that the tests do not fit with the unidimensional model and adding additional dimensions can significantly improve model fit. Moreover, cognitive domains of comprehension, problem solving and reasoning were recognized as three essential constructs which account for math skills and give more detailed information about items quality in clustering and analysis of the test items. This property of LCMIRT, in comparison with other approaches, increases the number of cognitive sub-clusters. At last, it is recommended that LCMIRT models are considered in constructing and analyzing educational and psychological tests, in order to more validly detect latent cognitive skills of the tests.
Keywords
Ackerman, T. (1989). Unidimensional IRT calibration of compensatory and
noncompensatory multidimensional items. Applied Psychological Measurement, 13 (2), 113-127.
Ackerman, T. (1996). Graphical representation of multidmensional item response theory analyses. Applied Psychological Measurement, 20, 311–329.
Adams, R. J., Wilson, M., & Wang, W. C. (1997). The multidimensional random coefficients multinomial logit model. Applied psychological measurement, 21 (1), 1-23.
Anderson,L.W.,&Krathwohl,D.R. (2001).A Taxonomy for learning,teaching,and assessing: A Revision of Bloom's Taxonomy of educational objectives.NewYork:Longman.
Ansley, T. N., & Forsyth, R. A. (1985). An examination of the characteristics of unidimensional IRT parameter estimates derived from two-dimensional data. Applied Psychological Measurement, 9, 37-48.
Bacci, S., Pandolfi, S., & Pennoni, F. (2014). A comparison of some criteria for states selection in the latent Markov model for longitudinal data. Advances in Data Analysis and Classification, 8 (2), 125-145.
Bartolucci, F. (2007). A class of multidimensional IRT models for testing unidimensionality and clustering items. Psychometrika, 72 (2), 141-157.
Bolt, D. M., & Lall, V. F. (2003). Estimation of compensatory and noncompensatory multidimensional item response models using Markov chain Monte Carlo. Applied Psychological Measurement, 27 (6), 395-414.
Bradlow, E. T., Wainer, H., & Wang, X. (1999). A Bayesian random effects model for testlets. Psychometrika, 64 (2), 153-168.
Bragg, L. A., Vale, C., Herbert, S., Loong, E., Widjaja, W., Williams, G., & Mousley, J. (2013, January). Promoting awareness of reasoning in the primary mathematics classroom. In MAV 2013: Mathematics of the planet earth: Proceedings of the MAV 50th Annual Conference 2013 (pp. 23-30). Mathematical Association of Victoria.
Brooks, S., Gelman, A., Jones, G., & Meng, X. L. (Eds.). (2011). Handbook of Markov Chain Monte Carlo. CRC press.
Carroll, J. B. (1945). The effect of difficulty and chance success on correlations between items or between tests. Psychometrika, 10 (1), 1-19.
Chen, W. H., & Thissen, D. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22 (3), 265-289.
Christensen, K. B., Bjorner, J. B., Kreiner, S., & Petersen, J. H. (2002). Testing unidimensionality in polytomous Rasch models. Psychometrika, 67 (4), 563-574.
De Champlain, A. F. (1996). The effect of multidimensionality on IRT true-score equating for subgroups of examinees. Journal of Educational Measurement, 33, 181-201.
De Champlain, A., & Gessaroli, M. E. (1998). Assessing the dimensionality of item response matrices with small sample sizes and short test lengths. Applied Measurement in Education, 11 (3), 231-253
Drasgow, F., & Lissak, R. I. (1983). Modified parallel analysis: A procedure for examining the latent dimensionality of dichotomously scored item responses. Journal of Applied psychology, 68 (3), 363-373.
Dubravka Svetina & Roy Levy (2014) A Framework for Dimensionality Assessment for Multidimensional Item Response Models, Educational Assessment, 19:1, 35-57.
Frederiksen, N. (Ed.). (1990). Diagnostic Monitoring of Skill and Knowledge Acquisition. Psychology Press.
Fraser, C., & McDonald, R. P. (1988). NOHARM II: A FORTRAN program for fitting unidimensional and multidimensional normal ogive models of latent trait theory. The University of New England, Armidale, Australia. : University of New England, Centre for Behavioral Studies.
Geramipour, M., & Shahmirzadi, N. (2018). Application and Comparison of Multidimensional Latent Class Item Response Theory on Clustering Items in Comprehension Tests. The Journal of AsiaTEFL, 15 (2), 479-490.
Goodman, L. A. (1974). Exploratory latent structure analysis using both identifiable and unidentifiable models. Biometrika, 61 (2), 215-231.
Gu, Z. (2011). Maximizing the potential of multiple-choice items for Cognitive Diagnostic Assessment (Doctoral dissertation, University of Toronto).
Hambleton, R. K., & Rovinelli, R. J. (1986). Assessing the dimensionality of a set of test items. Applied psychological measurement, 10 (3), 287-302.
Hambleton, R. K., & Swaminathan, H. (1985). Item response theory: Principles and applications (Vol. 7): Springer.
Hayton J. C., Allen D. G., and Scarpello V. (2004). Factor Retention Decisions in Exploratory Factor Analysis: A Tutorial on Parallel Analysis. Organizational Research Methods. 7 (2): 191–205
Heinen, T. (1996). Latent class and discrete latent trait models: Similarities and differences. Sage Publications, Inc.
Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30 (2), 179-185.
Hulin, C. L., Drasgow, F., & Parsons, C. K. (1983). Item response theory: Application to psychological measurement. Homewood, IL: Irwin.
Kim, H. R., & Stout, W. F. (1994). A new index for assessing the amount of multidimensionality and/or simple structure present in test data. In annual meeting of the American Educational Research Association, New Orleans.
Koretz, D. (2008). Measuring Up: What Educational Testing Really Tells Us. Cambridge, MA: Harvard University Press. pp. 7-14.
Langeheine, R., & Rost, J. (Eds.). (2013). Latent trait and latent class models. Springer Science & Business Media.
Lazarsfeld, P. F., & Henry, N. W. (1968). Latent structure analysis. Houghton Mifflin Co.
Lord, F. M. (1980). Applications of item response theory to practical testing problems:Mahwah, NJ: Erlbaum.
Lord, F. M., Novick, M. R., & Birnbaum, A. (1968). Statistical theories of mental test scores. (pp.359-382). Reading, MA: Addison-Wesley.
Masters, G. N. (1985). A comparison of latent trait and latent class analyses of Likert-type data. Psychometrika, 50 (1), 69-82.
Nandakumar, R. (1994). Assessing essential dimensionality of set of items. Journal of Educational Measurement,31:1.
Payne, DA (2003). Applied Educational Assessment (2nd ed.). US: Wads Worth.
Reckase, M. D. (1979). Unifactor latent trait models applied to multifactor tests: Results and implications. Journal of Educational and Behavioral Statistics, 4 (3), 207-230.
Reckase, M. (2009). Multidimensional item response theory (Vol. 150). New York: Springer.
Rost, J. (1988). Rating scale analysis with latent class models. Psychometrika, 53 (3), 327-348.
Rost, J. (1990). Rasch models in latent classes: An integration of two approaches to item analysis. Applied Psychological Measurement, 14 (3), 271-282.
Roznowski, M., Tucker, L. R., & Humphreys, L. G. (1991). Three approaches to determining the dimensionality of binary items. Applied Psychological Measurement, 15 (2), 109-127.
Santrock, J. W. (2004). Educational Psychology. New York: McGraw-Hill.
Shealy, R., & Stout, W. (1993). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58 (2), 159-194.
Sireci, S. G., Thissen, D., & Wainer, H. (1991). On the reliability of testlet‐based tests. Journal of Educational Measurement, 28 (3), 237-247.
Slavin, R. E., & Davis, N. (2006). Educational psychology: Theory and practice.
Stout, W. F. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52 (4), 589-617.
Sympson, J. B. (1978). A model for testing with multidimensional items. Paper presented at the Proceedings of the 1977 Computerized Adaptive Testing Conference, Minneapolis: University of Minnesota, Department of Psychology, Psychometric Methods Program.
Takane, Y., & De Leeuw, J. (1987). On the relationship between item response theory and factor analysis of discretized variables. Psychometrika, 52 (3), 393-408.
Traub, R. E., & Lam, Y. R. (1985). Latent structure and item sampling models for testing. Annual review of psychology, 36 (1), 19-48.
Walker, C. M., & Beretvas, S. N. (2003). Comparing multidimensional and unidimensional proficiency classifications: Multidimensional IRT as a diagnostic aid. Journal of Educational Measurement, 40 (3), 255-275.
Way, W. D., Ansley, T. N., & Forsyth, R. A. (1988). The comparative effects of compensatory and noncompensatory two-dimensional data on unidimensional IRT estimates. Applied Psychological Measurement, 12 (3), 239-252.
Whitely, S. E. (1980). Multicomponent latent trait models for ability tests. Psychometrika, 45 (4), 479-494.
Wilson, D., Wood, R., & Gibbons, R. (1987). TESTFACT [Computer program]. Mooresville IN: Scientific Software.
Van der Linden, W. J., & Hambleton, R. K. (1996). Handbook of modern item response theory: Springer.
von Davier, A. A., & Wilson, C. (2008). Investigating the population sensitivity assumption of item response theory true-score equating across two subgroups of examinees and two test formats. Applied Psychological Measurement, 32 (1), 11-26.
Yen, W. M. (1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of educational measurement, 30 (3), 187-214.
Zhang, J. (2004). Comparison of unidimensional and multidimensional approaches to IRT parameter estimation. ETS Research Report Series, 2004 (2), i-40.
Zwick, R. (1987). Assessing the dimensionality of NAEP reading data. Journal of Educational Measurement, 24 (4), 293-308.
)