Differential Item Functioning  Detection Rate Using Rasch Trees Model: A Simulated and Real Data Study of the NAJA High Stakes Tests

Karami, Hamidreza; Gramipour, Masoud; Minaei, Asghar

doi:10.22054/jem.2021.61694.2190

Document Type : Research Paper

Authors

¹ Faculty member of the Department of Counter Narcotics, University of Law Enforcement Sciences

² Associate Professor of Curriculum Planning, Kharazmi University, Tehran, Iran

³ Faculty

https://doi.org/10.22054/jem.2021.61694.2190

Abstract

Although rasch tree model analysis has been introduced to identify Bias questions in various tests, little research has been done. The purpose of this study was to use the rasch tree model and to investigate the intervening factors in revealing the differential action of the test questions. To answer the research questions, the method of simulation studies and data of the specific test of Amin University of Law Enforcement Sciences were used. The present research is applied in terms of purpose and descriptive in terms of method called psychometric research. The statistical population of the special exam of Amin University of Law Enforcement Sciences in 1398 with the number of 2414 people has been analyzed in full. The DIFtree package in R software was used to determine the detection rate of the differentiation Item Functioning using the rasch tree model. The results of the simulation study showed that the rasch tree model detects the differentiation Item Functioning in samples with a volume of more than 1000 subjects as 100%. The results also showed that in the specific exam of the University of Law Enforcement Sciences, 9 questions have differentiation Item Functioning, in the most important of which, 7 questions in the mathematics group with 18 years of age (second category) and 6 questions in the mathematics group with 19 years of age (category First) has a bias towards the experimental sciences group (third category) and the orientation of the bias has been in favor of the first and second category and to the detriment of the third category.

Keywords

References

-گرامی پور، مسعود (1393). ارزیابی توان آماری تحلیل رگرسیون لجستیک در آشکارسازی کنش افتراقی سؤال‌های آزمون، فصلنامه مطالعات اندازهگیری و ارزشیابی آموزشی، سال چهارم، شماره 8، 187-211. بازیابی از:

http://jresearch.sanjesh.org/.

- همبلتون، رونالد ک؛ سؤامیناتان، اچ؛ راجرز، اچ. جین (1991). مبانی نظریه پرسش- پاسخ. ترجمه محمدرضا فلسفی نژاد (1389). تهران: دانشگاه علامه طباطبایی.

References

- Aryadoust, Vahid.(2018). Using recursive partitioning Rasch trees to investigate differential item functioning in second language reading tests. Studies in Educational Evaluation 56. 197–204.

-Benjamini, Y. Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing, Journal of the Royal Statiatical Socirty, Series B(Methodological), Vol.57, No. 1, 289-300.

- Camilli, G. (2006). Test fairness. In R. Brennan (Ed.), Educational measurement (pp. 221-256). New York: American Council on Education & Praeger series on higher education.

-Elder, C.; Mc Namara, T. & Congdon, P. (2003). Rasch techniques for detecting bias in performance tests: An example comparing the performance of native and non-native speakers on a test of academic English. Journal of Applied Measurement, 4: 181–197.

- Ellis, B. B. & Raju, N. S. (2003). Test and item bias: What they are, what they aren’t, and how to detect them. Educational Resources information center (ERIC).

-Engelhard, G. Hansche, L. & Rutledge, K. (1990). Accuracy of bias review judges in identifying differential item functioning on teacher certification tests. Applied Measurement in Education, 3, 347–360

- Fischer G, Molenaar I (eds.) (1995). Rasch Models: Foundations, Recent Developments and Applications. Springer-Verlag, New York.

- Geramipour, Masoud (2020). Item-Focused Trees Approach in Differential Item Functioning (DIF) Analysis: A Case Study of an EFL Reading Comprehension Test, Vol. 7, No. 2, 2020,123-147.

- Hambleton, R, K. Swaminathan, H. & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage Publictions.

- Hidalgo, M. D. & López-Pina, J. P. (2004). Differential item functioning detection and effect size: A comparison between logistic regression and Mantel Haenszel procedures. Educational and Psychological Measurement, 64: 903–915.

- Liou M (1994). More on the Computation of Higher-Order Derivatives on the Elementary Symmetric Functions in the Rasch Model. Applied Psychological Measurement, 18(1), 53–62.

- Liu, Yanlou. Yin, Hao. Xin, Tao. Shao, Laicheng. Yuan, Lu (2019). A Comparison of Differential Item Functioning Detection Methods in Cognitive Diagnostic Models, doi: 10.3389/fpsyg.2019.01137.

- Merkle EC, Zeileis A (2013). Tests of Measurement Invariance without Subgroups: A Generalization of Classical Methods. Psychometrika, 78(1), 59–82.

- Millsap, R. E. (2012). Statistical approaches to measurement invariance. New York, NY: Routledge

- Moritz, B.(2020). Item Focussed recursive Partitioning for Simulataneous Selection of items and variables that induce Differential Item Functioning(DIF) in dichotomous or Polytomous items.

- Popham,W.J (2005). High-Stakes Tests: Harmful, Permanent, Fixable. American Educational Research Journal.6,p85.

- Strobl, Carolin. Kopf, Julia. Zeileis, Achim (2015). Rasch Trees: A New Method for Detecting Differential Item Functioning in the Rasch Model, Working Papers in Economics and Statistics.

- Tutz, G., & Berger, M. (2016). Item-focussed trees for the identification of items in differential item functioning. Psychometrika, 81, 727-750.

-Yüksel, Selcen. Halil Elhan, Atilla. Gökmen, Derya. Küçükdeveci, Ayşe Adile. Kutlay, Şehim (2019). Analyzing differential item functioning of the Nottingham Health Profile by Mixed Rasch Model, Turk J Phys Med Rehab 2018;64(4):300-307.

- Zeileis A, Hornik K (2007). Generalized M-Fluctuation Tests for Parameter Instability. Statistica Neerlandica, 61(4), 488–508.

- Zeileis A, Hothorn T, Hornik K (2008). Model-Based Recursive Partitioning. Journal of Computational and Graphical Statistics, 17(2), 492–514.

- Zumbo, B. D. (2007). Three generations of DIF analyses: Considering where it has been, where it is now, and where it is going. Language Assessmend Quarterly, 4(2), 223-233.

Quarterly of Educational Measurement

Differential Item Functioning Detection Rate Using Rasch Trees Model: A Simulated and Real Data Study of the NAJA High Stakes Tests

References

References

Volume 12, Issue 44
July 2021
Pages 1-30

Differential Item Functioning Detection Rate Using Rasch Trees Model: A Simulated and Real Data Study of the NAJA High Stakes Tests

References

References

Volume 12, Issue 44July 2021Pages 1-30

Volume 12, Issue 44
July 2021
Pages 1-30