Document Type : Research Paper

Author

SBU

Abstract

The objective of this study is to investigate the sources of systematic errors associated with the scores obtained from interviews of job applicants and to estimate the generalizability and dependability coefficients within the framework of generalizability theory. Accordingly, the data on the job competency assessment scores of the applicants for the three positions of one the public universities were analyzed using a two-way, fully crossed design. Then, the regular sources of error, including the questions, interviewers and interaction of errors were determined and measured, with the help of which the generalizability and dependability coefficients of the scores were estimated. The findings suggest that generalizability and dependability coefficients of the assessment scores are not equal in all job interviews. Moreover, since the generalizability and dependability coefficients are less than expected, the mean assessment score of one job should not be the basis for decision-making for all the three job selections. Based on the results, it is suggested that, before the adoption of the interviewers' opinions about the merits of applicants, the generalizability and dependability coefficients of the scores be estimated using generalizability theory and if the coefficients are low, the job interview scores should not be the basis for decision-making.

Keywords

Barrick, M. R., Shaffer, J., & DeGrassi, S. D. (2009). What you see may not be what you get: Relationships among self-presentation tactics and ratings of interview and job performance. Journal of Applied Psychology, 94(6), 1394–1412. doi:10.1037/a0016532.
Barrick, M. R., Swider, B. W., & Stewart, G. L. (2010). Initial evaluations in the interview: relationships with subsequent interviewer evaluations and employment offers.  Journal of Applied Psychology, 95(6), 1163-1172. doi:10.1037/a0019918.
Bögels, S. M., Van der Vleuten, C. P. M., Blok, G., Kreutzkamp, R., Melles, R., & Schmidt, H. G. (1995). Assessment and validation of diagnostic interviewing skills for the mental health professions. Journal of Psychopathology and Behavioral Assessment, 17(3), 217-230. doi:10.1007/BF02229299.
Bolino, M. C., Kacmar, K. M., Turnley, W. H., & Gilstrap, J. B. (2008). A multi-level review of impression management motives and behaviors. Journal of Management, 34(6), 1080-1109.
doi:10.1177/0149206308324325.
Brennan, R. L. (2001). Generalizability Theory. New York: Springer-Verlag.
Cardinet, J., Johnson, S., & Pini, G. (2010). Applying generalizability theory using EduG. Routledge.
Conway, J. M., Jako, R. A., & Goodman, D. F. (1995). A meta-analysis of interrater and internal consistency reliability of selection interviews. Journal of Applied Psychology, 80(5), 565 −579.
doi:apa.org/journals/apl/80/5/565.html.
Cook, M. (2016). Personnel Selection: Adding Value through People-A Changing Picture. John Wiley & Sons.
Dayan, K., Fox, S., & Kasten, R. (2008). The preliminary employment interview as a predictor of assessment center outcomes. International Journal of Selection and Assessment, 16(2), 102-111.
doi:10.1111/j.1468-2389.2008.00415.x.
Hanson, M. D., Kulasegaram, K. M., Woods, N. N., Fechtig, L., & Anderson, G. (2012). Modified personal interviews: resurrecting reliable personal interviews for admissions?. Academic Medicine, 87(10), 1330-1334. doi: 10.1097/ACM.0b013e318267630f.
Highhouse, S., & Schmitt, N. (2013). A snapshot in time: Industrial and organizational psychology today. In N. Schmitt & S. High house (Eds.), Handbook of Psychology (Volume 12: Industrial and Organizational). Wiley.
Huffcutt, A. I., Conway, J. M., Roth, P. L., & Klehe, U. C. (2004). The impact of job complexity and study design on situational and behavior description interview validity. International Journal of Selection and Assessment, 12(3), 262-273. doi: 10.1111/j.0965-075X.2004.280_1.x.
Huffcutt, A. I., Culbertson, S. S., & Weyhrauch, W. S. (2013). Employment Interview Reliability: New meta‐analytic estimates by structure and format. International Journal of Selection and Assessment, 21(3), 264-276. doi: 10.1111/ijsa.12036.
Judge, T.A., Cable, D. M., & Higgins, C. A. (2001). The employment interview: A review of recent research and recommendations for future research. Human Resource Management Review, 10(4), 383-406. doi.org/10.1016/S1053-4822 (00)00033-4.
Levashina, J., & Campion, M. A. (2007). Measuring faking in the employment interview: Development and validation of an interview faking behavior scale. Journal of applied psychology, 92(6), 1638-1656. doi:10.1037/0021-9010.92.6.1638.
Lievens, F., De Corte, W., & Brysse, K. (2003). Applicant perceptions of selection procedures: The role of selection information, belief in tests, and comparative anxiety. International Journal of Selection and Assessment, 11(1), 67-77. doi: 10.1111/1468-2389.00227.
Macan, T. (2009). The employment interview: A review of current studies and directions for future research. Human Resource Management Review, 19(3), 203-218. doi.org/10.1016/j.hrmr.2009.03.006.
Maurer, S. D. (2002). A practitioner-based analysis of interviewer job expertise and scale format as contextual factors in situational interviews. Personnel Psychology, 55(2), 307-327.
doi: 10.1111/j.1744-6570.2002.tb00112.x.
Onyon, C. L. A. R. E., Wall, D., & Goodyear, H. M. (2009). Reliability of multi-station interviews in selection of junior doctors for specialty training. Medical teacher, 31(7), 665-667.
Posthuma, R. A., Morgeson, F. P., & Campion, M. A. (2002). Beyond employment interview validity: A comprehensive narrative review of recent research and trends over time. Journal of Personnel Psychology, 55(1), 1-81. doi: 10.1111/j.1744-6570.2002.tb00103.x.
Roulin, N., Bangerter, A., & Levashina, J. (2014). Interviewers' perceptions of impression management in employment interviews. Journal of Managerial Psychology, 29(2), 141-163. doi.org/10.1108/JMP-10-2012-0295.
Rogelberg, S. G. (Ed.). (2006). Encyclopedia of industrial and organizational psychology. Sage publications.
Sebok, S. S., Luu, K., & Klinger, D. A. (2014). Psychometric properties of the multiple mini-interview used for medical admissions: findings from generalizability and Rasch analyses. Advances in Health Sciences Education, 19(1), 71-84. doi: 10.1007/s10459-013-9463-7.
Shavelson, R., & Webb, N. (1991). Generalizability theory: A primer. Newbury Park, C.A.: Sage.
Stewart, G. L., Dustin, S. L., Barrick, M. R., & Darnold, T. C. (2008). Exploring the handshake in employment interviews. Journal of Applied Psychology, 93(5), 1139–1146. doi:10.1037/0021-9010.93.5.1139.
Taylor, P. J., & Small, B. (2002). Asking applicants what they would do versus what they did do: A meta‐analytic comparison of situational and past behavior employment interview questions. Journal of Occupational and Organizational Psychology, 75(3), 277-294. doi: 10.1348/096317902320369712.