Proof of Unidimensionality in Cognitive Test Instrument for Evaluation Science Learning

Rizki Nor Amelia(1), Septiko Aji(2), Kriswantoro Kriswantoro(3), Hesty Sukmasari(4),

DOI: https://doi.org/10.46843/jiecr.v6i1.1897


(1) Science Education Study Program, Faculty of Mathematics and Natural Sciences, Universitas Negeri Semarang, Indonesia
(2) Science Education Study Program, Faculty of Mathematics and Natural Sciences, Universitas Negeri Semarang
(3) Chemistry Education Study Program, Faculty of Teacher Training and Educational Sciences, Universitas Jambi, Indonesia
(4) Doctoral in Educational Management, Faculty of Educational Sciences and Psychology, Universitas Negeri Yogyakarta, Indonesia

Abstract

Dimensional analysis to determine the construct that makes up the instrument as one of the requirements before the analysis is often ignored. Proving dimensions is important because the correct specification of the number of latent dimensions is directly related to construct validity and will determine the appropriate software, approach, and analysis model parameters. Therefore, this research aims to prove the unidimensionality of the Cognitive Test Instrument for Evaluation of Science Learning (CTIESL) through Exploratory Factor Analysis with the Principal Component Analysis Extraction Method. The instrument consisted of 13 essay questions and was responded to by science pre-service teachers who took the Science Learning Evaluation Course (98 people). The results of the analysis prove that the CTIESL has a unidimensional construct based on the Scree Test criteria (detected only one point before the elbow on the scree plot) and the cumulative percent of variance extracted, which shows a satisfactory value (79.232%) so that this instrument is proven only to measure one latent trait, namely the cognitive ability to evaluate the science learning. By fulfilling this unidimensional assumption, the psychometric characteristics of the CTIESL can be explored further with Item Response Theory: 1-Logistic Parameters, 2-Logistic Parameters, or 3-Logistic Parameters, depending on needs.

Keywords

exploratory factor analysis; principal component analysis; unidimensional

Full Text:

PDF

References

Aiken, L.R. (1985). Three coefficients for analyzing the reliability and validity of ratings. Educational and psychological measurement, 45(1), 131–142.

Allen, M. J., & Yen, W. M. (1979). Introduction to measurement theory. Monterey, CA: Brooks/Cole

Ayanwale, M. A., Chere-Masopha, J., & Morena, M.C. (2022). The classical test or item response measurement theory: The status of the framework at the Examination Council of Lesotho. International Journal of Learning, Teaching and Educational Research, 21 (8), 384-406

Bichi, A. A., & Talib, R. (2018). Item response theory: An introduction to latent trait models for testing and item development. International Journal of Evaluation and Research in Education, 7 (2), 142-151

Cappelleri, J.C., Lundy, J.J., & Hays, R.D. (2014). Overview of classical test theory and item response theory for the quantitative assessment of items in developing patient-reported outcomes measures. Clinical Therapeutics, 36 (5), 648-662

Cattell, R. B. (1966). The scree test for the number of factors. Multivariate Behavioral Research, 1 (2), 245-276

Chan, Z., & Ho, S. (2019). Good and Bad Practices in Rubrics: The Perspectives of Students and Educators. Assessment & Evaluation in Higher Education, 44(4), 533–545.

Costello, A.B., & Osborne, J.W. (2005). Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Practical Assessment Research & Evaluation, 10 (7), 1-9

Dahl, B. M., Vasset, F., & Frilund, M. (2023). Students’ approaches to scientific essay writing as an educational method in higher education: A mixed methods study. Social Sciences & Humanities Open, 7(1), 100389.

Eleje, L. I., Onah, F. E., & Abanobi, C. C. (2018). Comparative study of classical test theory and item response theory using diagnostic quantitative economics skill test item analysis results. European Journal of Educational and Social Sciences, 3 (1), 57-75

Erguven, M. (2013). Two approaches to psychometric process: Classical test theory and item response theory. Journal of Education, 2 (2), 23–30.

Fabrigar, L.R., Wegener, D.T., MacCallum, R.C., & Strahan, E.J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4 (3), 272-299.

Garrido, C.C., González, D.N., Seva, U.L., & Piera, P.J.F. (2019). Multidimensional or essentially unidimensional? A multi-faceted factor analytic approach for assessing the dimensionality of tests and items. Psychothema, 31 (4), 450-457.

Glas, C. A. (2008). Item response theory in educational assessment and evaluation. Mesure et évaluation en éducation, 31 (2), 19-34.

Gorsuch, R. (1983). Factor Analysis. Hillsdale Erlbaum.

Hagell, P. (2014). Testing rating scale unidimensionality using the principal component analysis (PCA)/t-test protocol with the Rasch model: The primacy of theory over statistics. Open Journal of Statistics, 4 (6), 456–465.

Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate data analysis. Pearson Education Inc.

Haro, A. V., Noroozi, O., Biemans, H. J., Mulder, M., & Banihashem, S. K. (2024). How does the type of online peer feedback influence feedback quality, argumentative essay writing quality, and domain-specific learning? Interactive Learning Environments, 32(9), 5459-5478.

Harrison, D. (1986). Robustness of IRT parameter estimation to violations of the unidimensionality assumption. Journal of Educational Statistics, 11, 91-115

Hattie, J. (1985). Methodology review: Assessing unidimensionality of tests and items. Applied Psychological Measurement, 9 (2), 139–164.

Hayton, J.C., Allen, D.G., & Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7 (2), 191-205.

Henson, R. K., & Roberts, J. K. (2006). Use of exploratory factor analysis in published research: Common errors and some comments on improved practice. Educational and Psychological Measurement, 66 (3), 393–416.

Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2) , 179-185.

Hu, Z., Lin, L., Wang, Y., & Li, J. (2021). The integration of classical testing theory and item response theory. Psychology, 12, 1397-1409.

Izquierdo, I., Olea, J., & Abad, F.J. (2014). Exploratory factor analysis in validation studies: Uses and recommendations. Psychothema, 26 , 395-400.

Kaiser, H.F. (1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20 (1), 141-151

Kline, R. B. (2023). Principles and practice of structural equation modeling. Guilford Publications

Ledesma, R.D., & Valero-Mora, P. (2007). Determining the number of factors to retain in EFA: An easy-to-use computer program for carrying out parallel analysis. Practical Assessment, Research & Evaluation, 12 (2), 1-11

Linn, R. L. (1968). A Monte Carlo approach to the number of factors problem. Psychometrika, 33, 37-71

Livingston, S. A. (2020). Basic concepts of item response theory: A nonmathematical introduction research memorandum (ETS RM-20-06). Educational Testing Service

Malik, A. R., Pratiwi, Y., Andajani, K., Numertayasa, I. W., Suharti, S., & Darwis, A. (2023). Exploring artificial intelligence in academic essay: higher education student's perspective. International Journal of Educational Research Open, 5, 100296.

Mardapi, D. (2012). Measurement of educational assessment and evaluation. Nuha Medika

Mardapi, D., & Kartowagiran, B. (2011). Pengembangan instrumen pengukur hasil belajar nirbias dan terskala baku. Jurnal Penelitian dan Evaluasi Pendidikan, 15(2), 326-341.

McBride, N. L. (2001). An item response theory analysis of the scales from the international personality item pool and the neo personality inventory-revised [Doctoral dissertation, Virginia Tech].

Mitee, T. L. (2019). Comparative study of classical test theory and item response theory using item analysis results of quantitative chemistry achievement test. The African Journal of Behavioral and Scale Development Research, 1 (1), 26-36

Muñiz, J., & Fonseca-Pedrero, E. (2019). Diez pasos para la construcción de un test [Ten steps for test development]. Psychothema, 31 (1), 7-16.

Netemeyer, R.G., Bearden, W.O., & Sharma, S. (2003). Scaling procedures: Issues and applications. Sage Publications

Ohiri, S. C., & Okoye, R. O. (2023). Application of classical test theory as linear modeling to test item development and analysis. International Research Journal of Modernization in Engineering Technology and Science, 5(1), 2152-2159.

Olson, J. M., & Krysiak, R. (2021). Rubrics as tools for effective assessment of student learning and program quality. In Curriculum Development and Online Instruction for the 21st Century (pp. 173-200). IGI Global.

Pasquali, L. (2003). Psicometria: Theory dos testes na psicologia and na educação. Petropolis Vozes.

Pasquali, L. (2007). Validade dos testes psicológicos: Será possível reencontrar o caminho? Psicologia: Theorya e Pesquisa, 23, 99-107.

Pett, M. A., Lackey, N. R., & Sullivan, J. J. (2003). Making sense of factor analysis: The use of factor analysis for instrument development in health care research. Sage Publications Inc.

Reckase, M.D. (1979). Unifactor latent trait models applied to multifactor tests: Results and implications. Journal of Educational Statistics, 4, 207-230

Rusch, T., Lowry, P.B., Mair, P., & Treiblmaier, H. (2017). Breaking free from the limitations of classical test theory: Developing and measuring information systems scales using item response theory. Information & Management, 54 (2), 189-203

Santoso, P. H., Istiyono, E., & Haryanto, H. (2022). Principal component analysis and exploratory factor analysis of the mechanical waves conceptual survey. Indonesian Journal of Psychological and Educational Measurement, 11 (2), 209-225

Scouller, K. (1998). The influence of assessment method on students' learning approaches: Multiple choice question examination versus assignment essay. Higher education, 35(4), 453–472.

Shrestha, N. (2021). Factor analysis as a tool for survey analysis. American Journal of Applied Mathematics and Statistics, 9 (1), 4–11.

Stout, W. F. (1987). A nonparametric approach for assessing latent trait dimensionality. Psychometrika, 52, 589-617

Sulla, N. (2011). Students taking charge: inside the learner—active, technology—infused classroom. London: Routledge Taylor & Francis Group

Suskie, L. (2018). Assessing student learning: A common sense guide (3rd ed.). Hoboken, NJ: Jossey-Bass.

Tabachnick, B. G., & Fidell, L. S. (2001). Using multivariate statistics. Needham Heights, MA: Allyn & Bacon

Tavakol, M., & Wetzel, A. (2020). Factor analysis: A means for theory and instrument development in support of construct validity. International Journal of Medical Education, 11, 245 -247.

Thompson, B. (2004). Exploratory and confirmatory factor analysis: understanding concepts and applications. Washington DC: American Psychological Association

Toni, P. M., Romanelli, E. J., & Salvo, C. G. (2004). Analyze da estrutura fatorial da Bateria Piaget Head. Boletim de Psicologia, 54 (120), 73-86

van Rijn, P.W., Sinharay, S., Haberman, S.J., & Johnson, M.S. (2016). Assessment of fit of item response theory models used in large-scale educational survey assessments. Large-Scale Assessments in Education, 4, 1-23.

Velicer, W. F., & Jackson, D. N. (1990). Component analysis versus common factor analysis - some further observations. Multivariate Behavioral Research, 25 (1), 97–114

Vincent, W., & Shanmugam, S. K. S (2020). The role of classical test theory to determine the quality of classroom teaching test items. Pedagogia: Journal of Education, 9 (1), 5-34

Wagbara, E.S., & Nna-Kue, N.L. (2024). Relevance of classical test theory in the assessment of learning in tertiary institutions in Nigeria. British Journal of Education, Learning, and Development Psychology, 7(2), 202-2013

Watkins, M. W. (2018). Exploratory factor analysis: A guide to best practice. Journal of Black Psychology, 44 (3), 219-246

Williams, B., Onsman, A., & Brown, T. (2010). Exploratory factor analysis: A five-step guide for novices. Australasian Journal of Paramedicine, 8, 1-13

Ziegler, M., & Hagemann, D. (2015). Testing the unidimensionality of items. European Journal of Psychological Assessment, 31(4), 231–237.

Zwick, W.R., & Velicer, W.F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99(3), 432-442.

Refbacks

  • There are currently no refbacks.


View My Stats