The author is a student in the Master of Physics Education Program, Department of Physics, Faculty of Mathematics and Natural Sciences, Universitas Negeri Padang.
JurnalPost.com - In the world of research, surveys, tests, and observation sheets are instruments frequently used to collect data. Students and researchers alike utilize these tools to measure various aspects, ranging from learning motivation and critical thinking skills to learning outcomes and student perceptions of the learning process.
However, there is one crucial question often overlooked before instruments are distributed to respondents: is the measurement tool truly capable of measuring what it is intended to study?
Instruments developed without quality checks have the potential to produce inaccurate data. Consequently, research conclusions may be imprecise and fail to fully reflect actual conditions.
Therefore, testing the quality of instruments, particularly for validity and reliability, is not merely a formality in research methodology, but a vital step to ensure the quality and trustworthiness of the collected data.
Validity: Ensuring the Instrument Measures the Right Thing
Validity relates to the extent to which evidence and theory support the interpretation of measurement results for a specific purpose.
For example, a student may want to research students' critical thinking skills. If the questions developed only ask students to memorize definitions, the instrument may not be able to capture their ability to analyze, evaluate, and provide logical reasoning.
Thus, the development of an instrument must consider the alignment between research objectives, the indicators being measured, and the types of questions used.
Research by Setiawan, Wagiran, and Alsamiri (2024) on evaluation instruments for the teaching process in higher education highlights the importance of checking content and construct validity.
The study, which involved 1,200 students, used the Aiken's V technique through expert assessment by six individuals as well as Confirmatory Factor Analysis (CFA).
The study reported an average Aiken's V value of 0.752.
These findings demonstrate that instrument development must account for the alignment of content with the indicators to be measured, as well as the construct structure that forms the basis of the measurement.
In other words, a good instrument does not just need questions that appear relevant; it must also be supported by evidence showing its alignment with research objectives.
Reliability: Measuring Consistently
In addition to validity, reliability is a critical aspect to consider in the development of research instruments.
Reliability relates to the consistency of measurement results. A reliable instrument is expected to provide relatively consistent results when used under comparable measurement conditions.
One statistical technique frequently used to assess the internal consistency of a scale is Cronbach's Alpha.
However, the Cronbach's Alpha value should not be understood as the sole evidence that an instrument has met measurement quality requirements.
Taber (2018), in an article for the journal Research in Science Education, warns that Cronbach's Alpha is often reported in research without sufficient explanation regarding the rationale for choosing the statistic or the interpretation of its results.
A high Alpha value does not automatically prove that an instrument has measured the correct construct. Nor does it inherently indicate that all question items measure only a single dimension.
Therefore, researchers need to understand the meaning of the statistics used so that test results do not become mere supplementary figures in a research report.
Statistical Tools Help Check Item Quality
Instrument quality testing can be performed through various statistical analysis approaches. Each approach has characteristics, assumptions, and assessment criteria that researchers must understand.
Tarigan and colleagues (2022), for instance, conducted research comparing the use of SPSS 22.0 software and the Rasch Model.
The study involved 40 test items administered to 40 students.
The analysis results using SPSS showed that 20 items were declared valid, while the analysis using the Rasch Model showed 17 items met the validity criteria used in the study.
The reported reliability values were 0.828 for the SPSS analysis and 0.69 for the Rasch Modelanalysis.
The difference in results indicates that the chosen analysis method can influence the evaluation of instrument quality.
Because of this, researchers should not rely solely on statistical software but must also understand the concepts, assumptions, and characteristics of each approach used.
Beyond evaluating test instruments, the Rasch Model can also be utilized in assessing the quality of questionnaires.
Loh, Jamil, and Ng (2022) conducted research on 24 program evaluation questionnaire items involving 1,891 participants.
The study reported a person reliability value of 0.87 and an item reliability value of 0.96.
Although the reliability results showed good values, the study still identified one item that was unsuitable and required improvement.
These findings show that a high reliability value does not eliminate the need to examine the quality of each item more deeply.
A more integrated testing approach can be found in research by Sofyan and colleagues (2026) regarding the development of evaluation instruments for multimedia-based learning products.
The study combined classical test theory, Structural Equation Modeling (SEM), and the Rasch Model involving 257 respondents.
The study reported an Aiken's V value of 0.89, Cronbach's Alpha of 0.93, Comparative Fit Index (CFI) of 0.96, Root Mean Square Error of Approximation (RMSEA) of 0.054, and Item Reliability of 0.96.
The use of these various analysis techniques provides broader evidence regarding instrument quality, covering aspects of content alignment, construct structure, internal consistency, and the characteristics of each item.
Don't Get Trapped by Statistical Figures
Based on these various studies, one important point must be understood: instrument quality cannot be determined based on a single statistical figure.
Validity and reliability have different functions, even though they are interrelated in the instrument development process.
An instrument may show good internal consistency but not necessarily be capable of measuring the intended construct accurately.
Conversely, an instrument that has content alignment with research objectives still requires reliability testing to determine the consistency of its measurement results.
Therefore, before distributing surveys or conducting tests, researchers should follow a systematic series of stages.
These stages begin with drafting an instrument grid based on research variables and indicators, requesting expert assessment, conducting limited trials, and analyzing the results using appropriate statistical techniques.
If problematic question items are found, researchers need to review and improve them based on statistical evidence and theoretical considerations.
Items showing poor results should not be automatically deleted without considering their relevance to the construct being measured.
Understanding this process is essential, especially for students working on undergraduate theses, master's theses, dissertations, or other scientific research.
Instrument testing is not just done to meet academic requirements; it is a form of scientific responsibility in producing trustworthy data.
Ultimately, the quality of research is not determined solely by the number of respondents or the complexity of the data analysis techniques used.
Good research actually begins with the use of appropriate measurement tools.
Trustworthy data is born from instruments designed carefully, tested through appropriate procedures, and justified based on scientific principles.
Acknowledgments
The author would like to thank the lecturers of the Physics Learning Media Development course, Prof. Dr. Desnita, M.Si., and Dr. Hamdi, M.Si., for their guidance and direction in the process of writing this popular scientific article.
Author Biography
Name: Betria Putri Syarita
Institution: Master of Physics Education Program, Department of Physics, Faculty of Mathematics and Natural Sciences, Universitas Negeri Padang
Email: [email protected]
References
Loh, S. L., Jamil, N. A., & Ng, K. T. (2022). Establishing the validity and reliability of a program evaluation questionnaire using Rasch Measurement Model. Dinamika Jurnal Ilmiah Pendidikan Dasar, 14(2), 149–157. https://doi.org/10.30595/dinamika.v14i2.14786
Setiawan, R., Wagiran, & Alsamiri, Y. (2024). Construction of an instrument for evaluating the teaching process in higher education: Content and construct validity. REID (Research and Evaluation in Education), 10(1), 50–63. https://doi.org/10.21831/reid.v10i1.63483
Sofyan, Kurniati, E., Widana, I. W., Sabariah, & Binti Muhammad, M. (2026). Development and validation of multimedia-based learning product instruments: SEM and Rasch model approaches. Indonesian Journal of Educational Development, 7(1), 82–95. https://doi.org/10.59672/ijed.v7i1.6343
Taber, K. S. (2018). The use of Cronbach's alpha when developing and reporting research instruments in science education. Research in Science Education, 48, 1273–1296. https://doi.org/10.1007/s11165-016-9602-2
Tarigan, E. F., Nilmarito, S., Islamiyah, K., Darmana, A., & Suyanti, R. D. (2022). Analisis instrumen tes menggunakan Rasch Model dan Software SPSS 22.0. Jurnal Inovasi Pendidikan Kimia, 16(2), 92–96. https://doi.org/10.15294/jipk.v16i2.30530
Comments 0
Write a Comment Share your response