FoCoa Shiny app for formative assessment using cognitive diagnosis modeling

  1. Susana Sanz 1
  2. Rodrigo S. Kreitchmann 1
  3. Pablo Nájera 1
  4. José David Moreno 1
  5. José Ángel Martínez-Huertas 2
  6. Miguel A. Sorrel 1
  1. 1 Universidad Autónoma de Madrid
    info

    Universidad Autónoma de Madrid

    Madrid, España

    ROR https://ror.org/01cby8j38

  2. 2 Universidad Nacional de Educación a Distancia, Madrid
Zeitschrift:
Psicología educativa

ISSN: 1135-755X

Datum der Publikation: 2023

Ausgabe: 29

Nummer: 2

Seiten: 149-158

Art: Artikel

DOI: 10.5093/PSED2022A14 DIALNET GOOGLE SCHOLAR lock_openOpen Access editor

Andere Publikationen in: Psicología educativa

Ziele für nachhaltige Entwicklung

Zusammenfassung

La combinación de evaluaciones formativas y sumativas podría mejorar la evaluación. El modelado de diagnóstico cognitivo (MDC) se ha propuesto para diagnosticar fortalezas y debilidades de estudiantes en la evaluación formativa. Sin embargo, ningún software permite implementarlo fácilmente. Así, se ha desarrollado FoCo (https://foco.shinyapps.io/FoCo/), permitiendo realizar análisis MDC y teoría clásica de tests. Se analizaron respuestas de 86 estudiantes de grado a un examen de métodos de investigación, diagnosticándose sus fortalezas y necesidades en cuanto a su dominio de los contenidos de la asignatura y las tres primeras competencias de la taxonomía de Bloom y se analizó la validez de los resultados. El análisis ha sido informativo, ya que para estudiantes con puntuaciones similares ha sido posible detectar diferentes fortalezas y debilidades. Además, se encontró que estos atributos predicen criterios relevantes. Se espera que FoCo facilite el uso de MDC en contextos educativos.

Bibliographische Referenzen

  • Abad, F. J., Olea, J., Ponsoda, V., & García, C. (2011). Medición en ciencias sociales y de la salud [Measurement in social and educational sciences]. Editorial Síntesis.
  • Akbay, L., & de la Torre, J. (2020). Estimation approaches in cognitive diagnosis modeling when attributes are hierarchically structured. Psicothema, 32(1), 122-129. https://doi.org/10.7334/psicothema2019.182
  • Attali, Y., & Fraenkel, T. (2000). The point-biserial as a discrimination index for distractors in multiple-choice items: Deficiencies in usage and an alternative. Journal of Educational Measurement, 37(1), 77.86. https:// doi.org/10.1111/j.1745-3984.2000.tb01077.x
  • Bennett, R. E. (2011). Formative assessment: A critical review. Assessment in Education: Principles, Policy & Practice, 18(1), 5-25. https://doi.org/10.1 080/0969594X.2010.513678
  • Bloom, B., Engelhart, M., Furst, E., Hill, W., & Krathwohl, D. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. David McKay.
  • Bradshaw, L., Izsák, A., Templin, J., & Jacobson, E. (2014). Diagnosing teachers’ understandings of rational numbers: Building a multidimensional test within the diagnostic classification framework. Educational Measurement: Issues and Practice, 33(1), 2-14. https://doi.org/10.1111/ emip.12020
  • Cerezo, R., Fernández, E., Amieiro, N., Valle, A., Rosário, P., & Núñez, J. C. (2019). Mediating role of self-efficacy and usefulness between selfregulated learning strategy knowledge and its use [El papel mediador de la autoeficacia y la utilidad entre el conocimiento y el uso de estrategias de autorregulación del aprendizaje]. Revista de Psicodidáctica, 24(1), 1-8. https://doi.org/10.1016/j.psicod.2018.08.001
  • Chang, W., Cheng, J., Allaire, J. J., Sievert, C., Schloerke, B., Xie, Y., Allen, J., McPherson, J., Dipert, A., & Borges, B. (2021). Shiny: Web application framework for R. R package version 1.6.0. https://CRAN.R-project.org/ package=shiny
  • Chen, H., & Chen, J. (2016). Retrofitting non-cognitive-diagnostic reading assessment under the generalized DINA model framework. Language Assessment Quarterly, 13(3), 218-230. https://doi.org/10.1080/154343 03.2016.1210610
  • Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. https://doi.org/10.1007/BF02310555
  • Chiu, C.Y., Sun, Y., & Bian, Y. (2018). Cognitive diagnosis for small educational programs: The general nonparametric classification method. Psychometrika, 83(2), 355-375. https://doi.org/10.1007/s11336-017- 9595-4
  • De la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199. https://doi.org/10.1007/s11336-011-9207-7
  • De la Torre, J., & Minchen, N. (2014). Cognitively diagnostic assessments and the cognitive diagnosis model framework. Psicología Educativa, 20(2), 89-97. https://doi.org/10.1016/j.pse.2014.11.001
  • Fan, T., Song, J., & Guan, Z. (2021). Integrating diagnostic assessment into curriculum: A theoretical framework and teaching practices. Language Testing in Asia, 11, Article 2. https://doi.org/10.1186/s40468-020- 00117-y
  • Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382. https://doi.org/10.1037/ h0031619
  • Fraile, J., Pardo, R., & Panadero, E. (2017). ¿Cómo emplear las rúbricas para implementar una verdadera evaluación formativa? Revista Complutense de Educación, 28(4), 1321-1334. https://doi.org/10.5209/RCED.51915
  • García-Martín, J., & García-Sánchez, J. N. (2018). The instructional effectiveness of two virtual approaches: processes and product [La eficacia instruccional de dos enfoques virtuales: procesos y producto] Revista de Psicodidáctica, 23(2), 117-127. https://doi.org/10.1016/j. psicoe.2018.02.003
  • Green, S. B., & Yang, Y. (2009). Reliability of summed item scores using structural equation modeling: An alternative to coefficient alpha. Psychometrika, 74(1), 155-167. https://doi.org/10.1007/s11336-008-9099-3
  • George, D., & Mallery, P. (2003). SPSS for Windows step by step: A simple guide and reference. 11.0 update (4th ed.). Allyn & Bacon.
  • Jang, E. E., Dunlop, M., Park, G., & van der Boom, E. H. (2015). How do young students with different profiles of reading skill mastery, perceived ability, and goal orientation respond to holistic diagnostic feedback? Language Testing, 32(3), 359-383. https://doi.org/10.1177/0265532215570924
  • Johnson, M. S., & Sinharay, S. (2020). The reliability of the posterior probability of skill attainment in diagnostic classification models. Journal of Educational and Behavioral Statistics, 45(1), 5-31. https://doi. org/10.3102/1076998619864550
  • Jurich, D. P., & Bradshaw, L. P. (2014). An illustration of diagnostic classification modeling in student learning outcomes assessment. International Journal of Testing, 14(1), 49-72. https://doi.org/10.1080/ 15305058.2013.835728
  • Katz, M. H. (2006). Multivariate analyses. A practical guide for clinicians. (2nd ed.). Cambridge University Press.
  • Kreitchmann, R. S., de la Torre, J., Sorrel, M. A., Nájera, P., & Abad, F. J. (2022) Improving reliability estimation in cognitive diagnosis modeling. Behavior Research Methods. Advance online publication. https://doi. org/10.3758/s13428-022-01967-5
  • Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. https://doi. org/10.2307/2529310
  • Li, H., & Suen, H. K. (2013). Constructing and validating a Q-matrix for cognitive diagnostic analyses of a reading test. Educational Assessment, 18(1), 1-25. https://doi.org/10.1080/10627197.2013.761522
  • Linstone, H. A., & Turoff, M. (Eds.). (1975). The Delphi method. Techniques and applications. Addison-Wesley.
  • Liu, Y., Tian, W., & Xin, T. (2016). An application of M2 statistic to evaluate the fit of cognitive diagnostic models. Journal of Educational and Behavioral Statistics, 41(1), 3-26. https://doi.org/10.3102%2F1076998615621293
  • Ma, W., Terzi, R., & de la Torre, J. (2021). Detecting differential item functioning using multiple-group cognitive diagnosis models. Applied Psychological Measurement, 45(1), 37-53. https://doi. org/10.1177/0146621620965745
  • Marchesi, A. & Martín, E. (2014). Calidad de la enseñanza en tiempos de crisis. Alianza Editorial.
  • Muthén, L. K., & Muthén, B. O. (1998-2017). Mplus user’s guide (8th ed.). Muthén & Muthén.
  • Nájera, P., Abad, F .J., & Sorrel, M. A. (2021). Determining the number of attributes in cognitive diagnosis modeling. Frontiers in Psychology, 12, Article. 614470. https://doi.org/10.3389/fpsyg.2021.614470
  • Nájera, P., Sorrel, M. A., & Abad, F. J. (2021). cdmTools: Useful tools for cognitive diagnosis modeling. R package version 1.0.0. https://cran.rproject.org/web/packages/cdmTools/
  • Nájera, P., Sorrel, M. A., de la Torre, J., & Abad, F. J. (2021). Balancing fit and parsimony to improve Q-matrix validation. British Journal of Mathematical and Statistical Psychology, 74(S1), 110-130. https://doi. org/10.1111/bmsp.12228
  • Paulsen, J., & Valdivia, D. S. (2021) Examining cognitive diagnostic modeling in classroom assessment conditions. The Journal of Experimental Education. Advance online publication. https://doi.org/10.1080/0022 0973.2021.1891008
  • R Development Core Team (2021). R: A language and environment for statistical computing. version 4.0.4. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0. http://www.Rproject.org/
  • Ren, H., Xu, N., Lin, Y., Zhang, S., & Yang, T. (2021). Remedial teaching and learning from A cognitive diagnostic model perspective: Taking the data distribution characteristics as an example. Frontiers in Psychology, 12, Article 628607. https://doi.org/10.3389/fpsyg.2021.628607
  • Rojas, G., de la Torre, J., & Olea, J. (2012, April). Choosing between general and specific cognitive diagnosis models when the sample size is small. Paper presented at the meeting of the National Council on Measurement in Education, Vancouver, Canada.
  • Santos, M. G., & Matas, S.G. (2007). Self-assessing primary competences in Psychology students [La autoevaluación de las competencias básicas del estudiante de Psicología]. Revista de Psicodidáctica, 12(1), 7-27.
  • Sen, S., & Cohen, A. S. (2020). Sample size requirements for application diagnostic classification models. Frontiers in Psychology, 11, Article 4050. https://doi.org/10.3389/fpsyg.2020.621251
  • Sessoms, J., & Henson, R. A. (2018). Applications of diagnostic classification models: A literature review and critical commentary. Measurement: Interdisciplinary Research and Perspectives, 16(1), 1-17. https://doi.or g/10.1080/15366367.2018.1435104
  • Sorrel, M. A., Abad, F. J., & Nájera, P. (2021). Improving accuracy and usage by correctly selecting: The effects of model selection in cognitive diagnosis computerized adaptive testing. Applied Psychological Measurement, 45(2), 112-129. https://doi.org/10.1177/0146621620977682
  • Sorrel, M. A., Abad, F. J., Olea, J., de la Torre, J., & Barrada, J. R. (2017). Inferential item-fit evaluation in cognitive diagnosis modeling. Applied Psychological Measurement, 41(8), 614-631. https://doi. org/10.1177/0146621617707510
  • Sorrel, M. A., Olea, J., Abad, F. J., de la Torre, J., Aguado, D., & Lievens, F. (2016). Validity and reliability of situational judgement test scores: A new approach based on cognitive diagnosis models. Organizational Research Methods, 19(3), 506-532. https://doi.org/10.1177/1094428116630065
  • Tatsuoka, K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345-354. http://www.jstor.org/stable/1434951
  • Templin, J., & Bradshaw, L. (2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification, 30(2), 251-275. https://doi.org/10.1007/s00357-013-9129-4
  • Testa, S., Toscano, A., & Rosato, R. (2018). Distractor efficiency in an item pool for a statistics classroom exam: Assessing its relation with item cognitive level classified according to Bloom’s taxonomy. Frontiers in Psychology, 9, Article 1585. https://doi.org/10.3389/fpsyg.2018.01585
  • Tjoe, H., & de la Torre, J. (2014). The identification and validation process of proportional reasoning attributes: An application of a cognitive diagnosis modeling framework. Mathematics Education Research Journal, 26, 237-255. https://doi.org/10.1007/s13394-013-0090-7
  • Vermunt, J. K., & Magidson, J. (2013). Latent GOLD 5.0 upgrade manual. Statistical Innovations Inc.
  • Von Davier, M., & Lee, Y. S. (Eds.). (2019). Handbook of diagnostic classification models. Springer.
  • Wang, W., Song, L., Chen, P., Meng, Y., & Ding, S. (2015). Attribute-level and pattern-llevel classification consistency and accuracy indices for cognitive diagnostic assessment. Journal of Educational Measurement, 52(4), 457-476. https://doi.org/10.1111/jedm.12096
  • Wu, H. M. (2019). Online individualised tutor for improving mathematics learning: A cognitive diagnosis model approach. Educational Psychology, 39(10), 1218-1232. https://doi.org/10.1080/01443410.201 8.1494819
  • Xu, X., & von Davier, M. (2006). Cognitive diagnosis for NAEP proficiency data (Research report ETS RR-06-08). ETS Research Report Series.
  • Yu, X., Cheng, Y., & Chang, H.-H. (2019). Recent developments in cognitive diagnostic computerized adaptive testing (CD-CAT): A comprehensive review. In M. von Davier & Y.-S. Lee (Eds.), Handbook of diagnostic classification models (pp. 307-331). Springer.
  • Zhan, P. (2020). Longitudinal learning diagnosis: Minireview and future research directions. Frontiers in Psychology, 11, Article 1185. https:// doi.org/10.3389/fpsyg.2020.01185
Fuente de los datos: Dialnet