FoCoa Shiny app for formative assessment using cognitive diagnosis modeling

  1. Susana Sanz 1
  2. Rodrigo S. Kreitchmann 1
  3. Pablo Nájera 1
  4. José David Moreno 1
  5. José Ángel Martínez-Huertas 2
  6. Miguel A. Sorrel 1
  1. 1 Universidad Autónoma de Madrid
    info

    Universidad Autónoma de Madrid

    Madrid, España

    ROR https://ror.org/01cby8j38

  2. 2 Universidad Nacional de Educación a Distancia, Madrid
Journal:
Psicología educativa

ISSN: 1135-755X

Year of publication: 2023

Volume: 29

Issue: 2

Pages: 149-158

Type: Article

DOI: 10.5093/PSED2022A14 DIALNET GOOGLE SCHOLAR lock_openOpen access editor

More publications in: Psicología educativa

Sustainable development goals

Abstract

Combining formative and summative evaluations could improve assessment. Cognitive diagnosis modeling (CDM) has been proposed as a tool for diagnosing students’ strengths and weaknesses in formative assessment. However, there is no user-friendly software to implement it. For this reason, a Shiny app, FoCo, has been developed (https://foco.shinyapps.io/FoCo/), to conduct CDM and classical test theory analyses. The responses from 86 undergraduate students to a research methods course examination were analyzed. Students’ strengths and needs were diagnosed concerning their dominance of the syllabus contents and the first three competencies in Bloom’s taxonomy. The validity of the results was analyzed. The exam showed acceptable about evaluating students’ knowledge, as students with similar scores showed different strengths and weaknesses. Additionally, these attributes were found to predict different relevant criteria. It is expected that FoCo’s easiness to use promotes the employment of CDM in real educational settings.

Bibliographic References

  • Abad, F. J., Olea, J., Ponsoda, V., & García, C. (2011). Medición en ciencias sociales y de la salud [Measurement in social and educational sciences]. Editorial Síntesis.
  • Akbay, L., & de la Torre, J. (2020). Estimation approaches in cognitive diagnosis modeling when attributes are hierarchically structured. Psicothema, 32(1), 122-129. https://doi.org/10.7334/psicothema2019.182
  • Attali, Y., & Fraenkel, T. (2000). The point-biserial as a discrimination index for distractors in multiple-choice items: Deficiencies in usage and an alternative. Journal of Educational Measurement, 37(1), 77.86. https:// doi.org/10.1111/j.1745-3984.2000.tb01077.x
  • Bennett, R. E. (2011). Formative assessment: A critical review. Assessment in Education: Principles, Policy & Practice, 18(1), 5-25. https://doi.org/10.1 080/0969594X.2010.513678
  • Bloom, B., Engelhart, M., Furst, E., Hill, W., & Krathwohl, D. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook I: Cognitive domain. David McKay.
  • Bradshaw, L., Izsák, A., Templin, J., & Jacobson, E. (2014). Diagnosing teachers’ understandings of rational numbers: Building a multidimensional test within the diagnostic classification framework. Educational Measurement: Issues and Practice, 33(1), 2-14. https://doi.org/10.1111/ emip.12020
  • Cerezo, R., Fernández, E., Amieiro, N., Valle, A., Rosário, P., & Núñez, J. C. (2019). Mediating role of self-efficacy and usefulness between selfregulated learning strategy knowledge and its use [El papel mediador de la autoeficacia y la utilidad entre el conocimiento y el uso de estrategias de autorregulación del aprendizaje]. Revista de Psicodidáctica, 24(1), 1-8. https://doi.org/10.1016/j.psicod.2018.08.001
  • Chang, W., Cheng, J., Allaire, J. J., Sievert, C., Schloerke, B., Xie, Y., Allen, J., McPherson, J., Dipert, A., & Borges, B. (2021). Shiny: Web application framework for R. R package version 1.6.0. https://CRAN.R-project.org/ package=shiny
  • Chen, H., & Chen, J. (2016). Retrofitting non-cognitive-diagnostic reading assessment under the generalized DINA model framework. Language Assessment Quarterly, 13(3), 218-230. https://doi.org/10.1080/154343 03.2016.1210610
  • Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297-334. https://doi.org/10.1007/BF02310555
  • Chiu, C.Y., Sun, Y., & Bian, Y. (2018). Cognitive diagnosis for small educational programs: The general nonparametric classification method. Psychometrika, 83(2), 355-375. https://doi.org/10.1007/s11336-017- 9595-4
  • De la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199. https://doi.org/10.1007/s11336-011-9207-7
  • De la Torre, J., & Minchen, N. (2014). Cognitively diagnostic assessments and the cognitive diagnosis model framework. Psicología Educativa, 20(2), 89-97. https://doi.org/10.1016/j.pse.2014.11.001
  • Fan, T., Song, J., & Guan, Z. (2021). Integrating diagnostic assessment into curriculum: A theoretical framework and teaching practices. Language Testing in Asia, 11, Article 2. https://doi.org/10.1186/s40468-020- 00117-y
  • Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382. https://doi.org/10.1037/ h0031619
  • Fraile, J., Pardo, R., & Panadero, E. (2017). ¿Cómo emplear las rúbricas para implementar una verdadera evaluación formativa? Revista Complutense de Educación, 28(4), 1321-1334. https://doi.org/10.5209/RCED.51915
  • García-Martín, J., & García-Sánchez, J. N. (2018). The instructional effectiveness of two virtual approaches: processes and product [La eficacia instruccional de dos enfoques virtuales: procesos y producto] Revista de Psicodidáctica, 23(2), 117-127. https://doi.org/10.1016/j. psicoe.2018.02.003
  • Green, S. B., & Yang, Y. (2009). Reliability of summed item scores using structural equation modeling: An alternative to coefficient alpha. Psychometrika, 74(1), 155-167. https://doi.org/10.1007/s11336-008-9099-3
  • George, D., & Mallery, P. (2003). SPSS for Windows step by step: A simple guide and reference. 11.0 update (4th ed.). Allyn & Bacon.
  • Jang, E. E., Dunlop, M., Park, G., & van der Boom, E. H. (2015). How do young students with different profiles of reading skill mastery, perceived ability, and goal orientation respond to holistic diagnostic feedback? Language Testing, 32(3), 359-383. https://doi.org/10.1177/0265532215570924
  • Johnson, M. S., & Sinharay, S. (2020). The reliability of the posterior probability of skill attainment in diagnostic classification models. Journal of Educational and Behavioral Statistics, 45(1), 5-31. https://doi. org/10.3102/1076998619864550
  • Jurich, D. P., & Bradshaw, L. P. (2014). An illustration of diagnostic classification modeling in student learning outcomes assessment. International Journal of Testing, 14(1), 49-72. https://doi.org/10.1080/ 15305058.2013.835728
  • Katz, M. H. (2006). Multivariate analyses. A practical guide for clinicians. (2nd ed.). Cambridge University Press.
  • Kreitchmann, R. S., de la Torre, J., Sorrel, M. A., Nájera, P., & Abad, F. J. (2022) Improving reliability estimation in cognitive diagnosis modeling. Behavior Research Methods. Advance online publication. https://doi. org/10.3758/s13428-022-01967-5
  • Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. https://doi. org/10.2307/2529310
  • Li, H., & Suen, H. K. (2013). Constructing and validating a Q-matrix for cognitive diagnostic analyses of a reading test. Educational Assessment, 18(1), 1-25. https://doi.org/10.1080/10627197.2013.761522
  • Linstone, H. A., & Turoff, M. (Eds.). (1975). The Delphi method. Techniques and applications. Addison-Wesley.
  • Liu, Y., Tian, W., & Xin, T. (2016). An application of M2 statistic to evaluate the fit of cognitive diagnostic models. Journal of Educational and Behavioral Statistics, 41(1), 3-26. https://doi.org/10.3102%2F1076998615621293
  • Ma, W., Terzi, R., & de la Torre, J. (2021). Detecting differential item functioning using multiple-group cognitive diagnosis models. Applied Psychological Measurement, 45(1), 37-53. https://doi. org/10.1177/0146621620965745
  • Marchesi, A. & Martín, E. (2014). Calidad de la enseñanza en tiempos de crisis. Alianza Editorial.
  • Muthén, L. K., & Muthén, B. O. (1998-2017). Mplus user’s guide (8th ed.). Muthén & Muthén.
  • Nájera, P., Abad, F .J., & Sorrel, M. A. (2021). Determining the number of attributes in cognitive diagnosis modeling. Frontiers in Psychology, 12, Article. 614470. https://doi.org/10.3389/fpsyg.2021.614470
  • Nájera, P., Sorrel, M. A., & Abad, F. J. (2021). cdmTools: Useful tools for cognitive diagnosis modeling. R package version 1.0.0. https://cran.rproject.org/web/packages/cdmTools/
  • Nájera, P., Sorrel, M. A., de la Torre, J., & Abad, F. J. (2021). Balancing fit and parsimony to improve Q-matrix validation. British Journal of Mathematical and Statistical Psychology, 74(S1), 110-130. https://doi. org/10.1111/bmsp.12228
  • Paulsen, J., & Valdivia, D. S. (2021) Examining cognitive diagnostic modeling in classroom assessment conditions. The Journal of Experimental Education. Advance online publication. https://doi.org/10.1080/0022 0973.2021.1891008
  • R Development Core Team (2021). R: A language and environment for statistical computing. version 4.0.4. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0. http://www.Rproject.org/
  • Ren, H., Xu, N., Lin, Y., Zhang, S., & Yang, T. (2021). Remedial teaching and learning from A cognitive diagnostic model perspective: Taking the data distribution characteristics as an example. Frontiers in Psychology, 12, Article 628607. https://doi.org/10.3389/fpsyg.2021.628607
  • Rojas, G., de la Torre, J., & Olea, J. (2012, April). Choosing between general and specific cognitive diagnosis models when the sample size is small. Paper presented at the meeting of the National Council on Measurement in Education, Vancouver, Canada.
  • Santos, M. G., & Matas, S.G. (2007). Self-assessing primary competences in Psychology students [La autoevaluación de las competencias básicas del estudiante de Psicología]. Revista de Psicodidáctica, 12(1), 7-27.
  • Sen, S., & Cohen, A. S. (2020). Sample size requirements for application diagnostic classification models. Frontiers in Psychology, 11, Article 4050. https://doi.org/10.3389/fpsyg.2020.621251
  • Sessoms, J., & Henson, R. A. (2018). Applications of diagnostic classification models: A literature review and critical commentary. Measurement: Interdisciplinary Research and Perspectives, 16(1), 1-17. https://doi.or g/10.1080/15366367.2018.1435104
  • Sorrel, M. A., Abad, F. J., & Nájera, P. (2021). Improving accuracy and usage by correctly selecting: The effects of model selection in cognitive diagnosis computerized adaptive testing. Applied Psychological Measurement, 45(2), 112-129. https://doi.org/10.1177/0146621620977682
  • Sorrel, M. A., Abad, F. J., Olea, J., de la Torre, J., & Barrada, J. R. (2017). Inferential item-fit evaluation in cognitive diagnosis modeling. Applied Psychological Measurement, 41(8), 614-631. https://doi. org/10.1177/0146621617707510
  • Sorrel, M. A., Olea, J., Abad, F. J., de la Torre, J., Aguado, D., & Lievens, F. (2016). Validity and reliability of situational judgement test scores: A new approach based on cognitive diagnosis models. Organizational Research Methods, 19(3), 506-532. https://doi.org/10.1177/1094428116630065
  • Tatsuoka, K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345-354. http://www.jstor.org/stable/1434951
  • Templin, J., & Bradshaw, L. (2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification, 30(2), 251-275. https://doi.org/10.1007/s00357-013-9129-4
  • Testa, S., Toscano, A., & Rosato, R. (2018). Distractor efficiency in an item pool for a statistics classroom exam: Assessing its relation with item cognitive level classified according to Bloom’s taxonomy. Frontiers in Psychology, 9, Article 1585. https://doi.org/10.3389/fpsyg.2018.01585
  • Tjoe, H., & de la Torre, J. (2014). The identification and validation process of proportional reasoning attributes: An application of a cognitive diagnosis modeling framework. Mathematics Education Research Journal, 26, 237-255. https://doi.org/10.1007/s13394-013-0090-7
  • Vermunt, J. K., & Magidson, J. (2013). Latent GOLD 5.0 upgrade manual. Statistical Innovations Inc.
  • Von Davier, M., & Lee, Y. S. (Eds.). (2019). Handbook of diagnostic classification models. Springer.
  • Wang, W., Song, L., Chen, P., Meng, Y., & Ding, S. (2015). Attribute-level and pattern-llevel classification consistency and accuracy indices for cognitive diagnostic assessment. Journal of Educational Measurement, 52(4), 457-476. https://doi.org/10.1111/jedm.12096
  • Wu, H. M. (2019). Online individualised tutor for improving mathematics learning: A cognitive diagnosis model approach. Educational Psychology, 39(10), 1218-1232. https://doi.org/10.1080/01443410.201 8.1494819
  • Xu, X., & von Davier, M. (2006). Cognitive diagnosis for NAEP proficiency data (Research report ETS RR-06-08). ETS Research Report Series.
  • Yu, X., Cheng, Y., & Chang, H.-H. (2019). Recent developments in cognitive diagnostic computerized adaptive testing (CD-CAT): A comprehensive review. In M. von Davier & Y.-S. Lee (Eds.), Handbook of diagnostic classification models (pp. 307-331). Springer.
  • Zhan, P. (2020). Longitudinal learning diagnosis: Minireview and future research directions. Frontiers in Psychology, 11, Article 1185. https:// doi.org/10.3389/fpsyg.2020.01185
Data source: Dialnet