Development of a Formative Assessment Chatbot for a 100-Level Course in Cybersecurity
DOI:
https://doi.org/10.63561/jca.v2i3.837Keywords:
Natural Language Processing (NLP), Formative Assessment, Educational Chatbots, Conversational Agents, Dialogue-Based AssessmentAbstract
This study presents Cypanion: a formative assessment chatbot developed to support first-year undergraduate students in learning foundational cybersecurity concepts through interactive dialogue and real-time feedback. Classroom observations revealed limited prior exposure to cybersecurity among students, often leading to disengagement and poor performance. To address this, Cypanion was built using the DIET-enabled RASA framework for natural language understanding, supported by Python, SQLite3, and the PyCharm API. The system was trained on domain-specific datasets comprising instructional content, question–answer pairs, and conversational tasks including registration, assessment, and formative feedback. Evaluation focused on two natural language understanding subtasks: intent classification and entity extraction. The chatbot achieved 95% precision, recall, and F1-score for intent classification, and 85% precision, 75% recall, and 80% F1-score for entity extraction. These results demonstrate Cypanion’s effectiveness in accurately interpreting student inputs and generating relevant educational responses. The findings suggest that formative assessment chatbots can enhance engagement, support personalized learning, and scale assessment delivery in technical domains like cybersecurity. Future improvements will include speech-based interfaces and exam security modules for deployment in remote learning environments.
References
Arevalillo-Herráez, M., Arnau-González, P., & Ramzan, N. (2022). On adapting the DIET architecture and the Rasa conversational toolkit for the sentiment analysis task. IEEE Access, 10, 107477-107487. DOI: https://doi.org/10.1109/ACCESS.2022.3213061
Bunk, T., Varshneya, D., Vlasov, V., & Nichol, A. (2020). DIET: Lightweight language understanding for dialogue systems. arXiv preprint arXiv:2004.09936. https://arxiv.org/abs/2004.09936
Candello, H., & Pinhanez, C. (2016). Designing conversational interfaces. Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais-IHC, 100.Graesser, A. C., Lu, S., Jackson, G. T., Mitchell, H. H., Ventura, M., Olney, A., & Louwerse, M. M. (2004). AutoTutor: A tutor with dialogue in natural language. Behavior Research Methods, Instruments, & Computers, 36(2), 180–192. https://doi.org/10.3758/BF03195563 DOI: https://doi.org/10.3758/BF03195563
Hsu Myat Mo, S., Sann Su Su Yee, & Soe, K. M. (2023). On using DIET architecture for users’ comments in Myanmar language [Paper presentation]. International Conference on Communication and Computer Research (ICCR 2023). Retrieved from https://www.researchgate.net/publication/379829156_DIET_Architecture_for_Users'_Comments_in _Myanmar_Language
Iniesto, F., Coughlan, T., & Lister, K. (2021, April). Implementing an accessible conversational user interface: Applying feedback from university students and disability support advisors. In Proceedings of the 18th International Web for All Conference (pp. 1–5). DOI: https://doi.org/10.1145/3430263.3452431
Lehman, B., & Jackson, G. T. (2018). Eliciting Deeper Evidence through Conversation-Based Assessments. In Deep comprehension (pp. 209-223). Routledge. DOI: https://doi.org/10.4324/9781315109503-17
Lenk, F., & Schmidt, S. (2022, January). The virtual tutor: Combining conversational agents with learning analytics to support formative assessment in online collaborative learning. In Proceedings of the 55th Hawaii International Conference on System Sciences (pp. 1–10). University of Hawai‘i at Mānoa. https://doi.org/10.24251/HICSS.2022.009 DOI: https://doi.org/10.24251/HICSS.2022.009
Lopez, A. A., Guzman-Orth, D., Zapata-Rivera, D., Forsyth, C. M., & Luce, C. (2021). Examining the accuracy of a conversation-based assessment in interpreting English learners’ written responses. ETS Research Report Series, 2021(1), 1–15. https://doi.org/10.1002/ets2.12315 DOI: https://doi.org/10.1002/ets2.12315
Mageira, K., Pittou, D., Papasalouros, A., Kotis, K., Zangogianni, P., & Daradoumis, A. (2022). Educational AI chatbots for content and language integrated learning. Applied Sciences, 12(7), 3239. https://doi.org/10.3390/app12073239 DOI: https://doi.org/10.3390/app12073239
Nemieboka, T. F., Onyenwe, I. E., & Asogwa, D. C. (2024). Development of an NLP-driven computer- based test guide for visually impaired students. arXiv preprint arXiv:2401.12375. DOI: https://doi.org/10.17148/IJARCCE.2023.12901
Okonkwo, C. W., & Ade-Ibijola, A. (2020). Python-bot: A chatbot for teaching Python programming. Engineering Letters, 29(1), 346–355.
Rosruen, N., & Samanchuen, T. (2018). Chatbot utilization for medical consultant system. In Proceedings of the 3rd Technology Innovation Management and Engineering Science International Conference (TIMES-iCON), 1–5. IEEE. DOI: https://doi.org/10.1109/TIMES-iCON.2018.8621678
Ruan, S., Jiang, L., Xu, J., Tham, B. J., Qiu, Z., Zhu, Y., Murnane, E. L., Brunskill, E., & Landay, J. A. (2019). QuizBot: A dialogue-based adaptive learning system for factual knowledge. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1–13). ACM. https://doi.org/10.1145/3290605.3300587 DOI: https://doi.org/10.1145/3290605.3300587
Ruan, S., Willis, A., Xu, Q., Davis, G. M., Jiang, L., Brunskill, E., & Landay, J. A. (2019). BookBuddy: Turning digital materials into interactive foreign language lessons through a voice chatbot. In Proceedings of the Sixth (2019) ACM Conference on Learning@ Scale (pp. 1–4). ACM. DOI: https://doi.org/10.1145/3330430.3333643
