Student Perceptions of MathTutor as an Intelligent Tutoring System in Higher Education: Learning Support, Progress Tracking, Educational Value, and Usability

Mirjana Ivanović , Aleksandra Klašnja-Milićević , Jelena Ilić

Faculty of Sciences, University of Novi Sad, Serbia.

How to cite: Ivanović, M.; Klašnja-Milićević, A.; Ilić, J.(2026). Student Perceptions of MathTutor as an Intelligent Tutoring System in Higher Education: Learning Support, Progress Tracking, Educational Value, and Usability. In: 12th International Conference on Higher Education Advances (HEAd’26). Valencia, 15-18 June 2026.https://doi.org/10.4995/HEAd26.2026.21692

Abstract

Intelligent Tutoring Systems (ITS) are increasingly used in higher education to provide personalised and adaptive learning. However, empirical evidence on university students’ perceptions of such systems remains limited. This study evaluates MathTutor in higher education mathematics, focusing on students’ perceptions of learning support features, progress tracking, educational value, and system usability. A quantitative survey was conducted with 45 undergraduate computer science students. The results indicate that students particularly value progress visualisation and content organisation, while perceptions of feedback and hints are more nuanced. MathTutor was perceived as providing meaningful educational value, especially in terms of understanding and perceived knowledge gains, whereas perceptions related to motivation were more varied. Overall usability was rated positively, yet it was not significantly associated with perceived learning support or educational value. The findings highlight the importance of examining students’ subjective experiences with ITS and provide design-relevant insights. However, the findings are limited by the small sample size and single-institution context.

Keywords: ITS; Student Perceptions; Self-Regulated Learning; Usability Evaluation.

1. Introduction

Modern higher education faces the challenge of providing individualised learning support in large-scale STEM courses, where students often require timely and adaptive feedback. Mathematics frequently represents a barrier to academic progression, while traditional instruction may not sufficiently address individual needs or support sustained motivation. Intelligent Tutoring Systems (ITS) offer a potential solution by providing personalised guidance, adaptive instruction, and immediate formative feedback. However, the effectiveness of ITS depends not only on their technical capabilities but also on students’ perceptions and acceptance, including perceived usefulness, feedback clarity, progress monitoring support, and overall usability.

MathTutor, an intelligent tutoring system developed at Carnegie Mellon University, supports mathematics learning through interactive problem-solving, step-by-step guidance, and real-time feedback. The system also includes progress tracking and visualization features to monitor performance. Against this background, the present study evaluates MathTutor in a higher education mathematics context, focusing on students’ perceptions of learning support, progress tracking and visualisation, educational value, and system usability. This study contributes to the ITS literature by presenting a student-centred evaluation of MathTutor in an authentic higher education context, with a focus on learning support features, progress visualisation, and perceived educational value. Specifically, the study addresses the following research questions about MathTutor:

RQ1: How do students perceive learning support features (hints, feedback, and tutorial resources) in terms of their usefulness for learning and task completion?

RQ2: How do students evaluate progress tracking and visualisation features as tools for monitoring their learning progress and performance?

RQ3: What are students’ perceptions of educational value regarding understanding, motivation to learn, and perceived improvement in knowledge compared with traditional learning?

RQ4: What is the overall usability as measured by System Usability Scale (SUS), and how is usability associated with perceived learning support and perceived educational value?

2. Related Work and Research Context

ITS have gained increasing attention in higher education, particularly in STEM disciplines where learning is cumulative and requires continuous practice, timely feedback, and adaptive support. Traditional instructional approaches often struggle to address differences in prior knowledge, learning pace, and problem-solving strategies, positioning ITS as a promising solution through personalised guidance, immediate feedback, and progress monitoring (Strielkowski et al., 2025; Wang et al., 2024; Klašnja-Milićević & Ivanović, 2021).

However, the educational value of ITS cannot be inferred solely from their technical sophistication. Prior research emphasises the importance of students’ perceptions, engagement, and perceived usefulness when evaluating ITS effectiveness in higher education (Ivanović &Klašnja-Milićević, 2024; Ilić et al., 2024). Although ITS effectiveness has been widely studied, much of the literature focuses on technical development and academic results, with limited attention to university students’ subjective experiences (Strielkowski et al., 2025). This gap is relevant in higher education, where learner autonomy and metacognitive engagement play a central role (Ateş, 2025).

ITS are designed to support “learning by doing” through adaptive feedback and hierarchical hints based on learner modelling (Aleven et al., 2025; Lin et al., 2025). While positive effects have been reported in primary and secondary mathematics education, systematic reviews continue to highlight the lack of higher education–specific evidence and limited focus on student-centred evaluation (Ilić et al., 2024). Recent large-scale ITS platforms for mathematics, such as MathTutor, demonstrate scalability and adaptive problem-solving support, yet evidence on their pedagogical value from students’ perspectives in authentic university contexts remains scarce (Lyu et al., 2026; Awang et al., 2025; Merino-Campos, 2025).

Students’ perceptions of learning support features, particularly feedback, hints, and progress visualisation, represent key dimensions of ITS evaluation. Adaptive feedback and dashboards can support self-regulated learning by enhancing metacognitive awareness, though their effectiveness depends on timing, relevance, and alignment with learners’ autonomy (Alvarez et al., 2022; Mehrvarz et al., 2025). In higher education, such tools primarily function as metacognitive aids rather than direct instructional scaffolds.

Finally, perceived educational value and usability are critical for technology acceptance and sustained engagement. Students’ perceptions of learning benefits influence attitudes toward ITS, while poor usability may introduce extraneous cognitive load and undermine instructional effectiveness (Bangor et al., 2008; Wang et al., 2023). Together, these findings underline the need for student-centred evaluation of ITS in higher education.

3. Research Methodology

Participants and Procedure. The study included 45 undergraduate computer science students from the Faculty of Sciences. Students used MathTutor to complete mathematics tasks and then voluntarily completed an anonymous post-use survey on their experience with the system.

Instruments and Measures. Data were collected using a structured questionnaire assessing students’ perceptions of (1) learning support features, (2) progress tracking and visualisation, and (3) perceived educational value. Responses were recorded on a 5-point Likert scale (1 = strongly disagree to 5 = agree). The internal consistency of the questionnaire was acceptable (Cronbach’s alpha > 0.80). The questionnaire consisted of 20 items organised into three sections corresponding to the constructs mentioned above and was developed based on relevant literature on intelligent tutoring systems and technology-enhanced learning.

Overall usability was measured using the System Usability Scale (SUS), a widely used standardised instrument consisting of 10 items.

Data Analysis. Data were analysed using IBM SPSS Statistics. Descriptive statistics were used to summarise responses, and Spearman’s rank-order correlations were computed to examine relationships between SUS scores and perceived learning support and educational value.

4. Results

This section presents descriptive and correlational results based on students’ questionnaire responses following direct use of the MathTutor system.

RQ1: Most students reported generally positive perceptions of MathTutor’s learning support features (Table 1). Progress visibility and content organisation received the highest mean, suggesting that students particularly valued features that help them navigate learning content and monitor completion. In contrast, evaluations of feedback and hints showed greater variability, indicating less consistent experiences with adaptive guidance across participants.

Table 1. Students’ Perceptions of the MathTutor ITS (5-point Likert scale)

Statement	Poor N (%)	Fair N (%)	Good N (%)	Very good N (%)	Excellent N (%)	Mean	SD
Track my progress	0 (0.0)	0 (0.0)	7 (15.6)	14 (31.1)	24 (53.3)	4.38	0.75
Feedback motivated me	1 (2.2)	2 (4.4)	18 (40.0)	15 (33.3)	9 (20.0)	3.64	0.93
Content organisation	0 (0.0)	0 (0.0)	11 (24.4)	21 (46.7)	13 (28.9)	4.04	0.74
Hints supported problem-solving	0 (0.0)	16 (35.6)	14 (31.1)	9 (20.0)	6 (13.3)	3.56	0.86

RQ2: Students evaluated MathTutor’s progress tracking and visualisation features positively (Table 2). Most respondents reported that progress views and reports supported monitoring their learning performance and identifying weak areas. Overall responses were predominantly in the upper range of the scale across all items.

Table 2. Descriptive statistics for progress tracking and visualisation features (5-point Likert scale)

Statement	Poor N (%)	Fair N (%)	Good N (%)	Very good N (%)	Excellent N (%)	Mean (M)	SD
Viewing personal progress in MathTutor	0 (0.0)	0 (0.0)	7 (15.6)	14 (31.1)	24 (53.3)	4.38	0.7
Identifying own weak points through the interface	0 (0.0)	0 (0.0)	13 (28.9)	17 (37.8)	15 (33.3)	4.04	0.8
Ease of understanding progress reports (Reports)	1 (2.2)	1 (2.2)	11 (24.4)	12 (26.7)	20 (44.4)	4.09	1.0

RQ3: Students perceived MathTutor as educationally beneficial compared with traditional learning (Table 3), particularly in terms of understanding and knowledge gains. Motivation-related perceptions were more mixed.

Table 3. Students’ perceptions of MathTutor’s educational value

Aspect	Response category	N (%)
ITS vs. traditional learning	Better than traditional methods	11 (24.4)
	Same as traditional methods	19 (42.2)
	Not sure	15 (33.3)
ITS improves understanding and motivation	Yes	35 (77.8)
	No	10 (22.2)
Impact of ITS on motivation	Motivation increased	20 (44.4)
	Motivation unchanged	24 (53.3)
	Motivation decreased	1 (2.2)
Impact of ITS on the knowledge level	Knowledge level increased	29 (64.4)
	No impact on knowledge level	16 (35.6)

RQ4: Correlation analysis showed no statistically significant associations between usability and perceived learning support or educational value measures (Tables 4). The correlational analysis included four learning support perception items and one item related to task accessibility, whereas the descriptive analysis also included the hints-based support item, which was not entered into the correlational model.

Table 4. Overall system usability

	N	Minimum	Maximum	Mean	Std. Deviation
SUS Score	45	50	88	74.83	9.526
Valid N (listwise)	45

For RQ4, the mean SUS score indicated good overall usability (Table 5).

Table 5. Correlations Between System Usability and Perceived Learning Support Features

Learning support item	ρ (SUS)	p
Feedback motivated me to continue learning	.190	.211
Interface helped me understand content organisation	−.050	.742
Interface helped me identify my weak points	.070	.647
Ease of accessing tasks	−.215	.155

Note. Spearman’s rho; no correlations were statistically significant (p > .05); N = 45.

5. Discussion

This study explored students’ perceptions of MathTutor as an ITS in a higher education mathematics context across four dimensions, presented below.

RQ1 (Learning support features). Students evaluated MathTutor’s learning support most positively when it supported task completion through progress visualisation and content organisation. In contrast, perceptions of feedback and hints were more variable, suggesting that adaptive guidance may be experienced differently among university learners. This aligns with prior research indicating that the perceived usefulness of feedback and hints depends on timing, specificity, and alignment with learner autonomy and self-regulated learning.

RQ2 (Progress tracking and visualisation). Progress tracking emerged as a salient component of students’ experience with MathTutor. Positive evaluations of progress views, reports, and the identification of weak areas indicate that these tools can support monitoring of performance and learning trajectories. In higher education, such features may primarily serve as metacognitive support by helping students reflect on progress and areas for improvement.

RQ3 (Educational value). Students generally perceived MathTutor as a meaningful complement to traditional instruction rather than a direct replacement. Perceived benefits were more consistent for cognitive outcomes (e.g., understanding and knowledge gains), while motivation-related perceptions were less uniform. This suggests that ITS may support learning effectiveness in higher education mathematics, whereas motivational outcomes may depend on additional contextual and individual factors.

RQ4 (Usability and associations). MathTutor achieved good usability ratings, but usability was not associated with perceived learning support or educational value. This suggests that students distinguish between technical usability and pedagogical value, and that usability may function as a baseline expectation for experienced university learners. Instead, features supporting self-regulated learning – such as progress visualisation and structured guidance –may play a more central role in shaping students’ learning experiences.

6. Implications, Limitations and Future Research

Impact and Implications. The findings highlight the value of evaluating intelligent tutoring systems from a student-centred perspective in higher education mathematics. For instructors, MathTutor can serve as a complementary tool that supports students’ monitoring of progress and structured engagement with learning content. For ITS designers, results emphasise the importance of progress visualisation and content organisation as high-impact features that support self-regulated learning, while hints and feedback should remain adaptive and autonomy-sensitive to accommodate differences in learners’ preferences and problem-solving strategies.
Limitations. This study is limited by the small and relatively homogeneous sample of undergraduate students, the reliance on self-reported perceptions, and the context of a single institution and one intelligent tutoring system (MathTutor). Therefore, the findings should be interpreted as context-specific and may not be generalisable to other disciplines, student populations, or ITS implementations.

Future Research. Future work should include larger and more diverse samples, combine student perceptions with objective learning measures (e.g., performance outcomes and system log data), and examine the impact of ITS use through longitudinal or experimental designs in higher education mathematics to better capture changes in learning and motivation over time.

7. Conclusion

This study evaluated MathTutor as an intelligent tutoring system in the context of higher education mathematics, focusing on students’ perceptions of learning support, progress tracking, educational value, and system usability. By adopting a student-centred perspective, the study contributes empirical evidence to an area where user-focused evaluations of ITS in higher education remain limited.

The findings indicate that MathTutor is generally perceived as a supportive and usable learning environment. Progress tracking and content organisation were particularly valued, highlighting the importance of transparent structures and clear feedback in supporting self-regulated learning among university students. While feedback and hints were also perceived as beneficial, students’ responses suggest that adaptive guidance should be carefully balanced to support learning without compromising learner autonomy. MathTutor was viewed primarily as a meaningful complement to traditional instruction rather than a replacement, supporting cognitive aspects of learning more consistently than motivational outcomes. Finally, system usability was evaluated positively, but its limited association with perceived learning support and educational value suggests that usability and pedagogical effectiveness should be considered as related yet distinct dimensions in the evaluation and design of intelligent tutoring systems for higher education.

References

Alvarez, R. P., Jivet, I., Perez-Sanagustin, M., Scheffel, M., & Verbert, K. (2022). Tools designed to support self-regulated learning in online learning environments: A systematic review. IEEE Transactions on Learning Technologies, 15(4), 508–522. https://doi.org/10.1007/978-3-319-98572-5_2

Aleven, V., Borchers, C., Huang, Y., Nagashima, T., McLaren, B., Carvalho, P., … Koedinger, K. (2025). An integrated platform for studying learning with intelligent tutoring systems: CTAT+ TutorShop. arXiv preprint arXiv:2502.10395. https://doi.org/10.48550/arXiv.2502.10395

Ateş, H. (2025). Integrating augmented reality into intelligent tutoring systems to enhance science education outcomes. Education and Information Technologies, 30(4), 4435-4470. https://doi.org/10.1007/s10639-024-12970-y

Awang, L. A., Yusop, F. D., & Danaee, M. (2025). Current practices and future direction of artificial intelligence in mathematics education: A systematic review. International Electronic Journal of Mathematics Education, 20(2), em0823. https://doi.org/10.29333/iejme/16006

Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the System Usability Scale. International Journal of Human–Computer Interaction, 24(6), 574–594. https://doi.org/10.1080/10447310802205776

Ilić, J., Ivanović, M., & Klašnja-Milićević, A. (2024, November). The Impact of ChatGPT on Student Learning Experience in Higher STEM Education: A Systematic Literature Review. In 2024 21st International Conference on Information Technology Based Higher Education and Training (ITHET) (pp. 1-9). IEEE. https://doi.org/10.1109/ITHET61869.2024.10837649

Ilić, J., Ivanović, M., & Klašnja-Milićević, A. (2024). The impact of intelligent tutoring systems and artificial intelligence on students’ motivation and achievement in STEM education: A systematic review. Journal of Educational Studies in Mathematics and Computer Science, 1(2), 5-18. https://doi.org/10.5937/JESMAC2402005I

Ivanovic, M., & Klašnja-Milićević, A. (2024). Smart Technology Enhanced Teaching and Learning of Programming—Several Case Studies. In International Conference on Smart Education and Smart E-Learning (pp. 129-140). Singapore: Springer Nature Singapore. https://doi.org/10.1007/978-981-97-4954-6_11

Klašnja-Milićević A, Ivanović M. (2021). E-learning Personalization Systems and Sustainable Education. Sustainability; 13(12):6713. https://doi.org/10.3390/su13126713

Lin, C. J., Lee, H. Y., Wang, W. S., Huang, Y. M., & Wu, T. T. (2025). Enhancing reflective thinking in STEM education through experiential learning: The role of generative AI as a learning aid. Education and Information Technologies, 30(5), 6315–6337. https://doi.org/10.1007/s10639-024-13072-5

Lyu, Q., Borchers, C., Xia, M., Xiao, K., Carvalho, P. F., Koedinger, K. R., & Aleven, V. (2026). Evaluating a Data-Driven Redesign Process for Intelligent Tutoring Systems. https://doi.org/10.48550/arXiv.2603.29094

Mehrvarz, M., Abdoli, S., Kahveci, E. N., Rahimi, S., & McLaren, B. M. (2025). The impact of hint modality in digital game-based learning: A systematic literature review. In Proceedings of the 19th International Conference of the Learning Sciences (ICLS 2025) (pp. 2939–2941). https://doi.org/10.22318/icls2025.723915

Merino-Campos, C. (2025). The impact of artificial intelligence on personalized learning in higher education: A systematic review. Trends in Higher Education, 4(2), 17. https://doi.org/10.3390/higheredu4020017

Strielkowski, W., Grebennikova, V., Lisovskiy, A., Rakhimova, G., & Vasileva, T. (2025). AI-driven adaptive learning for sustainable educational transformation. Sustainable Development, 33(2), 1921–1947. https://doi.org/10.1002/sd.3221

Wang, S., Wang, F., Zhu, Z., Wang, J., Tran, T., & Du, Z. (2024). Artificial intelligence in education: A systematic literature review. Expert Systems with Applications, 252, 124167 https://doi.org/10.1016/j.eswa.2024.124167