Background: Many programming environments include automated feedback in the form of hints to help novices learn programming autonomously. Some experimental studies investigated the impact of automated hints in the immediate performance and learning retention in that context. Automated feedback is also becoming a popular research topic in the context of formal specification languages, but so far no experimental studies have been conducted to assess the impact of hints while learning such languages. Objective: We aim to investigate the impact of different types of automated hints while learning a formal specification language, not only in terms of immediate performance and learning retention, but also in the emotional response of the students. Method: We conducted a simple one-factor randomised experiment in 2 sessions involving 85 undergraduate students majoring in CSE. In the 1st session students were divided in one control group and three experimental groups, each receiving a different type of hint while learning to specify simple requirements with the Alloy formal specification language. To assess the impact of hints on learning retention, in the 2nd session, 1 week later, all students had no hints while formalising requirements. Before and after each session the students answered a standard self-reporting emotional survey to assess their emotional response to the experiment. Results: Of the three types of hints we evaluated, only the ones that point to the precise location of an error had a positive impact on the immediate performance and none of them had significant impact in the learning retention. Hint availability also causes a significant impact on the emotional response, but no significant emotional impact exists once hints are no longer available (e.g. no significant deprivation effects were detected). Conclusion: Although none of the evaluated hints had an impact on learning retention, learning a formal specification language with an environment that provides hints with precise error locations seems to contribute to a better overall experience without apparent drawbacks. Further studies are needed to investigate if other kinds of hints, namely hints combined with some sort of self-explanation prompts, can have a positive impact in learning retention.

@inproceedings{CunhaMCMS:2024,
 author = {A. Cunha and N. Macedo and J.C. Campos and I. Margolis and E. Sousa},
 title = {Assessing the impact of hints in learning formal specification},
 booktitle = {ICSE-SEET 2024},
 year = {2024},
 pages = {145-155},
 publisher = {ACM},
 abstract = {Background: Many programming environments include automated feedback in the form of hints to help novices learn programming autonomously. Some experimental studies investigated the impact of automated hints in the immediate performance and learning retention in that context. Automated feedback is also becoming a popular research topic in the context of formal specification languages, but so far no experimental studies have been conducted to assess the impact of hints while learning such languages. Objective: We aim to investigate the impact of different types of automated hints while learning a formal specification language, not only in terms of immediate performance and learning retention, but also in the emotional response of the students. Method: We conducted a simple one-factor randomised experiment in 2 sessions involving 85 undergraduate students majoring in CSE. In the 1st session students were divided in one control group and three experimental groups, each receiving a different type of hint while learning to specify simple requirements with the Alloy formal specification language. To assess the impact of hints on learning retention, in the 2nd session, 1 week later, all students had no hints while formalising requirements. Before and after each session the students answered a standard self-reporting emotional survey to assess their emotional response to the experiment. Results: Of the three types of hints we evaluated, only the ones that point to the precise location of an error had a positive impact on the immediate performance and none of them had significant impact in the learning retention. Hint availability also causes a significant impact on the emotional response, but no significant emotional impact exists once hints are no longer available (e.g. no significant deprivation effects were detected). Conclusion: Although none of the evaluated hints had an impact on learning retention, learning a formal specification language with an environment that provides hints with precise error locations seems to contribute to a better overall experience without apparent drawbacks. Further studies are needed to investigate if other kinds of hints, namely hints combined with some sort of self-explanation prompts, can have a positive impact in learning retention.}
 }

Generated by mkBiblio 2.6.26