#### **on lectures fruition and teaching effectiveness**  Maria Cristiana Martini, Marco Furini, Giovanna Galli **Effects of an experimental online education support on lectures fruition and teaching effectiveness**

**Effects of an experimental online education support** 

Department of Communication and Economics, University of Modena and Reggio Emilia, Reggio Emilia, Italy Maria Cristiana Martini, Marco Furini, Giovanna Galli

### **1. Introduction**

The fruition of university courses has significantly changed in the last decade, in consequence of the higher accessibility of technological devices: universities, as well as for-profit companies, started to propose video lectures, as a substitution or in support of traditional lessons, and massive online open courses (MOOCs) have gained more and more importance in the education processes. This tendency comes as an answer to a growing need for flexibility expressed by working students, life-long learning processes, students who have families and care burdens, those with some forms of disability or special needs that make it difficult to attend classes.

Many authors have investigated the effectiveness of video lectures, primarily in comparison with face-to-face classes, with mixing results: some found no significant differences between online and face-to-face courses (Lim et al., 2007; Neuhauser, 2002; Nemetz et al., 2017), while others suggested higher outcomes in online courses (Soffer and Nachmias, 2018; Burkhardt et al., 2008; Connolly et al., 2007; Lim et al., 2008). The Covid-19 pandemic has magnified and accelerated the surge of online teaching, in a way that makes the change hardly reversible. The ongoing debate on the effectiveness of video lectures in higher education is meant to last and intensify.

In this paper, we describe and discuss the implementation, the acceptability, and the effectiveness of an experimental service designed to capture, record, edit and stream video lectures; this system was introduced with the principal aim of supporting, and not substituting, in-class learning. In detail, Section 2 illustrates the experimental service and the main usage behaviours; Section3 presents the main results in terms of effectiveness and usage models, while some conclusions are drawn in Section 4.

#### **2. ONELab: an experimental education support**

ONELab is a system designed to capture, record, edit and stream video lectures, introduces by the Department of Communication and Economics of the University of Modena and Reggio Emilia in September 2017. Traditional face-to-face classes were regularly held, but ONELab was intended to ease the educational experience of those students who cannot attend classes regularly, and to provide an additional support to traditional students. Each classroom is equipped with a video camera pointed on the teacher's desk, an audio system to capture and amplify the teacher's voice, a screen to display the slideshow, and a live video production system to capture, mix, record and stream the video signals (i.e. teacher's video and slideshow) and the audio. After a minimal post-processing, the video lectures are loaded to the online platform and made available for students (see Furini et al., 2018; 2020 for more details).

In the first year of experimentation, from September 2017 to June 2018, 1,376 video lectures were produced, covering the 49 courses offered in the first year of the five bachelor's and master's degrees supplied by the Department, for a total of 2,064 hours. In the academic year 2018/19 these numbers doubled, and further increased in 2019/20, as the courses offered in the second and third year joined the experimentation.

Maria Cristiana Martini, University of Modena and Reggio Emilia, Italy, cmartini@unimore.it

Marco Furini, University of Modena and Reggio Emilia, Italy, marco.furini@unimore.it, 0000-0003-1094-6521

FUP Best Practice in Scholarly Publishing (DOI 10.36253/fup\_best\_practice)

Maria Cristiana Martini, Marco Furini, Giovanna Galli, *Effects of an experimental online education support on lectures fruition and teaching effectiveness*, pp. 67-72, © 2021 Author(s), CC BY 4.0 International, DOI 10.36253/978-88-5518-461-8.13, in Bruno Bertaccini, Luigi Fabbris, Alessandra Petrucci (edited by), *ASA 2021 Statistics and Information Systems for Policy Evaluation. Book of short papers of the on-site conference*, © 2021 Author(s), content CC BY 4.0 International, metadata CC0 1.0 Universal, published by Firenze University Press (www.fupress.com), ISSN 2704-5846 (online), ISBN 978-88-5518-461-8 (PDF), DOI 10.36253/978-88-5518-461-8

<sup>57</sup> Giovanna Galli, University of Modena and Reggio Emilia, Italy, giovanna.galli@unimore.it

The analysis of the first year log files shows that students' reaction was enthusiastic, with an average of 8,323 video lectures played each month in the first year, from September 2017 to August 2018, and a peak of 14,483 views in January 2018, during the exams session. Overall, during the year students watched video lectures for 71,488 hours. The most watched lectures relate to technical subjects, such as mathematics and statistics, where students take advantage from the possibility to replay some difficult passages until they are clear.

Video lectures are watched mostly during the teaching semester, but a significant part of students resort to watch them when the semester is over, especially during the exams sessions. The usage analyses show that students watch video lessons mostly during the working hours, from Monday to Friday; however, 16% of the views happen during the week-ends, and 22% in the evening and during the night, suggesting that, when given the opportunity, students tend to customise the learning process to their needs and life-style.

Of the 1251 freshmen in the academic year 2017/18, only 319 (25.4%) never accessed the ONELab platform to watch video lessons during their first year, while 13.4% never accessed it, neither in the first or in the second year. Table 1 shows the percentage of non-users among different categories of students, separately for undergraduate and graduate students.


**Table 1.** Percentage of students who never accessed the ONELab video lectures during the first year and in the first two years, per students' characteristics.

*Significance level: \*\*\* 99.9%, \*\* 99%; \* 95%; ° 90%.* 

The use of ONELab is particularly popular among graduate students, while almost one undergraduate student out of four never watched any video lecture. Females are more conscientious than males, and look for every provided support to enhance their preparation, but the difference is statistically significant only among undergraduates. Students affected by Specific Learning Disorders are only 12, but none of them missed the new learning support, that allows for a certain degree of self-paced study, ensuring more control over their learning. On the other hand, undergraduate students coming from technical and vocational colleges are less organised in their study, and overlook video lectures to a greater extent, while non-EU foreigners miss this support both as undergraduate and graduate students.

The recourse to video lectures is extremely scarce among students who end up dropping out the university in the first year. To some extent, these students might have dropped out because they did not take advantage of ONELab to support their studies, but it might also be that some students decided to leave the university, or to transfer to a different degree, so early that they did not have time to try the video lectures.

### **3. Effectiveness of the video lectures**

The high percentage of usage is a first indirect indicator of effectiveness of the video lectures, but we aim at assessing the benefits of the video lectures in terms of learning outcomes, namely the number of acquired (European) credits and the final grades. We focus on students enrolled in 2017, and analyse data on their ONELab accesses and academic achievements during the academic years 2017/18 and 2018/19. We do not consider the third year, neither for three-year courses, because it corresponds to the Covid-19 pandemic outbreak, when face-to-face classes were totally replaced by video lectures in the second semester, which makes the situation incomparable.

Since early dropouts produce a low number of credits (or no credits at all) and a low level of access to the online platform, we remove them from the following analyses to exclude the existence of spurious relationships.

Table 2 shows that the average number of credits acquired by students who watched video lectures is largely bigger than for those who did not, both in the first and in the second year, and the difference is statistically significant. Students who accessed ONELab also performed better in terms of grades: the average grade of accessing students is higher than for non-accessing students, although the difference is only marginally significant in the second year. However, separate analyses carried out on undergraduate and master students demonstrate that students who accessed the video lectures show significantly better performances only in terms of acquired credits.


**Table 2.** Average number of acquired credits during the first year and average grade for ONELab users and non-users, per degree level.

The rough distinction between students who never watched video lectures and those who accessed the platform at least one time, although simplistic, has proven to be meaningful in explaining performance differences among students. We try to describe in more detail the different usage styles of those who accessed the platform at least one time in the two years through the following variables, separately measured on the first and the second year:

 *Total number of accesses*: this variable measures the general degree of usage during the first year. It varies between 0 and 864 for the first year, and between 0 and 1,885 in the second year; the average is respectively 75.3 and 112.7;


Some of these variables show unexpectedly high values (for example, students registering 1,885 total accesses, or students who played 698 times the video lectures of a single course), and the reason is twofold. First, every single access does not correspond to a complete play of the video-lecture; as reported by many students, "critical" passages, especially on some technical topics, have been repeatedly reloaded and re-played, and sometimes a lecture is erroneously played while looking for another one, or for a different part of the same recording. In addition, the platform was a novelty that probably raised curiosity among students, leading some to explore the resources far beyond the actual usage.

Based on the ten described variables, we perform an agglomerative hierarchical cluster analysis; the agglomeration criterion is the Ward's method that, at each step, merges the couple of units/clusters that leads to minimum increase in total within-cluster variance. The distance is the squared Euclidean. Given the different order of magnitude, all variables have been rescaled to the [0-1] range using min-max normalization.

This cluster analysis suggests the existence of four distinct groups; combining these clusters with the group of absolute non-users, we obtain the five profiles described in Table 3 (first year dropouts are excluded from the analysis):


users. They are probably students who discovered the video lectures only late in the first year, or approached them first without enthusiasm but found out they were useful than expected for their preparation.

5. Zealous users: They amount to 9.5% of students, and they accessed the platform hundreds of times; they accessed more or less all of the courses provided in their study program, and they were assiduous on most of them, playing video-lectures from each single course up to 70 times in the first year, and even up to 120 in the second year.



For each group of students, the learning performances are reported in Table 4. The level of performance increases with the frequency of usage of the ONELab services. Regarding the number of credits, all the group means are statistically different at least at a 95% significance level, except for Converted users, that are not significantly different from Regular users in the first year, and from Zealous users in the second year. The average grade shows only slight differences, nevertheless consistent with a better performance for regular and zealous users. Differences between graduate and undergraduate students are not noticeable.



# **4. Conclusions**

In this paper, we analysed the effectiveness of an experimental platform to provide university students with remote access to video lectures to support traditional face-to-face classes. Results show higher learning outcomes for students who regularly watched the video lectures, primarily in terms of the number of acquired credits. This is consistent with the conclusions drawn in Cagliero et al. (2017), who report higher student's success rates following the introduction of an analogous system to provide video-recorded lessons to complement in-class learning. In our experience, the beneficial is particularly pronounced for undergraduate students, although they show a more limited recourse to the platform than graduate students.

However, a more careful analysis of the principal beneficiaries of the implemented service casts a shadow on the capacity of the system to smooth learning ability differences and recover those students who have a hard time keeping pace with their studies and exams. Video lectures, in fact, are mainly watched by conscientious students, i.e. females, students coming from "lyceum" high school, and graduate students, who aim at improving their learning through additional educational material, while critical students are those who access the platform less. This suggests that the information about the new service should be conveyed to students in a more careful and focused way, addressing especially to students at risk of being left behind and dropping out. In this sense, given the strong connection between dropouts and video lectures (non) usage, monitoring and analysing the access data might help to detect critical students, and try to prevent them from dropping out.

Finally, a negative consequence of the introduction of this service was a dramatic decrease in the number of students attending classes, much before the university classrooms were emptied by the pandemic crisis. When face-to-face classes will return to normality after more than one year of online teaching, the problem is likely to become even more compelling, forcing teachers and pedagogues to rethink face-to-face classes in a more interactive and engaging format.

## **References**

