#### Emma Zavarronea , Maria Gabriella Grassiab , Rocco Mazzab , Alessia Forcinitib a Department of Humanities Studies, Iulm University, Milan, Italy. **Emergency remote teaching: an explorative tool**

**Emergency remote teaching: an explorative tool** 

b Department of Social Sciences, University Federico II, Naples, Italy. Emma Zavarrone, Maria Gabriella Grassia, Rocco Mazza, Alessia Forciniti

## **1. Introduction**

The worldwide rapid spread and severity of the infectious disease caused by Coronavirus forced the WHO to declare a global state of pandemic emergency during March 2020, by leading the governments around the world to adopt policies that created the widest rift of education systems in human history. As 85% of worldwide countries, also Italy has temporarily closed each educational institution, by causing the disruption of tertiary education for 16.89% of the Italian learner's population.

To ensure the "pedagogic continuity", universities adopted the transitioning from traditional face-to-face to online learning (e.g., Tallent-Runnels *et al*., 2006; Sangrà *et al*., 2012; Todri *et al.*, 2021). In particular, the shift to fully remote teaching solutions as response to crisis is called by Hodges *et al.* (2020) as *emergency remote teaching (ERT).* This paradigm shift created changes about the perception of the learning process (Lederman, 2020) and supposed significant didactic efforts in terms of digitalisation and interactive pedagogical approaches. The ERT main goal is not to re-design a long-term educational ecosystem, but to supply a rapid and temporary solution to a crisis condition (Appolloni *et al*., 2021) adopting a learning framework different from online ones.

This implicates venturing into uncharted territory with several logistical challenges and attitudinal modifications (Ribeiro, 2020), also in terms of teaching-learning assessment.

Thus, the evaluation of ERT on the quality of higher education becomes a sensitive issue. This paper raises the evaluation of the effectiveness of teaching delivery during the transition from a traditional model to the ERT one. The focus is to detect how ERT is perceived and how it can connect to students' performance and to the quality of education. The aspect of quality has been dealt out in terms of European Standards and Guidelines (ESG) adopted in 2005 by the Ministers of Higher Education of the countries participating in the Bologna Process (1999; Grano and Ricci, 2009). To ensure the quality of the tertiary education system, European countries have established monitoring agencies. In Italy, this agency is ANVUR, which received official accreditation by the European association for quality assurance in higher education (ENQA) in 2019. However, during the Coronavirus health emergency, ANVUR did not provide guidance to universities on how to manage distance learning and its evaluations, relying on the autonomy of universities which continued to adopt the traditional evaluation systems. In a higher education landscape dominated by quality assurance view for evaluating teaching quality and student satisfaction (Fabbris, 2007), the teaching-learning ERT solutions adopted could not be fit since it is affected by several new factors and the principles recalled by the Bologna Process may be not appropriate.

Thus, the paper focuses on an alternative simple tool for evaluating the quality of teachinglearning in ERT cases. Our research question has an explorative nature: we are interested in detecting empirical evidence about the learning assessment and engagement in higher education with focus on students' engagement and their success performance during ERT.

These dimensions have been represented in the ERT map inspired by perceptual maps of the consumer' theory (Whitlark & Smith, 2001; Gower *et al*., 2010). In our model, the ERT map has been realised by a data integration perspective which considers the university administrative and textual data in a multivariate scenario of methodologies. Textual information is represented by the student voice, since this provides essential information for Quality Assurance systems and for

51 Emma Zavarrone, IULM, International University of Languages and Media, Italy, emma.zavarrone@iulm.it, 0000-0001-9509-8773 Maria Gabriella Grassia, University of Naples Federico II, Italy, mariagabriella.grassia@unina.it, 0000-0002-7128-7323

Rocco Mazza, University of Naples Federico II, Italy, rocco.mazza@unina.it, 0000-0002-4901-5225 Alessia Forciniti, IULM, International University of Languages and Media, Italy, alessia.forciniti1@gmail.com

FUP Best Practice in Scholarly Publishing (DOI 10.36253/fup\_best\_practice)

Emma Zavarrone, Maria Gabriella Grassia, Rocco Mazza, Alessia Forciniti, *Emergency remote teaching: an explorative tool*, pp. 61-66, © 2021 Author(s), CC BY 4.0 International, DOI 10.36253/978-88-5518-461-8.12, in Bruno Bertaccini, Luigi Fabbris, Alessandra Petrucci (edited by), *ASA 2021 Statistics and Information Systems for Policy Evaluation. Book of short papers of the onsite conference*, © 2021 Author(s), content CC BY 4.0 International, metadata CC0 1.0 Universal, published by Firenze University Press (www.fupress.com), ISSN 2704-5846 (online), ISBN 978-88-5518-461-8 (PDF), DOI 10.36253/978-88-5518-461-8

monitoring and managing information for universities processes. It represents one of the central issues in the most recent version of the European Guidelines and Standards for Quality Assurance in the European Higher Education Area adopted at the Yerevan meeting in 2015 and in AVA 2017 (where AVA stands for Self-assessment, Evaluation, Accreditation; ANVUR, 2021).

In the following, section 2 introduces the theoretical framework; section 3 describes our model and data; section 4 shows the main findings; section 5 presents the future directions.

# **2. Theoretical framework**

During the last two years, the ERT condition caused by the *SARS-CoV-2* pandemic allowed to enrich the literature by several contributions aimed to propose methods and techniques to evaluate the aspects of online teaching-learning in higher education.

Among approaches to evaluate engagement and performance, Bawa (2020) examined the effects of the pandemic related ERT on learners' grades, using an experimental design to investigate the shift of online learning. The analysis has been realised by comparing the same course content and assessment methods for an experimental group formed of students enrolled during 2019-2020 and a control group with students who attended college before the health crisis. The results showed better outcomes in the experimental group than in the control group, above all for highest range performance. Dost *et al.* (2020) investigated attendance and perceptions of medical students across 40 UK schools during May 2020. By means of a cross-sectional study conducted on a national level via an online survey on a 20-items questionnaire measured on Likert scales, the study examined the experiences of online teaching, perceived benefits and barriers and the reached outcomes. From the findings, it emerges that online teaching platforms allow students to digest information in their own time and at the same time to discuss them with peers and, showed to be effective in terms of achieving learning outcomes. Huang *et al.* (2020) analysed the students' engagement by adopting a mixed-methods design: from a quantitative descriptive approach to a qualitative visual method.

At the end of course, all students were given a format inspired by the Motivated Strategies of Learning Questionnaire evaluated on Likert scales about four components: task value, metacognitive self- regulation, effort regulation, and peer learning. In addition, the demographic information and the answers to open- ended questions were coded and grouped into themes by qualitative approach. The results demonstrated the engagement does not depend on learning experience but on extrinsic goal orientation.

Therefore, in a landscape of emerging difficulties to evaluate the student's performance and engagement in ERT contexts as the Coronavirus one, our work proposes a strategy of quantitative analysis aimed to show empirical evidence about the learning assessment and engagement in higher education.

## **3. Model and data**

To assess the quality and ERT success our proposal is based on study of two dimensions: the students' engagement (SE) and success performance (SP) that represent the proxy variables used for constructing our model of analysis. SE has a textual nature comes from students' voice whilst SP uses the career data of students. We obtained our model (*Fig.*1) by integrating these sources of information: textual ones linked to the analysis of the strengths of the engagement and administrative ones related to the results of students' performance. We operated a multidimensional analysis to study the data from these different sources.

### *Fig.1: Model flowchart*

The measurement of the SE was focused on answers referring to the strengths of the engagement which have been analysed by means of the textual approach. We applied bags of words scheme to transform unstructured texts data in a structured data matrix to analyse. We operated the following pre-treatment operations: 1. Texts normalization; 2. Lemmatization; 3. Stopwords deleting. We built a Document-Term Matrix (DTM) without low frequency words (frequency cut at 5 minimum term frequency and 2 at document frequency of the feature) and empty documents. The DTM has texts for each row (755) and words in columns (790).

The SP measurement is connected to the quantification of the students' results and for reaching this goal the administrative data were used. To construct the Success Performance Indicator (SPI), we considered the average of marks (*M*) and ECTS (European Credit Transfer and Accumulation System) or credits (*C*) achieved before and during the pandemic. More precisely, for measuring each variation of marks and ECTS we used (1):

$$
\Delta \mathcal{M}\_{\ell} \mathcal{C}\_{\ell} = \frac{\mathcal{M}\_{\ell\_1} \mathcal{C}\_{\ell\_1} - \mathcal{M}\_{\ell\_0} \mathcal{C}\_{\ell\_0}}{\mathcal{M}\_{\ell\_0} \mathcal{C}\_{\ell\_0}} \tag{1}
$$

where *Mi0* and *Ci0* denote the average of marks and ECTS obtained until February 2020 for each *i-th* student, where *i*=*1, ... N; Mi1* and *Ci1* are the average of marks and ECTS achieved from February to November 2020.

Therefore, the SPI*<sup>i</sup>* was computed by (2):

$$SPI\_l = \frac{\Delta \,\, \mathcal{M}\_l \,\, \mathcal{C}\_l}{\max(\mathcal{M}, \mathcal{C})} \; 100\,\, \tag{2}$$

It considers each variation of average of marks and ECTS in relation to the maximum average of marks and ECTS (maximum of mark and ECTS number, 30x180=5400) reachable by 100.

To simplify the next steps, we recoded the SPI considering the quartile (Q) of SPI for each *i-th* student:

$$\text{SPI}\_{\mathbb{I}} = \begin{cases} 1 - \text{low if } \text{SPI}\_{\mathbb{I}} \le \text{Q}\_{\mathbf{1}\{\text{SPI}\_{\mathbb{I}}\}} \\ 2 - \text{medium if } \text{Q}\_{\mathbf{1}\{\text{SPI}\_{\mathbb{I}}\}} < \text{SPI}\_{\mathbb{I}} \le \text{Q}\_{\mathbf{2}\{\text{SPI}\_{\mathbb{I}}\}} \\ 3 - \text{high if } \text{SPI}\_{\mathbb{I}} > \text{Q}\_{\mathbf{3}\{\text{SPI}\_{\mathbb{I}}\}} \end{cases}$$

The SPI can be interpreted as: low performance when SPI*i* is lower or equal to the first quartile (SPI*i* ≤ *Q1*); medium performance if SPI*i* is between the first and the second quartile (*Q1* < SPI*i* ≤ *Q2*); high performance in the case of SPI*<sup>i</sup>* is greater to the third quartile (SPI*i* > *Q3*).

Using the SPIi recoded in terms of quartile and the lexical dimension of SE, we created a contingency table to cross the proxy variables of our model. We inserted into this matrix the performance information at the student level in the rows and lexical keyword extracted from

DTM in the columns. This was possible because the DTM matrix (from SE study) and SP measurement matrix had the same rows. With a multidimensional data analysis approach, we operated a dimensional reduction through Correspondence Analysis (CA). A factorial plan or perceptual map, where we plotted our features, has been created. With this strategy, we can study the association between the two dimensions taken into account in our model. The intersection of these two proxy variables on the factorial plan shows on the horizontal axis SE while on the vertical axis SP. According to our model three principal theoretical areas can be imagined:


The population is composed by the students enrolled in a three-year degree course at Iulm University of Milan (*N*=5000) during the academic year 2019-2020. The survey on ERT weaknesses and strengths had a response rate equal to 14% of the population. The investigated variables are related to the students' career: year of the course; type of high school; gender; average of the marks and ECTS obtained until February 2020, before the Coronavirus, and from February to November 2020, during ERT.

The female students are overrepresented. They respectively represent the 82.3% of the whole student population, the overall students enrolled in the first year are 44.3%. The Iulm University is composed by three Faculties: the 68% of the respondents studied at the Faculty of Communication and other respondents are equally split in other faculties.

## **4. Results**

For the SE, 755 texts were parsed and tokenized from the corpus. The results are a set of strings containing words used in documents. Subsequently, we reduced language variability to avoid possible sources of noise and to improve the effectiveness of the next analytical steps. The step consisted in the normalization of words, spelling and brought back each inflected word in its canonical form. Finally, we pruned non-informative words and non-alphabetic characters from the texts. The vocabulary size consisted of 790 types. Subsequently, we reduced the dimensionality of this matrix, we filtered sparse words (with a sparsity threshold of 2%). At the end of the process, each document was represented as a document-vector and the number of types was 190. The comparison wordcloud plot shows the most frequent terms for separate grouping level of SPI (*Fig. 2 (b)*). The wordcloud allows to capture some differences among the words related to the different level of SPI: terms like "riascoltare" "listen to the lesson again" characterized the low level of SPI, whilst terms with negative meaning in the SPI were in the high-level area.

As we affirm in Par.3, a contingency table was created, and we obtained the factorial plan through the CA (*Fig.3*) where the first dimension explains 54.4% and represents the ERT success while the second one explains 45.6% and denotes the SE. We can see three sections characterized by SPI levels splitting the map in the three horizontal levels: low, middle and high. On the contrary, the SE can be read easily from right to left, where we find an individual student engagement and the collective students' engagement respectively. At first glance, we discover that the individual engagement is at the high level of SPI. This puts the semantic dimension related to the individual experience close to the high and low performance factors, as far as an exploratory analysis is concerned. We want to highlight that the two polarities (high and low) referring to the performance are both in the same half plane. Obviously, both refer to two different ways of experiencing distance for students. The difference consists, for the high SPI, in the facilitated access to the technologies made available and the possibility of optimizing the time available for the study. The low performance is close to the properly teaching dimension and relative to the contents of the courses.

*Fig.3: Factorial plan* 

## **5. Future directions**

The work proposes the data integration approach to create the ERT map. The CA allows to explore this integration using the contingency table built on SPI and SE, proxies developed to detect the success of the performance and the level of student engagement respectively. Attention should be paid to the use of short texts, inspired by the dialogue on social media, which do not always allow- based on the Italian language- to extract the true underlying concept. For this reason, future developments are moving in two directions: creation of an Italian dictionary, specifically for evaluating ERT, and creation of indicators that can be used in an agile way for subsequent ERT evaluations. The indicators could also be useful for drop out screening and prevention by monitoring the level of collaborative engagement.

# **References**


Il%20Processo%20di%20Bologna.%20Documenti%20ufficiali.pdf. Last access: 15/04/2021.

