#### with a latent class approach G. Damiana Costanzo, Michelangelo Misuraca, Angela Coscarelli **Profiling students' satisfaction towards university courses with a latent class approach**

Profiling students' satisfaction towards university courses

Dep. of Business Administration and Law, University of Calabria, Arcavacata di Rende, Italy G. Damiana Costanzo, Michelangelo Misuraca, Angela Coscarelli

# 1. Introduction

Collecting and analysing students' opinions on their learning experiences during enrollment in an academic program is widely recognised as a key strategy for evaluating tertiary education quality. Academic institutions require students to participate every year in specific surveys, aiming to gather their viewpoints about the organisation of the single courses and their feelings about the teaching activity's traits and effectiveness. The *Standards and guidelines for quality assurance in the European Higher Education Area* (ESG, 2015), for example, underline the relevance of students' voices in the assessment processes. Likewise, students' and graduates' opinions constitute essential information for the quality assurance of the Italian university system. The National Agency for the Evaluation of Universities and Research Institutes (ANVUR), to harmonise the data collection in all the universities, provides guidelines to build the questionnaire administered in the surveys on students' opinions. Initially released in 2013, these guidelines were updated in 2017 and are currently in use. The survey, under article 1 of Law 370/1999, is mandatory and autonomously carried out every academic year by the different institutions, representing one of the fundamental sources for the so-called AVA system<sup>1</sup> (i.e., self-assessment, periodic assessment, accreditation), introduced in Italy by Law 240/2010 and Legislative Decree 19/2012. This system aims to improve teaching and research quality in universities, applying a model based on internal planning and management procedures.

This study investigates students' satisfaction towards courses delivered at the University of Calabria, focusing on the academic year 2020–2021. In this period, almost corresponding to the occurrence of the second and third waves of COVID-19 outbreak in Italy, traditional teaching methods were substantially disrupted by the social distancing actions pursued by the Italian government, enhancing the use of blended and hybrid learning (e.g., Aboagye et al., 2021; Chaturvedi et al., 2021). Here we considered the first-year courses since students enrolled in 2020–2021 programs experienced the completion of their previous educational degrees in the first wave of COVID-19. We carried out a latent class analytical strategy to profile students' satisfaction at a course level, taking into account their interest in each course and their perceptions about the course organisation and the instructor's behaviour. Since the items listed in the survey are expressed as 4-point balanced scales, we used the so-called Latent Profile Analysis (LPA) to identify unobserved course profiles, starting from students' responses to the continuous indicators concerning course satisfaction.

# 2. Theoretical framework and data structure

LPA is a statistical approach belonging to finite mixture models. It can be seen as a variant of latent class analysis (LCA, Oberski, 2016) aiming at identifying a set of discrete, exhaustive, and non-overlapping groups of subjects characterised by different patterns of responses on indicator items, typically represented by ordinal or continuous manifest variables. Each subject

Referee List (DOI 10.36253/fup\_referee\_list)

FUP Best Practice in Scholarly Publishing (DOI 10.36253/fup\_best\_practice)

G. Damiana Costanzo, Michelangelo Misuraca, Angela Cosca, *Profiling students' satisfaction towards university courses with a latent class approach*, © Author(s), CC BY 4.0, DOI 10.36253/979-12-215-0106-3.04, in Enrico di Bella, Luigi Fabbris, Corrado Lagazio (edited by), *ASA 2022 Data-Driven Decision Making. Book of short papers*, pp. 17-22, 2023, published by Firenze University Press and Genova University Press, ISBN 979-12-215-0106-3, DOI 10.36253/979-12-215-0106-3

<sup>1</sup>https://www.anvur.it/attivita/ava

G. Damiana Costanzo, University of Calabria, Italy, dm.costanzo@unical.it, 0000-0003-2295-3278 Michelangelo Misuraca, University of Calabria, Italy, michelangelo.misuraca@unical.it, 0000-0002-8794-966X Angela Coscarelli, University of Calabria, Italy, angela.coscarelli@unical.it

is assigned to the most likely latent group, i.e. an unobserved profile that generates patterns of responses on the indicators. LPA may be considered, therefore, a case-centred analytic tool focusing on similarities and differences among subjects rather than relations among variables (Bergman and Magnusson, 1997).

Assuming that the continuous (or ordinal) variables are normally distributed within each latent profile, a model of G components aims at representing the distributions of the observed subjects' scores on a set of indicator items x<sup>i</sup> (i = 1,...,n), given the latent categorical variable Θ, as a function of the probability of the subjects to be typed into a profile and the profilespecific normal density:

$$f(\mathbf{x}\_i|\Theta) = \sum\_{k=1}^{G} \pi\_k f\_k(\mathbf{x}\_i|\theta\_k) \tag{1}$$

where π<sup>k</sup> and θ<sup>k</sup> represent the probability of belonging to the k-th latent profile (with π<sup>k</sup> summing to 1 across the different profiles) and the estimation of the mean and the set of variances/covariances for k, respectively (Tein et al., 2013).

LPA has been recently used in the educational domain, for example, to identify students' time use profiles (Fosnacht et al., 2018), to explore motivation patterns in learning environments (Hodis and Hodis, 2020), to define types of social support for student resilience during the COVID-19 pandemic (Mai et al., 2021). In the following, LPA is used to identify academic course profiles, considering students' opinions about the courses they attended.

The yearly survey about students' satisfaction carried out at the University of Calabria (Italy) was used to build a dataset of course response patterns. The questionnaire administered in the survey is built following the ANVUR guidelines to harmonise the data collection in all the different Universities.

We focused, in particular, on the academic year 2020–2021. During this year, because of the COVID-19 pandemic, courses have been delivered in presence, in distance (via online platforms), or by mixing the two types. The total number of the collected questionnaires was 77,049 for the courses of 73 academic programs included in the entire university catalogue. After filtering only the questionnaires completed by first-year students that attended at least half of the lectures for each first-year course (24,064), we selected 10 different items concerning for each course the *Interest* of the students, the *Instructor* behaviour and the *Teaching* characteristics. Table 1 lists the three dimensions and the corresponding items.


Since the items listed in the survey are expressed as 4-point balanced scales (*definitively no*, *more no than yes*, *more yes than no*, *definitively yes*), we converted the ranks in scores from 1 to 4. Moreover, since the items INT 1 and INS 3 showed a certain number of missing data (10.8% and 6.7%, respectively), we performed a multiple imputation procedure in order to save all the selected cases in the dataset. Finally, the cases were collapsed at a course level by averaging the individual response patterns, and hence the course response patterns were standardised. The resulting 657 × 10 matrix was used in the analysis. The imputation of missing data was performed by using the R library mice, whereas LPA was performed by using the library mclust (van Buuren and Groothuis-Oudshoorn, 2011; Scrucca et al., 2016).

### 3. Model selection and main findings

A key question in finite mixture modelling is how many latent classes should be included. The selection of the best model was carried out by jointly evaluating the Bayesian Information Criterion (BIC) and the Integrated Complete-data Likelihood (ICL) criterion proposed by Biernacki et al. (2000). ICL appears more robust than BIC, adding a penalty on solutions with greater entropy or classification uncertainty.

In addition to the number of profiles, the model can be specified in terms of whether and how the variable variances and covariances are estimated. Geometric features (shape, volume, orientation) of the clusters are determined by the covariances. We estimated two kinds of models, considering in both cases profiles with equal volume and shape. In the *EEI* model, the indicator variables are set to have zero covariances within and across profiles. Indicator-variable variances are allowed to vary within profiles but are constrained to be equal between them. In the *EEE* model, the complete variable (co)variance matrix is estimated, with variances and covariances constrained to be the same across the profiles.

Table 2 shows the fit indices of the two models, including the log-likelihood ℓ (with the corresponding degrees of freedom), the BIC and the ICL.


Table 2: Data fit of EEI and EEE models.

The log-likelihood had lower values for each model that increased of one latent profile, with EEE models showing lower values than EEI models. Regarding the information criteria, we observed that the two indices confirmed that EEE models are better than EEI models. Jointly considering the results of the three selection criteria listed above, we selected the EEE model with 4 latent clusters. To validate our choice, we performed a bootstrap likelihood ratio test (BLRT) to verify the null hypothesis that a (k + 1)-profile model is equal to or better than a k-profile model, i.e. that an increase in the number of profiles increases fit (Nylund et al., 2007). Table 3 shows the results of the test and the corresponding p-values, suggesting that a 4-profile solution is optimal.

The four profiles allowed to identify different satisfaction patterns for the first-years courses under investigation. Table 4 reports the mean values of the different items per profile, looking at each as a factor score with a mean equal to 0 (due to standardisation).

Profile 1 (90.26% of the courses) showed for each item a score above 0, identifying courses with an average level of student satisfaction. Profile 2 (5.48% of courses) showed for the *Interest*



Table 4: Mean values of the ten items per profile.

and *Instructor* dimensions − as well as the items of the *Teaching* dimension related to course organisation − scores above 0, with values greater than the corresponding values of Profile 1. The only negative value was observed for the clarity of instructors, with a value just below 0. Nevertheless, the courses likely belonging to this profile showed negative scores for the adequacy of the prior knowledge required to attend the lectures and the learning materials, with a peak for the course workload that is not perceived as proportional to the ECTS. Profile 3 (2.74% of the courses) is characterised by a fair share of students' dissatisfaction, particularly concerning the instructors' activities. Courses likely belonging to this profile showed very negative scores for the interest stimulated by the instructor, the clarity of this latter in the explanations, and a peak for the punctuality of the instructors. On the other hand, the profile had positive scores for the adequacy of the prior knowledge and the workload. Profile 4 (1.52% of the courses), finally, was even more characterised by dissatisfaction, with scores significantly above 0 for all the items. In particular, the *Instructor* dimension showed very negative scores, together with a low score for the interest stimulated by the instructors themselves. At the same time, we observed negative scores for the items of the *Teaching* dimension, with an unfavourable evaluation of the coherence of the courses with respect to their syllabus and an inadequacy of prior knowledge and learning materials. A joint lecture on the profile membership and some covariates can offer valuable insights into the less satisfying courses.

To characterise the profiles, we focused at this stage of the research only on the nature of academic programs in which courses are embodied, taking into account if the courses belong to undergraduate and single-cycle programs (namely, *Laurea* and *Laurea Magistrale a Ciclo Unico*) or to master programs (namely, *Laurea Magistrale*). Focusing on the profiles that showed a certain degree of dissatisfaction, we observed that 83.3% of courses in Profile 2 are embodied in master programs. At the same time, the 66.7% and the 50.0% of courses in Profile 3 and Profile 4, respectively, are embodied in master programs. This aspect can help characterise the source of the satisfaction (or dissatisfaction) perceived by students, helping academic institutions perform targeted interventions on the courses showing specific shortcomings. Considering, for example, the courses belonging to Profile 2, preparing more effective learning materials and re-designing the course programs may improve students' satisfaction, taking into account the higher expectations of master students. Nevertheless, the data used in this study referred to the academic year 2020–2021, which occurred during the second and third waves of the COVID-19 pandemic in Italy. This means that a comparison with other academic years is necessary to detect potential structural weaknesses that deserve greater attention.

# 4. Final remarks and future research

This study analysed students' satisfaction towards courses attended during the first year of the academic programs delivered at the University of Calabria. The different course types were depicted using a latent class approach known as LPA, a finite mixture model able to measure the impact of an unobserved categorical variable defining different latent profiles on a set of continuous variables. The survey structure did not allow us to evaluate satisfaction at a student-level since each questionnaire is registered with a different ID due to the privacy policies implemented at the University of Calabria. For this reason, we evaluated satisfaction at a course level, trying to identify different course types and analyse their characteristics. By contrast, the data averaging did not allow considering the variability of students' opinions for each course, posing a possible limitation of the present study.

The response patterns expressing students' satisfaction/dissatisfaction levels to the different aspects concerning teaching and the organisation of learning activities characterised a 4-profile model, revealing for each one the most critical aspects. Remarkably, together with a profile encompassing the majority of courses and revealing a general degree of satisfaction, we identified three course profiles expressing a different degree of dissatisfaction instead. A focus on the academic programs the latter courses belong − considering if they were included in first-cycle, single-cycle or second-cycle programs − showed that graduate students have potentially higher expectations than undergraduate students, evaluating in a more critical way the course organisation and the workload required by each course. A noteworthy aspect is that these students experienced the rapid change of teaching induced by COVID-19 during the last year of their first-cycle degree, starting a new cycle of study in the uncertainty caused by the ongoing social distancing and the limitations established by the Italian government.

The analytical strategy employed in the study can be easily implemented as a visual tool helping academic institutions at a department level (e.g., by the Joint Teaching-Student Committees) or at a university level (e.g., by the Independent Evaluation Units) in the quality assurance systems, giving a hint of which courses have to be carefully monitored and at the same time of which aspect are perceived as more critical by the students. Currently, a different version of the survey has been tested in some universities by the National Agency for the Evaluation, and it is ready to be released in the short term a new version of the guidelines.

Several future developments of the study can be considered. First, the effect of some metadata (as covariates) may help better define and characterise the profiles, taking into account the cycle dimension and the attendance rate, and the domain of the courses (e.g., courses based on quantitative or qualitative methods). Second, a longitudinal analysis may help in evaluating how a shock like the COVID-19 pandemic or a change in the academic programs' regulation influences the perception of courses' quality, estimating the transition of a course (in a probabilistic fashion) from a latent profile to another one in different periods (Collins and Lanza, 2013). This latter variant of latent models can enrich the analytical strategy's informative power, allowing for evaluating the quality across time.

# References

