#### Beatrice Donati <sup>b</sup> <sup>a</sup> Department of Statistics, Computer Science, Applications, University of Florence, Florence, **A structural equation model to measure logical competences**

, Riccardo Bruni b, Federico Crescenzi <sup>a</sup>

,

, Bruno Bertaccini <sup>a</sup>

A structural equation model to measure logical competences

Italy; <sup>b</sup> Department of Humanities University of Florence, Florence, Italy; Silvia Bacci, Bruno Bertaccini, Riccardo Bruni, Federico Crescenzi, Beatrice Donati

## 1. Introduction

Silvia Bacci <sup>a</sup>

Logical abilities are a ubiquitous ingredient in all those contexts that take into account soft skills, argumentative skills, or critical thinking. However, there is a substantial lack of research that addresses the actual possession of such logical abilities by students. With this aim, since October 2020 the University of Florence has promoted a three-stage initiative to collect data in order to measure the logical abilities of students when enrolling at the University. The first stage is an entrance test for assessing the students' initial abilities. This test comprises ten questions, each investigating a specific reasoning construct.

At the second stage, students attend a short training course to strengthen their logical abilities. As third step, in order to evaluate the effectiveness of the course, they take an exit examination, replicating the structure and the difficulty of the entrance test.

This paper builds on the previous work by Bertaccini et al. (2021) where the effectiveness of the course was tested via Item Response Theory (DeMars, 2010; Bartolucci et al., 2019) and test-equating techniques (Battauz, 2015). Building on an enlarged database of students that took the training course and examinations in the second half of 2021 and leveraging auxiliary information about students' characteristics, we estimated a Structural Equation Model (SEM; Duncan, 2014; Bollen, 1989) to have a better comprehension and interpretation of the results reported by Bertaccini et al. (2021).

## 2. Data and methods

### Data

The data that we analyse in this work are obtained from the 80 students that took both the tests and the short training course in the second semester of the academic year 2021-2021. The items of each test aimed at investigating the same logical constructs, namely: Double negation (item code N); Disjunction negation (item code D); Conjunction negation (item code C); Hypothetical reasoning (item code IMPL); Sufficient and necessary conditions (item code NEC); Negation of the universal quantifier (item code NU); Negation of the existential quantifier (item code NE); Modus tollens (item code MT); Syllogism (item code S); Multiple steps deduction (item code DED). The students who respond correctly to a given item are given a score equal to 1, otherwise they are given a score equal to 0.

In addition, we were able to obtain exogenous information on students' characteristics such as their age, the grade obtained at the secondary school, the scientific area the student has enrolled in (i.e. science, social, technic, humanistic) and the years of university enrolment.

Compared to the previous work by Bertaccini et al. (2021), the novelty of this study consists in investigating the role of auxiliary information on students to explain their logic abilities and the effectiveness of the training course. The authors assessed the effectiveness of the course by

<sup>27</sup> Silvia Bacci, University of Florence, Italy, silvia.bacci@unifi.it, 0000-0001-8097-3870 Bruno Bertaccini, University of Florence, Italy, bruno.bertaccini@unifi.it, 0000-0002-5816-2964 Riccardo Bruni, University of Florence, Italy, riccardo.bruni@unifi.it, 0000-0003-2695-0058 Federico Crescenzi, University of Florence, Italy, federico.crescenzi@unifi.it, 0000-0002-0701-4398 Beatrice Donati, University of Florence, Italy, beatrice.donati@unifi.it, 0000-0002-4707-8476

FUP Best Practice in Scholarly Publishing (DOI 10.36253/fup\_best\_practice)

Silvia Bacci, Bruno Bertaccini, Riccardo Bruni, Federico Crescenzi, Beatrice Donati, *A structural equation model to measure logical competences*, pp. 37-41, © 2021 Author(s), CC BY 4.0 International, DOI 10.36253/978-88-5518-461-8.08, in Bruno Bertaccini, Luigi Fabbris, Alessandra Petrucci (edited by), *ASA 2021 Statistics and Information Systems for Policy Evaluation. Book of short papers of the on-site conference*, © 2021 Author(s), content CC BY 4.0 International, metadata CC0 1.0 Universal, published by Firenze University Press (www.fupress.com), ISSN 2704-5846 (online), ISBN 978-88-5518-461-8 (PDF), DOI 10.36253/978-88-5518-461-8

first estimating two Item Response Theory (IRT) models, one for the entrance test and one for the exit test, and then it tested whether there was a significant shift in the distribution of the logical abilities through a test equating procedure. Given the availability of auxiliary information, we opted for a one-step procedure based on SEM so as to take into account both the measurement issue of logical abilities before and after the training course and the structural relations among the observed (i.e., student characteristics) and the latent variables (i.e., logical abilities).

## Methods

A SEM is a multivariate technique used to test complex relationships between observed (manifest) and unobserved (latent) variables as well as relationships between two or more latent variables. Special observed variables, named indicators or items, are used to measure the latent variables. In turn, observed and latent variables distinguish in exogenous variables, which are not explained within the model, and endogenous variables that are affected by other variables in the model (plus an error term). A SEM is characterised by a system of multiple equations, discerning between two sub-models: (i) a structural model, designed to explain the relationships among latent variables as well as among endogenous latent variables and observed variables, and (ii) a measurement model, to link the latent variables to the items. In more detail, the structural model can be expressed by the following equation

$$
\eta = B\eta + \Gamma\xi + \zeta,\tag{1}
$$

where we model the latent logic ability at the exit test η as depending on the latent logic ability at entrance, ξ. Also, in (1) B is a matrix of regression coefficients of the endogenous latent variables; Γ is the matrix of regression coefficients among the endogenous and exogenous latent variables and ζ is vector of errors.

Figure 1: Structural part of the theoretical SEM.

The measurement model is defined by two equations, respectively for the endogenous (2) and exogenous (3) latent variables:

$$x = \Delta\_x \zeta + \sigma,\tag{2}$$

$$y = \Delta\_y \eta + \epsilon,\tag{3}$$

where y is a vector of the item responses, x is a vector of exogenous individual characteristics. In both (2) and (3) ∆<sup>y</sup> and ∆<sup>x</sup> are matrices of factor loading while δ and σ are vectors of error terms.

Note that when one or more exogenous variables are not affected by measurement errors, the structural (1) is simplified as:

$$
\eta = B\eta + \Gamma x + \zeta \tag{4}
$$

## 3. Results

The proposed SEM with all the significant variables is reported in Figure 1. More detailed estimates are shown in Table 1. The estimated SEM presents a good fit to the data with Comparative Fit Index (CFI) equal to 0.944, Tucker-Lewis Index (TLI) of 0.935 and a Root Mean Squared Error of Approximation (RMSEA) of 0.054. All estimates were obtained using the R-package Lavaan (Rosseel, 2012).

Figure 2: Final SEM.

Our results for the measurement part suggest that the course was indeed effective as the estimated items' coefficients are greater in magnitude after having attended the training course. (see Figure 1).

Regarding the regression part of the model, we found that the only significant determinant of the logical skill at entrance was the final grade obtained at the high-school. Also, we found that the only significant effect on the logical skill at exit was the logical skill at entrance. These results confirms that the short training course was indeed useful to sharp students' logical abilities and moreover it is consistent with the preliminary results obtained by Bertaccini et al. (2021).


Table 1: SEM results: measurement part, regression part, and covariances.

# 4. Conclusions

In this paper, we took extended the previous work study by Bertaccini et al. (2021) to offer a more comprehensive and a unified framework to test the effectiveness of the training course for the development of the logical skills of students enrolling at the University of Florence. The effectiveness of the course is confirmed thus making advisable for the University of Florence to design an internal policy so that it may become a standard tool of training and evaluation.

# References

