## healthcare Pietro Renziab, Alberto Franciab **EGIPSS model for the evaluation of performance in healthcare**

EGIPSS model for the evaluation of performance in

Department of Economics, Science and Law, University of Republic of San Marino, Republic of San Marino Pietro Renzi, Alberto Franci

Department of Economics, Society and Politics, Urbino University "Carlo Bo", Urbino, Italy

# 1. Introduction

a

b

Debate about the performance of healthcare systems has been amplified by the current Covid19 pandemic. The impact of this crisis has served to highlight the fragility of many such systems and the key need for policymakers and health service managers across the world to evaluate their performance. Arguably the strategic development of any healthcare system should aim to reduce health inequalities, and therefore, as a minimum, it is necessary to monitor its performance in addressing inequalities in both health and its social determinants. The situation in Italy is a case in point, where there are demands for better quality of care,

higher productivity, better responsiveness, more efficiency and better sustainability. All of these are expressions of the same question, viz. how to improve the performance of health services and health workers? However, measuring healthcare performance presents difficulties because of its multidimensional nature, which can easily lead to conceptual and methodological confusion. As a consequence, there is a scarcity of models which fully analyse performance at healthcare system level. Unsurprisingly, virtually all current performance frameworks include quality of care as a key element, with effectiveness, productivity and efficiency also being recurrent themes. Examples include the World Health Organisation's (WHO) World Health Report 2000, the Organisation for Economic Cooperation and Development's (OECD) framework (2004), and the Nuti's framework (2008). In contrast, social outcomes of healthcare and equity are missing or little developed in most frameworks, with Australian and Canadian national frameworks being notable exceptions. Given this situation, Sicotte et al. (1999) developed the comprehensive Evaluation Globale et Intégrée de la Performance des Systèmes de Santé (EGIPPS) framework for the assessment of the performance of Health Care Organizations (HCOs). Therefore, the main aims of this paper are:


# 2. Key features of the EGIPSS model

In the healthcare sector, one framework stands out: the EGIPSS framework developed by Sicotte et al. (cit.), which is a comprehensive approach to the assessment of performance of HCOs. It includes goal achievement, service production and adaptation to the environment as core dimensions of performance, and usefully adds a focus on values and culture. EGIPSS is geared towards North American settings and has been mainly used in OECD countries. For example, it acts as the basis of WHO-Europe's framework for assessing hospitals, to assess accreditation schemes, to analyse how actors and stakeholders of an HCO define performance and to explore how HCOs learn.

In this paper, the authors present a practical, simplified version of the EGIPSS framework. Keeping the key strengths of this framework, some elements were redefined based on

Pietro Renzi, University of San Marino, San Marino, pietro@unirsm.sm, 0000-0001-6200-7265

<sup>153</sup> Alberto Franci, University of Urbino Carlo Bo, Italy, alberto.franci@uniurb.it, 0000-0002-8157-9792

FUP Best Practice in Scholarly Publishing (DOI 10.36253/fup\_best\_practice)

Pietro Renzi, Alberto Franci, *EGIPSS model for the evaluation of performance in healthcare*, pp. 167-172, © 2021 Author(s), CC BY 4.0 International, DOI 10.36253/978-88-5518-461-8.32, in Bruno Bertaccini, Luigi Fabbris, Alessandra Petrucci (edited by), *ASA 2021 Statistics and Information Systems for Policy Evaluation. Book of short papers of the on-site conference*, © 2021 Author(s), content CC BY 4.0 International, metadata CC0 1.0 Universal, published by Firenze University Press (www.fupress.com), ISSN 2704-5846 (online), ISBN 978-88-5518-461-8 (PDF), DOI 10.36253/978-88-5518-461-8

concepts of integrated healthcare systems and public service. Inspiration was found in Parsons' social system action theory to develop an integrative framework of performance, with the performance of a HCO considered to be multi-dimensional. More specifically, it is the result of the interaction between four organisational functions (see Figure 1). Consequently, the success of an organisation depends not only on how each of these functions is organised, but also on how they are aligned with each other. Performance is therefore understood as something more comprehensive than merely efficiently producing desired outputs. Furthermore, it incorporates the managerial approach of the New Public Management (NPM). The framework also describes six equilibriums or alignments between these four functions, which can be best understood as tensions that may arise between the functions as a result of a change in one of them (Figure 1).

The tactical alignment links the Goal Achievement and Service Production function. This deals first with the appropriateness of the service provision in relation to the goals: "To what extent do the service production processes contribute to attaining the goals? Are they effectively producing the output needed to reach the goals?".

The allocative alignment links the Interaction with the environment and the Service Production function. It first deals with resource acquisition. Questions that can be used to assess this include: "Are the obtained resources adequate to organise the service production function? Is the service production function optimal in relation to available resources?".

The strategic alignment examines the link between the Goals that the HCO is pursuing and its Environment. Here, questions include whether the organisational goals correspond with the needs of the population and other key actors.

The legitimating alignment is about the congruence of the Goal Attainment function with the Culture and Values Maintaining function, and questions how the strategic choice of goals influences and shapes the organisational values.

The operational alignment covers the congruence of the Culture and Values Maintaining function with the Service Production modalities, and the impact of the Service Production system on the organisational culture and values.

Finally, the contextual alignment between Culture and Values Maintaining function and Adaptation to the environment deals with how the social, political and cultural dimensions of the environment influence the organisational culture and its core operational values.

Figure 1 below sets out the model:

Source: Sicotte et al. (1998)

## 3. Materials and methods

The research made use of various statistical sources (Istituto Nazionale di Statistica (ISTAT), Centro Studi Investimenti Sociali, Osservasalute (CENSIS), Istituto Superiore di Sanità and an array of survey methods. The indicators used in the performance evaluation model were determined through an in-depth study of the existing literature (Sicotte, cit.) and in collaboration with experts from the two locations involved in the study.

Data relating to the Republic of San Marino was provided by its Health Authority, its Istituto per la Sicurezza Sociale (ISS RSM) and by the Office of Statistics of the Republic. The data for the "AreaVasta" of the Marche Region were local and regional statistical sources, plus some internal information sources.

Once the list of indicators was identified, a "balise" (according to French terminology) or benchmarking (according to English terminology) of excellence was determined for each indicator. This represented a norm/guide/method against which results could be compared, thereby enabling opinions and judgements to be formed. The EGIPSS model and methodology incorporates performance indices that enable comparisons to be made based on excellence, which can then be weighted relative to each other within a set of dimensional and subdimensional categories. For example, the Adaptation function covers the dimension of 'Availability of resources' which has two subdimensions of 'Healthcare expenditure and financing' and 'Health workforce' (see tab.2 below). The weights used were based on the original weights provided by the model which were in turn validated by a panel of experts representing the various stakeholders of the healthcare systems studied, chosen according to their skills. This validation process incorporated the DELPHI method (Fabbris et al 2007).

One analytical issue involves establishing the relationship between an indicator and performance. The approach adopted was to determine a balise of excellence for each indicator, that involved values considered to be 'high performing'. The sense of variation in the relationship between an indicator and its associated performance can be positive, negative or parabolic. An overall performance achievement index for a subdimension and dimension is calculated by applying the assigned weights to the calculated percentage of achievement of the balise for each indicator and then aggregating the results. This can be done at each level, on the basis that if the weights are expressed as a percentage their sum within a subdimension, dimension and function must be equal to 100. The process can be repeated for the four functions provided by the model.

Once the percentage of achievement of the balise has been calculated, it is possible to assign a qualitative scale of performance. This serves to add precision, with the values used in this study shown in table 1:


Tab.1: Levels of performance

## 4. Results

This section presents a synthesis and summary of the results derived from the application of the EGIPSS model in the "Area Vasta" of the Marche region and the Republic of San Marino (the latter involves a more reduced version, due to a lack of certain data in local information systems).

Comparisons were made using the indicators relating to the Adaptation function, the Service Production function, Goal Attainment, and the Culture and Values Maintaining function, which are set out in tabs. 2, 3, 4 and 5, respectively. In addition to the above tables the authors sought a helpful diagrammatical presentation that enables the reader to judge the relative performance of different healthcare organisations in terms of 'Strategic equilibrium'. Figure 2 below illustrates this for the relationship between Infant Mortality and Healthcare Expenditure and Financing for the three healthcare organisations studied.



Tab. 3: Service Production function



# Tab. 4: Goal Attainment function


# Tab. 5: Culture and Values Maintaining function


## 5. Weaknesses and strengths of the results obtained

The model provides an overview of the performance, especially for the "Area Vasta" of the Marche, through a prism of 107 indicators. These indicators can present warning signs regarding accessibility, technical quality, efficiency and fairness of a system.

Among the inevitable weaknesses that can affect any frameworks of this nature it should be pointed out that the evaluation of performance is equivalent to comparing the result of an indicator, of a dimension or of a sub-dimension, to a given standard. Where such standards exist they have been used. Otherwise, the performance evaluation was set against external objectives (such as those determined by Canada, the WHO and OECD) or comparisons with the similar results from other countries. The authors' choice on Canada was justified by the fact that this country, by establishing its own empirical standards of excellence, utilised its own comparisons with the EU15 countries (Vrijens et al., 2016). This approach, the only practicable one available, made it possible to position the areas studied in relation to those states that have similar healthcare systems. However, it should be noted that the interpretation of relative performance between micro-areas and States appears very delicate due to the methodological and contextual differences that can compromise the validity of the comparisons. Further constraints were the absence of relevant indicators or the lack of data in the information systems of the two areas.

## 6. Conclusions

Performance evaluation is a process that enables the holistic analysis of healthcare systems utilising measurable indicators. Its role, therefore, is to improve the quality of the decisions being taken by all the staff in the healthcare arena and those services that can impact on people's health. The principles and orientation of the EGIPPS model are useful for assessing healthcare systems at any level, whether it be a country, a province or even a local community.

## 7. References


Nuti S. (2008), La valutazione della performance in sanità, Il Mulino Editore.

Vrijens F., Renard F., Camberlin C. et al. (2016), Performance of the Belgian health system – Report 2015, Belgian Health Care Knowledge Centre (KCE), KCE Reports 259C.