17

Dynamic Incentives for Optimal Control of Competitive Power Systems

KARLSRUHER BEITRÄGE ZUR REGELUNGS-UND STEUERUNGSTECHNIK

# Lukas Kölsch

# Dynamic Incentives for Optimal Control of Competitive Power Systems

Lukas Kölsch

**Dynamic Incentives for Optimal Control of Competitive Power Systems** Karlsruher Beiträge zur Regelungs- und Steuerungstechnik Karlsruher Institut für Technologie

Band 17

# **Dynamic Incentives for Optimal Control of Competitive Power Systems**

by Lukas Kölsch

Karlsruher Institut für Technologie Institut für Regelungs- und Steuerungssysteme

Dynamic Incentives for Optimal Control of Competitive Power Systems

Zur Erlangung des akademischen Grades eines Doktor-Ingenieurs von der KIT-Fakultät für Elektrotechnik und Informationstechnik des Karlsruher Instituts für Technologie (KIT) genehmigte Dissertation

von Lukas Kölsch, M.Sc.

Tag der mündlichen Prüfung: 10. Dezember 2021 Hauptreferent: Prof. Dr.-Ing. Sören Hohmann Korreferent: Prof. Dr.-Ing. Andreas Rehkopf

**Impressum**

Karlsruher Institut für Technologie (KIT) KIT Scientific Publishing Straße am Forum 2 D-76131 Karlsruhe

KIT Scientific Publishing is a registered trademark of Karlsruhe Institute of Technology. Reprint using the book cover is not allowed.

www.ksp.kit.edu

*This document – excluding parts marked otherwise, the cover, pictures and graphs – is licensed under a Creative Commons Attribution-Share Alike 4.0 International License (CC BY-SA 4.0): https://creativecommons.org/licenses/by-sa/4.0/deed.en*

*The cover page is licensed under a Creative Commons Attribution-No Derivatives 4.0 International License (CC BY-ND 4.0): https://creativecommons.org/licenses/by-nd/4.0/deed.en*

Print on Demand 2022 – Gedruckt auf 100 % Recyclingpapier mit dem Gütesiegel "Der Blaue Engel"

ISSN 2511-6312 ISBN 978-3-7315-1209-7 DOI 10.5445/KSP/1000147941

# **Preface**

This dissertation has been written during my work as research assistant at the Institute of Control Systems (IRS) at Karlsruhe Institute of Technology (KIT).

First and foremost, I would like to express my deepest gratitude to my doctoral supervisor Prof. Dr.-Ing. Sören Hohmann for giving me the opportunity of pursuing my PhD under his chair, his excellent scientific guidance throughout my time at the IRS and his permanent openness to explore and find new research fields. I also owe sincere gratitude to Prof. Dr.-Ing. Andreas Rehkopf for acting as a second examiner and for his personal encouragement during the final phase of my work.

I am highly indepted to my former master's students (and now colleagues) Felix Strehle, Pol Jané Soneira, and Lukas Rausche, whose exciting scientific acumen in general and careful proofreading of this dissertation in particular have made a crucial impact on shaping the manuscript in its present form. Likewise, I would like to thank Dr.-Ing. Stefan Krebs and Dr.-Ing. Martin Pfeifer for their ongoing interest and motivation, which always pushed me to find the right path. I am very thankful for the best office colleagues one could imagine, namely Florian Köpf, Simon Rothfuß, and Manuel Schwartz, with whom I was allowed to experience many prosperous discussions of both scientific and non-scientific nature, discovering the spontaneous self-therapeutic effect of collaborative whiteboard discussions. I would also like to thank Rishabh Vyas, who helped me numerous times with special linguistic questions about the final manuscript even at the most pathological times of the night, as well as all my other undergraduate and master's students, whose willingness to write their own thesis under my supervision was instrumental in generating numerous new ideas.

Furthermore, I am very grateful for the financial support provided by the German Research Foundation as part of the DFG Priority Programme "Hybrid and Multimodal Energy Systems".

Finally, my heartfelt and very special thanks go to my parents Viola and Roland for their unconditional mental support during the writing process and throughout my whole life.

Karlsruhe, December 2021 Lukas Kölsch

# **Abstract**

Incentive mechanisms can play a pivotal role in accelerating technologically challenging transformation processes such as the energy transition. The aim of those incentives is to provide an environment where the interplay of each competitor's individually optimal actions is also globally optimal in the sense of an overall objective. The dissertation at hand presents a feedback control framework towards optimal incentive mechanisms for today's and future power systems in the triad of system stability, economic efficiency, and grid support. A key innovation of the developed control scheme is the incorporation of temporally and spatially differentiated real-time price signals that result from the solution of static and dynamic optimization problems. Inclusion of locally available measurement information, consistent co-modeling of the underlying physical network including resistive losses, and continuoustime formulation of all subsystems pave the way towards a real *closed-loop* incentive control framework. Particular importance is attached to a rigorous separation between market and network participants, respecting the principle of unbundling. After comprehensive analysis of the resulting closed-loop system, the control scheme is further adapted to a novel realtime congestion management system. Two case studies illustrate practical advantages of the developed approach compared to existing concepts. The port-based system modeling, the absence of any centralized control authority, and the ability for automatic, decentralized regulation of all prices across the entire network finally enable seamless extensibility by additional incentive components.

# **Kurzfassung**

Technologisch herausfordernde Transformationsprozesse wie die Energiewende können durch passende Anreizsysteme entscheidend beschleunigt werden. Ziel solcher Anreize ist es hierbei, ein Umfeld idealerweise so zu schaffen, dass das Zusammenspiel aller aus Sicht der beteiligten Wettbewerber individuell optimalen Einzelhandlungen auch global optimal im Sinne eines übergeordneten Großziels ist. Die vorliegende Dissertation schafft einen regelungstechnischen Zugang zur Frage optimaler Anreizsysteme für heutige und zukünftige Stromnetze im Zieldreieck aus Systemstabilität, ökonomischer Effizienz und Netzdienlichkeit. Entscheidende Neuheit des entwickelten Ansatzes ist die Einführung zeitlich wie örtlich differenzierter Echtzeit-Preissignale, die sich aus der Lösung statischer und dynamischer Optimierungsprobleme ergeben. Der Miteinbezug lokal verfügbarer Messinformationen, die konsequente Mitmodellierung des unterlagerten physikalischen Netzes inklusive resistiver Verluste und die durchgängig zeitkontinuierliche Formulierung aller Teilsysteme ebnen den Weg von einer reinen *Anreiz-Steuerung* hin zu einer echten *Anreiz-Regelung*. Besonderes Augenmerk der Arbeit liegt in einer durch das allgemeine Unbundling-Gebot bedingten rigorosen Trennung zwischen Markt- und Netzakteuren. Nach umfangreicher Analyse des hierbei entstehenden geschlossenen Regelkreises erfolgt die beispielhafte Anwendung der Regelungsarchitektur für den Aufbau eines neuartigen Echtzeit-Engpassmanagementsystems. Weitere praktische Vorteile des entwickelten Ansatzes im Vergleich zu bestehenden Konzepten werden anhand zweier Fallstudien deutlich. Die port-basierte Systemmodellierung, der Verzicht auf zentralisierte Regeleingriffe und nicht zuletzt die Möglichkeit zur automatischen, dezentralen Selbstregulation aller Preise über das Gesamtnetz hinweg stellen schließlich die problemlose Erweiterbarkeit um zusätzliche optionale Anreizkomponenten sicher.

# **Contents**





# **List of Figures**



# **List of Tables**


# **Abbreviations and Symbols**

# **Abbreviations**



# **Latin Symbols**





# **Greek Symbols**



# **Calligraphic Symbols**



# **Script Symbols**



# **Doublestruck Symbols**


# **Typewriter Symbols**


# **Other Symbols**



# **Indices and Exponents**


## **Operators**


# **1 Introduction**

The harnessing of electrical energy is among the most groundbreaking achievements in human history. Its permanent and widespread local availability by means of an interconnected power system is a cornerstone of our modern society and has become an indispensable condition for any civilizational, i.e. cultural, technological, and economic progress. As a result, this civilizational progress itself has consequences for the energy supply system: Stemming from a growing awareness of the negative impact of fossil energy production on the environment, and provoked by the climate debate of recent years, the idea of *sustainability* has gained an increasing presence. Accordingly, our natural environment has faced the repercussions of technological progress, thus in particular the emissions of greenhouse gases as well as the depletion of natural resources, should be given greater focus [RLP<sup>+</sup>15]. This understanding has led to a number of worldwide political declarations of intent, according to which the ecological footprint is desired to be drastically reduced. As part of the European Green Deal, in 2021 the European Climate Law [Cou21] was passed, according to which the EU member states are obliged to reduce their net greenhouse gas emissions to zero by 2050. Moreover, it enforces a reduction in greenhouse gas emissions of minus 55%<sup>1</sup> by the year 2030.

The power generation sector holds a prominent role in climate protection, as it not only accounts for the largest share of global greenhouse emissions to date [BBM<sup>+</sup>14, p. 516; MEH<sup>+</sup>17, p. 7], but also has the greatest quantitative potential for substitution compared with other sectors [DKH19, p. 3]. As a consequence, many countries have set ambitious goals for the next decades to shift away from conventional fossil-fueled power generation towards renewable and low-carbon energy sources. Germany's reduction targets are consolidated in the *Climate Protection Plan 2050*. For the power generation sector, this directive imposes a drastic reduction from 280 to 175 million tonnes of CO<sup>2</sup> equivalents between 2020 and 2030<sup>2</sup> . These actions are intended to increase the share of renewables in gross electricity consumption in Germany<sup>3</sup> from 32% in 2015 to 65% in 2030 [Deu14, KM17, Deu20].

From a systems viewpoint, these ambitious policy roadmaps are accompanied by significant techno-economic disruptions, namely, the increasing displacement of conventional, fossilfueled power plants in favor of power generation from renewable energy sources (RESs), such as wind, photovoltaic, biomass or run-of-river plants, leading to a structural change in the feed-in characteristcs: Since the majority of RES power plants are based on energy sources whose availability is weather-dependent [SP18], no exact forecasts can be made about the expected feed-in. This growing uncertainty manifests itself in an increasing volatility of the *residual load*, meaning that the difference between the current demand and the current

<sup>1</sup> related to the reference year 1990.

<sup>2</sup> Sector-specific targets on an annual basis are imposed in a federal law, see [Deu19, Annex 2].

<sup>3</sup> Likewise, there are national measures and legislative acts for stimulating green electricity in the U.S. [EMG+21], China [HZZ20], Canada [Gov16], India [Int20], and Scandinavia [Nor17]. An overview of the measures in the individual EU member states is given in [Eur20].

generation by RES power plants fluctuates to an increasingly large extent. However, it is important to note that this increasing volatility has not only a temporal, but also a geographical dimension. Unlike conventional power plants, which can be located deliberately at load centers [Sch14, p. 11], renewable generation units are usually located wherever geographic conditions allow an economically viable harvest of e.g. wind or solar supply. This creates the undesirable phenomenon of *locally non-integrable power* during periods of high solar exposure or high wind intensity [MGMH14]. Likewise, load centers with geographic conditions that are expected to be less profitable for power plant operators may suffer from ongoing deficits in local net generation. This consequently will result in significant and increasing amounts of compensation flows via the transmission lines [ZPM13].

In light of this upcoming scenario of increasing temporal and local volatility, there is serious reason among decision-makers as well as in the scientific community to expect that the existing power transmission and distribution system, in its current form, will not be able to keep pace with these developments. Furthermore, it is evident that the existing technical and regulatory system infrastructure is noticeably becoming stressed. Until now, this fact has already manifested itself e.g. in increasingly volatile spot market prices [RN20], increased incidence of line congestion leading to critical grid interventions [Sta19, p. 42], reverse power flows [NCB<sup>+</sup>18, HRR20], and increasingly high amounts of net imports and exports of electricity over long distances [KO20, p. 164].

The most evident technical means of counteracting these issues is through the installation of local energy storages (see [DBK<sup>+</sup>18, Gür18, BMC<sup>+</sup>20] for comprehensive reviews of prospective technologies and their economic viability), accompanied by a massive grid expansion. However, apart from the long time horizons from planning to launch<sup>4</sup> , oversizing transmission lines according to a worst-case calculation is known to not be economically viable, especially if this worst-case scenario can only be roughly estimated. Apart from these technical challenges, there is also a strong need for redesign of the existing regulatory and market framework to accommodate the high intermittency of RESs while ensuring overall system stability [KO20, p. 164f.; Sch21, p. 6f.].

Moreover, increased *competition* among the flexible participants is a useful mechanism to stimulate the participants' own strive for innovation. At the same time, excesses in consumer prices have to be prevented in order to avoid undermining people's confidence in the viability of renewable energy generation and thus missing the targets of the whole energy transition. Consequently, a proper mechanism design should deliver *real-time signals* to all competitors that incentivize supply during times of particularly high demand and in geographic regions where it is of utmost necessity from a "global" network perspective. Although the current structure of the energy market, dominated by *normative interventions* such as feed-in tariffs or balancing group (BG) contracts, is not capable of providing such investment signals that internalize network effects, first studies also show that a mere adoption of locally differentiated prices may lead to false incentives as well, unless the slowly reacting settlement times of existing electricity markets are overcome (see e.g. [Con19]).

<sup>4</sup> Due to extensive tendering and approval processes, the average time required for a transmission line construction permit is 7 years, with an upper quartile of 14 years [BL11, p. 4]. Likewise, electrochemical storages with fast reaction times such as battery energy storage systems will presumably not be available on a large scale within the next 20 years [FSK+20, p. 2].

At this point, scientific contributions from the area of *automatic control theory* can contribute to "close the loop" between market and network (i.e., between the pricing mechanism and the physical interaction of participants across the power grid) and to rigorously analyze the interdependencies that arise in this process. The dissertation at hand is thus devoted to developing a holistic feedback control framework for large-scale interconnected power systems with competitive participants which ensures *stability* of the overall system. By applying a market-based architecture, the controller is intended to pursue *economically efficient* operating points without any need for additional supervision by a higher-level authority. At the same time, the control framework should be able to *support the grid* by taking the geographical component of momentary infeed into account. Thereby, continuous valuation of each participant's individual contribution to a stable network operation should provide instantaneous price incentives for local oversupply or undersupply. Despite this conceptual redesign towards real-time signaling, the proposed framework is intended to be a genuine *brownfield approach*, i.e., it should neither tighten the existing action space of competitive network participants nor require a redesign of *physical* infrastructure. Hence, the overall objective of this dissertation can be summarized as follows:

**Objective.** *Design of a unifying, market-based feedback control strategy to enforce a stable, economically efficient, and grid-supportive operation of future power systems.*

The outline of this dissertation is as follows (see Fig. 1.1). Chapter 2 reviews the current state of the art, formulates the existing research gaps, and outlines the specific contributions of the dissertation aligning with the above objective. Subsequently, these specific contributions are elaborated in Chapters 3–5. In Chapter 6, effectiveness and performance of the presented overall control framework is evaluated by means of case studies on a realistic IEEE 57-bus benchmark system. Finally, a conclusive discussion on the contributions is given in Chapter 7.

**Figure 1.1:** Outline of this dissertation.

# **2 Related Work and Research Contributions**

This chapter forms the basis for the scientific contributions of the subsequent chapters by providing a comprehensive survey on recent advances in competitive power systems. Section 2.1 analyzes the status quo of today's electricity markets and outlines recent tendencies. Section 2.2 surveys the state of research. Section 2.3 summarizes the main findings of the previous sections by means of three specific research gaps. Finally, Section 2.4 presents the main contributions of this dissertation that result from these research gaps.

# **2.1 Today's Liberalized Electricity Markets**

From the beginning of electricity production until the 1980s, the power grid was characterized by *vertically integrated* electricity companies, which concentrated all aspects of power supply, from generation and distribution to grid operation, under the same roof. However, as power systems became increasingly interconnected in the 1900s, this concept proved to be insufficient to take the increasing importance of power flows over long distances into consideration. For instance, a power plant operator (PPO) that simultaneously acts as a local system operator would always retain both the incentive and the opportunity to prevent competitors from entering the market. While it is economically desirable to create competition between PPOs, the system operators (i.e. owners of the transmission and distribution infrastructure), by contrast, constitute *natural monopolists* [GS17, p. 264f.] due to their high fixed costs and low variable costs<sup>5</sup> . From an economic perspective, the most effective way to handle natural monopolies is through *regulation*. For this purpose, the ownership of some part of the network is auctioned off by the state on the basis of long-term concessions, and a regulatory authority constantly monitors compliance with these contracts. This separation of different roles has led to the need of *unbundling*, which includes the following key aspects [BGM<sup>+</sup>14, p. 7]:


<sup>5</sup> Practically speaking, it is not useful to construct multiple power line infrastructures from different companies in parallel.

**Figure 2.1:** Status quo of key stakeholders in Germany's power system. The arrows mark the typical flow direction of physical power (solid lines), information (dashed lines), or capital (triangle lines). Color-coded components mark those effects that arise by increased infeed of RESs and are discussed in Subsection 2.1.2.

As a consequence, there exist individual companies with clearly defined roles and narrow information boundaries. In this process, power generation and supply continue to be left to free competition, while system operators are regulated by the state authority.

#### **2.1.1 Overview and Participants**

Fig. 2.1 shows a schematic overview of the relevant stakeholders in Germany's power system<sup>6</sup> as it exists today. The power network infrastructure can be divided into the *transmission network* (typically with nominal voltages above 100 kV) and the *distribution network* (nominal voltage below or equal to 100 kV). While the transmission network operates as a supraregional transport and interconnection layer, the distribution network is conceived for subordinate,

<sup>6</sup> Although nomenclature is based on the usual terminology in Germany, a fairly similar structure can also be found in other European countries as an outcome of EU-wide harmonization.



regional distribution of electrical energy to the point of consumption. Consequently, transmission grids tend to be more meshed than distribution grids, which are typically operated as open ring or radial grids, and are characterized by an R/X ratio close to zero.

Large-scale *conventional power plants* (CPPs) are directly connected to the transmission grid via step-up transformers. The aim of each CPP is to deliver a certain amount of electrical power within a specific period of time. There are various opportunities for selling the produced power: On one hand, the own supply can be offered on the Wholesale Electricity Market (WEM). The WEM is an *energy-only market*, which means that only the energy actually to be supplied at a certain time is offered. On the other hand, there exists a *balancing market* (BM), where each PPO's production capacity can alternatively be offered to system operators as a means for stabilizing interventions, e.g. after disturbances or faults. The BM is a *capacity market*, which means that it trades the *option* of having a certain amount of electrical power delivered for a certain period of time. In fact, the offered power packages are standardized in terms of spatial and temporal quantization, and depending on the specific time scale, both WEM and BM divide into specific sub-markets. Table 2.1 provides an overview of the existing electricity markets in Germany. In a fully unbundled power system, CPPs are owned by independent companies which are purely profit-driven. Whether marketing its own electrical power is more profitable on the WEM or the BM depends on the current market forecast of the individual CPP.

The demand side can be divided into *wholesale consumers*, which cover their own demand directly via the WEM, and *retail consumers*, which pay a fixed electricity price quoted by an *electricity supplier* (ES). The measurement of the actually consumed power is done over a 5-minute interval for wholesale consumers and at an annual interval for retail consumers. Between measurement intervals, consumption of the latter group is estimated based on standard load profiles (SLPs). The ES has to purchase respective amounts of power on the WEM in the exact amount corresponding to the (estimated or actual) consumption.

Each supply or demand point of the power system is assigned to a BG. The *balance responsible party* (BRP) has to provide a zero balance between supply and demand for each quarter-hour interval. This can be achieved optionally by means of the own assets, or with the help of purchases or sales on the WEM.

The *transmission system operator* (TSO) is the owner of a certain part of the transmission network infrastructure<sup>7</sup> and is responsible for its maintenance and expansion. Furthermore, the TSO is responsible for stability of the own network. For this purpose, it uses its own measurements in the network as well as information from the subordinate *distribution system operators* (DSOs)<sup>8</sup> . By aggregating the transmitted schedules of the BRPs, the TSO performs an after-market power flow calculation for the upcoming day and, if necessary, has the authority to intervene in the schedules of the power plants in the event of forecast line overloads (*redispatch*) or to prevent non-integrable generation by RESs on times of high net generation surpluses (*feed-in management*). Furthermore, the TSO shall ensure that any current imbalance is remedied by requesting balancing power from the BM or by applying its own stabilizing measures, such as staging of transformers or installation of compensator elements, e.g. static VAR compensators or flexible AC transmission systems (FACTSs).

#### **2.1.2 Recent Developments**

The increased penetration of RESs into the transmission or distribution grid (see red components in Fig. 2.1) has a significant implication on the existing market. The main developments are reported below.

#### **Fixed Remuneration of RESs Infeed**

To encourage a more rapid investment in RESs, numerous financial incentive instruments have been created. For instance, RES operators are either paid a fixed (i.e. guaranteed) remuneration or an additional premium per amount of power fed in (see orange arrows in Fig. 2.1). The power fed into the grid is marketed by the TSOs. However, since revenue from sales on the WEM is typically far less than the guaranteed remuneration for RESs [CHMG14, p. 304], the resulting extra costs must be compensated by retail consumers and wholesale consumers<sup>9</sup> paying an additional consumer levy to the local DSO. Likewise, if feed-in management is required due to an oversupply by renewables, the respective RESs are compensated for the lost profit, which is also covered by the consumer levy. This levy is taking an ever-increasing share of the net electricity price. In Germany, it averaged 20.4% for retail consumers in 2021 and is considered a significant driver of current and future electricity prices [Bun21, p. 19.]. This is a paradoxical fact, since the actual variable costs of electricity generation by RESs are significantly lower than those from fossil energy sources.

<sup>7</sup> In Germany, there are 4 TSOs. All of them are part of the *European Supergrid* of the European Network of Transmission System Operators for Electricity (ENTSO-E), consisting of 42 TSOs from 35 countries in total.

<sup>8</sup> Within the ENTSO-E network there are more than 2300 DSOs, 880 of which are located in Germany [TM15, p. 4; PFA+19, p. 11].

<sup>9</sup> If wholesale consumers manage to be classified as "energy-intensive", they can exempt themselves from the consumer levy [Deu14].

#### **Redispatch and Copper Plate Assumption**

The remuneration policy for RESs discussed above has another side effect that arises particularly in the case of weakly developed grids or long transmission paths: If CPPs are displaced from the merit order during periods of high solar and/or wind generation, there is a local undersupply of load centers. While these may be compensated arithmetically by the respective RESs, significant spatial disparities between the locations of generation and consumption may occur, e.g. when offshore wind turbines ramp up, thereby superseding CPPs far from the coast. If these disparities exceed the physical transmission capacity limits of the network, redispatch has to be ordered by the responsible TSO (see blue arrow in Fig. 2.1). This redispatch mechanism is *cost-based*, i.e. both the down-regulated and up-regulated power generators are each compensated pairwise for their lost profits or additional costs to be incurred. In fact, central Europe has seen a surge in the occurence of redispatch events over the last decade [Fek21, p. 5f.; EE21], and this trend is assumed to continue [Sta19, p. 42].

Cost-based redispatch provides numerous inappropriate incentives for PPOs from a macroeconomic and environmental perspective: For an individual PPO, the compensation mechanism makes the specific location of generation and thus the role of a certain power plant to the overall network completely irrelevant<sup>10</sup>. Instead, the PPOs' investment decisions are only governed by minimizing their own fixed costs, which often results in a preference for rural areas far from load centers. At the same time, PPOs have no incentive to replace existing, low-efficiency power plants at critical load centers with higher-efficiency power plants, since cost reimbursement by the TSO is always guaranteed, even if the bid falls well outside the merit order. Furthermore, studies have shown that PPOs can anticipate the occurence of a redispatch event with very good accuracy solely based on seasonal patterns and local weather measurements, even without any knowledge of the prevailing network conditions [Sta19, Chapter 5]. This creates several speculative trading opportunities for PPOs that are unfavorable from a global systems view: For instance, if it can be anticipated that a specific power plant will be curtailed by the TSO, it is advantageous for this PPO to submit a lower bid in order to be scheduled, but then to be reimbursed for the curtailment without having to physically deliver the amount of power. Conversely, if a certain power plant is likely to be requested after-market by the TSO for redispatch, strategic PPOs will bid too high to raise their own expected compensation payments. Further speculative trading strategies following from good predictability are discussed in [Sta19, p. 53].

#### **Balancing Energy**

The increasing volatility of power supply increases the difficulty for BRPs to keep their BGs balanced. If a BRP delivers less power than scheduled, the TSO is obliged to procure the difference on the BM and later charges its own costs as a *balancing energy* payment (see violet arrows in Fig. 2.1). Likewise, if the BRP delivers more power than scheduled, it receives a balancing energy payment in the same amount by the TSO. This mechanism is thus another speculative trading opportunity: In times of high spot market prices it is advantageous for the

<sup>10</sup> This phenomenon is called *copper plate assumption*, which refers to the fact that from the PPO's point of view, the whole power network is reduced to a single copper plate with unlimited transmission capacity.

BRP to sell its own power on the WEM instead of keeping its own BG balanced. Analogously, in times of low spot market prices, it is preferable to purchase power on the WEM and earn the difference to the received balancing energy payment as arbitrage profit [MRF11]. This exploitation pattern counteracts the efforts of the TSO to keep the network balanced and is likely to endanger the stability of the network. In recent years, several critical incidents in the ENTSO-E network have been reported which trace back to strategic behavior of the BRPs (cf. [JW15, KH19, PLd20]).

#### **2.1.3 Discussion**

The above examples show that under the impact of increasing RESs infeed, the existing electricity market in its current form is tending towards undesirable interactions between stakeholders. Thereby, the (normative) role of the TSO on electricity markets is increasing, such that the original requirement for unbundling is only partially fulfilled. In particular, while legal unbundling is still given, there are an increasing number of situations where the TSO has to intervene in PPOs' decision making or where the receivable information empowers PPOs to exploit their knowledge to the disadvantage of other participants, thus violating both operational and informational unbundling. These trends result in a more frequent occurrence of critical network situations and thus in an increasing instability of the network. The economic inefficiencies also create substantial extra costs for consumers (see Fig. 2.1). In addition, cost-based redispatch and copper plate assumption act as key drivers for PPOs to act contrary to what would be necessary and desirable from a network perspective.

In order to resolve the shortfalls shown in Subsection 2.1.2, a certain tendency towards even stronger normative intervention by the regulatory authority can be observed [EE19]. However, such interventions do not adequately address the intrinsic problem and often reach their target far too late<sup>11</sup> .

Moreover, numerous position papers suggest a shortening of the current time frames (see Table 2.1) towards a *real-time* interaction of the stakeholders. In a real-time scenario, the actions (i.e. outputs) of individual stakeholders are based on continuous measurements of the own state and environment. Thus the overall interaction of all participants on the physical network an via different markets can be interpreted as a *feedback control system* comprising a large number of internal control loops with different response speeds. However, there is also a wide consensus across the research community that retaining the *existing* WEM structure is not sufficient to prevent the tendencies described in Section 2.1.2. For instance, the occurence of redispatch is always an indicator of structural *market failure* [Ris21, p. 58], as it may trigger network participants at critical points to further aggravate the (anticipated) congestion.

Consequently, the following section provides a comprehensive overview over existing controltheoretic approaches which are able to make vital contributions towards a stable, economically efficient, and grid-supportive electricity market design.

<sup>11</sup> For example, after a particularly serious case of balancing energy speculation in Germany in June 2019, high penalties were imposed on the BRPs involved by the responsible Federal Network Agency [Bun20], albeit the majority of cases are likely to remain undetected [LC12, p. 259f.].

**Figure 2.2:** Overview of main phenomena in power systems [Sau11, MBB12, OW15, CLI+17]. The focus of this dissertation is on timescales between 10 ms and 1 d and is highlighted in gray.

## **2.2 State of Research**

Fig. 2.2 provides an overview of the main phenomena in electrical power systems ranging from 10−<sup>6</sup> s to 10<sup>9</sup> s. Typically, the common term *power system stability* is associated with timescales smaller than 15 min. Stability is said to be given, *"if, after being subjected to a disturbance, all state variables [of the network] recover to (possibly new) steady-state values which satisfy operational constraints [...] without the occurence of involuntary load shedding"* [FCSP<sup>+</sup>20]. According to the latest IEEE-PES definition, power system stability encompasses the sub-tasks *frequency stability*, *voltage stability*, *rotor angle stability*, *converterdriven stability*, and *resonance stability* [HMR<sup>+</sup>20]. Among these sub-tasks, frequency and voltage stability play the most important role for system operators<sup>12</sup>. By contrast, measures for *economically efficient* and *grid-supportive* operation take place on timescales above 1 s (see Fig. 2.2). They range from supply-side optimization (*economic dispatch*) and demand-side management to *day-ahead scheduling* and long-term *grid planning*.

In the remainder of this section, the main research branches devoted to one or more of the three sub-objectives from Chapter 1 are presented. First of all, Subsection 2.2.1 provides an overview of the main control paradigms and system architectures that are being discussed in the power systems literature. Focusing on slower timescales, Subsection 2.2.2 outlines the main ideas and results for the economic operation on supply- and demand-side. Finally, Subsection 2.2.3 reviews the latest research on congestion management. Subsequently, an in-depth review on the state of research on faster timescales is given for frequency control (Subsection 2.2.4), voltage control (Subsection 2.2.5), and the combination of both (Subsection 2.2.6).

<sup>12</sup> In particular, rotor angle and converter-driven stability are typically maintained by the owner of the respective facility by means of inner control loops on a faster time scale [Tv16, WBL+14, RYI+16]. Accordingly, both categories are not considered for the remainder of this dissertation. Likewise, resonance stability encompasses subsynchronous phenomena being addressed by constructional measures (cf. [NFM+11, LS15, VG16]), which is beyond the scope as well.

#### **2.2.1 Overview of Control Architectures**

Historically, the power system followed a strict top-down approach not only in its physical structure but also concerning the responsibility of system operators. In particular, the TSO alone was responsible for the procurement of stability, while the DSO only undertook subordinate control tasks such as transformer staging and power line switching. However, triggered by the replacement of CPPs by RESs, the distribution network has evolved from a weakly equipped *supply grid* towards an interactive and self-sufficient *smart grid*, encompassing decentralized communication via broadband connections, bidirectional *prosumers* with local storages, sector coupling, *smart meter* instrumentation for retail customers, and advanced cyber-security protection [SRRD15, DEA<sup>+</sup>17, KD18, ABAH<sup>+</sup>20]<sup>13</sup> .

The current literature suggests different approaches for transitioning the former top-down control hierarchy between TSO and DSO towards a more agile and autonomous mode of operation. The concept most widely discussed is the *microgrid* (MG) topology<sup>14</sup>. Originally presented in [Las02], a single MG denotes a specific section of the network<sup>15</sup>, which is connected to the superordinate grid by a single *point of common coupling*. In normal ("islanded") mode, each MG is isolated from the main network and thus both supply and stabilization are provided decentrally by means of local storage and generation capacity only. If islanded operation is no longer possible, e.g. in case of severe outages or load shifts, the MG switches to grid-connected mode, thus compensating the imbalance via the main grid. Through the principle of subsidiarity, each MG can be controlled separately without having to account for interdependencies with neighboring MGs. This factor has contributed to an increasing number of real-world studies where the MG concept is implemented<sup>16</sup>. A current overview of in-field experiences is given in [MMR<sup>+</sup>21, p. 4]. Major challenge in MG operation, however, is the re-synchronization with the superordinate grid, which is highly nontrivial and yet in addition often occurs especially at that moment when the MG is in a severe state [DRVA10, ZS10]. Moreover, self-sufficient operation in islanded mode necessitates that generation and storage capacity be generously sized in order to always meet its own demand. It thus has to be critically acclaimed that the physical separation into isolated portions comes at the price of considerably higher investment costs to guarantee worst-case redundancy separately for each MG—in fact, computational overhead is simply shifted to resource overhead.

From the perspective of market participants, the increased availability of communication infrastructure has enabled the development of *virtual power plants* (VPPs) [Dv03, MR11, NY21], virtually aggregating geographically diverse small-scale *distributed energy resources* (DERs) to one single asset. This allows the VPP operator to internally reduce volatility and

<sup>13</sup> All of these building blocks are either in process of field testing, or a nation-wide rollout in some countries is upcoming in the next few years. See the given references for practical evidence.

<sup>14</sup> To date, no formal definition for microgrid exists within the research community. The one presented here is adopted from [OMSE+14] and [SFO15, p. 52ff.]. However, there exist alternative definitions such as [TS12], which do not require a point of common coupling, but only require the MG to act as a single controllable entity with respect to the superordinate grid.

<sup>15</sup> Typically on the scale of a medium voltage network. Similar structures at lower voltage levels, e.g. within households, are defined as *nanogrid* by a majority of authors.

<sup>16</sup> In fact, even at the beginning of the electricity age in the late 19th century, all urban power grids were operated in islanded mode. Even today, this is still in practice in many remote areas e.g. in Canada or Russia. In this respect, it would be more appropriate to speak of a *revival* of the MG.

unpredictability through the effect of diversification, and thus be accessed similarly to largescale CPPs for the purpose of stability procurement. In contrast to the MG concept, the overall network remains physically interconnected. Several authors distinguish between the two different roles of *Technical VPPs* (TVPPs) and *Commercial VPPs (CVPPs)*. In this notion, TVPPs pool the regional generation capacity of small-scale DERs within a certain area. They have detailed measurement information about the state of the local network and use their own resources to provide system services to TSOs and DSOs. Conversely, CVPPs are aggregators of DERs from possibly *distant* regions. They have no information about the network and place bids on the WEM based on their aggregated generation profile. The conceptual separation into CVPPs and TVPPs has the advantage that small- and micro-scale DERs gain access for both the WEM and for procurement of ancillary services on the BM. However, the previous structure of TSOs, DSOs, and BRPs which was criticized in Section 2.1 (cf. Fig. 2.1) typically remains unchanged. An alternative approach to organize a fully decentralized power system without the need for large redundancies is the *web-of-cells* (WoC) concept [MBR<sup>+</sup>17, LHK19]<sup>17</sup> . Here, the overall network is partitioned into cells operated by autonomous *cell coordinators* (CCs), each exclusively responsible for stability within its own cell. The WoC concept is able to reduce complexity by means of information barriers between CCs, albeit without cutting the physical interconnections between cells. Furthermore, there exists no superordinate instance anymore, thus CCs communicate in a peer-to-peer manner. Although this approach is interesting, most studies have only focused on the conceptual formalism itself. Yet to date, the literature on specific control schemes tailored to the WoC scenario is scarce (see e.g. the contributions [RCT17, MRHD17]).

#### **2.2.2 Real-Time Economic Dispatch and Demand-Side Management**

A major implication of the copper plate assumption is that the existing WEM does not take into account the physical and operational conditions (e.g. power flow equations and line flow limits), but relies on the "*fiction of a congestion-free market area*" [HSMT19, p. 13]. Therefore, *real-time economic dispatch* approaches aim to replace the existing WEM with a (feedforward or feedback) controller, incorporating these physical and operational conditions in an *optimal power flow* (OPF) problem<sup>18</sup>, which is solved at specific points in time.

In the remainder of this subsection, we distinguish between *centralized* approaches, which pursue a single overall objective, approaches from *noncooperative game theory*<sup>19</sup>, where each network participant is modeled as a player, with no contracts possible between them, and approaches from *cooperative game theory*, which involve binding contracts between players. A comprehensive review of the recent state of research in economic dispatch and demandside management is provided in the survey paper [WYL21]. For a detailed discussion of continuous-time methods, see [DBSPG19].

<sup>17</sup> A number of sources refer this concept as *grid of MGs* or *networked MGs*, cf. [WL20].

<sup>18</sup> OPF is a well-studied problem class in both theory and application. Early OPF algorithms [DT68, AS74, HG91] are based on Newton-type methods, while today's solvers usually rely on active set and interior point methods [NW06, Chapter 14; Tay15, Chapter 3; FR16, p. 1185] or on convex relaxation techniques of the OPF problem, such as linear relaxation (DC-OPF) [TCL12] or semidefinite relaxation [Low14].

<sup>19</sup> An overview of basic concepts and main terminology of game theory used throughout this dissertation is summarized in Section A.2 in the appendix.

#### **Centralized and Model-Predictive Approaches**

Aggregating all objectives and constraints into a single, centralized OPF problem allows to circumvent the issues from Subsection 2.1.2, since no after-market interventions such as redispatch or consumer grid tariffs are necessary. *Model predictive control* (MPC) is an advanced control technique originally stemming from the process industry. It involves solving a repeated numerical solution of a snapshot of the optimization problem with a preferably fast sampling time, and then applying this numerical solution to the controller output within the current time interval upon repeating the calculation in the next time step. As the steady-state mappings derived from the dynamic constitutive equations of the physical system typically correspond to the AC power flow equations, the control architecture of MPC emerges as a repeated offline solution of static AC-OPF problems whose feasible set can be referred to as the nonconvex *power flow manifold* (see e.g. [BD15]). The ability to also incorporate predictive data like weather or load forecasts in the OPF problem makes MPC particularly suitable for optimal dispatch of power grids with high penetration of RESs [HMMD13, p. 5827f.]. Extensive research on the application of MPC for optimal dispatch of power networks is conducted for low voltage distribution networks [KPP<sup>+</sup>17, PRK<sup>+</sup>17, DSE<sup>+</sup>21], industrial consumers and households [SGAM18, SFC<sup>+</sup>18, FS21] and islanded power systems [PRG14, OCN<sup>+</sup>17, SSC<sup>+</sup>19]. While the authors in [PRG14] use computer simulations to show the effectiveness of the approach, power hardware-in-the-loop demonstrators are used in [PRK+17] and even fieldsite demonstrations are available in [FS21], which highlight the overall practicality of MPC for real-world optimal dispatch.

In spite of their widespread presence in the context of power networks, it must be critically acknowledged that the accuracy and thus feasibility of the obtained solutions is strongly dependent on the accuracy of the precomputed system model. Conversely, as the complexity of the underlying system model increases, so does the computational complexity [BBPS09]. This, in turn, prevents the choice of reasonably fast sampling times.

In response to these practical imperfections, which are very common in real-world scenarios (cf. [PZS<sup>+</sup>18]), *stochastic MPC* methods have gained a foothold in power system research over the last 20 years, employing the representation of model uncertainties as stochastic constraints. However, the robustness gained by including all possible model deviations and disturbances comes at the price of fairly "conservative" solutions that do not fully explore the entire feasible set and thus often yield profits which are significantly below the maximum achievable profits, whereby, the computational complexity for finding a solution increases strongly once again as compared to the deterministic framework. Beyond academic examples, the usability of stochastic feedforward models for real-time optimization of interconnected power systems with intermittent and volatile feed-in characteristics thus seems questionable.

Moreover, in both classical and stochastic MPC, a centralized, omniscient system operator is assumed to have both full knowledge about all system states and access to all actuators. This assumption is unfavorable for large-scale power systems both in terms of computational complexity and vulnerability, thus limiting the applicability of classical MPC to small-scale power systems with a moderate number of parameters.

Recent developments in *distributed model predictive control* (DMPC) methods [VHRW08, CSML13] allow to efficiently decompose the optimal dispatch problem into subproblems, such that local agents with geographically limited measurement and control access optimize only a part of the system. [SESJ13] proposes a DMPC algorithm for economic dispatch using Dantzig-Wolfe decomposition. [XXM19] presents a cost-minimal DMPC scheme for grid-connected low voltage networks, the performance of which is validated on a power hardware-in-the-loop platform. In [RCPP20], a dual decomposition method is applied for DMPC-based optimal scheduling of interconnected migrogrids. These approaches suffer from the fact that the distributed agents aim to jointly minimize a global objective function. However, in fully unbundled power systems, network participants usually are not committed to a common goal and do not take cooperative decisions. Instead, they operate selfishly by maximizing their own profits, i.e., try to minimize their own objective function. Distributed *continuous-time* controllers for radial power systems are presented in [ZP15]. In [ZLP15], the controllers are extended to meshed topologies. However, both approaches again rely upon the assumption of zero transmission losses.

In general, a proper accounting for transmission losses is a key challenge of all OPF-based methods, especially at distribution grid level, due to inherent nonconvexity of lossy AC-OPF problems and their resulting non-existence of appropriate Lyapunov candidate functions [SALB05; Sch15, Remark 6.2.2]. In [MNRQ17], losses are approximated by *robust loss coefficients* in order to provide a distributed solution to the economic dispatch problem. Improved loss coefficients formulas, which also consider reactive power flows, are presented in [HYC<sup>+</sup>18].

#### **Noncooperative Game-Theoretic Approaches**

Noncooperative game theory is concerned with selfishly driven network participants and provides tools and methods on how to maximize their own profit while taking into account the other players' strategies. The main solution concept in noncooperative game theory is the Nash equilibrium (cf. Definition A.6), which is used to characterize a state in which no player can achieve a higher profit by unilaterally changing its strategy.

Former research focused on noncooperative approaches with highly simplified models of network participants, which allows Nash equilibrium seeking to be resolved using linear programming techniques, see e.g. [SLL02, WY13] for generator dispatch or [ZHB12] for demand-side management strategies. Afterwards, stronger emphasis was placed on tractability and practicability aspects by applying parallelization and approximation techniques based on e.g. proximal decomposition [AOS<sup>+</sup>13a, AOS<sup>+</sup>13b] or distributed agreement [CLLV14].

In [XZN13], the applicability of DMPC for a linearized model of interconnected power systems with different objective functions is shown. The authors in [FS12] propose a *noncooperative* DMPC method, which requires an information exchange about the agents' planned trajectories via a non-manipulable information path. In addition, [RYNC14] proposes a noncooperative DMPC procedure, where each agent is equipped with a distributed Kalman Filter to estimate the overall state of the system. Both approaches are restricted to linear systems only. [LKF15] presents a *contract-based* DMPC method for agents with differing objective fuctions, which is applicable to nonlinear non-affine systems. However, it needs guaranteed information

about the future trajectories of coupling variables to be exchanged among agents. [VBGQ+20] presents a DMPC procedure for optimal dispatch of different network participants (in particular, conventional generators, RESs, and energy storage systems). However, the approach does not account for physical line limits and thus not prevent possible redispatch contingencies.

However, as the strategic interplay of network participants is typically represented by a full peer-to-peer competition among players, it must be assumed that each has complete information of the other players' strategies. Thus, a major drawback of classical noncooperative frameworks is their utterly high communication effort for broadcasting the relevant information in a non-discriminatory manner via non-manipulable information paths. Moreover, a calculation of the individual best response depending on each other player's strategy is computationally demanding and poorly scalable, which makes a real-time solution intractable, except for very simplified academical case studies.

In this context, *Stackelberg approaches* try to lighten this computational load by implementing a sequential sharing of information in a leader-follower manner. Stackelberg approaches are most common in the area of demand-side management. E.g. in [TZS<sup>+</sup>14, YH15, MWH<sup>+</sup>18, EES<sup>+</sup>19], a single-leader multiple-follower scenario is applied where a single ES is the pricesetting leader and multiple consumers are the price-taking and utility-maximizing followers. In the area of VPP operation, [MR11] presents a Stackelberg model for the optimal interaction between a single VPP operator (leader) and multiple micro-cogeneration units (followers). Multiple-leader multiple-follower games are investigated in [ALCB15, LGCZ15, MMO17], allowing multiple price-setting ESs to compete on their part. However, due to its high computational effort, Stackelberg approaches require a time discretization with sufficiently large time intervals, thus there is a possible conflict between real-time computability and real-time adaptivity. Moreover, the underlying assumption that followers have no means at all to affect the leaders' strategies seems rather disputable.

A compromise between the fully decentralized information pattern of peer-to-peer competition and a centralized scheme is undertaken by *price-based* frameworks featuring different models of competition. Here, individual competition among players is replaced by a real-time bidding process against a single market instance. The two main directions studied in the context of power systems are *Cournot competition* and *Bertrand competition*, whose main distiguishing feature is which quantity constitutes the strategic decision variable. In Cournot competition, each participant bids *quantities of power* while respecting the given price per quantity. [NSH15] and [CBK17] present Cournot-based demand-side management schemes where a central control authority ensures supply-demand balancing. The authors of [DM19] propose a fully distributed pricing mechanism for a Cournot model of competition between price-setting generators and price-taking loads. In Bertrand competition, by contrast, the *bid price* is the strategic decision variable. Network participants send bids to an *Independent System Operator* (ISO), which evaluates the bids centrally and then dispatches them. Two recent papers [CC20] and [CSD<sup>+</sup>20] develop a Bertrand competition model between price-setting generators resulting in a continuous-time bidding process against a centralized system operator, which is shown to provide economic efficiency. Transmission line congestion can be considered only for radial networks. In [SCD<sup>+</sup>19], a Bertrand competition between generators and ISO was implemented by means of a time-triggered update scheme of bids and generator setpoints. Another iterative algorithm is proposed in [MWY<sup>+</sup>19] for interaction between multiple price-setting consumers and a centralized generation company.

In spite of different terminology, e.g. *ISO* [CC20], *macro-station* [SHP11], *central distributor* [PLB<sup>+</sup>16], *non-profit central control authority* [CBK17], these approaches follow the same principle: The centralized instance acts as the only counterpart and antagonist for every single player, with all information on the other players' strategies being aggregated in the momentary price. In contrast to a generalized game-theoretic setup with peer-to-peer dependencies between all players, both models of competition provide better scalability, since the own individual strategy of each player depends only on the *aggregate* of the other strategies (Cournot) or on the strategy of the single ISO (Bertrand). This helps mitigate computational overhead, which is an advantage especially in large-scale power systems.

#### **Cooperative Game-Theoretic Approaches**

While noncooperative game theory supposes that no binding contracts are possible between players, *cooperative* game theory involves binding contracts between players and provides methods for allocating a jointly realized profit to the players involved. The two main branches in cooperative game theory are *coalition formation* and *bargaining*.

Within coalition formation, the goal is to find stable alliances of network participants that result in an individual improvement for all as compared to not collaborating. In the field of coalition formation, a conceptual profit allocation strategy invoking coooperation among independent power producers is presented in [JY03], and a first coalition formation algorithm for radial distribution grids is presented in [SHP11]. Moreover, there has been considerable research effort into using the three classical solution concepts *core*, *nucleolus* and *Shapley value*, see e.g. [LH16, CBP<sup>+</sup>19], [HMM19], and [LH16, CK19], respectively. These concepts represent different manifestations of a *stable equilibrium*, originated by different notions of individual gratification, which aim to prevent each participant from leaving the coalition. While calculation of core and nucleolus is computationally expensive and possibly non-unique, the Shapley value, in contrast, can be calculated considerably easier. In [NKR13], a conceptual approach for arbitration of conflicting interests between producers and system operators based on a coalitional game is proposed. Through an additional power routing function, the approach is capable of preventing network issues such as transmission congestion. [FMC17] develops a discrete-time model for dynamic coalition formation of smart grid prosumers including storage devices. The last two approaches discussed also allow to account for transmission losses. However, their high computational cost prevents real-time application to large-scale systems.

In contrast to coalition formation, bargaining approaches focus in particular on the sharing of the additional gains that exist as an outcome of cooperation. The classical minimum requirement for bargaining solutions is *Pareto efficiency* (cf. Definition A.5). Numerous papers deal with bargaining strategies among ESs, see e.g. [YTL12, FAP18, WH18]. In the area of power system dispatch, [WK09] presents a bargaining solution for small-scale DC power networks, where sources and loads may aim for different goals such as maximizing the own infeed of power or optimally following a certain voltage profile. However, since players do not act simultaneously but sequentially, the method is unsuitable for large-scale systems. Other solutions for economic generator dispatch based on simple cost functions are presented in [WLM15] and [WWM16]. One central issue in all bargaining approaches is the strong correlation between simplicity of the model and computational tractability of the bargaining problem. In particular, finding a solution for nonlinear or nonconvex problems is computationally expensive even for low-dimensional problems. Consequently, most papers dealing with bargaining approaches for economic dispatch rely on highly simplified models of both the market and its participants.

Generally speaking, approaches originating from cooperative game theory are subject to even more rigorous requirements e.g. with regard to revenue sharing mechanisms or multistage side payments. In practice, this implies the need for a centralized instance that has full knowledge about all states and control actions, and that is authorized to redistribute the profit over all players with respect to some predefined rules. However, this requirement is diametrically opposed to both operational and informational unbundling.

#### **2.2.3 Price-Based Congestion Management**

In the context of Cournot and Stackelberg approaches, it is shown e.g. in [YTN13, MWH<sup>+</sup>18] that the resulting *real-time pricing* controllers<sup>20</sup> incentivize stakeholders to shift their generation and consumption curves to off-peak hours and thus provoke a more balanced demand curve over time. However, a purely time-varying, but regionally uniform price is unable to reflect *regional* scarcity patterns since physical network constraints such as congestion cannot be internalized [TBW15, p. 212]. In particular, the methods discussed in the preceding subsection do not protect against after-market reallocation by the system operator due to transmission congestion. A remedy to this situation is the introduction of *locational*, i.e. regionally differentiated, prices in order to adequately signal local network conditions. Moreover, locational pricing is considered to reduce structural and regional mismatches in generation and consumption by providing long-term investment signals to deploy generation capacity in load centers [Sta19, p. 29ff.]. Initial conceptual studies were conducted in [SCTB88] and [Hog92]. Afterwards, nodal pricing frameworks have been implemented in numerous countries, e.g. the US, Scandinavia and Australia. Although being highly efficient in avoiding redispatch, nodal pricing tends to support the exploitation pattern of BRPs (cf. Section 2.1.2), since large generators may exploit their own impact on their nodal price e.g. by temporarily cutting supply to provoke high local spot market peaks [BHJ16]. Accordingly, there is an ongoing debate among both policymakers and researchers about the most effective grain size of price zones<sup>21</sup>. From a practical viewpoint, it has been found useful to choose zone boundaries along transmission lines which are prone to congestion [Sto97].

Besides numerous research contributions on a rather strategic level, which typically analyze the long-term perspective, a few papers deal with real-time pricing schemes to address congestion

<sup>20</sup> In energy-related literature, the term *real-time pricing* is typically used for closed-loop pricing strategies, while *time-of-use pricing* refers to open-loop pricing strategies.

<sup>21</sup> The grain size of price zones varies widely across countries and states: While the price zones in Norway (5), Sweden (4) and Denmark (2) are supra-regional, there are 11 price zones in the state of New York and even more than 4000 in the state of Texas.

management. [MY16] and [MSM17] present centralized strategies for cost-efficient generator rescheduling based on an offline calculation of sensitivity factors. In [SCA16, SBCA19], active power flow limits are included in a centralized real-time economic controller, while [ZP15, ZLP15] present a distributed controller representation and [SCD+19, CC20, CSD+20] incorporate the limits in a real-time bidding scheme. However, all of the above pricing approaches rely on simplified network models such as DC-OPF or decoupled AC-OPF, where the crucial effects of transmission losses and reactive power flows on congestion are both neglected, cf. [Wei17, p. 220ff.; JB20].

#### **2.2.4 Frequency Control**

In a power system fed by synchronous generators (SGs), deviations from the global active power balance cause a positive or negative mechanical torque on the rotating masses of the rotors<sup>22</sup> and thus a deviation of each nodal frequency from its nominal value. Vice versa, frequency control aims at restoring the nominal frequency by means of positive or negative active power injections. As of today, frequency control interventions are invoked by the TSO using a hierarchical sequence of actions based on the standard scenario of a large outage or load jump: Immediately after the failure, there is a change in the kinetic energy of the rotating masses and thus a steady decrease or increase in frequency. The rotational inertia prevents an immediate collapse of the grid, which is called *momentary reserve*. To stabilize the frequency, a decentralized proportional control (*primary control*) is activated within the first 30 s after the incident. Power plants participating in primary control are equally distributed over the interconnected grid. Using a specific *droop coefficient*, each power plant's own positive or negative active power injection can then be calculated directly from in-situ measurements of the nodal frequency deviation. Since the proportional controllers from primary control lead to a steady-state frequency deviation, *secondary control* is activated in the next step in order to regulate the frequency towards its nominal value. For this purpose, each TSO calculates its own *area control error* by integrating the frequency deviation and then centrally activates the responsible power plants so that the required active power is available in no later than 5 min. Lastly, *tertiary control* (sometimes also called *minute reserve*), which is available within 15 min, aims at restoring the pre-contingency balance among TSOs by activating additional positive or negative power injections or, if necessary, re-scheduling the setpoints of the secondary layer. All three control layers are marketed separately on the BM<sup>23</sup> (cf. Table 2.1), with a further separation between positive and negative power for the secondary and tertiary level.

With the ongoing transition from SG-dominated CPPs towards inverter-interfaced DERs, plenty of research has been undertaken to investigate how inverters can be seamlessly integrated into the existing hierarchy of frequency control. The most prominent approach is *inverter droop control*, where the droop characteristic of SGs with proportional relationship between active power infeed and frequency deviation is mimicked. Originally published in [CDA93], this approach has since been extensively studied, e.g. in [TJUM97, CCG02, BPP<sup>+</sup>08].

<sup>22</sup> The dynamic behavior during the self-synchronization process between the nodal frequencies of an interconnected power system can be analyzed in more detail by the theory of coupled Kuramoto oscillators [DCB13].

<sup>23</sup> Within the ENTSO-E supergrid, primary, secondary and tertiary frequency control are referred to as *frequency containment reserve* (FCR), *automatic frequency restoration reserve* (aFRR), and *manual frequency restoration reserve* (mFRR), respectively.

The main advantage of inverter droop control is its decentralized nature without the need for extensive communication and its full backward compatibility with a SG-dominated environment. However, as with the conventional droop control for SGs, a steady-state deviation from nominal frequency remains. Besides, the practical effort to correctly tune the droop parameters is supposed to be high [CS13]. Apart from droop-based approaches, several alternative control methods have been proposed, such as *virtual synchronous machine* [BH07, DSF15], *virtual oscillator control* [SDJD15], or *internal matching control* [JAD16, MDS+18], all with the goal of adopting the rotational inertia concept familiar from SGs to inverters.

Owing to the faster operating time scales of inverter-interfaced DERs and new topological concepts such as MG, VPP, and WoC, numerous papers have investigated how to overcome the traditional hierarchy of primary, secondary, and tertiary frequency control by means of a *unifying* control strategy [GCLL13]. It was derived in [LCZL14] and [LZC16] that economic dispatch can be integrated into a unifying secondary control scheme. Moreover, in [DSPB16] it was shown that each minimizer of an economic dispatch problem can be reached by a decentralized droop controller with properly chosen droop coefficients. To combine the primary and secondary control level, the authors in [ADJS13, SPDB13] present *distributed averaging proportional-integral* (DAPI) controllers where zero frequency deviation is reached via a neighbor-to-neighbor communication path. A gather-and-broadcast approach is derived in [DG17], where the integration between economic dispatch and frequency control is performed in a semi-decentralized fashion with locally aggregated measurements of the nodal frequency deviations. In [FP10], *primal-dual saddle-point flow* is applied to frequency control. It allows to unify all three control levels together with economic dispatch by characterizing the desired equilibrium as a minimizer of a constrained optimization problem. Since the dual variables of the cost functions can be interpreted as price signals, no generation setpoints by a superordinate authority are needed. This understanding has led to the idea of *continuous-time dynamic pricing* controllers for frequency regulation [ZTL12, SDv15, TBD16, MZL17, SDv17a]. Furthermore, inequality constraints can be incorporated either by using extended Lagrangians [SDv17b], projection operators [ZP15, SDv17b, WLP<sup>+</sup>19], or a saturation term [ZMLB18]. A rigorous performance analysis of primal-dual saddle-point flow applied to DC power flow can be found in [SPPMD16] and [CY17]. Detailed discussions of current research work on optimization-based frequency control are conducted in [DBSPG19].

Meanwhile, increased computing power has also permitted MPC-based and game-theoretic control methods to be considered for real-time or near-real-time purposes: A computationally tractable DMPC scheme for frequency control based on a linearized MG model is presented in [BT17]. The authors of [STY<sup>+</sup>18] present a DMPC strategy for heterogeneous power systems fed by CPPs and wind power plants, which is able to achieve frequency restoration by incorporating ultra short-term wind forecasts. In addition, there exist noncooperative concepts to frequency regulation which address the individual goals of stability providers [CYWL15, NOS<sup>+</sup>15, CSP<sup>+</sup>19]. While [CYWL15] focuses on load frequency control using a linear-quadratic differential game, [NOS<sup>+</sup>15] provides a real-time framework for frequency control by means of time-varying prices offered by a centralized ISO. Both approaches rely on a linearized system model. In [SCA16], an aggregate frequency error is used as additive feedback signal in a real-time WEM. In [CSP<sup>+</sup>19], advanced meta-heuristic optimization techniques are applied to a nonlinear system model in order to improve the frequency response characteristic on device level.

#### **2.2.5 Voltage Control**

Voltage control aims to keep voltage magnitudes at each node within a tolerance band around the specific nominal voltage. Unlike frequency control, local actions of voltage control only have a local impact, thus control actions are typically<sup>24</sup> accomplished in a three-stage process with increasing geo-spatial coverage: At the lowest level (*primary control*), the local voltage is regulated to a given setpoint. For this purpose, excitation systems of the SGs (*automatic voltage regulation*) as well as active and passive network components, e.g. phase shifters and *flexible AC transmission systems* (FACTSs) are applied. At the *secondary control* level, local setpoints within an entire zone are adjusted so that a designated *pilot point* matches its setpoint and, in addition, there is no voltage band violation at any node. Lastly, *tertiary control* aims to centrally coordinate the individual zones and adjust the setpoint voltages at pilot points.

To enable seamless integration of inverter-inferfaced DERs into the existing control hierarchy, numerous decentralized droop-based approaches for voltage control have been proposed [CDA93, CC18, LSZ18]. Here, local deviations from the voltage setpoint are compensated by additional *reactive* power injections similarly to the active power injections of the frequency droop approach. Another similarity to frequency droop is the nonzero steady-state deviation due to the purely proportional control architecture. Namely, it can be shown that in order to keep the steady-state deviation within an acceptable range, R/X ratio of all lines must be small, ideally zero. Yet, this assumption is not admissible for distribution networks [MMMT12].

There are several alternative methods for voltage control based on the formulation of optimization problems. As shown in [FCL13], the standard voltage droop control is inversely optimal to a convex optimization problem. Based on this finding, [ZC16] and [KZGB16] propose an iterative procedure for adjusting the droop characteristic in order to keep all voltages of a radial network within an acceptable range. [SPDB17] presents a *quadratic droop* controller as the solution to an optimization problem which minimizes a trade-off between voltage setpoint deviation and reactive power dissipation. [HGK18] presents an approach how to optimally tune the droop parameters.

Furthermore, several publications have appeared in recent years documenting distributed *non-droop-based* methods with neighbor-to-neighbor communication among nodes. [CTFS18] proposes a combination of sliding mode and DAPI control, where the weighted average voltage of the MG is provably equal to the weighted average of all voltage references. [THL19] develops an optimization-based procedure for voltage control in radial distribution grids, where the objective function reflects the setpoint deviation of voltages and where upper and lower limits on reactive power injection are specified by box constraints. The proposed method is based upon the assumption of a uniform R/X ratio throughout the network. In addition, [LSZ18] allows to specify the overall amount of reactive power injection as additional soft constraint and a linearized relation between voltage and reactive power as additional equality constraint. To enable an online computation, [THL19] uses the distributed accelerated dual descent algorithm, while [LSZ18] is based on the alternating direction method of multipliers. Both approaches rely on a linearized system model.

<sup>24</sup> The degree of automation of each level as well as the responsibilities of TSO and DSO highly depend on the specific grid code. The interested reader is referred to [vV08, MC11] for further elaborations.

In the context of MGs, several authors propose *passivity-based* approaches to voltage stabilization. In [ABEPCR18, ABMEPG18], primary voltage controllers are developed on the basis of a linearized *port-Hamiltonian* model<sup>25</sup> of the MG, such that a specific energy function of the closed-loop system has its minimum at the desired setpoint voltages. [AvJ17] presents a secondary voltage controller based on a nonlinear port-Hamiltonian model which provides zero setpoint deviation in lossless networks.

More recently, a number of game-theoretic methods addressing voltage stability have emerged. Most papers focus on sequential approaches, where a trade-off between the grid-forming generators and, if applicable, consumers can be found either iteratively [GA20] or in a leaderfollower manner [CZ18]. A simultaneous approach is proposed by [WHK18], where voltage stabilization is achieved by incentivizing PV owners with additional financial payments, if they follow a given desired voltage profile. Yet, the proposed method is applicable for radial networks only.

#### **2.2.6 Combined Frequency and Voltage Control**

Regulating frequency and voltage separately from each other always relies upon the assumption that line resistances (or line reactances<sup>26</sup>) are negligible. However, especially in medium voltage networks, which are neither dominantly inductive nor dominantly resistive, this assumption does not hold in practice. For this purpose, [GMG<sup>+</sup>07, LK09] use *virtual impedance* techniques which allow a modification of the setpoint voltages by means of a second feedback loop in order to compensate the cross-term. In [RLBR12, BS13], a *generalized droop control* is introduced, which provides coupled droop characteristics with cross-terms for active and reactive power. With the resulting equations, it is required that all line and load parameters are known. Moreover, similar to decoupled droop control, generalized droop control cannot regulate the steady-state deviation of frequency and voltage to zero.

In [SPSD<sup>+</sup>15], a DAPI controller for primary and secondary frequency and voltage control in inverter-based MGs is proposed. While the primary level relies upon a droop characteristic, the secondary controller uses a consensus protocol in order to restore the nominal frequency by means of an arbitrary connected communication path between the agents. Moreover, the presented controller in [SPSD<sup>+</sup>15] provides a tuning functionality for the conflicting goals of reactive power sharing and voltage regulation. [DMSD16] augments the DAPI controller by a modified integral control law to minimize a quadratic optimization problem for the active power generations. In order to achieve better reactive power sharing, [WSI18] combines a consensus protocol for secondary frequency and voltage control with an additive correction of the voltage references. A slightly different approach is presented in [SGV14]. Here, distributed consensus is reached via all-to-all communication among all agents in the network. The approach is capable of providing both frequency and voltage regulation without knowing the

<sup>25</sup> The port-Hamiltonian modeling framework [vJ14] is a powerful modeling tool for complex multi-domain physical systems that has gained increased attention from various fields of application in the last years [MMS09, FZO+13, AS17, ACRMA19, PCH+20]. A brief introduction to the basic terminology used in the context of this dissertation is given in Appendix A.4.

<sup>26</sup> If the network is assumed to be purely resistive, then frequency and voltage are decoupled as well, where frequency can be controlled by reactive power and voltage by active power [LSK05].

physical network parameters. However, both the amount of tunable parameters as well as overall communication effort is substantial.

Among non-droop-based methods, a number of contributions employ passivity theory to obtain frequency and voltage controllers with zero setpoint deviation: In [RSFT15, TFRFT16, TFT17], stabilizing controller parameter are found via a centralized solution of linear matrix inequalities. An extension is presented in [NFT19], which allows to include the dynamics of the power lines into the underlying MG model. In [SMKH19], the criterion for stabilizing controller parameters is further simplified to sufficient conditions which can be calculated a priori. Besides, [DE18] presents a first differential game formalism for joint frequency and voltage control of inverter-based islanded MGs. However, as pointed out by the authors, a high computational effort as well as scalability issues limit its applicability to small-scale systems only.

## **2.3 Discussion**

In light of the future developments and disruptions, the preceding section conducted an extensive review of current research trends towards a stable, economically efficient and gridsupportive operation as objected at the end of Chapter 1. Table 2.2 provides a summarizing overview of existing papers on distributed control methods that combine at least *two* of the subjects economic dispatch, frequency control, or voltage control.

First, we analyze the adequacy of the state of research in ensuring a *stable* operation of future power systems. Table 2.2 indicates that research in voltage control has tended to concentrate on internal and primary control methods specifically suited for inverter-based networks, and to overlook integrating secondary and tertiary voltage control strategies from an optimization viewpoint. Since the traditional control hierarchy described in Section 2.2.5 is typically kept untouched, it has to be assumed that a higher-level controller is able to provide optimal setpoints. Although a few papers, e.g. [LQD14, KZGB16, GA20, HBHD21a], have addressed the question of *autonomous* voltage control frameworks that unify the primary and secondary control layer, all of these approaches are subject to a centralized controller and thus exhibit a single point of failure.

In the area of frequency control, the rise of intermittent DERs and the new phenomenon of volatile residual loads have led to a new, more general understanding of the traditional hierarchy and its purpose: Originally defined as a scheduled cascade which has to be activated in a strict sequence in the event of a major power plant outage, the literature nowadays more generally defines primary control as frequency containment and secondary control as frequency restoration, which has led to many unifying control approaches in which both goals are addressed at the same time. Meanwhile, a strong connection between tertiary frequency control and economic dispatch has been exposed [LCZL14], which later resulted in the idea of real-time dynamic pricing.

However, it can be seen from Table 2.2 that all previous papers dealing with the combination of economic dispatch with frequency and voltage control are not capable in providing nonzero deviation of frequencies and voltages from their respective setpoints. This limitation is caused


**Table 2.2:** State of literature on multi-disciplinary real-time control methods for interconnected power systems.

♣ Network Connectors: S = SMs, I = inverters, L = loads.

‡ Economic Dispatch: <sup>=</sup> competitive goals with distributed controllers, G# <sup>=</sup> competitive goals with centralized controller, G# <sup>=</sup> centralized goal with distributed controllers, G#G# <sup>=</sup> centralized goal with centralized controller, −− <sup>=</sup> not provided.

§ Congestion Management: <sup>=</sup> fully provided, G# <sup>=</sup> partly provided, <sup>−</sup> <sup>=</sup> not provided.

¶ Frequency Control: <sup>=</sup> zero frequency deviation, G# <sup>=</sup> nonzero frequency deviation, <sup>−</sup> <sup>=</sup> not provided.

♭ Voltage Control: <sup>=</sup> fully provided, G# <sup>=</sup> partly provided, <sup>−</sup> <sup>=</sup> not provided.

by the fact that the controllers are designed under the simplifying assumption that line conductances can be neglegted. Yet this assumption, as mentioned earlier, is inadmissible especially for distribution grids [MMMT12]. As a consequence, a practical implementation of all of these controllers would still require some kind of additional frequency or voltage restoration by a superordinate controller.

Moreover, it was pointed out in e.g. [Sch15] that the *heterogeneity* of future power systems requires that frequency and voltage control have to be provided by both conventional SMs and inverter-interfaced DERs. However, referring to the first column of Table 2.2, the existing controllers are applicable only to a limited group of network connectors, for instance conventional SMs or grid-forming inverters only. These findings together expose the following research gap.

**Research Gap 1.** *For interconnected power systems with nonzero transmission losses, there are no control strategies for optimal dispatch of heterogeneous network connectors which ensure both frequency and voltage stability.*

With regard to the second sub-objective of *economic efficiency*, the existing controllers reveal several weaknesses: As Table 2.2 suggests, most control approaches follow a common goal, i.e. all network participants are assumed to be non-competitive. On the contrary, pricebased approaches such as [NOS+15, SCD+19, CSD+20], which allow for individual profitmaximizing agents, feature a centralized information and communication structure, with a single node that is responsible for the entire pricing process. The few existing approaches with both competitive agents and distributed communication all require to disclose their own strategy to all other competitors. Thus, no existing approach is able to fully amalgamate both market and control strategy while also achieving legal, operational, and informational unbundling. Even worse, due to their higher communication effort, these approaches are likely to further amplify the interferences and dependencies between competitive electricity providers (CPPs or inverter-interfaced RESs) and regulated infrastructure providers (system operators), that are already occuring at present (cf. Section 2.1.2).

Besides these structural inefficiencies, the employed game-theoretic solution concepts of Nash [SLL02, MWY+19], generalized Nash [AOS+13b, AOS+13a], or Stackelberg equilibrium [TZS<sup>+</sup>14, ALCB15, EES<sup>+</sup>19] only indicate the existence of *some* equilibrium, which no participant will unilaterally leave. In particular, these solution concepts are not able to properly assess whether the resulting equilibrium might be dominated by another equilibrium which is more desirable from the participants' points of view (i.e., generates higher profits for some of the participants).

As a further major drawback, most market-based control approaches also rely on very simplified models with zero line resistances. Due to the disregard of losses over transmission lines, the resulting market-clearing prices are infeasible, i.e. this again requires some sort of after-market mechanism with an exchange of side payments and/or additional grid tariffs, which would have had to be supervised by a higher-level authority. Therefore, to the best of the author's knowledge, all existing price-based control schemes are unable to balance the

resulting net payments to zero at market clearing, whenever the underlying power network has nonzero transmission losses.

**Research Gap 2.** *There is no systematic control framework which provides a trade-off between individual interests and market balancing while complying with the desirable information barriers between power plants and system operators.*

For the third sub-objective of *grid-supportive* operation of power systems, the existing pricebased control methods suffer from a number of pitfalls that prohibit to incentivize generation close to load centers. In particular, the vast majority of price-based approaches developed so far result in temporally different, but geographically uniform prices. Therefore, it has no *spatial control effect* in any case. On the other hand, iterative bidding approaches such as [SDv19, CC20], which allow for geographically non-uniform prices, do not account for transmission losses throughout the network. In this way, the copper plate assumption, which was problematized in Section 2.1.2, persists, thus there is no incentive for local energy supply near load centers (cf. [HSMT19, p. 14]). Likewise, the conceptual approaches for nodal and zonal pricing surveyed in [Wei17] always rely on simplifying DC-OPF models not considering transmission losses, which are not able to accurately reflect the actual states and physical relationships in the network, especially if the network is heavily loaded [Bak21, p. 266].

Considerably, no price-based control approach is able to combat congestion in real time. The majority of congestion management strategies are on a larger time frame, e.g. with time slices at an hourly basis and a daily planning horizon. With regard to real-time approaches, Table 2.2 shows that several authors integrate congestion management strategies as additional constraints into the economic dispatch problem. However, in most cases this economic dispatch results in a centralized optimization problem which is solved by non-competitive agents. The few competitive approaches considering congestion management [SCD<sup>+</sup>19, CSD<sup>+</sup>20] are only applicable for power systems with a tree topology. Moreover, all of the existing real-time congestion management strategies neglect the effect of reactive power flows on network congestion. At the same time, none of the purely competitive approaches with distributed communication (last three rows in Table 2.2) are able to properly reflect network congestion.

All in all, existing dynamic pricing controllers fail to implement a real-time mechanism in order to incorporate network effects to the local prices. Within the literature, there is a large gap between long-term, investment signaling schemes, which are purely conceptual and do not provide a real-time mechanisms, and real-time pricing controllers that neglect the physical characteristics and limitations of the network and thus provide no means for incentive signaling. Therefore, a holistic short-term control framework is lacking, that is able to hold grid participants accountable for their own impact on the momentary state of the network.

**Research Gap 3.** *There exists no constructive pricing mechanism to provide regional market incentives for power plants by evaluating the momentary positive or negative impact of local power generation.*

**Figure 2.3:** Outline of main contributions. "Higher-level" contributions on slower timescales are positioned at the top, while "lower-level" contributions acting on faster timescales are positioned at the bottom.

## **2.4 Statement of Contributions**

To overcome the limitations and gaps of existing methods, this dissertation is dedicated to developing a distributed, continuous-time control framework for competitive power systems based on real-time dynamic pricing. By consistently accounting for transmission losses, the resulting control scheme shall not require any kind of settlement payments or after-market clearing procedure. Furthermore, due to a rigid compliance with information barriers between economic players and system operators, the presented brownfield mechanism shall be incentive-compatible in a fully unbundled environment. All control laws are expressed as explicit differential equations. The main contributions of this dissertation and their relationships to the revealed research gaps are illustrated in Fig. 2.3 and briefly outlined in the following.

**Contribution 1.** *This dissertation develops a distributed, market-based feedback control strategy for lossy and heterogeneous power grids, incorporating both, frequency and voltage control, with optimal dispatch.*

The control methods presented in Chapter 3 of this dissertation allow a real-time optimization of the connected DERs as well as conventional SGs, with the primary goal of maintaining the nominal frequency and keeping all voltage magnitudes within predefined limits whilst minimizing an individual user-defined cost function. The proposed controller is unifying in the sense that the former hierarchical division into primary, secondary, and tertiary frequency and voltage control tasks are combined within a single controller (see bottom panel in Fig. 2.3). Furthermore, frequency and voltage stabilization is maintained simultaneously. The resulting market-based control structure is fully compliant with legal, operational, and informational unbundling. In particular, motivated from an economic perspective, no communication between competitive players, no generation of exogenous setpoints by a higher-level authority, and no disclosure of individual cost functions is necessary. The controller design is based

on a nonlinear, *network-preserving* model<sup>27</sup> of the power system, and allows for a distributed implementation.

**Contribution 2.** *This dissertation presents a dynamic pricing framework for vertically unbundled power systems that enforces Pareto efficiency and market-clearing allocations among competitive participants.*

The real-time pricing approach presented in Chapter 4 (see middle panel in Fig. 2.3) enables each network participant to selfishly perform an individual profit maximization by means of internal control actions. In the light of [Ahl18, p. 324f.], the game-theoretic notion of *Pareto efficiency* (cf. Definition A.5) is employed to ensure economically efficient operation of all network participants. Accordingly, the resulting closed-loop equilibrium is such that there exists no other equilibrium where one or more network participants take a higher profit. A distributed market balancing scheme ensures that the total revenues of producers are always kept identical to the payments of consumers. This permits the additional costs arising from momentary transmission losses to be reflected directly in the real-time price, without creating net deficits in overall generation or an accumulation of capital. The resulting (optimal) market balancing problem gives rise to the problem class of *(dynamic) optimal control problems and differential games subject to port-Hamiltonian systems* (PHSs) (cf. Appendix A.4), for which no solution concept in the sense of an explicit stabilizing control strategy exists yet.

**Contribution 3.** *This dissertation develops a continuous-time optimal control method for (nonlinear) port-Hamiltonian systems with* N *agents.*

In Chapter 5 of this dissertation, we propose a continuous-time adaptive control strategy for solving single- and multi-player noncooperative differential games with general (Lagrangian) performance indices and system dynamics modeled as general nonlinear input-state-output PHSs. The proposed control law (see right panel in Fig. 2.3) implements an online learning procedure which uses the Hamiltonian of the system as initial value function candidates. The feedback strategy of players is extended by adaptively weighting the individual value functions to ensure convergence to the Nash solution. Necessary and sufficient conditions for stability of the resulting controlled system are provided by employing Lyapunov stability theory. Since PHSs have received substantial attention as being a powerful tool for systematic modeling of dynamical multi-physics systems, the presented methodology qualifies for a broad range of applications apart from power systems.

**Contribution 4.** *This dissertation presents feedback control strategies for incentivizing grid-supportive behavior of power plants by continuous generation of price signals.*

In order to contribute to a grid-supportive mechanism design, this dissertation provides a distributed framework for real-time zonal pricing. Being fully compatible with the WoC

<sup>27</sup> A power system model is said to be network-preserving if each bus is modeled separately (possibly as a dynamic subsystem) and connected to the other buses via power flow equations. Conversely, the model is said to be network-reduced, if buses within a certain geographical region are aggregated and treated as a single bus.

concept introduced in Section 2.2.3, local CCs are enabled to allocate cell-specific, temporally and spatially differentiated prices. Thereby, higher-level goals such as congestion management or local investment signaling can be implemented in both the short term and long term (see top panel in Fig. 2.3). In fact, the presented mechanism allows to pursue those equilibria that are desirable from a global perspective by means of a distributed consensus protocol.

The above contributions are systematically elaborated in the subsequent Chapters 3–5, which are organized as follows: Chapter 3 develops a network-preserving power system and market model conforming with the WoC concept, and presents a price-based control framework with distributed *intra-cell* communication to satisfy Contribution 1. Chapter 4 presents a distributed control framework at *inter-cell* level for an automatic regulation of spatial price differences, working towards Contribution 2 and Contribution 4. Finally, Chapter 5 provides the solution of dynamic optimal control problems for PHSs to complete the remaining tasks of Contribution 2 along with Contribution 3.

# **3 Optimization-Based Control of Cellular Power Networks**

In this chapter, a WoC-based control framework for economic dispatch and power system stabilization of competitive network participants acting in an unbundled power system is developed. After introducing the general setup of the WoC-based infrastructure in Section 3.1, Section 3.2 presents a dynamic, network-preserving model of the physical power system. Section 3.3 investigates the competitive behavior of unbundled network participants and develops a conforming market mechanism. Section 3.4 derives price-based local controllers at intra-cell level and investigates the overall system dynamics resulting from the coupling with the physical system. Thereby Contribution 1 is provided. Section 3.5 examines the interaction between cells and gives an initial diretion towards inter-cell pricing strategies. Section 3.6 briefly discusses the main findings of this chapter<sup>28</sup> .

## **3.1 General Setup**

We consider an interconnected power system with meshed physical topology, which is modeled as a directed graph G<sup>p</sup> = (Vp, Ep). The nodes v ∈ V<sup>p</sup> correspond to n<sup>p</sup> buses and the edges e ∈ E<sup>p</sup> to m<sup>p</sup> power lines connecting the buses, where the direction of the edges indicates the direction of positive power flow. The incidence matrix (cf. Definition A.10) of G<sup>p</sup> is denoted by <sup>D</sup>p. The power system is divided into <sup>n</sup><sup>Z</sup> <sup>∈</sup> <sup>N</sup> cells operated by a CC <sup>k</sup> ∈ Z <sup>=</sup> {k1, k2, . . .}. Within its own cell boundaries, each CC acts as an ISO, being exclusively responsible to ensure the main objectives of stability, economic efficiency and grid-supportive operation outlined in Chapter 1. The responsibilities and information patterns assigned to the CCs as well as their interactions with other network participants are detailed in Section 3.3.

Regardless of cell boundaries, the power system is connected with *wholesale prosumers* (WPs) and *retail prosumers* (RPs). Both WPs and RPs are generic groups for network participants that produce and/or consume electrical energy. A major difference between both groups, however, is the presence or absence of *flexibility*: For (flexible) WPs, the own power supply or demand is the result of economic considerations (i.e. maximization of own profit) and thus supposed to be affected by prices. For (inflexible) RPs, on the other hand, the supply or demand of RPs is assumed to be uncontrollable. These two classes thus cover all possible types of energy producers and consumers.

<sup>28</sup> Preliminary versions of the results of this chapter have been published in the conference papers [KBKH19, KDB+20, KWKH20].

For the overall model, let <sup>W</sup> <sup>=</sup> {π1, π2, . . .} denote the set of <sup>n</sup><sup>W</sup> WPs<sup>29</sup>. Each node is associated to *at least* one WP π ∈ W. We denote the subset of nodes i ∈ V<sup>p</sup> associated to WP π ∈ W by VW,π ⊆ Vp. Moreover, each node i ∈ V<sup>p</sup> belongs to *exactly* one of the n<sup>Z</sup> *cells* <sup>V</sup>Z,k ⊆ Vp, <sup>k</sup> = 1, . . . , n<sup>Z</sup> , i.e. <sup>V</sup>Z,k ∩ VZ,l <sup>=</sup> <sup>∅</sup> for all <sup>k</sup> ̸<sup>=</sup> <sup>l</sup> and <sup>S</sup><sup>n</sup><sup>Z</sup> <sup>k</sup>=1 VZ,k = Vp. The edges in E<sup>p</sup> can further be divided into the set E b <sup>p</sup> of *inter-cell* or *boundary* edges representing power lines that connect buses located in different cells<sup>30</sup> and the set <sup>E</sup> nb <sup>p</sup> of *intra-cell* or *nonboundary* edges representing power lines that start and end within the same cell. The physical interconnection of different cells is described by the graph Gˆ <sup>p</sup> = (Z, E b p ) with incidence matrix Dˆ p.

A fundamental principle of the WoC concept is that measurement and control information is only shared *locally*. This is expressed by a separate communication infrastructure for each cell k ∈ Z represented by directed graphs Gc,k = (VZ,k, Ec,k) with incidence matrices Dc,k. Note that the direction of the edges of the communication infrastructures do not imply a one-way connection but rather the direction of positive "communication flows". The following assumption holds under mild conditions:

**Assumption 3.1.** *The graph of the physical system* G<sup>p</sup> *and all communication graphs* Gc,k*,* k ∈ Z *are weakly connected.*

Assumption 3.1 implies that Gˆ <sup>p</sup> is weakly connected as well.

The totality of all communication between nodes is represented by the communication graph

$$\mathcal{H}\_{\mathbf{c}} = \left(\mathcal{V}\_{\mathbf{p}}, \bigcup\_{k \in \mathcal{Z}} \mathcal{E}\_{\mathbf{c},k}\right). \tag{3.1}$$

This leads to the following formal definition:

**Definition 3.2 (Cell).** *A* cell *is a weakly connected component in* Gc*.*

Notably, the characteristics of the WoC scheme rely only on the *communication topology*, while the physical topology is not subject to any restrictions. In particular, power flows across the cell boundaries are allowed without any limits. In the following, we summarize the main properties and assumptions on the considered power system:

**Definition 3.3 (Competitive Power System with Cell-Based Infrastructure).** *We consider an interconnected power system with the following characteristics.*

• *We consider a lossy AC system with meshed topology,*

<sup>29</sup> Note that this framework is compatible to the VPP concept (cf. [NY21]), since the individual resources of prosumers may be located at geographically distant nodes in different cells.

<sup>30</sup> If there are multiple inter-cell power lines connecting the same cells, then Eˆ<sup>p</sup> contains an edge for each of these power lines.

**(a)** WoC topology **(b)** MG topology **Figure 3.1:** Differences between WoC and MG topology.


**Remark 3.4.** *Compared to our model from Definition 3.3, the definition of a* grid of MGs *[Sch15, Definition 3.2.1] differs by the following limiting characteristics:*


Remark 3.4 emphasizes that the MG concept is subject to more structural restrictions as well as more conservative capacity requirements (which are hard to verify in advance) than the WoC concept pursued in this work. Fig. 3.1 schematically summarizes the topological differences between these two concepts. Using the above notation, the topology of an islanded MG is regarded as the special case <sup>E</sup><sup>ˆ</sup> <sup>p</sup> = ∅. In the following, however, we will generally assume that <sup>G</sup><sup>c</sup> ̸<sup>=</sup> <sup>G</sup><sup>p</sup> and <sup>E</sup><sup>ˆ</sup> <sup>p</sup> ̸= ∅ hold in order to permit a global exchange of power. For reasons of comparison, though, the MG approach will occasionally serve as a reference scenario.

# **3.2 Modeling of the Physical Network**

The set of nodes can be partitioned as V<sup>p</sup> = V<sup>S</sup> ∪ V<sup>I</sup> ∪ VL, where each subset represents a certain type of network connectors, namely, *synchronous machine-type* (S) nodes, *inverter-type* (I), and *load-type* (L) nodes. S nodes are connected to synchronous machines (SMs) of e.g.


**Table 3.1:** Network connectors.

gas or hydro turbines. I nodes are connected to power electronics interfaced DERs. L nodes are connected to producers or consumers with a fixed active and reactive power supply or demand. S and I nodes i ∈ V<sup>S</sup> ∪ V<sup>I</sup> are each equipped with an active power generation input pg,i and an active power consumption input pℓ,i. L nodes i ∈ V<sup>L</sup> are equipped with active and reactive power consumption inputs pℓ,i and qℓ,i, respectively. Table 3.1 gives an overview of the three node types along with their specific independent (exogenous) and dependent (endogenous) variables. The set Np,i denotes all *neighboring* nodes j ∈ V<sup>p</sup> of i ∈ V<sup>p</sup> with (i, j) ∈ E<sup>p</sup> or (j, i) ∈ Ep.

**Remark 3.5.** S *and* I *nodes correspond to the notion of* PV nodes *from classical power flow calculation. In the presented framework, the voltage magnitudes are either controlled directly (at* I *nodes) or indirectly via the excitation voltages* Uf,i *(at* S *nodes).* L *nodes correspond to the notion of* PQ nodes*, where the active and reactive power generation or consumption is given and the voltage magnitudes and bus voltage angles* θ<sup>i</sup> *follow from Kirchhoff's laws.*

*In classical power flow calculation, apart from PV and PQ buses there typically exists one* slack bus i ∈ Vp*, which represents a large power plant or a superordinate network with infinite capacity of real and reactive power. The slack bus is characterized by fixed* U<sup>i</sup> *and* θ<sup>i</sup> *as independent and* p<sup>i</sup> *and* q<sup>i</sup> *as dependent variables. As the resulting power flows are invariant with respect to a uniform shift of all bus voltage angles* θ<sup>i</sup> *, the slack bus typically serves as bus voltage angle reference by setting* θ<sup>i</sup> = 0*. Yet in line with Definition 3.3, our model is intended to avoid any external sources of power. Thus, no slack bus is considered within our power system model. However, to provide for nonsingular solutions within calculations, all CCs and prosumers use bus voltage angle* differences ϑij = θ<sup>i</sup> − θ<sup>j</sup> *instead of explicitly dealing with the absolute values* θ<sup>i</sup> *and* θ<sup>j</sup> *.* ♢

The following subsections systematically derive a common model for the physical interaction of the power system. Subsection 3.2.1 considers the power flows over lossy transmission lines. Subsections 3.2.2–3.2.4 present an appropriate model for SM-type, inverter-type, and load-type nodes, respectively. On this basis, Subsection 3.2.5 synthesizes the dynamic model of the overall physical plant system.

#### **3.2.1 Power Lines**

Regarding the physical infrastructure, Definition 3.3 does not restrict the voltage level of the underlying power grid. In the following, however, we will focus on *medium voltage networks* (with nominal voltages typically between 10 kV and 100 kV), as the majority of DERs are connected at this voltage level. In addition to the properties from Definition 3.3, medium voltage networks can be assumed to have the following characteristics under normal operating conditions.

**Assumption 3.6.** *a) The network is operating around the nominal frequency. b) The network is a three-phased AC system which is operated under symmetrical conditions.*

Note that Assumption 3.6a) is only needed for the purpose of model validity and does not imply the (much stricter) requirement of frequency stability to be addressed in the following sections. Assumption 3.6b) implies that the network is both symmetrically configured and symmetrically operated, see [SFO15, Section 2.4.1] for basic terminology. With Assumption 3.6b), a single-phase equivalent circuit model [MBB12, p. 66] can be applied (see Fig. 3.2). Here,

**Figure 3.2:** Infinitesimally short line segment of length dh.

each phase of a power line connecting two nodes i, j ∈ V<sup>p</sup> is decomposed into an infinite number of increments dh (see Fig. 3.2). Each of these increments can be represented by a specific series resistance R′ ij characterizing the thermal losses, a specific series reactance X′ ij characterizing the flux linkage within the power line and the flux linkage to the other phases, a specific parallel susceptance B′ ij describing the potential difference between the individual phases, and a specific parallel conductance G′ ij .

With Assumption 3.6b) and provided that no line length exceeds 300 km, the Π-equivalent circuit model can be applied [And, p. 101; MBB12, p. 67]. It provides a lumped-parameter representation (see Fig. 3.3) with

$$\vec{Z}\_{\rm L,ij} = \sqrt{\frac{\vec{Z}\_{ij}^{\prime}}{\vec{Y}\_{ij}^{\prime}}} \cdot \sinh\left(\sqrt{\vec{Z}\_{ij}^{\prime}\vec{Y}\_{ij}^{\prime}} \cdot h\right),\tag{3.2}$$

$$
\vec{Y}\_{\rm L,ij} = \sqrt{\frac{\vec{Z}\_{ij}'}{\vec{Y}\_{ij}'}} \cdot \tanh\left(\sqrt{\vec{Z}\_{ij}'\vec{Y}\_{ij}'} \cdot h\right),
\tag{3.3}
$$

**Figure 3.3:** Π-equivalent circuit model for a power line between nodes i and j.

where Z⃗′ ij := R′ ij + ȷX′ ij and Y⃗ ′ ij = G′ ij + ȷB′ ij . Together with the nodal shunt admittance ⃗ysh,i = gsh,i+ȷbsh,i, we obtain ⃗y<sup>i</sup> = ⃗ysh,i+ P j∈Np,i Y⃗ <sup>L</sup>,ij . Consequently, the entire network of power lines can be described in vector-matrix notation by the *nodal network equation* [MBB12, p. 113]

$$
\vec{I}\_{\mathbf{p}} = \vec{Y}\_{\mathbf{p}} \vec{U}\_{\mathbf{p}} \tag{3.4}
$$

with ⃗I<sup>p</sup> = col<sup>i</sup>∈V<sup>p</sup> { ⃗Ii} and <sup>U</sup>⃗ <sup>p</sup> = col<sup>i</sup>∈V<sup>p</sup> {U⃗ <sup>i</sup>}. The nodal admittance matrix <sup>Y</sup>⃗ <sup>p</sup> describes the relationship between the nodal voltages U⃗ <sup>p</sup> and the currents ⃗I<sup>p</sup> entering the nodes. If node i is connected to node j by a line, then

$$
\left[\vec{\mathbf{Y}}\_{\rm p}\right]\_{ij} = \left[\vec{\mathbf{Y}}\_{\rm p}\right]\_{ji} = -\frac{1}{\vec{Z}\_{\rm L,ij}},\tag{3.5}
$$

$$\left[\vec{\mathbf{Y}}\_{\rm p}\right]\_{ii} = \vec{y}\_i + \sum\_{j \in \mathcal{N}\_{\rm p,i}} \frac{1}{\vec{Z}\_{\rm L,ij}}.\tag{3.6}$$

Using the nodal admittance matrix Y⃗ <sup>p</sup>, the apparent power balance can be determined to

$$\vec{s}\_i = \sum\_{i \in \mathcal{V}\_\mathbf{p}} \sum\_{j \in \mathcal{N}\_\mathbf{p}, i} \vec{U}\_i \vec{U}\_j^\* [\vec{\mathbf{Y}}\_\mathbf{p}]\_{ij}^\*,\tag{3.7}$$

where ⃗s<sup>i</sup> denotes the net power flow from node i ∈ V<sup>p</sup> to neighboring nodes. Now the equation can be split into its real and imaginary part [SFO15, p. 90]:

$$p\_i = \sum\_{i \in \mathcal{V}\_\mathbf{p}} \sum\_{j \in \mathcal{N}\_\mathbf{p}, i} U\_i U\_j Y\_{\mathbf{p},ij} \cos(\vartheta\_{ij} - \beta\_{ij}),\tag{3.8a}$$

$$q\_i = \sum\_{i \in \mathcal{V}\_\mathbf{p}} \sum\_{j \in \mathcal{N}\_{\mathbf{p},i}} U\_i U\_j Y\_{\mathbf{p},ij} \sin(\vartheta\_{ij} - \beta\_{ij}),\tag{3.8b}$$

where <sup>Y</sup>p,ij := <sup>|</sup>[Y⃗ <sup>p</sup>]ij <sup>|</sup> and <sup>β</sup>ij := arg([Y⃗ <sup>p</sup>]ij ).

A widely used representation of the power flows (3.7) are the *hybrid network equations*, where the nodal admittance matrix is given in rectangular coordinates using Y⃗ <sup>p</sup> =: G<sup>p</sup> + ȷBp, while the nodal voltages are given in polar coordinates using <sup>U</sup>⃗ =: <sup>U</sup> ◦ exp(ȷθ). This yields

the network equations at node i ∈ V<sup>p</sup> as

$$p\_i = U\_i^2 G\_{ii} + \sum\_{j \in \mathcal{N}\_{\mathbb{P},i}} U\_i U\_j \left( B\_{ij} \sin(\vartheta\_{ij}) + G\_{ij} \cos(\vartheta\_{ij}) \right), \tag{3.9}$$

$$q\_i = -U\_i^2 B\_{ii} + \sum\_{j \in \mathcal{N}\_{\rm p,i}} U\_i U\_j \left( G\_{ij} \sin(\vartheta\_{ij}) - B\_{ij} \cos(\vartheta\_{ij}) \right). \tag{3.10}$$

The (sending-end) power flow from node <sup>i</sup> ∈ V<sup>p</sup> to node <sup>j</sup> ∈ V<sup>p</sup> is given by S⃗ ij = Pij + ȷQij , where

$$P\_{ij} = -U\_i^2 G\_{ij} + U\_i U\_j G\_{ij} \cos(\vartheta\_{ij}) + U\_i U\_j B\_{ij} \sin(\vartheta\_{ij}),\tag{3.11}$$

$$Q\_{ij} = U\_i^2 B\_{ij} + U\_i U\_j B\_{ij} \cos(\vartheta\_{ij}) + U\_i U\_j G\_{ij} \sin(\vartheta\_{ij}).\tag{3.12}$$

Hence, the active transmission loss over line (i, j) ∈ E<sup>p</sup> equals

$$\Phi\_{ij} = P\_{ij} + P\_{ji} = -(U\_i^2 + U\_j^2)G\_{ij} + 2U\_i U\_j G\_{ij} \cos(\vartheta\_{ij}).\tag{3.13}$$

#### **3.2.2 Synchronous Machine-Type Nodes**

SMs form the backbone of today's power systems. Besides their main application of converting mechanical energy into electrical energy, e.g. as turbogenerators in bulk CPPs, they can also serve as constant-speed drives, e.g. for centrifugal pumps or air compressors.

The constant magnetic field in the *rotor* (typically governed by a DC exciter winding) interacts with the rotating field of the *stator* by inducing a *rotor voltage* (or polar wheel voltage) in the stator windings, where in *motor mode* the rotor follows the stator rotating field, while in *generator mode*, the stator rotating field follows the rotor. The rotor field winding of the SM is aligned with the *direct* (d-) axis, while the axis perpendicular to the d-axis is called *quadrature* (q-) axis. Both axes form a reference system inside the machine that rotates with the rotor. Under Assumption 3.6a), the electrical side of the SM at node i ∈ V<sup>S</sup> can be adequately described by the *sixth-order model* (3.14)–(3.16) from [MBB12, p. 455]

$$U\_{\rm gd,i} = U\_{\rm d,i}^{\prime\prime} - R\_{\rm s,i} I\_{\rm d,i} - X\_{\rm q,i}^{\prime\prime} I\_{\rm q,i},\tag{3.14a}$$

$$U\_{\rm gq,i} = U\_{\rm q,i}^{\prime\prime} + X\_{\rm d,i}^{\prime\prime} I\_{\rm d,i} - R\_{\rm s,i} I\_{\rm q,i},\tag{3.14b}$$

$$T\_{\rm d0,i}^{\prime}U\_{\rm q,i}^{\prime} = U\_{\rm f,i} - U\_{\rm q,i}^{\prime} + I\_{\rm d,i} \cdot (X\_{\rm d,i} - X\_{\rm d,i}^{\prime}),\tag{3.14c}$$

$$T\_{\mathbf{q}0,i}^{\prime} \dot{U}\_{\mathbf{d},i}^{\prime} = -U\_{\mathbf{d},i}^{\prime} + I\_{\mathbf{q},i} \cdot (X\_{\mathbf{q},i} - X\_{\mathbf{q},i}^{\prime}),\tag{3.14d}$$

$$T\_{\rm d0,i}^{\prime\prime} \dot{U}\_{\rm q,i}^{\prime\prime} = U\_{\rm q,i}^{\prime} - U\_{\rm q,i}^{\prime\prime} + I\_{\rm d,i} \cdot (X\_{\rm d,i}^{\prime} - X\_{\rm d,i}^{\prime\prime}),\tag{3.14e}$$

$$(T\_{\mathbf{q}0,i}^{\prime\prime}U\_{\mathbf{d},i}^{\prime\prime} = U\_{\mathbf{d},i}^{\prime} - U\_{\mathbf{d},i}^{\prime\prime} + I\_{\mathbf{q},i} \cdot (X\_{\mathbf{q},i}^{\prime} - X\_{\mathbf{q},i}^{\prime\prime}),\tag{3.14f}$$

where U ′′ (·) and U ′ (·) denote the subtransient and transient electromotive forces (internal voltages) induced at the stator coils and Ug(·) denotes the *terminal voltage* of the SM. Due to time-varying screening effects in the rotor, usually a distinction is made between *subtransient state*, *transient state*, and *steady state*, which is represented in (3.14) by the (sub-)transient and steady-state reactances X (·) (·),i and time constants <sup>T</sup> ′′ d0,i ≪ T ′ d0,i and T ′′ q0,i ≪ T ′ q0,i, each

modeled separately for d- and q-axis. For a detailed derivation of the model (3.14), the reader is referred to the comprehensive textbooks [Kun94, BV00, MBB12].

The mechanics of the rotor is described by a second-order system following from Newton's equation of motion

$$
\dot{\gamma}\_i = \Omega\_{\text{m},i},
\tag{3.15}
$$

$$D\_i \dot{\Omega}\_{\rm m,i} + D\_{\rm d,i} \Omega\_{\rm m,i} = M\_{\rm T,i} - M\_{\rm e,i},\tag{3.16}$$

where γ<sup>i</sup> denotes the *rotor angle* between the q-axis and a static (non-rotating) reference frame, Ji is the moment of inertia of the turbogenerator set, Dd,i is the damping-torque coefficient, MT,i is the turbine torque, and Me,i is the (counteracting) electromechanical torque.

We make the following simplifying assumptions for the SM model.


*c) Saliency of the rotor is neglected.*

Assumptions 3.7a) and 3.7b) are justified by the timescale limitation adopted in Section 2.2 (cf. Fig. 2.2). Assumption 3.7c) reflects the most common configuration of turbogeneraors in CPPs.

With Assumption 3.7a), all effects of the damper windings are neglected, yielding T ′′ d0,i = T ′′ q0,i = 0. Typically, the asynchronous torque of the damper windings is approximated by an additional damping term +dd,iωm,i in (3.16), which is proportional to the deviation ωm,i = Ωm,i − Ω 0 <sup>m</sup>,i of the mechanical angular velocity of the rotor from its nominal value Ω 0 <sup>m</sup>,i. In particular, inserting (3.14e) with T ′′ d0,i = 0 into (3.14a), and (3.14f) with T ′′ q0,i = 0 into (3.14a) yields

$$U\_{\rm gd,i} = U\_{\rm d,i}' - I\_{\rm q,i} X\_{\rm q,i}' - R\_{\rm s,i} I\_{\rm d,i},\tag{3.17a}$$

$$U\_{\rm gq,i} = U'\_{\rm q,i} + I\_{\rm d,i} X'\_{\rm d,i} - R\_{\rm s,i} I\_{\rm q,i},\tag{3.17b}$$

$$(T'\_{\rm d0,i}U'\_{\rm q,i} = U\_{\rm f,i} - U'\_{\rm q,i} + I\_{\rm d,i} \cdot (X\_{\rm d,i} - X'\_{\rm d,i}),\tag{3.17c}$$

$$T'\_{\mathbf{q}0,i}\dot{U}'\_{\mathbf{d},i} = -U'\_{\mathbf{d},i} + I\_{\mathbf{q},i} \cdot (X\_{\mathbf{q},i} - X'\_{\mathbf{q},i}),\tag{3.17d}$$

γ˙ <sup>i</sup> = Ωm,i, (3.17e)

$$J\_i \dot{\Omega}\_{\rm m,i} + D\_{\rm d,i} \Omega\_{\rm m,i} + d\_{\rm d,i} \omega\_{\rm m,i} = M\_{\rm T,i} - M\_{\rm e,i}.\tag{3.17f}$$

**Figure 3.4:** Equivalent circuit diagram of a SM in transient state connected to the grid.

With Assumption 3.7b), it holds that Xq,i = X′ <sup>q</sup>,i, U ′ <sup>d</sup>,i = 0, and Rs,i = 0. This leads to

$$U\_{\rm gd,i} = -I\_{\rm q,i} X\_{\rm q,i}',\tag{3.18a}$$

$$U\_{\rm gq,i} = U'\_{\rm q,i} + I\_{\rm d,i} X'\_{\rm d,i},\tag{3.18b}$$

$$T'\_{\rm d0,i} U'\_{\rm q,i} = U\_{\rm f,i} - U'\_{\rm q,i} + I\_{\rm d,i} \cdot (X\_{\rm d,i} - X'\_{\rm d,i}),\tag{3.18c}$$

$$
\dot{\gamma}\_i = \Omega\_{\text{m},i},
\tag{3.18d}
$$

$$J\_i \dot{\Omega}\_{\rm m,i} + D\_{\rm d,i} \Omega\_{\rm m,i} + d\_{\rm d,i} \omega\_{\rm m,i} = M\_{\rm T,i} - M\_{\rm e,i}.\tag{3.18e}$$

The dynamics of the electrical subsystem is illustrated in the left half of Fig. 3.4.

#### **Electromechanical Coupling**

On the mechanical side, it usually holds that Dd,i ≪ dd,i [MBB12, p. 455], thus Dd,i can be neglected in (3.18e). With the above assumption and Ω˙ <sup>0</sup> <sup>m</sup>,i = 0, we get Jiω˙ <sup>m</sup>,i + dd,iωm,i = MT,i − Me,i. Due to the fact that power equals torque times angular velocity, this yields

$$J\_i \dot{\omega}\_{\rm m,i} + d\_{\rm d,i} \omega\_{\rm m,i} = \frac{1}{\Omega\_{\rm m,i}} \left( p\_{\rm T,i} - p\_{\rm e,i} \right) \,. \tag{3.19}$$

Multiplication by Ω 0 <sup>m</sup>,i and Assumption 3.6a) lead to

$$
\Omega^{0}\_{\rm m,i} J\_i \dot{\omega}\_{\rm m,i} + \Omega^{0}\_{\rm m,i} \cdot d\_{\rm d,i} \omega\_{\rm m,i} = p\_{\rm T,i} - p\_{\rm e,i}.\tag{3.20}
$$

Now we switch from mechanical to electrical domain by choosing

$$
\Omega\_i = \frac{2\Omega\_{\rm m,i}}{n\_{\rm pp}}, \qquad \qquad \qquad \Omega\_{\rm n,i} = \frac{2\Omega\_{\rm m,i}^0}{n\_{\rm pp}}, \tag{3.21}
$$

where <sup>n</sup>pp <sup>∈</sup> <sup>N</sup> denotes the number of pole pairs. The deviation of the electrical angular velocity from its nominal value is thus given by ω<sup>i</sup> = Ω<sup>i</sup> − Ωn,i = 2ωm,i npp . Accordingly, we

get ω˙ <sup>i</sup> = Ω˙ <sup>i</sup> = 2Ω˙ <sup>m</sup>,i npp = 2 ˙ωm,i npp . Expressing the mechanical velocity in (3.20) by means of the electrical velocity ω<sup>i</sup> yields thus

$$\underbrace{J\_i \cdot \frac{\Omega\_{\rm n,i} \cdot n\_{\rm pp}}{2} \cdot \frac{n\_{\rm pp}}{2} \cdot \dot{\omega}\_i + \underbrace{d\_{\rm d,i} \cdot \frac{\Omega\_{\rm n,i} \cdot n\_{\rm pp}}{2} \cdot \frac{n\_{\rm pp}}{2}}\_{=:A\_i} \cdot \omega\_i = p\_{\rm T,i} - p\_{\rm e,i},\tag{3.22}$$

which is the classical second-order *swing equation*.

#### **Connection with the Power Network**

The SM is connected to the power network as depicted in the right half of Fig. 3.4, where U⃗ i denotes the phasor of the "external" nodal voltage U<sup>i</sup> at the coupling point with the power network. The active and reactive power injection at the coupling point are given by<sup>31</sup>

$$p\_{\rm inj,i} = \frac{U\_i' U\_i}{X\_{\rm dn,i}} \sin(\vartheta\_{\rm n,i}) + \frac{U\_i^2}{2} \frac{X\_{\rm dn,i} - X\_{\rm qn,i}}{X\_{\rm dn,i} X\_{\rm qn,i}} \sin(2\vartheta\_{\rm n,i}),\tag{3.23}$$

$$q\_{\rm inj,i} = \frac{U\_i' U\_i}{X\_{\rm dn,i}} \cos(\vartheta\_{\rm n,i}) - \frac{U\_i^2}{X\_{\rm dn,i} X\_{\rm qn,i}} \left( X\_{\rm dn,i} \sin^2(\vartheta\_{\rm n,i}) + X\_{\rm qn,i} \cos^2(\vartheta\_{\rm n,i}) \right). \tag{3.24}$$

In case of non-salient pole SMs, we have Xd,i = Xq,i and thus Xdn,i = Xqn,i. Therefore, the above equations simplify to

$$p\_{\rm inj,i} = \frac{U\_i' U\_i}{X\_{\rm dn,i}} \sin(\vartheta\_{\rm n,i}),\tag{3.25}$$

$$q\_{\rm inj,i} = \frac{U\_i' U\_i}{X\_{\rm dn,i}} \cos(\vartheta\_{\rm n,i}) - \frac{U\_i^2}{X\_{\rm dn,i}}.\tag{3.26}$$

The coupling between mechanical and electric domain is given by

$$p\_{\rm c,i} = p\_{\rm inj,i}, \quad \qquad \qquad \qquad \Omega\_i = \frac{2\Omega\_{\rm m,i}}{n\_{\rm pp}}.\tag{3.27}$$

With Ωm,i = Ω<sup>0</sup> <sup>m</sup>,i <sup>+</sup> <sup>ω</sup>m,i and <sup>γ</sup>˙ <sup>i</sup> = Ωm,i, we get ˙δ<sup>i</sup> <sup>=</sup> <sup>ω</sup>m,i. With Assumption 3.6a), it holds that npp · ˙δ<sup>i</sup> = ˙θ<sup>i</sup> , thus (3.18d) is equivalent to ˙θ<sup>i</sup> = ω<sup>i</sup> . This allows to write the overall equations compactly as

$$
\dot{\theta}\_i = \omega\_i,\tag{3.28a}
$$

$$
\Gamma\_i \dot{\omega}\_i = -A\_i \omega\_i + p\_{\Gamma,i} - p\_{\text{inj},i}, \tag{3.28b}
$$

$$T\_{\rm d0,i}^{\prime} \dot{U}\_i^{\prime} = U\_{\rm f,i} - U\_i^{\prime} + (X\_{\rm d,i} - X\_{\rm d,i}^{\prime})(U\_i^{\prime})^{-1} q\_{\rm inj,i}. \tag{3.28c}$$

<sup>31</sup> A detailed derivation of this physical relationship is provided in Appendix B.1.

**Figure 3.5:** Single-phase circuit diagram of a grid-forming inverter with capacitive inertia.

#### **3.2.3 Inverter-Type Nodes**

The vast majority of DERs fed by RESs are connected to the grid via inverter interfaces. The model for I nodes derived in the following is able to both represent the main physical characteristics of the inverter dynamics while also ensuring a seamless integrability into the existing SM infrastructure.

At each <sup>I</sup> node, we consider a *grid-forming*<sup>32</sup> inverter [RLBR12] with capacitive inertia and controllable DC current IDC,i. We assume that each inverter i ∈ V<sup>I</sup> fulfills the following assumptions regarding its inner control loop:

**Assumption 3.8.** *a) At AC side, the inverter is able to produce a perfect sinusoidal AC voltage* U⃗ i *.*


The validity of these assumptions is ensured as follows: To comply with Assumption 3.8a), the AC current can be processed through an appropriate LCL low-pass filter to suppress the harmonics caused by switching dynamics of the involved insulated gate bipolar transistors. Assumption 3.8b) is justified by the fact that the losses caused by the DC-side conductance GDC,i are significantly larger. The UDC ∼ ω matching controller that was established in Assumption 3.8c) allows to exploit the structural similarities between kinetic energy of the rotor of a SM and electric energy stored in the DC-side capacitor of the inverter.

A key difference to alternative concepts of mimicking SM dynamics, such as the virtual synchronous machine concept, is that the UDC ∼ ω matching controller directly uses the physically available electric capacitance CDC,i instead of relying on a separate storage and/or a microprocessor-based external emulation of SM dynamics which may be more susceptible to delays. By contrast, using the UDC ∼ ω matching controller as inner control loop for frequency and voltage control purposes has been explicitly promoted in the recent paper [DBSPG19].

<sup>32</sup> Inverters are said to be *grid-forming* if they operate as controllable voltage source. By contrast, the inverter is said to be in *grid-following* mode, if it acts as an (active and reactive) power source while U<sup>i</sup> and θ<sup>i</sup> are the dependent variables [SFO15, p. 69f.].

With Assumption 3.8a), the inverter can be represented by its single-phase equivalent circuit which is depicted in Fig. 3.5. The constitutive equation for the DC-side voltage UDC,i is

$$C\_{\rm DC,i} \dot{U}\_{\rm DC,i} = -\frac{1}{R\_{\rm DC,i}} U\_{\rm DC,i} - I\_{\rm x,i} + I\_{\rm DC,i}.\tag{3.29}$$

In grid-forming mode, the DC-side voltage UDC,i serves as independent variable which can be chosen freely. With Assumption 3.8c), the phase angle θ<sup>i</sup> of U⃗ i is chosen according to the integral controller scheme [JAD16, MDS<sup>+</sup>18] by

$$
\dot{\theta}\_i = k\_i U\_{\text{DC},i} \tag{3.30}
$$

with k<sup>i</sup> = Ωn,i/U<sup>⋆</sup> DC,i, where U ⋆ DC,i denotes the nominal DC link voltage. With (3.30), the DC side follows the dynamics

$$\frac{C\_{\rm DC,i}}{k\_i} \cdot \Omega\_i + \frac{1}{k\_i R\_{\rm DC,i}} \cdot \Omega\_i = -I\_{\rm x,i} + I\_{\rm DC,i}.\tag{3.31}$$

With Assumption 3.8b), it holds that Ix,iUx,i = Ix,iUDC,i = pDC,i = pinj,i. Furthermore, we choose

$$I\_{\rm DC,i} = \frac{\Omega\_{\rm n,i}}{R\_{\rm DC,i} \cdot k\_i} + \frac{p\_{\rm g,i}}{\Omega\_i},\tag{3.32}$$

where pg,i is a desired active power input or output. Since Ω˙ <sup>i</sup> = ˙ω<sup>i</sup> , (3.31) reads as

$$\frac{C\_{\rm DC,i}}{k\_i^2} \cdot \dot{\omega}\_i + \frac{1}{k\_i^2 \cdot R\_{\rm DC,i}} \cdot \omega\_i = \frac{1}{\Omega\_i} \left(p\_{\rm g,i} - p\_{\rm inj,i}\right) \,. \tag{3.33}$$

Multiplication with Ωn,i along with Assumption 3.6a) leads to

$$\underbrace{C\_{\rm DC}(U\_{\rm DC}^{\star})^2}\_{=:\Gamma\varepsilon\_{i}i} \cdot \dot{\omega}\_{i} + \underbrace{(U\_{\rm DC,i}^{\star})^2}\_{=:A\varepsilon\_{i}} \cdot \omega\_{i} = p\_{\rm g,i} - p\_{\rm inj,i},\tag{3.34}$$

where we can define ΓI,i := CDC,i(U ⋆ DC,i) 2 Ωn,i as the *virtual inertia* and AI,i := (U ⋆ DC,i) 2 Ωn,iRDC,i as the *virtual damping coefficient*. In summary, this yields the inverter AC side dynamics

$$
\dot{\theta}\_i = \omega\_i,\tag{3.35a}
$$

$$
\Gamma\_{\mathcal{Z},i}\dot{\omega}\_i = -A\_{\mathcal{Z},i}\omega\_i + p\_{\text{g},i} - p\_{\text{inj},i}.\tag{3.35b}
$$

In particular, we clearly see the structural equivalence between (3.35) and the swing dynamics (3.28). However, both the resulting (virtual) inertia ΓI,i and (virtual) damping AI,i are typically much smaller in magnitude than Γ<sup>i</sup> and A<sup>i</sup> , respectively (see [JAD16, Remark 2] and [MDS<sup>+</sup>18, Remark 2]). As a result, a higher damping would necessitate a larger parallel conductance GDC,i and thus provoke a higher power loss on the DC side. Likewise, a larger virtual inertia ΓI,i requires a larger capacitance CDC,i and/or a larger nominal DC voltage U ⋆ DC,i.

#### **3.2.4 Load-Type Nodes**

Load-Type nodes represent the power supply or demand of retail producers or consumers. These network participants are connected via step-down transformers or grid-feeding inverters, which are each modeled by a constant impedance ⃗yℓ,i = gℓ,i+ȷbℓ,i plus an uncontrollable active and reactive power demand<sup>33</sup> along with a frequency-dependent part with an unknown damping coefficient AL,i ≥ 0. Accordingly, we get

$$
\dot{\theta}\_i = \omega\_i,\tag{3.36}
$$

$$p\_{\rm inj,i} = -g\_{\ell,i}U\_i^2 - A\_{\mathcal{L},i}\omega\_i - p\_{\ell,i},\tag{3.37}$$

$$q\_{\rm inj,i} = -b\_{\ell,i} U\_i^2 - q\_{\ell,i}.\tag{3.38}$$

**Remark 3.9.** *Distributor nodes with* pℓ,i = qℓ,i = AL,i = 0 *can be eliminated from the model by Kron reduction [CT14; SFO15, p. 45ff.].* ♢

Despite the sign convention for L nodes, pℓ,i and qℓ,i may be negative as well.

#### **3.2.5 Overall Model**

In the following, the dynamic equations of S, I, and L nodes are combined to a novel, networkpreserving overall model. For all S and L nodes, we choose p<sup>i</sup> = pinj,i and q<sup>i</sup> = qinj,i. I nodes are connected via p<sup>i</sup> = pinj,i and UAC,i = U<sup>i</sup> . Moreover, we assume that all shunt admittances ⃗ysh,i of L nodes are incorporated in the diagonal elements of the nodal admittance matrix Y⃗ p.

Now we define the *extended* physical graph G = (V, E) with V = V<sup>p</sup> ∪ V′ S and E = E<sup>p</sup> ∪ E′ p , where V ′ S represents the SMs' internal voltages U ′ i , and E ′ p represents the SMs' connections between internal voltage U ′ i and external voltage U<sup>i</sup> . For convenience of notation, Xdn,i is included in the line impedance. Then, we can define the extended nodal admittance matrix Y⃗ = G + ȷB incorporating both the admittances of power lines and the reactances of SMs.

<sup>33</sup> According to the node type, we use the passive sign convention.

This finally leads to the compact notation

$$\begin{aligned} \dot{\vartheta}\_{ij} &= \omega\_i - \omega\_j, & i &\quad ij \in \mathcal{E}, \quad \text{(3.39a)}\\ \dot{L}\_i &= -A\_i \omega\_i + p\_{\text{g},i} - p\_{\ell,i} - U\_i^2 G\_{ii} \end{aligned}$$

$$-\sum\_{j \in \mathcal{N}\_i} U\_i U\_j \left( B\_{ij} \sin(\vartheta\_{ij}) + G\_{ij} \cos(\vartheta\_{ij}) \right), \qquad \qquad i \in \mathcal{V}\_\mathbf{g}, \tag{3.39b}$$

$$\begin{aligned} T\_{\mathrm{d0},i}^{\prime} \dot{U}\_i &= U\_{\mathrm{f},i} - U\_i - (X\_{\mathrm{d},i} - X\_{\mathrm{d},i}^{\prime}) \\ &\cdot \left( -U\_i B\_{ii} + \sum\_{j \in \mathcal{N}\_i} U\_j \left( G\_{ij} \sin(\vartheta\_{ij}) - B\_{ij} \cos(\vartheta\_{ij}) \right) \right), \qquad i \in \mathcal{V}\_{\mathcal{S}}^{\prime}, \end{aligned} \tag{3.39c}$$

$$\begin{aligned} 0 &= -A\_i \omega\_i - p\_{\ell,i} - U\_i^2 G\_{ii} \\ &- \sum\_{j \in \mathcal{N}\_i} U\_i U\_j \left( B\_{ij} \sin(\vartheta\_{ij}) + G\_{ij} \cos(\vartheta\_{ij}) \right), & i \in \mathcal{V}\_{\ell}, \end{aligned} \tag{3.39d}$$

$$0 = -q\_{\ell,i} + U\_i^2 B\_{ii} - \sum\_{j \in \mathcal{N}\_i} U\_i U\_j \left( G\_{ij} \sin(\vartheta\_{ij}) - B\_{ij} \cos(\vartheta\_{ij}) \right), \quad i \in \mathcal{V}\_\ell,\tag{3.39e}$$

where V<sup>g</sup> = V<sup>S</sup> ∪ VI, V<sup>ℓ</sup> = V<sup>L</sup> ∪ V<sup>S</sup> , L<sup>i</sup> = Γiω<sup>i</sup> , <sup>G</sup>ij <sup>=</sup> ℜ{[Y⃗ ]i,j}, <sup>B</sup>ij <sup>=</sup> ℑ{[Y⃗ ]i,j}, and N<sup>i</sup> denoting the set of neighboring nodes j ∈ V of the extended graph G without regard to edge directions.

By defining the *plant state vector* x<sup>p</sup> = col{ϑ, L<sup>S</sup> , LI, U<sup>S</sup> , ωℓ, Uℓ} of (3.39), where for all m ∈ E, j ∈ V′ S , k ∈ VI, l ∈ Vℓ:

$$\vartheta := \mathrm{col}\_{m} \{ \vartheta\_{m} \}, \qquad \qquad \qquad \mathbf{L}\_{\mathcal{S}} := \mathrm{col}\_{j} \{ L\_{j} \}, \tag{3.40}$$

$$L\_{\mathcal{T}} := \text{col}\_k\{L\_k\}, \qquad \qquad \qquad \mathcal{U}\_{\mathcal{S}} := \text{col}\_j\{U\_j\}, \tag{3.41}$$

$$
\omega\_{\ell} := \text{col}\{\omega\_{l}\}, \qquad \qquad \qquad \mathcal{U}\_{\ell} = \text{col}\_{l}\{U\_{l}\}, \tag{3.42}
$$

we can set up the *plant Hamiltonian* as

$$\begin{split} H\_{\mathbf{p}}(\mathbf{x}\_{\mathbf{p}}) &= \frac{1}{2} \sum\_{i \in \mathcal{V}\_{\mathcal{S}}'} \left( \Gamma\_{i}^{-1} L\_{i}^{2} + \frac{U\_{i}^{2}}{X\_{d,i} - X\_{d,i}'} \right) \\ &+ \frac{1}{2} \sum\_{i \in \mathcal{V}\_{\mathcal{E}}} \Gamma\_{i}^{-1} L\_{i}^{2} \\ &- \frac{1}{2} \sum\_{i \in \mathcal{V}} B\_{ii} U\_{i}^{2} - \sum\_{(i,j) \in \mathcal{E}} B\_{ij} U\_{i} U\_{j} \cos(\vartheta\_{ij}) \\ &+ \frac{1}{2} \sum\_{i \in \mathcal{V}\_{\mathcal{E}}} \omega\_{i}^{2} \end{split} \tag{3.43}$$

to describe the total energy stored in the system. The first row of (3.43) represents the shifted kinetic energy of the rotors and the magnetic energy of the SM circuits, the second row represents the "virtual" kinetic energy at inverter-type nodes, the third row represents the magnetic energy of transmission lines and the fourth row represents the local deviations of load nodes from nominal frequency.

Using the plant Hamiltonian (3.43) along with its *co-state* z<sup>p</sup> = ∇H(xp) and V<sup>g</sup> = V<sup>S</sup> ∪ VI, with

$$\frac{\partial H\_{\rm p}}{\partial \vartheta\_{ij}} = B\_{ij} U\_i U\_j \sin(\vartheta\_{ij}), \tag{3.44}$$

$$\frac{\partial H\_{\rm p}}{\partial L\_{i}} = L\_{i}/\Gamma\_{i} = \omega\_{i}, \quad \quad \qquad \qquad i \in \mathcal{V}\_{\rm g}, \tag{3.45}$$

$$\frac{\partial H\_{\rm p}}{\partial U\_{i}} = \frac{U\_{i}}{X\_{d,i} - X\_{d,i}'} - B\_{ii}U\_{i} - \sum\_{j \in \mathcal{N}\_{i}} B\_{ij}U\_{j} \cos(\vartheta\_{ij}), \qquad i \in \mathcal{V}\_{\rm S}', \tag{3.46}$$

$$\frac{\partial H\_{\rm p}}{\partial U\_{i}} = -\sum\_{j \in \mathcal{N}\_{i}} B\_{ij} U\_{j} \cos(\vartheta\_{ij}), \quad \qquad \qquad i \in \mathcal{V}\_{\ell}, \tag{3.47}$$

$$\frac{\partial H\_{\rm p}}{\partial \omega\_{i}} = \omega\_{i}, \quad \quad \quad \quad \quad \quad \quad \quad \quad i \in \mathcal{V}\_{\ell}, \tag{3.48}$$

we can set up a port-Hamiltonian representation for the novel physical plant model (3.39) as

 ϑ˙ L˙ U˙ S **0 0** | {z } x˙ <sup>p</sup> = 0 D<sup>T</sup> pg 0 D<sup>T</sup> pℓ 0 −Dpg 0 0 0 0 0 0 0 0 0 −Dpℓ 0 0 0 0 0 0 0 0 0 | {z } J<sup>p</sup> − 0 0 0 0 0 0 A<sup>g</sup> 0 0 0 0 0 R<sup>S</sup> 0 0 0 0 0 A<sup>ℓ</sup> 0 0 0 0 0 Uˆ ℓ | {z } R<sup>p</sup> zp − **0** ρg ϱS ρℓ ϱℓ | {z } r<sup>p</sup> + 0 0 0 0 I 0 0 −I ˆ g 0 Tˆ <sup>U</sup> 0 0 0 0 0 −I ˆ ℓ 0 0 −I 0 pg U<sup>f</sup> qℓ pℓ , (3.49)

where

L = col{L<sup>S</sup> , LI}, (3.50) A<sup>g</sup> = diagi{Ai}, i ∈ Vg, (3.51) A<sup>ℓ</sup> = diagi{Ai}, i ∈ Vℓ, (3.52) R<sup>S</sup> = diag<sup>i</sup> {(Xdi − X′ di) /τU,i} , i ∈ V′ S , (3.53) Uˆ <sup>ℓ</sup> = diagi{Ui}, i ∈ Vℓ, (3.54) ρ<sup>g</sup> = col<sup>i</sup> GiiU 2 <sup>i</sup> + X j∈N<sup>i</sup> GijUiU<sup>j</sup> cos(ϑij ) , i ∈ Vg, (3.55) ρ<sup>ℓ</sup> = col<sup>i</sup> GiiU 2 <sup>i</sup> + X j∈N<sup>i</sup> GijUiU<sup>j</sup> cos(ϑij ) , i ∈ Vℓ, (3.56) ϱ<sup>S</sup> = col<sup>i</sup> Rg,i X GijUiU<sup>j</sup> sin(ϑij ) , i ∈ V′ S , (3.57)

$$\mathfrak{g}\_{\ell} = \text{col}\_{i} \{ \sum\_{j \in \mathcal{N}\_{i}} G\_{ij} U\_{i} U\_{j} \sin(\vartheta\_{ij}) \}, \qquad \qquad i \in \mathcal{V}\_{\ell}, \tag{3.58}$$

$$
\hat{T}\_U = \text{diag}\_i\{1/T\_{\text{d0},i}\}, \tag{3.59}
\\
\qquad \qquad i \in \mathcal{V}'\_S, \tag{3.59}
$$

$$
\hat{\mathbf{I}}\_{\mathbf{g}} = \begin{bmatrix} \mathbf{I}\_{n\_{\mathbf{g}} \times n\_{\mathbf{g}}} & \mathbf{0}\_{n\_{\mathbf{g}} \times n\_{\ell}} \end{bmatrix}, \tag{3.60}
$$

$$
\hat{\mathbf{I}}\_{\ell} = \begin{bmatrix} \mathbf{0}\_{n\_{\ell} \times n\_{\mathbf{g}}} & \mathbf{I}\_{n\_{\ell} \times n\_{\ell}} \end{bmatrix}. \tag{3.61}
$$

The input vector in (3.49) is composed of the control input u<sup>p</sup> = col{p<sup>g</sup> , Uf} and the disturbance input d = col{q<sup>ℓ</sup> , pℓ}.

**Remark 3.10.** *Note that* (3.49) *is a nonlinear system, whereby the main nonlinearities appear in the co-state equations* (3.44)*,* (3.46)*, and* (3.47)*, caused by nonlinear electromagnetic relations. Moreover, there is a modulation of the dissipation matrix* R<sup>p</sup> *with the state vector* Uℓ*. The interconnection matrix* Jp*, however, contains only entries which are constant in time.* ♢

**Remark 3.11.** *Even in the case that certain parts of Assumption 3.7 do no longer apply, the port-Hamiltonian framework can principially be applied to formulate a corresponding (higherorder) system model for* S *nodes. In [SDv16], a port-Hamiltonian model for multi-machine power networks composed of* sixth-order *SM models* (3.14) *is established. A systematic derivation of port-Hamiltonian representations for* all *classical SG models based on first-principles modeling is conducted in [SDv19].* ♢

## **3.3 Modeling of Participants' Interaction**

This section presents a model for the network participants' interaction and competition on the energy market. The conceptual classification into WPs and RPs is based on the premise that WPs can shift their own production or consumption strategically to some extent<sup>34</sup> based on a profit-maximizing strategy. Thus, WPs are competing players on a cross-zone electricity market which pursue to maximize their own profits Pg,i through (positive or negative) power injections pg,i, while incurring power production costs Cg,i.

**Assumption 3.12.** *The cost functions* Cg,i(pg,i) *are strictly convex.*

**Remark 3.13.** *Assumption 3.12 is typically mild for both consumers and producers [BV00, ZP15]. This is due to the fact that WPs tend to internally work with a strictly convex surrogate function in order to apply their pursuit of profit maximization and to avoid being stuck in sub-optimal operating points (see the discussion in Appendix B.3)* ♢

By contrast, RPs typically perform their decision-making concerning power production or consumption independently of other RPs or WPs and have no means to displace their instantaneous (positive or negative) demand. Thus, RPs are assumed to be inelastic.

**Remark 3.14.** *The role of WPs is congruent with the definition of CVPPs (cf. Subsection 2.2.1), although the formalism presented in the following also allows negative power injections. Moreover, the duties and responsibilities of CCs as well as their geographic coverage strongly resemble those of a TVPP. However, the latter terminology is avoided in the context of this dissertation, since the attribute "virtual" negates the active role of CCs in the mutual coordination of supply and demand under consideration of physical network conditions. Furthermore, our concept of WPs does not require unidirectional power flows as might be suggested by the term "power plant".* ♢

We summarize the number of WP nodes as n<sup>g</sup> = n<sup>S</sup> + nI. WPs inject electrical power <sup>p</sup><sup>g</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup><sup>g</sup> into the power system and are able to influence the nodal voltages <sup>U</sup> <sup>∈</sup> <sup>R</sup> n<sup>g</sup> >0 . Both S and I nodes have to meet upper and lower limits on power generation pg,i and p <sup>g</sup>,i. Note that all buses i ∈ V may be equipped with RPs and thus with an uncontrollable active and reactive power demand pℓ,i and qℓ,i.

For compactness of notation, we introduce cell-aggregated variables for the power generation <sup>p</sup>ˆ<sup>g</sup> = col<sup>k</sup>∈Z {pˆg,k} ∈ <sup>R</sup><sup>n</sup><sup>Z</sup> where <sup>p</sup>ˆg,k <sup>=</sup> P i∈VZ,k pg,i, the power consumption <sup>p</sup>ˆ<sup>ℓ</sup> = col<sup>k</sup>∈Z {pˆℓ,k} ∈ <sup>R</sup><sup>n</sup><sup>Z</sup> where <sup>p</sup>ˆℓ,k <sup>=</sup> P i∈VZ,k pℓ,i, and the active power loss Φˆ = col<sup>k</sup>∈Z {Φˆ <sup>k</sup>} ∈ <sup>R</sup><sup>n</sup><sup>Z</sup> , where Φˆ <sup>k</sup> = P i∈VZ,k Φi .

The following subsections pave the way towards a market-based control scheme with zonal (i.e. spatially differentiated) prices <sup>λ</sup><sup>k</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup><sup>Z</sup> . Subsection 3.3.1 derives an incentive-compatible market mechanism which respects the requirement of unbundling. Subsections 3.3.2–3.3.4 model the behavior of WPs, RPs, and CCs, respectively, under the chosen market mechanism. Finally, Subsection 3.3.5 summarizes the consequences of market interactions on the upcoming controller design.

<sup>34</sup> Hence, spatially differentiated prices also provide medium- and long-term incentives for shifting capacity to other cells that, on average, provide higher prices for generators or lower cost fro consumers.

#### **3.3.1 Design of Market Mechanism**

In the classical model of energy economics, a *market equilibrium* of the competitive power system from Definition 3.3 is attained if the market price λ resulting from competition yields a balance between total demand **1** <sup>T</sup>p<sup>ℓ</sup> and total generation **1** <sup>T</sup>p<sup>g</sup> (see the common Definition A.23 in Appendix A.6). However, this definition makes no statement whether the achieved market equilibrium is "valuable" from the invididual participants' point of view or from a global perspective. The *individual* utility of an equilibrium is often certified by means of the Pareto efficiency (or Pareto optimality), cf. Definition A.5. According to the first theorem of welfare economics (cf. Theorem A.25), each market equilibrium is also Pareto efficient if the participants compete over a *perfect market*. This prototypical notion of an idealized market contains numerous simplifying assumptions and is summarized in Definition A.24.

For the normative evaluation of the *global* efficiency of an attained equilibrium solution, there exist different metrics such as the *utilitarian welfare function*<sup>35</sup> or the *Rawlsian welfare function* (cf. Appendix A.6). When adopting the utilitarian welfare function as a measure for global efficiency, the market equilibrium achieved in a perfect market is not only individually, but also globally efficient (cf. Remark A.27). It can thus be stated that such a perfect market should always be targeted by means of any appropriate market design. However, certain properties of real-world power systems are diametrically opposed to those required in Definition A.24:


These points emphasize that, when dealing with actual electricity markets under realistic operating conditions, a mere reference to the existing theorems of welfare economics is insufficient. In the subsequent sections each of the above challenges is therefore intensively studied when deducing a viable (i.e. network-preserving) market-based control scheme.

In the remainder of this subsection, we first address the issue of inelastic market participants (challenge (1) above). As outlined in Subsection 2.2.2, the two main philosophies for modeling distributed market-based interaction of noncooperative agents are Bertrand and Cournot competition (see Subsection 2.2.2 for a discussion of current research approaches related to

<sup>35</sup> The maximum of the utilitarian welfare function is called *social optimum* and is characterized by equal marginal profits of each participant.

electricity markets and [dD21, Chapter 1] for a more general economic framing). We thus investigate whether these two models of competition are able to generate (individually and/or globally) efficient market equilibria in case of inflexible RPs. For didactical reasons (yet without loss of generality), we exemplify these considerations on an atomistic model by requiring pg,i > 0 and pℓ,i > 0, and by temporarily neglecting both the system dynamics (challenge (2)) and the line losses (challenge (3)). Both challenges are then considered again from Subsection 3.3.2 onwards.

#### **Bertrand Competition with Inelastic Demand**

In Bertrand competition each WP <sup>i</sup> ∈ W bids a price <sup>λ</sup><sup>i</sup> <sup>∈</sup> <sup>R</sup> per amount of power to maximize its own profit

$$\mathbf{P}\_{\mathbf{g},i}(\lambda\_i, \lambda\_{\neg i}) = \lambda\_i p\_{\mathbf{g},i}(\lambda\_i, \lambda\_{\neg i}) - \mathbf{C}\_{\mathbf{g},i}(p\_{\mathbf{g},i}(\lambda\_i, \lambda\_{\neg i})),\tag{3.62}$$

where <sup>C</sup>g,i : <sup>R</sup> <sup>→</sup> <sup>R</sup>≥<sup>0</sup> represents the costs of WP <sup>i</sup> to generate <sup>p</sup>g,i. The power generation invoked by the system operator is given by

$$p\_{\mathbf{g},i}(\lambda\_i, \lambda\_{\neg i}) = \begin{cases} 0, & \exists j \in \mathcal{W} \backslash \{i\}: \lambda\_j < \lambda\_i, \\ \frac{\mathbf{1}^{\mathsf{T}} \mathbf{p}\_\ell}{N}, & \text{otherwise}, \end{cases},\tag{3.63}$$

where N is the number of WPs that bid the lowest price λ = mini{λi}. By (3.63), WP i is not able to sell any power if there exist other WPs that bid a lower price. Likewise, if WP i bids the lowest price λ<sup>i</sup> = λ, then the total demand 1 <sup>T</sup>p<sup>ℓ</sup> is equally shared among all WPs that bid the lowest price λ.

Bertrand competition has many undesirable properties. In [Das95], it is shown that even for strictly convex cost functions Cg,i(pg,i) (cf. Assumption 3.12), Bertrand competition may exhibit non-unique market equilibria. In [Wei06], it is elaborated that the range of acceptable prices is determined by λ<sup>i</sup> = [λ ◦ i , λ• i ], where the *outside option price*

$$\lambda\_i^\diamond = \frac{\mathbf{c}\_{\mathbf{g},i} \left( \frac{\mathbf{1}^\mathsf{T} \mathbf{p}\_\ell}{N} \right)}{\frac{\mathbf{1}^\mathsf{T} \mathbf{p}\_\ell}{N}} \tag{3.64}$$

is the price below which player i earns a negative profit when sharing the market, and the *competitive price limit* λ • i defined by

$$\mathbf{P}\_{\mathbf{g},i} \left( \frac{\mathbb{1}^{\mathsf{T}} \mathbf{p}\_{\ell}}{N}, \lambda\_{i}^{\bullet} \right) = \mathbf{P}\_{\mathbf{g},i} (\mathbb{1}^{\mathsf{T}} \mathbf{p}\_{\ell}, \lambda\_{i}^{\bullet}). \tag{3.65}$$

Bids λ ′ <sup>i</sup> > λ• <sup>i</sup> would always attract the other players to bid λ¬<sup>i</sup> = λ ′ <sup>i</sup> − ε, ε > 0, such that they share the total market exclusively among each other. Hence, the intersection of all those intervals constitutes the set of all possible Nash equilibria (cf. [Wei06])

$$\mathbb{L}\_{\text{nash}} = \{ \lambda : \lambda\_{\text{max}}^{\diamond} \le \lambda \le \lambda\_{\text{min}}^{\bullet} \}, \tag{3.66}$$

where λ ◦ max = max<sup>i</sup>∈<sup>P</sup>{λ ◦ i } and λ • min = min<sup>i</sup>∈<sup>P</sup>{λ • i }. This is demonstrated by the following two-player example.

#### **Example 1**

*Consider two WPs competing in a Bertrand competition, whose individual cost functions are given by* Cg,1(pg,1) = 4p 2 <sup>g</sup>,<sup>1</sup> −pg,<sup>1</sup> *and* Cg,2(pg,2) = 4p 2 g,2 *. This results in the profit functions*

$$\mathcal{P}\_{\mathbf{g},1}(\lambda\_1, \lambda\_2) = \lambda\_1 p\_{\mathbf{g},1}(\lambda\_1, \lambda\_2) - 4p\_{\mathbf{g},1}^2(\lambda\_1, \lambda\_2) + p\_{\mathbf{g},1}(\lambda\_1, \lambda\_2),\tag{3.67}$$

$$\mathcal{P}\_{\mathbf{g},2}(\lambda\_2, \lambda\_1) = \lambda\_2 p\_{\mathbf{g},2}(\lambda\_2, \lambda\_1) - 4p\_{\mathbf{g},2}^2(\lambda\_2, \lambda\_1). \tag{3.68}$$

*Let the total demand be* 1 <sup>T</sup>p<sup>ℓ</sup> = 1*, then* (3.63) *takes the form*

$$p\_{\mathbf{g},1}(\lambda\_1, \lambda\_2) = \begin{cases} 1, & \lambda\_1 < \lambda\_2, \\ 0.5, & \lambda\_1 = \lambda\_2, \\ 0, & \lambda\_1 > \lambda\_2, \end{cases} \qquad p\_{\mathbf{g},2}(\lambda\_2, \lambda\_1) = \begin{cases} 1, & \lambda\_2 < \lambda\_1, \\ 0.5, & \lambda\_2 = \lambda\_1, \\ 0, & \lambda\_2 > \lambda\_1, \end{cases} \tag{3.69}$$

*and inserting* (3.69) *into* (3.68) *yields*

$$\mathbf{P}\_{\mathbf{g},1}(\lambda\_1, \lambda\_2) = \begin{cases} \lambda\_1 - 3, & \lambda\_1 < \lambda\_2, \\ 0.5\lambda\_1 - 0.5, & \lambda\_1 = \lambda\_2, \\ 0, & \lambda\_1 > \lambda\_2, \end{cases}, \quad \mathbf{P}\_{\mathbf{g},2}(\lambda\_2, \lambda\_1) = \begin{cases} \lambda\_2 - 4, & \lambda\_2 < \lambda\_1, \\ 0.5\lambda\_2 - 1, & \lambda\_2 = \lambda\_1, \\ 0, & \lambda\_2 > \lambda\_1. \end{cases} \tag{3.70}$$

*The profit functions* (3.70) *are depicted in Fig. 3.6.*

**Figure 3.6:** Exemplary profit functions for WPs 1 (blue) and 2 (red).

*The blue lines correspond to WP 1 and the red lines correspond to WP 2. The solid lines show the WPs' profit if they were to capture the full market (*pg,i = 1*), while the dashed lines show their profit if they were to share the market (*pg,i = 0.5*). Possible market equilibria can now be determined by construction: From Fig. 3.6 it can be seen that WP 2 will never choose* λ<sup>2</sup> < 2 *as this would not result in a positive profit. Moreover, since for all* λ<sup>1</sup> ∈ [2, 5] *it is more profitable for WP 2 to share the market than to capture the full market or to capture nothing, the best option for WP 2 is to choose* λ<sup>2</sup> = λ1*. The same reasoning holds if WP 1 were to choose its price* λ<sup>1</sup> *freely given* λ<sup>2</sup> ∈ [2, 5]*. Consequently, all* λ ∈ [2, 5] *are market equilibria. By contrast, if WP 2 were to choose* λ<sup>2</sup> > 5*, WP 1 could make a higher profit by choosing* λ<sup>1</sup> < λ<sup>2</sup> *and capturing the full market, which is why* λ > 5 *cannot be a market equilibrium. The outside option prices (zeros of the dashed lines in Fig. 3.6) are* λ ◦ <sup>1</sup> = 1 *and* λ ◦ <sup>2</sup> = 2*, and the competitive price limits (intersection of dashed and solid lines) are* λ • <sup>1</sup> = 5 *and* λ • <sup>2</sup> = 6*. The intersection of the intervals* [λ ◦ 1 , λ• 1 ] *and* [λ ◦ 2 , λ• 2 ] *is thus equal to* [2, 5]*.*

**Remark 3.15.** *If all players are aware of each other (i.e., know the number of all competitors as well as well as their individual cost functions), it is the best choice for them to choose* λ<sup>i</sup> = λ • min*. If the players are not aware of each other, then* any *equilibrium* λ<sup>i</sup> = [λ ◦ max, λ• min] *is stable. This can be justified as follows: Let* λ <sup>♯</sup> <sup>∈</sup> [<sup>λ</sup> ◦ max, λ• min] *be some equilibrium. If player* i *performs a so-called* ε-undercutting *[Cab17, p. 105] by choosing* λ ♡ <sup>i</sup> = λ ♯ <sup>i</sup> − ε *to test if it can share the total market, this will result in lower profits* Pg,i(λ ♡ 1 , λ<sup>♯</sup> 2 ) < Pg,i(λ ⋆ 1 , λ<sup>♯</sup> 2 ) *<sup>36</sup>, thus no player has an incentive to refrain from* λ<sup>i</sup> = λ ♯ i *and hence* λ ♯ *is a Nash equilibrium.* ♢

The fact that there exists no unique market equilibrium is undesirable, as it causes the actions of market participants to be unpredictable and path-dependent. Consequently, it becomes difficult to incorporate tangible higher-level objectives into the market mechanism. Another issue with Bertrand competition is that the resulting equilibria may not be (individually) economically optimal. We show this by means of a second example.

**Example 2** *Consider two WPs with cost functions*

$$\mathfrak{C}\_{\mathbf{g},1}(p\_{\mathbf{g},1}) = p\_{\mathbf{g},1}^2, \qquad \qquad \qquad \mathfrak{C}\_{\mathbf{g},2}(p\_{\mathbf{g},2}) = \frac{1}{3} p\_{\mathbf{g},2}^2,\tag{3.71}$$

*which compete over a total demand* 1 <sup>T</sup>p<sup>ℓ</sup> = 10*. Following the same procedure as in Example 1 above, it can be shown that* λ<sup>1</sup> = λ<sup>2</sup> = 5 *is the unique market equilibrium. Both WPs generate* p ⋆ <sup>g</sup>,<sup>1</sup> = p ⋆ <sup>g</sup>,<sup>2</sup> = 5 *and their corresponding profits are* Pg,1(λ ⋆ 1 , λ<sup>⋆</sup> 2 ) = 0 *and* Pg,2(λ ⋆ 2 , λ<sup>⋆</sup> 1 ) = 16.33*. However, there exists an infinite amount of alternative equilibria which are Pareto dominating (both in terms of higher profits and lower costs for WPs). For example, if the generation were distributed* p ♯ <sup>g</sup>,<sup>2</sup> = 2.5 *and* p ♯ <sup>g</sup>,<sup>2</sup> = 7.5*, the individual profits were* P ♯ g,1 (λ ⋆ 1 , λ<sup>⋆</sup> 2 ) = 6.25 *and* P ♯ g,2 (λ ⋆ 1 , λ<sup>⋆</sup> 2 ) = 18.75*.*

Through the equal allocation of power generation<sup>37</sup> as in (3.63), Bertrand competition may cause large market barriers: If a new WP enters the market with n<sup>W</sup> WPs already competing in a Bertrand competition with equilibrium price λ ⋆ , this new WP now has to bid λ ⋆ and to generate 1 <sup>T</sup>pℓ/(N + 1) (which might be very large<sup>38</sup>), in order to still obtain positive profits. This requires WPs which want to enter the market to make large investments beforehand and therefore prevents many small-scale WPs such as RESs to participate in the energy market, thereby creating wrong incentives that are considered anti-competitive.

<sup>36</sup> since, figuratively speaking, the solid profit curve is below the dashed profit curve (cf. Fig. 3.6).

<sup>37</sup> Some recent approaches, e.g. [CC20], overcome the problems of a pure Bertrand competition by introducing a second step after the price bidding where the power generation p<sup>g</sup> is divided *unequally* according to the individual preferences of WPs. In particular, WPs disclose how much power they would prefer to generate given the equilibrium price λ. Afterwards, the total demand 1 <sup>T</sup>p<sup>ℓ</sup> is then distributed accordingly. This modification however has the main disadvantage that WPs have to disclose their preferences towards a centralized system operator, which is a contradiction to the overarching concept of a competitive power system with cell-based infrastructure which was adopted in Definition 3.3.

<sup>38</sup> If there is no WP that can meet the total demand at its marginal costs, it can be shown that there exists no equilibrium (*Edgeworth paradox*, cf. [KO93, p. 328]).

#### **Cournot Competition with Inelastic Demand**

In Cournot competition, each WP i ∈ W sets its power generation pg,i to maximize its own profit

$$\mathbf{P}\_{\mathbf{g},i}(p\_{\mathbf{g},i}, p\_{\mathbf{g},\neg i}) = \lambda (\mathbf{1}^{\mathsf{T}} \mathbf{p}\_{\mathbf{g}}) p\_{\mathbf{g},i} - \mathfrak{C}\_{\mathbf{g},i}(p\_{\mathbf{g},i}).\tag{3.72}$$

In contrast to Bertrand competition, where the power generation of each WP i ∈ W is a function of the price bids λ, in Cournot competition the unique market price λ is set by the system operator as a function of the total power generation, while each WP chooses its own power generation pg,i by itself.

For the classic Cournot competition, it is required that all consumers are *price-elastic* in order to arrive at a meaningful market equilibrium and deduce the price function λ(1 <sup>T</sup>p<sup>g</sup> ) (see e.g. [DM19]). However, as discussed at the beginning of Section 3.3, we choose to model consumers as price-inelastic. In a classic Cournot competition, this would trigger all WPs to tactically choose their pg,i such that **1** <sup>T</sup>p<sup>g</sup> < **1** <sup>T</sup>p<sup>ℓ</sup> in order to maximize profits. As a result, the prices would eventually grow infinitely large (yielding infinite profits for all WPs) and thus no market equilibrium could be reached at all. However, in order to still achieve a market equilibrium in a Cournot competition with price-inelastic consumers, we make the following assumption:

**Assumption 3.16.** *WPs do not know the total demand* 1 <sup>T</sup>p<sup>ℓ</sup> *nor how the market price* λ *depends on the power injections* p<sup>g</sup> *.*

Assumption 3.16 is met if strict observance of information barriers between WPs and CCs can be ensured (cf. [LPWL07, p. 24ff.]). As a result of Assumption 3.16, WPs act as pure *price-takers* as they are unaware of their own influence on the price λ. Note that it is not uncommon for WPs to be forced to act as price-takers in order to keep competition equitable, especially on critical services like power generation (cf. e.g. [Con14, p. 82]).

Owing to a price-taking behavior, each WP i ∈ W now solves

$$\max\_{\mathbf{p}\_{\mathbf{g},i}} \quad \mathsf{P}\_{\mathbf{g},i}(p\_{\mathbf{g},i}) = \lambda p\_{\mathbf{g},i} - \mathsf{C}\_{\mathbf{g},i}(p\_{\mathbf{g},i}) \tag{3.73}$$

in order to maximize its profit given the price λ, which is the solution of

$$\mathbb{1}^{\mathsf{T}}\mathfrak{p}\_{\mathsf{g}}(\lambda) = \mathbb{1}^{\mathsf{T}}\mathfrak{p}\_{\ell}.\tag{3.74}$$

As soon as the exact function λ(1 <sup>T</sup>p<sup>g</sup> ) is *not* known, Cournot competition has some very appealing features: First of all, the resulting market equilibrium is unique. Consider WPs with profit functions as in (3.73). Since the sensitivity of pg,i on λ is not known, WPs will always produce at marginal cost such that ∇Cg,i(pg,i) = λ. With Assumption 3.12, ∇Cg,i(pg,i) is strictly monotonically increasing and thus p ⋆ <sup>g</sup>,i(λ) = (∇Cg,i) −1 (λ). Under the condition of supply-demand matching, we therefore have P <sup>i</sup>∈<sup>P</sup> (∇Cg,i) −1 (λ) = 1 <sup>T</sup>p<sup>ℓ</sup> , which is again strictly monotonically increasing, thereby leading to a unique solution λ. More general statements on uniqueness of Cournot equilibria are provided e.g. by [MQ18, Theorem 4.1].

#### **Example 2 (***cont'd***)**

*Recall Example 2, where two WPs with cost functions* (3.71) *compete over a total demand* 1 <sup>T</sup>p<sup>ℓ</sup> = 10*. With Assumption 3.16, the market price* λ *is calculated as follows: For a fixed* λ*, the profit functions of WPs 1 and 2 with cost functions* (3.71) *are*

$$\mathbf{P}\_{\mathbf{g},1}(p\_{\mathbf{g},1}) = \lambda p\_{\mathbf{g},1} - p\_{\mathbf{g},1}^2,\qquad\qquad\qquad\mathbf{P}\_{\mathbf{g},2}(p\_{\mathbf{g},2}) = \lambda p\_{\mathbf{g},2} - \frac{1}{3}p\_{\mathbf{g},2}^2.\tag{3.75}$$

*With* (3.75)*, the optimal power generations in terms of* (3.73) *are*

$$p\_{\mathbf{g},1}^{\star} = \frac{1}{2} \lambda^{\star}, \qquad \qquad \qquad p\_{\mathbf{g},2}^{\star} = \frac{3}{2} \lambda^{\star}. \tag{3.76}$$

*Inserting* (3.76) *into* (3.74) *with* 1 <sup>T</sup>p<sup>ℓ</sup> = 10 *yields*

$$
\frac{1}{2}\lambda^\star + \frac{3}{2}\lambda^\star = 10 \qquad\qquad\implies\qquad\qquad\qquad\lambda^\star = 5.\tag{3.77}
$$

*Although* λ ⋆ *is the same as with Bertrand competition (cf. page 51), the resulting power generations are now given by* p ⋆ <sup>g</sup>,<sup>1</sup> = 2.5 *and* p ⋆ <sup>g</sup>,<sup>2</sup> = 7.5*, which are Pareto dominating among all market-clearing solutions fulfilling* (3.74)*.*

Despite the fact that both classical Bertrand and Cournot competition are unsuitable as soon as some fraction of the prosumers is inelastic, the above example suggests that using information barriers (in terms of unknown price sensitivities) to a Cournot model of competition result in the following benefits:


Note that both findings will be formalized to the case of general lossy networks with additional operational contraints later in Sections 3.4 and 3.5, respectively.

As will be elaborated in the remainder of this subsection, those information barriers are not only adequate to allow Cournot competition with inelastic demand, but also to approximately satisfy the conditions of a perfect market (cf. Definition A.24).

#### **Measures to Achieve a Perfect Market under Cournot Competition with Inelastic Demand**

Definiton A.24 is based upon idealized assumptions that can never be fully satisfied in reality. For instance, each real-world WP always has a certain (i.e. finitely small) influence on the overall market, and both information and power flows can never propagate at infinitely high speed through the network. However, through a reasonable design of market mechanisms and interactions, the characteristics of perfect competition can be pursued to a large extent:

Conditions 1) and 2) in Definition A.24 are trivially fulfilled. To satisfy condition 3), it has to be ensured that price formation over the network occurs sufficiently fast so that each market participant is equipped with the same information. To meet condition 4), we have to assume that the portions of electrical power which can be offered by suppliers are not quantized. From condition 5) it can be concluded that the resulting market mechanism must not incure any costs in total that are charged to the participants, i.e., the net profit of all CCs has to be zero. To comply with condition 6), WPs should at least act *as fast as possible*, such that no internal technical limits, e.g. start-up times, turbine dynamics, or ramp-up constraints are violated. Finally, condition 7) has the following implication:

**Assumption 3.17.** *No network participant is large enough to exercise market power.*

**Remark 3.18.** *Assumption 3.17 is justified if the overall number of WPs is high, since a high percentage of inelastic demand is always accompanied by an increased tendency to form monopolies and thus consequently higher prices [RT06]. Another means to satisfy Assumption 3.17 is the requirement that no WP should have a substantial market share in a specific cell<sup>39</sup> [Sta19, SHH19]. Alternatively, if there is a small number of market participants in a specific cell or throughout the network, these participants must not be aware that they have a relevant influence on the prices. Another implication of condition 7) is that any collusion among WPs has to be prevented [Küh01].* ♢

**Remark 3.19.** *With the increasing penetration by intermittent RESs, the price dynamics becomes unpredictable [RN20]. Thus, it is increasingly cumbersome for WPs to correctly predict the future price dynamics as well as the price sensitivities needed in* (3.72)*. This warrants, to a greater extent, the assumption that the momentary supply of WPs is at marginal cost, even if no WP has zero market power under real-world conditions.* ♢

**Remark 3.20.** *While Assumption 3.17 allows no oligopoly among WPs, there is no such requirement for the overall amount of CCs as they are assumed to be non-strategic network participants acting as the network's* neutral market facilitators *[dQQA19, p. 16]. See also Section 7.2 and references therein for conclusive statements on optimal cell partitioning.* ♢

Fig. 3.7 gives a summarizing overview of the obtained key stakeholders and their physical, informational, and monetary interaction. Compared to the status quo shown in Fig. 2.1, there is a drastic reduction in communication complexity along with rigorous information barriers between CCs as *sovereigns over the network* (complete information about their own network, but no means to physically interact with it) and WPs as *sovereigns over generation capacity* (able to physically interact with the network, but no knowledge about the network state). Moreover, it is most notable that any communication from CCs to WPs is exclusively via

<sup>39</sup> The establishment of distributed CVPPs is thus not only welcome in terms of portfolio diversification for the individual WPs, but also from an overall welfare point of view to prevent local monopoly formation.

**Figure 3.7:** Schematic diagram of key stakeholders arising from the proposed control scheme. The yellow box encompasses those components that are elaborated in detail within the scope of this dissertation. The arrows mark the typical flow direction of physical power (solid lines), information (dashed lines), or capital (triangle lines).

(wholesale) prices, Consequently, a full separation between the market-driven world (upper pane in Fig. 3.7) and the non-market-driven world (middle and lower pane in Fig. 3.7) is reestablished.

In the next Subsections 3.3.2–3.3.4, detailed and appropriate cost and communication structures are derived for WPs, RPs, and CCs, respectively, such that both requirements for perfect competition (cf. Definition A.24) as well as challenges (1)–(3) from Subsection 3.3.1 are properly addressed.

#### **3.3.2 Wholesale Prosumers**

WPs are *flexibility providers* [AP21], where the flexibility currently provided to the network corresponds to the momentary upper and lower power limits pg,i and p <sup>g</sup>,i. Throughout the following considerations, we suppose that all WPs behave *rationally* on an individual basis (cf. Assumption A.4) by means of a pure profit maximization strategy. To comply with the unbundling paradigm, we require the following:

**Claim 3.21.** *There should be no need for WPs to disclose their own cost or profit function to any opponent or CC.*

From Assumption 3.17 and Claim 3.21 it immediately follows that each WP π ∈ W neither has knowledge about the individual components nor about its own influence and thus seeks to maximize the momentary profits, since there is no personal benefit in pursing a dumping or markup strategy. Another beneficial implication of Claim 3.21 is that there is an intrinsic motivation for WPs to minimize their own cost structure. Namely, a flattening of Cg,i leads to WP π being allocated *more shares of total demand* due to the merit order effect [ZPE17, p. 279f.]. Consequently, every cost reduction of WP π ∈ W also increases the own revenue R<sup>π</sup> (while reducing the revenues of competitors).

To prevent any additional side payments, we further assume that all external costs are internalized appropriately into the cost functions Cg,i, e.g. that the individual greenhouse gas emissions are settled by a suitable emissions trading scheme (ETS) such as *EU-ETS* from the European Commission [ZW10]. These internalized costs can then be represented by an additional linear-affine component in the cost function. Typically it has a positive gradient for fossil-fueled CPPs that have to purchase allowances and a zero or negative slope for power generators from RESs that can sell their surplus allowances on the market<sup>40</sup>. Thus, typically, the actual costs for CPPs are higher than those of RESs. Note that these additional internalized costs do not affect the strict convexity assumption (cf. Assumption 3.12).

CCs purchase power from each WP of the own cell by paying the *zonal price* Λ<sup>k</sup> per unit of electrical power. To incorporate the marginal contribution of WPs on frequency stability, we propose the following remuneration scheme for WP π ∈ W:

$$\mathcal{R}\_{\pi} = \sum\_{i \in \mathcal{V}\mathbb{W}, \pi} \lambda\_i \cdot p\_{\mathbb{g},i} - \omega\_i \cdot p\_{\mathbb{g},i},\tag{3.78}$$

where λ<sup>i</sup> = Λ<sup>k</sup> for i ∈ VZ,k. The revenue R<sup>π</sup> consists of the frequency-independent payment for active power production at price λ<sup>i</sup> and a frequency-dependent part. The latter serves as an instantaneous *penalization* for active power generation during an overfrequency period and as an additional *reward* for active power generation during an underfrequency period. Subtracting the node-individual costs Cg,i incurred for power generation from (3.78), we get the overall profit

$$\mathbf{P}\_{\pi} = \sum\_{i \in \mathcal{V}\_{\mathcal{W}, \pi}} \mathbf{P}\_{\mathbf{g}, i} = \sum\_{i \in \mathcal{V}\_{\mathcal{W}, \pi}} -\mathbf{C}\_{\mathbf{g}, i}(p\_{\mathbf{g}, i}) + \lambda\_i \cdot p\_{\mathbf{g}, i} - \omega\_i \cdot p\_{\mathbf{g}, i} \tag{3.79}$$

for WP π ∈ W. In addition to maximizing Pπ, WPs must respect their internal operating limits

$$
\underline{p}\_{\mathbf{g},i} \le p\_{\mathbf{g},i} \le \overline{p}\_{\mathbf{g},i}, \tag{3.80}
$$

$$
\underline{U}\_i \le U\_i \le \overline{U}\_i \tag{3.81}
$$

at each node i ∈ V<sup>g</sup> (cf. challenge (2) in Subsection 3.3.1). Constraints (3.80) represent the remaining power generation capacity of WPs that is not contracted on the WEM (cf. Fig. 3.7),

<sup>40</sup> EU-ETS is based on the "cap and trade" principle, i.e., the quantity of emission allowances is capped and will be reduced over time. If a certain company cuts its emissions, it can sell its remaining emission certificates on the secondary market.

whereas constraints (3.81) are typically imposed by the grid code and monitored by the local CC.

**Remark 3.22.** *For the sake of simplicity we assume that the individual cost functions* Cg,i *and thus the profits* P<sup>π</sup> *are not dependent on the own voltage magnitudes* U<sup>i</sup> *. This is a common assumption for both CPPs and inverter-interfaced DERs, since changes applied to their output voltage usually do not incur additional cost [BV00, p. 405].* ♢

**Remark 3.23.** *Despite the fact that the above expressions feature a* producer-centric *notation, they also allow to incorporate elastic consumers at node* i *which aim to maximize their profits*

$$\mathbf{P}\_{\ell,i} = \mathbb{U}\_i(p\_{\ell,i}) - \lambda\_i \cdot p\_{\ell,i} + \omega\_i \cdot p\_{\ell,i} \tag{3.82}$$

*with* Ui(pℓ,i) *being a strictly concave utility function. These consumers can be modeled in a straightforward manner as producers with negative generation* pg,i = −pℓ,i ≤ 0*, since the profit maximization problem building upon* (3.82) *is structurally identical to* (3.79)*.* ♢

With Assumption 3.17, Claim 3.21, and Assumption 3.12, each WP π ∈ W thus aims at solving the constrained optimization problem

$$\max\_{\mathbf{P}\_{\mathbf{g},\pi},\mathbf{U}\_{\ell,\pi},\mathbf{U}\_{\mathcal{Z},\pi}} \mathbf{P}\_{\pi} \tag{3.83a}$$

$$\text{subject to} \quad \underline{p}\_{\mathbf{g},i} \le p\_{\mathbf{g},i} \le \overline{p}\_{\mathbf{g},i}, \quad i \in \mathcal{V}\_{\mathcal{W},\pi}, \tag{3.83b}$$

$$
\underline{U}\_i \le U\_i \le \overline{U}\_i, \qquad i \in \mathcal{V}\_{\mathcal{W}, \pi}. \tag{3.83c}
$$

**Remark 3.24.** *The reaction time of a particular WP to output a certain power* pg,i *or voltage* U<sup>i</sup> *varies in a wide range, depending on the specific electromechanical or electrochemical energy conversion technology used. For example, modern grid-forming inverters can provide the required quantities within less than one second [PEH18, p. 3662], while the turbogenerator of a classical large-scale thermal power plant may take several minutes until reaching maximum power [KRK*<sup>+</sup>*18]. We will account for such dynamic constraints directly in the WPs' controller equations to be derived in Section 3.4, so they need not be considered in the formulation of the optimization problem.* ♢

**Lemma 3.25.** *For each equilibrium of* (3.49)*, optimization problem* (3.83) *is equivalent to*

$$\max\_{\mathbf{P}\_{\mathbf{g},\pi}, \mathbf{U}\_{f,\pi}, \mathbf{U}\_{\mathcal{Z},\pi}} \mathbf{P}\_{\pi} \tag{3.84a}$$

$$\text{subject to} \qquad \underline{p}\_{\mathbf{g},i} \le p\_{\mathbf{g},i} \le \overline{p}\_{\mathbf{g},i}, \quad i \in \mathcal{V}\_{\mathcal{W},\pi}, \tag{3.84b}$$

$$\underline{U}\_i \le U\_i \le \overline{U}\_i, \qquad i \in \mathcal{V}\_{\mathcal{W}, \pi} \cap \mathcal{V}\_{\mathcal{T}}, \tag{3.84c}$$

$$
\underline{\Psi}\_i \le U\_{\rm f,i} \le \Psi\_i, \qquad i \in \mathcal{V}\_{\mathcal{W},\pi} \cap \mathcal{V}\_{\mathcal{S}}, \tag{3.84d}
$$

*where*

$$\underline{\Psi}\_{i} = U\_{i}^{\prime} - (X\_{\mathrm{d},i} - X\_{\mathrm{d},i}^{\prime}) \cdot \left(\frac{\underline{U}\_{i}}{X\_{\mathrm{n},i}} \cos(\vartheta\_{\mathrm{n},i}) - \frac{\underline{U}\_{i}^{2}}{U\_{i}^{\prime} X\_{\mathrm{n},i}}\right),\tag{3.85a}$$

$$\overline{\Psi}\_{i} = U\_{i}^{\prime} - (X\_{\mathrm{d},i} - X\_{\mathrm{d},i}^{\prime}) \cdot \left(\frac{\overline{U}\_{i}}{X\_{\mathrm{n},i}} \cos(\vartheta\_{\mathrm{n},i}) - \frac{\overline{U}\_{i}^{2}}{U\_{i}^{\prime}X\_{\mathrm{n},i}}\right). \tag{3.85b}$$

*Proof.* Let □<sup>⋆</sup> denote an equilibrium of (3.49) and let <sup>i</sup> ∈ V<sup>S</sup> . From (3.28c) it follows that

$$U\_{\mathbf{f},i}^{\star} = (U\_i^{\prime})^{\star} - (X\_{\mathrm{d},i} - X\_{\mathrm{d},i}^{\prime}) \cdot q\_{\mathrm{inj},i}^{\star} / (U\_i^{\prime})^{\star} =: \Psi\_i(U\_i^{\prime}, U\_i). \tag{3.86}$$

Inserting (3.26) in (3.86) and comparison with (3.85) yields Ψ<sup>i</sup> = Ψi(U ′ i , U<sup>i</sup> ) and Ψ<sup>i</sup> = Ψi(U ′ i ,Ui). With

$$\frac{\partial \Psi\_i}{\partial U\_i} = -\frac{X\_{\rm d,i} - X\_{\rm d,i}'}{X\_{\rm n,i}} \left( \cos(\vartheta\_{\rm n,i}) - \frac{2U\_i}{U\_i'} \right) > 0 \tag{3.87}$$

under normal operating conditions, it holds that Ψ<sup>i</sup> is strictly monotonically increasing with respect to U<sup>i</sup> , thus U ⋆ <sup>i</sup> ≤ U<sup>i</sup> ⇐⇒ Ψ<sup>i</sup> ≤ Ψ<sup>i</sup> and U<sup>i</sup> ≤ U ⋆ <sup>i</sup> ⇐⇒ Ψ<sup>i</sup> ≤ Ψ<sup>i</sup> . Accordingly, each optimizer of (3.84) is an optimizer of (3.83), and vice versa.

#### **3.3.3 Retail Prosumers**

RPs at node i ∈ V are characterized by an inelastic active and reactive power demand pℓ,i and qℓ,i. The cumulative cost for consumption in cell k ∈ Z are set to

$$\mathbf{C}\_{k}^{\ell} = \Lambda\_{\mathbb{R},k} \cdot \sum\_{i \in \mathcal{V}\_{\mathcal{E},k}} p\_{\ell,i}, \tag{3.88}$$

where ΛR,k denotes the *retail price* for electricity in cell k (cf. Fig. 3.7).

Note that small-scale power plants relying on uncontrollable power sources (e.g. single photovoltaic power stations or wind power plants that are not integrated into a VPP) can be modeled as RPs with pℓ,i ≤ 0. In addition, pℓ,i and qℓ,i are used to cover all positive or negative physical supplies arising from other contracts such as forward contracts or obligations due to the day-ahead market. However, from these markets only the current value of the resulting market clearing quantities is known to the local CC, while none of the other network participants are aware of it. Furthermore, it is completely unknown for all network participants which active and reactive power flows are to be expected in the future.

#### **3.3.4 Cell Coordinators**

The CCs' exclusive responsibility for stable, economically efficient and grid-supportive operation which was outlined in Section 3.1 has two connotations: On one hand, *exclusive* means

that no other network participant (neither CC nor WP) shares the same information or has the same responsibilities. On the other hand, it means that maintaining the above three objectives is the *sole purpose* of each CC. In particular, it does not have any economic goals. As pure service providers, CCs solely own a portion of the transmission and distribution infrastructure (cf. challenge (3) in Subsection 3.3.1) and, apart from that, neither storages nor any kind of generation capacity. This is a direct consequence from the unbundling requirement elaborated in Section 2.1. The geographic region that is covered by CC k ∈ Z is characterized by the weakly connected component Gc,k in Gc.

**Assumption 3.26.** *CCs have full knowledge about their cell's own line parameters (*Gij *,* Bij *,* g<sup>i</sup> *,* bi*) and maintain continuous, delay-free measurements of the voltage magnitudes* Ui *, the power injections* pg,i*, the power consumptions* pℓ,i*,* qℓ,i*, and the power flows* Pij *,* Qij *within their own cell.*

Assumption 3.26 can be justified as follows: The line parameters can be derived from manufacturers' specifications using (3.5) and (3.6). The power flows and voltage amplitudes can be measured directly in-situ by means of *phasor measurement units* (PMUs) [MBB12, p. 61f.]. In particular, note that the voltage angles θ<sup>i</sup> or ϑij do not need to be measured. Moreover, CCs are only aware of the current "grid-side" power flows as well as calculations of the momentary resistive transmission losses

$$\hat{\Phi}\_k = \sum\_{i \in \mathcal{V}\_{Z,k}} \left( G\_{ii} U\_i^2 + \sum\_{j \in \mathcal{N}\_i} G\_{ij} U\_i U\_j \cos(\vartheta\_{ij}) \right) \tag{3.89}$$

within cell k. However, CCs do not know the WPs' or RPs' internal parameters nor any of the individual network participants' profits or optimization strategies. Furthermore, if some of these quantities are not directly measurable, *state observers* can be used to reconstruct suitable *estimators* for each of the respective quantities. Such state observes are already being used successfully in today's power systems. Recently, researchers have also increased their scrutiny of new methodological research on state observers tailored specifically for power systems. Comprehensive overviews of current research branches in this field are provided in [ARO<sup>+</sup>18, WGCS19, KKŠ20].

For all i, j ∈ VZ,k, the responsible CC k ∈ Z seeks to choose the cell-specific price Λ<sup>k</sup> = λ<sup>i</sup> = λ<sup>j</sup> in such a way that its own (virtual) profit Pk, determined by the net active power balance within the own cell, is zero:

$$\mathbf{P}\_k = \Lambda\_k \cdot \left(\hat{\Phi}\_k + \sum\_{i \in \mathcal{V}\_k^Z} p\_{\ell,i}\right) - \Lambda\_k \cdot \sum\_{\substack{i \in \mathcal{V}\_{\mathcal{Z},k} \\ i \notin \mathcal{V}\_{\mathcal{L}}}} p\_{g,i} \stackrel{!}{=} 0. \tag{3.90}$$

#### **3.3.5 Summary**

Using the prototypical network participants WPs, CCs, and RPs, Subsections 3.3.2–3.3.4 have modeled the different operating maxims of stakeholders in a fully unbundled power network. Thereby, WPs π ∈ W constitute the players of an *electricity market game* with the goal of maximizing their own profit while complying with operating constraints e.g. for their own power injections and voltage magnitudes. While choosing their own actions as *best response* to given environmental conditions, WPs rely on information about their own cost structure and machine parameters as well as the current cell-specific price Λk. In the presented framework, there is no hierarchy in the execution of individual actions. As a consequence of the requirement of information barriers and unbundling, the individual players are not aware of the formation of prices. Thus, the profit functions of WPs are completely decoupled.

At the same time, CCs as pure service providers must ensure adherence with balancing constraint (3.90) through appropriate choice (i.e. real-time adjustment) of cell-specific prices Λ<sup>k</sup> while considering real-time measurements of the RPs' power consumptions as well as cell-specific transmission losses.

The dynamic interplay of these different network participants with time-varying environmental conditions (induced e.g. by load jumps or changes in the network structure) leads to a superposition of constrained optimization problems (3.83) and (3.90). The individual constraints include *input constraints* (e.g. limitations of power pg,i (3.83b) or inverter voltages U<sup>i</sup> in (3.83c)), *output constraints* such as limitations of voltage magnitudes at generator terminals in (3.83c), and *dynamic system constraints* which are enforced by the physical relationships of the interconnected plant system (3.49).

As already shown in Lemma 3.25, output constraints (here: constraints (3.83c) on the terminal voltages U<sup>i</sup> ) can be translated into input constraints (here: constraints on the excitation voltages Uf,i (3.84d)) provided that a suitable *steady-state map* (here Ψi(Ui) as in (3.85)) is known. While for input and output constraints *asymptotic* satisfaction is permissible, i.e., a temporary constraint violation is allowed in the course of solving the optimization problem, system constraints, on the other hand, are complex environmental conditions that typically have to be satisfied even during transients in order to achieve a result that is physically admissible at any point in time, i.e., in accordance with the laws of physics. However, due to the numerous incorporated information barriers, there is no central authority that has complete knowledge of the whole plant state xp. A suitable controller design must thus take these information barriers into account, but at the same time must exhibit favorable properties from an overall system perspective such as robustness to disturbances and sufficiently low computational effort of local calculations. With this in mind, the next section provides the main result in the sense of Contribution 1. In particular, the derivation of distributed controller equations enables network participants to achieve the overall goal of a stable overall power system by means of individual and local control actions as required by Contribution 1.

## **3.4 Controller Development**

For optimization-based control on constrained physical systems, arising from the interconnection of the physical plant system from Section 3.2 with the network participants from Section 3.3, two different branches are known in literature. The first branch emanates from a classical understanding of solving optimization problems in a *feedforward* manner, whereby there exists

a preferably exact steady-state model for the plant system, allowing its direct representation as an additional set of constraints. MPC (see Section 2.2.2) is a classical representative of a feedforward optimization approach. As discussed in Section 2.2.2, there is always a trade-off between constraint satisfaction (i.e. real-world feasibility of the resulting OPF solution) and computational complexity. Thereby, feedforward optimization methods for power system applications typically scale poorly not only with model order and number of iteration steps, but also with the number of nodes per WP or CC. However, imperfectly estimated parameters, non-modeled side effects, or sudden structural changes not captured in the model always cause physically infeasible solutions at the plant input without any further possibility of compensation.

As a second branch, *feedback*-based optimization converts the optimization algorithm into a continuous-time control law based on online measurements of the current plant state xp, while driving the physical system to its (constrained) optimum value<sup>41</sup>. By directly relying on measurement information, no perfect model knowledge is required. Instead, the physical constraints (along with possible disturbances or structural changes) manifest themselves directly in the measurement signal as they arise via the feedback branch, where *"the physical system [itself] acts as a constraint enforcer"* [HBHD21b, p. 2]. Since the system equations no longer need to be calculated separately by each local agent, this significantly reduces the problem size of the individual optimization problems, and thus also the computational effort for each network participant<sup>42</sup>. Furthermore, it is no longer necessary to conservatively model *all* possible realizations of a disturbance. Instead, only the *actual* materialization of the underlying disturbance process is employed for the participants' decision making via real-time measurements of the system state.

Due to these evident advantages, feedback optimization strategies will form the methodological basis of all controller design in the remainder of this dissertation. Subsection 3.4.1 presents the main types of feedback optimization strategies known in the literature and discusses their advantages and disadvantages. Subsection 3.4.2 then deduces distributed feedback control laws for WPs and CCs. Subsection 3.4.3 analyzes stability of the closed-loop equilibrium. Finally, Subsection 3.4.4 illustrates the main conclusions of this section by means of an academic example.

#### **3.4.1 Introduction to Feedback Optimization**

Feedback optimization methods interpret the governing optimization algorithm as a dynamic control system. The basic idea of re-thinking the classical discrete-time steepest descent

<sup>41</sup> Although terminologically related, *extremum seeking* refers to optimization-based control methods for the *model-free* exploration of the feasible domain by means of approproate dither signals, see e.g. [TMM+10] for an insightful literature review. However, since the necessary excitation with these dither signals is contrary to the requirement of economic efficiency and, moreover, an appropriate choice is highly difficult for higher-dimensional systems [HBHD21a, p. 6], extremum seeking-based control schemes will not be considered any further in the context of this dissertation.

<sup>42</sup> For quantitative robustness statements, see [CSPB19]. Practical experiments are conducted in [PBD20] and [BD19]. For an experimental validation of (feedforward) AC-OPF vs. feedback AC-OPF, the reader is referred to [OHC+20]. A general discussion of feedback vs. feedforward optimization of power systems is given e.g. in Section III of [DBSPG19].

approach for minimizing a certain (unconstrained) objective function J(u, y) as a timecontinuous *gradient flow* was already proposed in the seminal publications [Kos56, AHU58] dating back to the 1950s, and has just been extensively revisited and improved43. Typically, it relies upon the assumption of a fast-decaying plant system whose input-output behavior can be represented by a *steady-state input-output map* y = h(u) + d. For an arbitrary objective function J(u, y) which depends on both the control and system output, we then get

$$\min\_{\mathbf{u}} \qquad \qquad J(\mathbf{u}, \mathbf{y}) \tag{3.91a}$$

$$\text{subject to} \quad y = h(u) + d. \tag{3.91b}$$

Remarkably, from the controller's point of view, constraint (3.91b) does not need to be considered explicitly. As each output y applied to the controller already satisfies the system equation, it automatically holds that any input-output pair (u, y) is a solution of (3.91b). Thus, by defining <sup>J</sup>e(u) = <sup>J</sup>(u, <sup>h</sup>(u) + <sup>d</sup>), the own control strategy can be calculated by a gradient descent on <sup>J</sup>e. With the chain rule, we get

$$\dot{\boldsymbol{u}} = -\frac{\partial \tilde{J}}{\partial \boldsymbol{u}} = -\left(\frac{\partial J}{\partial \boldsymbol{y}}\right)^{\mathsf{T}} \frac{\partial \mathsf{h}}{\partial \boldsymbol{u}} - \frac{\partial J}{\partial \boldsymbol{u}}.\tag{3.92}$$

It is evident that only the sensitivities <sup>∇</sup>h(u) need to be known<sup>44</sup> .

Beyond the conceptually simplistic idea of continuous-time gradient flow, there are different approaches to dealing with possible additional input or output constraints of the form

$$
\mu \in \mathcal{U}', \tag{3.93a}
$$

$$y \in \mathcal{Y}',\tag{3.93b}$$

where U ′ and Y ′ denote the sets of feasible inputs and outputs, respectively. First of all, by defining U := U ′ ∩ h −1 (Y ′ ), (3.91)–(3.93) can be written as

$$\min\_{\mathbf{u}} \qquad \qquad J(\mathbf{u}) \tag{3.94a}$$

$$\text{subject to} \quad \mathfrak{u} \in \mathcal{U}. \tag{3.94b}$$

The approaches discussed in the literature differ in their strictness of *constraint enforcement*, in particular, whether or not each closed-loop equilibrium is always an "exact" optimizer of (3.94), and whether or not temporary violations of the constraint (3.94b) are allowed. Table 3.2 gives an overview of the main research branches, which are briefly discussed in the following.

<sup>43</sup> Interestingly, the main impetus for the renaissance of this general control scheme came from the power system community, see e.g. [DBSPG19] for a historical perspective. The main reasons for their popularity in power system applications are their robustness properties as well as their ease of implementation with relatively low computational effort for the individual controllers (cf. [CSPB19]).

<sup>44</sup> Trivially, if J only depends on the input, the gradient descent simplifies to u˙ = −∇J(u). For optimization problems which solely rely on y, see [JLv09, BDE12].


**Table 3.2:** Constraint enforcement schemes for continuous-time feedback optimization.

#### **Augmented Gradient Flow**

By augmenting <sup>J</sup>e(u) with a scalar *penalty* or *barrier* summand ℧(u) indicating constraint violation, the constrained optimization problem (3.94) turns into the unconstrained optimization problem

$$\min\_{\mathbf{u}} \quad \ddot{J}(\mathbf{u}) + \mathbb{U}(\mathbf{u}). \tag{3.95}$$

If the feasible domain U in (3.94b) can be characterized via inequality constraints of the form <sup>g</sup>(u) <sup>≤</sup> <sup>0</sup>, then for instance, ℧(u) = <sup>τ</sup>∥min{**0**, <sup>g</sup>(u)}∥<sup>2</sup> , τ > 0 provides a suitable penalty term and ℧(u) = −τ ln(g(u)), τ > 0 can be used as barrier term. As τ → ∞, possible constraint violations become more pronounced in the augmented objective function, and thus constraint satisfaction is imposed more forcefully. However, while penalty and barrier functions always impose some sort of outer and inner approximation of the feasible set, respectively, the augmentation of <sup>J</sup>e(u) typically leads to steady-state deviations between the closed-loop equilibrium and the optimizer u <sup>⋆</sup> of (3.94), especially if this optimizer is located at or near the boundary of U. But at the same time, τ must still be sufficiently low to ensure a reasonably timescale separation between a "fast" plant system and "slow" optimization dynamics.

#### **Projected Gradient Flow**

A refinement of the gradient flow was introduced by *projected gradient flow* [HBHD16]. The remarkable feature of this method is that u(t) ∈ U can be guaranteed at any time t, thus all constraints are always fulfilled even during transients. This is accomplished by requiring that all directional changes of u should point into the feasible set. For this purpose, the right-hand side of (3.92) is replaced by the *gradient projection* onto U, leading to

$$\dot{\boldsymbol{u}} = -\Pi\_{\mathcal{U}}(-\nabla \widetilde{J}(\boldsymbol{u})) = -\operatorname\*{arg\,min}\_{\boldsymbol{\xi} \in \mathcal{T}\_{\mathcal{U}}} \| -\nabla \widetilde{J}(\boldsymbol{u}) - \boldsymbol{\xi} \|,\tag{3.96}$$

where T<sup>U</sup> (u) := cl{d : d = α(u ′ − u),u ′ ∈ U, α ≥ 0} is the *tangent cone* of U at u. By definition, if <sup>u</sup> <sup>∈</sup> int <sup>U</sup>, then <sup>T</sup><sup>U</sup> (u) = <sup>R</sup><sup>p</sup> , while if u ∈ bd U, then T<sup>U</sup> (u) represents the set of all vectors starting from u and pointing into the feasible set U. The ability of *strict transient constraint enforcement* is a salient feature of projected gradient flow. In [HBHD21b] quantitative statements about asymptotic stability of the closed-loop system with projected gradient are provided, which, however, require a priori knowledge of appropriate Lipschitz constants of the system with respect to both the input u and the state x.

However, despite these advantages, projected gradient flow exhibits some major drawbacks, which hamper a practical application, especially in the light of the electricity market game mentioned in Subsection 3.3.5 with its intermittent nature of power consumption and/or production and a possibly large amount of distributed agents: Besides the fact that explicit expressions for the gradient projection (3.96) are often hard to find and generally not amenable to a distributed implementation, the main practical drawback of projected gradient flow is its tendency to "false equilibria" (i.e. equilibria of the closed-loop system that are no optimizers of (3.94)), whenever the current solution x(t) is infeasible with respect to (3.94b) at some point in time. We illustrate this effect by means of the following non-example:

#### **Example 3 (Projected Gradient Flow)**

*Consider the optimization problem*

$$\min\_{u} \qquad \qquad \frac{1}{2}u^2 \tag{3.97a}$$

$$\text{subject to} \quad u + 10 \le 0. \tag{3.97b}$$

*Note that* (3.97) *is a convex optimization problem with strictly convex objective function. Hence,* u <sup>⋆</sup> <sup>=</sup> <sup>−</sup><sup>10</sup> *is the unique optimizer of* (3.97)*. With* <sup>U</sup> <sup>=</sup> {<sup>u</sup> <sup>∈</sup> <sup>R</sup> : <sup>u</sup> ≤ −10}*, we get*

$$\mathcal{T}\_{\mathcal{U}}(u) = \begin{cases} \mathbb{R}\_{<0}, & u \ge -10, \\ \mathbb{R}, & u < -10. \end{cases} \tag{3.98}$$

*Now consider the trajectory of* (3.96) *starting at* <sup>u</sup><sup>0</sup> = 10*. With* <sup>T</sup><sup>U</sup> (10) = <sup>R</sup>*, we get*

$$\dot{u} = \operatorname\*{arg\,min}\_{\xi \in \mathcal{R}} \| -u - \xi \| = -u. \tag{3.99}$$

*Accordingly, a solution is given by* u(t) = 10 exp(−t) *and thus* lim<sup>t</sup>→∞ u(t) = 0*, which means that* u(t) *never converges to its optimum value* u <sup>⋆</sup> <sup>=</sup> <sup>−</sup>10*. Trivially, the same result holds for any other* u *starting in the positive orthant.*

With respect to the CCs' balance constraints (3.90) for instance, this shortcoming of projected gradient flow is highly problematic, since temporary constraint violations e.g. due to sudden load jumps are always possible, potentially causing the resulting closed-loop equilbrium to no longer satisfy (3.90).

#### **Saddle-Point Flow**

Besides penalty methods and projected gradient flow, another means for incorporating constraints is given by *primal-dual saddle-point flow*, originally proposed in [Kos56] and [AHU58]. It can be applied to optimization problems in *standard form* (cf. Definition A.18). The naming of primal-dual saddle-point flow stems from the fact that each optimizer of such optimization problems is a saddle point of the respective *(optimization) Lagrangian*. In particular, if (3.94b) can be represented by a set of <sup>r</sup> <sup>∈</sup> <sup>N</sup> ∪ {0} equality and <sup>s</sup> <sup>∈</sup> <sup>N</sup> ∪ {0} inequality constraints of the form h1(u) = **0** and h2(u) < **0**, respectively, then the Lagrangian of (3.94) takes the form <sup>L</sup> (u,λ, <sup>µ</sup>) = <sup>J</sup>e(u) + <sup>λ</sup> <sup>T</sup>h1(u) + µ <sup>T</sup>h2(u), where λ and µ denote the vector of dual variables for h1(u) and h2(u), respectively. Provided that some constraint qualification is fulfilled, the Karush-Kuhn-Tucker (KKT) conditions (cf. Definition A.20 in Appendix A.5) constitute first-order necessary conditions for the primal-dual optimizer (u ⋆ ,λ ⋆ , µ ⋆ ) with gradient descent for the primal variables u and gradient ascent for the dual variables λ, yielding

$$\dot{u} = -\frac{\partial \mathcal{L}(u, \lambda, \mu)}{\partial u},\tag{3.100a}$$

$$
\dot{\lambda} = + \frac{\partial \mathcal{L}^{\mathcal{Q}}(u, \lambda, \mu)}{\partial \lambda}. \tag{3.100b}
$$

The inequality constraints h2(u) ≤ 0 translate into complementary slackness condition (A.20c) and the nonnegativity of µ (cf. (A.20d)). Therefore, in [FP10], an additional projected gradient flow for the dual variables µ(t) onto the tangent cone of the nonnegative orthant

$$\dot{\mu} = +\Pi\_{\mathbb{R}^{s}\_{\geq 0}}(\nabla \mathcal{L}^{\rho}(\mu)) = +\Pi\_{\mathbb{R}^{s}\_{\geq 0}}(h\_{2}(u))\tag{3.101}$$

is used to enforce µ(t) ≥ **0** for all t. Since the projection of h1,i, i ∈ {1, . . . , s} onto the tangent cone T<sup>R</sup>≥<sup>0</sup> allows for the explicit expression

$$\Pi\_{\mathbb{R}\geq 0}(h\_{1,i}(\mathfrak{u})) = \langle h\_{1,i}\rangle\_{\mu\_i}^+ := \begin{cases} h\_{1,i}(\mathfrak{u}), & h\_{1,i} > 0 \vee \mu\_i > 0, \\ 0, & \text{otherwise}, \end{cases} \tag{3.102}$$

the dual ascent (3.101) of µ can be written compactly as

$$
\dot{\mu} = \langle h\_1(u) \rangle\_{\mu}^+,\tag{3.103}
$$

where ⟨·⟩<sup>+</sup> <sup>µ</sup> := col<sup>i</sup>∈{1,...,s}{⟨·⟩<sup>+</sup> µ<sup>i</sup> }.

In contrast to the "pure" projected gradient flow (3.96) without dualization, this *projected saddlepoint flow* (3.100), (3.103) yields a constraint satisfaction only at steady-state<sup>45</sup>. Particularly,

<sup>45</sup> Despite this temporary constraint violation of primal-dual saddle-point flow, it should be emphasized that the resulting controller always yields "physically feasible" results u(t) at all times t, since the actual constitutive equations of the plant system are exactly respected at all times due to the feedback interconnection and thus do not need to be part of the constraints, as discussed above.

even the gradient projection (3.102) does not prevent temporary constraint violations of h1(u) ≥ 0.

Despite this tendency towards temporary constraint deviation, however, especially in recent years, several control approaches in the area of networked and power systems rely on primaldual saddle-point flow and its derivations (see the list of most prevalent methods in Table 3.2). Likewise, some of the real-time economic dispatch and optimization-based frequency control approaches discussed in Subsections 2.2.2 and 2.2.4 resort to one of the above methods or a combination thereof for the real-time feedback optimization of their arising optimization problems. [MZL14, LZC16, TBD16, SDv17a, CY17] use the primal-dual saddle-point flow for optimization-based frequency control of S and/or L nodes. [SDv17b] incorporates additional inequality constraints on active power injections and power line capacities (for networks with tree topology only) by introducing additional projected gradient ascent (3.103) for the dual variables. Moreover, additional penalty terms are proposed allowing an extension to non-strictly convex objective functions. [HZB<sup>+</sup>17] uses projected gradient flow in conjunction with a penalty-augmented Lagrangian for a centralized voltage regulation with congestion management. The main reason for the popularity of primal-dual saddle-point flow approaches is their ease of implementation in contrast to the projected gradient method, especially if the underlying plant system exhibits a sparsity pattern. As discussed e.g. in [FP10, p. 1979], primal-dual saddle-point flow lends itself to a distributed implementation, which is highly desirable in view of the individual stakeholders defined in Section 3.3. Example 3 also stated and highlighted the "warm start" problem of projected gradient flow: If the controller variables are initialized as infeasible or if individual variables become temporarily infeasible due to external events, the closed-loop system tends to invoke false equilibria. By contrast, primaldual saddle-point flow can always be initialized even if the current state (u, y) does not satisfy all of the constraints. Moreover, the resulting controller equations (3.100) (and (3.103), if desired) always exist in explicit form, thus encouraging an automated controller design.

**Remark 3.27.** *There already exists a variety of* hybrid *forms between penalty, projected gradient flow or saddle-point flow methods (see Table 3.2). For instance, [Pol09a] suggests that further augmentation of* L (u,λ, µ) *by another penalty term (resulting in the definition of* augmented saddle-point flow*) may help in damping oscillations, yet at the price of losing optimality with respect to the original problem. [HBHD21a, p. 14] proposes a* mixed saddle-point flow *where some of the inequality constraints are enforced directly by (gradient) projection and some are enforced by projected gradient ascent of the dual variables as in* (3.103)*.* ♢

**Remark 3.28.** *In addition, the right-hand sides of the controller equations* (3.92)*,* (3.96)*, or* (3.100) *can be premultiplied with any positive semidefinite matrix* A(u) *without changing the equilibrium. The prominent choice* <sup>A</sup>(u) = (∇<sup>2</sup>Je(u))−<sup>1</sup> *, yielding a Newton-type gradient descent with improved convergence properties has thereby given rise to the formulation of* Newton gradient flows*,* projected Newton flow*, and* Newton saddle-point flow *[HBD21, HBHD21a].* ♢

With respect to practical application to electricity markets, a downside of augmentation methods and variations of saddle-point flow is that it usually sacrifices the distributed implementability mentioned above. In the light of this reasoning, particularly since temporary

constraint violations in the optimization problems (3.84) and (3.90) are tolerable according to the definition of power system stability presented at the beginning of Section 2.2, we will use projected saddle-point flow (with classical saddle-point flow for the equality constraints and gradient-projected dualization of the inequality constraints) as the basic building block for the local controllers, though we will discuss possible extensions where advisable.

#### **3.4.2 Resulting Control Laws**

Before deriving specific control laws for WPs and CCs by means of projected saddle-point flow, we give two general statements with reference to the optimization problem in standard form from Definition A.18.

**Lemma 3.29.** *Denote by* (u ⋆ ,λ ⋆ , µ ⋆ ) *an equlibrium of* (3.100)*,* (3.103)*, and suppose that the optimization problem* (A.19) *has a strictly convex objective function* f0(x) *and fulfills Slater's condition* (A.21)*. Then* u ⋆ *is a solution of* (A.19)*.*

*Proof.* Trivially, each (u ⋆ ,λ ⋆ , µ ⋆ ) fulfills the KKT conditions (A.20) for (A.19). Since f0(u) is strictly convex, the KKT conditions are also sufficient for optimality if Slater's condition holds (cf. [BV15, p. 244] and Lemma A.22). Thus, each u ⋆ is a solution of (A.19) .

**Lemma 3.30.** *Let* f0(u) *be strictly convex. Then the projected saddle-point flow* (3.100)*,* (3.103) *asymptotically converges to the solution of* (A.19)*.*

*Proof.* See [CMC16] for a proof using an extended version of LaSalle's invariance principle for Carathéodory solutions of discontinuous dynamical systems or [Goe17] for an alternative proof using maximal monotone mappings.

**Remark 3.31.** *If the strict convexity assumption of* f0(u) *from Lemma 3.30 does not hold, the application of saddle-point flow may cause the closed-loop system to exhibit limit-cycle behavior [HOBD20]. However, as shown in [HOBD20], the asymptotic convergence results of Lemma 3.30 can be regained by a suitable augmentation of* L (u,λ, µ)*.* ♢

Using Lemma 3.29 and Lemma 3.30, we now consider the individual optimization problems of WPs and CCs.

For WP π ∈ W, the Lagrangian of (3.84) equals

$$\begin{split} \mathcal{L}^{\rho}\_{\pi} &= -\mathsf{P}\_{\pi} + \boldsymbol{\mu}\_{\text{g}-,\pi}^{\mathsf{T}} (\underline{\mathbf{p}}\_{\text{g},\pi} - \mathbf{p}\_{\text{g},\pi}) + \boldsymbol{\mu}\_{\text{g}+,\pi}^{\mathsf{T}} (\mathbf{p}\_{\text{g},\pi} - \overline{\mathbf{p}}\_{\text{g},\pi}) \\ &+ \boldsymbol{\mu}\_{\text{S}-,\pi}^{\mathsf{T}} (\underline{\mathbf{U}}\_{\pi} - \mathbf{U}\_{\text{f},\pi}) + \boldsymbol{\mu}\_{\text{S}+,\pi}^{\mathsf{T}} (\mathbf{U}\_{\text{f},\pi} - \overline{\mathbf{V}}\_{\pi}) \\ &+ \boldsymbol{\mu}\_{\text{Z}-,\pi}^{\mathsf{T}} (\underline{\mathbf{U}}\_{\text{Z},\pi} - \mathbf{U}\_{\text{Z},\pi}) + \boldsymbol{\mu}\_{\text{Z}+,\pi}^{\mathsf{T}} (\mathbf{U}\_{\text{Z},\pi} - \underline{\mathbf{U}}\_{\text{Z},\pi}), \end{split} \tag{3.104}$$

where µ(·),π denote the vectors of Lagrange multipliers for the inequality constraints (3.84b)– (3.84d) and Ψ<sup>π</sup> := coli∈VW,π∪V′ S {Ψ<sup>i</sup> }, Ψ<sup>π</sup> := coli∈VW,π∪V′ S {Ψi}. Since (3.84) is convex and Slater's condition is fulfilled, the KKT conditions specifying a saddle point of L<sup>π</sup> can be applied to derive a necessary and sufficient condition for an optimizer of (3.84)46:

$$\mathbb{0} = \nabla \mathbb{C}\_{\pi}(\mathfrak{p}\_{\text{g},\pi}^{\star}) - \lambda\_{\pi} + \omega\_{\pi} - \mu\_{\text{g}-,\pi}^{\star} + \mu\_{\text{g}+,\pi}^{\star},\tag{3.105a}$$

$$0 = -\mu\_{\mathcal{S}-,\pi}^{\star} + \mu\_{\mathcal{S}+,\pi}^{\star},\tag{3.105b}$$

$$\begin{split} 0 &= -\mu\_{\mathcal{Z}-,\pi}^{\star} + \mu\_{\mathcal{Z}+,\pi}^{\star} - (\nabla \underline{\Psi}\_{\pi}(U\_{\mathcal{Z},\pi})) \circ \mu\_{\mathcal{S}-,\pi} + (\nabla \overline{\Psi}\_{\pi}(U\_{\mathcal{Z},\pi})) \circ \mu\_{\mathcal{S}+,\pi}, \tag{3.105c} \\ 0 &= (\mu^{\star} \quad \phantom{\cdot}) \circ (\mathsf{n} \quad \phantom{\cdot} - \mathsf{n} \quad \phantom{\cdot}) \end{split} \tag{3.105d}$$

$$\mathbb{0} = (\mu\_{\mathbf{g}-,\pi}^{\star}) \circ (\underline{\mathbf{p}}\_{\mathbf{g},\pi} - \mathbf{p}\_{\mathbf{g},\pi}),\tag{3.105d}$$

$$\mathbb{O} = (\mu\_{\text{g}+,\pi}^{\star}) \circ (\mathbf{p}\_{\text{g},\pi} - \overline{\mathbf{p}}\_{\text{g},\pi}),\tag{3.105e}$$

$$\mathbb{O} = (\mu\_{\text{g}}^{\star} \qquad \rangle \circ (\mathcal{U}\_{\text{e}} - \mathcal{U}\_{\text{e}} \quad )\tag{3.105}$$

$$\begin{aligned} 0 &= (\mu\_{\mathcal{S}-,\pi}^{\star}) \circ (\underline{\mathbf{U}}\_{\mathbf{f},\pi} - \mathbf{U}\_{\mathbf{f},\pi}), \\ 0 &= (\mu\_{\mathcal{S}+,\pi}^{\star}) \circ (\mathbf{U}\_{\mathbf{f},\pi} - \overline{\mathbf{U}}\_{\mathbf{f},\pi}), \end{aligned} \tag{3.105}$$

$$0 = (\mu\_{\mathcal{Z}-,\pi}^{\star}) \circ (\underline{\mathbf{U}}\_{\mathcal{Z},\pi} - \mathbf{U}\_{\mathcal{Z},\pi}),\tag{3.105h}$$

$$0 = (\mu\_{\mathcal{Z}+,\pi}^{\star}) \circ (\mathbf{U}\_{\mathcal{Z},\pi} - \overline{\mathbf{U}}\_{\mathcal{Z},\pi}),\tag{3.105i}$$

$$0 \le \mu\_{\mathcal{S}-,\pi}^{\star}, \mu\_{\mathcal{S}+,\pi}^{\star}, \mu\_{\mathcal{T}-,\pi}^{\star}, \mu\_{\mathcal{T}+,\pi}^{\star}, \mu\_{\mathcal{g}-,\pi}^{\star}, \mu\_{\mathcal{g}+,\pi}^{\star}, \tag{3.105j}$$

where ∇Cπ(pg,π) = col<sup>i</sup>∈VW,π {∇Cg,i(pg,i)}. Since no S ′ node is adjacent to any I node (cf. Subsection 3.2.5), it holds that ∇Ψ<sup>π</sup> (UI,π) = ∇Ψπ(UI,π) = **0**, thus application of the projected saddle-point flow (3.100), (3.103) yields the controller equations

$$\tau\_{\mathbf{g},\pi}\dot{\mathbf{p}}\_{\mathbf{g},\pi} = -\nabla \mathbb{C}\_{\pi}(\mathbf{p}\_{\mathbf{g},\pi}^{\star}) + \lambda\_{\pi} - \omega\_{\pi} + \mu\_{\mathbf{g}-,\pi} - \mu\_{\mathbf{g}+,\pi},\tag{3.106a}$$

$$
\pi\_{U\_{\ell,\pi}} \mathbf{U}\_{\ell,\pi} = \mu\_{\mathcal{S}-,\pi} - \mu\_{\mathcal{S}+,\pi},
\tag{3.106b}
$$

$$
\tau\_{U\_{\mathcal{Z},\pi}} \dot{\mathbf{U}}\_{\mathcal{Z},\pi} = \mu\_{\mathcal{Z}-,\pi} - \mu\_{\mathcal{Z}+,\pi},\tag{3.106c}
$$

$$
\tau\_{\mu\_{\text{g}-},\pi} \dot{\mu}\_{\text{g}-,\pi} = \langle \underline{\mathbf{p}}\_{\text{g},\pi} - \mathbf{p}\_{\text{g},\pi} \rangle\_{\mu\_{\text{g}-,\pi}}^{+},\tag{3.106d}
$$

$$
\tau\_{\mu\_{\rm g+,\pi}} \dot{\mu}\_{\rm g+,\pi} = \langle \mathbf{p}\_{\rm g,\pi} - \overline{\mathbf{p}}\_{\rm g,\pi} \rangle\_{\mu\_{\rm g+,\pi}}^{+},\tag{3.106e}
$$

$$
\tau\_{\mu\_{S-,\pi}} \dot{\mu}\_{S-,\pi} = \langle \underline{\mathbf{U}}\_{\mathbf{f},\pi} - \mathbf{U}\_{\mathbf{f},\pi} \rangle\_{\mu\_{S-,\pi}}^{+},\tag{3.106f}
$$

$$
\tau\_{\mu\_{\mathcal{S}+,\pi}} \dot{\mu}\_{\mathcal{S}+,\pi} = \langle \mathcal{U}\_{\mathbf{f},\pi} - \overline{\mathcal{U}}\_{\mathbf{f},\pi} \rangle\_{\mu\_{\mathcal{S}+,\pi}}^{+},\tag{3.106g}
$$

$$
\tau\_{\mu\_{\mathcal{Z}-,\pi}} \dot{\mu}\_{\mathcal{Z}-,\pi} = \langle \underline{\mathbf{U}}\_{\mathcal{Z},\pi} - \mathbf{U}\_{\mathcal{Z},\pi} \rangle\_{\mu\_{\mathcal{Z}-,\pi}}^{+},\tag{3.106h}
$$

$$
\tau\_{\mu\_{\mathcal{Z}+,\pi}} \dot{\mu}\_{\mathcal{Z}+,\pi} = \langle \mathbf{U}\_{\mathcal{Z},\pi} - \overline{\mathbf{U}}\_{\mathcal{Z},\pi} \rangle^{+}\_{\mu\_{\mathcal{Z}+,\pi}} \tag{3.106i}
$$

with suitable diagonal matrices τ (·) ≻ 0. The individual strictly positive diagonal elements in τ (·) are design parameters to be chosen by the individual WP. As outlined in Remark 3.28, premultiplication of the basic controller equations (3.100), (3.103) with an appropriate metric is helpful to adjust the convergence speed towards the equilibrium. The diagonal structure of τ (·) helps maintain a specific transient behavior for each primal or dual variable in (3.106) separately. In particular, the speed of the controller (along with possible overshoots) increases as ∥τ ∥ → 0. With regard to the practical application, each individual diagonal entry in τ must thus be sufficiently high to ensure that the respective actuator is capable of performing the

<sup>46</sup> cf. Definition A.21 and Lemma A.22 in Appendix A.5.

required work47. At the same time, it is reasonable to assume that no WP has an incentive to set τ (·) too high, since this would lead to an unnecessarily slow attainment of the own profit maximum. Besides these practical considerations, which are further discussed in Chapter 4, it is obvious that the specific choice of τ (·) does not alter the equilibrium of (3.106), cf. Remark 3.28.

The CCs' objective Pk(Λ<sup>⋆</sup> k ) = 0 from (3.90) can be accessed by means of the gradient flow Λ˙ <sup>k</sup> = −∇Pk(Λk), yielding

$$\dot{\Lambda}\_k = -\hat{\Phi}\_k - \sum\_{i \in \mathcal{V}\_{\mathcal{Z},k}} p\_{\ell,i} + \sum\_{\substack{i \in \mathcal{V}\_{\mathcal{Z},k} \\ i \notin \mathcal{V}\_{\mathcal{L}}}} p\_{\mathcal{g},i} \,. \tag{3.107}$$

The dynamics of the zonal price Λ<sup>k</sup> arise from the momentary active power balance of cell k through aggregation of all net power flows from cell k to neighboring cells. Illustratively, the price for power injection in cell k increases if the cell's momentary production is lower than its own demand caused by RPs and transmission losses, and vice versa. Yet, instead of aggregating all required information at a central point, (3.107) can be set up in a fully distributed manner by defining a *consensus protocol* via a cell-specific, weakly connected communication graph Gc,k = (VZ,k, Ec,k) <sup>48</sup>. In particular, let Dc,k denote the incidence matrix of Gc,k. Then with Lemma B.1 from Appendix B.2, (3.107) can be implemented in a distributed fashion by

$$
\tau\_{\lambda,k}\dot{\lambda}\_k = -\mathbf{p}\_{\mathbf{g},k} + \boldsymbol{\rho}\_k + \mathbf{p}\_{\ell,k} - \mathbf{D}\_{\mathbf{c},k}\boldsymbol{\nu}\_k,\tag{3.108a}
$$

$$
\tau\_{\nu,k}\dot{\nu}\_k = \left(\mathcal{D}\_{c,k}\right)^\dagger \lambda\_k,\tag{3.108b}
$$

where λ<sup>k</sup> = col<sup>i</sup>∈VZ,k {λi} and ρ<sup>k</sup> = col<sup>i</sup>∈VZ,k {ρi} with

$$\rho\_i = G\_{ii} U\_i^2 + \sum\_{j \in \mathcal{N}\_i} G\_{ij} U\_i U\_j \cos(\vartheta\_{ij}). \tag{3.109}$$

As above, τ λ,k and τ ν,k are real diagonal matrices of appropriate sizes with positive diagonal elements containing design parameters of the corresponding local controller. τ λ,k allows to adjust how fast-responding momentary power imbalances reflect in the prices, while the convergence speed of nodal prices λ<sup>k</sup> towards a single cell-specific price Λ<sup>k</sup> can be adjusted via τ ν,k.

#### **3.4.3 Analysis of the Closed-Loop System**

By virtue of using projected saddle-point flow, the initial state of the closed-loop system is not required to be located within the feasible set. However, in order to be able to attain an equilibrium, the following feasibility assumption must be made for the collection of optimization problems of WPs:

<sup>47</sup> Within the controller equations (3.106), the controller parameters τ <sup>g</sup>,π for active power injection are set to a comparatively high (i.e. "slow") value, whereby further differentiation of τ <sup>g</sup>,π in (3.106a) depends of the type of governing systems: While steam turbines of bulk CPPs require long ramp-up times and thus a very large τg,(·) , the respective parameters for inverter-interfaced DERs may be set to rather moderate values. By contrast, the parameters <sup>τ</sup> <sup>µ</sup>(·),π for the gradient ascent of dual variables can always be adjusted to a very fast value, as the dual variables correspond to "virtual" quantities that do not have to be generated physically.

<sup>48</sup> see Section B.2 in the appendix for some basic derivations.

**Assumption 3.32.** *The upper and lower bounds in* (3.83b)*–*(3.83c) *are chosen sufficiently loose such that there always exists a feasible solution that satisfies the constraints for all WPs* π ∈ W*.*

Let d <sup>⋆</sup> = col{<sup>q</sup> ⋆ ℓ , p ⋆ ℓ } be a given, constant disturbance input vector. We assume that the state vector x<sup>p</sup> of the physical plant system (3.49) fulfills the following regularity condition:

**Assumption 3.33 ( [TBD16, SDv17b]).** *The Hessian of* Hp(xp) *is positive definite at steady state* x ⋆ p *.*

Assumption 3.33, which is commonly known in energy-related literature as *existence of highvoltage solution* [DMSP17], can be ensured by satisfying e.g. the (relatively mild) condition presented in [SDv17a, Proposition 9] and originally derived in [DM16, Proposition 1]. The condition is satisfied if all voltage angle differences ϑij are small and if all generator reactances Xd,i − X′ <sup>d</sup>,i are small compared to the shunt susceptances b<sup>i</sup> .

The simultaneous execution of the optimization schemes (3.106) for WPs π ∈ W and (3.108) for CCs k ∈ Z leads to a superposition of the derived control equations (3.106) and (3.108). Consequently, the resulting closed-loop system consists of the physical system (3.39) along with the controller equations

$$
\tau\_{\rm g} \dot{p}\_{\rm g} = -\nabla \mathbf{C}(p\_{\rm g}) + \lambda - \omega\_{\rm g} + \mu\_{\rm g-} - \mu\_{\rm g+}, \tag{3.110a}
$$

$$
\tau\_{U\_\mathbf{f}} \dot{U}\_\mathbf{f} = \mu\_{\mathcal{S}-} - \mu\_{\mathcal{S}+},\tag{3.110b}
$$

$$
\tau\_{\mathbf{U}\_{\mathcal{T}}} \dot{\mathbf{U}}\_{\mathcal{T}} = \mu\_{\mathcal{T}-} - \mu\_{\mathcal{T}+},\tag{3.110c}
$$

$$
\tau\_{\mu\_{\rm g-}} \dot{\mu}\_{\rm g-} = \langle \underline{\mathbf{p}}\_{\rm g} - \mathbf{p}\_{\rm g} \rangle\_{\mu\_{\rm g-}}^{+},\tag{3.110d}
$$

$$
\tau\_{\mu\_{\rm g+}} \dot{\mu}\_{\rm g+} = \langle \mathbf{p}\_{\rm g} - \overline{\mathbf{p}}\_{\rm g} \rangle\_{\mu\_{\rm g+}}^{+},\tag{3.110e}
$$

$$
\tau\_{\mu\_{\mathcal{S}-}} \dot{\mu}\_{\mathcal{S}-} = \langle \underline{\mathbf{U}}\_{\text{f}} - \mathbf{U}\_{\text{f}} \rangle\_{\mu\_{\mathcal{S}-}}^{+},\tag{3.110f}
$$

$$
\tau\_{\mu\_{\mathcal{S}+}} \dot{\mu}\_{\mathcal{S}+} = \langle \mathcal{U}\_{\text{f}} - \overline{\mathcal{U}}\_{\text{f}} \rangle\_{\mu\_{\mathcal{S}+}}^{+},\tag{3.110g}
$$

$$
\tau\_{\mu\_{\mathcal{T}-}} \dot{\mu}\_{\mathcal{T}-} = \langle \underline{\mathbf{U}}\_{\mathcal{T}} - \mathbf{U}\_{\mathcal{T}} \rangle\_{\mu\_{\mathcal{T}-}}^{+},\tag{3.110h}
$$

$$
\tau\_{\mu\_{\mathcal{T}+}} \dot{\mu}\_{\mathcal{T}+} = \langle \mathbf{U}\_{\mathcal{T}} - \overline{\mathbf{U}}\_{\mathcal{T}} \rangle\_{\mu\_{\mathcal{T}+}}^{+},\tag{3.110i}
$$

$$
\tau\_{\lambda}\dot{\lambda} = -\mathbf{p}\_{\text{g}} + \boldsymbol{\rho} + \mathbf{p}\_{\ell} - \mathbf{D}\_{c}\boldsymbol{\nu},\tag{3.110}
$$

$$
\tau\_{\nu}\dot{\nu} = \mathbf{D}\_{\mathbf{c}}^{\sf T}\boldsymbol{\lambda},\tag{3.110\mathbf{k}}
$$

where the parameters and variables in (3.110) are stacked vectors or diagonal matrices of appropriate sizes for all π ∈ W and k ∈ Z, e.g. τ <sup>g</sup> = diag<sup>π</sup>∈W{τ <sup>g</sup>,π}, D<sup>c</sup> = col<sup>k</sup>∈Z {Dc,k}, and C(p<sup>g</sup> ) equals the sum of all cost functions Cπ(pg,π) in (3.79).

**Remark 3.34.** *Note that* D<sup>c</sup> *has a block diagonal structure. Thus, the corresponding communication graph* G<sup>c</sup> *is disconnected and contains* n<sup>Z</sup> *connected subgraphs.* ♢

#### **Analysis of the Closed-Loop Equilibrium: Optimality and Zero-Frequency Deviation**

At first, we point out a key result concerning the zero-frequency deviation of the closed-loop system (3.39), (3.110):

**Lemma 3.35 (Zero Frequency Deviation).** *At each equilibrium of* (3.39)*,* (3.110)*, it holds that* ω ⋆ <sup>i</sup> = 0 *for all* i ∈ V*, i.e. each node is operating at the nominal frequency.*

*Proof.* Eq. (3.39a) can be written in vector-matrix notation as ϑ˙ = D<sup>T</sup> <sup>p</sup> ω, where D<sup>p</sup> denotes the incidence matrix of the physical network Gp. Since G<sup>p</sup> is a weakly connected graph, each equilibrium ω <sup>⋆</sup> with **0** = D<sup>T</sup> <sup>p</sup>ω ⋆ fulfills ω <sup>⋆</sup> = ω ⋆ · **1**, i.e. the nodal frequencies are synchronized at steady state. Since **1** <sup>T</sup>ρ = Φ and **1** <sup>T</sup>D<sup>c</sup> = **0** <sup>T</sup>, left-multiplying (3.110j) by **1** T and inserting the equilibrium values implies

$$0 = -\sum\_{i \in \mathcal{V}\_{\mathbf{g}}} p\_{\mathbf{g},i}^{\star} + \sum\_{i \in \mathcal{V}} p\_{\ell,i} + \Phi. \tag{3.111}$$

Due to the fact that Φ = P <sup>i</sup>∈V <sup>p</sup><sup>i</sup> , a summation of all equations in (3.39b) and (3.39d) and insertion of the equilibrium values leads to

$$0 = -\sum\_{i \in \mathcal{V}\_{\mathbf{g}}} A\_i \omega\_i^\star + \sum\_{i \in \mathcal{V}\_{\mathbf{g}}} p\_{\mathbf{g},i}^\star - \sum\_{i \in \mathcal{V}} p\_{\ell,i} - \Phi. \tag{3.112}$$

Comparison between (3.111) and (3.112) yields

$$0 = -\sum\_{i \in \mathcal{V}\_{\mathfrak{g}}} A\_i \omega\_i^\star = -\omega^\star \cdot \sum\_{i \in \mathcal{V}\_{\mathfrak{g}}} A\_i. \tag{3.113}$$

Finally, since A<sup>i</sup> > 0 holds by definition, it follows that ω ⋆ is zero.

**Remark 3.36.** *As the control equations of WPs are absent in the proof of Lemma 3.35, zero frequency deviation of the closed-loop system can still be maintained in the presence of* clock drifts *[Sch15, Remark 5.2.11] of the frequency measurement units of WPs (where the real nodal frequency* ωi(t) *in* (3.79) *is pertubed by an additive disturbance* +∆ωi*).* ♢

The next lemma states that in each equilibrium of (3.110) marginal costs are equal at each node in a cell whenever the power injection constraints (3.80) are not binding.

**Lemma 3.37 (Equal Marginal Costs).** *Let* x <sup>⋆</sup> *denote an equilibrium of* (3.110) *and let* k ∈ Z*. Then, at any node* i, j ∈ Vg,k *with* p ⋆ <sup>g</sup>,i ∈ (p <sup>g</sup>,i, pg,i) *and* p ⋆ <sup>g</sup>,j ∈ (p <sup>g</sup>,j , pg,j )*, the marginal costs are equal, i.e.* ∇Cg,i(p ⋆ <sup>g</sup>,i) = ∇Cg,j (p ⋆ <sup>g</sup>,j )*.*

*Proof.* With (3.110k), it holds that **0** = D<sup>T</sup> <sup>c</sup> λ ⋆ at steady state. Since D<sup>c</sup> = diag{Dc,k} and since each Dc,k is the incidence matrix of a weakly connected graph, this implies that each row of λ ⋆ <sup>k</sup> has the same value, i.e. λ ⋆ <sup>k</sup> =: Λ<sup>k</sup> · <sup>1</sup>. Moreover, from (3.110d) and (3.110e), we get

$$\langle \underline{p}\_{\mathbf{g},i} - p\_{\mathbf{g},i}^{\star} \rangle\_{\mu\_{\mathbf{g}-,i}}^{+} = \langle p\_{\mathbf{g},i}^{\star} - \overline{p}\_{\mathbf{g},i} \rangle\_{\mu\_{\mathbf{g}+,i}}^{+} = 0. \tag{3.114}$$

If p ⋆ <sup>g</sup>,i ∈ (p <sup>g</sup>,i, pg,i), then both brackets in (3.114) contain negative arguments, which implies that µg−,i = µg+,i = 0 (cf (A.1)). Now, with ω <sup>⋆</sup> = **0** from Lemma 3.35, it follows via (3.110a) that ∇Cg,i(p ⋆ <sup>g</sup>,i) = λ ⋆ <sup>i</sup> = Λk. Hence, all marginal costs are equal to the cell-specific value Λk.

A trivial implication of Lemma 3.37 is that the special case of *quadratic cost functions*

$$\mathfrak{C}\_{\mathbf{g},i}(p\_{\mathbf{g},i}) = \frac{1}{2} \frac{p\_{\mathbf{g},i}^2}{\varpi\_i}, \quad \qquad \qquad \qquad \varpi\_i > 0, \quad i \in \mathcal{V}\_{\mathbf{g},k}, \tag{3.115}$$

yields *active power sharing* in cell k ∈ Z, which is defined as follows:

**Definition 3.38 (Active Power Sharing [SOA**<sup>+</sup>**14, p. 2465]).** *Let* <sup>ϖ</sup><sup>k</sup> = col<sup>l</sup>∈Vg,k {ϖk,l} > **0** *denote a vector of constant weighting factors. Then* active power sharing *applies in cell* k ∈ Z *if*

$$\frac{p\_{\mathbf{g},i}^{\star}}{\varpi\_{k,i}} = \frac{p\_{\mathbf{g},j}^{\star}}{\varpi\_{k,j}}, \quad \qquad \qquad \qquad i, j \in \mathcal{V}\_{\mathbf{g},k} \tag{3.116}$$

*holds.*

#### **Shifted Passivity and Stability**

Since the physical plant system is amenable to a port-Hamiltonian representation (3.49), it is convenient to evaluate the closed-loop stability based on passivity arguments. As discussed in [van17, Section 4.4], PHSs can be decomposed into an interconnection of smaller subsystems to simplify passivity analysis. For the sake of clarity, we thus conceptually segregate (3.110) into a *frequency controller* (3.110a), (3.110d)–(3.110e), (3.110j)–(3.110k) and a *voltage controller* (3.110b)–(3.110c), (3.110f)–(3.110i) and analyze each of these components separately.

**Lemma 3.39.** *The physical plant system* (3.49) *with output* y<sup>p</sup> = G<sup>T</sup> <sup>p</sup> z<sup>p</sup> *is (locally) shifted passive with respect to* x ⋆ p *, if*

$$\left[\left(\mathbf{z}\_{\mathrm{p}} - \mathbf{z}\_{\mathrm{p}}^{\star}\right)^{\mathrm{T}}\left[\mathcal{R}(\mathbf{z}\_{\mathrm{p}}) - \mathcal{R}(\mathbf{z}\_{\mathrm{p}}^{\star})\right] \geq 0\tag{3.117}$$

*with* R(xp) = Rpz<sup>p</sup> + r<sup>p</sup> *holds.*

*Proof.* The physical plant system is (locally) shifted passive with respect to x ⋆ p , if the shifted plant Hamiltonian [van17, p. 136] <sup>H</sup>e<sup>p</sup>(xe<sup>p</sup>) := <sup>H</sup>p(xp) <sup>−</sup> (xe<sup>p</sup>) <sup>T</sup> <sup>∇</sup>Hp(<sup>x</sup> ⋆ p ) − Hp(x ⋆ p ) with <sup>x</sup>e<sup>p</sup> <sup>=</sup> <sup>x</sup><sup>p</sup> <sup>−</sup> <sup>x</sup> ⋆ p satisfies <sup>H</sup>e<sup>p</sup>(xe<sup>p</sup>) <sup>≻</sup> <sup>0</sup> and

$$
\dot{\tilde{H}}\_{\rm p}(\tilde{x}\_{\rm p}) \le \tilde{y}\_{\rm p}^{\rm T} \tilde{u}\_{\rm p} \tag{3.118}
$$

with y<sup>p</sup> = G<sup>T</sup> <sup>p</sup> zp.

The positive definiteness condition is satisfied locally due to Assumption 3.33. To investigate (3.118), let the disturbance input d ⋆ be constant in time and define R(xp) = Rp∇Hp(xp)+rp. Then (3.49) can be written as

$$E\dot{x}\_{\rm p} = J\_{\rm p}\nabla H\_{\rm p}(x\_{\rm p}) - \mathcal{R}(x\_{\rm p}) + G\_{\rm p}u\_{\rm p} + G\_{\rm d}d^{\star} \tag{3.119}$$

with each equilibrium x ⋆ p fulfilling

$$0 = \mathcal{J}\_\mathbf{p} \nabla H\_\mathbf{p}(x\_\mathbf{p}^\star) - \mathcal{R}(x\_\mathbf{p}^\star) + \mathcal{G}\_\mathbf{p} u\_\mathbf{p}^\star + \mathcal{G}\_\mathbf{d} d^\star \tag{3.120}$$

for a constant input vector u ⋆ p . Since the shifted Hamiltonian <sup>H</sup>e<sup>p</sup>(xe<sup>p</sup>) is positive definite with minimum <sup>H</sup>e<sup>p</sup>(**0**) = 0, the shifted plant dynamics, i.e. (3.119) minus (3.120), can be expressed in terms of <sup>H</sup>e<sup>p</sup>(xe<sup>p</sup>) as follows:

$$E\dot{\tilde{x}}\_{\rm p}^{\dot{\cdot}} = J\_{\rm p} \nabla \tilde{H}\_{\rm p}(x\_{\rm p}) - \left[\mathcal{R}(x\_{\rm p}) - \mathcal{R}(x\_{\rm p}^{\star})\right] + G\_{\rm p} \tilde{u}\_{\rm p}.\tag{3.121}$$

With ˙H<sup>e</sup> <sup>p</sup>(xe<sup>p</sup>) = <sup>z</sup><sup>e</sup> T <sup>p</sup><sup>E</sup> ˙xe<sup>p</sup> and bearing in mind that <sup>J</sup><sup>p</sup> is skew-symmetric, the passivity condition (3.118) is equivalent to (3.117).

**Corollary 3.40.** *If the physical plant system* (3.49) *with output* y<sup>p</sup> = G<sup>T</sup> <sup>p</sup> z<sup>p</sup> *is lossless, then it is (locally) shifted passive with respect to* x ⋆ p *.*

*Proof.* For lossless grids, it holds that r<sup>p</sup> = **0**. Hence, (3.117) is always fulfilled due to the fact that R<sup>p</sup> ≽ 0. Accordingly, (local) shifted passivity of (3.49) follows from Lemma 3.39.

Next, we examine the shifted passivity of the frequency controller (3.110a), (3.110d)–(3.110e), (3.110j)–(3.110k):

**Lemma 3.41.** *The frequency controller* (3.110a)*,* (3.110d)*–*(3.110e)*,* (3.110j)*–*(3.110k) *with input* u<sup>c</sup> = −ω<sup>g</sup> *and output* yu<sup>1</sup> = p<sup>g</sup> *is (locally) shifted passive, if*

$$(\mathbf{p}\_{\rm g} - \mathbf{p}\_{\rm g}^{\star})^{\rm T} (\nabla \mathbf{C}(\mathbf{p}\_{\rm g}) - \nabla \mathbf{C}(\mathbf{p}\_{\rm g}^{\star})) \ge (\lambda - \lambda^{\star})^{\rm T} (\boldsymbol{\rho} - \boldsymbol{\rho}^{\star}).\tag{3.122}$$

*Proof.* By defining the frequency controller state ξ<sup>1</sup> = col{τ <sup>g</sup>p<sup>g</sup> , τ <sup>µ</sup>g<sup>−</sup> µg−, τ <sup>µ</sup>g+ µg+, τ <sup>λ</sup>λ, τ <sup>ν</sup>ν} and the frequency controller Hamiltonian H1(ξ<sup>1</sup> ) = <sup>1</sup> 2 ξ T 1 τ −1 <sup>c</sup> ξ<sup>1</sup> with τ <sup>c</sup> = diag{τ <sup>g</sup>, τ <sup>µ</sup>g<sup>−</sup> , τ <sup>µ</sup>g+ , τ <sup>λ</sup>, τ <sup>ν</sup>} ≻ 0, (3.110a), (3.110d)–(3.110e), (3.110j)–(3.110k) can be written in port-Hamiltonian form

$$\dot{\xi}\_1 = J\_1 \nabla H\_1(\xi\_1) - r\_1 + G\_{u1} u\_c + G\_{d1} p\_\ell,\tag{3.123}$$

$$y\_{u1} = \mathbf{G}\_{u1}^{\mathsf{T}} \boldsymbol{\zeta}\_1,\tag{3.124}$$

$$\boldsymbol{y}\_{d1} = \boldsymbol{G}\_{d1}^{\top} \boldsymbol{\zeta}\_{1},\tag{3.125}$$

where ζ<sup>1</sup> = ∇H1(ξ<sup>1</sup> ) = col{p<sup>g</sup> , µ<sup>g</sup>−, µg+,λ, ν} and

$$J\_{1} = \begin{bmatrix} \mathbf{0} & \mathbf{0} & \mathbf{0} & \hat{I}\_{g} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} & \mathbf{0} & \mathbf{0} & \mathbf{0} \\ \mathbf{0} & \mathbf{0} & \mathbf{0} & \mathbf{0} & \mathbf{0} \\ -\mathbf{I}\_{g}^{\top} & \mathbf{0} & \mathbf{0} & \mathbf{0} & D\_{c} \\ \mathbf{0} & \mathbf{0} & \mathbf{0} & -\mathbf{D}\_{c}^{\top} & \mathbf{0} \end{bmatrix}, \qquad \boldsymbol{r}\_{1} = \begin{bmatrix} \nabla \mathbf{C}(\mathbf{p}\_{\mathbf{g}}) \\ -\langle \mathbf{p}\_{\mathbf{g}} - \mathbf{p}\_{\mathbf{g}} \rangle\_{\mu\_{\mathbf{g},\mathbf{}}}^{+} \\ -\langle \mathbf{p}\_{\mathbf{g}} - \overline{\mathbf{p}}\_{\mathbf{g}} \rangle\_{\mu\_{\mathbf{g},\mathbf{}}}^{+} \\ -\boldsymbol{\rho} \\ \mathbf{0} \end{bmatrix}, \tag{3.126}$$
 
$$\mathbf{G}\_{u1} = \begin{bmatrix} I \\ \mathbf{0} \\ \mathbf{0} \\ \mathbf{0} \\ \mathbf{0} \end{bmatrix}, \qquad \qquad \mathbf{G}\_{d1} = \begin{bmatrix} \mathbf{0} \\ \mathbf{0} \\ \mathbf{0} \\ \mathbf{I} \\ \mathbf{0} \end{bmatrix}. \tag{3.127}$$

With the shifted controller Hamiltonian

$$
\tilde{H}\_1(\tilde{\xi}\_1) := H\_1(\xi\_1) - (\tilde{\xi}\_1)^\mathsf{T} \nabla H\_1(\xi\_1^\star) - H\_1(\xi\_1^\star), \tag{3.128}
$$

the shifted controller co-state <sup>ζ</sup><sup>e</sup> 1 equals

$$
\widetilde{\boldsymbol{\xi}}\_{1} = \nabla \widetilde{H}\_{1}(\widetilde{\boldsymbol{\xi}}\_{1}) = \nabla H\_{1}(\boldsymbol{\xi}\_{1}) - \nabla H\_{1}(\boldsymbol{\xi}\_{1}^{\star}) = \boldsymbol{\xi}\_{1} - \boldsymbol{\xi}\_{1}^{\star}.\tag{3.129}
$$

Since the disturbance input p<sup>ℓ</sup> is assumed to be constant, (3.123) can be expressed in shifted coordinates as

$$\dot{\tilde{\xi}}\_{1} = \mathcal{J}\_{1}\nabla\tilde{H}\_{1}(\tilde{\xi}\_{1}) - [r\_{1}(\xi\_{1}) - r\_{1}(\xi\_{1}^{\star})] + \mathcal{G}\_{u1}\tilde{u}\_{c},\tag{3.130}$$

$$
\tilde{\boldsymbol{y}}\_{u1} = \mathbf{G}\_{u1}^{\mathsf{T}} \ddot{\boldsymbol{\zeta}}\_{1}.\tag{3.131}
$$

Due to strict convexity of H1(ξ<sup>1</sup> ), positive definiteness <sup>H</sup>e<sup>1</sup>(eξ<sup>1</sup> ) ≻ 0 is always satisfied. Hence the frequency controller is (locally) shifted passive if

$$
\check{\tilde{H}}\_1(\tilde{\xi}\_1) = \left(\nabla \tilde{H}\_1(\tilde{\xi}\_1)\right)^\mathsf{T} \dot{\tilde{\xi}}\_1 = \tilde{\xi}\_1^\mathsf{T} [r\_1(\xi\_1) - r\_1(\xi\_1^\star)] + \tilde{\zeta}\_1^\mathsf{T} G\_{u1} \tilde{u}\_c \le \tilde{y}\_{u1}^\mathsf{T} \tilde{u}\_c. \tag{3.132}
$$

By inserting (3.124), we get

$$\begin{split} 0 & \stackrel{\circ}{0} \stackrel{\circ}{\sim} [r\_1(\xi\_1) - r\_1(\xi\_1^\*)] = -(\mathbf{p}\_\mathbf{g} - \mathbf{p}\_\mathbf{g}^\*)^\mathsf{T} (\nabla \mathbf{C}(\mathbf{p}\_\mathbf{g}) - \nabla \mathbf{C}(\mathbf{p}\_\mathbf{g}^\*)) \\ & + \tilde{\mathbf{p}}\_\mathbf{g}^\mathsf{T} \tilde{\mu}\_\mathbf{g} - \tilde{\mathbf{p}}\_\mathbf{g}^\mathsf{T} \tilde{\mu}\_\mathbf{g} + \tilde{\mu}\_\mathbf{g}^\mathsf{T} \langle \underline{\mathbf{p}}\_\mathbf{g} - \mathbf{p}\_\mathbf{g} \rangle\_{\mu\_\mathbf{g}-}^+ + \tilde{\mu}\_\mathbf{g+}^\mathsf{T} \langle \mathbf{p}\_\mathbf{g} - \overline{\mathbf{p}}\_\mathbf{g} \rangle\_{\mu\_\mathbf{g+}}^+ \\ & + (\lambda - \lambda^\star)^\mathsf{T} (\rho - \rho^\star). \end{split} \tag{3.133}$$

In [SDv15, Proposition 3], it is shown that <sup>µ</sup><sup>e</sup> T ⟨g⟩ + <sup>µ</sup> <sup>≤</sup> <sup>µ</sup><sup>e</sup> T <sup>g</sup> and <sup>µ</sup><sup>e</sup> T g <sup>⋆</sup> <sup>≤</sup> **<sup>0</sup>** hold for each convex function <sup>g</sup> = coli{gi}. Hence <sup>µ</sup><sup>e</sup> T <sup>g</sup>−⟨p<sup>g</sup> − p<sup>g</sup> ⟩ + µg<sup>−</sup> <sup>≤</sup> <sup>µ</sup><sup>e</sup> T <sup>g</sup>−(p<sup>g</sup> − p<sup>g</sup> ) = <sup>µ</sup><sup>e</sup> T <sup>g</sup>−(p<sup>g</sup> − p ⋆ <sup>g</sup> <sup>−</sup> <sup>p</sup>e<sup>g</sup> ) ≤ −µe T <sup>g</sup>−pe<sup>g</sup> . With the same procedure it can be calculated that <sup>µ</sup><sup>e</sup> T g+⟨p<sup>g</sup> − p<sup>g</sup> ⟩ + µg+ <sup>≤</sup> <sup>+</sup>µ<sup>e</sup> T g+pe<sup>g</sup> . Thus, the second line of (3.133) is less than or equal to zero, which leads to (3.122).

**Corollary 3.42.** *If the plant system* (3.49) *is lossless, then the frequency controller with input* u<sup>c</sup> = −ω<sup>g</sup> *and output* yu<sup>1</sup> = p<sup>g</sup> *is (locally) shifted passive.*

*Proof.* For lossless networks, it holds that ρ = **0**, hence (3.122) in Lemma 3.41 is always fulfilled due to the fact that C(p<sup>g</sup> ) is strictly convex by Assumption 3.12.

**Remark 3.43.** *Since the interconnection between frequency controller and physical plant system*

$$\boldsymbol{u}\_{c1} = -\boldsymbol{y}\_{p1} = -\boldsymbol{G}\_{p1}^{\top} \nabla H\_{\mathbf{p}}(\mathbf{x}\_{\mathbf{p}}) = -\omega\_{\mathbf{g}},\tag{3.134}$$

$$\boldsymbol{u}\_{p1} = \boldsymbol{y}\_{c1} = \boldsymbol{G}\_{u1}^{\top} \nabla H\_1(\boldsymbol{\xi}\_1) = \boldsymbol{p}\_g \tag{3.135}$$

*is power-preserving, i.e.* u T <sup>c</sup>1yc<sup>1</sup> + u T <sup>p</sup>1yp<sup>1</sup> = 0*, shifted passivity of the subsystems in terms of Lemmas 3.39 and 3.41 implies shifted passivity of the closed-loop system* (3.49)*,* (3.110a)*,* (3.110d)*–*(3.110e)*,* (3.110j)*–*(3.110k)*. In fact, the conditions stated in Lemmas 3.39 and 3.41 are only sufficient and not necessary, as an excess of passivity in one subsystem can compensate for the lack of passivity in an other subsystem, cf. [van17, p. 23].* ♢

Based on the previous conditions we now formulate a stability criterion for the lossy, physical plant system along with frequency and voltage controller:

**Theorem 3.44.** *Assume that the conditions of Lemmas 3.39 and 3.41 hold. For a constant input* d ⋆ *, let* (x ⋆ p , ξ ⋆ 1 , ξ ⋆ 2 ) *denote an equilibrium of* (3.49)*,* (3.110)*. Then there exists a neighborhood* B *around* (x ⋆ p , ξ ⋆ 1 , ξ ⋆ 2 ) *such that if* (xp, ξ<sup>1</sup> , ξ<sup>2</sup> ) ∈ B*, then the state asymptotically converges to* (x ⋆ p , ξ ⋆ 1 , ξ ⋆ 2 )*.*

*Proof.* With the voltage controller state ξ<sup>2</sup> = col{τ <sup>µ</sup>S− µS−, τ <sup>µ</sup>S<sup>+</sup> µS<sup>+</sup>, τ <sup>U</sup><sup>f</sup> U<sup>f</sup> , τ <sup>µ</sup>I− µI−, τ <sup>µ</sup>I<sup>+</sup> µI<sup>+</sup>, τ <sup>U</sup><sup>I</sup> UI}, let ξ ⋆ <sup>2</sup> denote an equilibrium of (3.110b)–(3.110c), (3.110f)–(3.110i) and define the Lyapunov function candidate <sup>V</sup>e<sup>2</sup>(eξ<sup>2</sup> ) = <sup>1</sup> 2 eξ T 2 τ −1 2 eξ T 2 , where τ <sup>2</sup> = diag{τ <sup>µ</sup>S− , τ <sup>µ</sup>S<sup>+</sup> , τ <sup>U</sup><sup>f</sup> , τ <sup>µ</sup>I− , τ <sup>µ</sup>I<sup>+</sup> , τ <sup>U</sup><sup>I</sup> } ≻ <sup>0</sup> and <sup>e</sup>ξ<sup>2</sup> <sup>=</sup> <sup>ξ</sup><sup>2</sup> <sup>−</sup> <sup>ξ</sup> ⋆ 2 . This allows to write out an overall Lyapunov function candidate <sup>V</sup>e(xe<sup>p</sup>, <sup>e</sup>ξ<sup>1</sup> , eξ2 ) for the overall closed-loop system as the sum of <sup>H</sup>e<sup>p</sup>, <sup>H</sup>e<sup>1</sup>, and <sup>V</sup>e<sup>2</sup>, i.e.

$$
\tilde{V}(\tilde{\mathbf{x}}\_{\rm p}, \tilde{\boldsymbol{\xi}}\_{1}, \tilde{\boldsymbol{\xi}}\_{2}) = \tilde{H}\_{\rm p}(\tilde{\mathbf{x}}\_{\rm p}) + \tilde{H}\_{1}(\tilde{\boldsymbol{\xi}}\_{1}) + \tilde{V}\_{2}(\tilde{\boldsymbol{\xi}}\_{2}) \succ 0. \tag{3.136}
$$

With respect to ˙ <sup>V</sup>e, we observe that

$$
\dot{\tilde{V}}(\tilde{x}\_{\rm p}, \tilde{\xi}\_{1}, \tilde{\xi}\_{2}) = \dot{\tilde{H}}\_{\rm p}(\tilde{x}\_{\rm p}) + \dot{\tilde{H}}\_{1}(\tilde{\xi}\_{1}) + \dot{\tilde{V}}\_{2}(\tilde{\xi}\_{2}), \tag{3.137}
$$

where the first summand of (3.137) equals

$$\begin{split} \overset{\star}{H}\_{\mathrm{p}}(\widetilde{\mathbf{z}}\_{\mathrm{p}}) &= (\nabla \widetilde{H}\_{\mathrm{p}}(\widetilde{\mathbf{z}}\_{\mathrm{p}})^{\sf T} \mathbf{J}\_{\mathrm{p}}(\nabla \widetilde{H}\_{\mathrm{p}}(\widetilde{\mathbf{z}}\_{\mathrm{p}})) - (\nabla \widetilde{H}\_{\mathrm{p}}(\widetilde{\mathbf{z}}\_{\mathrm{p}})^{\sf T} \left[\mathcal{R}(\mathbf{z}\_{\mathrm{p}}) - \mathcal{R}(\mathbf{z}\_{\mathrm{p}}^{\star})\right] \\ &= -(\mathbf{z}\_{\mathrm{p}} - \mathbf{z}\_{\mathrm{p}}^{\star})^{\sf T} \left[\mathcal{R}(\mathbf{z}\_{\mathrm{p}}) - \mathcal{R}(\mathbf{z}\_{\mathrm{p}}^{\star})\right] \end{split} \tag{3.138}$$

due to the skew-symmetry of Jp. The second summand of (3.137) is

$$\dot{\tilde{H}}\_1(\tilde{\xi}\_1) = -\tilde{\mathbf{p}}\_\mathbf{g}^\mathsf{T}(\nabla \mathbf{C}(\mathbf{p}\_\mathbf{g}) - \nabla \mathbf{C}(\mathbf{p}\_\mathbf{g}^\star)) + (\lambda - \lambda^\star)^\mathsf{T}(\rho - \rho^\star). \tag{3.139}$$

The third summand of (3.137) equals

$$
\begin{split}
\tilde{\boldsymbol{V}}\_{2}(\tilde{\boldsymbol{\xi}}\_{2}) &= \tilde{\boldsymbol{\mu}}\_{\mathcal{S}-}^{\mathsf{T}} \langle \underline{\boldsymbol{\Psi}} - \boldsymbol{U}\_{\operatorname{f}} \rangle\_{\boldsymbol{\mu}\_{\mathcal{S}-}}^{+} + \tilde{\boldsymbol{\mu}}\_{\mathcal{S}+}^{\mathsf{T}} \langle \boldsymbol{U}\_{\operatorname{f}} - \overline{\boldsymbol{\Psi}} \rangle\_{\boldsymbol{\mu}\_{\mathcal{S}+}}^{+} + \tilde{\boldsymbol{U}}\_{\operatorname{f}}^{\mathsf{T}} \tilde{\boldsymbol{\mu}}\_{\mathcal{S}-} - \tilde{\boldsymbol{U}}\_{\operatorname{f}}^{\mathsf{T}} \tilde{\boldsymbol{\mu}}\_{\mathcal{S}+} \\ &+ \tilde{\boldsymbol{\mu}}\_{\mathcal{T}-}^{\mathsf{T}} \langle \underline{\boldsymbol{U}} \underline{\boldsymbol{\mathcal{T}}} - \boldsymbol{U}\_{\operatorname{T}} \rangle\_{\boldsymbol{\mu}\_{\mathcal{T}-}}^{+} + \tilde{\boldsymbol{\mu}}\_{\mathcal{T}+}^{\mathsf{T}} \langle \boldsymbol{U}\_{\mathcal{T}} - \overline{\boldsymbol{U}}\_{\mathcal{T}} \rangle\_{\boldsymbol{\mu}\_{\mathcal{T}+}}^{+}.
\end{split}
\tag{3.140}
$$

In [SDv15, Proposition 3], it was shown that <sup>µ</sup><sup>e</sup> T ⟨g⟩ + <sup>µ</sup> <sup>≤</sup> <sup>µ</sup><sup>e</sup> T <sup>g</sup> and <sup>µ</sup><sup>e</sup> T g <sup>⋆</sup> <sup>≤</sup> <sup>0</sup> hold for each convex function <sup>g</sup> = coli{gi}. Hence <sup>µ</sup><sup>e</sup> T S−⟨Ψ − Uf⟩ + µS− <sup>≤</sup> <sup>µ</sup><sup>e</sup> T S−(<sup>Ψ</sup> <sup>−</sup> <sup>U</sup>f) = <sup>µ</sup><sup>e</sup> T S−(Ψ − U ⋆ <sup>f</sup> <sup>−</sup> <sup>U</sup><sup>e</sup> <sup>f</sup>) ≤ −µ<sup>e</sup> T S−U<sup>e</sup> <sup>f</sup> . With the same procedure it can be calculated for S+, I−, and I+ that the first two rows of (3.140) are less than or equal to zero and thus ˙ <sup>V</sup><sup>e</sup> <sup>2</sup>(eξ<sup>2</sup> ) ≤ 0. With the assumption from Lemma 3.39, the first row of (3.138) is ≤ 0 and with the assumption from Lemma 3.41, it also holds that ˙H<sup>e</sup> <sup>1</sup>(eξ<sup>1</sup> ) <sup>≤</sup> <sup>0</sup> in (3.139). Hence ˙ <sup>V</sup>e(xe<sup>p</sup>, <sup>e</sup>ξ<sup>1</sup> , eξ2 ) ≤ 0 in (3.137). Since <sup>V</sup><sup>e</sup> <sup>≻</sup> <sup>0</sup> and ˙ <sup>V</sup><sup>e</sup> <sup>≤</sup> <sup>0</sup>, the discontinuous closed-loop system (3.49), (3.110) fulfills the condition of [CMC16, Lemma 4.3]. According to [CMC16, Theorem 4.5], the limit set of each solution of (3.49), (3.110) starting in B is a singleton, hence (x ⋆ p , ξ ⋆ 1 , ξ ⋆ 2 ) is (locally) asymptotically stable [Kha02, Corollary 4.1].

#### **3.4.4 Summary and Illustrative Example**

In this section, we have introduced an optimization-based control scheme for WPs and CCs via projected saddle-point flow ensuring frequency and voltage regulation. While the latter can be ensured directly via local constraints (3.84b) and (3.84c), frequency regulation is achieved indirectly via the real-time price λ by incorporating ω in the profit functions of each WP<sup>49</sup> . The derived price-based control scheme provides a rigorous compliance with the unbundling paradigm: WPs as competitive players require only the nodal prices λ<sup>i</sup> transmitted by the local CC individually for each node. Apart from this communicated quantity, the decision-making of WPs is fully decentralized. Namely, there is no need for the WPs to reveal their own cost

<sup>49</sup> In contrast to this *native* implementation of the frequency control task, the optimization-based frequency control schemes presented e.g. in [TBD16, SDv17a, ZMLB18] artificially impose a frequency-dependent term as extra control input in addition to a saddle-point-based controller.

function C<sup>π</sup> to competitors or CCs, where dishonesty by the WPs would be likely anyway (see e.g. [TJ13, p. 562]). In addition, a distributed realization can be adopted for the control actions of CCs as an outcome of the reformulation (3.108). The resulting local controllers then only need to have knowledge about the local values <sup>p</sup>g,i <sup>−</sup> <sup>p</sup>ℓ,i and U⃗ i , voltages U⃗ <sup>j</sup> of direct neighbors j ∈ N<sup>i</sup> , as well as line parameters and communication variables νij of adjacent lines. Moreover, since no WP <sup>π</sup> ∈ W can unilaterally increase its profit<sup>50</sup> <sup>P</sup><sup>π</sup> whenever at steady-state pg,π = p ⋆ <sup>g</sup>,π, each p ⋆ g represents a Nash equilibrium of the multi-player game (3.83), cf. [BLMH+21].

Due to the port-Hamiltonian model of the physical plant system (3.49) and frequency controller (3.110a), (3.110d)–(3.110e), (3.110j)–(3.110k) as well as the nonnegativity property [SDv15, Proposition 3] which is exploited in the proofs of Lemma 3.41 and Theorem 3.44, closed-loop stability can easily be assessed by passivity arguments. Even if the control scheme is certainly intended for lossy networks, a strict mathematical proof that provides an a priori statement on closed-loop stability is only available for lossless networks. This is due to the fact that the conditions (3.117) and (3.122) can only be evaluated online, yet are always fulfilled in the lossless case. This result is in line with the concern that there exists no analytical energy-like function for "generic" power grids with nonzero transmission losses, which has been addressed by many authors over the last eight decades [Mag47, Nar84, KBP85, ASB01, SALB05, VT15]. In fact, energy functions which enable a Lyapunov-based stability analysis are only available for very specific academic test systems [BA03, GSAB05, OGA<sup>+</sup>05]. However, even for highly low-dimensional systems, there are counterexamples where all Lyapunov-like functions are not applicable [Nar84]. Yet an advantageous property of conditions (3.117) and (3.122) is their *modular* structure: If passivity could be verified for two smaller subsystems (3.49), their powerpreserving interconnection<sup>51</sup> always leads to a passive overall system, which potentially gives rise to a *plug-and-play* based approach to stability analysis, see [APM16, SP19, SMKH20] for more in-depth discussions. We illustrate some of the theoretical insights of Section 3.4 by the following academic example:

#### **Example 4**

*In this example, we investigate the frequency stability and active power sharing of the closedloop system resulting from the exemplary physical plant system from Fig. 3.8 along with the controller* (3.110)*. For the sake of brevity, we choose* <sup>G</sup> = (R/X)·<sup>B</sup> *with* (R/X) <sup>∈</sup> <sup>N</sup>∪ {0} *yet to be determined and set all power and voltage constraints equal to infinity. The units of all parameters are given in p.u. with* Sbase = 1 MVA *and* Ubase = 10 kV*. The parameter values of the plant system can be found in Appendix B.4. The controller parameters in* (3.110) *are all set to* τ <sup>c</sup> = 0.1 · I *and the cost functions are chosen to*

$$\mathbf{C}\_{\mathbf{g},i}(p\_{\mathbf{g},i}) = \frac{1}{2} \frac{1}{\varpi\_i} \cdot p\_{\mathbf{g},i}^2 \tag{3.141}$$

<sup>50</sup> Figuratively, this is the case since at equilibrium, the profit margins of all players are zero. Indeed, if they deviate from their strategy (p ⋆ <sup>g</sup>,π, U<sup>⋆</sup> f,π , U<sup>⋆</sup> <sup>I</sup>,π), the new strategy would either be infeasible with respect to the constraints (3.84b)–(3.84d) or they would be worse off as the additional cost ∇Cπ(pg,π) for power production exceeds the additional revenue ∇Rπ(pg,π).

<sup>51</sup> e.g. by coupling all boundary voltage phasors U⃗ b 1 , U⃗ b 2 and power injections p b ℓ,1 , p b ℓ,2 , q b ℓ,1 , q b ℓ,2 via U⃗ b <sup>1</sup> <sup>=</sup> <sup>U</sup>⃗ b 2 , p b ℓ,<sup>1</sup> = −p b ℓ,2 , and q b ℓ,<sup>1</sup> = −q b ℓ,2 , respectively.

*with weighting factors* ϖ<sup>1</sup> = 1*,* ϖ<sup>2</sup> = 1.1*,* ϖ<sup>3</sup> = 1.2*,* ϖ<sup>4</sup> = 1.3*, and* ϖ<sup>5</sup> = 1.4*. The network is initially in steady state with nominal frequency* Ωn,i = 50 Hz *and constant loads* pℓ,i*,* qℓ,i*. At* t = 10 s *and* t = 40 s*, a step load change of* ∆pℓ,i = +0.1*p.u. occurs at load nodes* 6 *and* 7*, respectively.*

**Figure 3.8:** Stylized representation of the exemplary physical plant system with n<sup>S</sup> = 2, n<sup>I</sup> = 3, and n<sup>L</sup> = 2. S nodes are depicted in black, I nodes are depicted in gray, and L nodes are depicted in white. Each bus is equipped with an uncontrollable load modeled by (uncontrollable) active and reactive power consumption.

*Fig. 3.9 shows that for* R/X ∈ {0, 1, 2}*, all nodal frequencies converge to the nominal frequency within* 10 s *after the load jumps. For* R/X = 3*, however, there is no synchronization of frequencies.*

**Figure 3.9:** Nodal frequencies for Example 4.

**Figure 3.10:** Active power injections for Example 4.

*For* R/X ∈ {0, 1, 2}*, the active power generations* pg,i *are equidistant for each post-fault equilibrium, i.e. active power sharing is maintained (see Fig. 3.10). As the* R/X *ratio increases,*

*also the final values of* p<sup>g</sup> *increase slightly. This can be traced back to the fact that the WPs have to compensate the increasingly high transmission losses. For* R/X = 3*, as already seen for the frequencies, there is no convergence to a final value* p ⋆ g *.*

**Figure 3.11:** Dissipations in the system for ascending R/X ratios. The system becomes unstable for R/X = 3.

*As apparent in Fig. 3.11, a negative (shifted) dissipation* (3.138) *can be observed for* R/X = 3*, while* (3.138) *is nonnegative for* R/X ∈ {0, 1, 2}*. The curve of* (3.138) *along with Figs. 3.9 and 3.10, suggests that the momentary dissipation rate is a useful tool to qualitatively assess the stability margin of the closed-loop system.*

*Now we reset* R/X *equal to* 1 *and assume that the measurement of* ω<sup>1</sup> *at node 1 constantly deviates from the actual frequency by* +0.1 Hz *to simulate a clock drift.*

*As depicted in Fig. 3.12, simulations show that the controller is robust in terms of clock drifts and is able to restore the nominal frequency* Ωn,i *(cf. Remark 3.36). However, active power sharing can no longer be achieved by node* 1*, which is in line with the findings from [Sch15, Remark 5.2.11] that clock drifts affect the ratios* (3.116) *of active power sharing. This can easily be verified by the fact that Lemma 3.37 concerning equal marginal costs is violated in the presence of clock drifts. However, as argued in [Sch15], the introduced quantitative errors in power sharing are negligible for most practical applications.*

**Remark 3.45.** *The fact that variations of* US,i *or* UI,i *typically come at no additional cost for the respective WP (cf. Remark 3.22) can be exploited by imposing additional responsibilities on* *the voltage controller* (3.110b)*–*(3.110c)*. For instance, the augmentation*

$$
\tau\_{U\_\mathbf{f}} \dot{U}\_\mathbf{f} = -\mathcal{U}\_\mathbf{f} + \mathcal{U}\_\mathbf{f}^{\text{set}} + \mu\_{\mathcal{S}-} - \mu\_{\mathcal{S}+},\tag{3.142a}
$$

$$
\boldsymbol{\tau}\_{\mathbf{U}\mathcal{I}} \dot{\mathbf{U}}\_{\mathcal{I}} = -\mathbf{U}\_{\mathcal{I}} + \mathbf{U}\_{\mathcal{I}}^{\text{set}} + \boldsymbol{\mu}\_{\mathcal{I}-} - \boldsymbol{\mu}\_{\mathcal{I}+} \tag{3.142b}
$$

*can be employed to follow a specific setpoint* U set <sup>f</sup> = coli∈V′ S {Ψi(U set i )}*,* U set <sup>I</sup> = coli∈V<sup>I</sup> {U set i }*. Alternatively, to also achieve* reactive power sharing *[SFO15, p. 60]*

$$\frac{q\_{\mathbf{g},i}^{\star}}{\chi\_{i}} = \frac{q\_{\mathbf{g},j}^{\star}}{\chi\_{j}}, \quad \quad \qquad \qquad \qquad i, j \in \mathcal{V}\_{\mathbf{g}} \tag{3.143}$$

*with* qg,i = qinj,i*,* qg,j = qinj,j *and* χ<sup>i</sup> , χ<sup>j</sup> > 0*, the distributed consensus-based control scheme from [Sch15, Section 5.3] can be applied<sup>52</sup> if the network* <sup>G</sup><sup>p</sup> *is mainly inductive (*<sup>G</sup> <sup>≈</sup> <sup>0</sup>*), yielding*

$$\tau\_{U\_{\mathbf{f},i}} \dot{U}\_{\mathbf{f},i} = -k\_i \sum\_{j \in \mathcal{N}\_i} \left( \frac{q\_{\mathbf{g},i}}{\chi\_i} - \frac{q\_{\mathbf{g},j}}{\chi\_j} \right) + \mu\_{\mathcal{S}-,i} - \mu\_{\mathcal{S}+,i}, \quad i \in \mathcal{V}\_{\mathcal{S}}', \quad k\_i > 0, \tag{3.144a}$$

$$\tau\_{U\mathbb{Z},i} \dot{U}\_{\mathbb{Z},i} = -k\_i \sum\_{j \in \mathcal{N}\_i} \left( \frac{q\_{\mathbb{g},i}}{\chi\_i} - \frac{q\_{\mathbb{g},j}}{\chi\_j} \right) + \mu\_{\mathbb{Z}-,i} - \mu\_{\mathbb{Z}+,i}, \quad i \in \mathcal{V}\_{\mathbb{Z}}, \quad k\_i > 0. \tag{3.144b}$$

*Note that even with one of these augmentations, steady-state compliance with the voltage amplitude constraints* (3.84c)*–*(3.84d) *is still preserved due to the projected gradient flow of the dual variables* (3.110f)*–*(3.110i)*.* ♢

## **3.5 Interconnection of Zonal Prices**

Lemma 3.37 has revealed that at each equilibrium of (3.110), nodal prices λ<sup>i</sup> , λ<sup>j</sup> ∈ VZ,k are equal to a common zonal price Λ<sup>k</sup> for each cell k ∈ Z. Incentives for increased or decreased power generation are thus imposed implicitly by means of differences in zonal prices. However, Remark 3.34 implies that there is no direct relationship between each individual Λ<sup>k</sup> to be adjusted. Therefore, in the following Subsection 3.5.1, specific couplings between CCs are introduced that enable such relationships. Subsequently, Subsection 3.5.2 reveals the connection of the resulting closed-loop equilibrium with a (notional) *centralized* optimization problem. On this basis, Subsection 3.5.3, which contains the main result of Section 3.5, evaluates the economic efficiency of this closed-loop equilibrium as a first milestone towards Contribution 2.

#### **3.5.1 Coupling of Zonal Prices**

To provide interdependencies between the zonal prices, additional constraints of the form

$$
\lambda\_i \stackrel{!}{=} \eta\_{ij} \cdot \lambda\_j,\tag{3.145}
$$

<sup>52</sup> For the sake of simplicity, we neglect the filter elements for measured active and reactive power injections pg,i, qg,i originally included in the controller equations of [Sch15, Section 5.3].

with an appropriate multiplier ηij > 0 can be imposed for pairs of nodes i, j ∈ V which are located in *different* cells. For this purpose, the equilibrium condition of (3.110k) along with the linear dependencies (3.145) can be written compactly as

$$\mathbb{0} = (\mathcal{D}\_{\mathbf{c}}^{+})^{\mathsf{T}} \boldsymbol{\lambda}. \tag{3.146}$$

Hence, the resulting controller equations (3.110j)–(3.110k) are given by

$$
\tau\_{\lambda}\dot{\lambda} = -\mathbf{p}\_{\text{g}} + \rho + \mathbf{p}\_{\ell} - \mathbf{D}\_{\text{c}}^{+}\nu,\tag{3.147a}
$$

$$
\tau\_{\nu}\dot{\nu} = (\mathcal{D}\_c^+)^\mathsf{T}\lambda.\tag{3.147b}
$$

With this notation, D<sup>+</sup> c can be interpreted as the incidence matrix of an extended, weighted communication graph G + <sup>c</sup> = (V,(Ec, E b c )), where E b c represents communication across cell boundaries. The weights of the edges of G + c are equal to ηij , if the two adjacent nodes belong to different cells, and equal to 1, if i and j are located in the same cell.

**Assumption 3.46.** *The multipliers* ηij *are chosen in a* feasible *sense such that* dim ker(D<sup>+</sup> c ) T > 0*. Practically speaking, we assume that there exists at least one* λ > **0** *fulfilling* (3.146)*.*

**Remark 3.47.** *Assumption 3.46 is fullfilled if and only if* log{η} ∈ im(D<sup>ˆ</sup> <sup>c</sup>)*. In this case, each cell* k ∈ Z *can be characterized with a specific* participation factor κ<sup>k</sup> > 0 *such that each* ηij *is calculated by* ηij = κ<sup>k</sup><sup>1</sup> /κ<sup>k</sup><sup>2</sup> *if* i *is located in cell* k<sup>1</sup> *and* j *is located in cell* k2*.* ♢

**Lemma 3.48 (Connectivity of Zonal Prices).** *If* G + c *is weakly connected, then*

$$\frac{\lambda\_1}{\kappa\_{k\_1}} = \frac{\lambda\_2}{\kappa\_{k\_2}} = \dots = \frac{\lambda\_n}{\kappa\_{k\_n}} =: \Lambda^\circ,\tag{3.148}$$

*where* κ<sup>k</sup><sup>i</sup> *denotes the corresponding participation factor belonging to the cell* k *where node* i ∈ V *is located.*

*Proof.* Define the auxiliary matrices K<sup>1</sup> = diagij∈(Ec,E<sup>b</sup> c ){ηij} ≻ 0 and K<sup>2</sup> = diag<sup>i</sup>∈V {κ<sup>k</sup><sup>i</sup> } ≻ 0. Then, (D<sup>+</sup> c ) <sup>⊤</sup>λ = **0** is equivalent to

$$\underbrace{\mathbf{K}\_{1}^{-1}\mathbb{0}}\_{\mathbb{0}} = \mathbf{K}\_{1}^{-1}(\mathcal{D}\_{c}^{+})^{\mathsf{T}}\lambda = \underbrace{\mathbf{K}\_{1}^{-1}(\mathcal{D}\_{c}^{+})^{\mathsf{T}}\mathbf{K}\_{2}\mathbf{K}\_{2}^{-1}\lambda}\_{=:(\mathcal{D}\_{c}^{\diamond})^{\mathsf{T}}}\lambda.\tag{3.149}$$

Inserting the definition of K<sup>1</sup> into (3.149) with ηij = κ<sup>k</sup><sup>1</sup> /κ<sup>k</sup><sup>2</sup> reveals that D◦ c is the incidence matrix of a new communication graph G ◦ c , which is equivalent to G + <sup>c</sup> with all edge weights reset to 1. Thus each solution λ ◦ of the resulting equation **0** = (D◦ c ) <sup>T</sup>λ ◦ is of the form λ ◦ <sup>=</sup> <sup>1</sup> · const, i.e. each component of <sup>λ</sup> ◦ has the same value. Finally, since λ ◦ = K<sup>−</sup><sup>1</sup> <sup>2</sup> λ = col<sup>i</sup>∈V {λi/κ<sup>k</sup><sup>i</sup> }, this leads to (3.148).

**Remark 3.49.** *If* G + c *is not weakly connected, then* (3.148) *holds separately for all nodes in each weakly connected subgraph in* G + c *.* ♢

**Remark 3.50.** *The uniform price* Λ ◦ *resulting when* κ<sup>k</sup> = 1 *for all* k ∈ Z *refers to the social optimum (cf. footnote 35), yielding a minimization of the overall costs of WPs for power generation as well as equal revenues for the same amount of active power generation irrespective of where it is located. Accordingly,* κ<sup>k</sup> *describes the multiplicity of the zonal price* Λ<sup>k</sup> *compared to the uniform price* Λ ◦ *.* ♢

In the following, we will discuss to what extent κ = col<sup>k</sup>∈Z {κk} can be applied to modify the impact of specific cells on the overall network.

#### **3.5.2 Comparison with Centralized Optimization**

Conceptions of an *overall utility* within competitive scenarios are strongly related to the idea of global efficiency, which was alluded to in Subsection 3.3.1. The information about which solution is considered "socially preferable" to which other solution can thus be expressed by means of welfare functions (cf. Appendix A.6). In this context, the next theorem reveals an interesting connection between the multiple optimization problems of WPs and CCs and a notional centralized optimization problem with a modified utilitarian welfare function which is a *weighted sum* of the individual WPs' profits:

**Theorem 3.51 (Equivalence to Centralized Optimization).** *Define the centralized optimization problem*

$$\max\_{\mathbf{p}\_{\mathbf{g}}, \mathbf{U}\_{\mathbf{f}}, \mathbf{U}\_{\mathbf{f}}} \mathbf{P}^{\kappa} \tag{3.150a}$$

subject to Φ = X i∈V<sup>g</sup> pg,i − X i∈V pℓ,i, (3.150b)

$$
\underline{\mathbf{p}}\_{\mathbf{g}} \le \mathbf{p}\_{\mathbf{g}} \le \overline{\mathbf{p}}\_{\mathbf{g}},\tag{3.150c}
$$

$$
\underline{\mathbf{U}}\_{\mathbf{f}} \le \mathbf{U}\_{\mathbf{f}} \le \overline{\mathbf{U}}\_{\mathbf{f}},\tag{3.150d}
$$

$$
\underline{\mathbf{U}}\_{\mathcal{T}} \le \mathbf{U}\_{\mathcal{T}} \le \overline{\mathbf{U}}\_{\mathcal{T}},\tag{3.150e}
$$

*where*

$$\mathbf{P}^{\kappa} = -\sum\_{k \in \mathcal{Z}} \sum\_{i \in \mathcal{V}\_{Z,k}} \frac{1}{\kappa\_k} \cdot \mathbf{G}\_{\mathbf{g},i}(p\_{\mathbf{g},i}) - \sum\_{i \in \mathcal{V}\_{\mathbf{g}}} \omega\_i \cdot p\_{\mathbf{g},i}.\tag{3.151}$$

*Then each optimizer*(p ⋆ g , U ⋆ f , U ⋆ I ) *of* (3.150) *is an equilibrium of* (3.110a)*–*(3.110i)*,* (3.147a)*–* (3.147b) *and vice versa. If all* Cg,i(pg,i) *in* (3.151) *are strictly convex, then* (p ⋆ g , U ⋆ f , U ⋆ I ) *is unique.*

*Proof.* Constraint (3.150b) is equivalent to (cf. [SDv17b, p. 2615])

$$
\hat{D}\_{\rm c} \hat{\nu} = \mathbf{p}\_{\rm g} - \mathbf{p}\_{\ell} - \rho,\tag{3.152}
$$

where <sup>ρ</sup> = coli{ρi} as in (3.55)–(3.56) and <sup>D</sup><sup>ˆ</sup> <sup>c</sup> is the incidence matrix of a weakly connected communication graph. If we choose Dˆ <sup>c</sup> = D◦ c and define C κ (p<sup>g</sup> ) = P k∈Z P i∈V<sup>Z</sup> k 1 κ<sup>k</sup> · Ci(pg,i), then the Lagrangian of (3.150) becomes

$$\begin{split} \mathcal{L}^{\kappa} &= \mathsf{C}^{\kappa}(\mathsf{p}\_{\mathsf{g}}) + \left(\sum\_{i \in \mathsf{V}\_{\mathsf{g}}} \omega\_{i} \cdot p\_{\mathsf{g},i}\right) + \hat{\boldsymbol{\lambda}}^{\top}(\mathsf{D}\_{\mathsf{c}}^{\complement} \boldsymbol{\hat{\nu}} - \mathsf{p}\_{\mathsf{g}} + \mathsf{p}\_{\ell} + \boldsymbol{\rho}) \\ &+ \hat{\boldsymbol{\mu}}\_{\mathsf{g}-}^{\top}(\underline{\mathbf{p}}\_{\mathsf{g}} - \mathsf{p}\_{\mathsf{g}}) + \hat{\boldsymbol{\mu}}\_{\mathsf{g}+}^{\top}(\mathsf{p}\_{\mathsf{g}} - \overline{\mathsf{p}}\_{\mathsf{g}}) + \hat{\boldsymbol{\mu}}\_{\mathcal{S}-}^{\top}(\underline{\mathbf{U}}\_{\mathsf{f}} - \mathsf{U}\_{\mathsf{f}}) + \hat{\boldsymbol{\mu}}\_{\mathsf{S}+}^{\top}(\mathsf{U}\_{\mathsf{f}} - \overline{\mathsf{U}\_{\mathsf{f}}}) \\ &+ \hat{\boldsymbol{\mu}}\_{\mathsf{T}-}^{\top}(\underline{\mathbf{U}}\_{\mathsf{T}} - \mathsf{U}\_{\mathsf{T}}) + \hat{\boldsymbol{\mu}}\_{\mathsf{T}+}^{\top}(\mathsf{U}\_{\mathsf{T}} - \overline{\mathsf{U}\_{\mathsf{T}}}), \end{split} \tag{3.153}$$

where <sup>λ</sup><sup>ˆ</sup> denotes the Lagrange multiplier for equality constraint (3.152) and <sup>µ</sup>ˆ(·) are the Lagrange multipliers for the inequality constraints (3.150c)–(3.150e). Since (3.150) is convex and Slater's condition is fulfilled, the primal-dual optimizer of (3.150) is given by the KKT point (·) <sup>♯</sup> defined by

$$\mathbb{0} = -\nabla \mathbb{C}^{\kappa}(\mathbf{p}\_{\mathbf{g}}^{\sharp}) - \omega\_{i} + \hat{\boldsymbol{\lambda}}^{\sharp} + \hat{\boldsymbol{\mu}}\_{\mathbf{g}-}^{\sharp} - \hat{\boldsymbol{\mu}}\_{\mathbf{g}+}^{\sharp},\tag{3.154a}$$

$$\mathbb{0} = \hat{\mu}\_{\mathcal{S}-}^{\sharp} - \hat{\mu}\_{\mathcal{S}+}^{\sharp},\tag{3.154b}$$

$$\mathbb{0} = -\hat{\mu}\_{\mathbb{Z}-}^{\sharp} + \hat{\mu}\_{\mathbb{Z}+}^{\sharp},\tag{3.154c}$$

$$0 = (\hat{\mu}\_{\text{g}-}^{\sharp}) \circ (\underline{\mathbf{p}}\_{\text{g}} - \underline{\mathbf{p}}\_{\text{g}}^{\sharp}),\tag{3.154d}$$

$$\mathbb{0} = (\hat{\mu}\_{\rm g+}^{\sharp}) \circ (\mathbf{p}\_{\rm g}^{\sharp} - \overline{\mathbf{p}}\_{\rm g}),\tag{3.154e}$$

$$\mathbb{0} = (\hat{\mu}\_{\mathcal{S}-}^{\sharp}) \circ (\underline{\mathbf{U}}\_{\mathrm{f}} - \mathbf{U}\_{\mathrm{f}}^{\sharp}),\tag{3.154}$$

$$\mathbb{0} = (\hat{\mu}\_{\mathcal{S}+}^{\sharp}) \circ (\mathbf{U}\_{\mathrm{f}}^{\sharp} - \overline{\mathbf{U}}\_{\mathrm{f}}),\tag{3.154g}$$

$$0 = (\hat{\mu}\_{\mathcal{T}-}^{\sharp}) \circ (\underline{U}\_{\mathcal{T}} - \underline{U}\_{\mathcal{T}}^{\sharp}),\tag{3.154h}$$

$$\mathbb{0} = (\hat{\mu}\_{\mathcal{T}+}^{\sharp}) \circ (U\_{\mathcal{T}}^{\sharp} - \overline{U}\_{\mathcal{T}}),\tag{3.154i}$$

$$\mathbb{U} = -\mathbf{p}\_{\mathrm{g}}^{\sharp} + \boldsymbol{\rho} + \mathbf{p}\_{\ell} - \mathbf{D}\_{\mathrm{c}}^{\diamond} \boldsymbol{\hat{\nu}}^{\sharp},\tag{3.154}$$

$$\mathbf{0} = (\mathbf{D}\_c^\diamond)^\mathsf{T} \hat{\boldsymbol{\lambda}}^\sharp,\tag{3.154k}$$

$$0 \le \hat{\mu}\_{\mathcal{S}-}^{\sharp}, \hat{\mu}\_{\mathcal{S}+}^{\sharp}, \hat{\mu}\_{\mathcal{T}-}^{\sharp}, \hat{\mu}\_{\mathcal{T}+}^{\sharp}, \hat{\mu}\_{\mathcal{g}-}^{\sharp}, \hat{\mu}\_{\mathcal{g}+}^{\sharp}. \tag{3.154l}$$

Inserting the definition (3.149) in (3.154k) and comparison with the right-hand side of (3.110k) yields λˆ ♯ = K<sup>−</sup><sup>1</sup> <sup>2</sup> λ ⋆ . Hence with ∇C κ (p ♯ g ) = K<sup>−</sup><sup>1</sup> <sup>2</sup> ∇C(p ♯ g ) in (3.154a) and by comparison between (3.154a)–(3.154l) and (3.110a)–(3.110k) we get the equivalences p ♯ <sup>g</sup> = p ⋆ g , U ♯ <sup>f</sup> = U ⋆ f , U ♯ <sup>I</sup> = U ⋆ I , µˆ ♯ <sup>g</sup><sup>−</sup> <sup>=</sup> <sup>K</sup><sup>−</sup><sup>1</sup> <sup>2</sup> µ ⋆ <sup>g</sup>−, <sup>µ</sup><sup>ˆ</sup> ♯ g+ = K<sup>−</sup><sup>1</sup> <sup>2</sup> µ ⋆ g+, µˆ ♯ S− = µ ⋆ S−, <sup>µ</sup><sup>ˆ</sup> ♯ <sup>S</sup><sup>+</sup> = µ ⋆ <sup>S</sup><sup>+</sup>, <sup>µ</sup><sup>ˆ</sup> ♯ I− = µ ⋆ I−, µˆ ♯ <sup>I</sup><sup>+</sup> = µ ⋆ <sup>I</sup><sup>+</sup>. Moreover, the dual optimizer <sup>ν</sup> ⋆ from (3.110j) (using the notation (3.147a)) and the primal optimizer νˆ ♯ of (3.152) are connected via the relationship −D◦ cνˆ <sup>♯</sup> + D<sup>+</sup> <sup>c</sup> ν <sup>⋆</sup> = **0**.

From (3.149) it follows that D◦ <sup>c</sup> = K2D<sup>+</sup> <sup>c</sup> K−<sup>1</sup> <sup>1</sup> with K1, K<sup>2</sup> ≻ 0. Accordingly, for each equilibrium λ ⋆ of (3.110k) there exists a λˆ ♯ fulfilling (3.154k), and vice versa.

In summary, for each primal-dual optimizer of (3.150) there exists exactly one corresponding equilibrium of (3.110a)–(3.110i), (3.147a)–(3.147b). In particular,(p ♯ g , U ♯ f , U ♯ I ) = (p ⋆ g , U ⋆ f , U ⋆ I ). Since (3.150) is a convex optimization problem, convergence of the trajectory (p<sup>g</sup> (t), Uf(t), UI(t)) to (p ⋆ g , U ⋆ f , U ⋆ I ) is guaranteed (cf. [Ant94, Theorem 2]).

If the cost functions Cg,i(pg,i) are strictly convex, then P κ is strictly concave. Hence, the equilibrium (p ♯ g , U ♯ f , U ♯ I ) from centralized optimization and therewith the equilibrium (p ⋆ g , U ⋆ f , U ⋆ I ) from distributed optimization are unique.

**Remark 3.52.** *If* κ = 1*, then the objective function of* (3.150) *is*

$$\mathbb{P}^{\kappa} = \sum\_{\pi \in \mathcal{P}} \mathbb{P}\_{\pi} - \sum\_{k \in \mathcal{Z}} \mathbb{C}\_{k}^{\ell} + \sum\_{k \in \mathcal{Z}} \mathbb{P}\_{k} = -\sum\_{i \in \mathcal{V}} \mathbb{C}\_{i}(p\_{\mathbf{g},i}),\tag{3.155}$$

*and thus equal to the sum of all payoffs of WPs* (3.79)*, RPs* (3.88)*, and CCs* (3.90)*. Accordingly, the distributed controller* (3.110a)*–*(3.110i)*,* (3.147a)*–*(3.147b) *leads to a constrained minimization of the overall costs of power production.* ♢

#### **3.5.3 Analysis of Pareto Efficiency**

As stated in Theorem 3.51, the interaction of distributed WPs and CCs, subject to cell-specific pricing via κk, leads to the same equilibrium (p ⋆ g , U ⋆ f , U ⋆ I ) as if a centralized authority with full knowledge of the whole plant system would solve the optimization problem (3.150) with modified cost C κ (p<sup>g</sup> ), where the cumulative cost functions of each cell k ∈ Z are divided by κk. This equivalence reveals some further key properties of the developed distributed controller scheme that allows the evaluation of the Pareto efficiency of the equilibrium from a multi-objective perspective.

**Remark 3.53.** *With the same reasoning as in the proof of Lemma 3.35, it can be proven that for each equilibrium solution of* (3.39)*,* (3.154)*, it holds that* ω = **0***, i.e. zero frequency deviation is preserved.* ♢

In particular, it can be shown that each κ > **0** achieves an *efficient* allocation in the sense that for a given κ ′ > 0, there is no possibility to find a "better" κ ′′ ̸= κ ′ , κ ′′ > 0 such that at least one cell is at a lower cost and no cell is at a higher cost. This is formalized in the next theorem.

**Theorem 3.54 (Pareto Efficiency of** κ**).** *For each* k ∈ Z*, consider the optimization problems*

$$\max\_{\mathbf{p}\_{\mathbf{g},k}, \mathbf{U}\_{\ell,k}, U\_{\mathcal{X},k}} \mathbf{P}\_k^{\text{total}} \tag{3.156a}$$

$$\text{subject to} \qquad (\mathbf{D}\_{\mathbf{c},k}^{+})^{\mathsf{T}} \lambda\_k = \mathbf{0}, \tag{3.156b}$$

$$
\underline{\mathbf{p}}\_{\mathbf{g},k} \le \mathbf{p}\_{\mathbf{g},k} \le \overline{\mathbf{p}}\_{\mathbf{g},k},\tag{3.156c}
$$

$$
\underline{\mathbf{U}}\_{\mathbf{f},k} \le \mathbf{U}\_{\mathbf{f},k} \le \overline{\mathbf{U}}\_{\mathbf{f},k},\tag{3.156d}
$$

$$
\underline{\mathbf{U}}\_{\mathcal{T},k} \le \mathbf{U}\_{\mathcal{T},k} \le \overline{\mathbf{U}}\_{\mathcal{T},k},\tag{3.156e}
$$

*where*

$$\mathbf{P}\_{k}^{\text{total}} = \left(\sum\_{i \in \mathcal{V}\_{Z,k}} \sum\_{\pi \in \mathcal{P}} \mathbf{P}\_{\pi,i}\right) - \mathbf{C}\_{k}^{\ell} + \mathbf{P}\_{k} \tag{3.157}$$

*denotes the net profits of all network participants in cell* <sup>k</sup> ∈ Z *and where* <sup>D</sup><sup>+</sup> <sup>c</sup>,k *in* (3.156b) *contains all elements of* (3.146) *that belong to the nodes in cell* k*. Let* κ<sup>1</sup> > **0** *be fixed and let the corresponding value of* P total k *for an equilibrium of* (3.156) *with parameter* κ *set to* κ<sup>1</sup> *be denoted by* P total⋆ k (κ1)*. Then, there exists no other* dominating κ<sup>2</sup> > **0** *with* κ<sup>2</sup> ̸= κ<sup>1</sup> *such that the following two conditions hold:*

$$\forall \, k \in \mathbb{Z}: \qquad \qquad \mathsf{P}\_{k}^{\mathrm{total}\star}(\kappa\_{\mathbf{2}}) \ge \mathsf{P}\_{k}^{\mathrm{total}\star}(\kappa\_{\mathbf{1}}),\tag{3.158a}$$

$$\exists \ k \in \mathbb{Z}:\qquad\qquad\mathsf{P}\_{k}^{\mathrm{total}\star}(\mathsf{\kappa\_{2}}) > \mathsf{P}\_{k}^{\mathrm{total}\star}(\mathsf{\kappa\_{1}}).\tag{3.158b}$$

*Proof.* Taking into account that (3.156) are concave optimization problems and following the same lines as in the proof of Theorem 3.51, it can be shown that each KKT point of (3.156) is given by (3.154a)–(3.154l) and vice versa. For each κ > **0**, the equivalent problem (3.150) is a linear scalarization of (3.156) with positive weights 1/κ<sup>k</sup> > 0. With [ED18, Proposition 9], it is implied that for each κ ′ > **0**, the solution of (3.156) is a part of the Pareto front, hence there exists no κ ′ > **0** which is dominated by another κ ′′ > 0 in terms of (3.158).

The observation of Theorem 3.54 that any market equilibrium resulting from a given κ > **0** yields a Pareto efficient allocation for players π ∈ W is fully consistent with the first fundamental theorem of welfare theory (see Theorem A.25 in Appendix A.6). However, as opposed to the "copper-plate based" statements of Appendix A.6, Theorem 3.54 provides a statement for *real-world* electricity markets which are faced with operational constraints and a lossy power network. We illustrate the effects of different values of κ by means of Example 4.

#### **Example 4 (***cont'd***)**

*Recall the illustrative 7-node example from Subsection 3.4.4. We now assume that the network is divided into two cells as depicted in Fig. 3.13a, with* κ<sup>1</sup> *set to* 1*.*

**Figure 3.13:** Exemplary 7-node system divided into two cells.

*Fig. 3.13b shows a scatter plot of the resulting steady-state* (P total 1 , P total 2 ) *after each of both load steps, and under different choices of* κ<sup>2</sup> ∈ [0.2, 20]*. It can be seen that in both cases, each point* (P total 1 , P total 2 ) *is part of the Pareto front, thus no* κ<sup>2</sup> *yields a solution* (P total 1 , P total 2 ) *which is dominated by another* κ ′ 2 *.*

## **3.6 Discussion**

In this chapter, we have elaborated a distributed intra-cell mechanism for optimization-based control in zonal electricity markets. The proposed mechanism allows a mixture of price-taking WPs, price-setting CCs, and inelastic RPs. The interplay of different optimization problems of WPs results in the overall control system (3.49), (3.110) for which frequency and voltage regulation is preserved. The entailed constraints for WPs are either technically motivated or used to incorporate other obligations of WPs, e.g. from day-ahead clearing or long-term contracts. In this regard, the feasible set of each WP's optimization problem (3.83) contains the capacity that is currently available to the real-time market and not contracted otherwise. In line with the brownfield paradigm addressed in Chapter 1, WPs are not obliged to disclose their own profit function or their momentary flexibility to local CCs or to competitors. The proposed control scheme thereby provides a valuable alternative and improvement to other concepts currently under discussion, which are always characterized by an increasingly strong normative interventionism (see Subsection 2.1.3).

Apart from the cell-based communication structure, all WPs compete in a common market, which helps reconcile individual profit and social welfare, thus limiting the *moral hazard* phenomenon (cf. [ZPE17, p. 180]). By pushing the market price to marginal cost level, the

allocation (p ⋆ <sup>g</sup>,π, U ⋆ <sup>f</sup>,π, U ⋆ <sup>I</sup>,π) resulting as a closed-loop equilibrium is individually rational from the perspective of each WP π ∈ W, since any other feasible allocation would lead to lower individual profits. Moreover, Theorem 3.54 guarantees that by permitting cell-specific couplings via some κ > **0**, any equilibrium is also Pareto efficient.

Despite these promising findings, however, Pareto efficiency is unsatisfactory if used as the sole solution concept, since no concrete statement has yet been attempted on the proper choice of particularly suitable or unsuitable <sup>κ</sup> <sup>∈</sup> <sup>R</sup> n<sup>Z</sup> >0 , which could theoretically lead to allocations with arbitrarily poor social welfare. In the next chapter, we shed light on such unique allocations by proposing different control schemes for the dynamic and distributed settlement of κ throughout the interconnected power system.

# **4 Real-Time Incentives by Zonal Pricing**

In this chapter, distributed control schemes for inter-cell pricing are developed, working towards Contribution 2 and 4 (see Fig. 2.3). Section 4.1 defines the requirements for the control framework to be developed. Section 4.2 gives an overview of the developed overall control structure along with the dependencies of the individual sub-controllers. Section 4.3 provides real-time control strategies for the automatic regulation of zonal price differences by means of κ in order to fulfill Contribution 4. Section 4.4 provides the main results for Contribution 2 by proposing a *dynamic balancing* controller which guarantees fair pricing from the perspective of RPs. Section 4.5 analyzes the overall closed-loop system resulting from the interconnection of the proposed controllers. Section 4.6 gives a conclusive statement about the main findings of this chapter<sup>53</sup> .

## **4.1 Introduction**

The market-based controller developed in Chapter 3 is able to achieve a solution that is *technically feasible* in the sense that each market-clearing solution provides compliance with all technical constraints as well as zero deviation from nominal frequency caused by local supplydemand matching. However, the resulting power injections and the generated wholesale and retail prices Λk,ΛR,k are usually not "monetarily feasible" in the sense that the overall (net) payments throughout the network balance to zero, i.e. that all revenues of WPs are covered by respective payments from RPs. Moreover, despite the global supply-demand matching, the prices generated by CCs are in general unable to reflect *local* scarcity, since geo-spatial network effects such as power flow limits cannot be internalized [TBW15]. In particular, the methods discussed so far do not prevent from after-market redispatch by the system operator due to transmission congestion. Therefore, in this dissertation the free parameter κ is used to design regional short-term incentives for the activation of additional power generation in undersupplied locations and the (market-based) cutback of power generation in oversupplied locations.

The aim of balancing the overall payments throughout the network can be formalized as follows:

<sup>53</sup> A preliminary version of the results of this chapter has been published in the journal paper [KZV 22]. <sup>+</sup>

**Definition 4.1 (Balance of Payments (BoP)).** *Let*

$$\sigma(t) = \int\_{0}^{t} \left( \sum\_{\pi \in \mathcal{W}} \mathsf{R}\_{\pi}(t') - \sum\_{k \in \mathcal{Z}} \sum\_{i \in \mathcal{V}\_{Z,k}} \mathsf{C}\_{k}^{\ell}(t') \right) \mathrm{d}t \tag{4.1}$$

*denote the time-integrated sum of revenues of WPs and RPs. Then the payments of the interconnected power system are said to be* balanced *if*

$$\lim\_{t \to \infty} \sigma(t) = 0,\tag{4.2}$$

*i.e. if the time-integrated revenues of all WPs are equal to the time-integrated costs of all RPs as* t → ∞*.*

A weaker requirement is that not the integrated, but only the momentary payments must tend towards zero:

**Definition 4.2 (BoP Containment)).** *The BoP of the interconnected power system is said to be* contained *if the momentary revenues of all WPs are equal to the momentary costs of all RPs as* t → ∞*:*

$$\lim\_{t \to \infty} \sum\_{\pi \in \mathcal{W}} \mathsf{R}\_{\pi}(t) - \sum\_{k \in \mathcal{Z}} \sum\_{i \in \mathcal{V}\_{\mathcal{Z},k}} \mathsf{C}\_{k}^{\ell}(t) = 0. \tag{4.3}$$

Trivially, Definition 4.2 is fulfilled, if Definition 4.1 is fulfilled, but not vice versa.

**Remark 4.3.** *If BoP is satisfied, then it immediately follows that the only net (i.e. external) payments by network participants are given by the sum of WPs' cost for power generation.* ♢

**Remark 4.4.** *If BoP containment is satisfied, then the BoP error is constant with respect to time.* ♢

While real-time zonal pricing is considered to reduce structural and regional mismatches in generation and consumption by signaling local grid conditions and thus providing investment signals for WPs to deploy generation capacity in load centers [Sta19], it should be noted that RPs in particular are also inflexible with regard to geographical displacements of their own power production or consumption<sup>54</sup>. Therefore, as a third measure towards fairness among network participants, we impose the following normative requirement with regard to the retail prices ΛR,k:

<sup>54</sup> Namely, forcing a hypothetical "competition on locations" among RPs would not only be meaningless, but also arbitrarily discriminating.

**Figure 4.1:** Schematic diagram of overall procedure for dynamic pricing.

**Claim 4.5.** *RPs should pay a* uniform *retail price* Λ<sup>R</sup> *per amount of power, independent of in which price zone they are located.*

As a consequence of Claim 4.5, we choose the uniform retail price as Λ<sup>R</sup> = Λ<sup>0</sup> (as opposed to the cell-specific wholesale prices Λ<sup>k</sup> = κ<sup>k</sup> · Λ 0 , cf. Lemma 3.48.)

## **4.2 Overall Control Structure**

Fig. 4.1 shows the overall procedure towards a purposeful and economically feasible κ as addressed in the following sections. At first, several approaches for the automatic generation of spatially different participation factors κ ′′ (*dynamic pricing*) are proposed in Section 4.3. Main focus is given to the development of a real-time congestion management strategy, which is derived and elaborated in detail in Section 4.3.1. Subsequently, Section 4.4 discusses the generation of an "adjusted" participation factor κ, which satisfies BoP. Inspired by the terminology used in frequency control (cf. footnote 23), we distinguish in the following between a "primary" controller for BoP containment, which has the goal of limiting the BoP error to a finite value (cf. Remark 4.4), and a "secondary/tertiary" controller for *BoP restoration*, which has the goal of optimally restoring the BoP error to zero, in order to also comply with (4.1). Hence, in Subsection 4.4.1, a *regularized* κ ′ is obtained, which already satisfies BoP containment (4.3), but not (4.1). Ultimately, in Subsection 4.4.2 the constant BoP error (cf. Remark 4.4) is compensated by a suitable *restoration signal* ϵ κ . All methodologies necessary to resolve this last component are then derived in Chapter 5. In accordance with our main principle adopted so far, all of the components considered in this chapter aim for a distributed communication structure. Fig. 4.2 provides an aggregate overview of the controller components to be developed in Sections 4.3 and 4.4 as well as their relation to the distributed frequency and voltage controller from Chapter 3. We can see a clear resemblance between the light gray colored controller components in Fig. 4.2 and the three layers in Fig. 2.3.

**Figure 4.2:** Outline of overall controller structure for real-time zonal pricing.

A throughout intent of the resulting overall control mechanism is to exercise and implement different notions of fairness among network participants: For the interaction between CCs, fairness means that regional price differences (imposed via non-uniform entries in κ ′′) are supposed to help reward power generation at locations that are favorable from a network perspective and to incentivize grid-supportive behavior in both short and long term. At individual cell level, fairness is achieved by using market balancing to prevent accumulations of positive or negative amounts of capital at any of the CCs.

# **4.3 Dynamic Locational Pricing in Real-Time**

Section 3.5 has revealed that each strictly positive κ ′′ is economically efficient in the sense of Theorem 3.54. By adjusting κ ′′ appropriately, specific desired relationships between zonal prices are imposed, which can serve as a real-time control mechanism for the reallocation of electricity supply from one cell to another.

In the following Subsection 4.3.1, we apply a feedback control strategy for κ ′′ to obtain continuous-time congestion management, which is conducted among neighboring CCs by adjusting the cell-specific prices. As discussed in the literature review in Subsection 2.2.3 (see also Table 2.2), current price-based congestion management techniques are either based on an offline calculation of linear sensitivity factors or rely on simplified, network-reduced power system models. By contrast, the presented control scheme puts special emphasis on deploying a network-preserving model (cf. footnote 27) of inter-cell interaction that considers both active and reactive power flows and accounts for the potential overloads on each individual boundary line. In order to further extend this principle to the nodal voltages, Subsection 4.3.2 introduces an additional feedback control strategy for adjusting the voltage setpoints U set S and U set I . Finally, Subsection 4.3.3 discusses various approaches for influencing κ ′′ in the sense of Contribution 4 in addition to congestion management.

#### **4.3.1 Real-Time Congestion Management with Zonal Pricing**

We describe the cell topology by the condensed graph Gˆ <sup>p</sup> (as defined in Section 3.1). Let Pˆ ij and Qˆ ij denote the sending-end active and reactive power flows respectively from node <sup>i</sup> ∈ VZ,k to node <sup>j</sup> ∈ Vz,l with <sup>k</sup> ̸<sup>=</sup> <sup>l</sup>. As it generally holds that <sup>P</sup><sup>ˆ</sup> ij ̸<sup>=</sup> <sup>−</sup>P<sup>ˆ</sup> ji and Qˆ ij ̸<sup>=</sup> <sup>−</sup>Q<sup>ˆ</sup> ji, we define

$$\hat{P}\_m = \begin{cases} |\hat{P}\_{ij}, & |\hat{P}\_{ji}| \ge |\hat{P}\_{ji}|, \\ -\hat{P}\_{ji}, & \text{otherwise}, \end{cases} \qquad \hat{Q}\_m = \begin{cases} |\hat{Q}\_{ij}, & |\hat{Q}\_{ji}| \ge |\hat{Q}\_{ij}|, \\ -\hat{Q}\_{ji}, & \text{otherwise}, \end{cases} \tag{4.4}$$

and S⃗<sup>m</sup> := <sup>P</sup>ˆ<sup>m</sup> <sup>+</sup> ȷQˆ<sup>m</sup> for each line <sup>m</sup> <sup>∈</sup> <sup>E</sup><sup>ˆ</sup> <sup>p</sup>. All complex power flows S⃗<sup>m</sup> are concatenated in the vector <sup>S</sup>⃗ = col<sup>m</sup>∈{1,...,|Eˆp|}{S⃗m}. With <sup>S</sup> max specifying the maximum permissible apparent power flow, the *congestion factor* <sup>C</sup><sup>m</sup> <sup>∈</sup> <sup>C</sup> of line <sup>m</sup> <sup>∈</sup> <sup>E</sup><sup>ˆ</sup> <sup>p</sup> is thus defined as

$$\mathcal{L}\_m := \frac{\vec{S}\_m}{S\_m^{\text{max}}}, \quad \qquad \qquad m \in \mathring{\mathcal{E}}\_{\text{p}}.\tag{4.5}$$

\*\*Definition 4.6\*\* (Line Congrestion).  $Line \ m \in \mathring{\mathcal{E}}\_{\rm p}$  is said to be congested, if 
$$|\mathcal{C}\_{m}| > 1. \tag{4.6}$$

For the sake of argument, the next example paves the way to a resonable choice of κ ′′ in case of line congestion.

#### **Example 5 (Congestion Management by Zonal Pricing)**

*Consider a WoC scenario with two cells interconnected by a single power line as depicted in Fig. 4.3a.*

**(a)** Network with two cells

**(b)** Network with three cells

**Figure 4.3:** Example networks to illustrate the influence of κ′′ in congestion management.

*We assume that the power line is congested (i.e.* <sup>|</sup>S⃗ <sup>1</sup>→2<sup>|</sup> > Smax 1→2 *) with* Pˆ <sup>1</sup>→<sup>2</sup> > 0*. Obviously, the active power flow from cell 1 to cell 2 can be mitigated by raising* κ ′′ 2 *(to stimulate active power injection in cell 1) and/or by lowering* κ ′′ 1 *(to restrain active power injection in cell 2). Conversely, if* Pˆ <sup>1</sup>→<sup>2</sup> < 0*, then raising* κ ′′ 1 *or lowering* κ ′′ 2 *is advisable.*

*Now consider a scenario of* three *interconnected cells as depicted in Fig. 4.3b and assume again that the power line from cell 1 to cell 2 is congested with* <sup>|</sup>S⃗ <sup>1</sup>→2<sup>|</sup> > Smax 1→2 *and* Pˆ <sup>1</sup>→<sup>2</sup> > 0*. While the qualitative statements on modifying* κ ′′ 1 *and* κ ′′ 2 *stay the same as in the two-cell scenario, it is not obvious whether increasing or decreasing* κ ′′ 3 *is more favorable in order to support the congestion management. Indeed, an increased power flow through parallel lines* 1 → 3 *and* 3 → 2 *can contribute to further relieving congestion on line* 1 → 2*, hence it is not reasonable to keep* κ ′′ 3 *completely unchanged. This fact applies even more to scenarios with several parallel branches or more than one adjacent cells whereby a purely heuristic determination of appropriate* κ ′′ k *certainly becomes increasingly complex.*

*Apparently, if* <sup>|</sup>S⃗ <sup>3</sup>→<sup>1</sup><sup>|</sup> <sup>=</sup> <sup>|</sup>S⃗ <sup>3</sup>→<sup>2</sup>|*, i.e. if an equal amount of power is exchanged between cells 3 and 1 and between cells 3 and 2, then an increased or decreased power injection in cell 3 (triggered by a changed participation factor* κ ′′ 3 *) has no effect on the power flow over the congested line* <sup>1</sup> <sup>→</sup> <sup>2</sup>*. On the contrary, if* <sup>|</sup>S⃗ <sup>3</sup>→<sup>1</sup>| ≫ |S⃗ <sup>3</sup>→<sup>2</sup>|*, then the influence of cell 3 on the congested line is quite similar to the influence of cell 1, thus cells 1 and 3 should have similar participation factors. Analogously, if* <sup>|</sup>S⃗ <sup>3</sup>→<sup>1</sup>| ≪ |S⃗ <sup>3</sup>→<sup>2</sup>|*, then the influence of cell 3 is highly similar to the influence of cell 2. Thus, the participation factors* κ ′′ 2 *and* κ ′′ 3 *should be similar. In summary, it emerges that if two cells exchange a high amount of power, then the respective participation factors should be tighly coupled. Conversely, the participation factors of two cells exchanging little power should be decoupled.*

*In order to keep all of the following participation factors limited to strictly positive values in* R>0*, we apply the transformation* ϕ<sup>i</sup> := ln(κ ′′ i )*. Then, based on the above discussion, the (transformed) participation factor of cell 3 can be calculated out of* ϕ<sup>1</sup> *and* ϕ<sup>2</sup> *as the weighted sum*

$$\phi\_3 = \frac{|\vec{S}\_{3\to1}|\phi\_1 + |\vec{S}\_{3\to2}|\phi\_2}{|\vec{S}\_{3\to1}| + |\vec{S}\_{3\to2}|}. \tag{4.7}$$

*Following the same principle, the participation factors for cells 1 and 2 are calculated as*

$$\phi\_1 = \frac{|\vec{S}\_{1\to 2}|\phi\_2 + |\vec{S}\_{1\to 3}|\phi\_3 - \psi\_1}{|\vec{S}\_{1\to 2}| + |\vec{S}\_{1\to 3}|}, \qquad \phi\_2 = \frac{|\vec{S}\_{2\to 1}|\phi\_1 + |\vec{S}\_{2\to 3}|\phi\_3 - \psi\_2}{|\vec{S}\_{2\to 1}| + |\vec{S}\_{2\to 3}|}, \tag{4.8}$$

*where* ψ<sup>1</sup> *and* ψ<sup>2</sup> *are suitable activation functions (yet to be determined) which indicate the congestion status of the boundary lines incident to the respective cell. In particular, choosing* ψ<sup>1</sup> > 0 *signals that increased power injection in cell 1 has an* undesirable *impact on congestion of the incident line* 1 → 2 *(i.e. the participation factor of cell 1 should be reduced), and* ψ<sup>2</sup> < 0 *signals that increased power injection in cell 2 has a* desirable *impact on congestion of the incident line* 1 → 2*, thus the participation factor of cell 2 should be increased.*

*Combining* (4.7) *and* (4.8) *finally yields*

$$\psi = \underbrace{\begin{bmatrix} -|\vec{S}\_{1\to2}| - |\vec{S}\_{1\to3}| & |\vec{S}\_{1\to2}| & |\vec{S}\_{1\to3}|\\ |\vec{S}\_{1\to2}| & -|\vec{S}\_{1\to2}| - |\vec{S}\_{2\to3}| & |\vec{S}\_{2\to3}|\\ |\vec{S}\_{1\to3}| & |\vec{S}\_{2\to3}| & -|\vec{S}\_{1\to3}| - |\vec{S}\_{2\to3}| \end{bmatrix}}\_{=:\mathcal{A}} \phi. \tag{4.9}$$

*It can be seen in* (4.9) *that* A *is the Laplacian matrix of the graph in Fig. 4.3b weighted by the apparent power flows between cells.*

Based on the insights gained from Example 5, we can thus set up

$$\mathbb{0} = -\underbrace{\hat{\mathbf{D}}\_{\text{p}} \text{diag}\{ |\vec{S}| \} \hat{\mathbf{D}}\_{\text{p}}^{\uparrow}}\_{\mathcal{A}} \phi - \psi \tag{4.10}$$

for the determination of participation factors κ ′′ = exp(ϕ). As discussed in Subsection 3.4.1, constraint (4.10) can be obtained in feedback form via the gradient flow

$$
\tau\_{\phi}\dot{\phi} = -\mathcal{A}\phi - \psi.\tag{4.11}
$$

However, while in Example 5 only line 1 → 2 was assumed to be congested, in a general setup potentially all lines can be more or less affected. In order to map the individual contribution of each cell to (possible) congestion on one or more lines, we choose the activation functions ψ to

$$
\psi = \hat{D}\_{\mathbf{p}} \mathbf{U},
\tag{4.12}
$$

where ℧ <sup>∈</sup> <sup>R</sup>|Eˆp<sup>|</sup> = col<sup>m</sup>∈Eˆ<sup>p</sup> {℧m} is an appropriate barrier function penalizing lines <sup>m</sup> <sup>∈</sup> <sup>E</sup><sup>ˆ</sup> p which are close to congestion. The signal flow diagram in Fig. 4.4 briefly summarizes the influence of line congestion on the prices λ(t) and thus the active power generation p<sup>g</sup> (t).

Before discussing two possible options how to properly select ℧m, we conclude with three additional remarks:

**Remark 4.7.** *Using the logarithmic quantities* ϕ *instead of* κ ′′ *also has the practical benefit that arithmetic averaging* (4.7) *or* (4.8) *clusters around the social optimum with* κ ′′ = 1 *and thus no upward outliers can occur.* ♢

**Remark 4.8.** *For high voltage networks where it is justified to assume that power lines are purely inductive, voltage profiles are flat (i.e.* U<sup>i</sup> ≈ 1 *for* i ∈ V*), and voltage angle differences* ϑij *are small, power line susceptances can be used to approximate power flows throughout the power system [vDD14]. Accordingly, it is then valid to use the approximation*

$$\mathcal{A} = \hat{\mathbf{D}}\_{\mathrm{p}} \text{diag}\{ |\vec{\mathcal{S}}| \} \hat{\mathbf{D}}\_{\mathrm{p}}^{\top} \approx -\beta \hat{\mathbf{D}}\_{\mathrm{p}} \text{diag}\{ \hat{\mathbf{B}} \} \hat{\mathbf{D}}\_{\mathrm{p}}^{\top},\tag{4.13}$$

$$\longrightarrow \vec{S}(t) \cdots \cdots \star \mathcal{L}(t) \cdots \cdots \star \mathcal{U}(t) \cdots \cdots \star \phi(t) \cdots \cdots \star \kappa''(t) \cdots \cdots \star \lambda(t) \cdots \cdots \star p\_{\mathbb{S}}(t) \cdots$$

**Figure 4.4:** Signal flow diagram illustrating the relationship between line congestion C(t) and prices λ(t) created by the proposed real-time congestion controller.

*where* β > <sup>0</sup> *and* <sup>B</sup><sup>ˆ</sup> <sup>∈</sup> <sup>R</sup> |Eˆp<sup>|</sup> <0 *is the vector of susceptances of inter-cell power lines (with the same sorting as the edges in* Dˆ <sup>p</sup>*). The approximation in* (4.13) *has the major advantage that* A *is now independent of the momentary power flows over inter-cell lines and thus can be calculated offline.* ♢

**Remark 4.9.** *An insightful interpretation of* (4.10) *can be made via generalized coordinates: In this regard,* ϕ<sup>k</sup> *(and thus* κ ′′ k *) can be interpreted as the* potential *(i.e. generalized effort) variable of cell* k*, while the apparent power flows* S⃗ k→l *, which indicate how closely the corresponding values of* κ ′ *should be coupled (cf. Example 5), act as* generalized conductances*. Accordingly,* ψ<sup>k</sup> *can be interpreted as local injection of generalized currents (i.e. as generalized* flow *variable), inducing* differences of potential *(i.e. generalized efforts)—in particular, an increase of the local potential (if* ψ<sup>k</sup> > 0*) or decrease of local potential (if* ψ<sup>k</sup> < 0*). Formulating the node-voltage method for the resulting* generalized circuit *with potentials* <sup>ϕ</sup>*, conductances* <sup>|</sup>S⃗ <sup>|</sup>*, and flows* <sup>ψ</sup> *also leads to* (4.10)*, where* A *acts as nodal admittance matrix.* ♢

#### **Congestion Controller without Integral Action**

As indicated in (4.10), ψ valuates the impact of each cell on overall congestion. By using (4.12), we can assemble this impact from line-wise "static" barrier functions to increasingly penalize those lines that are close to congestion. With regard to Example 5 where <sup>|</sup>S⃗ <sup>1</sup>→<sup>2</sup><sup>|</sup> <sup>&</sup>gt; S⃗max 1→2 and Pˆ <sup>1</sup>→<sup>2</sup> <sup>=</sup> ℜ{S⃗ <sup>1</sup>→<sup>2</sup>} > 0, the participation factor κ ′′ <sup>1</sup> has to be decreased in order to alleviate congestion between cells 1 and 2, while κ ′′ 2 should be increased. Conversely, if Pˆ <sup>1</sup>→<sup>2</sup> <sup>=</sup> ℜ{S⃗ <sup>1</sup>→<sup>2</sup>} < 0, then κ ′′ <sup>1</sup> has to be decreased and κ ′′ <sup>2</sup> has to be increased.

In this light, we propose a static barrier function for each line as follows:

$$\mathcal{U}\_m = \begin{cases} \Re\{\mathcal{C}\_m\} \cdot \frac{|\mathcal{C}\_m| - \mathcal{C}\_m^{\min}}{(1 - |\mathcal{C}\_m|)(1 - \mathcal{C}\_m^{\min})}, & |\mathcal{C}\_m| > \mathcal{C}\_m^{\min}, \\ 0, & \text{otherwise}, \end{cases} \tag{4.14}$$

where C min <sup>m</sup> ∈ (0, 1) is a (user-defined) threshold for control actions. As long as |Cm| is below this threshold, it holds that ℧<sup>m</sup> = 0. Moreover, |℧m| → ∞ as |Cm| → 1 <sup>55</sup> (see Fig. 4.5).

**Remark 4.10.** *Note that the individual values of* ϕ<sup>k</sup> *and* ψ<sup>k</sup> *in* (4.11) *can be calculated locally by the respective CC* k ∈ Z*, since the CC only requires information about neighboring* κ *values as well as power flows across its own cell boundary.* ♢

<sup>55</sup> Analogously, it can be stated that <sup>|</sup>℧m| → ±∞ as <sup>|</sup>S⃗m| → S⃗max <sup>m</sup> .

**Figure 4.5:** Exemplary plot of barrier function ℧m(Cm) with Cmin <sup>m</sup> = 0.5 from different viewing angles. The red circle highlights all points where |Cm| = Cmin <sup>m</sup> holds.

**Remark 4.11.** *The proposed barrier function* (4.14) *represents only one possible choice of feedback for* ψ = Dˆ <sup>p</sup>℧*. Basically, the only requirement which* ℧ *must certainly satisfy is* sgn(℧m) = sgn(ℜ{Cm}) *in order to ensure that* ψ *has the "correct" sign which helps to alleviate congestion. Furthermore,* |℧m| *should be at least monotonically increasing with respect to* |Cm|*.* ♢

#### **Congestion Controller with Integral Action**

Typically, the maximum rated power S⃗max <sup>m</sup> can be exceeded for a certain amount of time before a secure operation of the power system can no longer be guaranteed [CVV17]. Thus, the momentary apparent power flow should generally be permitted to temporarily exceed the maximum rated power. In order to capture this fact, we propose a more sophisticated feedback law for ψ by incorporating a simple model for line overheating to replace the static feedback (4.14), namely

$$\mathsf{U}\_{m} = \begin{cases} -\frac{1}{\tau\_{\mathsf{U}\_{m}}} (1 - |\mathcal{C}\_{m}|) \mathsf{U}\_{m}, & |\mathcal{C}\_{m}| < \mathcal{C}\_{m}^{\min}, \\ -\frac{1}{\tau\_{\mathsf{U}\_{m}}} (1 - |\mathcal{C}\_{m}|) \mathsf{U}\_{m} + \frac{1}{\tau\_{\mathsf{U}\_{m}}^{\tau}} (|\mathcal{C}\_{m}| - \mathcal{C}\_{m}^{\min}) \mathsf{R} \{ \mathcal{C}\_{m} \}, & \mathcal{C}\_{m}^{\min} \le |\mathcal{C}\_{m}| \le 1, \\ \frac{1}{\tau\_{\mathsf{U}\_{m}}} \mathfrak{R} \{ \mathcal{C}\_{m} \}, & |\mathcal{C}\_{m}| > 1. \end{cases} \tag{4.15}$$

Here, C min <sup>m</sup> denotes the threshold below which the line is defined as *cooling*. Accordingly, |℧m| can be interpreted as a measure of line overheating. In particular, if the line is congested (|Cm| > 1), then it heats up and |℧m| in (4.15) increases. If the line is not congested and |Cm| < C min <sup>m</sup> , then it cools down (such that |℧m| in (4.15) decreases) and the cooling rate

increases proportionally with less power flowing over the line. If C min <sup>m</sup> ≤ |Cm| ≤ 1, then both effects are overlaid. The design parameters τ − ℧<sup>m</sup> , τ <sup>+</sup> ℧<sup>m</sup> > 0 model the specific heating and cooling rates, which are dependent on e.g. the conductor type of the power line or local weather conditions [Dou88].

Both Remark 4.10 and Remark 4.11 analogously apply to the dynamic barrier function in (4.15).

#### **4.3.2 Adjustment of Voltage Setpoints**

If the voltage controller (3.110b)–(3.110c) is equipped with setpoints U set S , U set I as proposed in Remark 3.45, additional control of the *reactive* power flows can be implemented as a second measure for congestion management. For this purpose, the setpoints U set i can be used to manipulate voltage differences along congested lines and thus mitigate congestion by reducing reactive power flows where necessary.

Reconsider the exemplary scenario from Example 5 with line 1 → 2 assumed to be congested with <sup>|</sup>S⃗ <sup>1</sup>→<sup>2</sup><sup>|</sup> <sup>&</sup>gt; S⃗max 1→2 and Qˆ <sup>1</sup>→<sup>2</sup> <sup>=</sup> ℑ{S⃗ <sup>1</sup>→<sup>2</sup>} > 0. Here, the reactive power flow can be decreased by increasing the voltage amplitudes in cell 2 and/or by decreasing the voltage amplitudes in cell 1, while the voltage amplitudes in cell 3 do not need to be modified<sup>56</sup> .

In general, cell-specific setpoints U set k for each k ∈ Z can be controlled by the dynamic setpoint correction scheme

$$
\tau\_{U\_k^{\text{set}}} \dot{U}\_k^{\text{set}} = -U\_k^{\text{set}} + U\_k^{\text{nom}} + \overline{U}^{\Delta} \langle U\_k^{\Delta} \rangle\_{-1}^1,\tag{4.16}
$$

where U nom k denotes the nominal voltage of cell k ∈ Z, U ∆ > 0 is the maximum admissible deviation between the steady-state voltage setpoint and the nominal voltage, U ∆ k is the "desired" voltage deviation between U set k and U nom k , and

$$\langle U\_k^{\Delta} \rangle\_{-1}^1 := \begin{cases} 1, & U\_k^{\Delta} > 1, \\ U\_k^{\Delta}, & -1 \le U\_k^{\Delta} \le 1, \\\ -1, & U\_k^{\Delta} < 1 \end{cases} \tag{4.17}$$

is a saturation operator restrictingU ∆ k onto the domain [−1, 1]. With the summandU ∆ ⟨U ∆ k ⟩ 1 −1 in (4.16), it is guaranteed that the voltage setpoint U set k always remains in a sufficiently small neighborhood around the nominal voltage U nom k .

A simple yet effective feedback control scheme for the desired voltage deviations U <sup>∆</sup> can then be applied by choosing for example

$$U\_k^{\Delta} = \sum\_{\substack{m \in \mathcal{E}\_p^b\\m = i \to j, \, i \in \mathcal{Z}\_k}} \text{sgn}(\mathbb{S}\{\mathcal{C}\_m\}) \cdot |\mathcal{C}\_m|. \tag{4.18}$$

<sup>56</sup> An essential difference between active and reactive power flows is that reactive power flows are mainly local phenomena and thus should also be compensated by local interventions [vDD14, LAPG17].

Here, the desired voltage deviation U ∆ k is proportional to the absolute value |Cm| of the congestion factor. The multiplier sgn(ℑ{Cm}) reflects the direction of reactive power flow, which is required in order to yield the correct orientation for U ∆ k . With respect to all cells, we thus get the local setpoint correction dynamics

$$
\tau\_{U^{\rm sat}} \dot{\boldsymbol{U}}^{\rm set} = -\boldsymbol{\hat{U}}^{\rm set} + \boldsymbol{\hat{U}}^{\rm norm} + \overline{\boldsymbol{U}}^{\Delta} \langle \boldsymbol{U}^{\Delta} \rangle\_{-1}^{1},\tag{4.19a}
$$

$$\mathbf{U}^{\Delta} = \hat{\mathbf{D}}\_{\mathrm{p}} \left( \operatorname{sgn}(\mathbb{G} \{ \mathcal{C} \}) \circ | \mathcal{C} | \right), \tag{4.19b}$$

where τ <sup>U</sup>set = diag<sup>k</sup>∈Z {τUset k }, <sup>U</sup><sup>ˆ</sup> set = col<sup>k</sup>∈Z {U set k }, and <sup>U</sup><sup>ˆ</sup> nom = col<sup>k</sup>∈Z {U nom k }.

#### **4.3.3 Extensions, Combinations, and Long-Term Perspective**

In Subsection 4.3.1, the participation factor κ ′′ was exploited to establish a real-time congestion management strategy. In the following subsection, we briefly discuss several other possible means for regulating κ ′′ such that alternative notions of "grid-supportive behavior" as objected in Chapter 1 may be realized.

A somewhat classical approach is to ponder on the active power balances separately for each cell<sup>57</sup>. By e.g. setting

$$\kappa\_k'' = \frac{\sum\_{i \in \mathcal{V}\_{\mathcal{Z},k}} p\_{\ell,i}(t) + \hat{\Phi}\_k(t)}{\sum\_{i \in \mathcal{V}\_{\emptyset,k}} p\_{\mathbf{g},i}(t)},\tag{4.20}$$

the participation factor is equal to 1 as soon as the active power balance is fulfilled for cell k ∈ Z, while the revenues for power generation automatically increase if cell k is a load center. By this basic approach, regional incentives can be created to deploy additional generation capacity primarily where there is a momentary or permanent local undersupply. Conversely, cell-specific prices automatically drop in case there is a lot of available local generation capacity and/or few local consumption. Since all information required in (4.20) can be measured decentrally in cell k, there is actually no need for communication with neighboring CCs. An obvious disadvantage of (4.20) is that there exists no solution for κ ′′ k whenever there is no momentary power generation in cell k for some reason.

A considerably simpler approach, which invariably yields a nonsingular solution while requiring no measurements from WPs' power generations, is

$$\sum\_{k\_k^{\prime\prime}} \quad p\_{\ell,i}(t)$$

$$\kappa\_k^{\prime\prime} = \frac{\sum\_{i \in \mathcal{V}\_{Z,k}} p\_{\ell,i}(t)}{\sum\_{k \in \mathcal{Z}} \sum\_{i \in \mathcal{V}\_{Z,k}} p\_{\ell,i}(t)}.\tag{4.21}$$

<sup>57</sup> The concept of cell-specific balancing is closely related to the MG paradigm, which suggests that local power balancing helps to reduce (global) network losses, cf. [Sch15, p. 56].

In this case, the pricing scheme only accounts for the momentary distribution of consumption among all cells. To obtain meaningful results for this approach, cells should have nominally the same amount of consumers. In contrast to (4.20), it is necessary to obtain delay-free information about the consumption in all other cells<sup>58</sup> .

Besides the examples shown above, also a combination of pricing regimes can be applied such that the controller equations for κ ′′ k (t) (and consequently the incentive signaling of the pricing strategy) are composed of several components in an additive manner. For the remainder of this subsection, denote by κ ◦ k the "cell-individual" calculations governed e.g. by (4.20) or (4.21). Then, by introducing a *communality factor* ι ∈ [0, 1] and choosing

$$
\kappa\_k''(t) = \iota \cdot K + (1 - \iota) \cdot \kappa\_k^\circ(t), \tag{4.22}
$$

the *degree of excess* among cell-specific prices can be seamlessly adjusted from fully individualized prices κ ′′ k (t) = κ ◦ k (t) for ι = 0 up to a commonly shared price κ ′′ k (t) = K for ι = 1.

Besides a fixed common value K > 0, more sophisticated methods for the "commonly shared part" of (4.22) can be employed. The following lemma exemplarily shows how dynamic consensus among CCs can be reached by by invoking distributed averaging theory [DdGP10, SDF17].

**Lemma 4.12.** *Let* A<sup>κ</sup> *be the adjacency matrix of a weakly connected and possibly weighted communication graph* G<sup>κ</sup> = (Z, Eκ)*. Then for each equilibrium* ((κ ′′ k ) ⋆ , e<sup>⋆</sup> k ) *of the distributed-averaging-based controller*

$$
\kappa\_k'' = \iota \cdot e\_k + (1 - \iota) \cdot \kappa\_k^\diamond, \qquad \qquad \qquad k \in \mathbb{Z}, \tag{4.23a}
$$

$$\dot{e}\_k = -\sum\_{l \in \mathcal{Z}} \varpi\_{kl} (e\_k - \kappa\_l''), \qquad \qquad k \in \mathcal{Z} \tag{4.23b}$$

*with convex weights* ϖkl = [Aκ]kl/ P l∈N<sup>k</sup> [Aκ]kl *, it holds that* e ⋆ *is a weighted average of the neighboring participation factors* κ ′′ l *.*

*Proof.* By replacing ϖkl = [A]kl/ P l∈N<sup>k</sup> [A]kl in (4.23b) and equating the right-hand side of (4.23b) to zero, we get

$$e\_k^\star = \sum\_{l \in \mathcal{Z}} [\mathcal{A}]\_{kl} \cdot (\kappa\_l^{\prime\prime})^\star. \tag{4.24}$$

Thus, e ⋆ k is a weighted average of all neighboring participation factors (κ ′′ l ) ⋆ , whereby the weights are given by A.

**Remark 4.13.** *If* G<sup>κ</sup> *is complete, then* e ⋆ <sup>k</sup> = e ⋆ l *holds for all* k, l ∈ Z*, i.e. the commonly shared part of* κ ′′(t) *in* (4.23a) *has the same value throughout the whole network.* ♢

**Figure 4.6:** Zonal prices in case of load center in cell 7 under different choice of communality factor ι. The intensity of the red color in each individual cell signals the resulting increase in local prices and thus the added incentive for supply in this respective cell.

Fig. 4.6 schematically shows the effects of combined pricing (4.23) for a load center in cell 7 for different communality factors ι ∈ {0, 0.5, 1} with κ ◦ k (t) being set according to (4.21). It emerges that ι = 1 leads to a regionally independent (uniform) price (see Fig. 4.6a), while for ι = 0 the load surplus in cell 7 leads to an increased price in the same cell only, while all other cells remain unaffected (see Fig. 4.6d). For ι ∈ (0, 1), on the other hand, the specific properties of the weakly connected graph G<sup>κ</sup> become relevant: As Fig. 4.6b (complete graph Gκ) shows, all cells other than cell 7 share a unique price Λ¬<sup>7</sup> < Λ7, while in Fig. 4.6c (incomplete graph G<sup>κ</sup> = Gˆ <sup>p</sup>), the prices Λ<sup>k</sup> in cells k ̸= 7 gradually drop with their spatial distance to cell 7.

## **4.4 Dynamic Balancing of Payments**

While the previous section focused primarily on notions of fairness for WPs, this section is devoted to also ensuring fairness for RPs in terms of Claim 4.5 by further refinements on participation factors κ ′′. In particular, Subsection 4.4.1 combines the fairness objective of Claim 4.5 with the BoP containment from Definition 4.2. Subsequently, Section 4.4.2 introduces an optimization-based framework for BoP restoration to also comply with the BoP requirement as in Definition 4.1.

#### **4.4.1 Balance of Payments Containment**

Before considering specific actions for the observance of BoP containment, it is important to note the basic finding that the resulting power flows are invariant with regard to scalar multiplications of the participation vector κ ′′:

<sup>58</sup> Even if this approach is not decentralized, a fully distributed consensus-based control scheme can be derived by applying Lemma B.1 analogously to the control laws in Section 3.4, such that calculations of (4.21) are solely based on neighbor-to-neighbor communication among CCs.

,

**Lemma 4.14.** *Let* (p ♯ g , U ♯ f , U ♯ I ) *denote an equilibrium of* (3.110a)*–*(3.110i)*,* (3.147a)*–* (3.147b)*. For a given* κ1*, and let* (p ⋆ g , U ⋆ f , U ⋆ I ) *denote the corresponding equilibrium for* κ<sup>2</sup> = a · κ<sup>1</sup> *with* a > 0*. Then it holds that* (p ♯ g , U ♯ f , U ♯ I ) = (p ⋆ g , U ⋆ f , U ⋆ I )*.*

*Proof.* With Theorem 3.51 it follows that each equilibrium (p ♯ g , U ♯ f , U ♯ I ) of (3.110a)–(3.110i), (3.147a)–(3.147b) with κ = κ<sup>1</sup> is defined by (3.154) with (3.154a) equal to

$$\mathbb{0} = -\nabla \mathbb{C}^{\kappa\_1}(p\_\mathbf{g}^\sharp) - \omega^\sharp + \hat{\boldsymbol{\lambda}}^\sharp + \hat{\boldsymbol{\mu}}\_\mathbf{g}^\sharp - \hat{\boldsymbol{\mu}}\_\mathbf{g+}^\sharp,\tag{4.25}$$

while each equilibrium (p ⋆ g , U ⋆ f , U ⋆ I ) of (3.110a)–(3.110i), (3.147a)–(3.147b) with κ = κ<sup>2</sup> fulfills

$$0 = -\nabla \mathbb{C}^{\kappa\_2}(\mathbf{p}\_{\mathbf{g}}^{\star}) - \omega^{\star} + \hat{\boldsymbol{\lambda}}^{\star} + \hat{\boldsymbol{\mu}}\_{\mathbf{g}-}^{\star} - \hat{\boldsymbol{\mu}}\_{\mathbf{g}+}^{\star} = -\frac{1}{a} \cdot \nabla \mathbb{C}^{\kappa\_1}(\mathbf{p}\_{\mathbf{g}}^{\star}) - \omega^{\star} + \hat{\boldsymbol{\lambda}}^{\star} + \hat{\boldsymbol{\mu}}\_{\mathbf{g}-}^{\star} - \hat{\boldsymbol{\mu}}\_{\mathbf{g}+}^{\star},\tag{4.26}$$

a > 0, together with (3.154b), where each (·) ♯ in (3.154b)–(3.154l) is replaced by (·) ⋆ . With Lemma 3.35, it follows that ω <sup>♯</sup> = ω <sup>⋆</sup> = **0**. Multiplying (4.26) by a thus leads to

$$\mathbb{0} = -\nabla \mathbb{C}^{\kappa\_1} (\mathbf{p}\_{\mathbf{g}}^{\sharp}) + \hat{\boldsymbol{\lambda}}^{\sharp} + \hat{\boldsymbol{\mu}}\_{\mathbf{g}-}^{\sharp} - \hat{\boldsymbol{\mu}}\_{\mathbf{g}+}^{\sharp},\tag{4.27a}$$

$$=-\nabla \mathcal{C}^{\kappa\_1}(\mathbf{p}\_{\mathbf{g}}^{\star}) + a \cdot \hat{\boldsymbol{\lambda}}^{\star} + a \cdot \hat{\boldsymbol{\mu}}\_{\mathbf{g}-}^{\star} - a \cdot \hat{\boldsymbol{\mu}}\_{\mathbf{g}+}^{\star}.\tag{4.27b}$$

By comparison of coefficients<sup>59</sup> between (4.27a), (3.154b)–(3.154l) and (4.27b), (3.154b)–(3.154l)<sup>60</sup> it immediately follows that for each equilibrium (·) ♯ to κ<sup>1</sup> fulfilling (4.27a), (3.154b)–(3.154l), there exists an equilibrium (·) ⋆ to κ<sup>2</sup> = a · κ<sup>1</sup> with p ⋆ <sup>g</sup> = p ♯ g , λˆ ⋆ <sup>=</sup> <sup>a</sup> · <sup>λ</sup><sup>ˆ</sup> ♯ , µˆ ⋆ <sup>g</sup><sup>−</sup> = a · µˆ ♯ g−, µˆ ⋆ g+ = a · µˆ g+, µˆ ⋆ S− = µˆ ♯ S−, <sup>µ</sup><sup>ˆ</sup> ⋆ <sup>S</sup><sup>+</sup> = µˆ ♯ <sup>S</sup>+, <sup>µ</sup><sup>ˆ</sup> ⋆ I− = µˆ ♯ I−, <sup>µ</sup><sup>ˆ</sup> ⋆ <sup>I</sup><sup>+</sup> = µˆ ♯ <sup>I</sup>+, <sup>U</sup> ⋆ <sup>f</sup> = U ♯ f , U ⋆ <sup>I</sup> = U ♯ I , and νˆ <sup>⋆</sup> = νˆ ♯ . In particular, it holds that (p ♯ g , U ♯ f , U ♯ I ) = (p ⋆ g , U ⋆ f , U ⋆ I ).

Lemma 4.12 implies that there exists a remaining degree of freedom of dimension one which only affects the prices (and thus the costs and revenues of WPs and RPs), whilst leaving all physical quantities (i.e. power flows and nodal voltages) unchanged. This degree of freedom can be exploited to generate a *regularized* κ ′ (t) = a ⋆ · κ ′′(t), where a ⋆ is the *unique* scalar that induces BoP containment without altering the physical actuation (and thus the incentives originally governed by κ ′′(t)). With the observation that the requirement of BoP containment (4.3) can be written as ((κ ′ ) ⋆ ) Tpˆ ⋆ <sup>g</sup> = 1 Tpˆ ⋆ ℓ , the defining condition for a ⋆ thus equals

$$-a^\star \sum\_{k \in \mathbb{Z}} (\kappa\_k^{\prime\prime})^\star \hat{p}\_{\mathbf{g},k}^\star + \sum\_{k \in \mathbb{Z}} \hat{p}\_{\ell,k}^\star = 0. \tag{4.28}$$

<sup>59</sup> Note that strict convexity of C <sup>κ</sup>(p<sup>g</sup> ) implies that ∇C <sup>κ</sup>(p<sup>g</sup> ) is invertible.

<sup>60</sup> Again, each (·) ♯ in (3.154b)–(3.154l) has to be replaced by (·) ⋆.

Due to the separability of (4.28) with respect to κ ′′, we again employ the distributed consensusbased control approach derived in Appendix B.2 to obtain the distributed controller

$$
\tau\_a \dot{\mathbf{a}} = -\mathbf{a} \circ \kappa'' \circ \hat{\mathbf{p}}\_{\mathbf{g}} + \hat{\mathbf{p}}\_{\ell} + \hat{\mathbf{D}}\_{\mathbf{p}} \nu\_a,\tag{4.29a}
$$

$$
\boldsymbol{\tau}\_{\nu\_{a}} \dot{\boldsymbol{\nu}}\_{a} = -\boldsymbol{\hat{D}}\_{\text{p}}^{\text{T}} \boldsymbol{\nu}\_{a}.\tag{4.29b}
$$

Using (4.29), each individual CC can execute its control actions by relying solely on own local measurements as well as on neighbor-to-neighbor communication, thus in the same way as already implemented for the congestion controllers in Subsections 4.3.1 and 4.3.2.

**Remark 4.15.** *It is convenient to resort to the cell incidence matrix* Dˆ <sup>p</sup> *as the communication matrix in* (4.29)*, as it is already encountered for the settlement of cell-specific participation factors* κ ′′ k (4.11) *via* A *in* (4.10)*. Hence, existing communication infrastructure can be reused.* ♢

**Remark 4.16.** *If the distributed controller* (4.29) *is interpreted as primal-dual saddle-point flow (cf. Subsection 3.4.1), then each equilibrium* (a ⋆ , ν ⋆ a ) *of* (4.29) *is an optimizer of the convex optimization problem*

$$\min\_{\mathbf{k}'} \qquad \qquad \frac{1}{2} (\kappa')^\top \begin{bmatrix} \frac{\hat{p}\_{\mathbb{R},1}}{\kappa\_1'} & & \mathbf{0} \\ & \ddots & \\ \mathbf{0} & & \frac{\hat{p}\_{\mathbb{R},N}}{\kappa\_N'} \end{bmatrix} \kappa' \tag{4.30a}$$

$$\mathbf{i}\text{ subject to}\quad\left(\kappa'\right)^{\mathsf{T}}\hat{\boldsymbol{p}}\_{\mathsf{g}}=\mathbbm{1}^{\mathsf{T}}\hat{\boldsymbol{p}}\_{\ell},\tag{4.30b}$$

*where* a ⋆ <sup>k</sup> = (κ ′ k ) <sup>⋆</sup>/κ′′ k *and* ν ⋆ <sup>a</sup> *is the vector of Lagrange multipliers for the affine constraint* (4.30b)*. One possible generalization of* (4.30) *is to replace* (4.30a) *by an arbitrary convex objective function* J κ ′ (κ ′ )*. This optimization problem then also leads to participation factors fulfilling BoP containment, without the need to compute non-regularized participation factors* κ ′′*. However, since the relation* κ ′ = a · κ ′′ *with* κ ′′ > **0** *has been dropped, the strict positivity requirement is obeyed separately, yielding*

$$\min\_{\mathbf{x}',\mathbf{a}} \qquad \qquad \mathbf{J}^{\kappa'}(\mathbf{x}') \tag{4.31a}$$

subject to Dˆ <sup>p</sup>a − κ ′ ◦ pˆ<sup>g</sup> + pˆ<sup>ℓ</sup> = **0**, (4.31b)

$$
\kappa' - \varepsilon \ge 0,\tag{4.31c}
$$

*where* ε > **0** *is an arbitrarily small safety constant. Using* (4.31)*, the distributed consensus-based controller becomes*

$$
\tau\_{\kappa} \dot{\kappa}' = -\nabla \mathbf{J}^{\kappa'} (\kappa') + \nu\_{\kappa}^{\mathsf{T}} \hat{p}\_{\mathsf{g}} + \mu\_{\kappa}, \tag{4.32a}
$$

$$
\tau\_a \dot{a} = -\hat{D}\_{\text{p}}^{\text{T}} \nu\_{\kappa},\tag{4.32b}
$$

$$
\tau\_{\sigma} \dot{\nu}\_{\kappa} = \hat{\mathcal{D}}\_{\text{p}} \boldsymbol{a} - \kappa' \circ \hat{\mathcal{p}}\_{\text{g}} + \hat{\mathcal{p}}\_{\ell}, \tag{4.32c}
$$

$$
\tau\_{\mu\_{\kappa}} \dot{\mu}\_{\kappa} = \langle -\kappa' + \varepsilon \rangle^{+}\_{\mu\_{\kappa}},\tag{4.32d}
$$

*where* ν<sup>κ</sup> *and* µ<sup>κ</sup> *denote the Lagrange multipliers for constraints* (4.31b) *and* (4.31c)*, respectively.* ♢

From (4.28) in conjunction with (4.29), it immediately follows that a <sup>⋆</sup> = a ⋆ · <sup>1</sup>, where

$$a^\star = \left(\frac{\sum\_{k \in \mathcal{Z}} \sum\_{i \in \mathcal{V}\_{\mathcal{Z},k}} (\kappa\_k^{\prime\prime} \cdot p\_{\mathcal{g},i})}{\sum\_{i \in \mathcal{V}} p\_{\ell,i}}\right)^{-1} = \left(\frac{\sum\_{k \in \mathcal{Z}} \sum\_{i \in \mathcal{V}\_{\mathcal{Z},k}} (\kappa\_k^{\prime\prime} \cdot p\_{\ell,i})}{\sum\_{i \in \mathcal{V}} p\_{\ell,i}} + \frac{\sum\_{k \in \mathcal{Z}} \sum\_{i \in \mathcal{V}\_{\mathcal{Z},k}} (\kappa\_k^{\prime\prime} \cdot \hat{\Phi}\_k)}{\sum\_{i \in \mathcal{V}} p\_{\ell,i}}\right)^{-1} . \tag{4.33}$$

Equation (4.33) reveals some interesting connections between the steady-state wholesale prices Λ ⋆ <sup>k</sup> = κ ⋆ k ·(Λ<sup>0</sup> ) <sup>⋆</sup> = a ⋆ ·(κ ′′ k ) ⋆ · Λ ⋆ R and the steady-state retail prices Λ ⋆ <sup>R</sup> = (Λ<sup>0</sup> ) ⋆ , which are discussed below.

**Remark 4.17.** *In the lossless case, the inverse of* a ⋆ *is a weighted average of the individual cell-specific participation factors* (κ ′′) ⋆ *. Conversely,* (4.33) *suggests that for the case of uniform prices* κ ′ = 1*, BoP containment is only fulfilled for the (academic) case of lossless networks. In other words, the social optimum never produces a (monetarily) balanced solution in real-world power systems unless* κ ′′ *is regularized by the scalar multiplication* κ ′ = a ⋆ ·κ ′′ *with* 0 < a<sup>⋆</sup> < 1*. Without this regulation, the monetary imbalance increases as transmission losses increase.* ♢

**Remark 4.18.** *The regularization by means of* a ⋆ *(respectively* a ⋆ *) does not only account for the aggregate transmission losses61, but also ensures an automatic and fair monetary reallocation of these transmission losses to all RPs in real time<sup>62</sup>. Accordingly, since* 1 <sup>T</sup>pˆ<sup>g</sup> > 1 <sup>⊤</sup>pˆ<sup>ℓ</sup> *for lossy networks, there always exists at least one cell* k ∈ Z *with* κ ′ <sup>k</sup> < 1*, regardless of the choice of* κ ′′ <sup>∈</sup> <sup>R</sup> |Z| >0 *. Through the representation* (4.29)*, each CC is able to calculate the momentary value* a ⋆ *in a distributed way.* ♢

#### **4.4.2 Balance of Payments Restoration**

Recall the time-integrated sum of payments from Definition 4.1 with (3.78) and (3.88):

$$
\sigma(t) = \int\_0^t - \sum\_{k \in \mathcal{Z}} \left( \Lambda\_k \cdot \kappa\_k \cdot p\_{\mathbf{g},i} - \hat{\omega}\_k \cdot \hat{p}\_{\mathbf{g},k} \right) + \sum\_{k \in \mathcal{Z}} \Lambda^0 \cdot \hat{p}\_{\ell,k} \,\mathrm{d}t'.\tag{4.34}
$$

Applying the above "primary" control actions for BoP containment (4.3) leads to a finite, but generally nonzero end value σ ⋆ (cf. Remark 4.4). If σ <sup>⋆</sup> > 0, then there exists an excess of

<sup>61</sup> Transmission losses are already considered by the automatic dispatch of p<sup>g</sup> developed in Chapter 3 and therefore do not have to be explicitly contained in (4.33).

<sup>62</sup> By contrast, the current electricity market always requires additional grid tariffs to rebalance the resulting mismatch. See the discussion in Section 2.1.

capital since the RPs' payments are higher than the revenues that have been received by WPs. Conversely, if σ <sup>⋆</sup> < 0, then a deficit of payments exists because the revenues of WPs exceed the payments from RPs. In the following, we will derive the "secondary/tertiary" control actions to *optimally* regulate σ towards zero as t → ∞.

A reasonable choice for the additive restoration signals ϵ κ is given by the first-order lag relation

$$
\dot{\epsilon}\_k^\kappa = -\epsilon\_k^\kappa + \hat{p}\_{\mathbf{g},k} \cdot \sigma + u\_k, \quad \qquad \qquad k \in \mathbb{Z} \tag{4.35}
$$

where u<sup>k</sup> is a cell-specific control input yet to be determined, and where the positive sign of +ˆpg,k · σ ensures that an excess of overall payments by RPs (σ > 0) yields a *positive* correction ϵ κ <sup>k</sup> > 0 (and vice versa),

Now the overall balance (4.34) along with the cell-specific ϵ κ k dynamics (4.35) can be written compactly by the port-Hamiltonian representation

$$
\underbrace{\begin{bmatrix} \dot{\epsilon}^{\kappa} \\ \dot{\sigma} \end{bmatrix}}\_{\dot{\mathbf{x}}\_{\sigma}} = \left( \underbrace{\begin{bmatrix} \mathbf{0} & \hat{\mathbf{p}}\_{\mathbf{g}} \\ -\hat{\mathbf{p}}\_{\mathbf{g}}^{\mathsf{T}} & \mathbf{0} \end{bmatrix}}\_{J\_{\sigma}} - \underbrace{\begin{bmatrix} \mathbf{A}^{\prime} & \mathbf{0} \\ \mathbf{0}^{\mathsf{T}} & \mathbf{0} \end{bmatrix}}\_{R\_{\sigma}} \right) \nabla H\_{\sigma}(\mathbf{x}\_{\sigma}) + \underbrace{\begin{bmatrix} I \\ \mathbf{0}^{\mathsf{T}} \end{bmatrix}}\_{G\_{\sigma}} u + \begin{bmatrix} \mathbf{0} \\ d \end{bmatrix}, \tag{4.36a}
$$

$$H\_{\sigma}(\mathbf{z}\_{\sigma}) = \frac{1}{2} \cdot \Lambda^{0} \cdot (\boldsymbol{\epsilon}^{\kappa})^{\mathsf{T}} \boldsymbol{\epsilon}^{\kappa} + \frac{1}{2} \sigma^{2},\tag{4.36b}$$

where Λ ′ := (1/Λ 0 )· <sup>I</sup> <sup>≻</sup> <sup>0</sup>, and <sup>d</sup> = Λ<sup>0</sup> ·(1 <sup>T</sup>pˆ<sup>ℓ</sup> <sup>−</sup> <sup>p</sup><sup>ˆ</sup> T <sup>g</sup>κ ′ ) is a disturbance that decays to 0 as t → ∞.

**Lemma 4.19.** *Let the cell-specific power injections* pˆ<sup>g</sup> ̸= **0** *be constant and let* (4.3) *hold. Then the autonomous system* (4.36) *exhibits a globally asymptotically stable equilibrium at the origin.*

*Proof.* Consider the Lyapunov function candidate Hσ(xσ) ≻ 0 in (4.36b). The time derivative of Hσ(xσ) along the trajectories of (4.36) is given by H˙ <sup>σ</sup> = −(ϵ κ ) Tϵ κ . Now define the set <sup>E</sup> <sup>=</sup> {x<sup>σ</sup> <sup>∈</sup> <sup>R</sup>|Z|+1 : <sup>H</sup>˙ <sup>σ</sup>(xσ) = 0} <sup>=</sup> {x<sup>σ</sup> <sup>∈</sup> <sup>R</sup>|Z|+1 : <sup>ϵ</sup> <sup>κ</sup> <sup>=</sup> **<sup>0</sup>**}. Since the largest invariant set contained in E is the singleton {(ϵ <sup>κ</sup> <sup>=</sup> **<sup>0</sup>**,<sup>σ</sup> = 0)} and <sup>H</sup>σ(xσ) is radially unbounded, global asymptotic stability of the origin follows by LaSalle's invariance principle [Kha02, Corollary 4.2].

Now the "optimal" control input u = u ⋆ shall be chosen in such a way that the resulting closed-loop trajectory ((ϵ κ ) ⋆ (t), σ<sup>⋆</sup> (t)) of (4.36) is regulated towards zero so that a userdefined quadratic Lagrange-type performance index is minimized. In particular, we specify

$$\min\_{\mathbf{u}} \qquad \int\_{0}^{t\_{\mathrm{f}}} (\boldsymbol{\epsilon}^{\kappa})^{\mathsf{T}} \boldsymbol{Q}\_{1} \boldsymbol{\epsilon}^{\kappa} + \boldsymbol{Q}\_{2} \cdot \boldsymbol{\sigma}^{2} + \boldsymbol{u}^{\mathsf{T}} \boldsymbol{S} \boldsymbol{u} \, \mathrm{d}t \tag{4.37a}$$

subject to (4.36), (4.37b)

where <sup>Q</sup><sup>1</sup> <sup>≽</sup> <sup>0</sup> and <sup>Q</sup><sup>2</sup> <sup>≥</sup> <sup>0</sup> are weighting factors for the transient error of the local restoration signals ϵ κ (t) and the overall balance σ(t), respectively, and the control weight matrix S ≻ 0 penalizes the inputs u(t).

In fact, (4.37) entails an *optimal control problem* with quadratic performance index. This problem class is extensively studied in the following Chapter 5 by deriving explicit feedback control laws for the optimal control input u ⋆ . Furthermore, the proposed methodology will be applicable to general Lagrange-type performance indices (cf. Definition A.7 in Appendix A.2) subject to general (nonlinear) PHS dynamics.

# **4.5 Analysis of the Closed-Loop System**

This section studies the closed-loop equilibrium resulting from the interconnection between the physical plant system (3.49) and frequency and voltage controller (3.110) from Chapter 3 along with the dynamic pricing scheme from Chapter 4. It concludes with a stability analysis of the individual subsystems with respect to this equilibrium.

**Theorem 4.20.** *Each equilibrium of the closed-loop system* (3.49)*,* (3.110)*,* (3.142)*,* (4.29)*,* (4.11)*,* (4.15) *has the following properties:*

	- b) This statement immediately follows from Lemma 3.35.
	- c) For each i ∈ V<sup>S</sup> , the equilibrium condition of (3.110g) equals

$$0 = \langle U\_{\mathbf{f},i}^{\star} - \overline{U}\_{\mathbf{f},i} \rangle\_{\mu\_{\mathcal{S}+,i}}^{+} = \begin{cases} U\_{\mathbf{f},i}^{\star} - \overline{U}\_{\mathbf{f},i}, & (U\_{\mathbf{f},i}^{\star} - \overline{U}\_{\mathbf{f},i} > 0) \text{ and } (\mu\_{\mathcal{S}+,i}^{\star} > 0), \\ 0, & \text{otherwise.} \end{cases} \tag{4.38}$$

Since we assume that no voltage limits are active, it holds that Uf,i < Uf,i. Thus, (4.38) states that µ ⋆ <sup>S</sup>+,i = 0. The same reasoning can be applied to the steady-state conditions (3.110f), (3.110h), and (3.110i) to obtain µS− = **0**, µI− = **0**, and µI<sup>+</sup> = **0**. Using these results in conjunction with the equilibrium conditions of (3.142), we get U ⋆ <sup>g</sup>,i = (U set i ) ⋆ for all i ∈ Vg, which is equivalent to c).


**Remark 4.21.** *Requirement c) in Theorem 4.20 can always be fulfilled by choosing* U ∆ *sufficiently small, which prevents the limits on excitation and inverter output voltage from being reached.* ♢

**Theorem 4.22.** *If* κ ′′ > **0***,* pˆ<sup>g</sup> > **0** *and* pˆ<sup>ℓ</sup> *are constant, then the steady-state* (a ⋆ , ν ⋆ a ) *of the balancing controller* (4.29) *is globally asymptotically stable.*

*Proof.* Let ξ<sup>M</sup> := (a, νa) be the state of (4.29), and consider the Lyapunov function candidate

$$V(\xi\_{\mathbb{A}}) = \frac{1}{2} \Big( (a - a^\star)^\top \tau\_a (a - a^\star) + (\nu\_a - \nu\_a^\star)^\top \tau\_{\nu\_a} (\nu\_a - \nu\_a^\star) \Big).$$

Since τ (·) ≻ 0, it follows that V (ξ<sup>M</sup> ) > 0 for all ξ<sup>M</sup> ̸= ξ ⋆ M . The time derivative of V (ξ<sup>M</sup> ) along the trajectories of (4.29) equals

$$\begin{split} \dot{V}(\xi\_{\mathbb{K}}) &= -(\boldsymbol{a} - \boldsymbol{a}^{\star})^{\top}(\boldsymbol{\kappa}^{\prime\prime} \circ \hat{\boldsymbol{p}}\_{\mathrm{g}} \circ (\boldsymbol{a} - \boldsymbol{a}^{\star})) \\ &+ \underbrace{(\boldsymbol{a} - \boldsymbol{a}^{\star})^{\top}\hat{\boldsymbol{D}}\_{\mathrm{p}}(\nu\_{a} - \nu\_{a}^{\star}) - (\nu\_{a} - \nu\_{a}^{\star})^{\top}\hat{\boldsymbol{D}}\_{\mathrm{p}}^{\top}(\boldsymbol{a} - \boldsymbol{a}^{\star})}\_{=0} \end{split} \tag{4.39}$$

$$\hat{\mathbf{p}} = (\mathbf{a} - \mathbf{a}^\star)^\top \underbrace{-\text{diag}(\kappa'' \circ \hat{\mathbf{p}}\_\mathbf{g})}\_{T} (\mathbf{a} - \mathbf{a}^\star) . \tag{4.40}$$

As κ ′′ > **0** and pˆ<sup>g</sup> > **0** hold per definition, the matrix T in (4.40) is negative definite and thus <sup>V</sup>˙ (ξ<sup>M</sup> ) <sup>≤</sup> <sup>0</sup>. We define the set <sup>E</sup> <sup>=</sup> {ξ<sup>M</sup> <sup>∈</sup> <sup>W</sup> : <sup>V</sup>˙ (ξ<sup>M</sup> ) = 0} = {(a, νa) : a = a <sup>⋆</sup>}. The largest invariant set contained in E is {ξ ⋆ <sup>M</sup>}. As (4.29) is derived using distributed consensusbased control and the centralized steady-state variable a ⋆ is unique (i.e. the unique solution of (4.28)), it follows by Lemma B.1 that ξ ⋆ M is unique as well. Since the largest invariant set contained in E is a singleton and V (ξ<sup>M</sup> ) is radially unbounded, it thus follows from LaSalle's invariance principle that ξ ⋆ M is globally asymptotically stable [Kha02, Corollary 4.2].

**Theorem 4.23.** *If* ℧ *and* A *are constant, then the equilibrium* ϕ ⋆ *of the zonal pricing controller* (4.11) *is globally asymptotically stable.*

*Proof.* Define ϕ ′ (t) = <sup>τ</sup> <sup>ϕ</sup>ϕ(t) and note that <sup>τ</sup> <sup>ϕ</sup> <sup>≻</sup> <sup>0</sup>. From (4.11), it follows that <sup>1</sup> <sup>T</sup>τ <sup>ϕ</sup>ϕ˙ = 1 <sup>T</sup>ϕ˙ ′ = 0 for all t > <sup>0</sup> since <sup>A</sup> <sup>=</sup> <sup>D</sup><sup>ˆ</sup> <sup>p</sup>diag{|S⃗ |}D<sup>ˆ</sup> T p and 1 <sup>T</sup>Dˆ <sup>p</sup> = **0** <sup>T</sup>. This means 1 <sup>T</sup>ϕ ′ (t) = 1 <sup>T</sup>ϕ ′ (0) for all t > 0. Therefore the unique equilibrium of ϕ ′ (t) is ϕ ′<sup>⋆</sup> = ϕ¯ ′⋆ + ϕ¯′ · 1 with 1 <sup>T</sup>ϕ¯ ′⋆ = 0 and ϕ¯′ = 1 <sup>T</sup>ϕ(0)/N. Using the Lyapunov function candidate V (ϕ ′ ) = τ <sup>ϕ</sup> 2 (ϕ ′ − ϕ ′⋆ ) <sup>T</sup>(ϕ ′ − ϕ ′⋆ ), we get V˙ (ϕ ′ ) = −(ϕ ′ − ϕ ′⋆ ) <sup>T</sup>A(<sup>ϕ</sup> ′ − ϕ ′⋆ ) and V˙ (ϕ ′ ) < 0 for all ϕ ′ ̸= ϕ ′⋆ due to positive semidefiniteness of A and (ϕ ′ − ϕ ′⋆ ) ∈/ ker(A). As V (ϕ ′ ) is radially unbounded, it follows that ϕ ′⋆ (and thus ϕ ⋆ ) is globally asymptotically stable [Kha02, Theorem 4.2].

**Remark 4.24.** *For lossless power networks with* U<sup>i</sup> = const. *and* ϑij ≈ 0*, the Laplacian matrix can be assumed to be constant (cf.* (4.13) *in Remark 4.8). Thus, the zonal pricing controller* (4.11) *turns into a* distributed formation controller *[OSFM07], where the resulting state trajectory* ϕ(t) *is a weighted average of the (time-varying) reference signal given by the inputs* ψ(t)*, cf. [FYL06, Ren07, KvC*<sup>+</sup>*19].* ♢

Under the premise of timescale separation, the stability statements for the underlying frequency and voltage controller can be upheld. This feature is formalized in the next theorem.

**Theorem 4.25.** *Let* x = col{xp, p<sup>g</sup> , µ<sup>g</sup>−, µg+, U<sup>f</sup> , µS−, µS<sup>+</sup>, UI, µI−, µI<sup>+</sup>} *denote the state of* (3.49)*,* (3.110)*. Furthermore, let* κ *and* Uˆ set *be fixed, the Hessian of* Hp(xp) *be positive definite in a neighborhood around* x ⋆ p *and let the conditions of Lemmas 3.39 and 3.41 hold in a neighborhood around* x ⋆ *. Then the closed-loop equilibrium* x ⋆ *of the interconnection of the physical plant system* (3.49) *with the frequency controller* (3.110a)*,* (3.110d)*–* (3.110e)*,* (3.110j)*–*(3.110k) *and voltage controller* (3.110b)*–*(3.110c)*,* (3.110f)*–*(3.110i) *is stable in a neighborhood around* x ⋆ *.*

*Proof.* (see Appendix C.1)

**Corollary 4.26.** *Let* x = col{xp, p<sup>g</sup> , µ<sup>g</sup>−, µg+, U<sup>f</sup> , µS−, µS<sup>+</sup>, UI, µI−, µI<sup>+</sup>} *denote the state of* (3.49)*,* (3.110) *where* (3.49) *is* lossless*, i.e.* G = 0*. Furthermore, let* κ *and* Uˆ set *be fixed and the Hessian of* Hp(xp) *be positive definite in a neighborhood around* x ⋆ p *. Then the closed-loop equilibrium* x ⋆ *of the interconnection of the physical plant system* (3.49) *with the frequency controller* (3.110a)*,* (3.110d)*–*(3.110e)*,* (3.110j)*–*(3.110k) *and voltage controller* (3.110b)*–*(3.110c)*,* (3.110f)*–*(3.110i) *is stable in a neighborhood around* x ⋆ *.*

*Proof.* With G = 0, condition W2(x) ≤ 0 in the proof of Theorem 4.25 is automatically fulfilled, since for lossless grids it holds that <sup>ρ</sup> <sup>=</sup> **<sup>0</sup>** and <sup>R</sup>p(xp) = <sup>R</sup><sup>p</sup> <sup>≽</sup> <sup>0</sup>.

The requirements for constant parameters stated in Theorem 4.22, Theorem 4.23 and Theorem 4.25 clearly indicate the ubiquitous need for *sufficient timescale separation* between the individual subsystems (see Subsection 3.4.1 and references therein). Even if there is no quantitative certificate, sufficient timescale separation can be managed conveniently by the controller parameters τ (·) . In particular, to meet the requirements of Theorem 4.25 that κ and Uˆ set are constant, τ <sup>ϕ</sup> and τ <sup>U</sup>set must be chosen sufficiently large compared to the controller parameters in (3.110). In order to meet the requirements of Theorem 4.23, τ − ℧ and τ + ℧ must be larger than τ <sup>ϕ</sup>. In order to cope with Theorem 4.22, τ <sup>a</sup> and τ <sup>ν</sup><sup>a</sup> must be larger than τ <sup>ϕ</sup> and the other controller parameters in (3.110). Thus, the individual controllers operate on three different timescales<sup>63</sup> as depicted in Fig. 4.2. We can clearly see the correlation between the three components in Fig. 4.2 to Contributions 1, 2, and 4 outlined in Fig. 2.3. In Fig. 4.7, the individual dependencies and signal flows between these three components are plotted in more detail. A comparison with today's separation of timescales shown in Fig. 2.2 indicates that by unifying all tasks into three sub-tasks, there is a sufficiently large "gap" between the individual controller parameters possible—specifically, a factor of at least 100 can be chosen between each τ (·) . For further classification and conclusions, please refer to Chapter 7.

# **4.6 Discussion**

This chapter has presented a distributed control scheme for the automatic settlement of temporally and spatially differentiated prices. The developed continuous-time pricing scheme which is solely based upon neighbor-to-neighbor communication between CCs comprises of two conceptually distinct steps:

In the first step, a dynamic locational pricing scheme is provided by means of an automatic control of cell-specific participation factors κ ′′ k (t). By modulating the uniform price Λ 0 (t) locally, the resulting pricing strategies are able to account for the current *scarcity of (marginal) production* in a specific location at a specific point in time. In Section 4.3, we have introduced different approaches with and without integral action for the practical choice of κ ′′(t). In this context, special emphasis has been put on the development of a new real-time approach for distributed, flow-based congestion control. Through its "holistic" consideration of both active and reactive power flows along with integral action, the congestion controller enables the possibility to accurately incorporate (and thus exploit) the thermal load capacity of each separate inter-cell transmission line.

In the second step, *regularized* zonal prices κ ′ k (t) are generated out of κ ′′ k (t) such that each resulting closed-loop equilibrium satisfies BoP containment. A salient feature of this regularization mechanism is that the allocation of WPs' power injections obtained in the first step are left unaltered, while at the same time all RPs are provided with a retail price Λ<sup>R</sup> = λ ⋆ that is independent of the individual cell and by which all transmission losses are recompensed instantaneously. The combination with the optimization-based frequency and voltage controller

<sup>63</sup> Note that these three timescales do not correspond to the classical hierarchy of primary, secondary and tertiary control. Instead, the controller (3.110) presented in Sections 3.4 and 3.5 covers all aspects of the classical frequency and voltage control, while the above control layers shown in Fig. 4.2 address additional (higher-level) objectives as suggested in Contributions 2 and 4.

**Figure 4.7:** Controller structure for real-time zonal pricing with three different timescales. Subsystems that are depicted at the top act at slower timescales, those depicted at the bottom act at faster timescales.

(3.110) deployed in Chapter 3 provides economically efficient equilibria where the adherence of network constraints is fulfilled at market clearing without the need of a "global" (superordinate) authority, after-market settlements, or any other type of side payments. Moreover, due to the automatic regulation of wholesale and retail prices, no additional BRPs are necessary any longer.

So far, we have not developed an explicit procedure for determining the restoration signal ϵ κ (t) mentioned in Section 4.1 and Subsection 4.4.2, which allows the resulting participation factors κ<sup>k</sup> = κ ′ <sup>k</sup> +ϵ κ k to even fulfill (4.2), so that any residual imbalances of costs and revenues that may have occured during transient processes can be settled in an optimal way with regard to the optimal control problem (4.37). The methodological tools necessary for this task are derived in the next chapter.

# **5 Optimal Control of Port-Hamiltonian Systems**

This chapter is dedicated to the class of optimal control problems for input-state-output PHSs (A.13) with Lagrange-type objective functions (A.8) which is a generalization of the BoP restoration problem (4.37). Section 5.1 gives a brief introduction to main research branches in the field of PHS control. Section 5.2 presents a learning-based methodology for general optimal control problems with input-state-output PHSs. Section 5.3 extends the results from Section 5.2 to the multi-player case. The main results of Sections 5.2 and 5.3 give rise to Contribution 3. Section 5.4 first revisits the application of the proposed methodology to BoP restoration (4.37) in order to deliver the remaining steps of Contribution 2, and then outlines worthwhile avenues for further research<sup>64</sup> .

# **5.1 Introduction**

PHS modeling has had a major impact with respect to energy-based control design (see. e.g. [OvMM01, Kug01, OvME02, DBMS09, vJ14]. It opens a different perspective on viewing dynamical systems closer to physics and the fundamental concept of energy, which allowed for valuable advancements in science and engineering practice. Due to their specific energy-based structure with power-conjugated port variables (u, y) and a Hamiltonian H(x) representing the total stored energy in the system, the concept of energy plays a central role in controller design for PHSs. In particular, as the PHS representation has a strong link to passivity properties (which is exemplarily seen in Subsection 3.4.3), the vast majority of control methods for PHSs rely on *passivity-based control* (PBC).

Besides feedback interconnection of passive systems [SJK97, p. 107ff.], a common theme in standard PBC is the passivation and asymptotic stabilization (under an additional detectability condition) via static state feedback u(x) = −k(x) [SJK97, Chapter 2.4; van17, Chapters 5,7; OGC04, OvCA08]. Both approaches are comparable in the sense that the controller parameters are not unique, but must be found implicitly by solving *matching PDEs*, which entails a high number of degrees of freedom and, more severely, a high degree of computational complexity. In order to reduce this complexity, the degrees of freedom are usually confined at certain points by setting a subset of the free parameters to a fixed value, thus leading to sub-categories such as algebraic or (non-)parameterized IDA-PBC, see [van17, Chapter 7] for a comprehensive survey on PBC methods for PHSs. For the passivity-based state feedback controller design, an *inverse optimality* property can be characterized in both continuous [SJK97, p. 107ff.; van17, Theorem 3.5.1] and discrete time [MNC15]. For input-state-output PHSs, it is known

<sup>64</sup> Preliminary versions of the results of this chapter have been published in the journal papers [KJSSH21, KJSMH22].

that a static feedback controller is optimally stabilizing with respect to the specific performance index

$$\int \mathcal{J}\left(\mathbf{x}(t), \mathbf{u}(t)\right) = \frac{1}{2} \int\_0^{t\_\mathrm{f}} \ell(\mathbf{x}(t)) + \mathbf{u}^\top(t)\mathbf{u}(t) \,\mathrm{d}t \tag{5.1}$$

with

$$\ell(x(t)) = -\left(\frac{\partial V(x(t))}{\partial x}\right)^{\mathsf{T}} f(x(t)) + \frac{1}{2}k^{\mathsf{T}}(x(t))k(x(t)),\tag{5.2}$$

if and only if the open-loop system is output feedback passive with a positive definite storage function V (x(t)) and provided that an additional detectability condition is fulfilled, see [van17, p. 54ff.] and [SJK97, Theorem 3.30] for a detailed discussion. However, classical, PBC methods are in general not designed for optimal control problems in a more practical setup, where the "forward" solution to a given, arbitrary optimization problem with a general performance index (A.8) is sought. E.g. in (4.37), an optimal restoration signal (ϵ κ ) ⋆ (t) has to be determined that minimizes the sum of the state error (weighted by Q<sup>1</sup> and Q2) and control input error (weighted by S).

In the following section, we consider the general case to (4.37) with an arbitrary Lagrangian performance index. This problem class is equal to (A.8) with <sup>P</sup> <sup>=</sup> {1}, and thus refered to as the *single-player case* throughout this chapter.

## **5.2 Single-Player Case**

This section presents a new continuous-time adaptive feedback controller for the optimal control of input-state-output PHSs (A.13) with respect to general Lagrangian performance indices (A.8) with <sup>P</sup> <sup>=</sup> {1}. The proposed control law implements an online learning procedure which uses the Hamiltonian H(x) of the system as an initial value function candidate. Subsection 5.2.1 formalizes the problem class and briefly discusses related work. Subsection 5.2.2 presents an analytical solution to a *modified optimal control* problem. Subsection 5.2.3 provides necessary and sufficient conditions for H(x) to serve as initial value function candidate. Subsection 5.2.4 develops an adaptation strategy by which the modified optimal controller is able to learn the (original) optimal control law. For this adaptive optimal controller, Subsection 5.2.5 proves the asymptotic stability of the closed-loop equilibrium and gives an illustrative example. Subsection 5.2.6 summarizes the main result.

#### **5.2.1 Problem Definition and Related Work**

**Definition 5.1 (Optimal Control Problem for PHSs).** *Find the feedback control law* u(x) *as the solution of*

$$\min\_{\mathbf{u}(t)}\qquad\qquad\frac{1}{2}\int\_{0}^{t\_{\mathrm{f}}} \left(\ell(\mathbf{z}(t)) + (\mathbf{u}(t))^{\mathsf{T}}\mathbf{S}(\mathbf{z})\mathbf{u}(t)\right)\mathrm{d}t\tag{5.3a}$$

$$\text{subject to} \quad \dot{x}(t) = (\mathbf{J}(x(t)) - \mathbf{R}(x(t))) \frac{\partial H(x(t))}{\partial x(t)} + \mathbf{G}(x(t))u(t) \tag{5.3b}$$

*with state vector* <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>*, input vector* <sup>u</sup> <sup>∈</sup> <sup>R</sup><sup>m</sup>*, skew-symmetric interconnection matrix* <sup>J</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>×<sup>n</sup>*, positive semidefinite dissipation matrix* <sup>R</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>×<sup>n</sup>*, input matrix* <sup>G</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>×<sup>p</sup> *, positive definite Hamiltonian* <sup>H</sup>(x) : <sup>R</sup><sup>n</sup> <sup>→</sup> <sup>R</sup>*, positive definite* <sup>ℓ</sup>(x) : <sup>R</sup><sup>n</sup> <sup>→</sup> <sup>R</sup> *and positive definite* <sup>S</sup>(x) <sup>∈</sup> <sup>R</sup><sup>m</sup>×<sup>m</sup>*.*

In the case of *linear* input-state-output PHSs, the Hamiltonian H(x) is quadratic, which allows to calculate an optimal controller by using state-dependent Riccati equations [PZCL11]. In [MJMF<sup>+</sup>09, GCS09, MFMC<sup>+</sup>08], the necessary conditions which follow from Pontryagin's Maximum Principle are used to derive an explicit expression for the optimal feedback controller, provided the Hamiltonian of the system is quadratic. The authors in [WHLM18] provide fulland reduced-order LQR controllers for linear input-state-output PHSs. Further extensions of LQ-optimal control for PHSs are given for stochastic or infinite-dimensional spaces [LW18] and boundary control systems [LLL20].

While there is a rich theory available for linear input-state-output PHSs, the general solution of optimal control problems for *nonlinear* input-state-output PHSs remains challenging due to the necessity of explicitly solving the HJB equation which is a nonlinear PDE and thus hard to solve. Recent publications use *adaptive dynamic programming* (ADP) methods to find an iterative solution of the HJB equation. In [ASF<sup>+</sup>20], a combination of neuro-fuzzy and backstepping control is proposed for adaptive control of input-state-output PHSs. A profound overview on recent adaptive and learning-based control methods for PHSs can be found in [NLJB16]. If the performance index of the optimal control problem has a specific structure and the system dynamics are given by a Hamiltonian system with controlled Hamiltonian H(x,u), iterative learning control [Fuj03, FHS03] and iterative feedback tuning methods [FK08b] have been proposed. For fully actuated mechanical PHSs, the authors in [OFM<sup>+</sup>20] propose an adaptive path-following controller from a training trajectory using Bayesian estimation and the authors in [ZDC20] propose a deep learning-based algorithm for optimal estimation of the system dynamics. [NLJB14] and [SBNL15] use actor-critic reinforcement learning schemes to minimize the error between the resulting closed-loop system and a given desired closed-loop system without the need of explicitly solving the matching PDE of the employed passivitybased controller. However, these approaches suffer from the dissipation obstacle<sup>65</sup>, such that the Hamiltonian of the desired closed-loop system cannot be chosen freely.

<sup>65</sup> The dissipation obstacle [OvMM01] states that the Hamiltonian H(x) can only be shaped for coordinates that are not affected by physical damping. This limits the applicability of these controller classes to certain physical domains, see e.g. the discussion in [ZOJS15].

For the more general class of *continuous-time input-affine nonlinear systems*, ADP methods [BJ16] are proposed where the optimal value function V ⋆ (x) can be found iteratively using a weighted sum of basis functions [VL10, JJ13]. However, a proper set of initial weights leading to a stabilizing controller has to be found by educated guessing.

In summary, existing optimal control approaches of input-state-output PHSs hinge on very specific sub-classes of performance index, system dynamics, or both. Likewise, ADP methods for input-state-output PHSs as well as for the more general class of continuous-time, inputaffine nonlinear systems are non-constructive in the sense that they require the intransparent guessing of initial weights for a stabilizing value function candidate. Hence, to the best of the author's knowledge, there exist no explicit control schemes for dynamic optimal control problems with generalized Lagrangian performance index and a general input-state-output PHS.

#### **5.2.2 Modified Optimal Control for Port-Hamiltonian Systems**

The initial step of our controller design is based on a trick originally outlined in [SK00]. By multiplying the dynamic constraints (5.3b) with the gradient of a *control-Lyapunov function* (CLF) V (x), we obtain a MOC problem that allows for an analytical solution of an asymptotically stabilizing u(x(t)) = −k(x(t)).

**Definition 5.2 (Control-Lyapunov Function (CLF) [FK08a, p. 46]).** *Let* x˙ = f(x,u)*,* <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>*,* <sup>u</sup> <sup>∈</sup> <sup>R</sup><sup>m</sup> *with* <sup>f</sup>(**0**n, **<sup>0</sup>**m) = **<sup>0</sup>**n*. Then a radially unbounded, positive definite function* <sup>V</sup> : <sup>R</sup><sup>n</sup> <sup>→</sup> <sup>R</sup>*, fulfilling*

$$\forall x \neq 0\_n: \qquad \inf\_{u} \left\{ \left( \frac{\partial V}{\partial x} \right)^{\mathsf{T}} f(x, u) \right\} < 0 \tag{5.4}$$

*is said to be a* CLF*.*

**Remark 5.3.** *For input-affine nonlinear systems*

$$
\dot{x}(t) = f(x(t)) + g(x(t))u(t),\tag{5.5}
$$

*condition* (5.4) *is equivalent to [KA01, p. 641]*

$$\left(\frac{\partial V}{\partial x}\right)^{\mathsf{T}}\mathbf{G}(x) = \mathbb{0}\_{m} \Longrightarrow \left(\frac{\partial V}{\partial x}\right)^{\mathsf{T}}\mathbf{f}(x)\left\{ \begin{aligned} &<0, &\mathbf{z} \neq \mathsf{0}\_{n},\\ &=0, &\mathbf{z} = \mathsf{0}\_{n}. \end{aligned} \tag{5.6}$$

♢

It is well known that a CLF ensures the existence of an input u = −k(x) such that the closed-loop system is asymptotically stable at the origin<sup>66</sup> .

<sup>66</sup> The formal statement that the existence of a CLF is necessary and sufficient for stabilizability of a nonlinear system is given by Artstein's Theorem [Art83]. A first explicit control law that applies this finding in practice was provided by *Sontag's universal formula for stabilization* [Son89].

In particular, provided that a suitable CLF is given, it was shown in [SK00] that for general input-affine nonlinear systems (5.5), the following MOC problem can be solved explicitly:

**Definition 5.4 (Modified Optimal Control (MOC) Problem for PHSs).**

$$\min\_{\mathbf{u}} \qquad \qquad \frac{1}{2} \int\_{0}^{t\_{\ell}} \left( \ell(x) + \mathbf{u}^{\top} \mathbf{S}(x) \mathbf{u} \right) \mathrm{d}t \tag{5.7a}$$

$$\text{subject to} \quad \dot{V}(x) = \left(\frac{\partial V(x)}{\partial x}\right)^{\mathsf{T}} \left( (J(x) - R(x)) \frac{\partial H(x)}{\partial x} + G(x)u \right), \qquad \text{(5.7b)}$$

**Lemma 5.5 (Modified Optimal Control Law).** *Let* V (x) *be a CLF of* (5.5)*. Then an exact solution of the MOC problem* (5.7) *is given by the continuous control law*

$$u^\star = \begin{cases} -S^{-1}(x)G^\top(x)\frac{\partial V}{\partial x}\Upsilon(x), & S\gamma \neq 0, \\ 0, & S\gamma = 0, \end{cases} \tag{5.8}$$

*where*

$$\Upsilon(x) := \frac{f\_{\Upsilon} + \sqrt{(f\_{\Upsilon})^2 + Q\_{\Upsilon} \cdot S\_{\Upsilon}}}{S\_{\Upsilon}},\tag{5.9}$$

$$f\_{\Upsilon}(x) := \left(\frac{\partial V}{\partial x}\right)^{\Upsilon} f(x),\tag{5.10}$$

$$S\_{\Upsilon}(x) := \left(\frac{\partial V}{\partial x}\right)^{\Upsilon} G(x) S^{-1}(x) G^{\Upsilon}(x) \frac{\partial V}{\partial x},\tag{5.11}$$

$$Q\_{\Upsilon}(x) := \ell(x). \tag{5.12}$$

*Proof.* The proof follows the lines of [SK00, Theorem 4.4]. For optimization problem (5.7), we get the Hamiltonian

$$\mathcal{H}^{\ell}(x, u, \Upsilon) = \frac{1}{2}\ell(x) + \frac{1}{2}u^{\top}\mathcal{S}(x)u + \Upsilon \cdot \left(\frac{\partial V}{\partial x}\right)^{\top} \left(\mathcal{f}(x) + \mathcal{G}(x)u\right) \tag{5.13}$$

with scalar Lagrange multiplier Υ. Application of the control equation <sup>∂</sup><sup>H</sup> ∂u != **0** leads to (5.8). As stated in [SK00, Theorem 4.3], a free final time and fixed final state in (5.7a) implies that for each optimal trajectory (x ⋆ , Υ<sup>⋆</sup> ) it has to hold that H (x ⋆ ,u ⋆ , Υ<sup>⋆</sup> ) = 0, which is equivalent to

$$\frac{1}{2}(\Upsilon^\star)^2 \left(\frac{\partial V}{\partial x}\right)^\mathsf{T}(x^\star)\mathbf{G}(x^\star)S^{-1}(x^\star)\mathbf{G}^\top(x^\star)\frac{\partial V}{\partial x}(x^\star)$$

$$-\Upsilon^\star \left(\frac{\partial V}{\partial x}\right)^\mathsf{T}(x^\star)\mathbf{f}(x^\star) - \frac{1}{2}r(x^\star) = 0. \tag{5.14}$$

Since (5.14) is a quadratic function in Υ, it has the explicit solution

$$\int\_{\Gamma} \frac{f\_{\Upsilon} \pm \sqrt{(f\mathbf{r})^2 + Q\mathbf{r} \cdot S\mathbf{r}}}{S\mathbf{r}}, \quad \left(\frac{\partial V}{\partial x}\right)^{\mathsf{T}} G(x) \neq \mathbf{0},\tag{5.15a}$$

$$\Upsilon(x) = \begin{cases} \dots \\ -\frac{Q\_{\Upsilon}}{2f\_{\Upsilon}}, \end{cases} \qquad\qquad\qquad \begin{pmatrix} \dots \\ \frac{\partial V}{\partial x} \end{pmatrix}^{\top} \mathbf{G}(x) = \mathbf{0} \tag{5.15b}$$

with fΥ, QΥ, S<sup>Υ</sup> as in (5.10)–(5.12). Note that the "+" solution in (5.15a) implies Υ(x) > 0, thus the "−" solution in (5.15a) is discarded (cf. (5.9)) since it implies Υ(x) < 0, which always leads to an unstable solution. Moreover, note that ∂V ∂x <sup>T</sup> <sup>G</sup>(x) = **<sup>0</sup>** implies <sup>f</sup><sup>Υ</sup> <sup>&</sup>lt; <sup>0</sup>, since V (x) is a CLF. It can be shown by de L'Hospital's rule [Sac01, pp. 88, 186] that the Lagrange multiplier Υ(x) in (5.15) is continuous even for the case ∂V ∂x <sup>T</sup> <sup>G</sup>(x) = **<sup>0</sup>**. Hence, it can be fully described with the compact notation (5.9).

#### **5.2.3 Control-Lyapunov Functions for Port-Hamiltonian Systems**

In the following, we investigate under which conditions the Hamiltonian H(x) is a CLF.

**Lemma 5.6.** *Consider an input-state-output PHS as in* (A.13)*. Then* H(x) *is a CLF for* (A.13)*, if and only if*

$$\forall x \in \mathcal{X}\_G: \quad \left(\frac{\partial H}{\partial x}\right)^{\mathsf{T}} R(x) \frac{\partial H}{\partial x} > 0,\tag{5.16}$$

*where* <sup>X</sup><sup>G</sup> := {<sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup> : <sup>G</sup><sup>T</sup> (x) ∂H <sup>∂</sup><sup>x</sup> = 0, x ̸= **0**}*.*

*Proof.* Since J(x) is skew-symmetric and H(x) is a CLF, it holds by definition for all x ̸= **0** (cf. (5.4)) that

$$\begin{split} & \inf\_{\mathbf{u}} \{ \dot{H}(\mathbf{x}) \} \\ &= \inf\_{\mathbf{u}} \left\{ \left( \frac{\partial H}{\partial \mathbf{x}} \right)^{\mathsf{T}} \left( (\mathbf{J}(\mathbf{x}) - \mathbf{R}(\mathbf{x})) \frac{\partial H}{\partial \mathbf{x}} + \mathbf{G}(\mathbf{x}) u \right) \right\} \\ &= \inf\_{\mathbf{u}} \left\{ - \left( \frac{\partial H}{\partial \mathbf{x}} \right)^{\mathsf{T}} \mathbf{R}(\mathbf{x}) \frac{\partial H}{\partial \mathbf{x}} + \left( \frac{\partial H}{\partial \mathbf{x}} \right)^{\mathsf{T}} \mathbf{G}(\mathbf{x}) u \right\} \\ &< 0. \end{split} \tag{5.17}$$

If ∂H ∂x <sup>T</sup> <sup>G</sup>(x) ̸<sup>=</sup> **<sup>0</sup>**, then there always exists an input <sup>u</sup> ′ such that ∂H ∂x <sup>T</sup> <sup>G</sup>(x)<sup>u</sup> ′ < **0** is fulfilled. If ∂H ∂x <sup>T</sup> <sup>G</sup>(x) = **<sup>0</sup>**, then the first term inside the brackets in (5.17) needs to be negative whenever x ̸= **0**, i.e.

$$\left(\frac{\partial H}{\partial x}\right)^{\mathsf{T}}\mathbf{G}(x) = 0 \quad \Rightarrow \quad -\left(\frac{\partial H}{\partial x}\right)^{\mathsf{T}}\mathbf{R}(x)\frac{\partial H}{\partial x} < 0,\tag{5.18}$$

which is equivalent to (5.16). Conversely, if (5.16) is satisfied, then the definition of a CLF is automatically fulfilled since X<sup>G</sup> is the space where ∂V ∂x <sup>T</sup> <sup>G</sup>(x) = **<sup>0</sup>** in Definition 5.26 and hence (5.16) is equivalent to (5.6).

**Corollary 5.7.** *Consider an input-state-output PHS as in* (A.13)*. Then* H(x) *is a CLF for* (A.13)*, if and only if* (A.13) *is zero-state detectable.*

*Proof.* Since the input-state-output PHS (5.3b) is equipped with the passive output (A.13b), it holds that <sup>X</sup><sup>G</sup> <sup>=</sup> {<sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup> : <sup>y</sup>(x) = 0} and thus condition (5.16) is identical with the definition of a zero-state detectable input-affine nonlinear system (5.5) given in [van17, p. 47].

**Corollary 5.8.** *Consider an input-state-output PHS as in* (A.13)*. Then* H(x) *is a CLF, if* rank(R(x)) = <sup>n</sup> *holds for all* <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>*.*

*Proof.* Trivially, if R(x) has full rank, then ∂H ∂x <sup>T</sup> <sup>R</sup>(x) ∂H ∂x is positive and thus (5.16) is fulfilled for all <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>, which also implies for all <sup>x</sup> ∈ XG.

Next, we introduce a necessary and sufficient condition under which the Hamiltonian of a *linear* input-state-output PHS is a CLF:

\*\*Lemma 5.9.\*\* Consider the linear input-state-output PHS dynamics

$$\dot{x} = (J - R)\frac{\partial H}{\partial x} + \mathbf{G}(x)u \tag{5.19}$$

$$\text{with } \mathbf{J} = -\mathbf{J}^{\top}, R \succeq 0, H(x) = \frac{1}{2} \mathbf{z}^{\top} \mathbf{Q}x \text{ and } \mathbf{Q} \succ 0. \text{ Then } H(x) \text{ is a CLF, if and only if}$$

$$\ker \{ \mathbf{G}^{\top} \mathbf{Q} \} \cap \ker \{ \mathbf{Q}^{\top} \mathbf{R} \mathbf{Q} \} = \emptyset. \tag{5.20}$$

*Proof.* For linear input-state-output PHSs, it holds that ∂H <sup>∂</sup><sup>x</sup> <sup>=</sup> Qx and thus <sup>X</sup><sup>G</sup> <sup>=</sup> {<sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup> : G<sup>T</sup> ∂H <sup>∂</sup><sup>x</sup> <sup>=</sup> <sup>0</sup>} is equal to ker{G<sup>T</sup>Q}, since

$$\forall \, x \in \ker \{ \mathbf{G}^{\mathsf{T}} \mathbf{Q} \}: \qquad \qquad \mathbf{G}^{\mathsf{T}} \frac{\partial H}{\partial x} = \mathbf{G}^{\mathsf{T}} \mathbf{Q} x = \mathbf{0}. \tag{5.21}$$

Consequently, (5.16) in Lemma 5.6 reads as follows:

$$\forall \mathbf{x} \in \ker \{ \mathbf{G}^{\mathsf{T}} \mathbf{Q} \}: \qquad -\left(\frac{\partial H}{\partial \mathbf{z}}\right)^{\mathsf{T}} \mathbf{R} \frac{\partial H}{\partial \mathbf{z}} = -(\mathbf{Q}\mathbf{z})^{\mathsf{T}} \mathbf{R} \mathbf{Q} \mathbf{z} < 0. \tag{5.22}$$

First, it is important to note that −(Qx) <sup>T</sup>RQx in (5.22) is always <sup>≤</sup> <sup>0</sup> for <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>, since the product Q <sup>T</sup>RQ of positive semidefinite matrices is again positive semidefinite [HJ12, p. 431]. Secondly, since x <sup>T</sup>Mx = 0 ⇐⇒ <sup>x</sup> <sup>∈</sup> ker{M} for each <sup>M</sup> <sup>≽</sup> <sup>0</sup> (cf. [HJ12, Observation 7.1.6]), it follows that the equality x <sup>T</sup>Q <sup>T</sup>RQx = 0 holds if and only if <sup>x</sup> <sup>∈</sup> ker{Q <sup>T</sup>RQ}. Consequently, (5.22) holds if and only if ker{GTQ} and ker{<sup>Q</sup> <sup>T</sup>RQ} are disjoint.

Taking Lemma 5.6 into account, MOC can be applied to input-state-output PHSs in a straightforward manner67. However, as a consequence of the modification, the MOC law is optimal to an unintentionally modified objective function. The next lemma further evaluates to what extent the controller (5.8)–(5.12) is also optimal with respect to the original optimal control problem (5.3).

**Lemma 5.10.** *Consider an input-state-output PHS as in* (A.13)*. The (modified optimal) controller* (5.8)*–*(5.12) *is optimal with respect to* (5.3)*, if*

$$\forall x \in \mathbb{R}^n: \qquad \qquad \qquad \Upsilon(x) = 1. \tag{5.23}$$

*Proof.* If <sup>S</sup><sup>Υ</sup> ̸= 0, then it follows from (5.11) that <sup>G</sup><sup>T</sup> ∂V ∂x ̸= **0**. Thus (5.23) is equivalent to S<sup>Υ</sup> − 2f<sup>Υ</sup> = QΥ, and substitution of (5.10)–(5.12) leads to

$$\frac{\ell(x)}{2} - \frac{1}{2} \left( \frac{\partial V}{\partial x} \right)^{\mathsf{T}} \mathbf{G}(x) \mathbf{S}^{-1}(x) \mathbf{G}^{\mathsf{T}}(x) \frac{\partial V}{\partial x} + \left( \frac{\partial V}{\partial x} \right)^{\mathsf{T}} \mathbf{f}(x) = 0,\tag{5.24}$$

which is equivalent to the HJB equation for time-invariant systems. The function V (x) solving (5.24) is the value function V ⋆ (x). This means, if a CLF is found for which Υ(x) = 1 holds for all <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>, then this CLF is equal to the value function <sup>V</sup> ⋆ and thus (5.8) is an optimal control input for (5.3). If S<sup>Υ</sup> = 0, then G<sup>T</sup> (x) ∂V <sup>∂</sup><sup>x</sup> = **0**. Thus, (5.23) is equivalent to Υ(x) = − Q<sup>Υ</sup> 2f<sup>Υ</sup> != 1, which also leads to <sup>S</sup><sup>Υ</sup> <sup>−</sup> <sup>2</sup>f<sup>Υ</sup> <sup>=</sup> <sup>Q</sup>Υ.

A less restrictive requirement can be derived by allowing Υ(x) to have an arbitrary but fixed positive value:

**Corollary 5.11.** *The (modified optimal) controller* (5.8)*–*(5.12) *is optimal with respect to* (5.3)*, if*

$$\forall x \in \mathbb{R}^n: \qquad \qquad \Upsilon(x) = c, \qquad \qquad c \in \mathbb{R}\_{>0}.\tag{5.25}$$

*In this case, the value function* V ⋆ (x)*is a* c*-multiple of the CLF* V (x)*, i.e.* V ⋆ (x) = c·V (x)*.*

*Proof.* (trivial)

<sup>67</sup> For a special subclass of input-state-output PHSs, it can even be shown that apart from H(x), there exist *no other* viable CLF candidates (see Lemma D.1 in Appendix D.1).

**Remark 5.12.** *Lemma 5.10 states that if* Υ(x) ̸= 1*, then the chosen CLF* V (x) *does not solve the HJB equation* (5.24) *and is thus not equivalent to the value function of* (5.3)*. Consequently, the resulting controller* (5.8)*–*(5.12) *is not optimal with respect to* (5.3) *unless* Υ(x) = 1*.* ♢

**Remark 5.13.** *From Corollary 5.11 we can conclude that unless* Υ(x) *converges to a constant value and remains constant even after a disturbance, the chosen CLF cannot be equivalent to the value function. Consequently the resulting controller* (5.8)*–*(5.12) *is not optimal with respect to* (5.3)*. The stationarity of* Υ(t) *over time can therefore be interpreted as an* indicator of fitness *with respect to the original problem* (5.3)*.* ♢

#### **Example 6 (DC Motor)**

*Consider the linear model of a DC motor from [vJ14, p. 29].*

$$
\dot{\mathbf{x}} = \begin{bmatrix} -R & -K \\ K & -b \end{bmatrix} \frac{\partial H(\mathbf{x})}{\partial \mathbf{z}} + \begin{bmatrix} 1 \\ 0 \end{bmatrix} u,\tag{5.26a}
$$

$$y = \begin{bmatrix} 1 & 0 \end{bmatrix} \frac{\partial H(x)}{\partial x},\tag{5.26b}$$

*where* H(x) = 1/2(x 2 <sup>1</sup> + x 2 2 )*, with* u *being the input voltage,* y *being the output current,* b > 0 *being the mechanical damping constant,* R > 0 *being the resistance, and* K > 0 *being the gyrator constant. Now choose* b = R = K = 1 *and consider the MOC problem* (5.7) *with* ℓ(x) = x 2 <sup>1</sup> + x 2 2 *and* S(x) = 1 *subject to* (5.26)*. By numerically solving the algebraic Riccati equation, we can calculate the value function as* V ⋆ (x) = 1/2x <sup>T</sup>P x <sup>≈</sup> 0.21x 2 <sup>1</sup> + 0.02x1x<sup>2</sup> + 0.24x 2 2 *.*

*Fig. 5.1 shows the resulting trajectories of* Υ(t) *depending on different chosen CLFs* V (x) *in* (5.7)*:*

**Figure 5.1:** Trajectory of Υ for different CLFs V (x).

*It can be seen that as* V (x) *is closer to the value function* V ⋆ (x)*,* Υ *is closer to 1. Since* V4(x) *is equal to* V ⋆ (x)*,* Υ *in Fig. 5.1d is equal to 1.*

#### **5.2.4 Adaptive Optimal Control for Port-Hamiltonian Systems**

Even if H(x) qualifies as a CLF as described in Subsection 5.2.3, choosing V (x) = H(x) will generally result in Υ(x) ̸= 1. With Lemma 5.10, this implies that the HJB equation (5.24) is not fulfilled and thus the resulting controller (5.8)–(5.12) is not optimal with respect to the original problem (5.3). For this purpose, in this subsection, we propose an *extended* CLF V (x, w) as a linear combination of H(x) and a weighted sum of basis functions. Later in this subsection, we devise a gradient-based adaptation strategy of the weighting factors w such that V (x, w) fulfills the condition of Lemma 5.10, and present an adaptive optimal controller for (5.3) based on V (x, w). In the following Subsection 5.2.5, we then show that the equilibrium of the closed-loop system is (asymptotically) stable.

#### **Extended CLF**

**Definition 5.14 (Extended CLF).** *Let* <sup>Ξ</sup> : <sup>R</sup><sup>n</sup> <sup>→</sup> <sup>R</sup><sup>r</sup> *be a vector of basis functions. Then* <sup>V</sup> : <sup>R</sup><sup>n</sup> <sup>×</sup> <sup>R</sup><sup>r</sup> <sup>→</sup> <sup>R</sup> *with* <sup>V</sup> (x, <sup>w</sup>) = <sup>H</sup>(x) + <sup>w</sup><sup>T</sup>Ξ(x), w <sup>∈</sup> <sup>R</sup> r (5.27) *is said to be an* extended CLF*.*

**Assumption 5.15.** *The basis functions* Ξ(x) *in* (5.27) *are chosen such that there exists a vector* w<sup>⋆</sup> *of* optimal weights *that allows the actual value function to be parameterized as* V ⋆ (x) = H(x) + (w<sup>⋆</sup> ) <sup>T</sup>Ξ(x)*.*

**Remark 5.16.** *With the Weierstraß approximation theorem, Assumption 5.15 is admissible if the number of basis functions is large (see e.g. [VL10, p. 881; LLW14, p. 1020f.]). This fact will allow us to characterize the deviation from the optimal solution by the distance between* w(t) *and* w<sup>⋆</sup> *.* ♢

With V (x, w) as in (5.27), we obtain

$$\frac{\partial V(x, w)}{\partial x} = \frac{\partial H}{\partial x} + \left(\frac{\partial \Xi}{\partial x}\right)^{\mathsf{T}} w,\tag{5.28}$$

and accordingly the MOC law (5.8) takes the form

$$u^\star = \begin{cases} -S^{-1}(x)G^\top(x)\frac{\partial V}{\partial x}\Upsilon(x,w), & S^\prime\_\Upsilon \neq 0, \\ \mathbb{0}, & S^\prime\_\Upsilon = 0, \end{cases} \tag{5.29}$$

where

$$\Upsilon(x, w) := \frac{f\_{\Upsilon}' + \sqrt{(f\_{\Upsilon}')^2 + Q\_{\Upsilon}' \cdot S\_{\Upsilon}'}}{S\_{\Upsilon}'},\tag{5.30}$$

$$f\_{\Upsilon}^{\prime}(x) := \left(\frac{\partial V}{\partial x}\right)\_{\Upsilon}^{\Upsilon} (\mathbf{J}(x) - \mathbf{R}(x)) \frac{\partial H}{\partial x} - \left(\frac{\partial V}{\partial w}\right)^{\Upsilon} \dot{w},\tag{5.31}$$

$$S\_{\Upsilon}'(x) := \left(\frac{\partial V}{\partial x}\right)^{\Upsilon} K(x) \frac{\partial V}{\partial x},\tag{5.32}$$

$$Q'\_{\Upsilon}(x) := \ell(x),\tag{5.33}$$

$$K(x) := G(x)S^{-1}(x)G^{\dagger}(x). \tag{5.34}$$

Employing the same reasoning as in Lemma 5.10, we will study how to check whether a given CLF (5.27) is equivalent to the value function V ⋆ in more detail.

**Lemma 5.17.** *Let* V (x) = H(x) + (w<sup>⋄</sup> ) <sup>T</sup>Ξ(x) *be a given CLF with* <sup>w</sup><sup>⋄</sup> <sup>∈</sup> <sup>R</sup><sup>r</sup> *. Then* V (x) *is equivalent to the value function* V ⋆ (x) *of* (5.3)*, if and only if* <sup>∀</sup> <sup>x</sup> <sup>∈</sup> <sup>R</sup> n : (x, w<sup>⋄</sup> ) ∈ Q(x, w), (5.35) *where*

$$\mathcal{Q}(x, w) = \left\{ (x, w) \in \mathbb{R}^n \times \mathbb{R}^r : w^\top A(x)w + a^\top(x)w + a(x) = 0 \right\},\tag{5.36}$$

$$\mathbf{A}(x) = \frac{\partial \Xi}{\partial x} \mathbf{K}(x) \left(\frac{\partial \Xi}{\partial x}\right)^{\mathsf{T}},\tag{5.37}$$

$$\mathbf{a}(\mathbf{x}) = 2 \left( \left( \frac{\partial H}{\partial \mathbf{x}} \right)^{\mathsf{T}} \mathbf{K}(\mathbf{x}) \left( \frac{\partial \Xi}{\partial \mathbf{x}} \right)^{\mathsf{T}} - \left( \frac{\partial H}{\partial \mathbf{x}} \right)^{\mathsf{T}} (\mathbf{J}(\mathbf{x}) - \mathbf{R}(\mathbf{x})) \left( \frac{\partial \Xi}{\partial \mathbf{x}} \right)^{\mathsf{T}} \right), \tag{5.38}$$

$$a(\mathbf{x}) = \left(\frac{\partial H}{\partial \mathbf{x}}\right)^{\mathsf{T}} \mathbf{K}(\mathbf{x}) \frac{\partial H}{\partial \mathbf{x}} - 2\left(\frac{\partial H}{\partial \mathbf{x}}\right)^{\mathsf{T}} (\mathbf{J}(\mathbf{x}) - \mathbf{R}(\mathbf{x})) \frac{\partial H}{\partial \mathbf{x}} - \ell(\mathbf{x}).\tag{5.39}$$

*Proof.* Let <sup>w</sup><sup>⋆</sup> <sup>∈</sup> <sup>R</sup><sup>r</sup> be the optimal weighting vector. According to Lemma 5.10, it thus holds that Υ(x, w<sup>⋆</sup> ) = 1 for all <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>. As shown in the proof of Lemma 5.10, this condition is equivalent to S ′ <sup>Υ</sup> − 2f ′ <sup>Υ</sup> − Q′ Υ != 0. Substitution of (5.30)–(5.34) yields

$$\begin{split} 0 &= (\boldsymbol{w}^{\star})^{\top} \frac{\partial \Xi}{\partial x} \mathbf{K}(\boldsymbol{x}) \left(\frac{\partial \Xi}{\partial x}\right)^{\top} \boldsymbol{w}^{\star} + 2 \left( \left(\frac{\partial H}{\partial x}\right)^{\top} \mathbf{K}(\boldsymbol{x}) \left(\frac{\partial \Xi}{\partial x}\right)^{\top} \right. \\ &\quad - \left(\frac{\partial H}{\partial x}\right)^{\top} (\boldsymbol{J}(\boldsymbol{x}) - \mathbf{R}(\boldsymbol{x})) \left(\frac{\partial \Xi}{\partial x}\right)^{\top} \right) \boldsymbol{w}^{\star} \\ &\quad + \left(\frac{\partial H}{\partial x}\right)^{\top} \mathbf{K}(\boldsymbol{x}) \frac{\partial H}{\partial \boldsymbol{x}} - 2 \left(\frac{\partial H}{\partial \boldsymbol{x}}\right)^{\top} (\boldsymbol{J}(\boldsymbol{x}) - \mathbf{R}(\boldsymbol{x})) \frac{\partial H}{\partial \boldsymbol{x}} - \ell(\boldsymbol{x}). \end{split}$$

With Q(x, w) as in (5.36), condition (5.40) can be written as

$$\forall \boldsymbol{x} \in \mathbb{R}^n: \qquad \qquad \quad (\boldsymbol{x}, \boldsymbol{w}^\star) \in \mathcal{Q}(\boldsymbol{x}, \boldsymbol{w}).\tag{5.41}$$

Since the given CLF V (x) is equal to V ⋆ (x) if and only if w<sup>⋄</sup> = w<sup>⋆</sup> , this is equivalent to (5.35).

#### **Adaptation of the Extended Control-Lyapunov Function**

For each fixed <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>, the set <sup>Q</sup>(x, <sup>w</sup>) in (5.36) is a quadric. The basic idea of the following approach relies on the fact that, according to Lemma 5.17, the optimal weighting vector w<sup>⋆</sup> is contained in each quadric, i.e. (w<sup>⋆</sup> ) <sup>T</sup>A(x)w<sup>⋆</sup> + a <sup>T</sup>(x)w<sup>⋆</sup> + a(x) = 0 holds for all <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>. Thus for each arbitrary but fixed <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>, <sup>w</sup><sup>⋆</sup> can be characterized as the minimizer of an objective function J 0 <sup>w</sup>(x, w) := (w<sup>T</sup>A(x)w +a <sup>T</sup>(x)w +a(x))<sup>2</sup> . Moreover, with w<sup>⋆</sup> = arg min<sup>w</sup> J 0 <sup>w</sup>(x, w) and due to the fact that w<sup>⋆</sup> is contained in each quadric Q(x, w), it follows that

$$\left\{\boldsymbol{w}^{\star}\right\} \subseteq \bigcap\_{\boldsymbol{w} \in \mathbb{R}^{n}} \operatorname\*{arg\,min}\_{\boldsymbol{w}} \left\{\mathcal{J}\_{\operatorname{w}}^{0}(\boldsymbol{x}, \boldsymbol{w})\right\}.\tag{5.42}$$

If the basis functions are linearly independent, then the intersection of all quadrics is a singleton and thus

$$\left\{\boldsymbol{w}^{\*}\right\} \equiv \bigcap\_{\boldsymbol{w} \in \mathcal{R}^{n}} \operatorname\*{arg\,min}\_{\boldsymbol{w}} \left\{ \mathcal{J}^{0}\_{\operatorname{w}}(\boldsymbol{x}, \boldsymbol{w}) \right\}.\tag{5.43}$$

Since J 0 <sup>w</sup> is continuously differentiable, this allows for an adaptation of w(t) by a simple gradient flow

$$
\dot{w} = -\alpha \cdot \frac{\partial \mathcal{J}\_{\mathbf{w}}^{0}(x, w)}{\partial w},
\tag{5.44}
$$

where α > 0 defines the *learning rate*. However, note that J 0 <sup>w</sup> is in general *not* strictly convex around w<sup>⋆</sup> , which hampers convergence of (5.44) towards w<sup>⋆</sup> and thus necessitates additional conditions for a sufficient exploration of the state space. To circumvent these requirements, which are often very hard to evaluate, we formulate an extended objective function Jw(x, w) providing strict convexity with respect to w in a neighborhood of w<sup>⋆</sup> as follows:

#### **Definition 5.18 (Extended Objective Function).** *Let*

$$\mathbb{Q}(x, w) = w^{\top} A(x) w + a^{\top}(x) w + a(x) \tag{5.45}$$

*with* A(x)*,* a(x)*,* a(x) *as in* (5.37)*–*(5.39) *be the corresponding quadratic function to the quadric* <sup>Q</sup>(x, <sup>w</sup>) *in* (5.36)*. Then with* <sup>c</sup>1, . . . , <sup>c</sup><sup>r</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>*, the function*

$$\begin{split} \mathcal{J}\_{\mathbf{w}}(\mathbf{z}, \boldsymbol{w}) &= \mathcal{J}\_{\mathbf{w}}^{0}(\mathbf{z} + \mathbf{c}\_{1}, \boldsymbol{w}) + \dots + \mathcal{J}\_{\mathbf{w}}^{0}(\mathbf{z} + \mathbf{c}\_{r}, \boldsymbol{w}) \\ &= \left(\mathbb{Q}(\mathbf{z} + \mathbf{c}\_{1}, \boldsymbol{w})\right)^{2} + \dots + \left(\mathbb{Q}(\mathbf{z} + \mathbf{c}\_{r}, \boldsymbol{w})\right)^{2} \end{split} \tag{5.46}$$

*composed by a linear combination of shifted objective functions* J 0 <sup>w</sup>(x, w) *is said to be an* extended objective function*.*

**Lemma 5.19.** *Let* Jw(x, w)*be an extended objective function as in* (5.46)*. Then* Jw(x, w) *is (locally) strictly convex in an open neighborhood* <sup>M</sup> *of the optimal weights* <sup>w</sup><sup>⋆</sup> *for all* <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>*, if and only if the vectors*

$$w\_l = 2A(x + c\_l)w^\star + a(x + c\_l) \tag{5.47}$$

*with* l = 1, . . . , r *are linearly independent.*

*Proof.* Without loss of generality, we choose c<sup>1</sup> = **0**. As strict convexity with respect to w needs to be shown, the Hessian of J 0 <sup>w</sup> and J<sup>w</sup> (see(5.46)) are studied. The Hessian of J 0 <sup>w</sup> can be written as

$$\frac{\partial^2 \mathcal{J}\_{\mathbf{w}}^0(x, w)}{\partial w^2} = 2 \left( 2A(x)w + a(x) \right) \left( 2A(x)w + a(x) \right)^\top$$

$$+ 4A(x) \left( w^\top A(x)w + a(x)w + a(x) \right). \tag{5.48}$$

Since the optimal weights <sup>w</sup><sup>⋆</sup> need to be part of each quadric <sup>Q</sup>(x, <sup>w</sup>) regardless of the state <sup>x</sup> (see Lemma 5.17), the second summand in (5.48) is equal to zero for w = w<sup>⋆</sup> and accordingly

$$\left. \frac{\partial^2 \mathcal{J}\_{\rm w}^0(x, w)}{\partial w^2} \right|\_{w = w^\star} = 2v\_1 v\_1^\top \tag{5.49}$$

with v<sup>1</sup> = 2A(x)w<sup>⋆</sup> + a(x). Since the Hessian (5.49) is only composed by the multiplication of two vectors, which yields a matrix with identical but scaled row vectors, it is positive semidefinite and has rank one.

For the Hessian of the shifted objective function J 0 <sup>w</sup>(x+c2, w), we obtain in a similar manner

$$\left. \frac{\partial^2 \mathcal{J}\_{\rm w}^0(x + c\_2, w)}{\partial w^2} \right|\_{w = w^\*} = 2v\_2 v\_2^\top \tag{5.50}$$

with v<sup>2</sup> = 2A(x+c2)w<sup>⋆</sup>+a(x+c2). Note that the rank of the matrix (5.50) is one regardless of the shifting c2. The linear combination J 1 <sup>w</sup>(x, w) := J 0 <sup>w</sup>(x, w) + J 0 <sup>w</sup>(x + c2, w) leads to the Hessian

$$\left.\frac{\partial^2 \mathcal{J}\_{\mathbf{w}}^1}{\partial w^2}\right|\_{w=w^\*} = 2\boldsymbol{v}\_1 \boldsymbol{v}\_1^\top + 2\boldsymbol{v}\_2 \boldsymbol{v}\_2^\top. \tag{5.51}$$

The same applies to linear combinations with more than two summands due to the linearity property of differentiation. It can be seen that (5.51) has a maximum rank of two. As strict convexity of J<sup>w</sup> is required, full rank r needs to be satisfied for the Hessian of Jw(x, w) at w = w<sup>⋆</sup> . Thus the question arises, in which case the increase of summands implies a rank increase of the Hessian. Each matrix in (5.49) or (5.50) describes a linear map <sup>R</sup><sup>r</sup> <sup>→</sup> <sup>R</sup><sup>r</sup> of rank one and its image is a subspace of R<sup>r</sup> of dimension one. If the image of <sup>∂</sup> 2J 1 w <sup>∂</sup>w<sup>2</sup> in (5.51) is of dimension two, the rank automatically increases, since dim(im(M)) = rank(M) for an arbitrary matrix M. With the dimension formula for the sum of subspaces [Bos14, p. 47], it follows that dim(U<sup>1</sup> + U2) = dim(U1) + dim(U2) − dim(U<sup>1</sup> ∩ U2) for two arbitrary subspaces U<sup>1</sup> and U2. By setting U<sup>1</sup> = im(v1v<sup>1</sup> <sup>T</sup>) and <sup>U</sup><sup>2</sup> <sup>=</sup> im(v2v<sup>2</sup> <sup>T</sup>) it follows that only if dim(U<sup>1</sup> ∩ U2) = 0, the summation of the two matrices v1v<sup>1</sup> <sup>T</sup> and v2v<sup>2</sup> <sup>T</sup> leads to a rank increase. For each l ∈ {1, . . . , r}, the rank-one matrix vlv<sup>l</sup> <sup>T</sup> = (v<sup>l</sup><sup>1</sup> · v<sup>l</sup> , · · · , v<sup>l</sup><sup>r</sup> · vl) <sup>T</sup> is formed by weighted rows of v<sup>l</sup> <sup>T</sup> with the respective components v<sup>l</sup><sup>i</sup> , i = 1, . . . , r of v<sup>l</sup> ,

$$\begin{aligned} \boldsymbol{\upsilon}\_{l}\boldsymbol{\upsilon}\_{l}\boldsymbol{\upsilon}\_{l}^{\top} = \begin{bmatrix} \boldsymbol{\upsilon}\_{l\_{1}} \cdot \boldsymbol{\upsilon}\_{l} \prescript{\mathsf{T}}{}{\mathsf{T}}\\ \vdots\\ \boldsymbol{\upsilon}\_{l\_{r}} \cdot \boldsymbol{\upsilon}\_{l} \prescript{\mathsf{T}}{}{\mathsf{T}} \end{bmatrix}, \end{aligned} \tag{5.52}$$

and hence its image spans the subspace U<sup>l</sup> = im(vlv<sup>l</sup> <sup>T</sup>) = {<sup>µ</sup> · <sup>v</sup><sup>l</sup> <sup>|</sup> <sup>µ</sup> <sup>∈</sup> <sup>R</sup>}. With regard to J 1 <sup>w</sup> in (5.51) it can be seen that in order to let both sets U<sup>1</sup> and U<sup>2</sup> be disjunct, the linear independence of both vectors v<sup>1</sup> and v<sup>2</sup> is necessary. Graphically, the subspace im(vlv<sup>l</sup> <sup>T</sup>) is a straight line in R<sup>r</sup> , and linear independence leads to non-coinciding straight lines such that dim(U<sup>1</sup> ∩ U2) = 0. Applied to J<sup>w</sup> it becomes clear that J<sup>w</sup> has a Hessian with full rank, if the vectors v<sup>l</sup> in (5.47) with l = 1, . . . , r are linearly independent. Hence, each vector v<sup>l</sup> induces a matrix, implying an increase of one for the rank of the Hessian <sup>∂</sup> <sup>2</sup>J<sup>w</sup> <sup>∂</sup>w<sup>2</sup> , which leads to full rank and thus positive definiteness of <sup>∂</sup> <sup>2</sup>J<sup>w</sup> <sup>∂</sup>w<sup>2</sup> . Moreover, since J<sup>w</sup> is a C 2 function, positive definiteness of the Hessian is preserved for all <sup>w</sup> ∈ M around <sup>w</sup><sup>⋆</sup> .

**Remark 5.20.** *Lemma 5.19 states that the minimum number of summands in* J<sup>w</sup> *to achieve strict convexity is* r*. From a practical point of view, however, it is preferable to compose* J<sup>w</sup> *from more than* r *summands in order to enhance convergence properties.* ♢

Due to the fact that Q(x, w<sup>⋆</sup> ) = 0 is fulfilled for all <sup>x</sup> <sup>∈</sup> <sup>R</sup>n, the optimal weighing factor <sup>w</sup><sup>⋆</sup> is characterized by the strictly convex optimization problem <sup>w</sup><sup>⋆</sup> = arg min<sup>w</sup> {Jw(x, <sup>w</sup>)}. Thus, a given weighting vector w can be adapted by the gradient flow

$$
\dot{w} = -\alpha \cdot \frac{\partial \mathcal{J}\_{\mathbf{w}}(x, w)}{\partial w}, \qquad \qquad \qquad \alpha > 0. \tag{5.53}
$$

# **Example 6 (DC Motor) –** *cont'd Consider again the DC motor model* (5.26) *and choose the extended CLF*

$$V(x, w) = H(x) + \begin{bmatrix} w\_1 & w\_2 & w\_3 \end{bmatrix} \begin{bmatrix} x\_1^2 \\ x\_1 x\_2 \\ x\_2^2 \end{bmatrix} . \tag{5.54}$$

*Fig. 5.2 shows the contour plot of the resulting objective function* J 0 <sup>w</sup>(x, w) *for fixed* w<sup>3</sup> = w ⋆ 3 *and some states:*

**Figure 5.2:** Contour plot of J <sup>0</sup> <sup>w</sup>(x, w) with w<sup>3</sup> = w<sup>⋆</sup> 3 .

*The yellow dot marks the optimal point* (w ⋆ 1 , w<sup>⋆</sup> 2 ) <sup>T</sup> *in each case. It can be seen that for each of the states in Figs. 5.2a–c, the optimal point is a minimizer of* J 0 <sup>w</sup>(x, w)*. However,* J 0 <sup>w</sup>(x, w) *has no* strict *minimizer for any of the states in Fig. 5.2a–c, which indicates that* J 0 <sup>w</sup>(x, w) *is not strictly convex around* w<sup>⋆</sup> *.*

*Now consider the extended objective function* <sup>J</sup>w(x, <sup>w</sup>) *with shifts* <sup>c</sup><sup>0</sup> = (0, 0)<sup>T</sup>*,* <sup>c</sup><sup>1</sup> <sup>=</sup> (−0.5, <sup>−</sup>0.5)<sup>T</sup>*,* <sup>c</sup><sup>2</sup> = (0, <sup>1</sup>.5)<sup>T</sup>*, and* <sup>c</sup><sup>3</sup> = (1.5, 0)<sup>T</sup>*. Fig. 5.3 shows the contour plot of* Jw(x, w) *for fixed* w<sup>3</sup> = w ⋆ 3 *and some states:*

**Figure 5.3:** Contour plot of Jw(x, w) with w<sup>3</sup> = w<sup>⋆</sup> 3 .

*In Figs. 5.3a–c, the contour lines around* (w ⋆ 1 , w<sup>⋆</sup> 2 ) *are closed, indicating that* Jw(x, w) *is strictly convex around* w<sup>⋆</sup> *.*

*Now the adaptation procedure* (5.53) *is applied with both* J 0 <sup>w</sup>(x, w) *and* Jw(x, w)*. Fig. 5.4 shows the resulting trajectories of* w *and* Υ*.*

**Figure 5.4:** Adaptation results of the weights w depending on the chosen objective function.

*While in Fig. 5.4a,* w(t) *does not converge to the optimal weights, in Fig. 5.4b* w<sup>⋆</sup> *is reached after a learning phase of* 3 s*. Correspondingly, the Lagrange multiplier* Υ(t) *in Fig. 5.4a is instationary, while in Fig. 5.4b it converges to* 1 *after having completed the learning phase.*

*Fig. 5.5 shows the trajectories of* w(t) *and* Υ(t) *for the original objective function* J 0 <sup>w</sup>(x, w)*, if an additive voltage source* ud(t) = sin(t/2) *serves as an excitation signal of the DC motor* (5.26)*.*

**Figure 5.5:** Adaptation results of the weights w with J <sup>0</sup> <sup>w</sup>(x, w) and additive excitation input.

*In this case, convergence to* w = w<sup>⋆</sup> *and* Υ = 1 *is achieved after* 45 s*. This demonstrates that convergence and thus optimality with respect to the original problem* (5.3) *can obtained even with a locally non-strictly convex objective function* J 0 <sup>w</sup>(x, w) *in* (5.53)*, provided that there is an additive input signal which ensures a persistent excitation of* x(t) *and thus a persistent movement of the value of* J 0 <sup>w</sup>(x, w) *such that* w<sup>⋆</sup> *becomes the unique minimizer for all* t*.*

#### **5.2.5 Stability of the Closed-Loop System**

The combination of the open-loop input-state-output PHS (5.3b), the extended CLF (5.27), the MOC law (5.29), the adaptation procedure (5.53), and the shorthand notation (5.30)–(5.34) results in the following closed-loop system:

$$
\dot{x} = (J(x) - R(x))\frac{\partial H(x)}{\partial x} + G(x)u^\star,\tag{5.55a}
$$

$$u^\star = \begin{cases} S^{-1}(x)G^\top(x)\frac{\partial V}{\partial x} \cdot \Upsilon(x,w), & S^\prime\_\Upsilon \neq 0, \\ 0, & S^\prime\_\Upsilon = 0, \end{cases} \tag{5.55b}$$

$$
\dot{\boldsymbol{w}} = -\boldsymbol{\alpha} \cdot \frac{\partial \mathcal{J}\_{\rm w}(\boldsymbol{x}, \boldsymbol{w})}{\partial \boldsymbol{w}},
\tag{5.55c}
$$

$$\mathbf{x}\_0 = \mathbf{x}\_0,\tag{5.55d}$$

$$w\_0 = \mathbb{0}\_r.\tag{5.55e}$$

To perform a stability analysis of the equilibrium (**0**, w<sup>⋆</sup> ) of (5.55), V (x, w) is used as a Lyapunov function candidate to prove that


While the proof of (5.57) can be handled by the following Lemma 5.21, statement (5.56) (detailed later in Lemma 5.23) requires some additional preparatory work.

**Lemma 5.21.** *Consider the closed-loop system* (5.55) *starting at* (x0, <sup>w</sup>0) <sup>∈</sup> <sup>R</sup><sup>n</sup> <sup>×</sup> <sup>R</sup><sup>r</sup> *. Then*

$$\forall \, (x, w) \in \mathbb{R}^n \times \mathbb{R}^r: \qquad \qquad \dot{V}(x, w) \le 0,\tag{5.58}$$

*i.e.* V (x, w) *decreases monotonically over time.*

*Proof.* Applying the chain rule to (5.27) and inserting (5.55a), (5.55b), (5.55c), we get

$$\begin{split} \dot{V}(\boldsymbol{x},\boldsymbol{w}) &= \left(\frac{\partial V(\boldsymbol{x},\boldsymbol{w})}{\partial \boldsymbol{x}}\right)^{\mathsf{T}} \dot{\boldsymbol{x}} + \left(\frac{\partial V(\boldsymbol{x},\boldsymbol{w})}{\partial \boldsymbol{w}}\right)^{\mathsf{T}} \dot{\boldsymbol{w}} \\ &= \left(\frac{\partial V(\boldsymbol{x},\boldsymbol{w})}{\partial \boldsymbol{x}}\right)^{\mathsf{T}} (J(\boldsymbol{x}) - \boldsymbol{R}(\boldsymbol{x})) \frac{\partial H}{\partial \boldsymbol{x}} - \left(\frac{\partial V(\boldsymbol{x},\boldsymbol{w})}{\partial \boldsymbol{w}}\right)^{\mathsf{T}} \alpha \frac{\partial \mathcal{J}\_{\text{w}}(\boldsymbol{x},\boldsymbol{w})}{\partial \boldsymbol{w}} \\ &- \left(\frac{\partial V(\boldsymbol{x},\boldsymbol{w})}{\partial \boldsymbol{x}}\right)^{\mathsf{T}} G(\boldsymbol{x}) S^{-1}(\boldsymbol{x}) \mathbf{G}^{\mathsf{T}}(\boldsymbol{x}) \frac{\partial V(\boldsymbol{x},\boldsymbol{w})}{\partial \boldsymbol{x}} \boldsymbol{\Upsilon} \\ &\left(\partial V(\boldsymbol{x},\boldsymbol{w})\right)^{\mathsf{T}}{}\_{\boldsymbol{\mathsf{T}}}(\boldsymbol{V}(\boldsymbol{x},\boldsymbol{w})) \,\partial H \quad \left(\partial V(\boldsymbol{x},\boldsymbol{w})\right)^{\mathsf{T}} \,\partial \mathcal{J}\_{\text{w}}(\boldsymbol{x},\boldsymbol{w}) \end{split} \tag{5.60}$$

$$\begin{split} &= \left(\frac{\partial V(\mathbf{x}, w)}{\partial x}\right)'(J(x) - \mathbf{R}(x)) \frac{\partial H}{\partial x} - \left(\frac{\partial V(\mathbf{x}, w)}{\partial w}\right)' \alpha \frac{\partial \mathcal{J}\_{\mathbf{w}}(\mathbf{x}, w)}{\partial w} \\ &- S\_{\Upsilon}' \cdot \frac{f\_{\Upsilon}' + \sqrt{(f\_{\Upsilon}')^2 + Q\_{\Upsilon}' \cdot S\_{\Upsilon}'}}{S\_{\Upsilon}'} \\ &= f\_{\Upsilon}' - f\_{\Upsilon}' - \sqrt{(f\_{\Upsilon}')^2 + Q\_{\Upsilon}' \cdot S\_{\Upsilon}'} \\ &= -\sqrt{(f\_{\Upsilon}')^2 + Q\_{\Upsilon}' \cdot S\_{\Upsilon}'}. \end{split} \tag{5.61}$$

Since S ′ <sup>Υ</sup> ≥ 0 and Q′ <sup>Υ</sup> <sup>≥</sup> <sup>0</sup> holds (see (5.32) and (5.33)), <sup>V</sup>˙ (x, <sup>w</sup>) is nonpositive for all (x, <sup>w</sup>) <sup>∈</sup> <sup>R</sup><sup>n</sup> <sup>×</sup> <sup>R</sup><sup>r</sup> .

Next, statement (5.56) (positive definiteness of V (x, w)) has to be evaluated. Despite the fact that H(x) is positive definite by definition, V (x, w) = H(x) + w<sup>T</sup>Ξ(x) may be nonpositive, if "+w<sup>T</sup>Ξ(x)" is negative for some (x, <sup>w</sup>) <sup>∈</sup> <sup>R</sup><sup>n</sup> <sup>×</sup> <sup>R</sup><sup>r</sup> . In particular, we thus have to prove that V (x(t), w(t)) is still positive for all t ≥ t<sup>0</sup> with x ̸= **0**. This is stated in Lemma 5.23 with the help of the following Lemma 5.22.

**Lemma 5.22.** *Let* V ⋆ (x) = H(x) + (w<sup>⋆</sup> ) <sup>T</sup>Ξ(x) *be the value function of optimization problem* (5.3) *and let the conditions of Lemma 5.19 hold with* w<sup>0</sup> ∈ M*. Then, all weighting factors* w(t) *asymptotically converge to the optimal ones, i.e.* w(t) *fulfills*

$$\lim\_{t \to \infty} \|w(t) - w^\star\|\_2 = 0. \tag{5.62}$$

**Figure 5.6:** Contour plot of Jw(x, w) for fixed x.

*Proof.* If the conditions of Lemma 5.19 hold with w<sup>0</sup> ∈ M, then Jw(x, w) is strictly convex with respect to <sup>w</sup> in an open neighborhood <sup>M</sup> of the optimizer <sup>w</sup><sup>⋆</sup> for each arbitrary but fixed x, i.e. for all t ≥ 0 we have

$$\left( (\boldsymbol{w}\_1(t) - \boldsymbol{w}\_2(t))^\top \left( \frac{\partial \mathcal{J}\_\mathbf{w}(\boldsymbol{x}(t), \boldsymbol{w})}{\partial \boldsymbol{w}} \bigg|\_{\boldsymbol{w}\_1(t)} - \frac{\partial \mathcal{J}\_\mathbf{w}(\boldsymbol{x}(t), \boldsymbol{w})}{\partial \boldsymbol{w}} \bigg|\_{\boldsymbol{w}\_2(t)} \right) > 0. \tag{5.63}$$

With w1(t) = w(t) and w2(t) = w<sup>⋆</sup> , (5.63) reads as

$$(w(t) - w^\star)^\top \left( \frac{\partial \mathcal{J}\_\mathbf{w}(\mathbf{z}(t), \mathbf{w})}{\partial \mathbf{w}} \Big|\_{\mathbf{w}(t)} \right) > 0. \tag{5.64}$$

Insertion of (5.55c) in (5.64) yields (w(t) <sup>−</sup> <sup>w</sup><sup>⋆</sup> ) T <sup>w</sup>˙ (t) <sup>&</sup>lt; <sup>0</sup>. With <sup>w</sup><sup>e</sup> (t) := <sup>w</sup>(t) <sup>−</sup> <sup>w</sup><sup>⋆</sup> , this is equivalent to

$$\left(\left(\tilde{\boldsymbol{w}}(t)\right)^{\mathsf{T}}\dot{\tilde{\boldsymbol{w}}}(t) < 0. \tag{5.65}$$

By using the chain rule, (5.65) can be transformed to

$$\frac{1}{2} \cdot \frac{\text{d}}{\text{d}t} \left\{ \left( \tilde{\boldsymbol{w}}(t) \right)^{\mathsf{T}} \tilde{\boldsymbol{w}}(t) \right\} < 0,\tag{5.66}$$

where (w<sup>e</sup> (t))<sup>T</sup>w<sup>e</sup> (t) = <sup>∥</sup>w<sup>e</sup> (t)<sup>∥</sup> 2 2 . Multiplying (5.66) by two and applying the square root on both sides, we finally obtain <sup>d</sup> dt <sup>∥</sup>w<sup>e</sup> (t)∥<sup>2</sup> <sup>&</sup>lt; <sup>0</sup>, i.e. the distance <sup>∥</sup>w(t) <sup>−</sup> <sup>w</sup><sup>⋆</sup>∥<sup>2</sup> decreases strictly monotonically with time for all t ≥ 0. This results in (5.62).

With the help of the above Lemma 5.22, we can prove that V (x(t), w(t)) is indeed a positivedefinite function:

**Lemma 5.23.** *Let* V ⋆ (x) = H(x) + (w<sup>⋆</sup> ) <sup>T</sup>Ξ(x) *be the value function of optimization problem* (5.3) *and let the conditions of Lemma 5.19 hold with* w<sup>0</sup> ∈ M*. Then*

∀ t ≥ 0 : V (x(t), w(t)) ≻ 0. (5.67)

*Proof.* Let <sup>W</sup><sup>+</sup> <sup>=</sup> {<sup>w</sup> <sup>∈</sup> <sup>R</sup><sup>r</sup> : <sup>V</sup> (x, <sup>w</sup>) <sup>&</sup>gt; <sup>0</sup> <sup>∀</sup><sup>x</sup> ̸<sup>=</sup> **<sup>0</sup>**} denote the set of <sup>w</sup> <sup>∈</sup> <sup>R</sup><sup>r</sup> where <sup>V</sup> (x, <sup>w</sup>) <sup>&</sup>gt; <sup>0</sup> is fulfilled for *all* <sup>x</sup> ̸<sup>=</sup> **<sup>0</sup>**. As it trivially holds that <sup>w</sup><sup>⋆</sup> <sup>∈</sup> <sup>W</sup><sup>+</sup> and **<sup>0</sup>** <sup>∈</sup> <sup>W</sup><sup>+</sup>, and as <sup>W</sup><sup>+</sup> is an open set (see Lemma D.2 in Appendix D.1), it can be concluded that **<sup>0</sup>** <sup>∈</sup> int(W<sup>+</sup>) and <sup>w</sup><sup>⋆</sup> <sup>∈</sup> int(W<sup>+</sup>), which is illustrated in Fig. 5.6<sup>68</sup>. Hence, there exists an ε > <sup>0</sup> such that the ball <sup>B</sup>(w<sup>⋆</sup> , ε) = {<sup>w</sup> <sup>∈</sup> <sup>R</sup><sup>r</sup> : <sup>∥</sup><sup>w</sup> <sup>−</sup> <sup>w</sup><sup>⋆</sup>∥<sup>2</sup> <sup>≤</sup> <sup>ε</sup>} lies completely within <sup>W</sup><sup>+</sup>. According to (5.66), <sup>∥</sup>w(t) <sup>−</sup> <sup>w</sup><sup>⋆</sup>∥<sup>2</sup> is strictly decreasing with time. Consequently, there exists a T ≥ 0 such that <sup>∥</sup>w(T) <sup>−</sup> <sup>w</sup><sup>⋆</sup>∥<sup>2</sup> <sup>=</sup> <sup>ε</sup>, i.e. <sup>x</sup> intersects the surface of the ball. Since <sup>∥</sup>w(t) <sup>−</sup> <sup>w</sup><sup>⋆</sup>∥<sup>2</sup> is strictly decreasing, w(t) will remain within the ball for all t > T, thus

$$\forall t > T \;:\; \begin{array}{c} \\ \\ \end{array} \mathbf{w}(T) \in \mathbf{W}^+.\tag{5.68}$$

Since <sup>W</sup><sup>+</sup> is the set of parameters <sup>w</sup> where <sup>V</sup> (x, <sup>w</sup>) is positive definite for *all* <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>, (5.68) implies that V (x(t), w(t)) ≻ 0 holds for all t > T. With V (x(0), w(0)) = V (x0, **0**r) = <sup>H</sup>(x0) <sup>≻</sup> <sup>0</sup> and due to the fact that <sup>V</sup> (x(t), <sup>w</sup>(t)) is continuous and <sup>V</sup>˙ (x(t), <sup>w</sup>(t)) is monotonically decreasing according to Lemma 5.21, it holds that V (x(t), w(t)) ≻ 0 for all t ≥ 0.

As a consequence of Lemma 5.21 and Lemma 5.23, V (x(t), w(t)) is a suitable Lyapunov function. With this finding, the main statement of this section regarding stability and asymptotic stability of the closed-loop equilibrium can be formulated.

**Theorem 5.24.** *Let* V ⋆ (x) = H(x) + (w<sup>⋆</sup> ) <sup>T</sup>Ξ(x) *be the value function of optimization problem* (5.3) *and let the conditions of Lemma 5.19 hold with* w<sup>0</sup> ∈ M*. Then* x = **0**, w = w<sup>⋆</sup> *is a stable equilibrium of* (5.55)*.*

*If additionally one of the following conditions holds:*


*then* x = **0**, w = w<sup>⋆</sup> *is an asymptotically stable equilibrium of* (5.55)*.*

<sup>68</sup> Note that **0** ∈ int(W+) and w<sup>⋆</sup> ∈ int(W+) does not necessarily imply that w(t) ∈ W<sup>+</sup> holds for *all* t. Fig. 5.6 shows an illustrative contour plot of Jw(x, w) for a fixed x. Of course, w<sup>⋆</sup> = arg min Jw(x, w). However, depending on the shape of Jw, it may be possible that the descent direction −∇J<sup>w</sup> is pointing out of W+, which may yield w(t ′ ) ∈/ W<sup>+</sup> for some t ′ > t. Despite the fact that w(t) may be temporarily outside of W+, it follows from (5.68) that for a sufficiently large but finite T ≥ 0, w(t) always lies within W+.

*Proof.* According to Lemma 5.23, V (x, w) is positive definite and according to Lemma 5.21, V˙ (x, w)is negative semidefinite. As such, V (x, w)is a Lyapunov function for the equilibrium (**0**, w<sup>⋆</sup> ) of (5.55), which is consequently a stable equilibrium [Kha02, Theorem 4.1].

To prove asymptotic stability of (**0**, w<sup>⋆</sup> ), recall Lemma 5.22, which states that w(t) converges strictly monotonically to w<sup>⋆</sup> . Now let <sup>E</sup> <sup>=</sup> {<sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup> : <sup>V</sup>˙ (x(t), <sup>w</sup><sup>⋆</sup> ) = 0} be the set of states where V (x, w) is constant and w = w<sup>⋆</sup> . With regard to the individual summands in (5.61), we get <sup>E</sup> <sup>=</sup> {<sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup> : (<sup>f</sup> ′ <sup>Υ</sup> = 0)∧((Q′ <sup>Υ</sup> = 0)∨(S ′ <sup>Υ</sup> = 0)), <sup>w</sup> <sup>=</sup> <sup>w</sup>⋆}. With <sup>Q</sup>′ <sup>Υ</sup> = ℓ(x) ≻ 0, statement Q′ <sup>Υ</sup> = 0 is equivalent to x = **0**, which implies f ′ <sup>Υ</sup> = 0. Accordingly, E can be simplified to <sup>E</sup> <sup>=</sup> {<sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup> : (<sup>x</sup> <sup>=</sup> **<sup>0</sup>**) <sup>∨</sup> ((<sup>f</sup> ′ <sup>Υ</sup> = 0) ∧ (S ′ <sup>Υ</sup> = 0)), <sup>w</sup> <sup>=</sup> <sup>w</sup><sup>⋆</sup>}. Based on this definition, conditions (1) and (2) of Theorem 5.24 are then obtained as follows:


#### **Example 7 (Synchronous Motor)**

*We apply the presented method to the nonlinear model of a non-salient permanent magnet synchronous motor (PMSM) in rotating reference (dq) frame from [OPA*<sup>+</sup>*11]:*

$$L\_{\rm s} \dot{I}\_{\rm d} = -R\_{\rm s} I\_{\rm d} - \Omega L\_{\rm s} I\_{\rm q} + U\_{\rm d},\tag{5.69a}$$

$$L\_{\rm s} \dot{I}\_{\rm q} = -R\_{\rm s} I\_{\rm q} + \Omega L\_{\rm s} I\_{\rm d} - \Theta \Omega + U\_{\rm q},\tag{5.69b}$$

$$J\dot{\Omega} = n\_{\rm pp} \Theta I\_{\rm q} - f \Omega - M\_{\ell},\tag{5.69c}$$

*where* I<sup>d</sup> *and* I<sup>q</sup> *are the stator currents,* U<sup>d</sup> *and* U<sup>q</sup> *are the stator voltages,* Ω *is the angular velocity,* M<sup>ℓ</sup> *is the load torque,* L<sup>s</sup> *is the stator inductance,* R<sup>s</sup> *is the stator winding resistance,* Θ *is the magnetic flux,* J *is the moment of inertia,* f *is the mechanic friction constant, and* npp *is the number of pole pairs.*

*The objective of the control system to be designed is to follow a desired angular velocity* Ω des *with zero flux current* I des <sup>d</sup> = 0 *for a given load torque* Mdes ℓ *. Denote by* (U des d , Udes q , Ides q ) *the resulting open-loop equilibrium values and define* x := F(Id−i des d , Iq−i des q , Ω−Ω des) T *with* F := diag{Ls, Ls, J/npp}*, and* u := (Ud, Uq, Mℓ) <sup>T</sup>*. Then, the error dynamics of* (5.69) *can be modeled by the input-state-output PHS* (5.3b)*, where* G = diag{1, 1, −1/npp}*,*

$$\begin{array}{ll} H(\mathbf{x}) = 1/2(\mathbf{x}^{\top}\mathbf{F}^{-1}\mathbf{x}),\\ \mathbf{J}(\mathbf{x}) = \begin{bmatrix} 0 & \frac{Ln\_{\mathrm{pp}}x\_{3}}{J} + L\_{\mathrm{s}}\Omega^{\mathrm{des}} & \frac{L\_{\mathrm{s}}I\_{\mathrm{q}}^{\mathrm{des}}}{2} \\ -\frac{L\_{\mathrm{s}}I\_{\mathrm{p}}^{\mathrm{des}}}{J} - L\_{\mathrm{s}}\Omega^{\mathrm{des}} & 0 & -\Theta \\ -\frac{L\_{\mathrm{s}}I\_{\mathrm{q}}^{\mathrm{des}}}{2} & \Theta & 0 \end{bmatrix},\\ \mathbf{R} = \begin{bmatrix} R\_{\mathrm{s}} & 0 & -\frac{L\_{\mathrm{s}}I\_{\mathrm{q}}^{\mathrm{des}}}{2} \\ 0 & R\_{\mathrm{s}} & 0 \\ -\frac{L\_{\mathrm{s}}I\_{\mathrm{q}}^{\mathrm{des}}}{2} & 0 & \frac{f}{n\_{\mathrm{pp}}} \end{bmatrix}. \end{array}$$

*We choose* L<sup>s</sup> = 0.5*,* R<sup>s</sup> = 0.5*,* Θ = 1*,* J = 2*,* f = 0.1*,* npp = 2*, and consider the optimal control problem* (5.3)*, where*

**Figure 5.7:** Controller performance for PMSM example.

*Since the condition of Corollary 5.8 is fulfilled, the Hamiltonian* H(x) *is a CLF. The basis functions for the extended CLF are chosen to* Ξ(x) = (x 2 1 , x<sup>2</sup> 2 , x<sup>2</sup> 3 , x1x2, x2x3, x1x3) <sup>T</sup>*. For the above optimization problem, it can be shown that the exact solution of the HJB equation* (5.24) *is given by* V ⋆ (x) = 0.5(x 2 <sup>1</sup>+x 2 <sup>2</sup>+2x 2 3 )*, which implies* <sup>w</sup><sup>⋆</sup> = (−0.5, <sup>−</sup>0.5, <sup>−</sup>0.25, <sup>0</sup>, <sup>0</sup>, 0)T*. Now the control system* (5.55) *is initialized at* <sup>x</sup><sup>0</sup> <sup>=</sup> <sup>1</sup>*,* <sup>w</sup><sup>0</sup> <sup>=</sup> **<sup>0</sup>***. The shifts in* <sup>J</sup><sup>w</sup> *are set to* <sup>c</sup><sup>1</sup> <sup>=</sup> **<sup>0</sup>***,* <sup>c</sup><sup>2</sup> = (1, <sup>0</sup>, 0)<sup>T</sup>*,* <sup>c</sup><sup>3</sup> = (0, <sup>1</sup>, 0)<sup>T</sup>*,* <sup>c</sup><sup>4</sup> = (0, <sup>0</sup>, 1)<sup>T</sup>*,* <sup>c</sup><sup>5</sup> = (−1, <sup>−</sup>1, 0)<sup>T</sup>*,* <sup>c</sup><sup>6</sup> <sup>=</sup> (0, <sup>−</sup>1, <sup>−</sup>1)<sup>T</sup>*,* <sup>c</sup><sup>7</sup> = (−1, <sup>0</sup>, <sup>−</sup>1)<sup>T</sup>*, and the learning rate is set to* <sup>α</sup> = 0.5*. At* <sup>t</sup> = 10 *and* t = 30*, the load torque* Mdes ℓ *increases by* 1*, and at* t = 20*, the desired angular voltage* Ω des *increases by* 0.5*.*

*Fig. 5.7 shows the trajectories of* x(t)*,* u(t)*,* w(t)*, and* Υ*. It can be seen that after* 2 s*, the learning process is completed and both* w *and* Υ *have reached their optimal values. Correspondingly, the trajectories of* x *and* u *converge to the solution associated to the optimal controller, once the learning process is completed. This demonstrates that the proposed control strategy is capable of adapting the optimal parameters after a single learning phase.*

*To investigate the effects of an incorrect choice of basis functions, we repeat the simulation with* Ξ ′ (x) = (x 3 1 , x<sup>3</sup> 2 , x<sup>3</sup> 3 , x1x2, x2x3, x1x3) <sup>T</sup>, *i.e.* H(x) + w<sup>T</sup>Ξ ′ (x) *does not fit the structure of* V ⋆ (x) *and hence Assumption 5.15 is violated. The results are shown in Fig. 5.8.*

**Figure 5.8:** Controller performance if Assumption 5.15 is violated.

*The trajectory of* Υ *reveals a remarkable and distinct oscillatory behavior. However, after* 5 s*, the weighting factors* w *converge to a certain value* w<sup>⋄</sup> *. After the step increases of* Mdes ℓ *and* Ω des*, the weighting factors do not remain at their previous values. Both the oscillation of* Υ(t) *and the fluctuation of* w(t) *after the disturbances imply that* H(x) + (w<sup>⋄</sup> ) <sup>T</sup>Ξ ′ (x) *is not equal to the value function for this specific set of basis functions. However, the results show that even if* Ξ ′ (x) *is not accurate, the proposed controller is able to learn suitable weighting factors for a "suboptimal" control, with the oscillation of* Υ(t) *being interpreted as an indicator of lack of fitness for the chosen basis functions in* Ξ ′ (x) *(see the discussion in Remark 5.13).*

#### **5.2.6 Summary**

In this section, we have introduced a continuous-time adaptive feedback control scheme for dynamic optimization problems with generalized Lagrangian performance indices (A.8) and general input-state-output PHSs (A.13). Employing the initial value function guess V (x) = H(x) generates an admissible controller which is already stabilizing. While applying the CLF-based MOC approach to PHSs, this section shows that the port-Hamiltonian modeling gives access to a *natural CLF candidate*, which can serve as a helpful tool for constructing *universal formulae for stabilization* [Son89, CB04]<sup>69</sup> .

Subsequently, a gradient adaptation of the critic weights w based on the Lagrange multiplier Υ(t) enables a learning of the (unknown) value function V ⋆ (x) and thus the optimal control strategy u ⋆ . Asymptotic stability of the closed-loop equilibrium is shown by the main Theorem 5.24. Although a reasonable choice of basis functions is nontrivial, simulations show that even if Ξ(x) is inaccurate, the proposed controller is able to stabilize the system and provide near-optimal solution trajectories. Furthermore, the optimality of the adapted control law can be assessed for the case of a bad choice of basis functions via Υ(t). However, a rigorous perturbation analysis for systems where parameterization Ξ(x) does not fit the structure of V ⋆ (x) remains an open research question.

## **5.3 Multi-Player Case**

If a dynamical system is controlled by two or more actuators with differing interests and goals, a scenario emerges which can be modeled as a noncooperative differential game (see Definition A.2 in Appendix A.2). This section presents an adaptive controller for such noncooperative differential games with dynamics modeled as general nonlinear input-state-output PHSs. The proposed controller is able to learn feedback Nash strategies (cf. Definition A.6, see also Lemma A.8) for general Lagrange-type performance indices (A.8).

Subsection 5.3.1 formalizes the problem class by extending the single-player case from Section 5.2 to N players, and briefly discusses related work. Subsection 5.3.2 provides an analytical solution of the so-called *modified differential game*. Subsection 5.3.3 analyzes the asymptotic

<sup>69</sup> Sontag's universal formula for stabilization itself is a special case of the modified optimal controller (5.8). It can be obtained by choosing ℓ(x) = ∂V ∂x T G(x)(S(x))−1GT(x) ∂V ∂x and S(x) = I.

stability of the closed-loop equilibrium resulting from the modified differential game. Subsection 5.3.4 presents necessary and sufficient conditions for H(x) to serve as admissible CLF. Subsection 5.3.5 develops an adaptation strategy which enables to learn the original (non-modified) differential game (A.8). Subsection 5.3.6 proves the asymptotic stability of the resulting feedback Nash equilibrium and gives an illustrative example. Finally, Subsection 5.3.7 summarizes the main result of this section.

#### **5.3.1 Problem Definition and Related Work**

**Definition 5.25 (Multi-Player Optimal Control Problem for PHSs).** *For all* <sup>i</sup> <sup>∈</sup> <sup>P</sup>*, find the feedback control laws* ui(x) *as the solutions of*

$$\min\_{\mathbf{u}\_i(t)}\qquad\frac{1}{2}\int\_0^{t\_\ell} \ell\_i(\mathbf{x}(t)) + (\mathbf{u}\_i(t))^\mathsf{T} \mathbf{S}\_i(\mathbf{x}(t)) \mathbf{u}\_i(t) \,\mathrm{d}t\tag{5.70a}$$

$$\text{subject to} \quad \dot{x}(t) = (J(x(t)) - R(x(t))) \frac{\partial H(x(t))}{\partial x(t)} + \sum\_{i=1}^{N} \mathbf{G}\_i(x(t)) u\_i(t) \tag{5.70b}$$

*with set of players* <sup>P</sup> <sup>=</sup> {1, . . . , N}*, state vector* <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>*, input vectors* <sup>u</sup><sup>i</sup> <sup>∈</sup> <sup>R</sup><sup>p</sup><sup>i</sup> *, skew-symmetric interconnection matrix* <sup>J</sup>(x) <sup>∈</sup> <sup>R</sup><sup>n</sup>×<sup>n</sup>*, positive semidefinite dissipation matrix* <sup>R</sup>(x) <sup>∈</sup> <sup>R</sup><sup>n</sup>×<sup>n</sup>*, input matrices* <sup>G</sup>i(x) <sup>∈</sup> <sup>R</sup><sup>n</sup>×p<sup>i</sup> *, positive definite and radially unbounded Hamiltonian* <sup>H</sup> : <sup>R</sup><sup>n</sup> <sup>→</sup> <sup>R</sup>≥<sup>0</sup>*; smooth, positive definite* <sup>ℓ</sup><sup>i</sup> : <sup>R</sup><sup>n</sup> <sup>→</sup> <sup>R</sup>≥<sup>0</sup> *and positive definite* <sup>S</sup>i(x) <sup>∈</sup> <sup>R</sup><sup>p</sup>i×p<sup>i</sup> *. Without loss of generality, we assume that* H(x) *has its minimum at* x = **0** *with* H(**0**) = 0*.*

To find a feedback Nash strategy for (5.70), however, a system of HJB equations [Bre11] has to be solved in order to get a set of coupled value functions V ⋆ i (x) (one for each specific player), cf. Lemma A.8. Since an analytical solution of this problem is currently not known, it is crucial to exploit the structure of the optimization problems and/or the underlying plant system. For example, in linear-affine systems with quadratic objective functions (LQ games), the value functions are quadratic forms and thus the set of HJB equations translates into a tractable set of (algebraic) matrix-Riccati ODEs [PMC79, WSE99, AL13, WHLM18], for which extensive research has been undertaken in recent decades, see [Eng06] and [ES13] for a profound introduction to solution methods for LQ games.

For the more general case of input-affine nonlinear systems, methods derived from ADP provide viable means for solving the system of coupled HJB equations. In this case, approximators (critics) for the players' value functions are applied, whereby the weights of the critics are continuously adjusted based on measurements of the current state vector and the current inputs of the other players. Depending on the selected ADP approach, however, initial stabilizing weights [VL11, LLW14, MLS14, VMH16] or a suitable Lyapunov function [ZCL13, MNSS18] are required to guarantee convergence of the weights and stability of the closed-loop system. This makes the deduction of a general methodology difficult, especially for large-scale systems.

In the following, we present an adaptive feedback Nash strategy for the class of nonlinear input-state-output PHSs. In the previous Section 5.2, it was shown for the single-player case that H(x) inherently qualifies as a value function candidate. Again inspired by [SK00], in the remainder of this section particular attention is paid to determine the deviation of the current suboptimal solution of player i from the optimal (Nash) strategy using a certain Lagrange multiplier Υ<sup>i</sup> . Based on an evaluation of Υ<sup>i</sup> , the respective critic weights are updated simultaneously using a continuous-time gradient flow. Finally, the main theorem of this section (Theorem 5.53) will show convergence of the critic weights towards the Nash solution and asymptotic stability of the closed-loop system.

#### **5.3.2 Modified Differential Game**

As with the single-player case, an analytical solution of the original differential game (5.70) is generally intractable [Bre11, p. 390ff.], [LVS12, p. 278]. However, inspired by the basic idea of [SK00], projection of (5.70b) using a suitable CLF allows to formulate a set of modified optimization problems that makes obtaining an analytical solution considerably easier. For this purpose, we first introduce the new definition of a CLF in the context of differential games and then present an analytical solution of the set of modified optimization problems.

**Definition 5.26 (CLFs for Multi-Player Scenarios).** *A radially unbounded, smooth, positive definite function* V<sup>i</sup> : <sup>R</sup><sup>n</sup> <sup>→</sup> <sup>R</sup> *is said to be a* control-Lyapunov function associated to input <sup>i</sup> <sup>∈</sup> <sup>P</sup> *for the system*

$$\dot{x} = f(x, u\_1, \dots, u\_N), \qquad \quad x \in \mathbb{R}^n, \quad u\_i \in \mathbb{R}^{p\_i}, \quad i \in \mathbb{P}, \tag{5.71}$$

*if* f(**0**n, **0**<sup>p</sup><sup>1</sup> , . . . , **0**<sup>p</sup><sup>N</sup> ) = **0**<sup>n</sup> *and* ∀x ̸= **0**<sup>n</sup> : inf u<sup>i</sup> ( ∂V<sup>i</sup> ∂x T f(x,u1, . . . ,u<sup>N</sup> ) ) < 0. (5.72)

**Remark 5.27.** *For input-affine nonlinear systems*

$$\dot{x} = f(x) + \sum\_{i=1}^{N} G\_i(x)u\_i, \qquad \qquad x \in \mathbb{R}^n, \quad u\_i \in \mathbb{R}^{p\_i}, \tag{5.73}$$

*condition* (5.72) *is equivalent to*

$$\left(\frac{\partial V\_i}{\partial x}\right)^{\mathsf{T}} \mathbf{G}\_i(\boldsymbol{x}) = \mathbb{0}\_{p\_i} \quad \Longrightarrow \left(\frac{\partial V\_i}{\partial x}\right)^{\mathsf{T}} \left(\boldsymbol{f}(\boldsymbol{x}) + \sum\_{\begin{subarray}{c}j=1\\j\neq i\end{subarray}}^{N} \mathbf{G}\_j(\boldsymbol{x}) \boldsymbol{u}\_j\right) \begin{cases} < 0, & \boldsymbol{x} \neq \boldsymbol{0}\_n, \\ = 0, & \boldsymbol{x} = \boldsymbol{0}\_n. \end{cases} \tag{5.74}$$

♢

Now the original problem (5.70) is modified by left-multiplying (5.70b) with ∂V<sup>i</sup> ∂x T :

**Lemma 5.28.** *For all* <sup>i</sup> <sup>∈</sup> <sup>P</sup>*, let* <sup>V</sup>i(x) *be a CLF associated to input* <sup>u</sup><sup>i</sup> *of* (5.70)*. Then an exact solution of the modified differential game*

$$\min\_{\mathbf{u}\_i} \qquad \left\{ \frac{1}{2} \int\_0^{t\_f} \ell\_i(\mathbf{x}) + \mathbf{u}\_i^\mathsf{T} \mathbf{S}\_i(\mathbf{x}) \mathbf{u}\_i \mathbf{d}t \right\} \tag{5.75a}$$

$$\text{subject to}\quad \dot{V}\_i(\mathbf{x}) = \left(\frac{\partial V\_i}{\partial \mathbf{z}}\right)^\top \left((\mathbf{J} - \mathbf{R})\frac{\partial H}{\partial \mathbf{z}} + \sum\_{i=1}^N \mathbf{G}\_i(\mathbf{z})u\_i\right) \tag{5.75b}$$

*with* <sup>ℓ</sup>i(x) <sup>≽</sup> <sup>0</sup>*,* <sup>S</sup>i(x) <sup>≻</sup> <sup>0</sup> *is given by*

$$u\_i^\star(x) = -(S\_i(x))^{-1} \mathbf{G}\_i^\mathsf{T}(x) \frac{\partial V\_i(x)}{\partial x} \cdot \Upsilon\_i(t), \quad \qquad i \in \mathbb{P}, \tag{5.76}$$

$$0 = q\_i^0(\mathbf{z}) + \Upsilon\_i q\_i^1(\mathbf{z}) + \Upsilon\_i^2 q\_i^2(\mathbf{z}) + \Upsilon\_i \sum\_{\substack{j=1 \\ j \neq i}}^N q\_{ij}^2(\mathbf{z}) \Upsilon\_j, \qquad i \in \mathbb{P}, \tag{5.77}$$

*where*

$$q\_i^0(x) := \frac{1}{2}\ell\_i(x),\tag{5.78}$$

$$q\_i^1(\mathbf{z}) := \left(\frac{\partial V\_i}{\partial \mathbf{z}}\right)^\mathsf{T} (\mathbf{J} - \mathbf{R}) \frac{\partial H}{\partial \mathbf{z}},\tag{5.79}$$

$$q\_i^2(x) := -\frac{1}{2} \left(\frac{\partial V\_i}{\partial x}\right)^\top G\_i(x) (S\_i(x))^{-1} G\_i^\top(x) \frac{\partial V\_i}{\partial x},\tag{5.80}$$

$$q\_{ij}^2(x) := -\left(\frac{\partial V\_i}{\partial x}\right)^\dagger \mathbf{G}\_j(x) (\mathbf{S}\_j(x))^{-1} \mathbf{G}\_j^\top(x) \frac{\partial V\_j}{\partial x}.\tag{5.81}$$

*Proof.* For each <sup>i</sup> <sup>∈</sup> <sup>P</sup>, the Hamiltonian of optimization problem (5.75) is given by

$$\mathcal{H}\_{i}^{\prime}(\boldsymbol{x},\boldsymbol{u},\boldsymbol{\Upsilon}) = \frac{1}{2}\ell\_{i}(\boldsymbol{x}) + \frac{1}{2}\boldsymbol{u}\_{i}^{\top}\boldsymbol{S}\_{i}(\boldsymbol{x})\boldsymbol{u}\_{i} + \boldsymbol{\Upsilon}\_{i}\dot{V}\_{i}(\boldsymbol{x})\tag{5.82}$$

$$= \frac{1}{2}\ell\_{i}(\boldsymbol{x}) + \frac{1}{2}\boldsymbol{u}\_{i}^{\top}\boldsymbol{S}\_{i}(\boldsymbol{x})\boldsymbol{u}\_{i} + \boldsymbol{\Upsilon}\_{i}\left(\frac{\partial V\_{i}}{\partial \boldsymbol{x}}\right)^{\top}\left((\boldsymbol{J}-\boldsymbol{\mathcal{R}})\frac{\partial \boldsymbol{H}}{\partial \boldsymbol{x}} + \sum\_{j=1}^{N}\boldsymbol{G}\_{j}(\boldsymbol{x})\boldsymbol{u}\_{j}\right),\tag{5.83}$$

where Υ<sup>i</sup> is the (scalar) Lagrange multiplier corresponding to the constraint (5.75b) and Υ := coli{Υi}. The optimal control u ⋆ i can be computed by

$$\left.\frac{\partial \mathcal{H}\_i(x, u, \Upsilon)}{\partial u\_i}\right|\_{u\_i^\star} = 0,\tag{5.84}$$

which yields

$$u\_i^\star(x) = -(S\_i(x))^{-1} G\_i^\top(x) \frac{\partial V\_i}{\partial x} \Upsilon\_i. \tag{5.85}$$

From the HJB equation for time-invariant systems

$$\mathcal{H}\_i^{\ell}(x, u^\star, \Upsilon^\star) = 0, \qquad \qquad \qquad i \in \mathbb{P}, \tag{5.86}$$

we get

*. .*

$$\begin{split} \forall i \in \mathbb{P}: \quad 0 = \frac{1}{2} \ell\_i(\boldsymbol{x}) + \Upsilon\_i^\star \left(\frac{\partial V\_i}{\partial \boldsymbol{x}}\right)^\top (\boldsymbol{J} - \boldsymbol{R}) \frac{\partial H}{\partial \boldsymbol{x}} \\ \quad \quad \quad - (\Upsilon\_i^\star)^2 \frac{1}{2} \left(\frac{\partial V\_i}{\partial \boldsymbol{x}}\right)^\top \boldsymbol{G}\_i(\boldsymbol{x}) (\boldsymbol{S}\_i(\boldsymbol{x}))^{-1} \boldsymbol{G}\_i^\top(\boldsymbol{x}) \frac{\partial V\_i}{\partial \boldsymbol{x}} \\ \quad \quad \quad \quad - \Upsilon\_i^\star \left(\frac{\partial V\_i}{\partial \boldsymbol{x}}\right)^\top \sum\_{\begin{subarray}{c} j=1\\ j\neq i \end{subarray}}^N \boldsymbol{G}\_j(\boldsymbol{x}) (\boldsymbol{S}\_j(\boldsymbol{x}))^{-1} \boldsymbol{G}\_j^\top(\boldsymbol{x}) \frac{\partial V\_j}{\partial \boldsymbol{x}} \boldsymbol{\Upsilon}\_j^\star. \end{split} \tag{5.87}$$

With the definitions (5.78)–(5.81), this is equivalent to (5.76).

**Remark 5.29.** *The resulting closed-loop system*

$$\dot{\boldsymbol{x}} = (\boldsymbol{J} - \boldsymbol{R}) \frac{\partial \boldsymbol{H}}{\partial \boldsymbol{x}} - \boldsymbol{G}\_1 (\boldsymbol{S}\_1(\boldsymbol{x}))^{-1} \boldsymbol{G}\_1^\top \frac{\partial V\_1}{\partial \boldsymbol{x}} \boldsymbol{\Upsilon}\_1 - \dots - \boldsymbol{G}\_N (\boldsymbol{S}\_N(\boldsymbol{x}))^{-1} \boldsymbol{G}\_N^\top \frac{\partial V\_N}{\partial \boldsymbol{x}} \boldsymbol{\Upsilon}\_N,\tag{5.88}$$

$$0 = q\_1^0(\mathbf{z}) + \Upsilon\_1 q\_1^1(\mathbf{z}) + \Upsilon\_1^2 q\_1^2(\mathbf{z}) + \Upsilon\_1 \sum\_{j=2}^N q\_{1j}^2(\mathbf{z})\Upsilon\_j,\tag{5.89}$$

$$0 = q\_2^0(\mathbf{z}) + \Upsilon\_2 q\_2^1(\mathbf{z}) + \Upsilon\_2^2 q\_2^2(\mathbf{z}) + \Upsilon\_2 \sum\_{\substack{j=1 \\ j \neq 2}}^N q\_{2j}^2(\mathbf{z}) \Upsilon\_j,\tag{5.90}$$

$$\begin{aligned} \vdots \\ 0 = q\_N^0(\mathbf{x}) + \Upsilon\_N q\_N^1(\mathbf{z}) + \Upsilon\_N^2 q\_N^2(\mathbf{z}) + \Upsilon\_N \sum\_{\substack{j=1 \\ j \neq N}}^N q\_{Nj}^2(\mathbf{z}) \Upsilon\_j \end{aligned} \tag{5.91}$$

*constitutes a differential-algebraic system of equations of index 1.* ♢

According to Definition 5.26, the CLF property of Vi(x)is highly dependent on the other inputs u¬<sup>i</sup> . Thus, (5.72) must apply to all possible inputs u1, . . . ,u<sup>i</sup> , . . . ,u<sup>N</sup> . This is generally a very strict requirement, if those inputs are assumed to be unconstrained in any way. However, since in the context of the modified differential game (5.75) all inputs are restricted according to (5.85), a much less restrictive requirement on the vector (V1, . . . , V<sup>N</sup> ) <sup>T</sup> of CLFs can be imposed:

**Definition 5.30 (Admissible Vector of CLFs).** *A vector* V (x) = (V1(x), . . . , Vp(x))<sup>T</sup> *is said to be an* admissible vector of CLFs *for* (5.73)*, if* (5.72) *is fulfilled for all* <sup>i</sup> <sup>∈</sup> <sup>P</sup> *with* ui(x) = u ⋆ i (x) *according to* (5.85)*, i.e. if*

$$\forall x \neq 0\_n: \qquad \inf\_{\mathbf{u}\_i^\star} \left\{ \left( \frac{\partial V\_i}{\partial x} \right)^\mathsf{T} f(x, \mathbf{u}\_1^\star, \dots, \mathbf{u}\_N^\star) \right\} < 0 \tag{5.92}$$

*holds for all* <sup>i</sup> <sup>∈</sup> <sup>P</sup>*.*

**Remark 5.31.** *For input-affine nonlinear systems, condition* (5.92) *is equivalent to*

$$\begin{aligned} \left(\frac{\partial V\_i}{\partial x}\right)^\mathsf{T} \mathbf{G}\_i(\mathbf{z}) &= \mathbf{0}\_{pi} \\ \Longrightarrow \left(\frac{\partial V\_i}{\partial x}\right)^\mathsf{T} \left(\mathbf{f}(\mathbf{z}) - \sum\_{\begin{subarray}{c} j=1\\ j\neq i \end{subarray}}^N \mathbf{G}\_j(\mathbf{z}) (\mathbf{S}\_j(\mathbf{z}))^{-1} \mathbf{G}\_j^\mathsf{T}(\mathbf{z}) \frac{\partial V\_j}{\partial \mathbf{z}} \cdot \Upsilon\_j\right) \begin{cases} < 0, & \mathbf{z} \neq \mathbf{0}\_n, \\ = 0, & \mathbf{z} = \mathbf{0}\_n \end{cases} \end{aligned} \tag{5.93}$$

*for all* <sup>i</sup> <sup>∈</sup> <sup>P</sup>*,* <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>*.* ♢

Since the individual inputs ui(x) are interdependent due to coupling by the scalars Υ1, . . . , Υ<sup>N</sup> via the control laws (5.76)–(5.77), it is sufficient to require that V (x) is an admissible vector of CLFs instead of the (stricter) requirement that each individual Vi(x) is a CLF in accordance with Definition 5.26. Lemma 5.28 can thus be relaxed as follows:

**Corollary 5.32.** *Let* V (x) *be an admissible vector of CLFs for* (5.70)*. Then an exact solution of the modified differential game* (5.75) *is given by* (5.76)*–*(5.81)*.*

*Proof.* Trivially, if ui(x) is given by (5.76)–(5.77), then (5.74) and (5.93) are equivalent.

#### **Suboptimality of the Modified Differential Game**

With shorthand notations u = coli{ui} and Υ = coli{Υi}, the solution (u ⋆ , Υ<sup>⋆</sup> ) generated by the MOC law (5.76)–(5.81) can be characterized as the Nash equilibrium of a differential game with scaled objective functions JΥ,i(x):

**Lemma 5.33.** *Let* V (x) *be an admissible vector of CLFs for* (5.70)*. Then each solution* (u ⋆ , Υ) *fulfilling* (5.76)*–*(5.77) *constitutes a Nash equilibrium with regard to the scaled* *objective functions*

$$\mathcal{J}\boldsymbol{\Upsilon}\_{\boldsymbol{i},i}(\boldsymbol{x}) = \int\_{0}^{t\_{\rm{f}}} \frac{1}{2\boldsymbol{\Upsilon}\_{i}} \cdot \left(\ell\_{i}(\boldsymbol{x}(t)) + (\boldsymbol{u}\_{i}(t))^{\top} \boldsymbol{S}\_{i}(\boldsymbol{x}(t)) \boldsymbol{u}\_{i}(t)\right) \mathrm{d}t, \quad i \in \mathbb{P} \tag{5.94}$$

*subject to the dynamic constraint* (5.70b)*.*

*Proof.* For each <sup>i</sup> <sup>∈</sup> <sup>P</sup>, the Hamiltonian of the scaled optimization problem (5.94) subject to (5.70b) equals

$$\mathcal{H}\_{i}^{\text{scaled}}(x, u, \xi\_{i}) = \frac{1}{2\Upsilon\_{i}} \left( \ell\_{i}(x) + u\_{i}^{\text{T}} S\_{i}(x) u\_{i} \right) + \xi\_{i}^{\text{T}} \left( (J - \mathcal{R}) \frac{\partial H}{\partial x} + \sum\_{j=1}^{N} \mathcal{G}\_{j}(x) u\_{j} \right), \tag{5.95}$$

where <sup>ξ</sup><sup>i</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup> is the Lagrange multiplier corresponding to the individual optimization problem of player <sup>i</sup> <sup>∈</sup> <sup>P</sup>. For the optimal control <sup>u</sup> scaled⋆ i , it has to hold that

$$\frac{\partial \mathcal{A}\_i^{\text{scaled}}(x, u, \xi\_i)}{\partial u\_i} \bigg|\_{u\_i^{\text{scaled}}} = \mathbb{0}\_{p\_i},\tag{5.96}$$

which yields

$$u\_i^{\text{scaled}\star}(x) = -(\mathcal{S}\_i(x))^{-1} \mathcal{G}\_i^{\text{T}}(x) \xi\_i \Upsilon\_i. \tag{5.97}$$

From the HJB equation for time-invariant systems

$$
\mathcal{H}\_i^{\text{scaled}}(x, u^\star, \xi\_i^\star) = 0,\tag{5.98}
$$

we get

$$\forall i \in \mathbb{P}: \qquad 0 = \frac{1}{2\Upsilon\_i} \ell\_i(x) + (\xi\_i^\star)^\top (\mathbf{J} - \mathbf{R}) \frac{\partial H}{\partial x} - \frac{1}{2} \Upsilon\_i(\xi\_i^\star)^\top \mathbf{G}\_i(x) (\mathbf{S}\_i(x))^{-1} \mathbf{G}\_i^\top(x) \xi\_i^\star$$

$$- (\xi\_i^\star)^\top \sum\_{\substack{j=1 \\ j \neq i}}^N \mathbf{G}\_j(x) (\mathbf{S}\_j(x))^{-1} \mathbf{G}\_j^\top(x) \xi\_j^\star \Upsilon\_j. \tag{5.99}$$

Now recall the HJB equation for the modified differential game in (5.87). Define

$$
\boldsymbol{\xi}\_{i}^{\star} = \frac{\partial V\_{i}}{\partial \boldsymbol{x}}, \quad \qquad \qquad \qquad i \in \mathbb{P}, \tag{5.100}
$$

which results in (5.87) and (5.99) being equivalent. Accordingly, each solution Υ<sup>⋆</sup> of (5.87) corresponding to the modified differential game problem in (5.75) is a solution of (5.98), with (5.100) associated to the new, scaled problem (5.94) in Lemma 5.33, and vice versa. By inserting ξ ⋆ i and Υ<sup>⋆</sup> in (5.97), we can conclude that u ⋆ i (x) in (5.76) is a Nash strategy of player <sup>i</sup> <sup>∈</sup> <sup>P</sup> for the scaled differential game (5.94) subject to (5.70b), i.e. u ⋆ i (x) = u scaled⋆ i (x) holds for all <sup>i</sup> <sup>∈</sup> <sup>P</sup>.

**Corollary 5.34.** *Let* V (x) *be an admissible vector of CLFs for* (5.70)*. Then each solution* (u ⋆ , Υ<sup>⋆</sup> ) *fulfilling* (5.76)*–*(5.77) *constitutes a Nash equilibrium with regard to the original problem* (5.70)*, if* Υ<sup>⋆</sup> i (x) = 1*.*

*Proof.* Trivially, by comparison between the original differential game (5.70) and the scaled differential game (5.94) subject to (5.70b), which is enforced by the controller (5.76)–(5.77), it follows that all components in Υ must be equal to one so that both differential games are equivalent.

**Remark 5.35.** *From Corollary 5.34, it follows that, unless* Υ(x) *converges to a constant value* 1 *and remains constant even after a disturbance, the chosen set of CLFs* Vi(x) *cannot be equivalent to the players' value functions. In this case, the resulting controllers* (5.76)*–*(5.77) *do not constitute a Nash strategy with regard to the original problem* (5.70)*. Conversely, the fluctuation of* Υ(t) *over time can therefore again be interpreted as an indicator of lack of fitness of the control strategy* u ⋆ i (x) *with respect to* (5.70) *as already reasoned in Remark 5.13 for the single-player case.* ♢

#### **Example 8 (Nonlinear Differential Game)**

*Consider the two-player differential game* (5.70) *with*

$$\int \mathcal{J}\_1 = \frac{1}{2} \int\_0^{t\_\ell} \left( x^\top \begin{bmatrix} 16 + 8x\_2^2 & -2x\_2^2 \\ -2x\_2^2 & 5 \end{bmatrix} x + 4 \cdot u\_1^2 \right) \mathrm{d}t,\tag{5.101a}$$

$$\prescript{}{f}{\mathcal{J}}\_2 = \frac{1}{2} \int\_0^{t\_f} \left( x^\top \begin{bmatrix} 5 + 2x\_2^2 & 2.5\\ 2.5 & 12 \end{bmatrix} x + u\_2^2 \right) dt \tag{5.101b}$$

*subject to the nonlinear input-state-output PHS*

$$\dot{x} = \left( \underbrace{\begin{bmatrix} 0 & -\frac{1}{2} \\ \frac{1}{2} & 0 \end{bmatrix}}\_{J(x)} - \underbrace{\begin{bmatrix} 1 + x\_2^2 & \frac{1}{2} \\ \frac{1}{2} & 2 \end{bmatrix}}\_{R(x)} \right) \frac{\partial H}{\partial x} + \underbrace{\begin{bmatrix} 1 \\ 0 \end{bmatrix}}\_{G\_1(x)} u\_1 + \underbrace{\begin{bmatrix} 1 \\ 1 \end{bmatrix}}\_{G\_2(x)} u\_2,\tag{5.102}$$

*where* H(x) = <sup>1</sup> 2 x 2 <sup>1</sup> + 1 2 x 2 2 *.*

*The corresponding value functions of* (5.101a) *and* (5.101b) *are*

$$V\_1^\star(x) = 2x\_1^2 - 2x\_1x\_2 + x\_2^2,\tag{5.103a}$$

$$V\_2^{\star}(x) = \frac{1}{2}x\_1^2 + x\_2^2,\tag{5.103b}$$

*which can be calculated by using the converse HJB approach [NP96].*

*The system* (5.102) *is initialized at* x = (0, 0)<sup>T</sup>*. Fig. 5.9 shows the resulting trajectories of* Υ1(t) *and* Υ2(t) *for different vectors of CLFs* V (x)*.*

**Figure 5.9:** Trajectory of Υ for different vectors of CLFs V (x).

*In Figs. 5.9a–c,* Υ(t) *is varying with time, while in Fig. 5.9d it holds for all* t *that* Υ(t) = 1*. Moreover, it can be seen that the closer* V (x) *is to* V ⋆ (x)*, the closer* Υ *is to* 1*.*

*However,* Vi(x) = V ⋆ i (x) *does not necessarily result in* Υi(t) = 1*, which can be seen e.g. in Fig. 5.9b: Although* V2(x) = V ⋆ 2 (x) *holds,* Υ<sup>2</sup> *approaches a value* ≈ 1.1*.*

*This effect is based on the dynamic coupling of both players and the nature of a Nash equilibrium: Nash strategies optimize the individual objective function of each player while assuming that the other players also act in an optimal way. However, since* V1(x) ̸= V ⋆ 1 (x) *in Fig. 5.9, the strategy of player 2 is not a Nash strategy which is indicated by* <sup>Υ</sup>2(t) ̸<sup>=</sup> <sup>1</sup> *according to Corollary 5.34. The same holds for player 1 in Fig. 5.9c where* V1(x) = V ⋆ 1 (x) *and* Υ<sup>1</sup> ≈ 1.017*.*

#### **5.3.3 Stability of the Modified Differential Game**

In this section, we show the relation between the sign of Υ<sup>i</sup> and asymptotic stability of the closed-loop system (5.70b), (5.76), (5.77). In particular, we show in the next lemma that an asymptotically stabilizing controller can always be constructed by choosing a positive solution for Υ<sup>i</sup> . Subsequently, we study the existence of a positive solution in Lemma 5.37.

**Lemma 5.36.** *Let* V (x) *be an admissible vector of CLFs for* (5.70) *and* ℓi(x) ≻ 0*. Then the equilibrium* x = **0** *of the closed-loop system* (5.70b)*,* (5.76)*–*(5.77) *is globally asymptotically stable, if* Υ<sup>i</sup> > 0*.*

*Proof.* Let Υ<sup>i</sup> > 0 holds. Inserting ui(x) = u ⋆ i (x) in (5.82) and application of (5.86) yields

$$\begin{split} \forall i \in \mathbb{P}: \qquad \dot{V}\_i(\mathbf{z}) &= -\frac{1}{2\Upsilon\_i} \left( \ell\_i(\mathbf{z}) + (\mathbf{u}\_i^\star)^\mathsf{T}(\mathbf{z}) \mathbf{S}(\mathbf{z}) \mathbf{u}\_i^\star(\mathbf{z}) \right) \\ &\leq -\frac{1}{2\Upsilon\_i} \ell\_i(\mathbf{z}) \prec 0. \end{split} \tag{5.104}$$

Thus for each <sup>i</sup> <sup>∈</sup> <sup>P</sup>, the Lyapunov function candidate <sup>V</sup>i(x) fulfills <sup>V</sup>i(x) <sup>≻</sup> <sup>0</sup> and <sup>V</sup>˙ <sup>i</sup>(x) ≺ 0 and is radially unbounded, which means that x = **0** is a globally asymptotically stable equilibrium of (5.70b), (5.76)–(5.77) [Kha02, Theorem 4.2].

**Lemma 5.37.** *If* (5.77) *has a real solution, then there exists exactly one solution* Υ<sup>+</sup> *with* Υ<sup>+</sup> > **0***.*

*Proof.* Let <sup>Υ</sup> <sup>∈</sup> <sup>R</sup><sup>N</sup> be a solution of (5.77). For each <sup>i</sup> <sup>∈</sup> <sup>P</sup>, (5.77) takes the form

$$\underbrace{q\_i^2(\mathbf{z}) \cdot \Upsilon\_i^2 + \left(q\_i^1(\mathbf{z}) + \underbrace{\sum\_{j=1 \atop j \neq i}^N q\_{ij}^2(\mathbf{z}) \Upsilon\_j}\_{b\_i}\right) \cdot \Upsilon\_i + \underbrace{q\_i^0(\mathbf{z})}\_{c\_i} = 0. \tag{5.105}$$

Thus, a solution of (5.105) is given by

$$\Upsilon\_i = \begin{cases} \frac{-b\_i \pm \sqrt{b\_i^2 - 4a\_i c\_i}}{2a\_i}, & a\_i \neq 0, \\\ -\frac{c\_i}{b\_i}, & a\_i = 0. \end{cases} \tag{5.106}$$

Due to the fact that q 0 i (x) <sup>≻</sup> <sup>0</sup>, the inequality <sup>c</sup><sup>i</sup> <sup>&</sup>gt; <sup>0</sup> holds for all <sup>x</sup> ̸<sup>=</sup> **<sup>0</sup>**. Moreover, <sup>a</sup><sup>i</sup> <sup>≼</sup> <sup>0</sup>, which follows directly from (5.80). Thus if a<sup>i</sup> < 0, then the "−" solution in the first row of (5.106) always leads to Υ<sup>i</sup> > 0. Otherwise, if a<sup>i</sup> = 0, then with the CLF property (5.93) of Vi(x) we get

$$b\_i = \left(\frac{\partial V\_i}{\partial x}\right)^\top \left(f(x) + \sum\_{\substack{j=1 \\ j\neq i}}^N \mathbf{G}\_j(x) u\_j\right) < 0,\tag{5.107}$$

which again results in Υ<sup>i</sup> > 0.

In the following, we assume that the positive solution Υ<sup>+</sup> of (5.77) is chosen. We finish this subsection with two concluding remarks.

$$\begin{array}{cccc} \textbf{Remark 5.38.} & \text{The continuity of } \Upsilon\_i(\mathfrak{x}) \text{ can be shown by applying de L'Hospital's rule to (5.106).}\\ & & \diamondsuit \end{array}$$

**Remark 5.39.** *By using the alternative Lyapunov function candidate* P <sup>i</sup>∈<sup>P</sup> Vi(x)*, it can be shown that asymptotic stability is still preserved, if* <sup>ℓ</sup>i(x) <sup>≻</sup> <sup>0</sup>*,* <sup>ℓ</sup>¬i(x) <sup>≽</sup> <sup>0</sup>*, and correspondingly, if* Υ + <sup>i</sup> > 0*,* Υ + <sup>¬</sup><sup>i</sup> <sup>≥</sup> <sup>0</sup> *holds for* some <sup>i</sup> <sup>∈</sup> <sup>P</sup>*. However, for the sake of brevity and to avoid case distinctions in the proofs, we will proceed with the stricter requirement* ℓi(x) ≻ 0 *for all* <sup>i</sup> <sup>∈</sup> <sup>P</sup>*.* ♢

#### **5.3.4 Admissible Vectors of Control-Lyapunov Functions for Input-State-Output Port-Hamiltonian Systems**

As mentioned above, finding an appropriate CLF for a given system is in general very difficult, even in the single-player case. Therefore, in this subsection, we will investigate under which condition the Hamiltonian H(x) can be used as a CLF Vi(x), in particular, under which conditions the vector <sup>H</sup>(x) · <sup>1</sup> is an admissible vector of CLFs.

**Lemma 5.40.** *The vector* <sup>V</sup> <sup>0</sup>(x) = coli{H(x)} <sup>=</sup> <sup>H</sup>(x) · <sup>1</sup> *is an admissible vector of CLFs for the input-state-output PHS* (5.70b)*, if and only if*

$$\forall x \in \mathcal{X}\_{G\_i} \colon \left(\frac{\partial H}{\partial x}\right)^{\mathsf{T}} \mathbf{R}(x) \frac{\partial H}{\partial x} + \sum\_{\substack{j=1 \\ j \neq i}}^{N} \left(\frac{\partial H}{\partial x}\right)^{\mathsf{T}} \mathbf{G}\_j(x) (\mathbf{S}\_j(x))^{-1} \mathbf{G}\_j^{\mathsf{T}}(x) \frac{\partial H}{\partial x} \cdot \Upsilon\_j > 0,\tag{5.108}$$

*where*

$$\mathcal{X}\_{\mathbf{G}\_i} = \{ \mathbf{z} \in \mathbb{R}^n \, : \, \mathbf{G}\_i^{\sf T}(\mathbf{z}) \frac{\partial H}{\partial x} = \mathbf{0}, \mathbf{z} \neq \mathbf{0} \}. \tag{5.109}$$

*Proof.* With X<sup>G</sup><sup>i</sup> as in (5.109), condition (5.93) is equivalent to

$$\forall i \in \mathbb{P}, \ x \in X\_{\mathbf{G}\_i} : \quad \left(\frac{\partial V\_i}{\partial x}\right)^{\mathsf{T}} \left(f(x) + \sum\_{\substack{j=1 \\ j \neq i}}^{N} \mathbf{G}\_j(x) u\_j\right) \begin{cases} < 0, & x \neq 0\_n, \\ = 0, & x = 0\_n. \end{cases} \tag{5.110}$$

Now with

$$\begin{split} & \left(\frac{\partial V\_{i}}{\partial x}\right)^{\mathsf{T}} \Biggl( \mathbf{f}(\mathbf{x}) + \sum\_{\begin{subarray}{c} j=1\\ j\neq i \end{subarray}}^{N} \mathbf{G}\_{j}(\mathbf{x}) \mathbf{u}\_{j} \Biggr) \\ &= \left(\frac{\partial H}{\partial x}\right)^{\mathsf{T}} (\mathbf{J}(\mathbf{x}) - \mathbf{R}(\mathbf{x})) \frac{\partial H}{\partial \mathbf{x}} - \sum\_{\begin{subarray}{c} j=1\\ j\neq i \end{subarray}}^{N} \left(\frac{\partial H}{\partial \mathbf{x}}\right)^{\mathsf{T}} \mathbf{G}\_{j}(\mathbf{x}) (\mathbf{S}\_{j}(\mathbf{x}))^{-1} \mathbf{G}\_{j}^{\mathsf{T}}(\mathbf{x}) \frac{\partial H}{\partial \mathbf{x}} \cdot \Upsilon\_{j} \\ &= -\left(\frac{\partial H}{\partial \mathbf{x}}\right)^{\mathsf{T}} \mathbf{R}(\mathbf{x}) \frac{\partial H}{\partial \mathbf{x}} - \sum\_{\begin{subarray}{c} j=1\\ j\neq i \end{subarray}}^{N} \left(\frac{\partial H}{\partial \mathbf{x}}\right)^{\mathsf{T}} \mathbf{G}\_{j}(\mathbf{x}) (\mathbf{S}\_{j}(\mathbf{x}))^{-1} \mathbf{G}\_{j}^{\mathsf{T}}(\mathbf{x}) \frac{\partial H}{\partial \mathbf{x}} \cdot \Upsilon\_{j}, \end{split} \tag{5.111}$$

the vector <sup>V</sup> <sup>0</sup>(x) = coli{H(x)} <sup>=</sup> <sup>H</sup>(x) · <sup>1</sup> is an admissible vector of CLFs for the inputstate-output PHS (5.70b), if and only if (5.108) is fulfilled.

**Corollary 5.41.** *The vector* <sup>V</sup> <sup>0</sup>(x) = coli{H(x)} <sup>=</sup> <sup>H</sup>(x) · <sup>1</sup> *is an admissible vector of CLFs for the input-state-output PHS* (5.70b)*, if and only if* (5.70b) *with passive outputs*

$$y\_i = G\_i^{\mathbb{T}}(x) \frac{\partial H(x)}{\partial x} \tag{5.112}$$

*is zero-state detectable.*

*Proof.* Let <sup>V</sup> <sup>0</sup>(x) = coli{H(x)} <sup>=</sup> <sup>H</sup>(x) · <sup>1</sup> be an admissible vector of CLFs for the inputstate-output PHS (5.70b). Then (5.108) is fulfilled (and vice versa). Due to the fact that <sup>X</sup><sup>G</sup><sup>i</sup> <sup>=</sup> {<sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup> : <sup>G</sup><sup>T</sup> i (x) ∂H <sup>∂</sup><sup>x</sup> = **0**,x ̸= **0**}, (5.108) is equivalent to

$$\forall x \in \{x \in \mathbb{R}^n : y\_i(x) = 0\}: \qquad \qquad \dot{H}(x) < 0. \tag{5.113}$$

With the Lyapunov function candidate H(x) ≻ 0, this means that (5.70b), (5.112) is zero-state detectable (cf. [van17, p. 47]).

Although the lemmas presented so far are necessary and sufficient, they are generally difficult to verify. Therefore, the following lemma presents a more practical condition which can be directly evaluated based on the given system and input matrices in (5.70b):

**Lemma 5.42.** *The vector* <sup>V</sup> <sup>0</sup>(x) = coli{H(x)} <sup>=</sup> <sup>H</sup>(x) · <sup>1</sup> *is an admissible vector of CLFs for the input-state-output PHS* (5.70b)*, if* H(x) *is pseudoconvex and at least one of the following conditions holds:*


*Proof.* Since H(x) is pseudoconvex, there exists no x ′ ̸= **0** such that ∇H(x ′ ) = **0**. 1. If rank{R(x)} = n, then R(x) ≻ 0 and thus

$$\begin{split} & \left( \frac{\partial V\_{i}}{\partial x} \right)^{\mathsf{T}} \Bigg( \boldsymbol{f}(\boldsymbol{x}) + \sum\_{j=1 \atop j \neq i}^{N} \boldsymbol{G}\_{j}(\boldsymbol{x}) \boldsymbol{u}\_{j} \Bigg) \\ &= \left( \frac{\partial \boldsymbol{H}}{\partial \boldsymbol{x}} \right)^{\mathsf{T}} (\boldsymbol{J}(\boldsymbol{x}) - \boldsymbol{R}(\boldsymbol{x})) \frac{\partial \boldsymbol{H}}{\partial \boldsymbol{x}} - \sum\_{j=1 \atop j \neq i}^{N} \left( \frac{\partial \boldsymbol{H}}{\partial \boldsymbol{x}} \right)^{\mathsf{T}} \boldsymbol{G}\_{j}(\boldsymbol{x}) (\boldsymbol{S}\_{j}(\boldsymbol{x}))^{-1} \boldsymbol{G}\_{j}^{\mathsf{T}}(\boldsymbol{x}) \frac{\partial \boldsymbol{H}}{\partial \boldsymbol{x}} \cdot \boldsymbol{\Upsilon}\_{j} \\ &= - \underbrace{\left( \frac{\partial \boldsymbol{H}}{\partial \boldsymbol{x}} \right)^{\mathsf{T}} \boldsymbol{R}(\boldsymbol{x}) \frac{\partial \boldsymbol{H}}{\partial \boldsymbol{x}}}\_{>0} - \sum\_{j=1 \atop j \neq i}^{N} \underbrace{\left( \frac{\partial \boldsymbol{H}}{\partial \boldsymbol{x}} \right)^{\mathsf{T}} \boldsymbol{G}\_{j}(\boldsymbol{x}) (\boldsymbol{S}\_{j}(\boldsymbol{x}))^{-1} \boldsymbol{G}\_{j}^{\mathsf{T}}(\boldsymbol{x}) \frac{\partial \boldsymbol{H}}{\partial \boldsymbol{x}}}\_{\geq 0} \underbrace{\boldsymbol{\Upsilon}\_{j}}\_{>0} < 0 \end{split} \tag{5.114}$$

is always fulfilled for all <sup>i</sup> <sup>∈</sup> <sup>P</sup>, which implies that (5.108) is fulfilled.

2. If at least two input matrices G<sup>i</sup><sup>1</sup> (x), G<sup>i</sup><sup>2</sup> (x) have full rank <sup>n</sup>, then for all <sup>i</sup> <sup>∈</sup> <sup>P</sup> there exists at least one <sup>k</sup> ̸<sup>=</sup> <sup>i</sup> such that <sup>G</sup>k(x)(Sk(x))−<sup>1</sup>G<sup>T</sup> k (x) ≻ 0. Accordingly, for all <sup>i</sup> <sup>∈</sup> <sup>P</sup> the inequality

$$\begin{split} & \left(\frac{\partial H}{\partial x}\right)^{\mathsf{T}} (\boldsymbol{J}(\boldsymbol{x}) - \boldsymbol{\mathsf{R}}(\boldsymbol{x})) \underbrace{\partial H}\_{\partial \boldsymbol{x}} - \sum\_{\begin{subarray}{c} j=1\\ j\neq i \end{subarray}}^{N} \left(\frac{\partial H}{\partial \boldsymbol{x}}\right)^{\mathsf{T}} \boldsymbol{G}\_{j}(\boldsymbol{x}) (\boldsymbol{S}\_{j}(\boldsymbol{x}))^{-1} \boldsymbol{G}\_{j}^{\mathsf{T}}(\boldsymbol{x}) \frac{\partial H}{\partial \boldsymbol{x}} \cdot \boldsymbol{\Upsilon}\_{j} \\ &= -\underbrace{\left(\frac{\partial H}{\partial \boldsymbol{x}}\right)^{\mathsf{T}} \boldsymbol{R}(\boldsymbol{x}) \frac{\partial H}{\partial \boldsymbol{x}}}\_{\geq 0} - \underbrace{\left(\frac{\partial H}{\partial \boldsymbol{x}}\right)^{\mathsf{T}} \boldsymbol{G}\_{k}(\boldsymbol{x}) (\boldsymbol{S}\_{k}(\boldsymbol{x}))^{-1} \boldsymbol{G}\_{k}^{\mathsf{T}}(\boldsymbol{x}) \frac{\partial H}{\partial \boldsymbol{x}} \cdot \boldsymbol{\Upsilon}\_{k} \\ & \quad - \sum\_{\begin{subarray}{c} j=1\\ j\neq i,k \end{subarray}}^{N} \underbrace{\left(\frac{\partial H}{\partial \boldsymbol{x}}\right)^{\mathsf{T}} \boldsymbol{G}\_{j}(\boldsymbol{x}) (\boldsymbol{S}\_{j}(\boldsymbol{x}))^{-1} \boldsymbol{G}\_{j}^{\mathsf{T}}(\boldsymbol{x}) \frac{\partial H}{\partial \boldsymbol{x}} \cdot \boldsymbol{\Upsilon}\_{j} < 0 \end{split} \tag{5.115}$$

holds, which implies that (5.108) is fulfilled.

In the following, Lemma 5.43 gives a necessary and sufficient condition for the special case of (5.70b) being a *linear* input-state-output PHS:

**Lemma 5.43.** *For the linear input-state-output PHS*

$$
\dot{x} = (J - R)Qx + \sum\_{i=1}^{N} G\_i u\_i \tag{5.116}
$$

*with* H(x) = <sup>1</sup> 2 x <sup>T</sup>Qx*,* <sup>Q</sup> <sup>≻</sup> <sup>0</sup>*, the vector* <sup>V</sup> <sup>0</sup>(x) = coli{H(x)} <sup>=</sup> <sup>H</sup>(x) · <sup>1</sup> *is an admissible vector of CLFs, if and only if*

$$\ker(\mathbf{R}) \cap \left(\bigcap\_{i=1}^{N} \ker(\mathbf{G}\_i^{\sf T})\right) \cap \left(\bigcup\_{i=1}^{N} \bigcap\_{\substack{j=1 \\ j \neq i}}^{N} \ker(\mathbf{G}\_j \mathbf{S}\_j^{-1} \mathbf{G}\_j^{\sf T})\right) = \emptyset. \tag{5.117}$$

*Proof.* (see Appendix D.2).

#### **5.3.5 Adaptive Differential Game**

From the previous subsection, it can be concluded that the application of the modified differential game (5.75) with V (x) being an admissible vector of CLFs always allows the derivation of stabilizing controllers u ⋆ i (x) which constitute a Nash strategy with respect to (5.94) and (5.70b). However, since Υ<sup>i</sup> depends on V (x), the condition Υ<sup>i</sup> = Υ<sup>j</sup> = 1 in Corollary 5.34 is in general not fulfilled for all <sup>i</sup> <sup>∈</sup> <sup>P</sup>. Thus, according to Corollary 5.34, the modified differential game is not equal to the formulation (5.70).

To eliminate this discrepancy in the same spirit as for the single-player case, we thus introduce a vector of extended CLFs V (x, w) with additional weighted sum of basis functions. The aim is to achieve <sup>Υ</sup>i(t) <sup>≡</sup> <sup>1</sup> for all <sup>i</sup> <sup>∈</sup> <sup>P</sup>, so that according to Lemma 5.33, the scaled objective functions (5.94) being minimized by the resulting modified optimal controllers (5.76)–(5.77), are identical to the original objective functions (5.70a).

#### **Adaptation using Extended CLFs**

Each extended CLF Vi(x, wi) is supposed to be the sum of H(x) and a weighted sum of <sup>r</sup><sup>i</sup> <sup>∈</sup> <sup>N</sup> basis functions

$$V\_i(x, w\_i) = H(x) + w\_i^\dagger \Xi\_i(x),\tag{5.118}$$

where Ξ<sup>i</sup> : <sup>R</sup><sup>n</sup> <sup>→</sup> <sup>R</sup><sup>r</sup><sup>i</sup> and <sup>w</sup><sup>i</sup> <sup>∈</sup> <sup>R</sup><sup>r</sup><sup>i</sup> . We assume that all basis functions Ξi(x) are smooth and "properly chosen" in the sense that the (optimal) value functions V ⋆ i (x), i.e. the solutions of the corresponding HJB equations

$$
\omega \mathcal{H}\_i^{\ell} \left( x, u^\star, \frac{\partial V\_i^\star}{\partial x} \right) = 0,\tag{5.119}
$$

can be parameterized via Ξi(x) and a vector w<sup>⋆</sup> i of optimal weighting factors:

**Assumption 5.44.** *For each player* <sup>i</sup> <sup>∈</sup> <sup>P</sup>*, there exists a* <sup>w</sup><sup>⋆</sup> <sup>i</sup> <sup>∈</sup> <sup>R</sup><sup>r</sup><sup>i</sup> *such that* V ⋆ i (x) = H(x) + (w<sup>⋆</sup> i ) <sup>T</sup>Ξi(x). (5.120)

Analogously to Remark 5.16, Assumption 5.44 is justified by the Weierstraß approximation theorem whenever the number of basis functions is large (see also [VL11, p. 1559]).

Now the aim is to set up w<sup>i</sup> and thus Vi(x, wi) in such a way that w<sup>i</sup> = w<sup>⋆</sup> i in order to enable that V (x, w) = coli{Vi(x, wi)} constitutes a vector of value functions for (5.70), i.e. a solution of the set of HJB equations (5.119) associated to the original problem (5.70). This property can be checked using the following corollary:

**Corollary 5.45.** *For all* <sup>i</sup> <sup>∈</sup> <sup>P</sup>*, let* <sup>V</sup>i(x) = <sup>H</sup>(x) + (w<sup>⋄</sup> i ) <sup>T</sup>Ξi(x) *be a given CLF with* w<sup>⋄</sup> <sup>i</sup> <sup>∈</sup> <sup>R</sup><sup>r</sup><sup>i</sup> *. Then* V (x) *is equivalent to the vector of value functions* V ⋆ (x) = coli{V ⋆ i (x)} *of* (5.70)*, if and only if*

$$\forall i \in \mathbb{P}, \ x \in \mathbb{R}^n: \qquad \qquad \qquad (x, w\_i^\diamond) \in \mathcal{Q}\_i(x, w\_i), \tag{5.121}$$

*where*

$$\mathcal{Q}\_i(x, w) := \left\{ (x, w\_i) \in \mathbb{R}^n \times \mathbb{R}^{r\_i} : w\_i^\mathsf{T} A\_i(x) w\_i + a\_i^\mathsf{T}(x) w\_i + a\_i(x) = 0 \right\},$$

(5.122)

$$\mathbf{A}\_{i}(\mathbf{x}) \coloneqq \frac{\partial \Xi\_{i}}{\partial \mathbf{x}} \mathbf{K}\_{i}(\mathbf{x}) \left(\frac{\partial \Xi\_{i}}{\partial \mathbf{x}}\right)^{\mathsf{T}},\tag{5.123}$$
 
$$\mathbf{A}\_{i}(\mathbf{x}) \dots \mathbf{A}\_{i}(\mathbf{x}) \qquad , \quad \mathbf{x} \in \mathbb{R}^{n}, \quad \mathbf{x} \in \mathbb{R}^{n}, \qquad \mathbf{x} \in \mathbb{R}^{n}$$

$$\begin{split} \mathbf{a}\_{i}^{\mathsf{T}}(\boldsymbol{x}) := & 2 \left( \left( \frac{\partial H}{\partial \boldsymbol{x}} \right)^{\mathsf{T}} \mathbf{K}\_{i}(\boldsymbol{x}) \left( \frac{\partial \boldsymbol{\Xi}\_{i}}{\partial \boldsymbol{x}} \right)^{\mathsf{T}} + \left( \frac{\partial H}{\partial \boldsymbol{x}} \right)^{\mathsf{T}} (\mathbf{J}(\boldsymbol{x}) + \mathbf{R}(\boldsymbol{x})) \left( \frac{\partial \boldsymbol{\Xi}\_{i}}{\partial \boldsymbol{x}} \right)^{\mathsf{T}} \\ & - 2 \sum\_{j=1} \mathbf{u}\_{j}^{\mathsf{T}} \mathbf{G}\_{j}^{\mathsf{T}}(\boldsymbol{x}) \frac{\partial \boldsymbol{\Xi}\_{i}}{\partial \boldsymbol{x}} \right), \end{split} \tag{5.124}$$

$$\begin{split} a\_i(\mathbf{x}) & \coloneqq \left(\frac{\partial H}{\partial \mathbf{x}}\right)^{\mathsf{T}} \mathbf{K}\_i(\mathbf{x}) \frac{\partial H}{\partial \mathbf{x}} + 2 \left(\frac{\partial H}{\partial \mathbf{x}}\right)^{\mathsf{T}} \mathbf{R}(\mathbf{x}) \frac{\partial H}{\partial \mathbf{x}} - \ell\_i(\mathbf{x}) \\ & - 4 \sum\_{\begin{subarray}{c} j=1 \\ j \neq i \end{subarray}} u\_j^{\mathsf{T}} \mathbf{G}\_j^{\mathsf{T}}(\mathbf{x}) \frac{\partial H}{\partial \mathbf{x}}, \end{split} \tag{5.125}$$

$$\mathbf{K}\_i(x) \coloneqq \mathbf{G}\_i(x) (\mathbf{S}\_i(x))^{-1} \mathbf{G}\_i^{\mathsf{T}}(x). \tag{5.126}$$

*Proof.* (see Appendix D.2).

**Lemma 5.46.** *For all* <sup>i</sup> <sup>∈</sup> <sup>P</sup>*, let*

$$\mathfrak{Q}\_{i}(x, w\_{i}) := w\_{i}^{\mathsf{T}} A\_{i}(x) w\_{i} + a\_{i}^{\mathsf{T}}(x) w\_{i} + a\_{i}(x) \tag{5.127}$$

*with* Ai(x)*,* ai(x) *and* ai(x) *as in* (5.123)*–*(5.125) *be the corresponding quadratic function of the quadric* <sup>Q</sup>i(x, <sup>w</sup>i) *in* (5.122)*. Then for each arbitrary but fixed* <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>*, the objective*

*function*

$$\mathcal{J}\_{\mathbf{w},i}(x,w\_i) := \sum\_{j=1}^{r\_i} (\mathbb{Q}\_i(x+\mathbf{c}\_{i,j}))^2,\tag{5.128}$$

*consisting of a sum of squared quadratic functions* Q<sup>i</sup> *whose arguments are shifted by* <sup>c</sup>i,1, . . . , <sup>c</sup>i,r<sup>i</sup> <sup>∈</sup> <sup>R</sup><sup>N</sup> *, has a global minimizer at* <sup>w</sup><sup>i</sup> <sup>=</sup> <sup>w</sup><sup>⋆</sup> i *and is locally strictly convex in an open neighborhood* <sup>M</sup><sup>i</sup> *of* <sup>w</sup><sup>⋆</sup> i *, if and only if the matrices*

$$\hat{\boldsymbol{A}}\_{i} := \begin{bmatrix} 2(\boldsymbol{w}\_{i}^{\star})^{\mathsf{T}} \boldsymbol{A}\_{i}^{\mathsf{T}} (\boldsymbol{x} + \boldsymbol{c}\_{i,1}) + \boldsymbol{a}\_{i}^{\mathsf{T}} (\boldsymbol{x} + \boldsymbol{c}\_{i,1}) \\ \vdots \\ 2(\boldsymbol{w}\_{i}^{\star})^{\mathsf{T}} \boldsymbol{A}\_{i}^{\mathsf{T}} (\boldsymbol{x} + \boldsymbol{c}\_{i,r\_{i}}) + \boldsymbol{a}\_{i}^{\mathsf{T}} (\boldsymbol{x} + \boldsymbol{c}\_{i,r\_{i}}) \end{bmatrix} \tag{5.129}$$

*have full rank* r<sup>i</sup> *.*

*Proof.* The proof is a direct consequence of the proof of Lemma 5.19. Here it has to hold for each player separately.

**Remark 5.47.** *Let* <sup>w</sup> = coli{wi} *with* <sup>w</sup> <sup>∈</sup> <sup>R</sup><sup>r</sup> *and* <sup>r</sup> <sup>=</sup> <sup>r</sup>1+· · ·+r<sup>N</sup> *denote the concatenation of all weights* w<sup>i</sup> *,* <sup>i</sup> <sup>∈</sup> <sup>P</sup> *and let* <sup>J</sup>ˆw(x, <sup>w</sup>) = <sup>P</sup> <sup>i</sup>∈<sup>P</sup> <sup>J</sup>w,i(x, <sup>w</sup>i) *be the sum of all objective functions* Jw,i(x, w)*. Then*

$$\boldsymbol{w}^{\star} = \underset{\boldsymbol{w}}{\text{arg min}} \; \{\hat{\mathcal{J}}\_{\text{w}}(\boldsymbol{x}, \boldsymbol{w})\}. \tag{5.130}$$


**Remark 5.48.** *For simplicity of notation, it is pointed out that the condition in Lemma 5.46 concerning local strict convexity of objective functions* Jw,i(wi) *in an open neighborhood* M<sup>i</sup> *of* w<sup>⋆</sup> i *is equivalent to the statement that the compound objective function* <sup>J</sup>ˆw(x, <sup>w</sup>) *is locally strictly convex in an open neighborhood* <sup>M</sup> <sup>=</sup> <sup>M</sup><sup>1</sup> × · · · × M<sup>N</sup> *of* <sup>w</sup><sup>⋆</sup> *.* ♢

With Remarks 5.47 and 5.48, we can again apply the unconstrained gradient flow to adapt w, which finally leads to

$$\dot{w}\_i = -\alpha \cdot \frac{\partial \hat{\mathcal{J}}\_{\mathbf{w}}(x, w)}{\partial w\_i} = -\alpha \cdot \frac{\partial \mathcal{J}\_{\mathbf{w}, i}(x, w\_i)}{\partial w\_i},\tag{5.131}$$

where α > 0 specifies the learning rate.

#### **Closed-Loop System**

The system dynamics (5.70b) together with the adaptation procedure (5.131) result in an extended system

$$\dot{x} = (J(x) - R(x))\frac{\partial H(x)}{\partial x} + \sum\_{i=1}^{N} \mathbf{G}\_i(x)u\_i,\tag{5.132a}$$

$$\dot{w}\_i = -\alpha \cdot \frac{\partial \mathcal{J}\_{\mathbf{w},i}(x, w\_i)}{\partial w\_i}, \quad \quad \quad \quad \quad \quad \quad i \in \mathbb{P}. \tag{5.132b}$$

With (5.118), we obtain

$$\frac{\partial V\_i(x, w\_i)}{\partial x} = \frac{\partial H}{\partial x} + \left(\frac{\partial \Xi\_i}{\partial x}\right)^{\mathsf{T}} w\_i \tag{5.133}$$

and accordingly the modified differential game for the extended system (5.132) reads

$$\forall i \in \mathbb{P}: \qquad \min\_{\mathbf{u}\_i} \left\{ \frac{1}{2} \int\_0^{t\_\mathbf{f}} \ell\_i(\mathbf{z}) + (\mathbf{u}\_i)^\mathsf{T} S\_i(\mathbf{z}) \mathbf{u}\_i \, \mathrm{d}t \right\} \tag{5.134a}$$

$$\text{s.t.} \quad \dot{V}\_i(\mathbf{z}, \boldsymbol{w}) = \left( \frac{\partial V\_i(\mathbf{z}, \boldsymbol{w})}{\partial \boldsymbol{x}} \right)^\mathsf{T} \dot{\boldsymbol{x}} + \sum\_{i=1}^N \left( \frac{\partial V\_i(\mathbf{z}, \boldsymbol{w})}{\partial \boldsymbol{w}\_i} \right)^\mathsf{T} \dot{\boldsymbol{w}}\_i. \tag{5.134b}$$

For this modified differential game, it is straightforward to obtain the explicit control law from (5.9)

$$u\_i^\star = -(\mathcal{S}\_i(x))^{-1} \mathcal{G}\_i^\top(x) \frac{\partial V\_i}{\partial x} \cdot \Upsilon\_i(x, w\_i),\tag{5.135}$$

where

Υi(x, wi) = f ′ Υ,i ± q (f ′ <sup>Υ</sup>,i) <sup>2</sup> + Q′ <sup>Υ</sup>,i · S ′ Υ,i S ′ Υ,i , S′ <sup>Υ</sup>,i ̸= 0, − Q′ Υ,i 2f ′ Υ,i , S′ <sup>Υ</sup>,i = 0, (5.136) f ′ <sup>Υ</sup>,i(x, <sup>w</sup>) = ∂V<sup>i</sup> ∂x T (J(x) <sup>−</sup> <sup>R</sup>(x))∂H ∂x − 1 2 X N j=1 j̸=i ∂V<sup>i</sup> ∂x T K<sup>j</sup> (x) ∂V<sup>j</sup> ∂x · Υ<sup>j</sup> − ∂V<sup>i</sup> ∂w<sup>i</sup> T · α · ∂Jw,i ∂w<sup>i</sup> = ∂H ∂x T + w<sup>T</sup> i ∂Ξ<sup>i</sup> ∂x ! × · · · · · · × (J(x) <sup>−</sup> <sup>R</sup>(x))∂H ∂x − X N j=1 j̸=i Υ<sup>j</sup> 2 K<sup>j</sup> (x) ∂H ∂x + ∂Ξ<sup>j</sup> ∂x T w<sup>j</sup> ! ! − ∂V<sup>i</sup> ∂w<sup>i</sup> T · α · ∂Jw,i ∂w<sup>i</sup> (5.137) = ∂H ∂x T + w<sup>T</sup> i ∂Ξ<sup>i</sup> ∂x ! (J(x) <sup>−</sup> <sup>R</sup>(x))∂H ∂x + 1 2 X N j=1 j̸=i G<sup>j</sup> (x)u ⋆ j ! − ∂V<sup>i</sup> ∂w<sup>i</sup> T · α · ∂Jw,i ∂w<sup>i</sup> , (5.138) S ′ <sup>Υ</sup>,i(x, <sup>w</sup>) = ∂V<sup>i</sup> ∂x T Ki(x) ∂V<sup>i</sup> ∂x = w<sup>T</sup> i ∂Ξ<sup>i</sup> ∂x Ki(x) ∂Ξ<sup>i</sup> ∂x T <sup>w</sup> + 2 ∂H ∂x T Ki(x) ∂Ξ<sup>i</sup> ∂x T w<sup>i</sup> + ∂H T Ki(x) ∂H , (5.139)

∂x ∂x Q ′ <sup>Υ</sup>,i(x) = ℓi(x), (5.140)

$$\mathbf{K}\_i(x) = \mathbf{G}\_i(x)\mathbf{S}\_i^{-1}(x)\mathbf{G}\_i^{\mathsf{T}}(x). \tag{5.141}$$

Note that the "+" sign in the first row of (5.136) can be discarded with the same reasoning as in the proof of Lemma 5.37. Moreover, continuity of Υi(x, wi) is still preserved (cf. Remark 5.38). Finally, (5.137) and (5.138) are equivalent as it follows from (5.135) that

$$\left(\Upsilon\_i \cdot \mathbf{K}\_j(x) \left(\frac{\partial H}{\partial x} + \left(\frac{\partial \Xi\_j}{\partial x}\right)^\top w\_j\right) = -G\_j(x)u\_j. \tag{5.142}$$

(5.143c)

**Remark 5.49.** *From the controller equations* (5.135)*–*(5.136)*,* (5.138)*–*(5.141) *it becomes evident that neither* <sup>Ξ</sup><sup>j</sup> (x) *nor* <sup>w</sup><sup>j</sup> *of the other players* <sup>j</sup> ̸<sup>=</sup> <sup>i</sup> *must be known by* <sup>i</sup> <sup>∈</sup> <sup>P</sup>*.* ♢

By combining the modified optimal control law and the adaptation procedure (5.131), we get the closed-loop system

$$\dot{\boldsymbol{x}} = (\boldsymbol{J}(\boldsymbol{x}) - \boldsymbol{R}(\boldsymbol{x})) \frac{\partial \boldsymbol{H}}{\partial \boldsymbol{x}} - \sum\_{i=1}^{N} \Upsilon\_{i} \cdot \boldsymbol{G}\_{i}(\boldsymbol{x}) (\boldsymbol{S}\_{i}(\boldsymbol{x}))^{-1} \boldsymbol{G}\_{i}^{\top}(\boldsymbol{x}) \frac{\partial V\_{i}}{\partial \boldsymbol{x}},\tag{5.143a}$$

$$
\dot{\boldsymbol{w}}\_i = -\boldsymbol{\alpha} \cdot \frac{\partial \mathcal{J}\_{\mathbf{w},i}}{\partial \mathbf{w}\_i},
\tag{5.143b}
$$

$$0 = q\_i^0(\mathbf{z}) + \Upsilon\_i \cdot q\_i^1(\mathbf{z}, \mathbf{w}\_i) + \Upsilon\_i^2 q\_i^2(\mathbf{z}, \mathbf{w}\_i) + \Upsilon\_i \sum\_{\substack{j=1 \\ j \neq i}}^N q\_{ij}^2(\mathbf{z}, \mathbf{w}\_i, \mathbf{w}\_j) \Upsilon\_j, \quad i \in \mathbb{P}, \quad j$$

$$\mathbf{x}(0) = \mathbf{x}\_0,\tag{5.143d}$$

$$w\_i(0) = \mathbb{0}\_{r\_i},\tag{5.143e}$$

where

$$q\_i^0(\mathbf{z}) := \frac{1}{2} \ell\_i(\mathbf{z}),\tag{5.144}$$

$$q\_i^1(\mathbf{z}, \mathbf{w}\_i) := q\_i^1(\mathbf{z}) - \left(\frac{\partial V\_i}{\partial \mathbf{w}\_i}\right)^\mathsf{T} \cdot \alpha \cdot \frac{\partial \mathcal{J}\_{\mathbf{w},i}}{\partial \mathbf{w}\_i},\tag{5.145}$$

$$q\_i^2(x, w\_i) := -\frac{1}{2} \left(\frac{\partial V\_i}{\partial x}\right)\_w^\top G\_i(x) (S\_i(x))^{-1} G\_i^\top(x) \frac{\partial V\_i}{\partial x},\tag{5.146}$$

$$q\_{ij}^2(x, w\_i, w\_j) := -\left(\frac{\partial V\_i}{\partial x}\right)^\dagger G\_j(x) (\mathbf{S}\_j(x))^{-1} \mathbf{G}\_j^\top(x) \frac{\partial V\_j}{\partial x}.\tag{5.147}$$

#### **5.3.6 Stability of the Adaptive Differential Game**

To perform a stability analysis of the equilibrium (**0**n, w<sup>⋆</sup> ) of (5.143), we define the Lyapunov function candidate

$$\hat{V}(x, w) := \sum\_{i=1}^{N} V\_i(x, w) \tag{5.148}$$

and show that it is indeed a Lyapunov function for the closed-loop system (5.143).

**Lemma 5.50.** *Consider the closed-loop system* (5.143) *starting at* (x0, **0**r)*. The Lyapunov function candidate* Vˆ (x, w) *is strictly monotonically decreasing over time unless* (x, w) = (**0**n, w<sup>⋆</sup> )*.*

*Proof.* The time derivative of Vi(x, wi) equals

$$\begin{split} \dot{V}\_{i}(\boldsymbol{x},\boldsymbol{w}\_{i}) &= \left(\frac{\partial V\_{i}}{\partial \boldsymbol{x}}\right)^{\top} \dot{\boldsymbol{x}} + \left(\frac{\partial V\_{i}}{\partial \boldsymbol{w}\_{i}}\right)^{\top} \dot{\boldsymbol{w}}\_{i} \\ &= \left(\frac{\partial V\_{i}}{\partial \boldsymbol{x}}\right)^{\top} (\boldsymbol{J}(\boldsymbol{x}) - \boldsymbol{R}(\boldsymbol{x})) \frac{\partial H}{\partial \boldsymbol{x}} - \Upsilon\_{i} \cdot \left(\frac{\partial V\_{i}}{\partial \boldsymbol{x}}\right)^{\top} \boldsymbol{G}\_{i}(\boldsymbol{x}) (\boldsymbol{S}\_{i}(\boldsymbol{x}))^{-1} \boldsymbol{G}\_{i}^{\top}(\boldsymbol{x}) \frac{\partial V\_{i}}{\partial \boldsymbol{x}} \\ &- \sum\_{\begin{subarray}{c} j=1\\ j\neq i \end{subarray}}^{N} \Upsilon\_{j} \cdot \left(\frac{\partial V\_{j}}{\partial \boldsymbol{x}}\right)^{\top} \boldsymbol{G}\_{j}(\boldsymbol{x}) (\boldsymbol{S}\_{j}(\boldsymbol{x}))^{-1} \boldsymbol{G}\_{j}^{\top}(\boldsymbol{x}) \frac{\partial V\_{i}}{\partial \boldsymbol{x}} + \left(\frac{\partial V\_{i}}{\partial \boldsymbol{w}\_{i}}\right)^{\top} \dot{\boldsymbol{w}}\_{i}. \end{split} \tag{5.149}$$

Dividing (5.143c) by <sup>Υ</sup><sup>i</sup> <sup>&</sup>gt; <sup>0</sup> and inserting ˙ Vˆ <sup>i</sup> from (5.149) yields

$$\frac{1}{2}\frac{\ell\_i(x)}{\Upsilon\_i} + \dot{\hat{V}}\_i + \frac{\Upsilon\_i}{2} \cdot \left(\frac{\partial V\_i}{\partial x}\right)^{\mathsf{T}} \mathbf{G}\_i(x) (\mathbf{S}\_i(x))^{-1} \mathbf{G}\_i^{\mathsf{T}}(x) \frac{\partial V\_i}{\partial x} = 0,\tag{5.150}$$

which finally leads to

$$\dot{\hat{V}}\_{i} = -\underbrace{\frac{1}{2}\frac{\ell\_{i}(\mathbf{z})}{\Upsilon\_{i}}}\_{>0} - \underbrace{\frac{\Upsilon\_{i}}{2}}\_{>0} \cdot \underbrace{\left(\frac{\partial V\_{i}}{\partial \mathbf{z}}\right)^{\mathsf{T}} G\_{i}(\mathbf{z}) (\mathbf{S}\_{i}(\mathbf{z}))^{-1} G\_{i}^{\mathsf{T}}(\mathbf{z}) \frac{\partial V\_{i}}{\partial \mathbf{z}}}\_{\geqslant 0} \prec 0. \tag{5.151}$$

As a consequence, ˙ Vˆ = P<sup>N</sup> <sup>i</sup>=1 Vi(x, wi) < 0, unless x = **0**n.

**Remark 5.51.** *As already mentioned in Remark 5.39, it is sufficient for negative definiteness of* ˙ Vˆ (x, w) *(see* (5.151)*) and thus asymptotic stability of the closed-loop equilibrium to require that* <sup>ℓ</sup>i(x) <sup>≻</sup> <sup>0</sup>*,* <sup>ℓ</sup>¬<sup>i</sup>(x) <sup>≽</sup> <sup>0</sup> *holds for some* <sup>i</sup> <sup>∈</sup> <sup>P</sup>*.* ♢

**Lemma 5.52.** *Consider the closed-loop system* (5.143) *starting at* (x0, **0**r)*. Let the conditions of Lemma 5.46 hold with* w<sup>0</sup> ∈ M*. Then the Lyapunov function candidate* Vˆ (x, w) = 1 <sup>T</sup>V (x, w) *as in* (5.148) *fulfills*

$$\forall t > 0, (\mathbf{z}, \mathbf{w}) \neq (\mathbb{0}\_n, \mathbf{w}^\star) : \qquad \hat{V}(\mathbf{z}(t), \mathbf{w}(t)) > 0. \tag{5.152}$$

*Proof.* Vˆ (x, w) can be written in the form

$$
\hat{V}(x, w) = N \cdot H(x) + w^{\Upsilon} \hat{\Xi}(x),
\tag{5.153}
$$

where <sup>Ξ</sup><sup>ˆ</sup> (x) = coli{Ξi(x)}. Due to the fact that both <sup>V</sup><sup>ˆ</sup> (x, **<sup>0</sup>**r) <sup>≻</sup> <sup>0</sup> as well as <sup>V</sup><sup>ˆ</sup> (x, <sup>w</sup><sup>⋆</sup> ) ≻ 0 hold for all <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup> and <sup>J</sup>ˆw(x, <sup>w</sup>) is strictly convex, the conditions of Lemma 5.22 are fulfilled, which implies that

$$\lim\_{t \to \infty} \|w(t) - w^\star\|\_2 = 0. \tag{5.154}$$

Accordingly, from Lemma 5.23 it follows that <sup>V</sup><sup>ˆ</sup> (x(t), <sup>w</sup>(t)) <sup>≻</sup> <sup>0</sup> for all t > <sup>0</sup>.

$$\square$$

**Theorem 5.53.** *Consider the closed-loop system* (5.143) *starting at* (x0, **0**r)*. Let the conditions of Lemma 5.46 hold with* <sup>w</sup><sup>0</sup> ∈ M*. Then* <sup>x</sup> <sup>=</sup> **<sup>0</sup>**, <sup>w</sup> <sup>=</sup> <sup>w</sup><sup>⋆</sup> *is a locally asymptotically stable equilibrium of* (5.143)*.*

*Proof.* Define the shifted Lyapunov function candidate <sup>V</sup>e(x, <sup>w</sup>) = <sup>V</sup><sup>ˆ</sup> (x, <sup>w</sup> <sup>−</sup> <sup>w</sup><sup>⋆</sup> ). According to Lemma 5.52, <sup>V</sup>e(x, <sup>w</sup>) is positive definite and according to Lemma 5.50, ˙ <sup>V</sup>e(x, <sup>w</sup>) is negative definite. As such, <sup>V</sup>e(x, <sup>w</sup>) is a strict Lyapunov function for the equilibrium (**0**, <sup>w</sup><sup>⋆</sup> ) of (5.143), which is consequently a locally asymptotically stable equilibrium [Kha02, Theorem 4.1].

#### **Example 8 (Nonlinear Differential Game) –** *cont'd*

*Now the adaptation strategy* (5.143) *is applied to the nonlinear differential game* (5.101)*–* (5.102)*. As* R(x) *in* (5.102) *is positive definite, Lemma 5.42 states that* V <sup>0</sup>(x) = (H(x), H(x))<sup>T</sup> *is an admissible vector of CLFs.*

*The basis functions for the extended CLFs are chosen to* Ξ1(x) = Ξ2(x) = (x 2 1 , x1x2, x<sup>2</sup> 2 ) T*, which, after comparison of* (5.103) *and* (5.118)*, implies that the optimal weights are* w<sup>⋆</sup> <sup>1</sup> = (1.5, <sup>−</sup>2, <sup>0</sup>.5)<sup>T</sup> *and* <sup>w</sup><sup>⋆</sup> <sup>2</sup> = (0, 0, 0.5)<sup>T</sup>*.*

*The system is initialized at* <sup>x</sup><sup>0</sup> = (1, 1)<sup>T</sup>*,* <sup>w</sup><sup>0</sup> = (0, 0)<sup>T</sup> *and the shifts in* <sup>J</sup>w,i *(see* (5.128)*) are set to* <sup>c</sup>i,<sup>1</sup> = (0, 0)<sup>T</sup>*,* <sup>c</sup>i,<sup>2</sup> = (1, 0)<sup>T</sup>*,* <sup>c</sup>i,<sup>3</sup> = (0, 1)<sup>T</sup>*, and* <sup>c</sup>i,<sup>4</sup> = (−1, <sup>−</sup>1)<sup>T</sup>*. To investigate the behavior of the control system in case of an external excitation, we apply an additive disturbance input* <sup>d</sup> = (δ(<sup>t</sup> <sup>−</sup> 10), δ(<sup>t</sup> <sup>−</sup> 20))<sup>T</sup>*.*

*Fig. 5.10a shows the trajectories of* w1(t)*,* w2(t)*, and* Υ(t)*, if the learning rate* α *is set to* 0.5*, with the dashed lines indicating the optimal weights* w<sup>⋆</sup> 1 *and* w<sup>⋆</sup> 2 *, respectively. It can be seen that after* 2 s*,* x *has converged to its equilibrium value and both* Υ *as well as* w<sup>1</sup> *and* w<sup>2</sup> *have converged to their optimal values. Moreover, their values remain identical once the learning process is completed, even after the additive disturbance at* 10 s *and* 20 s*. Thus, it can be deduced that* V1(x, w1) *and* V2(x, w2) *converge to the respective value functions* V ⋆ 1 (x) *and* V ⋆ 2 (x)*, which means that the implemented control strategies* u ⋆ 1 (x) = − 1 S<sup>1</sup> · <sup>G</sup><sup>T</sup> 1 (x) ∂V<sup>1</sup> <sup>∂</sup><sup>x</sup> Υ<sup>1</sup> *and* u ⋆ 2 (x) = − 1 S<sup>2</sup> · <sup>G</sup><sup>T</sup> 2 (x) ∂V<sup>2</sup> <sup>∂</sup><sup>x</sup> Υ<sup>2</sup> *converge to feedback Nash strategies with regard to the original objective function* (5.70a)*.*

*Fig. 5.10b shows the trajectories, if* α *is reduced to* 0.05*. In this case, the equilibrium values of* w1*,* w2*, and* Υ *remain the same. However, convergence of* Υ*,* w1*, and* w<sup>2</sup> *is considerably slower, with a time to convergence of around* 20 s*. The overshoot of* Υ *is about ten times larger than in Fig. 5.10a. The disturbance at* t = 10 s *results in a slight displacement of* Υ*,* w1*, and* w2*, whereas there is virtually no displacement caused by the second disturbance at* t = 20 s*.* *This again shows that even after a disturbance,* w<sup>1</sup> *and* w<sup>2</sup> *remain at their optimal values once the learning process is completed.*

**Figure 5.10:** Adaptation results of the weights w and Lagrange multipliers Υ depending on the chosen learning rate α.

#### **5.3.7 Summary**

In this section, a novel approach for finding feedback Nash equilibria in N-player noncooperative differential games for input-state-output PHSs has been presented. The proposed design procedure is based on a modified differential game which yields Sontag-type explicit control laws for all players, provided that an admissible vector of CLFs can be found. First, necessary and sufficient conditions have been stated under which the Hamiltonian of the input-state-output PHS accomplishes this purpose. Then, in order to ensure the optimal controllers for the modified differential game to coincide with the optimal controllers for the original problem, a simple adaptation strategy based on gradient flow is applied which is proven to be asymptotically stabilizing and able to approximate the value functions of the individual players (cf. Theorem 5.53).

## **5.4 Discussion**

In this section, we restate the optimal balancing problem (4.37) from Subsection 4.4.2 and provide a conclusive statement about its explicit solution followed by a brief outline of prospective avenues of further research.

#### **5.4.1 Application to Balance of Payments Restoration**

Recall the optimal balancing problem from Subsection 4.4.2, yielding

$$\min\_{\mathbf{u}} \qquad \qquad \frac{1}{2} \int\_{0}^{t\_{\mathrm{f}}} \boldsymbol{x}\_{\sigma}^{\mathrm{T}} \underbrace{\begin{bmatrix} \boldsymbol{Q}\_{1} & \boldsymbol{0} \\ \mathbb{0}^{\mathsf{T}} & \boldsymbol{Q}\_{2} \end{bmatrix}}\_{\mathbf{Q}} \boldsymbol{x}\_{\sigma} + \boldsymbol{u}^{\mathsf{T}} \boldsymbol{S} \boldsymbol{u} \, \mathrm{d}t \tag{5.155a}$$

$$\text{subject to} \quad \dot{x}\_{\sigma} = (\mathbf{J}\_{\sigma} - \mathbf{R}\_{\sigma}) \frac{\partial H\_{\sigma}(x\_{\sigma})}{\partial x\_{\sigma}} + \mathbf{G}\_{\sigma}u. \tag{5.155b}$$

Note that <sup>Q</sup> <sup>≽</sup> <sup>0</sup> and <sup>S</sup> <sup>≻</sup> <sup>0</sup> hold by definition. Provided that the (mild) technical assumption of nontrivial zonal power injections pˆ<sup>g</sup> ̸= **0** holds, the following corollary gives the asymptotic stability of the resulting (closed-loop) equilibrium governed by the PHS dynamics (5.155b) and the adaptive optimal controller (5.55b), (5.55c).

**Corollary 5.54.** *Let* V ⋆ σ (xσ) = Hσ(xσ) + (w<sup>⋆</sup> ) <sup>T</sup>Ξ(xσ) *be the value function of optimization problem* (5.155) *and let the conditions of Lemmas 5.19 and 4.19 hold with* w<sup>0</sup> ∈ M*. Then* x<sup>σ</sup> = **0**, w = w<sup>⋆</sup> *is an asymptotically stable equilibrium of* (5.155b)*,* (5.55b)*,* (5.55c)*.*

*Proof.* If the conditions of Lemma 5.19 hold, then Theorem 5.24 can be applied to state that (5.155b), (5.55b), (5.55c) is at least (non-asymptotically) stable around (x<sup>σ</sup> = **0**, w = w<sup>⋆</sup> ). Moreover, if the conditions of Lemma 4.19 hold, then this lemma states that the (open-loop) PHS dynamics (5.155b) are globally asymptotically stable. Thus, asymptotic stability of the closed-loop system (5.155b), (5.55b), (5.55c) around (x<sup>σ</sup> = **0**, w = w<sup>⋆</sup> ) follows by Theorem 5.24.

#### **5.4.2 Possible Extensions**

The novel methodology for ADP-based solutions of optimal control problems and differential games with PHS dynamics is founded on the "Lyapunov-like" properties of the Hamiltonian H(x), which allow to obviate some common challenges in ADP literature, such as the need for initial stabilizing weights. However, for didactical reasons, a number of simplifying yet not necessarily essential assumptions are made in the context of this chapter. In the following we thus briefly discuss worthwhile extensions and generalizations of the previous findings.

#### **Extension to Weak CLFs**

A remarkable insight following from Corollary 5.54 is that the CLF property of H(x) is *not* a necessary condition for asymptotic stability of the closed-loop system. In particular, although Hσ(xσ) is only a *weak CLF* <sup>70</sup>, the special structure of Hσ(xσ) ensures that the largest invariant set contained in H˙ <sup>σ</sup>(xσ) is still equal to the origin, such that asymptotic stability can be addressed by invoking LaSalles invariance principle. In general, note that H(x) being a weak CLF is also sufficient for non-asymptotic stability of the closed-loop system for both the single- and multi-player case<sup>71</sup>. However, for asymptotic stability, it always has to be investigated "by hand" whether the largest invariant set contained in {<sup>x</sup> ̸<sup>=</sup> **<sup>0</sup>** : <sup>V</sup>˙ (x, <sup>w</sup>) = 0} is equal to the origin.

#### **Extension to Passive Input-Output Systems**

Although all considerations of this chapter focus on optimization problems in which system dynamics can be described by an input-state-output PHS, they can seamlessly be extended to the more general system class of *passive input-output systems* (cf. Definition A.15 in Appendix A.4). In particular, all stability statements made so far still hold as long as the storage function V (x) of the passive input-output system has the property of a (weak) CLF (cf. Definition 5.26 and Definition 5.30). A very practical benefit of PHSs, however, is that the Hamiltonian obtained there as a direct outcome of the energy-based modeling procedure always reflects the actual energy stored in the system. Therefore, H(x) highly qualifies as a natural CLF candidate, whereas in (generic) passive input-output systems a suitable storage function has to be found artificially by means of educated guessing and thus often has no intrinsic physical interpretability.

#### **Extension of Gradient Flow**

As stated in Remark 3.28, the gradient flows in the adaptation schemes in (5.55c), (5.131) can alternatively be replaced by the Newton gradient flows

$$
\dot{\boldsymbol{w}}\_{i} = -\boldsymbol{\alpha} \cdot \left(\frac{\partial^2 \mathcal{J}\_{\mathbf{w},i}}{\partial \mathbf{w}\_i^2}\right)^{-1} \frac{\partial \mathcal{J}\_{\mathbf{w},i}}{\partial \boldsymbol{w}\_i}, \qquad \qquad \boldsymbol{\alpha} > 0. \tag{5.156}
$$

If ill-conditioning of the Hessian in (5.156) hinders the numerical calculation of its inverse, there is a broad literature on alternative formulations of the Newton descent direction, such as *regularized Newton's method* [Pol09b] or the pseudo-inverse formulation [GK65]

$$\dot{\boldsymbol{w}}\_{i} = -\boldsymbol{\alpha} \cdot \left(\frac{\partial^{2} \mathcal{J}\_{\mathbf{w},i}}{\partial \mathbf{w}\_{i}^{2}}\right)^{\dagger} \frac{\partial \mathcal{J}\_{\mathbf{w},i}}{\partial \boldsymbol{w}\_{i}}, \qquad \qquad \boldsymbol{\alpha} > 0. \tag{5.157}$$

<sup>70</sup> H(x) is said to be a weak CLF if the requirement "< 0" in (5.6) is relaxed to "≤ 0".

<sup>71</sup> Provided that the conditions of Lemma 5.19 (or Lemma 5.46 for the N-player case) with w<sup>0</sup> ∈ M continue to apply.

Besides the standard least-squares pseudoinverse (5.157), there exists a large number of advanced approaches based on singular value decomposition such as truncated pseudoinverse or damped least-squares pseudoinverse, see [MCA95] for a discussion of alternative formulations.

# **6 Case Studies**

The aim of this chapter is to demonstrate the capability of the overall control scheme shown in Fig. 4.7 in its entirety. Section 6.1 introduces the employed benchmark simulation model, which is based on the IEEE 57-bus system. Section 6.2 demonstrates the performance of the cell-based frequency and voltage controller from Chapter 3 using a three-cell scenario and showcases the benefits of the WoC scheme compared to an islanded MG operation (*Case Study I*). Section 6.3 evaluates the efficiency of the overall control scheme by means of a ten-cell scenario (*Case Study II*). The latter Case Study Incorporates the frequency and voltage controller from Chapter 3, the price-based controllers for congestion management and BoP containment from Chapter 4, and the BoP restoration strategy developed in Chapter 5. Section 6.4 draws a conclusive statement about the main findings of both case studies.

## **6.1 Benchmark Model Setup**

The simulation is carried out on a modified IEEE 57-bus system depicted in Fig. 6.1. As part of the *IEEE power system test case* benchmark systems [IEE21] as well as the MATPOWER package [ZMST11], the IEEE 57-bus system was originally derived from a segment of a realworld power system localized in the midwestern US. It has already gained notable popularity and application in numerous works from different research directions. For instance, [JW16] employs the IEEE 57-bus test case to test a probabilistic OPF algorithm considering both uncertain infeed and consumption. In [ZC06], the same test system is used to demonstrate the effectiveness and performance of a new interline power flow controller approach. In [KPS15], a price-based optimization algorithm which minimizes the total cost of active power generation by minimizing the sum of active power losses in the network is tested on the IEEE 57-bus system.

Due to its complex structure with 80 power lines, a high mesh degree<sup>72</sup> and widely diverse line admittances, the IEEE 57-bus system is supposed to be a pertinent basis for the purpose of thoroughly validating the controller concepts derived in this dissertation. Yet, we have conducted the following modifications and amendments to the original model [IEE21] in order to provide a more insightful playground with regard to all possible network connectors and participants established in sections 3.2 and 3.3:

• Originally equipped with only 7 SGs and 42 loads, we have connected 38 controllable sources to the network (19 each of S and I type) as well as one (uncontrollable) load (pℓ,i , qℓ,i) per bus i ∈ V, modeling the connection to local low-voltage feeders. The

<sup>72</sup> The associated graph GIEEE has an average degree of 2.81.

**Figure 6.1:** Stylized representation of the IEEE 57-bus system [IEE21]. S nodes are depicted in black, I nodes are depicted in gray, and L nodes are depicted in white. Each bus is equipped with an uncontrollable load modelled by (uncontrollable) active and reactive power consumption.


**Table 6.1:** Node parameters.

assignment of buses to node types is illustrated in Fig. 6.1. A detailed list is provided by Table E.1 in Appendix E.1.

• Since the node parameters are not specified in [IEE21], they are chosen to be randomly distributed within the specific intervals shown in Table 6.1. The parameter values for S nodes, L nodes and transmission lines are based on those provided in [TBD16, KBKH19]. The parameter values of <sup>I</sup> nodes base upon [MDS<sup>+</sup>18]. A full list of all numerical values is given in Table E.1 in Appendix E.1.

Throughout this chapter, all of the following numerical values are given in p.u. with Ubase = 135 kV, Sbase = 100 MVA, and Cbase = 1 MU/Sbase.

All simulations conducted in this chapter are implemented using the software system Wolfram Mathematica (Version 12.0.0) with the numerical solver NDSolve. In general, the program components developed in the context of this dissertation exhibit a high degree of modularity, so that changes to the plant model and variations or partial replacements of the employed controllers are straightforward. This enables to conveniently select and evaluate a wide variety of combinations of network topologies, parameter ranges, and control strategies.

The numerical computations are conducted on a machine with an Intel Core i7-6600U and 12 GB of RAM.

# **6.2 Case Study I: Three Cells**

The objective of the first Case Study Is to investigate the performance of the frequency and voltage controllers developed in Chapter 3 and to evaluate the economic efficiency of the obtained results by examining different cell topologies.

#### **6.2.1 Overview and Stimulation Signal**

The IEEE 57-bus system is divided into three cells which are each interconnected via multiple boundary lines (see Fig. 6.2). Let DIEEE denote the incidence matrix of the IEEE 57-bus system and D<sup>3</sup> IEEE denote the respective incidence matrix if all |E<sup>ˆ</sup> <sup>p</sup>| = 8 boundary edges of adjacent cells (colored red in Fig. 6.2) are removed. In the following, we investigate the role of connectivity of G<sup>p</sup> and G<sup>c</sup> as well as the influence of κ by examining the following four different topological scenarios:

**Figure 6.2:** IEEE 57-bus system divided into three cells.


**Table 6.2:** List of scenarios elaborated in Chapter 6 (CM = congestion management).


A comparison chart summarizing the different scenarios of Case Study I is provided in the upper half of Table 6.2.

#### **Controller Parameters**

The controller parameters in τ <sup>g</sup> from (3.110a) are set to 0.1, while the controller parameters in τ (·) from (3.110b)–(3.110k) are set to 0.01. The lower and upper bounds of pg,i are set to p <sup>g</sup>,i = −0.002 and pg,i = 0.003 respectively, and the voltage limits are set to U<sup>i</sup> = 0.98 and U<sup>i</sup> = 1.02.

#### **Initialization**

At t = 0, the system is in synchronous mode with L = **0**. All controller variables p<sup>g</sup> , ν, λ, µ(·) are initialized to zero. Without loss of generality, we set W ≡ Z and choose the cost functions to

$$\mathbf{C}\_{\pi}(\mathbf{p}\_{\pi}) = \frac{1}{2} \cdot \left( \sum\_{i \in \mathcal{V}^{p}, \pi} \frac{1}{\varpi\_{i}} \cdot p\_{\mathbf{g},i}^{2} \right), \qquad \qquad \qquad \pi \in \mathcal{W}, \tag{6.1}$$

where ϖ<sup>i</sup> = 1 + 0.04 ·(i−1). The initial values of the voltage angle deviations ϑij are chosen to be randomly distributed within the interval [−0.04, 0.014] and all voltages U<sup>i</sup> are chosen to be randomly distributed within the interval [0.98, 1.02].

#### **Stimulation Signal**

The active and reactive power consumptions p<sup>ℓ</sup> and q<sup>ℓ</sup> are set piecewise constant in order to simulate a step change in generation or consumption in specific areas of the network. To this end, the initial values of p<sup>ℓ</sup> and q<sup>ℓ</sup> are chosen such that the AC power flow equations (3.8) are satisfied. Subsequently, step-wise load changes are applied at equal intervals of 5 min at the six L nodes 12, 13, 20, 27, 28, and 43. The selection of these nodes was made to ideally cover a wide range of load concentrations and thus to provoke diverse and reverted power flows. Nodes 12 and 13 as well as 27 and 28 belong to the same cell and are direct neighbors. Bus 20 also belongs to the same cell as 12 and 13, but is no direct neighbor of the latter. Bus 43 belongs to cell 2 and is situated more remotely. A total simulated time of t = 30 min is chosen, which incorporates the following five load jumps:


A graphical representation of the resulting active and reactive power consumptions is given in Fig. 6.3.

**Figure 6.3:** Step-wise jumps in power consumption (p<sup>ℓ</sup> , q<sup>ℓ</sup> ) for Case Study I. The thin gray lines indicate the (constant) power consumption at all other nodes.

#### **6.2.2 Numerical Results**

Figs. 6.4 and 6.6 show the resulting nodal prices λ, active power generations p<sup>g</sup> , nodal frequencies f and voltage magnitudes U for Scenarios I-a)–I-d), respectively.

In each of the four scenarios, no nodal price differences within each cell k ∈ {1, 2, 3} can be detected. A close-up plot of the nodal prices for Scenario I-a) after t<sup>1</sup> and t<sup>4</sup> (given in Fig. 6.5) shows that the nodal prices converge to a common (cell-specific) price Λ<sup>k</sup> in less than 10 s.

#### **Prices and Active Power Generation**

**Scenario I-a) (islanded cells)** It can be seen in the upper panel of Fig. 6.4a that due to physical decoupling, the loads steps in cell 3 at t1, t2, and t<sup>3</sup> only affect the price in cell 3 itself, while prices in other cells remain unaffected. Likewise, the load jump in cells 1 and 2 at t<sup>5</sup> only affects the prices in cells 1 and 2 itself, while the price in cell 3 retains its initial value. With regard to the active power generations (see the upper panel of Fig. 6.4b), it can be seen that the upper and lower limits p<sup>g</sup> and p<sup>g</sup> are respected except after t<sup>3</sup> in cell 3 and after t<sup>5</sup> in cell 2, where we see a significant oscillation of all generations around the lower limit. Moreover, the active power generations in cell 1 do not converge within the interval between t<sup>4</sup> and t5.

**Scenario I-b) (free zonal prices)** The resulting cell-specific prices shown in the second upper panel in Fig. 6.4a predominantly follow a similar course as in Scenario I-a). In particular, the load jumps in cell 3 at t1, t2, and t<sup>3</sup> lead to an increase in the prices in cell 3. Likewise, the load jumps in cells 1 and 2 at t<sup>4</sup> lead to an increase in the prices in cells 1 and 2. However, in contrast to Scenario I-a), load jumps in one cell also affect the prices in the other cells, signifying the effect of physical interconnection via inter-cell lines. Moreover, the spread between zonal prices is lower than in Scenario I-a). Similar to the price curves, the resulting active power generations in different cells are affected by each other and exhibit a lower

**Figure 6.4:** Nodal prices and active power generation for Scenarios I-a)–I-d).

**Figure 6.5:** Synchronization of nodal prices towards a zonal price.

spread than in Scenario I-a) (see the second upper panel in Fig. 6.4b). This illustrates that the other cells contribute to eliminating the imbalance in a specific cell.

**Scenario I-c) (uniform pricing)** The second lower panel in Fig. 6.4a reveals that in this scenario, all prices quickly synchronize to a common value after each load jump. Accordingly, load jumps in one specific cell impact the active power generations, nodal frequencies, and voltage magnitues in all cells. This corresponds to the individual active power generations being closer to each other, barely reaching their specific upper or lower limits, see the second lower panel in Fig. 6.4b.

**Scenario I-d) (constant participation factors)** As depicted in the lower panel of Fig. 6.4a, the cell-specific prices Λ<sup>k</sup> converge to distinct steady-state values, which invariably follow a ratio of 1:3:9 regardless of their absolute values. This is in line with the specified participation factors κ = (1, 3, 9)<sup>T</sup>. Accordingly, the individual share of active power generation for each cell (see the lower panel in Fig. 6.4b) is distributed proportially to *both* the weighting factors ϖ<sup>i</sup> in (6.1) *and* the cell-specific participation factors κ, while always respecting the specified constraints (p<sup>g</sup> , p<sup>g</sup> ) on active power generation as observed in the previous two scenarios. For the formation of prices (and thus the resulting active power generations), it is irrelevant in which region of the network the specific load jump occurs.

#### **Frequency and Voltage Control**

The resulting frequency and voltage curves show significant differences between Scenario I-a) and the other scenarios (see Fig. 6.6). As evident in the upper panel of Fig. 6.6a, in Scenario I-a), each load jump causes temporary deviations of the nodal frequencies in the involved cells, which are subsequently regulated back to their nominal value. The maximum overshoot during the entire simulated time is +40 mHz. After the steps at t<sup>4</sup> and t5, the performance of the frequency regulation in cell 1 is poor. By contrast, frequency regulation is maintained

**Figure 6.6:** Nodal frequencies and voltage magnitudes for Scenarios I-a)–I-d).

without exception for Scenarios I-b)–I-d), with maximum overshoots of +50 mHz (see the other subplots in Fig. 6.6a).

Fig. 6.6b shows the resulting voltage magnitudes in all cells. In Scenario I-a) (upper panel in Fig. 6.6b), all upper and lower limits are met except after t5, where most of the voltage magnitudes in cell 2 oscillate around their upper or lower limits. As with nodal frequencies, the issues with voltage magnitudes are limited to Scenario I-a), while in the other scenarios voltage regulation is consistently provided, i.e. voltage magnitudes are kept within their respective limits without any oscillations and/or constraint violations.

#### **6.2.3 Summary of Case Study I**

The presented case study demonstrates the general applicability of the proposed control scheme for the automatic regulation of frequency and voltage by means of price signals. However, the detailed results for Scenario I-a) show that WPs in cell 2 are unable to autonomously meet the local active and reactive power demand without violating the constraints (3.83b)– (3.83c). This result exemplifies that the operation as autonomous MGs creates *unnnecessarily reduced robustness*, unless each MG is over-equipped with instantaneous generation capacity at any point in time based on peak demand considerations. As discussed at the beginning of Subsection 2.2.1, the latter would be highly cost-ineffective. It can be seen in Fig. 6.4a that with islanded MG operation, price jumps only occur in cells where the load changes actually occur, and that these price jumps are far more severe than in the other scenarios. On the contrary, the WoC-based infrastructure with a physically interconnected overall network helps to indirectly couple producers and consumers from different regions and thus to increase the robustness of the grid against sudden load changes. Interestingly, deploying the new WoC infrastructure comes at almost no additional fixed cost compared to autonomous operation, since the (few) additional coupling lines necessary for the interconnection of cells are either already in place or can be retrofitted between nearby buses with little effort.

The cell-specific ratios of the free zonal prices resulting in Scenario I-b) provide a mixture of the results in islanded MG operation (Scenario I-a)) and those in uniform pricing (Scenario I-c)), but, as discussed in Remark 3.34, do not yet follow an overarching goal. Furthermore, the results of Scenario I-d) stress that in general, any given relationships between zonal prices can be specified via κ. In the subsequent Case Study II, such "non-academic" adjustments of κ as discussed in Chapter 4, will be examined in detail.

## **6.3 Case Study II: Ten Cells**

This case study considers the full control scheme as proposed in this dissertation, with zonal pricing through κ(t) along with frequency and voltage control. Special focus is put on real-time congestion management as well as on the attainment of BoP.

**Figure 6.7:** IEEE 57-bus system divided into ten cells.

#### **6.3.1 Overview and Stimulation Signal**

The modified IEEE 57-bus system is now divided into ten cells as illustrated in Fig. 6.7 whereby the assignment of buses to the node types and all physical parameters (see Table E.1) are left identical to Case Study I.

The communication networks represented by D<sup>c</sup> and Dˆ <sup>c</sup> are chosen identically<sup>73</sup> to the physical interconnection structure, i.e. D<sup>c</sup> = D<sup>p</sup> and Dˆ <sup>c</sup> = Dˆ <sup>p</sup>. We examine the following two different real-time pricing scenarios:


A comparison chart summarizing the scenarios of Case Study II is provided in the lower half of Table 6.2 in Subsection 6.2.1.

#### **Controller Parameters**

To ensure sufficient timescale separation (Fig. 4.7), the controller parameters are chosen as follows: The frequency and voltage controller (3.110) is given the fastest response times by setting all parameters τ(·) in (3.110) to 0.01. For the "higher-level" balancing (4.29) and pricing distribution controllers (4.11), we choose the components in τ <sup>a</sup> and τ <sup>ϕ</sup> to 0.1, and the components in τ <sup>ν</sup><sup>a</sup> to 10. Finally, we set the components in τ ℧ and τ <sup>U</sup>set to 100 for congestion controller (4.14) without integral action, and in τ − ℧ and τ + ℧ to 1000 for congestion controller (4.15) with integral action.

The lower and upper bounds of pg,i are set to p <sup>g</sup>,i = 0 and pg,i = 0.5 respectively, and the voltage limits are set to U<sup>i</sup> = 0.98 and U<sup>i</sup> = 1.02. The maximum admissible voltage setpoint deviation is chosen to U ∆ = 0.015. For both congestion controllers (4.14) and (4.15), the thresholds C min <sup>m</sup> , <sup>m</sup> <sup>∈</sup> <sup>E</sup><sup>ˆ</sup> <sup>p</sup> are chosen as C min <sup>m</sup> = 0.8. The maximum permissible power flow of line 35 → 35 is set to S max <sup>35</sup>→<sup>36</sup> = 0.2. For the other lines, it is set to S max <sup>i</sup>→<sup>j</sup> = 0.1. As state and controller weight matrices for BoP restoration, we choose Q<sup>1</sup> = 4 · I, Q<sup>2</sup> = 20, and S = 4 · I,

<sup>73</sup> As shown in further simulative studies in [KBKH19], the concrete choice of D<sup>c</sup> or Dˆ <sup>c</sup> has no influence on the resulting steady state, yet only effects on the transient behavior, provided that the respective communication graphs G<sup>c</sup> or Gˆ<sup>c</sup> are connected.


**Table 6.3:** Step-wise load changes for Case Study II.

respectively. The basis functions for BoP restoration are chosen to

$$\Xi(x\_{\sigma}) = \begin{bmatrix} (\epsilon\_1^{\kappa})^2 \\ \vdots \\ (\epsilon\_{10}^{\kappa})^2 \\ \sigma^2 \\ \prod\_{k=1}^{10} \epsilon\_k^{\kappa} \\ \sigma \cdot \left( \prod\_{k=1}^{10} \epsilon\_k^{\kappa} \right) \end{bmatrix}, \tag{6.2}$$

and as objective function for the adaptation (5.55c), we use

$$\mathcal{J}\_{\mathbf{w}}(x\_{\sigma}, w) = (\mathfrak{Q}(x\_{\sigma}, w))^2 + \sum\_{k=1}^{N} (\mathfrak{Q}(x\_{\sigma} + \mathbf{e}\_k, w))^2 + \sum\_{k=1}^{N} (\mathfrak{Q}(x\_{\sigma} - \mathbf{e}\_k, w))^2 \tag{6.3}$$

with a learning rate of α = 0.5.

#### **Initialization and Stimulation Signal**

The initial values of all controller states are set to 0, except for Uˆ set(0) = <sup>1</sup>. We set <sup>W</sup> <sup>=</sup> <sup>Z</sup> and choose the cost functions to

$$\mathbf{C}\_{\pi}(\mathbf{p}\_{\mathbf{g},\pi}) = \frac{1}{2} \left( \sum\_{i \in \mathcal{V}\_{\mathcal{P},\pi}} \frac{1}{\varpi\_{i}} p\_{\mathbf{g},i}^{2} \right), \qquad \qquad \qquad \qquad \pi \in \mathcal{W}, \tag{6.4}$$

where ϖ = 1 + 0.01 · (i − 1).

The external disturbance is given by step-wise load jumps at the three nodes 1, 28, 35 from different zones in the network as listed in Table 6.3 and illustrated in Fig. 6.8. In order to investigate the path dependence of the resulting solutions, the load jumps occurring at t1, t<sup>2</sup> and t<sup>3</sup> are compensated by reversed load jumps at t4, t5, and t<sup>6</sup> (in descending order with respect to the order of occurence).

**Figure 6.8:** Step-wise jumps in power consumption (p<sup>ℓ</sup> , q<sup>ℓ</sup> ) for Case Study II. The thin gray lines indicate the (constant) power consumption at all other nodes.

#### **6.3.2 Numerical Results**

Triggered by the step-wise load jumps in p<sup>ℓ</sup> and q<sup>ℓ</sup> , respectively, all voltage magnitudes, active power generations, and nodal prices attain a new steady state during each of the 15 min intervals (see Fig. 6.9). Notably, the jumps at t<sup>1</sup> and t<sup>5</sup> as well as those at t<sup>2</sup> and t<sup>3</sup> yield *exactly the same* steady state for each of the quantities shown in Fig. 6.9. All nodal frequencies converge to the nominal frequency 50 Hz within 10 min and exhibit a maximum deviation of −38 mHz after the step at t6. These overshoots of nodal frequencies tend to be higher in amplitude if the load jump is closer to the respective node. E.g. after t1, the load jump at node 1 in cell 1 causes a much more significant transient frequency deviation for the nodes in cell 1 than for those in the other cells. Furthermore, all specified voltage limits are maintained during the entire simulation.

Similarly to Case Study I, the nodal prices λ<sup>i</sup> show a very fast convergence of less than one second towards a zonal price Λk. At steady state, the cell-specific active power generations are equidistant to each other, while their absolute values depend on all current active and reactive power consumptions. These results indicate that active power sharing (cf. Definition 3.38) is given across the entire network. whereby the weighting factors ϖ<sup>i</sup> in (6.4) are identical to those in (3.116).

#### **Congestion Management**

Fig. 6.10 shows the resulting apparent power flows over all boundary lines <sup>m</sup> <sup>∈</sup> <sup>E</sup><sup>ˆ</sup> <sup>p</sup>. It can be clearly seen that in both scenarios all power flow limits S max <sup>m</sup> are respected at steady state. In Scenario II-a) (see Fig. 6.10a), there is no transient violation of any of the line flow limits. The participation factors κ synchronize within 2 min after initialization to a common value of less than 0.87 (see Fig. 6.11a). Subsequently, with the apparent power of line 38 → 48 approaching its specified maximum absolute value, the participation factors split over a range of 0.1, where the zonal pricing factor in cell 7 takes the highest value and the zonal pricing

**Figure 6.9:** Nodal frequencies, voltage magnitudes, active power generation, and nodal prices.

**Figure 6.10:** Apparent power flows over inter-cell lines Eˆp.

**Figure 6.11:** Participation factors κ.

factors in cells 8 and 10 take the lowest value. After the second jump at t2, the line between node 35 in cell 2 and node 36 in cell 4 reaches its limit of S max <sup>35</sup>→<sup>36</sup> = 0.2. Subsequently, <sup>κ</sup><sup>2</sup> increases significantly, while κ<sup>4</sup> decreases. After the third jump at t3, also the power line between node 21 in cell 10 and node 22 in cell 8 reaches its limit. At this point κ<sup>2</sup> slightly recedes, while κ<sup>10</sup> increases. After the occurence of the opposite jumps at t<sup>4</sup> and t5, the former steady states are recovered, before all participation factors κ converge to their initial value 0.87 after the last jump at t6.

Fig. 6.11a reveals that in Scenario II-a), the individual participation factors are overlaid by a ripple. By contrast, the individual participation factors in Scenario II-b) resulting from the congestion controller with integral action move more slowly and show almost no ripple during the transient process, but exhibit higher transient overshoots (see Fig. 6.11b). Accordingly, the resulting apparent power flows in Fig. 6.10b exhibit temporary constraint violations of a maximum duration of 10 min. Remarkably, the steady-state apparent power flows S<sup>38</sup>→<sup>48</sup>, S<sup>35</sup>→<sup>36</sup>, and S<sup>21</sup>→<sup>22</sup> of Scenario II-b) are exactly at their specified maximum limits, while in

**Figure 6.12:** BoP containment and restoration.

Scenario II-a) they only approach these respective limits. This can be elucidated by the fact that the barrier functions ℧<sup>m</sup> used for the zonal pricing controller for congestion management (4.14) without integral action do not allow for an exact convergence to the threshold C min m (and thus |Sm| = S max <sup>m</sup> ), while for the congestion controller (4.15) with integral action, the barrier function ℧<sup>m</sup> only requires that the *integrated constraint violation* has to be finite.

#### **Balancing of Payments**

The trajectories of σ indicating the BoP error are depicted in the upper panels of Fig. 6.12. In Scenario II-a), σ settles to a new steady-state value between −1.18 and −1.43 after each load jump. The resulting bias of σ depends on the magnitude of the applied load jump, although there is no clear relationship between the sign of the load jump and its impact on σ. However, BoP containment is given, which is shown by the bottom panel in Figure 6.12a. Although the associated time derived BoP deviations σ˙ (see ibid.) never exceed 0.016, their accumulation over a period of several minutes always causes a net deficit of more than 1. Thus, with respect to the sign convention in (4.34), we see that the overall payments of RPs exceed the WPs' overall revenues, thereby resulting in an unwanted accumulation of capital. By contrast, in Scenario II-b), σ recovers the desired steady-state value of σ <sup>⋆</sup> = 0 no later than 15 min after each load jump, thus BoP restoration (and hence of course also BoP containment) is accomplished.

**Figure 6.13:** Learning-based optimal controller for BoP restoration in Scenario II-b).

Fig. 6.13 shows the progression of the Lagrange multiplier Υ(t) of the employed adaptive optimal controller from Chapter 5 as well as the resulting weighting factors w(t) of the adaptation procedure (5.55c). The weighting factors w1, . . . , w<sup>10</sup> associated to the basis functions Ξ<sup>1</sup> = (ϵ κ 1 ) 2 to Ξ<sup>10</sup> = (ϵ κ <sup>10</sup>) 2 converge to the steady-state value w ⋆ <sup>1</sup> = · · · = w ⋆ <sup>10</sup> = 0.964 within 5 s after starting the simulation, and w<sup>11</sup> belonging to Ξ<sup>11</sup> = σ 2 converges to w ⋆ <sup>11</sup> = 9.4999. The weighting factors associated to the remaining basis functions converge to zero. After the adaptation is accomplished, the value of w(t) is no longer affected by subsequent load jump. Yet, at the same time, Υ(t) is slightly affected by each load jump, while always converging to a new steady-state value within 10 s which stays in a corridor between 0.978 and 0.985. These results indicate that the underlying HJB equation (5.24) is approximately (but not exactly) fulfilled and thus the learned, extended CLF V (x) = Hσ(xσ) + w<sup>⋆</sup>Ξ(xσ) is a good approximation of the value function. Occasionally, after Υ(t) has reached a steady state, the graph of Υ(t) reveals steep-flanked spikes of Υ(t) with a magnitude up to 0.12 from time to time. This is caused by a near-zero denominator in (5.55b), leading to a division by (almost) zero.

#### **Zonal Pricing**

Fig. 6.14 compares the resulting (cell-specific) wholesale prices λ(t) and the (cell-independent) retail price ΛR(t). Similar to Case Study I, nodal prices λ in each cell k ∈ Z converge to a cell-specific wholesale price Λ<sup>k</sup> within 1 s, such that no differences between nodes can be observed in Fig. 6.14. The retail price increases with increasing total demand at the L nodes, and vice versa. In line with the trajectories of κ(t) in Fig. 6.11b, there are *identical* wholesale prices, if none of the power flows along the boundary lines reaches its limit, and different wholesale prices, if at least one of the power lines reaches its limit (see Fig. 6.10). Interestingly, the retail price is hardly affected in cases where Λ = col<sup>k</sup>∈Z {Λk} is after a line flow limit has been reached, such as after 19 min through the congestion of line 38 → 48 or after 37 min triggered by line 35 → 36. Most of the time, each of the wholesale prices is *below* the retail price. This is caused by the BoP containment, which enforces the RPs to collectively compensate for the momentary nonzero resistive losses in addition to their own

**Figure 6.14:** Comparison between wholesale prices λ and retail price ΛR.

**Figure 6.15:** Apparent power flows and corresponding wholesale and retail prices in case of relaxed power flow limits for Scenario II-a).

power consumption. However, after the load jumps at t<sup>3</sup> and t5, the spread between wholesale prices is high enough that wholesale price Λ<sup>2</sup> of cell 2 even exceeds the retail price. This can be explained by the fact that cell 2 is located directly at the *critical line* 35 → 36, thus the zonal pricing generates a heavy increase in κ<sup>2</sup> (and hence Λ2) in order to combat congestion locally by means of incentivizing increased infeed by cell 2.

For comparison, Fig. 6.15 shows the resulting apparent power flows and price curves in case the congestion controller of Scenario II-b) is deactivated. In this case, there is a significant overload of the lines 38 → 48, 35 → 36, and 21 → 22 (see Fig. 6.15a) by up to 18%. Accordingly, all wholesale prices are equal and always lie below the retail price, as can be seen in Fig. 6.15b.

#### **6.3.3 Summary of Case Study II**

The zonal pricing controller used for congestion management in Case Study II causes a demandoriented, incentive-driven shift of power generation to regions that are "more favorable" from

a global grid perspective, whenever the purely merit-order-based network operation (with uniform prices) is no longer feasible. Compared to the alternative mechanism of cost-based redispatch, however, two major improvements are evident here: First of all, congestion is not alleviated "unidimensionally" by simply affecting two participants pairwise. Instead, a convenient and automatic adjustment of *all* cells is executed, as can be seen e.g. in Fig. 6.11 after the jump at t1. Secondly, the presented congestion controller is able to handle multiple congested lines at the same time, as e.g. demonstrated in Fig. 6.10 after the second and third jump. By contrast, cost-based redispatch strategies quickly become challenging in case of multiple congested lines due to the possibly very large number of combinations of increase/decrease pairs.

Case Study II also shows that the selected pricing strategy has only a minor influence on the retail prices, as can be seen by comparison between Λ<sup>R</sup> in Fig. 6.14 and Λ<sup>R</sup> in Fig. 6.15b. The reason for this is the appropriate settlement of *regularized* wholesale and retail prices through distributed balancing. In combination with the pricing strategy, it provides for additional competition between WPs, since, loosely speaking, an increasing κ<sup>k</sup> in one cell is automatically compensated by a decreasing κ<sup>k</sup> in some other cell. From this perspective, the automatic settlement of zonal price differences resulting from congested lines can be interpreted as a *dynamic rearrangement of the (copper plate-based) merit order, triggered by network utility considerations.*

While the presented schemes for frequency and voltage control, congestion management, and BoP containment are fully distributed, the developed adaptive optimal controller for BoP restoration is still operated in a centralized manner, since both the complete system dynamics as well as all inputs have to be known. Especially, if the state vector of the underlying system is very large, this considerably hinders the determination of suitable basis functions. Although the trajectories of Υ(t) and w(t) can be employed to qualitatively evaluate whether or not the value function candidate V (x, w) obtained with a *given* set of basis functions is a sufficiently good approximation of the value function, so far, there exists no constructive approach how to reasonably choose the basis functions. Therefore, further research is advisable on how to usefully incorporate existing a priori model knowledge into the choice of basis functions.

## **6.4 Discussion**

In this chapter, the main capabilities of the developed dynamic zonal pricing controller with combined frequency and voltage regulation have been systematically investigated using an IEEE benchmark model. Table 6.4 summarizes the respective control objectives substantiated in each of the simulation scenarios. Case Study I in Subsection 6.2 showcases the benefits of cellbased approaches for frequency and voltage control of large-scale systems. This is in addition to Example 4, which already demonstrates the integrated price-based frequency controller in its main features incorporating the classical primary, secondary, and tertiary control layers. Finally, Case Study II in Subsection 6.3 demonstrates the effectiveness of incentive-based dynamic pricing using participation factors for real-time control of line congestion. This unification of the classical aspects of power system stabilization and incentive-compatible, real-time dispatch represents a new, holistic framework for assigning a concrete, monetary


**Table 6.4:** Scenario chart for Case Studies I and II based on and in supplement to the literature comparison chart in Table 2.2. For abbreviations not listed, see ibid.

**Table 6.5:** Comparison of simulation times TCPU.


value to *grid support* in addition to a pure remuneration for power provision itself—something that has not been encountered in the literature so far.

Table 6.5 lists the computation times TCPU required for each of the simulated scenarios. It is evident that all computation times are far below the simulated time Tsim in each case, which shows the *real-time capability* of the overall control scheme. The individual durations shown in Table 6.5 along with further observations from the accompanying simulation experiments suggest that the simulation time is highly correlated with the number of active constraints. This correlation is plausible taking into account that the NDSolve routine, which is specialized for solving differential-algebraic equations, is a *variable step* solver which tends to a reduction of the step sizes (and consequently an increased computation time tCPU) whenever the simulated system becomes more stressed due to higher fluctuations of the state variables close to the active constraints. However, since the proposed control framework is distributed, in real-time applications, separate controllers with significantly lower computational complexity along with fixed-step solvers will be operated in parallel at every single node, thereby resulting in significantly lower computation times.

<sup>♠</sup> BoP: <sup>=</sup> BoP restoration, G# <sup>=</sup> BoP containment, <sup>−</sup> <sup>=</sup> not provided.

# **7 Conclusion and Outlook**

# **7.1 Conclusion**

Modern power systems exhibit a remarkable paradox: On one hand, increasing RES-based intermittent power generation requires close cooperation among all resources for efficiency and synergy reasons, in order to incorporate a "physical awareness" among all network participants about what actions are deemed most ideal from a global network perspective. On the other hand, the unbundling principle along with antitrust considerations impose, for good reason, that the exchange of information both between system operators and competitive prosumers, as well as between prosumers themselves, should be kept to a minimum.

The dissertation at hand provides a contribution towards bridging this gap by proposing an incentive-compatible, market-integrated control scheme with cell-based price consensus. Relying on and complying with the classical division into regulated system operators and profit-driven prosumers, the proposed brownfield framework allows each self-serving WP to autonomously make decisions in order to maximize its profit. In contrast to a copper plate-like market which does not consider the location of buyers and sellers [RS19], in the proposed framework any information about the temporal and spatial network state is indicated by the node distinct real-time price set by the local CCs which act as the network's neutral market facilitators. Inefficient allocations and normative add-on mechanisms such as cost-based redispatch or the BRP role, which are both critically discussed in Subsection 2.1.2, can thus be omitted.

Since all power system stability issues and physical constraints are already settled at market clearing, there is no further need of a capacity-based BM as compared to the baseline scenario. Thus, potential strategic market behavior as discussed in Subsection 2.1.2 is prevented. Consequently, there are no after-market costs to charge to the RPs<sup>74</sup>. In addition to the proposed real-time mechanism, the WEM may still be in place, which all network participants may use for their own risk hedging or for long-term agreements (cf. Fig. 3.7). If the proposed framework is adopted in practice, the only change for market participants is the way in which their own real-time prices for electricity generation or consumption have to be calculated.

For each of the individual network participants and for each of the technical and economic layers in Fig. 4.7, this dissertation has derived dedicated control schemes, with a special focus on optimization-based controller design and distributed controller communication. The major contributions of this dissertation and their relevance to the research gaps addressed in Section 2.3 are depicted in Fig. 7.1 and briefly summarized below.

<sup>74</sup> Remaining grid tariffs, if any, now only apply for construction and maintenance of the grid infrastructure itself.

**Figure 7.1:** Overview of main contributions.

In Chapter 3, we have worked towards a market-based integration of power system stability by developing a distributed, optimization-based frequency and voltage controller such that the actions of the local WPs are stimulated solely by their individual nodal price. By thoroughly accounting for network parameters in the CCs' calculations, the price calculated by CCs also covers all accompanying line losses, thereby closing Research Gap 1. A sufficient condition for stability of the closed-loop system has been provided in Theorem 3.44.

The rigorous pursuit of an economically efficient interplay between stakeholders has constituted a major focus of all endeavors. By introducing a novel, vector-valued zonal participation factor κ along with a distributed price consensus protocol, the equilibrium set of the resulting closed-loop system is restricted to the set of Pareto efficient allocations, which has been elaborated in Theorem 3.51 and Theorem 3.54. The dynamic *regularization* of all participation factors by means of distributed balancing, which has been discussed in Theorem 4.22, thereby forces only those solutions that provide BoP containment, turning the entire power infrastructure into a *monetarily self-contained system* without any accumulation of capital.

The methodological foundation for efficiently regulating the remaining steady-state deviations of aggregate net payments has been laid by introducing a new adaptive framework for optimal control of (generic) PHSs in Theorem 5.24. Since the class of PHSs has proven to be a universally applicable modeling tool for a variety of multi-physics or techno-economic systems, possible applications of the developed method reach far beyond the particular area of power system control applications presented earlier in this work. In Theorem 5.53, the developed methodology has been further extended to the case of N-player noncooperative differential games.

Thanks to distributed balancing via the time-dependent regularization factor a, all of the remaining |Z| − 1 degrees of freedom when choosing κ could be exploited to implement higher-level goals. Namely, control laws with and without integral action have been proposed in the context of this dissertation on how to usefully impose "incentive-signaling" spatial differences among zonal prices, while also respecting the obligation of treating all RPs equally (cf. Claim 4.5). Theoretical conclusions about the resulting closed-loop system have been stated in Theorems 4.20, 4.23 and 4.25, and the practicality of the overall control framework in automatic congestion control has been demonstrated in Case Study II of Chapter 6, finally closing Research Gap 3.

Since all individual controllers are systematically derived from optimization problems, there are no remaining degrees of freedom in the controller design except for the (inverse) controller gains τ(·) , which, however, do not affect the value of the closed-loop equilibrium. Reasonably, the controller gains can be used to provide the necessary timescale separation between the individual control layers. The overall controller thus features low efforts for parameter tuning.

By automatic control of the participation factors κ based on continuous measurements of the state of the network, this work has given rise to allocating marginal production in specific regions of the power system in real time. In contrast to a pure merit order system, the cell-specific multiplication by κ provides a real-time incentive mechanism that encourages profit-driven network participants with power sources and flexible loads to strategically shift their capacities to regions with higher average prices and thus to enhance the reliability of the system by inducing an additional *geographical awareness* among WPs.

As discussed in Subsection 2.1.2, all disruptions caused by the increasing RES infeed have so far been charged to the end customer in an intransparent manner via ever-increasing grid tariffs. Through the concept of spatially differentiated prices, which cluster around the (virtual) uniform WP price Λ 0 , the presented work marks a *paradigm shift towards a "polluter-pays" principle*. Exemplarily shown in Case Study II, where the social optimum gained by uniform pricing would fail to alleviate congestion at market clearing, WPs in unfavorable locations are subject to significantly lower prices through the dynamic pricing mechanism, which in turn pushes them out of the market in the long run. This creates an *ongoing market consolidation* that is favorable from an overall perspective, along with preferential investment in regions that are most desirable by virtue of the global pricing policy.

Remarkably, the proposed control framework does not require any internal model of the WPs involved (apart from the minor assumption of rationally chosen cost functions). This not only protects the privacy of the stakeholders involved, but also enables the integration of an arbitrary number of small-scale flexible prosumers, fully in line with the future vision of power networks as transactive *energy internet* (EI) platforms with minimum entry barriers [MKR<sup>+</sup>20, PCK<sup>+</sup>21].

## **7.2 Outlook**

This work advocates for a careful separation between *sovereignty over the grid* (held by CCs) and *sovereignty over electricity generation* (held by WPs) to maximize fairness without restricting the benefits of a free market. Nonetheless, there will always be the need for a certain normative framework set by policy makers, e.g. rules that ensure compliance with

Assumption 3.17 or the resolution of overriding long-term goals, in order to keep the results of the market mechanism within the bounds of what is socially desirable [NBM+20].

A cell-based infrastructure *without* cutting the physical network has proven to be advantageous compared to the MG concept, as it unites the merits of informational and operational unbundling with those of non-discriminatory power flows. However, the question of most effective cell size and appropriate definition of cell boundaries, which is still open and controversially discussed in academia, is not explicitly addressed in the context of this dissertation. On one hand, as it becomes apparent from the application to congestion control, a finer cellularization, especially along lines that may be vulnerable in terms of congestion, basically contributes to a more elaborate influence of specific line flows. On the other hand, a too fine cellularization counteracts the "no market power" imperative (cf. Assumption 3.17), as it opens up unwanted exploitation opportunities for WPs located at critical points in the network to affect prices unilaterally, leading to reduced competition in the overall network. Hence, an "optimal" cell partitioning must always trade-off between both requirements [Sto97]. While in this dissertation, the determination of appropriate price zones is done manually, based on a priori knowledge and in a fit-and-forget manner, recent lines of research advocate *dynamic* cell partitioning that, in contrast to static approaches, adaptively adjusts to the instantaneous needs of the network [XZK<sup>+</sup>16, p. 135ff.; Wan15, p. 97]. In [HDA21], a first practical algorithm is presented where the dynamic formation of *local energy markets* is adjusted on a daily basis to gather and reward the flexibility of end users.

The explicit control methods derived in this dissertation can all be applied regardless of the eventual (static or dynamic) cell partitioning and therefore constitute a practical and easily implementable toolbox for any superordinate cellularization and pricing strategy to conveniently realize the intended higher-level goals.

The explicit concepts for real-time pricing elaborated in this dissertation open up a broad field for extensions and amendments. For instance, by adequately extending κ ′′ (see Subsection 4.3.3), additional summands could not only incorporate generic long-term goals, but also account for *network effects anticipated for the future* right before their actual occurrence. This could contribute to an internalization of follow-up costs as current costs and therefore pave the way towards a *proactive energy infrastructure* [MS21]. In particular, the distributed balancing mechanism from Subsection 4.4.1, if active, always ensures that any of these additional price components which incorporate future tendencies are already fairly allocated among consumers as of today.

Especially for large-scale power infrastructures, aiming for a distributed controller implementation with neighbor-to-neighbor communication, as pursued in this work, is advantageous both in terms of computational complexity and cyber security. However, compared to the other control schemes presented in this dissertation, the novel adaptive methodology for optimal control of single- and multi-player PHSs introduced in Chapter 5 still requires a complete model of the physical plant system (along with perfect measurement information on the other players' control actions in the multi-player case) and therefore lacks a distributed implementation. Paving the way for the application to large-scale systems, future research should thus focus on optimal controllers and noncooperative differential games for PHSs with *limited information sharing* and non-exact model knowledge. In addition, further emphasis

could be put on more sophisticated adaptation strategies for the individual weighting factors to increase the speed of convergence while still guaranteeing asymptotic stability.

The major stumbling block towards a *comprehensive stability analysis of the overall system* dynamics is the general absence of Lyapunov-based a priori statements in case of lossy power networks (see also the discussion in Subsection 3.4.4 on ongoing research efforts). For this reason, the existing literature on PHS-based control of AC power systems is almost without exception confined to the lossless case. In the context of this work, a sufficient condition for stability has been presented by Theorem 4.25, which, however, contains an impractical criterion that requires a priori knowledge of the numerical steady-state value. This in turn requires the solution of an intractable (and possibly high-dimensional) nonlinear system of power flow equations, which in general is only given through approximation (cf. [Tay15]). A trend in recent research on power system stability and control is therefore towards the design of versatile controllers that preserve *equilibrium-independent passivity* of the overall system [SNM<sup>+</sup>21]. Future advances in this field may allow to derive controllers whose stabilizing properties do not depend upon a condition that is difficult to prove in advance yet at the price of a conservative controller design with limited performance and/or the abandonment of potential degrees of freedom.

Another requirement to be critically investigated by further research is the dynamic security requirement of "sufficient timescale separation" (cf. Section 4.5), which was necessary in the context of this work, but is generally difficult to certify. For this purpose, an even more rigorous *first-principles modeling* e.g. by means of a thorough port-Hamiltonian modeling of the physical plant system and all controller parts bears great potential towards a more native understanding of the (generalized) power flows of the individual subsystems along with their particular impact on overall system stability. Such a generalized stability analysis could eventually obviate the need for any artificial security assumptions regarding timescale separation.

Yet the goals of the energy transition can only be achieved if all parties are enabled in equal measure to be seen as part of the solution. Due to its variability over time, the retail price Λ<sup>R</sup> already introduced as part of this dissertation provides a thriving linkage point for research and innovation on demand-side oriented participation. Suitable price incentives and low-entry optimization tools for personal use might thus also encourage RPs to play a much more active role in future electricity markets. Potential approaches currently being piloted range from smart micro-storages at household level [MGR<sup>+</sup>18], widespread installation of every household with a smart meter infrastructure [BM17, CGM19, YLGW19], and app-based optimal scheduling to exploit *smart home* flexibilities [ATM<sup>+</sup>20], to the deployment of completely decentralized EI platforms for automated settlement of *smart contracts* among micro-scale energy resources, e.g. on the Ethereum blockchain [ACRK19, BAA<sup>+</sup>21, SKSJ21]. However, two factors are decisive to the success of such concepts: First, there must be a political debate about how much flexibility is tolerable for the individual and how much individual inflexibility is tolerable for society. Secondly, any innovation that enables the participation of RPs must be made accessible to everybody with lowest possible entry barriers. If unbureaucratic ways to involve end users can be enabled on a large scale and kept accessible for everyone, there is huge potential in turning the narrative of energy transition from a bare necessity to a collaborative societal project where everyone willing can participate and benefit.

# **A Mathematical Supplements**

## **A.1 Notation**

Throughout this dissertation, vectors and matrices are written in boldface. All vectors are column vectors a = coli{ai} = col{a1, a2, . . .} = (a1, a2, . . .) <sup>T</sup> with elements a<sup>i</sup> , i = 1, 2, . . .. The notation A = diag{a} = diagi{ai} means that A is a diagonal matrix with the entries of vector a on its diagonal. The all-zeros and all-ones vectors of dimension n are denoted by **<sup>0</sup>**<sup>n</sup> and <sup>1</sup>n, respectively. The (<sup>n</sup> <sup>×</sup> <sup>n</sup>)-identity matrix is denoted by <sup>I</sup>n. The unit vector of the k-th axis of a Cartesian coordinate system is denoted by ek. For vectors a, b of the same size, a ≥ b denotes that each component in a is greater than or equal to the corresponding component in b. Positive semidefinite and positive definite matrices or functions are denoted by <sup>≽</sup> <sup>0</sup> and <sup>≻</sup> <sup>0</sup>, respectively. Moreover, we write <sup>V</sup> (x) <sup>≻</sup> <sup>0</sup> if <sup>V</sup> (<sup>x</sup> ̸<sup>=</sup> **<sup>0</sup>**) <sup>&</sup>gt; <sup>0</sup> and <sup>V</sup> (**0**) = 0, and <sup>V</sup> (x) <sup>≽</sup> <sup>0</sup> if <sup>V</sup> (<sup>x</sup> ̸<sup>=</sup> **<sup>0</sup>**) <sup>≥</sup> <sup>0</sup> and <sup>V</sup> (**0**) = 0.

Optimizers or equilibria are marked with a star □<sup>⋆</sup> and shifted values with respect to the optimizer or equilibrium are marked with a tilde □e, i.e. <sup>x</sup>e(t) = <sup>x</sup>(t) <sup>−</sup> <sup>x</sup> ⋆ . Upper and lower bounds of a variable are denoted by □ and □, respectively. Phasor variables are marked with an arrow □⃗ and the complex conjugate of a variable is marked with an asterisk □<sup>∗</sup> .

To allow distinction from the Hamiltonian H of the input-state-output PHS (cf. Definition A.13), the Hamiltonian function of the optimization problem (also called *Pontryagin function*) is denoted by <sup>H</sup> . With the set of players <sup>P</sup>, the vector <sup>u</sup>¬<sup>i</sup> refers to the vector of all inputs <sup>j</sup> <sup>∈</sup> <sup>P</sup>\{i}. Where appropriate, we also the subscript index <sup>¬</sup><sup>i</sup> to denote *all elements from a given index set except* i, if the index set becomes clear from context. For clarity of presentation, the time dependence (t) of the variables is not explicitly mentioned, unless it is essential for transparency of the statements. If no dimension is specified for a certain vector or matrix, it can be derived from context.

For scalars a, b <sup>∈</sup> <sup>R</sup>, we define

$$\langle a \rangle\_b^+ = \begin{cases} a, & a > 0 \text{ or } b > 0, \\ 0, & \text{otherwise}, \end{cases} \tag{A.1}$$

and for vectors <sup>a</sup>, <sup>b</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>, (A.1) is applied component-wise, i.e. ⟨a⟩ + <sup>b</sup> = col{⟨a1⟩ + b1 , . . . , ⟨an⟩ + b<sup>n</sup> }. For a scalar <sup>a</sup> <sup>∈</sup> <sup>R</sup>, we define the saturation operator

$$\langle a \rangle\_{-1}^{1} = \begin{cases} 1, & a > 1, \\ a, & a \in [-1, 1], \\ -1, & a < -1, \end{cases} \tag{A.2}$$

and for vectors <sup>a</sup> <sup>∈</sup> <sup>R</sup>n, (A.2) is applied component wise, i.e. ⟨a⟩ 1 <sup>−</sup><sup>1</sup> = col(⟨a1⟩ 1 −1 , . . . ,⟨an⟩ 1 −1 ).

The operations absolute value | · |, natural logarithm ln(·), and exponential function exp(·) when applied to vectors <sup>a</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup> are applied component wise, e.g. <sup>|</sup>a<sup>|</sup> <sup>=</sup> col{|a1|, . . . , <sup>|</sup>an|} etc. The Hadamard product of two vectors is denoted by ◦, i.e. a ◦ b = col{a1b1, . . . , anbn} for each <sup>a</sup>, <sup>b</sup> <sup>∈</sup> <sup>R</sup>n.

The gradient of a function <sup>f</sup> : <sup>R</sup><sup>n</sup> <sup>→</sup> <sup>R</sup> with respect to <sup>x</sup> is denoted by <sup>∇</sup>f(x) and is the column vector of all partial derivatives of f(x), i.e. ∇f(x) = col{ ∂f(x) ∂x<sup>1</sup> , . . . , ∂f(x) ∂x<sup>n</sup> }. The Jacobian of <sup>Φ</sup> : <sup>R</sup><sup>n</sup> <sup>→</sup> <sup>R</sup><sup>r</sup> with <sup>x</sup> 7→ <sup>Φ</sup>(x) is defined as <sup>∂</sup><sup>Φ</sup> <sup>∂</sup><sup>x</sup> = (∇Φ1(x), . . . , <sup>∇</sup>Φr(x))<sup>T</sup>.

# **A.2 Basic Concepts of Game Theory**

The conceptual framework for the investigation of decision-making situations among multiple network participants with different, possibly conflicting goals is provided by game theory. In recent decades, game theory has been established as a lingua franca for almost all research on competitive energy markets [ANHMIT20, p. 86; NBM<sup>+</sup>20, p. 2], since it not only offers modeling tools for conflicting situations, but also explicit instructions for strategic actions of the participants as well as a normative assessment of desired and undesired operating points from participants' viewpoints. After first conceptual studies in the 1970s [MD77, BHK78], game-theoretic concepts modeling the interaction of competitive network participants have been developed at every timescale, from optimal day-ahead scheduling on an hourly basis [AOS<sup>+</sup>13b] to real-time strategies using differential games<sup>75</sup> [LLL<sup>+</sup>19]. Moreover, a wide variety of use cases and network participants have been considered, from optimal generator dispatch, demand-side management and demand response<sup>76</sup> [MWY<sup>+</sup>19], to optimal balancing of interconnected distribution systems and optimal utilization of shared resources [MMM20]. An early review on the applications of game theory for power systems is conducted in [SHPB12]. For more recent survey papers, see [CY19, ANHMIT20, NBM<sup>+</sup>20].

In the context of game theory, each network participant is modeled as element <sup>i</sup> <sup>∈</sup> <sup>P</sup> of a set of players. It aims at maximizing its own profit P<sup>i</sup> by choosing a strategy a<sup>i</sup> .

**Definition A.1 (Game [BO99; Bau16, p. 4f.]).** *<sup>A</sup>* game *is a tuple* (P,(Ai)<sup>i</sup>∈<sup>P</sup>,(Pi)<sup>i</sup>∈<sup>P</sup>)*, where* <sup>P</sup> <sup>=</sup> {1, . . . , N} *is the set of players,* <sup>A</sup><sup>i</sup> *is the set of actions of player* <sup>i</sup> <sup>∈</sup> <sup>P</sup>*, and*

$$\mathbf{P}\_i: \mathbb{A}\_1 \times \dots \times \mathbb{A}\_N \to \mathbb{R} \tag{A.3}$$

*is the profit function of player* <sup>i</sup> <sup>∈</sup> <sup>P</sup>*.*

For convenience of notation, let <sup>A</sup> <sup>=</sup> <sup>A</sup><sup>1</sup> × · · · × <sup>A</sup><sup>N</sup> denote the set of *action profiles*, where an action profile <sup>a</sup> <sup>∈</sup> <sup>A</sup> is the collection of all actions <sup>a</sup><sup>i</sup> of the players. If <sup>P</sup><sup>i</sup> depends not

<sup>75</sup> (cf. Definition A.2 in Appendix A.2)

<sup>76</sup> The term "demand-side management" encompasses all (long- and short-term) control actions at consumer side, while "demand response" specifically refers to the provision of short-term incentives for price-elastic consumers.

only on the actions <sup>a</sup><sup>i</sup> <sup>∈</sup> <sup>A</sup><sup>i</sup> but also on the state <sup>x</sup>(t) of a dynamical system, this leads to the notion of a differential game:

**Definition A.2 (Differential Game [SH69; BO99; Bau16, p. 90]).** *A* differential game *is a tuple* (P, <sup>S</sup>,(Ai)i∈P,(Pi)i∈P)*, where* <sup>P</sup> <sup>=</sup> {1, . . . , N} *is the set of players,* <sup>S</sup> *is a controlled dynamical system*

$$
\dot{x}(t) = f(x(t), u(t)), \tag{A.4a}
$$

$$x(0) = x\_0,\tag{A.4b}$$

*where* <sup>f</sup> : <sup>R</sup><sup>n</sup> <sup>×</sup> <sup>R</sup><sup>p</sup><sup>1</sup> × · · · × <sup>R</sup><sup>p</sup><sup>N</sup> *is Lipschitz continuous with state* <sup>x</sup> ∈ X ⊆ <sup>R</sup><sup>n</sup> *and inputs* <sup>u</sup><sup>i</sup> ∈ U<sup>i</sup> <sup>⊆</sup> <sup>R</sup><sup>p</sup><sup>i</sup> *.* A<sup>i</sup> *is the set of actions of player* <sup>i</sup> <sup>∈</sup> <sup>P</sup> *with* <sup>A</sup><sup>i</sup> <sup>=</sup> <sup>U</sup><sup>i</sup> *, and*

$$\mathbf{P}\_i: \mathbb{A} \times \mathcal{X} \to \mathbb{R} \tag{A.5}$$

*is the profit function of player* <sup>i</sup> <sup>∈</sup> <sup>P</sup>*.*

**Remark A.3.** *Independent to the notion of differential games, the term* dynamic game *encompasses those games where the players make their decisions* A<sup>i</sup> sequentially*, in contrast to* static *games, where players act simultaneously.* ♢

A common assumption in classical game theory is that all players are subject to the following basic behavioral pattern:

**Assumption A.4 (Rationality and Common Knowledge of Rationality [Aum76, AB95, Bic04]).** *Each player acts* rationally*, i.e. strives to maximize its own profit* P<sup>i</sup> *. Moreover, each player takes into account that the other players act rationally as well.*

An important implication of Assumption A.4 is that especially those strategies are preferable, where no player can achieve a higher profit without at least one other player achieving a lower profit. Such strategies are said to be *Pareto efficient* or *Pareto optimal.* A formal definition is given below:

**Definition A.5 (Pareto Efficiency [Bau16]).** *The action profile* a <sup>⋆</sup> = (a ⋆ 1 , . . . , a ⋆ i , . . . , a ⋆ N ) t ransp *is said to be* Pareto efficient *if there exists no other* <sup>a</sup> <sup>∈</sup> <sup>A</sup> *such that*

<sup>∃</sup><sup>i</sup> <sup>∈</sup> <sup>P</sup> : <sup>P</sup>i(a) <sup>&</sup>gt; <sup>P</sup>i(<sup>a</sup> ⋆ ) ∧ P¬<sup>i</sup>(a) ≥ P¬<sup>i</sup>(a ⋆ ). (A.6)

*If, instead, there exists an* <sup>a</sup> <sup>∈</sup> <sup>A</sup> *fulfilling* (A.6)*, then* <sup>a</sup> ⋆ *is said to be a* Pareto inefficient *or* Pareto dominated *strategy.*

For existence theorems on Pareto efficient solutions, see e.g. [Meh85].

Another prominent solution strategy arising from Assumption A.4 is to find *Nash equilibria* among all possible action profiles <sup>a</sup> <sup>∈</sup> <sup>A</sup>:

**Definition A.6 (Nash Equilibrium [Bau16, p. 8]).** *The action profile* a <sup>⋆</sup> = (a ⋆ 1 , . . . , a ⋆ N ) *is said to be a* Nash equilibrium*, if none of the players can make a higher profit by unilaterally deviating from its own action profile, i.e., if*

$$\forall \; a\_i \in \mathbb{A}\_i: \qquad \qquad \mathbb{P}\_i(a\_i, a\_{\neg i}) \ge \mathbb{P}\_i(a\_i^\star, a\_{\neg i}^\star) \tag{A.7}$$

*holds for all* <sup>i</sup> <sup>∈</sup> <sup>P</sup>*.*

Depending on the *information structure* of the given differential game, we distinguish between *open-loop strategies*, where each player only has knowledge of the initial state x<sup>0</sup> of (A.4), and *closed-loop strategies*, where each player is able to observe the actual state x(t) of (A.4).

If all profit functions P<sup>i</sup> are Lagrange-type, implicit solutions exists for both open-loop and closed-loop Nash strategies.

**Definition A.7 (Lagrange-Type Performance Index).** *A profit function is said to be a* Lagrange-type *performance index, if it has the following form.*

$$\mathbb{P}\_{i}(\mathbf{z}) = -\int\_{0}^{t\_{\mathrm{f}}} \mathcal{L}\_{i}(\mathbf{z}(t), \mathfrak{u}\_{1}(t), \dots, \mathfrak{u}\_{N}(t)) \, \mathrm{d}t. \tag{A.8}$$

In the open-loop case, the Nash strategy of each player is determined by solving a boundary value problem which follows from *Pontryagin's maximum principle* [Bre11]. The resulting open-loop control laws depends only on time, thus a disturbance leads most likely to suboptimal operation.

In the context of feedback control, where the actors base their strategic decisions on continuous measurements of the state vector and have limited information about other actors' objectives, emphasis is placed on finding closed-loop Nash strategies (also called feedback Nash strategies). Here, the players are able to react to disturbances by immediately noticing undesired deviations from the desired state. An implicit solution is given by a set of coupled Hamilton-Jacobi-Bellman (HJB) equations:

**Lemma A.8 (Sufficient Condition for Feedback Nash Equilibrium [Bre11, HKZ12]).** *For the differential game from Definition A.2 with fixed duration* [0, t<sup>f</sup> ]*, the action profile* a <sup>⋆</sup> = (u ⋆ i (t), . . . ,u ⋆ <sup>N</sup> (t))<sup>T</sup> *provides a feedback Nash equilibrium, if there exist continuously differentiable* value functions V ⋆ i : [0, T] × X → <sup>R</sup> *for each* <sup>i</sup> <sup>∈</sup> <sup>P</sup>*, such that the HJB (partial differential) equation*

$$-\frac{\partial V\_i^\star(t, x)}{\partial t} = \max\_{u\_i \in \mathcal{U}\_i} \left\{ \left( \frac{\partial V\_i^\star(t, x)}{\partial x} \right)^\top f(t, x, u\_i, u\_{\neg i}) - \mathcal{L}\_i(t, x, u\_i, u\_{\neg i}^\star) \right\} \tag{A.9}$$

*and the terminal condition*

$$V\_i^\star(t\_\mathbf{f}, \mathbf{z}) = 0 \tag{A.10}$$

*are satisfied for all* (t, x) ∈ [0, T] × X *.*

Note that for N = 1, the above differential game with Lagrange-type performance index is equivalent to the classical optimal control problem. For existence theorems for open-loop and feedback Nash equilibria, see [Bre11, p. 360ff.].

# **A.3 Algebraic Graph Theory**

**Definition A.9 (Graph, Directed Graph, Connected Graph, Complete Graph [Bap14, p. 10; HKK**<sup>+</sup>**05, p. 9; BM76, p. 4]).** *<sup>A</sup>* graph <sup>G</sup> *is a pair* (V, <sup>E</sup>)*, where* <sup>V</sup> *is the set of nodes and* E *is the set of edges. The graph is said to be* directed *if the edges* e ∈ E *are ordered pairs* e = (v1, v2)*, where* v<sup>1</sup> ∈ V *represents the tail of* e *and* v<sup>2</sup> ∈ V *represents the head of* e*. The graph is said to be* (weakly) connected*, if there exists an undirected path connecting every pair of nodes in* V*. The graph is said to be* complete*, if there exists an edge between each pair of nodes.*

**Definition A.10 (Incidence Matrix [Bap14, p. 13]).** *The* incidence matrix <sup>D</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>×<sup>m</sup> *of the graph* G *is defined by*

$$[\mathcal{D}]\_{i,j} = \begin{cases} 1, & i \text{ is the head of } e\_j, \\ -1, & i \text{ is the tail of } e\_j, \\ 0, & \text{otherwise.} \end{cases} \tag{A.11}$$

*If* G *is a weighted graph, then the non-zero entries in* (A.11) *are multiplied by the weights* <sup>w</sup>ij <sup>∈</sup> <sup>R</sup> *of the respective edge* (i, j) ∈ E*.*

**Definition A.11 (Laplacian Matrix [Bap14, p. 49]).** *The* Laplacian matrix <sup>A</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>×<sup>n</sup> *of the graph* G *is defined by*

$$\mathcal{A} = \mathbf{D} \mathbf{D}^{\mathsf{T}}.\tag{A.12}$$

From Definition A.11 it follows that [A]i,j = [A]j,i = −1, if nodes i and j are incident, and [A]i,i being equal to the *degree* of node i ∈ V (i.e. the total number of edges connected to node i). The incidence and Laplacian matrix of a connected graph have the following properties:

**Lemma A.12.** *If a graph* G = (V, E) *with* |V| = n *is connected, then the following statements hold:*


*Proof.* (see [Bap14, pp. 14, 50f.])

## **A.4 Port-Hamiltonian Systems**

In recent years, systematic modeling of dynamical multi-physics systems<sup>77</sup> in port-Hamiltonian form has become increasingly popular in a wide range of applications such as acoustics [FH16], aerospace [ACRMA19], robotics [GSC17, MMS07, MMS09], power electronics [BDST17, CGB<sup>+</sup>19], material science [LSM<sup>+</sup>20], or energy systems [FZO<sup>+</sup>13, SOA<sup>+</sup>14, CT14, HMM<sup>+</sup>20]. Among the broad class of implicit or explicit PHS representations (see e.g. [CGHM13] for a comprehensive overview), the state-space representation via *input-state-output PHSs*, in which the system dynamics are affected by the gradient field of a smooth, positive definite *Hamiltonian* H(x) representing the overall energy stored in the system, is very familiar for modeling and control purposes, as it allows direct access to the inputs <sup>u</sup> <sup>∈</sup> <sup>R</sup><sup>p</sup> and outputs <sup>y</sup> <sup>∈</sup> <sup>R</sup><sup>p</sup> of the system.

**Definition A.13 (Input-State-Output PHS [van17, Definition 6.1.1]).** *Let* <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup> *be a state vector. The state-space system*

$$
\dot{x} = (J(x) - R(x))\frac{\partial H(x)}{\partial x} + G(x)u,\tag{A.13a}
$$

$$y = G^{\top}(x)\frac{\partial H(x)}{\partial x} \tag{A.13b}$$

*is said to be an* input-state-output PHS (without feedthrough)*, if* J(x) = −J T (x) *and* R(x) = R T (x) <sup>≽</sup> <sup>0</sup>*. The gradient* <sup>z</sup> := <sup>∇</sup>H(x) *is said to be the vector of* co-states *or* generalized efforts *of* (A.13)*.*

The above input-state-output representation clearly visualizes a separation between the interconnection topology of the system and the constitutive relations of its components, which is an inherent property of PHSs [DBMS09]. The input-state-output representation of

<sup>77</sup> The PHS representation can be applied whenever energy plays a major role in the dynamical system to be modeled. Indeed, this modeling paradigm can also be abstracted to non-physical systems, e.g. by conceiving the momentary value of the objective function of gradient flow (cf. Subsection 3.4.1) as generalized *cyber-energy* stored in the system [JOv17, p. 109].

PHSs is thus particularly suitable for modeling of large-scale power systems with a repetitive structure. A generalization of input-state-output PHSs is given by the following definition:

**Definition A.14 (Input-State-Output PHS with Nonlinear Resistive Structure [van17, Definition 6.1.4]).** *Let* <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup> *be a state vector. The state-space system*

$$
\dot{x} = J(x)\frac{\partial H(x)}{\partial x} - \mathcal{R}(x) + \mathcal{G}(x)u,\tag{A.14a}
$$

$$y = \mathbf{G}^{\sf T}(x) \frac{\partial H(x)}{\partial x} \tag{A.14b}$$

*is said to be an* input-state-output PHS with nonlinear resistive structure*, if* J(x) = −J T (x) *and* z <sup>T</sup>R(x) <sup>≥</sup> <sup>0</sup>*.*

**Definition A.15 (Passivity, Shifted Passivity [van17, p. 94ff.]).** *A nonlinear statespace system*

$$
\dot{x} = f(x, u),
\tag{A.15a}
$$

$$y = g(x, u),\tag{A.15b}$$

*where* <sup>f</sup> : <sup>R</sup><sup>n</sup> <sup>×</sup> <sup>R</sup><sup>p</sup> <sup>→</sup> <sup>R</sup><sup>n</sup> *is locally Lipschitz,* <sup>g</sup> : <sup>R</sup><sup>n</sup> <sup>×</sup> <sup>R</sup><sup>p</sup> <sup>→</sup> <sup>R</sup><sup>p</sup> *is continuous,* f(**0**, **0**) = **0** *and* g(**0**, **0**) = **0** *is said to be* passive *if there exists a positive semidefinite* storage function V (x) *such that*

$$\dot{V} = \left(\frac{\partial V}{\partial x}\right)^{\mathsf{T}} f(x, u) \le u^{\mathsf{T}} y \tag{A.16}$$

*holds for all* (x,u) <sup>∈</sup> <sup>R</sup><sup>n</sup> <sup>×</sup> <sup>R</sup><sup>p</sup> *.*

*Let* (x ⋆ ,u ⋆ ) *with* f(x ⋆ ,u ⋆ ) = **0** *denote an equilibrium of* (A.15)*. Then* (A.15) *is said to be* shifted passive *with respect to* (x ⋆ ,u ⋆ ) *if there exists a positive semidefinite storage function* V (x) *with*

$$\dot{V} = \left(\frac{\partial V}{\partial x}\right)^{\mathsf{T}} f(x, u) \le (u - u^\star)^{\mathsf{T}} (y - y^\star). \tag{A.17}$$

The following corollary trivially follows by comparison of Definition A.15 with Definition A.13 and Definition A.14.

**Corollary A.16.** *The input-state-output PHSs* (A.13) *or* (A.14) *are passive, if and only if their Hamiltonians* H(x) *are bounded from below.*

**Remark A.17.** *Depending on the application, different sortings of the state vector* x *are chosen in the context of the dissertation. However, the PHS property is* invariant *with respect to these permutations. Consider e.g. the state transformation* x ′ <sup>=</sup> P x*, where* <sup>P</sup> <sup>∈</sup> <sup>R</sup>n×<sup>n</sup> *denotes an arbitrary permutation matrix. Then it can be directly inferred that the transformed input-stateoutput PHS*

$$\dot{x}' = (\underbrace{\mathbf{PJ}(x)\mathbf{P}^{-1}}\_{J(x')} - \underbrace{\mathbf{P}\mathbf{R}(x)\mathbf{P}^{-1}}\_{R(x')})\frac{\partial H(x)}{\partial x'} + \mathbf{P}G(x)u,\tag{A.18a}$$

$$y = (\mathbf{P}G(x))^{\top}\frac{\partial H(x)}{\partial x}\tag{A.18b}$$

$$y = (\mathbf{PG}(\mathbf{z}))^\top \frac{\mathbf{W}^{-\top}}{\partial \mathbf{z}}\tag{A.18b}$$

$$\mathbf{J}(\mathbf{z}') = -\mathbf{J}^\top(\mathbf{z}') \text{ and } \mathbf{B}(\mathbf{z}') = \mathbf{B}^\top(\mathbf{z}') \succ 0. \text{ This monotonicity also holds for the numerical solution}$$

*fulfills* J(x ′ ) = −J (x ′ ) *and* R(x ′ ) = R (x ′ ) ≽ 0*. This property also holds for the general case of input-state-output PHSs with nonlinear resistive structure.* ♢

# **A.5 Convex Optimization**

**Definition A.18 (Optimization Problem in Standard Form).**

µ

	- h<sup>j</sup> (x) = 0, j = 1, . . . , m (A.19c)

**Definition A.19 (Convex Optimization Problem).** *The optimization problem* (A.19) *is said to be a* convex optimization problem *if* fi(x)*,* i = 0, . . . , p *are convex functions and* h<sup>j</sup> (x)*,* j = 1, . . . , m *are affine functions.*

**Definition A.20 (Karush-Kuhn-Tucker (KKT) Point [BV15, p. 243ff.]).** *Consider the optimization problem* (A.19)*. Each* (x ⋆ ,λ ⋆ , µ ⋆ ) *is said to be a* Karush-Kuhn-Tucker point*, if*

$$\frac{\partial \mathcal{Q}}{\partial x}(x^\star, \lambda^\star, \mu^\star) = 0,\tag{A.20a}$$

$$\frac{\partial \mathcal{Q}}{\partial \lambda}(x^\star, \lambda^\star, \mu^\star) = 0,\tag{A.20b}$$

	- <sup>⋆</sup> <sup>≥</sup> **<sup>0</sup>**, (A.20d)

*where*

$$\mathcal{L}^{\rho}(x,\lambda,\mu) := f\_0(x) + \sum\_{i=1}^{p} \mu\_i f\_i(x) + \sum\_{j=1}^{m} \lambda\_j h\_j(x) \tag{A.21}$$

*denotes the* Lagrangian *of* (A.19)*.*

**Definition A.21 (Slater's Condition [BV15, p. 226f.]).** *The convex optimization problem* (A.19) *is said to fulfill* Slater's condition*, if there exists some* x ′ *which satisfies* (A.19b)*–* (A.19c) *and* fi(x ′ ) < 0 *for all* non-affine *functions* f<sup>i</sup> *.*

**Lemma A.22.** *Consider the the convex optimization problem* (A.19) *and assume that Slater's condition holds. Then* x *is a global optimizer of* (A.19)*, if and only if* (x ⋆ ,λ ⋆ , µ ⋆ ) *is a KKT point.*

*Proof.* (see e.g. [BV15, p. 226])

# **A.6 Basic Principles of Welfare Economics**

**Definition A.23 (Market Equilibrium, Market-Clearing Price, Market-Clearing Quantity [Deb59, Yu20]).** *A* market equilibrium *is a condition where a market price is settled through competition such that the amount of produced goods equals the amount of demanded goods. The price reached at market equilibrium is said to be the* market-clearing price *or* competitive price*, and the quantity reached at market equilibrium is said to be the* market-clearing quantity *or* competitive quantity*.*

**Definition A.24 (Perfect Market, Perfect Competition [Bor93, Ley18, Yu20]).** *A market structure is said to be a* perfect market *or* perfect competition*, if the following idealized conditions are fulfilled:*


*7. There is a large number of suppliers and demanders interacting with each other, so that none of them can individually influence market prices.*

A perfect market has the following feature.

**Theorem A.25 (First Fundamental Theorem of Welfare Economics [Moo06, p. 150]).** *In a perfect market, each market equilibrium leads to a Pareto efficient allocation of resources.*

In *monopolies* or *oligpolies*, individual suppliers have a considerable influence on prices<sup>78</sup> . However, if there are a sufficient number of players (*polypoly*), the assumption of price-taking behavior is justified, as a single player has only negligible influence on the overall price.

A simple but powerful welfare metric<sup>79</sup> is the sum of aggregate profits (*utilitarian* or *Benthamian welfare function*) [Var10, p. 635], which in our case equals

$$\mathbb{W} = \sum\_{\pi \in \mathcal{W}} \mathbb{P}\_{\pi} + \sum\_{k \in \mathcal{Z}} \mathbb{P}\_{k} - \sum\_{k \in \mathcal{Z}} \mathbb{C}\_{k}^{\ell}. \tag{A.22}$$

The usefulness of W becomes clear in the following result:

**Theorem A.26 (Pareto Efficiency of Market Equilibria under Perfect Competition [Var10, p. 636ff.]).** W *attains its maximum in an equilibrium of a perfect market.*

**Remark A.27.** *Theorem A.26 states that in a perfect market, the point of social optimum (i.e.* arg max W*) and the point of individual optimum (in the sense of individual Pareto efficiency) coincide. From a welfare point of view, it is therefore desirable that the conditions of Definition A.24 are satisfied to the best possible extent.* ♢

<sup>78</sup> Oligopolies with few participants are said to be *tight*, and those with many participants are said to be *wide*. The wider the oligopoly, the closer the market equilibrium is to the social optimum.

<sup>79</sup> Another welfare measure which is sometimes encountered in the context of energy economics is the *maximin* (or Rawlsian) welfare function [Var10, p. 636] W ′ = mini∈P{Pi}, where only the profit of the "worst-off" market participant is relevant. However, since the profits of the other market participants are completely disregarded and furthermore W ′ does not conform to the chosen profit functions for the case where WPs have different nominal outputs, this type of welfare functions is not considered further.

# **B Supplementary Material to** *Optimization-based Control of Cellular Power Networks* **(Chapter 3)**

# **B.1 Synchronous Machine-Type Nodes: Connection with the Power Network**

Fig. B.1 shows a vector diagram of the stator voltages U ′ i and Ug,i and their relationship to the external nodal voltage U<sup>i</sup> . For convenience, we define a synchronous rotating reference frame and denote the absolute rotor angle with respect to the same reference frame by δ<sup>i</sup> . φ<sup>i</sup> defines the *load angle* between stator current and nodal voltage<sup>80</sup> , ϑg,i is the *rotor angle* between U⃗ g,i and U⃗ ′ i , and ϑn,i is the (pseudo-) rotor angle with respect to the synchronous rotating reference frame of U⃗ i . The current and voltage phasors in synchronous rotating reference frame can be characterized by U⃗ ′ <sup>i</sup> = U ′ i exp ȷδ<sup>i</sup> = U ′ i exp ȷ(θ<sup>i</sup> + ϑn,i), U⃗ <sup>g</sup>,i = Ug,i exp(ȷ(θ<sup>i</sup> + ϑg,i)), U⃗ <sup>i</sup> <sup>=</sup> <sup>U</sup><sup>i</sup> exp(ȷθi), and ⃗I<sup>i</sup> <sup>=</sup> <sup>I</sup><sup>i</sup> exp <sup>ȷ</sup>(θ<sup>i</sup> <sup>−</sup> <sup>φ</sup>i) = <sup>I</sup><sup>i</sup> exp <sup>ȷ</sup>(θ<sup>i</sup> <sup>+</sup> <sup>ϑ</sup>n,i <sup>−</sup> <sup>φ</sup> ′ i ). From the right half of Fig. 3.4, it follows that

$$
\vec{U}'\_i - \vec{U}\_{\mathbf{g},i} = jX\_{\mathbf{d},i}\vec{I}\_{\mathbf{d},i} + jX\_{\mathbf{q},i}\vec{I}\_{\mathbf{q},i}.\tag{\text{B.1}}
$$

Moreover, it follows that Ugd,i − Ud,i = ȷXT,iIq,i + ȷXn,iIq,i and Ugq,i − Uq,i = ȷXT,iId,i + ȷXn,iId,i, hence

$$
\vec{U}'\_i - \vec{U}\_i = jX\_{\mathbf{d},i}\vec{I}\_{\mathbf{d},i} + jX\_{\mathbf{q},i}\vec{I}\_{\mathbf{q},i} + j(X\_{\mathbf{T},i} + X\_{\mathbf{n},i})(\vec{I}\_{\mathbf{d},i} + \vec{I}\_{\mathbf{q},i}).\tag{\text{B.2}}
$$

With the geometric relationships from Fig. B.1

$$I\_{\mathbf{d},i} = I\_i \sin(\varphi\_i'),\tag{\text{B.3a}}$$

$$I\_{\mathbf{q},i} = I\_i \cos(\varphi\_i'),\tag{\text{B.3b}}$$

$$U\_{\mathbf{q},i} = U\_i \sin(\vartheta\_{\mathbf{n},i}),\tag{\text{B.3c}}$$

$$U\_{\mathbf{d},i} = U\_i \cos(\vartheta\_{\mathbf{n},i}),\tag{\text{B.3d}}$$

we can calculate the active and reactive power injections

$$\begin{split} p\_{\text{inj},i} &= \Re(\vec{U}\_i \vec{I}\_i^\*) = U\_i I\_i \cos(\varphi\_i) = U\_i I\_i \cos(\varphi\_i' - \vartheta\_{\text{n},i}) \\ &= U\_i I\_i \sin(\varphi\_i') \sin(\vartheta\_{\text{n},i}) + U\_i I\_i \cos(\varphi\_i') \cos(\vartheta\_{\text{n},i}), \end{split} \tag{B.4}$$

$$\begin{split} q\_{\text{inj},i} &= \odot(\vec{U}\_{i}\vec{I}\_{i}^{\*}) = U\_{i}I\_{i}\sin(\varphi\_{i}) = U\_{i}I\_{i}\sin(\varphi\_{i}^{\prime} - \vartheta\_{\text{n},i}) \\ &= U\_{i}I\_{i}\sin(\varphi\_{i}^{\prime})\cos(\vartheta\_{\text{n},i}) - U\_{i}I\_{i}\cos(\varphi\_{i}^{\prime})\sin(\vartheta\_{\text{n},i}). \end{split} \tag{B.5}$$

<sup>80</sup> The SM acts as generator if ϑn,i > 0 and as motor if ϑn,i < 0.

**(a)** Synchronous generator (ϑn,i > 0)

**(b)** Synchronous motor (ϑn,i < 0)

**Figure B.1:** Vector diagram of a SM.

Inserting (B.3) into (B.4) and (B.5) yields

$$p\_{\rm inj,i} = U\_{\rm d,i} I\_{\rm d,i} + U\_{\rm q,i} I\_{\rm q,i},\tag{B.6}$$

$$q\_{\rm inj,i} = U\_{\rm q,i} I\_{\rm d,i} - U\_{\rm d,i} I\_{\rm q,i} \,. \tag{B.7}$$

Since ⃗Id,i and ⃗Iq,i are orthogonal, (B.2) can be written separately for both d- and q-axis, which yields

$$U\_{\mathbf{q},i}^{\prime} - U\_{\mathbf{q},i} = I\_{\mathbf{d},i}(X\_{\mathbf{d},i} + X\_{\mathbf{T},i} + X\_{\mathbf{n},i}) \tag{\text{B.8}}$$

$$U\_{\mathbf{d},i}' - U\_{\mathbf{d},i} = I\_{\mathbf{q},i}(X\_{\mathbf{q},i} + X\_{\mathbf{T},i} + X\_{\mathbf{n},i}) \tag{\text{B.9}}$$

With Assumption 3.7b), we have U ′ <sup>d</sup>,i = 0. Moreover, we define the auxiliary reactances Xdn,i := Xd,i + XT,i + Xn,i and Xqn,i := Xq,i + XT,i + Xn,i to get

$$I\_{\rm d,i} = \frac{U\_i' - U\_i \cos(\vartheta\_{\rm n,i})}{X\_{\rm dn,i}},\tag{\rm B.10a}$$

$$I\_{\mathbf{q},i} = \frac{U\_i \sin(\vartheta\_{\mathbf{n},i})}{X\_{\mathbf{q}\mathbf{n},i}}. \tag{\text{B.10b}}$$

Substituting (B.10) into (B.4) and (B.7) yields

$$p\_{\rm inj,i} = \frac{U\_i' U\_i}{X\_{\rm dn,i}} \sin(\vartheta\_{\rm n,i}) + \frac{U\_i^2}{2} \frac{X\_{\rm dn,i} - X\_{\rm qn,i}}{X\_{\rm dn,i} X\_{\rm qn,i}} \sin(2\vartheta\_{\rm n,i}),\tag{B.11}$$

$$q\_{\rm inj,i} = \frac{U\_i' U\_i}{X\_{\rm dn,i}} \cos(\vartheta\_{\rm n,i}) - \frac{U\_i^2}{X\_{\rm dn,i} X\_{\rm qn,i}} \left( X\_{\rm dn,i} \sin^2(\vartheta\_{\rm n,i}) + X\_{\rm qn,i} \cos^2(\vartheta\_{\rm n,i}) \right). \tag{B.12}$$

## **B.2 Distributed Consensus-Based Control**

In the area of networked systems, there often arise control laws of the *separable type*

$$\dot{u} = \sum\_{i=1}^{n} f\_i(u),\tag{B.13}$$

where the calculation of the global variable u is composed of a finite amount of additive components fi(u) with i = 1, . . . , n, representing "local" calculations, e.g. relying on parameters which are only locally available. For the local calculation of fi(u), however, each local agent needs instantaneous knowledge of the global variable u. Additionally, in order to calculate u˙, all fi(u) need to be known globally.

The idea of *consensus-based control* [OSFM07] is to design local controllers for agent i which achieve the same equilibrium as (B.13), but without requiring knowledge of u or f¬<sup>i</sup>(u). Instead, the agents communicate local variables in a distributed fashion via m neighbor-toneighbor communication links. The communication structure is represented by a weakly connected, directed graph with incidence matrix <sup>D</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>×<sup>n</sup>.

**Lemma B.1.** *Consider the centralized controller* (B.13) *and let the steady state* u ⋆ *of* (B.13) *be unique. Then, each equilibrium* (u ⋆ , ν ⋆ ) *of the distributed* consensus-based controller

$$
\tau\_{\mathsf{u}} \dot{\mathsf{u}} = \begin{bmatrix} f\_1(u\_1) \\ \vdots \\ \vdots \\ f\_n(u\_n) \end{bmatrix} + \mathsf{D}\nu,\tag{\mathsf{B}.14a}
$$

$$
\tau\_{\nu}\dot{\nu} = -\mathbf{D}^{\mathsf{T}}u \tag{\text{B.14b}}
$$

*with* τ <sup>u</sup>, τ <sup>ν</sup> ≻ 0 *is unique. Moreover, it holds that* u <sup>⋆</sup> = u ⋆ · 1*.*

*Proof.* The equilibrium of (B.14) is characterized by

$$\mathbb{0} = \begin{bmatrix} f\_1(u\_1^\star) \\ \vdots \\ \vdots \\ f\_n(u\_n^\star) \end{bmatrix} + \mathcal{D}\nu^\star,\tag{\text{B.15a}}$$

$$\mathbb{0} = -\mathbf{D}^{\mathsf{T}} u^{\star}.\tag{\text{B.15b}}$$

Since D is the incidence matrix of a connected graph, (B.15b) implies u ⋆ <sup>1</sup> = . . . = u ⋆ <sup>n</sup> =: ˆu ⋆ . Premultiplying (B.15a) with 1 <sup>T</sup> yields

$$0 = \mathbb{1}^{\mathsf{T}} \begin{bmatrix} f\_1(\hat{u}^\star) \\ \vdots \\ f\_n(\hat{u}^\star) \end{bmatrix} + \underbrace{\mathbb{1}^{\mathsf{T}}}\_{=0} \mathbf{D} \boldsymbol{\nu}^\star = \sum\_{i=1}^n f\_i(\hat{u}^\star), \tag{\mathsf{B.16}}$$

which is equivalent to (B.13), implying that uˆ ⋆ is equal to u ⋆ . To prove uniqueness of ν ⋆ , define <sup>ν</sup> =: <sup>ν</sup><sup>I</sup> <sup>+</sup> <sup>ν</sup>K, where <sup>ν</sup><sup>I</sup> <sup>∈</sup> im(Dτ <sup>−</sup><sup>1</sup> ν ) and <sup>ν</sup><sup>K</sup> <sup>∈</sup> ker(Dτ <sup>−</sup><sup>1</sup> ν ). We show that both ν ⋆ I and ν ⋆ <sup>K</sup> are unique, which implies that ν ⋆ is unique as well.

With regard to ν ⋆ <sup>K</sup>, note that for each <sup>b</sup> <sup>∈</sup> ker(Dτ <sup>−</sup><sup>1</sup> ν ) it holds that b T ν˙ = 0 and thus b T ν(t) = b T ν(0) for all t > 0. Moreover, with b T ν(t) = b T νK(t), it follows that ν ⋆ <sup>K</sup> = νK(0), thus ν ⋆ <sup>K</sup> is unique. With regard to ν ⋆ I , note that dim(im(Dτ <sup>−</sup><sup>1</sup> ν )) = rank(D) since τ <sup>ν</sup> has full rank. With (B.15a), we state that ν ⋆ I is the unique solution of

$$-\begin{bmatrix} f\_1(\hat{u}^\star) \\ \vdots \\ f\_n(\hat{u}^\star) \end{bmatrix} - \mathcal{D}\nu\_\mathcal{K}^\star = \mathcal{D}\nu\_\mathcal{I}^\star. \tag{B.17}$$

This completes the proof.

In summary, the distributed consensus-based controller (B.14) enables each agent to calculate a local estimator of the global variable u, which is communicated among agents in a distributed sense by using the communication structure represented by D.

**Remark B.2.** *If* (B.13) *has* υ > 0 *equilibria, then the distributed controller* (B.14) *also has* υ *equilibria. This fact can be proven by formulationg the proof of Lemma B.1 separately for each equilibrium* u ⋆ 1 , . . . , u<sup>⋆</sup> n *of* (B.13)*.* ♢

**Remark B.3.** *The results of Lemma B.1 remain the same if the incidence matrix* D *in* (B.14) *is replaced by an arbitrary Laplacian matrix* L *(cf. Definition A.11).* ♢

# **B.3 Extension to Nonconvex Cost Functions**

Strict convexity of Cg,i is an assumption that is very common in the literatur for simplification and regularity reasons. However, with a closer look, certain types of power plants exhibit a more delicate cost structure, whereby the cost function can be divided into three major areas which are caused by different levels of efficiency, see solid line in Fig. B.2a: The first area is characterized by relatively low efficiency, which is reflected by a rather steep increase of Cg,i(pg,i) per increment of power. What follows is a region of high efficiency, which causes a flattening of Cg,i(pg,i). At even higher levels of power, it becomes increasingly difficult (that is, increasingly expensive) to supply further amounts of power, with the consequence that Cg,i(pg,i) eventually rises with increasing steepness.

Let the (unique) inflection point of Cg,i(pg,i) be denoted by p ◦ g,i <sup>81</sup>. Then, also p ◦ <sup>g</sup>,i is an inflection point of Pg,i(pg,i). Now assume that the current price λ<sup>i</sup> fulfills λ<sup>i</sup> ≥ ∇Cg,i(p ◦ <sup>g</sup>,i) and define by p ♯ <sup>g</sup>,i := arg max Pg,i under the current price λ<sup>i</sup> . Then, it is rational for each profit-oriented

<sup>81</sup> This point corresponds to the operating point of the power plant with highest efficiency.

WP to apply a *virtual cost function* C˘g,i, which is strictly convex around p ♯ <sup>g</sup>,i in order to guarantee global convergence to the own profit maximum. If however λ<sup>i</sup> ≥ ∇Cg,i(p ◦ <sup>g</sup>,i), it is advisable for WPs to leave the market, since otherwise any nonzero power supply would result in a negative profit.

To obtain a continuously differentiable, strictly convex upper envelope of Cg,i, it is thus a reasonable choice to choose the virtual cost function equal to

$$\check{\mathbf{C}}\_{\mathbf{g},i} = \begin{cases} \mathbf{C}\_{\mathbf{g},i}, & p\_{\mathbf{g},i} \ge p\_{\mathbf{g},i}^{\diamond}, \\ p\_{\mathbf{g},i}^{2} - (p\_{\mathbf{g},i}^{\diamond})^{2} - \frac{m^{\diamond}}{2p\_{\mathbf{g},i}^{\diamond}}(p\_{\mathbf{g},i} - p\_{\mathbf{g},i}^{\diamond}) + \mathbf{C}\_{\mathbf{g},i}^{\diamond}, & p\_{\mathbf{g},i} < p\_{\mathbf{g},i}^{\diamond}, \end{cases} \tag{B.18}$$

where m◦ = ∇Cg,i(p ◦ <sup>g</sup>,i) denotes the steepness of Cg,i(pg,i) at the inflection point. We motivate this choice by a small example.

#### **Example 9**

*Consider the cost function*

$$\mathfrak{C}\_{\mathbf{g},i}(p\_{\mathbf{g},i}) = \frac{1}{10} \cdot \left( (p\_{\mathbf{g},i} - 2)^3 + (p\_{\mathbf{g},i} - 2)^2 + 2(p\_{\mathbf{g},i} - 2) + 8 \right). \tag{\mathbb{B}.19}$$

*Here,* p ◦ <sup>g</sup>,i = 5 3 *,* C ◦ <sup>g</sup>,i = 27 <sup>20</sup> *, and* m◦ <sup>i</sup> = 1 6 *. Fig. B.2 shows the curves of* Cg,i *and* C˘g,i *as well as the resulting profit functions* Pg,i *for different prices* λ<sup>i</sup> ∈ [5, 25]*. It can be seen that the replacement of* Cg,i *by the strictly convex, continuously differentiable function*

$$\mathbf{\breve{C}}\_{\mathbf{g},i} = \begin{cases} \mathbf{C}\_{\mathbf{g},i}, & p\_{\mathbf{g},i} \ge \frac{10}{6}, \\ p\_{\mathbf{g},i}^2 - \frac{1}{20} p\_{\mathbf{g},i} + \frac{509}{108}, & p\_{\mathbf{g},i} < \frac{10}{6} \end{cases} \tag{\text{B.20}}$$

*does not affect the respective value of the best response* p ♯ g,i*.*

**Figure B.2:** Plot of the nonconvex cost and profit functions (solid lines) and their respective "virtual" substitutes resulting from convexification (dashed lines). Notably, the convexified profit functions exhibit the same maxima as their nonconvex counterparts.

## **B.4 Parameter Values for Example 4**

The units of the parameters are given in p.u., except T ′ d0,i which is given in seconds.


**Table B.2:** Numerical values of the line parameters used in Example 4.


# **C Supplementary Material to** *Real-Time Incentives by Zonal Pricing* **(Chapter 4)**

## **C.1 Proof of Theorem 4.25**

*Proof.* Recall the physical plant system (3.121) and define K := (diag(κ))−<sup>1</sup> . Now consider the Lyapunov function candidate

<sup>V</sup> (xe) = <sup>H</sup>e<sup>p</sup>(xe<sup>p</sup>) + <sup>1</sup> 2 (peg ) <sup>T</sup>Kτ <sup>g</sup>pe<sup>g</sup> <sup>+</sup> <sup>µ</sup><sup>e</sup> T g+Kτ <sup>µ</sup>g+ <sup>µ</sup>eg+ <sup>+</sup> <sup>µ</sup><sup>e</sup> T <sup>g</sup>−Kτ <sup>µ</sup>g<sup>−</sup> <sup>µ</sup>e<sup>g</sup><sup>−</sup> + λe T <sup>τ</sup> <sup>λ</sup>λ<sup>e</sup> <sup>+</sup> <sup>ν</sup><sup>e</sup> T <sup>τ</sup> <sup>ν</sup>ν<sup>e</sup> <sup>+</sup> <sup>U</sup><sup>e</sup> T <sup>f</sup> <sup>τ</sup> <sup>U</sup><sup>f</sup> <sup>U</sup><sup>e</sup> <sup>f</sup> <sup>+</sup> <sup>µ</sup><sup>e</sup> T <sup>S</sup><sup>+</sup><sup>τ</sup> <sup>µ</sup>S<sup>+</sup> <sup>µ</sup>eS<sup>+</sup> <sup>+</sup> <sup>µ</sup><sup>e</sup> T S−<sup>τ</sup> <sup>µ</sup>S− <sup>µ</sup>eS− <sup>+</sup> <sup>U</sup><sup>e</sup> T <sup>I</sup><sup>τ</sup> <sup>U</sup><sup>I</sup> <sup>U</sup><sup>e</sup> <sup>I</sup> <sup>+</sup> <sup>µ</sup><sup>e</sup> T <sup>I</sup><sup>+</sup><sup>τ</sup> <sup>µ</sup>I<sup>+</sup> <sup>µ</sup>eI<sup>+</sup> <sup>+</sup> <sup>µ</sup><sup>e</sup> T I−<sup>τ</sup> <sup>µ</sup>I− <sup>µ</sup>eI− , (C.1)

where <sup>H</sup>e<sup>p</sup>(xe<sup>p</sup>) = <sup>H</sup>p(xp) <sup>−</sup> (xe<sup>p</sup>) <sup>T</sup>∇Hp(<sup>x</sup> ⋆ p ) − Hp(x ⋆ p ) is defined as in the proof of Lemma 3.39. Note that the Hessian of <sup>H</sup>e<sup>p</sup>(xe<sup>p</sup>) at <sup>x</sup>e<sup>p</sup> <sup>=</sup> **<sup>0</sup>** is equal to the Hessian of <sup>H</sup>p(xp) at <sup>x</sup><sup>p</sup> <sup>=</sup> <sup>x</sup> ⋆ p and thus positive definite by Assumption 3.33. Furthermore, since <sup>H</sup>e<sup>p</sup>(**0**) = 0 there exists a neighborhood around <sup>x</sup>e<sup>p</sup> <sup>=</sup> **<sup>0</sup>** with <sup>H</sup>e<sup>p</sup>(xe<sup>p</sup>) <sup>≻</sup> <sup>0</sup>. As all terms in (C.1) except <sup>H</sup>e<sup>p</sup>(xe<sup>p</sup>) are quadratic forms with positive metrics, <sup>H</sup>e<sup>p</sup>(xe<sup>p</sup>) <sup>≻</sup> <sup>0</sup> also implies that <sup>V</sup> (xe) <sup>≻</sup> <sup>0</sup> holds in a (possibly larger) neighborhood around <sup>x</sup><sup>e</sup> <sup>=</sup> **<sup>0</sup>**.

The derivative of <sup>V</sup> (xe) along the trajectories of (3.49), (3.110) equals

<sup>V</sup>˙ (xe) = <sup>z</sup><sup>e</sup> T <sup>p</sup> <sup>J</sup>pze<sup>p</sup> <sup>−</sup> <sup>z</sup><sup>e</sup> T p (Rp(xp) − Rp(x ⋆ p )) + <sup>z</sup><sup>e</sup> T <sup>p</sup>Gpue<sup>p</sup> − pe T <sup>g</sup>K(∇C(p<sup>g</sup> ) − ∇C(p ⋆ g )) <sup>−</sup> <sup>p</sup><sup>e</sup> T <sup>g</sup> <sup>ω</sup><sup>e</sup> <sup>g</sup> <sup>−</sup> <sup>p</sup><sup>e</sup> T <sup>g</sup>Kµeg+ + pe T <sup>g</sup>Kµe<sup>g</sup><sup>−</sup> <sup>+</sup> <sup>µ</sup><sup>e</sup> T g+K⟨p<sup>g</sup> − p<sup>g</sup> ⟩ + µg+ <sup>+</sup> <sup>µ</sup><sup>e</sup> T <sup>g</sup>−K⟨p<sup>g</sup> − p<sup>g</sup> ⟩ + µg<sup>−</sup> + λe T <sup>φ</sup><sup>e</sup> <sup>−</sup> <sup>U</sup><sup>e</sup> T <sup>f</sup> <sup>U</sup><sup>e</sup> <sup>f</sup> <sup>−</sup> <sup>U</sup><sup>e</sup> T <sup>f</sup> <sup>µ</sup>eS<sup>+</sup> <sup>+</sup> <sup>U</sup><sup>e</sup> T <sup>f</sup> <sup>µ</sup>eS− <sup>+</sup> <sup>µ</sup>eS<sup>+</sup>⟨U<sup>f</sup> <sup>−</sup> <sup>U</sup><sup>f</sup> ⟩ + µS<sup>+</sup> <sup>+</sup> <sup>µ</sup>eS−⟨U<sup>f</sup> <sup>−</sup> <sup>U</sup>f⟩ + µS− <sup>−</sup> <sup>U</sup><sup>e</sup> T <sup>I</sup>U<sup>e</sup> <sup>I</sup> <sup>−</sup> <sup>U</sup><sup>e</sup> T <sup>I</sup>µeI<sup>+</sup> <sup>+</sup> <sup>U</sup><sup>e</sup> T <sup>I</sup>µeI− <sup>+</sup> <sup>µ</sup>eI−⟨U<sup>I</sup> <sup>−</sup> <sup>U</sup>I⟩ + µI− <sup>+</sup> <sup>µ</sup>eI−⟨U<sup>I</sup> <sup>−</sup> <sup>U</sup>I⟩ + µI− . (C.2)

Rearrangement of the expressions in (C.2) yields

$$\dot{V}(x) = W\_1(x) + W\_2(x) + W\_3(x) + W\_4(x) + W\_5(x),\tag{C.3}$$

where

$$W\_1(\mathbf{z}) = \widetilde{\mathbf{z}}\_\mathbf{p}^\mathsf{T} \mathbf{J}\_\mathbf{p} \widetilde{\mathbf{z}}\_\mathbf{p},\tag{\text{C.4}}$$

$$W\_2(\mathbf{z}) = -\widetilde{\mathbf{z}}\_{\mathbf{p}}^{\sf T}(\mathcal{R}\_{\mathbf{p}}(\mathbf{z}\_{\mathbf{p}}) - \mathcal{R}\_{\mathbf{p}}(\mathbf{z}\_{\mathbf{p}}^{\star})) + \widetilde{\boldsymbol{\lambda}}^{\sf T}\widetilde{\boldsymbol{\varphi}},\tag{C.5}$$

$$W\_3(\mathbf{z}) = \widetilde{\mathbf{z}}\_\mathbf{p}^\mathsf{T} \mathbf{G}\_\mathbf{p} \widetilde{\mathbf{u}}\_\mathbf{p} - \widetilde{\mathbf{p}}\_\mathbf{g}^\mathsf{T} \widetilde{\boldsymbol{\omega}}\_\mathbf{g},\tag{\text{C.6}}$$

$$\begin{split} W\_{4}(\mathbf{z}) &= -\widetilde{\mathbf{p}}\_{\mathbf{g}}^{\sf T} \mathbf{K} (\nabla \mathbf{C}(\mathbf{p}\_{\mathbf{g}}) - \nabla \mathbf{C}(\mathbf{p}\_{\mathbf{g}}^{\sf \star})) - \widetilde{\mathbf{U}}\_{\mathbf{f}}^{\sf T} \widetilde{\mathbf{U}}\_{\mathbf{f}} - \widetilde{\mathbf{U}}\_{\mathbf{f}}^{\sf T} \widetilde{\mathbf{U}}\_{\mathbf{f}}, \\ W\_{5}(\mathbf{z}) &= -\widetilde{\mathbf{p}}^{\sf T} \mathbf{K} \widetilde{\mathbf{u}}\_{\mathbf{f}} + \widetilde{\boldsymbol{\mu}}^{\sf T} \mathbf{K} / \mathbf{n}\_{-} - \overline{\mathbf{n}} \ \rangle^{+} \end{split} \tag{C.7}$$

<sup>W</sup>5(x) = <sup>−</sup>p<sup>e</sup> <sup>g</sup>Kµeg+ <sup>+</sup> <sup>µ</sup><sup>e</sup> g+K⟨p<sup>g</sup> − p<sup>g</sup> ⟩ µg+ + pe T <sup>g</sup>Kµe<sup>g</sup><sup>−</sup> <sup>+</sup> <sup>µ</sup><sup>e</sup> T <sup>g</sup>−K⟨p<sup>g</sup> − p<sup>g</sup> ⟩ + µg<sup>−</sup> <sup>−</sup> <sup>U</sup><sup>e</sup> T <sup>f</sup> <sup>µ</sup>eS<sup>+</sup> <sup>+</sup> <sup>µ</sup>eS<sup>+</sup>⟨U<sup>f</sup> <sup>−</sup> <sup>U</sup>f⟩ + µS<sup>+</sup> <sup>+</sup> <sup>U</sup><sup>e</sup> T <sup>f</sup> <sup>µ</sup>eS− <sup>+</sup> <sup>µ</sup>eS−⟨U<sup>f</sup> <sup>−</sup> <sup>U</sup>f⟩ + µS− <sup>−</sup> <sup>U</sup><sup>e</sup> T <sup>I</sup>µeI<sup>+</sup> <sup>+</sup> <sup>µ</sup>eI<sup>+</sup>⟨U<sup>I</sup> <sup>−</sup> <sup>U</sup>I⟩ + µI<sup>+</sup> <sup>+</sup> <sup>U</sup><sup>e</sup> T <sup>I</sup>µeI− <sup>+</sup> <sup>µ</sup>eI−⟨U<sup>I</sup> <sup>−</sup> <sup>U</sup>I⟩ + µI− . (C.8)

Due to skew-symmetry of J<sup>p</sup> it holds that W1(x) = 0. The term W2(x) is less than or equal to zero whenever the conditions of Lemmas 3.39 and 3.41 are fulfilled. Moreover, with ze T <sup>p</sup>G<sup>p</sup> <sup>=</sup> <sup>∇</sup>Hp(L) = <sup>ω</sup><sup>g</sup> <sup>=</sup> <sup>ω</sup><sup>e</sup> <sup>g</sup>, we get <sup>W</sup>3(x) = 0. Due to (strict) convexity of <sup>C</sup>g,i(pg,i), the relationship

$$-\frac{1}{\kappa\_i}(p\_{\mathbf{g},i} - p\_{\mathbf{g},i}^{\star})(\nabla \mathsf{C}\_{\mathbf{g},i}(p\_{\mathbf{g},i}) - \nabla \mathsf{C}\_{\mathbf{g},i}(p\_{\mathbf{g},i}^{\star})) < 0\tag{\text{C.9}}$$

holds for each i ∈ Vg. Thus, the first summand in W4(x) is less than or equal to zero. The same holds for the quadratic terms −(U<sup>f</sup> − U ⋆ f ) <sup>T</sup>(U<sup>f</sup> <sup>−</sup> <sup>U</sup> ⋆ f ) and −(U<sup>I</sup> − U ⋆ I ) <sup>T</sup>(U<sup>I</sup> <sup>−</sup> <sup>U</sup> ⋆ I ), which yields W4(x) ≤ 0.

Finally, since the inequalities <sup>µ</sup><sup>e</sup> T ⟨g⟩ + <sup>µ</sup> <sup>≤</sup> <sup>µ</sup><sup>e</sup> T <sup>g</sup> and <sup>µ</sup><sup>e</sup> T g <sup>⋆</sup> <sup>≤</sup> **<sup>0</sup>** hold for each convex function g = coli{gi} [SDv15, Proposition 3] (see also the proofs of Lemma 3.41 and Theorem 3.44), it can be concluded that

$$(\mu\_{x\_i+} - \mu\_{x\_i+}^\star) \langle x\_i - \overline{x}\_i \rangle\_{\mu\_{x\_i+}}^+ \le (\mu\_{x\_i+} - \mu\_{x\_i+}^\star)(x\_i - x\_i^\star) \tag{C.10}$$

and

$$(\mu\_{x\_i-} - \mu\_{x\_i-}^\star) \langle \underline{x}\_i - x\_i \rangle\_{\mu\_{x\_i-}}^+ \le -(\mu\_{x\_i-} - \mu\_{x\_i-}^\star)(x\_i - x\_i^\star). \tag{C.11}$$

Accordingly, each row in (C.8) is less than or equal to zero.

All in all, it follows that <sup>V</sup>˙ <sup>≤</sup> <sup>0</sup> holds in a neighborhood around <sup>x</sup> ⋆ , thus x ⋆ is stable in the sense of Lyapunov [Kha02, Theorem 4.1].

# **D Supplementary Material to** *Optimal Control of Port-Hamiltonian Systems* **(Chapter 5)**

## **D.1 Single-Player Case**

H(x) **as Natural Control-Lyapunov Function Candidate**

**Lemma D.1.** *Consider the purely dissipative input-state-output PHS*

$$x = -\mathcal{R}(x)\frac{\partial H(x)}{\partial x} + G(x)u,\tag{D.1a}$$

$$y = G^{\top}(x)\frac{\partial H(x)}{\partial x} \tag{D.1b}$$

*with* R(x) = R T (x) <sup>≽</sup> <sup>0</sup> *and* <sup>H</sup>(x) <sup>≻</sup> <sup>0</sup>*. Then* (D.1) *is asymptotically stabilizable at the origin by means of a state feedback* u(x)*, if and only if* H(x) *is a CLF for* (D.1)*.*

*Proof.* Let <sup>g</sup>(x) <sup>∈</sup> <sup>R</sup><sup>n</sup> be an arbitrary but fixed vector. Then for all <sup>k</sup>(x) <sup>∈</sup> <sup>R</sup><sup>n</sup>\{**0**} it trivially holds that k T (x)g(x) = 0 is equivalent to g(x) = **0**. Negating this statement yields

$$\exists \; k(x) \in \mathbb{R}^n \; : \; k^\top(x)g(x) > 0 \qquad \Longleftrightarrow \qquad \qquad g^\top(x)g(x) > 0. \tag{D.2}$$

By choosing g(x) = G<sup>⊤</sup>(x)∇H(x), with G(x) and H(x) as in (D.1), equation (D.2) reads

$$\exists \, k(x) \in \mathbb{R}^n \colon \, k^\top(x) \mathbf{G}^\top(x) \nabla H(x) > 0 \iff \left(\nabla H(x)\right)^\top \mathbf{G}(x) \mathbf{G}^\top(x) \nabla H(x) > 0. \tag{\text{D.3}}$$

Alternatively, choosing g(x) = p R(x)∇H(x) with R(x) = R T (x) ≽ 0, (D.2) takes the form

$$\exists \, \hat{k}(x) \in \mathbb{R}^n \, : \, \hat{k}^\top(x) \sqrt{\mathcal{R}(x)} \nabla H(x) > 0 \quad \Longleftrightarrow \quad (\nabla H(x))^\top \mathcal{R}(x) \nabla H(x) > 0. \tag{D.4}$$

Now we restrict the vector k(x) in (D.2) to the form k(x) = G<sup>T</sup> (x)∇Ve(x) and <sup>k</sup>ˆ(x) to kˆ(x) = p <sup>R</sup>(x)∇V<sup>ˆ</sup> (x) with <sup>V</sup>e(x), <sup>V</sup><sup>ˆ</sup> (x) <sup>≻</sup> <sup>0</sup>, which leads to

$$\overbrace{\exists \, \mathring{V}(\boldsymbol{x}) > 0 : \left(\boldsymbol{\nabla}\mathring{V}(\boldsymbol{x})\right)^{\sf T} \boldsymbol{G}(\boldsymbol{x}) \boldsymbol{G}^{\sf T}(\boldsymbol{x}) \boldsymbol{\nabla}H(\boldsymbol{x}) > 0}^{\mathbbm{x}\_{1}} \implies \overbrace{\left(\boldsymbol{\nabla}H(\boldsymbol{x})\right)^{\sf T} \boldsymbol{G} \boldsymbol{G}^{\sf T} \boldsymbol{\nabla}H(\boldsymbol{x}) > 0,\tag{\text{D.5a}}}\_{\text{(D.5a)}}$$

$$\overbrace{\exists \, \mathring{V}(\boldsymbol{x}) > 0 : \left(\boldsymbol{\nabla}\mathring{V}(\boldsymbol{x})\right)^{\sf T} \boldsymbol{R}(\boldsymbol{x}) \boldsymbol{\nabla}H(\boldsymbol{x}) > 0 \implies \overbrace{\left(\boldsymbol{\nabla}H(\boldsymbol{x})\right)^{\sf T} \boldsymbol{R}(\boldsymbol{x}) \boldsymbol{\nabla}H(\boldsymbol{x}) > 0}\_{\text{(D.5b)}}.\tag{\text{D.5b}}$$

Conjunction of the two statements in (D.5) and contraposition yields X<sup>2</sup> ∧ X<sup>4</sup> =⇒ X<sup>1</sup> ∧ X3, which can be written as

$$\left(\nabla H(\mathbf{z})\right)^{\mathsf{T}} \left(\mathbf{R}(\mathbf{z}) + \mathbf{G}(\mathbf{z})\mathbf{G}^{\mathsf{T}}(\mathbf{z})\right) \nabla H(\mathbf{z}) = 0 \qquad \Longrightarrow \qquad \overline{\mathbf{x}\_{1}} \wedge \overline{\mathbf{x}\_{3}}.\tag{\mathsf{D.6}}$$

The left-hand side of (D.6) holds, if and only if H(x) is *no* CLF for the PHS (D.1). The right-hand side of (D.6) implies that there also exists no <sup>V</sup> (x) <sup>≻</sup> <sup>0</sup> with (∇<sup>V</sup> (x))<sup>T</sup>(R(x) + G(x)G<sup>T</sup> (x))∇H(x), i.e. there exists no CLF for the PHS (D.1). Thus, we conclude that there exists no CLF for (D.1), if H(x) is no CLF for (D.1). The inverse statement is trivial: If there exists no CLF for (D.1), then also H(x) is no CLF for (D.1). By Artstein's theorem [Art83] it is known that the existence of a CLF is necessary and sufficient for the asymptotic stabilizability of (D.1) at the origin. Hence we conclude that H(x) being a CLF is necessary and sufficient for asymptotic stabilizability of (D.1) at the origin

#### **Openness of** W<sup>+</sup>

**Lemma D.2.** <sup>W</sup><sup>+</sup> *is an open set, i.e. for each* <sup>w</sup> <sup>∈</sup> <sup>W</sup><sup>+</sup> *there exists an* ε > <sup>0</sup> *such that each* <sup>w</sup>′ <sup>∈</sup> <sup>R</sup><sup>r</sup> *with* <sup>∥</sup><sup>w</sup> <sup>−</sup> <sup>w</sup>′<sup>∥</sup> < ε *lies within* <sup>W</sup><sup>+</sup>*.*

*Proof.* To prove that W<sup>+</sup> is an open set, two auxiliary sets Y<sup>+</sup> and Y + cyl are introduced which can be proven to be open more conveniently. Finally, openness of W<sup>+</sup> is concluded by a canonical projection of Y + cyl:

Let <sup>Y</sup><sup>+</sup> := {(x, <sup>w</sup>) <sup>∈</sup> <sup>R</sup><sup>n</sup> <sup>×</sup> <sup>R</sup><sup>r</sup> : V (x, w) > 0} be the set of all (x, w) with V (x, w) > 0. Then Y<sup>+</sup> is the preimage of R<sup>+</sup>, i.e. Y<sup>+</sup> = V −1 (R<sup>+</sup>). Each preimage of a continuous function is open whenever the corresponding image is open [Cro05, Theorem 2.9]. Since V (x, w) is continuous and R<sup>+</sup> is open, Y<sup>+</sup> must also be open. Furthermore, we note that Y<sup>+</sup> is nonempty due to the fact that V (x, **0**r) = H(x) > 0 is fulfilled by definition for all <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>\{**0**}. Now consider the <sup>Y</sup><sup>+</sup>*-inner cylinder* (see Fig. D.1) with <sup>Y</sup> + cyl(w, w) := {(x, <sup>w</sup>) <sup>∈</sup> <sup>Y</sup><sup>+</sup> : <sup>w</sup> <sup>∈</sup>]w, <sup>w</sup>[,(xe, <sup>w</sup>) <sup>∈</sup> <sup>Y</sup><sup>+</sup> <sup>∀</sup> <sup>x</sup><sup>e</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>\{**0**}}. Obviously, <sup>Y</sup> + cyl(w, <sup>w</sup>) <sup>⊆</sup> <sup>Y</sup><sup>+</sup>, and according to Lemma D.3 (see below), Y + cyl(w, <sup>w</sup>) is open for each pair (w, <sup>w</sup>) <sup>∈</sup> <sup>R</sup><sup>r</sup> <sup>×</sup>R<sup>r</sup> .

**Figure D.1:** Y<sup>+</sup> with inner cylinder Y + cyl(w, w) ⊆ Y+.

**Figure D.2:** Each inner cylinder Y + cyl(w, w) is an open set.

With the help of Y + cyl(w, <sup>w</sup>), we can define the *maximum* <sup>Y</sup><sup>+</sup>*-inner cylinder*

$$\widehat{\mathbb{Y}}\_{\text{cyl}}^{+} \coloneqq \bigcup\_{\underline{\mathfrak{w}}, \overline{\mathfrak{w}} \in \mathbb{R}^{r}} \mathbb{Y}\_{\text{cyl}}^{+}(\underline{\mathfrak{w}}, \overline{\mathfrak{w}}) \tag{D.7}$$

as the union of all possible cylinders Y + cyl(w, w), see Fig. D.1. Since the union of open sets is open [Sin19, Theorem 1.1.9], <sup>Y</sup><sup>b</sup> <sup>+</sup> cyl is open as well. With the canonical projection proj : <sup>R</sup><sup>n</sup> <sup>×</sup>R<sup>r</sup> <sup>→</sup> <sup>R</sup><sup>r</sup> , the set <sup>W</sup><sup>+</sup> can be interpreted as <sup>W</sup><sup>+</sup> = proj(Y<sup>b</sup> <sup>+</sup> cyl), i.e. the canonical projection of <sup>Y</sup><sup>b</sup> <sup>+</sup> cyl in <sup>R</sup><sup>r</sup> . Since projection maps are open maps [Deo18, p. 5], W<sup>+</sup> is an open set.

**Lemma D.3.** *The* Y<sup>+</sup>*-inner cylinder* Y + cyl(w, w) := (x, <sup>w</sup>) <sup>∈</sup> <sup>Y</sup> <sup>+</sup> : <sup>w</sup> <sup>∈</sup>]w, <sup>w</sup>[,(xe, <sup>w</sup>) <sup>∈</sup> <sup>Y</sup> <sup>+</sup> <sup>∀</sup> <sup>x</sup><sup>e</sup> <sup>∈</sup> <sup>R</sup> n \{**0**} *with* <sup>w</sup>, <sup>w</sup> <sup>∈</sup> <sup>R</sup><sup>r</sup> *is an open set.*

*Proof.* If <sup>w</sup> <sup>≥</sup> <sup>w</sup>, then the interval ]w, <sup>w</sup>[ is improper, thus <sup>Y</sup> + cyl(w, w) = ∅. Since empty sets are trivially open, the proof is complete. If w < w, let (x ′ , w′ ) <sup>∈</sup> <sup>Y</sup> + cyl(w, w) be an arbitrary point within the cylinder (see Fig. D.2). Now define

$$\mathbf{x}^{\prime\prime} = \text{col}\_{i}\{\|x\_{i}^{\prime}\|\}, \tag{\text{\textquotedblleft}}, \qquad \qquad \qquad \qquad i = 1, \ldots, n,\tag{\text{\textquotedblleft}}$$

$$\mathbf{w}''' = \text{col}\_j \{ \min \{ \overline{w}\_j - w\_j', w\_j' - \underline{w}\_j \} \}, \qquad \qquad j = 1, \ldots, r,\tag{\text{D.9}}$$

where x ′′ denotes the component-wise distances between x ′ i and 0, and w′′ denotes the component-wise distances between w ′ j and the lower or upper bounds w<sup>j</sup> or w<sup>j</sup> , respectively.

w

From Fig. D.2 it can be seen that by definition of x ′′ and w′′, it holds that

$$(x' + \frac{1}{2}x'', w') \in \text{int}\,\mathbb{Y}\_{\text{cyl}}^{+}(\underline{w}, \overline{w}),\tag{D.10}$$

$$(x' - \frac{1}{2}x'', w') \in \text{int}\,\mathbb{Y}\_{\text{cyl}}^{+}(\underline{w}, \overline{w}),\tag{D.11}$$

$$(x', w' + \frac{1}{2}w'') \in \text{int}\,\mathbb{Y}\_{\text{cyl}}^{+}(\underline{w}, \overline{w}),\tag{D.12}$$

$$(x', w' - \frac{1}{2}w'') \in \text{int}\,\mathbb{Y}\_{\text{cyl}}^{+}(\underline{w}, \overline{w}).\tag{D.13}$$

Moreover, it can be seen that both the "upper" and "lower" subset of Y + cyl(w, w) are convex sets. Thus, it is obvious that we can always construct an open ball B ′ ((x ′ , w′ ), ε) around (x ′ , w′ ) with radius

$$\varepsilon = \frac{1}{2} \cdot \left\| \begin{bmatrix} x''\\ w'' \end{bmatrix} \right\|\_{\infty} \tag{D.14}$$

that lies completely in Y + cyl(w, <sup>w</sup>). Hence, <sup>Y</sup> + cyl(w, w) is an open set.

# **D.2 Multi-Player Case**

#### **Proof of Lemma 5.43**

*Proof.* For linear ISO-PHSs, it holds that

$$\mathcal{X}\_{G\_i} = \{x \in \mathbb{R}^n \, : \, G\_i^{\sf T} Q x = \mathbb{0}\} = \ker(G\_i^{\sf T} Q). \tag{D.15}$$

Moreover, in [HJ12, Observation 7.1.6] it is stated that for each positive semidefinite matrix <sup>A</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>×<sup>n</sup>, the equality <sup>x</sup> <sup>T</sup>Ax = 0 holds, if and only if <sup>x</sup> <sup>∈</sup> ker(A). Thus for each <sup>i</sup> <sup>∈</sup> <sup>P</sup>, condition (5.108) is equivalent to

$$\ker(\mathbf{G}\_i^\mathsf{T}\mathbf{Q}) \cap \ker(\mathbf{Q}^\mathsf{T}\mathbf{R}\mathbf{Q}) \cap \left(\bigcap\_{\substack{j=1\\j\neq i}}^N \ker(\mathbf{\Upsilon}\_j \cdot \mathbf{Q}^\mathsf{T}\mathbf{G}\_j \mathbf{S}\_j^{-1} \mathbf{G}\_j^\mathsf{T}\mathbf{Q})\right) = \emptyset. \tag{\text{D.16}}$$

Due to the fact that Υ<sup>j</sup> > 0 and ker(G<sup>T</sup> <sup>i</sup> Q) ⊆ ker(Q <sup>T</sup>GiS −1 <sup>i</sup> G<sup>T</sup> <sup>i</sup> Q), (D.16) can be written as follows:

$$\begin{split} & \bigcup\_{i=1}^{N} \left( \ker(\mathbf{G}\_{i}^{\mathsf{T}} \mathbf{Q}) \cap \ker(\mathbf{Q}^{\mathsf{T}} \mathbf{R} \mathbf{Q}) \cap \left( \bigcap\_{\begin{subarray}{c} j=1\\ j\neq i \end{subarray}}^{N} \ker(\mathbf{Q}^{\mathsf{T}} \mathbf{G}\_{j} \mathbf{S}\_{j}^{-1} \mathbf{G}\_{j}^{\mathsf{T}} \mathbf{Q}) \right) \right) \\ &= \ker(\mathbf{Q}^{\mathsf{T}} \mathbf{R} \mathbf{Q}) \cap \bigcup\_{i=1}^{N} \left( \ker(\mathbf{G}\_{i}^{\mathsf{T}} \mathbf{Q}) \cap \left( \bigcap\_{\begin{subarray}{c} j=1\\ j\neq i \end{subarray}}^{N} \ker(\mathbf{Q}^{\mathsf{T}} \mathbf{G}\_{j} \mathbf{S}\_{j}^{-1} \mathbf{G}\_{j}^{\mathsf{T}} \mathbf{Q}) \right) \right) \\ &= \ker(\mathbf{Q}^{\mathsf{T}} \mathbf{R} \mathbf{Q}) \cap \left( \bigcap\_{i=1}^{N} \ker(\mathbf{G}\_{i}^{\mathsf{T}} \mathbf{Q}) \right) \cap \left( \bigcup\_{i=1}^{N} \left( \bigcap\_{\begin{subarray}{c} j=1\\ j\neq i \end{subarray}}^{N} \ker(\mathbf{Q}^{\mathsf{T}} \mathbf{G}\_{j} \mathbf{S}\_{j}^{-1} \mathbf{G}\_{j}^{\mathsf{T}} \mathbf{Q}) \right) \right) \\ &= \emptyset. \end{split} \tag{\mathbf{D},17}$$

With ker(QA) = ker A for each regular matrix Q together with Lemma D.4 (see below), (D.17) is equivalent to (5.117).

$$\textbf{Lemma D.4.}\text{ For each positive definite matrix }\textbf{C}\in\mathbb{R}^{N}\text{, it holds that}$$

$$\ker(\textbf{AC})\cap\ker(\textbf{BC})=\emptyset \qquad\Longleftrightarrow\qquad \ker(\textbf{A})\cap\ker(\textbf{B})=\emptyset.\tag{\text{D.18}}$$

*Proof.* The left-hand side of (D.18) can be written as follows

$$
\exists x \in \mathbb{R}^n \backslash \{0\}: \qquad \qquad (\mathbf{AC}x = \mathbf{0}) \wedge (\mathbf{BC}x = \mathbf{0}).\tag{\text{D.19}}
$$

With z := Cx this is equivalent to

$$
\partial \vec{\omega} \in \mathbb{R}^n \\
\langle \{0\} : \qquad \qquad \qquad (\mathbf{A} \mathbf{z} = \mathbf{0}) \wedge (\mathbf{B} \mathbf{z} = \mathbf{0}). \tag{\text{D.20}}
$$

Accordingly, (D.20) states that the kernels of A and B are disjoint. This is equivalent to the right-hand side of (D.18) .

#### **Proof of Corollary 5.45**

*Proof.* As a consequence of (5.94), it follows that V (x) is equivalent to the vector of value functions, if and only if <sup>Υ</sup>i(t) <sup>≡</sup> <sup>1</sup> for all <sup>i</sup> <sup>∈</sup> <sup>P</sup>. With (5.105), this means that

$$q\_i^2(\mathbf{z}) + q\_i^1(\mathbf{z}) + \sum\_{\substack{j=1 \\ j \neq i}}^N q\_{ij}^2(\mathbf{z}) + q\_i^0(\mathbf{z}) = 0,\tag{D.21}$$

which is equivalent to

$$\begin{split} & -\frac{1}{2} \left( \frac{\partial V\_{i}}{\partial x} \right)^{\mathsf{T}} \mathbf{G}\_{i}(\boldsymbol{x}) (\mathbf{S}\_{i}(\boldsymbol{x}))^{-1} \mathbf{G}\_{i}^{\mathsf{T}}(\boldsymbol{x}) \frac{\partial V\_{i}}{\partial \boldsymbol{x}} - \sum\_{\begin{subarray}{c} j=1\\ j \neq i \end{subarray}}^{N} \left( \frac{\partial V\_{i}}{\partial \boldsymbol{x}} \right)^{\mathsf{T}} \mathbf{G}\_{j}(\boldsymbol{x}) (\mathbf{S}\_{j}(\boldsymbol{x}))^{-1} \mathbf{G}\_{j}^{\mathsf{T}}(\boldsymbol{x}) \frac{\partial V\_{j}(\boldsymbol{x})}{\partial \boldsymbol{x}} \\ & + \left( \frac{\partial V\_{i}}{\partial \boldsymbol{x}} \right)^{\mathsf{T}} (\mathbf{J}(\boldsymbol{x}) - \mathbf{R}(\boldsymbol{x})) \frac{\partial H}{\partial \boldsymbol{x}} + \frac{1}{2} \ell\_{i}(\boldsymbol{x}) \\ & = 0. \end{split} \tag{D.22}$$

By inserting (5.118), (5.126), and multiplying by −1, we get

$$\begin{split} & \quad \left( \left( \frac{\partial H}{\partial x} \right)^{\mathsf{T}} + w\_{i}^{\mathsf{T}} \frac{\partial \Xi\_{i}}{\partial x} \right) \mathbf{K}\_{i}(x) \left( \frac{\partial H}{\partial x} + \left( \frac{\partial \Xi\_{i}}{\partial x} \right)^{\mathsf{T}} w\_{i} \right) \\ & \quad + 2 \sum\_{\begin{subarray}{c} j=1 \\ j \neq i \end{subarray}}^{N} \left( \left( \frac{\partial H}{\partial x} \right)^{\mathsf{T}} + w\_{i}^{\mathsf{T}} \frac{\partial \Xi\_{i}}{\partial x} \right) \mathbf{K}\_{j}(x) \left( \frac{\partial H}{\partial x} + \left( \frac{\partial \Xi\_{j}}{\partial x} \right)^{\mathsf{T}} w\_{j} \right) \\ & \quad - 2w\_{i}^{\mathsf{T}} \frac{\partial \Xi\_{i}}{\partial x} (\mathbf{J}(x) - \mathbf{R}(x)) \frac{\partial H}{\partial x} + 2 \left( \frac{\partial H}{\partial x} \right)^{\mathsf{T}} \mathbf{R}(x) \frac{\partial H}{\partial x} - \frac{1}{2} \ell\_{i}(x) \\ & = 0. \end{split} \tag{D.23}$$

Rearranged by w<sup>i</sup> , this yields

 w<sup>T</sup> i ∂Ξ<sup>i</sup> ∂x Ki(x) ∂Ξ<sup>i</sup> ∂x T w<sup>i</sup> ! + 2 ∂H ∂x T Ki(x) ∂Ξ<sup>i</sup> ∂x T w<sup>i</sup> ! + 2X j=1 j̸=i ∂H ∂x T + w<sup>T</sup> j ∂Ξ<sup>j</sup> ∂x ! K<sup>T</sup> j (x) ∂Ξ<sup>i</sup> ∂x T w<sup>i</sup> + 2 ∂H ∂x T (J(x) + <sup>R</sup>(x)) ∂Ξ<sup>i</sup> ∂x T w<sup>i</sup> + ∂H ∂x T Ki(x) ∂H ∂x + 2X N j=1 j̸=i ∂H ∂x T + w<sup>T</sup> j ∂Ξ<sup>j</sup> ∂x ! K<sup>T</sup> j (x) ∂H ∂x + 2 ∂H ∂x T R(x) ∂H ∂x − 1 2 ℓi(x) = 0, (D.24)

thus (D.24) can be written in the form (5.121)–(5.126).

# **E Supplementary Material to** *Case Studies* **(Chapter 6)**

## **E.1 Benchmark Model Parameters**


**Table E.1:** Numerical values of the nodal parameters used in Case Studies I and II.


**Table E.1:** Numerical values of the nodal parameters used in Case Studies I and II (*cont'd*)

# **References**

# **Public References**





































[ZW10] Zhang, Yue-Jun; Wei, Yi-Ming: An overview of current research on EU ETS: Evidence from its operating mechanism and economic effect. In: *Applied Energy* 87 (2010), No. 6, p. 1804–1814

# **Own Publications and Conference Proceedings**


## **Supervised Theses**


#### **Karlsruher Beiträge zur Regelungs- und Steuerungstechnik (ISSN 2511-6312) Institut für Regelungs- und Steuerungssysteme**



Band 17 Kölsch, Lukas Dynamic Incentives for Optimal Control of Competitive Power Systems. ISBN 978-3-7315-1209-7

# INSTITUT FÜR REGELUNGS- UND STEUERUNGSSYSTEME

The increasing dominance of renewable energy sources involves unprecedented challenges for future electricity markets. In particular, stronger intermittency of power generation requires prices to not only respond more quickly to the prevailing demand-supply situation, but also to better internalize physical grid constraints.

This work presents a feedback control framework for generating continuous price signals which fully reflect the spatio-temporal scarcity of electricity in real time. For this purpose, the techno-economic interplay between optimization dynamics of individual profit-maximizing market participants and physical dynamics of the underlying grid infrastructure is conflated to an overall white-box model in port-Hamiltonian form. Completely automatable deduction of nonlinear controller equations along with a novel Lyapunov-based stability proof allow to consolidate frequency and voltage stabilization with higher-level goals such as economic efficiency and optimal congestion management. Particular emphasis is placed on a fully distributed implementation of the control framework with pure neighbor-to-neighbor communication for increased robustness and plug-and-play capability.

In addition, this work provides a new class of adaptive controllers for solving the general case of multi-player dynamic games subject to nonlinear port-Hamiltonian system dynamics, followed by a mathematically rigorous analysis of the resulting closed-loop system.

Dynamic Incentives for Optimal Control of Competitive Power Systems

ISSN 2511-6312 ISBN 978-3-7315-1209-7