**Alexandra Silva K. Rustan M. Leino (Eds.)**

# LNCS 12759

# **Computer Aided Verification**

**33rd International Conference, CAV 2021 Virtual Event, July 20–23, 2021 Proceedings, Part I**

### Lecture Notes in Computer Science 12759

#### Founding Editors

Gerhard Goos Karlsruhe Institute of Technology, Karlsruhe, Germany Juris Hartmanis Cornell University, Ithaca, NY, USA

#### Editorial Board Members

Elisa Bertino Purdue University, West Lafayette, IN, USA Wen Gao Peking University, Beijing, China Bernhard Steffen TU Dortmund University, Dortmund, Germany Gerhard Woeginger RWTH Aachen, Aachen, Germany Moti Yung Columbia University, New York, NY, USA

More information about this subseries at http://www.springer.com/series/7407

Alexandra Silva • K. Rustan M. Leino (Eds.)

# Computer Aided Verification

33rd International Conference, CAV 2021 Virtual Event, July 20–23, 2021 Proceedings, Part I

Editors Alexandra Silva University College London London, UK

K. Rustan M. Leino Automated Reasoning Group | AWS Seattle, WA, USA

ISSN 0302-9743 ISSN 1611-3349 (electronic) Lecture Notes in Computer Science ISBN 978-3-030-81684-1 ISBN 978-3-030-81685-8 (eBook) https://doi.org/10.1007/978-3-030-81685-8

LNCS Sublibrary: SL1 – Theoretical Computer Science and General Issues

© The Editor(s) (if applicable) and The Author(s) 2021. This book is an open access publication.

Open Access This book is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this book are included in the book's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the book's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use.

The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland

### Preface

It was our privilege to serve as the program chairs for CAV 2021, the 33rd International Conference on Computer-Aided Verification. CAV 2021 was held as a virtual conference during July 20–23, 2021. The tutorial days were on July 19 and July 24, 2021, and the pre-conference workshops were held during July 18–19, 2021. Due to the COVID-19 outbreak, all events took place online.

CAV is an annual conference dedicated to the advancement of the theory and practice of computer-aided formal analysis methods for hardware and software systems. The primary focus of CAV is to extend the frontiers of verification techniques by expanding to new domains such as security, quantum computing, and machine learning. This puts CAV at the cutting edge of formal methods research, and this year's program is a reflection of this commitment.

CAV 2021 received a very high number of submissions (290). We accepted 16 tool papers, 3 case studies, and 60 regular papers, which amounts to an acceptance rate of roughly 27%. The accepted papers cover a wide spectrum of topics, from theoretical results to applications of formal methods. These papers apply or extend formal methods to a wide range of domains such as concurrency, machine learning, and industrially deployed systems. The program featured keynote talks by Loris D'Antoni (UW-Madison), Corina Pasareanu (NASA), and Anna Slobodova (Centaur Technology, Inc.) as well as invited tutorials by Nate Foster (Cornell University), Zak Kincaid (Princeton) together with Tom Reps (UW-Madison), and Nadia Polikarpova (UC San Diego). Furthermore, we continued the tradition of Logic Lounge, a series of discussions on computer science topics targeting a general audience.

In addition to the main conference, CAV 2021 hosted the following workshops: Formal Approaches to Certifying Compliance (FACC), Formal Methods for ML-Enabled Autonomous Systems (FoMLAS), Formal Methods for Blockchains (FMBC), Numerical Software Verification (NSV), Theory and Practice of String Solving (TPSS), Verifying Probabilistic Programs (VeriProP), Synthesis (SYNT), Satisfiability Modulo Theories (SMT), and Verification Mentoring Workshop (VMW).

Organizing a flagship conference like CAV requires a great deal of effort from the community. The Program Committee for CAV 2021 consisted of 79 members — a committee of this size ensures that each member has to review only a reasonable number of papers in the allotted time. In all, the committee members wrote over 900 reviews while investing significant effort to maintain and ensure the high quality of the conference program. We are grateful to the CAV 2021 Program Committee for their outstanding efforts in evaluating the submissions and making sure that each paper got a fair chance. Like last year's CAV, we made the artifact evaluation mandatory for tool paper submissions and optional, but encouraged, for the rest of the accepted papers. This year saw an unprecedented number of 66 artifact submissions. The Artifact Evaluation Committee consisted of 72 members who put in significant effort to evaluate each artifact. The goal of this process was to provide constructive feedback to tool developers and help make the research published in CAV more reproducible. We are also very grateful to the Artifact Evaluation Committee for their hard work and dedication in evaluating the submitted artifacts.

CAV 2021 would not have been possible without the tremendous help we received from several individuals, and we would like to thank everyone who helped make CAV 2021 a success. First, we would like to thank Clément Pit-Claudel and Maria Schett for chairing the Artifact Evaluation Committee and John Cyphert for putting together the proceedings. We also thank Arie Gurfinkel for chairing the workshop organization, Bor-Yuh Evan Chang for managing sponsorship, Thomas Wies for arranging student fellowships, Norine Coenen for handling publicity, Leopold Haller for organising the Logic Lounge, and Peter Müller for putting together the Ask me Anything program. We also thank Jean-Baptiste Jeannin and Arjun Radhakrishna for chairing the Mentoring Committee. Putting together an online conference is a complex task and we are grateful to the virtualization chair Tiago Ferreira, the student volunteer coordinators Tobias Kappé and Tao Gu, the local organizers for the Asia timezone, Ichiro Hasuo and Krishna S, and the team at Slides Live for all their efforts. Last but not least, we would like to thank the members of the CAV Steering Committee (Kenneth McMillan, Aarti Gupta, Orna Grumberg, and Daniel Kroening) for helping us with several important aspects of organizing CAV 2021.

We hope that you will find the proceedings of CAV 2021 scientifically interesting and thought-provoking!

June 2021 Alexandra Silva Rustan Leino

### Organization

#### Steering Committee


#### Conference Co-chairs


#### Artifact Co-chairs


### Workshop Chair


### Verification Mentoring Workshop Organizing Committee


#### Logic Lounge Organizer


#### Ask Me Anything Organizer

### Publicity Chair


Sagar Chaki Mentor Graphics, USA Jennifer Davis Collins Aerospace, USA Rajeev Joshi Amazon, USA K. Rustan M. Leino (Co-chair) Ruzica Piskac Yale University, USA

Bor-Yuh Evan Chang University of Colorado Boulder and Amazon, USA Hana Chockler King's College London, UK Cristina David University of Bristol, UK Yuxin Deng East China Normal University, China Rayna Dimitrova CISPA Helmholtz Center for Information Security, Germany Alastair Donaldson Imperial College London, UK Constantin Enea Université de Paris, France Joao Fernandes University of Porto, Portugal Bernd Finkbeiner CISPA Helmholtz Center for Information Security, Germany Vijay Ganesh University of Waterloo, Canada Pierre Ganty IMDEA Software Institute, Spain Aarti Gupta Princeton University, USA Arie Gurfinkel University of Waterloo, Canada Ichiro Hasuo National Institute of Informatics, Japan Marieke Huisman University of Twente, Netherlands David N. Jansen Institute of Software, Chinese Academy of Sciences, China Jean-Baptiste Jeannin University of Michigan, USA Ranjit Jhala University of California, San Diego, USA Temesghen Kahsai The University of Iowa, USA Benjamin Lucien Kaminski University College London, UK Joost-Pieter Katoen RWTH Aachen University, Germany Guy Katz The Hebrew University of Jerusalem, Israel Laura Kovacs Vienna University of Technology, Austria Mitja Kulczynski Kiel University, Germany Mohit Kumar Tekriwal University of Michigan, USA Orna Kupferman The Hebrew University of Jerusalem, Israel Marta Kwiatkowska University of Oxford, UK Shuvendu Lahiri Microsoft Research, USA Akash Lal Microsoft Research, India Kim Larsen Aalborg University, Denmark Marijana Lazic Technical University of Munich, Germany Owolabi Legunsen University of Illinois at Urbana-Champaign, USA Amazon, USA Rupak Majumdar Max Planck Institute for Software Systems, Germany

Ruben Martins Carnegie Mellon University, USA Ken McMillan University of Texas at Austin, USA Aina Niemetz Stanford University, USA Sylvie Putot Ecole Polytechnique, France


#### Artifact Evaluation Committee


Luke Geeson Arm, UK Julien Lepiller Yale University, USA Marcel Moosbrugger TU Wien, Austria Marianela Morales Inria, France

Isabel Garcia-Contreras IMDEA Software Institute and Universidad Politecnica de Madrid, Spain Nick Giannarakis University of Wisconsin-Madison, USA Pablo Gordillo Universidad Complutense de Madrid, Spain Laura Graves University of Waterloo, Canada Zheng Guo University of California, San Diego, USA Vedad Hadžić Graz University of Technology, Austria Miguel Isabel Universidad Politécnica de Madrid, Spain Anastasiia Izycheva Technical University of Munich, Germany Chris Jenkins University of Iowa, USA Daniela Kaufmann Johannes Kepler University Linz, Austria Brian Kempa Iowa State University, USA Bettina Könighofer Graz University of Technology, Austria Mitja Kulczynski Kiel University, Germany Mohit Kumar Tekriwal University of Michigan, USA Stella Lau Massachusetts Institute of Technology, USA Chunxiao Li University of Waterloo, Canada Junyi Liu Institute of Software, Chinese Academy of Sciences, China Debasmita Lohar Max Planck Institute for Software Systems, Germany Makai Mann Stanford University, USA Roy Margalit Tel Aviv University, Israel Sidi Mohamed Beillahi Université de Paris and CNRS, France Jasper Nalbach RWTH Aachen University, Germany Andres Noetzli Stanford University, USA Mário Pereira Universidade NOVA de Lisboa, Portugal Mateo Perez University of Colorado Boulder, USA Elizabeth Polgreen University of California, Berkeley, USA Mathias Preiner Stanford University, USA Tim Quatmann RWTH Aachen University, Germany Bob Rubbens University of Twente, Netherlands Vimala S. Indian Institute of Technology, Madras, India Philipp Schröer RWTH Aachen University, Germany Joseph Scott University of Waterloo, Canada Amanda Stjerna Uppsala University, Sweden Zachary Susag University of Wisconsin-Madison, USA Hira Syeda Chalmers Universityof Technology, Sweden Martin Tappler Graz University of Technology, Austria Michael Tautschnig Queen Mary University of London, UK Saeid Tizpaz Niari University of Texas at El Paso, USA Hazem Torfah University of California, Berkeley, USA Deivid Vale Radboud University Nijmegen, Netherlands

Masaki Waga Kyoto University, Japan Peixin Wang Shanghai Jiao Tong University, China Sarah Winkler Free University of Bozen-Bolzano, Italy Tobias Winkler RWTH Aachen University, Germany Ali Younes Bauman Moscow State University, Russia Xiao-Yi Zhang National Institute of Informatics, Japan Yuhao Zhang University of Wisconsin-Madison, USA

#### Additional Reviewers

Ahmad, Hammad An, Jie Armborst, Lukas Almagor, Shaull Arenas, Puri Asadi, Sepideh Amir, Guy Arif, Fareed Asarin, Eugene Baanen, Anne Batz, Kevin Berzish, Murphy Bacci, Giovanni Baumeister, Jan Blicha, Martin Balasubramanian, A. R. Belo Lourenço, Cláudio Boker, Udi Barbosa, Haniel Bentkamp, Alexander Bønneland, Frederik M. Barwell, Adam Berger, Jana Brain, Martin Castellano, Ezequiel Chen, Mingshuai Coenen, Norine Castro-Pérez, David Chida, Nariyoshi Cogumbreiro, Tiago Cetinkaya, Ahmet Chipara, Octav Correas Fernández, Jesús Cheang, Kevin Dai, Gaoyang

Defourné, Antoine Downing, Mara Darwin, Oscar Dill, David Dunn, Isaac Dave, Vrunda Dohmen, Taylor Dureja, Rohit De Masellis, Riccardo Doveri, Kyveli Eberhart, Clovis Eiers, William Esen, Zafer Ebrahimi, Masoud Farzan, Azadeh Feng, Yuan Fleury, Mathias Fedyukovich, Grigory Ferraiuolo, Andrew Gardy, Patrick Godefroid, Patrice Graham-Lengrand, Stéphane Gehani, Ashish Gomez-Zamalloa, Miguel Grumberg, Orna Genaim, Samir Goorden, Martijn Guan, Ji Georgiou, Pamina Gordillo, Pablo Guha, Shibashis Giacobbe, Mirco Graf, Susanne Gupta, Ashutosh Giesl, Jürgen

Habermehl, Peter Helfrich, Martin Huang, Chengchao Hadzic, Vedad Hofmann, Jana Huber, Nikolaus Hark, Marcel Holík, Lukáš Hyvärinen, Antti Hecking-Harbusch, Jesko Hozzova, Petra Irfan, Ahmed Isabel, Miguel Jaber, Nouraldin Jha, Susmit Jovanović, Dejan Jensen, Mathias Claus Jiang, Xu Junges, Sebastian Jensen, Peter Gjøl Kadron, Burak Klikovits, Stefan Koenighofer, Bettina Kempa, Brian Klinkenberg, Lutz Kremer, Gereon Kheterpal, Nishant Klüppelholz, Sascha Kura, Satoshi Kim, Edward La Malfa, Emanuele Li, Jianlin Lin, Shaokai Lachnitt, Hanna Li, Yangjia Lorber, Florian Larraz, Daniel Li, Yong Lukina, Anna Lathouwers, Sophie Limperg, Jannis Luppen, Zachary Lee, Sang-Hwa Maderbacher, Benedikt Merayo, Alicia Mora, Federico

Madnani, Khushraj Metzger, Niklas Mueller, Peter Mallik, Kaushik Michelmore, Rhiannon Mundkur, Prashanth Mann, Makai Mohaqeqi, Morteza Murali, Vishnu Martin-Martin, Enrique Monti, Raul Möhle, Sibylle Mazzucato, Denis Moosbrugger, Marcel Nagisetty, Vineel Nenzi, Laura Noll, Thomas Narodytska, Nina Nikšić, Filip Nummelin, Visa Nejati, Saeed Otoni, Rodrigo Ozdemir, Alex Özkan, Burcu Overbeek, Roy Pant, Yash Vardhan Perez, Mateo Polgreen, Elizabeth Passing, Noemi Philipoom, Jade Poulsen, Danny Bøgsted Patane, Andrea Pick, Lauren Preiner, Mathias Pereira, Mário Piribauer, Jakob Purser, David Quatmann, Tim Reynolds, Andrew Rubbens, Bob Ryan, Megan Rowe, Reuben Sato, Sota Sebastiani, Roberto Stanford, Caleb Schupp, Stefan

Shah, Ameesh Stankovic, Miroslav Schurr, Hans-Jörg Solovyev, Alexey Stein, Benno Schwenger, Maximilian Spel, Jip Tabar, Asmae Torfah, Hazem Tsiskaridze, Nestan Tekriwal, Mohit Tschaikowski, Max Turrini, Andrea Tibo, Alessandro Unno, Hiroshi Vasconcelos, Vasco Vediramana Krishnan, Hari Govind Vukmirović, Petar Vazquez-Chanlatte, Marcell Venkatesan, Abinaya Waga, Masaki Wang, Qisheng

Wilson, Amalee Wagner, Christopher Weil-Kennedy, Chana Winkler, Tobias Wang, Benjie Welzel, Christoph Wu, Haoze Wang, Fang Wicker, Matthew Wu, Min Wang, Peixin Xue, Bai Yu, Emily Zeljić, Aleksandar Zhang, Linpeng Zhou, Mengchu Zhang, Hanwei Zhao, Hengjun Zuleger, Florian Zhang, Hengjun Zhou, Li

### Contents – Part I

#### Invited Papers


Automated Safety Verification of Programs Invoking Neural Networks . . . . . 201 Maria Christakis, Hasan Ferit Eniser, Holger Hermanns, Jörg Hoffmann, Yugesh Kothari, Jianlin Li, Jorge A. Navas, and Valentin Wüstholz



#### Hybrid and Cyber-Physical Systems


#### Security



### Contents – Part II

#### Complexity and Termination






# **Invited Papers**

### **NNREPAIR: Constraint-Based Repair of Neural Network Classifiers**

Muhammad Usman1(B) , Divya Gopinath2(B), Youcheng Sun3(B) , Yannic Noller4(B) , and Corina S. P˘as˘areanu2(B)

<sup>1</sup> University of Texas at Austin, Austin, USA muhammadusman@utexas.edu <sup>2</sup> KBR Inc., Nasa Ames, Mountain View, USA {divya.gopinath,corina.s.pasareanu}@nasa.gov <sup>3</sup> Queen's University Belfast, Belfast, UK youcheng.sun@qub.ac.uk <sup>4</sup> National University of Singapore, Singapore, Singapore yannic.noller@acm.org

**Abstract.** We present NNrepair, a constraint-based technique for repairing neural network classifiers. The technique aims to fix the logic of the network at an *intermediate layer* or at the *last layer*. NNrepair first uses *fault localization* to find potentially faulty network parameters (such as the *weights*) and then performs *repair* using *constraint solving* to apply small modifications to the parameters to remedy the defects. We present novel strategies to enable precise yet efficient repair such as inferring correctness specifications to act as oracles for intermediate layer repair, and generation of *experts* for each class. We demonstrate the technique in the context of three different scenarios: (1) Improving the *overall accuracy* of a model, (2) Fixing security vulnerabilities caused by *poisoning* of training data and (3) Improving the *robustness* of the network against *adversarial* attacks. Our evaluation on MNIST and CIFAR-10 models shows that NNrepair can improve the accuracy by 45.56% points on poisoned data and 10.40% points on adversarial data. NNrepair also provides small improvement in the overall accuracy of models, without requiring new data or re-training.

#### **1 Introduction**

Neural networks have many applications, being used for example in pattern analysis, image classification, or sentiment analysis for textual data, and also in medical diagnosis or perception and control in autonomous driving, which bring safety and security concerns [10]. These systems learn the network parameters (weights and biases) through *training* on a set of labeled examples. The performance of the trained networks is independently validated by computing the *accuracy* on a held-out labeled test set.

Just like other software systems, trained neural networks can have *defects* that need *repair*. For example, a trained neural network may have low accuracy which may be due to limited training data. One would like to repair the network by modifying its parameters (or a subset of them) to improve its overall accuracy, even in the absence of additional training data. In another scenario, the training data for a neural network has been *poisoned* by an adversary leading to high accuracy on normal data but poor accuracy on poisoned data [6,7,11]. In this case, one would like to repair the network to remedy the defect while still maintaining a high accuracy on non-poisoned data. In yet another scenario, a trained network may have high accuracy on the test set but may be vulnerable to adversarial perturbations, i.e., small modifications to the inputs that lead to unexpected outputs. Recent studies [8,15,20] show that this defect is very common even for highly trained, highly accurate networks. In this case, one would like to repair the network to make it *robust* against adversarial perturbations while at the same time retaining its accuracy on the normal, unperturbed test set.

Retraining could be used to alter the neural network parameters and repair for faults, but it can be very difficult and expensive subject to uncertainties, and may result in a network that is quite different from the original one, thus wasting the effort of the original training.

We present a novel constraint-solving based approach, NNrepair, to repair neural networks trained for the task of classification, with respect to all three scenarios described above. Similar to traditional program repair [5,13,22], NNrepair first uses *fault localization* to identify the network parameters that are the likely source of defects, followed by *repair*, which uses *constraint solving* to apply small modifications to the network parameters to remedy the defects.

Given a trained neural network model, the potentially faulty components could be the architecture of the model (which is fixed in the design stage) or the learn-able parameters such as the weights and the biases (which are determined during training). In this work, we focus on the learn-able parameters of a neural network model, specifically the weights on the edges connecting neurons. As observed in [9], changing the weights is a common fix for neural networks.

We leverage the organization of a neural network into layers and the natural decomposition of computation that each layer provides, and scope our work to focus the repair on a single layer of the network. Repairs across multiple layers are possible, but they would be less scalable and involve more complex modifications. We propose two types of repairs: *intermediate-layer repair* and *last-layer repair*. Intermediate-layer repair attempts to fix failures by modifying the behavior of neurons at an inner layer of the network. Last-layer repair, on the other hand, attempts to modify the decision constraints at the last layer.

Fault localization is used to mark one or more neurons at a layer as 'suspicious' and to find a sub-set of incoming edges to the suspicious neurons, whose weights will be the target for repair. The repair process involves solving constraints collected from the network, via a simple form of concolic execution [17]. For last-layer repair, the oracle of the repair is the desired label for every failing input and the repair constraints encode this decision. For intermediate-layer repair, we propose a novel use of activation patterns representing specifications of correct behavior at the layer [4] as oracles for repair. This enables us to keep the repair local to the layer and therefore efficient.

Furthermore, to make the *constraint solving* scalable, instead of solving for constraints for all classes at once, we propose to *decompose* the repair into a set of sub-tasks, one for each output class. Specifically, we set-up the constraint solving to correct a subset of the weights with the goal of improving accuracy of the model wrt a specific output class. The result of this repair is a set of *experts*, which are neural networks that improve accuracy of the network wrt specific output classes. We then combine the experts to obtain the final repaired model.

There are a few recent related techniques that propose to use constraint solving for neural network repair. We summarize them in Sect. 6. These techniques tend to focus on last layer repair while we also propose repair at an intermediate layer. Furthermore, we evaluate our initial prototype in three scenarios: improving accuracy, robustness and resilience towards poisoned data. None of the related techniques address all three (albeit potentially possible).

We summarize our contributions as follows.


#### **2 Background**

*Neural Networks.* In this work we focus on neural network classifiers. These networks take in an input, such as an image, and output a class (or label) specific to the problem they have been trained to solve. Networks are organized in *layers* of different types, including convolutional, activation, and pooling, each of which has a number of nodes. For this paper, we focus on activation layers. Each node from the previous layer will output into the associated node in the activation layer, which will apply an *activation function.* Common activation functions include linear rectification (a.k.a. ReLU) and sigmoid. For simplicity we discuss here ReLU activations but our work applies to arbitrary activations as discussed below. Let N(X) denote the value of a neuron as a function of the input. N(X) = - <sup>i</sup> <sup>w</sup><sup>i</sup> · <sup>N</sup>i(X) + <sup>b</sup> where <sup>N</sup>i's denote the values of the neurons in the previous layer of the network and the coefficients <sup>w</sup><sup>i</sup> and the constant <sup>b</sup> are referred to as *weights* and *bias*, respectively. If this function evaluates to a non-negative value, the node is *activated* and outputs that value, otherwise it outputs 0. A final decision (logits) layer produces the network decisions based on the real values computed by the network, by applying e.g., a softmax function; in our work we use the max function instead. For a comprehensive introduction to neural networks, see [3].

*Activation Patterns.* We leverage previous work [4] to infer network properties based on the *activation patterns* of neurons in the network. We will use these activation patterns as oracles for the intermediate layer repair. An activation pattern σ specifies an activation status (*on* or *off* ) for some subset of neurons at a layer in the network. All other neurons do not matter. We write *on*(σ) for the set of neurons marked *on*, and *off* (σ) for the set of neurons marked *off* in the pattern σ. Each activation pattern σ defines a predicate σ(X) that is satisfied by all inputs X whose evaluation achieves the same activation status for all neurons as prescribed by the pattern.

$$\sigma(X) \coloneqq = \bigwedge\_{N \in on(\sigma)} N(X) > 0 \land \bigwedge\_{N \in off(\sigma)} N(X) \le 0 \tag{1}$$

A decision pattern σ is a property wrt network F and postcondition P if:

$$\forall X \; : \sigma(X) \Rightarrow P(F(X)). \tag{2}$$

A postcondition for a classification network is that the top predicted class is C, i.e., P(Y ) := argmax(Y ) = C.

The previous work [4] also describes how to compute activation patterns. The idea is to observe the activation signatures of a large number of inputs and apply decision tree learning over them to infer activation patterns that are thus empirically valid. We adopt the same approach here. The *support* of a pattern is formed by all the inputs that satisfy the pattern. We are interested in computing high-support patterns as they are the most likely to reflect valid properties of the network.

#### **3 Example**

This section demonstrates *Intermediate-layer* and *Last-layer* repair on a simple example. Figure 1 shows a simple two-input network with two hidden layers; each containing two ReLU nodes (ReLU(x) = x (on) if x > 0, 0 (off) otherwise), and

**Fig. 1.** Example

**Table 1.** Data for example


two outputs, <sup>y</sup><sup>0</sup> and <sup>y</sup><sup>1</sup>. The weights are depicted on the edges between nodes. For simplicity we assume biases are 0. The input X, which is a two-element array denoted [x0, x<sup>1</sup>], is assigned class 0 if <sup>y</sup><sup>0</sup> > y<sup>1</sup> and 1 otherwise. Let us assume the model behaves correctly on the first four inputs shown in Table 1. The table also shows the decisions of the ReLU activations for nodes N<sup>0</sup>, N<sup>1</sup>, N<sup>2</sup>, N<sup>3</sup>, respectively. Whenever a ReLU node is on, the decision is indicated as a 1 and if it is off, then the decision value is shown as 0.

Consider now the input <sup>X</sup><sup>4</sup> = [1.5, <sup>2</sup>.0]. Assume this input is mis-classified; the output class is 1 but the ideal class is 0. The inaccuracy of the model could be a result of insufficient training. We then aim to build a repair, which in our case focuses on a single layer of the network and modifies the weights feeding into the neurons at that layer.

We keep the repair local to the layer by using activation patterns [4] in lieu of the decision constraints. The insight in [4] is that the logic that every layer implements could be captured as rules in terms of the activation patterns of the neurons. We can observe in the example, that for all inputs correctly classified with label 0, the neuron pair (N<sup>2</sup>, N<sup>3</sup>) in the second layer has the activation pattern (off, on). For the failing input, this pattern is not satisfied; in fact the activation for (N<sup>2</sup>, N<sup>3</sup>) for the failing input is (on, on). We use the above observation to fix the failure by performing *intermediate layer* repair. We

**Fig. 2.** Overview of the approach

aim to modify the neuron activations of the second layer on the failing input to satisfy the correct-label pattern for class 0 at the layer.

We aim to perform the repair by making minimal changes to the model. We identify the weights to be modified using an attribution-based approach and use constraint solving to compute the values of the new weights (see Sect. 4 for details). Changing the weight of a single edge, connecting <sup>N</sup><sup>1</sup> and <sup>N</sup><sup>2</sup> from <sup>−</sup>1.5 to <sup>−</sup>1.9 changes the activation pattern for (N2, N<sup>3</sup>) to (off, on) on the failing input, while preserving the behavior of the neurons (in terms of their activation pattern) and the output of the model on the passing inputs.

Consider now another input for the above-corrected network, <sup>X</sup><sup>5</sup> = [0.6, <sup>1</sup>.0]. This input is very close to <sup>X</sup><sup>1</sup> = [0.0, <sup>1</sup>.0] (correctly classified to 1) with a small change to <sup>x</sup><sup>0</sup> that makes the model mis-classify the input to 0. This represents a typical adversarial scenario where a correctly classified input is perturbed slightly to create an input that 'jumps' the decision boundary of the network leading to a mis-classification. It can be observed that the activation patterns of the internal layer neurons for <sup>X</sup><sup>5</sup> are the same as for the correctly classified input X<sup>1</sup>, thus an intermediate-layer repair would not work for this input. Therefore we perform *last-layer repair*. We localize the weights of the edges in the last layer that need repair. Changing the weight on the edge between <sup>N</sup><sup>3</sup> and <sup>y</sup><sup>1</sup> (from 1.5 to 1.6) corrects the class for the failing test to 1, while retaining the same labels for the other inputs.

#### **4 Approach**

Figure 2 gives an overview of our approach. We aim to repair a faulty trained neural network classifier, which is given as input. As in other repair approaches, we consider both positive and negative examples for the repair. The negative examples are used to guide the repair towards correcting the faults while the positive examples are used to constrain the repair to not damage the existing good functionality of the network. We aim for a repair strategy that is scalable and applies small changes to the network. We therefore target the repair on a single layer of the network. Repairs across multiple layers of the network are possible, but they would be less scalable and involve more complex modifications.

Unlike all previous work, which tends to focus the repair at the last layer (see Sect. 6) we propose here techniques for both intermediate and last layer repair. Intuitively, a last layer repair is easier as it aims to modify the weights that impact directly the decisions, and can use the network's output as an *oracle* to guide the repair. However the resulting repair may not generalize well and furthermore the network may be faulty at some intermediate layer. A repair at an intermediate layer can have a higher impact over the network's behavior but it is more difficult as it is not clear what *oracle* to use to guide the repair. One can use the output of the network as the oracle but this may result in an un-manageable large number of constraints to solve. In this work we propose a novel use of neuron activation patterns to act as oracles in intermediate layer repair.

As repairing for all the output classes at the same time can be very difficult, our proposed approach obtains instead a set of *expert networks*, one for each target class, which are easier to compute. These experts are combined to obtain a final repaired classifier. Specifically, our repair strategy has the following steps:


The solutions for the symbolic δ's obtained from the solver are used to update the weights of the network, thus obtaining an *expert* for a specific class.

4 *Combining Experts*: Finally the experts obtained for each class are combined to obtain the repaired classifier. This needs to be done carefully, to avoid redundant computations among experts and to not damage the overall accuracy and timing performance of the classification.

In the following we give more details about our approach.

#### **4.1 Intermediate-Layer Repair**

*Fault Localization.* We explore the usage of *activation patterns* of the network (Sect. 2) to act as oracles of correct behavior. We also use these patterns to guide the identification of potentially faulty neurons. Specifically, we use the decisiontree learning approach from [4] to extract *correct-label patterns* corresponding to every output class at an intermediate layer. Each pattern is satisfied by a group of inputs correctly classified to a certain label. Typically multiple correct-label patterns are generated. We select the ones with the highest support, which are mostly likely to hold true on the network for all inputs. Note also that the work in [4] considers ReLU activations but it could be extended to consider arbitrary linear or non-linear activation functions, by comparing the values of neurons with a threshold.

A *correct-label* pattern with high support at a layer indicates that there is a high chance that any input satisfying the pattern at the layer would be classified by the network to the corresponding label. Furthermore, a mis-classified input will not satisfy the *correct-label* pattern for the respective ideal label. For every failing input, we compare the activations of the neurons with those in the respective *correct-label* pattern and consider those *neurons* whose activations differ as the potentially faulty ones. The repair then aims to change the outputs of the neurons for each of the failing inputs, such that they satisfy the correctlabel pattern for their ideal labels.

In this work, we select a dense layer (i.e., a fully connected layer which receives input from every neuron in the previous layer) with ReLU activations. Typically such dense layers appear closer to the output and may impact the classification decision more than convolutional layers which process the input. Further, the number of neurons at fully connected layers is typically smaller than at other layers making the pattern-extraction process efficient.

Consider a mis-classified input, <sup>X</sup><sup>f</sup> with ideal label <sup>C</sup>. Let <sup>σ</sup><sup>C</sup> be the *correctlabel* pattern with highest support for C. Let L be the layer for this pattern, and let N denote a neuron at layer L. Then the set of suspicious or faulty neurons N<sup>f</sup> aulty can be defined as follows;

$$N \in \mathcal{N}\_{fault} \iff (N \in on(\sigma\_C) \land N(X\_f) \le 0) \lor (N \in off(\sigma\_C) \land N(X\_f) > 0) \tag{3}$$

Once the neurons whose outputs need to change are identified, we also need to identify the incoming edges to those neurons whose weights we aim to modify. We use a simple statistical method to identify the *important weights* which impact the respective neuron's output, more for the failing inputs as compared to the passing inputs.

Consider a set *Fail* of failing inputs with the same ideal label C and a set *Pass* of passing inputs. We use #(·) to denote the cardinality of the sets. The defect score for each edge is determined as follows.

$$Score(E\_i) \ ::= \frac{\sum\_{X \in Fail} |N\_i(X) \cdot w\_i|}{\#Fail} - \frac{\sum\_{X \in Pass} |N\_i(X) \cdot w\_i|}{\#Pass} \tag{4}$$

Here <sup>E</sup><sup>i</sup> denotes an incoming edge (for a faulty node <sup>N</sup>), <sup>N</sup><sup>i</sup> is the corresponding node in the preceding layer and <sup>w</sup><sup>i</sup> is the weight of the edge.

Thus, we take the average of the absolute values passing through the edge for all the negative examples for C and the average of the absolute values passing through the edge for all the positive examples and subtract them. The intuition is to identify the edges which have more influence on the incorrect decision of the network. We calculate the defect score for each incoming edge to each neuron (N) in <sup>N</sup>faulty. We then select the edges with top n% of the scores to create the set of faulty edges, for a small n.

*Concolic Execution.* We perform a simplified form of concolic execution to form symbolic constraints for suspicious neurons. For the weights of the suspicious edges, we add δ values that are set to 0 in concrete mode, but are designated as symbolic in the symbolic mode. The network is executed concolically along both positive and negative examples, to collect the values of neurons as weighted sums in terms of both concrete values and the symbolic δ values. The value of a neuron is computed as constraints of the following form:

$$Sym\_{N,X} = \sum\_{i} (w\_i + \delta\_i) \cdot N\_i(X) + \sum\_{j} w\_j \cdot N\_j(X) + b \tag{5}$$

Here *<sup>S</sup>*ymN,X is a fresh symbolic variable introduced to encode the symbolic value of neuron N for input X, w<sup>i</sup>'s denote the weights of the suspicious edges (in the suspicious layer) while the <sup>w</sup><sup>j</sup> 's denote the other weights, which do not need modification. Furthermore, <sup>N</sup><sup>i</sup>(X), <sup>N</sup><sup>j</sup> (X) represent the concrete values of the neurons coming from the previous layer. Note that no expensive constraint solving is needed in this step.

*Repair Constraints and Constraint Solving.* For intermediate-layer repair, we add the activation patterns constraints (that imply the decision constraints, see Eq. 1) to the set of constraints. Specifically, for each neuron N in <sup>N</sup>faulty, and for each (passing or failing) input <sup>X</sup> we add *<sup>S</sup>*ymN,X <sup>&</sup>gt; 0 if <sup>N</sup> <sup>∈</sup> *on*(σ<sup>C</sup> ) and we add *<sup>S</sup>*ymN,X <sup>≤</sup> 0 if <sup>N</sup> <sup>∈</sup> *off* (σ<sup>C</sup> ).

The solutions for the symbolic δ's obtained from the solver guarantee that all the inputs (both passing and failing) satisfy the pattern and are thus likely to be classified as C by the network. These solutions are then used to update the weights of the network, thus obtaining an *expert* for the class C.

*Example.* Let us consider the example from Sect. 3, the case of the intermediatelayer repair. As already discussed in Sect. 3, let us suppose we consider the activation pattern for class 0 at layer 2. We select <sup>N</sup><sup>2</sup> as the target for repair (since its activation along the failing test <sup>X</sup><sup>4</sup> is on instead of off) and we want the input to satisfy the pattern {off, on} for {N<sup>2</sup>, <sup>N</sup><sup>3</sup>}. We compute defect scores for the incoming edges to <sup>N</sup><sup>2</sup> using the failing input and all passing inputs for classes 0 and 1. The score of the edge between <sup>N</sup><sup>0</sup> and <sup>N</sup><sup>2</sup> is 2.0 while the score of the edge between <sup>N</sup><sup>1</sup> and <sup>N</sup><sup>2</sup> is 2.81, we therefore select the second edge as a target for repair. We then build the following constraints from the failing test. *S*ym*<sup>N</sup>*2*,*<sup>4</sup> = 2.0 · (−1.0 · 1.5+2.0 · 2.0) + (−1.5 +δ) · (0.5 · 1.5+1.0 · 2.0)∧*S*ym*<sup>N</sup>*2*,*<sup>4</sup> ≤ 0.0

Similarly, we build constraints from the passing tests that satisfy the pattern for label 0, <sup>X</sup><sup>0</sup> and <sup>X</sup><sup>2</sup>:

*S*ym*<sup>N</sup>*2*,*<sup>0</sup> = 2.0 · (−1.0 · 1.0+2.0 · 1.0) + (−1.5 +δ) · (0.5 · 1.0+1.0 · 1.0)∧ *S*ym*<sup>N</sup>*2*,*<sup>0</sup> ≤ 0.0∧ *S*ym*<sup>N</sup>*2*,*<sup>2</sup> = 2.0 · (−1.0 · 1.0+2.0 · 0.0) + (−1.5 +δ) · (0.5 · 1.0+1.0 · 0.0)∧ *S*ym*<sup>N</sup>*2*,*<sup>2</sup> ≤ 0.0

In practice we also add some constraints on δ to keep it small but we omit them here for simplicity. A solution for all the constraints is δ <sup>=</sup> <sup>−</sup>0.4 which is used to update the weight for the target repair resulting in an expert for class 0.

#### **4.2 Last-Layer Repair**

*Fault Localization.* In a classifier network the last layer typically contains as many neurons as the number of classes. An input is classified to label C, if the output of the respective neuron is greater than the values of all other output neurons. It is therefore natural to designate this neuron as suspicious for target class <sup>C</sup>. Let <sup>N</sup><sup>C</sup> denote the neuron at the last layer corresponding to a class C. We use the same technique as in intermediate layer repair (Eq. 4) to *localize edges* and short-list the important weights which are the target for repair.

*Concolic Execution.* Similar to the intermediate layer repair, we add symbolic δ values to the important weights and perform concolic execution along passing and failing tests to create the symbolic expression for the node *<sup>S</sup>*ym<sup>N</sup>*<sup>C</sup>* ,X (following Eq. 5).

*Repair Constraints and Constraint Solving.* We then add the decision constraints for the passing and failing inputs:

$$\bigwedge\_{C \neq C'} Sym\_{N\_C, X} > Sym\_{N\_{C'}, X} \tag{6}$$

The obtained solutions guarantee that all the inputs that were used in the repair (both positive and negative) are classified to the correct class. The solutions are used to build the expert for each class. We then combine the experts using the combination strategies outlined in the next section.

*Example.* Consider now the example from Sect. 3, the case of the last-layer repair. As we aim to repair for class 1 we select for repair the neuron named <sup>y</sup><sup>1</sup> in the figure. The score for the edge between <sup>N</sup><sup>2</sup> and <sup>y</sup><sup>1</sup> is <sup>−</sup>2.75 and the score for the edge between <sup>N</sup><sup>3</sup> and <sup>y</sup><sup>1</sup> is 0.45 so we select the latter for repair. We then build the following constraints based on the failed test (note that the expression for the second variable simplifies to a concrete value):

*S*ym*<sup>y</sup>*1*,*<sup>5</sup> = (2.5 · (2 · (−1 · 0.6+2 · 1.0) − 1.9 · (0.5 · 0.6+1 · 1.0)) + (1.5 + δ) · (−0.5 · (−1 · 0.6+2 · 1.0) + 3 · (0.5 · 0.6+1 · 1.0)))∧ *S*ym*<sup>y</sup>*0*,*<sup>5</sup> = (−1.5 · (2 · (−1 · 0.6+2 · 1.0) − 1.9 · (0.5 · 0.6+1 · 1.0)) + 2.0 · (−0.5 · (−1 · 0.6+2 · 1.0) + 3 · (0.5 · 0.6+1 · 1.0)))∧ *S*ym*<sup>y</sup>*1*,*<sup>5</sup> > *S*ym*<sup>y</sup>*0*,*<sup>5</sup>

Similar constraints are added for the positive inputs (we omit them here for brevity). Solving these constraints gives δ = 0.1 which is added to the weight for the edge between <sup>N</sup><sup>3</sup> and <sup>y</sup><sup>1</sup> to obtain an expert for class 1.

#### **4.3 Combining Experts**

We create experts for each label in the dataset. For example, for a neural network trained on the MNIST data set (which is used for the classification of handwritten digits from 0 to 9), we generate 10 experts – one expert per label. We propose three variants of how to combine these experts:


Variant (A) is an instance of *ensemble modeling* [1], which typically involves creating multiple models to predict an outcome. In our case, we start by executing all the experts for each input. This is done in a combined fashion, to avoid repeated execution of same code: before the repaired layer the model is executed with the original weights; starting from the repaired layer the execution is split up for the different experts. At the end of the execution, each expert classifies the input to a certain label. We need to combine the results from all the experts in order to classify the input to a single label.

Each expert can classify the input to any of the labels, however, each expert can be trusted to produce the correct result only for its own respective label. Therefore, we start by generating a set E including the experts that classify the inputs to their respective labels. Note that it could be that multiple experts report that the given input belongs to their respective class or it could be that no expert classifies the given input to the expert's class. If E is empty, then we select the label by the original model. If there is one expert in E, then we select this unique expert. If there are multiple experts in E, then we need to resolve the conflict between experts and choose one label, for which we propose three strategies:

*Naive:* This strategy simply falls back to the original model.


In variant (B), we propose to merge the experts before executing the model. For the intermediate-layer repair, for every weight that is considered faulty we update it with the one δ value, which is the *average* of the solutions from all the experts. This creates a single *merged* network. For the last-layer repair, we simply apply all the repairs at once; there is no need for an average as the nodes (and edges) that are targets for repair are disjoint.

In variant (C), instead of using all experts we select a subset of strong experts. Note that each expert is constructed from failing inputs only for the respective label. Therefore, when exposed to data which are supposed to be classified to the expert's label, the expert displays higher accuracy than the original model (higher recall). However, when exposed to data which can belong to different labels, the experts could display lower overall accuracy than the original model (lower precision) due to high false positives. Therefore, we determine which of the experts have both their precision and recall (*F1 score*), computed over all positive/negative inputs, higher than the original model and retain only those while filtering out the rest. The same combination strategies, variant (A), are used to obtain a single classification result for the input.

### **5 Evaluation**

We implemented our approach in the NNrepair tool pipeline, which is based on NeuroSPF [21]. It first translates a trained Keras model into Java, uses Symbolic PathFinder (SPF) [16] for concolic execution and z3 [14] for constraint solving. In this section we evaluate NNrepair by considering its application to three highly common scenarios; *Scenario 1:* improving accuracy, *Scenario 2:* fixing backdoor attacks, and *Scenario 3:* enhancing adversarial robustness. Our experiments use two commonly used datasets for image classification networks, MNIST and CIFAR-10. We consider two architectures for MNIST with 10 and 7 layers respectively. They are convolutional neural networks (CNNs) and have the typical structure of modern neural networks such as convolutional/dense, max-pooling and softmax layers. The first MNIST model has an accuracy of 96.34% on the standard test set, while the second model has an accuracy of 98.89%. We refer to these models as MNIST-LQ (low-quality) and MNIST-HQ (high-quality) respectively. The CIFAR-10 model is a 15-layer CNN with 890k trainable parameters and has an accuracy of 81.04%. In order to validate our approach, we consider the following research questions:

**RQ1** Is NNrepair successful in correcting the defects in all three scenarios?


#### **5.1 Scenarios**


**Fig. 3.** Example poisoned data for MNIST (left) and CIFAR-10 (right). The backdoor is embedded as the white square at the bottom right corner of each image. When the backdoor appears, the poisoned MNIST model will classify the input as "7" and the poisoned CIFAR-10 model will classify it as "horse".

to retain the accuracy on standard, un-poisoned data, which we measure on Test. In this scenario, the first 600 inputs in Train are poisoned (P-Train). We draw from these particular inputs to get the negative examples to focus the repair on the defect. We draw the positive examples from Train.

(3) For the last scenario, we apply adversarial perturbations over Train and Test using FGSM<sup>1</sup>, for = 0.05. This results in four data sets: Train, Adv-Train, Test and Adv-Test. The models have good overall accuracy on Train and Test, but poor accuracy on Adv-Train and Adv-Test. The goal of the repair here is to improve the accuracy on the adversarial data (which we measure on Adv-Test) without damaging too much the accuracy on standard data (which we measure on Test). We draw the negative examples to be used in repair from Adv-Train, while we use positive examples from both Adv-Train and Train. Since we use two separate sets to generate experts, when computing the F1-score for selection of experts, we explored two different options: computing F1 score over Adv-Train only and computing harmonic mean of the F1 scores computed over Train and Adv-Train separately. However, in practice there was no difference as same experts were filtered in both cases.

#### **5.2 Experiment Set-Up**

For each of the three scenarios, we experimented with both intermediate-layer and last-layer repairs. We evaluated all the combination strategies (Naive, Confidence, Voting, and Merged) with the F1-filtering option being OFF and ON. When F1-filtering is OFF, the experts for all labels are used in the combination strategies by default, while when it is ON, we only include those experts whose F1 score on Train is greater than the original model.

*Intermediate-Layer Repair:* We focused on the dense layer just before the output layer for both the MNIST and CIFAR models. The intuition for this selection is that dense layers appearing closer to the output potentially impact the classification decision more than convolutional layers closer to the input (which have the role of feature extraction). The MNIST models have 128 and 100 ReLU nodes and 576 and 400 incoming edges to each neuron at this layer respectively, while the CIFAR model has 512 ReLU neurons and 1,600 incoming edges at this

<sup>1</sup> https://www.tensorflow.org/tutorials/generative/adversarial fgsm.

layer. We extracted high support patterns for correct classification; the average support per label was within 1,013–2,502 across scenarios, out of around 6,000 inputs per label. The neurons short-listed in N<sup>f</sup> aulty using pattern-based localization varied between 1 and 10 in number. We focused on modifying the weights of the incoming edges having their scores within the top 10%.

We used the patterns extracted at the layer to select a subset of tests for the purpose of constraint solving. As explained in Sect. 4, we used decisiontree learning to extract patterns for correct classification for every label. We also extracted patterns for incorrect classification for each label, which represent neuron activations satisfied by inputs which should ideally be classified to the given label but get mis-classified. From the set of all failing tests for a given label, we select all inputs that satisfy the pattern for incorrect classification for the label. From the set of all passing tests for a given label, we select the subset of inputs that satisfy the pattern for correct classification. We then randomly select # failing tests + 100 inputs from this set. The subset of failing and passing tests selected using the procedure above is used for constraint solving.

*Last Layer Repair:* At the last layer, the two MNIST models have 10 ReLU nodes and 128 and 100 incoming edges to each neuron respectively, while the CIFAR model has 10 ReLU neurons and 512 incoming edges. For each label, we selected 5 failing and 5 passing inputs randomly from the respective datasets. For the first scenario, both these failing and passing inputs come from the Train set. For the last layer repair top 5 suspicious weights were made symbolic for each expert. We determined empirically that a larger number for symbolic weights and/or passing/failing inputs leads often to unsat constraints while a smaller number may not improve the network.

The poisoned (2) and adversarial (3) scenarios differ from scenario 1, in that they seek to address two challenges. The repaired model needs to have better accuracy than the original model on poisoned and adversarial inputs respectively (evaluated on the P-Test and Adv-Test sets), as well as the accuracy on normal inputs should not be degraded much (evaluated on the normal Test set). For this reason, for the purpose of constraint solving in addition to including passing tests from the respective poisoned and adversarial train sets, we also include passing tests from Train. We performed experiments increasing the number of passing tests included from the normal train set from 0 to 10, 50, and 100.

#### **5.3 Results**

Table 2 presents a summary of our results (please refer to the Appendix<sup>2</sup> for more detailed results). The table displays the results for MNIST and CIFAR models for the three scenarios. For each scenario, the results for both intermediate layer and last layer repair are presented in terms of the improvement in accuracy obtained over the original model. This is the best result corresponding to the improvement in accuracy on the respective test sets (normal Test for the first

<sup>2</sup> https://arxiv.org/abs/2103.12535.

scenario, P-Test for the second and Adv-Test for the third). The combination strategy and the F1-Filter setting (ON/OFF) used to obtain the best result are also displayed, along with the corresponding improvements in accuracy on the other train and test sets. For the repair, z3 was able to generate solutions for each expert within a minute. The constraint generation using SPF was the bottleneck and SPF generated constraints for each expert within 15–60 min, depending on the number of tests included. However, this could be improved since running SPF on all positive/negative inputs can be performed in parallel. Experiments were performed on a Windows 10.0 machine with Intel Core-i5 and 16 GB RAM. The code, constraint files along with Z3 solution files are available at https:// github.com/muhammadusman93/nnrepair.

**RQ1:** For this research question we seek to investigate if NNrepair is successful in correcting defects in all three scenarios. To measure success, we consider the improvement in accuracy provided by the repair in all three scenarios.

The effectiveness of NNrepair in improving accuracy (Scenario 1) can be analyzed by considering Table 2 (cases MNIST-LQ, MNIST-HQ and CIFAR10).

We observe that the best results provided by NNrepair for the MNIST-LQ model was +0.20, +0.02 for the MNIST-HQ model and +0.16 for the CIFAR10 model. This improvement (albeit small) was achieved without any new inputs or re-training. The quality of the improvement appears to degrade as the quality of the original model increases. We note that achieving improvement in the overall accuracy of an already high-quality model without new data is very challenging. In fact this improvement appears to be in line or better than related repair techniques (see Sect. 6). Note also that the complexity and size of the models do not seem to have an impact on the effectiveness of the repair. The MNIST-HQ architecture is simpler than MNIST-LQ and the CIFAR10 architecture is much bigger and more complex than the MNIST models.

For Scenario 2, on the MNIST-Pois model, NNrepair increased the accuracy from 10.38% to 55.94% on poisoned inputs (P-Test). The repair causes a slight decrease (−3.11) in accuracy on non-poisoned inputs (Test) but the repaired model still has a high accuracy (≥95.5%) on non-poisoned inputs. On the more challenging CIFAR10-Pois model, the best improvement provided by NNrepair is a +3.77 increase on poisoned inputs, and a small decrease in accuracy on non-poisoned inputs (−0.61). For Scenario 3, on the MNIST-Adv model, NNrepair increased the accuracy from 28.37% to 38.77% on adversarial inputs, while causing a small decrease (−3.14) in accuracy on non-adversarial inputs. For CIFAR10-Adv, the best result was an increase of +0.34 on adversarial inputs with a minor decrease of −0.07 on non-adversarial inputs.

For the last two scenarios, the primary goal is to improve accuracy on poisoned or adversarial data. Although ideally we would also want to preserve the original accuracy on normal data, this may not always be possible in practice. We experimented with varying number of passing tests from Train for scenarios 2 and 3. The results are presented in the first table in the Appendix. The accuracy of the resulting repair on the poisoned/adversarial test sets tends to decrease as the number of normal passing tests goes up. However, this also reduces the **Table 2.** Summary of NNrepair performance on all models. Repair column shows the type of repair, i.e., intermediate or last layer. Increase/decrease in accuracy shown in terms of the difference between the accuracy of the repaired model and the original model on the respective datasets. Accuracy of the original model is shown in brackets (in bold) below each data set. The *Strategy* column shows the combination strategy which work best for each scenario. *ALL* means that all strategies performed equally. *F1-Filter* shows if best results were obtained by turning F1-Filter ON or OFF. The number of experts used are shown in brackets.


degradation in the accuracy on normal test set. Previous studies in adversarial robustness [23] indicate that one can obtain robust networks but the price to be paid is a significant decrease in accuracy on normal data. Similar considerations apply to the poisoning case. Therefore, we tolerate small decrease in the accuracy on normal Test in our work as well.

The last two columns in Table 2 list the combination strategies and the F1 filtering option which work best for each scenario. The Merged strategy seems to work well for the CIFAR10 model for all the three scenarios. However, there is no clear winner for the MNIST models. In fact, for the last layer repair on CIFAR10, all the strategies gave the same improvement in accuracy. In practice, the users would need to use a separate validation set and try all the strategies to pick the best one for their application domain.

**Answer RQ1**: NNrepair shows benefit in all three scenarios. It can repair a network to make it robust against adversarial perturbations/poisoned inputs while at the same time retain a good accuracy on the normal, unperturbed/non-poisoned test set. NNrepair can also improve the overall accuracy of the models, however the effectiveness of the repair tends to decrease when the original accuracy is already high.

**RQ2:** Table 2 can be used to compare the performance of intermediate-layer and last-layer repair on the different scenarios. For the MNIST models, last-layer repair did not help in improving the overall accuracy. Repairing the dense layer before the output layer using the pattern-based repair helps in increasing the accuracy albeit by a small amount. For the CIFAR10 model, on the other hand, repairing the output layer increases the overall accuracy of the model by 0.16, which is better than intermediate-layer repair (+0.03).

For the poisoned and adversarial scenarios, on the MNIST models, last-layer repair performed better than intermediate layer-repair on the targeted test sets. Intermediate-layer repair increased the accuracy by 1.81 on the poisoned model and 3.87 on the adversarial model while last-layer repair increased the accuracy by 45.56 on the poisoned model and 10.40 on the adversarial model. For CIFAR10-Pois, intermediate layer repair increases the accuracy by 0.81 while last layer repair improves it by 3.77. Note that intermediate-layer repair seems to help better in retaining the accuracy on the standard Test, albeit providing smaller improvements on the target sets (detailed results in the Appendix). Furthermore, for CIFAR10-Adv, intermediate layer repair gives better results than last-layer repair (0.34 vs 0.27 respectively).

To summarize, focusing only on an inner layer of the network or just the output layer may not suffice to correct errors in all models and scenarios. We plan to investigate application of repair at more than one layer. Fault localization approaches may help determine the layer/s to focus on for effective repair for a given application.

**Answer RQ2:** Intermediate-layer repair helped more in improving the overall accuracy of the models (except for CIFAR10) and last-layer repair was more effective in repairing specific failures such as vulnerabilities to poisoned or adversarial inputs (except on CIFAR10 adversarial model). The take away is that there is not a specific type of repair (last-layer or intermediate layer) that works well consistently and different models and failure scenarios may necessitate repair at different layers.

**RQ3:** To understand the overhead introduced by running multiple experts and the combination logic, we conducted experiments on one of the models, MNIST-LQ. We executed the original model on the test set and compared the inference time with the model produced by a repair at an intermediate layer (i.e., layer 6) and by a repair at the final layer (i.e., layer 8). Additionally, we measured the inference time for an intermediate layer repair with F1-Filtering (i.e., layer6-F1). We performed this comparison for all 10,000 inputs in the test set.

The *Merge* combination strategy does not require any expert combination after model execution because this strategy merges the repairs in advance. Therefore, there is no change with regard to the original model execution except the weight values used in the calculations, and we did not observe any difference in terms of the inference time. We focus the remaining discussion on the strategies that require the execution of multiple experts. Our experiments show that the time for the expert combination after model execution (as necessary for *Naive*, *Confidence*, and *Voting* combination strategies) is negligible with around 0.0008 ms and also is similar for all these combination strategies. The main overhead is introduced by the additional calculations necessary to compute the multiple expert values at each layer. The box plot in Fig. 4 shows the total time for the model execution for the experts inclusive the time for the *Naive* expert combination.

**Fig. 4.** Inference time comparison (*Naive* Combination Strategy)

The repair at the last layer produces an average slowdown (compared to the original model) of 1.0383x. In contrast, the repair at the intermediate layer produces an average slowdown of 7.7638x. Therefore, it makes sense to apply some filtering of experts, which do not show good performance on the training set (see F1-score filtering in Sect. 4.3). For this experiment we kept 3 experts (see the plot with *layer6-F1* ). This reduced model produces an average slowdown of only 3.0742x.

**Answer RQ3:** The *Merge* combination strategy does not impact the inference time. All other combination strategies introduce a similar overhead. While the inference time for the last-layer repair is comparable with the original model, the inference time for an intermediate-layer repair is expensive. However, it can be significantly reduced with F1 filtering.

#### **5.4 Discussion**

The purpose of our evaluation was to showcase the versatility of NNrepair in different scenarios. The takeaway from the experiments is that there is not a specific type of repair (last-layer or intermediate layer) that works well consistently and different models and failure scenarios may necessitate repair at different layers. In particular, we believe that the intermediate-layer repair holds the most promise for scaling to large networks and we plan to further experiment with the technique in the future.

Generally, the best repair results are obtained on the poisoning task, where the accuracy can be increased by up to 45% and 3.7% on MNIST and CIFAR10, without a need for retraining, which can be expensive in practice. Furthermore, note that we do not assume knowledge of the poison, as our techniques only use information about correct and incorrect classification. In the future, we plan to perform more experiments with different poisoning scenarios.

We were able to obtain modest accuracy improvements on the high-quality models, while for the low-quality models, re-training can achieve better results (see comparison with MODE in the next section). More experimental comparison with retraining and/or fine-tuning the models is needed to further assess the merits of our constraint-based repair.

The gains in the adversarial setting are not very significant for the larger models. In this work, our goal was to demonstrate the feasibility of using localized constraints solving as a generic technique for addressing a wide range of challenges in deep learning. Adversarial attack is only one potential application scenario that is considered. There is a large body of research work on adversarial attacks and we can not claim in any way that we can cover all attacks.

We also note that the efficacy of NNrepair is evaluated statistically (over the test set) as our method does not provide any formal guarantees. In general, it is difficult to guarantee an improvement of the overall accuracy with formalisms, as there are no formal specifications for the image classification domain. Thus, in practice one builds (trains) a model using a statistical measure of accuracy.

#### **6 Related Work**

The emphasis of this paper is on neural network repair, where the goal is to "correct" the neural network and improve its performance, robustness and security, by using a small number of labeled inputs. There have been relatively few attempts for repairing a neural network. These neural network repair works can be classified given if re-training is needed and/or if there is a first step to prioritize neuron weights to fix. A number of fix patterns and challenges for neural network repair were collected in [9].

In MODE [12], a neural network is said to be buggy for a specific output label if its test accuracy is lower than the expectation. This is fixed by selecting features that are critical for the misbehavior via differential analysis using a subset of training data and then retraining by selecting inputs from the remaining unused training inputs based on the differential heat map. We ran MODE on the MNIST models from our study. The results are as follows:


NNrepair has similar performance, i.e., slightly better than MODE on MNIST-HighQuality and slightly worse on MNIST-LowQuality. Meanwhile, the re-training procedure in MODE led to varied performances for the repaired model. The results for MODE are the average outcome after 10 runs, none of which improved the accuracy of MNIST-HighQuality.

Unlike MODE that identifies ill-trained weights or buggy neurons, Apricot [24] first generates a set of models from the original neural network with a reduced set of training data and at each iteration of the training, Apricot adjusts each weight of the repaired model towards the average weight of these reduced models correctly classifying the input while away from the misclassifications. The approach from [19] uses constraint solving for repairing neural networks. It considers a two-dimension slice of the input space of ACAS Xu and uses SMT constraints to achieve weight changes for correct cases that are checked against the specification. We found it non-trivial to extend this approach to typically high-dimensional input space of the image classifiers that we study in this paper.

Typically, a software repair technique (including for neural networks) employs as a first step *fault localization* to determine the code entities that need to be fixed. DeepFault [2] is an approach to spectrum-based fault localization that aims to identify the neurons that are 'more' responsible to adversarial behaviours of a neural network. However, the aim of DeepFault is to generate more adversarial examples, which is the opposite to the repair purpose of our paper. Another related approach, Arachne [18], uses fault localization to identify neural weights (connected to the final output layer) to modify, using Particle Swarm Optimisation (PSO), for better weights to improve the model's accuracy on some particular label. As also noted in [18], increasing the prediction accuracy for a particular label often comes along with the decreasing prediction accuracy of the overall neural network model.

Our NNrepair work provides a general repair approach which can be applied for improving accuracy, enhancing robustness against adversarial attacks and fixing the backdoor security problems for neural networks. Although previous techniques could be presumably extended to these scenarios, in practice they were only demonstrated for improving the prediction accuracy of the neural network (in MODE and Apricot) or a particular label (in Arachne).

#### **7 Conclusion and Future Work**

We presented NNrepair, which uses constraint solving for intermediate-layer and last-layer repair of neural networks. We demonstrated NNrepair in three scenarios: improving the *overall accuracy*, fixing security vulnerabilities caused by *data poisoning* and improving the *adversarial robustness* of the networks.

In future work, we plan to experiment with different localization techniques and to evaluate our repair on larger networks and different architectures. Our method can also be applied to multiple layers but we restricted to single-layer for scalability. One avenue for research is to apply single-layer repair repeatedly or compositionally to handle correcting bugs across multiple layers.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### Balancing Automation and Control for Formal Verification of Microprocessors

Shilpi Goel, Anna Slobodova(B) , Rob Sumners, and Sol Swords

> Centaur Technology, Inc., Austin, TX, USA {shilpi,anna,rsumners,sswords}@centtech.com

Abstract. Formal methods are becoming an indispensable part of the design process in software and hardware industry. It takes robust tools and proofs to make formal validation of large scale projects reliable. In this paper, we will describe the current status of formal verification at Centaur Technology. We will explain our challenges and our methodology—how various proofs and verification artifacts are interconnected and how we keep them consistent over the duration of a project. We also describe our main engine—a powerful symbolic simulator with rewriting capabilities that is integrated in a theorem prover and proven correct.

Keywords: Hardware verification *·* Microprocessor verification *·* Microcode verification *·* Formal methods *·* ACL2 *·* Symbolic simulation *·* Decision procedures

#### 1 Introduction

The discussion of Formal Verification (FV) of software and hardware three decades ago was mostly about case studies or proofs of concept that required a lot of manual effort by researchers. Since then, FV has taken a transformational journey that has resulted in highly automated tools—equivalence checkers, model checkers, SMT solvers, and theorem provers. Large scale formal verification projects were first reported by hardware companies around ten years ago, e.g. Intel [28], IBM [36], ARM [34], and Centaur Technology [18,37]. Success stories of FV at software development companies followed. To name just a few, see Peter O'Hearn's keynote at *PLDI 2020* conference about incorrectness logic and static analysis his group applies at Facebook [30], David Dill's keynote at *CAV 2020* about the Libra project at Facebook [19] and their use of the Move Prover [44], or the invited talk by Byron Cook at *CAV 2018* about the application of formal methods at Amazon Web Services [16]. Formal methods are becoming a reliable and indispensable part of the design process in the commercial software and hardware industries. This newly elevated position of formal verification brings new responsibilities for those that develop tools and methods and those who build proofs. FV teams face various challenges:


These challenges can only be solved by building robust expandable proofs. In this paper, we will describe the approach taken by our FV team at Centaur Technology. Centaur is a relatively small company, of about one hundred employees, that designs x86 compatible microprocessors, focusing on the low cost, low power market. It might surprise many that our formal verification tools are based on a theorem prover. This is only possible because the theorem prover we use, ACL2 [8], has been designed with industrial applications in mind [24]. ACL2 has been successfully used not only at our company but also at many others: e.g. ARM, AMD [35], IBM [36], Rockwell-Collins [22], and Oracle [32]. All our proofs are done within the ACL2 system. ACL2 is used to write specifications, models, tools, and tests, as well as to generate documentation. Two features of ACL2 that are crucial to our work are fast execution and extensibility. Our x86 model [20] is not only one of most complete of its kind, but is capable of executing application programs at a speed of around 3 million instructions per second.

We will start with a brief description of the ACL2 system and the features that make it a good choice for a verification framework (Sect. 2). The reflective features of ACL2 allow us to build verified tools within the system. One such tool is FGL [39], our symbolic simulator equipped with rewriting capabilities. FGL is completely integrated into ACL2 as a verified clause processor. It provides a desirable balance between automation and user guidance. We will describe its mechanism in detail in Sect. 4. We also explain its usability as a highly programmable solver that is capable of proving complex conjectures about Register-Transfer Level (RTL) design and microcode in Sect. 3. FGL and its use within our framework are primary contributions of the work presented in this paper. The challenges enumerated above are illustrated with the process of verification for a single x86 instruction. We explain the complex interconnection of the various parts of the proofs, and describe how they are built and maintained.

#### 2 Our FV Tools

All formal verification at Centaur is done within the framework of ACL2 [8]. ACL2 is an untyped language (a subset of Common Lisp) and a theorem prover that supports first-order logic as expressed in this language. ACL2 also has some limited support for higher-order style definitions [29]. ACL2 is an open source software project that has an active community contributing to an extensive library of proofs and utilities. Centaur has contributed to many libraries that support hardware verification, including support for translating Verilog and System Verilog to ACL2 expressions [7,10] and libraries that support bit operations. ACL2 provides an interface through which it can be connected to trusted tools such as SAT solvers. There is also an integration of Z3 in ACL2 [5] and an interface to the ABC model checker [1,14].

Besides interfaces to trusted tools, ACL2 has a mechanism for extending its reasoning by admitting verified clause-processors [2]. We use this feature in several ways, notably for SVL [43], a routine that automates verification of multipliers, and for FGL, the core tool that provides automation for our microoperation execution and microcode proofs.

FGL, briefly, is a term rewriter geared toward transforming expressions acting on fixed-sized data into Boolean formulas. For example, a specification for an x86 instruction may be written in high-level ACL2. Processing a call of this specification function on variable arguments in FGL yields a result that expresses each of the bits of the writeback data, flags, etc., as a Boolean formula (represented in an and-inverter graph) whose inputs are the symbolic bits of the input variables. Similarly, FGL processing of the ACL2 model of the microcoded implementation for that instruction yields Boolean formula representations of the implementation's outputs. Equivalence checking these two sets of Boolean formulas is then sufficient to show that the implementation result matches the specification. We describe the FGL system in more detail in Sect. 4, showing how it transforms terms into hybrid term/Boolean-function objects and how its behavior may be programmed with rewrite rules.

#### 3 Challenges of Verifying a Single x86 instruction

An intuitive notion of the functional correctness of a microprocessor is that any sequence of bytes decoded as instructions either executes correctly or leads to an exception if byte sequence is illegal. For the x86 instruction set, parsing and decoding a sequence of bytes is a complex process due to the many instruction formats with varying lengths and field types. The Intel 64 and IA-32 Instruction Set Architecture (ISA) is defined by the Software Developer's Manuals [27], which have thousands of pages describing the expected impact of every instruction on the state of the machine. It is a living and growing specification, with new instructions and variants added constantly. The architectural specification does not dictate how the ISA is supposed to be implemented. Various implementationspecific choices, collectively called the *microarchitecture*, include:


and various others features of the microprocessor. In our previous work [21], we described what it means for an x86 instruction to be decoded and executed correctly and how our proofs capture this property. For illustrative purposes, we use the same example that was described in that work. Table 1 describes the x86 double-precision shift right instruction SHRD and Table <sup>2</sup> shows the microoperations that implement it<sup>1</sup>. In this paper, we will recall the individual steps of the verification with a different purpose—to discuss the challenges in each step and how we deal with them. In particular, we will focus on increasing the automation and reducing the time required of engineers to catch and debug problems while maintaining the proofs.

In the process of verification, we refer to two sets of formal specifications: the architectural specification of x86 [20] and a microarchitectural specification, which is a proprietary IP of Centaur and unique to each project. We refer to the former as the *x86 model* and the latter as the *microcode model*. Both of these models are written in ACL2 following an interpreter-style operational semantics approach. The x86 model includes the specification of x86 instructions that operate on the ISA state, and analogously, the microcode model includes the specifications of microoperations that operate on the microarchitectural state. Thanks to the high execution speed of the x86 model, it can be validated by running extensive code. The microcode model is directly compared to the RTL implementation. In addition, for data-intensive operations like floating-point arithmetic, we have the ability to run our models against existing x86 hardware from Intel and AMD. Again, the efficient execution of ACL2 code is crucial for the validation of these models.

Our verification is done on the Register-Transfer Level (RTL) of microprocessor design. We have two goals: to confirm that the RTL behaves as specified by our microarchitectural specification and to show that it implements instructions correctly with respect to our architectural specification.


Table 1. SHRD–-Double Precision Shift Right: irrelevant fields elided

#### 3.1 Front-End and Microcode Verification

The front-end of a microprocessor fetches, decodes, and then translates a sequence of bytes into a sequence of microoperations. For a modern x86 processor, this is one of the more complicated parts of the design. Writing and

<sup>1</sup> Note that this is not the actual implementation of SHRD in our current design.


Table 2. SHRD RCX, RDX, imm8: a concrete run

maintaining a formal specification for it would be impractical. The readability and complexity of such a specification would be similar to that of the implementation itself. How, then, do we go about its verification? We have one methodology to verify the decoding of byte sequences into legal/illegal instructions (with appropriate exceptions), and another one to show that legal instructions are implemented correctly via microoperations.

Listing 1.1. SHRD entry in inst.lst

```
(xINST "SHRD"
       (OP :OP #xFAC)
       (ARG :OP1 '(:MODR/M.R/M :GPR :MEM)
             :OP2 '(:MODR/M.REG :GPR)
             :OP3 '(:IMM8))
        '(X86 -SHLD/SHRD)
        '((:UD (UD-LOCK - USED))))
```
For illegal instructions, we make sure that all sequences of bytes that do not decode into a sequence of legal instructions are recognized as illegal and we verify that an appropriate exception is signaled. This is done by simulating the front-end on a symbolic sequence of bytes and proving that any input that does not map to a legal opcode (as defined by the decode specification in our x86 model) produces an exception. The decode specification in the x86 model relies heavily on inst.lst—a data structure defined by us that captures all the information needed to decode every x86 instruction. The initial version of inst.lst was mechanically extracted from the Intel manuals (Chaps. 3–5, Vol. 2) [27] by parsing the tables in the description pages of each instruction and transforming the contents into an ACL2-readable format. For instance, for the implementation in Table 2, the relevant entry in the Intel manuals is in Table 1 and that in inst.lst is in Listing 1.1. Since then, inst.lst has been inspected, enhanced, and validated against internal and external x86 decoders.

Next we focus on our process for verifying legal instructions. For each instruction, our goal is to prove that for any starting machine state and for any byte sequence representing a legal invocation of that instruction in that state, the front-end produces a sequence of microoperations which, when run on our microcode model, produce the same results as the instruction run on our x86 model<sup>2</sup>. To prove this, we simulate the front-end to generate the corresponding sequence of microoperations. Using FGL, we then prove that the sequence implements the instruction as defined by our x86 specification. FGL symbolically processes the sequence of microoperations as executed on our microcode model, resulting in a symbolic machine state where the bits of the written registers are represented as Boolean formulas in terms of the values read from the initial state. It likewise processes the instruction specification, reducing it to Boolean formulas as well. We can then show by Boolean equivalence checking that the front-end-generated sequence of microoperations has the same effect on the state as the x86 instruction specification. We discuss the process of symbolic simulation of microcode by FGL in Sect. 4. This FGL proof confirms that the front-end's operation is correct for this particular instruction.

This correctness has two caveats. First, it assumes that the individual microoperations are correctly implemented, i.e., in accordance with their specifications in our microcode model. Second, in the case of out-of-order processors, if the microoperations are executed in a different order, that sequence needs to be compared to the sequence generated by the front-end. Currently, we can

<sup>2</sup> Note that the microcode model is a proprietary formal model of the microarchitecture implemented by the design. Its validation is discussed later.

ensure only the former—for most microoperations, we have proved that their implementations in the processor's execution units matches their behavior in our microcode model; we discuss this further in Sect. 3.2. However, the latter the correctness of reordering of the microoperations—is work planned for the future.

There is another part to the verification story. The front-end generates only sequences of microoperations of a limited length. Some instructions are complex and require much longer sequences (e.g. instructions performing transcendental or cryptographic functions). For these instructions, the sequence of microoperations generated by the front-end is just the beginning of the microcode program. The rest is stored in a ROM and the front-end generates the entry-point of this code. That means our verification has to account for those microcode routines.

ROM instructions are more complex than microoperations and they may also be compressed in order to save valuable ROM space. As in the front-end, the specification of this compression and decoding of ROM instructions into sequences of microoperations is complex and also changes during the design process. Even if we could define it formally, the maintenance of such a specification would be very time consuming. Instead, we do the same trick as with the front-end—we symbolically simulate the part of the design that fetches ROM instructions and translates them into a sequence of microoperations. The rest is done similarly as for the sequence of operations generated by the front-end. These proofs implicitly verify the correctness of fetching from ROM and ROM instruction translation. We call this *implicit verification* because we do not have an explicit specification of the translation and fetching. However, we do have formal specifications of the instructions implemented by the microcode. Therefore, the proof of correctness of the instructions implies the correctness of the underlying design, including ROM fetching and translation. In other words, we can verify some parts of the design as black-boxes, without knowing exactly how they work, by reasoning about the overall observable effect on the machine state. The main advantage of this type of verification of both the front-end and microcode translator is that the maintenance of the proofs does not require either deep understanding of the design or writing and maintaining cumbersome specifications.

The microcode sequences generated from x86 instructions that we encountered so far were in the style of straight-line code. We do not expect this to be the case for all of them. In the past, we worked on some microcode stored in ROM that served other purposes [18]. This code had loops and jumps between loops and we were able to do invariant-style proofs. Our main problem at that time was that the proofs were not robust enough and very hard to maintain. Now we are in a much better position, having FGL and a methodology that keeps the microcode model in sync with the design. Hence, we are optimistic about our ability to bring the verification of most, if not all, legal x86 instructions to completion.

Finally, we note that in our previous work [21], we used GL—the predecessor of FGL—as our core verification tool. The benefits of switching to FGL have been considerable. GL had limited support for term rewriting, as a result of which symbolic simulation of the microcode model was difficult and debugging failed proof attempts even more so. As such, instead of programming GL to deal with symbolic machine states, we usually used ACL2's rewriter to "open up" the microcode model and played to GL's strengths by using it for the final equivalence proof that often required non-trivial arithmetic reasoning. In other words, we obtained ACL2 formulas corresponding to the written registers in terms of the values in the initial microcode state, and then used GL to prove that those formulas were equivalent to our specification functions. FGL easily allows us to do these tasks (symbolic simulation and equivalence checking) along with others within a common environment and thereby reduces overhead in our methodology.

#### 3.2 Verification of Execution Units

Everything that was said in Sect. 3.1 relies on the assumption that our microcode model is correct. Parts of that model—front-end decoding and ROM instruction fetch and translate—are implicitly verified. The other part, definitions of microoperations that form the base of the model, were explicitly defined in ACL2 and need to be validated. A large portion of our work lies in the proofs that confirm that RTL executes the microoperations in compliance with those specifications. In order to achieve that, we build a formal model of the respective RTL module [7], unroll it with respect to the latencies of the microoperations to be verified [6] and check conformance with the specification using FGL.

These microoperations are executed in various units, the number, timing, and organization of which differs based on the specific microarchitecture. We might have separate floating-point add and floating-point multiply units, or one unit that executes both. There might be a unit that implements string operations, another that implements integer operations, and yet another one devoted to SIMD operations, etc. The scope of proofs that confirm correctness of execution of microoperations is dictated by the capacity of the tools we use. During the first years of FV at Centaur, we limited the proof for each microoperation to the specific unit where it was executed [25,26,37]. Since then, improvements to our RTL modeling and symbolic simulation (i.e., FGL) allow us to do the proofs in the scope of the module containing all those units (we refer to that module as the execution module or EXE) [21]. Migration to a higher scope has a huge advantage for the stability of the proofs. First, proofs are robust with respect to the changes of the interfaces of submodules in EXE. For instance, when an interface of a floating-point sub-unit changes to accommodate extra control signals that simplify its logic, very likely the change is transparent to the inputoutput behavior of EXE and will not effect our proofs. Second, if timing of an internal unit changes, but overall timing of the EXE module does not, that is transparent to the proofs.

Having all microoperation proofs in the same scope has another advantage we can build just one formal model of the RTL, do one unrolling to the maximum latency, and store it as a constant that can be shared and loaded by individual proofs. A review of the assumptions about the interface and the maintenance of the assumptions is also simplified when all the proofs are done with respect to one module.

#### 3.3 Regressions

Regressions have become an indispensable part of the continuous integration. There are several reasons why we need to re-run our proofs regularly. Since we start to build our proofs early in the design process, design changes occur regularly and can introduce bugs that we need to catch. But proofs can be broken not only due to changes in the design but also because of changes in the specifications, tools, and libraries. While the ISA specification is relatively stable, the microarchitecture specification might change during the project as a result of feedback from back-end tools or better ideas from the designers. Proofs might also change as the design becomes more mature and we add more thorough checks. While the core ACL2 theorem prover is very stable, ACL2 libraries are growing and may be modified by developers outside our team. All of these verification artifacts are tightly interconnected and regressions ensure that we keep them consistent.

When a proof of the correctness of a microoperation fails, there are several possible reasons:


We need to investigate the reason for failure and either report a bug to designers, adjust the proofs, or change the specification of the microoperation.

When we change the specification of a microoperation, the new definition will then be used by our microcode proofs. If those fail, it may indicate that the change affected some instruction implementations in an undesirable way. In other cases, the failure might be a result of missing rewrite rules. Microcode proofs might also fail due to the changes to front-end design or fetch and translate from ROM that introduced a new bug.

Regressions can be scheduled for a specific frequency (daily, weekly, etc.), run manually, or triggered by changes in the design, specification, or tool suite. We use open-source tools like git and Jenkins, and ACL2-specific scripts that compute dependencies on ACL2 files. Regressions also automatically generate a documentation manual from our ACL2 proof scripts [17]. This documentation includes information about which proofs failed and which succeeded and as the result of it, which microoperations and instructions are covered by the successful proofs. This keeps the documentation in sync with the design as well as the proofs. We tag individual documentation topics to indicate their intended audience; e.g., the *General Audience* tag is used when an overview of a verification effort is presented, and the *FV Audience* tag is used when describing proof strategies and verification tools.

#### 4 FGL

Since FGL is the core proof engine used in our microcode and execution unit proofs, we will describe here how it works and how it may be programmed.

FGL [4,39] is part of the ACL2 libraries and publicly available [9]. It is a significant rewrite and extension of GL ("G in the Logic") [38,41,42], which was itself a rewrite and extension of the G System of Boyer and Hunt [13]. The idea behind all of these is to recursively transform ACL2 terms into symbolic objects that represent the values of these terms and that consist mostly of structures containing Boolean function objects. When successful, the result of transforming the body of a conjecture is a single Boolean function, which may be checked for validity. The G System supported Boolean functions represented as binary decision diagrams [15], and operated on symbolic input objects using symbolic counterpart functions derived mechanically from function definitions. GL used an interpreter to capture function behavior rather than translating definitions, and added support for an and-inverter graph (AIG) representation for Boolean functions along with links to external SAT solvers for resolving Boolean function validity. Later changes in GL added preliminary support for rewrite rules and termlike symbolic objects so as to allow for some abstraction.

FGL continues the trend toward user-definable rules displacing built-in behavior. It is a rewriter at its core, so user-defined rewrite rules are the basis of its reasoning system, rather than an add-on. Nevertheless, it comes with an extensive library of rules that replicates the automation provided by GL. Rewrite rules supported by FGL offer powerful capabilities such as programmable binding of free variables and visibility into the syntax of the rewriting targets [39]. FGL also replaces built-in primitive function symbolic counterparts with *meta rules* similar in spirit to ACL2's [23], which similarly allow directly programmable manipulation of the syntax of objects but may also be added by users. FGL adds support for incremental SAT, allowing multiple SAT checks of related formulas to share learned clauses and heuristic information. It also allows global simplification of the entire AIG using combinational circuit simplification methods. Both of these features may be invoked from within rewrite rules; e.g., if the author of a rewrite rule judges that a hypothesis of the rule is unlikely to be solved by rewriting alone, they may specify that incremental SAT should be used to prove it.

Many other projects have also aimed to allow interactive theorem provers to call on automatic decision procedures; too many such efforts exist to list them all. In higher-order logic proof assistants, several tools collectively called *hammers* translate queries into the language of an automated theorem prover

```
Listing 1.2. Semantics of a machine instruction
(defun run -inst (inst st)
  (let* ((instname (first inst))
         (args (rest inst))
         (x (first args))
         (y (second args))
         ( ans ( case instname
                (const y)
                (copy (get -st-reg y st))
                (add (+ (get -st-reg x st)
                              (get -st-reg y st)))
                (and (bitwise -and (get -st-reg x st)
                                         (get -st-reg y st)))
                (rshift (right - shift (get -st-reg y st)
                                         (get -st-reg x st ))))))
    (set -st-reg x ans st)))
```
Listing 1.3. Semantics of a straight-line code block (defun run -prog (insts st) (if (atom insts) st (let ((st (run -inst (first insts) st))) (run -prog (rest insts) st))))

and then translate the emitted proof back into a form acceptable by the original prover [12]. Several decision procedure integrations have also been carried out in ACL2. Reeber and Hunt [33] identified a decidable subclass of ACL2 list formulas and contributed a decision procedure that transforms such a formula into a SAT problem. Peng and Greenstreet [31] process a subclass of ACL2 formulas including integer and rational arithmetic, uninterpreted functions, and algebraic data structures, converting such problems to SMT queries. FGL differs by focusing on the efficient integration of user-extendible term rewriting and Boolean simplification and decision procedures.

#### 4.1 Example

We describe how FGL works at a high level by running through an example, the code of which is publicly available [40]. We define a simple machine model (Listings 1.2, 1.3) that has 16 32-bit registers and a few instructions defined, and use those instructions to implement (in straight-line code) an optimized routine to count the number of bits set in a 32-bit input (Listing 1.4), similar to implementations in *Bit Twiddling Hacks* [11]. We also define a straightforward ACL2 specification count-bits for the bit count operation (Listing 1.5). We prove that for any initial state, if we run this program on the machine, then the resulting state has its register 0 value equal to the count-bits of the value that was in register 0 before running the program (Listing 1.6).

The invocation of def-fgl-thm in Listing 1.6 causes the FGL rewriter to be applied to the conjecture. It begins by descending into the term and applying rewrite rules to subterms from the inside out. In many cases, these rules are just the definitional formulas of the functions we have introduced; for example, the definitions of run-prog, run-inst, and count-bits are used as rewrite rules, so that calls of these functions are replaced by their bodies. Rewriting the term Listing 1.4. BITCOUNT program listing

```
(defconst *bitcount*
  '((copy 10 0) ;; copy the operand to regs 10 and 11
    (copy 11 0)
    (const 5 # x55555555) ;; set reg 5 to the mask
    (and 10 5) ;; bitand the operand with the mask
    (const 0 1) ;; set reg 0 to 1
    (rshift 11 0) ;; right shift the operand by 1
    ( and 11 5) ;; mask the shifted operand
    ...
    (const 0 #x003f)
    (and 10 0) ;; mask the relevant bits of the result
    (copy 0 10))) ;; move the result to reg 0.
```
Listing 1.5. count-bits specification function

```
(defun count - bits (x)
  (if (or (not (integerp x)) (<= x 0))
      0
    (+ (nth -bit 0 x)
       (count - bits (right - shift 1 x)))))
```
while opening such definitions effectively conducts a symbolic simulation of the program and its specification. For some functions, it is preferable to avoid opening the definitions and instead use rules that rely on particular properties to simplify combinations of calls; for example, Listing 1.7 shows a rule that simplifies a read of a write of the machine state's register file.<sup>3</sup>

Rather than producing a new term as the result of rewriting each subterm, the FGL rewriter produces hybrid structures we call *symbolic objects* that may (like terms) contain function calls, variable references, and constants, but (unlike terms) also may contain *symbolic Booleans*, represented by a reference into an AIG defining a Boolean function, and *symbolic integers*, represented by a list of references into the AIG giving the two's-complement bits. Table 3 lists the variants of symbolic objects.

In order to prove this conjecture, we aim for the result of rewriting the conjecture to be a symbolic Boolean, which can then be proved valid by encoding its negation as a SAT problem. We therefore want to compute a Boolean formula equivalent to the equal comparison of the specification and implementation results. Working backwards from this goal, we can obtain this if we can represent the specification and implementation results as symbolic integers; the equal

```
Listing 1.6. Correctness theorem for BITCOUNT
(def -fgl -thm bitcount -implements - count - bits
  (let* ((input (get -st-reg 0 st))
         (final -st (run -prog * bitcount* st))
         (result (get -st-reg 0 final -st)))
    (equal result (count - bits input))))
```
<sup>3</sup> Since ACL2 is an untyped language, functions have well-defined behavior even on illtyped inputs. The uses of zero-extend in this rule reflect the choice of the definitions to coerce integers that don't fit in the allotted space into well-typed values by zeroextending them.

```
Listing 1.7. Read-over-write rule for get-st-reg
```

```
(def -fgl -rewrite get -st-reg -of-set -st-reg
  (equal (get -st-reg i (set -st-reg j v st))
          (if (equal (zero - extend 4 i) (zero - extend 4 j))
              (zero - extend 32 v)
            (get -st-reg i st))))
```
#### Table 3. Symbolic object variants

– (g-boolean *lit*) represents a Boolean, <sup>t</sup> or nil, as an AIG literal, *lit*


– (g-map *tag alist*) represents a table of key/value pairs with constant keys and symbolic values,

supporting fast lookups (see ACL2 documentation on fast alists [3])

comparison of these is the conjunction of the Boolean equivalences between all the corresponding bits. Working further backwards, we'll find that we can similarly compute these values given the bits of the intermediate integer values from which they are computed, etc., back to the original values that are components of the free variables of the conjecture. That is, generally speaking, we wish to represent every intermediate integer value as a symbolic integer. In the next two sections we will describe how to extract Boolean variables from the initial variables of the conjecture (Sect. 4.2) and how to build up Boolean formulas to represent the bits of intermediate values (Sect. 4.3).

#### 4.2 Extracting Boolean Variables

When rewriting a term in a Boolean context such as the test of an if expression, FGL will coerce the rewritten result to a symbolic Boolean object. The symbolic Boolean values of symbolic object types other than function calls and variables are easy to determine; for example, integers are non-nil and therefore considered true in ACL2. For function call and variable results, this coercion is accomplished by assigning a Boolean variable to the object, either a fresh one—a new primary input node in the underlying AIG— or an existing one when such an assignment has already been recorded for that object. These Boolean variables along with the constants t and nil are the base Boolean formulas. More complex formulas are built up from these variables by processing of if terms and by low-level meta-routines, introduced below.

The Boolean variables needed for the bitcount proof correspond to the bits of the accessed registers of the initial machine state st. We introduce rewrite rules that cause FGL to generate 32 Boolean variables for the bits of a 32-bit register when that register is accessed, composing these into a symbolic integer. The two rules involved are shown in Listing 1.8.

```
Listing 1.8. Rules for generating Boolean variables for initial register values
(def -fgl -rewrite get -st-reg -generate -bits
  (implies (syntaxp (fgl - object -case st :g-var))
            (equal (get -st-reg n st)
                    (zero - extend 32 (hide -get -st-reg n st )))))
(def -fgl -rewrite zero -extend -const - width
  (implies (syntaxp (integerp n))
            (equal (zero - extend n x)
                    (if (or (not (integerp n))
                             (<= n 0))
                        0
                      (intcons (intcar x)
                                (zero - extend (1- n) (intcdr x)))))))
```
The FGL rewriter will try to apply the first rule, get-st-reg-generate-bits, every time it encounters a call of get-st-reg, but due to its syntaxp hypothesis it will immediately fail if st is not syntactically a variable. In the case of the conjecture we're attempting to prove, this ensures that the rule will only apply to get-st-reg calls on the initial state. Such calls will be replaced by the zero-extend term of the right-hand side. In that term, hide-get-st-reg is an alias for get-st-reg; this avoids looping in the application of the rule. The construction of the 32-bit vector of Boolean variables is then accomplished by repeated application of the rule zero-extend-const-width. The functions intcar, intcdr, and intcons used here to access or construct bits of an integer as if it were a list of Booleans: intcar gets the Boolean value of the least-significant bit (LSB), intcdr right-shifts by 1 to remove the LSB, and intcons adds a new LSB to an integer, reversing the intcdr operation. The first argument to intcons is recognized by FGL as a Boolean context, so the rewriter will introduce Boolean variables corresponding to the terms that appear there, namely:

```
(intcar (intcdr ... (intcdr (hide-st-get k st)) ... ))
```
The association of each such termlike object with the corresponding Boolean variable is stored in a hash table. Each time a termlike object is found in a Boolean context, it is looked up in the table; if it has an existing entry, the corresponding Boolean variable is returned, and if not, a new Boolean variable is generated and stored.

After generating the new Boolean variable, the intcons call becomes a new symbolic integer that now includes that bit. The final value produced by the zeroextend is therefore a symbolic integer consisting of 32 fresh Boolean variables. If the same register were to be accessed again, the same process would occur except that the objects associated with the Boolean variables would be recognized and the same Boolean variables returned again.

#### 4.3 Composing Boolean Functions

The most basic way in which a new Boolean formula is computed from a previous one during FGL's rewriting process is by FGL's built-in handling of if. Specifically, if an if term occurs in which the two branches are both symbolic

```
Listing 1.9. Bitwise AND implementation rule
(def -fgl -rewrite fgl - bitwise -and
  (equal (bitwise -and x y)
         (if (int -endp - check x-endp x)
              (if (intcar x) (ifix y) 0)
            (if (int -endp - check y-endp y)
                (if (intcar y) (ifix x) 0)
              (intcons (and (intcar x)
                             (intcar y))
                        (bitwise -and ( intcdr x) (intcdr y )))))))
```
Boolean objects, the result is the Boolean if-then-else of the test formula and the two branch formulas. This if-then-else formula is built in the AIG and a reference to the resulting node is returned as the Boolean formula resulting from the if. If the two branches are both integer values represented either as symbolic integers or integer constants, then the result is a new symbolic integer, the bits of which are the if-then-elses of the test with the corresponding bits from the two branches.

As a simple example, the rule used to expand calls of bitwise-and is shown in Listing 1.9. This rewrites a call of bitwise-and on a pair of symbolic integers, producing a new symbolic integer in which each bit's formula is the AND of the corresponding bits of the inputs.

The rule applies to any call of bitwise-and. It first checks each of the inputs with int-endp-check. This is true if it can be syntactically determined that the input must be either *<sup>−</sup>*1 or 0—in particular, if the input's symbolic integer representation has only one bit. (The syntactic check works by binding its result to the free variable x-endp introduced within the form. The technical details of this rewriter feature are described elsewhere [39].) If this is true of either input, then the result is based on the one relevant bit of that input (the intcar): if it is true, then the input's value is -1 and the result is the other input (coerced to an integer value using ifix, which replaces non-integer values with 0); if false, then the input's value is 0 and therefore the result is too. In many cases, the intcar value will be a (non-constant) Boolean formula; the result of this if is then a new vector of Boolean formulas, each of which is the conjunction of the intcar formula with the corresponding bit of the other input.

If the int-endp-check test is false on both inputs, then the rule creates the first bit of the result by creating the and of the first bits of the two inputs. (In ACL2, (and x y) is really shorthand for (if x y nil), so this is actually another if merge operation.) It then makes another call of bitwise-and on the remaining bits of the two inputs, which will cause another application of this rule; this recurs until the bits of one of the inputs are exhausted.

The bitwise-and rule is a particularly simple example of how FGL can be programmed to compute complex Boolean formulas, but designing and proving these sorts of rules for other operations is a straightforward exercise in interactive theorem proving. FGL also includes a library of such rules which the user can safely extend with new rewrite rules as needed.

For some applications, the performance of stepping through iterative rules such as these using the rewriter is insufficient. For these cases, FGL supports creating custom rewriting procedures analogous to ACL2's metafunctions [23] and invoking them via rules similar to ACL2's meta rules. Metafunctions operate directly on the syntactic forms to be rewritten—symbolic objects in FGL, terms in ACL2. They return a resulting term (and substitution in FGL, though not in ACL2) that is equivalent to the input object. To allow a metafunction to be applied during rewriting, a meta rule is admitted, which requires proving a theorem stating that the metafunction produces correct results. It is noteworthy that FGL itself is proven in ACL2 to produce correct results even with user extension via rewrite rules or custom rewriting procedures.

#### 5 Conclusion

Over the past years, formal verification at Centaur has moved beyond its previous focus on data-path proofs for arithmetic modules. Our verification projects have expanded into the areas of front-end decoding and microcode, as well as the implementations of a rich set of microoperations. We engage with the design process in its early stages and maintain and expand our proofs throughout the whole life cycle of the project. Over the years, our tools have been improved and we have learned a few lessons.

We chose to use open-source tools and we are constantly contributing to ACL2 libraries. The ACL2 community has a tested way of collaboration between groups using git, peer reviewed commits, and a rich regression suite.

We write specifications that can be expanded and refined in response to design and microarchitectural changes. When the design is incomplete, the specifications are still useful when augmented by relevant assumptions. When a project requires additional flags or features, a modular style of specification allows for appropriate changes. We try to avoid complex specifications like those for the front-end decoder or ROM instruction decoder. These parts of the design are implicitly verified during microcode verification.

Scheduled, triggered, and manual regressions are an important safeguard to avoid breaking consistency among our proofs. They catch undesirable changes in the specifications, tools, and design.

A key to ensuring stability of the proofs is their scope—the bigger the scope, the more stable the proofs, because changes to interfaces of larger modules are less frequent than changes at lower levels. The transition from unit to clusterlevel proofs led to substantially higher robustness and easier maintenance. This has been possible due to improvements in the process of building our formal models and enhancements in FGL. We also benefit greatly from enhancements in modern SAT solvers.

We still have considerable work to do towards achieving our verification goals. Some of these goals could be achieved with more man power, whereas for others we do not have the right technology yet. There is a lot of microcode left to be verified. We have not verified the mechanisms of out-of-order microoperation scheduling, but we believe it is possible with our tools. We do not have a complete methodology for verification of memory access instructions yet. Our plan is to work on all these fronts.

### References


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Algebraic Program Analysis**

Zachary Kincaid1(B), Thomas Reps2(B), and John Cyphert2(B)

<sup>1</sup> Princeton University, Princeton, NJ 08540, USA zkincaid@cs.princeton.edu <sup>2</sup> University of Wisconsin, Madison, WI 53706, USA reps@cs.wisc.edu, jcyphert@wisc.edu

**Abstract.** This paper is a tutorial on algebraic program analysis. It explains the foundations of algebraic program analysis, its strengths and limitations, and gives examples of algebraic program analyses for numerical invariant generation and termination analysis.

#### **1 Introduction**

This tutorial provides an introduction to algebraic program analysis, focusing upon techniques for (numerical) invariant generation and termination analysis. By reading this paper, you will learn the answers to the following questions:


The origin of algebraic program analysis is the algebraic approach to solving path problems in graphs [1,6,48,59]: (1) compute a regular expression recognizing a set of paths of interest, and (2) interpret that regular expression within an algebraic structure corresponding to the problem at hand. Various path problems (e.g., computing shortest paths, path-finding problems, and dataflow analysis) can be solved by using different algebraic structures to interpret regular expressions.

In the context of program analysis, the graph of interest is a control flow graph for a program, and the algebra defines a space of summaries (approximations of program behavior) and a means for composing them. The algebraic approach amounts to computing a summary for a program in "bottom-up" fashion, building summaries for larger and larger subprograms by applying the operators of the summary algebra.

The general pattern of an algebraic program analysis is: given a system of (recursive) equations defining the semantics of a program, (1) symbolically compute a closed-form solution, and then (2) interpret the closed form within an algebraic structure corresponding to the analysis. The algebraic approach can be contrasted with classical iterative abstract interpretation, which also starts with a system of (recursive) equations defining the semantics of a program. However, the iterative approach is to (a) interpret the operations in the equations in an abstract domain, and then (b) solve the equations over the abstract domain by successive approximation. Thus, the classical approach is one of "interpret and then solve," whereas the algebraic approach is "solve and then interpret."

The algebraic approach can be applied to various kinds of equations and algebraic structures. Three cases we consider in this article, and the corresponding kind of program-analysis problems they can be used to solve, are:


*Why Algebraic Program Analysis?* Algebraic program analysis is a general framework for understanding compositional program analyses. The principle of compositionality states that "the meaning of a complex expression is determined by its structure and the meanings of its constituents" [57]. A program analysis is compositional when the result of analyzing a composite program is a function of the results of analyzing its components. Compositionality enables program analyses to scale to large programs, to be parallelized, to be applied incrementally, and to be applied to incomplete programs [18]. Algebraic program analysis provides a structure in which to think about how to design such an analysis.

Insistence upon compositionality also demands a different perspective on program analysis, which can suggest solutions to problems that may otherwise not be apparent. We demonstrate this principle with a series of examples that illustrate a variety of different ideas that are enabled by thinking of program analysis in compositional terms.

Last, the algebraic framework enables a style of reasoning about the behavior of program analyses themselves. By exploiting compositionality, it is possible to design effective algebraic analyses that satisfy certain laws (e.g., monotonicity— "more information in yields more information out"). Analyses can be classified on the basis of algebraic laws that they satisfy, and we can reason how program transformation affects analysis using these laws.

*Why Not Algebraic Program Analysis?* While compositionality brings many desirable properties, it comes at the price of losing *context*. Compositionality requires that the analysis of a program component is a function of the source code of that component, and therefore *cannot* depend on the surrounding context in which the component appears in the program. Many program analysis techniques make essential use of context, for example:


One of the main challenges of designing a good algebraic program analysis is to overcome this loss of contextual information.

Secondly, algebraic program analysis is less general than iterative program analysis, in the sense that any set of semantic (in)equations can be solved iteratively using the same basic algorithm, whereas each particular type of equation system requires a specialized algorithm. Some problems—e.g., resolving semantic equations of recursive procedures—have no known practical algebraic solutions.

#### **2 Regular Algebraic Program Analysis**

This section describes the algebraic approach to solving path problems in graphs [1,6,48,59]. The basic structure of the method is to use regular expressions to capture the set of paths of a graph, and then *interpret* these expressions to obtain a desired result. We illustrate the approach by considering the problem of computing shortest paths, and then show how it can be applied to numerical invariant generation.

First, we establish some basic definitions. The syntax of **regular expressions** over an alphabet Σ is as follows:

$$a \in \Sigma$$

$$R \in \mathsf{RegExp}(\Sigma) \text{ ::= } a \mid 0 \mid 1 \mid R\_1 + R\_2 \mid R\_1 \cdot R\_2 \mid R^\* \cdot \mathsf{R}$$

We will sometimes use juxtaposition <sup>R</sup>1R<sup>2</sup> (rather than <sup>R</sup><sup>1</sup> · <sup>R</sup>2) to denote concatenation.

The semantics of regular expressions over Σ is given by a Σ**-interpretation** *<sup>I</sup>* <sup>=</sup> **A**, f, which consists of *regular algebra* **<sup>A</sup>** and a *semantic function* <sup>f</sup>. A **regular algebra A** = - A, <sup>0</sup><sup>A</sup>, <sup>1</sup><sup>A</sup>, <sup>+</sup><sup>A</sup>, · <sup>A</sup>, <sup>∗</sup><sup>A</sup> is an algebraic structure consisting of a set A (called its *universe*) equipped with two distinguished elements <sup>0</sup><sup>A</sup>, <sup>1</sup><sup>A</sup> <sup>∈</sup> <sup>A</sup>, two binary operations +<sup>A</sup> (*choice*) and · <sup>A</sup> (*sequencing*), and a unary operation (−)∗<sup>A</sup> (*iteration*).<sup>1</sup> When the algebra is clear from context, we will drop the superscript. A **semantic function** <sup>f</sup> : <sup>Σ</sup> <sup>→</sup> <sup>A</sup> maps each letter in Σ to an element of **A**'s universe.

<sup>A</sup> <sup>Σ</sup>-interpretation *<sup>I</sup>* <sup>=</sup> **A**, f assigns to each regular expression <sup>R</sup> over <sup>Σ</sup> to an element *I* -R of **A** by interpreting each letter according to the semantic function and each regular operator using its counterpart in **A**:

$$\begin{aligned} \mathcal{J}\begin{bmatrix} 0 \end{bmatrix} &= 0^A & \mathcal{J}\begin{bmatrix} R\_1 \cdot R\_2 \end{bmatrix} &= \mathcal{J}\begin{bmatrix} R\_1 \end{bmatrix} \cdot ^A \mathcal{J}\begin{bmatrix} R\_2 \end{bmatrix} \\ \mathcal{J}\begin{bmatrix} 1 \end{bmatrix} &= 1^A & \mathcal{J}\begin{bmatrix} R\_1 + R\_2 \end{bmatrix} &= \mathcal{J}\begin{bmatrix} R\_1 \end{bmatrix} + ^A \mathcal{J}\begin{bmatrix} R\_2 \end{bmatrix} \\ \mathcal{J}\begin{bmatrix} a \end{bmatrix} &= f(a) & \text{For } a \in \Sigma & \mathcal{J}\begin{bmatrix} R^\* \end{bmatrix} = \mathcal{J}\begin{bmatrix} R \end{bmatrix}^A \end{aligned}$$

Notice that the interpretation is compositional: for any expression R, *I* -R is a function of the top-level operator in R and the interpretations of its subexpressions.

<sup>1</sup> Note that no particular laws are assumed to govern these operations. We will return to this issue in Sect. 3.

*Example 1 (Standard interpretation).* The *standard* interpretation of regular expressions is the language interpretation, *<sup>L</sup>* <sup>=</sup> **L**, where **<sup>L</sup>** is the regular algebra of languages. The universe of the interpretation is the set of regular languages over Σ, 0 - ∅ is the empty language, 1 - {} is the singleton language containing the empty word, and the operators are

$$\begin{aligned} X + Y &\triangleq X \cup Y & \text{Uniform} \\ X \cdot Y &\triangleq \{ xy : x \in X, y \in Y \} & \text{Concatenation} \\ X^\* &\triangleq \{ x\_1 x\_2 \ldots x\_n : x\_1, \ldots, x\_n \in X \} & \text{Kleene closure} \end{aligned}$$

The semantic function maps each letter <sup>a</sup> to the singleton language {a}. For any regular expression R, *L* -R is the (regular) set of words recognized by R. ⌟

We now describe how *non-standard* interpretations can be used to solve problems over directed graphs. A **directed graph** <sup>G</sup> <sup>=</sup> V,E consists of a finite set of vertices <sup>V</sup> and a finite set of directed edges <sup>E</sup> <sup>⊆</sup> <sup>V</sup> <sup>×</sup> <sup>V</sup> . A **path** in <sup>G</sup> is a finite sequence <sup>e</sup>1e<sup>2</sup> ...e<sup>n</sup> with <sup>e</sup><sup>i</sup> <sup>∈</sup> <sup>E</sup> such that for each <sup>i</sup>, the destination of <sup>e</sup><sup>i</sup> matches the source of ei+1. A **path expression** (in G) is a regular expression over the alphabet of edges E that recognizes a set of paths in G. For any pair of vertices u, v <sup>∈</sup> <sup>V</sup> , there is a path expression *PathExp*G(u, v) that recognizes exactly the set of paths in G that begin at u and end at v. There are several ways to compute path expressions. The classical method is Kleene's algorithm [44] for computing a regular expression for a finite state automaton (thinking of G as an automaton over the alphabet E with start state u and final state v). For sparse graphs, there are more efficient alternatives to Kleene's algorithm, in particular Tarjan's algorithm [58]. The insight of the algebraic approach to path problems is that these algorithms can be re-used for multiple purposes: first use a path expression algorithm to find a regular expression recognizing a set of paths of interest, and then compute a problem-dependent (non-standard) interpretation of that expression.

*Example 2 (Shortest paths).* Consider the integer-weighted graph depicted in Fig. 1a. Suppose that we wish to compute the length of the shortest path from a to c. We begin by computing a path expression recognizing all paths from a to c:

$$\left\{ \left< a, b \right> \left< b, d \right> \left( \left< d, e \right> \left< e, d \right> \right)^{\*} \left< d, a \right> \right\}^{\*} \left< a, b \right> \left( \left< b, c \right> + \left< b, d \right> \left( \left< d, e \right> \left< e, d \right> \right)^{\*} \left< d, c \right> \right) \right.$$

This path expression can be represented succinctly by the directed acyclic graph (DAG) pictured in Fig. 1b. Define the *distance interpretation D* where the semantic function maps each edge to its weight, and the algebra's universe consists of the integers along with ±∞, 0 is interpreted as ∞, 1 as 0, and the operators are as follows:

**Fig. 1.** An integer weighted graph and a path expression DAG representing the paths from a to c

$$d\_1 + d\_2 \stackrel{\Delta}{=} \min(d\_1, d\_2) \tag{Minimum} $$

$$d\_1 \cdot d\_2 \stackrel{\Delta}{=} d\_1 + d\_2 \quad \text{Addition} \tag{Addition}$$

$$d^\* \stackrel{\Delta}{=} \begin{cases} -\infty & \text{if } d < 0\\ 0 & \text{otherwise} \end{cases} \tag{Closure}$$

The weight of the shortest weighted path from a to c is *D*-*PathExp*G(a, c) = 1, which can be calculated efficiently by interpreting the path expression DAG "bottom-up" (see gray labels in Fig. 1b). ⌟

Algebraic path-finding can be used to generate invariants by representing a program by a *control flow graph*, and interpreting path expressions within an algebra of program summaries. A **control flow graph** (CFG) <sup>G</sup> <sup>=</sup> V, E, r, C is a directed graph V,E with a distinguished root (or entry) vertex <sup>r</sup> <sup>∈</sup> <sup>V</sup> , and where each edge <sup>e</sup> <sup>∈</sup> <sup>E</sup> is labeled by a command <sup>C</sup>(e); see Fig. 2a for an example. In the remainder of this section, we give examples of interpretations that can be used to generate (numerical) program summaries.

#### **2.1 Transition-Formula Interpretations**

Fix a finite set of variables, X, representing the variables of a program. A **transition formula** is a logical formula F(X, X ) whose free variables range over X

and a set of "primed copies" X - {x : <sup>x</sup> <sup>∈</sup> <sup>X</sup>}. For the purposes of this exposition, we further suppose that variables range over integers, and that transition formulas are expressed in the language of linear integer arithmetic. A transition formula can be interpreted as a binary relation <sup>→</sup><sup>F</sup> over states State - ZX, where <sup>s</sup> <sup>→</sup><sup>F</sup> <sup>s</sup> if and only if <sup>F</sup> is true when <sup>s</sup> is used to interpret the un-primed variables and s is used to interpret the primed variables. For example, if F is the transition formula

$$F \triangleq x' = x + 1 \land y = y' \land x < y \; ,$$

then we have

<sup>s</sup> <sup>→</sup><sup>F</sup> <sup>s</sup> ⇐⇒ <sup>s</sup> (x) = s(x)+1, s(y) = s (y), and s(x) < s(y) .

Suppose that <sup>G</sup> <sup>=</sup> V, E, r, C is a control flow graph, where commands range over assignments *x* := *e* and assumptions [*c*], where e is a linear integer term and c is a linear arithmetic formula. (An assumption [*c*] is a command that does not change the program state, but which can only be executed if the formula *c* holds.) We define a semantic function *tf* that maps each control flow edge into the universe of transition formulas by translating the command associated with the edge into logic:

$$\mathfrak{gl}(e) \triangleq \begin{cases} (x'=e) \wedge \left(\bigwedge\_{y \neq x \in X} y'=y\right) & \text{if } C(e) \text{ is } x := e\\ c \wedge \left(\bigwedge\_{y \in X} y'=y\right) & \text{if } C(e) \text{ is } \mathbb{I}\{c\} \end{cases}$$

We define an algebra of transition formulas as follows:

$$\begin{aligned} 0 & \triangleq false & \text{Empty relation} \\ 1 & \triangleq \bigwedge\_{x \in X} x' = x & \text{Identity relation} \\ F + G & \triangleq F \vee G & \text{Untion} \\ F \cdot G & \triangleq \exists X''.F(X, X'') \wedge G(X'', X') & \text{Relational composition} \end{aligned}$$

Above and elsewhere, we use positional notation for substitution; e.g., F(X, X) denotes the formula obtained by replacing all the X symbols with "double primed" symbols in X (and leaving the un-primed X symbols as they are). Intuitively, F<sup>∗</sup> should be interpreted as the reflective transitive closure of F. However, in general it is not possible to compute the reflexive transitive closure of a formula (nor even to *represent* it as a formula). Hence, we must be content with an over-approximate transitive closure operator. There are many different methods for over-approximating transitive closure, so we speak of the *family* of algebras of transition formulas, which have the same basic structure and differ only in the interpretation of the iteration operator. In the remainder of this section, we describe a selection of methods for implementing the iteration operator. *Disclaimer* : for each example, the presentation differs somewhat (sometimes substantially) from the cited source. The examples should be read as "how the cited analysis might be presented in the algebraic framework."

*Example 3 (Transitive Predicate Abstraction* [47]*).* Fix a set of variables X. Say that a transition formula p(X, X ) is

– *reflexive* if <sup>x</sup>∈<sup>X</sup> <sup>x</sup> <sup>=</sup> <sup>x</sup> <sup>|</sup><sup>=</sup> <sup>p</sup>(X, X ) – *transitive* if p(X, X ) <sup>∧</sup> <sup>p</sup>(X , X) <sup>|</sup><sup>=</sup> <sup>p</sup>(X, X)

Let P be a finite set of *candidate* reflexive and transitive transition formulas. For example we might choose

$$\begin{array}{l} P \triangleq \left\{ x \bowtie x' : x \in X, \bowtie \left\{ \leq, \geq \right\} \right\} \\ \cup \left\{ x \bowtie 0 \Rightarrow x' \bowtie 0 : x \in X, \bowtie \left\{ \leq, \geq, <, > \right\} \right\} \end{array}$$

We can define an iteration operator that over-approximates the reflexive transitive closure of a formula F by the conjunction of the subset of P that is entailed by F:

$$F^\* \triangleq \bigwedge \{ p \in P : F \mid = p \} \ . \tag{7}$$

*Example 4 (Interval analysis* [51]*).* Let F(X, X ) be a transition formula. An *inductive interval invariant* for <sup>F</sup> assigns to each variable <sup>x</sup> <sup>∈</sup> <sup>X</sup> a pair of integers <sup>a</sup>x, b<sup>x</sup> <sup>∈</sup> <sup>Z</sup> such that if <sup>s</sup> is a state such that <sup>s</sup>(x) <sup>∈</sup> [ax, bx] for all <sup>x</sup> <sup>∈</sup> <sup>X</sup> and <sup>s</sup> <sup>→</sup><sup>F</sup> <sup>s</sup> , then s (x) <sup>∈</sup> [ax, bx] for all <sup>x</sup> <sup>∈</sup> <sup>X</sup>. Monniaux showed that it is possible to determine optimal inductive interval invariants by posing the inductive-invariance condition symbolically and quantifying over the bounds [51].

Let <sup>P</sup> <sup>=</sup> {p<sup>x</sup> : <sup>x</sup> <sup>∈</sup> <sup>X</sup>} and <sup>Q</sup> - {q<sup>x</sup> : <sup>x</sup> <sup>∈</sup> <sup>X</sup>} be sets of fresh variables, which we use to the lower and upper bounds of intervals, respectively. The set of inductive interval invariants for a formula F can be represented by the formula

$$Inv(F, P, Q) \triangleq \forall X, X'. \left( F \land \bigwedge\_{x \in X} p\_x \le x \le q\_x \right) \Rightarrow \left( \bigwedge\_{x \in X} p\_x \le x' \le q\_x \right).$$

That is, the models of *Inv* (which assign integers to the lower and upper bound variables P and Q) are in one-to-one correspondence with the interval invariants of F. We may universally quantify over all inductive interval invariants to arrive at the following iteration operator:

$$F^\* \stackrel{\Delta}{=} \forall P, Q. \left( Inv(F, P, Q) \land \bigwedge\_{x \in X} p\_x \le x \le q\_x \right) \Rightarrow \left( \bigwedge\_{x \in X} p\_x \le x' \le q\_x \right).$$

In contrast to the typical iterative approach with classical widening and narrowing operators, this operator computes a formula that implies *all* (and therefore *most precise*) inductive interval invariants.<sup>2</sup> For example, for the

<sup>2</sup> Note that while the formula implies all interval invariants, it does not *itself* take the form of an interval invariant.

loop (**while** (*i* = *n*) **do** *i* := *i* + 1), this method yields the following overapproximation of the reflexive transitive closure of F:

$$F^\* \equiv n' = n \land i \le i' \land (i \le n \Rightarrow i' \le n).$$

If we suppose that i is initially 0 and n is initially 100, then this formula implies the loop invariant that n is equal to 100, and i is in the interval [0, 100]. ⌟

*Example 5 (Recurrence analysis* [4,27]*).* Let F(X, X ) be a transition formula, and let **x** and **x** denote vectors containing the variables X and X , respectively. A *linear recurrence inequation of* F is a formula of the form **ax** ≤ **ax** + b that is entailed by F. The idea behind recurrence analysis is to extract a set of linear recurrence inequations for a formula, {**a**- <sup>i</sup> **<sup>x</sup>** <sup>≤</sup> **<sup>a</sup>**- <sup>i</sup> **<sup>x</sup>** <sup>+</sup> <sup>b</sup>i}<sup>i</sup>∈<sup>I</sup> , and to use the closed form of those recurrences to over-approximate the transitive closure of F:

$$F^\* \triangleq \exists k. k \ge 0 \land \bigwedge\_{i \in I} \mathbf{a}\_i^\mathsf{T} \mathbf{x}' \le \mathbf{a}\_i^\mathsf{T} \mathbf{x} + k b\_i$$

For instance, consider the following loop:

**while** (*x* > 0) **do if** (*y* < 0) { *x* := *x* + *y*; *y* := *y* -1} **else** { *x* := *x* - 2; *y* := *y* - 3}

The loop exhibits the following recurrences

$$
\begin{array}{c}
\left(2x'-y'\right) \le \left(2x-y\right)-1 \\
y' \le y-1 \\
\end{array}
\quad \text{or in matrix form,}
\begin{bmatrix}2-1 \\ 0 \\ 0-1\end{bmatrix}
\begin{bmatrix}x' \\ y'\end{bmatrix} \le \begin{bmatrix}2-1 \\ 0 \\ 0-1\end{bmatrix}
\begin{bmatrix}x \\ y\end{bmatrix} + \begin{bmatrix}-1 \\ -1 \\ 3\end{bmatrix}.
$$

which yields the following transition formula that summarizes the loop:

$$\exists k. k \ge 0 \land (2x' - y') \le (2x - y') - k \land y' \le y - k \land -y' \le -y + 3k \quad \dots$$

The loop also exhibits other recurrences (such as <sup>x</sup> <sup>≤</sup> <sup>x</sup>−1); however, the three selected recurrences are complete in the sense that all implied recurrences are non-negative linear combinations of these three (e.g., <sup>x</sup> <sup>≤</sup> <sup>x</sup> <sup>−</sup> 1 is obtained by adding 1/2-times the first and second recurrences).

Such a complete set of recurrences exists for any transition formula F, which can be computed as follows. First, observe that the set of linear recurrences of F,

$$\operatorname{Rec}(F) \stackrel{\Delta}{=} \{ (\mathbf{a}, b) : F \mid = \mathbf{a}^\mathsf{T} \mathbf{x}' \le \mathbf{a}^\mathsf{T} \mathbf{x} + b \}$$

is closed under non-negative linear combinations (i.e., it is a *convex cone*). Our goal is to find a (finite) set of generators for *Rec*(F)—a finite set {(**a**i, bi)}<sup>i</sup>∈<sup>B</sup> such that

$$\operatorname{Rec}(F) = \left\{ (0, \lambda\_0) + \sum\_{i \in B} \lambda\_i (\mathbf{a}\_i, b\_i) : \lambda\_0 \ge 0, \lambda\_i \ge 0 \text{ for all } i \in B \right\}.$$

To compute generators for *Rec*(F), we first introduce a fresh set of "difference" variables, {δx}x∈<sup>X</sup> and form a formula

$$\Delta(F) \stackrel{\Delta}{=} \exists X, X'.F \land \bigwedge\_{x \in X} \delta\_x = x' - x \ . ,$$

Observe that (**a**, b) <sup>∈</sup> *Rec*(F) if and only if <sup>Δ</sup>(F) <sup>|</sup><sup>=</sup> **<sup>a</sup>**<sup>δ</sup> <sup>≤</sup> <sup>b</sup>. Thus, a set of generators for *Rec*(F) corresponds exactly to a half-space representation for the convex hull of Δ(F), which can be computed using the algorithm from [27].

The class of linear recurrence inequations considered in this example can be generalized in various ways to yield more powerful invariant generation procedures. In particular,


#### **2.2 Weak Interpretations**

Transition formulas are an appealing basis for algebraic program analysis, since all the operators (except the iteration operator) are *precise*—they simply encode the meaning of the program into logic. The significance of this is that transition formula algebras delay precision loss *as long as possible*, which helps to overcome loss of contextual information. However, there are algebraic analyses of interest that are defined on weak logical fragments that cannot precisely express union and/or relational composition.

*Example 6 (Affine relation analysis* [38]*).* An *affine relation* is a relation that corresponds to the set of models of a transition formula of the form A**x** = B**x**+c. Define the algebra of affine transition relations to be the regular algebra where the universe is the set of affine transition relations, 0 is interpreted as the empty relation, 1 is interpreted as the identity relation, + is interpreted as the affine hull of <sup>R</sup><sup>1</sup> <sup>∪</sup> <sup>R</sup><sup>2</sup> (the smallest affine relation that contains both <sup>R</sup><sup>1</sup> and <sup>R</sup>2), · is interpreted as relational composition, and ∗ is interpreted as the operation that sends any affine relation <sup>R</sup> to the limit of the sequence {Ri}<sup>∞</sup> <sup>i</sup>=0 defined by

$$R\_0 = 0 \qquad \qquad R\_{i+1} = R\_i + (R\_i \cdot R) \text{ for } i \ge 0$$

Since we have <sup>R</sup><sup>0</sup> <sup>⊆</sup> <sup>R</sup><sup>1</sup> <sup>⊆</sup> ... and if any <sup>R</sup>i+1 properly contains <sup>R</sup><sup>i</sup> the dimension of Ri+1 is strictly greater than that of Ri, this sequence must stabilize in finite time, so the operation R<sup>∗</sup> is computable. ⌟

#### **3 Semantic Foundations**

This section presents a general view of algebraic program analysis, with the goal of elucidating its underlying principles so that they may be understood outside the setting of graphs and regular expressions. This sets the stage for Sect. 4 and Sect. 5, wherein we will develop program analysis schemes that follow the same general "recipe" that we lay out in this section, but deviate from the instance of this recipe that we saw in Sect. 2.

Following the theory of abstract interpretation [22], we begin with a *concrete semantics* that defines the meaning of a program. The concrete semantics is specified as the least (or greatest) solution to a system of recursive equations. The concrete semantics is not computable—the goal of a program analysis is to *approximate* it. The way that this is accomplished in an algebraic analysis is by symbolically computing a closed-form solution to the semantic equations (i.e., a non-recursive system of equations whose (unique) solution coincides with the concrete semantics), and then interpreting that closed-form solution in an algebraic structure that approximates the algebra of the concrete semantics.

#### **3.1 Semantic Equations**

Given a control flow graph G, we can syntactically derive a system of equations E(G)—see Fig. 2. For each vertex v, we introduce a variable X<sup>v</sup> and an equation (X<sup>v</sup> = Rv) that relates that variable to the variables for v's predecessors. Notice that this system of equations can be viewed as a (left-)regular grammar, with each non-terminal symbol X<sup>v</sup> recognizing the set of paths from the root r to the vertex v. This is an instance of the more general concept of a solution to a system of equations over an algebraic structure. A *solution* to the system of equations <sup>E</sup>(G) = {X<sup>v</sup> <sup>=</sup> <sup>R</sup>v}<sup>v</sup>∈<sup>V</sup> over a regular interpretation *<sup>I</sup>* <sup>=</sup> **A**, f is a function <sup>σ</sup> that maps each variable to an element of **A** such that each equation is satisfied: for each equation (X<sup>v</sup> = Rv) in E(G), we have σ(X) = *I*σ-R, where *I*<sup>σ</sup> is the interpretation obtained by extending the semantic function to variables by interpreting them according to σ.

The prototypical concrete semantics of interest in algebraic analysis is the relational semantics. The **relational semantics** of a program associates to every control flow vertex <sup>v</sup> a reachability relation <sup>R</sup>v, which is the set of pairs s, s such that if the program begins at r in state s, then it may reach v with state s . The relational semantics may be obtained as the least solution to the system of semantic equations over the relational interpretation, which is defined as follows. The regular algebra of state relations, **R**, has binary relations on states as its universe, 0 is interpreted as the empty relation ∅, 1 is interpreted as the identity relation {s, s : <sup>s</sup> <sup>∈</sup> State}, · is interpreted as relational composition, + as union, and ∗ as reflexive, transitive closure. The **relational interpretation** *R* is the interpretation over the regular algebra of state relations where the semantic function maps each command to its associated transition relation; e.g., *<sup>i</sup>* := *<sup>i</sup>* + 2 is associated with the set of all pairs s, s such that <sup>s</sup> (*i*) = s(*i*)+1 and s (x) = <sup>s</sup>(x) for all <sup>x</sup> <sup>=</sup> *<sup>i</sup>*. The relational semantics of a CFG <sup>G</sup> is the least solution to E(G) over the relational interpretation.

Having formulated the concrete semantics as the solution to a system of equations, we must now solve the system symbolically. The classical algorithm is a variation of Gaussian elimination, given in Algorithm 1. This algorithm is essentially Kleene's algorithm [44] for computing a regular expression for a

$$\begin{aligned} X\_b &= \langle r, a \rangle \left\langle a, b \right\rangle \left( \left\langle b, c \right\rangle \left\langle c, d \right\rangle \left( \left\langle d, b \right\rangle + \left\langle d, e \right\rangle e, b \right) \right)^\* \\ X\_c &= \left\langle r, a \right\rangle \left\langle a, b \right\rangle \left( \left\langle b, c \right\rangle \left\langle c, d \right\rangle \left( \left\langle d, b \right\rangle + \left\langle d, e \right\rangle e, b \right) \right)^\* \left\langle b, c \right\rangle \\ X\_d &= \left\langle r, a \right\rangle \left\langle a, b \right\rangle \left( \left\langle b, c \right\rangle \left\langle c, d \right\rangle \left( \left\langle d, b \right\rangle + \left\langle d, e \right\rangle e, b \right) \right)^\* \left\langle b, c \right\rangle \left\langle c, d \right\rangle \\ X\_e &= \left\langle r, a \right\rangle \left\langle a, b \right\rangle \left( \left\langle b, c \right\rangle \left\langle c, d \right\rangle \left( \left\langle d, b \right\rangle + \left\langle d, e \right\rangle e, b \right) \right)^\* \left\langle b, c \right\rangle \left\langle c, d \right\rangle \left\langle d, e \right\rangle \\ X\_f &= \left\langle r, a \right\rangle \left\langle a, b \right\rangle \left( \left\langle b, c \right\rangle \left\langle c, d \right\rangle \left( \left\langle d, b \right\rangle + \left\langle d, e \right\rangle e, b \right) \right)^\* \left\langle b, f \right\rangle \end{aligned}$$

(c)

**Fig. 2.** (a) A control flow graph; (b) the corresponding systems of equations; and (c) a closed-form solution.

finite state automaton, recast in the language of equations. The front-solving step eliminates variables one-by-one, at each step i producing a system of equation of equations that is equivalent to the original, but in which the variable <sup>X</sup><sup>i</sup> does not appear in the right-hand-side of any equations <sup>X</sup><sup>j</sup> <sup>=</sup> <sup>R</sup><sup>j</sup> for <sup>j</sup> <sup>≥</sup> <sup>i</sup>. The back-solving step eliminates all variable occurrences from right-hand-sides, at each step replacing X<sup>i</sup> with its closed form R<sup>i</sup> in each equation X<sup>j</sup> = R<sup>j</sup> for j<i. An example illustrating the result of solving the system of equations in Fig. 2b symbolically appears in Fig. 2c. The significant difference to the familiar Gaussian elimination algorithm in linear algebra is the "loop-solving" step, which solves a single recursive equation X<sup>i</sup> = R<sup>i</sup> symbolically by re-arranging R<sup>i</sup> into the form XiA + B and taking BA<sup>∗</sup> to be the solution. The loop-solving step is justified under the relational interpretation, and more generally for any interpretation over a *Kleene algebra*. 3

<sup>3</sup> The laws of Kleene algebra are not minimal in this regard.

**Input :** Left-linear system of equations, <sup>E</sup> <sup>=</sup> {X<sup>i</sup> <sup>=</sup> <sup>R</sup>i}<sup>n</sup> i=1 **Output :** Closed-form solution to E **for** i *= 1 to* n **do** /\* Front-solving \*/ Re-arrange R<sup>i</sup> in the form XiA + B; <sup>R</sup><sup>i</sup> <sup>←</sup> BA<sup>∗</sup> ; /\* "Loop-solving" \*/ **foreach** j>i **do** <sup>R</sup><sup>j</sup> <sup>←</sup> <sup>R</sup><sup>j</sup> [<sup>X</sup> → <sup>R</sup>i] ; **end for** i *=* n *to 2* **do** /\* Back-solving \*/ **foreach** j<i **do** <sup>R</sup><sup>j</sup> <sup>←</sup> <sup>R</sup><sup>j</sup> [X<sup>i</sup> → <sup>R</sup>i] ; **end return** E; **Algorithm 1:** Gaussian elimination for left-linear systems of equations

**Definition 1.** *Let* **<sup>A</sup>** <sup>=</sup> A, <sup>+</sup>, ·, <sup>∗</sup>, <sup>0</sup>, <sup>1</sup> *be a regular algebra. We say that* **<sup>A</sup>** *is an idempotent semiring if it satisfies the following (for all* a, b, c,<sup>∈</sup> <sup>A</sup>*):*


*In any idempotent semiring, we may define a natural order* <sup>≤</sup>*, where* <sup>a</sup> <sup>≤</sup> <sup>b</sup> *iff* a + b = b*. Note that* + *is the least upper bound with respect to this order.*

*We say that* **A** *is a Kleene algebra if it is an idempotent semiring and the following hold (for all* a, x <sup>∈</sup> <sup>A</sup>*):*

> 1 + a(a∗) = a<sup>∗</sup> 1+(a∗)a = a<sup>∗</sup> *Unfolding* ax <sup>≤</sup> <sup>x</sup> <sup>⇒</sup> <sup>a</sup>∗<sup>x</sup> <sup>≤</sup> x xa <sup>≤</sup> <sup>x</sup> <sup>⇒</sup> xa<sup>∗</sup> <sup>≤</sup> <sup>x</sup> *Induction*

*Exercise 1.* Show that in any Kleene algebra, the least solution to a (left-)linear recursive equation X = a + Xb exists and is equal to ab<sup>∗</sup>

The sense in which Gaussian elimination computes a "closed-form solutions" to a system of left-linear equations E is that:


The connection between Gaussian elimination and graph algorithms like Floyd-Warshall inspired Tarjan's path-expression algorithm [58]. In the language of graphs, Tarjan's algorithm computes for each vertex v of a control flow graph G with root r a path expression *PathExp*G(r, v) that recognizes the set of paths from r to v; in the language of equations, it solves left-linear systems of equations symbolically. Tarjan's algorithm is preferred to Gaussian elimination in practice: is more efficient (nearly linear time for reducible flow graphs, compared to cubic time for Gaussian elimination) and produces simpler solutions. For expository purposes, we will continue to refer to Gaussian elimination for solving systems of equations, viewing Tarjan's method as an efficient variation.

#### **3.2 Abstract Interpretation**

Gaussian elimination can solve a system of left-linear equations over a Kleene algebra (e.g., relational semantics) symbolically. However, the solution cannot be interpreted in the concrete algebra, since operators are not effective (that is, they cannot be implemented by a machine). We approximate the concrete semantics by interpreting the closed-form solution in an effective *abstract* algebra (e.g., one of the transition-formula algebras from Sect. 2).

Following the theory of abstract interpretation [22], the correctness of this approach is justified by establishing a relationship between the "concrete" and "abstract" interpretations. In the algebraic framework, a natural way to express the relationship is via a *soundness relation* [24], which is a binary relation between two algebras that is preserved by the operations of the algebra. Membership of a (concrete, abstract) pair in the relation indicates that the concrete element is approximated by the abstract element.

**Definition 2 (Soundness relation).** *Given two* Σ*-interpretations I* = **A**, f *and I* = **A**, f *,* <sup>−</sup> − ⊆ <sup>A</sup> <sup>×</sup> <sup>A</sup> *is a soundness relation if* <sup>f</sup>(a) <sup>f</sup>(a) *for all* <sup>a</sup> <sup>∈</sup> <sup>Σ</sup> *and is a sub-algebra of the product algebra* **<sup>A</sup>** <sup>×</sup> **<sup>A</sup>***; i.e.,* <sup>0</sup> <sup>0</sup>*,* <sup>1</sup> <sup>1</sup>*, and for all* <sup>x</sup><sup>1</sup> <sup>y</sup><sup>1</sup> *and* <sup>x</sup><sup>2</sup> <sup>y</sup><sup>2</sup> *we have*

*–* x<sup>1</sup> + x<sup>2</sup> y<sup>1</sup> + y<sup>2</sup> *–* <sup>x</sup><sup>1</sup> · <sup>x</sup><sup>2</sup> <sup>y</sup><sup>1</sup> · y<sup>2</sup> *–* x∗- <sup>1</sup> <sup>y</sup>∗ 1

The definition of soundness relation generalizes to interpretations over other classes of algebraic structures in the natural way: it is a binary relation over two algebras of the same signature that is preserved by every operation in the signature.

*Example 7 (Transition formula overapproximation).* Let **R** denote the algebra of state relations and **TF** denote an algebra of transition formulas. The *overapproximation* relation is defined by

$$R \Vdash\_O F \iff \forall \langle s, s' \rangle \in R, s \to\_F s'.$$

Preservation of constants and the sequencing and choice operations is easily verified; to show that <sup>O</sup> is a soundness relation, we need only to show that R <sup>O</sup> F implies R∗**<sup>R</sup>** <sup>O</sup> <sup>F</sup>∗**TF** ; i.e., (−)∗**TF** over-approximates reflexive transitive closure. Of course, this proof depends on the particular implementation of the iteration operator.

The over-approximate soundness relation allows us to *verify* safety properties: if R <sup>O</sup> F and F entails some property P, then R satisfies P. ⌟ *Example 8 (Transition formula underapproximation).* The *under-approximation* relation is defined by

$$R \Vdash\_U F \iff \forall s, s'. s \to\_F s' \Rightarrow \langle s, s' \rangle \in R,$$

Preservation of constants and the sequencing and choice operations is again easily verified; to show that <sup>U</sup> is a soundness relation, we need only to show that R <sup>O</sup> F implies R∗**<sup>R</sup>** <sup>O</sup> <sup>F</sup>∗**TF** ; i.e., (−)∗**TF** under-approximates reflexive transitive closure. The iteration operators in Sect. 2 are all over-approximate. An example of an under-approximate iteration operator is

$$F^\* \triangleq \bigvee\_{i=0}^n \underbrace{F \circ \cdots \circ F}\_{i \text{ times}}$$

(for some fixed choice of n) which corresponds to bounded model checking [9], with an unrolling bound of n.

The under-approximate soundness relation allows us to *refute* safety properties: if R <sup>U</sup> F and F does not entail some property P, then R does not satisfy P. ⌟

The problem of "approximating the behavior of a program" can be formalized as follows:

Given a system of semantic equations over a set of variables X describing the concrete semantics of a program (i.e., its least solution σ over some interpretation *<sup>I</sup>* ), find some <sup>σ</sup> : X → **<sup>A</sup>** such that for each variable <sup>X</sup> ∈ X , we have <sup>σ</sup>(X) <sup>σ</sup>(X).

The algebraic approach to this problem is to compute for each variable X a closed form R<sup>X</sup> (such that σ(X) = *I* (RX)), and define σ(X) - *I* (RX). The correctness of this approach is justified by the following soundness lemma, which follows by induction on regular expressions.

**Lemma 1 (Soundness).** *Let* Σ *be an alphabet, let I* = **A**, f *and I* = **A**, f *be* <sup>Σ</sup>*-interpretations, and let* <sup>⊆</sup> <sup>A</sup> <sup>×</sup>A *be a soundness relation. Then for any regular expression* <sup>R</sup> <sup>∈</sup> *RegExp*(Σ)*, we have <sup>I</sup>* -R *I* -R

#### **3.3 Discussion**

A subtlety of algebraic program analysis is that most algebras of interest in program analysis are *not* Kleene algebras (for instance, none of the algebras in Sect. 2 are), and so in general, Gaussian elimination does not find solutions to systems of equations over "abstract" interpretations corresponding to program analyses. This technical difficulty is sidestepped by appealing to the concrete semantics (which typically *is* defined over a Kleene algebra, such as the algebra of state relations) to justify the use of path-expression algorithms, and a sound approximating algebra to interpret the resulting expressions. The fact that the abstract interpretation of the closed-form solution to the concrete system of equations does not yield a solution to the abstract system of equations is immaterial: our goal is to *overapproximate* the concrete rather than *solve* the abstract.

Formalizing a program analysis as an algebraic structure allows one to understand the behavior of program analyses in terms of algebraic laws, and use the language of algebra to reason about program analyses. For example, any transition formula algebra (in the family described in Sect. 2.1) is an idempotent semiring, and so any two ∗-free regular expressions that denote the same language have the same (up to logical equivalence) interpretation as a transition formula. While none of the iteration operators in Sect. 2.1 satisfy the *Unfolding* and *Induction* laws of Kleene algebra, they do satisfy weaker *pre-Kleene algebra* iteration laws:


A concrete use-case for these laws appears in [25], which develops regular expression transformation techniques that preserve concrete semantics but are guaranteed to produce (non-strictly) more precise abstract semantics.

Such laws can also be useful for users of program analysis tools. For example, since all operations are monotone (as a consequence of the monotonicity and idempotent-semiring laws), a user can rely on the principle that "more information in yields more information out." If a user alters a program P by adding additional *assume* commands to get a program P (e.g., expressing invariants that are found by some other automated invariant generation technique, user-provided hints, etc.), monotonicity means that they may rely on the fact that the analysis will produce summaries for P that are at least as precise as those for P.

*A Recipe for Algebraic Program Analysis.* We conclude this section by presenting a general view of algebraic program analysis, abstracted from the language of graphs and regular expressions:


Section 4 and Sect. 5 give two more instances of this generic recipe, generalizing beyond left-linear equations and regular-expressions as closed forms. Section 4 considers linear equations (and an appropriate language of closed forms); Sect. 5 considers another form of equation with ω-regular expressions as closed forms.

#### **4 Interprocedural Analysis**

Algebraic program analyses are oriented around computing summaries for program fragments, and are naturally suited to analyzing programs with procedures. Following Cousot & Cousot [23] and Sharir & Pnueli [56], the idea is to structure the analysis in two phases:


An example of a program with procedures is given in Fig. 3(a). The CFGs for its procedures are shown in Fig. 3(b) along with a set of equations corresponding to the CFGs (Fig. 3(c)). For Phase I, it is also useful to consider the following equations in which we have eliminated all variables except for those of the form Xs,x, which represent the procedure summaries.

$$\begin{array}{l} X\_{s\_1, x\_1} = \left( \langle s\_1, a \rangle \cdot X\_{s\_2, x\_2} + \langle s\_1, b \rangle \right) \cdot X\_{s\_2, x\_2} \\ X\_{s\_2, x\_2} = X\_{s\_3, x\_3} \cdot X\_{s\_3, x\_3} \\ X\_{s\_3, x\_3} = \langle s\_3, x\_3 \rangle \end{array} \tag{1}$$

This system of equations can be obtained either by a process of successively eliminating variables from Fig. 3(c), or they can be read off directly from each control-flow graph: sequential composition corresponds to ·, and branching corresponds to +.

We can also construct a graph of the dependencies among the variables in the equation system. In this case, we would have

$$X\_{s\_3, x\_3} \longrightarrow X\_{s\_2, x\_2} \longrightarrow X\_{s\_1, x\_1} \tag{2}$$

(which is also isomorphic to the program's call graph). Note that the equations in Eq. (1) are *not* left-linear. However, by eliminating variables in a topological order of Eq. (2), these systems can still be solved using Gaussian elimination (Algorithm 1).

$$\begin{array}{l} X\_{s3,x\_3} = \langle s\_3, x\_3 \rangle \\ X\_{s2,x\_2} = \langle s\_3, x\_3 \rangle \cdot \langle s\_3, x\_3 \rangle \\ X\_{s1,x\_1} = \left( \langle s\_1, a \rangle \cdot \langle s\_3, x\_3 \rangle \cdot \langle s\_3, x\_3 \rangle + \langle s\_1, b \rangle \right) \cdot \langle s\_3, x\_3 \rangle \cdot \langle s\_3, x\_3 \rangle \end{array} \tag{3}$$

Unfortunately, this strategy breaks down for programs with recursive procedures: the essential difficulty is in computing the summaries of procedures that are directly recursive or part of a set of mutually recursive procedures. We will return to this issue shortly, after a brief discussion of Phase II, which can be addressed via algebraic program analysis, regardless of whether the original equation system contains recursion.

**Fig. 3.** (a) A three-procedure program scheme. (b) Control-flow graphs for program (a). The edges labeled "X2" and "X3" represent calls to the respective procedures. (c) A system of equations corresponding to (b).

**Fig. 4.** Graph corresponding to the equation system used for Phase II for the program from Fig. 3.

With closed-form solutions for the procedure summaries in hand, Phase II can be addressed with Gaussian elimination. (Note that for a program with recursive procedures, the transformed Phase II system is still recursive. However, it is *left*-recursive, and so can be handled with regular expressions, and analyzed using the transition-formula interpretations of Sect. 2—the "loops" in Phase II correspond to sequences of recursive calls). Figure 4 shows the equation system

**Fig. 5.** (a) A two-procedure program scheme, where X<sup>1</sup> represents the main procedure, <sup>X</sup><sup>2</sup> represents a recursive subroutine, and <sup>C</sup>s1,a, <sup>C</sup>s2,x2, <sup>C</sup>s2,b, and <sup>C</sup>b,x2 represent four program statements. (b) Control-flow graphs for program (a). The three edges labeled "X2" represent calls to procedure X2. (c) A system of equations corresponding to (b).

used for Phase II for the program from Fig. 3 in graphical form. The graph is similar to Fig. 3(b) with (i) additional edges from each call-site to the start node of the called procedure, and (ii) the edges previously labeled with "X2" and "X3" are now labeled with the values from Eq. (3) for the corresponding procedure summaries: s3, x3·s3, x3 and s3, x3, respectively.

The remainder of this section focuses on Phase I: computing procedure summaries. Consider the two-procedure program shown in Fig. 5(a). CFGs for its procedures are shown in Fig. 5(b) along with a set of recursive equations corresponding to the interprocedural CFG. Unfortunately, equations like those in Fig. 5(c) do not fit naturally with the recipe given in Sect. 3.3. The essential difficulty is with item 3.3: "Design a suitable language of 'closed-form solutions' and an algorithm for computing them." In particular, we cannot use regular expressions and path-expression algorithms because the equations in Fig. 5(c) are not left-linear (and they cannot be put in left-linear form).

Two ideas are involved in using algebraic program analysis to summarize recursive procedures:


#### **4.1 Motivation: Newtonian Program Analysis**

To motivate why we are interested in the special case of linear equations (Sect. 4.2), this section provides a brief overview of how linear equations arise in NPA. Let <sup>E</sup> <sup>=</sup> {X<sup>i</sup> <sup>=</sup> <sup>R</sup>i} n <sup>i</sup>=1 be a system of equations, and fix an interpretation *<sup>I</sup>* over some algebra **<sup>A</sup>**. Define a function **<sup>f</sup>** : <sup>A</sup><sup>n</sup> <sup>→</sup> <sup>A</sup><sup>n</sup> by **<sup>f</sup>**(σ) = (*I*σ-R1,..., *I*σ-Rn) (i.e., the n-tuple of interpreted right-hand-sides, where variables are interpreted according to σ). NPA is an iterative method for program analysis that solves the following sequence of problems for ν:

$$\begin{array}{cc}\hline\nu^{(0)} = \mathbf{f}(\mathbf{0}) & \nu^{(i+1)} = \mathbf{Y}^{(i)}\\\hline\end{array}$$

where **Y**(i) is the value of **Y** in the least solution of

$$\boxed{\mathbf{Y} = \mathbf{f}(\nu^{(i)}) + \text{LinearCorrectionTerm}(E, \nu^{(i)}, \mathbf{Y})} \tag{5}$$

Thus, NPA is similar to Kleene iteration, except that on each iteration, **f**(ν(i)) is "corrected" by an amount controlled by LinearCorrectionTerm(E,ν(i), **Y**)—a function of **f**, the current approximation ν(i) , and (vector) variable **Y**—which nudges the next approximation ν(i+1) in the right direction at each step.

The linear correction term is the result of replacing each right-hand side R<sup>i</sup> = <sup>j</sup> <sup>R</sup><sup>j</sup> with a sum <sup>n</sup> <sup>i</sup>=0 <sup>R</sup>i,j,k, where each <sup>R</sup>i,j,k is obtained from <sup>R</sup>i,j by replacing all variables, except possibly one, with its interpretation in ν. (The formal definition can be found elsewhere [26, §3.2].) For example, consider the system of equations below, a simplified variant of Fig. 5(c) that is obtained by eliminating all variables except X<sup>s</sup>1,x<sup>1</sup> , X<sup>s</sup>2,b, X<sup>s</sup>2,x<sup>2</sup> :

$$\begin{array}{lcl}X\_{s\_1,x\_1} &= \langle s\_1, a \rangle \, X\_{s\_1,x\_2} \\ &X\_{s\_2,b} = \langle s\_2, b \rangle + X\_{s\_2,b} \cdot X\_{s\_2,x\_2} \cdot X\_{s\_2,x\_2} \cdot \langle d, b \rangle \\ &X\_{s\_2,x\_2} = \langle s\_2, x\_2 \rangle + X\_{s\_2,b} \, \langle b, x\_2 \rangle \end{array} \tag{6}$$

The transformation results in the following system (for brevity, we denote Y<sup>s</sup>1,x<sup>1</sup> , Y<sup>s</sup>2,b, Y<sup>s</sup>2,x<sup>2</sup> by Y1, Y2, Y3):

$$\begin{array}{l} Y\_1 = \langle s\_1, a \rangle \cdot Y\_3 \\ Y\_2 = \langle s\_2, b \rangle + Y\_2 \cdot \nu\_3 \cdot \nu\_3 \cdot \langle d, b \rangle + \underbrace{\nu\_2 \cdot Y\_3 \cdot \nu\_3 \cdot \langle d, b \rangle}\_{} + \underbrace{\nu\_2 \cdot \nu\_3 \cdot Y\_3 \cdot \langle d, b \rangle}\_{} \\ Y\_3 = \langle s\_2, x\_2 \rangle + Y\_2 \cdot \langle b, x\_2 \rangle \end{array} \tag{7}$$

Note that the two underlined summands are both truly *linear* : they are linear, but not left-linear nor right-linear.

The process of solving Eqs. (4) and (5) for ν(i+1), given ν(i), is called one *Newton round*. On the initial Newton round, we set ν(0) <sup>1</sup> , ν(0) <sup>2</sup> , ν(0) <sup>3</sup> ← 0, *<sup>I</sup>* s2, x2, *<sup>I</sup>* s3, x3. On round <sup>i</sup> + 1, we solve Eq. (7) for Y1, Y2, Y3 with ν1, ν2, ν3 set to the value ν(i) <sup>1</sup> , ν(i) <sup>2</sup> , ν(i) <sup>3</sup> obtained on round <sup>i</sup>, and then set ν(i+1) <sup>1</sup> , ν(i+1) <sup>2</sup> , ν(i+1) <sup>3</sup> ←Y1, Y2, Y3.

Operationally, the linearization transformation imposes a particular protocol for sampling the program's space of behaviors. For instance, in Fig. 5(b), the procedure X<sup>2</sup> has two call-sites along the loop through b. In Eq. (7), each right-hand-side summand in the equation for Y<sup>2</sup> has at most one variable: the transformation inserted ν<sup>2</sup> or ν<sup>3</sup> at various call-sites (considering Xs2,b as a pseudo-call-site corresponding to tail recursion), and left at most one variable Y<sup>i</sup> in each summand. In essence, during a given Newton round, the analyzer samples the behavior of **f** by taking the + of various paths through the transformation of **f**. Along each path through a (transformed) right-hand side, the summary for each pseudo-call-site X<sup>i</sup> encountered is held fixed at νi, except for possibly one pseudo-call-site on the path, which is explored by visiting (the linearized version of) the called procedure. The summaries ν1, ν2, ν<sup>3</sup> are updated according to the result of this exploration, and the algorithm performs the next Newton round.

The analogy between NPA and Newton's method in numerical analysis is that in both cases one creates a linear approximation of **f**(**X**) around the "point" (ν(i),**f**(ν(i))); the solution of the linear system is the next approximation of **X**.

#### **4.2 Algebraic Program Analysis for Linear Equations**

In this section, we instantiate the recipe for algebraic program analysis from Sect. 3.3 to solve a system of linear equations, such as the linearized problems that arise as Eq. (5) [53]. This goal may seem out of reach because item 3.3 of the recipe requires us to "design a suitable language of 'closed-form solutions' and an algorithm for computing them."

What is a suitable language of closed-form solutions of linear equations? Clearly the regular expressions and path-expression algorithms used in Sect. 2 and Sect. 3 will not do, because the least solution under the language interpretation to the (truly) linear equation X = aXb + 1 is ai <sup>b</sup><sup>i</sup> : <sup>i</sup> <sup>≥</sup> <sup>0</sup> , which is the canonical example of a linear-context-free language that is not regular. However, over fifty years ago, formal-language theorists established that linear-contextfree languages have certain similarities to regular languages [17,34,61], and we can make use of this property to design a language of closed forms for linear equations. Intuitively, ai <sup>b</sup><sup>i</sup> : <sup>i</sup> <sup>≥</sup> <sup>0</sup> can be obtained by (i) introducing *paired alphabet symbols*, such as (a, b), (ii) defining *concatenation of paired symbols* as (a, b)·(c, d) def = (ca, bd), (iii) defining Kleene-star in the natural way over pairedsymbol concatenation, so (a, b)<sup>∗</sup> is the language of paired words (ai , b<sup>i</sup> ) : <sup>i</sup> <sup>≥</sup> <sup>0</sup> , and (iv) applying an operation that concatenates the left word and right word of each paired word: (ai , b<sup>i</sup> ) : <sup>i</sup> <sup>≥</sup> <sup>0</sup> → ai <sup>b</sup><sup>i</sup> : <sup>i</sup> <sup>≥</sup> <sup>0</sup> .

For the purpose of algebraic program analysis, this idea can be formalized by introducing **tensored regular expressions** over an alphabet Σ, whose syntax is defined as follows:<sup>4</sup>

$$\begin{aligned} a &\in \Sigma\\ R \in \mathsf{RegExp}(\Sigma) &\coloneqq = a \mid 0 \mid 1 \mid R\_1 + R\_2 \mid R\_1 \cdot R\_2 \mid R^\* \mid S^\sharp\\ S \in \mathsf{RegExp}\_T(\Sigma) &\coloneqq = R\_1 \otimes R\_2 \mid \underline{\Omega} \mid \underline{\bot} \mid S\_1 \oplus S\_2 \mid S\_1 \odot S\_2 \mid S^\oplus \end{aligned}$$

We can now follow the pattern of Sect. 2, and define algebras suitable for interpreting tensored regular expressions.

**Definition 3.** *<sup>A</sup> tensor-product algebra* <sup>T</sup> <sup>=</sup> **A**, **<sup>T</sup>**, <sup>⊗</sup>, *consists of two regular algebras* **<sup>A</sup>** *and* **<sup>T</sup>** *along with an operation* <sup>⊗</sup> : <sup>A</sup> <sup>×</sup> <sup>A</sup> <sup>→</sup> <sup>T</sup>*, called* tensor product*, and an operation* : <sup>T</sup> <sup>→</sup> <sup>A</sup>*, called* detensor*.*

*Example 9 (Standard interpretation).* The standard interpretation from Example 1 can be extended to tensored regular expressions by defining a universe of languages over word pairs ("tensored words") T = 2<sup>Σ</sup>∗×Σ<sup>∗</sup> , whose operators are given by:

$$\begin{aligned} X \otimes Y &\stackrel{\Delta}{=} \{ \langle x, y \rangle : x \in X, y \in Y \} \\ Z^{\dagger} &\stackrel{\Delta}{=} \{ \underline{z}\overline{z} : \langle \underline{z}, \overline{z} \rangle \in Z \} \\ Z\_1 \odot Z\_2 &\stackrel{\Delta}{=} \{ \langle \underline{z}\_2 \underline{z}\_1, \overline{z}\_1 \overline{z}\_2 \rangle : \langle \underline{z}\_1, \overline{z}\_1 \rangle \in Z\_1, \langle \underline{z}\_2, \overline{z}\_2 \rangle \in Z\_2 \} \\ Z\_1 \oplus Z\_2 &\stackrel{\Delta}{=} Z\_1 \cup Z\_2 \\ Z^{\circledast} &\stackrel{\Delta}{=} \bigcup\_{i \in \mathbb{N}} Z^i \end{aligned}$$

Note that this interpretation allows tensored regular expressions to be used to capture linear context-free languages. For instance, the equation X = aXb + 1, whose least solution is ai <sup>b</sup><sup>i</sup> : <sup>i</sup> <sup>≥</sup> <sup>0</sup> can be written in closed form as X = ((<sup>a</sup> <sup>⊗</sup> <sup>b</sup>))-, and the equation X = aXa + bXb + 1, whose least solution is the language of even-length palindromes over {a, b}, can be written as <sup>X</sup> <sup>=</sup> (((<sup>a</sup> <sup>⊗</sup> <sup>a</sup>) <sup>⊕</sup> (<sup>b</sup> <sup>⊗</sup> <sup>b</sup>)))-. ⌟

*Example 10 (Relational interpretation).* The relational interpretation can be extended to tensored regular expressions by defining an algebra of binary statepair relations, as follows.<sup>5</sup> The universe is the set of relations on State <sup>×</sup> State (i.e., an element of the universe is a subset of State <sup>×</sup> State <sup>×</sup> State <sup>×</sup> State). Comparing with the standard interpretation, (in which an element p1, p2 consists of a "backwards path" p and a "forwards continuation") we may think of

<sup>4</sup> A warning about notation: in our previous papers, we used <sup>⊕</sup> and <sup>⊗</sup> for the two semiring operations, for tensor product, and ⊕<sup>T</sup> and ⊗<sup>T</sup> for the two tensoredsemiring operations. In this paper, we use + and · for the semiring operations, with circles around them for the tensored-semiring versions: ⊕ and . We use ⊗ for tensor product, which is consistent with usual mathematical notation.

<sup>5</sup> That is, an element of the algebra is a pair of pairs of states.

an element <sup>s</sup> 1 s2 , s1 s 2 of a state-pair relation as consisting of two pre/post state pairs: a "backwards" pair s <sup>1</sup> <sup>∗</sup> <sup>←</sup> <sup>s</sup><sup>1</sup> and a "forwards" pair <sup>s</sup><sup>2</sup> <sup>→</sup><sup>∗</sup> <sup>s</sup> <sup>2</sup>. In the algebra of state-pair relations, 0 is interpreted as the empty relation, 1 as the identity relation, and + as union. The remaining operators are given by:

$$\begin{aligned} R\_1 \otimes R\_2 &= \left\{ \left< \begin{pmatrix} s'\_1\\ s\_2 \end{pmatrix}, \begin{pmatrix} s\_1\\ s'\_2 \end{pmatrix} \right> : \left< s\_1, s'\_1 \right> \in R\_1, \left< s\_2, s'\_2 \right> \in R\_2 \right\} \\ T^\ddagger &= \left\{ \left< s, s' \right> : \exists s'', s'' . \left< \begin{pmatrix} s''\\ s''' \end{pmatrix}, \begin{pmatrix} s \\ s'' \end{pmatrix} \right> \in T \land s'' = s''' \right\} \\ T\_1 \odot T\_2 &= \left\{ \left< \begin{pmatrix} s\_1\\ s\_2 \end{pmatrix}, \begin{pmatrix} s'\_1\\ s'\_2 \end{pmatrix} \right> : \left< \begin{pmatrix} s\_1\\ s\_2 \end{pmatrix}, \begin{pmatrix} s''\_1\\ s''\_2 \end{pmatrix} \right> \in T\_1 \land \left< \begin{pmatrix} s''\_1\\ s''\_2 \end{pmatrix}, \begin{pmatrix} s'\_1\\ s'\_2 \end{pmatrix} \right> \in T\_2 \right\} \end{aligned} \tag{8}$$

Note that the tensored sequencing operation is just a form of relational composition (over tuples of stacked elements); similarly, tensored iteration is a form of reflexive transitive closure. ⌟

*Example 11 (Transition-formula interpretation).* Transition formulas can be used to interpret tensored regular expression in a way analogous to the relational interpretation (as one should expect, because there must be a soundness relation between them!). A tensored transition formula T is a formula over four vocabularies, representing the value of the variables before and after a pair of computations. The tensor and detensor operations are essentially the same as those from the relational interpretation, translated into logic:

$$\mathbb{P}\left(F\_1 \otimes F\_2\right)\left(\begin{pmatrix}X\_1'\\X\_2\end{pmatrix}, \begin{pmatrix}X\_1\\X\_2'\end{pmatrix}\right) \stackrel{\Delta}{=} F\_1(X\_1, X\_1') \wedge F\_2(X\_2, X\_2')\tag{9}$$

$$T^\sharp(X, X') \stackrel{\Delta}{=} \mathbb{P}\left(\begin{matrix}Y\_1\\Y\_2\end{matrix}\right) . T\left(\begin{pmatrix}Y\_1\\Y\_2\end{pmatrix}, \begin{pmatrix}X\\X'\end{pmatrix}\right) \wedge Y\_1 = Y\_2$$

In the Eq. (9), the vocabularies X1, X <sup>1</sup>, X2, and X <sup>2</sup> track the original role of the respective vocabulary in F<sup>1</sup> or F2. The "stacked" notation is intended to be suggestive of an interpretation of a tensored transition formula over a doubled vocabulary, where the variables are X <sup>1</sup> <sup>∪</sup> <sup>X</sup><sup>2</sup> and their "primed copies" are <sup>X</sup><sup>1</sup> <sup>∪</sup> <sup>X</sup> <sup>2</sup>. To make the connection with Sect. 2.1 more apparent, we shall define W<sup>1</sup> = X <sup>1</sup>, W<sup>2</sup> = X2, W <sup>1</sup> = X1, W <sup>2</sup> = X <sup>2</sup>. With this notation, the product operation can be defined as:

$$\left( \left( T\_1 \odot T\_2 \right) \left( \begin{pmatrix} W\_1 \\ W\_2 \end{pmatrix}, \begin{pmatrix} W\_1 \\ W\_2 \end{pmatrix} \right) \right) \triangleq \exists \left( \begin{pmatrix} W\_1 \\ W\_2 \end{pmatrix} \right)'' \left. T\_1 \left( \begin{pmatrix} W\_1 \\ W\_2 \end{pmatrix}, \begin{pmatrix} W\_1 \\ W\_2 \end{pmatrix} \right) \cap T\_2 \left( \begin{pmatrix} W\_1 \\ W\_2 \end{pmatrix} \right)'' \left( \begin{pmatrix} W\_1 \\ W\_2 \end{pmatrix} \right) \right)$$

As with the relational interpretation, the product operation is just a form of relational composition (over tuples of stacked elements).

Remarkably, the algebra of tensored transition formulas is the same as the algebra of untensored transition formulas, just over an extended set of variables. In particular, the iteration operators from Sect. 3 can be used to implement . For instance, consider the recursive procedure

$$foo(): \text{ if } \text{ (\*) } \text{ then } \text{ } a: \text{ = } a \text{ + 1; } foo\text{) ; } b: \text{ = } b \text{ + 1.}$$

The path to the recursive call of *foo* and the path from the recursive call to exit can be modeled by the transition formulas F and G, respectively:

$$F \stackrel{\triangle}{=} a' = a + 1 \land b' = b$$

$$G \stackrel{\triangle}{=} b' = b + 1 \land a' = a$$

A procedure summary for *foo* can be calculated by evaluating ((<sup>F</sup> <sup>⊗</sup>G))-, using recurrence analysis (Example 5) to implement the operator:

$$F \otimes G \stackrel{\Delta}{=} a\_1 = a\_1' - 1 \wedge b\_1 = b\_1' \wedge b\_2' = b\_2 + 1 \wedge a\_2' = a\_2$$

$$(F \otimes G)^{\oplus} \stackrel{\Delta}{=} \exists k.k \ge 0 \land a\_1 = a\_1' - k \wedge b\_1 = b\_1' \wedge b\_2' = b\_2 + k \wedge a\_2' = a\_2$$

$$((F \otimes G)^{\oplus})^{\underline{t}} \stackrel{\Delta}{=} \exists k.k \ge 0 \land a' = a + k \wedge b' = b + k$$

We now show how to compute closed forms for linear equations. First, we perform a *regularizing transformation*, which takes a system of linear equations ELin and converts it into a system of left-linear equations ELeftLin. The transformation takes each right-hand-side term of the form <sup>a</sup> · <sup>Y</sup> · <sup>b</sup> and converts it to <sup>Z</sup> (<sup>a</sup> <sup>⊗</sup> <sup>b</sup>), where <sup>Y</sup> and <sup>Z</sup> are variables whose values are elements of the regular algebras **<sup>A</sup>** and **<sup>T</sup>** of a tensor-product algebra **A**, **<sup>T</sup>**, <sup>⊗</sup>, .

**Definition 4.** *Given a linear equation system* E*Lin over the regular algebra* **A** *of a tensor-product algebra* <sup>T</sup> <sup>=</sup> **A**, **<sup>T</sup>**, <sup>⊗</sup>, *, the regularizing transformation* τ*Reg creates a left-linear equation system* E*LeftLin* = τ*Reg*(E*Lin*) *over* **T** *by transforming each equation of* E*Lin as follows:*

$$Y\_j = c\_j + \sum\_{i,k} \left( a\_{i,j,k} \cdot Y\_i \cdot b\_{i,j,k} \right)$$

$$\overline{Z\_j = \left( 1 \otimes c\_j \right) \oplus \bigoplus\_{i,k} \left( Z\_i \odot \left( a\_{i,j,k} \otimes b\_{i,j,k} \right) \right)} \quad \tau\_{\text{Reco}}$$

*where* Z<sup>i</sup> *and* Z<sup>j</sup> *are variables that take on values from* **T***.*

For instance, if the regularizing transformation is applied to the linear system of equations in Fig. 6a, the result is the system of equations Fig. 6b. Because Fig. 6b is left-linear, we can now use the approach from Sect. 2 and Sect. 3—that is, create a closed-form solution for each variable Z<sup>i</sup> by finding a path expression for the variable in the graph Fig. 6c. Finally, one gives a closed-form solution for each variable <sup>Y</sup><sup>i</sup> for the linear equation system in Fig. 6a by applying (−)- to each path expression—see Fig. 6d. This algorithm for computing closed-form solutions to linear equations is justified in the tensored-relational interpretation, and more generally, in any interpretation whose algebra forms what we dub a Kronecker algebra, defined as follows:

$$\begin{aligned} Y\_1 &= aY\_2b + cY\_2d \\ Y\_2 &= e + fY\_1g & Z\_2 &= \langle 1 \otimes e \rangle \oplus \langle Z\_1 \odot (f \otimes g) \rangle \\ &\text{(a)} \\ &\text{(b)} \\ (a \otimes b) &\oplus (c \otimes d) &\left(\sum\limits\_{(1 \otimes e)} \begin{pmatrix} 1 \otimes e \\\\ (f \otimes g) \end{pmatrix} \quad Y\_1 = \left(\begin{pmatrix} 1 \otimes e\\((a \otimes b) \oplus (c \otimes d)) \odot (f \otimes g) \end{pmatrix}^{\otimes}\right)^t \\ &\text{(b)} \\ &\text{(c)} \end{aligned}$$
 
$$\begin{aligned} \text{(d)} \end{aligned}$$
 
$$\text{(e)} \end{aligned} \qquad \begin{aligned} \text{(a \otimes b)} \begin{aligned} \text{(b \otimes e)} \begin{aligned} Y\_1 &= \left(\begin{pmatrix} 1 \otimes e\\((a \otimes b) \oplus (c \otimes d)) \odot (f \otimes g) \end{pmatrix}^{\otimes} \begin{aligned} \text{(c \otimes d)} \begin{pmatrix} 1 \otimes e\\(c \otimes d) \odot (c \otimes d) \end{pmatrix}^{\otimes} \begin{aligned} \text{(d)} \end{aligned} \end{aligned}$$
 
$$\begin{aligned} \text{(e)} \end{aligned}$$

**Fig. 6.** (a) A linear system of equations; (b) its regularization; (c) the graph corresponding to (b); (d) a closed-form solution for (a).

**Definition 5.** *<sup>A</sup> Kronecker algebra* **Kr** <sup>=</sup> A, <sup>+</sup>, ·, <sup>∗</sup>, <sup>0</sup>, <sup>1</sup>*,* T, <sup>⊕</sup>, , , <sup>0</sup>, <sup>1</sup>*,* <sup>⊗</sup>, *is a tensor-product algebra that consists of two Kleene algebras* A, <sup>+</sup>, ·, <sup>∗</sup>, <sup>0</sup>, <sup>1</sup> *and* T, <sup>⊕</sup>, , , <sup>0</sup>, <sup>1</sup> *such that (i) the natural order forms a complete lattice (i.e., both algebras have all infinite sums), and (ii) the following properties hold:*

*1.* 0 ⊗ 0=0 *2.* 1 ⊗ 1=1 *3.* (<sup>a</sup> <sup>⊗</sup> <sup>b</sup>)- <sup>=</sup> <sup>a</sup> · <sup>b</sup>*, for all* a, b <sup>∈</sup> <sup>A</sup> *4.* (a<sup>1</sup> <sup>⊗</sup> <sup>b</sup>1) (a<sup>2</sup> <sup>⊗</sup> <sup>b</sup>2)=(a<sup>2</sup> · <sup>a</sup>1) <sup>⊗</sup> (b<sup>1</sup> · <sup>b</sup>2)*, for all* <sup>a</sup>1, a2, b1, b<sup>2</sup> <sup>∈</sup> <sup>A</sup> *5.* (t<sup>1</sup> <sup>⊕</sup> <sup>t</sup>2)- = t - <sup>1</sup> + t - <sup>2</sup> *, for all* <sup>t</sup>1, t<sup>2</sup> <sup>∈</sup> <sup>T</sup>

*We assume that all distributivity properties of* A *and* T*, as well as item 5, hold for infinite sums. In particular, for item 5, we have*

$$\left(\bigoplus\_{i\in I} t\_i\right)^\sharp = \sum\_{i\in I} t\_i^\sharp \tag{10}$$

#### **4.3 Discussion**

*The Instantiation of the Recipe.* Returning to the recipe from Sect. 3.3, what we have done for a system of linear equations ELin is to instantiate the recipe as follows:


3. *(Interpretation)*. Tensored regular expressions can be interpreted as tensored transition formulas (Example 11), which are simply transition formulas over a "doubled" vocabulary.

*Two Lessons.* We would like to mention two lessons that we learned while working on this material over the years.

	- Algebraic program analysis `a la Sect. 2 solves a left-linear (or right-linear) system of equations using methods based on regular expressions.
	- NPA repeatedly creates a system of linear equations that needs to be solved. Such linear equations are related to linear context-free languages, such as the language {a<sup>i</sup> bi }, which is not regular.
	- Ergo, it is a non-starter to attempt to apply algebraic program analysis to the equations that arise on each round of NPA.

However, as shown in this section, it was possible to side-step this fundamental mismatch, by extending algebraic program analysis to systems of linear equations using Kronecker algebras, which have additional operations, such as tensor product and detensor.

Thus, beyond the technical details, perhaps a more important takeaway is "be careful how you apply sanity checks." There is a risk that a plausible-sounding sanity check could cause you to discard an idea that is worth pursuing.

2. In some sense, the solution using Kronecker algebras goes against the grain of what computer scientists typically preach, namely, create appropriate abstractions (in the sense of abstract data-types) for a problem at hand, and then program your solution, thinking of the chosen abstractions as the operations of an abstract machine. This style of thinking is considered central to managing complexity in computer science, and it is generally considered heresy to break an abstraction.

For algebraic program analysis, the abstraction is regular algebra, used with interpretations that are abstractions (in the sense of abstract interpretation [22]) of a program's concrete transition relations. However, the introduction of tensor product and detensor *breaks* that abstraction! To understand what we mean, consider the definition of <sup>F</sup> · <sup>G</sup> for transition relations in Boolean programs, i.e.,

> (<sup>F</sup> · <sup>G</sup>)(W, Z) -<sup>∃</sup>X, Y.F(W, X) <sup>∧</sup> <sup>G</sup>(Y,Z) <sup>∧</sup> (<sup>X</sup> <sup>=</sup> <sup>Y</sup> ),

and the definitions of <sup>F</sup> <sup>⊗</sup> <sup>G</sup> and <sup>T</sup> -, <sup>6</sup> namely,

$$\begin{array}{c} (F \otimes G)(W, X, Y, Z) \stackrel{\triangle}{=} F(W, X) \wedge G(Y, Z) \\ T(W, X, Y, Z)^{\circ} \stackrel{\triangle}{=} \exists X, Y. T(W, X, Y, Z) \wedge (X = Y) \end{array}$$

<sup>6</sup> Because we are trying to relate these operations to the untensored product operation

<sup>·</sup>, we do not make use of the stacked notation from Sect. 4.2.

The product operation <sup>F</sup> ·<sup>G</sup> has three distinct steps: (i) conjoin <sup>F</sup>(W, X) and G(Y,Z); (ii) conjoin the equality X = Y ; and (iii) project out vocabularies X and <sup>Y</sup> . In essence, tensor product and detensor break the abstraction of · as an indivisible operation: · is decomposed into two more-granular operations, <sup>⊗</sup> and . By performing <sup>F</sup> <sup>⊗</sup> <sup>G</sup>, we perform just the first step of ·, and only later, when is performed, do we "finish up" by applying the second and third steps of ·. The advantage is that we can operate on tensored values for some number of steps before "finishing" some earlier ·.

Again, beyond the technical details, the takeaway may be the *process* that we went through, which may be of value as a conceptual tool in other contexts:


#### **5 Termination Analysis**

This section describes how algebraic program analysis can be applied to termination analysis, based on the approach of [63]. The goal of termination analysis is to prove that a program has no infinite executions. Our high-level strategy is to exploit compositionality: we prove that a loop terminates by first computing a summary (e.g., a transition formula) for its body, and then finding a termination argument for the summary.

Following Sect. 3, we first formalize a concrete semantics as the (greatest) solution of a system of semantic equations. An appropriate notion of concrete semantics for termination analysis is the set of *non-terminating* states of the program (from which there exists an infinite execution)—the program terminates exactly when none of the program's initial states belong to this set. As in Sect. 3, this system of equations can be derived syntactically from a program's control flow graph—see Fig. 7 for an example. The non-terminating states of the program are the greatest solution to this system of equations over the algebra where the universe is the set of states, is interpreted as union (a state is non-terminating if it has at least one infinite execution) and is interpreted as preimage (a state is non-terminating iff it can reach a non-terminating state).<sup>7</sup>

<sup>7</sup> Despite the fact that this system of equations is right-linear, the method of Sect. 2 does not apply because the system of equations has two sorts instead of one; in particular, has type - : 2State×State×2State <sup>→</sup> <sup>2</sup>State, and so is not a binary operation on a set.

**Fig. 7.** A program represented by a control flow graph (a), abstract syntax tree (b), and system of equations (c).

A suitable language of "closed-form solutions" for the system of equations that arise in termination analysis is ω-regular expressions. The syntax of ω**regular expressions** over an alphabet Σ is as follows:

$$a \in \Sigma$$

$$R \in \mathsf{RegExp}(\Sigma) \; ::= a \mid 0 \mid 1 \mid R\_1 + R\_2 \mid R\_1 \cdot R\_2 \mid R^\* $$

$$S \in \omega \text{-} \mathsf{RegExp}(\Sigma) \; ::= R^{\omega} \mid S\_1 \boxplus S\_2 \mid R \boxminus S$$

The semantics of a (ω)-regular expressions is given by an interpretation over an ω-algebra and a regular algebra.


*Example 12 (Standard interpretation).* In the *standard interpretation* of ωregular expressions, the universe consists of sets of infinite sequences over the alphabet Σ, and the operations are

$$\begin{aligned} W\_1 \boxplus W\_2 &\stackrel{\Delta}{=} W\_1 \cup W\_2 & \text{Uniform} \\ X \boxminus W &\stackrel{\Delta}{=} \{xw : x \in X, w \in W\} & \text{Concatenation} \\ X^{\omega} \equiv \{x\_1 x\_2 \cdots : x\_1, x\_2, \cdots \in X\} & \text{Infinite repetition} \end{aligned}$$

For example, an ω-regular expression that recognizes all infinite paths in Fig. 7a starting at r is:

Outer loop ( r, a a, b b, c ( c, d d, e e, c) ∗ c, r) ω ( r, a a, b b, c ( c, d d, e e, c) ∗ c, r) ∗ r, a a, b b, c - ( c, d d, e e, c) ω Inner loop ⌟

*Example 13 (Nonterminating state interpretation).* The non-terminating state algebra is an ω-algebra over the algebra of state relations. Its universe consists of sets of states. The operators are

$$\begin{aligned} R \boxdot S & \stackrel{\scriptstyle \Delta}{=} \{ x : \exists y. \langle x, y \rangle \in R \land y \in S \} & \text{Primeage} \\ S\_1 \boxplus S\_2 & \stackrel{\scriptstyle \Delta}{=} S\_1 \cup S\_2 & \text{Union} \\ & \mathcal{B}^{\omega} \not\subseteq \{ \scriptstyle\_{\mathcal{B}^{\omega}} \subset \mathsf{Strate} \; : \; \exists x\_1, x\_2, \dots \; \} & \begin{array}{c} \\ \end{array} \end{aligned} \text{Function}$$

$$R^{\omega} \triangleq \left\{ x\_0 \in \mathbf{State} : \begin{array}{l} \exists x\_1, x\_2, \dots \\ \forall i. \,\langle x\_i, x\_{i+1} \rangle \in R \end{array} \right\} \quad \text{Non-terminating states of } R$$

Tarjan's path expression algorithm can be adapted to compute an ω-regular expression that recognizes the set of infinite paths in a graph beginning at a particular node [63]. The equational view of this algorithm is that it computes closed-form solutions to right-linear equations over *B¨uchi algebras* (e.g., the algebra of non-terminating states).

**Definition 7 (B¨uchi algebra).** *A B¨uchi algebra is an* ω*-algebra over a Kleene algebra satisfying the following:*


*where is the order defined by* <sup>a</sup> <sup>b</sup> *iff* <sup>a</sup> <sup>b</sup> <sup>=</sup> <sup>b</sup>*.*

*Exercise 2.* Show that in any B¨uchi algebra, the greatest solution to the equation X = (a X) z exists and is equal to X = a<sup>ω</sup> (a<sup>∗</sup> z).

Summarizing: we have modeled a program's non-terminating states as the greatest solution to a system of semantic equations, devised a language of "closed form solutions", and identified an algorithm for computing closed form solutions to the equations. It remains only to develop abstract interpretations of the language of closed forms which implements termination analysis.

#### **5.1 Non-terminating State-Formula Interpretations**

Just as transition formulas (over variables X and X ) can be used to represent state relations, state formulas (over the variables X) can be used to represent sets of (non-terminating) states. We can extend an algebra of transition formulas to an algebra of non-terminating state formulas by defining

$$F \boxminus P \stackrel{\Delta}{=} \exists X'. F(X, X') \land P(X') \tag{Premage}$$

$$P\_1 \boxplus P\_2 \stackrel{\Delta}{=} P\_1 \lor P\_2 \tag{\text{Union}}$$

Intuitively, the ω operator should compute the set of non-terminating states of a transition formula. Analogously to the ∗ operator in Sect. 2, this set is uncomputable, and we must be satisfied with an over-approximation (i.e., we aim to compute a state formula that contains all non-terminating states—the soundness relation of interest is the one defined by <sup>N</sup> <sup>S</sup> ⇐⇒ ∀<sup>s</sup> <sup>∈</sup> N.s <sup>|</sup><sup>=</sup> <sup>S</sup>). There are many ways of doing this, so we speak of the family of non-terminating state formula interpretations. In the remainder of this section, we give examples of ω-operators.

*Example 14 (Linear-lexicographic ranking functions* [32]*).* Let F(X, X ) be a transition formula. A *linear lexicographic ranking function* (LLRF) for F is a sequence of linear terms t1,...,t<sup>n</sup> over X such that for any states s and s such that <sup>s</sup> <sup>→</sup><sup>F</sup> <sup>s</sup> , each t<sup>i</sup> evaluates to a non-negative integer in s, and the integer n-tuple decreases in lexicographic order going from s to s . Since there are no infinite strictly descending chains of non-negative n-tuples of integers with respect to the lexicographic order, if F has an LLRF, then F has no nonterminating states. For example, the inner loop of Fig. 7 has a 1-dimensional LLRF k, and the outer loop has a 2-dimensional LLRF <sup>n</sup> <sup>−</sup> i, j.

The problem of determining whether a linear integer arithmetic formula has an LLRF is decidable [32]. If a formula does *not* have an LLRF, then we can use a coarse over-approximation of the non-terminating states of a formula (e.g., the set of states that have at least one outgoing transition). This yields the following interpretation of the ω operator:

$$F^{\omega} \stackrel{\triangle}{=} \begin{cases} false & \text{if there is an LLRF for } F \\ \exists X'. F(X, X') & \text{otherwise} \end{cases}$$

For Fig. 7, using recurrence analysis to implement the ∗ operator (Example 5), we get that every non-terminating state must satisfy *false*—the program terminates from any initial state. ⌟

*Example 15 (Unbounded trajectories* [63]*).* Let F(X, X ) be a transition formula. A necessary (but not sufficient) condition for a state s to be a non-terminating for a transition formula F is that there is a computation of F starting from s for every possible length. This condition is undecidable, but it can be approximated using an approximate transitive-closure operator such as the ones in Sect. 2.1. Suppose that (−)<sup>∗</sup> is an over-approximate transitive-closure operator. Letting <sup>k</sup>

and k be symbols that do not appear in F, we can create a transition formula exp(F) in one parameter k such that for any k , if there exists a sequence <sup>s</sup><sup>1</sup> <sup>→</sup><sup>F</sup> <sup>s</sup><sup>2</sup> <sup>→</sup><sup>F</sup> ···→<sup>F</sup> <sup>s</sup>k , then <sup>s</sup><sup>1</sup> <sup>→</sup>exp(<sup>F</sup> ) <sup>s</sup>k :

$$\exp(F) \triangleq (F \wedge k' = k + 1)^\* [k \mapsto 0]^\*$$

The states <sup>s</sup> for which there exists a computation <sup>s</sup> <sup>→</sup>exp(<sup>F</sup> ) <sup>s</sup> <sup>→</sup> <sup>s</sup> for all choices of the parameter k over-approximates the set of non-terminating states of F:

F <sup>ω</sup> - <sup>∀</sup>k <sup>≥</sup> <sup>0</sup>.∃X , X. exp(F) <sup>∧</sup> <sup>F</sup>(X , X)

For example, if ∗ is instantiated to recurrence analysis (Example 5), then on the transition formula

$$F \triangleq i \neq n \land i' = i + 2 \land n' = n$$

(corresponding to the program **while** (*i* = *n*) **do** *i* := *i* + 2), we have

$$F^{\omega} = i > n \lor (n - i) \bmod 2 = 1$$

Additional examples of termination analyses in the algebraic framework appear in [63] and [62].

#### **5.2 The Instantiation of the Recipe**

The recipe from Sect. 3.3 is instantiated for termination analysis as follows:


#### **6 Recap**

This section contains a few remarks about commonalities among the three kinds of problems and the techniques we have presented for applying algebraic program analysis to them. The paper has been structured around the three-part recipe for algebraic program analysis given in Sect. 3.3. Table 1 recaps how the recipe has been instantiated for the three kinds of problems considered.

Within this paper, all methods for computing closed-form solutions can be understood as some variation of Gaussian elimination, Algorithm 1 (in practice, they are variations of Tarjan's path-expression algorithm). The essential


**Table 1.** Instantiations of the recipe for algebraic program analysis from Sect. 3.3.

**Table 2.** "Loop-solving" steps.


difference between Sect. 2, Sect. 4, and Sect. 5 is the "loop-solving" step. Each requires the right-hand-side expression R to be in a particular form (left-linear, linear, right-linear), and each requires a different language of expressions in which to express closed forms (regular, tensored regular, ω-regular). Table 2 shows the respective "loop-solving" steps for computing a closed form. Note that in Table 2, the letters a, bi, ci, z range over expressions (which may involve variables other than X). For example, to apply the left-linear rule to the equation X = Xp + Xq + Y r + Z, we first re-arrange terms on the right-hand side as X(p + q)+(Y r + Z) and then compute the "closed-form" (Y r + Z)(p + q)∗.

#### **7 Related Work**

*Abstracting States Versus State Changes.* Classically, invariant generation is conceived as the problem of over-approximating the reachable states of a program. Computing invariants involves solving a system of equations of the form

$$\begin{cases} X[r] = v\_r & r \in Node, \text{ the root node} \\ X[n] = \sum\_{e\_{m,n} \in Edges} \mathcal{J}\{e\_{m,n}\}(X[m]) & n \in Node - \{r\} \end{cases} \tag{11}$$

for the unknowns <sup>X</sup>[n], <sup>n</sup> <sup>∈</sup> *Nodes*, where <sup>v</sup><sup>r</sup> represents the set of initial states and *I* -− provides an interpretation of each CFG edge as a state transformer. In a solution, X[n] holds a descriptor that represents a superset of the set of program states that can arise at program point n. Note that in Eq. (11), the function *I* em,n on edge em,n is *applied* to the value X[m] on node m.

Algebraic program analyses, in contrast, concern dynamics—state *changes* rather than states. The reason is that algebraic analyses are compositional: states do not compose, but state changes do.

A first step towards abstracting state *changes* was taken by Graham & Wegman [33], who gave a method to solve dataflow equations via *composition* of the state transformers on CFG edges. That is, their basic primitives were (i) composition of functions, and (ii) union of functions. If we adopt this outlook and define <sup>r</sup><sup>1</sup> · <sup>r</sup><sup>2</sup> to be <sup>r</sup><sup>2</sup> ◦ <sup>r</sup>1, <sup>r</sup><sup>1</sup> <sup>+</sup> <sup>r</sup><sup>2</sup> to be the union of <sup>r</sup><sup>1</sup> and <sup>r</sup>2, and 1 to be the identity function, instead of Eq. (11), the goal would be to solve the following equation system:

$$\begin{aligned} X[r] &= 1 & r \in Node, \text{ the root node} \\ X[n] &= \sum\_{e\_{m,n} \in Edges} X[m] \cdot \mathcal{J}\{e\_{m,n}\} & n \in Node - \{r\} \end{aligned} \tag{12}$$

where the unknowns X[n] are now function-valued. Note that the function *I* em,n on edge em,n is *composed* with the value X[m] on node m. From here because one is working over function-valued quantities—it is now natural to formulate interprocedural program-analysis problems by means of equations over unknowns that denote procedure summaries, as was done by Cousot and Cousot [23] and Sharir and Pnueli [56].

*"Interpret, Then Solve" Versus "Solve, Then Interpret."* The systems in Eqs. (11) and (12) are *interpreted*, in the sense that they are understood as semantic equations valued over a particular abstract domain, say D. Such a system <sup>E</sup> <sup>=</sup> {X<sup>i</sup> <sup>=</sup> <sup>R</sup>i}<sup>i</sup>∈<sup>I</sup> can be solved by an iterative method: compute a sequence <sup>σ</sup>0, σ1, ··· ∈ {Xi}<sup>i</sup>∈<sup>I</sup> <sup>→</sup> <sup>D</sup> of assignments abstract domain values to variables

$$\begin{array}{lcl} \sigma\_0(X\_i) \stackrel{\Delta}{=} 0 & \text{for all } i \in I\\ \sigma\_{n+1}(X\_i) \stackrel{\Delta}{=} \mathcal{J}\_{\sigma\_n}[R\_i] & \text{for all } n \ge 0 \text{ and all } i \in I \end{array}$$

Eventually this process converges—typically with the aid of widening to extrapolate to the limit—upon an assignment that over-approximates the least solution to E.

In algebraic program analysis, we think of a system of equations as an uninterpreted (syntactic) object. Equations are solved symbolically and then the solutions are interpreted in an algebraic structure to obtain an analysis result. The key step in this direction was made by Tarjan [59], who observed that once a solution to the path-expression problem was in hand, multiple dataflowanalysis problems could be solved merely by reinterpreting the alphabet symbols and operators of regular expressions in different algebras—i.e., "solve and then interpret."

Whereas the iterative framework for program analysis has a "built-in" algorithm for analyzing loops and recursive behavior (by computing the limit of a sequence), the algebraic framework does not prescribe any particular method, and it is up to the analysis designer to devise one. This obligation places an additional burden on the analysis designer, but also provides flexibility: the analysis designer may analyze loops in ways that may (Example 6) or may not (Examples 5 and 4) resemble iterative fixpoint computation.

*Iteration Operators and Loop Summarization.* In the computer-aided-verification community, there is a body of literature on loop summarization (or "loop leaping") and acceleration. Summarization aims to compute or approximate the behavior of (certain) loops, while acceleration aims to approximate the postimage of a set of states under a loop. These techniques have been incorporated into iterative abstract interpretation [28,31], abstraction-refinement-based software model checking [19,37], termination analysis [7,20,60], and resource bound analysis [10,64]. The most closely related techniques to algebraic program analysis are those that build summaries for whole programs in "bottom-up" fashion. Such analyses have been formalized in various ways, including: recursion on the abstract syntax tree (AST) of a program [51], AST rewriting [8], and graph rewriting [47,60]. Algebraic program analysis provides a unifying foundation for such analyses, in the same way that dataflow analysis [39] and (iterative) abstract interpretation [22] provide a unifying foundation for iterative program analyses.

There are several methods for loop summarization, based on finite-monoid affine transformations [11,12,29], difference-bound relations [15,21], octagonal relations [13,14,45], integer vector addition systems [35], fragments of the theory of arrays [2]. For the most part, these summarization methods are non-uniform in the sense that their input language differs from their output language (e.g., [13] takes as input an octagonal relation and produces as output a Presburger formula). This non-uniformity is the essential barrier that must be overcome to use such techniques to implement the iteration operator of an algebraic program analysis (e.g., we can define an iteration operator by using optimization modulo theories [55] to extract the octagonal hull of a Presburger formula, then use [13] to compute a Presburger formula representing its transitive closure).

*Elimination-Based Dataflow Analysis.* Elimination-based dataflow analysis is a family of dataflow analyses that computes analysis results using methods that resemble Gaussian elimination [3,33,36] (see [54] for a survey). Early methods were specialized to reducible control flow graphs, but operated faster than general Gaussian elimination. Tarjan's algorithm [58] is an elimination method with fast operation on reducible (and "nearly reducible") control flow graphs, but is applicable to arbitrary graphs.

*Weighted Graphs.* There is a vast literature on solving path problems on weighted graphs where the weights are drawn from a semiring [1,30,50]. Path problems can also be solved on semiring-weighted pushdown systems, which has applications to interprocedural dataflow analysis [52]. This work focuses on *iterative* techniques for solving path problems.

(Non-iterative) algorithms for path problems over algebraic structures with an explicit iteration operator were considered by Aho et al. [1], Backhouse & Carr´e [5], and Lehmann [48], and was implicit in previous work by Kleene [44], and McNaughton & Yamada [49]. Tarjan connected this line of work with program analysis [58,59].

#### **8 Open Problems**

We conclude with a list of challenges suggested by algebraic program analysis.

*Scaling SMT-Based Algebraic Program Analysis.* The bottom-up interpretation step of a closed-form expression is efficient, in that it operates in linear time and space in the size of the expression DAG in a model where each algebraic operation has unit cost. For logic-based interpretations, however, algebraic operations do *not* have unit cost: operators manipulate formulas, and the size of those formulas may grow as operators are applied. For example, the regular expression a2<sup>n</sup> can be represented by an expression DAG with n+1 nodes, with the following shape:

*... a*

If the letter <sup>a</sup> is interpreted as the transition formula <sup>x</sup> <sup>=</sup> <sup>x</sup> + 1 and · as relational composition, then the transition-formula interpretation of a2<sup>n</sup> has size O(2<sup>n</sup>). Scaling SMT-based algebraic program analysis to large programs requires techniques for generating *succinct* summaries, and/or efficient reasoning about compact formula representations involving λ-expressions.

*Recursive Procedures.* Section 4.2 shows how the algebraic approach can be applied to summarize *linearly* recursive procedures. But to compute summaries for generally recursive procedures, current-generation algebraic-programanalysis tools fall back on another non-algebraic scheme (such as hybrid iterative/algebraic, like Kleene or Newton iteration [40,53], or the template-based approach of [16]). This raises the question: is there a practical algebraic method for analyzing general recursion? The essential challenge is in devising a language of "closed forms" that (1) can represent arbitrary context-free languages, and (2) is amenable to an effective interpretation in logic.

*Beyond Numerical Domains.* To date, all algebraic program analyses have been numerical in nature—they abstract away aspects of program behavior that cannot be captured by integer variables. It remains to be seen whether the algebraic approach can yield practical analyses for reasoning about features like strings, arrays, and the heap. Reasoning about memory manipulation is particularly challenging in a compositional setting, since we cannot rely on the context of a program fragment to resolve aliasing relationships. One possible avenue is to incorporate abductive reasoning to make educated guesses about the shape of memory, as in [18].

*Property Refutation.* Algebraic program analysis is typically conceived as a method for generating over-approximate summaries. The nature of overapproximation is that the summaries can be used to verify that a program *does* satisfy a property of interest, but not prove that it *doesn't*. An interesting direction for future work is to devise methods by which algebraic program analyses can refute properties, perhaps based on bounded model checking [9], under-approximate loop summarization [46], or symbolic execution [43].

**Acknowledgments.** Supported, in part, by a gift from Rajiv and Ritu Batra; by a Facebook Research Award; by NSF under grant number 1942537, and by ONR under grants N00014-17-1-2889 and N00014-19-1-2318. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the authors, and do not necessarily reflect the views of the sponsoring entities.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### Programmable Program Synthesis

Loris D'Antoni, Qinheping Hu, Jinwoo Kim(B) , and Thomas Reps

> University of Wisconsin-Madison, Madison, USA {ldantoni,qhu28,jkim934,treps}@wisc.edu

Abstract. Program synthesis is now a reality, and we are approaching the point where domain-specific synthesizers can now handle problems of practical sizes. Moreover, some of these tools are finding adoption in industry. However, for synthesis to become a mainstream technique adopted at large by programmers as well as by end-users, we need to design programmable synthesis frameworks that (*i*) are not tailored to specific domains or languages, (*ii*) enable one to specify synthesis problems with a variety of qualitative and quantitative objectives in mind, and (*iii*) come equipped with theoretical as well as practical guarantees. We report on our work on designing such frameworks and on building synthesis engines that can handle program-synthesis problems describable in such frameworks, and describe open challenges and opportunities.

#### 1 Introduction

#### 1.1 A Synthesis Tale

Monica, a software engineer, is trying to write a program for transforming data she has stored in an array of integer numbers. Monica needs to zero-out all the negative entries from the array (they represent irrelevant data points) and add 10 to all the positive entries (this is a normalization step needed in Monica's API). Of course, Monica is a great engineer and she could write this program herself, but since Monica knows that similar problems arise often in her company (i.e., reformatting arrays to match certain APIs), Monica decides to try out this new thing everyone is talking about: *program synthesis*.

Monica wants a tool that takes as input some examples of the desired transformation and a set of operators the program can use, and magically outputs the intended program. In fact, Monica already has an input, a unit test, that she wants to process using her newly synthesized program: [−1, <sup>2</sup>, <sup>3</sup>, <sup>10</sup>, <sup>31</sup>, <sup>−</sup>14, <sup>−</sup>11], for which the output should be [0, <sup>12</sup>, <sup>13</sup>, <sup>20</sup>, <sup>41</sup>, <sup>0</sup>, 0].

Monica also knows that the final program will look like a loop that iterates over the input array arr, which leads her to develop the grammar in Fig. 1. Monica thinks this grammar is general enough that it will cover a reasonable range of programs for similar tasks but limited enough that it will not result in spurious programs that overfit too much to the examples.

Quickly, Monica discovers that using program synthesis is not so straightforward. There are so many different tools! And they all take different kinds of c The Author(s) 2021

$$\begin{array}{c} \text{Start} \rightarrow x = \mathsf{len}(\mathsf{arr}) - 1; \,\mathsf{while}\,\mathsf{1e}\,\mathsf{x} \succeq \mathsf{0} \text{ do } S\\\quad S \rightarrow \mathsf{arr}\,\mathsf{[E]} = \mathsf{arr}\,\mathsf{[E]} + E \mid \mathsf{arr}\,\mathsf{[E]} = E\\\quad x = E \mid S; \,\,S \mid \mathsf{if}\,\mathsf{array}\,\mathsf{[x]} \rhd \mathsf{0} \text{ then } S \,\mathsf{else}\,\,S\\\quad E \rightarrow \mathsf{0} \mid \mathsf{1} \mid \mathsf{x} \mid E + E \mid E - E \end{array}$$

Fig. 1. The grammar G*ex* Monica has in mind for synthesizing programs that iterate over an input array (Start is the starting nonterminal). G*ex* is general enough to cover most programs that iteratively normalize entries in an array.

inputs. After a bit more research, Monica decides to go for one of the many tools, UltraSynth™, and encodes her problem. UltraSynth is written in a C-like language and Monica has mostly programmed in Python for her job. However, Monica decides to give UltraSynth a try and after a few days of learning the ins and outs of UltraSynth, she finally manages to encode her transformationsynthesis problem in UltraSynth. To achieve her goal, Monica had to tweak a bit what the grammar looks like to provide it to UltraSynth, which only accepted grammars without unbounded recursion (i.e., without infinitely many terms) and had to encode the examples in a way that was accepted by the tool.

The time has come and Monica manages to run UltraSynth on an instance of the synthesis problem. UltraSynth outputs the program in Fig. 2b, which is correct on the example. However, this program is needlessly large and contains many unneeded operations.


Fig. 2. Two possible solutions for Monica's synthesis problem.

Monica has already invested a lot of time in learning UltraSynth, so she tries to figure out a way to avoid such problematic programs. Monica astutely realizes that the needless computations in Line 3 of Fig. 2b are due to repeated applications of the minus operator. Monica would like to ask UltraSynth to synthesize the program that contains as few minus operators as possible, but UltraSynth does not support a way to "prefer" one possible program over another. To bypass this limitation, Monica decides to remove the production E <sup>→</sup> E - E in order to suppress these programs.

Monica reruns UltraSynth after removing E <sup>→</sup> E - E from the grammar, and to her surprise, UltraSynth continues to run for hours and eventually times out without providing a solution. After investigating the matter, Monica finds out that she has made a mistake and disallowed too many programs—there is no longer a valid solution to the synthesis problem because without subtraction, the variable x cannot be decremented in line 5. UltraSynth was unable to report, or even detect this simple mistake—Why is it so difficult to program a synthesizer and why can't synthesis tools detect the simplest of mistakes?

Monica has finally had enough of synthesis. She goes back to her daily routine and just writes the 7-line piece of code that applies the transformation she intended (Fig. 2a).

#### 1.2 Programmable Synthesis Frameworks

The story of Monica is a common one in program synthesis, where most of the recent focus has been on solving problems rather than building general algorithms, tools, and methodologies. Existing synthesis frameworks are not programmable as they lack at least one of the following properties:


This state of affairs is unfortunate because synthesis is very general; if synthesis were easier to use, it would benefit many domains. The potential generality, which is currently held back by the need for better support for usability, underscores the need to answer the following question:

# Can we make synthesis more programmable?

In this paper, we present the steps we have undertaken in the direction of making synthesis more programmable, including some of the challenges that we faced, and some of the opportunities that the work has opened.

#### 2 An Overview of Programmable Program Synthesis

The goal of enlarging the scope of synthesis has focused our attention on the need to have a framework in which synthesis problems can be addressed. By a *framework*, we mean the conceptual underpinnings that allow one to build tools to automate the creation of solutions for problems in some domain, in this case, program-synthesis problems. The canonical example is how the theory of parsing underlies the yacc tool [13], which automates the construction of parsers. For instance, consider the problem that yacc addresses:

	- Input: a context-free grammar that describes language L.
	- Output: a parser, yyparse(), such that invoking yyparse() on s computes Parse(L,s).

One consideration for building a framework is the existence of a well-defined "engine" (or collection of engines) for performing the desired task—in this case, parsing s with respect to L, once both L and s are at hand. Yacc supports just a single engine, which parses a string with respect to a grammar that is LALR(1). In principle, yacc could have been a more general tool by having it perform various tests on L to determine what grammar family L belongs to (e.g., LALR(1), LR(1), LL(1), LL(\*)), and then emitting a parser that makes use of an appropriate parsing algorithm for that family, falling back on Generalized LR parsing [19] in case L is not in one of the specialized families supported.

Another aspect illustrated by yacc is that the parameters to the problem have different "binding times". In this case, string s changes more frequently than language L—i.e., L is bound early, and s is bound late. The framework implementation can exploit the known value of the early-bound parameter to create a more efficient implementation. In the case of yacc, it compiles L to tables used by a table-driven LALR(1) parsing algorithm.

#### 2.1 Why Isn't Existing Work in Synthesis Programmable?

There do exist synthesis tools (mostly, solver-aided languages [27,28]) that allow one to control some aspects of a synthesis problem in a programmable fashion. However, the nature of existing synthesis tools also forces an association between how a synthesis problem is written and how it is solved. For instance, in Sect. 1, the fact that solvers are tightly coupled to some specification language prevented Monica from trying out a different tool after UltraSynth produced an inadequate answer. The current state of program-synthesis tools is depicted in Fig. 3.

Fig. 3. Program synthesis today, where the lack of separation between specification and solver causes a user to have to encode a problem multiple times to use different tools.

This situation is in direct conflict with the principles articulated at the end of Sect. 1, namely, that a user should be able to program the various aspects of a synthesis problem using a formalism that is both (i) *domain-agnostic* and (ii) *solver-agnostic*. The first property addresses generality: the formalism should be powerful enough to capture a wide variety of synthesis problems (e.g., SQL, regular expressions, and imperative programs). The second property opens the door for synthesis-problem specifications to be fed—possibly after a compilation/translation step—to different specialized solvers, or to multiple solvers with different capabilities.

Another example that one may consider a programmable framework is Syntax-Guided Synthesis (SyGuS) [1], which is a successful synthesis framework targeted at expressions. The defining characteristic of SyGuS, compared to other synthesis approaches such as solver-aided languages, is that it allows one to write synthesis problems in a completely logical format.

*Example 1.* Consider the simple problem of synthesizing the maximum max of two input variables, x and y. There are two parts to a SyGuS problem: a syntactic part, written as a context free grammar such as the example <sup>G</sup><sup>S</sup> below:

$$G\_S \text{ ::= }Start \to \mathbf{x} \mid \mathbf{y} \mid Start \text{ \*} starts \mid \text{if } \mathbf{x} \text{<} \mathbf{y} \text{ then } Start \text{ \*} \mathbf{1} \text{se } Start$$

and the specification part, which is written as a Boolean formula <sup>ψ</sup><sup>S</sup>:

$$
\psi\_S \equiv \forall x, y. \max(x, y) \ge x \land \max(x, y) \ge y \land (\max(x, y) = x \lor \max(x, y) = y)
$$

<sup>A</sup> SyGuS problem is simply the pair sy = (G<sup>S</sup>, ψ<sup>S</sup>), where a solution to the SyGuS problem sy is a term <sup>t</sup> <sup>∈</sup> <sup>L</sup>(G) such that <sup>ψ</sup><sup>S</sup> holds. For example, the following term is a solution for the function max in the problem we just described:

$$\textbf{i} \text{ б } \textbf{x} \text{ б } \textbf{then} \text{ у } \textbf{e} \textbf{1} \text{эe } \textbf{x}.$$

The advantage of such a logic-based formalism is that it achieves a separation from solver and specification, which allows SyGuS to be *solver-agnostic*. Several different SyGuS solvers have been developed (e.g., [4,7,21,26]), many of which use drastically different internal algorithms that have different strengths for solving different kinds of problems. Moreover, a user of SyGuS need not consider the differing input languages or characteristics of these solvers, and instead can encode their problem just once in the SyGuS format to have access to all the different solvers.

While SyGuS achieves—and shows the benefits of—solver-agnosticity, it *fails to achieve domain-agnosticity* because the framework is targeted specifically at expressions. For example, consider the scenario from Sect. 1: Monica would be unable to encode her problem in SyGuS, because the grammar <sup>G</sup>ex in Fig. <sup>1</sup> contains a production with a while loop, and loops, which require a custom semantics, cannot be expressed in any decidable theory—a key restriction of SyGuS. SyGuS also does not allow one to express intent outside of the behavioral specification ψ, which would have prevented Monica from trying to optimize the program obtained from UltraSynth in Fig. 2b.

All in all, the current state of program synthesis is an unsatisfactory mess, as depicted in Fig. 3. There are multiple non-interoperable solvers with different input languages, targeting different synthesis domains with varying degrees of overlap. SyGuS, by virtue of solver-agnosticity, provides a unified approach to synthesizing expressions, which forms the basis of multiple solvers. However, while SyGuS is a bright spot, it fails to be general: it does not cope with (1) the *variety of domains* used in synthesis, required to deal with arbitrary languages (e.g., SQL, regular expressions, and imperative programs), and (2) the *variety of collateral considerations* that arise for different domains (e.g., types, quantitative objectives, and probabilities).

#### 2.2 What Does a Programmable Synthesis Framework Look Like?

Our vision of programmable synthesis can be summed up as follows:

#### programmable synthesis

== easily instantiable, domain-agnostic, solver-agnostic synthesis framework.

In contrast with Fig. 3, what we would like to have is depicted in Fig. 4, where both user and solver work with a unified general format, regardless of domain or solving technique. Such an approach would allow one to specify a synthesis problem once and for all, without having to worry about the underlying solving strategy. To achieve this goal, it is necessary to distill out the essence of many program-synthesis problems into a specification formalism that is ground in formal methods (e.g., automata and logic) and is agnostic to any specific domain of application. This degree of abstraction also opens the opportunity to lift certain synthesis algorithms and ideas to a higher level that makes these algorithms reusable across different tools. Our framework can then interface to

Fig. 4. Programmable program synthesis, where a synthesis problem with arbitrary constraints can be written once and for all in a general format, which can then be dispatched to compatible solvers.

different solving tools (backend solvers) in a way that allows one to easily swap one solver for another, or to use multiple solvers in tandem. If our vision is achieved, the capabilities that would be available to tool designers—discussed in greater detail in Sect. 5—would allow synthesis tools to be created that have the kind of flexibility that Monica expected and needed in Sect. 1.1.

Let us now be more concrete about the requirements for such a framework for synthesis. Following the pattern for yacc given above, a framework for synthesis could follow a similar scheme:

	- Input: an <sup>F</sup>syntax specification of a language's syntax, and an <sup>F</sup>semantics specification of the language's semantics.
	- Output: a function SynthL,-·*<sup>L</sup>* (·) that takes <sup>ϕ</sup> as input and computes Synthesize(L, -·<sup>L</sup>, ϕ).

To be even more concrete, <sup>F</sup>syntax could be a regular-tree grammar [5],<sup>1</sup> and <sup>F</sup>semantics would be defined over the grammar in a compositional manner, production by production. What we have called collateral considerations (types, quantitative objectives, probabilities, etc.) would be handled as part of the <sup>F</sup>syntax or <sup>F</sup>semantics specifications, depending on the issue at hand. For instance, constraints on program behavior, such as refinement types [24], minimizing/bounding evaluation resources usage [11,15], and probabilistic behavior

<sup>1</sup> The grammar would also be equipped with production-by-production pretty-printing rules to specify how to convert a tree to its textual representation.

[22], are semantic concerns that would be part of Fsemantics. Other considerations would be part of Fsyntax, such as bounds on the use of syntactic constructs [12], or the use of probabilistic generative models of syntactic structures [3,17]. For instance, for these two issues, one could weight the productions of the grammar with values from a semiring, and place a (possibly learned) distribution on the productions, respectively.

The scheme in the box above would allow us to meet the goals of being both domain-agnostic and solver-agnostic,<sup>2</sup> as long as (*i*) the formalisms for <sup>F</sup>syntax and <sup>F</sup>semantics are sufficiently powerful to qualify as "domain-agnostic," and (*ii*) specifications in these formalisms can be analyzed and broken down into components that can be farmed out to existing solvers (or perhaps to new implementations of the kinds of algorithms used in existing solvers).

*Who benefits from such a framework?* The existence of a domain- and solveragnostic framework benefits two parties: (*i*) users of synthesis tools such as Monica, and (*ii*) designers of synthesis tools, such as the team behind Ultra-Synth. Both scenarios can be illustrated by making an analogy with LLVM [20] which provides an intermediate representation for compilation that is similarly both domain- and solver-agnostic. Users of LLVM, which are front-end language designers, benefit from two facts: (*i*) that the LLVM IR is rich enough to support the range of features their language might have, and (*ii*) that once their language is compiled down into LLVM IR, the entire library of LLVM IR optimizations is accessible to them. Similarly, a programmable synthesis framework benefits users in two ways: (*i*) by supporting the full range of features that may be required for a synthesis problem, and (*ii*) by putting multiple solvers within reach for problems written in the framework. Additionally, a well-defined framework also facilitates *reuse* of problem components: for example, Monica can reuse <sup>G</sup>*ex* for synthesizing other array transformations.

On the other hand, backend optimization designers of LLVM benefit from the fact that once their optimization is written in LLVM, all LLVM users may easily access those optimizations if need be. Similarly, tool designers for a programmable synthesis framework rest easy knowing that once their tool supports the framework, those who need it will find it accessible and easy to use regardless of what internal techniques they decide to use. Note that while the framework intends to be general, tools that interface with the framework can choose to be selective in the problems they support—it is up to the users, or perhaps the framework designers, to match a problem with an appropriate solver (similar to how language designers mix and match backend optimizations for their language in LLVM). In addition, advances at the framework-level—such as

<sup>2</sup> We also acknowledge that even the scheme given above, which was modeled on the one for yacc, is open to revision. In particular, the additional degree of parameterization for synthesis (L, -·*L*, and ϕ) opens the door for a variety of alternatives, based on different "binding times" for L, -·*L*, and ϕ. For instance, a solver that uses different abstract domains as part of a refinement-based search strategy [29] would have L and ϕ fixed, but vary -·*L*. Similarly, when one has quantitative syntactic objectives [12], the solver would carry out its search with ϕ fixed, L varying, and -·*<sup>L</sup>* induced as L changes.

the development of meta-algorithms, as illustrated in Sect. 4.2—instantly benefit all tools that support the framework.

*This Paper.* New technical challenges, as well as new opportunities, come along with our broader goals. In this paper, we present some of the work that we have done toward building the kind of framework sketched out above.


These steps are just the beginning of what we expect to be a multi-year journey into designing a framework that achieves our goals, and solvers for such a framework. We discuss some of the open challenges and opportunities in Sect. 5.

### 3 Programmable-Synthesis Specifications

Designing synthesis frameworks that are *programmable* requires one to formally abstract the essence of how one specifies different program-synthesis problems. While we do not claim to have developed a completely unified framework that can capture all synthesis problems yet, in this section we present two ideas for programming many practical synthesis problems: (*i*) SemGuS, a framework that uses logic and formal methods to make the search space and specifications of all synthesis problems easy to program in arbitrary domains (Sect. 3.1), and (*ii*) an extension of SemGuS that allows one to specify *quantitative objectives* over the syntactic structure of a synthesized program (Sect. 3.2).

#### 3.1 Semantics-Guided Synthesis

Existing work on program synthesis [1] typically identifies two main components to a synthesis problem: (*i*) a *search space* of candidate programs, which is in essence a small programming language, and (*ii*) a *behavioral specification*, which describes what the synthesized program should do. A programmable synthesis framework must represent (at the very least) these two components in a domainand solver-agnostic way. Take the syntax-guided synthesis (SyGuS) framework, for example: SyGuS achieves solver-agnosticity by representing the search space as a regular tree grammar, and the specification as a Boolean formula in a decidable background theory.

Then why is SyGuS, and this particular combination of representations, unable to achieve domain-agnosticity? The syntactic component of SyGuS the grammar—actually does achieve some degree of domain-agnosticity, in the sense that one is free to define a language of one's own. However, SyGuS requires that the specified grammar be contained within a fixed background theory, which are terms with a pre-defined fixed and standardized *semantics*. While this design choice makes the solutions to SyGuS problems easy to verify (using an SMT solver), it limits the *programmability of the search space*.

For example, let us reconsider the example in Sect. 1. If Monica attempted to write her example as a SyGuS problem, she would have been unable to use loops because loops are not part of the supported background theory. What if Monica wanted a solution that operates over a DSL, or had some pre-defined components that she wanted to use (like len(arr))? What if Monica wanted to synthesize regular expressions, or some other programs with relatively nonstandard semantics?

One can intuitively understand these scenarios as synthesis problems over different programming languages (search spaces)—a DSL, library functions, regular expressions. To support different programming languages, a synthesis framework needs more than the ability to accept a syntax, it needs the ability to accept a semantics for a language as well. Therefore, developing a programmable synthesis framework capable of supporting all these scenarios requires designing a solver-agnostic way of specifying the semantics of such arbitrary programming languages. SyGuS has shown that regular tree grammars are an effective formalism for programming the syntax of a search space; we extend this with a formalism to program the semantics of the search space as well, which, to achieve true domain-agnosticity, need not be constrained to a fixed background theory.

*Semantics as Constrained Horn Clauses.* Our solution to this challenge is the Semantics-Guided Synthesis (SemGuS) framework [14], which allows users to customize the syntax and semantics of the search space. To see how Sem-GuS supports programmable semantics, let us consider the production *Start* → while x>=0 do S from Fig. <sup>1</sup> as an example. This production is a while loop, and part of the semantics for a term produced by this production can be expressed using the inference rule below<sup>3</sup> (where Γ represents a state that maps variables to integer values):

$$\frac{\begin{bmatrix} \mathbf{x} \mathbf{\succ} \mathbf{0} \end{bmatrix} (\varGamma) = \mathbf{T} \mathbf{r} \mathbf{u} \quad \|s\| (\varGamma) = \varGamma\_1 \quad \|\mathbf{wh1e \mathbf{x}} \mathbf{x} \mathbf{\succ} \mathbf{0} \text{ do } s\| (\varGamma\_1) = \varGamma\_2}{\|\mathbf{wh1e \mathbf{x}} \mathbf{x} \mathbf{\succ} \mathbf{0} \text{ do } s\| (\varGamma) = \varGamma\_2} \tag{1}$$

Such semantics are supported in the SemGuS framework by expressing the inference rule in Eq. (1) as a Constrained Horn Clause (CHC). CHCs are logical formulas, and more precisely, they are implications where one is only allowed

<sup>3</sup> A similar rule must be added for the case in which the guard evaluates to false.

to have a single relation in the conclusion, and a conjunction of relations along with one constraint in the premise:

Definition 1 (Constrained Horn Clauses.). *A* Constrained Horn Clause *is a first-order formula of the form*

$$(\forall \overrightarrow{x^{\cdot}}, \overrightarrow{x\_1^{\cdot}}, \dots, \overrightarrow{x\_n^{\cdot}}. (\phi \land R\_1(\overrightarrow{x\_1^{\cdot}}) \land \dots \land R\_n(\overrightarrow{x\_n^{\cdot}}) \implies H(\overrightarrow{x^{\cdot}})),$$

*where* φ *is a constraint over some background theory that may contain variables from* −→x , <sup>−</sup>x <sup>→</sup><sup>1</sup>,..., <sup>−</sup>x <sup>→</sup>n*, and* <sup>R</sup><sup>1</sup>,...,R<sup>n</sup> *and* <sup>H</sup> *are uninterpreted relations.*

In SemGuS, search spaces are represented as regular tree grammars, where productions have associated semantics. In Eq. (1), the semantics of a term x>=0 is represented using the semantic function -·. SemGuS, assumes that each nonterminal <sup>N</sup> appearing in the grammar has a corresponding logical relation sem<sup>N</sup> , which we refer to as the *semantic relation*, that represents the behavior of the semantic function -· in Eq. (1). For example, the expression <sup>s</sup>(Γ) = <sup>Γ</sup><sup>1</sup> from Eq. (1) can be translated into the relation sem<sup>S</sup>( s, Γ, Γ<sup>1</sup>).

*Example 2 (Semantic Rules as CHCs).* The following CHC captures how one would express in SemGuS the semantics of the production *Start* → while x>=0 do S shown in Eq. (1):

$$\frac{\Gamma[\mathbf{x}] \ge 0 \quad \mathsf{sem}\_S(\langle s, \varGamma \rangle, \varGamma\_1) \quad \mathsf{sem}\_{Start}(\langle \mathbf{while} \,\mathbf{1e} \,\mathbf{x} \succeq \mathbf{0} \text{ do } s, \varGamma\_1 \rangle, \varGamma\_2)}{\mathbf{sem}\_{Start}(\langle \mathbf{while} \,\mathbf{1e} \,\mathbf{x} \succeq \mathbf{0} \text{ do } s, \varGamma \rangle, \varGamma\_2)}\tag{2}$$

One can read Eq. (2) as the following implication:

$$\mathsf{sem}\_S(\langle s, \varGamma \rangle, \varGamma\_1) \land \mathsf{sem}\_{Start}(\langle \mathsf{while1ex} \times \mathsf{0} \text{ do } s, \varGamma\_1 \rangle, \varGamma\_2) \land \varGamma[\mathsf{x}] \ge 0 \implies \qquad \qquad (3)$$

$$\mathsf{sem}\_{Start}(\langle \mathsf{while1ex} \times \mathsf{0} \text{ do } s, \varGamma \rangle, \varGamma\_2)$$

Equation (3) is a CHC where sem*Start* and sem*<sup>S</sup>* are relations, and <sup>Γ</sup>[x] <sup>≥</sup> <sup>0</sup> corresponds to the first-order constraint φ.

SemGuS allows one to specify multiple such CHCs<sup>4</sup> for each *production* in the grammar. CHCs are the logical formalism of choice for expressing these semantics in a language-agnostic way, which are an intuitive and expressive format.

*The* SemGuS *Framework.* Once a user has understood how to define a semantics for their grammar, a SemGuS problem then can be specified simply as a synthesis problem over a grammar equipped with such a semantics.

Definition 2 (SemGuS). *A* SemGuS *problem over a theory* T *is a tuple* sem = (G-·, ψ(x, f(x)))*, where:*

*–* G *is a regular tree grammar equipped with the semantics* -·*,*

<sup>4</sup> The ability to define multiple semantic rules for a production is useful for productions such as while loops, which are commonly equipped with two rules that describe looping and loop termination.


*<sup>A</sup> solution to the* SemGuS *problem* sem *is a term* s <sup>∈</sup> L(G-·) *such that* ψ(x, s(x)) *holds.*

*Example 3 (Monica's Synthesis Problem in* SemGuS*).* Consider the synthesis problem Monica had in Sect. 1. Let Gex-· be the grammar <sup>G</sup>ex from Fig. 1, equipped with semantic rules such as the one defined in Eq. (2). Let E <sup>=</sup> {[−1, <sup>2</sup>, <sup>3</sup>, <sup>10</sup>, <sup>31</sup>, <sup>−</sup>14, <sup>−</sup>11]}, the input array Monica considered for her task. Let ψ(arr, f(arr)) be a formula over the theory of arrays and CLIA describing what it means for the program f to be correct on an input arr:

$$\psi(arr, f(arr)) \equiv \bigwedge\_{0 \le i < len(arr)} f(arr)[i] = ITE(arr[i] > 0, arr[i] + 10, 0).$$

Then semex = (Gex-·, arr∈<sup>E</sup> <sup>ψ</sup>(arr, f(arr))) is a SemGuS problem defined over a background theory of arrays and CLIA—the behavioral specification requires that the final program satisfies all the examples in E. <sup>5</sup> Moreover, semex is written in a completely logical format, and is thus not tied to a specific tool like UltraSynth and can be dispatched to multiple backend solvers (assuming tooling) as Monica pleases.

The ability to customize the semantics for a language in a framework allows that framework to support a plethora of different synthesis problems. One can define synthesis problems over regular expressions, domain-specific languages, imperative languages, or any other language that has a semantics definable as CHCs within the framework, all of which can be tested using different solvers utilizing different strategies.

*Example 4 (Regular Expressions Synthesis in* SemGuS*).* Synthesis problems over regular expressions can be expressed succinctly in SemGuS. The grammar of regular expressions can be captured with the following grammar, where c is a character and φ the empty set:

$$R \to c \mid \epsilon \mid \phi \mid R + R \mid R \cdot R \mid R^\* $$

Using CHCs, one can also naturally express the semantics of terms r <sup>∈</sup> L(R). For example, the semantics of Kleene star can be given as the following two CHCs:

$$\begin{array}{cc} \mathsf{sem}\_R(r^\*, \epsilon) & \mathsf{sem}\_R(r^\*, s\_2) & s = s\_1 s\_2 \\ \hline - \end{array}$$

<sup>5</sup> In this example, one could have used a formula simply describing the input/output examples instead of a more complex logical formula. We chose the latter option to illustrate how the behavioral specification can involve terms in interesting theories e.g., CLIA and arrays.

The rules are based on the expansion r<sup>∗</sup> <sup>→</sup> <sup>+</sup> r · r∗: the first rule lets r<sup>∗</sup> accept , and the second rule accepts a string s by finding two substrings s1, s2, such that <sup>s</sup><sup>1</sup> is accepted by <sup>r</sup>, <sup>s</sup><sup>2</sup> is accepted by <sup>r</sup>∗, and the concatenation <sup>s</sup><sup>1</sup> · <sup>s</sup><sup>2</sup> is equal to s. The specification of the problem can then use expressions of the form sem<sup>R</sup>(r, s) and <sup>¬</sup>sem<sup>R</sup>(r, s) to denote whether an example s is positive or negative, respectively.

#### 3.2 Adding Quantitative Syntactic Objectives

In the example discussed in Sect. 1.1, the original synthesis problem Monica posed to the solver was under-constrained and caused the underlying tool to synthesize an undesirable solution that contained unnecessary operations. While the logical-specification mechanism is powerful, it can only capture the functional requirements of the synthesis problem—e.g., the program should perform correctly on a given set of input/output examples. When multiple possible programs can satisfy the specification, a programmable synthesis framework should provide a way to prefer one to the other—i.e., the user of the framework should be able to describe a quantitative objective. In this section, we show how the formal foundations of SemGuS (i.e., the use of grammars and logic) allow us to easily extend the framework to incorporate quantitative objectives over the syntax of the synthesized program. The ideas we present were originally described in the context of SyGuS [12]; here we show how they can also be applied to SemGuS.

*Adding Quantitative Objectives Using Weighted Grammars.* Recall that a Sem-GuS problem is given along with a *regular tree grammar* specifying the search space. In our running example, Monica would like to synthesize a program that has few occurrences of the minus operator. A natural way to express this intent is allowing Monica to tag productions involving such an operator with a cost, let's say 1. Our quantitative extension of SemGuS builds on this intuition and allows users to add weights/costs to productions in the grammar. This extension leads to a well-studied formalism, weighted tree grammars, keeping the SemGuS framework general. Intuitively, a *weighted tree grammar* is a grammar in which each production p has an associated weight/cost μ(p).

Intuitively, the weight of a derivation tree is the *sum* of the weights of all productions.<sup>6</sup> For simplicity, in this paper, we assume that the domain of weights is the natural numbers, and that their sum is the usual application of the + operator. We use <sup>w</sup>G(t) to denote the weight of a term <sup>t</sup> with respect to the weighted grammar G.

With the weights specified by the weighted grammars, users can specify quantitative objectives as constraint objectives and optimization objectives. A constraint objective is a predicate ω(v) over a numerical variable v; we say that

<sup>6</sup> Weights have to come equipped with operators that tell us how to combine weights of individual productions to obtain the weights of terms. Formally, the weights must be from a semiring; we refer the reader to the original work on this topic [12] for details.

a term t satisfies the constraint objective if ω(wG(t)) holds. An optimization objective is a flag opt ∈ {True, False} indicating whether we want to minimize the weight of the solution.

*Example 5.* Recall that in the example introduced in Sect. 1, Monica wants to avoid redundant occurrences of the minus ( - ) operator. To express this intent in SemGuS, Monica can utilize the following weighted grammar.

$$\begin{array}{l} \text{Start} \rightarrow x = \texttt{len}(\texttt{arrr}) - 1; \texttt{while} \texttt{x} \texttt{z} \texttt{do} \ S\\ \quad S \rightarrow \texttt{arr}\texttt{[E]} = \texttt{arr}\texttt{[E]} + E \mid \texttt{arr}\texttt{[E]} = E \mid \\ \quad x = E \mid S; \; S \mid \texttt{if} \ \texttt{arr}\texttt{[x]} \texttt{>0} \ \texttt{then} \ S \ \texttt{[} S\\ \quad E \rightarrow \texttt{0} \mid \texttt{1} \mid \texttt{x} \mid E + E \mid E - E / 1 \end{array}$$

In the weighted grammar, only the rule E <sup>→</sup> E - E is assigned the weight 1. All other rules are assigned the weight 0 (omitted for readability). The weight of a term <sup>t</sup> with respect to this grammar is the number of occurrence of the minus operator in t. If Monica wants to restrict the number of occurrences of the minus operators to be less than <sup>5</sup>, she can use the constraint objective ω(v) = v < <sup>5</sup>. Furthermore, if she want to minimize the occurrences of the minus operator, she can set the flag opt to True.

To summarize, a SemGuS problem with quantitative syntactic objectives is a tuple sem = (W-·, ψ(x, f(x)), ω, opt) where W-· is a weighted grammar with a corresponding semantics, ψ is a Boolean formula like before, ω is the constraint objective, and the flag opt is the optimization objective. The goal is to find a solution that not only satisfies the specification ψ, but also the quantitative objective ω, and is of minimal cost if opt is set to True.

Quantitative syntactic objectives are useful in applications such as programming by examples [10] and program repair [6], where it is desirable to produce small programs with fewer constants, because such programs are more likely to generalize to examples and test cases outside of the set of examples given by the user. When allowing real-valued weights, syntactic objectives can be also used to find the most likely solution with respect to a given probability distribution. We can assign productions weights that represent their probabilities; the weight of a candidate solution is its likelihood.

#### 4 Programmable-Synthesis Solvers

While a programmable synthesis framework as discussed in Sect. 3 is certainly desirable, it is of little practical use if one is unable to solve the problems that are written in such a framework. In this section, we show that SemGuS problems can be solved practically. We first describe two general solving techniques for SemGuS (Sect. 4.1) and then present new algorithmic solving techniques enabled by the SemGuS framework (Sect. 4.2).

#### 4.1 General Solving Procedures for **SemGuS**Problems

We start off by presenting two solving procedures for general SemGuS problems we implemented as a tool, rooted in strategies commonly used in existing program synthesizers: enumeration (used in the tool MESSY-Enum) and constraint solving (used in the tool MESSY). Specifically, we will be considering SemGuS-with-examples problems: SemGuS problems where the specification is given in terms of a finite set of examples E. An algorithm for solving SemGuSwith-examples problems can be combined with counterexample-guided inductive synthesis (CEGIS) [27], which generates counterexamples in case a synthesized answer does not meet the general specification, to iteratively increase the example set E and eventually obtain a correct program.

MESSY-Enum: A Basic Enumerator for **SemGuS** Problems. Because SemGuS also relies on a grammar to specify the syntax of valid terms, like SyGuS, one can employ a simple enumerator that generates terms of increasing size from the grammar and test the enumerated terms against the behavioral specification. With SemGuS, a term (representing a program) cannot be executed directly, because the semantics to ascribe to it has been specified in the semantic specification. However, because the semantics is specified with CHCs, the term can be executed with a level of interpretation supplied by an off-theshelf CHC solver. Therefore, MESSY-Enum employs an off-the-shelf CHC solver such as [18] to check if the CHCs are consistent with the specification.<sup>7</sup>

Concretely, given a term <sup>t</sup><sup>e</sup> to test, one can use the following CHC to check whether <sup>t</sup><sup>e</sup> meets the specification:

$$\frac{\bigwedge\_{e\_i \in \mathsf{E}} \mathsf{sem}\_{Start}(\langle e\_i, t\_e \rangle, o\_i)}{Realizable} \text{ } \mathsf{Query} \tag{4}$$

The Query rule in Eq. (4) exactly encodes the specification as a CHC: it asks whether the semantics of <sup>t</sup><sup>e</sup> computed by semStart is consistent with the set of input-output examples E. If so, the conclusion *Realizable* is provable using the existing set of CHCs—i.e., <sup>t</sup><sup>e</sup> is a solution to the synthesis problem.

Because we cannot directly execute candidate terms and instead rely on CHC solvers (which may be treated as a blackbox), it is difficult to employ common enumeration optimizations, such as behavioral equivalence caching, or equality saturation. Developing an enumeration-based solver capable of utilizing these ideas would require generating an explicit and efficiently executable interpreter from the given semantics, which is an interesting research challenge and future direction that we discuss in Sect. 5.

<sup>7</sup> One can treat CHC solving as akin to a proof search, where the objective is to prove that a specific query holds (in this case, *Realizable* from Eq. (4)) using the provided CHCs.

MESSY: **SemGuS** Problem Solving as CHC-Solving. MESSY-Enum uses a CHC solver to check whether an enumerated term <sup>t</sup><sup>e</sup> is consistent with the specification or not—however, CHC solvers are also capable of automatically searching for terms that satisfy the specification, as well. Our next solver, MESSY, takes advantage of this fact by expressing *both* the syntax of the search space and the semantics using CHCs. Once the entire search space is modeled this way, one can then slightly modify the Query rule to accommodate this change and directly use a CHC solver to solve the entire SemGuS problem. In essence, Messy reduces solving the SemGuS problem into finding a configuration of variables for which the set of CHC rules (containing syntax, semantics, and specification) is valid—similar to how constraint-based methods in existing synthesizers reduce the synthesis problem to one of solving a set of constraints.

*Example 6 (MESSY Encoding).* We show how the syntax and semantics used in the production *Start* <sup>→</sup> while x>=0 do S from Fig. <sup>1</sup> can be captured using CHCs. This production states that one can obtain a syntactically valid term while x>=0 do s <sup>∈</sup> L(*Start*) for the nonterminal *Start*, given a valid term s <sup>∈</sup> L(S). Equation (5) encodes this idea as a CHC using the *syntax relations* syn<sup>S</sup>, and syn*Start*, which capture whether the supplied arguments are valid terms that may be derived from the corresponding nonterminals *S*, and *Start*.

$$\frac{\text{sym}\_S(s)}{\text{sym}\_{Start}(\text{while } \mathbf{x} \text{:= 0 do } s)}\tag{5}$$

Because the syntax relations provide a way to guarantee that a term t is a valid term in the syntax of a SemGuS problem, one can rewrite the Query rule from Eq. (4) to use this relation instead of an explicitly enumerated term <sup>t</sup><sup>e</sup>.

$$\frac{\mathsf{sym}\_{Start}(t) \quad \bigwedge\_{e\_i \in \mathsf{E}} \mathsf{sem}\_{Start}(\langle e\_i, t \rangle, o\_i)}{\mathsf{Realizable}} \text{ Quey} \tag{6}$$

The new Query rule in Eq. (6) has the term t as a free variable—i.e., proving *Realizable* amounts to finding a term t <sup>∈</sup> L(*Start*) that is consistent on the input-output examples. A CHC solver presented with this rule, in tandem with the syntax and semantic rules, will then attempt to find a configuration of t such that *Realizable* holds. If the solver can prove that the premises of Equation Eq. (4) hold, then the term t is a solution to the SemGuS problem.

One of the advantages of using such a CHC-based method is when dealing with cases where there is no answer to the synthesis problem, i.e., when there exists no t such that Realizable holds. In this case, the SemGuS problem contains *no* answer satisfying the specification within its search space; we say that such a problem is *unrealizable*. Proving unrealizability is something that many existing solvers fail to consider, but is important: for example, Monica would not have had to wait for several hours after modifying the grammar in Sect. 1 if her solver had been able to show that the problem was unrealizable.

#### 4.2 Meta Algorithms for Solving **SemGuS**Problems

Now that we have shown how to build solvers for general SemGuS problems (that do not involve quantitative objectives), we turn to 'meta'-algorithms for solving SemGuS problems, which are 'meta' in the sense that they (*i*) may be used atop any general SemGuS solver, (*ii*) generate modified SemGuS problems (rather than solutions) that can be easier to solve than the original SemGuS problem or can be used to solve SemGuS problems with quantitative objectives. The key component behind these meta-algorithms is the customizability of the search-space description in SemGuS.

A Meta Solver for Quantitative Objectives. We first present an algorithm for solving SemGuS problems with quantitative objectives [12]—i.e., where productions in the grammar have weights. We assume, for simplicity, that the only quantitative objective is to find the program of *least cost* that satisfies the specification. The idea of the algorithm is to iteratively reduce the SemGuS problem with a quantitative objective to a sequence of SemGuS problems without quantitative objectives, which are used to iteratively find a solution that has least cost—i.e., at each step of the sequence the cost of the solution is improved.

The algorithm operates as follows. Initially, we are given a SemGuS problem sem with a weighted grammar W (we omit the semantic information for brevity) and with the minimization objective opt set to true. <sup>8</sup> The first step of the algorithm is to construct an unweighted grammar G<sup>W</sup> by merely erasing all the weights in W. We can now use any SemGuS solver to solve the resulting SemGuS problem and obtain a term t<sup>0</sup>. This term will have a weight <sup>c</sup> according to the weighted grammar W, but it might not be the term of least cost that satisfies the specification. Our algorithm therefore tries to find out whether a solution with a lower weight exists, and accordingly constructs an (unweighted) grammar G<sup>W</sup> <c such that a term <sup>t</sup> is accepted by the grammar <sup>G</sup><sup>W</sup> <c if and only if the weight of t according to W is less than c. When the weights are natural numbers, this construction is always possible [12]. We now have again an unweighted grammar, and we can use a SemGuS solver to solve the resulting problem. This procedure can be repeated until no better solution exists.

*Example 7.* Consider the weighted grammar W we presented in Example 5. In particular, let us focus our attention on the following subset of productions that involve non-zero weights:

$$E \to \mathbf{0} \mid \mathbf{1} \mid \mathbf{x} \mid E \star E \mid E \text{ -- } E/1$$

The grammar G<sup>W</sup> <sup>&</sup>lt;3, which accepts all terms of weight less than 3 is as follows:

$$\begin{array}{l} E \rightarrow E\_2 \mid E\_1 \mid E\_0\\ E\_2 \rightarrow E\_1 \text{ - } E\_0 \mid E\_0 \text{ - } E\_1\\ E\_1 \rightarrow E\_0 \text{ - } E\_0\\ E\_0 \rightarrow \mathbf{0} \mid \mathbf{1} \mid \mathbf{x} \mid E\_0 \star E\_0 \end{array}$$

<sup>8</sup> For simplicity, we assume no further quantitative objectives are present, but the general case can be handled using similar ideas [12].

Intuitively, each non-terminal <sup>E</sup><sup>i</sup> produces all and only terms with exactly <sup>i</sup> minus operators.

The meta solver for quantitative objectives shows how using a solver-agnostic specification formalism—i.e., grammars—enables algorithms that operate at the specification level and can be reused across multiple solvers.

Underapproximating Semantics with **SemGuS**. The previous section showed how the programmability of the search-space syntax (i.e., the grammar) allows us to design meta-algorithms to solve SemGuS problems involving quantitative objectives. In this section, we show how the programmability of the search-space *semantics* can be used to build meta-algorithms that can make synthesis faster. The key idea is to generate "simpler" variants of the original SemGuS problem that use an *underapproximating semantics*, where an underapproximating semantics is defined as a subset of the original semantics that must be precise on the subset on which it is defined.

Definition 3. *For a grammar* G *equipped with a semantics* -·*, we say* -·- underapproximates -· *on* G*, or that* -· *is an* underapproximating semantics *for* G *with respect to* -·*, if for every term* t <sup>∈</sup> L(G)*, every state* Γ*, and every value* v *on which* -· is defined*,* t-(Γ, v) = t(Γ, v)*.*

One easy way to underapproximate a semantics is to simply "eliminate" certain operators from a grammar by not defining semantic rules for them. However, the concept of underapproximation need not be bounded to eliminating operators from a grammar—it may have a fully semantic meaning instead, for example, a bound on the number of possible loop iterations. The key intuition is that underapproximation is sound for use in synthesis—if a term t is the answer to a synthesis problem sy, sy actually does not need to contain any syntax or semantics outside of what is used to define and compute t. (In contrast, overapproximation is sound for proving unrealizability.)

*Example 8.* Recall, once again, the synthesis problem Monica has in Sect. 1. The grammar <sup>G</sup>ex of Fig. <sup>1</sup> contains a while loop, which has a complex semantics that can be expensive to compute and, most importantly, allows nonterminating behavior. Most existing synthesizers [27,28] explicitly prohibit nontermination by only considering finitely many unrollings for loops (because most answers to a synthesis problem will indeed terminate).

Fortunately, Monica knows that on her example [−1, <sup>2</sup>, <sup>3</sup>, <sup>10</sup>, <sup>31</sup>, <sup>−</sup>14, <sup>−</sup>11], the loop should iterate no more than 7 times to process every element of the array. Monica may then choose to supply the synthesizer with an underapproximating semantics that limits the number of loop iterations to 7, which could greatly reduce the amount of computation the synthesizer must perform—for example, a naive enumerator might get stuck on a nonterminating loop when using the precise semantics, while terminating quickly when using the underapproximating semantics. Such a semantics can be expressed easily by adding a loop counter c to the semantics of loops given in Eq. (2), yielding the following CHC:

$$\begin{array}{llll} c \geq 0 & I^r[\mathbf{x}] \geq 0 & \mathsf{sens}(\langle s, \varGamma, c - 1 \rangle, \varGamma\_1) & \mathsf{sens}\_{Start}(\langle \mathtt{while} \, \mathbf{1e} \, \mathbf{x} \succeq \mathbf{0} \, \mathbf{do} \, s, \varGamma\_1, c - 1 \rangle, \varGamma\_2) \\\hline & & \mathsf{sens}\_{Start}(\langle \mathtt{while} \, \mathbf{1e} \, \mathbf{x} \succeq \mathbf{0} \, \mathbf{do} \, s, \varGamma, c \rangle, \varGamma\_2) \end{array} \tag{7}$$

Setting c = 7 in the Query rule now ensures that loops run at most 7 iterations.

Abstract Semantics with **SemGuS**. Similar to how we used underapproximating semantics to find solutions to a SemGuS problem, abstract (overappoximating) semantics can be used to prove that a SemGuS problem is unrealizable.

Definition 4. *For a grammar* G *equipped with a semantics* -·*, we say* -·# *is an* abstract semantics *for* G *with respect to* -· *if there exists an abstraction function* α *and a concretization function* γ*, such that for all* t <sup>∈</sup> L(G)*, if* t(Γ, v) *holds, then* t#(α(Γ), α(v)) *holds, and* Γ <sup>∈</sup> γ(α(Γ))*,* v <sup>∈</sup> γ(α(v))*, i.e.,* α *and* γ *form a Galois connection.*

In contrast to underapproximating semantics, abstract semantics are sound when used to prove *unrealizability*—i.e., that a synthesis problem has no solution that satisfies its specification within its search space. Consider the use of abstract interpretation in program analysis: abstract interpretation is most often used to prove that a program cannot reach a certain set of bad states, while often being unable to guarantee that a program will produce a specific value, due to lack of precision. Similarly, an abstract semantics will often be unable to guarantee that a synthesized program satisfies the specification, due to lack of precision—but it can guarantee that all programs in the search space will *never* be able to produce a certain set of values, which can be used to prove unrealizability.

*Example 9.* Consider the scenario from Sect. 1, in which Monica removed subtraction from her grammar in an attempt to simplify the synthesized program. The removal of subtraction made the problem unrealizable—and UltraSynth ran for hours on end because it could not prove that this was the case. While proving unrealizability can be very difficult in general, a solver capable of reasoning about abstract domains and semantics could have utilized an (abstract) semantic rule such as Eq. (8) below:

$$\frac{\mathsf{sem}\_E(\langle e\_1, \varGamma \rangle, \{pos\}) \quad \mathsf{sem}\_E(\langle e\_2, \varGamma \rangle, \{pos\})}{\mathsf{sem}\_E(\langle e\_1 + e\_2, \varGamma \rangle, \{pos\})} \tag{8}$$

Equation (8) is defined on the abstract domain {pos, zero, neg}—corresponding to positive, zero, and negative values—and captures the fact that the sum of two positive numbers will always be positive. This rule will be able to prove that <sup>G</sup>*ex* without subtraction will *never* be able to modify x to a negative value, and thus that no program in the search space will terminate (leading to unrealizability).

Unrealizability is a property that is ignored by many current synthesizers, but it is a very important property nonetheless. One practical way to think about unrealizability is as a sanity check, like a type system: the fact that a synthesis problem provided by an end user is unrealizable means that the synthesis problem is malformed in the sense that the user has got some of their specifications wrong. Similar scenarios happen daily with ordinary programming, and we expect them to happen with synthesis as well—thus, it is desirable that synthesizers be able to detect these problems, and report them early on if possible, without running indefinitely, as in Sect. 1. Unrealizability also has applications in computing optimal solutions, as in Sect. 4.2: unrealizability given a grammar with a lower weight bound ensures that the current solution is optimal.

#### 5 The Future of Programmable Synthesis and **SemGuS**

We hope we have convinced the reader that synthesis could use more programmability, and that SemGuS addresses many of the programmability issues of existing synthesis work. But what lies ahead? How can we make programmable synthesis truly practical? In this section, we first outline some of the steps we are undertaking to answer this question (Sect. 5.1).

More importantly, we would also like to emphasize that *the vision of programmable program synthesis can only be realized through a community effort*. We will conclude this section with ideas to involve the synthesis community to help us realize our vision (Sect. 5.2).

#### 5.1 What Are We Working on Next?

In this section, we present some of the directions our group is pursuing in extending SemGuS to richer objectives and building better solvers for it. We also describe some open problems related to SemGuS.

*Interfacing Existing Program Synthesizers with* SemGuS. The bulk of our discussion in Sect. 3 was about achieving domain-agnosticity by building upon the ideas that SyGuS used in achieving solver-agnosticity. However, there also exist synthesis tools that are already domain-agnostic; most notably, solver-aided languages such as Sketch [27], Rosette [28], MiniKanren [8], and Prose [25]. While these tools are not solver-agnostic, they can in principle be used as SemGuS solvers by virtue of their domain-agnosticity.

To use such existing tools as SemGuS solvers, one must develop a compiler of sorts to translate a SemGuS problem (written in the logical format from Sect. 3) to the specific front-end language of the tool. This task is not trivial for a number of reasons. First, each of these tools implement restrictions on the types of synthesis problems they accept; these restrictions are what enables their fast algorithms. For example, Rosette, Sketch, and MiniKanren only support finite search spaces (i.e., finite grammars), and this fact is encoded in different ways for different tools (e.g., by imposing bounds on the search depth or by imposing syntactic bounds on the search space). Second, some of these solvers implicitly use limited semantics—e.g., Sketch limits how many times a loop can be executed. Third, some of these solvers require special inputs that are useful to guide the synthesis engine—e.g., Prose requires the user to provide a semantics for each operator in the input language as well as an *inverse semantics* that can be executed *backwards*; the inverse semantics is used to perform efficient enumeration.

Soundly compiling SemGuS problems to these tools requires one to modify the original problems to fit these restrictions. Thankfully, the flexibility of Sem-GuS comes to our aid! In Sect. 4.2, we have described ideas for transforming SemGuS problems using restricted grammars or underapproximating semantics. These transformations are sound for synthesis—i.e., a solution to the transformed synthesis problem, which satisfies the restrictions of a particular external tool, is still a solution to the original problem—and thus can be used to interface with external tools. We are currently working on automating such translations.

The case of Prose is particularly interesting in that it requires inverse semantics, which are not immediately available from a SemGuS problem. However, because SemGuS semantics are expressed logically as CHCs, one can automatically invert these semantics starting from the CHCs—we are currently developing a tool that performs this inversion automatically and uses the inverse semantics to interface with Prose.

Other more specialized solvers, such as those for synthesizing regular expressions [23], could also be interfaced with our framework, with the limitation that they will only be able to handle specific problems. The more general question here is: how can we determine whether a specific SemGuS problem is compatible with a specialized solver? We are working on designing "theories" that describe specific semantics for which specialized solvers exist. For example, if one were to use SemGuS to work with regular expressions, they could import the regular-expression theory, which by design would enable compatibility with certain solvers. Note that this approach is still solver-agnostic because any general SemGuS solver would still be able to use this problem definition.

*Lifting Existing Synthesis Algorithms to Work with* SemGuS. While interfacing existing synthesizers with SemGuS is one straightforward way of creating SemGuS solvers, we envision that higher efficiency can be achieved by designing solvers that take advantage of the structure of SemGuS problems. Is it possible to lift algorithms (not tools) that have previously been successful with SyGuS or other synthesizers up into SemGuS?

For example, consider the problem of building an efficient enumeration algorithm for SemGuS, an algorithmic technique that is now successfully employed in most SyGuS solvers [2,4,21]. The success of enumeration has been driven by a number of clever ideas for efficiently pruning the search space of relevant programs. An example was mentioned in Sect. 4.1, where we discussed the challenges with employing strategies such as behavioral equivalence caching or equality saturation on SemGuS due to the lack of an executable semantics—i.e., in Sem-GuS, evaluating a term on an input requires a costly call to a CHC solver. We are currently building an enumeration algorithm for SemGuS that addresses this limitation. Our algorithm first synthesizes an executable interpreter from the SemGuS problem semantics, and then uses this executable interpreter to guide the search. To scale, our approach must handle other challenges, which we are also working on—e.g., discovering which operators have a semantics that is associative or commutative can help us avoid enumerating equivalent terms.

While the generality of SemGuS is an obstacle to adapting some well-known algorithms, the same generality also helps SemGuS provide a natural interface to express other algorithms, such as program synthesis using abstraction refinement [29]. The approach taken here is to synthesize programs that work on an abstract domain, and repeatedly refine the abstract domain until a program is found that is correct under the concrete semantics. This approach, in a sense, uses a meta-algorithm that can be expressed naturally in SemGuS, as discussed in Sect. 4.2. We believe that SemGuS will naturally be able to express many such meta-algorithms, and further accelerate the development of new meta-algorithms.

*Supporting Richer Specifications.* Beyond the basic specification mechanisms, SemGuS already supports syntactic quantitative objectives through weighted grammars (Sect. 3.2). To capture the breadth of specifications appearing in modern synthesis applications, the SemGuS framework will have to evolve over time. While we are investigating a number of complex objectives that will require extensions to the framework (e.g., probabilistic specifications), in the following paragraph we describe a specification mechanism the current SemGuS framework can already capture for free: types.

Consider the problem of synthesizing a program that meets a given time complexity (or asymptotic resource usage in general) [11,16]. In existing work, such bounds are specified (and proven correct) using a dependent type system. The solver uses the type system to guide the search, by enumerating only terms that satisfy a certain type. We observe that the SemGuS framework is already able to capture such type-based specifications! In particular, types are a form of *static* semantics that can be associated with terms and, in most cases, typing rules can be encoded as CHCs, similarly to how one encodes semantic rules. For example, the following dependent type rule can be captured using a CHC where each typing judgment t : type is described using a relation r(t, type).

$$\frac{a:\{\text{Int}\mid\varphi\_a(v)\}\quad b:\{\text{Int}\mid\varphi\_b(v)\}\quad +: x:\text{Int}\to y:\text{Int}\to\{\text{Int}\mid v=x+y\}}{a+b:\{\text{Int}\mid v=x+y\land\varphi\_a(x)\land\varphi\_b(y)\}}\tag{9}$$

#### 5.2 What Can the Synthesis Community Do?

As we mentioned at the beginning of this section:

#### *The vision of programmable program synthesis can only be realized through a community effort.*

We discuss problems the community can help with in this concluding section.

*A Broader Scope for Synthesis.* The scope and potential of synthesis is very broad, in fact even broader than what has been discussed in this paper. An invited paper by Gulwani began [9]

Program Synthesis is the task of discovering an executable program from user intent expressed in the form of some constraints.

However, we feel that this viewpoint is actually somewhat narrow. We believe that insight on many problems can be obtained via the "lens" of synthesis: for many computing tasks, the goal is to produce some artifact to which some semantics is attached, and the process of producing that artifact can be thought of as a synthesis problem. For instance, in an AI planning problem, the artifact is a plan—i.e., Monica from Sect. 1 is a robot, and the sought-for program must navigate her from point A to point B (e.g., minimizing power consumption and time, while avoiding collisions and satisfying other safety guarantees). Closer to home for the CAV community, inside many tools for statically checking assertions in programs (such as SLAM or BLAST), the key component is one that creates an abstracted model of the program that is sufficiently precise to show that an assertion violation is not possible. Among the artifacts that may need to be synthesized are inductive invariants, abstract transformers, function summaries, and interface specifications. Thus, we conclude by offering the following wider definition of synthesis, which connects this broader outlook with the semantics-based perspective that we have presented in this paper:

Synthesis is the task of discovering a syntactic object—selected from some formalism in which each syntactic object has a rigorously defined semantics—from an "intent" expressed in the form of some kind of constraint.

We believe that the issues discussed in this paper will be increasingly important if synthesis is to be applied successfully to the creation of artifacts that have semantics, but are not programs *per se*.

The generality of our framework can bridge the gap between the many applications of synthesis, and we hope that the community will engage in our work by modeling their synthesis problems in SemGuS, and by adapting their solvers to work with SemGuS. Such contributions will result in new benchmarks and solvers, contributing to the programmability and effectiveness of SemGuS.

*Standardization and Competitions.* We believe that the idea of a programmable synthesis framework, and SemGuS, the start of such a framework, represents a step forward in program synthesis. Similarly to what happened with SyGuS, SemGuS must be standardized, other researchers should build solvers for it, and these solvers should compete annually in SemGuS competitions.

We hope that this paper will encourage readers to experiment with and advance the ideas presented here, in three ways: First, we hope that the generality of the framework will make it easy for people to use it on various problems, which in turn will make it easy to collect large and diverse sets of benchmarks that will make the design of new solvers focused and effective. Second, we hope that researchers will build new algorithms and techniques that are general and can solve problems built in this framework. Third, we hope to soon create a yearly competition that will foster further interest in building general synthesizers for our framework. More than anything, this paper is a call-to-arms—an invitation to help broaden the scope and abilities of program synthesis, toward an era where Monica uses synthesizers just as much as Python during her daily work.

Acknowledgments. Supported, in part, by a gift from Rajiv and Ritu Batra; by multiple Facebook Research Awards; by a Microsoft Faculty Fellowship; by NSF under grants 1420866, 1763871, and 1750965; by ONR under grants N00014-17-1-2889 and N00014-19-1-2318; and a grant from the Korea Foundation for Advanced Studies. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the authors, and do not necessarily reflect the views of the sponsoring entities.

### References


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Deductive Synthesis of Programs with Pointers: Techniques, Challenges, Opportunities (Invited Paper)**

Shachar Itzhaky<sup>1</sup>, Hila Peleg<sup>1</sup>, Nadia Polikarpova2(B), Reuben N. S. Rowe<sup>3</sup>, and Ilya Sergey4,5

> <sup>1</sup> Technion, Haifa, Israel {shachari,hilap}@cs.technion.ac.il <sup>2</sup> University of California, San Diego, USA npolikarpova@eng.ucsd.edu <sup>3</sup> Royal Holloway, University of London, Egham, UK reuben.rowe@rhul.ac.uk <sup>4</sup> Yale-NUS College, Singapore, Singapore ilya.sergey@yale-nus.edu.sg <sup>5</sup> National University of Singapore, Singapore, Singapore

**Abstract.** This paper presents the main ideas behind deductive synthesis of heap-manipulating program and outlines present challenges faced by this approach as well as future opportunities for its applications.

#### **1 Introduction**

Just like a journey of a thousand miles begins with a single step, an implementation of a working operating system, cryptographic library, or a compiler begins with writing a single function. This is not quite so for verified software, whose development starts with three "steps": a function *specification* (or, *spec*), followed by its *implementation*, and then by a *proof* that the implementation satisfies the spec. Although recent years have seen an explosion of increasingly diverse and sophisticated verified systems [14,20,26,31,41,48,71,73,96], their cost remains high, owing to the effort required to write formal specifications and proofs in addition to writing the code.

The good news is that in many cases the aforementioned three steps can be replaced by just one of them: writing the spec. The rest can be delegated to *deductive program synthesis* [52]—an emerging approach to automated software development, which takes as input a specifications, and searches for a corresponding program *together with its proof*.

Past approaches to deductive synthesis typically avoided low-level programs with pointers [43,69,83], which are notoriously difficult to reason about, making these approaches inapplicable to automating the development of verified systems code. The few techniques that did handle the heap [47,72] had significant limitations in terms of expressiveness and/or efficiency. Our prior work on the SuSLik

A. Silva and K. R. M. Leino (Eds.) CAV 2021, LNCS 12759, pp. 110–134, 2021. https://doi.org/10.1007/978-3-030-81685-8\_5

synthesizer [70], has introduced an alternative approach to synthesis of pointermanipulating programs, whose key enabling component is the use of Separation Logic (SL) [66,75] as the specification formalism. Due to its proof scalability, Separation Logic enabled modular verification of low-level imperative code and has been implemented in a large number of automated and interactive program verifiers [4,7,18,37,57,62,64,68]. The main novelty of SuSLik was an observation that the structure of SL specifications can be used to efficiently guide the search for a program and its proof. Since then, our follow-up work has explored automatic discovery of recursive auxiliary functions [34], generating independently checkable proof certificates for synthesized programs [93], and giving the user more control over the synthesis using concise mutability annotations [19].

As an appetizer for SL-powered deductive program synthesis consider the problem of flattening a binary tree data structure into a doubly-linked list. Assume also that the programmer would prefer to perform this transformation *in-place*, without allocating new memory, which they conjecture is possible because the nodes of the two data structures have the same size (both are records with a payload and two pointers). With SuSLik, the programmer can describe this transformation using the following Hoare-style SL specification:

# *{*tree(x*, S*)*}* flatten (loc x) *{*dll(x*, y, S*)*}* (1)

Here the precondition asserts that initially x points to the root of a tree, whose contents are captured by a set S. The postcondition asserts that after the execution of flatten, the same location x is a head of a doubly-linked list, with the same elements S as the initial tree (y denotes the existentially quantified back-pointer of the list). The definitions of the two predicates, tree and dll, which constrain the symbolic heaps in the pre- and postcondition are standard for SL [75] and will be shown in Sect. 2.

Given the spec (1), SuSLik takes less than 20 s to generate the program in Fig. 1, written in a core C-like language, as well as a formal proof that the program satisfies the spec. Several things are noteworthy about this program. First, the code indeed does not perform any allocation, and instead accomplishes its goal by switching pointers (in lines 17, 18, 23, and 25); this makes it economical in terms of memory usage as only a low-level program can be: similar code written in a functional language like OCaml would inevitably rely on garbage collection. Second, the main function flatten relies on

```
1 flatten(loc x) {
2 if (x == 0) {
3 } else {
4 let l = *(x + 1);
5 let r = *(x + 2);
6 flatten(l);
7 flatten(r);
8 helper (r, l, x);
9 }
10 }
11
12 helper(loc r, loc l,
13 loc x) {
14 if (l == 0) {
15 if (r == 0) {
16 } else {
17 *(r + 2) = x;
18 *(x + 1) = r;
19 }
20 } else {
21 let v = *l;
22 let w = *(l + 1);
23 *(l + 2) = r;
24 helper(r, w, l);
25 *(l + 2) = x;
26 }
27 }
```
**Fig. 1.** Flattening a tree into a DLL.

an auxiliary recursive function helper, which the programmer did not anticipate; in fact the need for this auxiliary—and its specification—is discovered by SuSLik completely automatically. All the programmer has to do to obtain a provably correct implementation of flatten is to write the spec (1) and define the two SL predicates it uses, which are, however, reusable across different programs.

At this point, a critical reader might be wondering whether this technology is mature enough to move past hand-crafted benchmarks and assist them in developing the next CompCert [48] or CertiKOS [31]. For one, the program in Fig. 1 does not seem optimal: a closer look reveals that the role of helper is to concatenate the lists obtained by flattening the two subtrees, resulting in the overall O(n<sup>2</sup>) complexity wrt. the size of the original tree.<sup>1</sup> Apart from performance of synthesized programs, the reader might have the following concerns:


The goal of this manuscript is precisely to illustrate the remaining challenges in SL-based synthesis of heap-manipulating programs and outline some future research directions towards addressing these challenges. In the remainder of this paper we provide the necessary background and a survey of the results to date (Sect. 2); we then zoom in on the promising techniques for improving proof search (Sect. 3); in Sect. 4 we discuss the completeness of synthesis, outlining the work that needs to be done in order to formally characterize the class of programs that can and cannot be generated; in Sect. 5 we talk about possible extensions to the synthesis procedure for improving the quality of synthesized programs; finally, in Sect. 6 we discuss possible applications of SL-based synthesis, such as program repair, data migration, and concurrent programming.

#### **2 State of the Art**

#### **2.1 Specifications**

SuSLik takes as input a Hoare-style specification, i.e., a pair of a pre- and a postcondition. Consider, for example, a specification for a function swap, <sup>2</sup> which swaps the values of two pointers:

$$\{\mathbf{x} \mapsto a \ast \mathbf{y} \mapsto b\} \text{ אוןa\n\n(1oc\ x,\ 1oc\ y)}\tag{2} \\ \{\mathbf{x} \mapsto b \ast \mathbf{y} \mapsto a\} \tag{2}$$

The precondition x -→ a ∗ y -→ b states that the relevant part of the heap contains two memory locations, x and y, which store values a and b, respectively. We also know that and x = y, because the semantics of separating conjunction (∗) require that the two heaps it connects be disjoint. The postcondition x -→ b ∗ y -→ a demands that after

<sup>1</sup> In Sect. 4 we show what it takes to derive an alternative, linear-time solution.

<sup>2</sup> Our language has no return statement, hence all functions have return type void, which is omitted from the spec; return values are emulated by writing to the heap.

executing the function, the values stored in x and y be swapped. This specification also implicitly guarantees that swap always terminates and executes without memory errors (e.g., null-pointer dereferencing). Note that x and y also appear as parameters to swap, and hence are program variables, i.e., can be mentioned in the synthesized program; the payloads a and b, on the other hand, are logical variables, implicitly universally quantified, and must not appear in the program. In the rest of this paper, we distinguish program variables from logical variables by using monotype font for the former.

In general, in a specification {P} f(...) {Q}, assertions P, Q both have the form φ; P, where the spatial part P describes the shape of the heap, while the pure part φ is a plain first-order formula that states the relations between variables (in (2) the pure part in both pre- and postcondition is trivially true, and hence omitted). For the spatial part, SuSLik employs the standard symbolic heap fragment of Separation Logic [66,75]. Informally, a symbolic heap is a set of atomic formulas called heaplets joined with separating conjunction (∗). The simplest kind of heaplet is a points-to assertion x -→ e, describing a single memory location with address x and payload e. An empty symbolic heap is represented with emp.

To capture linked data structures, such as lists and trees, SuSLik specifications use inductive heap predicates, which are standard in Separation Logic. For instance, the tree predicate used in (1) is inductively defined as follows:

$$\begin{aligned} \mathsf{tree}\langle x, S \rangle \triangleq x = 0 &\Rightarrow \{ S = \emptyset; \mathsf{emp} \} \\ \mid \ x \neq 0 &\Rightarrow \{ S = \{ v \} \cup S \cup S\_{r}; \\ \mid x, 3 \mid \*x &\mapsto v \ast \langle x, 1 \rangle \mapsto l \ast \langle x, 2 \rangle \longmapsto r \ast \mathsf{tree}\langle l, S\_{l} \rangle \ast \mathsf{tree}\langle r, S\_{r} \rangle \} \end{aligned} \tag{3}$$

The predicate is parametrized by the root pointer x and the set of tree elements S. This definition consists of two guarded clauses: the first one describes the empty tree (and applies when the root pointer is null), and the second one describes a non-empty tree. In the second clause, a tree node is represented by a three-element record starting at address x. Records are represented using a generalized form of the points-to assertion with an offset: for example, the heaplet x, 1 -→ l describes a memory location at the address x + 1. The block assertion [x, 3] is an artifact of C-style memory management: it represents a memory block of three elements at address x that has been dynamically allocated by malloc (and hence can be de-allocated by free). The first field of the record stores the payload v, while the other two store the addresses l and r of the left and right subtrees, respectively. The two disjoint heaps tree(l, Sl) and tree(r, Sr) store the two subtrees. The pure part of the second clause indicates that the payload of the whole tree consists of v and the subtree payloads, S<sup>l</sup> and Sr.

#### **2.2 The Basics of Deductive Synthesis**

The formal underpinning of SuSLik is a deductive inference system called Synthetic Separation Logic (SSL). Given a pre-/postcondition pair P, Q, deductive synthesis proceeds by constructing a derivation of the SSL synthesis judgment, denoted {P}-{Q} | c, for some program c. In this derivation, c is the output program, constructed while searching for the proof of the synthesis goal {P} - {Q}. Intuitively, the output program c should satisfy the Hoare triple {P} c {Q}. The derivation is constructed by applying inference rules, a subset of which is presented in Fig. 2, and every inference rule "emits" a program fragment corresponding to this deduction.

**Fig. 2.** Selected SSL rules (simplified). **Fig. 3.** Derivation of swap.

Figure 3 shows an SSL derivation for swap, using inference rules of Fig. 2. The derivation, read bottom-up, starts with the pre/post pair from (2) as the synthesis goal; each rule application simplifies the goal until both the pre- and the post-heap are empty, and might also prepend a statement (highlighted in grey) to the output program. In the initial goal, the Read rule can be applied to the heaplet <sup>x</sup> -→ a to read the logical variable a from location x into a fresh program variable a1; the second application of Read similarly reads from the location y. At this point, the Write rule is applicable to the post-heaplet x -→ b1 because its right-hand side only mentions program variables and can be directly written into the location x; note that this rule equalizes the corresponding heaplets in the pre- and post-condition. After two applications of Write, the pre- and the post-heap become equal and can be simply cancelled out by the Frame rule, leaving emp on either side of the goal; the terminal rule Emp then concludes the derivation. Although very simple, this example demonstrates the secret behind SuSLik's efficiency: the shape of the specification restricts the set of applicable rules and thereby guides program synthesis.

#### **2.3 Synthesis with Recursion and Auxiliary Functions**

We now return to our introductory example—flattening a binary tree into a doublylinked list—whose specification (1) we repeat here for convenience:

{tree(x, S)} flatten(loc x) {dll(x, y, S)}

The definition of the tree predicate has been shown above (3); the predicate dll(x, y, S) describes a doubly-linked list rooted at x with back-pointer y and payload set S:

$$\begin{aligned} \mathsf{dll}\{x, y, S\} \triangleq x = 0 &\Rightarrow \{S = \emptyset; \mathsf{emp}\} \\ \mid \ x \neq 0 &\Rightarrow \{S = \{v\} \cup S'; \\ \mid x, 3 \mid \*x &\mapsto v\*\langle x, 1\rangle \mapsto n\*\langle x, 2\rangle \mapsto y\*\mathsf{dll}\langle n, x, S'\rangle\} \end{aligned} \tag{4}$$

Note that in the spec (1) both the set S and the back-pointer y are logical variables, but S is implicitly universally quantified (a so-called ghost variable), because it occurs in the precondition, while y is existentially quantified (a so-called existential variable), because it only occurs in the postcondition. The reader might be wondering why use an existential here instead of a null pointer: as we show below, such weakening is required

**Fig. 4.** Intermediate synthesis state when deriving flatten.

to obtain the solution in Fig. 1; we discuss the alternative spec and corresponding solution in Sect. 4.

At a high level, the synthesis of flatten proceeds by eagerly making recursive calls on the left and the right sub-trees, l and r, as illustrated in Fig. 4, which leads to the following synthesis goal:

$$\begin{array}{c} \{ [\mathbf{x}, 3] \ast \mathbf{x} \longmapsto v \ast \langle \mathbf{x}, 1 \rangle \longmapsto 1 \ast \langle \mathbf{x}, 2 \rangle \longmapsto r \ast \mathsf{call}(\mathbf{1}, y\_r, S\_l) \ast \mathsf{all}(\mathbf{r}, y\_r, S\_r) \} \\ \qquad \leadsto \{ \mathsf{ell} \mathsf{l}(\mathbf{x}, y, \{v\} \cup S\_l \cup S\_r) \} \end{array} \tag{5}$$

Now the synthesizer must concatenate the two doubly-linked lists, rooted at l and r, together with the parent node x into a single list. Since the spec gives us no access to the last element of either of the two lists, this concatenation requires introducing a recursive auxiliary function to traverse one of the lists to the end. We now demonstrate how SuSLik synthesizes recursive calls and discovers the auxiliary using a single mechanism we call cyclic program synthesis [34], inspired by cyclic proofs in Separation Logic [11, 76]. The main idea behind cyclic proofs is that, in addition to reaching a terminal rule like Emp, a sub-goal can be "closed off" by forming a cycle to an identical companion goal earlier in the derivation; in SSL these cycles give rise to recursive calls.

Figure 5 depicts a cyclic derivation of flatten. For now let us ignore the applications of the Proc rule, which do not modify the synthesis goal; their purpose will become clear shortly. Given the initial goal (1), SuSLik first applies the Open rule, which unfolds the definition of tree in the precondition and emits a conditional with one branch per clause of the predicate. The first branch (x = 0) is trivially solved by skip, since a null pointer is both an empty tree and an empty list. The second branch is shown in Fig. 5: its precondition contains two predicate instances tree(l, Sl) and tree(r, Sr) for the two sub-trees of x.

Now SuSLik detects that either of those instances can be unified with the precondition tree(x, S) of the top-level goal, so it fires the Call rule, which uses cyclic reasoning to synthesize recursive calls. More specifically, Call has two premises: the first one synthesizes a recursive call and the second one the rest of the program after the call. The spec of the first premise must be identical to some earlier goal, so that it can be closed off by forming a cycle; in our example, the back-link (1) connects the first premise back to the top-level goal. Once a companion goal is identified, SuSLik inserts an application of Proc right above it: its purpose is to delineate procedure boundaries, or, in other words, give a name to the piece of code that the Call rule is trying to call. To ensure that recursion is terminating, we must prove that tree(l, Sl) in the precondition of the Call's premise is strictly smaller than tree(x, S) in the pre-

**Fig. 5.** Derivation of flatten and its recursive auxiliary helper.

condition of the companion (see [34] for more details about our termination checking mechanism).

After the second application of Call (to tree(r, Sr)), SuSLik arrives at the goal (5), with two lists in the precondition (marked (a) in Fig. 5). Ignoring again the application of Proc, which will be inserted later, SuSLik proceeds by unfolding the list dll(l, yl, Sl) via Open, eventually arriving at the goal (b): this goal again has two lists in the precondition but one of them is now smaller (it is the tail of dll(l, yl, Sl)). At this point Call detects that (a sub-heap of) goal (b) can be unified,<sup>3</sup> with goal (a) thus forming the cycle (3), which this time links to an internal goal in the derivation instead of the top-level goal. As before SuSLik inserts an application of the Proc rule just above the companion goal (a), thereby abducing an auxiliary procedure with a fresh name.

#### **2.4 Implementation and Empirical Results**

The most up-to-date implementation of SuSLik is publicly available at:

https://github.com/TyGuS/suslik

Table 1 collects the results of running SuSLik on benchmarks from our prior work [19, 34,70,93] as well as seven new benchmarks, which we added to illustrate various challenges discussed in subsequent sections.<sup>4</sup> Most existing benchmarks had been adapted from the literature on verification and synthesis [24,47,50,72]. In addition to standard textbook data structures, our benchmarks include operations on two less common data structures, which to the best of our knowledge cannot be handled by other synthesizers.

<sup>3</sup> This is where we rely on the existential back-pointer in (1): if we replace y<sup>l</sup> with 0, then dll(l, 0, Sl) and dll(w, yw, Sw) would not unify.

<sup>4</sup> The code and benchmarks accompanying this paper are available online [35].

**Table 1.** SuSLik benchmarks and results. We report the number of Procedures generated, total number Stmt of statements in those procedures, the ratio Code/Spec of code to specification (in AST nodes), and the synthesis time in seconds for standard SuSLik (Time), with a simpler cost function (TimeSC) and with no bounds on predicate unfolding and calls (TimeNB). "-" denotes timeout after 30 minutes. Footnotes indicate the sources of benchmarks.



**Table 1.** (continued)

A rose tree [51] is a variable-arity tree, where child nodes are stored in a linked list; it is described in SL by two mutually recursive predicates (rtree for the tree and children for the list of children), and our synthesized operations on rose trees are also mutually recursive. A packed tree is a binary tree serialized into an array; it is interesting because operations on packed trees use non-trivial pointer arithmetic (we discuss them in Sect. 6).

Apart from the size of each program (in statements), we also report the ratio of code size to spec size (both in AST nodes) as a measure of synthesis utility. For the majority of the benchmarks the generated code is larger than the specification, sometimes significantly (up to 12x); the only exceptions are benchmarks with very convoluted specs, such as BST rotations (benchmarks 43 and 44), or extremely simple programs, such swap from Fig. 3 (benchmark 1) and prepending an element to a sorted list (benchmark 19).

A number of benchmarks generate more than one procedure: those programs require recursive auxiliaries [34], such as our running example flatten from Fig. 1 (benchmark 40). It is worth mentioning that benchmarks 37 through 41 encode different versions of flattening a binary tree into a singly or doubly-linked list: 37 and 38 are simplified versions that do not require discovering auxiliaries because they contain additional hints from the user (a library function for appending lists in 37 and an inductive specification for flatten with a list accumulator in 38); 39 is similar to 40 but returns a singly-linked list (and hence requires allocation). Finally 41 is a version of 40 that uses 0 instead of y as the back-pointer of the output list; this precludes SuSLik from generating an auxiliary for appending two lists, and instead it discovers a slightly more complex, but linear-time solution, which we discuss in Sect. 4.

The missing synthesis times for some benchmarks indicate that they could not be synthesized automatically after 30 min, but were possible to solve in an "interactive" mode, where the search has been given hints on how to proceed in the case of multiple choices. We elaborate on the possibility of generating those programs automatically in subsequent sections. Apart from regular SuSLik time we also report time for two variations discussed in Sect. 3.

#### **3 Proof Search**

Similarly to existing deductive program synthesizers [43], SuSLik adopts best-first And/Or search [54] to search for a program derivation. The search space is represented as a tree with two types of nodes. An Or-node corresponds to a synthesis goal, whose children are alternative derivations, any of which is sufficient to solve the goal. An And-node corresponds to a rule application, whose children are premises, all of which need to be solved in order to build a derivation. Each goal has a cost, which is meant to estimate how difficult it is to solve. The search works by maintaining a worklist of Or-nodes that are yet to be explored. In each iteration, the node with the least cost is dequeued and expanded by applying all rules enabled according to a proof strategy; the node's children are then added back to the worklist.

The proof strategy and the cost function are crucial to the performance of the proof search. In current SuSLik implementation both are ad-hoc and brittle; in the rest of the section we outline possible improvements to their design.

#### **3.1 Pruning via Proof Strategies**

A proof strategy is a function that takes in a synthesis goal and its ancestors in the search tree, and returns a list of rules enabled to expand that goal. Without strategies, the branching factor of the search would be impractically large. SuSLik's strategies are based on the observation that some orders of rule applications are redundant, and hence can be eliminated from consideration without loss of completeness. Identifying redundant orders is non-trivial and is currently done informally, increasing the risk of introducing incomplete strategies.

For example, SuSLik's proof strategy precludes applying Call if Close (a rule that unfolds a predicate in the postcondition) has been applied earlier in the derivation. The reasoning is that Call only operates on the precondition, while Close only operates on the postcondition, hence the two rule applications must be independent, and can always be reordered so that Call is applied first. But it gets more complicated once we let Call abduce auxiliaries: now applying Call after Close could be useful to give it access to more companion goals, whose postconditions differ from that of the top-level goal. Consider for example copying a rose tree with the following spec:

{r -<sup>→</sup> <sup>x</sup> <sup>∗</sup> rtree(x, S)} void rtcopy(loc r) {<sup>r</sup> -<sup>→</sup> <sup>y</sup> <sup>∗</sup> rtree(y, S) <sup>∗</sup> rtree(x, S)} (6) Copying a rose tree seems to require two mutually-recursive procedures: the main one (6) that copies an rtree and an auxiliary one that copies the list of its children, and hence has children instead of rtree in its postcondition. To our surprise, however, our proof strategy does not preclude the derivation of rtcopy (see benchmark 52 in Table 1): in this derivation, the auxiliary returns two rtrees, which are then unfolded after the call to extract the relevant children.

*Future Directions.* To develop more principled yet efficient strategies, we need to turn to the proof theory community, which has accumulated a rich body of work on efficient proof search. One technique of particular interest—focusing [53]—defines a canonical representation of proofs in linear logic [29] (more precisely, a canonical ordering on the application of proof rules, which can be enforced during the search by tracing local properties). Existing program synthesis work [27,79] has leveraged ideas from focusing, but only in the setting of type inhabitation for pure lambda calculi. SuSLik takes advantage of some of these ideas, too: it designates some rules, such as Read and logical normalization rules, to be invertible; these rules can be applied eagerly and need not be backtracked. Beyond focusing, we might explore the applicability of more advanced canonical representations of programs and proofs [1,33,79]. We believe that these techniques will help us formalize and leverage inherent SSL symmetries, such as that two programs operating on disjoint parts of the heap can be executed in any order.

#### **3.2 Prioritization via a Cost Function**

When selecting the next goal to expand, SuSLik's best-first search relies on a heuristic cost function of the form (with p, w > 1):

$$\begin{array}{ccc} \mathsf{cost}(\{\phi, P\} \sim \{\psi, Q\}) = p \ast \mathsf{cost}(P) + \mathsf{cost}(Q) & \mathsf{cost}(\mathfrak{p}(\overline{e})^{u,c}) = w \ast (1 + u + c) \\ \mathsf{cost}(P \ast Q) = \mathsf{cost}(P) + \mathsf{cost}(Q) & \mathsf{cost}(\mathfrak{.}) = 1 \end{array}$$

In other words, a cost of a synthesis goal is a (weighted) total cost of all heaplets in its pre- and postcondition. The intuition is that the synthesizer needs to eliminate all these heaplets in order to apply the terminal Emp rule, so each heaplet contributes to the goal being "harder to solve". Predicates are more expensive than other heaplets, because they can be unfolded and produce more heaplets. In addition, for each predicate instance p(e) u,c SuSLik keeps track of the number of times it has been unfolded (u) or has gone through a call (c); factoring this into the cost prevents the search from getting stuck in an infinite chain of unfolding or calls. Finally, it can be useful to give a higher weight to the heaplets in the precondition, because many rules that create expensive search branches (most notably Call) operate on the precondition.

Our implementation currently uses p = 3, w = 2, which is a result of manual tuning. Column TimeSC in Table 1 shows how synthesis times change if we set p = 1. As you can see, SuSLik's performance is quite sensitive even to this small change: four benchmarks, which originally took under 30 s, now time out after 30 minutes, while benchmark 24, on the contrary, is solved five times faster. These results suggest that different synthesis tasks benefit from different search parameters, and that we might need a mechanism to tune SuSLik's search strategy for a given synthesis task.

In addition, because the cost heuristic is not efficient enough at guiding the search, we introduce hard bounds on the number of unfoldings and calls u and c for a predicate instance. Column TimeNB in Table 1 shows the results of running SuSLik without these bounds: as you can see, 19 benchmarks time out (compared to only two in the original setup). The requirement to guess sufficient bounds for each benchmark hampers the usability of SuSLik, hence in the future we would like to replace them with a better cost function.

*Future Directions.* To guide the search in a more intelligent and flexible way, we turn to extensive recent work on using learned models to guide proof search [8,28,49,78,95] and program synthesis [5,15,39,46,55,82]. Guiding deductive synthesis would most likely require a non-trivial combination of these two lines of work.

In the area of proof search, existing techniques are used to select the next strategy in a proof assistant script [59,60,78,95], or select a subset of clauses to use in a first-order resolutions proof [9,49]. Although these techniques are not directly applicable to our context, we can likely borrow some high-level insights, such as two-phased search [49], which applies a slow neural heuristic to make important decisions in early stages of search (e.g., which predicate instances to unfold), and then less accurate but much faster hand-coded heuristics take over. Among the many techniques for guiding program synthesis, neural-guided deductive search (NGDS) [39] might be the natural place to start, since it shows how to condition the next synthesis step on the current synthesis sub-goal.

At the same time we also expect the limited size of the available dataset (i.e., the benchmarks from Table 1) would hamper the application of deep learning to SuSLik. An alternative approach is to encode feature extractors [58] and apply machine learning algorithms to the result of such feature extractors. Another approach is to learn a coarse-grained model from available data and then adjust it during search, based on the feedback from incomplete derivations, as in [6,15,82].

#### **4 Completeness**

Soundness and completeness are desirable properties of synthesis algorithms. In our case, it is natural to formalize these properties relative to an underlying verification logic, which defines Hoare triples {P} c {Q}, with the total correctness interpretation "starting from a state satisfying P, program c will execute without memory errors and terminate in a state satisfying Q". This logic can be defined in the style of Smallfoot [7], using a combination of symbolic execution rules and logical rules, with the addition of cyclic proofs to handle recursion [76].

Relative soundness means that any solution SuSLik finds can be verified: ∀P, Q, c. P -Q | c ⇒ {P} c {Q}. Relative completeness means that whenever there exists a verifiable program, SuSLik can find one: <sup>∀</sup>P, Q.(∃c.{P} <sup>c</sup> {Q}) <sup>⇒</sup> (∃c- . P -Q | c- ). Proving relative soundness is rather straightforward, because SSL rules are essentially more restrictive versions of verification rules, hence an SSL derivation can be rewritten by translating every P -Q | c into {P} c {Q}. <sup>5</sup> Completeness on the other hand is quite tricky, exactly because SSL rules impose more restrictions on the pre- and postconditions, in order to avoid blind enumeration of programs and instead guide synthesis by the spec. In the rest of this section we look into two major sources of relative incompleteness of SSL: recursive auxiliaries and pure reasoning.

<sup>5</sup> In our recent work we have developed an automatic translation from SSL derivations into three Coq-based verification logics [93].

#### **4.1 Recursive Auxiliaries**

A common assumption and source of incompleteness in recursive program synthesis [43,67,69] is that (1) synthesis is performed one function f at a time: if auxiliaries are required, their specifications are supplied explicitly; and (2) the specification Φ of f is inductive: one can prove that Φ holds of f's body assuming it holds of each recursive call. This restriction hampers the usability of synthesizers, because the user must guess all required auxiliaries and possibly generalize Φ to make it inductive, which in most cases requires knowing the implementation of f. As we have shown in Sect. 2, SuSLik mitigates these limitations to some extent, as it is able to discover auxiliary functions, such as helper in Fig. 1, automatically. To make the search tractable, however, cyclic synthesis restricts the space of auxiliary specifications considered by SuSLik to synthesis goals observed earlier in the derivation. Although this restriction is easy to state, we still do not have a formal characterization (or even a firm intuitive understanding) of the class of auxiliaries that SSL fundamentally can and cannot derive. Below we illustrate the intricacies on a series of examples.

```
1 intersect (loc r, y)
2 {
3 let x = *r;
4 if (x == 0) {
5 } else {
6 let v = *x;
7 let n = *(x + 1);
8 *r = n;
9 intersect(r, y);
10 insert(v, x, r, y);
11 }
12 }
                               13 insert(int v, loc x, r, y) {
                               14 let z = *r;
                               15 if (y == 0) { free(x); }
                               16 else {
                               17 let vy = *y;
                               18 let n = *(y + 1);
                               19 if (v == vy) {
                               20 *(x + 1) = z;
                               21 *r = x;
                               22 } else {
                               23 insert(v, x, r, n);
                               24 }}}
```
**Fig. 6.** Intersection of lists with unique elements. This implementation cannot be synthesized from (7), but a slight modification of it can, as explained in the text.

*Generalizing Pure Specs.* One reason SuSLik might fail to abduce an auxiliary is that the pure part of the companion's goal might be too specific for the recursive call. Let us illustrate this phenomenon using the list intersection problem (benchmark 16 in Table 1) with the following specification, where ulist denotes a singly-linked list with unique elements:

$$\{\mathbf{r} \mapsto x \ast \text{ulist}(x, S\_x) \ast \text{ulist}(\mathbf{y}, S\_y)\} \sim \{\mathbf{r} \mapsto z \ast \text{ulist}(z, S\_x \cap S\_y) \ast \text{ulist}(\mathbf{y}, S\_y)\} \tag{7}$$

Given this specification, we expected SuSLik to generate the program shown in Fig. 6. To compute the intersection of two input lists rooted in x and y, this program first computes the intersection of y and the tail of x (line 9). The auxiliary insert then traverses y to check if it contains v (the head of x), and if so, inserts it into the intermediate result z (line 23), and otherwise, de-allocates the node x (line 15). This program, however, cannot be derived by SSL; to see why let us take a closer look at the synthesis goal after line 9, which serves as the spec for insert:

$$\begin{aligned} \{S\_x = \{\mathbf{v}\} \cup S\_1 \land \mathbf{v} \notin S\_1 \land S\_z = S\_1 \cap S\_y; \mathbf{r} \mapsto z \ast \text{ulist}(z, S\_z) \ast \text{ulist}(\mathbf{y}, S\_y) \; \ast \\ \mathbf{x} \mapsto \mathbf{v} \ast \dots \} \sim \{S'\_z = S\_x \cap S\_y; \mathbf{r} \mapsto z' \ast \text{ulist}(z', S'\_z) \ast \text{ulist}(\mathbf{y}, S\_y) \} \end{aligned} \tag{8}$$

The issue here is that the pure spec is too specific: the precondition S<sup>z</sup> = S<sup>1</sup> ∩ S<sup>y</sup> and the postcondition S- <sup>z</sup> = S<sup>x</sup> ∩ S<sup>y</sup> define the behavior of this function in terms of the elements of input lists x and y, but the recursive call in line 23 replaces y with its tail n so these specifications do not hold anymore. The solution is to generalize the pure part of spec (8), so that it does not refer to Sx:

$$\begin{aligned} \{\mathbf{v}\notin S\_z; \mathbf{r} &\longmapsto z\*\text{ulist}(z, S\_z)\*\text{ulist}(\mathbf{y}, S\_y)\*\mathbf{x} \longmapsto \mathbf{v}\*\dots\} \\ &\sim \{S'\_z = S\_z \cup (\{\mathbf{v}\} \cap S\_y); \mathbf{r} \longmapsto z'\*\text{ulist}(z', S'\_z)\*\text{ulist}(\mathbf{y}, S\_y)\} \end{aligned} \tag{9}$$

Alas, such a transformation of the pure spec is beyond SuSLik's capabilities.

To our surprise, SuSLik was nevertheless able to generate a solution to this problem by finding an alternative implementation for insert, shown on the right. This implementation appends v to z instead of prepending it; more specifically, insert starts by traversing z, and once it reaches the base case, it calls another auxiliary, intersectOne (omitted for brevity), which traverses y and returns a list whose elements are {v} ∩ S<sup>y</sup> (i.e., a list with at most one element), which is then appended to the intersection. At a first glance it is unclear how this superfluous traversal of z can possibly help with generalizing the spec (8); the key to this mystery lies in the recursive call in line 22: note that as the second parameter, instead of the input list x, it actually uses z after replacing its head element

```
13 insert(int v, loc x,
      r, y) {
14 let z = *r;
15 if (z == 0) {
16 intersectOne(v,
         x, r, y)
17 } else {
18 let vz = *z;
19 let n = *(z + 1);
20 *r = n;
21 *z = v;
22 insert(v, z, r,
         y);
23 ...
24 }}
```
with v! This substitution makes the overly restrictive spec of (8) actually hold.

Of course this implementation is overly convoluted and inefficient, so in the future we plan equip SuSLik with the capability to generalize pure specs. To this end, we plan to combine deductive synthesis with invariant inference techniques via biabduction [86]. For instance, whenever the Call rule identifies a companion goal, we can replace its pure pre- and post-condition φ and ψ with unknown predicates U<sup>φ</sup> and Uψ. During synthesis, we would maintain a set of Constrained Horn Clauses over these unknown predicates (starting with: φ ⇒ U<sup>φ</sup> and U<sup>ψ</sup> ⇒ ψ); these constraints can be solved incrementally, like in our prior work [69], pruning the current derivation whenever the constraints have no solution. If synthesis succeeds, the assignment to U<sup>φ</sup> and U<sup>ψ</sup> corresponds to the inductive generalization of the original auxiliary spec. Since only the pure part of the spec is generalized, the spatial part can still be used to guide synthesis.

*Accumulator Parameters.* It is common practice to introduce an auxiliary recursive function to thread through additional data in the form of "accumulator" inputs or outputs. Cyclic program synthesis has trouble conjuring up arbitrary accumulators, since it constructs auxiliary specifications from the original specification via unfolding and making recursive calls.

Consider linked list reversal (23 in Table 1): SuSLik generates an inefficient, quadratic version of this program, which reverses the tail of the list and then appends its head to the result (hence discovering "append element" as the auxiliary). The canonical linear-time version of reversal requires an auxiliary with two list arguments—the already reversed portion and the portion yet to be reversed—and hence is outside of SuSLik's search space: cyclic synthesis cannot encounter a precondition with two lists, as it starts with a single list predicate in the precondition, and neither unfolding nor making a call can duplicate it.


**Fig. 7.** Flattening a tree into a DLL in linear time.

There are examples, however, where SuSLik surprized us by inventing the necessary accumulator parameters. Consider again our running example, flattening a tree into a doubly-linked list. Recall that given the spec (1), SuSLik synthesizes an inefficient implementation with quadratic complexity. A canonical linear-time solution requires an auxiliary that takes as input a tree and a list accumulator, and simply prepends every traversed tree element to this list; because of the accumulator parameter, discovering this auxiliary seems to be outside of scope of cyclic synthesis. To our surprise, SuSLik is actually able to synthesize a linear-time version of flatten, shown in Fig. 7 (and encoded as benchmark 41 in Table 1), given the following specification:

$$\{\text{tree}(\mathbf{x}, S)\} \text{ \textbf{1}\textbf{at}\textbf{ten} } \begin{Bmatrix} \text{loc } \mathbf{x} \end{Bmatrix} \begin{Bmatrix} \text{d}\mathbb{I}(\mathbf{x}, 0, S) \end{Bmatrix} \tag{10}$$

Compared with (1), the existential back-pointer y of the output list is replaced with the null-pointer 0, precluding SuSLik from traversing the output of the recursive call (cf. Sect. 2), which in this case comes in handy, since it enforces that every tree element is traversed only once.

The new solution starts the same way as the old one, by flattening the left sub-tree l, which leads to the following synthesis goal after line 6:

$$\{\mathsf{dll}(\mathbf{1},0,S\mathbf{i}) \ast \mathsf{tree}(\mathbf{r},S\_{\mathbf{r}}) \ast [\mathbf{x},3] \ast \mathbf{x} \mapsto v \ast \dots \} \sim \{\mathsf{dll}(\mathbf{x},0,\{v\} \cup S\mathbf{i} \cup S\_{\mathbf{r}})\}\tag{11}$$

As you can see, the precondition now contains a tree and a list! Since it cannot recurse on the list dll(l, 0, Sl), the synthesizer instead proceeds to unfold the tree tree(r, Sr) and then use (11) as a companion for two recursive calls on r's sub-trees, turning (11) into a specification for helper in Fig. 7.

#### **4.2 Pure Reasoning**

To enable synthesis of the wide range of programs demonstrated in Sect. 2, SuSLik must support a sufficiently rich logic of pure formulas. Our implementation currently supports linear integer arithmetic and sets, but the general idea is to make SuSLik parametric wrt. the pure logic (as long as it can be translated into an SMT-decidable theory), and outsource all pure reasoning to an SMT solver.

In the context of synthesis, however, outsourcing pure reasoning is trickier than it might seem (or at least trickier than in the context of verification). Consider the following seemingly trivial goal:

$$\{\mathbf{x} \mapsto a + 10\} \sim \{\mathbf{x} \mapsto a + 11\} \tag{12}$$

This goal can be solved by incrementing the value stored in x, i.e., by the program let a1 = \*x; \*x = a1 + 1. Verifying this program is completely straightforward: a typical SL verifier would use symbolic execution to obtain the final symbolic state {x -→ a + 10 + 1}, reducing verification to a trivial SMT query ∃a.a + 10 + 1 = a + 11. Synthesizing this program, on the other hand, requires guessing the program expression a1 + 1, which does not occur anywhere in the specification.

To avoid blind enumeration of program expressions, SuSLik attempts to reduce the goal (12) to a syntax-guided synthesis (SyGuS) query [2]:

$$\exists f. \forall x, a, a\_1. a\_1 = a + 10 \implies f(x, a\_1) = a + 11$$

Queries like this can be outsourced to numerous existing SyGuS solvers [3,32,46,77]; SuSLik uses CVC4 [74] for this purpose. Because SyGuS queries are expensive, the challenge is to design SSL rules to issue these queries sparingly.

**Fig. 8.** SSL derivation for goal (12).

Figure <sup>8</sup> shows how two pure reasoning rules, <sup>∃</sup>-Intro and Solve-∃, work together to solve the goal (12). <sup>∃</sup>-Intro is triggered by the postcondition heaplet <sup>x</sup> -→ a + 1, whose right-hand side is a ghost expression, which blocks the application of Write. <sup>∃</sup>-Intro replaces the ghost expression with a program-level existential variable <sup>y</sup> (i.e., an existential which can only be instantiated with program expressions). Now Solve-<sup>∃</sup> takes over: this rule constructs a SyGuS query using all existentials in the current goal as unknown terms and the pure pre- and post-condition as the SyGuS specification. In this case, the SyGuS query succeeds, replacing the existential y with the program term a1 + 1. From here on, the regular Write rule finishes the job.

Note that although the goal (12) is artificially simplified, it is extracted from a real problem: benchmark 32 in Table 1, length of a list of lists. In fact the versions of SuSLik reported in our previous work were incapable of solving this benchmark because they were lacking the <sup>∃</sup>-Intro rule, which we only introduced recently. Although the current combination of pure reasoning rules works well for all our benchmarks, it is still incomplete (even modulo the completeness of the pure synthesizer), because, for efficiency reasons, Solve-<sup>∃</sup> only returns a single solution to the SyGuS problem, even if the pure specification allows for many. This might be insufficient when Solve-<sup>∃</sup> is called before the complete pure postcondition is known (for example, to synthesize actual arguments for a call). Developing an approach to outsourcing pure reasoning that is both complete and efficient is an open challenge for future work.

#### **5 Quality of Synthesized Programs**

Should we hope that the output of deductive synthesis will be directly integrated into high-assurance software, we need to make sure that the code it generates is not only correct, but also efficient, concise, readable, and maintainable. The current implementation of SuSLik does not take any of these considerations into account during synthesis; in this section we discuss two of these challenges, and outline some directions towards addressing them.

#### **5.1 Performance**

We have already mentioned examples of SuSLik solutions with sub-optimal asymptotic complexity in Sect. 4: for example, SuSLik generates quadratic programs for linked list reversal and tree flattening instead of optimal linear-time versions. Although a lineartime solution to tree flattening from Fig. 7 is actually within SuSLik's search space (even with the more general spec (1)), SuSLik opts for the sub-optimal one simply because it has no ability to reason about performance and hence has no reason to prefer one over the other.

To enable SuSLik to pick the more efficient of the two implementations, we can integrate SSL with a resource logic, such as [56], following the recipe from our prior work on resource-guided synthesis [44]. One option is to annotate each points-to heaplet x -<sup>→</sup><sup>p</sup> <sup>e</sup> with non-negative potential <sup>p</sup>, which can be used to pay for execution of statements, according to a user-defined cost model. Predicate definitions can describe how potential is allocated inside the data structure; for example, we can define a tree with p units of potential per node as follows:

tree(x, S, p) <sup>x</sup> = 0 ⇒ {<sup>S</sup> <sup>=</sup> <sup>∅</sup>; emp} | x = 0 ⇒ {S = {v} ∪ S<sup>l</sup> ∪ Sr; [x, 3] ∗ x -<sup>→</sup><sup>p</sup> <sup>v</sup> <sup>∗</sup>x, <sup>1</sup> -→ l ∗ x, 2 -<sup>→</sup> <sup>r</sup> <sup>∗</sup> tree(l, Sl, <sup>p</sup>) <sup>∗</sup> tree(r, Sr, <sup>p</sup>)}

We can now annotate the specification (1) with potentials as follows:

$$\{\text{tree}(\mathbf{x}, S, 2)\} \text{ \*\*1atten\*\* } \{\text{loc } \mathbf{x}\} \text{ \*\*\{dll(\mathbf{x}, y, S, 0)\} \text{ }} \tag{13}$$

If we define the cost of a procedure call to be 1, and the cost of other statements to be 0, this specification guarantees that flatten only makes a number of recursive calls that is linear in the size of the tree (namely, two calls per tree element). With this specification, the inefficient solution in Fig. 1 does not verify: since helper traverses the list r, it must assign some positive potential to every element of this list in order to pay for the call in line 24, but the specification (13) assigns no potential to the output list. On the other hand, the efficient solution in Fig. 7 verifies: after the recursive call to flatten in line 6 we obtain {dll(l, y, Sl, 0) <sup>∗</sup> tree(r, Sr, 2) <sup>∗</sup> ...}; helper verifies against this specification since it only traverses the tree r and hence can use the two units of potential stored in its root to pay for the two calls in lines 21 and 24. In fact, the user need not guess the precise amount of potential p = 2 in the spec (13): any constant p ≥ 2 would work to reject the quadratic solution and admit the linear one.

#### **5.2 Readability**

Although readability is hard to quantify, we have noticed several patterns in SuS-Lik-generated code that are obviously unnatural to a human programmer, and hence need to be addressed. Perhaps the most interesting problem arises due to inference of recursive auxiliaries: because SuSLik has no notion of abstraction boundaries, the allocation of work between the different procedures is often sub-optimal. One example is benchmark 39 in Table 1, which flattens a binary tree into a singly-linked list. This example is discussed in detail in our prior work [34]; the solution is similar to flatten from Fig. 1, except that this transformation cannot be performed in-place: instead, the original tree nodes have to be deallocated, and new list nodes have to be allocated. Importantly, in SuSLik's solution, tree nodes are deallocated inside the helper function, whose main purpose is to append two lists. A better design would be to perform deallocation in the main function, so that helper has no knowledge of tree nodes whatsoever. To address this issue in the future we might consider different quality metrics when abducing specs for auxiliaries, such as encouraging all heaplets generated by unfolding the same predicate to be processed by the same procedure.

#### **6 Applications**

#### **6.1 Program Repair**

In our statement of the synthesis problem, complete programs are generated from scratch from Hoare-style specifications. But what if the program is already written previously but is buggy—would it be possible to automatically find a fix for it if we know what its specification is? This line of research, employing deductive synthesis for automated program repair [30], known as deductive program repair, has been explored in the past for functional programs [42] and simple memory safety properties [90], and only recently has been extended to heap-manipulating programs using the approach pioneered by SuSLik [63].

The SL-based deductive repair relies on existing automated deductive verifiers [17] to identify a buggy code fragment (which breaks the verification), followed by the discovery of the correct specification, which is used for the subsequent synthesis of the patch. The main shortcoming of the existing SL-based repair tools is the need to provide the top-level specs for the procedures in order to enable their verification (and potential bug discovery) in the first place. As a way to improve the utility of those tools, a promising direction is to employ existing static analyzers, such as Infer [12], to derive those specifications by abducing them from the usages of the corresponding functions [13].

#### **6.2 Data Migration and Serialization**

The pay-off of deductive synthesis is especially high for programs like tree flattening, which change the internal representation of a data structure without changing its payload; these programs usually have a simple specification, while their implementations can get much more intricate. One example where such programs can be useful is migration of persistent data: thanks to recent advancements in non-volatile memory (NVM) [40,45,84], large amounts of data are now persistently stored in memory, in arbitrary programmer-defined data structures. If the programmer decides to change the data structure, data has to be migrated between the old and the new representations, and writing those migration functions by hand can be tedious. In addition, reallocating large data structures is often prohibitively expensive, so the migration needs to be performed in-place, without reallocation. As we have demonstrated in our running example, this is something that can be easily specified and synthesized in SuSLik.

**Fig. 9.** (Left) Pointer-based and packed representations of the same binary tree. (Right) An SL predicate for packed trees.

Another real-world application of this kind of programs is data serialization and de-serialization, where data is transformed back and forth between a standard pointerbased representation and an array so that it can be written to disk or sent over the network [16,91]. For example, Fig. 9 shows a pointer-based full binary tree and its serialized (or packed) representation, where the nodes are laid out sequentially in preorder [92]. The right-hand-side of the figure shows an SL predicate ptree that describes packed trees: every node x starts with a tag that indicates whether it is a leaf; if x is not a leaf, its left child starts at the address x + 2 and its right child at x + 2 · (1 + nl), where n<sup>l</sup> is the size of the left child, which is typically unknown at the level of the program.

Imagine a programmer wants to synthesize functions that translate between these two representations, i.e., pack and unpack the tree. The most natural specification for unpack would be:

$$\{\mathbf{r} \mapsto x \ast \mathsf{packed}(x, sz, S)\} \mathtt{unpack\\_sim\'{p1oc\ }\mathbf{r}\} \begin{Bmatrix} \mathbf{r} \mapsto y \ast \mathsf{packed}(x, sz, S) \\ \ast \mathsf{tree}(y, sz, S) \end{Bmatrix} \tag{14}$$

This specification, however, cannot be implemented in SSL: when x is an internal node, we do not know the address of its right subtree, so we have nothing to pass into the second recursive call. Instead unpack must traverse the packed tree and discover the address of the right subtree by moving past the end of the left subtree; this can be implemented by returning the address past the end of the ptree together with the root of the newly built tree, as a record:

{r -→ x ∗ r, 1 -→ ∗ ...} unpack(loc r) {r -→ x + 2 · sz ∗ r, 1 -→ y ∗ ...} (15)

With this specification, SuSLik is able to synthesize unpack in 20 s (benchmark 54 in Table 1); as for pack (benchmark 53), it is within the search space (which we confirmed in interactive mode) but automatic search currently times out after 30 minutes. In the future, it would be great if SuSLik could automatically discover an auxiliary with specification (15), given only (14) as inputs; this is similar to the problem of discovering accumulator parameters, which we discussed in Sect. 4, and is outside of capabilities of cyclic synthesis at the moment.

#### **6.3 Fine-Grained Concurrency**

Finally, we envision that deductive logic-based synthesis will make it possible to tackle the challenge of synthesizing provably correct concurrent libraries. The most efficient shared-memory concurrent programs implement custom synchronization patterns via fine-grained primitives, such as compare-and-set (CAS). Due to sophisticated interference scenarios between threads, reasoning about such programs is particularly challenging and error-prone, and is the reason for the existence of many extensions of Concurrent Separation Logic (CSL) [10,65] for verification of fine-grained concurrency [22,23,36,38,61,85,87–89].

For instance, Fine-Grained Concurrent Separation Logic (FCSL) [61,80,81], takes a very specific approach to fine-grained concurrency verification, following the tradition of logics such as LRG [25] and CAP [22] and building on the idea of splitting the specification of a concurrent library to a resource protocol and Hoare-style pre/postconditions. State-of-the art automated tools for fine-grained concurrency verification require one to describe both the protocol and Hoare-style pre/postconditions for the methods to be verified [21,94]. We believe, it should be possible to take those two components and instead synthesize the concurrent method implementations. The resource protocol will provide an extended set of language primitives to compose programs from. Those data structure-specific primitives can be easily specified in FCSL and contribute derived inference rules describing when these primitives can be used safely.

**Acknowledgements.** We thank Andreea Costea and Yutaka Nagashima for their feedback on the drafts of this paper. This research was supported by the National Science Foundation under Grant No. 1911149, by the Israeli Science Foundation (ISF) Grants No. 243/19 and 2740/19, by the United States-Israel Binational Science Foundation (BSF) Grant No. 2018675, by Singapore MoE Tier 1 Grant No. IG18-SG102, and by the Grant of Singapore NRF National Satellite of Excellence in Trustworthy Software Systems (NSoE-TSS).

#### **References**

1. Acclavio, M., Straßburger, L.: From syntactic proofs to combinatorial proofs. In: Galmiche, D., Schulz, S., Sebastiani, R. (eds.) IJCAR 2018. LNCS (LNAI), vol. 10900, pp. 481–497. Springer, Cham (2018). https://doi.org/10.1007/978-3-319- 94205-6 32


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **AI Verification**

### **DNNV: A Framework for Deep Neural Network Verification**

David Shriver(B) , Sebastian Elbaum , and Matthew B. Dwyer

University of Virginia, Charlottesville, VA, USA *{*dls2fc,selbaum,matthewbdwyer*}*@virginia.edu

**Abstract.** Despite the large number of sophisticated deep neural network (DNN) verification algorithms, DNN verifier developers, users, and researchers still face several challenges. First, verifier developers must contend with the rapidly changing DNN field to support new DNN operations and property types. Second, verifier users have the burden of selecting a verifier input format to specify their problem. Due to the many input formats, this decision can greatly restrict the verifiers that a user may run. Finally, researchers face difficulties in re-using benchmarks to evaluate and compare verifiers, due to the large number of input formats required to run different verifiers. Existing benchmarks are rarely in formats supported by verifiers other than the one for which the benchmark was introduced. In this work we present DNNV, a framework for reducing the burden on DNN verifier researchers, developers, and users. DNNV standardizes input and output formats, includes a simple yet expressive DSL for specifying DNN properties, and provides powerful simplification and reduction operations to facilitate the application, development, and comparison of DNN verifiers. We show how DNNV increases the support of verifiers for existing benchmarks from 30% to 74%.

**Keywords:** Deep neural networks · Formal verification · Tool

#### **1 Introduction**

Deep neural networks (DNN) are being applied increasingly in complex domains including safety critical systems such as autonomous driving [3,7]. For such applications, it is often necessary to obtain behavioral guarantees about the safety of the system. To address this need, researchers have been exploring algorithms for verifying that the behavior of a trained DNN meets some correctness property. In the past few years, more than 20 DNN verification algorithms have been introduced [2,4,6,8–11,15,21,22,24–27,29–34,36], and this number continues to grow. Unfortunately, this progress is hindered by several challenges.

First, DNN verifier developers must contend with a rapidly changing field that continually incorporates new DNN operations and property types. While supporting more properties and operations may increase the applicable scope


**Table 1.** The network and property formats supported by each verifier. A \* indicates that only a subset of the full input format specification is supported.

of verifiers to real-world problems, it also increases a verifier's complexity. For example, for a verifier such as *DeepPoly*, supporting additional operations requires non-trivial effort to define and prove correctness of new abstract transformers. For verifiers such as *Reluplex* or *Neurify*, supporting new property types requires implementing a mapping from those properties onto internal verifier structures.

Second, DNN verifier users carry the burden of re-writing property specifications and transforming their models to match a chosen verifier's supported format. That burden is compounded by the diversity of input formats required by each verifier, as illustrated in Table 1. There is little overlap between input formats for verifiers (only *DeepZono* and *DeepPoly* or *BaB* and *BaBSB* which are algorithmically similar), and even when using the same format (as in the case of the popular ONNX format) we find that the underlying operations supported are different. This makes it difficult and costly to run multiple verifiers on a given problem since the user must understand the requirements of each verifier and translate inputs to their formats. While two new formats, VNNLIB [13] and SOCRATES [20], have been introduced in an attempt to standardize DNN verifier input formats, their expressiveness is currently limited and they can require writing new conversion tools for networks, as we discuss at the end of Sect. 3.1.

Finally, DNN verifier researchers face challenges in re-using benchmarks to evaluate and compare verifiers. Most benchmarks exist in the format of the verifier for which they were introduced, and running other verifiers on that benchmark requires writing custom tooling to translate the benchmark to other formats, or writing new input parsers for verifiers to support the given benchmark format. For example, the ACAS Xu benchmark (described in Sect. 5), was originally specified with networks in *Reluplex*-NNET format, and properties hardcoded into the verifier. The benchmark was converted, for example, into RLV format for *BaB* and *BaBSB*, as well as into ONNX with hard-coded properties for *RefineZono*. Other benchmarks, such as the DAVE benchmark used by *Neurify*, has networks specified in *Neurify*-NNET, and properties hard-coded into the verifier. Due to its format, this potentially great benchmark has not been used by other verifiers.

*We introduce a framework,* DNNV*, to reduce the burden on verifier researchers, developers, and users.* DNNV helps to create and run more re-usable verification benchmarks by standardizing a network and property format, and it increases the applicability of a verifier to richer properties and real-world benchmarks by performing property reductions and simplifying DNN structures.

**Fig. 1.** DNNV architecture

As shown in Fig. 1, DNNV takes as input a network in the common ONNX input format, a property written in an expressive domain-specific language DNNP, and the name of a target verifier. Using the framework and plugins for the target verifier, DNNV transforms the problem by simplifying the network and reducing the property to enable the application of verifiers that otherwise would be unable to run. DNNV then translates the network and property to the input format of the desired verifier, runs that verifier on the transformed problem, and returns the results in a standardized format.

The primary contributions of this work are: (1) the DNNV framework to reduce the burden on DNN verifier researchers, developers, and users; DNNV includes a simple yet expressive DSL for specifying DNN properties, and powerful simplification and reduction operations to increase verifiers' scope of applicability, (2) an open source tool implementing DNNV<sup>1</sup>, with support for 13 verifiers, and extensive documentation, and (3) an evaluation demonstrating the cost-effectiveness of DNNV to increase the scope of applicability of verifiers.

#### **2 Background**

A deep neural network N encodes an approximation of a target function <sup>f</sup> : <sup>R</sup>*<sup>n</sup>* <sup>→</sup> <sup>R</sup>*<sup>m</sup>*. A DNN can be represented as a directed graph <sup>G</sup><sup>N</sup> <sup>=</sup> V<sup>N</sup> , E<sup>N</sup> , where nodes, <sup>v</sup> <sup>∈</sup> <sup>V</sup><sup>N</sup> , represent operations and edges, <sup>e</sup> <sup>∈</sup> <sup>E</sup><sup>N</sup> , represent input

<sup>1</sup> https://github.com/dlshriver/DNNV.

arguments to operations. A node without any incoming edges is an input to the DNN. The output of a DNN can be computed by looping over nodes in topological order and computing the value of the node given its inputs. The literature on machine learning has developed a broad range of rich operation types and explored the benefits of different combinations of operations in realizing accurate approximations of different target functions, e.g., [12].

Given a DNN, <sup>N</sup> : <sup>R</sup>*<sup>n</sup>* <sup>→</sup> <sup>R</sup>*m*, a property, <sup>φ</sup>(<sup>N</sup> ), defines a set of constraints over the inputs, <sup>φ</sup><sup>X</sup> – the pre-condition, and a set of constraints over the outputs, <sup>φ</sup><sup>Y</sup> – the post-condition. Verification of <sup>φ</sup>(<sup>N</sup> ) seeks to prove or falsify: <sup>∀</sup><sup>x</sup> <sup>∈</sup> <sup>R</sup>*<sup>n</sup>* : <sup>φ</sup><sup>X</sup> (x) <sup>→</sup> <sup>φ</sup><sup>Y</sup> (<sup>N</sup> (x)).

A widely studied class of properties is *robustness*, which originated with the study of adversarial examples [28,35]. These properties specify that inputs from a specific region of the input space must all produce the same output class. Detecting violations of robustness properties has been widely studied, and they are a common type of property for evaluating verifiers [10,25,26,29,30]. Another common class of properties is *reachability*, which define the post-condition using constraints over output values. Reachability properties specify that inputs from a given region of the input space must produce outputs within a given region of the output space. Such properties have been used to evaluate several DNN verifiers [16,17,30].

A recent survey on DNN verification [18] classifies these approaches based on their type: reachability, optimization, or search, or a combination of these. Reachability-based methods compute a representation of the reachable set of outputs from an encoding of the set of inputs that satisfy the pre-condition. The computed output set is often an over-approximation of the true reachable output region. The precision of the computed output region depends on the symbolic representation used, e.g., hyper-rectangles, zonotopes, polyhedra. Reachabilitybased methods include [11,22,24–27,34]. Optimization-based methods formulate property violations as a threshold for an objective function and use optimization algorithms to attempt to satisfy that threshold. Optimization-based methods include [2,9,21,29,33]. Search-based methods explore regions of the input space where they then formulate reachability or optimization sub-problems. Searchbased methods include [6,10,15,16,31,32].

#### **3 DNNV Overview**

DNNV remedies several key challenges faced by the DNN verification community. A general overview of DNNV is shown in Fig. 1. DNNV takes in a property and network in a standard format, simplifies the network, reduces the property, translates the network and property to the input format of the verifier, runs the verifier, and translates its output. Each of these components can be customized by verifier specific plugins. We explain these components in more detail below.

As shown in Table 1, existing verifiers do not support a consistent, common input format for networks and properties. DNNV standardizes the input and output formats to aid the community in creating and running verification benchmarks.

**ONNX.** For specifying general deep neural network architectures, we choose the open source DNN format ONNX [19]. ONNX can

represent real-world networks, is supported by many common frameworks (e.g., PyTorch, MXNet) and conversion tools are available for other frameworks (e.g., TensorFlow, Keras). Our current implementation supports a subset of the ONNX specification that subsumes the subsets of ONNX implemented by the supported verifiers. Table 2 shows the number of ONNX operations supported by each of the verifiers included in DNNV. DNNV supports 40% more operations than the verifier with the next highest support. The ONNX subset supported by DNNV is sufficient for almost all existing verification benchmarks, as well as many realworld networks including VGG16 and ResNet34.

**DNNP.** Due to the lack of a standard format for specifying DNN properties, we develop a Python-embedded DSL for DNN properties, which we call DNNP. DNNP is designed to express any property that can be verified by existing DNN verifiers in a form that is independent of the network. DNNP is described in more detail in Appendix A of the extended version of this paper [23].

We demonstrate DNNP with an example of a local robustness property, shown in Fig. 2. The property specifies that, for all inputs, x (Lines 14–23), in the input space (Line 18) and within a hyper-rectangle of

 from dnnv.properties import \* from torchvision.datasets import FashionMNIST from torchvision.transforms import ToTensor 4 N = Network("N") data = FashionMNIST("/tmp", download=True, transform=ToTensor()) mean = 0.2860 std = 0.3530 i = Parameter("data\_idx", type=int, default=1) x = (data[i][0][None, :].numpy() - mean) / std e = Parameter("epsilon", type=float) / std 13 14 Forall( 15 x\_, 16 Implies( 17 And( (-mean / std) <= x\_ <= ((1 - mean) / std),

**Fig. 2.** Example of a local robustness property specified with DNNP.

radius e centered at the given input x (Line 19), the network should predict the same maximum class for both x and x (Line 21). For Fashion MNIST, this means that for all images within an <sup>L</sup><sup>∞</sup> distance of <sup>e</sup> (specified on Line 12) from

**3.1 Input Formats Table 2.** The number of ONNX operations supported by each verifier.


19 (x - e) < x\_ < (x + e), 20 ), 21 argmax(N(x\_)) == argmax(N(x)), 22 ), 23 )

image 1 of the dataset (selected on Lines 10–11), the network should classify all of these images the same as it does for image 1. We first import several Python packages that will be useful for specifying the property (Lines 1–3), including the dataset used to train the network, and a method for data manipulation. Because DNNP allows importing arbitrary Python packages, it enables re-use of the same data loading and manipulation methods used to train a network. After importing the necessary utilities, we define several variables that will be used in the final property expression (Lines 5–12). Two of these variables, i on Line 10 and e on Line 12 are declared as parameters, which allows them to be specified on the command line at run time. The value for e must be provided at run time, since no default value is provided. Finally, we define the semantics of the property specification, using methods provided by DNNP, as well as variables defined above (Lines 14–23).

**Fig. 3.** Batch Normalization Simplification simplifies a batch norm following a convolution operation to an equivalent single convolution operation with modified weights and bias, while maintaining the strides and pads.

**Other Input Formats.** Since the creation of DNNV, two new input formats, VNNLIB [13] and SOCRATES [20], have emerged in an attempt to standardize the verifier input space. The current draft of VNNLIB also uses ONNX as the DNN input format, however it supports a much smaller set of operations than DNNV, supporting only 17 ONNX operations. The VNNLIB property format is a subset of SMTLIB in which variables of the form X*<sup>i</sup>* are implicitly mapped to network inputs and variables of the form Y*<sup>i</sup>* are implicitly mapped to network outputs. In its current form, this specification only supports DNN models with a single flat input tensor and single flat output tensor, whereas DNNP and ONNX can support DNN models with multiple inputs and output tensors of any shape. SOCRATES proposes a JSON format containing both the property and network specifications. Because DNNV treats networks and properties independently, properties can be re-used for multiple networks, and only a single network must be stored to check multiple properties, resulting in a lower storage cost, especially for large models. Additionally, while the custom JSON format used by SOCRATES requires new DNN translation tools to be written to convert to the required format, the ONNX format used by DNNV is commonly available in most machine learning frameworks. While we believe that ONNX and DNNP are currently the most expressive and easily accessible input formats currently proposed, DNNV can provide benefits to any format through DNN simplification and property reduction to increase the applicability of all verifiers.

#### **3.2 Network Simplification**

In order to allow verifiers to be applied to a wider range of real world networks, DNNV provides tools for network simplification. Network simplification takes in an operation graph and applies a set of semantics preserving transformations to the operation graph to remove unsupported structures, or to transform sequences of operations into a single more commonly supported operation.

An operation graph <sup>G</sup><sup>N</sup> <sup>=</sup> V<sup>N</sup> , E<sup>N</sup> is a directed graph where nodes, <sup>v</sup> <sup>∈</sup> <sup>V</sup><sup>N</sup> represent operations, and edges <sup>e</sup> <sup>∈</sup> <sup>E</sup><sup>N</sup> represent inputs to those operations. Simplification, *simplify* : G→G, transforms an operation graph <sup>G</sup><sup>N</sup> ∈ G, to an equivalent DNN with more commonly supported structure, *simplify*(G<sup>N</sup> ) = <sup>G</sup>N- , such that the resulting DNN has the same behavior as the original <sup>∀</sup>x.<sup>N</sup> (x) = <sup>N</sup> (x), and uses more commonly supported structures.

One such simplification is *batch normalization simplification*, which removes batch normalization operations from a network by combining them with a preceding convolution operation or generalized matrix multiplication (GEMM) operation. This is possible since batch normalization, convolution, and GEMM operations are all affine operations. The simplification of a batch normalization operation following a convolution operation is shown in Fig. 3. If no applicable preceding layer exists, the batch normalization layer is converted into an equivalent convolution operation. This simplification enables the application of verifiers without explicit support for batch normalization operations, such as *Neurify* and *Marabou*, to networks with these operations.

**Fig. 4.** Property reduction to a local robustness property adds a suffix that classifies outputs as violations or non-violations of the original output constraints, and changing the property to a common form of robustness property.

DNNV currently includes 6 additional DNN simplifications, enumerated and described in more detail in Appendix B of the extended version of this paper [23].

#### **3.3 Property Reduction**

In order to allow verifiers to be applied to more general safety properties, DNNV provides tools to reduce properties to a supported form. For instance, properties can be translated to local robustness properties, which are required by *MIPVerify* or reachability properties which are required by *Reluplex*.

Property reduction takes in a verification problem, which is comprised of a property specification and a network, and encodes it as an equivalid set of verification problems with properties in a form supported by a given verifier.

<sup>A</sup> *verification problem* is a pair, <sup>ψ</sup> <sup>=</sup> N , φ, of a DNN, <sup>N</sup> , and a property specification <sup>φ</sup>, formed to determine whether N |<sup>=</sup> <sup>φ</sup> is *valid*. Reduction, reduce : <sup>Ψ</sup> <sup>→</sup> <sup>P</sup>(Ψ), aims to transform a verification problem, N , φ <sup>=</sup> <sup>ψ</sup> <sup>∈</sup> <sup>Ψ</sup>, to an equivalid form, reduce(ψ) = {N1, φ1,...,N*k*, φ*k*}, in which property specifications are in a common supported form. As defined, reduction has two key properties. The first property is that the set of resulting problems is equivalid with the original verification problem. The second property is that the resulting set of problems all use the same property type. Applying reduction enables verifiers to support a large set of verification problems by implementing support for a single property type.

For example, given a network that classifies images of clothing items, a user may want to specify that, if the network classifies an image as a coat, then the score given to the class of a pullover is not less than the score for the sneaker class. The property is specified in the bottom left of Fig. 4. Such a verification problem can be difficult to specify for many verifiers. For example, *Neurify* would require writing code to specify linear constraints for the property and re-compiling the verifier, and *MIPVerify* cannot support this property as is. DNNV can reduce this verification problem to an equivalent problem with a robustness property.

A high level overview of this reduction is shown in Fig. 4; a more detailed description is provided in Appendix C of the extended version of this paper [23].

#### **3.4 Input and Output Translation**

Because of the large variety of input formats required by the verifiers, one of the primary components of DNNV translates from its internal representation of properties and networks to the input formats of each verifier.

DNNV also requires an output translator that can parse the results of running a verifier and returns sat, unsat, or unknown. If the result is sat, indicating a violation was found, DNNV also returns a counter example to the property, and validates that it does violate the property by performing inference with the network and confirming that the input and output do not satisfy the property.

#### **4 Implementation**

DNNV is written in 8400 lines of Python code and is available for download and re-use at https://doi.org/10.5281/zenodo.4883626. Python was chosen due to its ubiquitous use for developing deep neural networks. DNNV currently supports 13 verifiers, and was designed to facilitate the integration of new verifiers. The currently supported verifiers are shown in Table 1, along with their original input formats, and algorithmic approach. Around 2000 LOC (of the 8400 total LOC) are used to integrate these 13 verifiers into DNNV, with *Planet* requiring the most effort at 437 lines, and *BaB* and *BaBSB* requiring the least effort with 89 lines of code due to re-use of the *Planet* input translator.

#### **4.1 Supporting Reuse and Extension**

DNNV is designed to facilitate the integration of new verifiers. The 5 primary components of DNNV, DNN simplification, property reduction, input translation, verifier execution, and output translation are designed to be re-usable, and to facilitate the implementation of new components by providing utilities for traversing and manipulating operation graphs and properties.

Networks are represented as an operation graph, where nodes represent operations in the DNN and edges represent inputs and outputs to those operations. The operation graph can also be traversed using a visitor pattern. This pattern is particularly useful for the development of DNN simplifications and input translators. It allows developers to easily traverse computation graphs in order to translate operations to the required format. We provide built-in utilities for converting from our internal network representation to ONNX, PyTorch, and TensorFlow models. The implementation also includes utilities for performing pattern matching on operation graphs. We utilize this feature to provide utilities that transform a network from an operation graph representation to a sequential layer representation, which is particularly useful for the network input translator of *Neurify*, which requires DNNs to have a regular structure of a set of convolutional layers followed by fully connected layers, all with relu activations.

#### **4.2 Usage**

DNNV can be run from the command line as follows: python -m dnnv <prop> <verifier> --network <name> <path>, where the arguments correspond to a DNN model in the ONNX format, a property written in DNNP, and the verifier to run. Many additional options can be seen by specifying the -h option.

After execution, for each verifier, DNNV reports the verification result as one of sat (if the property was falsified), unsat (if the property was proven to hold), unknown (if the verifier is incomplete and could not prove the property holds), or error, along with the reason for error, if an error occurs during DNN and property translation, or during verifier execution. DNNV also reports the time to translate and verify the property.

#### **5 Study**

We now examine the applicability of verifiers to existing verification benchmarks with and without DNNV. A verification benchmark consists of a set of verification problems which are used to evaluate the performance of a verifier. A problem is made of a DNN and a property specification and asks whether the property is valid for the given DNN. We consider a verifier to support a benchmark if it can be run on that benchmark out of the box. We consider a verifier to have support for a benchmark through DNNV if DNNV can be run on that benchmark with networks specified using ONNX and properties specified in DNNP, and can reduce, simplify, and translate the problem to work with the target verifier.

**Benchmarks.** To evaluate benchmark support, we collected the benchmarks used by each of the 13 verifiers supported by DNNV, and determined whether each verifier can run on the benchmark out of the box, and also whether they could be run on the benchmark when DNNV is applied. The verification benchmarks are shown in Table 3 and are also described in more detail in Appendix D of the extended version of this paper [23]. Each row of the table corresponds to a benchmark, to which we assign a short key for identifying the benchmark. For each benchmark, we give the name, some of the verifiers it evaluated, the number of properties (#P) and networks (#N ), and features that can make it challenging for verifiers. These features include whether any properties cannot represent their input constraints using hyper-rectangles (¬HR), whether any network in the benchmark contains convolution operations (C), whether any network contains residual structures (R), and whether any network uses any non-ReLU activation functions (¬ReLU).

**Results.** The support of verifiers for each benchmark is shown in Table 4. Each row of this table corresponds to one of the 13 verifiers supported by DNNV, and each column corresponds to one of the 19 benchmarks identified in Table 3. Each


**Table 3.** Verifier benchmarks.

**Table 4.** Benchmark support by each verifier. The left half of the circle is black if the verifier can support the benchmark out of the box, and is white otherwise. The right half is black if the verifier supports the benchmark through DNNV, and is white otherwise. An absent circle indicates that the verifier can not be made to support some aspect of the benchmark.


cell of the table may contain a circle that identifies the support of the verifier for the benchmark. The left half of the circle is black if the verifier can support the benchmark out of the box, and is white otherwise. The right half is black if the verifier supports the benchmark through DNNV, and white otherwise. An absent circle indicates that the verifier can not be made to support some aspect of the benchmark. For the benchmarks shown here, this is always due to the presence of non-ReLU activation functions in some of the networks in the benchmarks.

As shown in Table 4, DNNV can dramatically increase the support of verifiers for benchmarks. For example, the *Planet* verifier could originally be run on 5 of the 19 benchmarks, but could be run on 16 using DNNV. Similarly, the *nnenum* verifier, could originally only be run on 1 of the existing benchmarks, but could be run on 13 using DNNV. **Of the 223 pairs of verifiers and benchmarks for which support may be possible, 166 of them are currently supported by DNNV, an increase of over 2.4 times the 68 pairs supported without DNNV.**

### **6 Conclusion**

We present the DNNV framework for reducing the burden on DNN verifier researchers, developers, and users. DNNV standardizes input and output formats, includes a simple yet expressive DSL for specifying DNN properties, and provides powerful simplification and reduction operations to facilitate the application, development, and comparison of DNN verifiers. Our study showed the potential of DNNV and we made its implementation available, with support for 13 verifiers, and extensive documentation.

**Acknowledgment.** This material is based in part upon work supported by the National Science Foundation under Grant Number 1900676 and 2019239.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Robustness Verification of Quantum Classifiers**

Ji Guan1(B) , Wang Fang1,2, and Mingsheng Ying1,3,4

> <sup>1</sup> State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences, Beijing 100190, China

{guanj,fangw}@ios.ac.cn <sup>2</sup> University of Chinese Academy of Sciences, Beijing 100049, China <sup>3</sup> Center for Quantum Software and Information, University of Technology Sydney, Ultimo, NSW 2007, Australia mingsheng.ying@uts.edu.au

<sup>4</sup> Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China

**Abstract.** Several important models of machine learning algorithms have been successfully generalized to the quantum world, with potential speedup to training classical classifiers and applications to data analytics in quantum physics that can be implemented on the near future quantum computers. However, quantum noise is a major obstacle to the practical implementation of quantum machine learning. In this work, we define a formal framework for the robustness verification and analysis of quantum machine learning algorithms against noises. A robust bound is derived and an algorithm is developed to check whether or not a quantum machine learning algorithm is robust with respect to quantum training data. In particular, this algorithm can find adversarial examples during checking. Our approach is implemented on Google's TensorFlow Quantum and can verify the robustness of quantum machine learning algorithms with respect to a small disturbance of noises, derived from the surrounding environment. The effectiveness of our robust bound and algorithm is confirmed by the experimental results, including quantum bits classification as the "Hello World" example, quantum phase recognition and cluster excitation detection from real world intractable physical problems, and the classification of MNIST from the classical world.

**Keywords:** Quantum machine learning · Robustness verification · Adversarial examples · Robust bound

#### **1 Introduction**

In the last few years, the successful interplay between machine learning and quantum physics shed new light on both fields. On the one hand, machine learning has been dramatically developed to satisfy the need of the industry over the past two decades. At the same time, many challenging quantum physical problems have been solved by automated learning. Notably, inaccessible quantum many-body problems have been solved by neural networks, one instance of machine learning [1]. On the other hand, as the new model of computation under quantum mechanics, quantum computing has been proved that it can (exponentially) speed up classical algorithms for some important problems [2]. This motivates the development of quantum machine learning and provides the possibility of improving the existing computational power of machine learning to a new level (see the review papers [3,4] for the details). After that, quantum machine learning was integrated into solving real world problems in quantum physics. One essential example is that quantum convolutional neural networks inspired by machine learning were proposed to implement quantum phase recognition [5]. Quantum phase recognition asks whether a given input quantum state belongs to a particular quantum phase of matter. At the same time, more provable advantages of quantum machine learning than the classical counterpart have been reported. For instance, the training complexity of quantum models has an exponential improvement on certain tasks [6]. Stepping into industries, Google recently built up a framework *TensorFlow Quantum* for the design and training of quantum machine learning within its famous classical machine learning platform—TensorFlow [7].

Even though quantum machine learning outperforms the classical counterpart in some way, the difficulties in the classical world are expected to be encountered in the quantum case. Classical machine learning has been found to be vulnerable to intentionally crafted adversarial examples (e.g. [8,9]). Adversarial examples are inputs to a machine learning algorithm that an attacker has crafted to cause the algorithm to make a mistake. One essential mission of machine learning is to prove the absence of or detect adversarial examples used in the defense strategy—adversarial training [10]—appending adversarial examples to the training dataset and retraining the machine learning algorithm to be robust to these examples. However, this goal is not easily achieved [11]. The machine learning community has developed several interesting ideas on designing specific attack algorithms (e.g. [10,12]) to generate adversarial examples, which is far from measuring the robustness against any adversary. Recently, the formal method community has taken initial steps in this direction [13–16], by verifying the robustness of classical machine learning algorithms in a provable way: either a formal guarantee that the algorithms are robust for a given input or a counter-example (adversarial example) is provided if an input is not robust. Some tools have been developed, such as VerifAI [17] and NNV [18]. This phenomenon of vulnerability is more common in the quantum world since quantum noise is inevitable in quantum computation, at least in the current NISQ (Noisy Intermediate-Scale Quantum) era, and thus led to a series of recent works on quantum machine learning robustness against specific noises. For example, Lu et al. [19] studied the robustness to various classical adversarial attacks; Du et al. [20] proved that by appending depolarization noise in quantum circuits for classifications, a robust bound against adversaries can be derived; Liu and Wittek [21] gave a robust bound for the quantum noise coming from a special unitary group. Very recently, Weber et al. [22] formalized a link between binary quantum hypothesis testing [23] and robust quantum machine learning algorithms for classification tasks.

Up to our best knowledge, the existing studies of quantum machine learning robustness only consider the situation of a *known* noise source. However, a fundamental difference between quantum and classical machine learning is that the quantum attacker is usually the surroundings instead of humans in the classical case, and the information of the environment is unknown. To protect against an *unknown* adversary, we need to derive a robust guarantee against a worst-case scenario, from which the commonly-assumed known noise sources (e.g. depolarization noise [20]) are usually far. Yet in the case of unknown noise, several basic issues are still unsolved:


In this work, we define a formal framework for the robustness verification and analysis of quantum machine learning algorithms against noises in which the above problems can be studied in a principled way. More specifically, we choose to use fidelity as the metric measuring the robustness as it is one of the most widely used quantities to quantify the uncertainty of noise in the process of quantum computation, and commonly used in quantum engineering and experimental communities (e.g. [25,26]). Based on this, an analytical robust bound for any quantum machine learning classification algorithm is obtained and can be applied to approximately checking the robustness of quantum machine learning algorithms. Furthermore, we show that computing the optimal robust bound can be reduced to solving a Semidefinite Programming (SDP) problem. These results lead to an algorithm to exactly and efficiently check whether or not a quantum machine learning algorithm is robust with respect to the training data. A special strength of this algorithm is that it can identify useful new training data (adversarial examples) during checking, and these data can be used to implement adversarial training as the same as classical robustness verification. The effectiveness of our robust bound and algorithms is confirmed by the case studies of quantum bits classification as the "Hello World" example of quantum machine learning algorithms, quantum phase recognition and cluster excitation detection from real world intractable physical problems, and the classification of MNIST from the classical world.

In summary, the main technical contributions of the paper are as follows.

– *Computing the optimal robust bound* of quantum machine classification algorithms is reduced to an SDP (Semidefinite Programming) problem;


#### **2 Quantum Data and Computation Models**

For the convenience of the reader, in this section, we recall some basic concepts of quantum data (states) and the quantum computation model.

The basic data of classical computers are bits, represented by two digits 0 and 1. In quantum computing, quantum bits (qubit) play the same role. A qubit is expressed by a normalized complex vector |φ- = a b = a|0- + b|1 with complex numbers a and b satisfying the normalization condition |a| <sup>2</sup> <sup>+</sup> <sup>|</sup>b<sup>|</sup> <sup>2</sup> = 1. Here, |0- = -1 0 , |1- = -0 1 correspond to bits 0, 1 respectively, and {|0-, |1-} is an orthonormal basis of a 2-dimensional Hilbert (linear) space. In general, for a quantum computer consisting of n qubits, a quantum datum is a normalized complex vector |ψ in a 2<sup>n</sup>-dimensional Hilbert space <sup>H</sup>. Such a <sup>|</sup>ψ is usually called a pure state in the literature of quantum computation.

As a model for computation, a quantum circuit consists of a sequence of, say m quantum logic gates. Each quantum gate can be mathematically represented by a unitary matrix U<sup>i</sup> on H, i.e., U† <sup>i</sup> U<sup>i</sup> = UiU† <sup>i</sup> = I, where U† <sup>i</sup> is the conjugate transpose of U<sup>i</sup> and I is the identity matrix on H. Then the circuit is represented by the unitary matrix U = U<sup>m</sup> ···U1. If the quantum datum |ψ is inputted to the circuit, then the output is a quantum datum:

$$|\psi'\rangle = U|\psi\rangle. \tag{1}$$

In practice, a quantum datum may not be completely known and can be thought of as a mixed state or ensemble {(pk, |ψk-)}k, meaning that it is at |ψk- with probability pk. Mathematically, it can be described by a density operator <sup>ρ</sup> (Hermitian positive semidefinite matrix with unit trace<sup>1</sup>) on <sup>H</sup>:

$$
\rho = \sum\_{k} p\_k |\psi\_k\rangle\langle\psi\_k|,\tag{2}
$$

where ψk| is the conjugate transpose of |ψk-, i.e., |ψk- = ψk| †. In this case, the model of quantum computation is tuned to be a super-operator E, i.e. a mapping from matrices to matrices. It can be written as

$$
\rho' = \mathcal{E}(\rho). \tag{3}
$$

<sup>1</sup> ρ has unit trace if tr(ρ) = 1, where trace tr(ρ) of ρ is defined as the summation of diagonal elements of ρ.

Here, ρ and ρ are the input and output data (mixed states) of quantum computation E, respectively. Not every super-operator E is meaningful in physics. It is required to satisfy the following conditions:


Such a super-operator E admits a Kraus matrix form [2]: there exists a set of matrices {Ek}<sup>k</sup> on H such that

$$\mathcal{E}(\rho) = \sum\_{k} E\_k \rho E\_k^\dagger.$$

Here {Ek}<sup>k</sup> is called Kraus matrices of E [2].

The behind dynamics of quantum computers is governed by quantum mechanics, which is applied at the microscopic scale (near or less than 10−<sup>9</sup> meters). At this level, we cannot directly readout the quantum data as the same as the classical counterpart. The only way to extract information from it is through a quantum measurement, which is mathematically modeled by a set {Mk}<sup>m</sup> <sup>k</sup>=1 of matrices on its state (Hilbert) space H with <sup>k</sup> M† <sup>k</sup>M<sup>k</sup> = I. This observing process is probabilistic: if the system is currently in state ρ, then a measurement outcome k is obtained with probability

$$p\_k = \text{tr}(M\_k^\text{I} M\_k \rho). \tag{4}$$

After the measurement, the system's state will be collapsed (changed), depending on the measurement outcome k, which is vitally different from the classical computation. If the outcome is k, the post-measurement state becomes

$$
\rho\_k' = \frac{M\_k \rho M\_k^\dagger}{\text{tr}(M\_k^\dagger M\_k \rho)}.\tag{5}
$$

This special property makes it hard to accurately estimate the distribution {pk}<sup>k</sup> unless enough many copies of ρ are provided.

In summary, quantum data have two different forms—pure state |ψ and mixed state ρ corresponding to the computation model as a unitary matrix U or a super-operator E, respectively. Not surprisingly, the latter is a generalization of the former by putting:

$$\rho = |\psi\rangle\langle\psi|, \qquad \mathcal{E}(\rho) = U\rho U^\dagger.$$

Because of this, the results obtained for mixed states ρ can also be applied to pure states |ψ-. Thus, in this paper, we mainly consider mixed states as the quantum data and super-operators as the model of quantum computation.

#### **3 Quantum Classification Algorithms**

In this section, we briefly recall quantum classification algorithms. They are designed for *classification of quantum data*. Essentially, they share the same basic ideas with their classical counterparts but deal with quantum data in the quantum computation model.

#### **3.1 Basic Definitions**

In this paper, we focus on a specific learning model called quantum supervised classification. Given a Hilbert space H, we write D(H) for the set of all (mixed) quantum states on H (see its definition in Eq. (2)).

**Definition 1.** *A quantum classification algorithm* A *is a mapping* D(H) → C*, where* C *is the set of classes we are interested in.*

Following the training strategy of classical machine learning, the classification A is learned through a dataset T instead of being pre-defined. This training dataset <sup>T</sup> <sup>=</sup> {(ρi, ci)}<sup>N</sup> <sup>i</sup>=1 consists of N < ∞ pairs (ρi, ci), meaning that quantum state ρ<sup>i</sup> belongs to class ci. To learn A, we initialize a *quantum learning model*—a parameterized quantum circuit (including measurement control) E<sup>θ</sup> and a measurement {Mk}<sup>k</sup>∈C. Mathematically, the circuit can be modelled as a quantum super-operator E<sup>θ</sup> (see its definition in Eq. (3)), and θ is a set of free parameters that can be tuned. Then for each k ∈ C, we can compute the probability of the measurement outcome being k:

$$f\_k(\theta,\rho) = \text{tr}(M\_k^\dagger M\_k \mathcal{E}\_\theta(\rho)).\tag{6}$$

It is worth noting that, as we mentioned before, measuring quantum state ρ is probabilistic and ρ will be changed after measuring. So, in practice, accurately estimating fk(θ, ρ) for all k ∈ C requires enough many copies of ρ, which is not the same as the classical case, where a single copy of classical data often meets the training process.

The quantum classification algorithm A outputs the class label c for a quantum state ρ using the following condition:

$$\mathcal{A}(\theta,\rho) = \arg\max\_{k} \text{tr}(M\_k^\dagger M\_k \mathcal{E}\_\theta(\rho)). \tag{7}$$

The learning is carried out as θ is optimized to minimize the empirical risk

$$\min\_{\theta} \frac{1}{N} \sum\_{i=1}^{N} \mathcal{L}(f(\theta, \rho\_i), c\_i), \tag{8}$$

where L refers to a predefined loss function, f(θ, ρ) is a probability vector with each fk(θ, ρ), k ∈ C as its element, and c<sup>i</sup> is also seen as a probability vector with the entry corresponding to c<sup>i</sup> being 1 and others being 0. The goal is to find the optimized parameters θ<sup>∗</sup> minimizing the risk in Eq. (8) for the given dataset T. Mean-squared error (MSE) is the most popular instance of the empirical risk, i.e., the loss function L is squared error:

$$\mathcal{L}(f(\theta,\rho\_i),c\_i) = \frac{1}{C} \| f(\theta,\rho\_i) - c\_i \|\_{2}^{2},$$

where C is the number of classes in C, and ·<sup>2</sup> is the l2-norm.

As one can see in the above learning process, the main differences between classical and quantum machine learning algorithms are the learning models and data.

In this paper, we focus on the well-trained quantum classification algorithm A, usually called a quantum classifier. Here, A is said to be well-trained if training and validation accuracy are both high (≥95%). The training (validation) accuracy is the frequency that A successfully classifies the data in a training (validation) dataset. A validation dataset is mathematically equivalent to a training dataset but only for testing A rather than learning A. In this context, θ<sup>∗</sup> is naturally omitted, i.e., A(ρ) = A(θ∗, ρ) and E(ρ) = E<sup>θ</sup><sup>∗</sup> (ρ). Briefly, A only consists of a super-operator E and a measurement {Mk}k, denoted by A = (E, {Mk}k).

#### **3.2 An Illustrative Example**

Let us further illustrate the above definitions by a concrete example—Quantum Convolutional Neural Networks (QCNNs) [5], one of the most popular and successful quantum learning models. QCNN extends the main features and structures of the Convolutional Neural Networks (CNNs) to quantum computing.

**Fig. 1.** Simple example of CNN and QCNN. QCNN, like CNN, consists of a convolution layer that finds a new state and a pooling layer that reduces the size of the model. Here, MCUG stands for measurement control unitary gate, i.e., unitary matrix V<sup>1</sup> is applied on the circuit if and only if the measurement outcome is 1.

The model of QCNN applies the convolution layer and the pooling layer from CNNs to quantum systems, as shown in Fig. 1(b). The layout proceeds as follows:


The input of QCNNs is an unknown quantum state ρin and the output is obtained by measuring a fixed number of output qubits. As in the classical case, the learning model (defined as the number of convolution and pooling layers) is fixed, but the involved quantum gates (i.e. unitary matrices) Ui, V<sup>j</sup> , F themselves are learned by the above learning process.

*Remark 1.* Quantum machine learning can also be used to do classical machine learning tasks. Image classification, for example, is one of the most successful applications of Neural Networks (NNs). To explore the possible advantage of quantum computing, Quantum Neural Networks (QNNs) have been used to classify images in [27,28]. It is shown that by encoding images to a quantum state ρin, QNNs can achieve high accuracy in image classification. We will present a quantum classifier for the classification of MNIST as an example in the evaluation section.

#### **4 Robustness**

An important issue in classical machine learning is: how robust is a classification algorithm to adversarial perturbations. A similar issue exists for quantum classifiers against quantum noise. Intuitively, the robustness of quantum classifier A is the ability to make correct classification with a small perturbation to the input states. Then a quantum state σ is considered as an adversarial example if it is similar to a benign state ρ, but ρ is correctly classified and σ is classified into a class different from that of ρ. Formally,

**Definition 2 (Adversarial Example).** *Suppose we are given a quantum classifier* A(·)*, an input example* (ρ, c)*, a distance metric* D(·, ·) *and a small enough threshold value* ε > 0*. Then* σ *is said to be an* ε*-adversarial example of* ρ *if the following is true*

$$(\mathcal{A}(\rho) = c) \land (\mathcal{A}(\sigma) \neq c) \land (D(\rho, \sigma) \le \varepsilon) .$$

The leftmost condition A(ρ) = c asserts that ρ is correctly classified, the middle condition A(σ) = c means that σ is incorrectly classified, and the rightmost condition D(ρ, σ) ≤ ε indicates that ρ and σ are similar (i.e., their distance is small). Sometimes, without any ambiguity, σ is called an adversarial example of ρ if ε is preset. Notably, by the above definition, if A incorrectly classifies ρ, then we do not need to consider the corresponding adversarial examples. This is the correctness issue of quantum classifier A rather than the robustness issue. Hence, in the following discussions, we only consider the set of all correctly recognized states.

The absence of adversarial examples leads to robustness.

**Definition 3 (Adversarial Robustness).** *Let* A *be a quantum classifier. Then* ρ *is* ε*-robust for* A *if there is no adversarial example of* ρ*.*

The major problem concerning us in this paper is the following:

*Problem 1 (Robustness Verification Problem).* Given a quantum classifier A(·) and an input example (ρ, c). Check whether or not A(σ) = c for all σ ∈ Nε(ρ), where Nε(ρ) is the ε-neighbourhood of ρ as

$$\mathcal{N}\_{\varepsilon}(\rho) = \{ \sigma \in \mathcal{D}(\mathcal{H}) : D(\rho, \sigma) \le \varepsilon )\}.$$

If not, then an adversarial example (counter-example) σ ∈ Nε(ρ) is provided.

Obviously, if δ is a robust bound for an input example (ρ, c) such that A(σ) = c for any state σ ∈ Nδ(ρ), then for any ε ≤ δ (i.e. Nε(ρ) ⊆ Nδ(ρ)), there is no ε-adversarial example of ρ. It is a challenging problem to compute the optimal robust bound δ<sup>∗</sup> = max δ so that there is no ε-adversarial example if and only if ε ≤ δ∗.

The above adversarial robustness of quantum states can be generalized to a notion of robustness for quantum classifiers:

**Definition 4 (Robust Accuracy).** *Let* A *be a quantum classifier. The* ε*robust accuracy of* A *is the proportion of* ε*-robust states in the training dataset.*

*Remark 2.* Here, the robust accuracy is defined with respect to the training dataset. In some applications, the dataset can be chosen as another set of quantum states with correct classifications, such as a validation dataset or a combination of it with the training dataset.

The reader should notice that the above definitions of robustness for quantum classifiers are similar to those for classical classifiers. But an intrinsic distinctness between them comes from the choice of distance D(·, ·). In the classical case, humans play the role of the adversary, and then such a distance should promise that a small perturbation is imperceptible to humans, and vice versa. Otherwise, we cannot take the advantage of machine learning over human's distinguishability. For instance, in image recognition, the distance should reflect the perceptual similarity in the sense that humans would consider adversarial examples generated by it perceptually similar to benign image [24]. In the quantum case, it is essential to choose a distance D that is meaningful in quantum physics. In this paper, we choose to use the distance:

$$D(\rho, \sigma) = 1 - F(\rho, \sigma)$$

defined by fidelity

$$F(\rho, \sigma) = \left[ \text{tr}(\sqrt{\sqrt{\rho}\sigma\sqrt{\rho}}) \right]^2.$$

Here √ρ = k <sup>√</sup>λk|ψkψk| if ρ admits the spectral decomposition <sup>k</sup> λ<sup>k</sup> |ψkψk|. Fidelity is one of the most widely used quantities to quantify such uncertainty of noise by the experimental quantum physics and quantum engineering communities (see e.g. [29,30]).

*Remark 3.* The trace distance has been used in recent literature (e.g. [20]) for some issues related to quantum robustness verification:

$$T(\rho, \sigma) = \frac{1}{2} ||\rho - \sigma||\_{tr} = \frac{1}{2} \text{tr}[\sqrt{(\rho - \sigma)^\dagger (\rho - \sigma)}].$$

It is a generalization of the total variation distance, which is a distance measure for probability distributions. So far, to the best of our knowledge, there is no discussion about which distance is better in the literature. Here, we argue that fidelity is better than trace distance in the context of quantum machine learning against quantum noise. As we know, state distinguishability is the basis of measuring the effect of noise on quantum computation. The main difference between trace distance T(ρ, σ) and fidelity F(ρ, σ) is the number of copies of states ρ and σ as the resource required in the experiments for distinguishing them. More precisely, trace distance quantifies the maximum probability of correctly guessing through a measurement whether ρ or σ was prepared, while fidelity asserts the same quantity whence infinitely many samples of ρ and σ can be supplied (See Appendix A of the extended version of this paper [31] for more details). In quantum machine learning, a large enough number of copies of the states are the precondition of statistics in Eq. (6) for learning and classification. Thus, fidelity is more suitable than trace distance for our purpose.

#### **5 Robust Bound**

In this section, we develop a theoretic basis for robustness verification of quantum classifiers. After setting the distance D to be the one defined by fidelity, a robust bound can be derived.

**Lemma 1 (Robust Bound).** *Given a quantum classifier* A = (E, {Mk}<sup>k</sup>∈C) *and a quantum state* ρ*. Let* p<sup>1</sup> *and* p<sup>2</sup> *be the first and second largest elements of* {tr(M† <sup>k</sup>MkE(ρ))}k*, respectively. If* <sup>√</sup>p<sup>1</sup> <sup>−</sup> <sup>√</sup>p<sup>2</sup> <sup>&</sup>gt; <sup>√</sup>2ε*, then* <sup>ρ</sup> *is* <sup>ε</sup>*-robust.*

*Proof.* See Appendix B of the extended version of this paper [31].

The above robust bound gives us a quick robustness verification by the measurement outcomes of ρ without searching any possible adversarial examples. Furthermore, it also can be used to compute an under-approximation of the robust accuracy of A by one-by-one checking the robustness of quantum states in the training dataset. We will see that the robust bound and the induced robust accuracy scales well in the later experiments. However, <sup>√</sup>p<sup>1</sup> <sup>−</sup> <sup>√</sup>p<sup>2</sup> <sup>&</sup>gt; <sup>√</sup>2<sup>ε</sup> is not a necessary condition of <sup>ε</sup>-robustness. Fortunately, when <sup>√</sup>p<sup>1</sup> <sup>−</sup> <sup>√</sup>p<sup>2</sup> <sup>≤</sup> <sup>√</sup>2ε, we can compute the optimal robust bound by Semidefinite Programming (SDP). Recall that SDP is a convex optimization concerned with the optimization of a linear objective function over the intersection of the cone of positive semidefinite matrices with an affine space. It has the form

$$\begin{aligned} \min & \quad \text{tr}(CX) \\ \text{subject to} & \quad \text{tr}(A\_k X) \le b\_k, \quad \text{for } k = 1, \dots, m \\ & \quad X \ge 0 \end{aligned}$$

where C, A1,...,A<sup>m</sup> are all Hermitian n × n matrices (i.e. A† = A), and X is the optimization variable n × n matrix with X ≥ 0, i.e., X is positive semidefinite. Many efficient solvers have been developed for solving SDPs—not only compute the minimal value, but also output a corresponding optimal solution X. The following two theorems show that checking ε-robustness and computing optimal robust bound of quantum states can both be reduced to an SDP.

**Theorem 1 (**ε**-robustness Verification).** *Let* A = (E, {Mk}<sup>k</sup>∈C) *be a quantum classifier and* ρ *be a state with* A(ρ) = l*. Then* ρ *is* ε*-robust if and only if for all* k ∈ C *and* k = l*, the following problem has no solution (feasibility problem):*

$$\begin{aligned} \min\_{\sigma \in \mathcal{D}(\mathcal{H})} \quad & 0 \\ subject \; to \quad \sigma \ge 0 \\ & \text{tr}(\sigma) = 1 \\ & \text{tr}[(M\_l^\dagger M\_l - M\_k^\dagger M\_k)\mathcal{E}(\sigma)] \le 0 \\ & 1 - F(\rho, \sigma) \le \varepsilon \end{aligned}$$

*Proof.* See Appendix C of the extended version of this paper [31].

Actually, the objective function 0 in the above theorem can be chosen as any constant number.

**Theorem 2 (Optimal Robust Bound).** *Let* A *and* ρ *be as in Theorem 1 with* A(ρ) = l*, and let* δ<sup>k</sup> *be the solution of the following problem:*

$$\begin{aligned} \delta\_k &= \min\_{\sigma \in \mathcal{D}(\mathcal{H})} \quad 1 - F(\rho, \sigma) \\quad & \text{to } \quad \sigma \ge 0 \\ & & \text{tr}(\sigma) = 1 \\ & & \text{tr}[(M\_l^\dagger M\_l - M\_k^\dagger M\_k)\mathcal{E}(\sigma)] \le 0 \end{aligned}$$

*where if the problem is unsolved, then* δ<sup>k</sup> = +∞. *Then* δ = min<sup>k</sup>=<sup>l</sup> δ<sup>k</sup> *is the optimal robust bound of* ρ*.*

*Proof.* The proof is similar to Theorem 1.

*Remark 4.* One may wonder why checking ε-robustness and computing the optimal robust bound can always be reduced to an SDP. This is indeed implied by the *basic quantum mechanics postulate* of linearity; more specifically, all of the superoperators and measurements used in quantum machine learning algorithms are linear. In contrast, the functions represented by the neural networks in classical machine learning may be nonlinear as the pooling layer is not linear. As a result, the reduced optimization problem for the robustness verification is not convex (e.g. [32]). For overcoming this difficulty, many different methods have been developed to encode the nonlinear activation functions as linear constraints. Examples include NSVerify [33], MIPVerify [34], ILP [35] and ImageStar [13].

**Algorithm 1.** StateRobustnessVerifier(A, ε, ρ, l)

**Require:** A = (E, {M*k*}*<sup>k</sup>*∈C) is a well-trained quantum classifier, ε < 1 is a real number, (ρ, l) is an element of the training dataset of A

**Ensure: true** indicates ρ is ε-robust or **false** with an adversarial example σ indicates ρ is not ε-robust

1: **for each** k ∈ C and k = l **do**

2: By an SDP solver, compute δ*<sup>k</sup>* with an optimal state σ*<sup>k</sup>* in the SDP of Theorem 2 3: **end for**

4: Let δ = min*<sup>k</sup>* δ*<sup>k</sup>* and k<sup>∗</sup> = arg min*<sup>k</sup>* δ*<sup>k</sup>*


#### **6 Robustness Verification Algorithms**

In this section, we develop several algorithms for verifying the robustness of quantum classifiers based on the theoretic results presented in the last section.

First, let us consider the robustness of a given quantum state ρ. In many applications (as shown in our experiments in Sect. 7), we are required to check whether ρ is ε-robust for an arbitrarily given threshold ε. Note that once we computed the optimal robust bound δ, checking ε-robustness of ρ is equivalent to compare ε and δ; that is, ε ≤ δ if and only if ρ is ε-robust. Combining with this simple observation with Theorem 1, we obtain Algorithm 1 for checking the ε-robustness of ρ and finding the minimum adversarial perturbation δ caused by quantum noise. The main cost of Algorithm 1 incurs in solving SDPs in Line 2, which scales as O(n<sup>6</sup>.<sup>5</sup>) by interior-point methods [36], where n is the number of rows of the semidefinite matrix ρ in SDP, i.e., the dimension of Hilbert space of the quantum states in our case. As we need to apply an SDP solver for |C| − 1 times in Line 1, the total complexity is as follows.

**Theorem 3.** *The worst case complexity of Algorithm <sup>1</sup> is* <sup>O</sup>(|C| · <sup>n</sup><sup>6</sup>.<sup>5</sup>)*, where* <sup>n</sup> *is the dimension of input state* ρ *and* |C| *is the number of the set* C *of classes we are interested in.*

Now we turn to consider the robustness of a quantum classifier A. Algorithm 2 is designed for checking robustness of A by combining Algorithm 1 with Lemma 1 (see the discussion in the paragraph after Lemma 1). A major benefit of formal robustness verification for classical classifiers is perhaps that it can be used to detect a counter-example (adversarial example) for a given input (see e.g. [13–16]). This benefit is kept in Algorithm 2 for the robustness verification of quantum classifiers. In particular, we are able to extend the technique of adversarial training in classical machine learning [10] into the quantum case: an adversarial example σ is automatically generated once ε-robustness of ρ fails, and then by appending (σ, l) into the training dataset, we can retrain A to improve the robustness of the classifier.

**Algorithm 2.** RobustnessVerifier(A, ε, T)

**Require:** A = (E, {M*k*}*<sup>k</sup>*∈C) is a well-trained quantum classifier, ε < 1 is a real number, T = {(ρ*i*, l*i*)} is the training dataset of A

**Ensure:** The robust accuracy RA and a set R = {< σ*<sup>j</sup>* , i*<sup>j</sup>* >}, where for each j, ρ*<sup>j</sup>* is an ε-adversarial example of ρ*<sup>i</sup><sup>j</sup>* ; R can be an empty set if all states in T are ε-robust.

1: R = ∅ be an empty set. *// Recording adversarial examples and corresponding indexes of states in training dataset* T

```
2: for each (ρi, li) ∈ T do
```
3: Let p<sup>1</sup> and p<sup>2</sup> be the first and second largest elements of {tr(M† *<sup>k</sup>*M*k*E(ρ*i*))}*k*, respectively.

4: **if** <sup>√</sup>p<sup>1</sup> <sup>−</sup> <sup>√</sup>p<sup>2</sup> <sup>≤</sup> <sup>√</sup>2<sup>ε</sup> **then** *// Applying the robust bound in Lemma <sup>1</sup>* 5: **if** StateRobustnessVerifier (A, ε, ρ*i*, l*i*) == **false then** 6: σ be the output state of StateRobustnessVerifier (A, ε, ρ*i*, l*i*) 7: R = R ∪ {(σ, i)}


10: **end for** 11: **return** RA = 1 <sup>−</sup> <sup>|</sup>*R*<sup>|</sup>

<sup>|</sup>*<sup>T</sup>* <sup>|</sup> , <sup>R</sup> *//* <sup>|</sup>R<sup>|</sup> = 0 *if* <sup>R</sup> *is an empty set*

To analyze the complexity of Algorithm 2, we first see by Theorem 2 that for evaluating the robustness of A—computing its robust accuracy and finding its adversarial examples, one need to call Algorithm 1 for each quantum state in the training dataset, which costs <sup>O</sup>(|C| · <sup>n</sup><sup>6</sup>.<sup>5</sup>). Thus, the total complexity of robustness verification is <sup>O</sup>(|T|·|C|·n<sup>6</sup>.<sup>5</sup>), where <sup>|</sup>T<sup>|</sup> is the number of elements in the training dataset T. However, the robust bound given in Lemma 1 can help to speed up the process by quickly finding all potential non-robust states, as the complexity of finding the bound is only <sup>O</sup>(|C| · <sup>n</sup><sup>3</sup>), which is the cost of |C| times of the multiplication of two n × n matrices. In practice, this bound scales well, as confirmed by our experiments presented in Sect. 7. Therefore, a good strategy for implementing the robustness verification is that we first use robust bound to pick up all potential non-robust states from the given training dataset T and store them in a set T . Then we check all left candidates in the training dataset T one-by-one using Algorithm 1 and use a set R to record the found adversarial examples and the corresponding indexes of states. This strategy can significantly reduce the complexity to O(|T | · |C| · <sup>n</sup><sup>6</sup>.<sup>5</sup>). Indeed, our experiments show that the robust bound given in Lemma 1 scales very well in the sense of |T ||T|.

*Remark 5.* Thanks to the linearity of the quantum learning model determined by the basic postulate of quantum mechanics, the robustness verification of quantum classifiers can be done in an efficient way (with polynomial time complexity in the size of the input state). It is usually not the case in verifying the robustness of classical machine learning algorithms. For example, DNNs are often non-linear and non-convex, and verifying even some simple properties of them can be an NP-complete problem [37].

Surprisingly, the robustness verification problem for quantum classifiers becomes much harder if we are required to find adversarial examples in *pure states*. Roughly speaking, the reason is that the set of all pure states is not convex, and thus computing the optimal robust bound for pure states is not an SDP, as in Theorem 2. We can prove that it is a Quadratically Constrained Quadratic Program (QCQP), an optimization problem where both the objective function and the constraints are quadratic functions (see Appendix D of the extended version of this paper [31] for the proof), which is NP-hard. Algorithm 1 can be adapted to this pure state robustness verification by calling a QCQP solver instead of an SDP solver in Line 2. Subsequently, Algorithm 2 can use this new version of Algorithm 1 as a subroutine to compute the corresponding robust accuracy and find adversarial examples of pure states. We will evaluate the QCQP-based robustness verification in the case study of MNIST classification in which handwritten digits are encoded in pure states.

#### **7 Evaluation**

Algorithm 2 is implemented on *TensorFlow Quantum*—a platform of Google for designing and training quantum machine learning algorithms, by calling an SDP solver—CVXPY: Python Software for Disciplined Convex Programming [38]. This section aims to evaluate our approach with experiments on some concrete examples. This section is arranged as follows. In Subsects. 7.1–7.4, we present several well-trained quantum classifiers. Then the evaluation is carried out in Subsect. 7.5 by applying Algorithm 2 to check the robustness verification of these classifiers and find their adversarial examples if existing.

To demonstrate our method as sufficiently as possible, we check the robustness of four quantum classifiers. We begin with a "Hello World" example—qubits classification, and then we step in two quantum classifiers applied to real world tasks—quantum phase recognition and cluster excitation detection, which are both fundamental and hard problems in quantum physics. At last, to compare with classical robustness verification, we consider the classification of MNIST by encoding handwritten digital images into quantum data. *These experiments cover all illustrated examples of TensorFlow Quantum*.

#### **7.1 Quantum Bits Classification**

A "Hello World" example of quantum machine learning is quantum bits classification [7]. The aim is to implement a binary classification for regions on a single qubit, i.e., a perceptron for qubits. Specifically, two random normalized vectors |a and |b- (pure states) in the X-Z plane of the Bloch sphere are chosen. Around these two vectors, we randomly sample two sets of quantum data points; the objective is to learn a quantum gate to distinguish the two sets. A concrete instance of this type is shown in Fig. 2. In this example, the angles with |0- (Zaxis) of the two states |a and |b are θ<sup>a</sup> = 1 and θ<sup>b</sup> = 1.23, respectively; see the first figure in Fig. 2. Around these two vectors, we randomly sample two sets (one for category "a" and one for category "b") of quantum data points on the sphere, forming a dataset. The dataset consists of 800 samples for the training and 200 samples for the validation. As shown in Fig. 2, we use a parameterized rotation gate <sup>R</sup>y(θ) = <sup>e</sup>−iσ*y*θ/<sup>2</sup> and a measurement <sup>M</sup> <sup>=</sup> {M<sup>a</sup> <sup>=</sup> <sup>|</sup>0-0|, M<sup>b</sup> = |1-1|} to do the classification. Targeting to minimizing the MSE form of Eq. (8), we use Adam optimizer [39] to update θ. After training, we achieve both 100% training and validation accuracy, and the final parameter θ is 0.4835.

**Fig. 2.** Training model of quantum bits classification: the left figure shows the samples of the quantum training dataset represented on the Bloch sphere. Samples are divided into two categories, marked by red and yellow, respectively. The vectors are the states around which the samples were taken. The first part of the right figure is a parameterized rotation gate, whose job is to remove the super-positions in the quantum data. The second part is a measurement M along the Z-axis of the Bloch sphere converting the quantum data into classes. (Color figure online)

#### **7.2 Quantum Phase Recognition**

Quantum phase recognition (QPR) of one dimensional many-body systems has been attacked by quantum convolutional neural networks (QCNNs) proposed by Cong et al. [5]. Consider a Z<sup>2</sup> ×Z<sup>2</sup> symmetry-protected topological (SPT) phase P and the ground states of a family of Hamiltonians on spin-1/2 chain with open boundary conditions:

$$H = -J\sum\_{i=1}^{N-2} Z\_i X\_{i+1} Z\_{i+2} - h\_1 \sum\_{i=1}^{N} X\_i - h\_2 \sum\_{i=1}^{N-1} X\_i X\_{i+1}$$

where Xi, Z<sup>i</sup> are Pauli matrices [2] for the spin at site i, and the Z2×Z<sup>2</sup> symmetry is generated by Xeven(odd) = <sup>i</sup>∈even(odd) <sup>X</sup>i. The goal is to identify whether the ground state |ψ of H belongs to phase P when H is regarded as a function of (h1/J, h2/J). For small N, a numerical simulation can be used to exactly solve this problem [5]; See Fig. 4a in Appendix E of the extended version of this paper [31] for the exact phase boundary points (blue and red diamonds) between SPT phase and non-SPT (paramagnetic or antiferromagnetic) phase for N = 6. Thus the 6-qubit instance is an excellent testbed for different new methods and techniques of QPR. Here, we train a QCNN model to implement 6-qubit QPR in this setting.

To generate the dataset for training, we sample a serials of Hamiltonian H with h2/J = 0, uniformly varying h1/J from 0 to 1.2 and compute their corresponding ground states; see the gray line of Fig. 4a in Appendix E of the extended version of this paper [31]. For the testing, we uniformly sample a set of validation data from two random regions of the 2-dimensional space (h1/J, h2/J); see the two dashed rectangles of Fig. 4a. Finally, we obtain 1000 training data and 400 validation data. Our parameterized QCNN circuit is shown in Fig. 4b in Appendix E of the extended version of this paper [31], and the unitaries Ui, V<sup>j</sup> , F are parameterized with generalized Gell-Mann matrix basis [40]: U = exp(−i <sup>j</sup> θjΛ<sup>j</sup> ), where Λ<sup>j</sup> is a matrix and θ<sup>j</sup> is a real number; the total number of parameters θ<sup>j</sup> , Λ<sup>j</sup> is 114. For the outcome measurement of one qubit, we use measurement M = {M<sup>0</sup> = |+-+|, M<sup>1</sup> = |−-−|} to predict that input states belongs to P with output 0, where |±- = <sup>√</sup> 1 <sup>2</sup> (|0-±|1-). Targeting to minimizing the MSE form of Eq. (8), we use Adam optimizer to update the 114 parameters. After training, 97.7% training accuracy and 95.25% validation accuracy are obtained. At the same time, our classifier conducts a phase diagram (the colorful figure in Fig. 4a), where the learned phase boundary almost perfectly matches the exact one gotten by the numerical simulation. All these results indicate that our classifier is well-trained.

#### **7.3 Cluster Excitation Detection**

The task of cluster excitation detection is to train a quantum classifier to detect if a prepared cluster state is "excited" or not [7]. Excitations are represented with a X rotation on one qubit. A large enough rotation is deemed to be an excited state and is labeled by 0, while a rotation that isn't large enough is labeled by 1 and is not deemed to be an excited state. Here, we demonstrate this classification task with 6 qubits. We use the circuit shown in Fig. 5a of Appendix E in the extended version of this paper [31] to generate training (840) and validation (360) samples. The circuit generates a cluster state by performing a X rotation (we omit angle θ) on one qubit. The rotation angle θ is ranging from −π to π and if −π/2 ≤ θ ≤ π/2, the label of the output state is 1; otherwise, the label is 0. The classification circuit model (a quantum convolutional neural network) uses the same structure in TensorFlow Quantum [7], shown in Fig. 5b of Appendix E in the extended version of this paper [31]. The explicit parameterization of Ci, P<sup>j</sup> can be found in [7]. The final measurement M = {M<sup>0</sup> = |0-0|, M<sup>1</sup> = |1-1|}. Targeting to minimizing the MSE form of Eq. (8), we use Adam optimizer to update all Ci, P<sup>j</sup> . We achieve 99.76% training accuracy and 99.44% validation accuracy.

#### **7.4 The Classification of MNIST**

Handwritten digit recognition is one of the most popular tasks in the classical machine learning zoo. The archetypical training and validation data come from the MNIST dataset which consists of 55,000 training samples handwritten digits [41]. These digits have been labeled by humans as representing one of the ten digits from number 0 to 9, and are in the form of gray-scale images that contains 28 × 28 pixels. Each pixel has a grayscale value ranging from 0 to 255. Quantum machine learning has been used to distinguish a too simplified version of MNIST by downscaling the image sizes to 8 × 8 pixels. Subsequently, the numbers represented by this version of MNIST can not be perceptually recognized [7]. Here, we build up a quantum classifier to recognize a MNIST version of 16 × 16 pixels (see second column images of Fig. 3). As demonstrated in [7], we select out 700 images of number 3 and 700 images of number 6 to form our training (1000 images) and validation (400 images) datasets. Then we downscale those 28 <sup>×</sup> 28 images to 2<sup>4</sup> <sup>×</sup> <sup>2</sup><sup>4</sup> images (fitting the size of quantum data), and encode them into the pure states of 8 qubits via amplitude encoding. Amplitude encoding uses the amplitude of computational basis to represent vectors with normalization:

$$(x\_0, x\_2, \dots, x\_{n-1}) \to \sum\_{i=0}^{n-1} \frac{x\_i}{\sqrt{\sum\_{j=0}^{n-1} |x\_j|^2}} |i\rangle \dots$$

where {|i-} is a set of orthogonal basis of the 8 qubits state space. The normalization doesn't change the pattern of those images. For learning a quantum classifier, we use the QCNN model in Fig. 6 of Appendix E in the extended version of this paper [31] and use measurement M = {M<sup>0</sup> = |+-+|, M<sup>1</sup> = |−-−|}. The output of measurement M indicates the numbers: output 1 for number 3 and output 0 for number 6. The explicit parameterization of those Ci, P<sup>j</sup> can be found in [7]. Again we use Adam optimizer to update the model parameters minimizing the MSE form of Eq. (8). We finally achieve 98.4% training accuracy and 97.5% validation accuracy.

#### **7.5 Robustness Verification**

Now, we start to check the ε-robustness for the above four well-trained classifiers presented in the previous four subsections.

In practical applications, the value of robustness ε in Definition 3 represents the ability of state preparation by quantum controls. For example, the state-ofthe-art is that a single qubit can be prepared with fidelity 99.99% (e.g. [29,30]). Here, we choose four different values of ε in each experiment.

To show the scalability of our robust bound given in Lemma 1, we use it to develop an algorithm (Algorithm 3 in Appendix F of the extended version of this paper [31]) to under-approximate the robust accuracy, which is computed by Algorithm 2. Algorithm 3 is a subroutine of Algorithm 2 without calling an SDP solver (whenever a potential non-robust state can be detected by the robust bound in Lemma 1). We compare the verification times by Algorithms 2 and 3.


**Table 1.** Verification results of quantum bits classification

**Table 2.** Verification results of quantum phase recognition.


The experiments are done on a computer with the following configurations: Intel(R) Core(TM) i7-9700 CPU @ 3.00 GHz × 8 Processor, 15.8 GiB Memory, Ubuntu 18.04.5 LTS, with CVXPY: Python Software for Disciplined Convex Programming [38] for solving SDP, and a SciPy solver for finding the minimum of constrained nonlinear multivariable function for solving QCQP.

The experimental results are given in Tables 1, 2, 3 and 4. As an example, we illustrate the details of the result for the case of ε = 0.001 in Table 1. First, we only apply our robust bound in Lemma 1 to pick up all potential non-robust states from the 800 points in the training dataset. Then 95 points are left. Thus, the under-approximation of the robust accuracy computed by Algorithm 3 (in Appendix F of the extended version of this paper [31]) is 88.13%. Next, we check the 0.001-robustness by Algorithm 2. Indeed, only 80 of the points detected by the above robust bound are non-robust and the exact robust accuracy is 90.00%. We also compare the verification time of the two approaches to the robust accuracy. See the second column in Table 1 for the detail, and other experiment results of ε-robustness are also summarized in the same table. Tables 1, 2, 3 and 4 for the verification results show that in all of these experiments, the robust bound obtained in Lemma 1 scales very well, and the robustness verification by Algorithm 3 costs significantly less time (<2 s) than the way of computing the optimal robust bound by Algorithm 2. For example, for quantum phase recog-


**Table 3.** Verification results of cluster excitation detection

**Table 4.** Verification results of the classification of MNIST


nition, for ε = 0.0001, 0.0002 and 0.0003, the under-approximation of the robust accuracy is the same as the real value. Even for the last case of ε = 0.0004, only the 0.1% difference is got. Furthermore, from the tables, the verification time of Algorithm 2 is increasing with the value of ε, while the running time of the method by the robust bound is almost unchanged. This is because the former algorithm uses an SDP or QCQP solver to search all possible adversarial examples for the potential non-robust states picked up by the robust bound, and the number of these states are growing up with the value of ε. These counterexamples detected by the algorithm confirm that our robustness framework is effective. For instance, see Fig. 3 for two visualized adversarial examples generated by Algorithm 2 with a QCQP solver. As we can see, the benign and adversarial images are perceptually similar. This also proves that our robustness verification algorithm can detect not only quantum but also classical adversarial examples.

**Fig. 3.** Two training states and their adversarial examples generated by Algorithm 2 with a QCQP solver: the first column images are 28×28 benign data from MNIST; The second column shows the two downscaled 16 × 16 grayscale images; The last column images are decoded from adversarial examples founded by Algorithm 2. The third column images are the grayscale difference between benign and adversarial images.

#### **8 Conclusion**

In this work, we initiate the research of the formal robustness verification of quantum machine learning algorithms against unknown quantum noise. We found an analytical robustness bound which can be efficiently computed to under-approximate the robust accuracy in practical applications. Furthermore, we developed a robustness verification algorithm that can exactly verify the εrobustness of quantum machine learning algorithms and provides useful counterexamples for the adversarial training.

For topics for future research, it should be useful in practical applications to find an efficient method that over-approximates the robust accuracy of quantum classifiers. Combined with the under-approximation approach developed in this work, it can help us to more accurately and fast estimate the robust accuracy. In classical machine learning, there exist some works in the literature to achieve this task. For instance, ImageStars, a new set representation, was introduced in [13] to perform efficient set-based analysis by combining operations on concrete images with linear programming, which leads to efficient over-approximative analysis of classical convolutional neural networks.

Tensor networks are one of the best-known data structures for implementing large-scale quantum classifiers (e.g. QCNNs with 45 qubits in [5]). For practical applications, we are going to incorporate tensor networks into our robustness verification algorithm so that it can scale up to achieve the demand of NISQ devices (of ≥50 qubits).

More generally, further investigations are required to better understand the role of robustness in quantum machine learning, especially through more experiments on real world applications like learning phases of quantum many-body systems.

**Acknowledgment.** We would like to thank the anonymous reviewers for their insightful comments. This work was partly supported by the National Key R&D Program of China (Grant No: 2018YFA0306701), the National Natural Science Foundation of China (Grant No: 61832015) and the Australian Research Council (ARC) under grant No.DP210102449.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### BDD4BNN**: A BDD-Based Quantitative Analysis Framework for Binarized Neural Networks**

Yedi Zhang1, Zhe Zhao1, Guangke Chen1, Fu Song1,2(B) , and Taolue Chen3

> <sup>1</sup> ShanghaiTech University, Shanghai, China songfu@shanghaitech.edu.cn

<sup>2</sup> Shanghai Engineering Research Center of Intelligent Vision and Imaging, Shanghai, China <sup>3</sup> Birkbeck, University of London, London, UK

**Abstract.** Verifying and explaining the behavior of neural networks is becoming increasingly important, especially when they are deployed in safety-critical applications. In this paper, we study verification and interpretability problems for Binarized Neural Networks (BNNs), the 1-bit quantization of general realnumbered neural networks. Our approach is to encode BNNs into Binary Decision Diagrams (BDDs), which is done by exploiting the internal structure of the BNNs. In particular, we translate the input-output relation of blocks in BNNs to cardinality constraints which are in turn encoded by BDDs. Based on the encoding, we develop a quantitative framework for BNNs where precise and comprehensive analysis of BNNs can be performed. We demonstrate the application of our framework by providing quantitative robustness analysis and interpretability for BNNs. We implement a prototype tool BDD4BNN and carry out extensive experiments, confirming the effectiveness and efficiency of our approach.

### **1 Introduction**

Deep neural networks (DNNs) have achieved human-level performance in several tasks, and are increasingly being incorporated into various application domains such as autonomous driving [4] and medical diagnostics [53]. Modern DNNs usually contain a great many parameters which are typically stored as 32/64-bit floating-point numbers, and require a massive amount of floating-point operations to compute the output for a single input [60]. As a result, it is often challenging to deploy them on resourceconstrained, embedded devices. To mitigate the issue, quantization, which quantizes 32/64-bit floating-points to low bit-width fixed-points (e.g., 4-bits) with little accuracy loss [23], emerges as a promising technique to reduce resource requirements. In particular, binarized neural networks (BNNs) [27] represent the case of 1-bit quantization using the bipolar binaries ±1. BNNs can drastically reduce memory storage and execution time with bit-wise operations, hence substantially improve the time and energy efficiency. BNNs have been demonstrated to achieve a high accuracy for a wide variety of applications [34,41,52].

This work is supported by the National Natural Science Foundation of China (NSFC) under Grants No.: 62072309, and an oversea grant from the State Key Laboratory of Novel Software Technology, Nanjing University (KFKT2018A16).

DNNs have been shown to lack robustness [11,14,36,49,59] and interpretability of the predictions they make [25,43]. Various formal techniques and heuristics have been proposed to analyze DNNs and interpret their behaviors, most of which focus on *real-numbered* DNNs only. Verification of *quantized* DNNs has not been thoroughly explored so far, although recent results have highlighted its importance: it was shown that a quantized DNN does not necessarily preserve the properties satisfied by the realnumbered DNN before quantization [14,22]. Indeed, the fixed-point number semantics effectively yields a discrete state space for the verification of quantized DNNs whereas real-numbered DNNs feature a continuous state space. The discrepancy could invalidate current verification techniques for real-numbered DNNs when they are directly applied to the quantized counterparts (e.g., both false negative and false positive could occur). Therefore, specialized techniques are required for rigorously verifying quantized DNNs.

Broadly speaking, the existing techniques for quantized DNNs make use of constraint solving which is based on either SAT/SMT or (reduced, ordered) binary decision diagrams (BDDs). A majority of work resorts to SAT/SMT solving. For the 1-bit quantization (i.e., BNNs), typically BNNs are transformed into Boolean formulas where SAT solving is harnessed [12,33,45,46]. Some recent work also studies variants of BNNs [28,48], i.e., BNNs with ternary weights. For quantized DNNs with multiple bits (i.e., fixed-points), it is natural to encode them as quantifier-free SMT formulas, e.g., using bit-vector and fixed-point theories [7,22,24], so that off-the-shelf SMT solvers can be leveraged. In another direction, BDD-based approaches currently can tackle BNNs only [54]. In a nutshell, they encode a BNN and an input region as a BDD, based on which various analyses can be performed via queries on the BDD. The crux of the approach is how to generate the BDD efficiently. In the work [54], the BDD is constructed by BDD learning [44], thus, currently limited to toy BNNs (e.g., 64 input size, 5 hidden neurons, and 2 output size) with relatively small input regions.

On the other hand, existing work mostly focuses on *qualitative* verification, which asks whether there exists an input *x* (in a specified region) for a neural network such that a property (e.g., local robustness) is violated. In many practical applications, checking only the existence is not sufficient. Indeed, for local robustness, such an (adversarial) input almost surely exists which makes a qualitative answer less meaningful. Instead, *quantitative* verification, which asks how often a property φ is satisfied or violated, is far more useful yet more challenging as it could provide a probabilistic guarantee of the behavior of neural networks. Such a quantitative guarantee is essential to certify, for instance, certain implementations of neural network based perceptual components against safety standards of autonomous vehicles [29,32]. Quantitative analysis of general neural networks, however, is challenging, hence received little attention and for which the results are rather limited so far. DeepSRGR [69] presented an abstract interpretation based quantitative robustness verification approach for DNNs which is sound but incomplete. For BNNs, approximate SAT model-counting solvers (SAT) are leveraged [6,47] based on the SAT encoding for the qualitative counterpart. Though probably approximately correct (PAC) style guarantees can be provided, verification cost is usually prohibitively high to achieve higher precision and confidence.

**Main Contributions.** We propose a BDD-based framework BDD4BNN to support quantitative analysis of BNNs. The main challenge is how to efficiently build BDDs from BNNs [47]. In contrast to previous work [54] which is learning-based and largely treats the BNN as a blackbox, we *directly* encode a BNN and the associated input region into BDDs. In a nutshell, a BNN is a sequential composition of multiple internal blocks and one output block. Each block comprises 3 layers and captures a function *f* : {+1, −1} *<sup>n</sup>* → {+1, −1} *<sup>m</sup>*, where *n* (resp. *m*) denotes the number of inputs (resp. outputs) of the block. Technically, the function *f* can be alternatively rewritten as a function over the standard Boolean domain, i.e., *f* : {0, 1} *<sup>n</sup>* → {0, 1} *<sup>m</sup>*. A key stepping-stone of our encoding is the observation that the *i*-th output *yi* of the block can be captured by a cardinality constraint of the form *n <sup>j</sup>*=<sup>1</sup> *<sup>j</sup>* ≥ *k* such that *yi* = +1 ⇔ *n <sup>j</sup>*=<sup>1</sup> *<sup>j</sup>* ≥ *k*, where each literal *<sup>j</sup>* is either *xj* or ¬*xj* for the input variable *xj*, and *k* is a constant. We then present an algorithm to encode a cardinality constraint *n <sup>j</sup>*=<sup>1</sup> *<sup>j</sup>* ≥ *k* as a BDD with *O*((*n* − *k*) · *k*) nodes in *O*((*n* − *k*) · *k*) time. As a result, the input-output relation of each block can be encoded as a BDD, the composition of which yields the BDD for the entire BNN. A distinguished advantage of our BDD encoding lies in its support of incremental encoding. In particular, when different input regions are of interest, there is no need to construct the BDD of the entire BNN from scratch.

Encoding BNNs as BDDs enables a wide variety of applications in security analysis and decision explanation of BNNs. In this paper, we highlight two of them within our framework, i.e., robustness analysis and interpretability. It was shown that DNNs have been suffering from poor robustness to adversarial examples [49,50,59]. We consider two quantitative variants of the problem: (1) how many adversarial examples does the BNN have in the input region, and (2) how many of them are misclassified to each class? We further provide an algorithm to incrementally compute the (locally) maximal Hamming distance within which the BNN satisfies the desired robustness properties.

Interpretability is an issue arisen as a result of the blackbox nature of DNNs [25,43]. In application domains such as medical diagnosis, understanding the decisions made by DNNs is a must. We consider two problems: (1) why some inputs are (mis)classified into a class by the BNN and (2) are there any essential features in the input region that are common for all samples classified into a class?

**Experimental Results.** We implement our framework as a prototype tool BDD4BNN using the CUDD package [58], which scales to BNNs with up to 4 internal blocks, 200 hidden neurons, and 784 input size. To the best of our knowledge, it is the first work to precisely and quantitatively analyze such large BNNs that go significantly beyond the state-of-the-art. The experimental results show that BDD4BNN is significantly more efficient and scalable than the learning-based technique [54]. Furthermore, we demonstrate how BDD4BNN can be used in quantitative robustness analysis and decision explanation of BNNs. For quantitative robustness analysis, our experimental results show that BDD4BNN is considerably (5× to 1, 340×) faster and more accurate than the state-of-the-art approximate SAT-based approach [6]. It can also compute precisely the distribution of predicated classes of the images in the input region as well as the locally maximal Hamming distances on several BNNs. For decision explanation, we show the effectiveness of BDD4BNN in computing prime-implicant explanations and essential features of the given input region for some target classes. Note that this work focuses on quantitative verification and interpretability of BNNs and may underperform SAT/SMT-based methods [12,33,45,46] for qualitative verification of BNNs.

In general, our main contributions can be summarized as follows.

**Fig. 1.** Architecture of a BNN with *d* + 1 blocks


### **2 Preliminaries**

In this section, we briefly introduce binarized neural networks (BNNs) and (reduced, ordered) binary decision diagrams (BDDs).

We denote by <sup>R</sup>, <sup>N</sup>, <sup>B</sup>, and <sup>B</sup>±<sup>1</sup> the set of real numbers, the set of natural numbers, the standard Boolean domain {0, <sup>1</sup>} and the integer set {+1, <sup>−</sup>1}. For *<sup>n</sup>* <sup>∈</sup> <sup>N</sup>, we denote by [*n*] the set {1, ··· , *n*}. We will use *W*, *W* ,... to denote (2-dimensional) matrices, *x*, *v*, ··· to denote (row) vectors, and *x*, *v*,... to denote scalars. We denote by *Wi*,: and *W*:, *<sup>j</sup>* the *i*-th row and *j*-th column of the matrix *W*. Similarly, we denote by *x<sup>j</sup>* and *Wi*, *<sup>j</sup>* the *j*-th entry of *x* and *Wi*,: respectively. In this work, Boolean values 1/0 will be used as integers 1/0 in arithmetic computations without typecasting.

#### **2.1 Binarized Neural Networks**

A binarized neural network (BNN) [27] is a neural network where weights and activations are predominantly binarized over the domain <sup>B</sup>±1. In this work, we consider feed-forward BNNs. As shown in Fig. 1, a BNN can be seen as a sequential composition of several internal blocks and one output block. Each internal block comprises 3 layers: a linear layer (LIN), a batch normalization layer (BN), and a binarization layer (BIN). The output block comprises a linear layer and an ARGMAX layer. Note that the input/output of internal blocks and the input of the output block are all vectors over <sup>B</sup>±1.


**Table 1.** Definitions of layers in BNNs, where *nd*+<sup>2</sup> = *s* and arg max(·) returns the index of the largest entry which occurs first.

**Definition 1.** *A BNN* <sup>N</sup> : <sup>B</sup>*n*<sup>1</sup> <sup>±</sup><sup>1</sup> <sup>→</sup> <sup>B</sup>*<sup>s</sup> with s classes is given by a tuple of blocks* (*t*1, ··· , *td*, *td*+1) *such that* N = *td*+<sup>1</sup> ◦ *td* ◦···◦ *t*1,


*where tbin <sup>i</sup> , tbn <sup>i</sup> , tlin <sup>i</sup> for i* ∈ [*d*]*, tlin <sup>d</sup>*+<sup>1</sup> *and tam <sup>d</sup>*+<sup>1</sup> *are given in Table 1.*

Intuitively, a LIN layer is a linear transformation. A BN layer following a LIN layer is used to standardize and normalize the output of the LIN layer. A BIN layer is used to binarize the real-numbered output vector of the BN layer. In this work, we consider the sign function which is widely used in BNNs to binarize real-numbered vectors. An ARGMAX layer follows a LIN layer and outputs the index of the largest entry as the predicted class which is represented by a one-hot vector. (In case there is more than one such entry, the first one is returned.) Formally, given a BNN N = (*t*1, ··· , *td*, *td*+1) and an input *<sup>x</sup>* <sup>∈</sup> <sup>B</sup>*<sup>n</sup>*<sup>1</sup> ±<sup>1</sup>, <sup>N</sup>(*x*) <sup>∈</sup> <sup>B</sup>*<sup>s</sup>* is a one-hot vector in which the index of the non-zero entry is the predicated class.

#### **2.2 Binary Decision Diagrams**

A BDD [9] is a rooted acyclic directed graph where non-terminal nodes *v* are labeled by Boolean variables var(*v*) and terminal nodes (leaves) *v* are labeled with values val(*v*) ∈ B, referred to as the 1-leaf and the 0-leaf respectively. Each non-terminal node *v* has two outgoing edges: hi(*v*) meaning var(*v*) = 1 and lo(*v*) meaning var(*v*) = 0. We will also refer to hi(*v*) and lo(*v*) as the hi and lo children of *v* respectively. Moreover, assuming that *x*1, ··· , *xm* is the variable ordering, for each node *v* with var(*v*) = *xi* and each *v* ∈ {hi(*v*), lo(*v*)} with var(*v* ) = *xj*, we have *i* < *j*. In the graphical representation of BDDs, hi(*v*) and lo(*v*) are depicted by solid and dashed lines respectively. Multi-Terminal Binary Decision Diagrams (MTBDDs) are a variant of BDDs in which the

**Fig. 2.** The reduced BDD for *f*(*x*1, *y*1, *x*2, *y*2) = (*x*<sup>1</sup> ⇔ *y*1) ∧ (*x*<sup>2</sup> ⇔ *y*2)

**Table 2.** Some basic BDD operations, where *op* ∈ {And, <sup>O</sup>r, <sup>X</sup>or, <sup>X</sup>nor}


terminal nodes are not restricted to be 0 or 1. A BDD is *reduced* if it (1) has only one 1-leaf and one 0-leaf, (2) does not contain a node *v* such that hi(*v*) = lo(*v*), and (3) does not contain two distinct non-terminal nodes *v* and *v* such that var(*v*) = var(*v* ), hi(*v*) = hi(*v* ) and lo(*v*) = lo(*v* ). For example, Fig. 2 shows the reduced BDD for the Boolean function *f*(*x*1, *y*1, *x*2, *y*2) = (*x*<sup>1</sup> ⇔ *y*1) ∧ (*x*<sup>2</sup> ⇔ *y*2). Hereafter, we assume that BDDs are reduced.

Bryant [9] showed that BDDs can serve as a canonical form of Boolean functions. Given a BDD over variables *x*1, ··· , *xm*, each non-terminal node *v* with var(*v*) = *xi* represents a Boolean function *fv* = (*xi* ∧ *f*hi(*<sup>v</sup>*)) ∨ (¬*xi* ∧ *f*lo(*<sup>v</sup>*)). Operations on Boolean functions can usually be efficiently implemented via manipulating their BDD representations. A good variable ordering is crucial for the performance of BDD manipulations while the problem of finding an optimal ordering for a function is NP-hard. To store and manipulate BDDs efficiently, the nodes are stored in a hash table and the recent computed results are stored in a cache to avoid duplicated computations. In this work, we will use some basic BDD operations such as ITE for If-Then-Else, Xor for exclusive-OR, Xnor for exclusive-NOR (i.e., *<sup>a</sup>* <sup>X</sup>nor *<sup>b</sup>* <sup>=</sup> <sup>¬</sup>(*<sup>a</sup>* <sup>X</sup>or *<sup>b</sup>*)) and SatAll(*fv*) for the set of all solutions of the Boolean formula *fv*. We denote by <sup>L</sup>(*v*) the set SatAll(*fv*). For easy reference, more operations are given in Table 2. By *op*(*v*, *v* ) we denote the operation Apply(*v*, *v* , *op*).

#### **3** BDD4BNN **Design**

#### **3.1** BDD4BNN **Overview**

An overview of BDD4BNN is depicted in Fig. 3. BDD4BNN comprises four main components: Region2BDD, BNN2CC, BDD Model Builder, and Query Engine. For a fixed BNN N = (*t*1, ··· , *td*, *td*+1) and a region *R* of the input space of N, BDD4BNN constructs the BDDs (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*s*] to encode the input-output relation of N in the region *R*, where the BDD *Gout <sup>i</sup>* corresponds to the class *i* ∈ [*s*]. Technically, the region *R* is partitioned into *s* parts represented by (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*<sup>s</sup>*]. For each property query, BDD4BNN analyzes (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*s*] and outputs the query result.

**Fig. 3.** Overview of BDD4BNN

**Fig. 4.** Graphic representation of BDDs using Algorithm 1

The general workflow of our approach is as follows. First, Region2BDD builds up a BDD *Gin <sup>R</sup>* from the region *R* which represents the desired input space of N for analysis. Second, BNN2CC transforms each block of the BNN N into a set of cardinality constraints (CCs) similar to [6,46]. Third, BDD Model Builder builds the BDDs (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*s*] from all the cardinality constraints and the BDD *Gin <sup>R</sup>* . Finally, Query Engine answers queries by analyzing the BDDs (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*<sup>s</sup>*]. Our Query Engine currently supports two types of application queries: robustness analysis and interpretability.

In the rest of this section, we first introduce the key sub-component CC2BDD, which provides an encoding of cardinality constraints into BDDs. We then provide details of the components Region2BDD, BNN2CC, and BDD Model Builder. The Query Engine will be described in Sect. 4.

#### **3.2 CC2BDD: Cardinality Constraints to BDDs**

A *cardinality constraint* is a constraint of the form *n <sup>j</sup>*=<sup>1</sup> *<sup>j</sup>* ≥ *k* over a vector *x* of Boolean variables with length *n*, where the literal *<sup>j</sup>* is either *x<sup>j</sup>* or ¬*x<sup>j</sup>* for each *j* ∈ [*n*]. Note that constraints of the form *n <sup>j</sup>*=<sup>1</sup> *<sup>j</sup>* > *k*, *n <sup>j</sup>*=<sup>1</sup> *<sup>j</sup>* ≤ *k* and *n <sup>j</sup>*=<sup>1</sup> *<sup>j</sup>* < *k* are equivalent to *n <sup>j</sup>*=<sup>1</sup> *<sup>j</sup>* ≥ *k* + 1, *n <sup>j</sup>*=<sup>1</sup> ¬ *<sup>j</sup>* ≥ *n* − *k* and *n <sup>j</sup>*=<sup>1</sup> ¬ *<sup>j</sup>* ≥ *n* − *k* + 1, respectively. We assume that 1 (resp. 0) is a special cardinality constraint that always holds (resp. never holds).

To encode *n <sup>j</sup>*=<sup>1</sup> *<sup>j</sup>* ≥ *k* as a BDD, we observe that all the possible solutions of *n <sup>j</sup>*=<sup>1</sup> *<sup>j</sup>* ≥ *k* can be compactly represented by a BDD-like graph shown in Fig. 4(a), where each node is labeled by a literal, and a solid (resp. dashed) edge from a node labeled by *<sup>j</sup>* means that the value of the literal *<sup>j</sup>* is 1 (resp. 0). Thus, each path from the 1-node to the 1-leaf through the *<sup>j</sup>*-node (where 1 ≤ *j* ≤ *n*) captures a set of valuations where *<sup>j</sup>* followed by a (horizontal) dashed line is set to be 0 while *<sup>j</sup>* followed by

#### **Algorithm 1:** BDD Construction for cardinality constraints

 **Proc** CC2BDD(CC : *n <sup>j</sup>*=<sup>1</sup> *<sup>j</sup>* ≥ *k*) *Gk*+1,<sup>1</sup> <sup>=</sup> *Gk*+1,<sup>2</sup> <sup>=</sup> ··· <sup>=</sup> *Gk*+1,*n*−*k*+<sup>1</sup> <sup>=</sup> <sup>C</sup>onst(1); *<sup>G</sup>*1,*n*−*k*+<sup>2</sup> <sup>=</sup> *<sup>G</sup>*2,*n*−*k*+<sup>2</sup> <sup>=</sup> ··· <sup>=</sup> *Gk*,*n*−*k*+<sup>2</sup> <sup>=</sup> <sup>C</sup>onst(0); **for** (*i* = *k*; *i* ≥ 1; *i* − −) **do for** (*j* = *n* − *k* + 1; *j* ≥ 1; *j* − −) **do if** (*<sup>i</sup>*+*j*−<sup>1</sup> == *x<sup>i</sup>*+*j*−<sup>1</sup>*)* **then** *Gi*, *<sup>j</sup>* = ITE(*x<sup>i</sup>*+*j*−<sup>1</sup>,*Gi*+1, *<sup>j</sup>*,*Gi*, *<sup>j</sup>*+1); **else** *Gi*, *<sup>j</sup>* = ITE(*x<sup>i</sup>*+*j*−<sup>1</sup>,*Gi*, *<sup>j</sup>*+<sup>1</sup>,*Gi*+1, *<sup>j</sup>*); **return** *G*1,<sup>1</sup>

a (vertical) solid line is set to be 1, and all the other literals which are not along the path can take arbitrary values. Clearly, for each of these valuations, there are at least *k* positive literals, hence the constraint *n <sup>j</sup>*=<sup>1</sup> *<sup>j</sup>* ≥ *k* holds.

Based on the above observation, we build the BDD for *n <sup>j</sup>*=<sup>1</sup> *<sup>j</sup>* ≥ *k* using Algorithm 1. It builds a BDD for each node in Fig. 4(a), row-by-row (the index *i* in Algorithm 1) and from right to left (the index *j* in Algorithm 1). For each node at the *i*-th row and *j*-th column, the label of the node must be the literal *i*+*j*−1. We build the BDD *Gi*, *<sup>j</sup>* = ITE(*xi*+*j*−1,*Gi*+1, *<sup>j</sup>*,*Gi*, *<sup>j</sup>*+1) if *i*+*j*−<sup>1</sup> is of the form *xi*+*j*−<sup>1</sup> (Line 6), otherwise we build the BDD *Gi*, *<sup>j</sup>* = ITE(*xi*+*j*−1,*Gi*, *<sup>j</sup>*+1,*Gi*+1, *<sup>j</sup>*) (Line 7). Finally, we obtain the BDD *G*1,<sup>1</sup> that encodes the solutions of *n <sup>j</sup>*=<sup>1</sup> *<sup>j</sup>* ≥ *k*. Consider *x*1+¬*x*2+*x*3+¬*x*4+*x*5+¬*x*<sup>6</sup> ≥ 3, Fig. 4(b) shows its BDD by Algorithm 1.

**Lemma 1.** *For each cardinality constraint n <sup>j</sup>*=<sup>1</sup> *<sup>j</sup>* ≥ *k, a BDD G with O*((*n* − *k*) · *k*) *nodes can be computed in O*((*n*−*k*)· *k*) *time such that* L(*G*) *is the set of all the solutions of n <sup>j</sup>*=<sup>1</sup> *<sup>j</sup>* ≥ *k.*

Compared with prior works [8,42] which transform general arithmetic constraints into BDDs, we devise a dedicated BDD encoding algorithm for the cardinality constraints without applying reduction, hence it is more efficient.

#### **3.3 Region2BDD: Input Regions to BDDs**

In this paper, we consider the following two types of input regions.


Note that both *R*(*u*, *n*1) and *R*(*u*, [*n*1]) denote the entire input space B*<sup>n</sup>*<sup>1</sup> ±1.

Recall that each input sample is an element from B*<sup>n</sup>*<sup>1</sup> ±<sup>1</sup>. To represent the region *<sup>R</sup>* by a BDD, we transform each value ±1 into a Boolean value 1/0. To this end, for each input *<sup>u</sup>* <sup>∈</sup> <sup>B</sup>*<sup>n</sup>*<sup>1</sup> <sup>±</sup><sup>1</sup>, we create a new sample *<sup>u</sup>*(*b*) <sup>∈</sup> <sup>B</sup>*<sup>n</sup>*<sup>1</sup> such that for every *<sup>i</sup>* <sup>∈</sup> [*n*1], *<sup>u</sup><sup>i</sup>* <sup>=</sup> 2*u*(*b*) *<sup>i</sup>* − 1. Therefore, *R*(*u*,*r*) and *R*(*u*, *I*) will be represented by *R*(*u*(*b*) ,*r*) and *R*(*u*(*b*) , *I*), respectively. The transformation functions *t lin <sup>i</sup>* , *t bn <sup>i</sup>* , *t bin <sup>i</sup>* and *t am <sup>d</sup>*+<sup>1</sup> of the LIN, BN, BIN, and ARGMAX layers (cf. Table 1) will be handled accordingly. Note that for convenience, vectors over the Boolean domain B may be directly given by *u* or *x* when it is clear from the context.

**Region Encoding Under Hamming Distance.** Given an input *<sup>u</sup>* <sup>∈</sup> <sup>B</sup>*<sup>n</sup>*<sup>1</sup> and an integer *r*, the region *R*(*u*,*r*) can be expressed by a cardinality constraint *n*1 *<sup>j</sup>*=<sup>1</sup> *<sup>j</sup>* ≤ *r* (which is equivalent to *n*1 *<sup>j</sup>*=<sup>1</sup> ¬ *<sup>j</sup>* ≥ *n*<sup>1</sup> − *r*), where for every *j* ∈ [*n*1], *<sup>j</sup>* = *x<sup>j</sup>* if *u<sup>j</sup>* = 0, otherwise *<sup>j</sup>* = ¬*xj*. For instance, consider *u* = (1, 1, 1, 0, 0) and *r* = 2, we have:

HD(*u*, *x*) = 1 ⊕ *x*<sup>1</sup> + 1 ⊕ *x*<sup>2</sup> + 1 ⊕ *x*<sup>3</sup> + 0 ⊕ *x*<sup>4</sup> + 0 ⊕ *x*<sup>5</sup> = ¬*x*<sup>1</sup> + ¬*x*<sup>2</sup> + ¬*x*<sup>3</sup> + *x*<sup>4</sup> + *x*5.

Thus, *R*((1, 1, 1, 0, 0), 2) can be expressed by the cardinality constraint ¬*x*1+¬*x*2+¬*x*3+ *x*<sup>4</sup> + *x*<sup>5</sup> ≤ 2, or equivalently *x*<sup>1</sup> + *x*<sup>2</sup> + *x*<sup>3</sup> + ¬*x*<sup>4</sup> + ¬*x*<sup>5</sup> ≥ 3.

By Algorithm 1, the cardinality constraint of *R*(*u*,*r*) can be encoded by the BDD *Gin <sup>u</sup>*,*r*, such that L(*Gin <sup>u</sup>*,*r*) = *R*(*u*,*r*). Following Lemma 1, we get that:

**Lemma 2.** *For an input region R given by an input <sup>u</sup>* <sup>∈</sup> <sup>B</sup>*n*<sup>1</sup> *and an integer r, a BDD Gin <sup>u</sup>*,*<sup>r</sup> with O*(*r*·(*n*<sup>1</sup> −*r*)) *nodes can be computed in O*(*r*·(*n*<sup>1</sup> −*r*)) *time such that* L(*Gin <sup>u</sup>*,*r*) = *R*(*u*,*r*)*.*

**Region Encoding Under Fixed Indices.** Given an input *<sup>u</sup>* <sup>∈</sup> <sup>B</sup>*n*<sup>1</sup> and a set of indices *<sup>I</sup>* <sup>⊆</sup> [*n*1], the region *<sup>R</sup>*(*u*, *<sup>I</sup>*) <sup>=</sup> {*<sup>x</sup>* <sup>∈</sup> <sup>B</sup>*n*<sup>1</sup> | ∀*<sup>i</sup>* <sup>∈</sup> [*n*1] \ *<sup>I</sup>*. *<sup>u</sup><sup>i</sup>* <sup>=</sup> *<sup>x</sup>i*} can be represented by the BDD *Gin <sup>u</sup>*,*<sup>I</sup>* - <sup>A</sup>nd*i*∈[*n*1]\*<sup>I</sup>* (*u<sup>i</sup>* == 1)?Var(*xi*):Not(Var(*xi*)) . Intuitively, *Gin <sup>u</sup>*,*<sup>I</sup>* states that the value at the index *i* ∈ [*n*1] \ *I* should be the same as the one in *u* while the value at the index *i* ∈ *I* is unrestricted. For instance, consider *u* = (1, 0, 0, 0) and *I* = {3, 4}, we have:

*R*((1, 0, 0, 0), {3, 4}) = {(1, 0, 0, 0), (1, 0, 0, 1), (1, 0, 1, 0), (1, 0, 1, 1)} = *x*<sup>1</sup> ∧ ¬*x*2.

**Lemma 3.** *For an input region R given by an input <sup>u</sup>* <sup>∈</sup> <sup>B</sup>*<sup>n</sup>*<sup>1</sup> *and indices I* <sup>⊆</sup> [*n*1]*, a BDD Gin <sup>u</sup>*,*<sup>I</sup> with O*(*n*<sup>1</sup> − |*I*|) *nodes can be computed in O*(*n*1) *time such that* L(*Gin u*,*I* ) = *R*(*u*, *I*)*.*

#### **3.4 BNN2CC: BNNs to Cardinality Constraints**

As mentioned before, to encode the BNN N = (*t*1, ··· , *td*, *td*+1) as BDDs, we transform the BNN N into cardinality constraints from which the desired BDDs (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*s*] are constructed. To this end, we first transform each internal block *ti* : B*ni* <sup>±</sup><sup>1</sup> <sup>→</sup> <sup>B</sup>*ni*+<sup>1</sup> <sup>±</sup><sup>1</sup> into *ni*+<sup>1</sup> cardinality constraints, each of which corresponds to one of the outputs of *ti*. Then we transform the output block *td*+<sup>1</sup> : B*nd*+<sup>1</sup> <sup>±</sup><sup>1</sup> <sup>→</sup> <sup>B</sup>*<sup>s</sup>* into *<sup>s</sup>*(*<sup>s</sup>* <sup>−</sup> 1) cardinality constraints, where one output class yields (*s* − 1) cardinality constraints.

For each vector-valued function *t*, we denote by *t*<sup>↓</sup> *<sup>j</sup>* the (scalar-valued) function returning the *j*-th entry of the output of *t*.

**Transformation for Internal Blocks.** Consider the internal block *ti* : B*ni* <sup>±</sup><sup>1</sup> <sup>→</sup> <sup>B</sup>*ni*+<sup>1</sup> <sup>±</sup><sup>1</sup> for *<sup>i</sup>* <sup>∈</sup> [*d*]. Recall that for every *<sup>j</sup>* <sup>∈</sup> [*ni*+1] and *<sup>x</sup>* <sup>∈</sup> <sup>B</sup>*ni* <sup>±</sup>1, *ti*<sup>↓</sup> *<sup>j</sup>*(*x*) <sup>=</sup> *<sup>t</sup> bin <sup>i</sup>* (*t bn <sup>i</sup>* ( *x*,*W*:, *<sup>j</sup>* +*bj*)), and each value <sup>±</sup>1 of an input *<sup>u</sup>* <sup>∈</sup> <sup>B</sup>*<sup>n</sup>*<sup>1</sup> <sup>±</sup><sup>1</sup> is replaced by 1/0 (cf. Sect. 3.3). To be consistent, the function *ti*<sup>↓</sup> *<sup>j</sup>* : <sup>B</sup>*ni* <sup>±</sup><sup>1</sup> <sup>→</sup> <sup>B</sup>±<sup>1</sup> is reformulated as the function *<sup>t</sup>* (*b*) *<sup>i</sup>*<sup>↓</sup> *<sup>j</sup>* : <sup>B</sup>*ni* <sup>→</sup> <sup>B</sup> such that for every *<sup>x</sup>* <sup>∈</sup> <sup>B</sup>*ni* , *t* (*b*) *i*↓ *j* (*x*) = 0.5 × (*t bin <sup>i</sup>* (*t bn <sup>i</sup>* ( 2*x* − **1**,*W*:, *<sup>j</sup>* + *bj*)) + 1), where **1** denotes the vector of 1's with the width *ni*.

Let *Ci*, *<sup>j</sup>* be the following cardinality constraint:

$$C\_{i,j} \triangleq \begin{cases} \sum\_{k=1}^{n\_i} \ell\_k \ge \lceil \frac{1}{2} \cdot (n\_i + \mu\_j - \mathbf{b}\_j - \frac{\gamma\_j \cdot \sigma\_j}{\alpha\_j}) \rceil, & \text{if } \alpha\_j > 0;\\ 1, & \text{if } \alpha\_j = 0 \land \gamma\_j \ge 0;\\ 0, & \text{if } \alpha\_j = 0 \land \gamma\_j < 0;\\ \sum\_{k=1}^{n\_i} \neg \ell\_k \ge \lceil \frac{1}{2} \cdot (n\_i - \mu\_j + \mathbf{b}\_j + \frac{\gamma\_j \cdot \sigma\_j}{\alpha\_j}) \rceil, & \text{if } \alpha\_j < 0; \end{cases}$$

where for every *k* ∈ [*ni*], *<sup>k</sup>* is *x<sup>k</sup>* if *Wk*, *<sup>j</sup>* = +1, and *<sup>k</sup>* is ¬*x<sup>k</sup>* if *Wk*, *<sup>j</sup>* = −1.

**Proposition 1.** *t* (*b*) *<sup>i</sup>*<sup>↓</sup> *<sup>j</sup>* <sup>⇔</sup> *Ci*, *j.*

Proof refers to [71].

**Transformation for the Output Block.** For the output block *td*+<sup>1</sup> : B*nd*+<sup>1</sup> <sup>±</sup><sup>1</sup> <sup>→</sup> <sup>B</sup>*<sup>s</sup>* , since *td*+<sup>1</sup> = *t am <sup>d</sup>*+<sup>1</sup> ◦ *t lin <sup>d</sup>*+<sup>1</sup>, then for every *<sup>j</sup>* <sup>∈</sup> [*s*], we can reformulate *td*+1<sup>↓</sup> *<sup>j</sup>* : <sup>B</sup>*nd*+<sup>1</sup> <sup>±</sup><sup>1</sup> <sup>→</sup> <sup>B</sup> as the function *t* (*b*) *<sup>d</sup>*+1<sup>↓</sup> *<sup>j</sup>* : <sup>B</sup>*nd*+<sup>1</sup> <sup>→</sup> <sup>B</sup> such that for every *<sup>x</sup>* <sup>∈</sup> <sup>B</sup>*nd*+<sup>1</sup> , *<sup>t</sup>* (*b*) *d*+1↓ *j* (*x*) = *td*+1<sup>↓</sup> *<sup>j</sup>*(2*x* − **1**). For every *j* ∈ [*s*] \ {*j*}, we define the cardinality constraint *Cd*+1, *<sup>j</sup>* as follows:

$$C\_{d+1,j'} \triangleq \begin{cases} \sum\_{k=1}^{n\_{d+1}} \ell\_{d+1,k} \ge \frac{1}{4} (\mathbf{b}\_{j'} - \mathbf{b}\_{j} + \sum\_{k=1}^{n\_{d+1}} (\mathbf{W}\_{k,j} - \mathbf{W}\_{k,j'})) + 1 + \sharp \mathsf{Neg}, \\ \qquad\text{ if } j' < j \text{ and } \frac{1}{4} (\mathbf{b}\_{j'} - \mathbf{b}\_{j} + \sum\_{k=1}^{n\_{d+1}} (\mathbf{W}\_{k,j} - \mathbf{W}\_{k,j'})) \text{ is an integer;} \\\\ \sum\_{k=1}^{n\_{d+1}} \ell\_{d+1,k} \ge \lceil \frac{1}{4} (\mathbf{b}\_{j'} - \mathbf{b}\_{j} + \sum\_{k=1}^{n\_{d+1}} (\mathbf{W}\_{k,j} - \mathbf{W}\_{k,j'})) \rceil + \sharp \mathsf{Neg}, & \text{otherwise;} \end{cases}$$

where Neg = |{*k* ∈ [*nd*+1] | *Wk*, *<sup>j</sup>* − *Wk*, *<sup>j</sup>* = −2}|, *<sup>d</sup>*+1,*<sup>k</sup>* is *x<sup>d</sup>*+1,*<sup>k</sup>* if *Wk*, *<sup>j</sup>* − *Wk*, *<sup>j</sup>* = +2, *<sup>d</sup>*+1,*<sup>k</sup>* is ¬*x<sup>d</sup>*+1,*<sup>k</sup>* if *Wk*, *<sup>j</sup>* − *Wk*, *<sup>j</sup>* = −2, and *<sup>d</sup>*+1,*<sup>k</sup>* is 0 if *Wk*, *<sup>j</sup>* − *Wk*, *<sup>j</sup>* = 0.

**Proposition 2.** *t* (*b*) *<sup>d</sup>*+1<sup>↓</sup> *<sup>j</sup>* ⇔ *<sup>j</sup>* ∈[*s*], *j <sup>j</sup> Cd*+1, *<sup>j</sup> .*

Proof refers to [71].

For each internal block *ti* : B*ni* <sup>±</sup><sup>1</sup> <sup>→</sup> <sup>B</sup>*ni*+<sup>1</sup> <sup>±</sup><sup>1</sup> , we denote by BNN2CC(*ti*) the cardinality constraints {*Ci*,1, ··· ,*Ci*,*ni*+<sup>1</sup> }. For each output class *j* ∈ [*s*], we denote by BNN2CC*<sup>j</sup>* (*td*+1) the cardinality constraints{*Cd*+1,1, ···*Cd*+1, *<sup>j</sup>*−<sup>1</sup>,*Cd*+1, *<sup>j</sup>*+<sup>1</sup>, ··· ,*Cd*+1,*s*}. By applying the above transformation to all the blocks of the BNN N = (*t*1, ··· , *td*, *td*+1), we obtain its cardinality constraint form N(*b*) = (*t* (*b*) <sup>1</sup> , ··· , *t* (*b*) *<sup>d</sup>* , *t* (*b*) *<sup>d</sup>*+<sup>1</sup>) such that for each *i* ∈ [*d*], *t* (*b*) *<sup>i</sup>* = BNN2CC(*ti*), and *t* (*b*) *<sup>d</sup>*+<sup>1</sup> <sup>=</sup> (BNN2CC1(*td*+1), ··· ,BNN2CC*<sup>s</sup>* (*td*+1)). Given an input *<sup>u</sup>* <sup>∈</sup> <sup>B</sup>*<sup>n</sup>*<sup>1</sup> , we denote by <sup>N</sup>(*b*) (*u*) the index *j* ∈ [*s*] such that all the cardinality constraints in BNN2CC*<sup>j</sup>* (*td*+1) hold under the valuation *u*. It is straightforward to verify:

**Theorem 1.** *<sup>u</sup>* <sup>∈</sup> <sup>B</sup>*<sup>n</sup>*<sup>1</sup> <sup>±</sup><sup>1</sup> *is classified into the class j by the BNN* <sup>N</sup> *<sup>i</sup>*ff <sup>N</sup>(*b*) (*u*(*b*) ) = *j.* *Example 1.* Consider the BNN N = (*t*1, *t*2) with one internal block *t*<sup>1</sup> and one output block *t*<sup>2</sup> as shown in Fig. 5 (left-bottom), where the elements of the Weight matrix *W* are associated to the edges, and the other parameters are given in the left-up table. The transformation functions of blocks *t*<sup>1</sup> and *t*<sup>2</sup> are given in the right-up table, and their cardinality constraints are given in the right-bottom table.

For instance, for each input *<sup>x</sup>* <sup>∈</sup> <sup>B</sup><sup>3</sup> <sup>±</sup>1, *<sup>y</sup>*<sup>1</sup> <sup>=</sup> sign(−*x*<sup>1</sup> <sup>+</sup> *<sup>x</sup>*<sup>2</sup> <sup>+</sup> *<sup>x</sup>*<sup>3</sup> <sup>+</sup> <sup>2</sup>.7), i.e., *<sup>y</sup>*<sup>1</sup> = +<sup>1</sup> <sup>⇔</sup> −*x*<sup>1</sup> + *x*<sup>2</sup> + *x*<sup>3</sup> +2.7 ≥ 0. By replacing *xi* with 2× *x* (*b*) *<sup>i</sup>* −1 and *x* (*b*) <sup>1</sup> with 1−¬*x* (*b*) <sup>1</sup> , we have: *y*<sup>1</sup> = +1 ⇔ (−*x* (*b*) <sup>1</sup> + *x* (*b*) <sup>2</sup> + *x* (*b*) <sup>3</sup> + 0.85 ≥ 0) ⇔ (¬*x* (*b*) <sup>1</sup> + *x* (*b*) <sup>2</sup> + *x* (*b*) <sup>3</sup> ≥ 0.15). Thus we get *y* (*b*) <sup>1</sup> ⇔ ¬*x* (*b*) <sup>1</sup> + *x* (*b*) <sup>2</sup> + *x* (*b*) <sup>3</sup> ≥ 1 (note that *y* (*b*) <sup>1</sup> = 0 ⇔ ¬*x* (*b*) <sup>1</sup> + *x* (*b*) <sup>2</sup> + *x* (*b*) <sup>3</sup> < 1). Similarly, we can deduce that *o*<sup>1</sup> ⇔ *y*<sup>1</sup> − *y*<sup>2</sup> ≥ 0.7, and thus *o*<sup>1</sup> ⇔ *y* (*b*) <sup>1</sup> − *y* (*b*) <sup>2</sup> ≥ 0.35 ⇔ *y* (*b*) <sup>1</sup> + ¬*y* (*b*) <sup>2</sup> ≥ 2.

#### **3.5 BDD Model Builder**

The construction of the BDDs (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*s*] from the BNN N(*b*) and the input region *R* is done iteratively throughout the blocks. Initially, the BDD for the first block is built, which can be seen as the input-output relation for the first internal block. In the *i*-th iteration, as the input-output relation of the first (*i*−1) internal blocks has been encoded into the BDD, we compose this BDD with the BDD for the block *ti* which is built from its cardinality constraints *t* (*b*) *<sup>i</sup>* , resulting in the BDD for the first *i* internal blocks. Finally, we obtain the BDDs (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*s*] of the BNN N, with respect to the input region *R*.

**Fig. 5.** An illustrating example

**Design Choice.** There are several design choices for efficiency consideration which we discuss as follows. First of all, to encode the input-output relation of an internal block *ti* into BDD from its cardinality constraints *t* (*b*) *<sup>i</sup>* = {*Ci*,1, ··· ,*Ci*,*ni*+<sup>1</sup> }, we need to compute And*<sup>j</sup>*∈[*ni*+1]CC2BDD(*Ci*, *<sup>j</sup>*). A simple and straightforward approach is to initially compute a BDD *G* = CC2BDD(*Ci*,1) and then iteratively compute the conjunction *G* = <sup>A</sup>nd(*G*,CC2BDD(*Ci*, *<sup>j</sup>*)) of *<sup>G</sup>* and CC2BDD(*Ci*, *<sup>j</sup>*) for 2 <sup>≤</sup> *<sup>j</sup>* <sup>≤</sup> *ni*+1.

Alternatively, we use a divide-and-conquer strategy to recursively compute the BDDs for the first half and the second half of the cardinality constraints respectively, and then apply the AND-operation. Our preliminary experimental results show that the latter approach often performs better (about 2 times faster) than the former one, although they generate the same BDD.

Second, constructing the BDD directly from the cardinality constraints *t* (*b*) *<sup>i</sup>* = {*Ci*,1, ··· ,*Ci*,*ni*+<sup>1</sup> } becomes prohibitively costly when *ni* and *ni*+<sup>1</sup> are large, as the BDDs CC2BDD(*Ci*, *<sup>j</sup>*) for *<sup>j</sup>* <sup>∈</sup> [*ni*+1] need to consider all the inputs in <sup>B</sup>*ni* . To improve efficiency, we apply feasible input propagation. Namely, when we construct the BDD for the block *ti*+1, we only consider its possible inputs with respect to the output of the block *ti*. Our preliminary experimental results show that the optimization could significantly improve the efficiency of the BDD construction.

Third, instead of encoding the input-output relation of the BNN N as a sole BDD or MTBDD, we opt to use a family of *s* BDDs (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*<sup>s</sup>*], each of which corresponds to one output class of N. Recall that each output class *i* ∈ [*s*] is represented by (*s* − 1) cardinality constraints. Then, we can build a BDD *Gi* for the output class *i*, similar to the BDD construction for internal blocks. By composing *Gi* with the BDD of the entire internal blocks, we obtain the BDD *Gout <sup>i</sup>* . Building a single BDD or MTBDD for the BNN is possible from (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*<sup>s</sup>*], but our approach gives the flexibility especially when a specific target class is interested, which is common for robustness analysis.


**Overall Algorithm.** The overall BDD construction procedure is shown in Algorithm 2. Given a BNN N = (*t*1, ··· , *td*, *td*+1) with *s* output classes and an input region *R*(*u*, τ), the algorithm outputs the BDDs (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*<sup>s</sup>*], encoding the input-output relation of the BNN N with respect to the input region *R*(*u*, τ).

The procedure BNN2BDD first builds the BDD representation *Gin <sup>u</sup>*,τ of the input region *R*(*u*, τ) and the cardinality constraints from BNN N(*b*) (Line 1). The first forloop builds a BDD encoding the input-output relation of the entire internal blocks w.r.t. *Gin u*,τ. The second for-loop builds the BDDs (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*<sup>s</sup>*], each of which encodes the input-output relation of the entire BNN for a class *i* ∈ [*s*] w.r.t. *Gin <sup>u</sup>*,τ. The procedure <sup>B</sup>lock2BDD receives the cardinality constraints {*Cm*, ··· ,*Cn*}, a BDD *<sup>G</sup>in* representing the feasible inputs of the block and the block index *i* as inputs, and returns a BDD *G*. If *i* = *d*+1, namely, the cardinality constraints {*Cm*, ··· ,*Cn*} are from the output block, the resulting BDD *G* encodes the subset of *Gin <sup>u</sup>*,τ that satisfy all the cardinality constraints {*Cm*, ··· ,*Cn*}. If *i d* + 1, then the BDD *G* encodes the input-output relation of the Boolean function *fm*,*<sup>n</sup>* such that for every *x<sup>i</sup>* ∈ L(*Gin*), *fm*,*n*(*x<sup>i</sup>* ) is the truth vector of the cardinality constraints {*Cm*, ··· ,*Cn*} under the valuation *x<sup>i</sup>* . When *m* = 1 and *n* = *ni*+1, *fm*,*<sup>n</sup>* is the same as *t* (*b*) *<sup>i</sup>* , hence <sup>L</sup>(*G*) <sup>=</sup> {*x<sup>i</sup>* <sup>×</sup> *<sup>x</sup>i*+<sup>1</sup> <sup>∈</sup> *<sup>G</sup>in* <sup>×</sup> <sup>B</sup>*ni*+<sup>1</sup> <sup>|</sup> *<sup>t</sup>* (*b*) *<sup>i</sup>* (*x<sup>i</sup>* ) = *xi*+1}. Detailed explanation refers to [71].

**Theorem 2.** *Given a BNN* N *with s output classes and an input region R*(*u*, τ)*, we can compute s BDDs* (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*s*] *such that the BNN* N *classifies an input x* ∈ *R*(*u*, τ) *into the class i* ∈ [*s*] *i*ff *x*(*b*) ∈ L(*Gout <sup>i</sup>* )*.*

Algorithm 2 explicitly involves *O*(*d* + *s*) RelProd-operations, *O*(*s*<sup>2</sup> + - *<sup>i</sup>*∈[*d*] *ni*) Andoperations and *O*(*d*) Exists-operations.

#### **4 Applications: Robustness Analysis and Interpretability**

In this section, we present two applications within BDD4BNN, i.e., robustness analysis and interpretability of BNNs.

#### **4.1 Robustness Analysis**

**Definition 2.** *Given a BNN* N *and an input region R*(*u*, τ)*, the BNN is (locally) robust w.r.t. the region R*(*u*, τ) *if each sample x* ∈ *R*(*u*, τ) *is classified into the same class as the ground-truth class of u.*

*An adversarial example in the region R*(*u*, τ) *is a sample x* ∈ *R*(*u*, τ) *such that x is classified into a class, that di*ff*ers from the ground-truth class of u.*

As mentioned in Sect. 1, qualitative verification which checks whether a BNN is robust or not is insufficient in many practical applications. In this paper, we are interested in *quantitative* verification of robustness which asks *how many adversarial examples are there in the input region of the BNN for each class*. To answer this question, given a BNN N and an input region *R*(*u*, τ), we first obtain the BDDs (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*s*] by applying Algorithm 2 and then count the number of adversarial examples for each class in the input region *R*(*u*, τ). Note that counting adversarial examples amounts to computing |*R*(*u*, τ)| − |L(*Gout <sup>g</sup>* )|, where *g* denotes the ground-truth class of *u*, and |L(*Gout <sup>g</sup>* )| can be computed in time *O*(|*Gout <sup>g</sup>* |).

In some applications, more refined analysis is needed. For instance, it may be acceptable to misclassify a dog as a cat, but unacceptable to misclassify a tree as a car. This suggests that the robustness of BNNs may depend on the classes to which samples are misclassified. To capture this, we consider the notion of targeted robustness.

**Definition 3.** *Given a BNN* N*, an input region R*(*u*, τ)*, and the class t, the BNN is t*target-robust *w.r.t. the region R*(*u*, τ) *if every sample x* ∈ *R*(*u*, τ) *is never classified into the class t. (Note that we assume that the ground-truth class of u di*ff*ers from the class t.)*

The quantitative verification problem of *t*-target-robustness of a BNN asks *how many adversarial examples in the input region R*(*u*, τ) *are misclassified to the class t by the BNN* N. To answer this question, we first obtain the BDD *Gout <sup>t</sup>* by applying Algorithm 2 and then count the number of adversarial examples by computing |L(*Gout <sup>t</sup>* )|.

Note that, if one wants to compute the (locally) maximal safe Hamming distance that satisfies a robustness property for an input sample (e.g., the proportion of adversarial examples is below a threshold), our framework can incrementally compute such a distance without constructing the BDD models of the entire BNN from scratch.

**Definition 4.** *Given a BNN* N*, input region R*(*u*,*r*) *and threshold* ≥ 0*, r*<sup>1</sup> *is the (locally) maximal safe Hamming distance of R*(*u*, τ)*, if one of the follows holds:*

*– if Pr*(*R*(*u*,*r*)) > *, then Pr*(*R*(*u*,*r*1)) ≤ *and Pr*(*R*(*u*,*r* )) > *for r* : *r*<sup>1</sup> < *r* < *r; – if Pr*(*R*(*u*,*r*)) ≤ *, then Pr*(*R*(*u*,*r*<sup>1</sup> + 1)) > *and Pr*(*R*(*u*,*r* )) ≤ *for r* : *r* < *r* ≤ *r*1*;*

*where Pr*(*R*(*u*,*r*)) *is the probability* - *i*∈[*s*].*i<sup>g</sup>* |L(*Gout <sup>i</sup>* )| <sup>|</sup>*R*(*u*,*r*)<sup>|</sup> *for g being the ground-truth class of u, assuming a uniform distribution of adversarial samples.*

Algorithm 3 shows the procedure to incrementally compute the maximal safe Hamming distance for a given threshold ≥ 0, input region *R*(*u*,*r*), and ground-truth class *g* of *u*. Remark that *Pr*(*R*(*u*,*r*)) may not be monotonic w.r.t. the Hamming distance *r*.

#### **4.2 Interpretability**

In general, interpretability addresses the question of *why some inputs in the input region are (mis)classified by the BNN into a specific class?* We consider the interpretability of BNNs using two complementary explanations, i.e., prime implicant explanations and essential features.

**Definition 5.** *Given a BNN* N*, an input region R*(*u*, τ) *and a class g, a* prime implicant explanation *(PI-explanation) of decisions made by the BNN* N *on the inputs* L(*Gout <sup>g</sup>* ) *is a minimal set of literals* {1, ··· , *k*} *such that for every x* ∈ *R*(*u*, τ)*, if x satisfies* <sup>1</sup> ∧···∧ *k, then x is classified into the class g by the BNN* N*.*

**Algorithm 3:** Compute the maximal safe Hamming distance

**<sup>1</sup> Proc** <sup>M</sup>axHD(BNN : <sup>N</sup> <sup>=</sup> (*t*1, ··· , *td*, *td*+1), Region : *<sup>R</sup>*(*u*,*r*), Threshold : , Class : *<sup>g</sup>*) **<sup>2</sup>** (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*s*] =BNN2BDD(N, *R*(*u*,*r*)); **<sup>3</sup> if** ( - *i*∈[*s*].*i<sup>g</sup>* |L(*Gout <sup>i</sup>* )| <sup>|</sup>*R*(*u*,*r*)<sup>|</sup> <sup>&</sup>gt;) **then** // decrease *<sup>r</sup>* **<sup>4</sup> while** (*r* ≥ 0) **do <sup>5</sup>** *r* = *r* − 1; **<sup>6</sup>** (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*s*] <sup>=</sup> (And(*Gin u*,*r*,*Gout <sup>i</sup>* ))*<sup>i</sup>*∈[*<sup>s</sup>*]; **<sup>7</sup> if** ( - *i*∈[*s*].*i<sup>g</sup>* |L(*Gout <sup>i</sup>* )| <sup>|</sup>*R*(*u*,*r*)<sup>|</sup> <sup>≤</sup> ) **then return** *<sup>r</sup>*; **<sup>8</sup> else** // increase *r* **<sup>9</sup> while** (*r* ≤ *n*1) **do** // *n*<sup>1</sup> is the input size of the BNN N **<sup>10</sup>** *r* = *r* + 1; **<sup>11</sup>** (*Bout <sup>i</sup>* )*<sup>i</sup>*∈[*s*] =BNN2BDD(N, *R*(*u*,*r*) \ *R*(*u*,*r* − 1)); **<sup>12</sup>** (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*s*] <sup>=</sup> (Or(*Bout <sup>i</sup>* ,*Gout <sup>i</sup>* ))*<sup>i</sup>*∈[*<sup>s</sup>*]; **<sup>13</sup> if** ( - *i*∈[*s*].*i<sup>g</sup>* |L(*Gout <sup>i</sup>* )| <sup>|</sup>*R*(*u*,*r*)<sup>|</sup> <sup>&</sup>gt;) **then return** *<sup>r</sup>* <sup>−</sup> 1; **<sup>14</sup> return** *r*

Intuitively, a PI-explanation {1, ··· , *k*} indicates that {var(1), ··· , var(*k*)} are key features, namely, if fixed, the predication is guaranteed no matter how the remaining features change. Remark that there may be more than one PI-explanation for a set of inputs L(*Gout <sup>g</sup>* ). When *g* is set to be the class of the benign input *u*, a PI-explanation on *Gout <sup>g</sup>* suggests why these samples are classified into *g* by the BNN N.

**Definition 6.** *Given a BNN* N*, an input region R*(*u*, τ) *and a class g, the* essential features *for the inputs* L(*Gout <sup>g</sup>* ) *are literals* {1, ··· , *k*} *such that every x* ∈ *R*(*u*, τ)*, if x is classified into the class g by the BNN* N*, then x satisfies* <sup>1</sup> ∧···∧ *k.*

Intuitively, the essential features {1, ··· , *k*} denote the key features such that all samples *x* ∈ *R*(*u*, τ) that are classified into the class *g* by the BNN N must agree on these features. Essential features differ from PI-explanations, where the former can be seen as a necessary condition, while the latter can be seen as a sufficient condition.

BDD libraries (e.g., CUDD [58]) usually provide APIs to identify prime implicants (e.g., Cudd bddPrintCover and Cudd FirstPrime) and essential variables (e.g., Cudd FindEssential). Therefore, prime implicants and essential features can be computed via queries on the BDDs (*Gout <sup>i</sup>* )*<sup>i</sup>*∈[*<sup>s</sup>*].

#### **5 Evaluation**

We have implemented our framework as a prototype tool BDD4BNN based on the CUDD package [58]. BDD4BNN is implemented with Python as the front-end to preprocess BNNs and C++ as the back-end to perform the BDD encoding and analysis. In this section, we report the experimental results, including BDD encoding, robustness analysis based on hamming distance, and interpretability.

**Experimental Setup.** The experiments were conducted on a machine with Intel Xeon Gold 5118 2.3GHz CPU, 64-bit Ubuntu 20.04 LTS operating systems, 128 G RAM. Each BDD encoding executed on one core limited by 8-h.

**Benchmarks.** We use the PyTorch (v1.0.1.post2) deep learning platform provided by NPAQ [6] to train and test BNNs. We trained 12 BNN models (P1-P12) with varying sizes using the MNIST dataset [35]. The MNIST dateset contains 70,000 gray-scale 28 × 28 images (60,000 for training and 10,000 for testing) of handwritten digits with 10 classes. In our experiments, we downscale the images (28 × 28) to some selected input size *<sup>n</sup>*<sup>1</sup> (i.e., the corresponding image is of the size <sup>√</sup>*n*<sup>1</sup> <sup>×</sup> <sup>√</sup>*n*1) and then binarize the normalized pixels of the images.

Details of the BNN models are listed in Table 3, each of which has 10 classes (i.e., *s* = 10). Column 1 shows the name of the BNN model. Column 2 shows the architecture of the BNN model, where *n*<sup>1</sup> : ··· : *nd*+<sup>1</sup> : *s* denotes that the BNN model has *d* + 1 blocks, *n*<sup>1</sup> inputs and *s* outputs; the *i*-th block for *i* ∈ [*d* + 1] has *ni* inputs and *ni*+<sup>1</sup> outputs with *nd*+<sup>2</sup> = *s*. Recall that each internal block has 3 layers while the output block has 2 layers. Therefore, the number of layers ranges from 5 to 14, the dimension of inputs ranges from 9 to 784, and the number of hidden neurons per linear layer ranges from 10 to 100. Column 3 shows the accuracy of the BNN model on the test set of the MNIST dataset. (We can observe that the accuracy increases with the size of inputs, the number of layers, and the number of hidden neurons per layer.) We randomly choose 10 images from the training set of the MNIST dataset (one image per class) to evaluate our approach.

#### **5.1 Performance of BDD Encoding**

We evaluate BDD4BNN on the BNNs listed in Table 3 using different input regions.

**BDD Encoding Using Full Input Space.** We evaluate BDD4BNN on the BNNs (P1– P5), where B*<sup>n</sup>*<sup>1</sup> <sup>±</sup><sup>1</sup> is used as the input region. The results are shown in Table 4, where <sup>|</sup>*G*<sup>|</sup> denotes the number of BDD nodes in the BDD manager. We can observe that both the execution time and the number of BDD nodes increase with the size of BNNs.

**BDD Encoding Under Hamming Distance.** We evaluate BDD4BNN on the BNNs (P5–P12). In this case, an input region is given by one of the 10 images and a Hamming distance *r* ranging from 2 to 6. The average results are shown in Table 5, where [*i*] (resp. (*i*)) indicates the number of cases that BDD4BNN runs out of memory (resp. time). Overall, the execution time and the number of BDD nodes increase with *r*. BDD4BNN succeeded on all the cases when *r* ≤ 4, 75 cases out of 80 when *r* = 5, and 48 cases out of 80 when *r* = 6. We observe that the execution time and number of BDD nodes increase with the number of hidden neurons (P6 vs. P7, P8 vs. P9, and P11 vs. P12), while the effect of the number of layers is diverse (P6 vs. P8 vs. P10, and P7 vs. P9). From P9 and P10, we observe that the number of hidden neurons per layer is likely the key impact factor of the efficiency of BDD4BNN. Interestingly, our tool BDD4BNN works well on BNNs with large input sizes (i.e., on P11 and P12).


**Table 3.** BNN benchmarks

**Table 4.** BDD encoding using full input space


These results demonstrate the efficiency and scalability of BDD4BNN on BDD encoding of BNNs. We remark that, compared with the learning-based approach [54], our approach is considerably more efficient and scalable. For instance, the learningbased approach takes 403 s to encode a BNN with 64 input size, 5 hidden neurons, and 2 output size when *r* = 6, while ours takes about 3 s even for a larger network P5.

#### **5.2 Robustness Analysis**

We evaluate BDD4BNN on the robustness of BNNs, including robustness analysis under different input regions and maximal safe Hamming distance computing.

**Robustness Verification with Hamming Distance.** We evaluate BDD4BNN on BNNs (P7, P8, P9, and P11) using the 10 images. The input regions are given by the Hamming distance *r* ranging from 2 to 4, resulting in 120 instances. To the best of our knowledge, NPAQ [6] is the only work that supports quantitative robustness verification of BNNs to which we compare BDD4BNN. Recall that NPAQ only provides PAC-style guarantees. Namely, it sets a tolerable error ε and a confidence parameter δ. The final estimated results of NPAQ have the bounded error ε with confidence of at least 1 − δ, i.e.,

$$\Pr[\left(1+\varepsilon\right)^{-1}\text{ReaLNum} \leq \text{Es} \cdot \text{t.inat.eed} \text{Num} \leq \left(1+\varepsilon\right)\text{ReaLNum}\right] \geq 1-\delta \tag{1}$$

In our experiments, we set ε = 0.8 and δ = 0.2, as done in [6].


**Table 5.** BDD encoding under Hamming distance

**Table 6.** Robustness verification under Hamming distance


The results on the average of the images are shown in Table 6. NPAQ ran out of time on 5 instances (which occur in P9 with *r* = 4 and P11 with *r* = 3 and *r* = 4), while BDD4BNN successfully verified all the 120 instances. Table 6 only shows the results of 115 instances that can be solved by NPAQ. Columns 3, 4, and 5 (resp. 6, 7, and 8) show the number of adversarial examples, the execution time, and the proportion of adversarial examples in the input region. Column 9 shows the error rate RealNum−EstimatedNum EstimatedNum , where RealNum is from our result, and EstimatedNum is from NPAQ. Column 10 shows the speedup of BDD4BNN compared with NPAQ. Remark that the numbers of adversarial examples are 0 for P11 on input regions with *r* = 3 and *r* = 4 that can be solved by NPAQ. There do exist input regions for P11 that cannot be solved by NPAQ but have adversarial examples (see below). On BNNs that were solved by both NPAQ and BDD4BNN, BDD4BNN is significantly (5× to 1, 340×) faster and more accurate than NPAQ. From Table 5 and Table 6, we also found that most of the verification time is spent on BDD encoding while the rest is usually less than 10 s.

**Details of Robustness and Targeted Robustness.** Figure 6(a) (resp. Fig. 6(b) and Fig. 6(c)) depicts the distributions of classes on P8 with Hamming distance *r* = 2 (resp. P8 with *r* = 3 and P11 with *r* = 2), where on the x-axis *i* = 0, ··· , 9 denotes the input

**Fig. 6.** Details of robustness verification with Hamming distance

region that is within the respective Hamming distance to the image of digit *i* (called *i*-region). We can observe that P8 is robust for the 0-region when *r* = 2 and robust for the 6-region when *r* = 2 and *r* = 3, but is not robust for the other regions. (Note P8 is not robust for 0-region when *r* = 3, which is hard to be visualized in Fig. 6(b) due to the small number of adversarial examples.) Most of the adversarial examples in the 1-region and 5-region are misclassified into the digit 3 by P8. P11 is not robust for the 1-region or the 5-region, but is robust for all the other regions. Though P8 and P11 are not robust on some input regions, indeed they are *t*-target-robust for many target classes *t*, e.g., P11 is *t*-target-robust for the 1-region when *t* - 2, and the 5-region when *t* - 3. (The raw data are given in [71].)

**Quality Validation of** NPAQ. Figure 6(d) shows the distribution of error rates of NPAQ, where the x-axis is the range of the error rate and the y-axis is the corresponding number of instances. There are 19 instances where the estimated number of adversarial examples exceeds (1+ ) of the real number of the adversarial examples and 7 instances where the estimated number of adversarial examples is less than (1 + ) <sup>−</sup><sup>1</sup> of the real number of the adversarial examples. This means that out of 115 instances, only in 89 instances the estimated number is within the allowed range, which is less than 1 − δ = 0.8.

**Maximal Safe Hamming Distance.** As a representative of such an analysis, we evaluate BDD4BNN on 4 BNNs (P7, P8, P9, and P11) with 10 images for 2 robustness thresholds ( = 0 and = 0.03). The initial Hamming distance *r* is 3. Intuitively, = 0 (resp. = 0.03) means that up to 0% (resp. 3%) samples in the input region can be adversarial.

Table 7 shows the results, where columns SD and Time give the maximal safe Hamming distance and the execution time, respectively. BDD4BNN solved 74 out of 80 instances. (For the remaining 6 instances, BDD4BNN ran out of time or memory, but


**Table 7.** Maximal safe Hamming distance

**Fig. 7.** Graphic representation of essential features and PI-explanations

it was still able to compute a larger safe Hamming distance.) We can observe that the maximal safe Hamming distance increases with the threshold on several BNNs and input regions. We can also observe that P11 is more robust than others, which is consistent with their accuracies (cf. Table 3). Remark that SD = −1 indicates that the input image itself is misclassified.

#### **5.3 Interpretability**

To demonstrate the ability of BDD4BNN on interpretability, we consider the analysis of the BNN P12 and the image *u* of digit 1.

**Essential Features.** For the input region given by the Hamming distance *r* = 4, we compute two sets of essential features for the inputs L(*Gout* <sup>2</sup> ) and L(*Gout* <sup>5</sup> ), i.e., the adversarial examples in the region *R*(*u*, 4) that are misclassified into the classes 2 and 5 respectively. The essential features are depicted in Figs. 7(a) and 7(b), where black (resp. blue) color means that the value of the corresponding pixel is 1 (resp. 0), and yellow color means that the value of the corresponding pixel can take arbitrary values. Figure 7(a) (resp. Fig. 7(b)) indicates that the inputs L(*Gout* <sup>2</sup> ) (resp. L(*Gout* <sup>5</sup> )) must agree on these black- and blue-colored pixels.

**PI-Explanations.** For demonstration, we assume that the input region is given by the fixed set of indices *I* = {1, 2, ··· , 28} which denotes the first row of pixels of 28 × 28 images. We compute two PI-explanations of the inputs L(*Gout* <sup>2</sup> ) and L(*Gout* <sup>5</sup> ). The PIexplanations are depicted in Figs. 7(c) and 7(d). Figure 7(c) (resp. Fig. 7(d)) suggests that, by the definition of the PI-explanation, all the images in the region *R*(*u*, *I*) obtained by assigning arbitrary values to the yellow-colored pixels are always misclassified into the class 2 (resp. class 5), while changing one black-colored or blue-colored pixel would change the predication result since a PI-explanation is a minimal set of literals.

#### **6 Related Work**

In this section, we discuss the related work on qualitative/quantitative analysis and interpretability of DNNs. As there is a vast amount of literature regarding these topics, we will only discuss the most related ones to BDD4BNN.

**Qualitative Analysis of DNNs.** For real-numbered DNNs, various formal verification approaches have been proposed. Typical examples include constraint solving based approaches [17,26,30,31,51], optimization based approaches [10,13,15,16,40,61,67, 68], and program analysis based approaches [2,3,18,20,37–39,55–57,62–64,69].

Existing techniques for quantized DNNs are mostly based on constraint solving, in particular, SAT/SMT solving [12,33,45,46]. Following this line, verification of BNNs with ternary weights [28,48] and quantized DNNs with multiple bits [7,22,24] were also studied. Recently, the SMT-based framework Marabou for real-numbered DNNs [31] has also been extended to support BNNs [1].

**Quantitative Analysis of DNNs.** Comparing to qualitative analysis, quantitative analysis of neural networks is currently very limited. Two sampling-based approaches were proposed to certify the robustness for both DNNs and BNNs [5,65]. Yang et al. [69] proposed a spurious region-guided refinement approach for real-numbered DNN verification, claiming to be the first work of the quantitative robustness verification of DNNs with soundness guarantees.

Following the SAT-based qualitative analysis of BNNs [45,46], SAT-based quantitative analysis approaches were also proposed [6,21,47]. In particular, approximate SAT model-counting solvers are utilized. Shih et al. [54] also proposed a BDD-based approach to tackle BNNs, similar to our work in spirit. However, our approach is able to handle BNNs of considerably larger sizes than their learning-based method.

**Interpretability of DNNs.** Though interpretability of DNNs is crucial for explaining predictions, it is very challenging to tackle due to the blackbox nature of DNNs. There is a large body of work on the interpretability of DNNs (cf. [25,43] for a survey). Almost all the existing approaches are heuristic-based and restricted to finding explanations that are local in an input region. Some of them tackle the interpretability of DNNs by learning an interpretable model, such as binary decision trees [19,70] or finite-state automata [66]. In contrast to ours, they target at DNNs and only approximate the original model in the input region. The BDD-based approach [54] mentioned above has been used to compute the PI-explanation, but essential features were not considered therein.

### **7 Conclusion**

In this paper, we have proposed a novel BDD-based framework for quantitative verification of BNNs. We implemented the framework as a prototype tool BDD4BNN and conducted extensive experiments on 12 BNN models with varying sizes and input regions. Experimental results demonstrated that BDD4BNN is more scalable than the existing BDD-learning based approach, and significantly more efficient and accurate than the existing SAT-based approach NPAQ. This work represents the first, but a key, step of the long-term program to develop an efficient and scalable BDD-based quantitative analysis framework for BNNs.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

,

### Automated Safety Verification of Programs Invoking Neural Networks

Maria Christakis<sup>1</sup> , Hasan Ferit Eniser1, Holger Hermanns , Jörg Hoffmann<sup>2</sup> Yugesh Kothari<sup>1</sup> , Jianlin Li , Jorge A. Navas<sup>4</sup> , and Valentin Wüstholz<sup>5</sup> <sup>1</sup> MPI-SWS, Kaiserslautern and Saarbrücken, Germany {maria,hfeniser,ykothari}@mpi-sws.org <sup>2</sup> Saarland University, Saarland Informatics Campus, Saarbrücken, Germany {hermanns,hoffmann}@cs.uni-saarland.de <sup>3</sup> SKLCS, Institute of Software, Chinese Academy of Sciences, Beijing, China ljlin@ios.ac.cn <sup>4</sup> SRI International, Menlo Park, USA jorge.navas@sri.com <sup>5</sup> ConsenSys, Kaiserslautern, Germany valentin.wustholz@consensys.net 6 Institute of Intelligent Software, Guangzhou, China <sup>7</sup> University of Chinese Academy of Sciences, Beijing, China , 3, , 2 6 2 7

Abstract. State-of-the-art program-analysis techniques are not yet able to effectively verify safety properties of heterogeneous systems, that is, systems with components implemented using diverse technologies. This shortcoming is pinpointed by programs invoking neural networks despite their acclaimed role as innovation drivers across many application areas. In this paper, we embark on the verification of system-level properties for systems characterized by interaction between programs and neural networks. Our technique provides a tight two-way integration of a program and a neural-network analysis and is formalized in a general framework based on abstract interpretation. We evaluate its effectiveness on 26 variants of a widely used, restricted autonomous-driving benchmark.

#### 1 Introduction

Software is becoming increasingly heterogeneous. In other words, it consists of more and more diverse software components, implemented using different technologies such as neural networks, smart contracts, or web services. Here, we focus on programs invoking neural networks, in response to their prominent role in many upcoming application areas. Examples from the forefront of innovation include a controller of a self-driving car that interacts with a neural network identifying street signs [43,48], a banking system that consults a neural network for credit screening [3], or a health-insurance system that relies on a neural network to predict people's health needs [51]. There are growing concerns regarding the effects of integrating such heterogeneous technologies [40].

Despite these software advances, state-of-the-art program-analysis techniques cannot yet effectively reason across heterogeneous components. In fact, program analyses today focus on homogeneous units of software in isolation; for instance, to check the robustness of a neural network (e.g., [37,27,36,66,65,64,57,41]), or safety of a program invoking a neural network while—conservatively, but imprecisely—treating the neural network as if it could return arbitrary values. This is a fundamental limitation of prevalent program-analysis techniques, and as a result, we cannot effectively analyze the interaction between diverse components of a heterogeneous system to check system properties.

Many properties of heterogeneous systems depend on components correctly interacting with each other. For instance, consider a program that controls the acceleration of an autonomous vehicle by invoking a neural network with the current direction, speed, and LiDAR image of the vehicle's surroundings. One might want to verify that the vehicle's speed never exceeds a given bound. Even such a seemingly simple property is challenging to verify automatically due to the mutual dependencies between the two components. On the one hand, the current vehicle direction and speed determine the feasible inputs to the neural network. On the other hand, the output of the neural network controls the vehicle acceleration, and thereby, the speed. To infer bounds on the speed (and ultimately prove the property), an automated analysis should therefore analyze how the two components interact.

Our approach. In this paper, we make the first step in verifying safety of heterogeneous systems, and more specifically, of programs invoking neural networks. Existing work on verification of neural networks has either focused on the network itself (e.g., with respect to robustness) or on models (e.g., expressed using differential equations) that invoke the network, for example as part of a hybrid system [24,59]. In contrast, our approach is designed for verifying safety of a C (or ultimately LLVM) program interacting with the network. In comparison to models, C programs are much more low-level and general, and therefore require an intricate combination of program and neural-network analyses.

More specifically, our approach proposes a symbiotic combination of a program and a neural-network analysis, both of which are based on abstract interpretation [18]. By treating the neural-network analysis as a specialized abstract domain of the program analyzer, we are able to use inferred invariants for the neural network to check system properties in the surrounding program. In other words, the program analysis becomes aware of the network's computation. For this reason, we also refer to the overall approach as neuro-aware program analysis. In fact, the program and neural-network analyses are co-dependent. The former infers sets of feasible inputs to the neural network, whereas the latter determines its possible outputs given the inferred inputs. Knowing the possible neural-network outputs, in turn, enables proving system safety.

We evaluate our approach on 26 variants of Racetrack, a benchmark from related work that originates in AI autonomous decision making [4,5,32,46,52,53]. Racetrack is about the problem of navigating a vehicle on a discrete map to a goal location without crashing into any obstacles. The vehicle acceleration (in discrete directions) is determined by a neural network, which is invoked by a controller responsible for actually moving the vehicle. In Sect. 4, we show the effectiveness of our approach in verifying goal reachability and crash avoidance for 26 Racetrack variants of varying complexity. These variants constitute a diverse set of verification tasks that differ both in the neural network itself and in how and for what purpose the program invokes the neural network.

Despite our evaluation being focused on this setting, the paper's contribution should not be mistaken as being about Racetrack verification. Instead, it is about neuro-aware program analysis of heterogeneous systems for autonomous decision making. While Racetrack is a substantially simplified blueprint for the autonomous-driving context, it features the crucial co-dependent program architecture that is characteristic across the entire domain.

Contributions. Overall, we make the following contributions:


#### 2 Overview

We now illustrate neuro-aware program analysis on a high level by describing the challenges in verifying safety for a variant of the Racetrack benchmark. This variant serves as our running example for the class of programs that invoke one or more neural networks to perform a computation affecting program safety.

In general, Racetrack is a heterogeneous system that simulates the problem of navigating a vehicle to a goal location on a discrete map without crashing into any obstacles. It consists of a neural network, which predicts the vehicle acceleration toward discrete directions, and a controller (implemented in C) that actually moves the vehicle on the map. Alg. 1 shows pseudo-code for our running example, a variant of Racetrack that incorporates additional nondeterministic noise to make verification harder.

Line 1 non-deterministically selects a state from the map as the currentState, and line 2 assumes it is a start state for the vehicle, i.e., it is neither a goal nor an obstacle. On line 3, we initialize the result of navigating the vehicle as stuck, i.e., the vehicle neither crashes nor does it reach a goal. The loop on line 5 iterates until either a predefined number of steps N is reached or the vehicle is no longer stuck (i.e., crashed or at a goal state). The if-statement on line 6 adds non-determinism to the controller by either zeroing the vehicle acceleration or invoking the neural network (NN) to make a prediction. Such non-deterministic noise illustrates one type of variant we created to make the verification task more difficult (see Sect. 4.1 for more details on other variants used in our evaluation). Line 10 moves the vehicle to a new currentState according to acceleration, and the if-statement on line 11 determines whether the vehicle has crashed or reached a goal. The assertion on line 16 denotes the system properties of goal reachability

#### Algorithm 1: An example Racetrack variant.

```
1 currentState ← ?
2 assume IsStartState(currentState)
3 result ← stuck
4 i ← 0
5 while i < N and result = stuck do
6 if ? then
7 acceleration ← 0
8 else
9 acceleration ← NN(currentState)
10 currentState ← Move(currentState, acceleration)
11 if IsCrash(currentState) then
12 result ← crash
13 else if IsGoal(currentState) then
14 result ← goal
15 i ← i + 1
16 assert result = goal
```
and crash avoidance. In case this assertion does not hold but we do prove the result to be stuck, then we have only verified crash avoidance.

Note that these are safety, and not liveness, properties due to the bounded number of loop iterations (line 5)—N is 50 in our evaluation, thus making bounded model checking [8,15] intractable.

Challenges. Verifying safety of this heterogeneous system with state-of-the-art program-analysis techniques, such as abstract interpretation, is a challenging endeavor.

When considering the controller in isolation, the analysis is sound if it assumes that the neural network may return any output (>). More specifically, the abstract interpreter can ignore the call to the neural network and simply havoc its return value (i.e., consider a non-deterministic value). In our running example, this means that any vehicle acceleration is possible from the perspective of the controller analysis. Therefore, it becomes infeasible to prove a system property such as crash avoidance. In fact, in Sect. 4, we show this to be the case even with the most precise controller analysis.

On the other hand, when considering the neural network in isolation, the analysis must assume that any input is possible (>) even though this is not necessarily the case in the context of the controller. More importantly, without analyzing the controller, it becomes infeasible to prove properties about the entire system; as opposed to properties of the neural network, such as robustness. Our approach. To address these issues, our approach tightly combines the controller and neural-network analyses in a two-way integration based on abstract interpretation.

In general, an abstract interpreter infers invariants at each program state and verifies safety of an asserted property when it is implied by the invariant inferred in its pre-state. In the presence of loops, as in Racetrack (line 5 in Alg. 1), inference is performed for a number of iterations in order to reach a fixpoint, that is, infer invariants at each program state that do not change when performing additional loop iterations.

For our running example, to compute the fixpoint of the main loop, the controller analysis invokes the neural-network analysis instead of simply abstracting the call to the neural network by havocking its return value. The invariants inferred by the controller analysis in the pre-state of the call to the network are passed to the neural-network analysis; they are used to restrict the input space of the neural network. In turn, the invariants that are inferred by the neuralnetwork analysis are returned to the controller analysis to restrict the output space. This exchange of verification results at analysis time significantly improves precision. By making the program analysis aware of the network's computation, neuro-aware program analysis is able to prove challenging safety properties of the entire system.

Our implementation combines off-the-shelf, state-of-the-art abstract interpreters, namely, Crab [34] for the controller analysis and DeepSymbol [41] or Eran [27,56,57] for the neural-network analysis. Crab<sup>8</sup> is a state-of-the-art analyzer for checking safety properties of LLVM bitcode programs. Its modular high-level architecture is similar to many other abstract interpreters, such as Astrée [9], Clousot [26], and Infer [11], and it supports a wide range of different abstract domains, such as Intervals [17], Polyhedra [19], and Boxes [33]. Specialized neural-network analyzers, such as DeepSymbol or Eran, have only very recently been developed to deal with the unique challenges of precisely checking robustness of neural networks; for instance, the challenge of handling the excessive number of "branches" induced by cascades of ReLU activations.

The technical details of this combination are presented in the following section. Note, however, that our technical framework does not prescribe a neuralnetwork analysis that is necessarily based on abstract interpretation. Specifically,

it could integrate any sound analysis that, given a set of (symbolic) input states, produces a set of output states over-approximating the return value of the neural network. We also discuss how our approach may integrate reasoning about other complex components, beyond neural networks. Our program analysis is also not inherently tied to Crab, but could be performed by other abstract interpreters that use the same high-level architecture, such as Astrée [9].

The Racetrack map on the right, which is borrowed from related work [5,32], shows the verification results achieved by our approach when combining Crab and DeepSymbol. Gray cells marked with 'x' denote obstacles, and yellow cells

<sup>8</sup> https://github.com/seahorn/crab

marked with 'g' denote goal locations. Recall from Alg. 1 that we can consider any cell, which is neither an obstacle nor a goal, as a possible start location.

In our evaluation, we run a separate analysis for each possible start state to identify all start locations from which the vehicle is guaranteed to reach a goal; in other words, the analysis tries to prove that result = goal holds (line 16 of Alg. 1) for each possible start location. Note that verifying a single start state already constitutes a challenging verification problem since, due to noise, the number of reachable states grows exponentially in the number of loop iterations (the vehicle can navigate to any feasible position). This setting of one start state is common in many reinforcement-learning environments, e.g., Atari games, Procgen [16], OpenAI Gym MiniGrid<sup>9</sup> , etc.

Maps like the above are used throughout the paper to display the outcome of a verification process per cell. We color locations for which the process succeeds green in all shown maps. Similarly, we color states from which the vehicle might crash into an obstacle red; i.e., one or more states reachable from the start state may lead to a crash, and the analysis is not able to show that result 6= crash holds before line 16. Finally, states from which the vehicle is guaranteed not to crash but might not reach a goal are colored in blue; i.e., the analysis is able to show that result 6= crash holds before line 16, but it is not able to show that result 6= stuck also holds.

As shown in the map, our approach is effective in verifying goal reachability and crash avoidance for the majority of start locations. Moreover, the verification results are almost identical when combining Crab with a different neural-network analyzer, namely Eran (see Sect. 4). Note that, since the analysis considers individual start states, the map may show a red start state that is surrounded by green start states. One explanation for this is that the vehicle never enters the red state from the surrounding green states or that it only enters the red state with a "safe" velocity and direction—imagine that the vehicle velocity when starting from the red state is always 2, whereas when entering it from green states, the velocity is always less. In general, whether a trajectory is safe largely depends on the neural-network behavior, which can be brittle.

#### 3 Approach

As we discussed on a high level, our approach symbiotically combines an existing program analysis (PA) with a neural-network analysis (NNA). The result is a neuro-aware program analysis (NPA) that allows for precisely analyzing a program that invokes neural networks (see Fig. 1). In the following, we focus on a single network to keep the presentation simple. As shown in Fig. 1, the two existing analyses are extended to pass information both from PA to NNA (Φ in the diagram) and back (Ψ in the diagram).

In the following, we describe neuro-aware program analysis in more detail and elaborate on how the program analysis drives the neural-network analysis to verify safety properties of the containing heterogeneous system. Since the

<sup>9</sup> https://github.com/maximecb/gym-minigrid

Figure 1: Overview of neuro-aware program analysis.

program analysis drives the neural-network analysis, we will explain the analysis in a top-down fashion by focusing on the program analysis before going into the details of the network analysis. In other words, our description of the program analysis assumes that we have a network analysis that over-approximates the behavior of the neural network.

#### 3.1 Neuro-Aware Program Analysis

For our presentation, we assume imperative programs P with standard constructs, such as loops, function calls, arithmetic, and pointer operations (our implementation targets LLVM bitcode). In addition, we assume a special function call o := nn(i1, . . . , in) that calls a neural network with input parameters i1, . . . , i<sup>n</sup> and returns the result of querying the network in return value o. We also assume that the query does not have side effects on the program state. We denote programs P augmented with special calls to neural networks as Pnn.

We assume an abstract domain D consisting of a set of abstract elements d ∈ D. Domain D is equipped with the usual binary operators hv, t, u, ` ai, where the ordering between elements is given by v. ⊥<sup>D</sup> represents the smallest domain element and ><sup>D</sup> the largest (smallest and largest relative to the ordering imposed by v). The least upper bound (greatest lower bound) operator is denoted by t (u). As usual, if the abstract domain is not finite or the number of elements is too large, then we also assume the domain to be equipped with widening (` ) and narrowing (<sup>a</sup> ) operators to ensure termination of the fixpoint computation. Moreover, we assume the abstract forget : D × V 7→ D operation that removes a set of variables from the abstract state, and its dual project : D × V 7→ D that projects the abstract state onto a set of variables. Finally, we assume the semantics function [[.]] : P 7→ D 7→ D that, given a pre-state, computes the abstract semantics of a program to obtain its post-state; it does so recursively, by induction over the syntax of the program. We do not require that there exists a Galois connection [18] between the abstract domain D and the concrete domain C. The only requirement is that D over-approximates C, i.e., [[.]]<sup>C</sup> ⊆ γ ◦ [[.]] ◦ α where [[.]]<sup>C</sup> is the concrete semantics and γ : D 7→ C and α : C 7→ D are the concretization and abstraction functions, respectively.

We can then trivially define [[c.]] : <sup>P</sup>nn 7→ <sup>D</sup> 7→ <sup>D</sup> to deal with <sup>P</sup>nn as follows:

$$\widehat{[Cmd]}(d) = \begin{cases} [\mathtt{o} := \mathtt{nn}(\mathtt{i}\_{1}, \ldots, \mathtt{i}\_{\mathtt{n}})](d) \text{ if } Cmd \equiv \mathtt{o} := \mathtt{nn}(\mathtt{i}\_{1}, \ldots, \mathtt{i}\_{\mathtt{n}})\\ [Cmd](d) & \text{otherwise} \end{cases}$$

$$\mathsf{f}(\mathsf{o} := \mathsf{nn}(\mathsf{i}\_1, \dots, \mathsf{i}\_n)](d) = \begin{cases} \bot\_D & \text{if } d = \bot\_D \\ \mathsf{forget}(d, o) \text{ otherwise} \end{cases}$$

However, this definition of [[c.]] is not very useful since it conservatively approximates the neural network by havocking its return value o.

To obtain a more precise approximation, we can integrate a designated neuralnetwork analysis. Specifically, we view the neural-network analysis as another abstract domain Dnn, where, in practice, we do not require any other operation from Dnn except the transfer function for o := nn(i1, . . . , in) that soundly approximates the semantics of the neural network (see Sect. 3.2 for more details):

$$\begin{aligned} \mathsf{l}\left[\mathsf{o} := \mathsf{nn}(\mathsf{i}\_{1}, \ldots, \mathsf{i}\_{n})\right](d) &= \begin{cases} \bot\_{D} & \text{if } d = \bot\_{D} \\ \mathsf{1et}\ d\_{nn} = \mathsf{convert}(\mathsf{project}(d, i\_{1}, \ldots, i\_{n})) & \mathsf{in} \\ \mathsf{1et}\ d\_{nn}' = \left\lbrack \mathsf{o} := \mathsf{nn}(\mathsf{i}\_{1}, \ldots, \mathsf{i}\_{n})\right\rbrack\_{D\_{nn}}(d\_{nn}) & \mathsf{in} \\ \mathsf{forget}(d, o) \sqcap \mathsf{convert}^{-1}(d\_{nn}') & \text{otherwise} \end{cases} \end{aligned}$$

Intuitively, this more precise transfer function performs the following steps (unless d is ⊥D). First, it converts from D to Dnn to invoke the transfer function of Dnn on the converted value dnn. It then havocs the return value o and conjoins the inferred return value after converting d 0 nn back to D. In the above definition, functions convert : D 7→ Dnn and convert<sup>−</sup><sup>1</sup> : Dnn 7→ D convert from one abstract domain (D) to the other (Dnn) and back. We allow for conversions to result in loss of precision, that is, ∀x ∈ D · x v convert<sup>−</sup><sup>1</sup> (convert(x)).

It is important to realize here that the implementation of functions convert and convert<sup>−</sup><sup>1</sup> , however precise, may still trigger a fatal loss of precision. After all, the abstract domains D and Dnn must also be expressive and precise enough to capture the converted values. For example, assume that, in a given program, the function call o := nn(i1, . . . , in) invokes a neural network to obtain the next move of a vehicle (encoded as a value from 0 to 7). Suppose the abstract return value d 0 nn is the set of moves {1, 7}. In this case, given a domain D that cannot express these two moves as disjuncts, the implementation of function convert<sup>−</sup><sup>1</sup> has no choice but to abstract more coarsely; for instance, by expressing the return value as the interval [1, . . . , 7]. Depending on the program, this may be too imprecise to prove safety. This happened to be the case for the Racetrack variants we analyzed in Sect. 4, which is why we chose to use a disjunctive domain for the analysis; more specifically, we use Boxes [33], which allows us to track Boolean combinations of intervals.

Nevertheless, the considerations and approach described so far are still not precise enough for verifying the safety properties in the variants of Racetrack that we consider. This is because the controller makes extensive use of (multidimensional) arrays, whose accesses are not handled precisely by the analysis. However, we observed that these arrays are initialized at the beginning of the system execution (for instance, to store maps indexed by x-y coordinates), and after initialization, they are no longer modified. Handling such array reads precisely is crucial in our context since over-approximation could conservatively indicate that the vehicle may crash into an obstacle.

Common array domains that summarize multiple elements using one or a small number of abstract elements fail to provide the needed precision. Even a fully expanded array domain [9] that separately tracks each array element loses precision if the index of an array read does not refer to a single array element; in such cases, the join of all overlapping elements will be returned. In addition, the Clang compiler—used to produce the LLVM bitcode that Crab analyzes desugars multi-dimensional arrays into single-dimensional arrays. This results in additional arithmetic operations (in particular, multiplications) for indexing the elements; these are also challenging to analyze precisely.

Interestingly, to address these challenges, we follow the same approach as for neural networks, in other words, by introducing a designated and very precise analysis to handle reads from these pre-initialized arrays. More formally, we introduce a new statement to capture such reads, o := ar(i1, . . . , in), where i<sup>k</sup> is the index for the k-th dimension of an n-dimensional array. Note that this avoids index conversions for multi-dimensional arrays since indices of each dimension are provided explicitly. Moreover, it is structurally very similar to the nn(. . .) statement we introduced earlier. In particular, the specialized transfer function for D differs only in the two conversion functions and the specialized transfer function [[.]]Dar :

$$\begin{aligned} \left[\mathbf{o} := \mathbf{a}\mathbf{r}(\mathbf{i}\_1, \dots, \mathbf{i}\_n)\right](d) &= \begin{cases} \bot\_D & \text{if } d = \bot\_D\\ \mathbf{1}\mathbf{et}\ d\_{ar} = \mathsf{convert}\_{\omega}(\mathsf{project}(d, i\_1, \dots, i\_n)) & \text{in} \\ \mathsf{1}\mathbf{et}\ d\_{ar}' = \left\[\mathbf{o} := \mathbf{a}\mathbf{r}(\mathbf{i}\_1, \dots, \mathbf{i}\_n)\right]l\_{ar}(d\_{ar}) & \text{in} \\ \mathsf{forget}(d, o) \sqcap \mathsf{convert}\_{\omega}^{-1}(d\_{ar}') & \text{otherwise} \end{cases} \end{aligned}$$

To keep this transfer function simple, its input is a set of concrete indices and its output a set of concrete values that are retrieved by looking up the indexed elements in the array (after initialization). This makes it necessary for convertar to concretize the abstract inputs to a disjunction of (concrete) tuples (i<sup>1</sup> , . . . , in) for the read indices. Similarly, convert<sup>−</sup><sup>1</sup> ar converts the disjunction of (concrete) values back to an element of domain D.

Let us consider the concrete example in Fig. 2 to illustrate this more clearly. Line 1 initializes an array that is never again written to. On line 2, a nondeterministic value is assigned to variable idx, and the subsequent assumestatement constrains its value to be in the interval from 0 to 6. The assertion on line 5 checks that element elem, which is read from the array (on line 4),

```
1 int arr [] = {0 , 1 , 1 , 2 , 3 , 5 , 8 , 13};
2 int idx = *;
3 assume (0 <= idx && idx <= 6);
4 int elem = arr [ idx ];
5 assert ( elem < 13);
```
Figure 2: Example illustrating the specialized array domain.

is less than 13. Let us assume that we want to analyze the code by combining the numerical Intervals domain with our array domain Dar ; in other words, we assume D is instantiated with Intervals. In the pre-state of the array read, the analysis infers that the abstract value for idx is interval [0, 6]. When computing the post-state for the read operation, the analysis converts this interval to the concrete set of indices {0, 1, 2, 3, 4, 5, 6} via convertar. The transfer function for the array domain then looks up the (concrete) elements for each index to obtain the (concrete) set {0, 1, 2, 3, 5, 8}. Before returning this set to the Intervals domain, the analysis applies convert<sup>−</sup><sup>1</sup> ar to obtain the abstract value [0, 8]. This post-state allows the numerical domain to prove the assertion.

Note that this array domain is not specific to controllers such as the one used in our Racetrack variants. In fact, one could consider using it to more precisely analyze other programs with complex arrays that are initialized at runtime; a concrete example would be high-performance hash functions that often rely on fixed lookup tables.

Even more generally, the domains we sketched above suggest that our approach is also applicable to other scenarios; for instance, when a piece of code is too challenging to handle by a generic program analysis, and a simple summary or specification would result in unacceptable loss of precision.

#### 3.2 Neural-Network Analysis

AI<sup>2</sup> [27] was the first tool and technique for verifying robustness of neural networks using abstract interpretation. Eran is a successor of AI<sup>2</sup> ; it incorporates specialized transfer functions and abstract domains, such as DeepZ [56] (a variant of Zonotopes [28]) and DeepPoly [57] (a variant of Polyhedra [19]). Meanwhile, DeepSymbol [41] extended AI<sup>2</sup> with a novel symbolic-propagation technique. In the following, we first provide an overview of the techniques in Eran and DeepSymbol. Then, we describe how their domains can be used to implement the specialized transfer function from Dnn that was introduced in Sect. 3.1. On a high level, even though we are not concerned with robustness properties in this work, we re-purpose components of these existing tools to effectively check safety properties of heterogeneous systems that use neural networks.

The main goal behind verifying robustness of a neural network is to provide guarantees about whether it is susceptible to adversarial attacks [31,12,50,44]. Such attacks slightly perturb an original input (e.g., an image) that is classified correctly by the network (e.g., as a dog) to obtain an adversarial input that is classified differently (e.g., as a cat). Given a concrete input (e.g., an image), existing tools detect such local-robustness violations by expressing the set of all perturbed inputs (within a bounded distance from the original according to a metric, such as L<sup>∞</sup> [35]) and "executing" the neural network with this set of inputs to obtain a set of outcomes (or labels). The network is considered to be locally robust if there are no more than one possible outcome.

Existing techniques use abstract domains to express sets of inputs and outputs, and define specialized transfer functions to capture the operations (e.g., affine transforms and ReLUs) that are required for executing neural networks. For instance, Eran uses the DeepPoly [57] domain that captures polyhedral constraints and incorporates custom transfer functions for affine transforms, Re-LUs, and other common neural-network operations. DeepSymbol propagates symbolic information on top of abstract domains [65,41] to improve its precision. The key insight is that neural networks make extensive use of operations that apply linear combinations of arguments, and symbolic propagation is able to track linear-equality relations between variables (e.g., activation values of neurons).

Both Eran and DeepSymbol have the following in common: they define an abstract semantics for reasoning about neural-network operations and for computing an abstract set of outcomes from a set of inputs. We leverage this semantics to implement the specialized transfer function [[o := nn(i1, ..., in)]]Dnn (dnn) from Sect. 3.1.

#### 4 Experimental Evaluation

To evaluate our technique, we aim to answer the following research questions:


RQ3: How does a more complex benchmark affect the verification results?

RQ4: How does the neural-network analyzer affect the verification results?

#### 4.1 Benchmarks

We run our experiments on variants of Racetrack, which is a popular benchmark in the AI community [4,5,32,46,52,53] and implements the pseudo-code from Alg. 1 in C (see Sect. 2 for a high-level overview of the benchmark).

The Racetrack code<sup>10</sup> is significantly more complicated than the pseudocode in Alg. 1 would suggest; more specifically, it consists of around 400 lines of C code and invokes a four-layer fully connected neural network—with 14 inputs, 9 outputs, and 64 neurons per hidden layer (using ReLU activation functions). To name a few sources of complexity, the currentState does not just denote a single value, but rather the position of the vehicle on the map, the magnitude and direction of its velocity, its distance to goal locations, and its distance to obstacles. As another example, the Move function runs the trajectory of the

<sup>10</sup> https://github.com/Practical-Formal-Methods/Racetrack-Benchmark

vehicle from the old to the new state while determining whether there are any obstacles in between.

For simplicity, the code does not use floating-point numbers to represent variables, such as position, velocity, acceleration, and distance. Therefore, the program analyzer does not need to reason about floating-point numbers, which is difficult for Crab and most other tools. However, this does not semantically affect the neural network or its analysis, both of which do use floats. An interface layer converts the input features, tracked as integers in the controller, to normalized floats for the neural-network analysis. The output from the neuralnetwork analysis is a set of floating-point intervals, which are logically mapped to integers representing discrete possible actions at a particular state.

We evaluate our approach on 26 variants of Racetrack, which differ in the following aspects of the analyzed program or neural network.

Maps. We adopt three Racetrack maps of varying complexity from related work [5,32], namely barto-small (bs) of size 12 × 35, barto-big (bb) of size 30 × 33, and ring (r) of size 45 × 50. The size of a map is measured in terms of its dimensions (i.e., width and height). The map affects not only the program behavior, but also the neural network that is invoked. The latter is due to the fact that we train custom networks for different maps.

Neural-network quality. The neural network (line 9 of Alg. 1) is trained, using reinforcement learning [60], to predict an acceleration given a vehicle state, that is, the position of the vehicle on the map, the magnitude and direction of its velocity, its distance to goal locations, and its distance to obstacles. As expected, the quality of the neural-network predictions depends on the amount of training. In our experiments, we use well (good), moderately (mod), and poorly (poor) trained neural networks. We use the average reward at the end of the training process to control the quality. More details are provided in RQ2.

Noise. We complicate the Racetrack benchmark by adding two sources of non-determinism, namely environment (env) and neural-network (nn) noise. Introducing such noise is common practice in reinforcement learning, for instance, when modeling real-world imperfections, like slippery ground.

When environment noise is enabled, the controller might zero the vehicle acceleration (in practice, with a small probability), instead of applying the acceleration predicted by the neural network for the current vehicle state. This source of non-determinism is implemented by the if-statement on line 6 of Alg. 1. Environment noise may be disabled for states that are too close to obstacles to allow the vehicle to avoid definite crashes by adjusting its course according to the neural-network predictions. The amount of environment noise is, therefore, controlled by the distance to an obstacle (od) at which we disable it. For example, when od = 3, environment noise is disabled for all vehicle states that are at most 3 cells away from any obstacle. Consequently, when od = 1, we have a more noisy environment. Note that we do not consider od = 0 since the environment would be too noisy to verify safety for any start state.

Note that environment noise is not meant to represent realistic noise, but rather to make the verification task more challenging. However, it is also not entirely unrealistic and can be viewed as "necessarily rectifying steering course close to obstacles". Non-deterministically zeroing acceleration is inspired by related work [32].

For a given vehicle state, the neural network computes confidence values for each possible acceleration; these values sum up to 1. Normally, the predicted acceleration is the one with the largest confidence value, which however might not always be high. When neural-network noise is enabled, the network analyzer considers any acceleration for which the inferred upper bound on the confidence value is higher than a threshold . For example, when = 0.25, any acceleration whose inferred confidence interval includes values greater than 0.25 might be predicted by the neural network. Consequently, for lower values of , the neural network becomes more noisy. Such probabilistic action selection is widely used in reinforcement learning [55].

Each of these two sources of noise—env and nn noise—renders the verification of a neural-network controller through enumeration of all possible execution paths intractable: due to the non-determinism, the number of execution paths from a given initial state grows exponentially with the number of control iterations (e.g., the main loop on line 5 of Alg. 1). In our Racetrack experiments, the bound on the number of loop iterations is 50, and as a result, the number of execution paths from any given initial state quickly becomes very large. By statically reasoning about sets of execution paths, our approach is able to more effectively handle challenging verification tasks in comparison to exhaustive enumeration.

Lookahead functionality. We further complicate the benchmark by adding lookahead functionality (not shown in Alg. 1), which aims to counteract incorrect predictions of the neural network and prevent crashes. In particular, when this functionality is enabled, the controller simulates the vehicle trajectory when applying the acceleration predicted by the neural network a bounded number of additional times (denoted la). For example, when la = 3, the controller invokes the neural network 3 additional times to check whether the vehicle would crash if we were to consecutively apply the predicted accelerations. If this lookahead functionality indeed foresees a crash, then the controller reverses the direction of the acceleration that is predicted for the current vehicle state on line 9 of Alg. 1. Conceptually, the goal behind our lookahead functionality is similar to the one behind shields [2]. While lookahead is explicitly encoded in the program as code, shields provide a more declarative way for expressing such safeguards.

#### 4.2 Implementation

For our implementation<sup>11</sup>, we extended Crab to support specialized abstract domains as described in Sect. 3. To integrate DeepSymbol and Eran, we implemented a thin wrapper around these tools to enable their analysis to start

<sup>11</sup> Our source code can be found at https://github.com/Practical-Formal-Methods/ clam-racetrack and an installation at https://hub.docker.com/r/practicalformalmethods/ neuro-aware-verification.

Figure 3: Verification results for RQ1, where env(od = 3). The maps on the left are bs (top) and bb (bottom), and the map on the right is r.

from a set of abstract input states and return a set of abstract output states. Moreover, our wrappers provide control over the amount of neural-network noise (through threshold ).

#### 4.3 Setup

We use deep Q-learning [47] to train a neural network for each Racetrack variant. We developed all training code in Python using the TensorFlow<sup>12</sup> and Torch<sup>13</sup> deep-learning libraries.

We configure Crab to use the Boxes abstract domain [33], DeepSymbol to use Intervals [18] with symbolic propagation [41], and Eran to use DeepPoly [57]. When running the analysis, we did not specify a bound on the available time or memory; consequently, none of our analysis runs led to a time-out or mem-out. Regarding time, we report our results in the following, and regarding memory, our technique never exceeded 13.5GB when analyzing all start states of any map.

We performed all experiments on a 48-core Intel <sup>R</sup> Xeon <sup>R</sup> E7-8857 v2 CPU @ 3.6GHz machine with 1.5TB of memory, running Debian 10 (buster).

#### 4.4 Results

We now present our experimental results for each research question.

RQ1: How effective is our technique in verifying goal reachability and crash avoidance? To evaluate the effectiveness of our technique in proving these system properties, we run it on the following benchmark variants: bs, bb, and r maps, good neural networks, env noise with od = 1, 2, 3, and la = 0 (i.e., no lookahead). The verification results are shown in Figs. 3, 4, and 5 (see Sect. 2 for the semantics of cell colors). These results are achieved when combining Crab with DeepSymbol, but the combination with Eran is comparable (see RQ4).

As shown in Fig. 3, for the vast majority of initial vehicle states, our technique is able to verify goal reachability and crash avoidance. This indicates that our integration of the controller and neural-network analyses is highly precise. As

<sup>12</sup> https://www.tensorflow.org

<sup>13</sup> http://torch.ch/

Figure 4: Verification results for RQ1, where env(od = 2).

Figure 5: Verification results for RQ1, where env(od = 1).


Table 1: Performance results for RQ1.

expected, the more env noise we add (i.e., the smaller the od values), the fewer states we prove safe (see Figs. 4 and 5).

Tab. 1 shows the performance of our technique. The first four columns of the table define the benchmark settings, the fifth the neural-network analyzer, and the last three show the total running time of our technique for all start states, the average time per state, and the percentage of this time spent on the neural-network analysis. Note that we measure the total time when running the verification tasks (for each start state) in parallel<sup>14</sup>; the average time per state is independent of any parallelization. We do not show performance results for different od values since environment noise does not seem to have a significant impact on the analysis time.

Recall from Sect. 2 that, without our technique, it is currently only possible to verify properties of a heterogeneous system like Racetrack by considering the controller in isolation, ignoring the call to the neural network, and havocking its return value. We perform this experiment for all of the above benchmark variants and find that Crab alone is unable to prove goal reachability or crash

<sup>14</sup> https://doi.org/10.5281/zenodo.1146014

Figure 6: Verification results for RQ2, with mod neural networks.

Figure 7: Verification results for RQ2, with poor neural networks.

avoidance for any initial vehicle state; in other words, all states are red. This is the case even when replacing Boxes with Polyhedra—these two domains perform the most precise analyses in Crab.

RQ2: How does the quality of the neural network affect the verification results? To evaluate this research question, we run our technique on the following benchmark variants: bs, bb, and r maps, mod and poor neural networks, env noise with od = 3, and la = 0. The verification results are shown in Figs. 6 and 7; they are achieved by combining Crab with DeepSymbol.

In deep Q-Learning (see Sect. 4.3), a neural network is trained by assigning positive or negative rewards to its predictions. A properly trained network learns to collect higher rewards over a run. Given this, we assess the quality of networks by considering average rewards over 100 runs from randomly selected starting states. If the network collects more than 70% of the maximum achievable reward, we consider it a good agent. If it collects ca. 50% (or respectively, ca. 30%) of the maximum reward, we consider it a mod (respectively, poor) agent.

In comparison to Fig. 3, our technique proves safety of fewer states since the quality of the networks is worse. Analogously, more states are verified in Fig. 6 than in Fig. 7. Interestingly, for bb, our technique proves crash avoidance (blue cells) more often when using a poor neural network (Fig. 7) instead of a mod one (Fig. 6). We suspect that this is due to the randomness of the training process and the training policy, which penalizes crashes more than getting stuck; so, a poor neural network might initially only try to avoid crashes.

Regarding performance, the analysis time fluctuates when using mod and poor neural networks. There is no pattern even when comparing the time across

Figure 8: Verification results for RQ3, where la = 0, 1, 3 from left to right.

Table 2: Performance results for RQ3.


different map sizes for equally trained networks. This is to be expected as neural networks may behave in unpredictable ways when not trained properly (e.g., the vehicle may drive in circles), which affects the performance of the analysis.

RQ3: How does a more complex benchmark affect the verification results? We complicate the benchmark by adding lookahead functionality, i.e., resulting in la additional calls to the neural network per vehicle move (see Sect. 4.1 for more details). Since well trained neural networks would benefit less from this functionality, we use mod networks in these experiments. In particular, we run our technique on the following benchmark variants: bs map, mod neural networks, env noise with od = 3, and la = 0, 1, 3. The verification results are shown in Fig. 8; they are achieved by combining Crab with DeepSymbol.

As la increases, the benchmark becomes more robust, yet more complex. We observe that, for larger values of la, our technique retains its overall precision despite the higher complexity; e.g., there are states that are verified with la = 3 or 1 but not with 0. However, there are also few states that are verified with la = 1 but not with 3. In these cases, the higher complexity does have a negative impact on the precision of our analyses.

Tab. 2 shows the performance of our technique for these experiments. As expected, the analysis time increases as the benchmark complexity increases.

RQ4: How does the neural-network analyzer affect the verification results? We first compare DeepSymbol with Eran on the following benchmark variants: bs, bb, and r maps, good neural networks, env noise with od = 3, and la = 0. The verification results achieved when combining Crab with Eran are shown in Fig. 9; compare this with Fig. 3 for DeepSymbol.

We observe the results to be comparable. With DeepSymbol, we color 216 cells green and 1 blue for bs, 455 green for bb, and 499 green and 6 blue for r. With Eran, the corresponding numbers are 214 cells green and 7 blue for bs, 459 green and 4 blue for bb, and 485 green and 71 blue for r. We observe similar results for other benchmark variants, but we omit them here.

Comparing the two neural-network analyzers becomes more interesting when we enable nn noise. More specifically, we run our technique on the following benchmark variants: bs, bb, and r maps, good networks, nn noise with = 0.25,

Figure 9: Verification results for RQ4 with Eran, where env(od = 3).

Figure 10: Verification results for RQ4 with DeepSymbol, where nn( = 0.25).

Figure 11: Verification results for RQ4 with Eran, where nn( = 0.25).

and la = 0. Fig. 10 shows the verification results when combining Crab with DeepSymbol, and Fig. 11 when combining Crab with Eran.

As shown in the figures, the verification results are slightly better with Eran. In particular, with DeepSymbol, we color 170 cells green for bs, 109 green for bb, and 195 green for r. With Eran, the corresponding numbers are 181 cells green for bs, 131 green for bb, and 203 green for r. Despite this, the performance of our technique can differ significantly depending on whether we use DeepSymbol or Eran, as shown in Tab. 3. One could, consequently, imagine a setup where multiple neural-network analyzers are run in parallel for each verification task. If time is of the essence, we collect the results of the analyzer that terminates first. If it is more critical to prove safety, then we could combine the results of all analyzers once they terminate.


Table 3: Performance results for RQ4.

#### 5 Related Work

The program-analysis literature provides countless examples of powerful analysis combinations. To name a few, dynamic symbolic execution [29,10] and hybrid fuzzing [45,58,69] combine random testing and symbolic execution, numerous tools broadly combine static and dynamic analysis [6,20,21,49,30,13,14,22,61], and many tools combine different types of static analysis [7,1,34]. In contrast to neuro-aware program analysis, almost all these tools target homogeneous, instead of heterogeneous, systems. Concerto [61] is a notable exception that targets applications using frameworks such as Spring and Struts. It combines abstract and concrete interpretation, where, on a high level, concrete interpretation is used to analyze framework code, whereas abstract interpretation is used for application code. Instead of building on existing analyzers, as in our work, Concerto introduces a designated technique for analyzing framework code.

There is recent work that focuses specifically on verifying hybrid systems with DNN controllers [24,59]. Unlike in our work, they do not analyze programs that interact with the network, but models; in one case, ordinary differential equations describing the hybrid system [24], and in the other, a mathematical model of a LiDAR image processor [59]. In this context of hybrid systems with DNN controllers, there is also work that takes a falsification approach to the same problem [62,70,23]. They generate corner test cases that cause the system to violate a system-level specification. Moreover, existing reachability analyses for neural networks [25,42,67,68] consider linear or piecewise-linear systems, instead of programs invoking them.

Kazak et al. [39] recently proposed Verily, a technique for verifying systems based on deep reinforcement learning. Such systems have been used in various contexts, such as adaptive video streaming, cloud resource management, and Internet congestion control. Verily builds on Marabou [38], a verification tool for neural networks, and aims to ensure that a system achieves desired service-level objectives (expressed as safety or liveness properties). Other techniques use abstract interpretation to verify robustness [27,57,41] or fairness properties [63] of neural networks. Furthermore, there are several existing techniques for checking properties of neural networks using SMT solvers [37,38,36] and global optimization techniques [54]. In contrast to our approach, they focus on verifying properties of the network in isolation, i.e., without considering a program that queries it. However, we re-purpose two of the above analyzers [57,41] to infer invariants over the neural-network outputs. Gros et al. [32] make use of statistical model checking to obtain quality-assurance reports for a neural network in a noisy environment. Their approach provides probabilistic guarantees about checked properties, instead of definite ones like in our work, and also does not analyze a surrounding system.

#### 6 Conclusion

Many existing software systems are already heterogeneous, and we expect the number of such systems to grow further. In this paper, we present a novel approach to verifying safety properties of such systems that symbiotically combines existing program and neural-network analyzers. Neuro-aware program analysis is able to effectively prove non-trivial system properties of programs invoking neural networks, such as the 26 variants of Racetrack.

Acknowledgements. We are grateful to the reviewers for their constructive feedback. This work has been supported by DFG Grant 389792660 as part of TRR 248 (see https://perspicuous-computing.science). Jorge Navas has been supported by NSF Grant 1816936. Holger Hermanns and Jianlin Li have been supported by Guangdong Province Science Grant 2018B010107004.

#### References


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/ 4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Scalable Polyhedral Verification of Recurrent Neural Networks**

Wonryong Ryou1(B) , Jiayu Chen<sup>1</sup>, Mislav Balunovic<sup>1</sup>, Gagandeep Singh<sup>2</sup>, Andrei Dan<sup>3</sup>, and Martin Vechev<sup>1</sup>

<sup>1</sup> Department of Computer Science, ETH Zürich, Zurich, Switzerland wryou@ethz.ch

<sup>2</sup> VMWare Research and Department of Computer Science, UIUC, Champaign, USA <sup>3</sup> Hitachi Power Grids Research, Zurich, Switzerland

**Abstract.** We present a scalable and precise verifier for recurrent neural networks, called Prover based on two novel ideas: (i) a method to compute a set of polyhedral abstractions for the non-convex and nonlinear recurrent update functions by combining sampling, optimization, and Fermat's theorem, and (ii) a gradient descent based algorithm for abstraction refinement guided by the certification problem that combines multiple abstractions for each neuron. Using Prover, we present the first study of certifying a non-trivial use case of recurrent neural networks, namely speech classification. To achieve this, we additionally develop custom abstractions for the non-linear speech preprocessing pipeline. Our evaluation shows that Prover successfully verifies several challenging recurrent models in computer vision, speech, and motion sensor data classification beyond the reach of prior work.

**Keywords:** Robustness verification · Polyhedral abstraction · Recurrent neural networks · Long short-term memory · Abstraction refinement · Speech classifier verification

#### **1 Introduction**

Recurrent neural networks (RNNs) are widely used to model long-term dependencies in lengthy sequential signals [11,27,43]. Prior work has demonstrated the susceptibility of RNNs to adversarial perturbations of its inputs [28], exposing security vulnerabilities of state-of-the-art RNNs when used in domains such as speech recognition [8,22], malware detection [16], and others. Thus, verifying the robustness of recurrent architectures is critical for their safe deployment. While there has been considerable interest in certifying the robustness of feedforward image classifiers [4,12,13,23,32,37,39,47], less attention has been given to recurrent architectures. As a result, current certification solutions do not scale beyond simple models and datasets, which limits their practical applicability. Further, there has been no work on verifying real-world use cases of RNNs. In this paper, we address both of these challenges and present the first precise and scalable verifier for RNNs based

**Fig. 1.** Certification of recurrent architectures using Prover: utterance "stop" with perturbations is correctly classified. Possible perturbations are captured and propagated through the system, then refined backward for improved precision. (Color figure online)

on abstract interpretation [10], which enables us to certify robustness of realistic speech recognition systems.

We illustrate the problem setting and overall flow in Fig. 1. Here, a speech recognition model based on the Long Short-Term Memory (LSTM) architecture [15] receives a signal encoding the utterance of "stop" by a human. As such models are usually employed in noisy environments, they must robustly classify variations (e.g., voice changes) to the utterance "stop". However, recent work [8] has shown the model may be fooled into classifying the utterance as "go". It is important to prove such mis-classifications are not possible, thus avoiding a potential exploitation by an adversary, for instance in automated traffic control settings (which can lead to accidents). Our goal is to design a verifier that can formally establish the robustness of such models against noise-induced perturbations. We focus on LSTMs, as they are the most widely used form of RNNs, but our methodology can be easily extended to other architectures (e.g., Gated Recurrent Unit (GRU) [9]). Figure 1 shows how our proposed verifier, called Prover, (Polyhedral Robustness Verifier of RNNs) automatically verifies the robustness of the model. Here, the labeled rectangles represent operations in the network. The "Preprocess" box captures domain-specific pre-processing operations (typically present when using RNNs, e.g., speech processing). In our method, we first compute a polyhedral abstraction capturing all speech signals given as input to the model under the given perturbation budget. At each timestep *i*, the pre-processing operation receives a polyhedron *s*(*i*) and produces an output polyhedron *x*(*i*) . This shape is then propagated symbolically through the LSTM and the post-processing stage, resulting in a polyhedral output shape, denoted as *z* (blue shape in Fig. 1).

**Key Challenge: Polyhedral Abstractions for LSTMs.** The main challenge in certifying LSTMs is the design of precise and scalable polyhedral abstract transformers for the non-linear operations employed in LSTMs: given a polyhedral shape capturing hidden states *h*(*i*−1), to produce the shape capturing the next set of hidden states *h*(*i*). A recent method [21] computes this based on gradient-based optimization but suffers from two main limitations. First, the optimization procedure is computationally expensive and does not scale to realistic use cases. Second, the method lacks convergence and optimality guarantees. To address these issues, we introduce a novel technique based on a combination of sampling, linear programming, and Fermat's theorem [1], which significantly improves the precision and scalability compared to prior work [21], while offering asymptotic guarantees of convergence towards the optimal solution.

**Refinement via Optimization.** To certify robustness, we must verify that each concrete point in the output shape *z* corresponds to the correct label "stop". However, *z* can contain, due to over-approximation, spurious incorrect concrete points (it intersects the red region representing incorrect outputs). To address this issue, we form a loss based on the output shape, backpropagate the gradient of this loss through the timesteps and adjust the polyhedral abstractions in each LSTM unit to decrease the loss. The goal is to refine the abstraction, guided by the certification task. We illustrate this process in Fig. 1 using the purple backward arrow with the refined polyhedral abstraction shown in purple. Using the refined abstraction, the new output shape *z* (purple polygon) lies completely inside the green region of the output space, meaning it provably contains only correct output vectors (corresponding to "stop"), and hence certification succeeds. Overall, our method significantly increases the precision of end-to-end RNN certification without introducing high runtime costs.

#### **Key Contributions.** Our main contributions are:


#### **2 Related Work**

While the first adversarial examples for neural networks were found in computer vision [6,41], recent work also showed the vulnerability of RNNs [28]. Modern speech recognition systems, based on RNNs, were shown susceptible to small noise crafted by an adversary using white-box attacks [7,8], achieving a 100% success rate against DeepSpeech [14], a state-of-the-art speech-to-text engine. These were later followed by attacks based on universal perturbation [26] and temporal dependency [46]. Recent work [22,31] demonstrates that adversarial

examples for audio classifiers are realizable in the real-world. While giving an empirical estimate of the vulnerability of RNNs, these works do not provide any formal guarantees, which is the goal of our work.

There have also been recent works on the verification of RNNs. [2] propose the certification of RNNs based on mixed-integer linear programming, which only works for ReLU-based networks and does not consider LSTMs, which use sigmoid and tanh activations. [45] propose an input discretization method to certify video models that are a combination of CNNs and RNNs. However, discretization does not scale to the perturbations we consider in our work. [18] propose to verify RNNs by automatically inferring temporal homogeneous invariants using binary search. However, their approach is limited to vanilla RNNs and does not apply to the more commonly used LSTM networks considered in this work. [19] propose the statistical variant of Angulin's algorithm [3] for probabilistic verification and counterexample generation for RNNs, however they cannot provide deterministic guarantees as our work. The work most related to ours is POPQORN [21] which uses expensive gradient-based optimizations for every operation in the network. We experimentally show that it does not scale to practical applications such as speech classification.

#### **3 Background**

We first define the threat model and then present all operations that are part of the verification procedure, including speech preprocessing and LSTM updates.

#### **3.1 Threat Model**

We use a threat model based on the *L*∞-norm, where an attacker can change each element of a correctly classified input vector *s* by an amount ≤ *-* <sup>∈</sup> <sup>R</sup> [8]. Therefore, our input region can be represented as a conjunction of intervals [*s<sup>i</sup>* − *-, s<sup>i</sup>* + *-*], where *s<sup>i</sup>* is the *i*-th element of *s*. The measure of signal distortion in this setting are decibels (dB) defined as:

$$dB(\mathbf{s}) = \max\_{i} 20 \cdot \log\_{10}(|s\_i|);\ dB\_s(\boldsymbol{\delta}) = dB(\boldsymbol{\delta}) - dB(\mathbf{s})$$

The quieter the perturbation is, the smaller *dBs*(*δ*) is. We fix the *dBs*(*δ*) =: *-* as *dB perturbation* and focus on verifying that the model classifies correctly all signals *s* possible under our threat model.

#### **3.2 Long Short-Term Memory (LSTM)**

LSTM architectures [15] are popular for handling sequential data as they can utilize long-term dependencies. These dependencies are passed through time using two state vectors for the timestep *t*: cell state *c*(*t*) and hidden state *h*(*t*). These state vectors are updated using the following formulas:

**Fig. 2.** LSTM cell: *f*(*t*) <sup>0</sup> *, i* (*t*) <sup>0</sup> *, <sup>o</sup>*(*t*) <sup>0</sup> , and *c*˜ (*t*) <sup>0</sup> represent the pre-activated gates.

$$\begin{cases} \mathbf{f}\_{0}^{(t)} = [\mathbf{z}^{(t)}, \mathbf{h}^{(t-1)}] \mathbf{W}\_{f} + \mathbf{b}\_{f} & \quad \mathbf{i}\_{0}^{(t)} = [\mathbf{z}^{(t)}, \mathbf{h}^{(t-1)}] \mathbf{W}\_{i} + \mathbf{b}\_{i} \\\ \mathbf{o}\_{0}^{(t)} = [\mathbf{z}^{(t)}, \mathbf{h}^{(t-1)}] \mathbf{W}\_{o} + \mathbf{b}\_{o} & \quad \mathbf{\tilde{c}}\_{0}^{(t)} = [\mathbf{z}^{(t)}, \mathbf{h}^{(t-1)}] \mathbf{W}\_{\tilde{c}} + \mathbf{b}\_{\tilde{c}} \\\ \mathbf{c}^{(t)} = \sigma(\mathbf{f}\_{0}^{(t)}) \odot \mathbf{c}^{(t-1)} + \sigma(\mathbf{i}\_{0}^{(t)}) \odot \tanh(\tilde{c}\_{0}^{(t)}) \qquad \mathbf{h}^{(t)} = \sigma(\mathbf{o}\_{0}^{(t)}) \odot \tanh(\mathbf{c}^{(t)}) \end{cases}$$

where [·*,* ·] is the horizontal concatenation of two row vectors, *W*· and *b*· are the kernel and bias of the cell, respectively, and *σ* is the sigmoid function. At timestep *t*, vectors *f*(*t*) <sup>0</sup> *, i* (*t*) <sup>0</sup> *, <sup>o</sup>*(*t*) <sup>0</sup> *, c*˜ (*t*) <sup>0</sup> represent pre-activations of the forget gate, input gate, output gate and the candidate gate, respectively. We show an illustration of an LSTM cell in Fig. 2. We treat *σ* and tanh as forms of activation functions, which is why we define the LSTM using pre-activations.

Intuitively, the *input gate* transforms the input vector, the *forget gate* filters the information from the previous cell state, the *candidate gate* prepares the candidate cell state, and the *output gate* transforms the current hidden state. All of these gates receive as input the hidden state *h*(*t*−1) of the previous cell and the input *x*(*t*) representing the current frame. This recurrent architecture allows inputs with arbitrary length, enabling LSTMs to handle temporal data, e.g., speech processing.

#### **3.3 Speech Preprocessing**

Though there have been various works that operate directly on the raw signal [29,36], speech signals are commonly preprocessed using the *filterbank* or *log Melfilterbank energy* methods. The result is a vector of coefficients whose elements contain log-scaled values of filtered spectra, one for every Mel-frequency. This method models the non-linear human acoustic perception as power spectrum filters based on Mel-frequencies. The input signal is split into several (possibly overlapping) frames for granular analysis, and the following steps are applied:

1. *Pre-emphasizing and windowing* are preprocessing stages on the raw signals. Speech signals tend to have larger and smoother low-frequency samples and smaller and fluctuating high-frequency samples. *Pre-emphasizing* is a process

of subtracting the adjacent sampled values multiplied by a scalar parameter (*s*(*i*) *<sup>j</sup>* <sup>−</sup>*αs*(*i*) *<sup>j</sup>*−<sup>1</sup>, commonly *<sup>α</sup>* = 0*.*97). This alleviates the unbalanced distribution of signal strength along with the frequency. *Windowing* involves multiplication of each sampled value and 'windows' according to their indices. The window here refers to a Hamming window, which is a bell-like curve with peak in the middle of the frame and drops at the side. It reduces the border effects on each frame by suppressing the values near the border with smaller values.


Following [35], each step can be represented as a distinct matrix operation. It allows us to decompose and rearrange the steps into slightly different stages:


We use the resulting *<sup>X</sup>* = [*x*(1) ··· *<sup>x</sup>*(*T*)] as the input to the neural network.

#### **3.4 Verification Using DeepPoly Abstract Domain**

DeepPoly [39] is a sub-polyhedral abstract domain that associates a lower and an upper polyhedral bound and interval bounds per neuron. It is faster than Polyhedra [40] and more precise than other weakly relational domains such as Octagons [25], Zones [24], and Zonotopes [38] when analyzing neural networks. Previously, it has been suceesfully applied for verifying feedforward networks in [4,39]. Formally, let X = {*x*1*, x*2*,...,xn*} be an ordered set of neurons such that the neurons in layer *l* appear before the neurons of layer *l > l*. DeepPoly associates with each neuron *x<sup>j</sup>* , both interval *l<sup>j</sup>* ≤ *x<sup>j</sup>* ≤ *u<sup>j</sup>* and polyhedral bounds - *i<j a<sup>i</sup>* ·*x<sup>i</sup>* +*b* ≤ *x<sup>j</sup>* ≤ - *i<j a <sup>i</sup>* ·*x<sup>i</sup>* +*b* where *l<sup>j</sup> , u<sup>j</sup> , ai, a <sup>i</sup>, b, b* <sup>∈</sup> <sup>R</sup>∪ {∞}. DeepPoly is exact for affine transformations which are frequently applied both in the speech preprocessing pipeline and the LSTM unit. DeepPoly loses precision

for the non-linear operations in LSTMs. We note that computing polyhedral bounds on their output is more challenging than for feedforward networks.

The precision of the DeepPoly approximation for the non-linear operations depends on the tightness of the interval bounds of the neurons that are input to the non-linear operations. DeepPoly provides a scalable and precise method called *backsubstitution* for optimizing a linear expression within a region defined by the set of DeepPoly constraints. It does so by recursively substituting the bounding linear expressions of target neurons with the polyhedral bounds of previous layers' neurons until reaching the input neurons. It then uses the concrete bounds of the input neurons for computing the result. Backsubstitution is used for computing the interval bounds of neurons input to the non-linear operations as well as for bounding the difference between the neurons in the output layer needed to prove robustness. We refer the reader to [39] for details of the backsubstitution.

#### **4 Overview of Prover**

This section illustrates the workings of Prover on a small example. Our goal is to certify the robustness of a single LSTM cell on the input *x* ∈ [−1*.*2*,* 1*.*2]. For this example, we assume that there are two output classes and all intermediate LSTM gates {*i, f, c*˜*, o*} share the same weights and biases:

$$\{\dot{\mathfrak{a}}, \mathfrak{f}, \check{\mathfrak{c}}, \mathfrak{o}\} = \begin{bmatrix} 1 \\ 0.5 \end{bmatrix} x + \begin{bmatrix} 0 \\ 1 \end{bmatrix}, \qquad \mathfrak{c} = \sigma(\dot{\mathfrak{e}}) \odot \tanh(\check{\mathfrak{c}}), \qquad \mathfrak{h} = \sigma(\mathfrak{o}) \odot \tanh(\mathfrak{c}).$$

The correct output here is *h*<sup>2</sup> and to certify robustness we need to prove that *h*<sup>2</sup> − *h*<sup>1</sup> *>* 0 holds for all inputs *x*. In other words, min *h*<sup>2</sup> − *h*<sup>1</sup> *>* 0.

**Polyhedral Abstraction.** We build our verifier based on the DeepPoly [39] abstraction since DeepPoly outperforms the interval analysis and other competitive domains, as Sect. 3.4 states.

**Challenges in Computing Polyhedral Bounds for LSTMs.** The composed binary non-linear operations applied in LSTMs such as *σ*(*x*) tanh(*y*) and *σ*(*x*)*y* are significantly more complex to handle than the ReLU, Sigmoid, and Tanh activations originally handled by [39]. This is because the non-linear operations in LSTMs mentioned above involve transcendental functions yielding non-linear 3D curves that are neither convex nor concave. The optimal polyhedral bounds for these operations have no closed-form solution and cannot be calculated by simple geometry or algebra. Further, obtaining such bounds is computationally expensive [21]. For example, obtaining the lower linear plane for bounding *σ*(*x*) tanh(*y*) is equivalent to solving a Lagrangian with 6 variables - 3 linear coefficients, 2 interval-bounded coordinates and 1 Lagrange multiplier for the constraint. In contrast, the optimal polyhedral bounds for ReLU, Sigmoid, and Tanh have closed form solutions, easily visualized in 2D.

**Precise Polyhedral Bounds via LP.** To overcome these challenges, we propose a generic approach based on linear programming (LP) to compute precise

polyhedral bounds. We illustrate our approach for calculating a lower polyhedral bound of *h*<sup>2</sup> = *σ*(*o*2) tanh(*c*2). First, we calculate the concrete intervals for the two target variables via *backsubstitution* [39], briefly described in Sect. 3.4. In our case, the target variables are *o*<sup>2</sup> and *c*<sup>2</sup> and the backsubstitution yields *o*<sup>2</sup> ∈ [0*.*4*,* 1*.*6] and *c*<sup>2</sup> ∈ [−0*.*79*,* 0*.*62]. Our abstraction can represent the affine transformations exactly. Therefore, we obtain the exact interval for *o*<sup>2</sup> = 0*.*5·*x*+1 via the backsubstitution whereas the obtained interval for *c*<sup>2</sup> is an overapproximate one. Then, we uniformly sample a set of points {(*x*1*, y*1)*, ...,*(*xn, yn*)} from the input domain [0*.*4*,* 1*.*6] × [−0*.*79*,* 0*.*62]. We solve the following optimization problem to calculate the lower polyhedral bound of *h*2:

$$\min\_{A\_l, B\_l, C\_l \in \mathbb{R}} \sum\_{i=1}^n \left( \sigma(x\_i) \tanh(y\_i) - \left( A\_l \cdot x\_i + B\_l \cdot y\_i + C\_l \right) \right),$$

subject to the constraint that *A<sup>l</sup>* · *x<sup>i</sup>* + *B<sup>l</sup>* · *y<sup>i</sup>* + *C<sup>l</sup>* ≤ *σ*(*xi*) tanh(*yi*) for each *i*. This is a linear program over three variables (*Al, Bl, Cl*) that can be solved efficiently in polynomial time. However, the obtained bound may not be sound as the sampled points do not fully cover the continuous input domain. To address this, we shift the plane downwards by an offset (decreasing *Cl*) equal to the maximum violation between *A<sup>l</sup>* ·*x*+*B<sup>l</sup>* ·*y*+*C<sup>l</sup>* and *h*<sup>2</sup> based on Fermat's theorem. After solving the linear program and the adjustment, we obtain *A<sup>l</sup>* = 0*.*04*, B<sup>l</sup>* = 0*.*46*, C<sup>l</sup>* = 0*.*01 which results in the following lower polyhedral bound to *h*2: *h*<sup>2</sup> ≥ *LB<sup>h</sup>*<sup>2</sup> = 0*.*04 · *o*<sup>2</sup> + 0*.*46 · *c*<sup>2</sup> + 0*.*01. We compute the upper bound to *h*<sup>2</sup> : *h*<sup>2</sup> ≤ *UB<sup>h</sup>*<sup>2</sup> analogously. After computing a polyhedral abstraction of each neuron, we calculate the lower bound of *h*<sup>2</sup> − *h*<sup>1</sup> via backsubstitution as follows:

$$\begin{aligned} &\min h\_2 - h\_1 \ge LB\_{h\_2} - UB\_{h\_1} \\ &\ge (0.04 \cdot o\_2 + 0.46 \cdot c\_2 + 0.01) - (-0.09 \cdot o\_1 + 0.66 \cdot c\_1 + 0.14) \\ &\ge 0.04 \cdot o\_2 + 0.46 \cdot (0.07 \cdot i\_2 + 0.27 \cdot g\_2 + 0.09) \\ &\quad + 0.09 \cdot o\_1 - 0.66 \cdot (-0.04 \cdot i\_1 + 0.38 \cdot g\_1 + 0.25) - 0.14 \\ &\ge 0.20 \cdot (0.5 \cdot x + 1) - 0.13 \cdot x - 0.10 \ge -0.03 \cdot x - 0.08 \ge -0.11. \end{aligned}$$

The precision of the bounds generated by our LP-based method increases with the number of samples yielding optimal bounds (in the sense of small gap) asymptotically. For our example, the computed bounds are optimal.

While our optimal bounds significantly improve precision compared to intervals, they are not sufficient to certify robustness. Prior work for ReLU networks [5,12,23] showed that the greedy approach of always selecting the optimal bounds minimizing the gap can yield less precise results than an adaptive strategy which computes bounds guided by the certification problem. Based on this observation, we introduce a novel approach based on splitting and gradient descent that computes polyhedral abstractions for non-linearities employed in LSTMs informed by the certification problem and proves that min *h*<sup>2</sup> − *h*<sup>1</sup> *>* 0 actually holds.

**Abstraction Refinement via Splitting and Gradient Descent.** While our method based on LP offers an efficient way to compute polyhedral abstraction of activation functions, its main limitation is that the abstraction cannot be refined based on the certification goal. In this work, we introduce a novel method where we first compute mutiple sound bounds for the neuron using our LP method and then automatically obtain a combination of the computed bounds that improves the lower bound of our certification objective *h*<sup>2</sup> −*h*<sup>1</sup> for each input example. As before, we use the backsubstitution to obtain the interval bounds for the input variables. Since the output of our LP method is sensitive to the choice of the sampled points, we split the original input region [*lx, ux*]×[*ly, uy*] to sample more effectively from smaller sub-regions thereby reducing the chances of missing an outlier. We found that splitting along the two diagonals of [*lx, ux*] × [*ly, uy*] into four triangular zones, denoted as T*k*, *k* ∈ {1*,* 2*,* 3*,* 4}, performs the best in our evaluation. We use T<sup>0</sup> to denote the original input region. Next, we calculate four additional planes, for both the upper and lower bounds, by sampling each subregion T*<sup>k</sup>* and then applying our LP method as before. We refer to each plane as a *candidate bound*:

$$\begin{aligned} \min\_{A\_l, B\_l, C\_l \in \mathbb{R}} & \sum\_{i=1}^n \left( \sigma(x\_i) \tanh(y\_i) - (A\_l \cdot x\_i + B\_l \cdot y\_i + C\_l) \right) \\ \text{subject to } & \bigwedge\_{i=1}^n A\_l \cdot x\_i + B\_l \cdot y\_i + C\_l \le \sigma(x\_i) \tanh(y\_i) \text{ where } (x\_i, y\_i) \sim \mathcal{T}\_k \end{aligned}$$

Using our LP based method, we obtain the following corresponding candidate polyhedral abstraction for *h*2, *LB<sup>k</sup> <sup>h</sup>*<sup>2</sup> for each T*<sup>k</sup>* in our example:

*<sup>h</sup>*<sup>2</sup> <sup>≥</sup> *LB*<sup>0</sup> *<sup>h</sup>*<sup>2</sup> = 0*.*<sup>04</sup> · *<sup>o</sup>*<sup>1</sup> + 0*.*<sup>46</sup> · *<sup>c</sup>*<sup>1</sup> + 0*.*01*, h*<sup>2</sup> <sup>≥</sup> *LB*<sup>1</sup> *<sup>h</sup>*<sup>2</sup> = 0*.*04 · *o*<sup>1</sup> + 0*.*46 · *c*<sup>1</sup> + 0*.*01 *<sup>h</sup>*<sup>2</sup> <sup>≥</sup> *LB*<sup>2</sup> *<sup>h</sup>*<sup>2</sup> = 0*.*<sup>13</sup> · *<sup>o</sup>*<sup>1</sup> + 0*.*<sup>63</sup> · *<sup>c</sup>*<sup>1</sup> <sup>−</sup> <sup>0</sup>*.*17*, h*<sup>2</sup> <sup>≥</sup> *LB*<sup>3</sup> *<sup>h</sup>*<sup>2</sup> = 0*.*04 · *o*<sup>1</sup> + 0*.*46 · *c*<sup>1</sup> + 0*.*01 *<sup>h</sup>*<sup>2</sup> <sup>≥</sup> *LB*<sup>4</sup> *<sup>h</sup>*<sup>2</sup> = 0*.*13 · *o*<sup>1</sup> + 0*.*63 · *c*<sup>1</sup> − 0*.*17

Note that *LB*<sup>0</sup> *<sup>h</sup>*<sup>2</sup> denotes the polyhedral abstraction calculated for the whole region, and there might be duplicate *LB*'s when the curve in the given subregion is concave. The final bound *LB<sup>h</sup>*<sup>2</sup> is a linear combination of *LB<sup>k</sup> h*2 :

$$LB\_{h\_2} = \sum\_{k=0}^{4} \lambda\_i \cdot LB\_{h\_2}^k, \qquad \sum\_{k=0}^{4} \lambda\_i = 1.$$

Our optimization algorithm, explained in Sect. 5.2, learns the values of *λ<sup>i</sup>* via gradient descent that maximizes min *h*<sup>2</sup> − *h*1. For our example, we obtain *λ* = (0*.*09*,* 0*.*13*,* 0*.*34*,* 0*.*09*,* 0*.*35) as the set of coefficients which results in a new lower bound of *h*<sup>2</sup> ≥ 0*.*10 · *o*<sup>2</sup> + 0*.*58 · *c*<sup>2</sup> − 0*.*11 for the neuron *h*2. We improve the bounds for other neurons in a similar fashion. Using the new bounds, we obtain *h*<sup>2</sup> −*h*<sup>1</sup> ≥ 0*.*01 *>* 0 which enables us to correctly certify the predicate of interest. If the certification still fails, it is possible to further refine the abstraction by increasing the number of splits and repeating the procedure above.

Compared to [21], which uses a single bound, our method is more flexible and can tune *λ* parameters to find a combination of different bounds for each neuron that yields the most precise certification result for each certification instance. Our method is also faster as it performs expensive gradient-based optimization for only the output layer whereas [21] performs this step for each neuron in the LSTM twice. [5,12,23] also suggest a similar idea of bounding ReLU's lower bound using gradient descent, but their approach is limited to unary functions with trivial candidates, not applicable to our setting which requires handling complex binary operations with non-trivial initial bounds.

**Generality of Our Method.** Our method is generic and can be easily extended to obtain polyhedral bounds for the non-linear operations in other architectures such as transformers [42] and capsule networks [34].

#### **5 Scalable Certification of LSTMs**

Next, we formally describe our scalable verifier for LSTM networks. As mentioned in Sect. 4, we build our verifier based on the DeepPoly abstract domain [39] introduced in Sect. 3.4. For simplicity, we focus on computing the polyhedral bounds for the output of non-linear operations. Note that the computed polyhedral bounds contain only the neurons from the previous layers. This restriction is required for backsubstitution used for computing the interval bounds of the inputs, which is an approximate algorithm for solving an LP (e.g. maximize or minimize *x<sup>j</sup>* ) within a polyhedral region defined by DeepPoly constraints. In Sect. 5.1, we show how to obtain tight, asymptotically optimal polyhedral bounds on key operations in the LSTM unit: *σ*(*x*) tanh(*y*) and *σ*(*x*)*y*. Section 5.2 describes a novel method to dynamically choose between different polyhedral bounds for increasing verifier precision.

#### **5.1 Computing Polyhedral Abstractions of LSTM Operations**

Our goal is to bound the products of *sigmoid and tanh* and *sigmoid and identity*, using lower and upper polyhedral planes parameterized by coefficients *Al*, *Bl*, *C<sup>l</sup>* and *Au*, *Bu*, *Cu*, respectively. Let *f*(*x, y*) = *σ*(*x*) tanh(*y*) and *g*(*x, y*) = *σ*(*x*)*y*. For *h* ∈ {*f,g*} we describe how to obtain the lower and upper bounds of *h*:

$$A\_l \cdot x + B\_l \cdot y + C\_l \le h(x, y) \le A\_u \cdot x + B\_u \cdot y + C\_u$$

We formulate the search for a lower bound of *h*(*x, y*) as an optimization problem that minimizes the volume between the bound and the function, subject to the (soundness) constraint that the lower bound is below the function value:

$$\min\_{A\_l, B\_l, C\_l} \int\_{(x, y) \in B} (h(x, y) - (A\_l \cdot x + B\_l \cdot y + C\_l)) $$

$$\text{subject to } A\_l \cdot x + B\_l \cdot y + C\_l \le h(x, y), \forall (x, y) \in B. \tag{1}$$

We denote *B* = [*lx, ux*] × [*ly, uy*] as the boundaries of input neurons *x* and *y* obtained using backsubstitution. We next describe our method to solve Eq. (1). **Step 1: Approximation via LP.** We solve an approximation of the intractable optimization problem from Eq. (1), obtaining potentially unsound constraints. Unsoundness implies that there can be points in region *B* which violate the bounds. We build on the approach from [4], which proposes to approximate the objective in Eq. (1) using Monte Carlo sampling. Let *D* = {(*x*1*, y*1)*,...,*(*xn, yn*)} be a set of points from *B* sampled uniformly at random. We phrase the following optimization problem:

$$\min\_{A\_l, B\_l, C\_l \in \mathbb{R}} \sum\_{i=1}^n \left( h(x\_i, y\_i) - \left( A\_l \cdot x\_i + B\_l \cdot y\_i + C\_l \right) \right)$$

$$\text{subject to } \bigwedge\_{i=1}^n A\_l \cdot x\_i + B\_l \cdot y\_i + C\_l \le h(x\_i, y\_i). \tag{2}$$

Figure 3 shows an input region with Monte Carlo samples as red circles and summands in the LP objective as vertical lines. As this is a low-dimensional linear program (LP), we can solve it exactly in polynomial time using off-the-shelf LP solvers. We compute a candidate upper bound analogously.

**Step 2: Adjusting the Offset to Guarantee Soundness.** Since we compute the lower bound from a subset of points in *B*, there can be a point in *B* where the value of *h*(*x, y*) is less than our computed lower bound. To ensure soundness, we compute *Δ<sup>l</sup>* = min(*x,y*)∈*<sup>B</sup> h*(*x, y*)−(*Al*·*x*+*Bl*·*y*+*Cl*) and then adjust the lower bound by updating the offset *C<sup>l</sup>* ← *C<sup>l</sup>* + *Δl*, resulting in a sound lower bound plane. While the

**Fig. 3.** Visualization of the *z* = *σ*(*x*) tanh(*y*) curve and the lower bound computed by linear programming. Red crosses represent the sampled points and dashed lines show the difference between the curve and the plane (summands in the optimization). (Color figure online)

method of [4] also performs offset calculation for obtaining sound bounds, they perform certification of image classifiers against geometric perturbations using expensive branch and bound for calculating the offset. In contrast, we exploit the structure of non-linearities used in LSTMs obtaining a closed-form formula for the offset yielding an exact solution. We now provide details of our offset adjustment method for *f*(*x, y*) = *σ*(*x*) tanh(*y*) and *g*(*x, y*) = *σ*(*x*)*y*.

**Offset Calculation for** *f*(*x, y*) = *σ*(*x*) tanh(*y*)**:** Let *A<sup>l</sup>* · *x* + *B<sup>l</sup>* · *y* + *C<sup>l</sup>* be the initial lower bounding plane obtained from LP in region *B*. We define *F*(*x, y*):

$$F(x, y) = \sigma(x)\tanh(y) - \left(A\_l \cdot x + B\_l \cdot y + C\_l\right)$$

To find *Δ<sup>l</sup>* = min(*x,y*)∈*<sup>B</sup> F*(*x, y*), we first find the extreme points by computing partial derivatives.

$$\frac{\partial F}{\partial x} = \sigma(x)\tanh(y)(1-\sigma(x)) - A\_l \tag{3}$$

$$\frac{\partial F}{\partial y} = \sigma(x)(1 - \tanh^2(y)) - B\_l \tag{4}$$

We consider three cases:

– *Case 1: x* ∈ {*lx, ux*} and *y* ∈ [*ly, uy*] Under this condition, we denote *S<sup>x</sup>* := *σ*(*x*) as a constant. To ease notation, let *t* = tanh(*y*) where *t* ∈ [tanh(*ly*)*,*tanh(*uy*)]. Then *∂F ∂y* ! = 0 can be rewritten as:

$$1 - t^2 = B\_l / S\_x \tag{5}$$

– *Case 2: y* ∈ {*ly, uy*} and *x* ∈ [*lx, ux*] Here we set *T<sup>y</sup>* := tanh(*y*) and *s* = *<sup>σ</sup>*(*x*)*, x* <sup>∈</sup> [*σ*(*lx*)*, σ*(*ux*)] analogously. *∂F ∂x* ! = 0 is rewritten to:

$$s(1-s) = A\_l/T\_y \tag{6}$$

– *Case 3: otherwise* Otherwise, we consider both *∂F ∂x* ! = 0 and *∂F ∂y* ! = 0. By combining Eq. (3) and Eq. (4), we reduce tanh(*y*) and obtain:

$$s^4 + (-2 - B\_l)s^3 + (1 + 2B\_l)s^2 + (-B\_l)s - A\_l^2 \stackrel{!}{=} 0\tag{7}$$

Given that *F*(*x, y*) is differentiable and the region *B* is compact, Fermat's theorem (stationary points) [1] states that *F* achieves its extremum at either the roots of Eq. (5), Eq. (6), and Eq. (7), or at the 4 corners of *B*. We evaluate *F* at these points to get *Δl*. We adjust the offset by replacing *C<sup>l</sup>* ← *C<sup>l</sup>* + *Δl*. The adjusted *F* is no less than 0 on any point in *B*, which means that the plane with adjusted *C<sup>l</sup>* becomes a sound lower bound of the *σ*(*x*) tanh(*y*) curve.

**Offset Calculation for** *g*(*x, y*) = *σ*(*x*)*y***:** We next calculate the offset for *σ*(*x*)*y*. We define the differentiable function *G*(*x, y*) = *σ*(*x*)*y* − (*A<sup>l</sup>* · *x* + *B<sup>l</sup>* · *y* + *Cl*) over the compact set *B* and compute:

$$\frac{\partial G}{\partial x} = \sigma(x)y(1 - \sigma(x)) - A\_l \tag{8}$$

$$\frac{\partial G}{\partial y} = \sigma(x) - B\_l \tag{9}$$

We use Fermat's theorem and consider three cases:

– *Case 1: x* ∈ {*lx, ux*} and *y* ∈ [*ly, uy*] When *σ*(*x*) is fixed, Eq. (9) is constant, which means *G* is monotonous in this case.

– *Case 2: y* ∈ {*ly, uy*} and *x* ∈ [*lx, ux*] Denote *s* = *σ*(*x*) where *s* ∈ [*σ*(*lx*)*, σ*(*ux*)], then setting Eq. (8) ! = 0 becomes

$$s(1-s) = A\_l/y\tag{10}$$

– *Case 3: otherwise* If there is a local extremum in the region, the Hessian of *G* must be either positive-definite or negative-definite.

$$\frac{\partial^2 G}{\partial x^2} = \sigma(x)y(1 - \sigma(x))(1 - 2\sigma(x)),\\\frac{\partial^2 G}{\partial y^2} = 0,\\\frac{\partial^2 G}{\partial x \partial y} = \sigma(x)(1 - \sigma(x))$$

$$\frac{\partial^2 G}{\partial x^2} \cdot \frac{\partial^2 G}{\partial y^2} - \left(\frac{\partial^2 G}{\partial x \partial y}\right)^2 = -\left(\sigma(x)(1 - \sigma(x))\right)^2 < 0$$

Hence, there is no local extremum inside the boundaries.

To summarize, we only need to consider the roots of Eq. (10) to calculate the minimum of *G* to obtain *Δ<sup>l</sup>* for *σ*(*x*)*y*. Figure 3 shows the lower bound plane obtained after solving the LP and adjusting the offset. We update the upper bound analogously.

**Asymptotic Optimality.** We can prove that, similarly to [4], as we increase the number of samples *n*, the solution of the LP asymptotically approaches the solution of the original problem from Eq. (1). Rephrasing and simplifying the theorem from [4]:

**Theorem 1.** *Let N be the number of points sampled in the algorithm. Let* (*ωl, bl*) *be our lower constraint (linear constraints and bias, respectively) and let L*(*ω*∗*, b*∗) *be the true minimum of function L. For every δ >* 0 *there exists N<sup>δ</sup> such that* |*L*(*ωl, bl*) − *L*(*ω*∗*, b*∗)| *< δ for every N>Nδ, with high probability. Analogous result holds for upper constraint* (*ωu, bu*) *and function U.*

We denote *L* = (*x,y*) *F*(*x, y*) and (*ωl, bl*) are our *Al, Bl, Cl*. Following the theorem, our sampling method guarantees the asymptotic optimality of our bounds. The theorem can be extended analogously for the upper bound.

#### **5.2 Abstraction Refinement via Optimization**

While our approach based on sampling, linear programming, and Fermat's theorem allows us to obtain (asymptotically) optimal bounds, it still has a fundamental limitation that it produces a *single* bound. Further, this approach is, in a sense, greedy: when considering the *entire* network, it is possible that selecting non-optimal planes for each neuron yields more precise results at the end. Neither the method from [21] nor the method in Sect. 5.1 achieves this. We present the first approach to learn an abstraction refinement that increases the end-to-end precision of certification.

**Step 1: Compute a Set of Candidate Bounds.** We adapt our approach from Sect. 5.1 to compute a set of candidate planes, instead of a single plane. We run the sampling procedure multiple times, each time on a different subregion of the original region *B* = [*lx, ux*] × [*ly, uy*], with the constraints still enforced over the entire region *B*. We define four different triangular subdomains: T<sup>1</sup> and T<sup>2</sup> are triangles resulting from splitting *B* along the main diagonal, while T<sup>3</sup> and T<sup>4</sup> are triangles resulting from splitting *B* along the other diagonal. We additionally define T<sup>0</sup> = *B*. For each *k* ∈ {0*,* 1*,* 2*,* 3*,* 4}, we perform sampling and optimization as in Eq. (2), this time sampling from T*<sup>k</sup>* to obtain candidate lower bounds:

$$\begin{aligned} \min\_{A\_l, B\_l, C\_l \in \mathbb{R}} & \sum\_{i=1}^n \left( \sigma(x\_i) \tanh(y\_i) - (A\_l \cdot x\_i + B\_l \cdot y\_i + C\_l) \right) \\ \text{subject to } & \bigwedge\_{i=1}^n A\_l \cdot x\_i + B\_l \cdot y\_i + C\_l \le \sigma(x\_i) \tanh(y\_i) \text{ where } (x\_i, y\_i) \sim \mathcal{T}\_k \end{aligned}$$

For each neuron *i*, this yields 5 candidate lower bound and upper bound planes, *LB<sup>k</sup> <sup>i</sup>* and *UB<sup>k</sup> <sup>i</sup>* for *k* ∈ {0*,* 1*,* 2*,* 3*,* 4}. These five candidate planes for each of the *N* neurons are shown in Fig. 4.

**Fig. 4.** Learning to combine linear bounds via gradient descent. Here the five candidate planes multiplied by *λ* are depicted either in green or red, or both. Green represents the sampled domain, T*k*, and red is the extension of the obtained green plane out of the domain. With the linear combination of the planes, we compute the bound, calculate the loss, and backpropagate. (Color figure online)

**Step 2: Find the Optimal Combinations of the Bounds.** Next, our goal is to learn a linear combination of the computed candidate bounds which yields the highest end-to-end certification precision for the given input region. To do this, we define the lower and upper bound of neuron *i* as a linear combination of the proposed five bounds:

$$LB\_i = \sum\_{k=0}^{4} \lambda\_i^{LB} \cdot LB\_i^k, \quad \sum\_{k=0}^{4} \lambda\_i^{LB} = 1,\\UB\_i = \sum\_{k=0}^{4} \lambda\_i^{UB} \cdot UB\_i^k, \quad \sum\_{k=0}^{4} \lambda\_i^{UB} = 1.$$

Recall that we formulate robustness certification as proving that for all labels *i* different from the ground truth label *t*: *z<sup>t</sup>* − *z<sup>i</sup> >* 0. The lower bound on *zt*−*z<sup>i</sup>* is computed using backsubstitution [39], as shown in our overview example in Sect. 4. However, this lower bound now depends on the coefficients *λ*, so we define the function *f*(*x, -, i,λ*) which computes the lower bound of the expression *z<sup>t</sup>* − *z<sup>i</sup>* when using *λ* to combine the neuron bounds.

We describe our approach to find the best coefficients *λ* in Algorithm 1. Consider the number of possible labels *m* and the number of binary operations of interest *Nops*. To find *λ*, we solve the optimization problem for each label *i*:

$$|z\_t - z\_i| > \max\_{\lambda} f(x, \epsilon, i, \lambda).$$

If the solution to the above optimization problem is positive, then we proved robustness with respect to class *i*. In the algorithm, we initialize *λ*˜, a prenormalized vector of *λ*, for each neuron, uniformly between –1 and 1. Then, in each epoch, we compute the normalized *λ* by applying softmax to *λ*˜ and run certification using *λ*, obtaining a loss L equal to the value −*f*(*x, -, i,λ*). We perform gradient descent update on *λ*˜ based on the loss. If the loss is negative, we have found *λ* which proves the robustness and the algorithm terminates. The core updating flow is shown in Fig. 4.


**Given** input *x*, label *y*, model M, perturbation Initialize the polyhedral abstractions and candidate bounds based on *x*, M and . **for** *i* ← 1 **to** *m* **where** *i* = *y* **do** Initialize *<sup>λ</sup>*˜ <sup>∼</sup> [−1*,* 1]*<sup>N</sup>ops*×<sup>5</sup>, *epoch* <sup>←</sup> <sup>0</sup> **repeat** *<sup>λ</sup>* <sup>←</sup> SoftMax(*λ*˜), L←−*f*(*x, , i, <sup>λ</sup>*), *<sup>λ</sup>*˜ <sup>←</sup> *<sup>λ</sup>*˜ <sup>−</sup> *<sup>α</sup>*∇*λ*˜L, *epoch* <sup>←</sup> *epoch* + 1 **until** *epoch* = *max*\_*epoch* or L *<* 0 **if** L ≥ 0 **then return** not certified **end if end for return** certified

#### **6 Certification of Speech Preprocessing**

Speech preprocessing transforms the original set of perturbed speech signals, represented via intervals, through complex pipeline operations, into a non-linear and non-convex set. Propagating this set through the network is computationally expensive (infeasible for large models). To address this issue, we define precise overapproximations of key non-linear operations found in the speech preprocessing pipeline, such as Square and Log, expressed in the DeepPoly abstraction. These approximate bounds are computed via constant time closed form formulas based on concrete bounds of the inputs. We note that the first and third stages of the pipeline described in Sect. 3.3 involve an affine transformation, captured exactly using DeepPoly. Overall, when combined with our LSTM verifier, this method yields more precise end-to-end certification results than using intervals for approximating speech preprocessing.

**Square.** The lower and upper polyhedral bounds of the output of the square function *<sup>y</sup>* <sup>=</sup> *<sup>x</sup>*<sup>2</sup> where *<sup>x</sup>* <sup>∈</sup> [*lx, ux*] are shown in Fig. 5a. We first consider the bounds for *y* which minimize the area in the *xy*-plane. The upper bound *UB<sup>y</sup>* is obtained by computing the chord joining the end points (*lx, l*<sup>2</sup> *x*) and (*ux, u*<sup>2</sup> *<sup>x</sup>*). The lower bound is a line parallel to *UB<sup>y</sup>* passing through a point ((*u<sup>x</sup>* +*lx*)*/*2*,*((*u<sup>x</sup>* + *lx*)*/*2)<sup>2</sup>) in the middle of the curve.

$$LB\_y = \left(u\_x + l\_x\right) \cdot x - \left(\left(u\_x + l\_x\right)/2\right)^2,\\UB\_y = \left(u\_x + l\_x\right) \cdot x - u\_x \cdot l\_x.$$

While the above bounds would be sufficient in any other domain, they do not work for the speech domain as the subsequent Log requires that the input is strictly non-negative, as it is not defined for negative inputs. Also, we should carefully consider the floating point errors during calculations. Hence, we introduce the additional parameter *<sup>δ</sup>* <sup>∈</sup> <sup>R</sup>, a small threshold value to ensure the lower bound stays non-negative. In our experiments, we set *<sup>δ</sup>* = 1 <sup>×</sup> <sup>10</sup>−<sup>5</sup>. Upper and lower bounds for *<sup>y</sup>* <sup>=</sup> *<sup>x</sup>*<sup>2</sup> are computed as *UB<sup>y</sup>* = (*u<sup>x</sup>* <sup>+</sup>*lx*)·*x*−*u<sup>x</sup>* ·*l<sup>x</sup>* and *LB<sup>y</sup>* <sup>=</sup>

**Fig. 5.** Two polyhedral abstractions for the speech preprocessing stage.

$$\begin{cases} 2\cdot(l\_x + \sqrt{l\_x^2 - \delta})\cdot x - (l\_x + \sqrt{l\_x^2 - \delta})^2 & 3\cdot l\_x^2 + 2\cdot l\_x, u\_x - u\_x^2 \le 4\cdot \delta, \sqrt{\delta} \le l\_x \\ 2\cdot(u\_x - \sqrt{u\_x^2 - \delta})\cdot x - (u\_x - \sqrt{u\_x^2 - \delta})^2 & 3\cdot u\_x^2 + 2\cdot u\_x, l\_x - l\_x^2 \le 4\cdot \delta, u\_x \le -\delta \\ 0 & l\_x \le \sqrt{\delta}, -\sqrt{\delta} \le u\_x \\ (u\_x + l\_x)\cdot x - ((u\_x + l\_x)/2)^2 & o.w. \end{cases}$$

**Log.** We define the polyhedral abstraction of the output *y* = log(*x*) of the log operation where *x* ∈ [*lx, ux*], as shown in Fig. 5b. Our abstractions are optimal and minimize the area in the *xy*-plane. The lower bound *LB<sup>y</sup>* is the chord joining the end points (*lx,* log(*lx*)) and (*ux,* log(*ux*)). The upper bound *UB<sup>y</sup>* is obtained by computing a line parallel to *LB<sup>y</sup>* passing through the middle of the curve at ((*u<sup>x</sup>* + *lx*)*/*2*,* log((*u<sup>x</sup>* + *lx*)*/*2)). Our final abstractions are:

$$LB\_y = \log(l\_x) + \frac{x - l\_x}{u\_x - l\_x} \log(\frac{u\_x}{l\_x}),\\UB\_y = \frac{2 \cdot x}{u\_x + l\_x} - 1 + \log(\frac{u\_x + l\_x}{2}).$$

#### **7 Experimental Evaluation**

We implemented our approach in a verifier called Prover, using PyTorch [30] and Gurobi 9.0 for solving linear programs. The code is available in https:// github.com/eth-sri/prover. We evaluate Prover on speech classifiers for FSDD [17] and GSC v2 [44] datasets. Then, we compare Prover against POPQORN [21] on the MNIST image classification task proposed by it. We note that POPQORN does not scale to the speech classifiers considered in our work. We demonstrate further scalability by verifying large motion sensor sequence classifier trained on HAPT [33] dataset containing 256 hidden dimensional 4 layered LSTM units.

**Setup.** GSC dataset experiments ran on an Nvidia GeForce RTX 2080, while the rest ran on a single Tesla V100. Following convention from prior work [39], we consider only those inputs that are classified correctly without perturbation. We use the same set of hyperparameters for the experiments unless specifically mentioned. We use 100 sampling points for constructing the linear program and optimize *λ* parameters using Adam [20] for 100 epochs. During optimization, we initialize the learning rate to 100 and multiply it by 0*.*98 after every epoch.

#### **7.1 Speech Classification**

We certify the robustness of two speech classifiers for the FSDD and GSC v2 datasets. FSDD consists of recordings of digits spoken by six different speakers, recorded at 8 kHz. GSC has 35 distinct labels of single word utterances at 16 kHz. We compare our base method based on sampling and linear programming (Sect. 5.1), denoted as Prover (LP), and our method using abstraction refinement via optimization (Sect. 5.2), denoted as Prover (OPT).

**Preprocessing.** A key challenge in speech classification, not encountered in the image domain, is the complex preprocessing stage before the LSTM network. The preprocessing stage in this experiment consists of FFT and Mel-filter transformations. Preprocessed input then passes through the fully connected layer with ReLU activation followed by the LSTM unit.

**FSDD Certification.** We used the following parameters for the preprocessing: we slice the raw wave signal with length 256 using a stride of 200 with 10 Melfrequencies. For this experiment, we trained an LSTM network with two LSTM layers and 32 hidden units each, preceded by a 40 ReLU-activated fully-connected layer. This network achieves an accuracy of 83.6% on the FSDD task. The average number of frames was 14.7. We verify the first 100 correctly classified inputs for each perturbation. Our perturbation metric on speech classification tasks is described in Sect. 3.1. Our results are shown in Fig. 6a and Fig. 6b. We vary the decibel perturbation between –90 dB and –70 dB and evaluate the precision and runtime of Prover. Figure 6a shows the percentage of certified samples: our method based on optimizing the bounds (OPT) performs best, e.g., certifying twice as many samples compared to LP, for a significant perturbation of –70

**Fig. 6.** Performance plots for the FSDD and GSC datasets with different perturbations. All tests are done with the same architecture described in the text.

dB. In terms of runtime, Fig. 6b shows that the OPT runtime increases with the perturbation magnitude, meaning that the optimizer needs more iterations to converge to the resulting bounds.

**Interval vs. Polyhedral Abstraction for Speech Preprocessing.** We studied experimentally the importance of designing precise polyhedral abstractions of the speech preprocessing pipeline. If we replace the polyhedral bounds for the square and logarithm operations with interval constraints, the precision of Prover (LP) drops from 86% to 61% on –90 dB and from 70% to 20% on –80 dB. This shows the importance of keeping relational information while overapproximating the speech preprocessing pipeline.

**GSC Certification.** We used the following parameters for the preprocessing: we downsample the raw input to 8 kHz, sliced the signal in length 1024, followed by 10 Mel-frequency filterbanks. As with the FSDD architecture, we used two layers of LSTM and 50 hidden units each, preceded by a 50 ReLU-activated fully-connected layer. This network achieves accuracy of 80% on the GSC task.

Certifying the GSC classifier is more challenging than FSDD: this dataset has 35 labels, compared to 10 in FSDD. The larger label set size requires Prover to compare 34 output differences - acquiring the lower bounds of *lGT* − *lF L* where each term stands for the final output score for the ground truth and false label, respectively. Figure 6c shows the percentage of certified samples: 75% on –110 dB and 46% on –100 dB with Prover (OPT), again higher precision than Prover (LP). Figure 6d shows the longer running time for Prover (OPT) than on FSDD, due to its larger label set size.

#### **7.2 Image Classification**

Based on the setup from [21], we flatten each image into a vector of dimension 784. This vector is partitioned into a sequence of *f* frames (*f* depends on the experiment). Next, the LSTM uses this frame as an input.

**Comparison with POPQORN.** We compare the precision and scalability of Prover against POPQORN [21]. We trained an LSTM network containing 1 layer with 32 hidden units using standard training, achieving an accuracy of 96.5%. The network receives a sequence of *f* = 7 image slices as input and predicts a digit corresponding to the image.

As POPQORN is slow, we used only ten correctly classified images randomly sampled from the test set. For each frame index *i* and each method, we compute the maximum perturbation bound  such that the method can certify that the LSTM classifier is robust to perturbations up to  in the *L*∞-norm of the *i*-th slice of the image. We determine the maximum  using the same binary search procedure as in [21].



Figure 7 presents the results of this experiment. We observe that, for all three methods, early frames have smaller  certified perturbation bounds than the later frames. The reason is that the approximation error on frame 1 propagates through the later frames to the classifying layer, while the error on frame 7 only affects the last layer. Across all frames, both our LP and OPT methods significantly outperform POPQORN, meaning that Prover enables a more precise abstraction than POPQORN. As for speech classifiers, OPT is more precise than LP. We compare running times of the three methods on perturbations in the first frame – most challenging as it requires propagating through all timesteps. Here, Prover (LP), Prover (OPT), and POPQORN

**Fig. 7.** Results for the comparison between Prover and POPQORN. Plotted points represent the maximum *L*<sup>∞</sup> norm perturbation for each frame index 1 through 7.

take 65,348, and 2,160 s respectively per example on average. We conclude that both variants of Prover are more precise than POPQORN while being 33.2× and 6.21× more scalable for LP and OPT respectively.

**Effect of Model Size.** We evaluate the scalability of Prover by certifying several recurrent architectures, with varying number of frames *F*, hidden units *H* and LSTM layers *L*. For each network, we certify the first 100 correctly classified images using the same perturbation *-* = 0*.*01 for all frames, with 3 repetitions. While in the previous experiment we certified each frame separately to closely follow the setup from [21], it is more natural to assume the adversary is able to perturb the entire input. The results are shown in Table 1. We observe that the precision of Prover is affected mostly by the number of frames, as the precision loss accumulates along the frames. Naturally, the running time increases with the number of neurons and frames, as Prover is optimizing the bounds for each *σ*(*x*) tanh(*y*) operation. However, we also observe a counter-intuitive phenomenon that Prover (OPT) performs better with multi-layer models than with the single-layer model. The precision from Prover (LP) drops with the number of LSTM layers unlike those from Prover (OPT). We hypothesize that an increased number of trainable parameters enhances the flexibility of the bounds for the optimization, allowing us to find more combinations of the bounds that certify the input. Prover (LP) has non-flexible bounds, so the error propagates.

**Effect of Perturbation Budget.** We certify the robustness of the MNIST classifier for different  values. We again evaluated 100 correctly classified samples from the test set. Figure 8 shows the experimental results. The OPT version has

**Fig. 8.** Results on MNIST with different epsilons and *F* = 4*, H* = 32*, L* = 2.

significantly higher precision than LP: i.e., for *-* = 0*.*013 in Fig. 8a, LP proves 39% while OPT certifies 89% of samples with a higher runtime in Fig. 8b.

#### **7.3 Motion Sensor Data Classification**

We further demonstrate the scalability of Prover by considering a large classifier containing 4 LSTM layers with 256 hidden units each for the human activity recognition dataset HAPT [33]. Each input in the dataset consists of recorded triaxial linear accelerations and angular velocities, sampled 50 Hz. Here, we restricted HAPT to six activity classes and we trimmed angular velocities to at most 6 s after the point of prediction. Each input sequence is sliced into sliding windows of 0.5 s, which are then passed as an input to the classifier. The trained classifier achieves 88% test accuracy. Identical to the other experiments, we run Prover on the first 100 correctly classified inputs.

**Fig. 9.** Results on HAPT with different epsilons and *H* = 256*, L* = 4.

Results, shown in Fig. 9, indicate that Prover (OPT) verifies more inputs than Prover (LP), for all perturbation budgets. Although the number of parameters has increased, Fig. 9b shows smaller running times compared to Fig. 6b and Fig. 6d. This is because of the smaller number of classes in HAPT, as the verification needs to perform the backsubstitution for each incorrect class. This result shows that Prover (i) is applicable to LSTM classifiers in various domains, and (ii) scales to the large models.

### **8 Conclusion**

We introduced a novel approach for certifying RNNs based on a combination of linear programming and abstraction refinement. The key idea is to compute a polyhedral abstraction of the non-linear operations found in the recurrent cells and to dynamically adjust this abstraction according to each input example being certified. Our experimental results show that Prover is more precise and scalable than prior work. These advances enable Prover to certify, for the first time, the robustness of LSTM-based speech classifiers.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Verisig 2.0: Verification of Neural Network Controllers Using Taylor Model Preconditioning**

Radoslav Ivanov(B), Taylor Carpenter, James Weimer, Rajeev Alur, George Pappas, and Insup Lee

> University of Pennsylvania, Philadelphia, PA 19104, USA {rivanov,carptj,weimerj,alur, pappasg,lee}@seas.upenn.edu

**Abstract.** This paper presents Verisig 2.0, a verification tool for closedloop systems with neural network (NN) controllers. We focus on NNs with tanh/sigmoid activations and develop a Taylor-model-based reachability algorithm through Taylor model preconditioning and shrink wrapping. Furthermore, we provide a parallelized implementation that allows Verisig 2.0 to efficiently handle larger NNs than existing tools can. We provide an extensive evaluation over 10 benchmarks and compare Verisig 2.0 against three state-of-the-art verification tools. We show that Verisig 2.0 is both more accurate and faster, achieving speed-ups of up to 21x and 268x against different tools, respectively.

#### **1 Introduction**

Following their increasing popularity, neural networks (NNs) have been recently introduced to various new domains, including safety-critical systems such as autonomous cars [4] and airborne collision avoidance systems [21]. At the same time, NNs have been shown to be greatly susceptible to input perturbations: minor input changes can cause a NN's outputs to vary drastically, as is the case with adversarial examples [26]. Such issues have emphasized the need to formally analyze NN-based systems and assure their safety before they are deployed.

A number of formal verification approaches have been proposed in the last few years to analyze closed-loop systems with NN components. On the one hand, several techniques have been developed for reachability analysis. These works

This work was supported by the Air Force Research Laboratory (AFRL) and the Defense Advanced Research Projects Agency (DARPA) under Contract No. FA8750- 18-C-0090, and by the Army Research Office (ARO) under Grant Number W911NF-20-1-0080, and by the Office of Naval Research (ONR) award N00014-20-1-2115. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of AFRL, ARO, DARPA, ONR, or the Department of Defense, or the United States Government.

**Fig. 1.** Overview of the closed-loop system considered in this paper.

handle the NN reachability problem in a variety of ways: by converting the NN into a hybrid system [19]; by casting the problem into a satisfiability modulo convexity problem [25]; by approximating the NN with a Taylor model [8,11,16,20]; or by propagating NN reachable sets using star sets [27,28]. Multiple falsification techniques have been developed as well: these approaches work by adapting existing hybrid-system falsifiers [2,6] to the NN case [7,29,33]; methods for systematic testing through scenario specification languages have been proposed as well [14]. Finally, a number of techniques have been developed to analyze properties of the NN in isolation (e.g., input-output properties) [9,10,12,15,22,30–32], though it is challenging to use these tools in a closed-loop setting as it is unclear what NN specification ensures closed-loop safety in general.

While existing reachability techniques have shown impressive performance, scalability remains an obstacle to applying these tools to realistic systems. In particular, these methods have been evaluated mostly on low-dimensional systems, i.e., systems with several states and at most 41 measurements [18]. The main scalability challenge stems from the fact that reachability is undecidable even for linear hybrid systems [1]. Thus, all approaches overapproximate the true reachable sets using a computationally convenient representation such as polytopes [13] or Taylor models [5]. At the same time, this overapproximation, known as the wrapping effect, leads to quick error accumulation over time, thus making it challenging to verify complex specifications over a longer time horizon.

To address these limitations, we present Verisig 2.0, a scalable tool for verifying safety properties of closed-loop systems with NN controllers. We combine ideas from NN reachability with ideas from hybrid system verification. In particular, we adopt the idea of approximating NNs with Taylor models (TMs) [11,16,20], and we alleviate the wrapping effect through TM preconditioning and shrink wrapping [3,23,24]. Finally, we note that the NN reachability computation can be parallelized since each neuron in a layer can be analyzed independently. We have implemented our tool in conjunction with the hybrid system tool Flow\* [5], which enables us to handle general hybrid system models with NN components.

We compare Verisig 2.0 against three tools, namely Verisig [20], NNV [28], and ReachNN\* [11]. We use 10 benchmarks that illustrate various challenges, such as hybrid models, non-linear systems and systems with high-dimensional observations. The results indicate that Verisig 2.0 is significantly faster (achieving speed-ups of up to 21x and 268x against Verisig and ReachNN\*, respectively) and produces tighter reachable set approximations on all benchmarks.

In summary, this paper has three contributions: 1) a Taylor-modelbased NN reachability method using TM preconditioning and shrink wrapping; 2) an efficient implementation that allows for parallel execution; 3) an extensive comparison against existing tools on 10 diverse benchmarks. The source code to reproduce the results is available online (github.com/ rivapp/CAV21 repeatability package) as well as in the main Verisig repository (github.com/Verisig/verisig).

#### **2 Problem Statement**

This section outlines the reachability problem addressed in this paper. We consider a closed-loop system, illustrated in Fig. 1, consisting of: a) a plant with states x modeled as a hybrid system; b) measurements y produced as a function of x; c) an NN controller h that takes y as input and produces controls u.

*Plant Model.* We assume the plant is modeled as a standard hybrid system. In particular, the state space <sup>X</sup> <sup>=</sup> <sup>X</sup><sup>D</sup> <sup>×</sup> <sup>X</sup><sup>C</sup> consists of continuous variables <sup>X</sup><sup>C</sup> and discrete locations <sup>X</sup><sup>D</sup> <sup>=</sup> {q1,...,q<sup>m</sup>}. When in location <sup>q</sup> <sup>∈</sup> <sup>X</sup><sup>D</sup>, the system evolves according to differential equations <sup>f</sup><sup>q</sup>, i.e., ˙<sup>x</sup> <sup>=</sup> <sup>f</sup><sup>q</sup>(x, u), where <sup>x</sup> <sup>∈</sup> <sup>X</sup><sup>C</sup> . Each location <sup>q</sup> <sup>∈</sup> <sup>X</sup><sup>D</sup> has an associated invariant <sup>I</sup>(q) <sup>⊆</sup> <sup>X</sup><sup>C</sup> that must hold true in that location. Transitions between locations are enabled by guards, which are boolean predicates on the continuous variables. Finally, each continuous variable may be reset to a new value when transitioning to a new location.

*Observation Model.* The system produces observations y <sup>=</sup> g(x), where g : X <sup>→</sup> <sup>R</sup><sup>p</sup>. Note that some benchmarks in this paper use state feedback only, i.e., y <sup>=</sup> x.

*Controller.* The controller h is a fully-connected feedforward NN with sigmoid/tanh activations. Formally, h can be represented as a composition of its L layers:

$$h(y) = h\_L \diamond h\_{L-1} \diamond \cdots \diamond h\_1(y),\tag{1}$$

where each <sup>h</sup><sup>i</sup>(y) = <sup>a</sup>(W<sup>i</sup><sup>y</sup> <sup>+</sup> <sup>b</sup><sup>i</sup>) performs a linear function, with parameters <sup>W</sup><sup>i</sup> and <sup>b</sup><sup>i</sup> identified during training, followed by a sigmoid/tanh activation <sup>a</sup>.

*Composed System.* Although the hybrid system formulation places no restrictions on the controller/plant composition, in the interest of clarity we assume the controller is executed in a time-triggered fashion, with sampling period T, as follows: <sup>u</sup>(t) = <sup>h</sup>(y(t<sup>k</sup>)), for <sup>t</sup> <sup>∈</sup> [t<sup>k</sup>, t<sup>k</sup> <sup>+</sup> <sup>T</sup>), where <sup>t</sup><sup>k</sup> <sup>=</sup> kT and <sup>k</sup> = 0, <sup>1</sup>, <sup>2</sup>,...

*Closed-Loop Reachability Problem.* Let S be a composed system. Given an initial set of states x(0) <sup>∈</sup> X<sup>0</sup>, the reachability problem, expressed as property <sup>φ</sup>, is to verify a property ψ of the reachable states of S:

$$
\phi(X\_0) \equiv (x(0) \in X\_0) \Rightarrow \psi(x(t)), \ \forall t \ge 0. \tag{2}
$$

#### **3 Background: Neural Networks as Taylor Models**

As described in Sect. 1, in this work we adopt a TM-based approach for propagating NN reachable sets. There are two main reasons for this: 1) TMs can approximate any differentiable function over a bounded range given a high enough order; 2) TMs are very effective at approximating hybrid system reachable sets, which allows for a smooth composition between the NN and the rest of the system. The rest of this section formalizes TMs and summarizes the existing approaches to using TMs for NN reachability.

*Taylor Model Definition.* Intuitively, a TM of a function f is a polynomial approximation p, together with a worst-case error bound I. A j-degree polynomial approximation p of a j times continuously differentiable function f around a point <sup>x</sup>, written <sup>p</sup>(x) <sup>≡</sup><sup>j</sup> <sup>f</sup>(x), is a polynomial <sup>p</sup> of degree <sup>j</sup> such that all partial derivatives of f and p coincide at x. Let <sup>I</sup> be the set of all intervals I = [a, b] and let f : D <sup>→</sup> <sup>R</sup> be a function of n variables defined over a domain D <sup>∈</sup> <sup>I</sup><sup>n</sup>. Then a Taylor model of f over D of degree j is a pair (p, I) of a polynomial approximation p and an error bound I (also known as a remainder) such that:

$$\begin{aligned} (1) & f(c) \equiv\_j p(c) \text{, where } c \text{ is the center of } D, \\ (2) & \forall x \in D, \ f(x) \in \{p(x) + e \mid e \in I\}. \end{aligned}$$

*Taylor Model Arithmetic.* Let TM<sup>1</sup> = (p1, I<sup>1</sup>) and TM<sup>2</sup> = (p2, I<sup>2</sup>) be two TMs defined over a domain D. Addition and multiplication are defined as follows [5]:

$$\begin{aligned} TM\_1 + TM\_2 &= (p\_1 + p\_2, I\_1 + I\_2) \\ TM\_1 \times TM\_2 &= (p\_1 \times p\_2, \mathbf{Int}(p\_1)I\_2 + \mathbf{Int}(p\_2)I\_1 + I\_1 \times I\_2), \end{aligned}$$

where Int(p) is an interval bound of p over D.

TMs have shown impressive performance in hybrid system reachability problems due to their ability to approximate any continuously differentiable function given a high enough order [5]. Another appealing feature is that TMs can be used to approximate solutions of differential equations through Picard iteration [5]. Thus, it is natural to try to use TMs to approximate NN reachable sets as well.

Two classes of approaches for approximating NNs with TMs have been developed in the literature. The first one is sampling-based: given a TM TM<sup>y</sup> of the inputs <sup>y</sup> to <sup>h</sup>, these methods sample points <sup>Z</sup> from TM<sup>y</sup> and corresponding outputs h(Z) to perform polynomial regression [8] or approximation [16]. While these approaches work well for systems with several state variables, they cannot handle higher-dimensional problems due to insufficient sampling.

A second approach to using TMs for NN reachability is to propagate the TMs through each neuron in the NN. Specifically, let TM<sup>y</sup> = (p, I) be the TM for y and consider a neuron ν that computes the function σ(wy <sup>+</sup> b), where σ denotes the sigmoid. One can use TM arithmetic [5] to obtain TM<sup>L</sup> = (wp <sup>+</sup> b, wI) for the linear map in ν. For the sigmoid TM, TM<sup>σ</sup>, one could obtain a Taylor series expansion of <sup>σ</sup> around the center of TM<sup>L</sup> and get remainder bounds using Taylor's theorem [20]. Thus, the final TM for <sup>ν</sup> is TM<sup>ν</sup> <sup>=</sup> TM<sup>σ</sup> ◦ TML. The benefit of propagating TMs in this fashion is that no sampling is necessary since the NN is approximated directly. On the other hand, scalability challenges manifest in a different way, namely the TM remainders may grow quickly depending on the NN architecture (explained in more detail in Sect. 4).

We adopt the latter approach to approximating NN as TMs due to its improved scalability. The next section describes our approach to reducing the TM remainder size through TM preconditioning and shrink wrapping.

**Fig. 2.** The wrapping effect for different taylor model orientations.

**Fig. 3.** Illustration of the shrink wrapping method.

#### **4 Taylor Model Preconditioning and Shrink Wrapping**

This section presents our approach to limiting the remainder growth as TMs are propagated through the NN. We explore two complementary techniques, namely TM preconditioning and shrink wrapping. Both of these ideas were originally developed for the purpose of reachability analysis of hybrid systems [3,23] – in this paper, we adapt them to the NN case.

#### **4.1 Taylor Model Preconditioning**

As noted in Sect. 3, although propagating the TM through the NN is preferred since it captures the functional representation of each neuron, it may suffer from quick remainder growth. The following example illustrates this process.

*Example 1.* Let <sup>y</sup><sup>1</sup> and <sup>y</sup><sup>2</sup> be inputs to the NN <sup>h</sup> with corresponding TMs TM<sup>y</sup><sup>1</sup> = (p<sup>1</sup>, I<sup>1</sup>) and TM<sup>y</sup><sup>2</sup> = (p<sup>2</sup>, I<sup>2</sup>) over domain <sup>D</sup> <sup>∈</sup> <sup>I</sup><sup>n</sup>. Let <sup>ν</sup> be a neuron in the first layer implementing the function <sup>ν</sup>(y<sup>1</sup>, y<sup>2</sup>) = <sup>σ</sup>(w<sup>1</sup>y<sup>1</sup> <sup>+</sup> <sup>w</sup><sup>2</sup>y<sup>2</sup> <sup>+</sup> <sup>b</sup>). The TM for the linear part of ν is

$$TML := (p\_L, I\_L) = (w\_1 p\_1 + w\_2 p\_2 + b, w\_1 I\_1 + w\_2 I\_2).$$

Let TM<sup>σ</sup> <sup>=</sup> <sup>σ</sup>(a) +σ (a)(TM<sup>L</sup> <sup>−</sup>a) +σ(a)(TM<sup>L</sup> <sup>−</sup>a)<sup>2</sup>/2 +I<sup>σ</sup> be a second-order Taylor series expansion of the sigmoid around point a, with remainder I<sup>σ</sup>. Using TM arithmetic [5], the TM for <sup>ν</sup> is TM<sup>ν</sup> = (p<sup>ν</sup>, I<sup>ν</sup>), where

$$\begin{aligned} p\_\nu &= \sigma''(a)p\_L^2 + (\sigma'(a) - a\sigma''(a))p\_L - (\sigma'(a) - 0.5a\sigma''(a))a + \sigma(a) \\ I\_\nu &= \sigma''(a)(2\mathtt{Int}(p\_L)I\_L + I\_L^2) + (\sigma'(a) - a\sigma''(a))I\_L + I\_\sigma. \end{aligned}$$

*Remark 1.* In order to compute a TM<sup>σ</sup> = (pσ, Iσ) for the sigmoid/tanh, one can follow the procedure described in prior work [20]. In summary, the following three steps need to be performed, assuming the input TM is denoted by TML:


As shown in Example 1, the remainder is propagated using interval analysis, where a major contributor is the Int(p<sup>L</sup>) term, i.e., the interval bounds of p<sup>L</sup>. Since this term approximates the (potentially high-dimensional) input TM with a box, it may introduce significant wrapping effect if the input TM is not a box, as illustrated in Fig. 2. The natural way to address this wrapping effect is through rotating the TM in order to align it with the axes [23,24].

#### **Algorithm 1.** NN Verification Using Taylor Model Preconditioning

**Input:** Measurement TM Vector *TMV*y, NN *h* with *L* layers, and sigmoid activations. 1: *TMV*<sup>0</sup> ← *TMV*<sup>y</sup> 2: **for each** *i* in {1*,...,L*} **do** 3: *TMV* <sup>L</sup> <sup>i</sup> ← *W*<sup>i</sup> ∗ *TMV*<sup>i</sup>−<sup>1</sup> + *b*<sup>i</sup> 4: (*Q* + *c,* **0**) ◦ (*R* + *Q*-*N,Q*-**<sup>I</sup>**) <sup>←</sup> *T aylorModelP reconditioning*(*TMV* <sup>L</sup> <sup>i</sup> ) 5: *TMV* <sup>ν</sup> <sup>i</sup> ← *T aylorModelF orSigmoid*((*Q,* **0**)) //Taylor series approximation 6: *TMV*<sup>i</sup> <sup>←</sup> *TMV* <sup>ν</sup> <sup>i</sup> ◦ (*R* + *Q*-(*c* + *N*)*,* **I**) 7: **end for** 8: **return** *TMV*<sup>L</sup>

Since the set represented by a TM is the image of a polynomial over a given domain, it is challenging to choose an appropriate rotation matrix. However, as discussed in prior work [23,24], if one first normalizes the TM so that the domain is [−1, 1]<sup>n</sup>, then the linear terms become the largest contributors to interval analysis overapproximation (since higher order terms are less than 1 in magnitude). Thus, a good choice for a rotation matrix is the matrix formed by the linear terms of the (normalized) TM.

To formalize the above concept, let us decompose a TM vector TMV = (**p**, **<sup>I</sup>**) into TMV = (c <sup>+</sup> M <sup>+</sup> N, **<sup>I</sup>**), where c denotes the constant terms, M denotes the linear terms and N denotes the higher-order terms. The idea of preconditioning is to decompose M <sup>=</sup> QR, where Q is an orthonormal matrix and R is upper-triangular. This is achieved by splitting TMV into a composition of two TM vectors: TMV = (Q+c, **<sup>0</sup>**)◦(R+QN,Q**I**).<sup>1</sup> Then, each neuron's computation is performed on Q only, which alleviates the wrapping effect introduced by Int(p<sup>L</sup>) in Example <sup>1</sup> since <sup>Q</sup> is orthonormal.

<sup>1</sup> Note that the new remainder may need to be enlarged to also include numerical errors due to the computation of *Q*.

The algorithm is presented in Algorithm 1. Note that preconditioning is performed in each layer, followed by again composing the two parts into the full TM. While it is possible to represent the final TM as a composition of individual layer TMs, the benefits of preconditioning would decrease after the first layer, since most of the variability is captured in the right-most TM.

#### **4.2 Shrink Wrapping**

In systems where verification over a longer time horizon is required, avoiding large remainders may be impossible even with effective preconditioning. In such cases, one could use shrink wrapping in order to refactor the TM into one that results in slower remainder accumulation in the future [3,24].

The high-level idea of shrink wrapping is illustrated in Fig. 3. If the remainder becomes a significant part of the set described by the TM, then TM arithmetic degrades into standard interval analysis. In this case, it helps to transform the TM into a new TM that contains the original one but has no remainder. Thus, even if the new TM is slightly larger, it is propagated symbolically using TM arithmetic, which results in smaller error accumulation in the long run.

The choice of new TM is not obvious and is affected by the system in consideration. The standard approach in related work [3,24] is to focus on the linear terms (assuming the TM is normalized so that D = [−1, 1]<sup>n</sup>). Specifically, suppose that the system's state x is described by the TM vector TMV<sup>x</sup> = (**p**, **<sup>I</sup>**)=(<sup>c</sup> <sup>+</sup> <sup>M</sup> <sup>+</sup> N, **<sup>I</sup>**). One option for the new TM vector is to premultiply TMV<sup>x</sup> by <sup>M</sup><sup>−</sup><sup>1</sup>, thereby reducing the linear terms to the identity matrix, <sup>I</sup>. Then a shrink wrap factor q is chosen such that the image of the higher-order terms contains the remainder of the initial TM vector, i.e., TMV new <sup>x</sup> = (<sup>c</sup> <sup>+</sup> <sup>I</sup> <sup>+</sup> qM<sup>−</sup>1N, **<sup>0</sup>**).<sup>2</sup>

While it is possible to choose q by finding bounds on the partial derivatives of the higher-order terms M<sup>−</sup>1<sup>N</sup> [3], our initial experiments indicated that a more straightforward approach leads to no loss in precision. In particular, we represent the new TM vector as TMV new <sup>x</sup> = (<sup>c</sup> + diag(**q**), **<sup>0</sup>**), where **<sup>q</sup>** <sup>=</sup> Int(TMV<sup>x</sup>). The last consideration is when to perform the TM conversion: if it is applied too often, more error could be introduced by the frequent elimination of useful information in the TMs. In our experiments, shrink wrapping is triggered when the remainder is larger than 1e<sup>−</sup><sup>6</sup> and larger than 1% of the total TM range.

#### **5 Implementation**

We implemented our approach in conjunction with the Flow\* tool [5], for easy integration with standard hybrid system models. We provide similar TM functions to the ones existing in Flow\*, adapted to the case of NNs. In addition to modified data structures, a main difference in our implementation is the option to parallelize the TM vector propagation, i.e., Line 5 in Algorithm 1. This parallelization is possible since each neuron in a layer only depends on the input

<sup>2</sup> The new remainder may be greater than 0 due to round-off error during the inversion.

TMs, thus each computation can be done on a separate core. As illustrated in Sect. 7, this implementation brings great benefits, especially in the case of larger NNs, where multiple neuron computations can be performed in parallel.

#### **6 Benchmarks**

We use 10 benchmarks to evaluate the proposed approach. These benchmarks were compiled from the related literature [17,19,20,28] and were selected in order to cover a wide variety of systems and controllers: 1) continuous and hybrid systems; 2) systems with state feedback and systems with measurements as a function of the states; 3) low-dimensional systems as well as systems with highdimensional measurements; 4) controllers with both tanh and sigmoid activations and with a number of neurons varying from 16 to 200 per layer.

Table 1 presents the dynamics and the initial set for each benchmark. For simplicity, all properties are reachability properties (i.e., the problem is to verify whether a goal set is reached from all initial states), though safety properties can be handled as well. In particular, the goal regions are as follows:

– <sup>B</sup><sup>1</sup> : <sup>x</sup><sup>1</sup> <sup>∈</sup> [0, <sup>0</sup>.2], x<sup>2</sup> <sup>∈</sup> [0.05, <sup>0</sup>.3]; <sup>B</sup><sup>2</sup> : <sup>x</sup><sup>1</sup> <sup>∈</sup> [−0.3, <sup>0</sup>.1], x<sup>2</sup> <sup>∈</sup> [−0.35, <sup>0</sup>.5];

**Table 1.** List of benchmarks. Benchmarks *B*1−*B*<sup>5</sup> and Tora were introduced by Huang et al. [17]; adaptive cruise control (ACC) was presented by Tran et al. [28]; mountain car (MC), quadrotor with model-predictive control (QMPC) and F1/10 were introduced by Ivanov et al. [20]. We use *V* to denote the measurement dimension. In F1/10, *y* is a 21-dimensional LiDAR scan.



#### **7 Experiments**

We compare our tool, named Verisig 2.0, against three state-of-the-art tools, namely Verisig [19,20], ReachNN\* [11,17], and NNV [27,28]. We selected these tools because they handle NNs with sigmoid/tanh activations. For each benchmark, we record whether each tool could verify the property (or return Unknown due to large approximation error). In addition, we compare the verification times between the different tools. While Verisig and NNV do not support parallel execution,<sup>3</sup> ReachNN\* has been optimized for GPU execution, so a comparison in terms of verification times is fair (all experiments were run on an Intel Xeon Gold 6248 running at 2.5 GHz and with an Nvidia GeForce RTX 2080 Ti GPU). Finally, we provide a comparison in terms of reachable sets.

Verification outcomes and times are reported in Table 2. Multiple controllers were used in some benchmarks in order to test a variety of NNs. We present the

**Table 2.** Verification evaluation. The notation *tanh/sig ( n* × *k)* indicates a NN with tanh/sig activations, *n* hidden layers and *k* neurons per layer. For each tool, we provide the verification time in seconds; if a property could not be verified, it is marked as Unknown. If a tool crashed on a benchmark, it is marked as DNF.


<sup>3</sup> NNV is parallelized in the case of ReLU activations, but not for smooth activations.

results for Verisig 2.0 as used with one and with 40 cores, in order to illustrate the benefit of parallelization. Note that parallelization helps the most in systems with wider NNs, e.g., the MC benchmark, since a larger part of the computation is devoted to NN computation (relative to plant computation) in these systems.

*Comparison with Verisig.* Verisig is the closest method to Verisig 2.0, as it also propagates TMs through the NN. Thus, Verisig can be seen as a baseline for our approach, so this comparison illustrates most clearly the benefits of preconditioning and shrink wrapping. Firstly, note that Verisig takes significantly more time to compute reachable sets (21 times slower in the case of the <sup>B</sup><sup>5</sup> benchmark) on all but one benchmark – the MC benchmark is peculiar because the NN is very small, hence most of the computation is spent on the plant. Furthermore, Verisig is unable to verify some properties due to increasing error. As shown in Fig. 4, the reachable sets computed by Verisig introduce more approximation error, especially in the challenging ACC benchmark, where preconditioning is particularly useful due to the larger input space.

(b) *B*<sup>5</sup> benchmark, sigmoid. (c) MC benchmark, 2 *×* 200.

**Fig. 4.** Comparison between the reachable sets produced by Verisig (blue) and Verisig 2.0 (green). Example simulated trajectories are plotted in red. The goal set is shown in magenta. Note that the goal is not reached in the *B*<sup>5</sup> benchmark. (Color figure online)

(a) *B*<sup>1</sup> benchmark, sigmoid. (b) *B*<sup>5</sup> benchmark, tanh. (c) Tora benchmark, sigmoid.

**Fig. 5.** Comparison between the reachable sets produced by ReachNN\* (blue) and Verisig 2.0 (green). Simulated trajectories are plotted in red (not shown in the Tora benchmark to improve visibility). The goal set is shown in magenta. (Color figure online)

**Fig. 6.** Comparison between the reachable sets produced by NNV (blue) and the Verisig 2.0 approach (green) on three of the benchmarks from Table 2. Example simulated trajectories are plotted in red. The goal set is shown in magenta. (Color figure online)

**Fig. 7.** Verisig 2.0 remainder growth (for position, *x*1) on the MC benchmark as we increase the NN size. The remainder is reset to 0 after shrink wrapping.

*Comparison with ReachNN\*.* ReachNN\* is a sampling-based approach to NN verification, so it is expected to work well on low-dimensional systems and encounter difficulties as the dimension increases. As can be seen in Table 2, Verisig 2.0 is faster on all but one benchmark, and the difference is especially pronounced on the four-dimensional Tora benchmark, where ReachNN\* is 268 times slower. Note that ReachNN\* cannot handle hybrid models, so no comparison could be made on those benchmarks. Finally, as shown in Fig. 5, Verisig 2.0 also results in tighter reachable sets – the benefit of shrink wrapping can be observed in Fig. 5a, where the ReachNN\* reachable sets eventually start to grow fast whereas Verisig 2.0 is able to maintain low approximation error over time.

*Comparison with NNV.* Note that NNV is unable to verify any of the properties considered in this paper due to high approximation error. This is mostly due to the fact that NNV is optimized for networks with ReLU activations, where the star set method used in NNV is effective and parallelizable. Figure 6 shows the reachable computed by each tool, where it is clear that Verisig 2.0 maintains tight reachable sets whereas the NNV approximation error grows quickly.

*Scalability Evaluation.* Finally, we also evaluate the scalability of Verisig 2.0 as we increase the NN size on the MC benchmark. Figure 7 illustrates how the remainder grows over time for the <sup>x</sup><sup>1</sup> (position) state. We observe that the larger NN results in significantly larger remainder growth. At the same time, interpreting the remainder growth in isolation may be misleading since it also depends on the size and shape of the true reachable sets. We leave a rigorous analysis of the effect of NN size on scalability for future work.

### **8 Conclusion**

This paper presented Verisig 2.0, a parallelized tool for NN verification. We developed a Taylor-model-based approach in which we reduce the approximation error in reachable sets through Taylor model preconditioning and shrink wrapping. Finally, we provided an extensive evaluation over 10 benchmarks and showed that our method is significantly more accurate and faster than state-of-the-art tools, resulting in 21x and 268x speed-ups on some benchmarks, respectively. For future work, we will investigate which NN architectures are more amenable for verification, both in terms of size and number of layers as well as in terms of weight magnitude and direction.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Robustness Verification of Semantic Segmentation Neural Networks Using Relaxed Reachability**

Hoang-Dung Tran1(B) , Neelanjana Pal<sup>2</sup>, Patrick Musau<sup>2</sup>, Diego Manzanas Lopez<sup>2</sup>, Nathaniel Hamilton<sup>2</sup>, Xiaodong Yang<sup>2</sup>, Stanley Bak<sup>3</sup>, and Taylor T. Johnson<sup>2</sup>

> <sup>1</sup> University of Nebraska-Lincoln, Lincoln, USA <sup>2</sup> Vanderbilt University, Nashville, USA <sup>3</sup> Stony Brook University, Stony Brook, USA

**Abstract.** This paper introduces robustness verification for semantic segmentation neural networks (in short, semantic segmentation networks [SSNs]), building on and extending recent approaches for robustness verification of image classification neural networks. Despite recent progress in developing verification methods for specifications such as local adversarial robustness in deep neural networks (DNNs) in terms of scalability, precision, and applicability to different network architectures, layers, and activation functions, robustness verification of semantic segmentation has not yet been considered. We address this limitation by developing and applying new robustness analysis methods for several segmentation neural network architectures, specifically by addressing reachability analysis of up-sampling layers, such as transposed convolution and dilated convolution. We consider several definitions of robustness for segmentation, such as the percentage of pixels in the output that can be proven robust under different adversarial perturbations, and a robust variant of intersection-over-union (IoU), the typical performance evaluation measure for segmentation tasks. Our approach is based on a new relaxed reachability method, allowing users to select the percentage of a number of linear programming problems (LPs) to solve when constructing the reachable set, through a relaxation factor percentage. The approach is implemented within NNV, then applied and evaluated on segmentation datasets, such as a multi-digit variant of MNIST known as M2NIST. Thorough experiments show that by using transposed convolution for upsampling and average-pooling for down-sampling, combined with minimizing the number of ReLU layers in the SSNs, we can obtain SSNs with not only high accuracy (IoU), but also that are more robust to adversarial attacks and amenable to verification. Additionally, using our new relaxed reachability method, we can significantly reduce the verification time for neural networks whose ReLU layers dominate the total analysis time, even in classification tasks.

#### **1 Introduction**

*Image segmentation* is the process of partitioning an image into multiple portions, or segments, which are sets of pixels, and in short is referred to as segmentation [30]. Segmentation has broad applications, ranging from perception in autonomous cyber-physical systems (e.g., identifying pedestrians, lanes, vehicles, etc. in images) and medical imaging (e.g., identifying tumors, measuring tissue, etc. in X-rays and other medical scans) [31]. *Semantic segmentation* additionally classifies each pixel into a *class* from a set of classes, and hence, can be viewed as a generalization of image classification, the robustness of which has been studied deeply in recent years.

State-of-the-art segmentation approaches typically rely on neural networks, known as *semantic segmentation networks (SSNs)*. Typically SSN architectures take an image as input and are composed of two major portions: a sequence of *down-sampling* layers to extract features from the input image into a latent space, followed by another sequence of *up-sampling* layers, which in essence map the features (roughly corresponding to the classes) from the latent space to the image's pixels, such that each pixel is associated with a class. However, just as neural networks for image classification are well-known to be vulnerable to adversarial perturbations, so too are SSNs [45]. Although deep neural networks (DNN) verification is emerging into an established research area with many tools and techniques proposed to verify safety and robustness specifications of DNNs [22,43] and neural network controlled systems [15,17,34,37], most state-of-art verification techniques for robustness verification of DNNs focus on variants of classification<sup>1</sup>, frequently for images [1,5,7,11,19,24,26,29,32,33,46].

To our knowledge, there are no existing methods that can verify robustness of SSNs, which perform a more complex task than image classification, as the output space dimensionality is (typically) of the same order of size as that of the input space (e.g., the output is an image with the width and height of the input image, but with identified classes in the output instead of color bit depth; see Figs. 5 and 7 for examples). We review some existing testing-based robustness evaluation methods in our related work section.

*Overview and Contributions.* In this paper, we present the first formal approach for verifying SSN robustness using reachability analysis. Our approach's central idea is, if an input image is attacked (perturbed) with some bounded disturbance, we construct a reachable output set that contains all possible classes for each pixel. From the reachable output set, we can formally guarantee an SSN's robustness at the pixel-level, i.e., each pixel is provably classified correctly. Our approach focuses on two effective SSN architectures, including dilated CNNs and transposed CNNs, which to our knowledge, are not supported in any other existing neural network verification approaches. We evaluate our approach on a

<sup>1</sup> The ACAS-Xu benchmarks [18] common in neural network verification can be viewed as a form of classifier: the networks produce advisories (weak left, etc.), which in essence are classes, but do not have images as their inputs.

set of SSNs trained with different architectures on the MNIST [21] and M2NIST data sets, the latter of which is a multi-digit variant of MNIST suitable for segmentation evaluation. Additionally, we define and evaluate several metrics for robustness, as the robustness evaluation is more sophisticated for segmentation.

Our reachability-based approach builds on ImageStars, which are an efficient data structure for verifying convolutional neural networks (CNNs) [33], to construct the input set and compute the reachable set layer-by-layer throughout the SSN. The ImageStar approach offers both exact and approximate reachability schemes for analyzing the robustness of CNNs. Although the approximate scheme obtains a tighter reachable set in comparison with the zonotope [28] and new polytope methods [29] by using optimized ranges, in practice, we do not need a tight reachable set in many cases. Indeed, we only need a "tight enough" reachable set to verify a property. Therefore, it is reasonable to let users have the freedom to choose an appropriate level of relaxation in constructing the reachable set for their applications. More relaxation comes with a coarser reachable set and vice versa. To fulfill this need, we also present a new relaxed ImageStar approach to allow users to choose a specific relaxation level defined by a *relaxation factor* (RF) percentage when constructing the reachable set for their applications. This relaxed reachability method can help reduce the verification time of SSNs significantly (up to 99%) in some cases.

In summary, the main contributions of this paper are: 1) the first formal approach for robustness verification of SSNs, 2) a new relaxed ImageStar reachability method, 3) the implementation of the approach in a prototype software tool, 4) thorough assessment of these methods on different network architectures, and 5) insight on how to train robust SSNs that are amenable to verification.

#### **2 Preliminaries and Problem Formulation**

#### **2.1 ImageStars**

In this section, we review the ImageStar data structure and its properties [33].

**Definition 1.** *An* ImageStar Θ *is a tuple* c, V, P, l, u *where* <sup>c</sup> <sup>∈</sup> <sup>R</sup><sup>h</sup>×w×nc *is the* anchor image*,* <sup>V</sup> <sup>=</sup> {v1, v2, ··· , v<sup>m</sup>} *is a set of* <sup>m</sup> *images in* <sup>R</sup><sup>h</sup>×w×nc *called* generator images*,* <sup>P</sup> : <sup>R</sup><sup>m</sup> → {, ⊥} *is a* predicate*,* <sup>l</sup> *and* <sup>u</sup> *are the* lower bound *and* upper bound *vectors of the* predicate variables*, and* h, w, nc *are the* height*,* width*, and* number of channels *of the images, respectively. The generator images are arranged to form the ImageStar's* h × w × nc × m basis array*. The set of images represented by the ImageStar is given as:*

$$\left[\Theta\right] = \{x \mid x = c + \Sigma\_{i=1}^{m}(\alpha\_i v\_i) \text{ such that } P(\alpha\_1, \dots, \alpha\_m) = \top, l\_i \le \alpha\_i \le u\_i\}.$$

*We may refer to both the tuple* Θ *and the set of states* -Θ *as* Θ*. In this work, we restrict the predicates to be a conjunction of linear constraints,* P(α) - Cα ≤ d *where, for* <sup>p</sup> *linear constraints,* <sup>C</sup> <sup>∈</sup> <sup>R</sup><sup>p</sup>×<sup>m</sup>*,* <sup>α</sup> *is the vector of* <sup>m</sup>*-variables, i.e.,* α = [α1, ··· , αm] <sup>T</sup> *, and* <sup>d</sup> <sup>∈</sup> <sup>R</sup><sup>p</sup>×<sup>1</sup>*. An ImageStar is the empty set if and only if* P(α) *subject to* l ≤ α ≤ u *is empty.*

**Lemma 1 (Affine mapping of an ImageStar).** *An affine mapping of an ImageStar* Θ = c, V, P, l, u *with a scale factor* γ *and an offset image* β *is another ImageStar* Θ = c , V , P , l , u *in which the new anchor, generators and predicate are as follows:*

$$c' = \gamma \cdot c + \beta, \quad V' = \gamma \cdot V, \quad P' \equiv P, \quad l' \equiv l, \quad u'. \equiv u.$$

*Note that, the scale factor* γ *can be a scalar or a vector containing scalar scale factors in which each factor is used to scale one color channel in the ImageStar.*

#### **2.2 Range of a Specific Input in an ImageStar**

We slightly alter the original definition of an ImageStar, [33], by introducing lower bound and upper bound vectors to the predicate variables. Specifically, if we want to find the range of an input x(i, j, k) (where 1 ≤ i ≤ h, 1 ≤ j ≤ w, 1 ≤ k ≤ nc) in an ImageStar Θ, we need to solve the following LP problem.

$$x\_{min} = \min(c(i, j, k) + \Sigma\_{p=1}^m \alpha\_i v\_p(i, j, k)) \text{ s.t. } C\alpha \le d, l \le \alpha \le u,\tag{1}$$

$$x\_{max} = \max(c(i, j, k) + \Sigma\_{p=1}^m \alpha\_i v\_p(i, j, k)) \text{ s.t. } C\alpha \le d, l \le \alpha \le u. \tag{2}$$

However, if we only want to estimate roughly the range of the neuron without solving the LP optimization problem, we can compute the estimated range quickly as follows.

$$x\_{min}^{est} = c(i, j, k) + \Sigma\_{p=1}^{m} l\_p \max(v\_p(i, j, k), 0) + \Sigma\_{q=1}^{m} u\_q \min(v\_q(i, j, k), 0), \tag{3}$$

$$x\_{max}^{est} = c(i, j, k) + \Sigma\_{p=1}^{m} u\_p \max(v\_p(i, j, k), 0) + \Sigma\_{q=1}^{m} l\_q \min(v\_q(i, j, k), 0). \tag{4}$$

#### **2.3 Semantic Segmentation Networks and Reachability**

**Definition 2.** *A* semantic segmentation network (SSN) f *is a nonlinear function that maps each pixel* x(i, j) *of a multichannel input image* x *to a target class* y(i, j) *from a set of classes* L = {1, 2,...,L}*:*

$$\begin{aligned} f: \ x \in \mathbb{R}^{h \times w \times nc} \to y \in \mathcal{L}^{h \times w} \\ x(i, j) \to y(i, j) \in \mathcal{L}, \end{aligned} \tag{5}$$

*where* h, w, nc *are the height, width, and number of channels of the input image, respectively, and* (i, j) ∈ {1,...,h}×{1,...,w} *are the pixel height and width indices, respectively.*

**Definition 3.** *Reachability analysis (or shortly, Reach) of a SSN* f *on an ImageStar input set* I *is the process of computing all possible classes corresponding to every pixel in all input images* x *in the ImageStar input set* I*:*

$$\begin{aligned} \operatorname{Reach}(f, I) &: \ I \to \mathcal{R}\_f \\ &x \to y = f(x) . \end{aligned} \tag{6}$$

*We call* R<sup>f</sup> (I) *the* pixel-class reachable set *of the SSN corresponding to the input set* I *(or just* R<sup>f</sup> *when* I *is clear from context), in which each pixel-class* pc(i, j) ∈ R<sup>f</sup> *at each pixel* (i, j) ∈ {1,...,h}×{1,...,w} *may contain more than one class, i.e.,* pc(i, j) = {l1,...,lm}⊆L*, for* L ≥ m ≥ 1*.*

#### **2.4 Adversarial Attacks and Robustness**

**Definition 4.** *An adversarial attack is where a set of* n *noise images* xnoise = [xnoise <sup>1</sup> *,* ... *,* <sup>x</sup>noise <sup>n</sup> ] *and corresponding coefficient vector* = [1,...,n] <sup>T</sup> *are added to input image* x *to change the classification result of a network.*

*Mathematically, an adversarial attack is a linear parameterized function* g-,x*noise* (·) *that takes an image as an input and produces the corresponding adversarial image.*

$$x^{adv} = g\_{\epsilon, x^{noise}}(x) = x + \Sigma\_{i=1}^n \epsilon\_i \cdot x\_i^{noise} \tag{7}$$

In this paper, we focus on the robustness analysis of SSNs under adversarial attacks. We refer readers to [45] for a survey of state-of-art attack and defenses approaches, mostly for classification.

**Definition 5.** *An* unknown, bounded adversarial attack (UBAA) *is an adversarial attack where the value of the coefficient vector is unknown but bounded in a range* [, ]*, i.e.,* <sup>i</sup> ≤ <sup>i</sup> ≤ i*. An UBAA can be defined formally as a tuple* A = -, , xnoise*.*

**Proposition 1 (UBAA as an ImageStar).** *Applying an UBAA* A = -, , <sup>x</sup>noise *on an image* <sup>x</sup> *creates a set of images, which can be represented as an ImageStar* I = <sup>c</sup> <sup>≡</sup> x, V <sup>≡</sup> <sup>x</sup>noise, P(α) <sup>≡</sup> <sup>P</sup>() <sup>≡</sup> <sup>≤</sup> <sup>≤</sup> *.*

**Definition 6.** *Given a SSN* f *and an input image* x*, a pixel* x(i, j) ∈ x *is called* robust *to an UBAA* A *if and only if:* ∀ g-,x*noise* ∈ A*,* <sup>f</sup>(xadv(i, j)) = <sup>f</sup>(x(i, j))*, where* <sup>x</sup>adv(i, j) <sup>∈</sup> <sup>x</sup>adv <sup>=</sup> <sup>g</sup>-,x*noise* (x)*. If* ∃ g-,x*noise* ∈ A *such that* <sup>f</sup>(xadv(i, j)) <sup>=</sup> f(x(i, j))*, the pixel* x(i, j) *is called* non-robust*.*

**Definition 7.** *The* robustness value (RV) *of a SSN corresponding to an UBAA applied to an input image is defined as* RV = <sup>N</sup>*robust* <sup>N</sup>*pixels* × 100%*, where* Nrobust *is the total number of robust pixels under the attack, and* Npixels = h·w *is the total number of pixels of the input image.*

**Definition 8.** *The* robustness sensitivity (RS) *of a SSN corresponding to an UBAA applied to an input image is defined as* RS = <sup>N</sup>*nonrobust*+N*unknown* <sup>N</sup>*attackedpixels , where* Nnonrobust *is the total number of non-robust pixels under the attack,* Nunknown *is the total number of pixels whose robustness is unknown (may or may not be robust), and* Nattackedpixels *is the total number of attacked pixels of the input image.*

**Definition 9.** *The* robust IoU (Intersection-over-Union) *(*RIoU *) of a SSN corresponding to an UBAA applied to an input image is defined as the average IoU of all labels that are robust under the attack. Let* x *be a segmentation* *ground-truth image,* y *be the verified segmentation image under the attack, and* IoU<sup>p</sup> *be the IoU (also known as Jaccard index) of the* pth *label in the label images* x *and* y*, then the robust IoU of the SSN is computed by:*

$$R\_{IoU} = \frac{\Sigma\_{p=1}^{L} IoU\_p}{L}.\tag{8}$$

The robust IoU definition is quite similar to traditional IoU, which is a core metric to evaluate the accuracy in training SSNs. However, instead of assessing the accuracy, we use the robust IoU concept in combination with the robustness value and robustness sensitivity as core metrics to evaluate the robustness of a SSN under adversarial attack in the verification context.

#### **2.5 Robustness Verification Problem Formulation**

We consider two robustness verification problems.

*Problem 1.* Given a SSN f, an image x, and an UBAA A, prove for every pixel x(i, j) ∈ x that x(i, j) is robust or non-robust to the attack A.

*Problem 2.* Given a SSN f, a set of N test images {x1,...,x<sup>N</sup> }, and an UBAA A, compute the average robustness value RV , the average robustness sensitivity RS, and the average robust IoU RIoU of the SSN (corresponding to A).

The core step in solving these problems is to prove the robustness of a SSN f under an UBAA A at the pixel-level, i.e., Problem 1, which can be solved using reachability analysis computing the "pixel-class reachable set" R<sup>f</sup> = Reach(f, I) that contains all possible classes of every pixel in the input set I constructed by applying the attack A on an image x (Proposition 1). Next, we investigate a new relaxed ImageStar reachability method for the ReLU layer, the up-sampling layers, including transposed convolution, dilated convolution, and pixel-classification. We note that the softmax layer can be neglected in the analysis [33].

### **3 Reachability of SSNs Using Relaxed ImageStars**

In this section, we build on the original ImageStar method to develop reachability analysis for the transposed convolution and dilated convolution layers, and propose a new relaxed ImageStar reachability method for the ReLU and pixel-classification layers. The reachability algorithms for other layers can be handled using existing methods, such as those in [33]. Thus, we highlight handling the up-sampling layers, which requires overcoming significant challenges, and has not previously been done. Handling up-sampling layers is necessary for SSN robustness verification.

#### **3.1 Reachability of a Transposed (Dilated) Convolutional Layer**

Transposed (dilated) convolutions are frequently used for up-sampling in image segmentation applications to generate an output feature map that has a spatial

**Fig. 1.** An example of a transposed convolution operation.

**Fig. 2.** Example 1.

dimension greater than that of the input feature map. A transposed convolution operation consists of four main steps, depicted in Fig. 1, and is defined by its kernel size k, padding p, and stride s. A dilated convolution operation is defined by its kernel size k, padding p, stride s and dilation factor d.

**Lemma 2.** *The reachable set of a transposed (dilated) convolutional layer with an ImageStar input set* I = c, V, P *is another ImageStar, specifically* I = c , V , P *where* c = T Conv(c) *(*c = DConv(c)*) is the transposed (dilated) convolution operation applied to the anchor image,* V = {v 1,...,v <sup>m</sup>}*,* v <sup>i</sup> = T ConvZeroBias(vi) *(*v <sup>i</sup> = DConvZeroBias(vi)*) is the transposed (dilated) convolution operation with zero bias applied to the generator images, i.e., using only the weights of the layer. Each of these are affine operations, see [30] for details, and as shown in Lemma 1, ImageStars are closed under affine operations.*<sup>2</sup>

#### **3.2 Relaxed Reachability of a ReLU Layer**

In this section, we present the relaxed ImageStar reachability of a ReLU layer. Like the original approximate reachability method [33], the relaxed ImageStar approach computes an overapproximate reachable set of a ReLU layer. However,

<sup>2</sup> In most neural network frameworks, transposed and dilated convolution are implemented as convolution with particular choices of padding, stride, and dilation factor as illustrated in Fig. 1 for transposed convolution, which is well-known to be affine.

it allows users to construct a "tight enough" reachable set sufficient to prove properties for their applications via a user-specified relaxation factor scaled from 0% to 100% that reduces verification time. In this paper, we focus on this process for ReLU layers. We use a small example depicted in Fig. 2 to illustrate the reachability of a ReLU layer using the relaxed ImageStar method. In this example, we have a 2 × 2 (4 neurons) ImageStar input set I with the anchor image c and two generator images v<sup>1</sup> and v2, and we want to compute an overapproximation of ReLU(I). To do that, we apply the triangle overapproximation rule [10,36] for the ReLU activation function at each neuron of the input set in the following.

**Lemma 3.** *For any input* x ∈ [l, u]*, the output set* Y = {y| y = ReLU(x)} *satisfies: (1) If* l ≥ 0*, then* y = x*; (2) If* u ≤ 0*, then* y = 0*; or (3) If* l < 0 *and* u > <sup>0</sup>*, then* <sup>Y</sup> <sup>⊂</sup> <sup>Y</sup>¯ <sup>=</sup> {y<sup>|</sup> <sup>y</sup> <sup>≥</sup> <sup>0</sup>, y <sup>≤</sup> <sup>u</sup>(x−l) <sup>u</sup>−<sup>l</sup> , y <sup>≥</sup> <sup>x</sup>}*.*

Using the predicate variable's bounds, we can quickly estimate the ranges of all neurons in the ImageStar set in Fig. 2 without solving any linear programming (LP) optimization problems (by using Eq. 3). From the estimated ranges, we see ReLU(n21)=0(n<sup>21</sup> ≤ 0) and ReLU(n22)=2−α<sup>1</sup> +α<sup>2</sup> (n<sup>22</sup> > 0). Therefore, to overapproximate ReLU(I), we need only perform the overapproximation rule on neurons n<sup>11</sup> and n12, which is where the user-defined relaxation can be applied. In the original approximate reachability approach [33], we use the exact ranges to construct the triangle overapproximation of the ReLU activation function, which requires solving 4 LPs to find the exact ranges for n<sup>11</sup> and n12, which are [−0.5, 1.5] and [−1, 1] respectively in this example. Now, if users want to *reduce the number of LPs* solved in constructing the overapproximate reachable set to speed up verification, which LPs should be chosen to solve to construct a sufficiently tight overapproximate reachable set? For example, if the users want to relax 50% number of LPs for Example 1, then only 4−(50% ×4) = 2 LPs are solved to construct an overapproximate reachable set. So, which two LPs should be chosen?

The answer is found by combining the exact ranges obtained by solving LPs and the estimated ranges to construct the overapproximate reachable set. This can be done using on of the following heuristic approaches. These approaches select which neurons and their corresponding lower (upper) bounds should be obtained exactly to construct an as-tight-as-possible overapproximate reachable set with a given allowable number of LPs. Some of these heuristic approaches are based on the estimated ranges information.

#### **3.2.1 Randomly Relaxed Reachability**

This approach randomly selects some LPs in the LPs pool to solve to obtain the lower (upper) bounds for some (random) neurons. For Example 1, the LPs pool is as follows.

$$\begin{aligned} LP\_{pool} &= \{ \min(n\_{11}), \max(n\_{11}), \min(n\_{22}), \max(n\_{22}), \\ subject &\ to: P = C\alpha \le d, \, l \le \alpha \le u \}. \end{aligned}$$

**Fig. 3.** Overapproximation areas at neurons n<sup>11</sup> and n<sup>12</sup> using estimated ranges.

If users relax 50% of the LPs, then the randomly relaxed reachability algorithm selects aimlessly two LPs in the LP pool to solve, and then combines the obtained lower (upper) ranges with the estimated ranges to construct an overapproximate reachable set using the triangle overapproximation rule, i.e., Lemma 3.

From Fig. 2, we can see that the estimated lower ranges of neurons n<sup>11</sup> and n<sup>12</sup> are the same as the exact ones. Therefore, if the randomly relaxed reachability algorithm selects min(n11) and min(n12) to solve, the final ranges used for constructing the reachable set exactly match the estimated ranges. This means solving min(n11) and min(n12) wastes time and does not reduce the conservativeness of the overapproximate reachable set, as no tighter ranges are obtained. In another case, if the algorithm selects max(n11) and max(n12), then we can obtain the exact ranges of two neurons by solving only two LPs (instead of four LPs), when combining the estimated lower ranges, i.e., −0.5 for n<sup>11</sup> and −1 for n<sup>12</sup> with the optimized upper ranges, i.e., 1.5 for n<sup>11</sup> and 1 for n12. In this case, the randomly relaxed algorithm can obtain the tightest overapproximate reachable set by solving only 50% of the LPs.

#### **3.2.2 Area-Based Relaxed Reachability**

The area-based relaxed reachability approach finds and optimizes the ranges of neurons with the potentially largest triangle overapproximation areas. Figure 3 illustrates the areas of the triangle overapproximation at neurons n<sup>11</sup> and n<sup>12</sup> using the estimated ranges. We see the overapproximation area of n<sup>12</sup> (S˜<sup>n</sup><sup>12</sup> = 0.75) is larger than that of n<sup>11</sup> (S˜<sup>n</sup><sup>11</sup> = 0.625). Therefore, if users relax 50% of the LPs, the area-based relaxed reachability algorithm will use two LPs to optimize the range of neuron n12, i.e., solving min(n12) and max(n12). With this optimized range, the overapproximation area of the neuron n<sup>12</sup> reduces from S˜<sup>n</sup><sup>12</sup> = 0.75 to S<sup>n</sup><sup>12</sup> = 0.5. If users relax 75% of the LPs, then the algorithm will use two LPs to optimize the range of the neuron n<sup>12</sup> and one LP to optimize the upper bound of the neuron <sup>n</sup>11, because ˜u<sup>11</sup> = 2.<sup>5</sup> <sup>&</sup>gt; <sup>|</sup>˜l11<sup>|</sup> = 0.5.

#### **3.2.3 Range-Based Relaxed Reachability**

The range-based relaxed reachability approach finds the neurons with the potentially widest ranges to optimize their ranges. For Example 1, unlike the area-based approach, the range-based approach will use two LPs to optimize the range of neuron n11, i.e., solving min(n11) and max(n11), whose estimated range (ER) is widest (ERn<sup>11</sup> <sup>=</sup> <sup>|</sup>u˜n<sup>11</sup> <sup>−</sup> ˜ln<sup>11</sup> <sup>|</sup> <sup>=</sup> <sup>|</sup>2.<sup>5</sup> <sup>−</sup> (−0.5)<sup>|</sup> = 3 > ERn<sup>12</sup> = 2.5). After optimizing the range of neuron n11, the overapproximation area at this neuron reduces from S˜n<sup>11</sup> = 0.625 to Sn<sup>11</sup> = 0.375. The improvement in terms of overapproximation area reduction of the range-based method is equivalent to the above area-based approach in this case, i.e., ΔSn<sup>11</sup> = ΔSn<sup>12</sup> = 0.25.

#### **3.2.4 Bound-Based Relaxed Reachability**

The bound-based relaxed reachability approach finds neurons with the potentially largest (lower or upper) bounds to optimize their bounds. For Example 1, the algorithm will use two LPs to optimize the upper bounds of the neurons n<sup>11</sup> and n12, i.e., solving max(n11) and max(n12), because their estimated upper bounds are the ones with largest absolute values. Thus, |u˜<sup>n</sup><sup>11</sup> | = 2.5 > |u˜<sup>n</sup><sup>12</sup> | = <sup>1</sup>.<sup>5</sup> <sup>&</sup>gt; <sup>|</sup>˜l<sup>n</sup><sup>12</sup> <sup>|</sup> = 1 <sup>&</sup>gt; <sup>|</sup>˜l<sup>n</sup><sup>11</sup> <sup>|</sup> = 0.5. After optimizing these upper bounds, the overapproximation areas at neurons n<sup>11</sup> and n<sup>12</sup> reduces to 0.375 and 0.5 respectively. In this case, we can see that the bound-based relaxed approach is the best approach compared to the others since it reduces the overapproximation errors at both neurons n<sup>11</sup> and n12, effectively reducing the overapproximation areas by ΔS<sup>n</sup><sup>11</sup> = ΔS<sup>n</sup><sup>12</sup> = 0.25. It is worth noting the obtained overapproximate reachable set is the same as the one obtained by the original approximate ImageStar reachability because the estimated and optimized lower bounds are the same.

#### **3.3 Reachability of a Pixel-Classification Layer**

The last layer in an SSN is a pixel-classification layer, which assigns a specific class (label) to each pixel of an input image. Given an h × w × nc input image, the size of the input x to the pixel-classification layer is h × w × L, where L is the number of classes (labels) of the network (we neglect the softmax layer in the analysis). To assign a specific class l, 1 ≤ l ≤ L to a pixel x(i, j) ∈ x, 1 ≤ i ≤ h, 1 ≤ j ≤ w, the value of the pixel x(i, j) at channel l, i.e., x(i, j, l), needs to be the maximum one among L channels. When the input to the network is an ImageStar set instead of a single image, the input to the pixel-classification layer is a h × w × L ImageStar set. Depending on the value of the predicate variables in the input set, a pixel x(i, j) in the set may be assigned to more than one class. For example, if l1,...,l<sup>m</sup> are the cross-channel max-point candidates of the pixel x(i, j) in L channels, the pixel-class reachable set of the layer at the considered pixel is pc(i, j) = {l1,...,l<sup>m</sup>}. By determining all cross-channel max-point candidates of all pixels in the input set, we can obtain the pixelclass reachable set of the layer, which is also the reachable set of the SSN, R<sup>f</sup> = [pc(i, j)]<sup>h</sup>×<sup>w</sup>, i.e., the collection of pixel classes at every index (i, j).

Similar to the max-pooling layer [33], determining all cross-channel maxpoint candidates of all pixels in the input set can be done via solving linear programming (LP) optimization problems, which is time-consuming due to the number of LPs required (or equivalently the size of the LP). To reduce computation time, we estimate the lower and upper bounds of the ImageStar input to the layer using only the ranges of the predicate variables. These bounds are then used to predict all possible cross-channel max-point candidates of all pixels.

#### **4 Verification Algorithm**

Our reachability-based verification algorithm for SSNs is presented in Algorithm 4.1. The algorithm takes an SSN f, an input image x, an UBAA A, and a reachability method (exact or approximate) as inputs, then returns the pixelclass reachable set R<sup>f</sup> , the robustness value RV , sensitivity RS, and robust IoU RIoU of the SSN. The algorithm works as follows. First, it constructs the input set corresponding to the attack using Proposition 1 (line 2). Then, it computes the pixel-class reachable set of the SSN using reachability analysis layer-bylayer (line 3). Using the pixel-class reachable set, it verifies the robustness of each pixel in the reachable set by comparing its classes with the non-attacked (ground truth) output segmentation image, i.e., y = f(x). If R<sup>f</sup> (i, j) = y(i, j), the pixel x(i, j) is robust under the attack (line 10). If R<sup>f</sup> (i, j) = y(i, j)∧y(i, j) ⊂ R<sup>f</sup> (i, j), the pixel x(i, j) is non-robust under the attack (line 12). Otherwise, the robustness of the pixel x(i, j) is *unknown* (may be robust or non-robust), due to overapproximation. Beyond verifying the robustness of each pixel in the reachable set, it also counts the numbers of 1) robust pixels Nrobust (line 10), 2) non-robust pixels Nnonrobust (line 12), and 3) pixels with unknown robustness Nunknown (line 13). Finally, it computes the robustness value, sensitivity and robust IoU of the SSN (lines 12, 13 and 14). The robustness of a SSN under an UBAA should be evaluated on a set of test images (Problem 2).

**Algorithm 4.1.** Robustness verification of a semantic segmentation network.

**Input:** f, x, A, RF, method SSN, input image, attack, relaxation factor, relaxation method **Output:** R<sup>f</sup> , RV, RS pixel-class reachable set, robustness value, robustness sensitivity 1: **procedure** [R<sup>f</sup> , RV, RS] = verify(f, x, A, RF, method) 2: I = constructInputSet(x, A) construct an ImageStar input set 3: R<sup>f</sup> = Reach(f, I, method) compute the pixel-class reachable set 4: y = f(x) compute non-attacked output segmentation image 5: h = x.Height, w = x.W idth 6: Nrobust = 0, Nnonrobust = 0, Nunknown = 0, Nattackedpixels = 0 7: **for** i =1: h **do** 8: **for** j =1: w **do** 9: **if** <sup>A</sup>.xnoise(i, j) -= 0 **then** Nattackedpixels = Nattackedpixels + 1 10: **if** R<sup>f</sup> (i, j) = y(i, j) **then** Nrobust = Nrobust + 1 pixel <sup>x</sup>(i, j) is robust 11: **else** 12: **if** y(i, j) -⊂ R<sup>f</sup> (i, j) **then** Nnonrobust = Nnonrobust + 1 pixel <sup>x</sup>(i, j) is non-robust 13: **else** Nunknown = Nunknown + 1 pixel x(i, j) robustness is unknown 14: RV = (Nrobust/(h · w)) · 100%) robustness value 15: RS = (Nnonrobust + Nunknown)/Nattackedpixels robustness sensitivity 16: RIoU = getAverageIoU(y, R<sup>f</sup> ) robust IoU

#### **5 Evaluation**

*Experimental Setup.* The approach is implemented in the NNV software tool for verification of deep neural networks<sup>3</sup>. We evaluate our approach by verifying the robustness of a set of SSNs trained on the MNIST [21] and M2NIST datasets shown in Table 1, where class "ten" corresponds to the background, and the other classes to the corresponding digits. The experiments were performed on a computer with an Intel Core i7-6700 CPU at 3.4 GHz with 8 cores and 64 GB Memory running Windows 10. The over-approximating reachability method and 6 cores are used for computing the pixel-class reachable sets.

We randomly selected 100 MNIST images (of size 28 × 28) and 100 M2NIST images (of size 64 × 84) to evaluate the robustness of the trained SSNs. We attack each image x in these two test sets using an UBAA brightening attack. Particularly, we darken a pixel x(i, j) in the image if its value is larger than a threshold <sup>d</sup>, i.e. if <sup>x</sup>(i, j) > d <sup>→</sup> <sup>x</sup>adv(i, j) = <sup>a</sup> <sup>d</sup>. Mathematically, the adversarial darkening attack on an image x can be described as:

$$\begin{aligned} x^{adv} &= x + \epsilon \cdot x^{noise}, \; 1 - \Delta\_{\epsilon} \le \epsilon \le 1, \\ x^{noise}(i, j) &= -x(i, j), \; \text{if } x(i, j) > d, \; \text{otherwise } x^{noise}(i, j) = 0. \end{aligned}$$

For = 1, we completely darken all the pixels whose values are larger than d (=150 in our experiments), i.e., xadv(i, j) = 0. The size of the input set caused by the attack is defined by Δ-. Generally, we have a large input set when Δ is large. To evaluate the average robustness values (RV ) and sensitivities (RS) of the SSNs (on the test sets) in the connection with the number of attacked pixels, we further restrict the maximum allowable number of attacked pixels by Nmax.

We focus our evaluation and discussion on three aspects: 1) the robustness and sensitivity of different SSN architectures under adversarial attacks, 2) the effect of SSN architectures and input size on verification performance, and 3) the improvement of the new relaxed reachability method in terms of verification results and performance. For the first two aspects, we use the relaxed reachability method with relaxation factor RF = 0%, i.e., no relaxation, to construct the reachable sets of the SSNs.

<sup>3</sup> The examples and tool are available: https://github.com/verivital/nnv/tree/ cav2021/code/nnv/examples/Submission/CAV2021. An archival version is available on Zenodo: https://doi.org/10.5281/zenodo.4726346.

**Table 1.** Semantic Segmentation Network Benchmarks. Notation: 'I': input, 'C': convolution, 'TC': transposed convolution, 'DC': dilated convolution, 'R': ReLU, 'B': batch normalization, 'AP': average-pooling, 'MP': max-pooling, 'S': softmax, 'L': label (pixel classification).


**Fig. 4.** The average robustness value, sensitivity, and IoU of MNIST SSNs.

#### **5.1 Robustness and Sensitivity of Different Network Architectures**

**Max-Pooling vs. Average-Pooling.** Max-pooling is the preferred choice over average-pooling for training SSNs because of its nonlinear characteristics. We investigate whether max-pooling is actually better than average-pooling in terms of accuracy and robustness of deep SSN. Figure 4 illustrates the average robustness and sensitivities of MNIST SSNs under different numbers of attacked pixels (Fig. 4a, 20 images are used) and input sizes (Fig. 4b, 10 images are used). We focus on the first two SSNs, i.e. N<sup>1</sup> and N2. These SSNs have the same architectures (with 21 layers). The only difference is N<sup>1</sup> uses average-pooling for down-sampling while N<sup>2</sup> uses max-pooling for the same task (both SSNs use two transposed convolutional layers for up-sampling). With training, we experienced that N<sup>1</sup> is more accurate than N2, (0.87 IoU vs. 0.85 IoU, see Table 1). Interestingly, N<sup>1</sup> is also more robust than N<sup>2</sup> since it has a larger average robustness value (Figs. 4a-a, 4b-a), a higher average robust IoU (Figs. 4a-c, 4b-c), and more robust pixels (Figs. 4a-d, 4b-d). One can also see that the average-pooling-based SSN is less sensitive to the attack than the max-pooling-based SSN (Figs. 4a-(b, e, f), 4b-(b, e, f)). Notably, when more pixels are attacked or larger input sizes are used, the max-pooling-based SSN (i.e., N2) produces more pixels with unknown

**Fig. 5.** Example pixel-class reachable sets of MNIST SSNs. The max-pooling-based SSN N<sup>2</sup> produces more unknown pixels than the average-pooling-based SSN N<sup>1</sup> (19 vs. 6).

robustness (Figs. 4a-f, 4b-f, and 5). Lastly, when the input size increases, the robustness of the max-pooling-based SSN drops more quickly than the averagepooling-based SSNs (Fig. 4b (a,d)) and its sensitivity increases faster (Fig. 4b -b). We believe the main reason causing the max-pooling-based SSN to be more sensitive to the attack is its high nonlinearity using max-pooling layers. It is quite interesting that even the max-pooling-based SSN N<sup>2</sup> has a higher accuracy (0.85) than the non-max-pooling SSN N<sup>3</sup> (0.83), the average robust IoU of the SSN N<sup>2</sup> is smaller than the one of N<sup>3</sup> (Figs. 4a-c, 4b-c).

**Accuracy vs. Robustness; Deeper Networks and ReLU Layer Robustness.** Accuracy (and for segmentation, IoU) is one of the most important factors for evaluating deep neural networks. We investigate whether more accurate and deeper SSNs are more robust compared to other architectures. To determine this, we analyze the robustness of two SSNs with different architectures and accuracy trained on the M2NIST data set. The first SSN N<sup>4</sup> is based on dilated convolution with 16 layers and 0.62 (IoU) accuracy (Table 1). The second SSN N<sup>5</sup> is based on transposed convolution with 22 layers and 0.75 (IoU) accuracy. Here, the second SSN is deeper and more accurate than the first SSN. We run the robustness analysis on these two SSNs on a set of 20 M2NIST images. The results are depicted in Fig. 6. In terms of robustness, the more accurate and deeper SSN N<sup>5</sup> is worse than the less accurate one N<sup>4</sup> as it has a smaller average robustness value and IoU (Figs. 6-(a,c), 7). Additionally, N<sup>5</sup> is also more sensitive to the attack than N<sup>4</sup> (Fig. 6-(b,e)) when we increase the number of attacked pixels. The main reason for this result is, the more accurate SSN contains many ReLU layers (8 ReLU layers) compared with the less accurate one (3 ReLU layers). Similar to the max-pooling layer, using many ReLU layers increases the nonlinearity of the SSN to capture complex features of images. Unfortunately, it also makes the SSN more sensitive to the attack.

**Fig. 6.** The average robustness value, sensitivities, IoU, verification time (Δ- = 10*−*<sup>5</sup>) and reachability times (blue for ReLU layers and orange for others, Δ- = 6 <sup>×</sup> <sup>10</sup>*−*<sup>5</sup>) of M2NIST SSNs. (Color figure online)

**Fig. 7.** Example pixel-class reachable sets of M2NIST SSNs. The more accurate and deeper SSN N<sup>5</sup> produces more non-robust pixels than the less accurate SSN N<sup>4</sup> (51 vs. 43).

**Dilated Convolution vs. Transposed Convolution.** Dilated convolution and transposed convolution are typical choices for semantic segmentation tasks. We compare these techniques in terms of accuracy and robustness. On MNIST SSNs, although the transposed-convolution SSNs N<sup>1</sup> and the dilated-convolution SSN N<sup>3</sup> have the same number of layers (21 layers with 3 ReLU), N<sup>3</sup> is less accurate than N<sup>1</sup> (0.83 vs. 0.87 IoU, see Table 1). In terms of robustness, N<sup>3</sup> is also less robust and more sensitive to the attack than N1, as it has smaller average RV and IoU, and larger sensitivities (Fig. 4). On M2NIST SSNs, by considering 21-layer (8 ReLU) transposed-convolution SSN N<sup>5</sup> and 24-layer (4 ReLU) dilated-convolution SSN N6, one can see that even with more layers, N<sup>6</sup> is less accurate than N<sup>5</sup> (0.72 vs. 0.75 IoU, see Table 1). Also, N<sup>6</sup> is less robust and more sensitive to the attack than N5, since it has smaller average RV and IoU, and larger sensitivities (Fig. 6).

#### **5.2 Verification Performance**

**Dilated Convolution vs. Transposed Convolution.** In general, more attacked pixels and larger input size leads to greater verification time, as depicted in Figs. 8a, 8b and 6b-(a). Interestingly, these show that the dilated-convolutionbased SSNs require greater verification time than the ones using transposed convolution. For example, the verification time of N<sup>3</sup> is larger than N<sup>2</sup> when they have the same number of layers.

**Fig. 8.** Verification time is proportional to the number of attacked pixels and input size. The max-pooling-based N<sup>2</sup> and dilated convolution-based N<sup>3</sup> SSNs require more verification time than the average-pooling and transposed convolution-based SSN N1. The reachability times of ReLU layers (blue) dominates the total reachability time (other layers reachability times are in orange). (Color figure online)

**Max-Pooling and ReLU Layers.** Using max-pooling layer for down sampling not only decreases the robustness of an SSN but also causes a dramatic increase in time and memory consumption in verification. Figure 8 shows that the verification time (in seconds) of the max-pooling-based SSN N<sup>2</sup> grows significantly compared with the average-pooling-based SSN N<sup>1</sup> when increasing the number of attacked pixels Nattackedpixels or the input size Δ-. When dealing with more number of attacked pixels or larger input size, the max-pooling layer introduces more predicate variables to overapproximate the reachable set, which causes the increase both in computation time and memory usage [33]. Similar to the maxpooling layer, the ReLU layer is also the main source of robustness degradation. Additionally, it may also dominate the reachability time of a SSN, as shown in Fig. 8c. This leads to an increase in the verification time for SSNs with many ReLU layers.

#### **5.3 Reducing Verification Time with Relaxation**

When ReLU layer analysis dominates the total verification time significantly, as in the case of MNIST SSNs shown in Fig. 8c and not in the case of M2NIST SSNs depicted in Fig. 6b-(b), we can use the relaxed ImageStar reachability methods to speed up the verification process. Table 2 presents the decrease in the verification times in percentage when applying different relaxation heuristics for ReLU layers. We note that due to the small input size and a small number of attacked pixels, we do not see any changes in the robustness value, sensitivity, and IoU compared with the non-relaxation method, i.e., the original approximate ImageStar method. However, there is a significant improvement in verification time when we apply the relaxed ImageStar reachability for non-max-pooling SSNs N<sup>1</sup> and N3. More relaxation leads to a higher reduction in the verification time: up to 99% of the verification time can be reduced with 100% relaxation in the reachability of ReLU layers.

Interestingly, using relaxation for the max-pooling-based SSN N<sup>2</sup> decreases the verification performance, i.e., leading to higher verification time. The main reason is that the relaxed reachable sets after ReLU layers become increasingly conservative. At the max-pooling layer, a more conservative reachable set leads to more local max-point candidates that need to be determined via solving more LPs, which causes an increase in the verification time. Additionally, if a local region has more than one max-point candidate, a new predicate variable and its corresponding generator image are introduced [33]. The increase in the number of predicate variables and generator images causes the explosion in the memory

**Table 2.** The relaxed ImageStar reachability methods can reduce significantly the verification time (in seconds) of MNIST SSN networks except for the one containing max-pooling layers, i.e., N2. The maximum allowable number of attacked pixels is Nmax = 50 for N<sup>1</sup> and N<sup>2</sup> and Nmax = 20 for N3.


**Fig. 9.** The conservativeness of different relaxation heuristics. The area-based and range-based relaxation strategies outperform others in terms of conservativeness.

usage for the analysis. In the worst case, it can lead to a memory error as shown in Table 2. Therefore, it is important to have relaxation strategies for max-pooling layers, which will be investigated in our future work.

#### **5.4 Conservativeness of Different Relaxation Heuristics**

We have four relaxation heuristics that can be used in the reachability analysis of ReLU layers. The verification time improvement of these methods is quite similar, as shown in Table 2. It is interesting to see how good they are in terms of conservativeness. Unfortunately, we cannot see it clearly via verification of SSNs. Although increasing the number of attacked pixels and input size can eventually show the difference in conservativeness of these methods, it requires a more powerful computer with massive memory for verification. Therefore, to determine the best relaxation heuristic in terms of conservativeness, we evaluate image classification robustness that has been studied extensively recently, and illustrates the benefits of the relaxation method beyond SSN verification. We apply our four relaxation heuristics to verify robustness of an MNIST classification network [29] that is trained by the DiffAI robust training framework under the L∞-norm attack, where all pixels of an input image are attacked independently by a bounded disturbance defined by <sup>4</sup>. The robustness of the network is quantified in percentage stating how many images of 100 randomly selected images are provably robust under the attack, i.e., classified correctly.

**Fig. 10.** When the relaxation factor (RF) ≤ 0.5, the area-based relaxed reachability is less conservative than DeepZono [28] and DeepPoly [29]. It is also faster than these approaches when the disturbance is small, i.e., ≤ 0.11.

Figure 9 illustrates the conservativeness of different relaxation methods. One can see that the area-based and range-based relaxation strategies consistently outperform others in terms of conservativeness since their provable numbers of robust images (in 100 images) under the different sizes of the L<sup>∞</sup> norm attacks are higher than others in all cases. Figure 10 illustrates the conservativeness and verification time of our area-based relaxed reachability (with different relaxation

<sup>4</sup> These benchmarks were used in VNN-COMP'20.

factors (RF)) in comparison with DeepZono [28] and DeepPoly [29]. In terms of conservativeness, the area-based relaxed reachability is better than DeepZono and DeepPoly when we choose a relaxation factor RF ≤ 0.5. When the disturbance is large, DeepZono and DeepPoly may become very conservative. For example, when the disturbance bound = 0.2, the only 5 and 14 (over 100) images are proved robust by DeepZono and DeepPoly, respectively. Meanwhile, without relaxation, i.e., relaxation factor RF = 0, the area-based relaxed reachability can prove 54 images are robust under the attack. It can prove robustness of 48 and 23 images when the relaxation factors are 0.25 and 0.5, respectively. In terms of verification time, when the disturbance is small, i.e., ≤ 0.11, the areabased relaxed reachability is faster than DeepZono and DeepPoly. It is slower than DeepPoly for larger disturbance (except for the case when the relaxation factor is 1). This increase in the verification time is apparent since DeepZono and DeepPoly do not solve any LPs for constructing the overapproximate reachable set of the network while our approach does. Due to using only estimated ranges of the neurons in constructing the reachable set, DeepZono and DeepPoly are overly conservative for a large disturbance, proving only a few images are robust. This reflects the fact that more computation time for optimization is needed to prove more images robust.

#### **6 Related Work**

To enable neural networks use in safety-critical scenarios, many methods have recently been proposed to improve their robustness and temper their susceptibility to adversarial attacks. The following section surveys the landscape of these approaches in order to better contextualize our work.

*SSN Robustness.* SSNs are used in visual understanding systems in numerous contexts, recent works aim to improve the robustness of these models [13,20,23,25], albeit none that provide worst-case guarantees, as our approach does. For instance, recent work develops rigorous testing-based approaches to evaluate the robustness of SSNs, considering a wide range of architectures, and offering an insightful discussion about the comparative robustness of these modalities against various adversarial attacks [2]. Kamann et al. conducted an extensive evaluation of a state-of-the-art SSN using over 400,000 images and issued a series of recommendations aimed at improving robustness to common perturbations. Zhou et al. presented an automated method for evaluating robustness of SSNs within visual systems for autonomous vehicles, which leverages an additional sensor to generate ground truth labels so that an examination of the classification accuracy of an SSN can be evaluated at runtime[47]. Robust training techniques that incorporate image corruptions and architecture modalities have also been developed for SSNs [20]. Even though such works provide better understanding, potential defenses against adversarial perturbations, run-time evaluation, and comparative robustness measures, they cannot provide formal verification guarantees for SSN robustness as our work does.

*Neural Network Verification and Falsification.* The bulk of neural network verification approaches have been aimed at verifying input-output properties of DNNs. These methods include SMT [18,19], polyhedral [35,44], mixed integer linear programming (MILP) [9], interval arithmetic [38], zonotope [28], linearization[39], and abstract-domain [29] approaches. There have also been a number of works aimed at testing the robustness of networks with respect to bounded input perturbations such as feature-guided search, global optimization, and game theory [16,42]. One such example is the work of Dreossi et al. where the authors proposed a general definition of robustness for DNNs [8]. Their work categorizes the existing literature into approaches that consider local robustness properties [6], and those that focus on verifying the global robustness of the networks [14]. Most of the existing research in this area focuses on robustness of classification neural networks, specifically image classification. While many approaches aim at verification, methods also exist for falsification of system specifications, in which robustness properties are included [12]. However, to the best of our knowledge, no existing approaches consider verification for SSNs, as we do in this paper.

*Sequence Model Verification and Robustness Analysis.* Aside from classification tasks, there are several verification approaches for sequence models. Unlike SSN and classification networks, the output of sequence models such as recurrent neural networks (RNNs) depends on spatially or temporally ordered data [4,41]. While some of these efforts are similar in spirit to our work in expanding the classes of problems and models for verification, the verification tasks and approaches differ.

*Scalability and Specifications.* Finally, verification of DNNs is challenging, and presently the most complex networks remain inaccessible to the majority of methods. However, several recent approaches have focused on improving the efficiency of existing methods via parallelization and other techniques [3,35,40]. As verification work is only meaningful when paired with high-quality specifications, there has been significant work on the importance of semantics when defining system specifications against adversarial attacks [27], and our paper contributes to this direction through our formulation of robustness specifications and metrics for segmentation tasks.

### **7 Conclusion**

We present the first formal approach to verify robustness of SSNs using relaxed reachability analysis. Our evaluation has analyzed the robustness and sensitivity under adversarial attacks on a set of SSNs with typical architectures. From our experiments, we show that while max-pooling and ReLU layers are useful in training highly accurate SSNs, they are also the main sources of robustness and verification performance degradation. SSNs using average-pooling for downsampling and transposed convolution for up-sampling seem to be an optimal choice for achieving high accuracy, robustness, and verification performance. Additionally, our relaxed reachability approach can help to reduce significantly the total verification time for networks where the reachability time of ReLU layers dominates the network's reachability time, and are applicable to other networks, such as CNNs used for classification. In the future, we will investigate new relaxation heuristics for the max-pooling layer and extend this work to cope with the encoder-decoder SSN architecture where max-unpooling layers are used for up-sampling operations, instead of dilated/transposed convolution as we considered in this paper.

**Acknowledgments.** The material presented in this paper is based upon work supported the Defense Advanced Research Projects Agency (DARPA) through contract number FA8750-18-C-0089, the Air Force Office of Scientific Research (AFOSR) award FA9550-19-1-0288, and the National Science Foundation (NSF) through grant numbers 1910017 and 2028001. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of DARPA, AFOSR or NSF.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **PEREGRiNN: Penalized-Relaxation Greedy Neural Network Verifier**

Haitham Khedr(B), James Ferlez, and Yasser Shoukry

University of California, Irvine, USA *{*hkhedr,jferlez,yshoukry*}*@uci.edu

**Abstract.** Neural Networks (NNs) have increasingly apparent safety implications commensurate with their proliferation in real-world applications: both unanticipated as well as adversarial misclassifications can result in fatal outcomes. As a consequence, techniques of formal verification have been recognized as crucial to the design and deployment of safe NNs. In this paper, we introduce a new approach to formally verify the most commonly considered safety specifications for ReLU NNs – i.e. polytopic specifications on the input and output of the network. Like some other approaches, ours uses a relaxed convex program to mitigate the combinatorial complexity of the problem. However, unique in our approach is the way we use a convex solver not only as a linear feasibility checker, but also as a means of penalizing the amount of relaxation allowed in solutions. In particular, we encode each ReLU by means of the usual linear constraints, and combine this with a convex objective function that penalizes the discrepancy between the output of each neuron and its relaxation. This convex function is further structured to force the largest relaxations to appear closest to the input layer; this provides the further benefit that the most "problematic" neurons are conditioned as early as possible, when conditioning layer by layer. This paradigm can be leveraged to create a verification algorithm that is not only faster in general than competing approaches, but is also able to verify considerably more safety properties; we evaluated PEREGRiNN on a standard MNIST robustness verification suite to substantiate these claims.

**Keywords:** Machine learning/AI · Decision procedures and solvers

#### **1 Introduction**

Neural Networks have become an increasingly central component of modern machine learning systems, including those that are used in safety-critical cyberphysical systems such as autonomous vehicles. The rate of this adoption has exceeded the ability to reliably verify the safe and correct functioning of these components, especially when they are integrated with other components such as

This work was sponsored by the NSF awards #CNS-2002405 and #CNS-2013824.

c The Author(s) 2021 A. Silva and K. R. M. Leino (Eds.) CAV 2021, LNCS 12759, pp. 287–300, 2021. https://doi.org/10.1007/978-3-030-81685-8\_13

controllers. Thus, there is an increasing need to verify that NNs reliably produce safe outputs, especially subject to malicious adversarial inputs [16,20,27,28].

In this paper, we propose PEREGRiNN, an algorithm for efficiently and formally verifying the input/output behavior of ReLU NNs. In this context, PERE-GRiNN falls into the broad category of sound and complete *search and optimization* NN verifiers [22]. The *search* aspect of PEREGRiNN involves iterating over different combinations of neuron activation patterns to verify that each is compatible with the specified safety constraints (on the input and output of the network). Like other algorithms in this category, PEREGRiNN combines this search with *optimization* techniques to make inferences about the feasibility of full-network activation patterns on the basis of activation patterns of only a subset of neurons. The optimization in question reformulates the original NN feasibility problem into a relaxed convex feasibility problem to allow sound inferences: i.e. if the convex relaxation is infeasible, then the original NN problem may soundly be concluded to be infeasible. In this relaxed feasibility problem, the output of each individual neuron is assigned a relaxation variable that is decoupled from the actual output of that neuron. PEREGRiNN also uses a type of reachability analysis (symbolic interval analysis) both to enhance the optimization-based inference described above and as a source of additional sound inference itself. For this reason, PEREGRiNN's search procedure searches neurons in a layer-by-layer fashion, preferring to fix the phases of neurons closest to the input layer first.

In contrast to other search and optimization algorithms, however, PERE-GRiNN *augments* each convex feasibility query with a (convex) penalty function in order to obtain better guidance on which activation patterns to search next. In particular, we note that the amount of relaxation needed on a neuron can be regarded as a *quasi-measure* of how close the convex solver came to operating the associated neuron in a valid regime – i.e. at a valid evaluation of that neuron on a particular input. In this sense, the amount of relaxation in aggregate can be regarded as a quasi-measure of how close the solver came to finding a valid evaluation of the network as a whole. Inversely, the largest distance between a relaxation variable and its neuron's closest ReLU constraint intuitively corresponds in some sense to how "problematic" that neuron is with regard to obtaining such a valid evaluation. These distances we refer to as the *"slacks"* for each neuron. Thus, PEREGRiNN may be regarded as *greedily* minimizing a *slack-based penalty*.

Finally, we evaluated the performance of PEREGRiNN by using it to verify the adversarial robustness of networks trained on the MNIST [21] dataset. Our experiments show that PEREGRiNN is on average 1.27× faster than Neurify [31], 1.24× faster than Venus [6], 1.15× faster than nnenum [4], and 1.65× faster than Marabou [19]. It also proves 27%, 19%, 10%, and 51% more properties than the other solvers, respectively. PEREGRiNN's unique convex penalty augmentations are also considered in ablation experiments to validate their benefits.

*Related Work.* Since PEREGRiNN is a sound and complete verification algorithm, we restrict our comparison to other sound and complete algorithms. NN verifiers can be grouped into roughly three categories: (i) SMT-based methods, which encode the problem into a Satisfiability Modulo Theory problem [11,18,19]; (ii) MILP-based solvers, which directly encode the verification

**Fig. 1.** Block diagram of the PEREGRiNN algorithm

problem as a Mixed Integer Linear Program [3,5–8,14,23,29]; (iii) Reachability based methods, which perform layer-by-layer reachability analysis to compute the reachable set [4,13,15,17,30,32,34,35]; and (iv) convex relaxations methods [10,31,33]. In general, (i), (ii) and (iii) suffer from poor scalability. On the other hand, convex relaxation methods depend heavily on pruning the search space of indeterminate neuron activations; thus, they generally depend on obtaining good approximate bounds for each of the neurons in order to reduce the search space (the exact bounds are computationally intensive to compute [9]). These methods are most similar to PEREGRiNN: for example, [7,25,32] recursively refine the problem using input splitting, and [31] does so via neuron splitting. Other search and optimization methods include: Planet [11], which combines a relaxed convex optimization problem with a SAT solver to search over neurons' phases; and Marabou [19], which uses a modified simplex algorithm.

#### **2 Problem Formulation**

In this paper, we will consider Rectified Linear Unit (ReLU) NNs. An n-layer ReLU network, is a composition of <sup>n</sup> ReLU layer functions: i.e. N N <sup>=</sup> <sup>f</sup><sup>n</sup> ◦ <sup>f</sup><sup>n</sup>−<sup>1</sup> ◦···◦ <sup>f</sup><sup>1</sup> where the <sup>i</sup> th ReLU layer function is defined as <sup>f</sup><sup>i</sup> : <sup>y</sup> <sup>∈</sup> <sup>R</sup><sup>k</sup>*i−*<sup>1</sup> → max{W<sup>i</sup><sup>y</sup> <sup>+</sup> <sup>b</sup><sup>i</sup>, <sup>0</sup>} ∈ <sup>R</sup><sup>k</sup>*<sup>i</sup>* . We refer to <sup>f</sup><sup>1</sup> as the *input layer*. Finally, to refer to individual neurons, we use the notation (z)<sup>j</sup> to indicate the <sup>j</sup>th element of <sup>z</sup>.

**Verification Problem.** Let N N be an n-layer NN as defined above. Furthermore, let <sup>P</sup><sup>y</sup><sup>0</sup> <sup>⊂</sup> <sup>R</sup><sup>k</sup><sup>0</sup> be a convex polytope in the input space of N N , and let <sup>P</sup><sup>y</sup>*<sup>n</sup>* <sup>⊂</sup> <sup>R</sup><sup>k</sup>*<sup>n</sup>* be a convex polytope in the output space of N N . Finally, let h- : <sup>R</sup><sup>k</sup><sup>0</sup>×R<sup>k</sup>*<sup>n</sup>* <sup>→</sup> <sup>R</sup>, - = 1,...,m be convex functions defining joint input/output constraints on N N . Then the verification problem is to decide whether

$$\left\{ x \in \mathbb{R}^{k\_0} \, \middle| \, x \in P\_{y\_0} \land \mathcal{NN}(x) \in P\_{y\_n} \land \left( \bigwedge\_{\ell=1}^m h\_\ell(x, \mathcal{NN}(x)) \le 0 \right) \right\} = \emptyset. \tag{1}$$

#### **3 PEREGRiNN Overview**

The general structure of PEREGRiNN is depicted in Fig. 1. Like other search and optimization based NN verifiers it has two main components: a *search component* and an *inference component*, and PEREGRiNN iterates back and forth

between these these two components until termination. In particular, the search and inference components interact in the following way. The search component successively iterates over all possible on/off activations for each neuron; this is done by fixing these activations one neuron at a time, starting from the input layer and working towards the output layer. The process of fixing a neuron's activation is referred to as *conditioning its phase*: each neuron can be in either its active phase (operating linearly) or inactive phase (outputting zero). Thus, the search component provides the inference component a subset of neurons, each of which has been conditioned; the inference component then attempts to soundly reason about whether the remaining, unconditioned neurons can be operated in such a way as to violate the safety constraint. If the inference component soundly concludes safety for all possible activations of the remaining unconditioned neurons, then the search component backtracks, oppositely reconditioning one of the neurons that was already conditioned. Otherwise, if a sound safe conclusion is not made, then the search component uses information from the inference component to decide on a new neuron to condition, and the process repeats. The algorithm terminates if either a counterexample to safety is found, or else all possible neuron activations are considered without finding such a counterexample.

The convex program inference block is at the heart of the inference component and PEREGRiNN itself. In this block, PEREGRiNN, like other search and optimization solvers, uses a relaxed linear feasibility program where the output of each individual neuron is assigned a relaxation variable that is decoupled from the actual output of that neuron. In the notation of Sect. 2, such a linear feasibility program can be written as follows, where the vector variables y<sup>i</sup>, i = 0 are the relaxation variables.

$$\begin{cases} y\_i \ge 0, \ y\_i \ge W\_i y\_{i-1} + b\_i & \forall i = 1, \ldots, n \\ y\_0 \in P\_{y\_0}, \ y\_n \in P\_{y\_n}^{\mathfrak{c}}, \ \bigwedge\_{\ell=1}^m h\_\ell(y\_0, y\_n) \le 0 \end{cases} \tag{2}$$

Importantly, if (2) is infeasible, then the original NN problem in (1) may be soundly concluded to be infeasible as well – and hence, safe. However, as described above, the primary function of the convex feasibility program is to use a set of conditioned neurons supplied by the search component in order to soundly reason about the remaining neurons. To do this, the conditioned neurons supplied by the search component are incorporated into the feasibility program (2) as *equality* constraints in the following way:

$$\underset{\text{i.v.}}{\text{Neuron}} \ (y\_i)\_j \text{ ON:} \quad (y\_i)\_j = (W\_i y\_{i-1} + b\_i)\_j \land (y\_i)\_j \ge 0 \tag{3}$$

$$\text{Neuron } (y\_i)\_j \text{ OFF: } \quad (y\_i)\_j = 0 \land (W\_i y\_{i-1} + b\_i)\_j \le 0. \tag{4}$$

Inferences created by the symbolic interval inference block using Symbolic Interval Analysis [32] are also incorporated using equality constraints like (3) and (4).

Of the remaining blocks, the "Backtracking & Reconditioning" block is essentially described above. The "Condition New Neuron" and "Sampling Inference" blocks have features unique to PEREGRiNN that are described in Sect. 4; the

former implements a novel neuron prioritization, and the latter is a unique approach to quickly obtaining initial safety counterexamples.

#### **4 PEREGRiNN Enhancements**

#### **4.1 Sum-of-Slacks Penalty**

The core enhancement in PEREGRiNN is the inclusion of a specific objective function in the convex program used by the inference component. As per the discussion above, this objective function is interpreted as a *penalty* on how far away a particular solution is from a valid input/output response of the network (and activation pattern on all hidden neurons). Specifically, this penalty function penalizes the sum of all of the "slack" variables for the entire network, where each neuron's slack variable is defined as <sup>s</sup><sup>i</sup> <sup>y</sup><sup>i</sup>−(W<sup>i</sup> ·y<sup>i</sup>−<sup>1</sup>+b<sup>i</sup>). That is the distance between a relaxation variable <sup>y</sup><sup>i</sup> and the linear response of its associated neuron. During each feasibility/inference call, this has the obvious effect of incentivizing the convex solver to choose an actual input/output response of the network.

In addition, this penalty is effectively the L<sup>1</sup>-norm of the *vector* of all the slack variables, since the slack variables are non-negative. The L<sup>1</sup>-norm of a vector, used as a penalty function, is well known to effectively encourage *sparsity* on the resulting optimal solution. Thus, the sum-of-slacks effectively incentivizes the convex solver to leave as *few* neurons as possible indeterminate in the solution. That is a sum-of-slacks penalty effectively encourages the convex solver to fix the phases of as many neurons as possible.

#### **4.2 Max-Slack Conditioning Priority**

As noted above, the search component of PEREGRiNN operates layer-wise from input layer to output layer in order to leverage Symbolic Interval Analysis for additional inference. Hence, the search component always chooses the next neuron to be searched (i.e. conditioned) from among those as-yet-unconditioned neurons that are closest to the input layer. It further makes sense to only consider conditioning neurons that the convex solver was unable to operate at valid inputs/output. However, the convex solver typically returns several neurons to choose from with this property, and it is necessary to choose which of them to search next. Given the interpretation of a neuron's "slack" variable as a measure of how "problematic" that neuron was for the solver to obtain a valid evaluation of the network, PEREGRiNN's search component chooses the next neuron to condition based on slack-order ranking of those neurons that are not being operated at valid input/output points. This "max-slack" heuristic choice is unique to PEREGRiNN; compare to the output gradient heuristic employed in [31].

#### **4.3 Layer-wise-Weighted Penalty**

PEREGRiNN takes the "max-slack" neuron search priority one step further, though. Using techniques similar to those in [26], it is possible to show that

there exists weights <sup>q</sup><sup>1</sup>,...,q<sup>n</sup> such that solving (2) with the penalty

$$\min\_{y\_0, \dots, y\_n} \sum\_{i=0}^n \sum\_{j=1}^{k\_i} q\_i s\_{ij} \tag{5}$$

will result in a solution that is guaranteed to concentrate the most total slack in the earliest (unconditioned) layer. Thus, by using the layer-wise weighted sum-ofslacks penalty in (5), PEREGRiNN is uniquely able to force the (unconditioned) layer closest to the input layer to have the *largest* total slack among all the layers. As a consequence, PEREGRiNN effectively concentrates the most "problematic" neurons in the layer where the next conditioning choice will be made. This scheme makes it much more likely that the neuron with the highest slack among *all* of the neurons will be among the next neurons considered for conditioning – in effect, often guiding the search component to condition on the most problematic neuron in the whole network (although this is not guaranteed).

As noted above, SMC [26] can be used to obtain layer-wise weights that guarantee concentration of slack in the earliest (shallowest) layer. However, these weights are often very large, since they depend on bounding the slack variables (most readily by over-approximation); the effect of this is possible computational instability in the convex program. Thus, as an *implementation* matter, we instead select these weights using a heuristic scheme characterized by two real-valued hyperparameters, <sup>λ</sup><sup>0</sup> and <sup>γ</sup>. In particular, the weight of the <sup>i</sup> th layer, q<sup>i</sup>, is selected as <sup>q</sup><sup>i</sup> <sup>=</sup> <sup>λ</sup><sup>0</sup> · <sup>γ</sup><sup>i</sup> . In our experiments, we found the values <sup>λ</sup><sup>0</sup> = 10−<sup>7</sup> and γ = 10<sup>3</sup> to effectively achieve the maximum slack concentration in the earliest layers.

#### **4.4 Initial Counterexample Search by Sampling**

Finally, PEREGRiNN extends a simple idea first introduced in [32] to rapidly identify counterexamples by means of sampling. The basic idea is to sample within a known region of the input to the NN (or the input to some deeper layer), and evaluate the NN (sub-NN) exactly on those samples in order to rapidly identify a counterexample; this approach help identify un-safe networks/properties early on. However, whereas [32] samples from within hyper-rectangle sets derived by symbolic interval analysis, PEREGRiNN uses the Volesti [12] Python library to uniformly sample points within the *polytopic* input constraint set, <sup>P</sup><sup>y</sup><sup>0</sup> , and thus applies to be more general input constraint sets in (1).

#### **5 Experiments**

We evaluated the performance and effectiveness of PEREGRiNN at verifying the adversarial robustness of NNs trained to recognize digits using the standard MNIST dataset. This verification problem fits into the general NN verification problem described in Sect. 2, and it is described subsequently in detail. In this context, we evaluated PEREGRiNN with two objectives described as follows.


**Table 1.** Architecture of the NN models used in the experiments


**Implementation.** We implemented PEREGRiNN in Python, and used an offthe-shelf Gurobi 9.1 [1] convex optimizer for solving linear programs; the Volesti [12] Python interface was used to sample from the input polytope for the sampling inference block. For the other NN verifiers, we used publicly available implementations that were published by their creators (citations are included below). Each instance of of any verifier was run within its own single-core Virtual Box VM with 30 GB of memory; no more than 4 VMs were run concurrently on a host machine with 48 hyperthreaded cores and 256 GB of memory.

#### **5.1 Adversarial Robustness Verification Task**

Subsequent experiments used the testbench we describe in this section; it is largely identical to the PAT-FCN test in the VNN-COMP 2020 competition [2].

**Neural Networks.** We used three ReLU NNs to recognize digits using the standard MNIST training database; these NNs are exactly as in the PAT-FCN portion of [2]. The sizes of these fully-connected networks are described in Table 1. Each entry in the "Architecture" column of Table 1 is the number of number of neurons in a layer, from input layer on the left to output layer on the right.

**Verification Properties.** We created a number of NN verification tasks based on proving whether the above described networks were robust against max-norm perturbations of their inputs. In particular, each verification task involves proving whether a particular input image, x , always results in the same classification when it is subjected to a max-norm perturbation of at most some fixed size, > 0. Thus, each such verification problem is parameterized by both the specified input image, x , and the maximum amount of perturbation, .

Formally, let x be a given image in category t ∈ {1,...,M}, and let > <sup>0</sup> be a specified maximum amount of max-norm perturbation of x . Then we say that a NN with M classification outputs, N N , is robust if for each classification category m ∈ {1,...,M}\{t} the set of inputs yielding classification of x as <sup>m</sup>

$$\phi\_m \triangleq \{ x \mid x \in \mathbb{R}^{k\_0}, \|x - x'\|\_{\infty} \le \epsilon, \, z \in \mathbb{R}^{k\_n}, \max\_{i=1,\ldots,n} \mathcal{N} \mathcal{N}(x)\_i = \mathcal{N} \mathcal{N}(x)\_m \} \tag{6}$$

is empty. Note that each instance of (6) is compatible with the problem in (1).

**Adversarial Robustness Verifier Testbench.** Our verification testbench was then constructed by selecting 50 test images from the MNIST test dataset; this set of test images includes the 25 used in the PAT-FCN portion of [2]. Each test instance was then a combination of one of those images, one of the networks from Table <sup>1</sup> and one the following two max-norm perturbations, = 0.02 or = 0.05; these perturbations are same ones used in PAT-FCN [2]. Thus, each verification test in our testbench can be identified by one of 300 tuples of the form: (net, image, perturb.) <sup>∈</sup> *TB* -{FC1, FC2, FC2}×{1,..., <sup>50</sup>}×{0.02, <sup>0</sup>.05}.

#### **5.2 Ablation Experiments**

In this series of experiments we evaluated the contribution that each of the primary PEREGRiNN enhancements made to its overall performance. This was done by comparing the full PEREGRiNN algorithm – as described in Sect. 4 – with altered versions that replace exactly one of those enhancements at a time. *Note:* removing core features of PEREGRiNN often resulted in much longer run times, so the experiments in this section use a testbench *TB* ⊂ *TB* that excludes all tests with one of the larger networks FC2 or FC3 *and* = 0.05.

**Penalty Function Ablation.** Our first ablation experiment evaluated the contribution of PEREGRiNN's unique penalty function features; see Sect. 4.1 and Sect. 4.3. In particular, we ran different variants of PEREGRiNN with the following penalty functions used inside the convex program inference block:


Figure 2a shows a cactus plot of the number of proved cases vs. the timeout permitted to the algorithm: i.e. to prove at least a specified number of the test cases, each algorithm must have its timeout set at to the value of its curve in Fig. 2a. Figure 2b shows a histogram of the number of times each of the algorithm variants needed to call the convex solver in order to terminate; this quantifies each algorithm's cost in a well-known unit of computation, also the single most computationally costly part of PEREGRiNN. Figure 2b plots the number of convex solver calls required for evenly spaced bins of convex solver calls.

**Fig. 2.** Performance of PEREGRiNN variants with different objective functions

*Conclusions:* Figure 2a demonstrates that PEREGRiNN's weighted sum of slacks has a clear benefit over both a uniformly weighted sum-of-slacks penalty and a plain feasibility convex program. For timeouts of longer than <sup>≈</sup> <sup>1</sup>.2 seconds, PEREGRiNN overtakes the other two in terms of number of properties proved; even the uniform sum-of-slacks penalty considerably outperforms the feasibility convex program at similar timeouts. Note that *reversing* the layer-wise weights of PEREGRiNN's penalty function incurs a *performance hit*, especially for timeouts >1.2 s. This suggests that driving slacks toward shallower layers, where the next neuron is conditioned, is the correct heuristic to apply. Figure 2b also shows that going from feasibility to sum-of-slacks to weighted sum-of-slacks significantly reduces the number of test cases that require between 425 and 525 calls to the convex solver. This order of comparison shows a concomitant net influx of tests into the lowest bin of < 25 convex calls; PEREGRiNN has the most test cases in this category, with <sup>≈</sup>130 test cases proved in < 25 convex solver calls.

**Neuron Conditioning Priority Ablation.** In the second ablation experiment, we evaluated the contribution of PEREGRiNN's maximum-slack neuron conditioning priority (see Sect. 4.2). To that end, we ran variants of PERE-GRiNN with three different neuron conditioning priorities for the search component:


The performance of these algorithm variants is shown in Fig. 3a and Fig. 3b. As in the previous ablation experiment, Fig. 3a shows a cactus plot of the number

of proved cases vs. the timeout, and Fig. 3b shows a histogram of the number of calls to the convex solver required under each of the conditioning priorities.

*Conclusions:* Figure 3a shows that PEREGRiNN's max-slack neuron priority allows it to prove slightly more properties than either a random neuron choice priority or the minimum-slack priority. The maximum slack priority also required the fewest total convex calls across all instances: it used 178 fewer than minimum slack and 686 fewer than a random choice. Thus, we conclude PEREGRiNN's max-slack heuristic slightly improves performance on this testbench.

#### **5.3 Comparison with Other NN Verifiers**

In this experiment, we evaluated PEREGRiNN with respect to a number of state-of-the-art NN verifiers on our adversarial robustness testbench, *TB*. In particular, we ran the following tools on *TB*: Venus [6]; Marabou [19]; Neurify [31]; and nnenum [4]. Venus was run with st ratio=0.4, depth power=4, offline deps = True, online deps = True, and ideal cuts = True; Marabou and Neurify were used with default parameters but THREADS = 1; and nnenum had ADVERSARIAL SEARCH turned off. Each algorithm had its own one-core VM.

(a) Cactus plot; proved cases vs. timeout (b) Histogram; number convex calls used

**Fig. 3.** Performance of PEREGRiNN variants with different conditioning priorities

Figure 4 contains a cactus plot showing the results for each of these algorithms, including PEREGRiNN. For a given number of test cases to be proved, Fig. 4 depicts the corresponding timeout required for each of the algorithm to prove that many cases. Of all the algorithms, PEREGRiNN was able to prove the most properties within the timeout limit of 600 s: PEREGRiNN was able to prove 190 properties; it was followed by nnenum, which proved 172; Venus, which proved 159; Neurify, which proved 149; and Marabou, which proved 125. Marabou consistently performed the worst, proving fewer cases than any other algorithm at every timeout. By contrast, Neurify was able to prove significantly more test cases than any other algorithm for extremely short timeouts, but it failed to prove more than 150 out of 300 test cases across the whole experiment. nnenum performed worse than Neurify on the way to proving 150 test cases, but it fared significantly better than either PEREGRiNN or Venus, which had more or less similar performance below this threshold. However, after ≈150 test cases,

PEREGRiNN significantly outperformed all other algorithms: as the timeout was increased, PEREGRiNN proved additional properties at a rate significantly outpacing its closest competitor in this regime, nnenum. We further note that all algorithms proved a mixture of SAT and UNSAT properties.

This data, taken as a whole, suggests that PEREGRiNN suffers from a worse "best-case" performance than several other algorithms, especially nnenum and Neurify. However, PEREGRiNN's performance seems to be much more consistent across different test cases. This allows it to prove more properties in aggregate at the expense of being slower on a smaller subset of them. This further suggests that PEREGRiNN is significantly less sensitive to peculiarities of particular test cases on the *TB* testbench. This will likely be a considerable advantage, on average, when faced with verifying unknown networks and properties of this type.

#### **6 Discussion: Analogy to SAT Solvers**

It is possible to draw a loose analogy between SAT solvers and search-andoptimization NN verifiers such as PEREGRiNN. Indeed, since each neuron has two phases, the operational phase of each neuron can be captured by a binary variable; then any valuation of *all* these variables can be interpreted as SAT or UNSAT based on the Input/Output properties to be verified on the network (subject to that conditioning). Thus, the neuron conditioning step in PERE-GRiNN is analogous to variable splitting in a SAT solver, and the *backtrack and re-condition* block (see Fig. 1) functions analogously to backtracking. In this analogy, infeasibility of the convex program and symbolic interval analysis function roughly like unit resolution in a SAT solver: they soundly reason about the overall property before all neurons have been conditioned (i.e. variables split).

**Fig. 4.** Cactus plot of various solvers on 300-case testbench, *TB*

However, the main contribution of PEREGRiNN is a heuristic for deciding which neuron to condition next: it is thus analogous to a heuristic for choosing the next variable to split in a SAT solver. Specifically, PEREGRiNN's heuristic provides a numerical ranking of the as-yet-unconditioned neurons, and therefore has a functional similarity to variable-ranking heuristics in SAT solvers (e.g. VSIDS [24]). On the other hand, PEREGRiNN's neuron ranking comes directly

from the output of the convex solver, which we argued reveals some information about the underlying verification problem – this has no direct SAT-solver analog.

### **7 Conclusion**

In this paper, we introduced PEREGRiNN, a new tool for formally verifying input/output properties for ReLU NNs. PEREGRiNN compares favorably with other state-of-the-art NN verifiers, thanks to a number of unique algorithmic features. The benefits of these features were established with ablation experiments.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Concurrency and Blockchain**

### Isla: Integrating Full-Scale ISA Semantics and Axiomatic Concurrency Models

Alasdair Armstrong1(B) , Brian Campbell<sup>2</sup>, Ben Simner<sup>1</sup>, Christopher Pulte<sup>1</sup>, and Peter Sewell<sup>1</sup>

> <sup>1</sup> University of Cambridge, Cambridge, UK alasdair.armstrong@cl.cam.ac.uk <sup>2</sup> University of Edinburgh, Edinburgh, UK

In this paper we present a tool, Isla, for computing the allowed behaviours of concurrent litmus tests with respect to full-scale ISA definitions, in Sail, and arbitrary axiomatic relaxed-memory concurrency models, in the Cat language. It is based on a generic symbolic engine for Sail ISA specifications, which should be valuable also for other verification tasks. We equip the tool with a web interface to make it widely accessible, and illustrate and evaluate it for Armv8-A and RISC-V.

By using full-scale and authoritative ISA semantics, this lets one evaluate litmus tests using arbitrary user instructions with high confidence. Moreover, because these ISA specifications give detailed and validated definitions of the sequential aspects of *systems* functionality, as used by hypervisors and operating systems, e.g. instruction fetch, exceptions, and address translation, our tool provides a basis for developing concurrency semantics for these. We demonstrate this for the Armv8-A instructionfetch model and self-modifying code examples of Simner et al.

#### 1 Introduction

A processor architecture should define, for any initial machine state, the set of all architecturally allowed observable executions—thus specifying the basic assumptions for programming and for software verification, and the correctness criterion for hardware verification. Architecture specifications have two main parts: the sequential and relaxed-memory concurrent aspects of instruction behaviour, each of which have been studied in previous work. For Armv8- A and RISC-V, Armstrong et al. have established full-scale sequential models in Sail [10,15], a domain-specific language for instruction-set architecture (ISA) specification, that are complete enough to boot real-world operating systems such as Linux. For Armv8-A this model is automatically derived from the authoritative Arm-internal specification [24], while for RISC-V it has been hand-written and adopted by RISC-V International. On the concurrency side, relaxed-memory semantics can be specified in two main styles: either as *abstractmicroarchitectural operational* models, characterising observable behaviour with explicit out-of-order execution and buffering, or as *axiomatic* models, expressed as a predicate over complete candidate executions represented as graphs of memory events. For Armv8-A and RISC-V "user" concurrency, both exist [1,7,8,22], along with a "Promising ARM" variant [23]. For Armv8-A they have been proved equivalent [21,22]; the authoritative vendor definition is the axiomatic one.

However, while an architecture *should* define the set of allowed executions for arbitrary programs, hitherto there has been no integration of full-scale ISA definitions with axiomatic concurrency models, either in mathematics or in tools (for operational models, this has only been done for RISC-V; other operational models have used small ISA fragments). Research and industry practice for relaxed memory semantics rely on making the semantics *executable as a test oracle*: not just a paper definition (in prose or mathematics), but tool-supported definitions that for small litmus-test examples can *compute* the set of all allowed executions, that can then be compared against experimental data. Many tools have been developed for operational and axiomatic architectural concurrency models [4,6,8,12,14,17–20,25,26,28–32], with axiomatic tools notably including the Herd tool of Alglave and Maranget [4,6,8], that can evaluate litmus tests w.r.t. axiomatic memory models specified in a relational-algebra style in the Cat language [2]. However, all of these previous tools for axiomatic models have (at best) used hard-coded ISA semantics that cover only small fragments of the complete architecture. For example, Zhang et al. [32] use a SMT solver based approach for SoC verification, with a user-specified memory model (TSO or SC), however the instruction level abstractions (ILAs) they use are much more abstract than the ISA semantics we consider.

In this paper we describe a tool, Isla, that integrates full-scale ISA specifications, in Sail, with arbitrary axiomatic models, in the Cat language. We first build a generic symbolic execution library for Sail specifications—which should also be valuable for other verification tasks. We use this to construct a tool for symbolically running binary litmus tests for any Sail ISA under any (non-recursive) Cat axiomatic memory model, using an SMT solver. We equip it with a web interface to make it widely accessible, and illustrate and evaluate all this for Armv8-A and RISC-V. Isla is available at https://isla-axiomatic.cl. cam.ac.uk and https://github.com/rems-project/isla. An extended version of the paper [11], available at https://www.cl.cam.ac.uk/~pes20/isla/, includes appendices showing the main parts of the full Sail/ASL semantics of a sample Armv8-A instruction (add x4, x3, #1); the Armv8-A axiomatic concurrency model (combining the official Arm specification for user concurrency [9,13] with the additions for instruction fetch semantics by Simner et al. [27]); and examples of the latter.

Our approach has several key advantages, which all follow from the fact that mainstream industry ISAs are surprisingly large and intricate. The Armv8-A ISA specification is around 100k lines. It defines the sequential behaviour of the full instruction set in all its detail, including e.g. instruction decoding, behaviour at each exception level, register banking, floating-point, vector instructions, system registers, exceptions, address translation, virtualisation, security extensions, and a host of optional architectural features. Simple litmus tests developed to investigate user concurrency have historically used only very few instructions and very little of this, and hand-written ISA models have sufficed, but even a 'simple' ADD instruction can, in reality, involve surprisingly much of the specification. If one wants to examine arbitrary compiler-generated code one needs many more instructions; and to develop systems concurrency semantics, e.g. covering the concurrency behaviour of instruction fetch, exceptions, or address translation, one might need any of the specification—and it would be exceedingly laborious and error-prone to reproduce it by hand in a hard-coded semantics. By handling the full authoritative Armv8-A ISA, we automatically support litmus tests that use arbitrary instructions, and we enable research on systems concurrency, with high confidence that the ISA follows the vendor specification. We demonstrate this by applying our tool to the model and examples for self-modifying code by Simner et al. [27], and our integration has also identified several places where the ISA specification needs modifications to correctly give the intended behaviour in a concurrent setting, e.g. to remove or enforce additional ordering. Because this is based on authoritative Arm and RISC-V ISA specifications, the work should enable relaxed-memory behaviour to be included in the standard test-edit-debug cycle used in the development of such large and critical specifications.

#### 2 Implementation

Axiomatic relaxed-memory concurrency models, being expressed as logical constraints over candidate execution graphs, lend themselves to solver-based tool implementations. For the instruction-semantics part of such a tool, the most direct approach would be to translate the ISA semantics (for the instructions that occur in a litmus test) directly into SMT and combine that with the axiomaticmodel constraints, roughly along the lines of Alglave et al. [3]. That approach was followed by Simner et al. [27], who compiled Sail directly into SMT to test an axiomatic model for instruction-fetch tests, but using a small handwritten Arm fragment, rather than the full Sail model derived from the Arm-internal model. The problem with this direct approach is one of scale: as one covers more of the Arm semantics, the resulting SMT problem simply becomes too large to be practicable. For example, for a load instruction, the virtual address must be translated into a physical address, which is a complex process with a great deal of configurability—there may be zero, one, or two stages of address translation, the page size may vary, the number of levels used in the page table may differ, etc. This approach also required the top level fetch-execute-decode loop to be handled specially, as one cannot translate such an unbounded loop directly into SMT, which imposes significant constraints on the shape of allowable tests.

In contrast, here we build and use a generic symbolic evaluation for Sail definitions using the Z3 SMT solver, which lets us compute the possible symbolic thread-local traces of each instruction, and hence of each thread (treating memory read values as unknowns, left to the concurrency model constraints). It also lets us use the same fetch-decode-execute loop that is used for emulation and co-simulation (which embodies various architecture-specific subtleties).

#### 2.1 Symbolic Execution for Sail

Sail is attractive for symbolic execution for several reasons. First, it is an intentionally simple language, lacking many of the features found in general-purpose languages. Second, it has to support very few programs, just the specifications of major ISAs, so (unlike tools for conventional programming languages) we can tune the execution to them. Third, almost all of the loops in these programs are bounded. Our starting point is the translation of Sail to C, for emulation, by Armstrong et al. [10]. This goes via a simple goto-language intermediate representation which is already well-suited for this task.

Static Function Linearisation. Our symbolic execution always creates a new task when we hit a branch, and we do not ever merge these tasks at join points. This is a good strategy for instruction semantics, as it simplifies the symbolic execution engine significantly, but it does mean some code can cause unnecessary branching. To avoid this we have a static rewrite that can take a function with if statements and rewrite it into a 'linear' form, e.g. as below:

```
var x = 2;
if undefined {
  x=x+1
} else {
  x=x+2
};
return x
                   ⇒
                              let x0 = 2;
                              let b = undefined;
                              let x1 = x0 + 1;
                              let x2 = x0 + 2;
                              let x3 = ite(b, x1, x2);
                              return x3
```
This works by translating the body of the function into SSA form, then replacing the φ-functions with if-then-else (ite) functions that translate into the SMT ite. This results in a more complex SMT expression, but less branching in the symbolic execution, so it is a trade-off, but often worthwhile.

Per-Thread Candidate Executions. For each litmus-test thread this symbolic execution will produce a number of *candidate executions*, each of which is a sequence of memory events (memory reads and writes, fences, register accesses, and so on) with the symbolic values of these events potentially being constrained by some SMT formula for the overall execution. For example, consider the Armv8-A instruction add x4, x3, #1. For this instruction, our symbolic evaluator generates an execution:

```
(declare-const input (_ BitVec 64))
(read-reg |R3| nil input)
(define-const output (bvadd input #x0000000000000001))
(write-reg |R4| nil output)
```
where the SMTLIB formula is defined by the declare-const and define-const statements, with read-reg and write-reg effects indicating which variables in the SMT formula correspond to the values read and written to registers (which are otherwise just global variables) by the instruction. We simplify here for brevity, omitting the negative, zero, carry and overflow flags that the model computes. For more complex instructions, there are additional effects for memory accesses, cache maintenance events, barriers, and so on.

#### 2.2 Checking a Litmus Test

Figure 1 shows the overall process of checking a litmus test. Tests can be supplied either in the .litmus format of previous axiomatic and operational tools [4,5, 14], reusing the parser from [4], or as a TOML file (a standard configuration file format, with libraries available for most languages). We first assemble the test with a conventional assembler into an ELF binary and load it into the representation of memory that will be used, before initialising the model with the program counter set to the entry point for each thread, then we symbolically execute the instructions in each thread separately, using the Sail semantics for each instruction, plus the same fetch-execute-decode loop in Sail we would use for emulation, to produce sets of per-thread traces as above. Treating litmus tests essentially as binaries, rather than the more-or-less ad hoc fragments of assembly abstract syntax used by earlier tools, accommodates the fact that the Armv8-A model does not define an abstract syntax, and reduces the gap between what the tool evaluates and what is run in experimental testing. Note that the Arm assembly in Fig. 1, as well as subsequent assembly snippets in this paper, use the standard Arm convention that x0 and w0 refer to the same register, where w0 refers to the lower 32-bits of the register, and x0 refers to the full 64-bit width.

We then generate an SMT problem for every combination of the candidate executions of each thread. This problem consists of the per-thread SMT formulae concatenated together (renaming variables as necessary to avoid name-clashes), combined with the axiomatic memory model (described in more detail below).

Finally, we need to generate some 'glue' SMT that connects the per-thread semantics with the memory model. For every effect in the per-thread SMT semantics we generate an enumeration of *events*, e.g. for an execution with two reads and two writes:

```
(declare-datatypes ((Event 0)) (((R1) (R2) (W1) (W2) (IW))))
```
The event IW is a special write event that represents the initial state. We generate relations such as value-of that relate events to their values as determined by the effects in the per-thread semantics, so if the second read event R2 read the value #xABCD, (value-of R2 #xABCD) would be true. We generate *syntactic dependency*

Fig. 1. Overview of process for checking the allowed executions of a litmus test

*relations* for address, data, and control dependencies, discussed in more detail in Sect. 2.3. Finally, there is a constraint on the final state of each test which specifies values expected in registers and memory after all threads have executed.

The Cat language represents axiomatic memory models as definitions of relations over the above events, and constraints over those relations, e.g. that specific relations are irreflexive, acyclic, or empty (or the negation of any of these). Relations are defined in a point-free relation-algebraic style, in terms of standard relational operators such as composition, intersection and union. The memory models we consider are all multi-copy-atomic, and all recursion in their definitions can trivially be replaced with (reflexive)-transitive closure. Herd's let rec construct computes the least solution to a set of equations [2], which is tricky to represent in SMT, so we do not support it. We believe even relations such as Power's (mutually recursive) preserved program order are nevertheless representable as SMT, so this limitation is mostly in our translation from Cat—we would likely want to use a different syntax to represent these relations for Isla.

A satisfiable solution to the overall SMT problem described above thus represents an execution permitted by the architecture. Parsing the model generated by the SMT solver allows us to generate a graph of the execution by instantiating each relation in the model with the various events. If all generated SMT problems are unsatisfiable for every combination of per-thread candidate executions then there are no permitted executions. If desired we can repeatedly ask the SMT solver for additional distinct models until we have all permitted executions.

#### 2.3 Syntactic Dependency Analysis

Axiomatic memory models for relaxed hardware architectures rely heavily on notions of address, data, and control dependencies between instructions. For example, consider the following assembly:

```
ldr w0, [x1] // load 32 bits from address in x1 into x0
 cbnz w0, LC01 // compare and branch if non-zero to LC01
LC01:
 mov w2, #1 // load 1 into x2
 str w2, [x3] // store 32 bit-value in x2 to the address in x3
```
Here there is a control dependency between the load (ldr) and the store (str), as the value read by the load is used to determine whether the branch instruction cbnz that precedes the store is taken or not. This control dependency exists irregardless of whether the branch is taken or not—its existence is purely determined by the syntactic structure of the above code.

In general, existing ISA descriptions do not cover this aspect of the architecture well, as they are principally developed only to describe the sequential behaviour. Previous tools have either hand-coded dependency information, which is acceptable for cut-down ISA models but too laborious and error-prone at the scale of the ISA models we use, or used a heavyweight taint-tracking interpreter [15]. Our approach avoids both of these. It is similar to the latter, computing dependencies from the ISA specification, but building the footprint analysis atop our symbolic execution library requires only around 500 LoC.

To express dependencies, we need to associate each event in our candidate executions with the syntactic instruction/opcode that generated them. To do this we use a Sail function , called in each architecture's fetch-decode-execute loop just after fetching an instruction; this adds a special effect to the candidate execution recording the instruction opcode. We also have another special effect that delimits each fetch-decode-execute cycle, so each effect such as read-mem and write-mem that would give rise to an event can be associated with an opcode, as well as an index in the program order relation for its thread.

For each instruction we also need to know its *footprint*: data about the instruction including which input registers it reads, which output registers it writes, whether it is a branch instruction, and so on. It also contains *taint* information—we need to know which registers writes may contain data 'tainted' by a memory read performed by a load, or which input registers 'taint' data written to memory. The Sail ISA specifications do not explicitly describe this footprint, so we are forced to derive it from the specification.

To do this we symbolically evaluate each opcode independently in a suitably unconstrained environment so as to capture all its possible behaviours. This can be computationally expensive due to the number of possible behaviours some instructions have, so we build a footprint cache to avoid re-computing this where possible. It turns out to be hard to distinguish ordinary branches from instructions that can cause an exception to occur, so we add a special branch address announce effect, created by a Sail function that we call in branch instructions. This also enables the taint tracking for branch addresses we need for control dependencies as described above. The taint tracking is achieved simply by looking at what sub-expressions in the generated SMT problem contain variables that also appear in the various effects in each trace.

Once we have this footprint information we can analyse it for the opcodes between each read and write effect and derive the necessary dependency relations over their events. Note that this dependency relation must be exact. If we underapproximate, we will allow executions that should be forbidden, and if we overapproximate we will forbid executions that should be allowed.

In some cases the current Arm-provided ISA specification does not include enough information to identify the architecturally respected dependencies, and our dependency analysis would identify a dependency when there should not be one. To solve this we add some special Sail functions that give fine-grained control of the dependency calculation. For example, in indirect branches we ignore any dependency between the target register <sup>X</sup>n and the link register X30 by including a function in the Sail definition that tells the footprint analysis to ignore any relation it finds between the two registers.

```
if branch_type == BranchType_INDCALL then {
    ignore_dependency_edge(n, 30);
    X(30) = PC() + 4
};
```
This works by adding a special annotation in the candidate execution trace which can be used by the footprint analysis—for all other purposes it is a noop. This information should properly become part of the architecture specification, as mistakes in the dependency calculations could be a source of soundness bugs. The lack of support for this information in existing ISA specifications can partly be explained by the lack of tooling to properly explore the integration of ISA specifications with concurrency, something we hope a tool such as ours can address.

#### 2.4 Web Interface

Figure 2 shows the web interface we have developed for our tool, based on the web interface for the C memory model tool Cerberus-BMC by Lau et al. [16]. This can either be run locally, or via a website, https://isla-axiomatic.cl.cam.ac. uk.

#### 3 System Litmus Tests

As mentioned previously, one advantage of our tool is that, because it supports the full sequential ISA, it enables easy experimentation with tests and models

Fig. 2. Web interface for the tool

outside the scope of previous tools, e.g. involving new systems features. For example, Simner et al. developed semantics for Arm instruction fetch and I/D cache maintenance [27]. Consider the litmus test in Fig. 3 [27, §3.3], a simple test involving self-modifying code. In order to run this test and the others in [27] our tool required only minimal changes: we had to add support for data-cache and instruction-cache maintenance events and relations for them in our Cat to SMT translation. Additionally we needed to generalise how we generated the rf (reads-from) relation to generate both the regular rf relation and the new irf (instruction-reads-from) relation. Because our tool already runs tests using a fetch-execute-decode loop, all the instruction fetch events were already available—we in fact filter them out when running user-mode tests.

When generating candidate executions for a thread we normally do not assume anything about what other threads may be doing, but for self-modifying code this would clearly be problematic, as it would imply that any other thread could modify any of this thread's instructions arbitrarily. We therefore mark the memory locations that contain instructions that can be modified and provide in advance all the possible values they might take.

#### 4 Results and Comparisons

We evaluate our tool for correctness and performance with respect to Herd using previous corpora of tests.


Fig. 3. Self-modifying code litmus test SM+cachesync-isb

We select 3798 litmus tests for both Armv8-A and RISC-V to compare between our tool and Herd—these tests include a representative set of features such as barriers and atomics, while exercising all of the basic litmus test shapes. All tests were run on a 2.6GHz Intel Xeon Gold 6240 CPU with 36 physical cores and 400GB of RAM. The tests are split into rough categories based on the contents of the tests. We ran 36 concurrent instances of both our tool and Herd across each set of tests, running Herd with the -speedcheck fast flag which causes it to stop enumerating executions when it resolves the final assertion in each test, which is the closest behaviour to how our tool behaves by default.

To assess correctness, we use a set of golden references for these above tests, for all of which the previous operational RMEM [14] and axiomatic Herd models and tools agree, and which have been extensively validated against hardware implementations. We confirm that our tool produces the same expected results as those models for all the litmus tests, including when run in exhaustive mode.

To assess performance, the table below gives the total real execution time for each batch of tests.


In general Herd is faster for nearly all tests, but this is not surprising given the amount of detail in the full-scale instruction semantics that we are using, particularly for Armv8-A. Our goal is not to be faster, but to support those full-scale ISA semantics while remaining fast enough for practical purposes. We achieve this: most tests take only a second or so to run, which is perfectly usable interactively. For example, given the Armv8-A basic 3-thread tests, for a single sequential run of the tests, the shortest took 872 ms to run, while the longest took 1231 ms. The above batch times are similarly perfectly usable for (e.g.) regression testing while editing a model.

We also evaluate our tool with respect to that of Simner et al., for the instruction-fetch tests (which are currently not supported by Herd) in Sect. 6 of their paper. Our tool returns the expected results for all these tests, including the two tests (FOW and SM.F+ic) that were unsupported by their tool. In terms of performance, we note that their tool took 30 min to run just 90 of the 1377 basic 2-thread tests above, which is awkwardly slow for using a tool in practice, whereas when limiting our tool to 8 cores (to more closely match their experimental setup) our tool will execute all 1377 in under 3 min. We were additionally able to provide further validation that the Simner et al. model behaves as the standard Armv8-A model for non-self-modifying tests by showing that it behaves identically for all 3798 of the non-self-modifying tests above.

Acknowledgement. This work was partially supported by the UK Government Industrial Strategy Challenge Fund (ISCF) under the Digital Security by Design (DSbD) Programme, to deliver a DSbDtech enabled digital platform (grant 105694), ERC AdG 789108 ELVER, EPSRC programme grant EP/K008528/1 REMS, an Arm iCASE award, Arm, and Google. Approved for public release; distribution is unlimited. This work was supported by the Defense Advanced Research Projects Agency (DARPA) and the Air Force Research Laboratory (AFRL), under contract FA8650- 18-C-7809 ("CIFV"). The views, opinions, and/or findings contained in this report are those of the authors and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.

#### References


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Summing up Smart Transitions**

Neta Elad<sup>1</sup> , Sophie Rain2(B) , Neil Immerman<sup>3</sup> , Laura Kov´acs<sup>2</sup> , and Mooly Sagiv<sup>1</sup>

> <sup>1</sup> Tel Aviv University, Tel Aviv, Israel <sup>2</sup> TU Wien, Vienna, Austria <sup>3</sup> UMass Amherst, Amherst, USA

**Abstract.** Some of the most significant high-level properties of currencies are the sums of certain account balances. Properties of such sums can ensure the integrity of currencies and transactions. For example, the sum of balances should not be changed by a transfer operation. Currencies manipulated by code present a verification challenge to mathematically prove their integrity by reasoning about computer programs that operate over them, e.g., in Solidity. The ability to reason about sums is essential: even the simplest ERC-20 token standard of the Ethereum community provides a way to access the total supply of balances.

Unfortunately, reasoning about code written against this interface is non-trivial: the number of addresses is unbounded, and establishing global invariants like the preservation of the sum of the balances by operations like transfer requires higher-order reasoning. In particular, automated reasoners do not provide ways to specify summations of arbitrary length.

In this paper, we present a generalization of first-order logic which can express the unbounded sum of balances. We prove the decidablity of one of our extensions and the undecidability of a slightly richer one. We introduce first-order encodings to automate reasoning over software transitions with summations. We demonstrate the applicability of our results by using SMT solvers and first-order provers for validating the correctness of common transitions in smart contracts.

#### **1 Introduction**

A basic challenge in smart contract verification is how to express the functional correctness of transactions, such as currency minting or transferring between accounts. Typically, the correctness of such a transaction can be verified by proving that the transaction leaves the sum of certain account balances unchanged.

Consider for example the task of minting an unbounded number of tokens in the simplified ERC-20 token standard of the Ethereum community [32], as illustrated in Fig. 1<sup>1</sup>. This example deposits the minted amount (n) into the receiver's address (a) and we need to ensure that the mint operation *only* changed the bal-

<sup>1</sup> The old- prefix denotes the value of a function before the mint transition, and the new- prefix denotes the value afterwards.

c The Author(s) 2021

A. Silva and K. R. M. Leino (Eds.) CAV 2021, LNCS 12759, pp. 317–340, 2021. https://doi.org/10.1007/978-3-030-81685-8\_15


**Fig. 1.** Minting n tokens in ERC-20.

ance of the receiver. To do so, in addition to (i) proving that the balance of the receiver has been increased by n, we also need to verify that (ii) the account balance of every user address a different than a has not been changed during the mint operation and that (iii) the sum of all balances changed exactly by the amount that was minted. The validity of these three requirements (i)-(iii), formulated as the post-conditions of Fig. 1, imply its functional correctness.

Surprisingly, proving formulas similar to the post-conditions of Fig. 1 is challenging for state-of-the-art automated reasoners, such as SMT solvers [6,7,9] and first-order provers [11,19,34]: it requires reasoning that links local changes of the receiver (a) with a global state capturing the sum of all balances, as well as constructing that global state as an aggregate of an unbounded but finite number of Address balances. Moreover, our encoding of the problem uses discrete coins that are minted and deposited, whose number is unbounded but finite as well.

In this paper we address verification challenges of software transactions with aggregate properties, such as preservation of sums by transitions that manipulate low-level, individual entities. Such properties are best expressed in higher-order logic, hindering the use of existing automated reasoners for proving them. To overcome such a reasoning limitation, we introduce *Sum Logic* (SL) as a generalization of first-order logic, in particular of Presburger arithmetic. Previous works [12,21,31] have also introduced extensions of first-order logic with aggregates by counting quantifiers or generalized quantifiers. In Sum Logic (SL) we only consider the special case of integer sums over uninterpreted functions, allowing us to formalize SL properties with and about unbounded sums, in particular sums of account balances, without higher-order operations (Sect. 3). We prove the decidability of one of our SL extensions and the undecidability of a slightly richer one (Sect. 4). Given previous results [21], our undecidability result is not surprising. In contrast, what may be unexpected is our decidability result and the fact that we can use our first-order fragment for a convenient and practical new way to verify the correctness of smart contracts.

We further introduce first-order encodings which enable automated reasoning over software transactions with summations in SL (Sect. 5). Unlike [5], where SMT-specific extensions supporting higher-order reasoning have been introduced, the logical encodings we propose allow one to use existing reasoners without any modification. We are not restricted to SMT reasoning, but can also leverage generic automated reasoners, such as first-order theorem provers, supporting first-order logic. We believe our results ease applying automated reasoning to smart contract verification even for non-experts.

We demonstrate the practical applicability of our results by using SMT solvers and first-order provers for validating the correctness of common financial transitions appearing in *smart contracts* (Sect. 6). We refer to these transitions as *smart transitions*. We encode SL into pure first-order logic by adding another sort that represents the tokens of the crypto-currency themselves (which we dub "coins").

Although the encodings of Sect. 5 do not translate to our decidable SL fragment from Sect. 4, our experimental results show that automated reasoning engines can handle them consistently and fast. The decidability results of Sect. 5 set the boundaries for what one can expect to achieve, while our experiments from Sect. 5 demonstrate that the unknown middle-ground can still be automated.

While our work is mainly motivated by smart contract verification, our results can be used for arbitrary software transactions implementing sum/aggregate properties. Further, when compared to the smart contract verification framework of [33], we note that we are not restricted to proving the correctness of smart contracts as finite-state machines, but can deal with semantic properties expressing financial transactions in smart contracts, such as currency minting/ transfers.

While ghost variable approaches [14] can reason about changes to the global state (the sum), our approach allows the verifier to specify only the local changes and automatically prove the impact on the global state.

*Contributions.* In summary, this paper makes the following contributions:


#### **2 Preliminaries**

We consider many-sorted first-order logic (FOL) with equality, defined in the standard way. The equality symbol is denoted by ≈.

We denote by STRUCT [Σ] the *set of all structures* for the vocabulary Σ. A structure A ∈ STRUCT [Σ] is a pair (D, I), where for each sort *s*, its domain in A is D(*s*), and for each symbol S, its interpretation in A is I(S). Note that *models* of a formula ϕ over a vocabulary Σ are structures A ∈ STRUCT [Σ].

A *first-order theory* is a set of first-order formulas closed under logical consequence. We will consider, the first-order theory of the natural numbers with addition. This is Presburger arithmetic (PA) which is of course decidable [27]. We write <sup>N</sup> to denote the set of natural numbers. We consider 0 <sup>∈</sup> <sup>N</sup> and write N<sup>+</sup> to explicitly exclude 0 from N. The vocabulary of PA is ΣPresburger = - 0, 1, c1,...,cl, +<sup>2</sup> , with all constants 0, 1, c<sup>i</sup> of sort Nat. A structure A = (D, I) ∈ STRUCT [ΣPresburger] is called a *Standard Model of Arithmetic* when <sup>D</sup>(Nat) = <sup>N</sup> and +<sup>2</sup> is interpreted as the standard binary addition + function over the naturals. The vocabulary ΣPresburger can be extended with a total order relation, yielding Σ<sup>∗</sup> Presburger = - <sup>0</sup>, <sup>1</sup>, <sup>+</sup><sup>2</sup>, <sup>≤</sup><sup>2</sup> , where <sup>≤</sup><sup>2</sup> is interpreted as the binary relation ≤ in Standard Models of Arithmetic.

#### **3 Sum Logic (SL)**

We now define *Sum Logic (* SL*)* as a generalization of Presburger arithmetic, extending Presburger arithmetic with unbounded sums. SL is motivated by applications of financial transactions over cryptocurrencies in smart contracts. Smart contracts are decentralized computer programs executed on a blockchainbased system, as explained in [28]. Among other tasks, they automate financial transactions such as transferring and minting money. We refer to these transactions as *smart transitions*. The aim of this paper and SL in particular is to express and reason about the post-conditions of smart transitions similar to Fig. 1.

SL expresses smart transition relations among sums of accounts of various kinds, e.g., at different banks, times, etc. Each such kind, j, is modeled by an uninterpreted function symbol, b<sup>j</sup> , where b<sup>j</sup> (a) denotes the balance of a's account of kind j, and a constant symbol s<sup>j</sup> , which denotes the sum of all outputs of b<sup>j</sup> . As such, our SL generalizes Presburger arithmetic with (i) a sort Address corresponding to the (unbounded) set of account *addresses*; (ii) *balance* functions b<sup>j</sup> mapping account addresses from Address to account values of sort Nat; and (iii) *sum constants* s<sup>j</sup> of sort Nat capturing the total sum of all account balances represented by b<sup>j</sup> . Formally, the vocabulary of SL is defined as follows.

#### **Definition 1 (**SL **Vocabulary).** *Let*

$$
\Sigma\_{+,\leq}^{l,m,d} = \left( a\_1, \dots, a\_l, b\_1^1, \dots, b\_m^1, c\_1, \dots, c\_d, s\_1, \dots, s\_m, 0, 1, +^2, \leq^2 \right)
$$

*be a* sorted first-order vocabulary of SL *over sorts* {Address, Nat}*, where*



**Table 1.** ERC-20 token standard


*–* <sup>≤</sup><sup>2</sup> *is a binary relation over* Nat <sup>×</sup> Nat*.*

In what follows, when the cardinalities in an SL vocabulary are clear from context, we simply write Σ instead of Σl,m,d <sup>+</sup>,<sup>≤</sup> . Further, by <sup>Σ</sup>l,m,d ✚+, ✚<sup>≤</sup> we denote the sub-vocabulary where the crossed-out symbols are not available. Note that even when addition is not available, we still allow writing numerals larger than 1.

We restrict ourselves to *universal sentences* over an SL vocabulary, with quantification only over the Address sort.

We now extend the Tarskian semantics of first-order logic to ensure that the sum constants of an SL vocabulary (s1,...,sm) are equal to the sum of outputs of their associated balance functions (b<sup>j</sup> for each s<sup>j</sup> ) over the respective entire domains of sort Address.

Let Σ be an SL vocabulary. An SL structure A = (D, I) ∈ STRUCT [Σ] representing a model for an SL formula ϕ is called an SL *model* iff

$$\mathcal{Z}(s\_j) = \sum\_{a \in \mathcal{D}(\mathsf{Iddraus})} \left[ \mathcal{Z}(b\_j) \right](a), \quad \text{for each } 1 \le j \le m. \tag{\text{Sum Property}}$$

We write A -SL ϕ to mean that A is an SL model of ϕ. When it is clear from context, we simply write A ϕ.

*Example 1 (Encoding ERC-20 in* SL*).* As a use case of SL, we showcase the encoding of the ERC-20 token standard of the Ethereum community [32] in SL. To this end, we consider an SL vocabulary Σl,2,d. We respectively denote the balance functions and their associated sums as b, b , s, s in the SL structure over Σl,2,d. The resulting instance of SL can then be used to encode ERC-20 operations/smart transitions as SL formulas, as shown in Table 1. Using this encoding, the post-condition of Fig. 1 is expressed as the SL formula

$$b'(a) \approx b(a) + n \land \forall a' \not\approx a.b'(a') \approx b(a') \land s' \approx s + n \tag{1}$$

formalizing the correctness of the smart transition of minting n tokens in Fig. 1. In the applied verification examples in Sect. 6, rather than verifying the low-level implementation of built-in functions such as mintn, we assume their correctness by including suitable axioms.

#### **4 Decidability of SL**

We consider the decidability problem of verifying formulas in SL. We show that when there are several function symbols b<sup>j</sup> to sum over, the satisfiability problem for SL becomes undecidable<sup>2</sup>. We first present, however, a useful decidable fragment of SL.

#### **4.1 A Decidable Fragment of SL**

We prove decidability for a fragment of SL, which we call the (l, 1, d)-FRAG fragment of SL (Theorem 4). For doing so, we reduce the fragment to Presburger arithmetic, by using regular Presburger constructs to encode SL extensions, that is the uninterpreted functions and sum constants of SL.

The first step of our reduction proof is to consider distinct models, which are models where the Address constants a<sup>i</sup> represent distinct elements in the domain D(Address). While this restriction is somewhat unnatural, we show that for each vocabulary and formula that has a model, there exists an equisatisfiable formula over a different vocabulary that has a *distinct* model (Theorem 1). The crux of our decidability proof is then proving that (l, 1, d)-FRAG has *small* Address *space*: given a formula ϕ, if it is satisfiable, then there exists a model where |D(Address)| ≤ κ(|ϕ|), |ϕ| is the length of ϕ, and κ(.) is some computable function (Theorem 3)<sup>3</sup>.

**Distinct Models.** An SL structure A is considered *distinct* when the l Address constants represent l distinct elements in D(Address). I.e.,

$$\left| \{ \mathcal{T}(a\_1), \dots, \mathcal{T}(a\_l) \} \right| = l \dots$$

Since each SL model induces an equivalence relation over the Address constants, we consider partitions P over {a1,...,al}. For each possible partition P we define a transformation of terms and formulas T<sup>P</sup> that substitutes equivalent Address constants with a single Address constant. The resulting formulas are defined over a vocabulary that has |P| Address constants. We show that given an SL formula ϕ, if ϕ has a model, we can always find a partition P such that each of its classes corresponds to an equivalence class induced by that model.

**Theorem 1 (Distinct Models).** *Let* ϕ *be an* SL *formula over* Σ*, then* ϕ *has a model iff there exists a partition* P *of* {a1,...,al} *such that* T<sup>P</sup> (ϕ) *has a* distinct *model.* 

**Small** Address **Space.** In order to construct a reduction to Presburger arithmetic, we bound the size of the Address sort. For a fragment of SL to be decidable, we therefore need a way to bound its models upfront. We formalize this requirement as follows.

<sup>2</sup> Proofs of our results are given in the appendix of [10].

<sup>3</sup> The function κ(.) is defined per decidable fragment of SL, and not per formula.

**Definition 2 (Small** Address **Space).** *Let* FRAG *be some fragment of SL over vocabulary* Σ = Σl,m,d <sup>+</sup>,<sup>≤</sup> *.* FRAG *is said to have* small Address space *if there exists a computable function* κΣ(.)*, such that for any* SL *formula* ϕ ∈ FRAG*,* ϕ *has a distinct model iff* ϕ *has a distinct model* A = (D, I) *with* small Address space*, where* |D(Address)| ≤ κΣ(|ϕ|)*.*

*We call* κΣ(.) *the* bound function *of* FRAG*; when the vocabulary is clear from context we simply write* κ(.)*.*

One instance of a fragment (or rather, family of fragments) that satisfies this property is the (l, 1, d)-FRAG fragment: the simple case of a *single* uninterpreted "balance" function (and its associated sum constant), further restricted by removing the binary function + and the binary relation ≤. Therefore, we derive the following theorem:

#### **Theorem 2 (Small** Address **Space of** (l, 1, d)*-*FRAG**).**

*For any* l*,* d*, it holds* (l, 1, d)*-*FRAG*, the fragment of* SL *formulas over the* SL *vocabulary*

$$
\Sigma^{l,1,d}\_{\nearrow \underline{k}} = \left( a\_1, \dots, a\_l, b^1, c\_1, \dots, c\_d, s, 0, 1 \right) \; ,
$$

*has* small Address space *with bound function* κ(x) = l + x + 1*.* 

An attempt to trivially extend Theorem 2 for a fragment of SL with two balance functions falls apart in a few places, but most importantly when comparing balances to the sum of a different balance function. In Sect. 4.2 we show that these comparisons are essential for proving our undecidability result in SL.

**Presburger Reduction.** For showing decidability of some FRAG fragment of SL, we describe a Turing reduction to pure Presburger arithmetic. We introduce a transformation τ (.) of formulas in SL into formulas in Presburger arithmetic. It maps universal quantifiers to disjunctions, and sums to explicit addition of all balances. In addition, we define an auxiliary formula η(ϕ), which ensures only valid addresses are considered, and that invalid addresses have zero balances. The formal definitions of τ (.) and η(ϕ) can be found in [10].

By relying on the properties of *distinctness* and *small* Address *space* we get the following results.

**Theorem 3 (Presburger Reduction).** *An* SL *formula* ϕ *has a* distinct*,* SL *model with small* Address *space iff* τ (ϕ) ∧ η(ϕ) *has a Standard Model of Arithmetic.* 

**Theorem 4 (**SL **Decidability).** *Let* FRAG *be a fragment of* SL *that has* small Address space*, as defined in Definition 2. Then,* FRAG *is decidable.*

*Proof (Theorem* 4*).* Let ϕ be a formula in FRAG. Then ϕ has an SL model iff for some partition P of {a1,...,al}, T<sup>P</sup> (ϕ) has a *distinct* SL model. For any P, the formula T<sup>P</sup> (ϕ) is in FRAG, therefore T<sup>P</sup> (ϕ) has a *distinct* SL model iff it has a *distinct* SL model with *small* Address *space*.

From Theorem 3, we get that for any P, ϕ<sup>P</sup> T<sup>P</sup> (ϕ) has a *distinct* SL model iff τ (ϕ<sup>P</sup> )∧η(ϕ<sup>P</sup> ) has a Standard Model of Arithmetic. By using the PA decision procedure as an oracle, we obtain the following *decision procedure for a* FRAG *formula* ϕ:


*Remark 1.* Our decision procedure for Theorem 4 requires B<sup>l</sup> Presburger queries, where B<sup>l</sup> is Bell's number for all possible partitions of a set of size l.

Using Theorem 4 and Theorem 2, we then obtain the following result.

**Corollary 1.** (l, 1, d)*-*FRAG *is decidable.* 

#### **4.2 SLUndecidability**

We now show that simple extensions of our decidable (l, 1, d)-FRAG fragment lose its decidability (Theorem 5). For doing so, we encode the halting problem of a two-counter machine using SL with 3 balance functions, thereby proving that the resulting SL fragment is undecidable.

Consider a two-counter machine, whose transitions are encoded by the Presburger formula π(c1, c2, p, c 1, c 2, p ) with 6 free variables: 2 for each of the three registers, one of which being the program counter (pc). We assume w.l.o.g. that all three registers are within N<sup>+</sup>, allowing us to use addresses with a zero balance as a special "separator". In addition, we assume that the program counter is 1 at the start of the execution, and that there exists a single halting statement at line H. That is, the two-counter machine halts iff the pc is equal to H.

**Reduction Setting.** We have 4 Address elements for each time-step, 3 of them hold one register each, and one is used to separate between each group of Address elements (see Table 2). We have 3 uninterpreted functions from Address to Nat ("balances"). For readability we denote these functions as c, l, g (instead of b1, b2, b3) and their respective sums as sc, sl, sg:



**Table 2.** Transition system of a 2-counter machine, array view.

Each group representing a time-step is a 4 Address element, ordered as follows:


In addition we have 2 Address constants, a<sup>0</sup> and a<sup>1</sup> which represent the pc value at the start and at the end of the execution. The element a<sup>1</sup> also holds the maximal value of l, that is, l(a1)+1 ≈ sc. Further, a<sup>0</sup> holds the fourthminimal value, since its the last element of the first group, and each group has four elements.

**Formalization Using a Two-Counter Machine.** We now formalize our reduction, proving undecidability of SL.

(i) We impose an injective labeling

$$
\varphi\_1 = \forall x, y. \left( l(x) \approx l(y) \right) \to \left( x \approx y \right),
$$

(ii) We next formalize properties over the program counter pc. The Address constant that represents the program counter pc value of the last time-step is set to have the maximal labeling, that is

$$
\varphi\_2 = \forall x. l(x) \le l(a\_1).
$$

Further, the Address constant that represents the pc value of the first time-step has the fourth labeling, hence

$$\varphi\_3 = l(a\_0) \approx 3$$

Finally, the first and last values of the program counter are respectively 1 and H, that is

$$\varphi\_4 = g(a\_0) \approx 1 \land g(a\_1) \approx H$$

(iii) We express *cardinality constraints* ensuring that there are as many Address elements as the labeling of the last Address constant (a1) + 1. We assert

$$
\varphi\_5 = (s\_c \approx l(a\_1) + 1) \land \forall x. \,(c(x) \approx 1),
$$

(iv) We encode the transitions of the two-counter machine, as follows. For every 8 Address elements, if they represent two sequential time-steps, then the formula for the transitions of the two-counter machine is valid for the registers it holds. As such, we have

$$\begin{aligned} \varphi\_6 &= \forall x\_1, \dots, x\_8. \left( F1 \land F2 \land F3 \right) \\ &\to \pi \left( g(x\_2), g(x\_3), g(x\_4), g(x\_6), g(x\_7), g(x\_8) \right) \end{aligned}$$

where the conjunction F1∧F2∧F3 expresses that x1,...,x<sup>8</sup> are two sequential time-steps, with F1, F2 and F3 defined as below. In particular, F1, F2 and F3 formalize that x1,...,x<sup>8</sup> have sequential labeling, starting with one zerovalued Address element ("separator") and continuing with 3 non-zero elements, as follows:

– Sequential:

$$l(x\_2) \approx l(x\_1) + 1 \land \dots \land l(x\_8) \approx l(x\_7) + 1\tag{F1}$$

– Time-steps:

$$g(x\_1) \approx 0 \land g(x\_2) > 0 \land g(x\_3) > 0 \land g(x\_4) > 0 \,\, , \tag{F2}$$

$$g(x\_5) \approx 0 \land g(x\_6) > 0 \land g(x\_7) > 0 \land g(x\_8) > 0 \tag{F3}$$

Based on the above formalization, the formula ϕ = ϕ<sup>1</sup> ∧···∧ϕ<sup>6</sup> is satisfiable iff the two-counter machine halts within a finite amount of time-steps (and the exact amount would be given by <sup>s</sup>*<sup>c</sup>* <sup>4</sup> ). Since the halting problem for two-counter machines is undecidable, our SL, already with 3 uninterpreted functions and their associated sums, is also undecidable.

**Theorem 5.** *For any* <sup>l</sup> <sup>≥</sup> <sup>2</sup>, m <sup>≥</sup> <sup>3</sup> *and* <sup>d</sup>*, any fragment of* SL *over* <sup>Σ</sup>l,m,d <sup>+</sup>,<sup>≤</sup> *is undecidable.* 

*Remark 2.* Note that in the above formalization the only use of associated sums comes from expressing the size of the set of Address elements. As for our uninterpreted function c(.) we have ∀x.c(x) ≈ 1, its sum s<sup>c</sup> is thus the amount of addresses. Hence, we can encode the halting problem for two-counter machines in an almost identical way to the encoding presented here, using a generalization of PA with two uninterpreted functions for l(.) and g(.), and a *size operation* replacing c(.) and its associated sum.

#### **5 SLEncodings of Smart Transitions**

The definition of SL models in Sects. 3 and 4 ensured that the summation constants s<sup>j</sup> were respectively equal to the actual summation of all balances b<sup>j</sup> (.). In this section, we address the challenge to formalize relations between s<sup>j</sup> and b<sup>j</sup> (.) in a way that the resulting encodings can be expressed in the logical frameworks of automated reasoners, in particular of SMT solvers and first-order theorem provers.

In what follows, we consider a single transaction or one time-step of multiple transactions over s<sup>j</sup> , b<sup>j</sup> (.). We refer to such transitions as *smart transitions*. Smart transitions are common in smart contracts, expressing for example the minting and/or transferring of some coins, as evidenced in Fig. 1 and discussed later.

Based on Sect. 3, our smart transitions are encoded in the Σl,2,d fragment of SL. Note however, that neither decidability nor undecidability of this fragment is implied by Theorem 4, nor Theorem 5. In this section, we show that our SL encoding of smart transitions is expressible in first-order logic. We first introduce a sound, *implicit* SL *encoding*, by "hiding" away sum semantics and using invariant relations over smart transitions (Sect. 5.1). This encoding does not allow us to directly assert the values of any balance or sum, but we can prove that this implicit encoding is complete, relative to a translation function (Sect. 5.2).

By further restricting our implicit SL encoding to this relative complete setting, we consider counting properties to explicitly reason with balances and directly express verification conditions with unbounded sums on s<sup>j</sup> and b<sup>j</sup> (.). This is shown in Sect. 5.3, and we evaluate different variants of the *explicit* SL *encoding* in Sect. 6, showcasing their practical use and relevance within automated reasoning.

To directly present our SL encodings and results in the smart contract domain, in what follows we rely on the notation of Table 1. As such, we respectively denote b, b by old-bal, new-bal and write old-sum, new-sum for s, s . As already discussed in Fig. 1, the prefixes old- and new- refer to the entire state expressed in the encoding before and after the smart transition. We explicitly indicate this state using old-world, new-world respectively. The non-prefixed versions bal and sum are stand-ins for *both* the old- and new- versions—Fig. 2 illustrates our setting for the smart transition of minting one coin.

With this SL notation at hand, we are thus interested in finding first-order formulas that verify smart transition relations between old-sum and new-sum, given the relation between old-bal and new-bal. In this paper, we mainly focus on the smart transitions of minting and transferring money, yet our results could be used in the context of other financial transactions/software transitions over unbounded sums.

*Example 2.* In the case of minting n coins in Fig. 1, we require formulas that (a) describe the state before the transition (the old-world, thus pre-condition), (b) formalize the transition (the relation between old-bal and new-bal; (i)-(ii) in

**Fig. 2.** Implicit SL encoding of mint1, where Addr is short for Address.

Fig. 1) and (c) imply the consequences for the new-world ((iii) in Fig. 1). These formulas verify that minting and depositing n coins into some address result in an increase of the sum by n, that is new-sum = old-sum+n, as expressed in the functional correctness formula (1) of Fig. 1.

#### **5.1 SLEncoding Using Implicit Balances and Sums**

The first encoding we present is a set of first-order formulas with equality over sorts {Coin, Address}. No additional theories are considered. The Coin sort represents money, where one coin is one unit of money. The Address sort represents the account addresses as before. As a consequence, balance functions and sum constants only exist implicitly in this encoding. As such, the property sum = <sup>a</sup>∈Address bal(a) *cannot be directly expressed in this encoding*. Instead, we formalize this property by using so-called *smart invariant* relations between two predicates has-coin and active over coins c ∈ Coin and a ∈ Address, as follows.

**Definition 3 (Smart Invariants).** *Let* has-coin ⊆ Address × Coin *and consider* active ⊆ Coin*. A* smart invariant *of the pair* (has-coin, active) *is the conjunction of the following three formulas*

*1. Only active coins* c *can be owned by an address* a*:*

$$\forall c: \mathsf{Coin}. \exists a: \mathsf{Add} \mathsf{ress}. \mathsf{has} \mathsf{-coin}(a, c) \to \mathsf{active}(c) \,. \tag{11}$$

*2. Every active coin* c *belongs to some address* a*:*

$$\forall c: \mathsf{Coin}. \mathsf{a} \mathsf{c:tive}(c) \to \exists a: \mathsf{Add} \mathsf{r} \mathsf{s} \mathsf{s}. \mathsf{has} \mathsf{-coin}(a, c) \,. \tag{\text{I2}}$$

*3. Every coin* c *belongs to at most one address* a*:*

$$\forall c: \mathsf{Coin}. \forall a, a': \mathsf{Add} \mathsf{ress}.$$

$$\left(\mathsf{has-coin}(a, c) \land \mathsf{has-coin}(a', c) \to a \approx a'\right). \tag{\text{I3}}$$

*We write* inv(has-coin, active) *to denote the smart invariant (I1)*∧*(I2)*∧*(I3) of* (has-coin, active)*.*

Intuitively, our *smart invariants* ensure that a coin c is *active* iff it is *owned* by precisely one address a. Our smart invariants imply the soundness of our implicit SL encoding, as follows.

**Theorem 6 (Soundness of** SL **Encoding).** *Given that* sum = |active| *and for every* a ∈ Address *it holds* bal(a) = |{c ∈ Coin | (a, c) ∈ has-coin}|*, then* inv(has-coin, active) =⇒ sum = <sup>a</sup>∈Address bal(a)*.* 

We say that a *smart transition preserves smart invariants*, when

inv(old-has-coin, old-active) ⇐⇒ inv(new-has-coin, new-active),

where old-has-coin, old-active and new-has-coin, new-active respectively denote the functions has-coin, active in the states before and after the smart transition. Based on the soundness of our implicit SL encoding, we formalize smart transitions preserving smart invariants as first-order formulas. We only discuss smart transitions implementing minting n coins here, but other transitions, such as transferring coins, can be handled in a similar manner. We first focus on miniting a single coin, as follows.

**Definition 4 (Transition** mint1(a, c)**).** *Let there be* c ∈ Coin, a ∈ Address*. The transition* mint1(a, c) *activates coin* c *and deposits it into address* a*.*

*1. The coin* c *was inactive before and is active now:*

¬old-active(c) ∧ new-active(c) . (M1)

*2. The address* a *owns the new coin* c*:*

$$\mathsf{new}\mathsf{-}\mathsf{has}\mathsf{-}\mathsf{co}\mathsf{in}(a,c)\land\forall a':\mathsf{Add}\mathsf{pres}\mathsf{s}.\ \mathsf{ \neg \mathsf{old}\mathsf{-}\mathsf{has}\mathsf{-}\mathsf{co}\mathsf{in}(a',c)\ .\tag{\mathsf{M2}}$$

*3. Everything else stays the same:*

$$\forall c': \mathsf{Coin}. c' \not\not\models c \rightarrow (\mathsf{new-active}(c') \leftrightarrow \mathsf{old-active}(c'))\,,\tag{\text{M3}}$$

∀c : Coin. ∀a : Address. (c ≈ c ∨ a ≈ a) → (M4)

(new-has-coin(a , c ) ↔ old-has-coin(a , c )) .

*The transition* mint1(a, c) *is defined as (M1)* ∧ *(M2)* ∧ *(M3)* ∧ *(M4).*

By minting one coin, the balance of precisely one address, that is of the receiver's address, increases by one, whereas all other balances remain unchanged. Thus, the expected impact on the sum of account balances is also increased by one, as illustrated in Fig. 2. The following theorem proves that the definition of mint<sup>1</sup> is *sound*. That is, mint<sup>1</sup> affects the implicit balances and sums as expected and hence mint<sup>1</sup> preserves smart invariants.

**Theorem 7 (Soundness of** mint1(a, c)**).** *Let* c ∈ Coin*,* a ∈ Address *such that* mint1(a, c)*. Consider balance functions* old-bal*,* new-bal : Address <sup>→</sup> <sup>N</sup>*, non-negative integer constants* old-sum*,* new-sum*, unary predicates* old-active*,* new-active ⊆ Coin *and binary predicates* old-has-coin*,* new-has-coin ⊆ Address × Coin *such that*


*and for every address* a *, we have*

$$\begin{aligned} \mathtt{o1d-ba1}(a') &= |\{c' \in \mathtt{coin} \mid (a', c') \in \mathtt{o1d-has-coin}\}| \ , \\ \mathtt{new-ba1}(a') &= |\{c' \in \mathtt{coin} \mid (a', c') \in \mathtt{new-has-coin}\}| \ . \end{aligned}$$

*Then,* new-sum = old-sum+ 1*,* new-bal(a) = old-bal(a)+1*. Moreover, for all other addresses* a = a*, it holds* new-bal(a ) = old-bal(a )*.* 

Smart transitions minting an arbitrary number of n coins, as in our Fig. 1, is then realized by repeating the mint<sup>1</sup> transition n times. Based on the soundness of mint1, ensuring that mint<sup>1</sup> preserves smart invariants, we conclude by induction that n repetitions of mint1, that is *minting* n *coins, also preserves smart invariants.* The precise definition of mint<sup>n</sup> together with the soundness result is stated in [10].

#### **5.2 Completeness Relative to a Translation Function**

Smart invariants provide sufficient conditions for ensuring soundness of our SL encodings (Theorem 6). We next show that, under additional constraints, smart invariants are also necessary conditions, establishing thus *(relative) completeness of our encodings.*

A straightforward extension of Theorem 6 however does not hold. Namely, only under the assumptions of Theorem 6, the following formula is not valid:

$$\texttt{sum} = \sum\_{a \in \texttt{Add} \texttt{read}} \texttt{ba1}(a) \quad \Longleftrightarrow \quad \texttt{inv}(\texttt{has-coin}, \texttt{active}) \,.$$

As a counterexample, assume (i) sum = |active|, (ii) for every a ∈ Address it holds that bal(a) = |{c ∈ Coin | (a, c) ∈ has-coin}|, that is the assumptions of Theorem 6. Further, let (iii) the smart invariants inv(has-coin, active) hold for all but the coins c1, c<sup>2</sup> ∈ Coin and all but the addresses a1, a<sup>2</sup> ∈ Address. We also assume that (iv) c<sup>1</sup> is active but not owned by any address and (v) c<sup>2</sup>

is active and owned by the two distinct addresses a1, a2. We thus have sum = <sup>a</sup>∈Address bal(a), yet inv(has-coin, active) does not hold.

To ensure completeness of our encodings, we therefore introduce a translation function <sup>f</sup> that restricts the set <sup>F</sup> <sup>2</sup>Address×Coin <sup>×</sup> <sup>2</sup>Coin of (has-coin, active) pairs, as follows. We exclude from F those pairs (has-coin, active) that violate smart invariants by both (i) not satisfying (I2), as (I2) ensures that there are not too many active coins, and by (ii) not satisfying at least one of (I1) and (I3), as (I1) and (I3) ensure that there are not too few active coins. The required translation function f (as in [10]) now assigns every pair (bal, sum) the set of all (has-coin, active) ∈ F that satisfy sum = |active|, bal(a) = |{c ∈ Coin | has-coin(a, c)}| for every address a and have not been excluded.

**Theorem 8 (Relative Completeness of** SL **Encoding).** *Let* (bal, sum) ∈ <sup>N</sup>Address <sup>×</sup> <sup>N</sup> *and let* (has-coin, active) <sup>∈</sup> <sup>f</sup>(bal, sum) *be arbitrary. Then,*

$$\mathsf{sum} = \sum\_{a \in \mathsf{Add} \mathsf{read} \mathsf{es}} \mathsf{ba1}(a) \quad \Longleftrightarrow \quad \mathsf{inv}\left(\mathsf{has-coin}, \mathsf{active}\right) \,.$$

 

#### **5.3 SLEncodings Using Explicit Balances and Sums**

We now restrict our SL encoding from Sect. 5.1 to explicitly reason with balance functions during smart transitions. We do so by expressing our translation function f from Sect. 5.2 in first-order logic. We now use the summation constant sum <sup>∈</sup> <sup>N</sup> and the balance function bal : Address <sup>→</sup> <sup>N</sup> in our SL encoding. In particular, we use our smart invariants inv(has-coin, active) in this explicit SL encoding together with two additional axioms (Ax1, Ax2), ensuring that sum = |active| and bal(a) = |{c ∈ Coin | has-coin(a, c)}| for all a ∈ Address.

To formalize the additional properties, we introduce two counting mechanisms in our SL encoding. The first one is a bijective function count : Coin <sup>→</sup> <sup>N</sup><sup>+</sup> and the second one is a function idx : Address <sup>×</sup> Coin <sup>→</sup> <sup>N</sup><sup>+</sup>, where idx(a, .) : Coin <sup>→</sup> <sup>N</sup><sup>+</sup> is bijective for every <sup>a</sup> <sup>∈</sup> Address. To ensure that count and idx(a, .) count coins, we impose the following two properties:

$$\forall c: \texttt{Coin.} \texttt{ actize}(c) \iff \texttt{count}(c) \le \texttt{sum}\,,\tag{Ax1}$$

$$\forall c: \mathsf{Coin}. \,\forall a: \mathsf{Add} \mathsf{cases}. \,\mathsf{has-coin}(a, c) \iff \mathsf{idx}(a, c) \le \mathsf{ba1}(a) \,\,. \,\quad (\mathsf{Ax2})$$

Figure 3 illustrates our revised SL encoding for our smart transition mint1. We next ensure soundness of our resulting explicit encoding for summation, as follows.

**Theorem 9 (Soundness of Explicit** SL **Encodings).** *Let there be a pair* (bal, sum) <sup>∈</sup> <sup>N</sup>Address <sup>×</sup>N*, a pair* (has-coin, active) ∈ F*, and functions* count : Coin <sup>→</sup> <sup>N</sup><sup>+</sup> *and* idx : Address <sup>×</sup> Coin <sup>→</sup> <sup>N</sup><sup>+</sup>*.*

*Given that* count *is bijective,* idx(a, .) : Coin <sup>→</sup> <sup>N</sup><sup>+</sup> *is bijective for every* <sup>a</sup> <sup>∈</sup> Address*, and that (Ax1), (Ax2) and* inv (has-coin, active) *hold, then,* sum = |active| *and* bal(a) = |{c ∈ Coin : has-coin(a, c)}|*, for every* a ∈ Address*.*

*In particular, we have* sum = <sup>a</sup>∈Address bal(a)*.* 

When compared to Sect. 5.1, our explicit SL encoding introduced above uses our smart invariants as axioms of our encoding, together with (Ax1) and (Ax2). In our explicit SL encoding, the post-conditions asserting functional correctness of smart transitions express thus relations among old-sum to new-sum. For example, for mint<sup>n</sup> we are interested in ensuring

$$\mathtt{mint}\_{n} \Rightarrow \mathtt{new}\text{-sum} = \mathtt{o1d-sum} + n \,. \tag{2}$$

By using two new constants old-total, new-total <sup>∈</sup> <sup>N</sup>, we can use sum <sup>=</sup> total as smart invariant for mintn. As a result, the property to be ensured is then

$$\begin{array}{l} \mathsf{(old\text{-}sum} = \mathsf{old\text{-}total} \land \mathsf{new-}\mathsf{total} = \mathsf{old\text{-}total} + n \land \mathsf{init}\_{n}) \\ \Rightarrow (\mathsf{new-}\mathsf{sum} = \mathsf{new-}\mathsf{total}) \, . \end{array} \tag{3}$$

It is easy to see that the negations of (2) and (3) are equisatisfiable. We note however that the additional constants old-total, new-total used in (3) lead to unstable results within automated reasoners, as discussed in Sect. 6.

**Fig. 3.** Explicit SL encoding of mint1, where Addr is short for Address.

#### **6 Experiments**

**From Theory to Practice.** To make our explicit SL encodings handier for automated reasoners, we improved the setting illustrated in Fig. 3 by applying the following restrictions without losing any generality.

(i) The predicates has-coin and active were removed from the explicit SL encodings, by replacing them by their equivalent expressions (Ax1)-(Ax2).

(ii) The surjectivity assertions of count and idx were restricted to the relevant intervals [1, sum], [1, bal(a)] respectively.

(iii) Compared to Fig. 3, only one mutual count and one mutual idx functions were used. We however conclude that we do not lose expressivity of our resulting SL encoding, as shown in [10].

(iv) When our SL encoding contains expressions such as ∀c : Coin. idx(a0, c) ∈ [l0, u0] ⇐⇒ idx(a1, c) ∈ [l1, u1], with a0, a<sup>1</sup> being distinct addresses such that either u<sup>i</sup> ≤ bal(ai) or l<sup>i</sup> > bal(ai), i ∈ {0, 1}, then it can be assumed that the coins in those intervals are in the same order for both functions [10].

Based on the above, we derive three different explicit SL encodings to be used in automated reasoning about smart transitions. We respectively denote these explicit SL encodings by int, nat and id, and describe them next.

**Benchmarks.** In our experiments, we consider four smart transitions mint1, mintn, transferFrom<sup>1</sup> and transferFromn, respectively denoting minting and transferring one and n coins. These transitions capture the main operations of linear integer arithmetic. In particular, mint<sup>n</sup> implements the smart transition of our running example from Fig. 1.

For each of the four smart transitions, we implement four SL encodings: the implicit SL encoding uf from Sect. 5.1 using only uninterpreted functions and three explicit encodings int, nat and id as variants of Sect. 5.3. We also consider three additional arithmetic benchmarks using int, which are not directly motivated by smart contracts. Together with variants of int and nat presented in the sequel, our benchmark set contains 31 examples altogether, with each example being formalized in the SMT-LIB input syntax [1]. In addition to our encodings, we also proved consistency of the axioms used in our encodings.

**Fig. 4.** Linked lists in id.

**SL Encodings and Relaxations.** Our explicit SL encoding int uses linear integer arithmetic, whereas nat and id are based on natural numbers. As naturals are not a built-in theory in SMT-LIB, we assert the axioms of Presburger arithmetic directly in the encodings of nat and id.

In our id encodings, inductive datatypes are additionally used to order coins. There exists one linked list of all coins for count and one for each idx(a, .), a ∈ Address. Additionally, there exists a "null" coin, which is the first element of every list and is not owned by any address. As shown in Fig. 4, the numbering of each coin is defined by its position in the respective list. This way surjectivity for count and idx can respectively be asserted by the formulas ∃c : Coin. count(c) ≈ sum and ∀a : Address. ∃c : Coin. idx(a, c) ≈ bal(a). However, asserting surjectivity for int and nat cannot be achieved without quantifying over N<sup>+</sup>. Such quantification would drastically effect the performance of automated reasoners in (fragments of) first-order logics. As a remedy, within the default encodings of int and nat, we only consider relevant instances of surjectivity.

Further, we consider variations of int and nat by asserting proper surjectivity to the relevant intervals of idx and count (denoted as *surj*) and/or adding the total constants mentioned in Sect. 5.3 (denoted as *with* total, *no* total) . These variations of int and nat are implemented for mint<sup>1</sup> and transferFrom1.

**Experimental Setting.** We evaluated our benchmark set of 31 examples using SMT solvers Z3 [7] and CVC4 [6], as well as the first-order theorem prover Vampire [19]. Our experiments were run on a standard machine with an Intel Core i5-6200U CPU (2.30 GHz, 2.40 GHz) and 8 GB RAM. The time is given in seconds and we ran all experiments with a time limit of 300 s. Time out is indicated by the symbol ×. The default parameters were used for each solver, unless stated otherwise in the corresponding tables<sup>4</sup>.

**Experimental Analysis.** We first report on our experiments using different variations of int and nat. Table 3 shows that asserting complete surjectivity for int and nat is computationally hard and indeed significantly effects the performance of automated reasoners. Thus, for the following experiments only


**Table 3.** Results of mint<sup>1</sup> and transferFrom<sup>1</sup> using nat and int, with/without the total constants and with/without surjectivity.

<sup>4</sup> The precise calls and encodings are available at github.com/SoRaTu/SmartSums.


**Table 4.** Smart transitions using implicit/explicit SL encodings.

relevant instances of surjectivity, such as ∃c : Coin. count(c) = sum were asserted in int and nat. Table 3 also illustrates the instability of using the total constant. Some tasks seem to be easier even though their reasoning difficulty increased strictly by adding additional constants.

Our most important experimental findings are shown in Table 4, demonstrating that *our* SL *encodings are suitable for automated reasoners. Thanks to our explicit* SL *encodings, each solver can certify every smart transition in at least one encoding.* Our explicit SL encodings are more relevant than the implicit encoding uf as we can express and compare any two non-negative integer sums, whereas for uf handling arbitrary values n can only be done by iterating over the mint<sup>1</sup> (or transferFrom1) transition. This iteration requires inductive reasoning, which currently only Vampire could do [15], as indicated by the superscript ∗. Nevertheless, the transactions mint1, transferFrom1, which involve only one coin in uf, require no inductive reasoning as the actual sum is not considered; each of our solvers can certify these examples.

We note that the tasks mint<sup>n</sup> and transferFrom<sup>n</sup> from Table 4 yield a huge search space when using their explicit SL encodings within automated reasoners. We split these tasks into proving intermediate lemmas and proved each of these lemmas independently, by the respective solver. In particular, we used one lemma for mint<sup>n</sup> and four lemmas for transferFromn. In our experiments, we only used the recent theory reasoning framework of Vampire with split queues [13] and indicate our results in by superscript †.

We further remark that our explicit SL encoding id using inductive datatypes also requires inductive reasoning about smart transitions and beyond. The need of induction explains why SMT solvers failed proving our id benchmarks, as shown in Table 4. We note that Vampire found a proof using built-in induction [15] and theory-specific reasoning [13], as indicated by superscript ‡.

We conclude by showing the generality of our approach beyond smart transitions. It in fact enables fully automated reasoning about any two summations


**Table 5.** Arithmetic reasoning in the explicit SL encoding int.

 <sup>i</sup>∈<sup>I</sup> <sup>g</sup>(i), <sup>i</sup>∈<sup>I</sup> <sup>h</sup>(i) of non-negative integer values <sup>g</sup>(i), <sup>h</sup>(i) (<sup>i</sup> <sup>∈</sup> <sup>I</sup>) over a mutual finite set I. The examples of Table 5 affirm this claim.

#### **7 Related Work**

*Smart Contract Safety.* Formal verification of smart contracts is an emerging hot topic because of the value of the assets stored in smart contracts, e.g. the DeFi software [3]. Due to the nature of the blockchain, bugs in smart contracts are irreversible and thus the demand for provably bug-free smart contracts is high.

The K interactive framework has been used to verify safety of a smart contract, e.g. in [23]. Isabelle [22] was also shown to be useful in manual, interactive verification of smart contracts [17]. We, however, focus on automated approaches.

There are also efforts to perform deductive verification of smart contracts both on the source level in languages such as Solidity [4,14,33] and Move [35], as well as on the the Ethereum virtual machine (EVM) level [2,29]. This paper improves the effectiveness of these approaches by developing techniques for automatically reasoning about unbounded sums. This way, we believe we support a more semantic-based verification of smart contracts.

Our approach differs from works using ghost variables [14], since we do not manually update the "ghost state". Instead, the verifier needs only to reason about the local changes, and the aggregate state is maintained by the axioms. That means other approaches assume (a) the local changes and (b) the impact on ghost variables (sum), whereas we only assume (a) and automatically prove a ⇒ b. This way, we reduce the user-guidance in providing and proving (b).

Our work complements approaches that verify smart contracts as finite state machines [33] and methods, like ZEUS [18], using symbolic model checking and abstract interpretation to verify generic safety properties for smart contracts.

The work in [30] provides an extensive evaluation of ERC-20 and ERC-721 tokens. ERC-721 extends ERC-20 with ownership functions, one of which being "approve". It enables transactions on another party's behalf. This is independent of our ability to express sums in first-order logic, as the transaction's initiator is irrelevant to its effect.

*Reasoning about Financial Applications.* Recently, the Imandra prover introduced an automated reasoning framework for financial applications [24–26]. Similarly to our approach, these works use SMT procedures to verify and/or generate counter-examples to safety properties of low- and high-level algorithms. In particular, results of [24–26] include examples of verifying ranking orders in matching logics of exchanges, proving high-level properties such as transitivity and anti-symmetry of such orders. In contrast, we focus on verifying properties relating local changes in balances to changes of the global state (the sum). Moreover, our encodings enable automated reasoning both in SMT solving and first-order theorem proving.

*Automated Aggregate Reasoning.* The theory of first-order logic with aggregate operators has been thoroughly studied in [16,21]. Though proven to be strictly more expressive than first-order logic, both in the case of general aggregates as well as simple counting logics, in this paper we present a practical way to encode a weakened version of aggregates (specifically sums) in first-order logic. Our encoding (as in Sect. 5) works by expressing particular sums of interest, harnessing domain knowledge to avoid the need of general aggregate operators.

Previous works [5,20] in the field of higher-order reasoning do not directly discuss aggregates. The work of [20] extends Presburger arithmetic with Boolean algebra for finite, unbounded sets of uninterpreted elements. This includes a way to express the set cardinalities and to compare them against integer variables, but does not support uninterpreted functions, such as the balance functions we use throughout our approach.

The SMT-based framework of [5] takes a different, white-box approach, modifying the inner workings of SMT solvers to support higher-order logic. We on the other hand treat theorem provers and SMT solvers as black-boxes, constructing first-order formulas that are tailored to their capabilities. This allows us to use any off-the-shelf SMT solver.

In [8], an SMT module for the theory of FO(Agg) is presented, which can be used in all DPLL-based SAT, SMT and ASP solvers. However, FO(Agg) only provides a way to express functions that have sets or similar constructs as inputs, but not to verify their semantic behavior.

#### **8 Conclusions**

We present a methodology for reasoning about unbounded sums in the context of *smart transitions*, that is transitions that occur in smart contracts modeling transactions. Our sum logic SL and its usage of sum constants, instead of fully-fledged sum operators, turns out to be most appropriate for the setting of smart contracts. We show that SL has decidable fragments (Sect. 4.1), as well as undecidable ones (Sect. 4.2). Using two phases to first implicitly encode SL in first-order logic (Sect. 5.1), and then explicitly encode it (Sect. 5.3), allows us to use off-the-shelf automated reasoners in new ways, and automatically verify the semantic correctness of smart transitions.

Showing the (un)decidability of the SL fragment with two sets of uninterpreted functions and sums is an interesting step for further work, as this fragment supports encoding smart transition systems. Another interesting direction of future work is to apply our approach to different aggregates, such as minimum and maximum and to reason about under which conditions these values stay above /below certain thresholds. A slightly modified setting of our SL axioms can already handle min/max aggregates in a basic way, namely by using ≥ and ≤ instead of equality and dropping the injectivity/surjectivity (respectively) axioms of the counting mechanisms.

Summing upon multidimensional arrays in various ways is yet another direction of future research. Our approach supports the summation over all values in all dimensions by adding the required number of parameters to the predicate idx and by adapting the axioms accordingly.

**Acknowledgement.** We thank Petra Hozzov´a for fruitful discussions on our encodings and Sharon Shoham-Buchbinder for her insights and contributions to this paper. This work was partially funded by the ERC CoG ARTIST 101002685, the ERC StG SYMCAR 639270, the United States-Israel Binational Science Foundation (BSF) grant No. 2016260, Grant No. 1810/18 from the Israeli Science Foundation, Len Blavatnik and the Blavatnik Family foundation, the FWF grant LogiCS W1255-N23, the TU Wien DK SecInt and the Amazon ARA 2020 award FOREST.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Stateless Model Checking Under a Reads-Value-From Equivalence**

Pratyush Agarwal<sup>1</sup>, Krishnendu Chatterjee<sup>2</sup>, Shreya Pathak<sup>1</sup>, Andreas Pavlogiannis<sup>3</sup>, and Viktor Toman2(B)

> IIT Bombay, Mumbai, India IST Austria, Klosterneuburg, Austria viktor.toman@ist.ac.at Aarhus University, Aarhus, Denmark

**Abstract.** Stateless model checking (SMC) is one of the standard approaches to the verification of concurrent programs. As scheduling non-determinism creates exponentially large spaces of thread interleavings, SMC attempts to partition this space into equivalence classes and explore only a few representatives from each class. The efficiency of this approach depends on two factors: (a) the coarseness of the partitioning, and (b) the time to generate representatives in each class. For this reason, the search for coarse partitionings that are efficiently explorable is an active research challenge.

In this work we present RVF-SMC, a new SMC algorithm that uses a novel *reads-value-from (RVF)* partitioning. Intuitively, two interleavings are deemed equivalent if they agree on the value obtained in each read event, and read events induce consistent causal orderings between them. The RVF partitioning is provably coarser than recent approaches based on Mazurkiewicz and "reads-from" partitionings. Our experimental evaluation reveals that RVF is quite often a very effective equivalence, as the underlying partitioning is exponentially coarser than other approaches. Moreover, RVF-SMC generates representatives very efficiently, as the reduction in the partitioning is often met with significant speed-ups in the model checking task.

#### **1 Introduction**

The verification of concurrent programs is one of the key challenges in formal methods. Interprocess communication adds a new dimension of non-determinism in program behavior, which is resolved by a scheduler. As the programmer has no control over the scheduler, program correctness has to be guaranteed under all possible schedulers, i.e., the scheduler is adversarial to the program and can generate erroneous behavior if one can arise out of scheduling decisions. On the other hand, during program testing, the adversarial nature of the scheduler is to hide erroneous runs, making bugs extremely difficult to reproduce by testing alone (aka Heisenbugs [1]). Consequently, the verification of concurrent programs rests on rigorous model checking techniques [2] that cover all possible program behaviors that can arise out of scheduling non-determinism, leading to early tools such as VeriSoft [3,4] and CHESS [5].

To battle with the state-space explosion problem, effective model checking for concurrency is stateless. A stateless model checker (SMC) explores the behavior of the concurrent program by manipulating traces instead of states, where each (concurrent) trace is an interleaving of event sequences of the corresponding threads [6]. To further improve performance, various techniques try to reduce the number of explored traces, such as context bounded techniques [7–10] As many interleavings induce the same program behavior, SMC partitions the interleaving space into equivalence classes and attempts to sample a few representative traces from each class. The most popular approach in this domain is partialorder reduction techniques [6,11,12], which deems interleavings as equivalent based on the way that conflicting memory accesses are ordered, also known as the Mazurkiewicz equivalence [13]. Dynamic partial order reduction [14] constructs this equivalence dynamically, when all memory accesses are known, and thus does not suffer from the imprecision of earlier approaches based on static information. Subsequent works managed to explore the Mazurkiewicz partitioning optimally [15,16], while spending only polynomial time per class.

The performance of an SMC algorithm is generally a product of two factors: (a) the size of the underlying partitioning that is explored, and (b) the total time spent in exploring each class of the partitioning. Typically, the task of visiting a class requires solving a consistency-checking problem, where the algorithm checks whether a semantic abstraction, used to represent the class, has a consistent concrete interleaving that witnesses the class. For this reason, the search for effective SMC is reduced to the search of coarse partitionings for which the consistency problem is tractable, and has become a very active research direction in recent years. In [17], the Mazurkiewicz partitioning was further reduced by ignoring the order of conflicting write events that are not observed, while retaining polynomial-time consistency checking. Various other works refine the notion of dependencies between events, yielding coarser abstractions [18–20]. The work of [21] used a reads-from abstraction and showed that the consistency problem admits a fully polynomial solution in acyclic communication topologies. Recently, this approach was generalized to arbitrary topologies, with an algorithm that remains polynomial for a bounded number of threads [22]. Finally, recent approaches define value-centric partitionings [23], as well as partitionings based on maximal causal models [24]. These partitionings are very coarse, as they attempt to distinguish only between traces which differ in the values read by their corresponding read events. We illustrate the benefits of value-based partitionings with a motivating example.

#### **1.1 Motivating Example**

Consider a simple concurrent program shown in Fig. 1. The program has 98 different orderings of the conflicting memory accesses, and each ordering corresponds to a separate class of the Mazurkiewicz partitioning. Utilizing the readsfrom abstraction reduces the number of partitioning classes to 9. However, when taking into consideration the values that the events can read and write, the number of cases to consider can be reduced even further. In this specific example, there is only a single behaviour the program may exhibit, in which both read events read the only observable value.


**Fig. 1.** Concurrent program and its underlying partitioning classes.

The above benefits have led to recent attempts in performing SMC using a value-based equivalence [23,24]. However, as the realizability problem is NPhard in general [25], both approaches suffer significant drawbacks. In particular, the work of [23] combines the value-centric approach with the Mazurkiewicz partitioning, which creates a refinement with exponentially many more classes than potentially necessary. The example program in Fig. 1 illustrates this, where while both read events can only observe one possible value, the work of [23] further enumerates all Mazurkiewicz orderings of all-but-one threads, resulting in 7 partitioning classes. Separately, the work of [24] relies on SMT solvers, thus spending exponential time to solve the realizability problem. Hence, each approach suffers an exponential blow-up a-priori, which motivates the following question: is there an efficient *parameterized* algorithm for the consistency problem? That is, we are interested in an algorithm that is exponential-time in the worst case (as the problem is NP-hard in general), but efficient when certain natural parameters of the input are small, and thus only becomes slow in extreme cases.

Another disadvantage of these works is that each of the exploration algorithms can end up to the same class of the partitioning many times, further hindering performance. To see an example, consider the program in Fig. 1 again. The work of [23] assigns values to reads one by one, and in this example, it needs to consider as separate cases both permutations of the two reads as the orders for assigning the values. This is to ensure completeness in cases where there are write events causally dependent on some read events (e.g., a write event appearing only if its thread-predecessor reads a certain value). However, no causally dependent write events are present in this program, and our work uses a principled approach to detect this and avoid the redundant exploration. While an example to demonstrate [24] revisiting partitioning classes is a bit more involved one, this property follows from the lack of information sharing between spawned subroutines, enabling the approach to be massively parallelized, which has been discussed already in prior works [21,23,26].

#### **1.2 Our Contributions**

In this work we tackle the two challenges illustrated in the motivating example in a principled, algorithmic way. In particular, our contributions are as follows.


### **2 Preliminaries**

**General Notation.** Given a natural number <sup>i</sup> <sup>≥</sup> 1, we let [i] be the set {1, <sup>2</sup>,...,i}. Given a map <sup>f</sup> : <sup>X</sup> <sup>→</sup> <sup>Y</sup> , we let dom(f) = <sup>X</sup> denote the domain of <sup>f</sup>. We represent maps <sup>f</sup> as sets of tuples {(x, f(x))}<sup>x</sup>. Given two maps <sup>f</sup>1, f<sup>2</sup> over the same domain <sup>X</sup>, we write <sup>f</sup><sup>1</sup> <sup>=</sup> <sup>f</sup><sup>2</sup> if for every <sup>x</sup> <sup>∈</sup> <sup>X</sup> we have <sup>f</sup>1(x) = <sup>f</sup>2(x). Given a set <sup>X</sup> <sup>⊂</sup> <sup>X</sup>, we denote by <sup>f</sup>|X the restriction of <sup>f</sup> to <sup>X</sup> . A binary relation <sup>∼</sup> on a set <sup>X</sup> is an *equivalence* iff <sup>∼</sup> is reflexive, symmetric and transitive.

#### **2.1 Concurrent Model**

Here we describe the computational model of concurrent programs with shared memory under the Sequential Consistency (SC) memory model. We follow a standard exposition of stateless model checking, similarly to [14,15,21–23,28],

**Concurrent Program.** We consider a concurrent program <sup>H</sup> <sup>=</sup> {thri}<sup>k</sup> <sup>i</sup>=1 of <sup>k</sup> deterministic threads. The threads communicate over a shared memory <sup>G</sup> of global variables with a finite value domain D. Threads execute *events* of the following types.


Additionally, threads can execute local events which do not access global variables and thus are not modeled explicitly.

Given an event <sup>e</sup>, we denote by thr(e) its thread and by var(e) its global variable. We denote by E the set of all events, and by R (W) the set of read (write) events. Given two events <sup>e</sup>1, e<sup>2</sup> ∈ E, we say that they *conflict*, denoted e<sup>1</sup> e2, if they access the same global variable and at least one of them is a write event.

**Concurrent Program Semantics.** The semantics of H are defined by means of a transition system over a state space of global states. A global state consists of (i) a memory function that maps every global variable to a value, and (ii) a local state for each thread, which contains the values of the local variables and the program counter of the thread. We consider the standard setting of Sequential Consistency (SC), and refer to [14] for formal details. As usual, H is execution-bounded, which means that the state space is finite and acyclic.

**Event Sets.** Given a set of events <sup>X</sup> ⊆ E, we write <sup>R</sup>(X) = <sup>X</sup> ∩ R for the set of read events of <sup>X</sup>, and <sup>W</sup>(X) = <sup>X</sup> ∩W for the set of write events of <sup>X</sup>. Given a set of events <sup>X</sup> ⊆ E and a thread thr, we denote by <sup>X</sup>thr and <sup>X</sup>=thr the events of thr, and the events of all other threads in <sup>X</sup>, respectively.

**Sequences and Traces.** Given a sequence of events τ = e1,...,e<sup>j</sup> , we denote by <sup>E</sup>(<sup>τ</sup> ) the set of events that appear in <sup>τ</sup> . We further denote <sup>R</sup>(<sup>τ</sup> ) = <sup>R</sup>(E(<sup>τ</sup> )) and <sup>W</sup>(<sup>τ</sup> ) = <sup>W</sup>(E(<sup>τ</sup> )).

Given a sequence <sup>τ</sup> and two events <sup>e</sup>1, e<sup>2</sup> ∈ E(<sup>τ</sup> ), we write <sup>e</sup><sup>1</sup> <sup>&</sup>lt;<sup>τ</sup> <sup>e</sup><sup>2</sup> when <sup>e</sup><sup>1</sup> appears before <sup>e</sup><sup>2</sup> in <sup>τ</sup> , and <sup>e</sup><sup>1</sup> <sup>≤</sup><sup>τ</sup> <sup>e</sup><sup>2</sup> to denote that <sup>e</sup><sup>1</sup> <sup>&</sup>lt;<sup>τ</sup> <sup>e</sup><sup>2</sup> or <sup>e</sup><sup>1</sup> <sup>=</sup> <sup>e</sup>2. Given a sequence <sup>τ</sup> and a set of events <sup>A</sup>, we denote by <sup>τ</sup> <sup>|</sup><sup>A</sup> the *projection* of <sup>τ</sup> on <sup>A</sup>, which is the unique subsequence of <sup>τ</sup> that contains all events of <sup>A</sup>∩E(<sup>τ</sup> ), and only those events. Given a sequence <sup>τ</sup> and a thread thr, let <sup>τ</sup>thr be the subsequence of <sup>τ</sup> with events of thr, i.e., <sup>τ</sup> |E(<sup>τ</sup> )thr. Given two sequences <sup>τ</sup><sup>1</sup> and <sup>τ</sup>2, we denote by <sup>τ</sup><sup>1</sup> ◦ <sup>τ</sup><sup>2</sup> the sequence that results in appending <sup>τ</sup><sup>2</sup> after <sup>τ</sup>1.

A (concrete, concurrent) *trace* is a sequence of events σ that corresponds to a concrete valid execution of <sup>H</sup>. We let enabled(σ) be the set of enabled events after <sup>σ</sup> is executed, and call <sup>σ</sup> *maximal* if enabled(σ) = <sup>∅</sup>. As <sup>H</sup> is bounded, all executions of H are finite and the length of the longest execution in H is a parameter of the input.

**Reads-From and Value Functions.** Given a sequence of events τ , we define the *reads-from function* of <sup>τ</sup> , denoted RF<sup>τ</sup> : <sup>R</sup>(<sup>τ</sup> ) → W(<sup>τ</sup> ), as follows. Given a read event <sup>r</sup> ∈ R(<sup>τ</sup> ), we have that RF<sup>τ</sup> (r) is the latest write (of any thread) conflicting with <sup>r</sup> and occurring before <sup>r</sup> in <sup>τ</sup> , i.e., (i) RF<sup>τ</sup> (r) <sup>r</sup>, (ii) RF<sup>τ</sup> (r) <sup>&</sup>lt;<sup>τ</sup> <sup>r</sup>, and (iii) for each <sup>w</sup> ∈ W(<sup>τ</sup> ) such that <sup>w</sup> <sup>r</sup> and w <<sup>τ</sup> <sup>r</sup>, we have <sup>w</sup> <sup>≤</sup><sup>τ</sup> RF<sup>τ</sup> (r). We say that <sup>r</sup> reads-from RF<sup>τ</sup> (r) in <sup>τ</sup> . For simplicity, we assume that <sup>H</sup> has an initial salient write event on each variable.

Further, given a trace σ, we define the *value function* of σ, denoted val<sup>σ</sup> : <sup>E</sup>(σ) → D, such that val<sup>σ</sup>(e) is the value of the global variable var(e) after the prefix of <sup>σ</sup> up to and including <sup>e</sup> has been executed. Intuitively, val<sup>σ</sup>(e) captures the value that a read (resp. write) event e shall read (resp. write) in σ. The value function val<sup>σ</sup> is well-defined as <sup>σ</sup> is a valid trace and the threads of H are deterministic.

#### **2.2 Partial Orders**

In this section we present relevant notation around partial orders, which are a central object in this work.

**Partial Orders.** Given a set of events <sup>X</sup> ⊆ E, a *(strict) partial order* <sup>P</sup> over <sup>X</sup> is an irreflexive, antisymmetric and transitive relation over <sup>X</sup> (i.e., <sup>&</sup>lt;<sup>P</sup> <sup>⊆</sup> <sup>X</sup> <sup>×</sup> <sup>X</sup>). Given two events <sup>e</sup>1, e<sup>2</sup> <sup>∈</sup> <sup>X</sup>, we write <sup>e</sup><sup>1</sup> <sup>≤</sup><sup>P</sup> <sup>e</sup><sup>2</sup> to denote that <sup>e</sup><sup>1</sup> <sup>&</sup>lt;<sup>P</sup> <sup>e</sup><sup>2</sup> or <sup>e</sup><sup>1</sup> <sup>=</sup> <sup>e</sup>2. Two distinct events <sup>e</sup>1, e<sup>2</sup> <sup>∈</sup> <sup>X</sup> are *unordered* by <sup>P</sup>, denoted <sup>e</sup><sup>1</sup> <sup>P</sup> <sup>e</sup>2, if neither <sup>e</sup><sup>1</sup> <sup>&</sup>lt;<sup>P</sup> <sup>e</sup><sup>2</sup> nor <sup>e</sup><sup>2</sup> <sup>&</sup>lt;<sup>P</sup> <sup>e</sup>1, and *ordered* (denoted <sup>e</sup><sup>1</sup> <sup>P</sup> <sup>e</sup>2) otherwise. Given a set <sup>Y</sup> <sup>⊆</sup> <sup>X</sup>, we denote by <sup>P</sup>|<sup>Y</sup> the *projection* of <sup>P</sup> on the set <sup>Y</sup> , where for every pair of events <sup>e</sup>1, e<sup>2</sup> <sup>∈</sup> <sup>Y</sup> , we have that <sup>e</sup><sup>1</sup> <sup>&</sup>lt;<sup>P</sup> <sup>|</sup><sup>Y</sup> <sup>e</sup><sup>2</sup> iff <sup>e</sup><sup>1</sup> <sup>&</sup>lt;<sup>P</sup> <sup>e</sup>2. Given two partial orders P and Q over a common set X, we say that Q *refines* P, denoted by <sup>Q</sup> <sup>P</sup>, if for every pair of events <sup>e</sup>1, e<sup>2</sup> <sup>∈</sup> <sup>X</sup>, if <sup>e</sup><sup>1</sup> <sup>&</sup>lt;<sup>P</sup> <sup>e</sup><sup>2</sup> then <sup>e</sup><sup>1</sup> <sup>&</sup>lt;<sup>Q</sup> <sup>e</sup>2. A *linearization* of P is a total order that refines P.

**Lower Sets.** Given a pair (X, P), where X is a set of events and P is a partial order over <sup>X</sup>, a *lower set* of (X, P) is a set <sup>Y</sup> <sup>⊆</sup> <sup>X</sup> such that for every event <sup>e</sup><sup>1</sup> <sup>∈</sup> <sup>Y</sup> and event <sup>e</sup><sup>2</sup> <sup>∈</sup> <sup>X</sup> with <sup>e</sup><sup>2</sup> <sup>≤</sup><sup>P</sup> <sup>e</sup>1, we have <sup>e</sup><sup>2</sup> <sup>∈</sup> <sup>Y</sup> .

**Visible Writes.** Given a partial order <sup>P</sup> over a set <sup>X</sup>, and a read event <sup>r</sup> <sup>∈</sup> <sup>R</sup>(X), the set of *visible writes* of <sup>r</sup> is defined as

$$\begin{aligned} \mathsf{V visibleW}\_P(r) &= \{ \begin{array}{l} w \in \mathcal{W}(X): \ (\text{i)} \ r \not\simeq w \text{ and (ii)} \ r \not\preceq\_P w \text{ and (iii)} \text{ for each} \\\ w' &\in \mathcal{W}(X) \text{ with } r \not\simeq w', \text{ if } w < p \ w' \text{ then } w' \not\preceq\_P r \end{array} \} \end{aligned}$$

i.e., the set of write events w conflicting with r that are not "hidden" to r by P. **The Program Order** PO**.** The *program order* PO of <sup>H</sup> is a partial order <sup>&</sup>lt;PO⊆E×E that defines a fixed order between some pairs of events of the same thread, reflecting the semantics of H.

A set of events <sup>X</sup> ⊆ E is *proper* if (i) it is a lower set of (E, PO), and (ii) for each thread thr, the events <sup>X</sup>thr are totally ordered in PO (i.e., for each distinct <sup>e</sup>1, e<sup>2</sup> <sup>∈</sup> <sup>X</sup>thr we have <sup>e</sup><sup>1</sup> PO <sup>e</sup>2). A sequence <sup>τ</sup> is *well-formed* if (i) its set of events <sup>E</sup>(<sup>τ</sup> ) is proper, and (ii) <sup>τ</sup> respects the program order (formally, <sup>τ</sup> PO|E(<sup>τ</sup> )). Every trace <sup>σ</sup> of <sup>H</sup> is well-formed, as it corresponds to a concrete valid execution of <sup>H</sup>. Each event of <sup>H</sup> is then uniquely identified by its PO predecessors, and by the values its PO predecessor reads have read.

**Fig. 2.** A trace σ, the displayed events E(σ) are vertically ordered as they appear in σ. The solid black edges represent the program order PO. The dashed red edges represent the reads-from function RFσ. The transitive closure of all the edges then gives us the causally-happens-before partial order -→σ.

**Causally-Happens-Before Partial Orders.** A trace σ induces a *causallyhappens-before* partial order →<sup>σ</sup> ⊆ E(σ) × E(σ), which is the weakest partial order such that (i) it refines the program order (i.e., →<sup>σ</sup> PO|E(σ)), and (ii) for every read event <sup>r</sup> ∈ R(σ), its reads-from RF<sup>σ</sup>(r) is ordered before it (i.e., RF<sup>σ</sup>(r) →<sup>σ</sup> <sup>r</sup>). Intuitively, →<sup>σ</sup> contains the causal orderings in <sup>σ</sup>, i.e., it captures the flow of write events into read events in σ together with the program order. Figure 2 presents an example of a trace and its causal orderings.

#### **3 Reads-Value-From Equivalence**

In this section we present our new equivalence on traces, called the *reads-valuefrom* equivalence (RVF equivalence, or <sup>∼</sup>RVF, for short). Then we illustrate that ∼RVF has some desirable properties for stateless model checking.

**Reads-Value-From Equivalence.** Given two traces σ<sup>1</sup> and σ2, we say that they are *reads-value-from-equivalent*, written <sup>σ</sup><sup>1</sup> <sup>∼</sup>RVF <sup>σ</sup>2, if the following hold.


Figure <sup>3</sup> presents an intuitive example of RVF-(in)equivalent traces.

**Fig. 3.** Three traces σ1, σ2, σ3, events of each trace are vertically ordered as they appear in the trace. Traces <sup>σ</sup><sup>1</sup> and <sup>σ</sup><sup>2</sup> are RVF-equivalent (σ<sup>1</sup> <sup>∼</sup>RVF <sup>σ</sup>2), as they have the same events, same value function, and the two read events are causally unordered in both. Trace σ<sup>3</sup> is not RVF-equivalent with either of σ<sup>1</sup> and σ2. Compared to σ<sup>1</sup> resp. σ2, the value function of σ<sup>3</sup> differs (r(y) reads a different value), and the causal orderings of the reads differ (r(x)-→<sup>σ</sup><sup>3</sup> r(y)).

**Soundness.** The RVF equivalence induces a partitioning on the maximal traces of H. Any algorithm that explores each class of this partitioning provably discovers every reachable local state of every thread, and thus RVF is a sound equivalence for local safety properties, such as assertion violations, in the same spirit as in other recent works [21–24]. This follows from the fact that for any two traces <sup>σ</sup><sup>1</sup> and <sup>σ</sup><sup>2</sup> with <sup>E</sup>(σ1) = <sup>E</sup>(σ2) and val<sup>σ</sup><sup>1</sup> <sup>=</sup> val<sup>σ</sup><sup>2</sup> , the local states of each thread are equal after executing σ<sup>1</sup> and σ2.

$$\begin{array}{c c c} & \mathsf{k} \text{ \textquotedblleft \textquotedblright \textquotedblright} & \mathsf{\hkern-9.12ex} \\ \mathsf{\hkern-12ex} \mathsf{\hkern-12ex} & \mathsf{\hkern-12ex} & \mathsf{\hkern-12ex} \\ \mathsf{\hkern-12ex} & \mathsf{\hkern-12ex} & \mathsf{\hkern-12ex} \\ \mathsf{\hkern-12ex} & \mathsf{\hkern-12ex} & \mathsf{\hkern-12ex} \end{array} \qquad \begin{array}{c c c} & \mathsf{\hkern-12ex} & \mathsf{\hkern-12ex} & \mathsf{\hkern-12ex} \\ \mathsf{\hkern-12ex} & \mathsf{\hkern-12ex} & \mathsf{\hkern-12ex} \\ \mathsf{\hkern-12ex} & \mathsf{\hkern-12ex} & \mathsf{\hkern-12ex} \end{array} \\ \end{array}$$

**Fig. 4.** SMC trace equivalences. An edge from X to Y signifies that Y is always at least as coarse, and sometimes coarser, than X.

**Coarseness.** Here we describe the coarseness properties of the RVF equivalence, as compared to other equivalences used by state-of-the-art approaches in stateless model checking. Figure 4 summarizes the comparison.

The SMC algorithms of [22] and [28] operate on a *reads-from equivalence*, which deems two traces σ<sup>1</sup> and σ<sup>2</sup> equivalent if


The above two conditions imply that the induced causally-happens-before partial orders are equal, i.e., →<sup>σ</sup><sup>1</sup> = →<sup>σ</sup><sup>2</sup> , and thus trivially also →<sup>σ</sup><sup>1</sup> |R = →<sup>σ</sup><sup>2</sup> |R. Further, by a simple inductive argument the value functions of the two traces are also equal, i.e., valσ<sup>1</sup> <sup>=</sup> valσ<sup>2</sup> . Hence any two reads-from-equivalent traces are also RVF-equivalent, which makes the RVF equivalence always at least as coarse as the reads-from equivalence.

The work of [23] utilizes a *value-centric equivalence*, which deems two traces equivalent if they satisfy all the conditions of our RVF equivalence, and also some further conditions (note that these conditions are necessary for correctness of the SMC algorithm in [23]). Thus the RVF equivalence is trivially always at least as coarse. The value-centric equivalence preselects a single thread thr, and then requires two extra conditions for the traces to be equivalent, namely:


Both the reads-from equivalence and the value-centric equivalence are in turn as coarse as the *data-centric equivalence* of [21]. Given two traces, the datacentric equivalence has the equivalence conditions of the reads-from equivalence, and additionally, it preselects a single thread thr (just like the value-centric equivalence) and requires the second extra condition of the value-centric equivalence, i.e., equality of orderings for each conflicting pair of events outside of thr.

Finally, the data-centric equivalence is as coarse as the classical *Mazurkiewicz equivalence* [13], the baseline equivalence for stateless model checking [14,15,29]. Mazurkiewicz equivalence deems two traces equivalent if they consist of the same set of events and they agree on their ordering of conflicting events.

While RVF is always at least as coarse, it can be (even exponentially) coarser, than each of the other above-mentioned equivalences. We illustrate this in Appendix B of [30]. We summarize these observations in the following proposition.

**Proposition 1.** RVF *is at least as coarse as each of the Mazurkiewicz equivalence* [15]*, the data-centric equivalence* [21]*, the reads-from equivalence* [22]*, and the value-centric equivalence* [23]*. Moreover,* RVF *can be exponentially coarser than each of these equivalences.*

In this work we develop our SMC algorithm RVF-SMC around the RVF equivalence, with the guarantee that the algorithm explores at most one maximal trace per class of the RVF partitioning, and thus can perform significantly fewer steps than algorithms based on the above equivalences. To utilize RVF, the algorithm in each step solves an instance of the verification of sequential consistency problem, which we tackle in the next section. Afterwards, we present RVF-SMC.

#### **4 Verifying Sequential Consistency**

In this section we present our contributions towards the problem of verifying sequential consistency (VSC). We present an algorithm VerifySC for VSC, and we show how it can be efficiently used in stateless model checking.

**The** VSC **Problem.** Consider an input pair (X,GoodW) where


<sup>A</sup> *witness* of (X,GoodW) is a linearization <sup>τ</sup> of <sup>X</sup> (i.e., <sup>E</sup>(<sup>τ</sup> ) = <sup>X</sup>) respecting the program order (i.e., <sup>τ</sup> PO|X), such that each read <sup>r</sup> ∈ R(<sup>τ</sup> ) reads-from one of its good-writes in <sup>τ</sup> , formally RF<sup>τ</sup> (r) <sup>∈</sup> GoodW(r) (we then say that <sup>τ</sup> *satisfies* the good-writes function GoodW). The task is to decide whether (X,GoodW) has a witness, and to construct one in case it exists.

VSC **in Stateless Model Checking.** The VSC problem naturally ties in with our SMC approach enumerating the equivalence classes of the RVF trace partitioning. In our approach, we shall generate instances (X,GoodW) such that (i) each witness <sup>σ</sup> of (X,GoodW) is a valid program trace, and (ii) all witnesses <sup>σ</sup>1, σ<sup>2</sup> of (X,GoodW) are pairwise RVF-equivalent (σ<sup>1</sup> <sup>∼</sup>RVF <sup>σ</sup>2).

**Hardness of** VSC**.** Given an input (X,GoodW) to the VSC problem, let <sup>n</sup> <sup>=</sup> <sup>|</sup>X|, let <sup>k</sup> be the number of threads appearing in <sup>X</sup>, and let <sup>d</sup> be the number of variables accessed in X. The classic work of [25] establishes two important lower bounds on the complexity of VSC:


The first bound eliminates the possibility of any algorithm with time complexity O(n<sup>f</sup>(k) ), where f is an arbitrary computable function. Similarly, the second bound eliminates algorithms with complexity O(n<sup>f</sup>(d) ) for any computable f.

In this work we show that the problem is parameterizable in k + d, and thus admits efficient (polynomial-time) solutions when both variables are bounded.

#### **4.1 Algorithm for VSC**

In this section we present our algorithm VerifySC for the problem VSC. First we define some relevant notation. In our definitions we consider a fixed input pair (X,GoodW) to the VSC problem, and a fixed sequence <sup>τ</sup> with <sup>E</sup>(<sup>τ</sup> ) <sup>⊆</sup> <sup>X</sup>.

**Active Writes.** A write <sup>w</sup> ∈ W(<sup>τ</sup> ) is *active* in <sup>τ</sup> if it is the last write of its variable in <sup>τ</sup> . Formally, for each <sup>w</sup> ∈ W(<sup>τ</sup> ) with var(w ) = var(w) we have <sup>w</sup> <sup>≤</sup><sup>τ</sup> <sup>w</sup>. We can then say that <sup>w</sup> is the active write of the variable var(w) in <sup>τ</sup> . **Held Variables.** A variable <sup>x</sup> ∈ G is *held* in <sup>τ</sup> if there exists a read <sup>r</sup> <sup>∈</sup> <sup>R</sup>(X) \ E(<sup>τ</sup> ) with var(r) = <sup>x</sup> such that for each its good-write <sup>w</sup> <sup>∈</sup> GoodW(r) we have <sup>w</sup> <sup>∈</sup> <sup>τ</sup> . In such a case we say that <sup>r</sup> *holds* <sup>x</sup> in <sup>τ</sup> . Note that several distinct reads may hold a single variable in τ .

**Executable Events.** An event <sup>e</sup> ∈ E(X) \ E(<sup>τ</sup> ) is *executable* in <sup>τ</sup> if <sup>E</sup>(<sup>τ</sup> ) ∪ {e} is a lower set of (X, PO) and the following hold.


**Memory Maps.** A *memory map* of τ is a function from global variables to thread indices MMap<sup>τ</sup> : G → [k] where for each variable <sup>x</sup> ∈ G, the map MMap<sup>τ</sup> (x) captures the thread of the active write of x in τ .

**Witness States.** The sequence τ is a *witness prefix* if the following hold.


Intuitively, τ is a witness prefix if it satisfies all VSC requirements modulo its events, and if each read not in τ has at least one good-write still available to readfrom in potential extensions of τ . For a witness prefix τ we call its corresponding event set and memory map a *witness state*.

Figure 5 provides an example illustrating the above concepts, where for brevity of presentation, the variables are subscripted and the values are not displayed.

**Fig. 5.** Event set X, and the good-writes function GoodW denoted by the green dotted edges. The solid nodes are ordered vertically as they appear in τ . The grey dashed nodes are in X \ E(τ ). Events r<sup>x</sup> and w- <sup>x</sup> are executable in τ . Event r<sup>y</sup> is not, its goodwrite is not active in τ . Event w<sup>y</sup> is also not executable, as its variable y is held by ry. The memory map of τ is MMap<sup>τ</sup> (x) = 1 and MMap<sup>τ</sup> (y) = 3. τ is a witness prefix, and E(τ ) with MMap<sup>τ</sup> together form its witness state.

#### **Algorithm 1:** VerifySC

**Input**: Proper event set <sup>X</sup> and good-writes function GoodW: <sup>R</sup>(X) <sup>→</sup> <sup>2</sup>W(X) **Output**: A witness <sup>τ</sup> of (X, GoodW) if (X, GoodW) has a witness, else <sup>τ</sup> <sup>=</sup> <sup>⊥</sup> **<sup>1</sup>** S←{}; Done ← {} **<sup>2</sup> while** S = ∅ **do <sup>3</sup>** Extract a sequence τ from S **<sup>4</sup> if** E(τ ) = X **then return** τ ; // All events executed, witness found **5 foreach** *event* e *executable in* τ **do <sup>6</sup>** Let τ<sup>e</sup> ← τ ◦ e // Execute <sup>e</sup> **<sup>7</sup> if** ∃τ - <sup>∈</sup> Done *s.t.* <sup>E</sup>(τe) = <sup>E</sup>(<sup>τ</sup> - ) *and* MMap<sup>τ</sup>*<sup>e</sup>* = MMap<sup>τ</sup> **then <sup>8</sup>** Insert <sup>τ</sup><sup>e</sup> in <sup>S</sup> and in Done // New witness state reached **<sup>9</sup> return** ⊥ // No witness exists

**Algorithm.** We are now ready to describe our algorithm VerifySC, in Algorithm <sup>1</sup> we present the pseudocode. We attempt to construct a witness of (X,GoodW) by enumerating the witness states reachable by the following process. We start (Line 1) with an empty sequence as the first witness prefix (and state). We maintain a worklist <sup>S</sup> of so-far unprocessed witness prefixes, and a set Done of reached witness states. Then we iteratively obtain new witness prefixes (and states) by considering an already obtained prefix (Line 3) and extending it with each possible executable event (Line 6). Crucially, when we arrive at a sequence τe, we include it only if no sequence τ with equal corresponding witness state has been reached yet (Line 7). We stop when we successfully create a witness (Line 4) or when we process all reachable witness states (Line 9).

**Correctness and Complexity.** We now highlight the correctness and complexity properties of VerifySC, while we refer to Appendix C of [30] for the proofs. The soundness follows straightforwardly by the fact that each sequence in S is a witness prefix. This follows from a simple inductive argument that extending a witness prefix with an executable event yields another witness prefix. The completeness follows from the fact that given two witness prefixes τ<sup>1</sup> and τ<sup>2</sup> with equal induced witness state, these prefixes are "equi-extendable" to a witness. Indeed, if a suffix <sup>τ</sup> <sup>∗</sup> exists such that <sup>τ</sup><sup>1</sup> ◦ <sup>τ</sup> <sup>∗</sup> is a witness of (X,GoodW), then <sup>τ</sup><sup>2</sup> ◦ <sup>τ</sup> <sup>∗</sup> is also a witness of (X,GoodW). The time complexity of VerifySC is bounded by <sup>O</sup>(n<sup>k</sup>+1 · <sup>k</sup><sup>d</sup>+1), for <sup>n</sup> events, <sup>k</sup> threads and <sup>d</sup> variables. The bound follows from the fact that there are at most <sup>n</sup><sup>k</sup> · <sup>k</sup><sup>d</sup> pairwise distinct witness states. We thus have the following theorem.

**Theorem 1.** VSC *for* <sup>n</sup> *events,* <sup>k</sup> *threads and* <sup>d</sup> *variables is solvable in* <sup>O</sup>(n<sup>k</sup>+1 · k<sup>d</sup>+1) *time. Moreover, if each variable is written by only one thread,* VSC *is solvable in* O(n<sup>k</sup>+1) *time.*

**Implications.** We now highlight some important implications of Theorem 1. Although VSC is NP-hard [25], the theorem shows that the problem is parameterizable in k + d, and thus in polynomial time when both parameters are bounded. Moreover, even when only k is bounded, the problem is fixed-parameter tractable in d, meaning that d only exponentiates a constant as opposed to n (e.g., we have a polynomial bound even when d = log n). Finally, the algorithm is polynomial for a fixed number of threads regardless of d, when every memory location is written by only one thread (e.g., in producer-consumer settings, or in the concurrent-read-exclusive-write (CREW) concurrency model). These important facts brought forward by Theorem 1 indicate that VSC is likely to be efficiently solvable in many practical settings, which in turn makes RVF a good equivalence for SMC.

#### **4.2 Practical Heuristics for VerifySC in SMC**

We now turn our attention to some practical heuristics that are expected to further improve the performance of VerifySC in the context of SMC.

**1. Limiting the Search Space.** We employ two straightforward improvements to VerifySC that significantly reduce the search space in practice. Consider the for-loop in Line 5 of Algorithm 1 enumerating the possible extensions of τ . This enumeration can be sidestepped by the following two greedy approaches.


The enumeration of Line 5 then proceeds only if neither of the above two techniques can be applied for τ . This extension of VerifySC preserves completeness (not only when used during SMC, but in general), and it can be significantly faster in practice. For clarity of presentation we do not fully formalize this extended version, as its worst-case complexity remains the same.

**2. Closure.** We introduce *closure*, a low-cost filter for early detection of VSC instances (X,GoodW) with no witness. The notion of closure, its beneficial properties and construction algorithms are well-studied for the *reads-from consistency verification* problems [21,22,31], i.e., problems where a desired reads-from function is provided as input instead of a desired good-writes function GoodW. Further, the work of [23] studies closure with respect to a good-writes function, but only for partial orders of Mazurkiewicz width 2 (i.e., for partial orders with no triplet of pairwise conflicting and pairwise unordered events). Here we define closure for all good-writes instances (X,GoodW), with the underlying partial order (in our case, the program order PO) of arbitrary Mazurkiewicz width.

Given a VSC instance (X,GoodW), its closure <sup>P</sup>(X) is the weakest partial order that refines the program order (<sup>P</sup> PO|X) and further satisfies the following conditions. Given a read <sup>r</sup> ∈ R(X), let Cl(r) = GoodW(r)∩VisibleW<sup>P</sup> (r). The following must hold.


If (X,GoodW) has no closure (i.e., there is no <sup>P</sup> with the above conditions), then (X,GoodW) provably has no witness. If (X,GoodW) has closure <sup>P</sup>, then each witness <sup>τ</sup> of VSC(X,GoodW) provably refines <sup>P</sup> (i.e., <sup>τ</sup> <sup>P</sup>).

Finally, we explain how closure can be used by VerifySC. Given an input (X,GoodW), the closure procedure is carried out before VerifySC is called. Once the closure <sup>P</sup> of (X,GoodW) is constructed, since each solution of VSC(X,GoodW) has to refine <sup>P</sup>, we restrict VerifySC to only consider sequences refining P. This is ensured by an extra condition in Line 5 of Algorithm 1, where we proceed with an event e only if it is minimal in P restricted to events not yet in the sequence. This preserves completeness, while further reducing the search space to consider for VerifySC.

**3. VerifySC Guided by Auxiliary Trace.** In our SMC approach, each time we generate a VSC instance (X,GoodW), we further have available an auxiliary trace σ-. In σ-, either all-but-one, or all, good-writes conditions of GoodW are satisfied. If all good writes in GoodW are satisfied, we already have <sup>σ</sup> as a witness of (X,GoodW) and hence we do not need to run VerifySC at all. On the other hand, if case all-but-one are satisfied, we use <sup>σ</sup> to guide the search of VerifySC, as described below. of the worklist <sup>S</sup> in Algorithm 1. We use the auxiliary trace <sup>σ</sup>with E(σ-

We guide the search by deciding the order in which we process the sequences ) = X. We use S as a last-in-first-out stack, that way we search for a witness in a depthfirst fashion. Then, in Line 5 of Algorithm 1 we enumerate the extension events in the reverse order of how they appear in <sup>σ</sup>-. We enumerate in reverse order, as each resulting extension is pushed into our worklist S, which is a stack (last-infirst-out). As a result, in Line 3 of the subsequent iterations of the main while loop, we pop extensions from <sup>S</sup> in order induced by <sup>σ</sup>-.

#### **5 Stateless Model Checking**

We are now ready to present our SMC algorithm RVF-SMC that uses RVF to model check a concurrent program. RVF-SMC is a sound and complete algorithm for local safety properties, i.e., it is guaranteed to discover all local states that each thread visits.

RVF-SMC is a recursive algorithm. Each recursive call of RVF-SMC is argumented by a tuple (X,GoodW, σ, <sup>C</sup>) where:


(4) <sup>C</sup> : R → Threads <sup>→</sup> <sup>N</sup> is a partial function called *causal map* that tracks implicitly, for each read r, the writes that have already been considered as reads-from sources of r.

Further, we maintain a function ancestors: <sup>R</sup>(X) → {true, false}, where for each read <sup>r</sup> ∈ R(X), ancestors(r) stores a boolean *backtrack signal* for <sup>r</sup>. We now provide details on the notions of causal maps and backtrack signals.

**Causal Maps.** The causal map C serves to ensure that no more than one maximal trace is explored per RVF partitioning class. Given a read <sup>r</sup> <sup>∈</sup> enabled(σ) enabled in a trace <sup>σ</sup>, we define forbids<sup>C</sup> <sup>σ</sup>(r) as the set of writes in <sup>σ</sup> such that <sup>C</sup> forbids <sup>r</sup> to read-from them. Formally, forbids<sup>C</sup> <sup>σ</sup>(r) = <sup>∅</sup> if <sup>r</sup> <sup>∈</sup> dom(C), otherwise forbids<sup>C</sup> <sup>σ</sup>(r) = {<sup>w</sup> ∈ W(σ) <sup>|</sup> <sup>w</sup> is within first <sup>C</sup>(r)(thr(w)) events of <sup>σ</sup>thr}. We say that a trace <sup>σ</sup> *satisfies* <sup>C</sup> if for each <sup>r</sup> ∈ R(σ) we have RF<sup>σ</sup>(r) <sup>∈</sup> forbids<sup>C</sup> <sup>σ</sup>(r). trace σ-

**Backtrack Signals.** Each call of RVF-SMC (with its GoodW) operates with a satisfying GoodW that has only reads as enabled events. Consider one of those enabled reads <sup>r</sup> <sup>∈</sup> enabled(σ-). Each maximal trace satisfying GoodW shall contain r, and further, one of the following two cases is true: write of W(σ-


Whenever we can prove that the first above case is true for r, we can use this fact to prune away some recursive calls of RVF-SMC while maintaining completeness. Specifically, we leverage the following crucial lemma, and present the proof in Appendix D of [30]. **Lemma 1.** *Consider a call* RVF-SMC(X,GoodW, σ, <sup>C</sup>) *and a trace* <sup>σ</sup><sup>σ</sup> *maximally such that no event of the extension is a read. Let* <sup>r</sup> <sup>∈</sup> enabled(σ-

 *extending* ) *such that* <sup>r</sup> <sup>∈</sup> dom(C)*. If there exists a trace* <sup>σ</sup> *that (i) satisfies* GoodW *and* <sup>C</sup>*, and (ii) contains* <sup>r</sup> *with* RF<sup>σ</sup>- (r) ∈ W(σ-)*, then there exists a trace* σ *that (i) satisfies* GoodW *and* <sup>C</sup>*, (ii) contains* <sup>r</sup> *with* RF<sup>σ</sup>(r) ∈ W(σ-)*, and (iii) contains a write* <sup>w</sup> ∈ W(σ-) *with* r <sup>w</sup> *and* thr(r) <sup>=</sup> thr(w)*.* read <sup>r</sup> <sup>∈</sup> enabled(σ-

We then compute a boolean *backtrack signal* for a given RVF-SMC call and ) to capture satisfaction of the consequent of Lemma 1. If the computed backtrack signal is false, we can safely stop the RVF-SMC exploration of this specific call and backtrack to its recursion parent.

**Algorithm.** We are now ready to describe our algorithm RVF-SMC in detail, Algorithm <sup>2</sup> captures the pseudocode of RVF-SMC(X,GoodW, σ, <sup>C</sup>). First, in Line <sup>1</sup> we extend <sup>σ</sup> to <sup>σ</sup> maximally such that no event of the extension is a read. Then in Lines 2–5 we update the backtrack signals for ancestors of our current recursion call. After this, in Lines 6–11 we construct a sequence of reads enabled in σ-. Finally, we proceed with the main while-loop in Line 13. In each while-loop iteration we process an enabled read r (Line 14), and we perform no more while-loop iterations in case we receive a false backtrack signal for <sup>r</sup>.

# **Algorithm 2:** RVF-SMC(X,GoodW, σ, <sup>C</sup>)

```
Input: Proper set of events X, good-writes function GoodW, valid trace σ that
        is a witness of (X, GoodW), causal map C.
1 σ-
   ← σ ◦ σ where σ extends σ maximally such that no event of σ is a read
2 foreach w ∈ E(σ) do // All extension events are writes
3 foreach r ∈ dom(ancestors) do // All ancestor mutations are reads
4 if r -
         w and thr(r) = thr(w) then // Potential new source for r to read-from
5 ancestors(r) ← true // Set backtrack signal to true
6 mutate ←  // Construct a sequence of enabled reads
7 foreach r ∈ enabled(σ-
                   ) do // Enabled events in σ-
                                                       are reads
8 if r ∈ dom(C) then // Causal map C is defined for r
9 mutate ← mutate ◦ r // Insert r to the end of mutate
10 else // Causal map C is undefined for r
11 mutate ← r ◦ mutate // Insert r to the beginning of mutate
12 backtrack ← true
13 while backtrack = true and mutate =  do
14 r ← pop front of mutate // Process next read of mutate
15 if r ∈ dom(C) then
16 backtrack ← false
17 Fr ← VisibleWPO|E(σ-
                   )(r) \ forbidsC
                            σ-
                             (r) // Visible writes not forbidden by C
18 Dr ← {valσ-
            (w) : w ∈ Fr} // The set of values that r may read
19 foreach v ∈ Dr do // Process each value
20 X-
        ← X ∪ E(σ-
                ) ∪ {r} // New event set
21 GoodW-
            ← GoodW ∪ {(r, { w ∈ Fr | valσ-
                                   (w) = v })} // New good-writes
22 σ-
       ← VerifySC(X-

                   , GoodW-

                         ) // VerifySC guided by σ-
                                                            ◦ r
23 if σ-
         = ⊥ then // (X-

                                              , GoodW-

                                                    ) has a witness
24 C-
        ← C
25 ancestors(r) ← backtrack // Record ancestor
26 RVF-SMC(X-

                 , GoodW-

                        , σ-

                          , C-

                            )
27 backtrack ← ancestors(r) // Retrieve backtrack signal
28 delete r from ancestors // Unrecord ancestor
29 foreach thr ∈ Threads do // Update causal map C(r) for each thread
30 C(r)(thr) ← |E(σ-
                 )thr| // Number of events of thr in σ-
```
When processing r, first we collect its viable reads-from sources in Line 17, then we group the sources by value they write in Line 18, and then in iterations of the for-loop in Line 19 we consider each value-group. In Line 20 we form the event set, and in Line 21 we form the good-write function that designates the value-group as the good-writes of r. In Line 22 we use VerifySC to generate a witness, and in case it exists, we recursively call RVF-SMC in Line 26 with the newly obtained events, good-write constraint for r, and witness.

To preserve completeness of RVF-SMC, the backtrack-signals technique can be utilized only for reads <sup>r</sup> with undefined causal map <sup>r</sup> <sup>∈</sup> dom(C) (cf. Lemma 1). The order of the enabled reads imposed by Lines 6–11 ensures that subsequently, in iterations of the loop in Line 13 we first consider all the reads where we can utilize the backtrack signals. This is an insightful heuristic that often helps in practice, though it does not improve the worst-case complexity.

**Fig. 6.** RVF-SMC (Algorithm 2). Circles represent nodes of the recursion tree. Below each circle is its corresponding event set <sup>E</sup>(σ-) and the enabled reads (dashed grey). Writes with green background are good-writes (GoodW) of its corresponding-variable read. Writes with red background are forbidden by C for its corresponding-variable read. Dashed arrows represent recursive calls. (Color figure online)

*Example.* Figure 6 displays a simple concurrent program on the left, and its corresponding RVF-SMC (Algorithm 2) run on the right. We start with RVF-SMC(∅, <sup>∅</sup>, , <sup>∅</sup>) (A). By performing the extension (Line 1) we obtain the events and enabled reads as shown below (A). First we process read r<sup>1</sup> (Line 14). The read can read-from w<sup>1</sup> and w3, both write the same value so they are grouped together as good-writes of r1. A witness is found and a recursive call to (B) is performed. In (B), the only enabled event is r2. It can read-from w<sup>2</sup> and w4, both write the same value so they are grouped for r2. A witness is found, a recursive call to (C) is performed, and (C) concludes with a maximal trace. Crucially, in (C) the event w<sup>5</sup> is discovered, and since it is a potential new reads-from source for r1, a backtrack signal is sent to (A). Hence after RVF-SMC backtracks to (A), in (A) it needs to perform another iteration of Line 13 while-loop. In (A), first the causal map <sup>C</sup> is updated to forbid <sup>w</sup><sup>1</sup> and <sup>w</sup><sup>3</sup> for <sup>r</sup>1. Then, read <sup>r</sup><sup>2</sup> is processed from (A), creating (D). In (D), r<sup>1</sup> is the only enabled event, and w<sup>5</sup> is its only C-allowed write. This results in (E) which reports a maximal trace. The algorithm backtracks and concludes, reporting two maximal traces in total. **Theorem 2.** *Consider a concurrent program* <sup>H</sup> *of* <sup>k</sup> *threads and* <sup>d</sup> *variables, with* <sup>n</sup> *the length of the longest trace in* <sup>H</sup>*.* RVF-SMC *is a sound and complete algorithm for local safety properties in* H*. The time complexity of* RVF-SMC *is* <sup>k</sup><sup>d</sup> · <sup>n</sup>O(k) · <sup>β</sup>*, where* <sup>β</sup> *is the size of the* RVF *trace partitioning of* <sup>H</sup>*.*

**Novelties of the Exploration.** Here we highlight some key aspects of RVF-SMC. First, we note that RVF-SMC constructs the traces incrementally with each recursion step, as opposed to other approaches such as [15,22] that always work with maximal traces. The reason of incremental traces is technical and has to do with the value-based treatment of the RVF partitioning. We note that the other two value-based approaches [23,24] also operate with incremental traces. However, RVF-SMC brings certain novelties compared to these two methods. First, the exploration algorithm of [24] can visit the same class of the partitioning (and even the same trace) an exponential number of times by different recursion branches, leading to significant performance degradation. The exploration algorithm of [23] alleviates this issue using the causal map data structure, similar to our algorithm. The causal map data structure can provably limit the number of revisits to polynomial (for a fixed number of threads), and although it offers an improvement over the exponential revisits, it can still affect performance. To further improve performance in this work, our algorithm combines causal maps with a new technique, which is the backtrack signals. Causal maps and backtrack signals together are very effective in avoiding having different branches of the recursion visit the same RVF class.

**Beyond RVF Partitioning.** While RVF-SMC explores the RVF partitioning in the worst case, in practice it often operates on a partitioning coarser than the one induced by the RVF equivalence. Specifically, RVF-SMC may treat two traces <sup>σ</sup><sup>1</sup> and <sup>σ</sup><sup>2</sup> with same events (E(σ1) = <sup>E</sup>(σ2)) and value function (val<sup>σ</sup><sup>1</sup> <sup>=</sup> val<sup>σ</sup><sup>2</sup> ) as equivalent even when they differ in some causal orderings (→<sup>σ</sup><sup>1</sup> |R = →<sup>σ</sup><sup>2</sup> |R). To see an example of this, consider the program and the RVF-SMC run in Fig. 6. The recursion node (C) spans all traces where (i) r<sup>1</sup> reads-from either w<sup>1</sup> or w3, and (ii) r<sup>2</sup> reads-from either w<sup>2</sup> or w4. Consider two such traces σ<sup>1</sup> and <sup>σ</sup>2, with RF<sup>σ</sup><sup>1</sup> (r2) = <sup>w</sup><sup>2</sup> and RF<sup>σ</sup><sup>2</sup> (r2) = <sup>w</sup>4. We have <sup>r</sup>1→<sup>σ</sup><sup>1</sup> <sup>r</sup><sup>2</sup> and <sup>r</sup>1→<sup>σ</sup><sup>2</sup> <sup>r</sup>2, and yet σ<sup>1</sup> and σ<sup>2</sup> are (soundly) considered equivalent by RVF-SMC. Hence the RVF partitioning is used to upper-bound the time complexity of RVF-SMC. We remark that the algorithm is always sound, i.e., it is guaranteed to discover all thread states even when it does not explore the RVF partitioning in full.

#### **6 Experiments**

In this section we describe the experimental evaluation of our SMC approach RVF-SMC. We have implemented RVF-SMC as an extension in Nidhugg [27], a state-of-the-art stateless model checker for multithreaded C/C++ programs that operates on LLVM Intermediate Representation. First we assess the advantages of utilizing the RVF equivalence in SMC as compared to other trace equivalences. Then we perform ablation studies to demonstrate the impact of the backtrack signals technique (cf. Sect. 5) and the VerifySC heuristics (cf. Sect. 4.2).

In our experiments we compare RVF-SMC with several state-of-the-art SMC tools utilizing different trace equivalences. First we consider VC-DPOR [23], the SMC approach operating on the value-centric equivalence. Then we consider Nidhugg/rfsc [22], the SMC algorithm utilizing the reads-from equivalence. Further we consider DC-DPOR [21] that operates on the data-centric equivalence, and finally we compare with Nidhugg/source [15] utilizing the Mazurkiewicz equivalence.<sup>1</sup> The works of [22] and [32] in turn compare the Nidhugg/rfsc algorithm with additional SMC tools, namely GenMC [28] (with reads-from equivalence), RCMC [29] (with Mazurkiewicz equivalence), and CDSChecker [33] (with Mazurkiewicz equivalence), and thus we omit those tools from our evaluation.

There are two main objectives to our evaluation. First, from Sect. 3 we know that the RVF equivalence can be up to exponentially coarser than the other equivalences, and we want to discover how often this happens in practice. Second, in cases where RVF does provide reduction in the trace-partitioning size, we aim to see whether this reduction is accompanied by the reduction in the runtime of RVF-SMC operating on RVF equivalence.

**Setup.** We consider 119 benchmarks in total in our evaluation. Each benchmark comes with a scaling parameter, called the *unroll* bound. The parameter controls the bound on the number of iterations in all loops of the benchmark. For each benchmark and unroll bound, we capture the number of explored maximal traces, and the total running time, subject to a timeout of one hour. In Appendix E of [30] we provide further details on our setup.

**Fig. 7.** Runtime and traces comparison of RVF-SMC with VC-DPOR.

<sup>1</sup> The MCR algorithm [24] is beyond the experimental scope of this work, as that tool handles Java programs and uses heavyweight SMT solvers that require fine-tuning.

**Fig. 8.** Runtime and traces comparison of RVF-SMC with Nidhugg/rfsc.

**Fig. 9.** Runtime and traces comparison of RVF-SMC with DC-DPOR.

**Fig. 10.** Runtime and traces comparison of RVF-SMC with Nidhugg/source.

**Results.** We provide a number of scatter plots summarizing the comparison of RVF-SMC with other state-of-the-art tools. In Fig. 7, Fig. 8, Fig. 9 and Fig. 10 we provide comparison both in runtimes and explored traces, for VC-DPOR, Nidhugg/rfsc, DC-DPOR, and Nidhugg/source, respectively. In each scatter plot, both its axes are log-scaled, the opaque red line represents equality, and the two semi-transparent lines represent an order-of-magnitude difference. The points are colored green when RVF-SMC achieves trace reduction in the underlying benchmark, and blue otherwise.

**Discussion: Significant Trace Reduction.** In Table 1 we provide the results for several benchmarks where RVF achieves significant reduction in the tracepartitioning size. This is typically accompanied by significant runtime reduction, allowing is to scale the benchmarks to unroll bounds that other tools cannot handle. Examples of this are 27 Boop4 and scull loop, two toy Linux kernel drivers.

In several benchmarks the number of explored traces remains the same for RVF-SMC even when scaling up the unroll bound, see 45 monabsex1, reorder <sup>5</sup> and singleton in Table 1. The singleton example is further interesting, in that while VC-DPOR and DC-DPOR also explore few traces, they still suffer in runtime due to additional redundant exploration, as described in Sects. 1 and 5.


**Table 1.** Benchmarks with trace reduction achieved by RVF-SMC. The unroll bound is shown in the column **U**. Symbol "–" indicates one-hour timeout. Bold-font entries indicate the smallest numbers for respective benchmark and unroll.


**Table 2.** Benchmarks with little-to-no trace reduction by RVF-SMC. Symbol † indicates that a particular benchmark operation is not handled by the tool.

**Discussion: Little-to-no Trace Reduction.** Table 2 presents several benchmarks where the RVF partitioning achieves little-to-no reduction. In these cases the well-engineered Nidhugg/rfsc and Nidhugg/source dominate the runtime.

**RVF-SMC Ablation Studies.** Here we demonstrate the effect that follows from our RVF-SMC algorithm utilizing the approach of backtrack signals (see Sect. 5) and the heuristics of VerifySC (see Sect. 4.2). These techniques have no effect on the number of the explored traces, thus we focus on the runtime. The left plot of Fig. 11 compares RVF-SMC as is with a RVF-SMC version that does not utilize the backtrack signals (achieved by simply keeping the backtrack flag in Algorithm <sup>2</sup> always true). The right plot of Fig. <sup>11</sup> compares RVF-SMC as is with a RVF-SMC version that employs VerifySC without the closure and auxiliary-trace heuristics. We can see that the techniques almost always result in improved runtime. The improvement is mostly within an order of magnitude, and in a few cases there is several-orders-of-magnitude improvement.

Finally, in Fig. 12 we illustrate how much time during RVF-SMC is typically spent on VerifySC (i.e., on solving VSC instances generated during RVF-SMC).

**Fig. 11.** Ablation studies of RVF-SMC. The left plot compares RVF-SMC with and without backtrack signals. The right plots compares RVF-SMC with and without the closure and auxiliary-trace heuristics of Sect. 4.2.

**Fig. 12.** Histogram that illustrates the percentage of time spent solving VSC instances during RVF-SMC.

#### **7 Conclusions**

In this work we developed RVF-SMC, a new SMC algorithm for the verification of concurrent programs using a novel equivalence called *reads-value-from (RVF)*. On our way to RVF-SMC, we have revisited the famous VSC problem [25]. Despite its NP-hardness, we have shown that the problem is parameterizable in k+d (for k threads and d variables), and becomes even fixed-parameter tractable in d when k is constant. Moreover we have developed practical heuristics that solve the problem efficiently in many practical settings.

Our RVF-SMC algorithm couples our solution for VSC to a novel exploration of the underlying RVF partitioning, and is able to model check many concurrent programs where previous approaches time-out. Our experimental evaluation reveals that RVF is very often the most effective equivalence, as the underlying partitioning is exponentially coarser than other approaches. Moreover, RVF-SMC generates representatives very efficiently, as the reduction in the partitioning is often met with significant speed-ups in the model checking task. Interesting future work includes further improvements over the VSC, as well as extensions of RVF-SMC to relaxed memory models.

**Acknowledgments.** The research was partially funded by the ERC CoG 863818 (ForM-SMArt) and the Vienna Science and Technology Fund (WWTF) through project ICT15-003.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Gobra: Modular Specification and Verification of Go Programs**

Felix A. Wolf<sup>1</sup> , Linard Arquint<sup>1</sup> , Martin Clochard<sup>1</sup>, Wytse Oortwijn<sup>2</sup> , Jo˜ao C. Pereira1(B) , and Peter M¨uller<sup>1</sup>

> <sup>1</sup> Department of Computer Science, ETH Zurich, Zurich, Switzerland *{*felix.wolf,linard.arquint,martin.clochard, joao.pereira,peter.mueller*}*@inf.ethz.ch <sup>2</sup> ESI (TNO), Eindhoven, The Netherlands wytse.oortwijn@tno.nl

**Abstract.** Go is an increasingly-popular systems programming language targeting, especially, concurrent and distributed systems. Go differentiates itself from other imperative languages by offering structural subtyping and lightweight concurrency through goroutines with messagepassing communication. This combination of features poses interesting challenges for static verification, most prominently the combination of a mutable heap and advanced concurrency primitives.

We present Gobra, a modular, deductive program verifier for Go that proves memory safety, crash safety, data-race freedom, and userprovided specifications. Gobra is based on separation logic and supports a large subset of Go. Its implementation translates an annotated Go program into the Viper intermediate verification language and uses an existing SMT-based verification backend to compute and discharge proof obligations.

**Keywords:** Separation logic · Program logics · Channel-based concurrency · Interfaces · Deductive verification · Automated verification

#### **1 Introduction**

Go is an increasingly popular systems programming language targeting, especially, concurrent and distributed systems such as web applications. It combines standard features of imperative languages, such as mutable heap data structures, with less common concepts, such as structural subtyping and lightweight concurrency through goroutines with message-passing communication.

This combination of features poses interesting challenges for static verification, most prominently the combination of a mutable heap and advanced concurrency primitives. Prior research on Go verification handles some of these features, but not their combination. For instance, Lange et al. [14,15] verify safety and

liveness of Go's message-passing, but do not consider functional properties about the heap state, whereas Perennial [4] supports heap data structures, but neither channels nor interfaces.

We present Gobra, an automated, modular verifier for heap-manipulating, concurrent Go programs. Gobra supports a large subset of Go, including Go's interfaces and primitive data structures, both of which have not been fully supported in previous work. Gobra verifies memory safety, crash safety, data-race freedom, and user-provided specifications. It takes as input a Go program annotated with assertions such as pre and postconditions and loop invariants. Verification proceeds by encoding the annotated programs into the intermediate verification language Viper [17] and then applying an existing SMT-based verifier. In case verification fails, Gobra reports at the level of the Go program which assertions it could not verify.

Gobra's assertion language builds on established concepts: Gobra uses separation logic style permissions [19] to reason locally about heap data structures. It supports recursive predicates and specification methods to abstract over (possibly unbounded) data structures and their contents. In particular, Gobra has first-class predicates that enable a natural specification of concurrency primitives, for instance, to parameterize a lock by an invariant.

Gobra is intended for the verification of substantial, real-world code, and is currently used to verify the Go implementation of the SCION internet architecture [23]. Our tool paper makes the following technical contributions:


**Outline.** We demonstrate key features of Gobra on examples (Sect. 2), give an overview of the encoding into Viper (Sect. 3), and provide an experimental evaluation of Gobra (Sect. 4). Lastly, Sect. 5 discusses related work and concludes.

#### **2 Gobra in a Nutshell**

This section illustrates Gobra's specification language on simple examples and shows how we handle interfaces and concurrency.

#### **2.1 Basics**

Gobra uses a variant of separation logic [19] in order to reason about mutable heap data structures and concurrency. Separation logics associate an access permission with each heap location. Access permissions are held by method executions and transferred between methods upon call and return. A method may access a location only if it holds the associated permission. Permission to a shared location v is denoted in Gobra by acc(&v), which is analogous to separation logic's <sup>v</sup>-→ . Gobra provides an expressive permission model supporting fractional permissions [3] to allow concurrent read accesses while still ensuring exclusive writes, (recursive) predicates to denote access to unbounded data structures, and quantified permissions (also called iterated separating conjunction) to express permissions to random-access data structures such as arrays and slices.

```
1 requires ∀ k int :: 0 ≤ k < len (s) =⇒ acc(&s[k])
2 ensures ∀ k int :: 0 ≤ k < len (s) =⇒ acc(&s[k])
3 ensures ∀ k int :: 0 ≤ k < len (s) =⇒ s[k] == old(s[k]) + n
4 func incr (s [] int , n int ) {
5 invariant 0 ≤ i ≤ len (s)
6 invariant ∀ k int :: 0 ≤ k < len (s) =⇒ acc(&s[k])
7 invariant ∀ k int :: i ≤ k < len (s) =⇒ s[k] == old(s[k])
8 invariant ∀ k int :: 0 ≤ k < i =⇒ s[k] == old(s[k]) + n
9 for i := 0; i < len(s); i += 1 {
10 s[i] = s[i] + n
11 }
12 }
```
**Fig. 1.** A simple Gobra example showing method and loop contracts.

The example in Fig. 1 illustrates the use of permissions. Method incr increases all elements of a given slice s by an amount n. (Slices are data types that can intuitively be seen as shared arrays of variable length.) The method requires permission to all slice elements (via its precondition) and returns them to the caller (via its first postcondition).

Functional properties are expressed via standard assertions, which include side-effect free Go expressions (including calls to pure methods, as we explain below) as well as universal quantification and old-expressions to refer to the value an expression had in the pre-state of a method. In our example, the second postcondition uses these assertions to express the functional behavior of the method. The loop invariants are analogous to the method contracts and are needed for verification.

In Go, any memory location can either be *shared* or *exclusive*. Shared locations reside on the heap and can, thus, be accessed by multiple methods and threads; reasoning about shared locations requires permissions to ensure race freedom and to enable framing, i.e., preserving information across heap changes. On the other hand, exclusive locations are accessed exclusively by one method execution and may be allocated on the stack; they can be reasoned about as local variables. The Go compiler determines automatically whether a location is

shared or exclusive, for instance by determining whether its address is taken at some point of the execution. To make verification independent of a particular compiler analysis, Gobra requires shared locations to be decorated with an extra annotation @ at the declaration point, as illustrated by the following client of incr:

```
1 a@ := [4] int { 1, 2, 4, 8 }
2 incr(a[2:], 2)
3 assert a == [4] int { 1, 2, 6, 10 }
```
The first line declares a Go array a of fixed length 4, with values 1, 2, 4, and 8. This array is sliced on line 2 using the syntax a[2:], thereby omitting the first two elements of a from the created slice. Since a is used in a context in which it is sliced, it is a shared location, which is made explicit via the @ annotation. Consequently, the array creation will produce permissions to the array elements, which are required by incr's precondition. Omitting the @ annotation will cause a verification error.

#### **2.2 Interfaces**

Go supports polymorphism through *interfaces*, named sets of method signatures. Subtyping for interfaces is structural: a type implements an interface iff every method of the interface is implemented by the type. The subtype relationship is determined by the type checker, without any declarations from the programmer<sup>1</sup>.

Calls on an interface value are dynamically dispatched. In settings with nominal subtyping, dynamic dispatch is handled by proving behavioral subtyping [16]: each subtype declaration requires a proof that the specifications of subtype methods refine the specifications of the corresponding supertype methods. Since structural subtypes are not declared explicitly, we adapt this approach as follows.

Whenever a Go program assigns a value to a variable of an interface type, Gobra requires an *implementation proof*, that is, a proof that each method of the subtype satisfies the specification of the corresponding method in the interface. Implementation proofs are inferred automatically by Gobra in simple cases; userprovided implementation proofs are required especially when they include ghost operations, for instance, to manipulate predicates.

The example in Fig. 2 illustrates this approach. Interface stream (lines 1–8) declares an interface with two methods, hasNext and next. The latter may return values of an arbitrary type, which is denoted by an empty interface. Since interfaces do not contain an implementation, their specification must be fully abstract. To this end, stream introduces an abstract predicate memory, whose definition is provided by the subtypes of the interface. The functional behavior of interface methods can be expressed in terms of pure (that is, side-effect free) abstract methods, here, hasNext, which will also be defined in subtypes.

Next, lines 10–16 show an implementation of the interface in the form of a counter. The counter has a current f and maximum max value. As long as the

<sup>1</sup> For the sake of simplicity, we omit *embeddings*, Go's construct for delegation; an extension is straightforward.

```
1 type stream interface {
2 pred memory()
3 requires acc(memory(), _) // arbitrary fraction of memory()
4 pure hasNext() bool
5 requires memory() && hasNext()
6 ensures memory()
7 next() interface {}
8 }
9
10 type counter struct { f int ; max int }
11 requires acc(&x.f, _) && acc(&x.max , _)
12 pure func (x *counter) hasNext() bool { return x.f < x.max }
13 requires acc(&x.f) && acc(&x.max , 1/2) && x.hasNext()
14 ensures acc(&x.f) && acc(&x.max , 1/2) && x.f == old(x.f)+1
15 ensures typeOf(y) == int && y.( int) == old(x.f)
16 func (x *counter) next() (y interface {}) { x.f++; return x.f-1 }
17
18 pred (x *counter) memory() { acc(&x.f) && acc(&x.max) }
19 (*counter) implements stream {
20 pure (x *counter) hasNextProof () bool {
21 return unfolding acc(x.memory(), _) in x.hasNext()
22 }
23 (x *counter ) nextProof () (res interface {}) { ... }
24 }
```
**Fig. 2.** An interface specification for a stream (lines 1–8) together with an implementation (lines 10–16) and an implementation proof (lines 18–24). We write acc(p, \_) to denote an arbitrary, positive amount of predicate p, and simply p for acc(p, 1/1). At line 14, the fractional permission to &x.max entails that x.max is not modified.

maximum value is not reached, next will increase the current value. At line 16, an integer can be assigned to the empty interface since behavioral subtyping holds trivially. The specification at line 15 expresses that the returned interface value contains an integer with the old value of the f field.

The counter implementation is completely independent of the stream interface. Their connection is established only in the implementation proof (lines 18–24). This proof defines the memory predicate from the stream interface for receivers of type counter (line 18). Moreover, an implementation proof verifies that the specification of each method implementation refines the specification of the corresponding interface method. This proof checks that, assuming the precondition of an interface method, a call to the implementation method with identical arguments establishes the postcondition of the interface method. This format is enforced syntactically and permits ghost operations before and after the call to manipulate predicates. For instance, the proof on line 21 for hasNext temporarily unfolds the memory predicate to obtain permission to x, which is required by the implementation method, and conversely after the call.

Implementation proofs can be written explicitly, imported from other packages, and also inferred automatically when no explicit proof exists in the current scope. Currently, Gobra does not infer ghost operations such as the unfolding on line 21; our experiments suggest that already simple heuristics can deal with

many cases occurring in practice. For instance, many implementation proofs we have encountered follow the same pattern: First, the interface predicate instances of the precondition are unfolded. Second, the implementation method is called. Lastly, the interface predicate instances of the postcondition are folded. This pattern can be generated automatically to alleviate the annotation burden.

Gobra's implementation proofs enable one to reason about interfaces without enforcing subtype declarations in either the interface or the declaration, which would defeat the purpose of structural subtyping. This solution allows one to reason about dynamically-dispatched calls. For instance, the following code snippet verifies in Gobra:

```
1 x := &counter{0, 50}
2 var y stream = x
3 fold y.memory()
4 var z interface {} = y.next()
```
In particular, Gobra is able to determine that next's precondition hasNext() holds because y.hasNext() is equal to x.hasNext(), and the latter follows from the definition of hasNext (line 12) and the initial value of x.f. This intuitive reasoning is enabled by an intricate underlying encoding, which is not exposed to users. Users do not have to know how interface predicates are encoded and can treat interface predicates the same as any other separation-logic predicate.

#### **2.3 Concurrency**

Go supports concurrency through *goroutines*, lightweight threads started by prefixing a method call with the go keyword. Go offers the usual synchronization primitives, but goroutines idiomatically synchronize via *channels*. Buffered channels provide asynchronous communication, where sending a message blocks only when the buffer is full. Unbuffered channels offer rendez-vouz communication.

Gobra enables verification of concurrent programs by associating Go's synchronization primitives with predicates that do not only express properties of data but also express how permissions to shared memory get transferred between threads. For instance, lock invariants may include properties as well as permissions to the data protected by the lock, and channel invariants include properties and permissions of the data sent over a channel. These invariants are specified via ghost operations when the synchronization primitive is initialized.

Figure 3 illustrates Gobra's concurrency support using an excerpt from a parallel search-and-replace algorithm (see the full paper [22] for the complete example). Method searchAndReplace spawns a series of worker threads and then sends each of them a chunk of the input slice to process. The worker threads are joined via a wait group wg. Method worker implements the worker threads.

Gobra associates channels (like c in the example) with a predicate to specify properties and permissions of the sent data. The call c.Init(...) on line 10 takes this predicate as an argument. As expressed on line 2, it includes permissions to the chunk a worker operates on. For synchronous channels, an additional predicate can specify permissions transferred in the opposite direction, from the

```
1 pred messagePerm(wg *sync.WaitGroup , chunk [] int , x, y int ) {
2 ( ∀ i int :: 0 ≤ i < len (chunk) =⇒ acc(&chunk[i]) ) && ...
3 }
4 requires ∀ i int :: 0 ≤ i < len (s) =⇒ acc(&s[i])
5 func searchAndReplace(s [] int , x, y int ) {
6 var wg@ sync.WaitGroup
7 ghost wg.Init()
8 c := make ( chan [] int ,4)
9 // predicate -name{..., _, ...} is syntax for partial application
10 ghost c.Init(messagePerm {&wg , _, x, y})
11 // Spawn workers
12 invariant acc(c.RecvChannel(), _)
13 invariant c.RecvGotPerm () == messagePerm {&wg, _, x, y}
14 for i := 0; i < numOfWorkers; i++ { go worker(c, wg , x, y) }
15 // Split slice into chunks , which are sent to workers
16 invariant c.SendChannel()
17 invariant c.SendGivenPerm () == messagePerm {&wg, _, x, y}
18 invariant ∀ i int :: offset ≤ i < len (s) =⇒ acc(&s[i])
19 invariant ... // constraints on offset and nextOffset
20 for offset := 0; offset != len (s); offset = nextOffset {
21 nextOffset = ...
22 wg.Add(1)
23 fold messagePerm {&wg , _, x, y}(s[offset:nextOffset ])
24 c <- s[offset:nextOffset]
25 }
26 wg.Wait()
27 }
28 requires acc(c.RecvChannel(), _)
29 requires c.RecvGotPerm () == messagePerm{wg , _, x, y};
30 func worker(c <- chan [] int , wg *sync.WaitGroup , x, y int ) {
31 invariant acc(c.RecvChannel(), _)
32 invariant c.RecvGotPerm () == messagePerm{wg , _, x, y};
33 invariant ok =⇒ messagePerm{wg , _, x, y}(chunk)
34 for chunk , ok := <- c; ok; chunk , ok = <-c {
35 unfold messagePerm{wg, _, x, y}(chunk)
36 ... // replace x with y in chunk
37 wg.Done() // same as wg.Add(-1)
38 }
39 }
```
**Fig. 3.** Excerpt showing goroutines, channels, and wait groups. The code spawns workers (line 14), sends slice chunks through a channel to the workers (line 24), and then waits on a wait group (line 26). A worker receives a chunk (line 34), processes it, and then notifies the wait group (line 37). For the sake of simplicity, some details were omitted.

receiver to the sender. Initializing a channel also creates send and receive permissions for the channel, which are used to control which threads may access it. In our example, we transfer a fraction of the receive permission to each worker (line 28).

The workers receive permission to the chunk they operate on via a message sent on line 24 and received on line 34. The transfer back is orchestrated through a wait group, which implements an abstract shared counter. Wait groups are used as follows: The main thread adds to the counter the number of units of work to be done in spawned goroutines (line 22). Each spawned goroutine decreases the counter each time a unit of work is done (via a call to Done, line 37). The

master can await the counter to reach 0 via a call to Wait (line 26). Gobra uses dedicated permissions to express the obligation of a thread to perform units of work before decreasing the counter; each time this happens, permissions are transferred to the wait group and, eventually to the main thread calling Wait. We omit the details here for brevity.

In our example, this mechanism allows the main thread to recover the permissions to the entire slice once the workers have terminated. The example in Fig. 3 illustrates only the permission aspect of the verification. Functional correctness can be verified easily based on the explained machinery, by specifying a stronger channel invariant that includes the work obligation for each worker. We omit the details here, but see the full paper [22] for the complete example.

### **3 Encoding**

Gobra encodes an annotated Go program into a Viper program verifying only if the input program is correct. Many features of Gobra are also present in Viper, making parts of the encoding straightforward. For instance, methods, pure methods, and predicates are encoded to their Viper counterpart. Viper's permission model (including fractions, wildcards, and quantifiers) is similar to Gobra's, but memory is represented differently; Viper's heap is object-based, where each object contains all declared fields. Viper's fields store primitive values (including references). To encode Go's compound values such as structs, arrays, slices, and interface values, we use Viper's mechanism to declare mathematical types (such as tuples) using uninterpreted types, uninterpreted functions, and appropriate axioms. Exclusive Go values are directly represented using these mathematical types. For shared values, there is an indirection via the Viper heap to permit aliasing and apply permission-based reasoning.

**Interfaces.** As explained in Sect. 2.2, our treatment of Go interfaces relies on interface predicates, specification methods, and implementation proofs. We explain how we handle the former two here; based on this encoding, the encoding of implementation proofs is analogous to methods.

Intuitively, we encode interface predicates as a case split over all possible implementations. All implementations not present in the current scope are subsumed by an abstract default case. Consequently, adding an implementation does not invalidate existing proofs, which enables modular reasoning. The predicate for the stream example (Fig. 2) is encoded as follows:

```
predicate memory(x: -
                      interface {}) {
  -
  typeOf(x) == *counter ? -
                             acc(x.(*counter)) : unknownMemory(x)
}
predicate unknownMemory(x: -
                             interface {})
function hasNext(x: -
                      interface {}) returns (y: -
                                                  bool)
  req -
       acc(x.memory(), _)
  ens -
       typeOf(x) == *counter =⇒ y == hasNextProof(-
                                                      x.(*counter))
```
The body of the predicate branches on the dynamic type of x, with a single case for the (only) given implementation. The abstract predicate unknownMemory encodes the default case. The encoding of pure methods such as hasNext uses an analogous case split, but uses hasNextProof , which is part of the implementation proof (Fig. 2 line 20) and couples the interface and implementation method. Our encoding of interface predicates is an instance of an *abstract predicate family* [18]. For Go, we have crafted a variant that is well-suited for implementation proofs, pure interface methods, and structural subtyping.

**First-Class Predicates.** Our support for concurrency uses first-class predicates, for instance, to specify channel invariants (see Sect. 2.3). We encode firstclass predicate values as mathematical types, using defunctionalization. Predicate instances are represented by abstract predicates that take the predicate value as an argument. First-class predicates enable us to use library stubs to support concurrency primitives such as mutexes and wait groups. These stubs allow us to encode the use of these concurrency primitives via standard method calls. Go's native channel operations are represented analogously.

#### **4 Implementation and Evaluation**

The Gobra implementation consists of a parser and type checker for annotated Go programs and a translation of those programs into the Viper intermediate verification language. The resulting Viper program is verified using Viper's symbolic execution backend, which in turn uses the Z3 SMT solver [7]. Verification errors are translated back to the Go level, such that users are not exposed to the internal encodings. Users never have to inspect the encoding. Error messages contain the failing assertion and a reason describing why the assertion failed. Gobra's test suite contains 407 verification tests (with and without errors) with a total of 10'030 LOCs (Go code and annotations) that take 14.9 min to verify.

We evaluated Gobra on 14 interesting verification problems, which include well-known algorithms and data structures, and cover Go's main features, such as interfaces (Examples 7–9) and concurrency primitives (Examples 13 and 14), including goroutines, mutexes, wait groups, and channels. For each example, Gobra verifies memory safety and functional correctness properties. To assess Gobra's performance on failing verifications, we have additionally constructed two incorrect variations of each example, one with a seeded error in the specification and one in the implementation.

All experiments were executed on a warmed-up JVM on a MacBook Pro with a 2.3 GHz 8-Core Intel Core i9 CPU and 32 GB of RAM, running macOS 11.1 and OpenJDK 11. For each experiment, we measured its verification time using Viper's symbolic execution backend and averaged the duration of twelve executions, excluding the slowest and fastest outlier.

Figure 4 summarizes the results, including the required annotations and verification times for the three variants of each example. The annotation overhead


**Fig. 4.** Experimental results. For each experiment, we list the number of lines of Go code (LOC), number of lines of specification and proof annotations (Spec), and the average verification time in seconds for correct examples (T), errors in the specification (Tspec error), and errors in the implementation (Timpl error). A line containing both, code and annotations, is counted as one line of Go code and one line of annotation.

ranges between 0.3 and 3.1 lines of annotations per line of code, which is typical for SMT-based deductive verifiers. Verification times range between a second and a minute per example. The verification times are significantly higher when the verified code uses concurrency features; these examples require quantitatively more and more-complex specifications, which complicates reasoning. Lastly, there is hardly any difference between successful and failed verification attempts. Consistent performance is crucial when verifiers are used interactively, where users run them frequently, especially on programs that do not yet verify.

#### **5 Related Work and Conclusion**

Besides Gobra, we are aware of two other verification approaches for Go. Perennial [4] reasons about concurrent, crash-safe systems. Their core techniques are an extension to the Iris framework [13] and independent of Go. They connect their theory to Go programs with Goose, a shallow embedding of Go into Coq [5], which proves that Go code complies with a given transition system. In contrast to Gobra, Perennial does not support core Go features such as channels and interfaces.

Several prior works [9,14,15] infer behavioral types [12] to reason about Go's channel-based message passing. After they infer behavioral types for a given program, they check safety and liveness properties on the inferred types, using model checkers such as mCRL2 [6]. Some works use additional analyses to strengthen the provided guarantees. Lange et al. [15] add a termination analysis to enable one to verify unbounded properties under certain conditions. Gabet

and Yoshida [9] extend this work by inferring behavioral types on shared variables and locks to additionally reason about data-race freedom, lock safety, and lock liveness. The approaches by Lange et al. [15] and Gabet and Yoshida [9] are vastly different from Gobra. They do not verify code contracts, but instead verify global properties such as deadlock and data-race freedom. Their automation is high and annotation overhead minimal, but their analyses are not modular and do not verify functional properties of code. Furthermore, they do not verify properties about the state of the heap.

There are some prior works that can handle channel-based concurrency and heap-manipulating programs, but these do not apply directly to Go. Villard et al. [20] introduce a powerful contract mechanism to specify protocols that channels must adhere to. Their channel specification language is more expressive than the one presented in this paper. Their contracts are finite state machines and thus can have multiple phases. However, their channels are always shared between two peers whereas Go supports more advanced concurrency patterns where both channel endpoints are shared between an unbounded number of peers. Actris [10,11] is a concurrent separation logic built on top of the Iris framework to reason about session types in an interactive theorem prover. Actris can go beyond two peers, but to do so, it requires a memory model that is incompatible with Go's memory model. Actris models the sharing of channel endpoints via Iris' ghost locks, which to our knowledge, implies sequentialization of sends, and dually receives, which is not guaranteed by Go's memory model.

Gobra's verification logic and encoding into Viper have been inspired by several other Viper-based verifiers, such as Nagini [8] for Python, Prusti [1] for Rust, and VerCors [2] for Java. None of these verifiers address the Go-specific features that Gobra supports.

**Conclusion.** We introduced Gobra, the first modular verifier for Go that supports reasoning about a crucial aspect of the language: the combination of channel-based concurrency and heap-manipulating constructs. Moreover, Gobra is the first verifier to support Go's version of interfaces and structural subtyping. In future work, we will expand the properties that can be verified with Gobra, in particular to liveness and hyper-properties. Furthermore, we are applying Gobra to verify the implementation of a full-fledged network router [23]. Gobra is hosted on Github at https://github.com/viperproject/gobra.

**Acknowledgements.** This project has received funding from the European Union's Horizon 2020 research and innovation program within the framework of the NGI-POINTER Project funded under grant agreement No. 871528.

#### **References**

1. Astrauskas, V., M¨uller, P., Poli, F., Summers, A.J.: Leveraging rust types for modular specification and verification. In: Object-Oriented Programming Systems, Languages, and Applications (OOPSLA), vol. 3, pp. 147:1–147:30. ACM (2019)


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Delay-Bounded Scheduling Without Delay!**

Andrew Johnson1(B) and Thomas Wahl1,2(B)

<sup>1</sup> Northeastern University, Boston, MA 02115, USA aj3189@princeton.edu <sup>2</sup> GrammaTech Inc., Bethesda, USA

**Abstract.** We consider the broad problem of analyzing safety properties of asynchronous concurrent programs under arbitrary thread interleavings. *Delay-bounded deterministic scheduling*, introduced in prior work, is an efficient bug-finding technique to curb the large cost associated with full scheduling nondeterminism. In this paper we first present a technique to *lift the delay bound* for the case of finite-domain variable programs, thus adding to the efficiency of bug detection the ability to prove safety of programs under arbitrary thread interleavings. Second, we demonstrate how, combined with predicate abstraction, our technique can both refute and verify safety properties of programs with unbounded variable domains, even for unbounded thread counts. Previous work has established that, for non-trivial concurrency routines, predicate abstraction induces a highly complex abstract program semantics. Our technique, however, never statically constructs an abstract parametric program; it only requires some abstract-states set to be closed under certain actions, thus eliminating the dependence on the existence of verification algorithms for abstract programs. We demonstrate the efficiency of our technique on many examples used in prior work, and showcase its simplicity compared to earlier approaches on the unbounded-thread Ticket Lock protocol.

#### **1 Introduction**

Asynchronous concurrent programs consist of a number of threads executing in an interleaved fashion and communicating through shared variables, message passing, or other means. In such programs, the set of states reachable by one thread depends both on the behaviors of the other threads, and on the order in which the threads are interleaved to create a global execution. Since the thread interleaving is unknown to the program designer, analysis techniques for asynchronous programs typically assume the worst case, i.e., that threads can interleave arbitrarily; we refer to this assumption as *full scheduling nondeterminism*. In order to prove safety properties of such programs, we must therefore ultimately investigate all possible interleavings.

Partially supported by the US National Science Foundation under grant #1718235.

c The Author(s) 2021

A. Silva and K. R. M. Leino (Eds.) CAV 2021, LNCS 12759, pp. 380–402, 2021. https://doi.org/10.1007/978-3-030-81685-8\_18

Proposed about a decade ago, *delay-bounded deterministic scheduling* [10] is an effective technique to curb the large cost associated with exploring arbitrary thread interleavings. The idea is that permitting a limited number of scheduling *delays*—skipping a thread when it is normally scheduled to execute— in an otherwise deterministic scheduler approximates a fully nondeterministic scheduler from below. Delaying gives rise to a new thread interleaving, potentially reaching states unreachable to the deterministic scheduler. In the limit, i.e., with unbounded delays, the delaying and the fully nondeterministic scheduler permit the same set of executions and, thus, reach the same states.

Prior work has demonstrated that delay-bounded scheduling can "discover concurrency bugs efficiently" [10], in the sense that such errors are often detected for a small number of permitted delays. The key is that few delays means to explore only few interleavings. Thus, under moderate delay bounds, the reachable state space can often be explored exhaustively, resulting—if no errors are found in a delay-bounded verification result.

We build on the empirical insight of efficient delay-*bounded* bug detection (testing) or verification, and make the following contributions.

**1. Delay-bounded scheduling without delay.** If no bug is found while exhaustively exploring the given program for a given delay budget, we "feel good" but are left with an uncertainty as to whether the program is indeed bugfree. We present a technique to remove this uncertainty, as follows. We prove that the set R(d) of states reached under a delay bound d equals the set R of reachable states under *arbitrary* thread interleavings if two conditions are met:


In some cases, the set of "critical program actions" may be definable statically at the language level; in others, this must be determined per individual action. To increase the chance that the above two conditions eventually hold, we typically work with conservative abstractions of R; the (precisely computed) abstract reachability set R is then used to decide whether the program is safe.

**2. Efficient delay-unbounded analysis.** We translate the above foundational result into an efficient delay-unbounded analysis algorithm. It starts with a deterministic Round-Robin scheduler, parameterized by the number of rounds r it runs and of delays d it permits, and increases r and d in a delicate schedule *weak-until* the two conditions above hold (it is not guaranteed that they ever will). The key for efficiency is that the reachability sets under increasing r and d are *monotone*. We therefore can determine reachability under parameters r ≥ r and d ≥ d starting from a *frontier* of the states reached under bounds r and d. We present this algorithm and prove it correct. We also prove its termination (either finding a bug or proving correctness), under certain conditions.

**3. Delay-unbounded analysis for general infinite-state systems.** We demonstrate the power of our technique on programs with unbounded-domain variables and unbounded thread counts. The existence of integer-like variables suggests the use of a form of predicate abstraction. Prior work has shown that predicate abstraction for unbounded-thread concurrent programs leads to complex abstract program semantics [8,15], going beyond even the rich class of well-quasiordered systems [1]. Our delay-unbounded analysis technique does not require an abstract program. Instead, we add to the idea of reachability analysis under increasing r and d a third dimension n, representing increasing thread counts, enjoying a similar convergence property. Circumventing the static construction of the abstract program simplifies the verification process dramatically.

In summary, this paper presents a technique to lift the bound used in delaybounded scheduling, while (empirically) avoiding the combinatorial explosion of arbitrary thread interleavings. Our technique can therefore find bugs as well as prove programs bug-free. We demonstrate its efficiency using concurrent pushdown system benchmarks, as well as known-to-be-hard infinite-state protocols such as the Ticket Lock [3]. We offer a detailed analysis of internal performance aspects of our algorithm, as well as a comparison with several alternative techniques. We attribute the superiority of our method to the retained parsimony of limited-delay deterministic-schedule exploration.

A full version of this paper, with proofs omitted here and other supplementary information, can be found in an accompanying Technical Report [14].

#### **2 Delay-Bounded Scheduling**

#### **2.1 Basic Computational Model**

For the purposes of introducing the idea behind delay-bounded scheduling, we define a deliberately broad asynchronous program model. Consider a multithreaded program P consisting of n threads. We fix this number throughout the paper up to and including Sect. 5.1, after which we consider parameterized scenarios. Each thread runs its own procedure and communicates with others via shared program variables. A "procedure" is a collection of *actions* (such as those defined by program statements). We define a shared-states set G and, for each thread, a local-states set L*<sup>i</sup>* (0 ≤ i<n). A global program state is therefore an element of G×*n*−1 *<sup>i</sup>*=0 L*i*. In addition, a finite number of states are designated as *initial*. (Finiteness is required in Sect. 4 for a termination argument [Lemma 11].)

The execution model we assume in this paper is asynchronous. A *step* is a pair (s, s ) of states such that there exists a thread i (0 ≤ i<n) such that s and s agree on the local states of all threads j = i; the local state of thread i may have changed, as well as the shared state. We say thread i *executes* during the step, by executing some action of its procedure.<sup>1</sup> The execution semantics within the procedure is left to the thread (e.g., there may be multiple enabled actions in a state, an action may itself be nondeterministic, etc.). Without loss of generality for safety properties, we assume that the transition relation induced

<sup>1</sup> If only the shared state changes, it is possible that the identity of the executing thread is not unique. This small ambiguity is inconsequential for this paper.

by each thread's possible actions be total. That is, instead of an action x being disabled for a thread in state s, we stipulate that firing x from s results in s.

A *path* is a sequence p = (s0,...,s*l*) of states such that, for 0 ≤ i<l, (s*i*, s*i*+1) is a step. This path has length l (= number of steps taken). A state s is *reachable* if there exists a path from some initial state to s. We denote by R the (possibly infinite) set of states reachable in P. Note that these definitions permit arbitrary asynchronous thread interleavings.

#### **2.2 Free and Round-Robin Scheduling**

We formalize the notion of a scheduling policy indirectly, by parameterizing the concept of reachability by the chosen scheduler. A state s is *reachable under free scheduling* if there exists a path p = (s0,...,s*l*) from some initial state s<sup>0</sup> to s*<sup>l</sup>* = s. A free scheduler is simulated in state space explorers using full nondeterminism. State s is *reachable under* n*-thread Round-Robin scheduling with round bound* r if there exists a path p = (s0,...,s*l*) from some initial state s<sup>0</sup> to s*<sup>l</sup>* = s such that

1. l/n ≤ r, and 2. for 0 ≤ i<l, thread i (mod n) executes during step (s*i*, s*i*+1).

#### **2.3 Delay-Bounded Round-Robin Scheduling**

We approximate the set of states reachable under free scheduling from below, using a relaxed Round-Robin scheduler. The scheduler introduced so far is, however, deterministic and thus *vastly* underapproximates the free scheduler, even for unbounded r. The solution proposed in earlier work is to introduce a limited number d of scheduling *delays* [10]. A delayed thread is skipped in the current round and must wait until the next round.

**Definition 1.** *State* s *is reachable under Round-Robin scheduling with round bound* r *and delay bound* d *("* reachable under *RR*(r, d) scheduling*" for short) if there exists a path* p = (s0,...,s*l*) *from some initial state* s<sup>0</sup> *to* s*<sup>l</sup>* = s *and a function* f : {0,...,l − 1}→{0,...,n − 1}*, called* scheduling function*, such that*


Variable d*<sup>p</sup>* from 1. quantifies the total delay, compared to a perfect Round-Robin scheduler, that the scheduling along path p has accumulated. Consider the case of n = 4 threads T0,. . . ,T3. Then the scheduling sequence (f(0),...,f(11)) below on the left, of l = 12 steps and involving 13 states, follows a perfect Round-Robin schedule of r = 3 rounds (separated by |):

0123|0123|0123 0123|0123|0123 × ××

The sequence on the right of l = 9 steps follows a Round-Robin scheduling of r = 3 rounds and a total of d*<sup>p</sup>* = 3 delays: one after the second step (T2 is delayed: 3 − 1 − 1 mod 4 = 1), another two delays after the sixth step (T3 and T0 are delayed: 1 − 2 − 1 mod 4 = 2). The final state of this path is reachable under *RR*(3, 3) scheduling. Note that delays effectively shorten rounds.

We denote by R(r, d) the set of states reachable in P under *RR*(r, d) scheduling. (Note that this set is finite, for any program P.) It is easy to see that, given sufficiently large r and d, *any* schedule can be realized under *RR*(r, d) scheduling:

**Theorem 2.** *State* s *is reachable under free scheduling iff there exist* r, d *such that* s *is reachable under RR*(r, d) *scheduling:* R = *r,d*∈<sup>N</sup> <sup>R</sup>(r, d)*.*

State-space exploration under free scheduling can therefore be reduced to enumerating the two-dimensional parameter space (r, d) and computing states reachable under *RR*(r, d) scheduling. This can be used to turn a Round Robin-based state explorer into a semi-algorithm, dubbed *delay-bounded tester* in [10].

An important property of the round and delay bounds is that increasing them can only increase the reachability sets:

**Property 3 (Monotonicity in** r **&** d**).** *For any round and delay bounds* r *and* d*:*

$$R(r,d) \subseteq R(r+1,d) \quad , \quad R(r,d) \subseteq R(r,d+1) \;. \tag{1}$$

This follows from the ... ≤ r and ... ≤ d constraints in Definition 1. The property relies on r and d being external to the program, not accessible inside it. Under this provision, monotonicity in any kind of resource bound is a fairly natural *yet not always guaranteed* property; we give a counterexample in Sect. 5.2.

#### **3 Abstract Closure for Delay-Bounded Analysis**

The goal of this paper is a technique to prove safety properties of asynchronous programs under arbitrary thread schedules. Theorem 2 affords us the possibility to reduce the exploration of such arbitrary schedules to certain bounded Round-Robin schedules, but we still need to deal with those bounds. In this section we present a closure property for bounded Round-Robin explorations.

#### **3.1 Respectful Actions**

Let S be the set of global program states of P, and let α: S→A be an *abstraction function*, i.e., a function that maps program states to elements of some abstract domain A. Function α typically hides certain parts of the information contained in a state, but the exact definition is immaterial for this subsection.

A key ingredient of the technique proposed in this paper is to identify actions of the program executed by a thread with the property that the abstract successor of an abstract state under such an action does not depend on concrete-state information hidden by the abstraction.

**Definition 4.** *Let* x *be a program action, and let the relation* s *<sup>i</sup>* : *<sup>x</sup>* −→ <sup>s</sup> *denote that* s → s *is a step during which thread* i *executes* x*. Action* x *respects* α *if, for all states* s1, s2, s 1, s <sup>2</sup> ∈ S *and all* i : 0 ≤ i<n*:*

$$\alpha(s\_1) = \alpha(s\_2) \; \land \; s\_1 \stackrel{i,x}{\longrightarrow} s\_1' \; \land \; s\_2 \stackrel{i,x}{\longrightarrow} s\_2' \; \Rightarrow \; \; \alpha(s\_1') = \alpha(s\_2') \; . \tag{2}$$

Intuitively, "x respects α" means that successors under action x of α-equivalent states all have the same unique abstraction. Note the special case s<sup>1</sup> = s2, s <sup>1</sup> = s <sup>2</sup>: for nondeterministic actions x to respect α, multiple successors s 1, s <sup>2</sup> of the same concrete state s<sup>1</sup> = s<sup>2</sup> under x also must have the same abstraction.

**Example 5.** *Consider* n*-thread* concurrent pushdown systems *(CPDS), an instance of the asynchronous computational model presented in Sect. 2.1. We have a finite set of shared states readable and writeable by each thread. Each thread also has a finite-alphabet stack, which it can operate on by (i) overwriting the top-of-the-stack element, (ii) pushing an element onto the stack, or (iii) popping an element off the top of the non-empty stack. The classic pointwise top-of-the-stack abstraction function is defined by*

$$\alpha(g, w\_0, \dots, w\_{n-1}) = (g, \sigma\_0, \dots, \sigma\_{n-1})\ \ , \tag{3}$$

*where* g *is the shared state (unchanged by* α*),* w*<sup>i</sup> is the contents of the stack of thread* i*, and* σ*<sup>i</sup> is the top of* w*<sup>i</sup> if* w*<sup>i</sup> is non-empty, and empty otherwise [18]. Note that the domain into which* α *maps is a finite set.*

*Push and overwrite actions respect* α*, while pop actions disrespect it: consider the case* n = 1 *and* s<sup>1</sup> = (g, w0) = (0, 10) *and* s<sup>2</sup> = (0, 11)*, with stack contents* 10 *and* 11*, resp. (left = top). While* α(s1) = α(s2) = (0, 1)*, the (unique) successor states of* s<sup>1</sup> *and* s<sup>2</sup> *after a* pop *are not* α*-equivalent: the elements* 0 *and* 1 *emerge as the new top-of-the-stack symbols, respectively, which* α *can distinguish.*

The notion of respectful actions gives rise to a condition on sets of abstract states that we will later use for convergence proofs:

**Definition 6.** *An abstract-state set* A *is closed under actions disrespecting* α *if, for every a* ∈ A *and every successor a of a under a disrespectful action, a* ∈ A*.*

For maximum precision: a is said to be a successor of a under a disrespectful action if there exist concrete states s and s , a thread id i and an action x such that α(s) = a, α(s ) = a , x disrespects α, and s *<sup>i</sup>* : *<sup>x</sup>* −→ <sup>s</sup> . If abstraction α is clear from the context, we may just say "closed under disrespectful actions".

#### **3.2 From Delay-Bounded to Delay-Unbounded Analysis**

We now present our idea to turn a round- and delay-bounded tester into a (partial) verifier, namely by exploring the given asynchronous program for a number of round and delay bounds until we have "seen enough". Recall the notations R and R(r, d) defined in Sect. 2. We also use R and R(r, d) short for α(R) and α(R(r, d)), i.e. the respective abstract reachability sets. (Note that R is *not* an abstract fixed point—instead, it is the result of applying α to the concrete reachability set R; see discussion in Sect. 7.)

**Theorem 7.** *For any* r, d <sup>∈</sup> <sup>N</sup>*, if* <sup>R</sup>(r, d) = <sup>R</sup>(<sup>r</sup> + 1, d <sup>+</sup> <sup>n</sup> <sup>−</sup> 1) *and* <sup>R</sup>(r, d) *is closed under actions disrespecting* α*, then* R(r, d) = R*.*

The theorem states: if the set of *abstract* states reachable under *RR*(r, d) scheduling does not change after increasing the round bound by 1 and the delay bound by n − 1, and it is closed under disrespectful actions, then R(r, d) is in fact the *exact* set R of abstract states reachable under a *free* scheduler: no approximation, no rounds, no delays, no Round-Robin.

**Proof.** of Theorem 7: we have to show that R(r, d) is closed under the abstract image function *Im* induced by α, defined as

$$\overline{Im}(a) \;= \; \{ a' \; : \; \exists s, s' \; : \; \alpha(s) = a, \; \alpha(s') = a', \; s \to s' \} \; . $$

That is, we wish to show *Im*(R(r, d)) ⊆ R(r, d), which proves that no more abstract states are reachable. Consider a ∈ R(r, d) and a ∈ *Im*(a), i.e. we have states s, s such that α(s) = a, α(s ) = a , and s *<sup>i</sup>* : *<sup>x</sup>* −→ <sup>s</sup> for some thread <sup>i</sup> and some action x. The goal is to show that a ∈ R(r, d).

To this end, we distinguish flavors of x. If x disrespects α, then a ∈ R(r, d), since the set is closed under disrespectful actions.

So x respects α. Since a ∈ R(r, d), there exists a state s<sup>0</sup> ∈ R(r, d) with α(s0) = a. Suppose for a moment that thread i is scheduled to run in state s0. Then it can execute action x; any successor state s <sup>0</sup> satisfies s <sup>0</sup> ∈ R(r, d), and:

$$a' \stackrel{(\text{def }a')}{=} \alpha(s') \stackrel{(x \text{ resp. } \alpha)}{=} \alpha(s'\_0) \stackrel{(\text{def }s'\_0)}{\in} \alpha(R(r,d)) = \overline{R}(r,d) \text{ .} $$

**But what if** the thread scheduled to run in state s<sup>0</sup> under *RR*(r, d) scheduling, call it j, **is not thread** i**?** Then we *delay* any threads that are scheduled before thread i's next turn; if i<j, this "wraps around", and we need to advance to the next round. The program state has not changed—we are still in s0. Let s <sup>0</sup> be the successor state obtained when thread i now executes action x, and λ(i, j) = 1 if i<j, 0 otherwise. Then we have s <sup>0</sup> ∈ R(r + λ(i, j), d + (j − i) mod n), and:

$$a' \stackrel{\text{(def }a')}{=} \alpha(s') \stackrel{\text{(\$x\$ run. \$a\$)}}{=} \alpha(s'\_0) \stackrel{\text{(def }s'\_0, a)}{\in} \overline{R}(r + \lambda(i, j), d + (j - i) \text{ mod } n) \\ \stackrel{\text{(monot. \$r, d)}}{\subseteq} \overline{R}(r+1, d + n - 1) \stackrel{\text{(Thm. \$r\$)}}{=} \overline{R}(r, d) \ .$$

This concludes the proof of Theorem 7.

**Example 8.** *Consider a simple 3-thread system with a shared-states set* G = {0, 1, 2}*. The local state of each thread is immaterial; function* α *just returns the shared state:* α(g,l0, l1, l2) = g*. The threads' procedures consist of the following actions, which update only the shared state:*

*Thread T0:* 0 → 1 *Thread T1:* 0 → 1 *Thread T2:* 0 → 2 *.*

*Table 1 shows the set of reachable states for different round and delay bounds. For example, with one round and zero delays, the only feasible action is T0's. The reachable states are* 0 *(initial) and* 1 *(found by T0). The table shows a path to a pair* (r, d) *that meets the conditions of Theorem 7. From* (r, d) = (1, 0) *we increment* r *to find a plateau in* r *of length* 1*. We then increase* d *to try to find a plateau in* d *of length* n − 1=2*. This example shows that a delay plateau of length* 1 *is not enough, as* 2 *is only reachable at least* 2 *delays. At* (2, 2) *we find a new state (*2*), so we restart the search for plateaus in* r *and* d*. At* (3, 4)*, the plateau conditions for Theorem 7 are met. There are no disrespectful transitions, so by Theorem 7, we know that* R(3, 4) = R*.*

**Table 1.** Reachable states in Example 8 under various round and delay bounds. The boxed set passes the convergence test suggested by Theorem 7


#### **4 Efficient Delay-Unbounded Analysis**

Turning Theorem 7 into a reachability algorithm requires efficient computation of the sets R(r, d). This section presents an approach to achieve this, by expanding only *frontier* states when either the round or the delay parameter is increased.

To this end, let C be a state property (such as an assertion) that respects α, in the sense that, for any states s1, s2, if α(s1) = α(s2), then s<sup>1</sup> |= C iff s<sup>2</sup> |= C. From now on, we further assume the domain A of abstraction function α to be finite, which will ensure termination of our algorithm (see Lemma 11 later).

Our verification scheme for C is shown in Algorithm 1, which uses Algorithm 2 as a subroutine. In the rest of this paper, we also refer to Algorithm 1 as *Delay- (and round-) UnBounded Analysis*, DrUBA for short.

The main data structure used in the algorithms is that of a State, which stores both program variables and scheduling information, in the attributes *finder* , *rounds taken*, and *delays taken*. For a state s, variables s.*rounds taken* and s.*delays taken* represent the number of times the scheduler started a round and delayed a thread, resp., to get to s. Variable s.*finder* contains the index of the thread whose action produced s. This is enough information to continue

**Algorithm 1.** Verifying property C against all reachable states of program P

```
Input: n-thread asynchronous program, property C
Output: "safe", "violation of C", or "unknown"
1: Reached := (finite) set of initial states -
                                             Reached: states reached so far
2: r := 0; d := 0
3: repeat
4: Frontier := {s ∈ Reached : s.rounds taken = r}
5: r++
6: for s ∈ Frontier do
7: Reached := Reached ∪ FinishRounds(s, r + 1, C)
8: r++
9: until round plateau of length 1
10: repeat
11: Frontier := {s ∈ Reached : s.delays taken = d}
12: d++
13: for s ∈ Frontier do
14: s-
          := s -
                                                          copy of state s
15: s-

          .delays taken++
16: s-

          .finder := (s-

                      .finder + 1) mod n
17: if s-

            .finder mod n = 0 then
18: s-

             .rounds taken++
19: Reached := Reached ∪ FinishRounds(s-

                                          , r, C)
20: if new abstract state found during for loop in Line 13 then
21: goto 3 -
                                   abort second repeat loop; go back to first
22: until delay plateau of length n − 1
23: if α(Reached) is closed under disrespectful actions then
24: return "safe"
25: else
26: return "unknown"
```
**Algorithm 2.** *FinishRounds*(s, r, C)


the execution from s later, starting with the thread after *finder* . For the initial states, *rounds taken* and *delays taken* are zero, and *finder* is n−1 (the latter so that expanding the initial states starts with thread (n − 1) + 1 mod n = 0). For set membership testing, two states are considered equal when they agree on their finders and on program variables. The *rounds taken* and *delays taken* variables are for scheduling purposes only and ignored when checking for equality.

As mentioned in Prop. 3, the sequence of reachability sets is monotone with respect to both rounds and delays, for any program. This entails two useful properties for Algorithm 1. First, we can increase the bounds in any order and at individual rates. Second, it suffices to expand states at the frontier of the exploration, without missing new schedules. When adding a new delay, we only need to delay those states that were (first) found in schedules using the maximum delays. When adding a round, we only need to expand states that were (first) found in the last round of a schedule.

Algorithm 1 first advances the round parameter r until a round plateau has been reached (Lines 3–9). It does so by running the *FinishRounds* function on *frontier states* s: those that were reached in the final round r of the previous round iteration. *FinishRounds* (Algorithm 2) explores from the given state s, Round-Robin style, up to the given round, without delaying any thread. The actual expansion of a state happens in function *Image* (Line 8 of Algorithm 2), which computes a state's successors and initializes their scheduling variables: *rounds taken* and *delays taken* are copied from u, the *finder* of the successor is the next thread (+1 mod n). If this wraps around, *rounds taken* is incremented as well.

Back to the main Algorithm 1: we have reached a round plateau of length 1 if the entire **for** loop in Line 6 sees no new *abstract* states (no new elements in α(*Reached*)). If so, we are not ready yet to perform the convergence test (recall Example 8). Instead, Algorithm 1 now similarly advances the delay parameter d (Lines 10–22). For each frontier state (*delays taken* = d), we delay the thread scheduled to execute from this state (by incrementing (mod n) the *finder* variable), and record the taken delay (Line 15). Then we again call the *FinishRounds* function and merge in the states found. Importantly, these merges preserve states already in *Reached*, meaning that the algorithm will keep states found earlier in the exploration (with smaller r, d).

The loop beginning in Line 10 repeats until a delay plateau of length n − 1 is encountered (as required by Theorem 7). This means that during n − 1 consecutive **repeat** iterations, the **for** loop in 13 did not find any new abstract states. When the round and delay plateaus have the required lengths (1 and n−1, resp.), we invoke the convergence test (Line 23), which amounts to applying Theorem 7. If the test fails, Algorithm 1 returns "unknown".

Towards proving partial correctness of Algorithm 1, we first show that the states eventually collected in set *Reached* by the algorithm correspond exactly to the round- and delay-bounded reachability sets R(r, d), and that—after the two main **repeat** loops—a plateau of sufficient length has been generated. As a corollary, the algorithm is partially correct, i.e. it returns correct answers if it terminates.

**Lemma 9.** *If Algorithm 1 reaches Line 23, the current values of* r *and* d *satisfy: (i) Reached* = R(r, d)*, and (ii)* R(r − 1, d − (n − 1)) = R(r, d)*.*

**Corollary 10.** *The answers "safe" and "violation of* C*" returned by Algorithm 1 are correct.*

The algorithm won't return either "safe" or "violation of C" in one of two situations: when the convergence test fails in Line 23 (it gives up), and when it fails to ever reach this line. The latter can be prevented using a finite-domain α:

**Lemma 11.** *If the domain* A *of abstraction function* α *is finite, Algorithm 1 terminates on every input.*

Since abstraction α approximates the information contained in a state, a plateau may be *intermediate*, e.g. R(1, 0) - R(1, 1) = R(2, 2) - R(2, 3). Thus, stopping the exploration simply on account of encountering a plateau—even of lengths (1, n−1)—is unsound. Intermediate plateaus make our algorithm (unavoidably) incomplete: if the test in Line 23 fails, then there are known-to-be-reachable abstract states with abstract successors whose reachability cannot be decided at that moment. If we knew the plateau to be intermediate, we could keep exploring the sets R(r, d) for larger values of r and d until the next plateau emerges, hoping that the convergence test succeeds at that time. In general, however, we cannot distinguish intermediate from final plateaus.

#### **5 DrUBA with Unbounded-Domain Variables**

In addition to unbounded control structures like stacks, which come up in pushdown systems and were discussed in Ex 5, infinite state spaces in programs are often due to (nominally) unbounded-domain program variables. This presents no problem for the computation of the concrete reachability sets *Reached* in Algorithm 1: for any round and delay bounds (r, d), the set of concrete reachable states *RR*(r, d) is finite and thus explicitly computable (no symbolic data structures are needed).<sup>2</sup> On the other hand, termination of the same algorithm requires that it eventually reach a plateau in r and d of sufficient length. This is guaranteed by an abstraction function α that maps concrete states into a finite abstract space. A finite abstract domain is therefore highly desirable.

A generic abstraction that reduces an unbounded data domain to a finite one is predicate abstraction [4,13, see [14] for a short primer]. The goal in this section is to demonstrate how the simple scheme of delay-unbounded analysis can be combined with predicate abstraction to verify unbounded-thread programs.

<sup>2</sup> Contrast this to a *context-switch* bound, under which reachability sets can be infinite.

#### **5.1 The Fixed-Thread Case**

Consider program P in Fig. 1 on the left [8, page 4: program P]. Intuitively, variable m counts the number of threads spawned to execute P concurrently. It is easy to see that "the assertion in [P] cannot be violated, no matter how many threads execute [P], since no thread but the first will manage to" [8] enter the *true* branch of the **if** statement and reach the assertion.

**Fig. 1. Left:** program P; **Right:** how Algorithm 1 operates on (an abstraction of) it

Previous work has shown that even the 1-thread version of this program cannot be proved correct using predicate abstraction **unless** we permit predicates that depend on both shared and local variables [9, for the unprovability result], which have been referred to as *mixed* [8]. An example is the predicate p :: (s = l), which comes up in the assertion. The dependence of p on both shared and thread-local data causes standard solutions that track the truth value of p in a shared or local Boolean variable to be unsound. The solution proposed in [8] is to use *broadcast* instructions to have the executing thread notify all other threads whenever the truth value of p changes. This solution comes with two disadvantages: (i) the resulting Boolean broadcast programs are more expensive to analyze than strictly asynchronous Boolean programs, and (ii) the solution cannot be extended to the unbounded-thread case.

Let us consider how we can verify this program using Algorithm 1, for the fixed-thread case; we consider n = 2 threads. We will have to use mixed predicates as in [8], but since we never execute the abstract Boolean program, there is no need for constructing it. As a result, there is no need for broadcast instructions.

The program generates an unbounded number of reachable concrete states, but we explore it only under round and delay bounds r and d. As per Algorithm 1, we increase these bounds until we have reached plateaus of lengths 1 and n − 1 = 1, resp. Plateaus are determined over the abstract-state set, so we need a function α.

*First attempt: a single predicate.* We define α as follows, for a concrete state c:

$$\alpha\_1(c) \;=\; (c.pc\_0, \; c.s = c.l\_0) \in \mathbf{0}, \; \dots, \mathbf{4} \times \{0, 1\} \;, \; c$$

where c.*pc*0, c.l0, and c.s are the values of thread 0's *pc* and local variable l, and of shared variable s, in state c, respectively. The function extracts from a concrete state the current program location of thread 0 and the value of predicate p :: (s = l) for thread 0.<sup>3</sup> The only statement *not* respecting α<sup>1</sup> is the **if** statement in Line 1: here, the new value of the *pc* cannot be determined from the current values of *pc* and predicate p alone. All other statements respect α1.

We can now perform an iterative exploration of this program—bounded but exhaustive within each bound. In Fig. 1 on the right, red arrows denote "new abstract state reached". A red horizontal arrow (r++) means: "keep increasing r". A red vertical arrow (d++) means: "switch to increasing r". In other words, following a red arrow—no matter the direction—we always go "right" (r++). The green horizontal arrow followed by a green vertical arrow at the end indicates that we have reached the first plateaus of length 1 in *both* directions: at (r, d) = (7, 4).

At this point we have reached a total of 7 abstract states. State (3, 0) (*pc* = 3, s = l) is not among them, so the assertion has not been violated so far. We run the convergence test, to determine whether set R(7, 4) is closed under disrespectful actions. Since the **if** in Line 1 is the only disrespectful statement, we only need to check successors of abstract states of the form (1, ?) (i.e., with *pc* = 1). Unfortunately, R(7, 4) contains abstract state (1, 0) (a reachable abstract state) but not its abstract successor (2, 0). This state is unreachable, but we do not know that at this point. This causes Algorithm 1 to return "unknown".

*Second attempt: two predicates.* The disrespectful action causing the failure suggests that we need to keep track of whether the branch in Line 1 can be taken, i.e. whether m = 1. We refine our abstraction using this (non-mixed) predicate:

$$\alpha\_2(c) \quad = \quad (c.pc\_0, \ c.s = c.l\_0, \ c.m = 1) \in 0, \ldots, 4 \times \{0, 1\}^2 \, . \tag{4}$$

The abstract successors of the **if** statement can now be decided based only on knowledge provided by α2, i.e. the statement respects α2. There is, however, another statement disrespecting α2, and only one: the increment m++ in Line 0. If m = 1, we cannot decide whether m = 1 will be true after the increment.

We again perform our iterative exploration of this program, and find the first suitable plateau at the same point (r, d) = (7, 4). This time, however, we have reached a total of 12 abstract states (all of them "safe"). We run the convergence test: we only need to check already reached abstract states of the form (0, ?, 0) (*pc* = 0, m = 1). Set R(7, 4) contains exactly one state of this form: (0, 1, 0), which m++ can turn into (1, 1, 0) and (1, 1, 1)—note that the next *pc* value is unambiguous (1), and predicate s = l is not affected. **The good news** is now that both abstract states (1, 1, 0) and (1, 1, 1) are contained in R(7, 4). This proves this set closed under disrespectful actions; Algorithm 1 terminates: the assertion is safe for any execution schedule, for the case of n = 2 threads.

<sup>3</sup> Tracking these values for thread 0 suffices: the multi-threaded program is symmetric.

We summarize that, in our solution above, we assumed a lucky hand in picking predicates—the question of predicate discovery is orthogonal to the delayunbounded analysis scheme. However, the proof obtained using Algorithm 1 does not involve costly broadcast operations, previously proposed as an ingredient to extend predicate abstraction to concurrent programs. A second, more powerful advantage is that, unlike the earlier broadcast solution, Algorithm 1 extends gracefully to the unbounded-thread case. This is the topic of the rest of this section.

#### **5.2 The Unbounded-Thread Case**

The goal now is to investigate whether an asynchronous unbounded-domain variable program is safe for *arbitrary* thread counts (and thread interleavings).

*Existing solutions.* We are aware of only one general technique that combines predicate abstraction with unbounded-thread concurrency [15]. That technique can achieve the above goal, roughly as follows. In addition to standard and mixed predicates used also in the fixed-thread case, we now permit *inter-thread* predicates, which quantify over all threads other than the executing one. Such predicates allow us to express, for example, that a thread's local variable l's value is larger than that of any other thread: ∀i : i = *self* : l>l*i*. Predicates of this type are provably required during predicate abstraction to verify the safety of the Ticket Lock algorithm [3,15].

Abstraction against inter-thread predicates leads to a *dual-reference program* [15], a process that is already far more complex than standard sequential or even fixed-thread predicate abstraction. But we pay another price for using these predicates: namely, the loss of *monotonicity* of the transition relation w.r.t. a standard well-quasiordering on infinite state sets of unboundedthread Boolean programs. In this context, monotonicity states, roughly, that adding passive threads to a valid transition keeps the transition intact.

This price is heavy, since monotonicity w.r.t. would have given us a well-quasiordered infinite-state transition system, for which local-state reachability properties are decidable [1]; working implementations exist. The abovementioned prior work attempts to salvage the situation, by adding a set of transitions (the *non-monotone fragment*) to the dual-reference program that restore monotonicity and further overapproximate but without affecting the reachability of unsafe states [15].

*Alternative solution.* We now propose a solution that uses the same type of interthread predicates (this is inevitable), but renders dual-reference programs, the monotone closure of the transition relation and all other "overhead" introduced in [15] unnecessary. We will use Algorithm 1 as a sub-routine.

The idea is as follows. Sect. 5.1 suggests a way to verify fixed-thread asynchronous programs, using a combination of predicate abstraction and Algorithm 1 . To handle the unbounded-thread case, we wrap another layer of incremental resource bounding around this combined algorithm—the "resource" this time is the number n of threads executing the program. For each member of a sequence of increasing fixed thread counts we compute the set of abstract states reachable under arbitrary thread interleavings. This is purely a sub-routine; we will use the method proposed in Sect. 5.1 (others are possible, e.g. [8]).

The incremental (in n) analysis proceeds until we have reached a thread plateau *of length 1*, and then run the convergence test: we check the current abstract reachability set for closure under disrespectful actions. This time, the abstract transitions must take into account that the number of executing threads is unknown. It is easy to see that a plateau of length 1 is sufficient: we compute the set of abstract states reachable under *arbitrary* thread schedules; thus, the obstacle of non-schedulability of thread i in the proof of Theorem 7 that forced us to wait for a (delay) plateau of length n − 1 does not apply here.

#### **A non-monotone resource parameterization**

Before we demonstrate this idea on program P, we justify our strategy of combining resource bounds. The idea presented above can be viewed as a multi-resource analysis problem where we increment r and d in an "inner loop" (represented by Algorithm 1 as a sub-routine to compute fixed-thread reachability sets), and n in an outer loop. Both loops compute monotonously increasing reachability sequences: for "inner" this is Prop. 3; for "outer" this is easy to see. Theorem 7 relies upon the monotonicity: without it, the test R(r, d) = R(r + 1, d + n − 1) makes the algorithm unsound.

The way we nest the three involved resource parameters is not arbitrary: Round-Robin reachability under an increasing thread count is not monotone. More precisely, making the thread-count parameter n explicit, let R(r, d, n) denote the set of states reachable in the n-thread program P under *RR*(r, d) scheduling. Then R(r, d, n) ⊆ R(r, d, n + 1) is **not** valid. The following example illustrates this (at first counter-intuitive) monotonicity violation:

**Example 12.** *Consider the asynchronous Boolean program over shared varinables* s *and* t *on the right. Here we have* R(3, 0, 1) ⊆ R(3, 0, 2)*: given 1 thread (sequential execution), a state with*


s = 1 *is reachable. With 2 symmetric threads, under delay-free Round-Robin scheduling (*d = 0*), the first and second thread will repeatedly flip* t *to* 1 *and back to* 0*, resp., before either one has a chance to get past the guard in* Line 1.

*A stronger result is: for all* <sup>r</sup> <sup>∈</sup> <sup>N</sup>*,* <sup>R</sup>(3, <sup>0</sup>, 1) ⊆ <sup>R</sup>(r, <sup>0</sup>, 2)*, i.e. we cannot make up for the poor scheduling of the second thread by adding more rounds.*

The consequence for us is that we cannot compute, for fixed r, d, the sets R(r, d,∞), using the closure-under-disrespectful-actions paradigm. Instead we must, for each n, compute R(∞,∞, n) (using Algorithm 1 or otherwise) and increase n in the outer loop.

#### **Verifying program** P **for unbounded thread count**

We recall that, given the two predicates shown in Eq. (4) and the pc, we were able to verify program P correct (under arbitrary thread interleavings) for n = 2 threads; a total of 12 abstract states were reached (out of 5 · <sup>2</sup><sup>2</sup> = 20 possible). Advancing the outer loop, we invoke Algorithm 1 for n = 3 threads. This reveals another reachable abstract state, namely *pc* = 0, s = l, m = 1. Unfortunately, this state causes Algorithm 1 to return "unknown": under α2, one currently unreached abstract successor is *pc* = 1, s = l, m = 1, violating closure. Observing that a thread executing Line 1 with m = 1 must be the first thread executing, we try tracking the initial value of m:

$$\alpha\_3(c) \quad = \quad (c.pc\_0, \ c.s = c.l\_0, \ c.m = 1, \ c.m = 0) \in 0, \ldots, 4 \times \{0, 1\}^3. \tag{5}$$

Interestingly, *all actions (statements) of program* P *respect abstraction* α3. This means that the test for closure under disrespectful actions is *vacuously true*—we can stop as soon as we have reached a plateau in n of length 1. We don't have to wait long for this plateau: we invoke Algorithm 1 for n = 3 and n = 4 under abstraction α3. (Note that n = 4 requires a longer plateau than n = 3.) The abstract reachability sets consist of the same 14 abstract states in both cases. We report the program safe, for arbitrary interleavings and arbitrary thread counts. We can also report the exact set of 14 reachable abstract states.

We again summarize that, while we still (and unavoidably) use mixed predicates, we do not construct a thread-parameterized abstract program, which would require broadcast statements [8] and a rather involved dual-reference transition semantics [15]. In fact, we did not even need to test for closure under any abstract images, since the chosen abstraction enjoys respect from all actions.

#### **6 Evaluation**

Our goal for the evaluation of DrUBA was to answer the following questions:


Questions 1 and 2 serve to compare DrUBA against other techniques; Questions 3 and 4 investigate features of Algorithm 1.

To this end we implemented, in Java 11, a verifier using Algorithm 1 that takes concurrent pushdown systems as input; we refer to this verifier as DrUBA in this section.<sup>4</sup> We also implemented the AI approach in Java 11. For the comparison with the context-unbounded approach, we used a publicly available tool<sup>5</sup>. Our experiments are based on the concurrent benchmark programs also used in [18]. The experiments are performed on a 3.20GHz Intel i5 PC. The memory limit was 8GB, with a timeout of 1 h.

<sup>4</sup> DrUBA implementation available at https://doi.org/10.5281/zenodo.4726301.

<sup>5</sup> https://github.com/lpzun/cuba.

#### **6.1 Results**

Table 2 reports the benchmark names, the thread counts, and the size of the reachable abstract state space (columns 1–3). The second part of the table shows the time it took each verifier to fully explore the state space and confirm convergence. For the AI approach, we check whether the abstract state space is closed under *all* operations each time either r or d is incremented. Algorithm 1 was faster than "AI" on every example except Stefan-4,5. Stefan is the only program that actually does not require any delays to discover all reachable abstract states. The results indicate that the AI approach spends approximately half of its computation time doing repeated convergence tests after each bound increment. Furthermore, as state sets increase in size, AI seems to take even longer, as with the Bluetooth3 (2+3) example. The convergence test needed for "AI" includes checking closure under both respectful and disrespectful actions, making it more costly than the one used in Algorithm 1.

Algorithm 1 also improved on the results with "CUBA". For examples that took longer than a few seconds, DrUBA was able to run in less time on the same benchmark. The difference on small examples is likely due to a different implementation language (C++ vs. Java). DrUBA does not explore as many schedules, and explores fewer as the delay and round bounds approach their cutoff values (as noted below). Additionally, DrUBA was less memory-intensive for large examples for which the CUBA approach cannot prove that the set of reachable states per context bound is finite. In this case, "CUBA" requires the use of more expensive symbolic representations of states sets. Algorithm 1 does not suffer from this problem—the reachability sets in each iteration are finite. For the Stefan-5 example, "CUBA" ran out of memory after 23 min. DrUBA was able to prove convergence for this example (as was "AI").

Table 3 reports the number of times Algorithm 1 computed the image (successors) of a state until reaching the final r-d-plateau (Col. 3) and during the final plateau (Col. 4), as well as the total number of image computations without the *frontier* optimization (Col. 5). The table offers convincing evidence to support our heuristic that waiting for a long d-plateau at the end of exploration is not costly, answering Question 3.. On most benchmarks, the amount of computation done during the plateau (Col. 4) was negligible. This included our largest example, Bluetooth3 (2+3). The exception to this is the Stefan examples, which—as mentioned earlier—do not require any delays to reach the full abstract state set (the d-plateau starts at (rmax,0)). Finally, a naive implementation that does not take advantage of monotonicity, forgoing the frontier approach to expanding the state set, was orders of magnitude worse. This is because it has to recompute the whole set for every iteration of r or d. This answers Question 4..

Comparing Col. 7 in Table 2 to the cutoff context-switch bounds from [18], we find that, while the r and d bounds were large, not all programs that needed large bounds took a long time to verify. For example, the Bluetooth3 (2+1) example took much less time than Stefan-5, despite requiring 21 more delays (with similar rounds). A hint for the reason can be found in Table 3. Once the set of abstract states is close to the R, there are very few new states on the

**Table 2.** Benchmark description and running times for different algorithms. Threads: # of threads (a + b: the respective numbers of threads from two different templates); R: number of reachable abstract states; Time: running time (sec) for each algorithm ("—": timeout or memory-out); rmax, dmax: round and delay counts at the *end* of each plateau when convergence was detected.


frontier. We can see this in the small numbers in Col. 4, but it also applies to the round bound. If a state is rediscovered, it is not expanded in further round increments. Once the round bound is large enough, there are few deep schedules of maximum possible length (nr) that produce new concrete states.

#### **6.2 Unbounded-Thread Experiments**

We implemented Algorithm 1 in combination with predicate abstraction as detailed in Sect. 5.2 to check the effectiveness of our technique on a tricky concurrent program that requires unbounded variable domains. The Ticket Lock

**Table 3.** Detailed analysis of Algorithm 1, measuring the number of times the program computed the successors of a state. Col. 3 reports the image operations Algorithm 1 performed before reaching the FP (final plateau), Col. 4—the number of additional image operations computed until the program ended. Col. 5 shows the image operations without the *frontier* improvement, requiring recomputing each R(r, d) from the initial states.


protocol [3] and the predicates used to prove its correctness are shown in Fig. 2. In Line 0, threads wait to enter the Critical Section, whose code is at the beginning of Line 1; the rest of Line 1 is exit code to prepare the thread for re-entry. In the predicates on the right, subscript i denotes thread i's copy of a local variable.

This example has been shown to require significant adjustments to predicate abstraction to accommodate fixed-thread concurrency [8], and has been claimed to require an entirely new theory to cope with the unbounded-thread case [15]. We rely on the same predicates used in earlier work, and it is clear what motivates each predicate. P1 ensures t is a"new" ticket larger than previous ones, P2 is

**shared int** *s* := 0, *t* := 0 **local int** *l* := fetch and add(*t*) 0: **while** *<sup>s</sup>* -= *l* **do**; wait for *s* = *l* 1: *critical-section code here* inc(*s*) *l* := fetch and add(*t*) **goto** 0 **P4:** <sup>∀</sup>*<sup>i</sup>* : *<sup>i</sup>* -

**P1:** <sup>∀</sup>*<sup>i</sup>* : *t>l<sup>i</sup>* **P2:** |{*<sup>i</sup>* : *pc<sup>i</sup>* = 1}| ≥ <sup>2</sup> **P3:** *s* = *l* <sup>=</sup> *self* : *<sup>l</sup>* -= *l<sup>i</sup>*

**Fig. 2. Left:** the Ticket Lock protocol; **Right:** four predicates used to prove it correct

used to check the safety property, P3 tracks the condition in Line 0, and P4 means that the operating thread's l is unique. DrUBA finds four abstract states for both 2 threads and 3 threads using Algorithm 1. This is an n-plateau of length 1.

To prove convergence for both Algorithm 1 and the "outer loop" incrementing n, we used the ACL2s theorem prover [7]. We specified the data in a concrete state, and the four abstract states that were found. Only the second statement disrespects this abstraction w.r.t. r, d and n, as we know the value of the test in the first statement for an abstract state. Given these, ACL2s was able to verify that the set of abstract states is closed under the semantics of statement 1. As a result, we can report that Ticket Lock is safe (P2 is invariantly false), for an arbitrary number of threads and arbitrary thread interleavings.

#### **7 Discussion of Related Work**

This work is inspired from two angles. The first is clearly the delay-bounded scheduling (DBS) technique [10]. The authors formalize this concept and show its effectiveness as a testing scheme. Their computational model of a dynamic task buffer is somewhat different from ours. We have not discussed dynamic thread creation here; it can be simulated by creating threads up-front and delaying them until such time as they are supposed to come into existence. The DBS paper also presents a sequentialization technique that can be turned into a symbolic verifier via verification-condition generation and SMT solving. This, however, requires bounding loops and recursion. Our approach combines exhaustive finite-state model exploration with convergence detection and thus does not suffer from these restrictions.

The second inspiration comes from an earlier context-unbounded analysis technique [18]. Similar in spirit to the present work, [18] started from a yet earlier context-bounded analysis technique and describes a condition under which a chosen context bound is sufficient to reach all states reachable under some abstraction. For the case of concurrent pushdown systems (CPDS)—the verification target of [18]—, the pop operation plays a crucial role in establishing this condition; note that, in our work, pop actions disrespect the top-of-the-stack abstraction commonly used for CPDS.

Our work has a number of advantages over [18]. First, and crucially, the set of states reachable under a context bound can be infinite (a single context can already generate infinitely many states); its determination thus requires more expensive symbolic reachability methods. In contrast, the reachability set under Round-Robin scheduling with a round- and a delay bound *is always finite*; moreover, it can be computed very easily, even for complex programs. This makes our technique a prime choice for lifting existing testing schemes to verifiers. A second advantage over [18] is that we retain much of the efficiency of the "almost deterministic" exploration delay-bounded scheduling, as demonstrated in Sect. 6. A downside of our work is that our convergence condition is sound only after a plateau has emerged of length roughly equal to the number of running threads; this is not required in [18]. However, as also demonstrated in Sect. 6, our efforts to compute reachable states for increasing r, d in a *frontier-driven* way nearly annihilates this drawback: in most cases, only a small number of image computations happen along the plateau.

An alternative to our verification approach is a classical analysis based on abstract interpretation [6]. Given function α, such analysis interprets the entire program abstractly, and then computes a fixed point under the abstract program's transition relation. This fixed point, if it exists, overapproximates the set of reachable abstract states. Hence, the absence of error states in the fixed point implies safety, but the presence of errors does not immediately permit a conclusion. In contrast, our technique interleaves *concrete* state space exploration (enabling genuine testing) with *abstraction-based* convergence detection. We believe this to be a useful approach in practical programming environments, where abstract proof engines with poorly understood bug-finding capabilities may be met with skepticism. A more detailed discussion of DrUBA vs. Abstract Interpretation can be found in the Appendix of [14].

Underapproximating program behaviors using bounding techniques is a widespread solution to address undecidability of safety verification problems. Examples include depth- [12] and context-bounding [16,17,20], delay-bounding [10], bounded asynchrony [11], preemption-bounding [19], and phase-bounded analysis [2,5]. Many of these bounding techniques admit decidable analysis problems [16,17,20] and thus have been successfully used in practice for bug finding. Round- and delay-bounded Round-Robin scheduling trivially renders safety decidable, since the delay-program is finite-state. In addition, it is very easy to implement, avoiding, for example, the need for symbolic data structures and algorithms to represent and process intermediate reachability sets.

#### **8 Conclusion**

We have presented an approach to enhancing delay-bounded scheduling in asynchronous programs with a convergence test that, if successful, certifies that all states from some chosen abstract domain have been reached. The resulting algorithm inherits from earlier work the capability to detect bugs efficiently, but can also prove safety properties, under arbitrary thread interleavings. It exploits the monotonicity of delay-bounded reachability sets to expand states and test for convergence only when needed. We have further demonstrated that, combined with predicate abstraction using powerful predicates, tricky unbounded-thread routines over unbounded data, such as the Ticket Lock, can be verified using substantially less machinery than proposed in earlier work. We have shown the experimental competitiveness of our approach against several related techniques.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# Checking Data-Race Freedom of GPU Kernels, Compositionally

Tiago Cogumbreiro1(B) , Julien Lange<sup>2</sup> , Dennis Liew Zhen Rong<sup>1</sup> , and Hannah Zicarelli<sup>1</sup>

<sup>1</sup> University of Massachusetts Boston, Boston, USA {tiago.cogumbreiro,zhenrong.liew001,hannah.zicarelli001}@umb.edu <sup>2</sup> Royal Holloway, University of London, Egham, UK julien.lange@rhul.ac.uk

Abstract. GPUs offer parallelism as a commodity, but they are difficult to program correctly. Static analyzers that guarantee data-race freedom (DRF) are essential to help programmers establish the correctness of their programs (kernels). However, existing approaches produce too many false alarms and struggle to handle larger programs. To address these limitations we formalize a novel compositional analysis for DRF, based on access memory protocols. These protocols are behavioral types that codify the way threads interact over shared memory.

Our work includes fully mechanized proofs of our theoretical results, the first mechanized proofs in the field of DRF analysis for GPU kernels. Our theory is implemented in Faial, a tool that outperforms the state-ofthe-art. Notably, it can correctly verify at least 1*.*42× more real-world kernels, and it exhibits a linear growth in 4 out of 5 experiments, while others grow exponentially in all 5 experiments.

Keywords: GPU · Data-race · Static analysis · Behavioural types

#### 1 Introduction

GPUs are massively parallel devices that promise a great return on investment at a cost: they are notably difficult to program. In GPU programming, hundreds of lightweight threads share portions of arrays in parallel (without locks)—very different from the programming model of multithreaded programs written in C or Java with heavy-weight heterogeneous threads. Data-race freedom (DRF) analysis aims to guarantee that for all possible executions, every array cell being written by one thread cannot be concurrently accessed by another thread.

In the field of static analysis of DRF in GPU programs, there is a tension between efficiency and correctness (no missed data-races and no false alarms) that thus far is unresolved. Bug finding tools [26,27,33] favor correctness over efficiency: they provide correct results at small scales, by simulating the program execution. Such tools are incapable of handling certain parameters symbolically (*e.g.*, array size) and can easily exhaust users' resources (*e.g.*, loops with long iteration spaces or unknown bounds). Approaches based on Hoare logic [5,7,22]

Fig. 1. Work-flow of the verification.

can cope with medium-sized programs, do not miss data-races, and do not require array size information; however, they suffer from a high-rate of false alarms and require code annotations written by concurrency experts. Finally, tools that can cope with larger programs and do not require array size information either miss data-races [24] or overwhelm the user with false alarms [37].

To appease this tension, we introduce a novel static DRF analysis that can handle larger programs and produce fewer false alarms than related work, without missing data-races. Additionally our analysis does not require code annotations or array size information. Our verification framework hinges on *access memory protocols*, a new family of behavioral types [1] that codify the way threads interact through shared memory. Our behavioral types also make evident two aspects of the analysis that can be made separate: concurrency analysis (*i.e.*, could these two expressions run in parallel?) and data-race conflict detection (*i.e.*, do these array indices match?).

*Contributions and Synopsis.* This paper includes the following contributions.


Listing 2.1 Examples of racy kernels, l.h.s. is from [34] and r.h.s. simplifies l.h.s. for clarity, with one-dimensional array and thread identifier, and 1-stride loops.


Our paper is accompanied by an implementation (Faial), an evaluation framework (inc. datasets), and proof scripts (in Coq) for each theorem. All of these are available in our artifact [9].

#### 2 Overview

This section gives an overview of our approach by examining a data-race we found in published work [17,34]. We discuss the challenges that such examples pose to the state-of-the-art of DRF analysis. Then we introduce a verification framework based on *access memory protocols*: behavioral types [1] that codify the way threads interact via shared memory. Figure 1 gives an overview of the verification pipeline. We start from CUDA kernels, from which we infer access memory protocols. Protocols are then checked for well-formedness and transformed in three steps into formulas that are verified by an SMT solver.

#### 2.1 Challenges of GPU Programming

GPU Programming Model. The key component of GPU programming is the *kernel* program, or just kernel, that runs according to the Single-Instruction-Multiple-Thread (SIMT) execution model, where multiple threads run a single instruction concurrently. A kernel is parameterized by a special variable that holds a thread identifier, henceforth named tid. In parallel, each member of a group of threads runs an instantiated copy of the kernel by supplying its identifier as an argument. Threads communicate via shared memory (arrays) and mediate communication via barrier synchronization (an execution point where all threads must wait for each other before advancing further). Writes are only guaranteed to be visible to other threads after a barrier synchronization.

GPU programming platforms usually group threads hierarchically in multiple levels, across which no inter-groups synchronization is possible. In both the literature [6,24] and this work, the focus is on intra-group communication.

Challenges. We motivate the difficulty of analyzing data-races by studying a programming error found in the wild, reported in Listing 2.1 (left). This excerpt comes from a tutorial [34] on optimizing numeric algorithms for GPUs. The code listing transposes a matrix N-times with an outer loop indexed by variable r.

Remarkably, the tutorial [34] does not inform the readers that Listing 2.1 contains a subtle *data-race*: one transpose-operation starts (the writes to tile Listing 2.2 Minimal representative example of an access memory protocol highlighting the data-race in Listing 2.1.

```
1 // r = 0
2 forU j in 0..M // for ( int j = 0; j<M; j++)
3 {rd[ tid+j ]}; // _ = tile [ tid+i ];
4 // r = 1
5 forU i in 0..M // for ( int i = 0; i<M; i++)
6 {wr[ tid ]} // tile [ tid ] = _;
```
in line 3) without awaiting the termination of the previous transpose-operation (the reads from tile in line 6), thus corrupting the data over time and possibly skewing the timing of the optimization to appear faster than it should be. We found a related data-race in [17], which reuses code from [34].

Our tool, Faial, successfully identifies the program state that triggers the data-race in Listing 2.1: when <sup>r</sup>=<sup>1</sup> and <sup>N</sup>=2. However, state-of-the-art tools struggle to accurately analyze Listing 2.1, as evaluated in Sect. 6 (Claim 1: Test 1). Symbolic execution tools, such as [26,27], timeout for N>1, and, in general, cannot handle symbolic (unknown) bounds. GPUVerify [6], a tool based on Hoare logic, reports a false alarm: a spurious data-race when <sup>r</sup>=<sup>0</sup> and <sup>N</sup>=1. PUG [24] incorrectly identifies the example as DRF, as its analysis appears to ignore dataraces originating from different iterations of a loop.

#### 2.2 Memory Access Protocols by Example

We now investigate the data-race in Listing 2.1 with an access memory protocol. For presentation purposes, we focus our discussion on Listing 2.1 (r.h.s.), that simplifies the l.h.s. whilst retaining the root cause of its data-race, which stems from the interaction between both loops. We discuss how we support multi-dimensional arrays, multi-dimensional thread identifiers, and arbitrary loop strides in Sect. 5. In our Coq formalism the notion of "accesses" (and their dimensions) is a parameter of the theory, thus orthogonal to the theory presented here.

Consider the execution of the end of the first iteration (r=0) and the beginning of the second (r=1) iteration of the outer-loop. In this case, the execution of the <sup>j</sup>-loop when <sup>r</sup>=<sup>0</sup> is not synchronized with the execution of the <sup>i</sup>-loop when <sup>r</sup>=<sup>1</sup> as there is no call to \_\_syncthreads() in between.

The access memory protocol in Listing 2.2 captures this *partial* execution from the viewpoint of array tile. By design access memory protocols over approximate kernels by abstracting away *what* data is being written to/read from an array, to focus on *where* data is being written. The protocol models the two problematic loops of Listing 2.1, *i.e.*, the <sup>j</sup>-loop when <sup>r</sup>=<sup>0</sup> and the <sup>i</sup>-loop when <sup>r</sup>=1. The first loop reads (rd[tid+j]) from the array, while the second writes (wr[tid]) to it. Evaluation of a protocol follows the SIMT model: each thread evaluates wr[tid] by instantiating tid with their unique identifier (hereafter, an integer), *e.g.*, thread <sup>0</sup> yields wr[0] and thread <sup>1</sup> yields wr[1].

Analysis of Unsynchronized Protocols. We say that a protocol is DRF when all concurrent accesses are pair-wise DRF, *i.e.*, when issued by different threads on the same index, then neither access is a write. For instance the respective sets of concurrent accesses of threads 0 and 1 in Listing 2.2 is given below

$$\begin{array}{c} \text{tid} = 0\\ \{\text{rd}[j] \mid 0 \le j < M\} \cup \{\text{wr}[0] \} \end{array} \\ \begin{array}{c} \text{tid} = 1\\ \text{l} \text{r} \text{l} \mid 0 \le j < M \end{array} \\ \begin{array}{c} \text{tid} = 1\\ \text{l} \text{r} \text{l} \mid 0 \le j < M \end{array} \\ \begin{array}{c} \text{tid} = 1\\ \text{l} \text{r} \text{r} \mid 0 \le j < M \end{array} \\ \begin{array}{c} \text{tid} = 1\\ \text{l} \text{r} \text{r} \mid 0 \le j < M \end{array} $$

When M>1, thread 0 (l.h.s) accesses rd[1] and thread 1 (r.h.s) accesses wr[1]. Thus, there is a data-race on index 1 of the array.

A fundamental challenge of static DRF verification is how to handle loops. Symbolic execution approaches that unroll loops, *e.g.*, [26,27], cannot handle large nor symbolic iteration spaces. Static approaches that use Hoare logic, *e.g.*, [5,7,22], require user-provided loop invariants. Another approach is to reduce loops to verifying the satisfiability of a corresponding universally quantified formula, *e.g.*, [25,30]. This has the advantage of being fast and not requiring invariants. However, its previous application to GPU programming, *i.e.*, PUG, is unsound due to the interaction between barrier synchronizations and loops, *e.g.*, PUG misses the data-race in Listing 2.1. We give more details in Sect. 6.

*Our Approach.* A key contribution of our work is to identify conditions that allow a kernel to be reduced to a first-order logic formula, by precisely characterizing the effect of barrier synchronization in loops. To this end, the language of access memory protocols distinguishes syntactically between protocol fragments that synchronize from those that do not. For instance, the protocol in Listing 2.2 is identified as *unsynchronized*, as it does not include any synchronization.

In Sect. 4, we show that the DRF analysis of unsynchronized protocols can be precisely reduced to a first-order logic formula, where universally quantified formulae represent loops, thus obviating the need to unroll them explicitly. For instance, we reduce the verification of Listing 2.2 to asking whether for all M, t1, and t2, where t<sup>1</sup> -= t<sup>2</sup> are thread identifiers, the following holds:

$$\begin{aligned} \{\forall j\_1, i\_1, j\_2, i\_2 \colon 0 \le j\_1 < M \land 0 \le i\_1 < M \land 0 \le j\_2 < M \land 0 \le i\_2 < M \implies \\ \{\operatorname{rd}[t\_1 + j\_1] \} \quad \cup \{\operatorname{wr}[t\_1] \} \quad \operatorname{DRF} \ \operatorname{with} \end{aligned} \quad \begin{aligned} \{\operatorname{rd}[t\_2 + j\_2] \} \quad \xrightarrow{\sim} \{\operatorname{rd}[t\_2 + j\_2] \} \end{aligned}$$

This formula is *unprovable* since rd[t<sup>1</sup> <sup>+</sup> <sup>j</sup>1] races with wr[t2] when, *e.g.*, <sup>t</sup><sup>1</sup> = 0, t<sup>2</sup> = 1, j<sup>1</sup> = 1, and M > 1. Hence, our technique flags Listing 2.2 as racy.

Analysis of Synchronized Protocols. The protocol in Listing 2.3 (left) models *all* the interactions over the shared array tile from Listing 2.1. This protocol consists of one outer loop r that contains two inner loops separated by a barrier synchronization (sync). The first inner loop writes (wr[tid]) to the array, while the second reads (rd[tid <sup>+</sup> <sup>j</sup>]) from the array.

Listing 2.3 access memory protocols (left) of array tile from Listing 2.1 and its aligned version (right).


This protocol illustrates how our language syntactically differentiates between protocols fragments that synchronize from those that do not. Concretely, our language precludes an unsynchronized loop (for<sup>U</sup> <sup>x</sup> <sup>∈</sup> n..m {u}) from calling sync anywhere in <sup>u</sup>, and it requires that a synchronized loop (for<sup>S</sup> <sup>x</sup> <sup>∈</sup> n..m {p}) includes at least one occurrence of sync. The superscript <sup>U</sup> (resp. <sup>S</sup>) stands for *<sup>s</sup>*ynchronized (resp. *u*nsynchronized). This distinction can be inferred automatically and yields a compositional analysis, as we explain below.

The behavior of synchronized loops is difficult to analyse because they may contain data-races that span more than one iteration. For instance an instruction of iteration <sup>r</sup> in Listing 2.3 may race with an instruction of iteration <sup>r</sup>+1.

*Our Approach.* In this work we show that the DRF analysis of synchronized protocols can safely be reduced to a first-order logic formula when such loops are *aligned*, *i.e.*, when there is at least one synchronization exactly before the loop and one at the end of its body. In Sect. 4.1 we show how to transform an arbitrary access memory protocol into an aligned protocol using a syntax-driven transformation technique called *barrier aligning*. Intuitively, barrier aligning normalizes loops so that they do not "leak" accesses between iterations. The right-hand side of Listing 2.3 shows the result of applying *barrier aligning* on the protocol from Listing 2.3 (left). Observe that the fragment before the aligned loop (line 1) corresponds to the unsynchronized part of the original loop (before sync). The original loop itself is rearranged so that the part succeeding sync is moved to the beginning of the aligned loop (lines 3–6). The fragment following the aligned loop (line 7) corresponds to the unsynchronized loop that appears after the sync in the original protocol.

In Sect. 4.1 we show that aligned protocols enable *compositional* verification: protocol fragments between two barriers can be analyzed independently. This compositional analysis is possible because (i) there is no causality between instructions, except through sync and (ii) aligned protocols syntactically delimit the causality induced by sync. For instance, the aligned protocol in Listing 2.3 can be reduced to analyzing the following three protocol fragments independently:

$$\text{for}^{\mathsf{U}} \, i \in 0..M \, \{\mathsf{wr}[\text{tid}]\} \qquad \text{for}^{\mathsf{U}} \, j \in 0..M \, \{\mathsf{rd}[\text{tid}+j]\}$$

$$\text{for}^{\mathsf{S}} \, r \in 1..N \, \{\mathsf{for}^{\mathsf{U}} \, j \in 0..M \, \{\mathsf{rd}[\text{tid}+j]\}; \text{for}^{\mathsf{U}} \, i \in 0..M \, \{\mathsf{wr}[\text{tid}]\}; \text{succ}\}$$

The first two protocols are handled like Listing 2.2 because they are unsynchronized. Representing a synchronized loop as a formula becomes possible when the protocol is *aligned*: both threads must share the same value for r at each iteration. Hence, we reduce the verification to asking whether for all N, M, t1, and t<sup>2</sup> where t<sup>1</sup> -= t<sup>2</sup> and the following holds:

$$\begin{aligned} \forall \overline{\eta}, j\_1, i\_1, j\_2, i\_2 &\colon 1 \le r < N \choose 0 \le j\_1 < M \land 0 \le i\_1 < M \land 0 \le j\_2 < M \land 0 \le i\_2 < M\\ \implies \{\mathrm{rd}[t\_1 + j\_1] \} &\cup \{\mathrm{wr}[t\_1] \} \quad DRF \; with \, \theta \quad \{\mathrm{rd}[t\_2 + j\_2] \} \to \{\mathrm{wr}[t\_2] \} \end{aligned}$$

Our technique identifies Listing 2.3 as racy since this formula is *unprovable*, *i.e.*, rd[t1+j1] races with wr[t2] when <sup>r</sup> = 1, <sup>t</sup><sup>1</sup> = 0, <sup>t</sup><sup>2</sup> = 1, <sup>j</sup><sup>1</sup> = 1, N > <sup>1</sup> and M > <sup>1</sup>.

#### 3 Access Memory Protocols

An access memory protocol describes the interaction between a group of threads and a single shared-memory location. A protocol records *where* in memory accesses take place, but abstracts away from *what* data is read from/written to memory. The language of protocols distinguishes between an unsynchronized protocol fragment u ∈ U, that disallows synchronization, from a synchronized fragment p ∈ S that must include a synchronization. The syntax and semantics of access memory protocols is given in Figure 2. Our operational semantics is inspired by the synchronous, delayed semantics (SDV) from Betts et al. [6], where threads execute independently and communicate upon reaching a barrier.

Hereafter, i, j, k are metavariables over non-negative integers picked from the set N. An arithmetic expression n is either: an integer variable x, an integer i, or a binary operation on integers that yields an integer. A boolean expression b is either a boolean literal, an arithmetic comparison , or a propositional logic connective ◦. We write n ↓ i when expression n evaluates to integer i, where evaluation is defined in the natural way. We overload the notation for Boolean expressions, *e.g.*, b ↓ true means that expression b evaluates to true.

*Unsynchronized Fragment.* A protocol <sup>u</sup> ∈ U either does nothing (skip), accesses a shared memory location o[i] (reads from/writes to index i), performs sequential composition, or loops. Figure 2 gives the semantics of unsynchronized protocols, which is parameterized by a set of thread identifiers T ⊆ <sup>N</sup>, where |T | ≥ <sup>2</sup>.

Evaluation of an unsynchronized protocol u by a thread identifier i, written <sup>u</sup> <sup>↓</sup>*<sup>i</sup>* <sup>P</sup>, yields a *phase*, *i.e.*, a set <sup>P</sup> ∈ P of *access values* <sup>α</sup> <sup>∈</sup> <sup>A</sup>. Each access value, or just access, notation i:o[j], consists of its issuing thread identifier i, an access mode <sup>o</sup> (read/write), and an index <sup>j</sup>. Protocol skip produces no accesses. A memory access o[n] evaluates the index and creates a singleton phase. Sequencing and looping are standard. Loop ranges include the lower bound and exclude the upper bound. Similarly to SDV, Rule U-par executes a copy of the unsynchronized code <sup>u</sup> for each thread <sup>i</sup> ∈ T by replacing the special variable tid by the thread identifier, <sup>u</sup>[tid := <sup>i</sup>], which results in the union of the accesses of all threads. To simplify the presentation we omit the unsynchronized conditionals, however they are included in our Coq formalism and are fully supported by Faial, see Sect. 5.

Syntax


Big-step semantics for U u ↓*<sup>i</sup>* P u ↓ S

$$\begin{array}{cccc} \mathcal{U}\text{-SKIP} & \mathcal{U}\text{-ACC} & \mathcal{U}\text{-SEQ} \\ \hline \mathsf{skip }\downarrow\_{i}\varnothing & & \overline{o[n]\downarrow\_{i}\{i:o[j]\}} & \overline{\mathit{u}\_{1}\downarrow\_{i}P\_{1}} & \mathcal{u}\_{2}\downarrow\_{i}\mathit{P}\_{2} \\ \hline \end{array}$$

$$\begin{array}{cc} \mathcal{U}\text{-FOR-2} \\ \hline (n < m) \downarrow \mathtt{true} & u[x:=n] \downarrow\_i P\_1 & \text{for}^\Downarrow \ x \in n + 1..m \ \{u\} \downarrow\_i P\_2 \\ \hline \end{array}$$

$$\begin{array}{c} \mathcal{U}\text{-PAR} \\ \hline S = \bigcup \{u[\text{tid}:=i] \downarrow\_i P\_i \mid i \in \mathcal{T} \} \\ \hline \end{array}$$

History concatenation and merging H · H H

[P<sup>1</sup> ...P*n*] · [P*n*+1 ...P*n*+*k*]=[P<sup>1</sup> ...P*n*+*k*] (H · [P]) ([P- ] · H- ) = H · [P ∪ P- ] · H- Big-step semantics for S p ↓ H

$$\begin{array}{ccccc} \mathcal{S}\text{-SYNC} & \mathcal{S}\text{-PAR} & \mathcal{S}\text{-SEQ} & \mathcal{S}\text{-FOR-1} \\ \hline \text{sync}\downarrow[\emptyset,\emptyset] & & u\downarrow\_{T}\,P & & p\downarrow H & q\downarrow H' \\ \hline \text{sync}\downarrow[\emptyset,\emptyset] & & & p\downarrow H\odot H' & \text{f}\text{-FOR-1} \\ \end{array} \qquad \begin{array}{ccccc} \mathcal{S}\text{-FOR-1} & \mathcal{S}\text{-FOR-1} & \mathcal{S}\text{-FOR-1} \\ & (n\geq m)\downarrow\,\text{true} & \\ \hline \text{for } \mathcal{S} & x\in n.m \;\{p\}\downarrow[\emptyset] \\ \end{array}$$

$$\frac{\frac{\text{S-FOR-2}}{\text{( $n < m$ )} \downarrow \text{true}}{\text{ }} \quad p[x := n] \downarrow H \qquad \text{for }^{\text{g}} x \in n + 1..m \text{ } \{p\} \downarrow H'} $$

Structurally well-formed protocols *swf* (p)

$$\begin{array}{ccccc} swf(u;\text{sync}) & \frac{swf(p)}{swf(p;q)} & \frac{swf(p)}{swf(u\_1;\text{for}^\\$\ x\in n.\text{.}\{p:\}a/)} \\ \text{Data-race, safe phase, and safe history} & \boxed{\alpha \# \beta} & \boxed{safe(P)} & \boxed{safe(H)} \\ \hline \frac{wr\in\{o,o'\}}{:o[k]\#\ j:o'[k]} & \frac{\forall \alpha,\beta\in P\colon-(\alpha\#\beta)}{safe(P)} & \boxed{\frac{\forall P\in H:safe(P)}{safe(H)}} \end{array}$$

#### Fig. 2. Syntax, semantics, and properties of access memory protocols.

H

U-for-1

(n ≥ m) ↓ true for<sup>U</sup> <sup>x</sup> <sup>∈</sup> n..m {u} ↓*<sup>i</sup>* <sup>∅</sup>

*Synchronized Fragment.* A protocol p ∈ S may perform barrier synchronization sync, run unsynchronized code <sup>u</sup>, perform sequential composition, and loop. Figure 2 gives the semantics of a protocol, notation p ↓ H. Evaluation of a protocol p yields a *history* (ranged over by H), *i.e.*, a list of phases (P) that records how memory was accessed. We use :: as list constructor and · for the usual list concatenation operator. Histories are merged using the special -operator.

A barrier synchronization creates two empty phases, corresponding to phases before and after synchronization. Running an unsynchronized protocol yields a single phase containing all accesses performed by that protocol. Sequencing two synchronized protocols p with q merges the last phase of the former with the first phase of the latter, as these two phases run concurrently. The base case of a synchronized loop produces a singleton history containing the empty phase. Running one iteration of a synchronized loop sequences the history of the first iteration with the rest of the loop, by merging the two histories.

Next, we introduce the notion of well-formed protocols, a restriction of structurally well-formed protocols, see *swf* (p) in Figure 2. We discuss how wellformedness is enforced in Sect. 5. We write *fv*(p) (resp. *fv*(n)) for the free variables of p (resp. n).

Definition 1 (Well-formed protocol, p ∈ W). *We say that a protocol is wellformed, notation* <sup>p</sup> ∈ W*, when swf* (p)*, fv*(p) ⊆ {tid}*, and every synchronized loop executes at least one iteration.*

DRF is formalized at the bottom of Figure 2. Two accesses are in a data-race (or racy) when there exist two different threads that access the same index k, and one of these accesses is a write. Our definition does not distinguish between harmful and *benign* data races, a data-race in which both threads write the same value. Phase P is *safe* iff each pair of accesses it contains is not racy. History P is *safe* when all of its phases are safe. We say that p is DRF iff p ↓ H and *safe*(H).

#### 4 DRF-Preserving Transformations of Protocols

This section presents the main steps of the DRF analysis summarized in Figure 1: barrier aligning (Sect. 4.1) and splitting (Sect. 4.2).

This section also includes our key theoretical results. We establish that these steps preserve and reflect data-races (*i.e.*, any and all data-races are found), see Theorem 1 and Theorem 3. We make precise the notion of compositionality that makes our approach scalable in Theorem 2.

#### 4.1 Aligning Protocols

The first transformation step normalizes protocols by aligning synchronized loops, which in turn enables a form of compositional verification. The goal of the transformation is to produce protocols which belong to A, see top of Figure 3.

*Barrier aligning* (or just aligning) is performed by function *align*, given in the bottom half of Figure 3. The function returns a pair whose first element is an Aligned protocols p ∈ A <sup>u</sup> ;sync ∈ A <sup>p</sup> ∈ A <sup>q</sup> ∈ A p ; q ∈ A p ∈ A q ∈ A <sup>p</sup> ; for<sup>S</sup> <sup>x</sup> <sup>∈</sup> n..m {q}∈A Sequencing aligned protocols <sup>o</sup> <sup>9</sup> : U→A→A <sup>o</sup> <sup>9</sup> : (A×U) → (A×U) →A×U u <sup>o</sup> <sup>9</sup> (u- ;sync)=(u ; u- ) ;sync u <sup>o</sup> <sup>9</sup> (p ; q)=(u <sup>o</sup> <sup>9</sup> p) ; q (p, u) <sup>o</sup> <sup>9</sup> (q, u- )=(p ;(u <sup>o</sup> <sup>9</sup> q), u- ) Aligning protocols *align* : W→A×U *align*(u ;sync)=(u ;sync, skip) *align*(p ; q) = *align*(p) <sup>o</sup> <sup>9</sup> *align*(q) *align*(p)=(q, u3) q<sup>1</sup> = u<sup>1</sup> <sup>o</sup> <sup>9</sup> q[x := n] u = u<sup>3</sup> ; u<sup>2</sup> *align*(u<sup>1</sup> ; for<sup>S</sup> <sup>x</sup> <sup>∈</sup> n..m {<sup>p</sup> ; <sup>u</sup>2})=(q<sup>1</sup> ; for<sup>S</sup> <sup>x</sup> <sup>∈</sup> <sup>n</sup>+1..m {u[<sup>x</sup> := <sup>x</sup>−1] <sup>o</sup> <sup>9</sup> q}, u[x := m−1])

Fig. 3. Aligning protocols.

aligned and synchronized protocol, and whose second element is an unsynchronized protocol. Intuitively, the pair represents a sequence which we delimitate syntactically. We note that the output of *align*, say (q, u), can be trivially made into an aligned protocol: <sup>q</sup>; <sup>u</sup>;sync. The case for synchronization is simple, *align* returns the input protocol as the first component of the pair and skip as the second component (the input protocol is already fully aligned). The case for sequence consists of the sequential composition of the pair aligned with unsynchronized code using operator (<sup>o</sup> <sup>9</sup>). Sequencing two pairs (p, u) <sup>o</sup> <sup>9</sup> (q, u ) amounts to sequencing u to the outer-most piece of unsynchronized code present in q.

Dealing with synchronized loops is more involved. Given a loop <sup>u</sup>1; for<sup>S</sup> <sup>x</sup> <sup>∈</sup> n..m {p; u2}, we produce a protocol consisting of the fragment preceding the loop and the synchronized part of its first iteration (q1), an aligned loop starting at n+1, and the unsynchronized part of its last iteration (u[x := m−1]). See Listing 2.3 for an example of protocol aligning. We note that we can always unroll the loop because the analysis only considers non-empty synchronized loops; we discuss how to enforce this assumption in Sect. 5.

We now state two fundamental properties of barrier aligning: preserving and reflecting DRF (Theorem 1), and enabling compositional verification (Theorem 2). Theorem 1 states that verifying DRF of a well-formed protocol p is equivalent to verifying DRF of its aligned counterpart.

Theorem 1 (Correctness of Align). *If* p ∈ W *and align*(p)=(q, u)*, then* p *is DRF if and only if* q; u *is DRF.*

To state our compositionality result, we introduce a language of contexts:

<sup>C</sup> ::= [ \_ ] <sup>|</sup> <sup>u</sup>;sync <sup>|</sup> <sup>p</sup>; C|C; <sup>p</sup> | C; for<sup>S</sup> <sup>x</sup> <sup>∈</sup> n..m {p} | <sup>p</sup>; for<sup>S</sup> <sup>x</sup> <sup>∈</sup> n..m {C}

Syntax

L <sup>h</sup> ::= skip <sup>|</sup> <sup>n</sup>:o[m] <sup>|</sup> <sup>h</sup> ; <sup>h</sup> <sup>|</sup> var <sup>x</sup> in n..m; <sup>h</sup>

Product of histories H ⊗ H

$$H\_1 \otimes H\_2 = \left[P\_1 \cup P\_2 \: \mid \: \left(P\_1, P\_2\right) \in H\_1 \times H\_2\right]$$

Big-step semantics h ⇓ H

skip ⇓ [∅] n ↓ i m ↓ j n:o[m] ⇓ [{i:o[j]}] h<sup>1</sup> ⇓ H<sup>1</sup> h<sup>2</sup> ⇓ H<sup>2</sup> h<sup>1</sup> ; h<sup>2</sup> ⇓ H<sup>1</sup> ⊗ H<sup>2</sup> (n ≥ m) ↓ true var <sup>x</sup> in n..m; <sup>h</sup> ⇓ [∅] (n<m) <sup>↓</sup> true <sup>h</sup>[<sup>x</sup> := <sup>n</sup>] ⇓ <sup>H</sup><sup>1</sup> var <sup>x</sup> in <sup>n</sup> + 1..m; <sup>h</sup> ⇓ <sup>H</sup><sup>2</sup> var <sup>x</sup> in n..m; <sup>h</sup> ⇓ <sup>H</sup><sup>1</sup> · <sup>H</sup><sup>2</sup>

Projection *trace* : U→L

$$\begin{array}{cc} \mathit{trace}(o[n]) = \mathit{td:o}[n] & \mathit{trace}(\mathit{for}^{\mathsf{U}} \ x \in n..m \ \{u\}) = \mathsf{var} \ x \ \mathsf{in} \ n..m; \mathit{true}(u) \\\\ \mathit{trace}(u\_1; u\_2) = \mathit{trace}(u\_1); \mathit{trace}(u\_2) & \mathit{trace}(\mathsf{skip}) = \mathsf{skip} \\\\ \mathit{Splitting}(\mathit{protok}) & \overline{\mathit{spotocols}} \end{array}$$

$$split(p;q) = split(p) \cdot split(q)$$

$$\begin{array}{lcl} \hline t\_1, t\_2 \text{ fresh} & h\_1 = \text{trace}(u)[\text{tid} := t\_1] & h\_2 = \text{trace}(u)[\text{td} := t\_2] \\ \hline \\ \text{split}(u; \text{sync}) = [\text{var } t\_1 \text{ in } 1..|\mathcal{T}|; \text{var } t\_2 \text{ in } 0..t\_1; h\_1 ; h\_2] \\ \hline \\ \text{split}(p; \text{for}^{\text{g}} \ x \in n. m \text{ } \{q\}) & = \text{split}(p) \cdot [\text{var } x \text{ in } n. m; h \ | \text{ } h \in \text{split}(q)] \\ \hline \end{array}$$

The base cases correspond to a hole [ \_ ] or an unsynchronized protocol (followed by sync). The other cases follow the structure of access memory protocols.

Theorem 2 (Compositionality). *Let* <sup>C</sup> *be a context, s.t.* <sup>C</sup>[skip;sync] *is DRF. For all* <sup>p</sup> ∈ A*, if* <sup>p</sup> *is DRF, fv*(p) ⊆ {tid}*, then* <sup>C</sup>[p] ∈ A *and* <sup>C</sup>[p] *is also DRF.*

Compositionality allows Faial to analyze each fragment of an aligned protocol independently, by splitting the given protocol into multiple symbolic traces.

#### 4.2 Splitting Protocols into Symbolic Traces

The second verification step, *splitting*, consists in transforming an aligned protocol into *symbolic traces*, *i.e.*, symbolic representations of sets of memory accesses which occur between two synchronizations.

*Symbolic Traces.* Intuitively, symbolic traces are a thin abstraction over an SMT formula. We describe how to translate a symbolic trace to a formula in Sect. 5.

We give the syntax and semantics of symbolic traces in Figure 4. Expression skip terminates a trace. Expression <sup>n</sup>:o[m] states that thread <sup>n</sup> accesses index m with mode o. Expression h1; h<sup>2</sup> composes two symbolic traces using operator <sup>⊗</sup>, also given in Figure 4. Expression var <sup>x</sup> in n..m; <sup>h</sup> binds variable <sup>x</sup> in h, where variable x is an integer in the range induced from n and m. The semantics of a symbolic trace yields a history with a phase for each possible variable assignment. Expression skip yields a single empty phase. Expression <sup>n</sup>:o[m] evaluates to a singleton set that contains the access value that results from evaluating the thread-identifier expression n and the index expression m. Sequencing histories h1; h<sup>1</sup> consists of performing the product of phases (operator ⊗), which consists of merging every phase of H<sup>1</sup> with every phase of H2. A variable binder behaves like a skip when the range of values is empty. Otherwise, we fork two histories H<sup>1</sup> and H2. We assign the lower bound of the set in H1, and we recursively evaluate a variable binder where we increment its lower bound in H2.

*Barrier splitting* is the transformation from aligned protocols to symbolic traces, performed via functions *trace* and *split*, defined in Figure 4. Function *trace* extracts the symbolic trace of an unsynchronized program for a single thread. Memory accesses are tagged with the owner thread tid, and unsynchronized loops are converted into variable bindings. Function *split* returns a list of symbolic traces. The case for p; q is trivial (operator · stands for list concatenation). The base case of *split* is for unsynchronized protocol fragment u, which produces a list containing a single symbolic trace. It introduces fresh variables t<sup>1</sup> and t<sup>2</sup> that represent two (distinct) symbolic thread identifiers. The rest of the trace consists of the trace of u instantiated to the first thread identifier t<sup>1</sup> followed by its instantiation to the second thread identifier t2. The case for synchronized loops simply reinterprets the loop as a variable binder. Function *split* leads to an exponential blow up wrt. nesting of synchronized loops, but this has not posed problems in practice, *c.f.*, Claim 2.

*Example 1.* Let <sup>p</sup><sup>ˆ</sup> <sup>=</sup> wr[tid + 1];rd[tid + 2];sync. We have that *split*(ˆp) returns:

$$\text{var } t\_1 \text{ in } 1..|T|; \\ \text{var } t\_2 \text{ in } 0..t\_1; \\ t\_1 ; \text{var}[t\_1+1]; \\ t\_1 ; \text{rd}[t\_1+2]; \\ t\_2 ; \text{var}[t\_2+1]; \\ t\_2 ; \text{rd}[t\_2+2]$$

We show that barrier splitting preserves and reflects DRF.

Theorem 3. *Let* p ∈ A*, such that* p ↓ H1*, and* H<sup>2</sup> = [H | h ∈ *split*(p)∧h ⇓ H]*, then safe*(H1) *if and only if safe*(H2)*.*

Hence we have established that aligning (Theorem 1) and splitting (Theorem 3) preserve and reflect data-races, *i.e.*, any and all data-races are found. Thus, the only source of approximation in our analysis stems from the inference of protocols from CUDA kernels, which we discuss in the next section. Theorem 3 highlights the compositionality of our analysis, as each symbolic trace resulting from function *split* can be analyzed independently.

#### 5 Implementation

In this section we present our tool, Faial, that implements the steps described in Figure 1. Faial takes a CUDA kernel as input and produces results that either identify the kernel as DRF or list specific data-races. In this section, we describe the implementation of the protocol inference, well-formedness checks, and transformation to SMT.

*Inference.* This step transforms a CUDA kernel into access memory protocols (one for each shared array). It uses libclang [23] to parse the kernel, a standard single static assignment (SSA) transformation to simplify the analysis of indices and arrays, and code slicing to only retain code related to *shared* array accesses. We note that Faial supports constructs of the CUDA programming model that are not directly modeled by access memory protocols, *e.g.*, unstructured loops, conditionals, function calls, and multi-dimensional arrays. To support multidimensional thread identifiers, we extend the language of protocols to support multiple thread identifiers, and adapt function *split* accordingly. The main challenges are related to loops and function calls.

Whenever possible loops are transformed to loops with a stride of 1 following ideas from loop normalization [24] and abstraction [30]. For instance, in for(int i=lb;i<ub;i+=s){S} we change the stride from <sup>s</sup> into <sup>1</sup> by executing the loop body S when the loop variable i is divisible by stride, *i.e.*, the loop becomes for(int i=lb;i<ub;i++) if((i+lb)%s==0){S}. Similarly, a loop ranging over powers of <sup>n</sup>, *e.g.*, for(int i=lb;i<ub;i\*=s), becomes for(int <sup>i</sup>=lb;i<ub;i++) if(powerof(i,s)){S}, where function powerof(i,s) tests whether <sup>i</sup> is a power of base s. We approximate whiles as a structured loop with an unknown upper bound.

Function calls that manipulate shared memory are uncommon in GPU programming. Additionally auxiliary functions that manipulate shared memory have a compiler annotation to inline their bodies, hence we can inline such calls easily. Faial cannot handle recursive functions, but these rarely occur in practice. Function calls that do not access shared memory are simply discarded.

*Well-Formedness.* This step ensures that kernels Faial analyzes meet the wellformedness conditions, *i.e.*, p ∈ W, including the assumptions that synchronized loops iterate at least once, see Definition 1. First, Faial annotates loops with a synchronized/unsynchronized tag according to the presence of sync in the loop body, then adjusts the precedence of sequencing to group all unsynchronized code preceding a sync or a synchronized loops. Synchronized loops of well-formed protocols cannot manipulate thread-local variables (*i.e.*, tid), an assumption shared by the CUDA programming model. Hence, Faial flags such kernels as erroneous. Next, Faial adds assertions before/after synchronized loops to check that the loop range is non-empty, *i.e.*, loops execute at least once. Similarly to loops, conditionals are tagged as synchronized or unsynchronized. Then, Faial inlines synchronized conditionals, *i.e.*, when a synchronized conditional is found, two copies of the input program are created and each copy is prefixed by a global assertion corresponding to the condition. Faial does not support synchronized conditionals that appear within synchronized loops. We have not found realworld kernels that include such a construction.

*Quantification.* This step transforms each symbolic trace (Figure 4) into an SMT formula, to check for *safety*, *c.f.*, Figure 2. The presented formalism assumes a decidable fragment. However, as CUDA programs may include multiplication in index expressions, Faial uses an undecidable logic (SMTLib's QF\_LIA). Essentially, the generated formula guarantees that the indices of array accesses are distinct when there is at least one write. We illustrate this straightforward transformation with Example 2.

*Example 2.* The formula generated from the trace in Example 1 is given below:

$$\begin{aligned} \left( \forall t\_1, t\_2 \colon 1 \le t\_1 < |\mathcal{T}| \land 0 \le t\_2 < t\_1 \land \left( \mathfrak{m}\_1 = \mathsf{w}r \lor \mathfrak{m}\_2 = \mathsf{w}r \right) \implies \\ \left( \left( \mathsf{idx}\_1 = t\_1 + 1 \land \mathfrak{m}\_1 = \mathsf{w}r \right) \lor \left( \mathsf{idx}\_1 = t\_1 + 2 \land \mathfrak{m}\_1 = \mathsf{rd} \right) \right) \\ \land \left( \left( \mathsf{idx}\_2 = t\_2 + 1 \land \mathfrak{m}\_2 = \mathsf{w}r \right) \lor \left( \mathsf{idx}\_2 = t\_2 + 2 \land \mathfrak{m}\_2 = \mathsf{rd} \right) \right) \land \left( \mathsf{idx}\_1 \neq \mathsf{idx}\_2 \right) \end{aligned}$$

where each symbolic access is translated to a conjunction representing its index (idx) and access mode (m). Observe that the formula enforces that indices idx<sup>1</sup> and idx<sup>2</sup> (executed by distinct threads) are different.

For multi-dimensional arrays, we generate one pair of indices per dimension, and check that at least one pair is distinct.

### 6 Experimental Evaluation

We evaluate Faial over several datasets and show how it fares against existing approaches. We structure this evaluation in three claims.


*Benchmarking Environment.* To make our evaluation reproducible, we developed a benchmarking framework to automate our experiments over the different tools and datasets. For Claim 1 and Claim 3, we designed a tool-agnostic file format for kernel functions and associated metadata (*e.g.*, expected result of DRF analysis,

Table 1. Results for Claim 1. DRF indicates that a (static analysis) tool reported a test case as DRF. NRR indicates that a (symbolic execution) tool did not report any data-race. Label *x/y* indicates that the tool reported *y* data-races, *x* of which are actual races. Label *timeout* indicates that the tool did not terminate within 90s. A test passes if the tool returns the expected result and all reported races are valid.


grid and block dimensions, and include directives). And for Claim 2, we created a tool that generates kernels according to given templates, *e.g.*, see Figure 7.

We evaluate Faial against the following verification tools: GPUVerify [5] v2018- 03-22; PUG [24] v0.2; and, GKLEE [26] and SESA [27] v3.0. Experiments for Claim 1 use an Inteli5-6500CPU, 7.7 GBRAM, andFedora 33OS,whileClaim2andClaim3 use an Intel i7-10510U CPU, 16 GB RAM, and Pop! OS.

*Excluded Tools.* We excluded ESBMC-GPU [33] and Simulee [37] from the evaluation because we were unable to get them to run satisfactorily. Both tools have rudimentary support for verifying arbitrary CUDA kernels. ESBMC-GPU did not find a single data-race in our benchmarks, while Simulee produced false alarms for every DRF-kernel given.

#### Claim 1: Correctness

We have selected a set of tricky kernels to expose false alarms and missed dataraces in Faial, GPUVerify, PUG, GKLEE, and SESA. Our results are reported in Table 1. The dataset consists of 5 tests, each consisting of two variations of the same kernel: one racy and one DRF. The racy version of Test 1 (*c.f.*, Listing 2.1) contains an inter-iteration data-races. The DRF version adds a sync after the second inner loop. Tests 2 to 4 expose various loop-related data-races. Their protocols are given in Figure 5. In the racy version of Test 2 wr[tid + 1] conflicts with wr[tid] of the first iteration. Similarly, in the racy version of Test 3, wr[tid + 1] of the last iteration races with wr[tid]. In the racy version of Test 4 the last iteration of a nested loop races with the first iteration of the following loop. Test 5 exposes the abstraction gap between kernel and access memory protocols (which abstract away array elements), see Figure 6.

Faial passes more tests than any other tool. Failed Test 5 is caused by access memory protocols abstracting away from *what* data is being read from/written

Fig. 5. Protocols for Tests 2 to 4, *c.f.*, Claim 1, where N is a free thread-global variable. Yellow shaded code only appears in the DRF version of first-iter and last-iter. Red shaded code only appears in the racy version of last-iter-first-iter (Color figure online).


Fig. 6. Kernels and protocols for Test 5 (read-index), *c.f.*, Claim 1; x becomes a free thread-local variable as protocols do not model array elements.

to arrays, *i.e.*, array elements. In each case, Faial reports one spurious data race (*0/1*). We report on performance trade-offs wrt. tracking array elements in Claim 2.

GPUVerify passes Test 5 because it tracks array elements, but fails the remaining 4 tests. Some reported false alarms are ill-formed, *e.g.*, on the racy component of Test 2, the report (0 : wr[tid]; 16 : wr[tid]) has disjoint indices.

PUG obtains the worst score amongst static tools. Notably, the tool misses a data-race in Test 1, demonstrating its unsoundness, *c.f.*, Sect. 2.1.

GKLEE and SESA timeout for tests that include loops, as the loop bounds are unknown.Both tools miss the data-race in Test 5. Symbolic tools may be able to report data-races when the bound is known, *e.g.*, timeouts start in Test 1 when the bound is at least 2, in Test 2 when the bound is at least 23, 000.

#### Claim 2: Scalability

We evaluate the scalability of our approach with a synthetic dataset that aims at demonstrating how different kernel constructs affect run time and memory usage of Faial, GKLEE, GPUVerify, PUG, and SESA. Our dataset is divided into five categories, one per syntactical construct in the language of access memory protocols, as well as conditionals, which are supported by our inference step, *c.f.*, Sect. 5. Figure 7 shows the protocols of the kernel patterns we generate in each category: (i) repeated accesses (read then write), (ii) repeated barrier synchronizations separated by writes, (iii) repeated conditionals, (iv) increasingly nested unsynchronized loops, and (v) increasingly nested synchronized loops. In each category, we vary the problem size by repeating a pattern from 1 to 50 times. Note that all kernels generated this way are DRF.


Fig. 7. Synthetic protocols generated for Claim 2. N is a free thread-global variable, and n1, n2. . . are positive integer literals.

Figure 8 shows the average run time and memory usage over five runs on logarithmic and linear scales, respectively. For each run, we set a timeout of 90s and we exclude any run that times out or reports a false alarm. Cutoffs in the memory plots are determined by the cutoffs in the run time plots.

Overall Faial is the most scalable tool. In 4 out of 5 categories, Faial has the slowest growth for all experiments, and verifies all tests within 0.46 s. In the largest problem sizes, our tool is the fastest in 3 categories (access, conditional, unsynchronized loop), 2nd for barriers, and 3rd for synchronized loops. Overall, the memory usage of Faial is competitive with other tools. Faial is the only tool with a near constant time/memory for up to 50 unsynchronized loops, indicating the scalability of reducing unsynchronized loops to universally quantified formulas. Faial only times out for kernels which consists of >17 nested synchronized loops. However such kernels are uncommon, *e.g.*, the levels of nested synchronized loops in the real-word kernels studied in Claim 3 are at most 3.

GPUVerify remains stable in the barrier and conditional categories but is affected negatively by loops and accesses. Loops are a known bottleneck in GPUVerify [2]. In the access category there is an exponential slowdown due to GPUVerify keeping track of what data is being written to/read from array.

PUG tool remains stable with the number of barrier synchronizations but is affected negatively by the number of conditionals and loops. PUG is the fastest tool with smaller inputs, but it raises false alarms in the access category, hence these measurements are omitted from the corresponding plots.

We discuss GKLEE and SESA together since SESA processes GKLEE's NVCC byte code output by concretizing variables, before passing it to GKLEE itself. There are two main factors that affect negatively these symbolic execution tools: (i) the number of loops, since they unroll each loop; and (ii) the amount of bookkeeping required to keep track of what is read from/written to memory. Figure 8 shows clear exponential curves for the access and barrier synchronization categories. Observe that these tools timeout immediately in the loop categories.

#### Claim 3: Real-World Usability

We evaluate the usability of our approach by comparing Faial with other static verification tools (GPUVerify and PUG) on real-world kernels wrt. rate of false alarm and run time. We curated a set of CUDA kernels from [2], which consists of 3 benchmark suites (totaling 227 CUDA kernels): NVIDIA GPU Computing

Fig. 8. Results for Claim 2. Run time (left plots) are given on a logarithmic scale, and memory (right plots) are given on a linear scale. Flatter and lower curve is better. Tools annotated with a triangle are excluded due to timeouts or errors.

SDK v2.0 (8 CUDA kernels); NVIDIA GPU Computing SDK v5.0 (166 CUDA kernels); Microsoft C++ AMP Sample Projects (20 kernels); gpgpu-sim benchmarks (33 kernels). All kernels are DRF and have been pre-processed by the authors of [2] to facilitate verification. Each kernel is in a distinct file, all dependencies are available, and kernels are annotated with minimal pre-conditions to allow for automatic analysis (*e.g.*, thread count is given).

(d) Run time (top) and memory usage (bottom) of true-positives. Time (resp. memory) is cropped at 10s (resp. 100MB) and plotted on a logarithmic (resp. linear) scale.

Fig. 9. Results for Claim 3, on a set of 227 DRF CUDA kernels.

As we aim to evaluate fully automatic verification of three tools, we removed code annotations (pre-conditions and loop invariants) specific to GPUVerify. Additionally, we made minor changes to some kernels to meet the limitations of the front-end of Faial and PUG. For instance we converted nested array lookups to use temporary variables and inlined functions calls that operate on arrays in 22 kernels. Another 8 kernels were modified to simplify their control flows. Our curated dataset will be included in our artifact submission.

Figures 9a, b, and c give the correctness results of Faial, GPUVerify, and PUG, respectively. Correct refers to the true-positive rate, *i.e.*, when the tool correctly identifies the kernel as DRF. False Alarm refers to the false alarm rate, *i.e.*, when the tool incorrectly identifies the kernel as racy. A kernel is Unsupported if it makes the tool crash. A Timeout occurs when the tool exceeds the limit of 60s to verify a kernel. The values shown are an average calculated over five runs. Figure 9d shows the average run time and memory usage of every true-positive report (we omit invalid reports) across the three tools.

Overall Faial has the highest rate of true-positives at 96%. Our tool is second in terms of run time and memory usage, showing a good compromise w.r.t. time and space. Faial verifies most kernels within 1s, and all kernels that need more time are only verified by Faial. GPUVerify shows lower memory usage at the cost of a higher verification run time. PUG verifies the lowest number of kernels (34.8%), as most kernels are unsupported (62.6%).

### 7 Related Work

*SMT-Based DRF Analyses.* Li and Gopalakrishnan propose a direct encoding of DRF analysis of GPU programs in SMT, with PUG [24,25]. Both PUG and Faial follow a similar approach of barrier splitting: having a symbolic representation of a canonical interleaving, and dividing up the analysis over barrier intervals. The two major distinctions are that (1) PUG misses inter-thread data-races in synchronized loops, *e.g.*, Listing 2.1, and (2) the algorithms of PUG are unspecified and lack soundness proofs. In [24, Sect. 6.3] the authors identify the challenge of detecting inter-thread data-races, but do not elaborate a solution. Ma *et al.* [30] present a similar technique to detect data-races and deadlocks in OpenMP programs (CPU-based parallelism). However, their work does not guarantee DRF, and they do not formalize their algorithms. In [8], Prasanth *et al.* propose a polyhedral encoding of DRF for OpenMP programs, which is only applicable to programs with affine array accesses. However the prevalence of linearized array expressions in GPU kernels is known to stump polyhedral analysis [16].

*Hoare-Logic-Based DRF Analyses.* The main drawback of Hoare-logic based tools is their high rate of false alarms. They also require code annotations from a concurrency expert to handle loops. GPUVerify [2,3,5,6,12] can verify CUDA and OpenCL kernels using Boogie [4] as a backend. GPUVerify also relies on a two-thread abstraction (pen and paper proof)—in this paper, we present the first *machine-checked* proof of the two-thread abstraction idea. VeriCUDA [20,21] focuses on reasoning about the functional correctness of GPU programs using Hoare-logic. In [22] the authors extend VeriCUDA to proving DRF. In a similar vein, VerCors [7] uses separation logic to prove the functional correctness and DRF of GPU kernels. Both VeriCUDA and VerCors expect a tool-specific language, hence cannot handle real-world kernels directly.

*Data-Race Finders.* These include dynamic data-race detection, symbolicexecution, and model-checking. Such techniques are better suited for highly detailed analysis in smaller kernels, and typically are unable to prove DRF. Dynamic data-race detection executes a kernel to find data-races on a fixed input, *e.g.*, [14,18,19,28,32,38,39]. This technique only reports real data-races, but suffers from a slowdown of at least 10× compared to the non-instrumented program, and requires the kernel input data, which might be unavailable or unknown. Symbolic execution and model checking have been extended to detect data-races [10,11,26,33,37]. These techniques do without the kernel input data and can detect more data-races than dynamic data-race detection.

*Miscellaneous.* Ferrel *et al.* introduce a machine-checked formalism to reason about the semantics of CUDA assembly [15]. Dabrowski *et al.* mechanize the DRF-analysis of multithreaded programs [13]. Muller and Hoffmann present a logic to reason about the evaluation cost of CUDA kernels [31].

Other behavioral types have been used to verify parallel and multithreaded systems that communicate via message-passing [29,35,36]. However these do not capture shared memory (only message-passing), thus cannot address data-races.

#### 8 Conclusion

We tackle the problem of statically checking DRF in GPU kernels, with a new family of behavioral types, *i.e.*, access memory protocols. We provide a novel compositional analysis of access memory protocols, along with fully mechanized proofs and an implementation. Our evaluation explores challenging and diverse benchmarks (229 real-world and 258 synthetic kernels) to demonstrate that our approach is more precise (false alarms and missed alarms), scalable (time/memory growth), and usable (real-world kernels correctly verified) than other tools.

Acknowledgements. We thank Rumyana Neykova, Stephen Chang, and the anonymous reviewers for their insightful feedback on earlier versions of this work.

#### References


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **GENMC: A Model Checker for Weak Memory Models**

Michalis Kokologiannakis(B) and Viktor Vafeiadis

MPI-SWS, Kaiserslautern, Germany *{*michalis,viktor*}*@mpi-sws.org

**Abstract.** GenMC is an LLVM-based state-of-the-art stateless model checker for concurrent C/C++ programs. Its modular infrastructure allows it to support complex memory models, such as RC11 and IMM, and makes it easy to extend to support further axiomatic memory models.

In this paper, we discuss the overall architecture of the tool and how it can be extended to support additional memory models, programming languages, and/or synchronization primitives. To demonstrate the point, we have extended the tool with support for the Linux kernel memory model (LKMM), synchronization barriers, POSIX I/O system calls, and better error detection capabilities.

#### **1 Introduction**

For any software developer or verification engineer, it is no news that concurrent programming is difficult, that concurrent software is often buggy, and that therefore verification of concurrent programs has attracted a lot of research interest. Within the verification community at least, it is also common knowledge that verification of concurrent programs is challenging because of the huge number of interleavings of the threads comprising a concurrent program.

What has changed in the last decade, however, is the importance of *weak memory consistency* [6,11,13,14,21,25,32,36,40,41] as a key factor contributing to the complexity of concurrent programming. Weak memory models do not simply increase the number of thread interleavings; they also confound programmers, who typically have little intuition about how to reason about the behaviors induced by these additional interleavings.

GenMC is a fully automatic verification tool meant for such programmers. It is a *stateless model checker* (SMC) [23] that can be used to verify bounded clients of intricate concurrent algorithms, such as implementations of synchronization primitives and shared data structures (e.g., queues, sets, and maps). It accepts as input a C/C++ program using C/C++11 atomics and/or the concurrency primitives from the pthread library, and reports any data races, assertion violations, or other errors encountered. By default, verification is performed with respect to the RC11 memory model [32], but there are command line options for selecting other models, such as IMM [41] and LKMM [10].

Since the theory underlying GenMC has already been published elsewhere [28,29,31], this paper focuses on the overall design of the tool and on various enhancements implemented in it. Our main design goals of GenMC were:


**Usability:** The tool should provide useful and readable error messages.

**Extensibility:** The tool should be easily adaptable to support additional models and synchronization primitives, and to tweak its performance. Extensibility is key to achieving the other goals, since it allows gradual improvements to the tool in terms of coverage, performance, and error detection/reporting.

These goals are achieved by a combination of techniques:

GenMC's core SMC algorithm [29,31] is parametric in the choice of the memory model—subject to a few minimal constraints (see Sect. 2).

The implementation is based on LLVM, a versatile intermediate language for multiple programming languages.

GenMC follows a modular architecture minimizing dependencies across components (see Sect. 3), which makes it easy to extend with support for additional memory models (Sect. 4) and synchronization primitives (Sect. 5).

Its architecture contains hooks to provide fast approximate consistency checks, which are exploited by the memory model implementations (see Sect. 4).

GenMC contains a number of optimizations that provide noticeable performance benefits on common workloads (Sect. 7).

GenMC keeps additional metadata so as to present error messages in terms of variables names appearing in the source code (Sect. 6).

GenMC has been applied to a few industrial settings, where it has found bugs and/or verified bounded correctness of concurrent libraries [39].

*Related Work.* There has been extensive work on SMC, with most tools focusing on sequential consistency [7,8,15,23,37]. Tools that support weak memory models include CDSChecker [38] that verifies C/C++11 programs under the original C11 memory model, Tracer [5] that verifies C/C++11 programs under the RA model, RCMC [27] that verifies C programs under RC11 [32], and Nidhugg [1,2,4,12,13] that supports SC, TSO, PSO and provides limited support for the POWER and ARMv7 memory models. In contrast to GenMC, which uses the same core algorithm for all memory models, Nidhugg uses multiple different algorithms depending on the memory model.

There has also been work on adapting SAT/SMT-based bounded model checking (BMC) techniques for weak memory models [9,17,22]. Dartagnan [22] is a BMC tool that is parametric in the choice of the memory model, as it accepts the memory model as input in the litmus format [11].

#### **2 Memory Model Requirements**

GenMC's core algorithm is parametric in the choice of the memory model provided that it can be expressed in an axiomatic way and satisfies a few basic requirements that we describe below.

Axiomatic memory models represent the executions of a concurrent program as *execution graphs* [11] that satisfy a certain consistency predicate. Execution graphs comprise a set of *events* (nodes) that represent the individual memory accesses performed by the program, and some relations on these events (edges). Example relations included in all memory models are the *preserved program order* (ppo) and *reads-from* (rf) relations: ppo relates events in the same thread that are ordered (e.g., by a chain of dependencies or a fence), while rf relates writes to reads reading from them.

GenMC can be used to verify programs under such a model as long as the model's consistency predicate fulfills the following requirements:


These requirements are satisfied by almost all axiomatic memory models (e.g., TSO [40], PSO [42], Power [11], ARMv7 [11], ARMv8 [21], RC11 [32], IMM [41], LKMM [10]). The only known axiomatic memory model that does not satisfy these requirements is the original formulation of the C/C++11 model [13], which has been criticized for its flaws [32,43].

Although these requirements cannot be satisfied by more advanced memory models that cannot be defined in an axiomatic fashion (e.g., [14,24,25,33]), there is ongoing work to support such a model.

#### **3 Tool Architecture**

Verification with GenMC comprises three stages (cf. Fig. 1, left).

The first stage invokes clang to compile the source C/C++ program to LLVM-IR. To accommodate programs written in different languages, GenMC also accepts LLVM-IR as its input, provided that it adheres to certain conventions about thread creation.

The second stage transforms the LLVM-IR code to make verification more effective by replacing spinloops by assume statements, bounding infinite loops,

**Fig. 1.** Overal architecture (left) and dynamic components (right).

and performing sound optimizations, such as dead allocation elimination. It also collects additional debugging information to enable better error reporting.

The third stage invokes the verification procedure, which explores all the executions of the program. If an error is found during this stage, the execution is halted and an error report is produced (see Sect. 6).

The architectural subcomponents of this stage are depicted in Fig. 1 (right). At the center lies the verification *driver*, which owns three independent components: an execution graph, a work set, and an interpreter.

The *execution graph* records the visited execution trace, and has routines for calculating various relation on the graph, such as the happens-before relation. As each memory model comprises different relations, the execution graph contains multiple calculators that are dynamically populated when the graph is created, and the consistency predicate is calculated as a fixpoint of all the selected relations, whenever this is requested by the driver.

The *work set* records alternate options for later exploration, the precise definition of which can depend on the memory model.

The *interpreter* merely executes the user program, notifying the driver each time a "visible" action (e.g., a load/store to shared memory) is encountered. It is directly based on the LLVM interpreter lli [35], and is the only part of our code base that heavily depends on LLVM. In turn, the driver modifies accordingly the execution graph, possibly pushes some items to the work set, and returns control back to the interpreter, along with a value that will be used by the interpreter, if necessary (e.g., in the case of a load). In effect, the driver and the interpreter can be thought of as *coroutines* [18]. The interpreter calls the driver whenever it encounters a visible action or finishes running a thread, while the driver monitors execution consistency, schedules the program threads, and discovers alternative exploration options, which are pushed to the work set.

The aforementioned components are all parameterized by the user's configuration options. The most important of these options is the memory model, which also determines whether dependencies between instructions should be tracked by the interpreter and stored in the execution graph. Another important option is when and how consistency is to be calculated. Since checking consistency at each step can be expensive for some memory models, it is possible to provide an approximate consistency check to be applied at each step and only perform the full consistency check once an error is detected.

To facilitate memory-model-specific optimizations, the driver is overridden for each memory model. Each instance sets up the (approximate) consistency checks and can provide specialized methods for crucial verification components.

#### **4 Supporting New Memory Models**

Adding support for a new memory model entails three basic steps.

First, one has to provide definitions for any memory model primitives that the interpreter should intercept beyond those already supported (i.e., plain memory accesses and C/C++11 atomics). One can either provide a header file mapping these primitives to LLVM-IR instructions or create special event types for them.

Second, one has to provide calculators for the memory model's relations that are not already supported by GenMC. Depending on the memory model, this step may require a variable amount of effort, but it effectively boils down to translating relational calculations into matrix operations.

Third, one can also provide approximations for the consistency checks. Such approximations entail storing crucial information about a memory model's relations as vector clocks (e.g., causally preceding events, for some notion of causality), but deciding what to store is up to the user to decide and encode. Importantly, GenMC's performance depends not only on the calculators provided in the previous step, but also on the effectiveness of the approximations, which quickly filter out inconsistent exploration options. For instance, GenMC's current RC11 driver treats SC accesses as release-acquire (RA) accesses (the consistency of which can be quickly determined), and only checks for full RC11 consistency when an error has been triggered, a heuristic that seems to work well in practice for programs that have both SC and non-SC accesses.

All in all, adding support for a memory model largely depends on the complexity of the model. Adding support for models like SC or RA is trivial, since such accesses are already supported as part of RC11 and IMM. In contrast, adding support for LKMM involved much more work, as we describe below.

#### **4.1 Supporting the Linux Kernel Memory Model (LKMM)**

LKMM [10] is a memory model that encompasses a variety of different architectures supported by the Linux kernel. As LKMM differs substantially from RC11 and IMM, supporting it required all steps described above as well as a few other engineering decisions, the most important of which are discussed below.

First, LKMM uses complex constraints for checking consistency of an execution graph. As repeatedly calculating these constraints can be expensive, we designed approximations for them. Unlike most other memory models, LKMM does not define a suitable happens-before relation for checking coherence and detecting races. (Its hb relation cannot be used for this purpose.) We thus defined a custom happens-before relation that can rule out inconsistent executions very quickly, and use it to approximate coherence and race detection checks.

Second, although LKMM dictates that non-atomic accesses (called plain in LKMM's jargon) only conditionally contribute to ppo, we incorporate such accesses in GenMC's ppo (thus arriving at a stronger notion of ppo), mostly for technical reasons. Specifically, the calculation of dependencies between only non-plain accesses is difficult because each non-plain access in the source-code level may map to several plain and non-plain accesses in LLVM-IR level.

To increase confidence in our implementation, we ran all litmus tests distributed along with LKMM as part of the Linux kernel (32 tests in total), and compared our results with the results of the Herd [11] memory model simulator. Both tools explored the same number of executions for all tests.

In addition, we extracted some manually written tests from LKMM's supplementary repository [34] (categories atomic and kernel). We picked these categories as they contain tests written in C pseudocode (thus easily translatable to C) and do not contain tests with plain accesses, which, as described, GenMC treats slightly differently from what LKMM dictates. In total, these categories amount to another 84 tests, from which we excluded two tests containing unsupported primitives, one test for which Herd did not terminate within 42 h, and three tests that cannot be cleanly translated to C. Out of the remaining 78 tests, GenMC explores the same number of executions for 75 tests. The discrepancies observed in the three remaining tests are due to the different way the two tools produce and calculate dependencies. (In GenMC, control dependencies extend to all subsequent memory accesses of the same thread, whereas in Herd they extend only to the merge point of a conditional statement.)

We note that Herd took about 18 min to run all the above tests, while GenMC needed less than 2 s.

#### **5 Supporting New Languages and Libraries**

Supporting additional programming languages is straightforward as long as they can be compiled to LLVM. This was, for example, the case when we extended GenMC to accept C++ (the initial version accepted only C input). All we had to do was to create stub header files for the C++ library, and to extend the interpreter to recognize the memory (de)allocation calls generated by clang.

Supporting different runtime environments (e.g., JVM bytecode) requires constructing a new interpreter for the desired runtime system that calls the driver whenever a visible action is encountered. In addition, since the driver and the interpreter communicate using the LLVM type information, it may be necessary to add a translation layer between the interpreter(s) and the driver.

Supporting new concurrency libraries requires localized changes. If the library's semantics can be implemented in terms of memory accesses, one has to construct an appropriate header file or extend the interpreter to provide the mapping from library calls to the relevant memory access events. If this is not possible and/or if native support for a library is desirable (e.g., for performance reasons), then the execution graph has to be extended with new kinds of events and the consistency checks have to be adapted accordingly.

Next, we present two such library extensions, one mapping its calls to individual memory accesses, and the other creating new kinds of events.

*System Calls.* As part of [26], we extended GenMC with support for system calls, such as open(), close(), read() and write(), which can be modeled by making multiple primitive calls (reads and writes) to a different address space.

There are two ways one could implement these system calls: either by providing an actual implementation (which would then be compiled to LLVM-IR) or by adding support in the interpreter to internally implement those calls and communicating multiple times with the driver.

We preferred the latter solution because it is more portable. An external implementation would have to be manually ported whenever support for more languages is added. In contrast, the internal implementation needs no change. Further, even if a new interpreter for a different runtime system is added, it should be simple to decouple the system calls from the interpreter, and have the different runtime systems share the infrastructure that handles system calls.

*Barriers. N*-way barriers are a widely-used synchronization primitive. They have two functions: barrier init and barrier wait. The former initializes a barrier object with the number of threads that will rendezvous at the barrier, while the latter is called every time a thread reaches the barrier. A thread that is calling barrier wait blocks until the initially specified number of threads reaches the barrier, at which point all threads will be simultaneously unblocked, and the barrier value will reset to the one specified with barrier init.

Barriers can be straightforwardly implemented with a shared variable counting the number of threads that have called barrier wait. But doing so yields poor model checking performance. For *N* threads calling barrier wait, there are *N*! possible orders in which they can update the shared counter, thus crippling the performance of the tool. Tracking the order of these updates is not only expensive but also completely unnecessary. For many real-world use cases of barriers (e.g., scatter-gather workloads), the order in which different threads reached the barrier is irrelevant, and the thread that reached last unimportant.

We leverage this intuition and provide built-in support for barrier init and barrier wait calls that does not track the relative ordering among barrier wait calls synchronizing with one another, thereby achieving an exponential reduction in verification time. Concretely, in the simple program below where *N* threads execute barrier wait concurrently, GenMC explores only one execution instead of *N*! executions:

```
barrier wait(); ... barrier wait();
```
Our extension is called BAM (Barrier-Aware Model-checking) and is detailed and evaluated in a companion paper [30].

#### **6 Error Detection and Reporting**

GenMC detects a number of different kinds of errors: violations of user-supplied regular and persistency assertions, data races, memory errors and simple cases of termination errors. It reports errors by printing an offending execution graph and highlighting the event(s) that caused the violation. Upon request, GenMC can also print a total ordering of the instructions that lead to the violation, or produce the offending execution in the DOT graph description language.

*Persistency Errors.* To verify persistency properties of programs performing file I/O, we allow user programs to contain a special *recovery routine* [26], which would typically check some invariant over the persisted state.

When such a routine is present, GenMC simulates all possible ways in which the program could have crashed because of a power failure, executing the recovery routine at the end of every such execution. Of course, to avoid the obvious state-space explosion, the simulation of all the possible failures is done in an optimized fashion, driven by the memory accesses of the recovery routine.

The performance of GenMC when verifying persistency properties of programs under the ext4 filesystem has been evaluated at [26].

*Memory Errors.* Memory errors refers to accessing uninitialized, unallocated or deallocated memory. In models like RC11 [32], reasoning about memory safety can be tricky at times, as demonstrated by the example below:

*p* := **alloc**(); ∗*p* :=rlx 42; *x* :=rlx 1; **if** *x* = 1 **then** *a* :=rlx *p*; *b* :=rlx ∗*a*;

This example is erroneous under RC11 because the allocation of *p* is not guaranteed to have propagated to the second thread by the time it is dereferenced. (Since all accesses are relaxed, there is no synchronization between the threads.)

GenMC also accounts for more complicated scenarios such as *p* being concurrently freed when accessed, *p* being freed twice, or *p* being the address of a local (stack) variable that might not be alive when accessed.

*Refining Error Reports.* It is often useful to refine the error reporting. For example, in memory models that treat data races as errors (such as RC11), GenMC by default detects data races and reports them as errors. This, however, can be costly in terms of verification time or even prohibit the verification of programs that use compiler/custom primitives to access shared memory, as such programs would almost certainly be considered racy.

To deal with such cases, GenMC provides switches that disable race detection and refine the range of errors that will be reported to the user. Switches of the latter kind are especially useful when dealing with programs that contain system calls. By default, when such system calls fail, GenMC reports an error, which is inconvenient for programs that contain proper error handling, as some system errors are rather benign (e.g., a file not existing). With the appropriate switch, in case of system errors, an appropriate value is written in errno, as dictated by the POSIX standard.

*Case Study.* We demonstrate the error reporting capabilities of GenMC with a real use case. We consider a flat-combining queue [19] that has been proposed to be ported in Rust's crossbeam library.

This queue serves as a nice case study for a couple of reasons. First, it contains loops that can diverge, and so its verification requires loop bounding, which GenMC can do automatically. Second, it is implemented using compiler primitives for concurrent accesses, and so its verification requires disabling race detection. Third, while experimenting with it, we found it to be buggy.

The error report produced by GenMC can be seen in Fig. 2. The error is quite intricate: it requires three threads to manifest, each of which executes a large number of instructions. The error is due to an ordering bug (relaxed accesses are used instead of release/acquire), which demonstrates the need for model checking tools that handle weak memory models.

```
Error detected: Attempt to read from uninitialized memory!
Event (3, 63) in graph:
<-1, 0> main:
<0, 1> thread_n:
  (1, 18): Urel (cmb.queue, 0) [(0, 36)] L.169: combiner.c
  (1, 19): Urel (cmb.queue, 2565579352) L.169: combiner.c
  (1, 96): Racq (m.msg._meta.next, 2565579416) [(2, 26)] L.228: combiner.c
  (1, 112): Wrlx (cmb.takeover, 2565579416) L.158: combiner.c
<0, 2> thread_n:
  (2, 26): Wrel (m.msg._meta.next, 94798317999592) L.167: combiner.c
<0, 3> thread_n:
  (3, 18): Urel (cmb.queue, 2565579352) [(1, 19)] L.169: combiner.c
  (3, 19): Urel (cmb.queue, 2565579480) L.169: combiner.c
  (3, 50): Rrlx (cmb.takeover, 2565579416) [(1, 112)] L.87: combiner.c
  (3, 63): Racq (m.msg._meta.next, 0) [BOTTOM] L.187: combiner.c
Number of complete executions explored: 2795
Number of blocked executions seen: 6001
Total wall-clock time: 2.12s
```
We note that the error report contains helpful debugging information, such as the names of variables accessed (e.g., m.msg. meta.next) and the values read/written. To display this information, GenMC maintains a mapping from addresses to program variables using the additional debugging information collected in the "Transformation" phase.

#### **7 Other Performance Enhancements to GenMC**

In this section, we briefly discuss two recent changes to the driver to optimize its performance for certain kinds of programs.

*Symmetry Reduction.* Many programs, such as the flat-combining queue of Sect. 6, have a symmetric structure: each thread runs the same code. In such cases, many execution graphs are equivalent up to some thread relabeling—a property that is exploited by *symmetry reduction* (SR) [16,20].

We implemented a simple SR algorithm that detects whether multiple threads with the same code are spawned with no intervening memory accesses, and avoids exploring executions for which a symmetric one (by relabeling such threads) has already been explored. This can yield exponential improvements. For example, a program with *N* threads incrementing a shared variable atomically has *N*! executions; employing SR yields only one execution. With SR, the verification time of the corrected flat-combining queue drops from 15 s to 2.5 s.

To further demonstrate the benefits of SR, we measured the performance of GenMC with and without SR on some realistic lock implementations adapted from the literature. The results can be seen in Table 1. All reported times are in seconds, unless mentioned otherwise. We ran both GenMC versions three times for each benchmark, with an increasing number of threads each time (the initial thread number for each benchmark is provided in the second column). As it can be seen, SR leads to a significant performance improvement in all cases.


**Table 1.** Testing lock implementations (1 h timeout; 4 GB memory limit)

*Lock-Aware Partial Order Reduction.* A common problem with locking is that of false sharing, where *N* threads contend to acquire the same lock even if it is unnecessary for correctness. In such cases, GenMC's partial order reduction algorithm [29] will explore all *N*! orders in which the lock can be acquired even though they all lead to the same outcome.

We have implemented *lock-aware partial order reduction* (LAPOR) [28], an enhancement to partial order reduction that does not track ordering among locks unless their critical regions have conflicting accesses, in which case the lock ordering is induced from the ordering among those accesses. With LAPOR, GenMC achieves exponential improvements in lock-based implementations of concurrent libraries that have false sharing, such as search trees with coarsegrained or hand-over-hand locking. LAPOR has been evaluated at [28].

#### **8 Conclusion**

We presented GenMC, a state-of-the-art stateless model checker that can be used to verify consistency and persistency properties of C/C++ programs. We described its architecture, and how its modular design can be leveraged to account for new features and memory models. To widen the applicability of GenMC, we have extended it with support for LKMM, basic system calls and additional synchronization primitives. We have also improved its performance with optimizations, such as symmetry reduction and lock-aware partial order reduction that can exponentially decrease its search space.

In the future, we plan to implement a DSL for memory models, so as to make it easier to extend GenMC with new models and quickly tweak their approximation strategies. We are also planning to incorporate further optimizations into the tool to enable more effective verification of lock-free algorithms.

**Acknowledgements.** We thank the anonymous reviewers for their feedback. This work was supported by a European Research Council (ERC) Consolidator Grant for the project "PERSIST" under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 101003349).

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

**Hybrid and Cyber-Physical Systems**

## **Synthesizing Invariant Barrier Certificates via Difference-of-Convex Programming**

Qiuye Wang1,2(B) , Mingshuai Chen3(B) , Bai Xue1,2(B) , Naijun Zhan1,2(B) , and Joost-Pieter Katoen3(B)

<sup>1</sup> SKLCS, Institute of Software, CAS, Beijing, China <sup>2</sup> University of Chinese Academy of Sciences, Beijing, China {wangqye,xuebai,znj}@ios.ac.cn <sup>3</sup> RWTH Aachen University, Aachen, Germany {chenms,katoen}@cs.rwth-aachen.de

**Abstract.** A barrier certificate often serves as an inductive invariant that isolates an unsafe region from the reachable set of states, and hence is widely used in proving safety of hybrid systems possibly over the infinite time horizon. We present a novel condition on barrier certificates, termed the *invariant barrier-certificate condition*, that witnesses unbounded-time safety of differential dynamical systems. The proposed condition is by far the least conservative one on barrier certificates, and can be shown as the weakest possible one to attain inductive invariance. We show that discharging the invariant barrier-certificate condition thereby synthesizing invariant barrier certificates—can be encoded as solving an *optimization problem subject to bilinear matrix inequalities* (BMIs). We further propose a synthesis algorithm based on differenceof-convex programming, which approaches a local optimum of the BMI problem via solving *a series of convex optimization problems*. This algorithm is incorporated in a branch-and-bound framework that searches for the global optimum in a divide-and-conquer fashion. We present a weak completeness result of our method, in the sense that a barrier certificate is guaranteed to be found (under some mild assumptions) whenever there exists an inductive invariant (in the form of a given template) that suffices to certify safety of the system. Experimental results on benchmark examples demonstrate the effectiveness and efficiency of our approach.

#### **1 Introduction**

Hybrid systems are mathematical models that capture the interaction between continuous physical dynamics and discrete switching behaviors, and hence are widely used in modelling cyber-physical systems (CPS). These CPS may be

This work has been partially funded by the NSFC under grant No. 61625206, 61732001, 61872341, and 61836005, by the ERC Advanced Project FRAPPANT under grant No. 787914, and by the CAS Pioneer Hundred Talents Program.

complex and safety-critical, with sensitive variables of the environment in its sphere of control. Everyday examples include process control at all scales, ranging from household appliances to nuclear power plants, or embedded systems in transportation domain, such as autonomous driving maneuvers in automotive, aircraft collision-avoidance protocols in avionics, or automatic train control applications, as well as a broad range of devices in health technologies, such as cardiac pacemakers.

The safety-critical feature of these CPS, with increasingly complex behaviors, has initiated automatic safety or, dually, reachability verification of hybrid systems [1,15]. The problem of reachability verification is undecidable in general [1], albeit with decidable families of sub-classes (see, e.g., [2,16–18,31]) identified in the literature. The hard core of the verification problem lies in reasoning about the continuous dynamics, which are often characterized by ordinary differential equations (ODEs). In particular, when nonlinearity arises in the ODEs, the explicit computation of the exact reachable set is usually intractable even for purely continuous dynamics [49].

Therefore in the literature, a plethora of approximation schemes, as surveyed in [15], for reachability analysis of hybrid systems has been developed, including an invariant-style reasoning scheme known as *barrier certificate* [41]. A barrier certificate often serves as an inductive invariant that isolates an unsafe region from the reachable set, thereby witnessing safety of hybrid systems possibly over the infinite time horizon. A common way to synthesize barrier certificates is to reduce the condition defining barrier certificates to a numerical optimization or constraint solving problem. There is, however, a trade-off between the expressiveness of the barrier-certificate condition and the efficiency in discharging the reduced constraints. Hence, to enable efficient algorithmic synthesis of barrier certificates via, e.g., linear programming (LP), second-order cone programming (SOCP), semidefinite programming (SDP) and interval analysis [11,30], the general condition on inductive invariance (that a barrier certificate defines an invariant, see [8,51]) has been strengthened into a spectrum of different shapes, e.g., [8,29,51,60,62]. It has been, nevertheless, a long-standing challenge *to find a barrier-certificate condition that is as weak as possible while admitting efficient synthesis algorithms*.

In this paper, we present a new condition on barrier certificates, termed the *invariant barrier-certificate condition*, based on the sufficient and necessary condition on being an inductive invariant [36]. Our invariant barrier-certificate condition is by far, to the best of our knowledge, the least conservative one on barrier certificates, and can be shown as the weakest possible one to attain inductive invariance. We show, by leveraging Putinar's Positivstellensatz [32], that discharging the invariant barrier-certificate condition —thereby synthesizing invariant barrier certificates— can be encoded as solving an optimization problem subject to *bilinear matrix inequalities* (BMIs). We further show that general bilinear matrix-valued functions can be decomposed as a difference of two psd-convex (extension of convexity to matrix-valued functions) functions using eigendecomposition, thus resulting in a synthesis algorithm as per *difference-ofconvex programming* (DCP) [33,52], which solves a series of convex sub-problems (in the form of *linear matrix inequalities* (LMIs)) that approaches (arbitrarily close to) a local optimum of the BMI problem. This algorithm is incorporated in a branch-and-bound framework that searches for the global optimum in a divideand-conquer fashion. We present a weak completeness result of our method, in the sense that a barrier certificate is guaranteed to be found (under some mild assumptions) whenever there exists an inductive invariant (in the form of a given template) that suffices to certify the system's safety. A similar result on completeness is previously provided only by symbolic approaches, yet to the best of our knowledge, not by methods base on numerical constraint solving, e.g., [4,60,61]. Experiments on a collection of examples suggested that our invariant barriercertificate condition recognizes more barrier certificates than existing conditions, and that our DCP-based algorithm is more efficient than directly solving the BMIs via off-the-shelf solvers.

Due to space restrictions, proofs and benchmark details have been omitted; they are found in an extended version of this paper [57].

#### **2 A Bird's-Eye Perspective**

We use the following example to give a bird's-eye view of our approach.

*Example 1 (* overview [11]*).* Consider the following continuous-time dynamical system modelled by an ordinary differential equation:

$$
\dot{\mathbf{x}} = \begin{pmatrix} \dot{x}\_1 \\ \dot{x}\_2 \end{pmatrix} = \begin{pmatrix} x\_1 + x\_2 \\ x\_1 x\_2 - 0.5 x\_2^2 + 0.1 \end{pmatrix}.
$$

The verification obligation is to show that the system trajectory originating from any state in the initial set <sup>X</sup><sup>0</sup> <sup>=</sup> {**<sup>x</sup>** | I(**x**) <sup>≤</sup> <sup>0</sup>} with <sup>I</sup>(**x**) = <sup>x</sup><sup>2</sup> <sup>1</sup> + (x<sup>2</sup> <sup>−</sup> 2)<sup>2</sup> <sup>−</sup> <sup>1</sup> will never enter the unsafe set X<sup>u</sup> = {**x** | U(**x**) ≤ 0} with U(**x**) = x<sup>2</sup> + 1. -

A barrier certificate satisfying our condition in Definition 4 serves as an inductive invariant that suffices to isolate the unsafe region X<sup>u</sup> from the set of reachable states from X0, thereby proving safety of the system over the infinite time horizon. To this end, we proceed in the following steps.

**1) Encode as Sum-of-Squares (SOS) Constraints.** We set a (polynomial) barrier-certificate template <sup>B</sup>(**a**, **<sup>x</sup>**) = ax<sup>2</sup> with unknown coefficient <sup>a</sup> <sup>∈</sup> <sup>R</sup>. According to Theorem 1, we only need to consider Lie derivatives up to order <sup>N</sup>B,*<sup>f</sup>* = 1, i.e., <sup>L</sup><sup>0</sup> *<sup>f</sup>* <sup>B</sup>(**a**, **<sup>x</sup>**) = ax<sup>2</sup> and <sup>L</sup><sup>1</sup> *<sup>f</sup>* <sup>B</sup>(**a**, **<sup>x</sup>**) = <sup>a</sup>(x1x<sup>2</sup> <sup>−</sup> <sup>0</sup>.5x<sup>2</sup> <sup>2</sup> + 0.1).

By Theorem 5, B(**a**, **x**) is an invariant barrier certificate if there exists a polynomial v(**x**), SOS polynomials σ(**x**), σ (**x**) and a constant > 0 such that

$$-\underbrace{ax\_{B}+\sigma(\mathbf{x})\underbrace{\left(x\_{1}^{2}+(x\_{2}-2)^{2}-1\right)}\_{\mathcal{I}}}\_{\mathcal{L}\_{f}^{1}B},\tag{1.1.\text{initial}}$$

$$\underbrace{-a\underbrace{\left(x\_{1}x\_{2}-0.5x\_{2}^{2}+0.1\right)}\_{\mathcal{L}\_{f}^{1}B}+v(\mathbf{x})\underbrace{ax\_{2}}\_{\mathcal{L}\_{f}^{0}B},\tag{1.2.\text{Lioconsecution}}\tag{1.2.\text{Lioconscection}}}$$

are SOS polynomials. We set = 0.01 in this example.

**2) Reduce to a BMI Optimization Problem.** Observe that the above SOS constraints can be formulated as BMI constraints. For instance, let us assume that (1.2) is an SOS polynomial of degree at most 2 and v(**s**, **x**) = s0+s1x1+s2x<sup>2</sup> is a template polynomial with unknown coefficients **s**. Then constraint (1.2) is equivalent to the BMI constraint

$$\mathcal{F}\_2(\mathbf{a}, \mathbf{s}) = -\begin{pmatrix} -0.1a & 0 & 0.5as\_0 \\ 0 & 0 & 0.5(as\_1 - a) \\ 0.5as\_0 \ 0.5(as\_1 - a) & as\_2 + 0.5a \end{pmatrix} \preceq 0$$

meaning that the bilinear matrix (LHS of ) is negative semidefinite. Note that the bilinearity arises due to the coupling of the unknown coefficients **a** and **s**.

Constraints (1.1) and (1.3) can be reduced to BMI constraints in an analogous way<sup>1</sup>, yielding <sup>F</sup><sup>1</sup> and <sup>F</sup>3. It then follows that, to solve the SOS constraints, we need to find a feasible solution (**a**, **s**) such that<sup>2</sup>

$$\mathcal{F}\_1(\mathbf{a}, \mathbf{s}) \preceq 0 \land \mathcal{F}\_2(\mathbf{a}, \mathbf{s}) \preceq 0 \land \mathcal{F}\_3(\mathbf{a}, \mathbf{s}) \preceq 0. \tag{2}$$

To exploit well-developed optimization techniques, the feasibility problem (2) is transformed to an optimization problem subject to BMI constraints:

$$\begin{array}{ll}\underset{\lambda,\mathbf{a},\mathbf{s}}{\text{maximize}} & \lambda\\\text{subject to} & \mathcal{B}\_{i}(\lambda,\mathbf{a},\mathbf{s}) \triangleq \mathcal{F}\_{i}(\mathbf{a},\mathbf{s}) + \lambda I \preceq 0, \quad i = 1,2,3\end{array} \tag{3}$$

where I is the identity matrix with compatible dimensions. Note that problem (2) has a feasible solution if and only if the optimal value λ<sup>∗</sup> in (3) is non-negative.

**3) Decompose as Difference-of-Convex Problems.** The problem (3) contains non-convex constraints and hence does not admit efficient (polynomialtime) algorithms tailored for convex optimizations. However, by our technique presented in Sect. 5, a non-convex function B<sup>i</sup>(λ, **a**, **s**) can be decomposed as the difference of two psd-convex (defined later) matrix-valued functions:

$$\mathcal{B}\_i(\lambda, \mathbf{a}, \mathbf{s}) = \mathcal{B}\_i^+(\lambda, \mathbf{a}, \mathbf{s}) - \mathcal{B}\_i^-(\lambda, \mathbf{a}, \mathbf{s}). \tag{4}$$

The decomposition of B2(λ, **a**, **s**), for instance, gives

```
B+
 2 (λ, a, s) =
    1
    8
     ⎛
     ⎜⎝
       8λ + 0.08a + a2 + 0.408s2
                            0 0.408s0s1 −2as0 + 0.816s0s2
              0.408s0s1 8λ + a2 + 0.408s2
                                             1 4a − 2as1 + 0.816s1s2
          −2as0 + 0.816s0s2 4a − 2as1 + 0.816s1s2 8λ − 4a + 2.449a2 − 4as2 + s2
                                                                          0 + s2
                                                                              1 + 1.632s2
                                                                                       2
                                                                                        ⎞
                                                                                        ⎟⎠
B−
 2 (λ, a, s) =
    1
    8
     ⎛
     ⎜⎝
         a2 + 0.408s2
                  0 0.408s0s1 2as0 + 0.816s0s2
          0.408s0s1 a2 + 0.408s2
                                  1 2as1 + 0.816s1s2
       2as0 + 0.816s0s2 2as1 + 0.816s1s2 2.449a2 + 4as2 + s2
                                                     0 + s2
                                                         1 + 1.632s2
                                                                  2
                                                                   ⎞
                                                                   ⎟⎠ .
```
<sup>1</sup> Despite that no bilinearity is involved in constraints (1.1) and (1.3), they can be processed in the same way as (1.2), yielding LMI constraints.

<sup>2</sup> Extra constraints on σ(**x**) and σ- (**x**) being SOS polynomials can be encoded analogously in the feasibility problem, yet are omitted here for the sake of simplicity.

**4) Solve a Series of Convex Sub-problems.** Now, we apply a standard iterative procedure in difference-of-convex programming [10] as follows. Given a feasible solution **z**<sup>k</sup> = (λk, **a**k, **s**k) to the BMI optimization problem (3), the concave part −B<sup>−</sup> <sup>i</sup> (λ, **a**, **s**) in (4) is linearized around **z**k, thus yielding a series of convex programs (k = 0, 1,...):

$$\begin{array}{ll}\underset{\lambda,\mathbf{a},\mathbf{s}}{\text{maximize}} & \lambda\\\text{subject to} & \mathcal{B}^+\_i(\mathbf{z}) - \mathcal{B}^-\_i\left(\mathbf{z}^k\right) - \mathcal{D}\mathcal{B}^-\_i\left(\mathbf{z}^k\right)\left(\mathbf{z} - \mathbf{z}^k\right) \preceq 0, \quad i = 1,2,3\end{array} \tag{5}$$

where DB<sup>−</sup> <sup>i</sup> denotes the derivative of the matrix-valued function B<sup>−</sup> i .

The soundness of our approach asserts that the feasible set of the linearized program (5) under-approximates the feasible set of the original BMI program (3). Therefore, if <sup>λ</sup><sup>k</sup> <sup>≥</sup> 0 after iteration <sup>k</sup>, we can safely claim that (**a**<sup>k</sup>, **<sup>s</sup>**<sup>k</sup>) is a feasible solution to (2). A barrier certificate B(**x**) is then obtained by substituting **a**<sup>k</sup> in B(**a**, **x**). Moreover, if we take the optimum **z**∗,k of (5) to be the next linearization point **<sup>z</sup>**<sup>k</sup>+1, the solution sequence {**z**<sup>k</sup>}<sup>k</sup>∈<sup>N</sup> converges to a local optimum of (3).

We show that the linearized program (5) is equivalent to an LMI optimization problem admitting polynomialtime algorithms, say the well-known *interior-point methods* supported by most off-the-shelf SDP solvers. Our iterative procedure starts with a strictly feasible initial solution **z**<sup>0</sup> to program (3) and terminates with <sup>λ</sup><sup>2</sup> <sup>≥</sup> <sup>0</sup> (subject to numerical round-off) and <sup>a</sup><sup>2</sup> <sup>=</sup> <sup>−</sup>0.00363421, yielding the barrier certificate

$$B(\mathbf{a}^2, \mathbf{x}) = -0.00363421x\_2 \le 0.1$$

Figure 1 depicts the system dynamics and the synthesized barrier certificate.

We remark that the aforementioned

**Fig. 1.** Phase portrait of the system in Example 1. The arrows indicate the vector field and the solid curves are randomly sampled trajectories.

iterative procedure on solving a series of convex optimizations converges only to a local optimum of the BMI problem (3). This means that, in some cases, it may miss the global optimum that induces a non-negative λ∗. We will present in Sect. 6 a solution to this problem by incorporating our iterative procedure into a branch-and-bound framework that searches for the global optimum in a divide-and-conquer fashion.

#### **3 Mathematical Foundations**

**Notations.** Let N, N<sup>+</sup>, R, R<sup>+</sup> and R<sup>+</sup> <sup>0</sup> be respectively the set of natural, positive natural, real, positive real and non-negative real numbers. For a vector **<sup>x</sup>** <sup>∈</sup> <sup>R</sup><sup>n</sup>, <sup>x</sup><sup>i</sup> refers to its <sup>i</sup>-th component and **x** denotes the <sup>2</sup>-norm; for a matrix <sup>A</sup> <sup>∈</sup> Rn×m, A(i, j) refers to its (i, j)-th element. Let R[**x**] be the polynomial ring in **<sup>x</sup>** over the field <sup>R</sup>. A polynomial <sup>h</sup> <sup>∈</sup> <sup>R</sup>[**x**] is *sum-of-squares* (SOS) iff there exist polynomials <sup>g</sup>1,...,g<sup>k</sup> <sup>∈</sup> <sup>R</sup>[**x**] such that <sup>h</sup> <sup>=</sup> <sup>k</sup> <sup>i</sup>=1 g<sup>2</sup> <sup>i</sup> . We denote by <sup>Σ</sup>[**x**] <sup>⊂</sup> <sup>R</sup>[**x**] the set of SOS polynomials over **<sup>x</sup>**x. <sup>S</sup><sup>n</sup> denotes the space of <sup>n</sup>×<sup>n</sup> real, symmetric matrices. For <sup>A</sup> ∈ Sn, <sup>A</sup> 0 means that <sup>A</sup> is *positive semidefinite* (psd, for short)<sup>3</sup>, i.e., <sup>∀</sup>**<sup>x</sup>** <sup>∈</sup> <sup>R</sup><sup>n</sup> : **<sup>x</sup>**TA**<sup>x</sup>** <sup>≥</sup> 0. A matrix-valued function <sup>B</sup>: <sup>R</sup><sup>n</sup> → S<sup>m</sup> is *psdconvex* on a convex set C ⊆ <sup>R</sup><sup>n</sup> if <sup>∀</sup>**x**1, **<sup>x</sup>**<sup>2</sup> ∈ C. <sup>∀</sup><sup>μ</sup> <sup>∈</sup> (0, 1): <sup>B</sup>(μ**x**<sup>1</sup> + (1 <sup>−</sup> <sup>μ</sup>)**x**2) μB(**x**1) + (1 − μ)B(**x**2).

**Differential Dynamical Systems.** We consider a class of continuous dynamical systems modelled by ordinary differential equations of the autonomous type:

$$
\dot{\mathbf{x}} = f(\mathbf{x}) \tag{6}
$$

where **<sup>x</sup>** <sup>∈</sup> <sup>R</sup><sup>n</sup> is the *state* vector, **<sup>x</sup>**˙ denotes its temporal derivative d**x**/dt, with <sup>t</sup> <sup>∈</sup> <sup>R</sup><sup>+</sup> <sup>0</sup> modelling time, and *<sup>f</sup>* : <sup>R</sup><sup>n</sup> <sup>→</sup> <sup>R</sup><sup>n</sup> is a polynomial *flow field* (or *vector field*) that governs the evolution of the system. A polynomial vector field is local Lipschitz, and hence for some <sup>T</sup> <sup>∈</sup> <sup>R</sup><sup>+</sup> ∪ {∞}, there exists a unique *solution* (or *trajectory*) *<sup>ζ</sup>***<sup>x</sup>**<sup>0</sup> : [0, T) <sup>→</sup> <sup>R</sup><sup>n</sup> originating from any initial state **<sup>x</sup>**<sup>0</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup> such that (1) *<sup>ζ</sup>***<sup>x</sup>**<sup>0</sup> (0) = **<sup>x</sup>**0, and (2) <sup>∀</sup><sup>τ</sup> <sup>∈</sup> [0, T): <sup>d</sup>*ζ***x**<sup>0</sup> dt <sup>t</sup>=<sup>τ</sup> = *f*(*ζ***<sup>x</sup>**<sup>0</sup> (τ )). We assume in the sequel that T is the maximal instant up to which *ζ***<sup>x</sup>**<sup>0</sup> exists for all **x**0.

*Remark 1.* Our techniques on synthesizing barrier certificates in this paper focus on differential dynamics of the form (6). However, we foresee no substantial difficulties in extending the results to multi-mode hybrid systems where extra constraints on the system evolution, e.g., guards, are present.

**Safety Verification Problem.** Given a domain set X ⊆ <sup>R</sup><sup>n</sup>, an initial set X<sup>0</sup> ⊆ X and an unsafe set X<sup>u</sup> ⊆ X , the *reachable set* of a dynamical system of the form (6) at time instant <sup>t</sup> <sup>∈</sup> [0, T) is defined as <sup>R</sup><sup>X</sup><sup>0</sup> (t) <sup>=</sup> {*ζ***<sup>x</sup>**<sup>0</sup> (t) <sup>|</sup> **<sup>x</sup>**<sup>0</sup> ∈ X0}. We denote by R<sup>X</sup><sup>0</sup> the aggregated reachable set, i.e., the union of R<sup>X</sup><sup>0</sup> (t) over <sup>t</sup> <sup>∈</sup> [0, T)<sup>4</sup>. The system is said to be *safe* iff <sup>R</sup><sup>X</sup><sup>0</sup> ∩ X<sup>u</sup> <sup>=</sup> <sup>∅</sup>, and *unsafe* otherwise. For simplicity, we consider <sup>X</sup> <sup>=</sup> <sup>R</sup><sup>n</sup> throughout this paper.

To avoid the explicit computation of the exact reachable set, which is usually intractable for nonlinear hybrid systems (cf., e.g., [15]), barrier-certificate methods make use of a partial differential operator, termed the *Lie derivative*, to capture the evolution of a barrier function along the vector field:

**Definition 1 (Lie Derivative** [28]**).** *Given a vector field <sup>f</sup>* : <sup>R</sup><sup>n</sup> <sup>→</sup> <sup>R</sup><sup>n</sup> *over* **<sup>x</sup>***, the* Lie derivative *of a polynomial function* <sup>B</sup>(**x**) *along <sup>f</sup>,* <sup>L</sup><sup>k</sup> *<sup>f</sup>* <sup>B</sup> : <sup>R</sup><sup>n</sup> <sup>→</sup> <sup>R</sup> *of order* <sup>k</sup> <sup>∈</sup> <sup>N</sup>*, is*

$$\mathcal{L}\_f^k B(\mathbf{x}) \stackrel{\frown}{=} \begin{cases} B(\mathbf{x}), & k = 0, \\ \left\langle \frac{\partial}{\partial \mathbf{x}} \mathcal{L}\_f^{k-1} B(\mathbf{x}), f(\mathbf{x}) \right\rangle, & k > 0, \end{cases}$$

<sup>3</sup> More generally, for A, B ∈ S<sup>n</sup>, <sup>A</sup> <sup>B</sup> indicates that <sup>B</sup> <sup>−</sup> <sup>A</sup> is positive semidefinite. <sup>4</sup> This subsumes the problem of unbounded-time safety verification where a unique solution exists over the infinite time horizon [0, ∞).

*where* ·, · *is the inner product of vectors, i.e.,* **u**, **<sup>v</sup>** <sup>=</sup> <sup>n</sup> <sup>i</sup>=1 <sup>u</sup>iv<sup>i</sup> *for* **<sup>u</sup>**, **<sup>v</sup>** <sup>∈</sup> <sup>R</sup>n*.*

The Lie derivative <sup>L</sup><sup>k</sup> *<sup>f</sup>* B(**x**) is essentially the k-th temporal derivative of the (barrier) function B(**x**), and thus captures the change of B(**x**) over time.

An *inductive invariant* <sup>Ψ</sup> <sup>⊆</sup> <sup>R</sup><sup>n</sup> of a dynamical system is a set of states such that all the trajectories starting from within Ψ remain in Ψ:

**Definition 2 (Inductive Iinvariant** [40]**).** *Given a system* (6)*, a set* <sup>Ψ</sup> <sup>⊆</sup> <sup>R</sup><sup>n</sup> *is an* inductive invariant *of system* (6) *if and only if*

$$\forall \mathbf{x}\_0 \in \Psi. \forall t \in [0, T) \colon \zeta\_{\mathbf{x}\_0}(t) \in \Psi. \tag{7}$$

In the sequel, we refer to inductive invariants simply as invariants. In [36], a sufficient and necessary condition on being a polynomial invariant is proposed:

**Theorem 1 (Invariant condition** [36]**).** *Given a polynomial* <sup>B</sup> <sup>∈</sup> <sup>R</sup>[**x**]*, its* zero sub-level set {**<sup>x</sup>** <sup>|</sup> <sup>B</sup>(**x**) <sup>≤</sup> <sup>0</sup>} *is an invariant of system* (6) *if and only if* <sup>5</sup>

$$B \le 0 \implies \bigvee\_{i=0}^{N\_{B,f}} \left( \left( \bigwedge\_{j=0}^{i-1} \mathcal{L}\_f^j B = 0 \right) \wedge \mathcal{L}\_f^i B < 0 \right) \vee \bigwedge\_{i=0}^{N\_{B,f}} \mathcal{L}\_f^i B = 0 \tag{8}$$

*where* <sup>N</sup>B,*<sup>f</sup>* <sup>∈</sup> <sup>N</sup><sup>+</sup> *is a completeness threshold, i.e., a finite positive integer that bounds the order of Lie derivatives, which can be computed using Gr¨obner bases*6*.*

In contrast, a *barrier certificate* is a function whose zero sub-level set isolates an unsafe region X<sup>u</sup> from the reachable set R<sup>X</sup><sup>0</sup> w.r.t. some initial set X0:

**Definition 3 (Semantic Barrier Certificate** [51]**).** *Given a system* (6)*, an initial set* X<sup>0</sup> *and an unsafe set* X<sup>u</sup>*, a* barrier certificate *of* (6) *is a differentiable function* <sup>B</sup> : <sup>R</sup><sup>n</sup> <sup>→</sup> <sup>R</sup> *satisfying*

$$\forall \mathbf{x}\_0 \in \mathcal{X}\_0. \forall t \in [0, T) \colon B\left(\xi\_{\mathbf{x}\_0}(t)\right) \le 0 \quad \text{and} \quad \forall \mathbf{x} \in \mathcal{X}\_0 \colon \mathbf{B}(\mathbf{x}) > 0. \tag{9}$$

The existence of such a barrier certificate trivially implies safety of the system. Moreover, one may readily verify that if some set Ψ = {**x** | B(**x**) ≤ 0} is an invariant and satisfies (X<sup>0</sup> ⊆ Ψ)∧(Ψ ∩X<sup>u</sup> = ∅), then B(**x**) is a barrier certificate.

As observed in [51], however, the semantic statement in Definition 3 encodes merely the general *principle of barrier certificates* [8], yet in itself is not that useful for safety verification because it explicitly involves the system solutions. Therefore, in order to enable efficient synthesis, the semantic condition on barrier certificates has been strengthened into a handful of different shapes (see, e.g., [8, 29,41,60], which all imply inductive invariance). It has been yet a long-standing challenge *to find a barrier-certificate condition that is as weak as possible while admitting efficient synthesis algorithms*.

Our BMI encoding of the invariant barrier-certificate condition (cf. Sect. 4) roots in Putinar's Positivstellensatz, which characterizes positivity of polynomials on a semi-algebraic set defined by a system of polynomial inequalities:

<sup>5</sup> In (8), i−1 <sup>j</sup>=0 <sup>L</sup><sup>j</sup>

*<sup>f</sup>* <sup>B</sup> = 0 is true for <sup>i</sup> = 0 by default. This applies in the sequel. <sup>6</sup> <sup>N</sup>B,*<sup>f</sup>* is the minimal <sup>i</sup> such that <sup>L</sup><sup>i</sup>+1 *<sup>f</sup>* B is in the polynomial ideal generated by L0 *<sup>f</sup>* B, <sup>L</sup><sup>1</sup> *<sup>f</sup>* B,..., <sup>L</sup><sup>i</sup> *<sup>f</sup>* B. The ideal membership can be decided via Gr¨obner basis.

**Theorem 2 (Putinar's Positivstellensatz** [32]**).** *Let* K = {**x** | <sup>m</sup> <sup>i</sup>=1 gi(**x**) ≥ <sup>0</sup>} *be a compact semi-algebraic set defined by* <sup>g</sup>1,...,g<sup>m</sup> <sup>∈</sup> <sup>R</sup>[**x**]*. Assume the* Archimedean condition *holds*<sup>7</sup>*, i.e., there exists* <sup>L</sup> <sup>∈</sup> <sup>R</sup><sup>+</sup> *such that* <sup>L</sup> − **x**<sup>2</sup> <sup>=</sup> <sup>σ</sup>0(**x**) + <sup>m</sup> <sup>i</sup>=1 <sup>σ</sup>i(**x**)gi(**x**) *for some* <sup>σ</sup>0,...,σ<sup>m</sup> <sup>∈</sup> <sup>Σ</sup>[**x**]*. If* <sup>h</sup> <sup>∈</sup> <sup>R</sup>[**x**] *is strictly positive on* K*, then*

$$h(\mathbf{x}) = \sigma\_0(\mathbf{x}) + \sum\_{i=1}^{m} \sigma\_i(\mathbf{x}) g\_i(\mathbf{x})$$

*holds for some SOS polynomials* σ0,...,σ<sup>m</sup> ∈ Σ[**x**]*.*

We now recall a key technique used in our reduction to semidefinite optimizations. Given a symmetric matrix <sup>X</sup> ∈ S<sup>n</sup> partitioned as <sup>X</sup> <sup>=</sup> - A C C<sup>T</sup> D with invertible <sup>A</sup>, the *Schur complement* of <sup>A</sup> in <sup>X</sup> is defined as X/A <sup>=</sup> <sup>D</sup>−CTA−1C. An important property of the Schur complement X/A is that it characterizes the positive semidefiniteness of the block matrix X:

**Theorem 3 (Schur Complement** [3]**).** *If* A 0*, then* X 0 *iff* X/A 0*.*

We apply the Schur complement in Sect. 5 to transform nonlinear convex constraints into linear constraints.

#### **4 Invariant Barrier-Certificate Condition as BMIs**

In this section, we present our *invariant barrier-certificate condition* (see Definition 4) based on the necessary and sufficient condition on being an inductive invariant (cf. Theorem 1), and show how to encode it as BMI constraints.

#### **4.1 Invariant Barrier-Certificate Condition**

**Definition 4 (Invariant Barrier Certificate).** *Given a system* (6)*, an initial set* <sup>X</sup><sup>0</sup> *and an unsafe set* <sup>X</sup><sup>u</sup>*, a polynomial function* <sup>B</sup> : <sup>R</sup><sup>n</sup> <sup>→</sup> <sup>R</sup> *is an* invariant barrier certificate *of system* (6) *if and only if*

*1. (initial):* ∀**x** ∈ X<sup>0</sup> : B(**x**) ≤ 0*;*

*2. (consecution):* <sup>∀</sup>**<sup>x</sup>** <sup>∈</sup> <sup>R</sup><sup>n</sup> : <sup>N</sup>B,*<sup>f</sup>* <sup>i</sup>=1 <sup>i</sup>−<sup>1</sup> <sup>j</sup>=0 <sup>L</sup><sup>j</sup> *<sup>f</sup>* <sup>B</sup>(**x**)=0 <sup>=</sup>⇒ L<sup>i</sup> *<sup>f</sup>* B(**x**) ≤ 0 *;*

*3. (separation):* ∀**x** ∈ X<sup>u</sup> : B(**x**) > 0*.*

Notice that the consecution constraint in Definition 4 involves Lie derivatives of orders up to <sup>N</sup>B,*<sup>f</sup>* <sup>∈</sup> <sup>N</sup><sup>+</sup>, as is the case in Theorem 1. Our invariant barriercertificate condition hence generalizes existing conditions on barrier certificates, e.g., [4,60,63], which consider Lie derivatives only up to the first order.

The consecution condition in Definition 4 is in fact equivalent to the invariant condition (8) in Theorem 1 (cf. [57, Lemma 2]), thereby revealing the relation between an inductive invariant and an invariant barrier certificate:

<sup>7</sup> This condition can be met by adding a (redundant) constraint <sup>g</sup>m+1(**x**) = <sup>L</sup><sup>0</sup> <sup>−</sup> **x**<sup>2</sup> <sup>≤</sup> 0, provided that a bound <sup>L</sup><sup>0</sup> <sup>∈</sup> <sup>R</sup><sup>+</sup> is known such that <sup>∀</sup>**<sup>x</sup>** ∈ K: <sup>L</sup>0−**x**<sup>2</sup> <sup>≥</sup> 0.

**Theorem 4 (Inductive Invariance).** *Given a system* (6)*, an initial set* X<sup>0</sup> *and an unsafe set* Xu*. If* B(**x**) *is an invariant barrier certificate, then* Ψ = {**x** | B(**x**) ≤ 0} *is an invariant. Conversely, if* Ψ = {**x** | B(**x**) ≤ 0} *is an invariant satisfying* X<sup>0</sup> ⊆ Ψ *and* Ψ ∩ X<sup>u</sup> = ∅*, then* B(**x**) *is an invariant barrier certificate.*

It follows from Theorem 4 that our invariant barrier-certificate condition is the least conservative one on barrier certificates to attain inductive invariance.

*Remark 2.* We do not employ the invariant condition (8) in Theorem 1 as the constraint on the consecution of Lie derivatives. This is because our consecution condition in Definition 4 is simpler, and in particular, amenable to more straightforward transformations to SOS constraints via Putinar's Positivstellensatz, as shown later in Subsect. 4.2.

*Remark 3.* For a fixed 0 < N < NB,*<sup>f</sup>* , the consecution condition in Definition 4 can be strengthened in the following way while preserving inductive invariance:

$$\begin{aligned} \forall \mathbf{x} \in \mathbb{R}^n &\colon \bigwedge\_{i=1}^{\mathfrak{N}-1} \left( \left( \bigwedge\_{j=0}^{i-1} \mathcal{L}\_f^j B(\mathbf{x}) = 0 \right) \implies \mathcal{L}\_f^i B(\mathbf{x}) \le 0 \right) \land \\ &\qquad \left( \left( \bigwedge\_{j=0}^{\mathfrak{N}-1} \mathcal{L}\_f^j B(\mathbf{x}) = 0 \right) \implies \mathcal{L}\_f^{\mathfrak{N}} B(\mathbf{x}) < 0 \right) \end{aligned}$$

where for the <sup>N</sup>-th Lie derivative, one needs <sup>L</sup><sup>N</sup> *<sup>f</sup>* <sup>B</sup>(**x**) <sup>&</sup>lt; 0 (rather than <sup>L</sup><sup>N</sup> *<sup>f</sup>* B(**x**) ≤ 0). In practice, using such a strengthened consecution condition —with less subconstraints to solve— may yield more efficient synthesis.

#### **4.2 Encoding as BMI Optimizations**

Next, we show how to encode synthesizing an invariant barrier certificate (cf. Definition 4) as an optimization problem subject to BMIs. To this end, we first recast the invariant barrier-certificate condition into a collection of SOS constraints<sup>8</sup>.

**Theorem 5 (Sufficient Condition for Invariant Barrier Certificate).** *Given a system* (6)*, an initial set* X<sup>0</sup> = {**x** | I(**x**) ≤ 0} *and an unsafe set* <sup>X</sup><sup>u</sup> <sup>=</sup> {**<sup>x</sup>** | U(**x**) <sup>≤</sup> <sup>0</sup>}*. A polynomial* <sup>B</sup> <sup>∈</sup> <sup>R</sup>[**x**] *is an invariant barrier certificate of* (6) *if for some* <sup>∈</sup> <sup>R</sup><sup>+</sup>*, there exist* <sup>v</sup>i,j <sup>∈</sup> <sup>R</sup>[**x**] *and SOS polynomials* <sup>σ</sup>(**x**), σ (**x**) *s.t.*

1.  $-B(\mathbf{x}) + \sigma(\mathbf{x})\mathcal{Z}(\mathbf{x})$ ,
2.  $\text{ for all } 1 \le i \le N\_{B,f},$ 
 $-\mathcal{L}\_f^i B(\mathbf{x}) + \sum\_{j=0}^{i-1} v\_{i,j}(\mathbf{x})\mathcal{L}\_f^j B(\mathbf{x}),$ 
3.  $B(\mathbf{x}) + \sigma'(\mathbf{x})\mathcal{U}(\mathbf{x}) - \epsilon$ 

*are SOS polynomials.*

By enforcing the Archimedean condition and applying Putinar's Positivstellensatz, we further derive a necessary condition of invariant barrier certificate:

<sup>8</sup> For simplicity, we assume that <sup>X</sup><sup>0</sup> and <sup>X</sup><sup>u</sup> are both captured by a single polynomial. Our formulations, however, apply also to cases with basic semi-algebraic X<sup>0</sup> or Xu.

**Theorem 6 (Necessary Condition for Invariant Barrier Certificate).** *Given a system* (6)*, an initial set* X<sup>0</sup> = {**x** | I(**x**) ≤ 0} *and an unsafe set* <sup>X</sup><sup>u</sup> <sup>=</sup> {**<sup>x</sup>** | U(**x**) <sup>≤</sup> <sup>0</sup>}*. If* <sup>B</sup> <sup>∈</sup> <sup>R</sup>[**x**] *is an invariant barrier certificate of* (6)*, then for some* <sup>∈</sup> <sup>R</sup><sup>+</sup>*, there exist* <sup>v</sup>i,j <sup>∈</sup> <sup>R</sup>[**x**] *and SOS polynomials* σ(**x**), σ (**x**), ρ(**x**), ρ (**x**), ρ <sup>i</sup> (**x**) *s.t. for any* <sup>L</sup> <sup>∈</sup> <sup>R</sup><sup>+</sup>*,*

$$\begin{cases} 1. \ -B(\mathbf{x}) + \rho(\mathbf{x})(\|\mathbf{x}\|^2 - L) + \sigma(\mathbf{x})\mathcal{Z}(\mathbf{x}) + \epsilon, \\ 2. \text{ for all } 1 \le i \le N\_{Bf}, \ -\mathcal{L}\_f^i B(\mathbf{x}) + \rho\_i^{\prime\prime}(\mathbf{x})(\|\mathbf{x}\|^2 - L) + \sum\_{j=0}^{i-1} v\_{i,j}(\mathbf{x})\mathcal{L}\_f^j B(\mathbf{x}) + \epsilon, \\ 3. \ B(\mathbf{x}) + \rho^\prime(\mathbf{x})(\|\mathbf{x}\|^2 - L) + \sigma^\prime(\mathbf{x})\mathcal{U}(\mathbf{x}) \end{cases}$$

#### *are SOS polynomials.*

Notice that a polynomial B(**x**) satisfying the sufficient condition in Theorem 5 suffices as an invariant barrier certificate that witnesses safety of the system. In contrast, a polynomial B(**x**) satisfying the necessary condition in Theorem 6 may serve as a candidate invariant barrier certificate, and safety of the system can be concluded via a posterior check<sup>9</sup> of B(**x**) per Definition 4.

Next we show how to encode an SOS constraint of the shape "h(**x**) ∈ Σ[**x**]" in Theorems 5 and 6 as a BMI constraint. To this end, we first set a *template polynomial* <sup>10</sup> B(**a**, **x**) parameterized by unknown real coefficients **a** as the barrier certificate. We then proceed by setting templates for the remaining unknown polynomials (e.g., vi,j (**x**)) and SOS polynomials (e.g., σ(**x**) and ρ(**x**)) in h(**x**), with all the parameters in these templates grouped into **s**. Observe that the parameterized SOS polynomial h(**a**, **s**, **x**) is of a bilinear form on the parameter spaces, i.e., h(**a**, **s**, **x**) is linear in **a** and **s** separately. However, nonlinearity arises in the combined parameter space (**a**, **s**) due to the product couplings of **a** and **s**, i.e., <sup>v</sup>i,j (**s**i,j , **<sup>x</sup>**)L<sup>j</sup> *<sup>f</sup>* B(**a**, **x**) in the consecution constraint.

Now the problem of synthesizing an invariant barrier certificate boils down to searching for an instantiation of the parameters **a** and **s** such that the sufficient condition in Theorem 5 holds (or alternatively, the necessary condition in Theorem 6 holds and the posterior check passed). Such an instantiation of **a** (making B(**a**, **x**) an invariant barrier certificate) will be called *valid* in the sequel.

Suppose that a parameterized SOS polynomial h(**a**, **s**, **x**) is of degree at most <sup>2</sup>d, with user-specified <sup>d</sup> <sup>∈</sup> <sup>N</sup>. Then <sup>h</sup>(**a**, **<sup>s</sup>**, **<sup>x</sup>**) can always be written in *quadratic form* as h(**a**, **s**, **x**) = **b**<sup>T</sup>Q(**a**, **s**)**b**, where **b** = (1, x1, x2, x1x2,...,x<sup>d</sup> <sup>n</sup>) is the *basis vector* of size p = <sup>n</sup>+<sup>d</sup> n containing all monomials of degree up to d, and Q(**a**, **s**) ∈ <sup>S</sup><sup>p</sup> is a parameterized real symmetric matrix known as the *Gram matrix* [6] 11. An important fact states that h(**a**, **s**, **x**) is SOS if and only if Q(**a**, **s**) 0.

Let F(**a**, **s**) = −Q(**a**, **s**). As per h(**a**, **s**, **x**), the matrix-valued function F(**a**, **s**) is bilinear in (**a**, **s**). Observe that h(**a**, **s**, **x**) *is SOS if and only if the BMI constraint* F(**a**, **s**) 0 *holds*. See Example 1 for an illustration of this BMI encoding.

<sup>9</sup> Such a check inherits decidability of the first-order theory of real-closed fields [53].

<sup>10</sup> A template polynomial g(**a**, **x**) is required to be linear in its parameters **a**.

<sup>11</sup> Extracting the Gram matrix amounts to solving a system of linear equations resulting from coefficient matching. The derived Gram matrix may contain extra unknowns if the system of linear equations admits multiple solutions, which nevertheless can be encoded in our subsequent workflow by enumerating the basis of its null space.

In general, F(**a**, **s**) can be flattened in an expanded bilinear form as

$$\mathcal{F}(\mathbf{a}, \mathbf{s}) = F + \sum\_{i=1}^{m} a\_i H\_i + \sum\_{j=1}^{n} s\_j G\_j + \sum\_{i=1}^{m} \sum\_{j=1}^{n} a\_i s\_j F\_{i,j}$$

where <sup>m</sup> and <sup>n</sup> are the size of **<sup>a</sup>** and **<sup>s</sup>**, respectively; F,Hi, G<sup>j</sup> , Fi,j ∈ S<sup>p</sup> are constant matrices. Discharging the conditions of invariant barrier certificates hence amounts to solving the BMI feasibility problem of finding **a** and **s** s.t.

$$\mathcal{F}\_{\iota}(\mathbf{a}, \mathbf{s}) \preceq 0, \quad \iota = 1, 2, \dots, l. \tag{10}$$

Here F(**a**, **s**) is indexed by ι and l is the number of SOS constraints involved.

To exploit well-developed techniques in optimization, the feasibility problem (10) is transformed to an optimization problem subject to BMI constraints:

$$\begin{array}{ll}\underset{\lambda,\mathbf{a},\mathbf{s}}{\text{maximize}} & \lambda\\\text{subject to} & \mathcal{F}\_{\iota}(\mathbf{a},\mathbf{s}) + \lambda I \preceq 0, \quad \iota = 1,2,\ldots,l. \end{array} \tag{11}$$

A solution (λ, **a**, **s**) to (11) is *feasible* if it satisfies the BMIs in (11), and *strictly feasible* if all the BMIs are satisfied with strict inequalities. We sometimes drop the λ component in the solution when it is clear from the context. Notice that *problem* (10) *has a feasible solution if and only if the optimal value* λ<sup>∗</sup> *in the BMI optimization problem* (11) *is non-negative*.

To achieve (weak) completeness of our method in subsequent sections on solving the BMI optimization problem, we make the following assumption on the boundedness of the search space (**a**, **s**) of the optimization.

**Assumption 1 (Boundedness on the Parameters).** *Every feasible solution* (**a**, **s**) *to the BMI problem* (11) *is in a compact set with non-empty interior, i.e.,*

$$\mathcal{C}(\mathbf{a}, \mathbf{s}) \in \mathcal{C}\_{\mathbf{a}} \times \mathcal{C}\_{\mathbf{s}} = \left\{ (\mathbf{a}, \mathbf{s}) \mid \left\| \mathbf{a} \right\|^2 \le L\_{\mathbf{a}}, \left\| \mathbf{s} \right\|^2 \le L\_{\mathbf{s}} \right\}.$$

*for some known bounds* <sup>L</sup>**a**, L**<sup>s</sup>** <sup>∈</sup> <sup>R</sup><sup>+</sup>*.*

*Remark 4.* The boundedness on **a** in Assumption 1 makes sense in practice since we usually prefer barrier certificates with bounded coefficients. Moreover, when the bilinear functions F<sup>ι</sup>(**a**, **s**) in (11) are affine in **a** and **s**, i.e., with a zero constant matrix F, the parameters **a** and **s** can be scaled independently by any positive factor. Therefore in this case, w.l.o.g, one may simply set L**<sup>a</sup>** = L**<sup>s</sup>** = 1.

#### **5 Solving BMI Optimizations via DCP**

The BMI optimization problem (11), derived from the synthesis problem, is known to be NP-hard and contains non-convex constraints [55], and hence is not amenable to efficient (polynomial-time) algorithms committed to solving convex optimizations. In this section, we present an algorithm for solving general BMI optimizations via difference-of-convex programming [33,52], which solves a series of convex sub-problems that approaches a local optimum of (11).

For brevity, we consider optimization problems with a single BMI constraint<sup>12</sup>:

maximize **<sup>z</sup>**=(**x**,**y**) <sup>g</sup>(**z**)

$$\text{subject to} \quad \mathcal{B}(\mathbf{x}, \mathbf{y}) \hat{=} F + \sum\_{i=1}^{m} x\_i H\_i + \sum\_{j=1}^{n} y\_j G\_j + \sum\_{i=1}^{m} \sum\_{j=1}^{n} x\_i y\_j F\_{i,j} \preceq 0 \quad (12)$$

where the objective function <sup>g</sup> : <sup>R</sup><sup>m</sup>+<sup>n</sup> <sup>→</sup> <sup>R</sup> is linear in **<sup>z</sup>** = (**x**, **<sup>y</sup>**); F,Hi, G<sup>j</sup> , Fi,j ∈ S<sup>p</sup> are constant symmetric matrices.

#### **5.1 Difference-of-Convex Decomposition**

The key challenge in solving the BMI problem (12) is its non-convexity, that is, the matrix-valued function B(**x**, **y**) is, in general, not psd-convex.

There have been attempts, most pertinently in [10], to decompose a bilinear function as a difference between two psd-convex functions, known as the *difference-of-convex* (DC) *decomposition*, such that the optimization in its decomposed form enjoys well-established techniques in difference-of-convex programming [33,52]. The DC decomposition in [10], however, is confined to BMIs of a specific structure, namely, <sup>X</sup>T<sup>Y</sup> <sup>+</sup> <sup>Y</sup> <sup>T</sup><sup>X</sup> 0, where <sup>X</sup> and <sup>Y</sup> are matrix variables containing variables x<sup>i</sup> and y<sup>j</sup> , respectively. The more general bilinear function B(**x**, **y**) in (12) does unfortunately not admit straightforward forms of decomposition such as those in [10, Lemma 3.1].

In what follows, we present a difference-of-convex decomposition of the matrix-valued function B(**x**, **y**), inspired by [58], using eigendecomposition.

First, observe that the function B(**x**, **y**) can be written as

$$\mathcal{B}(\mathbf{x}, \mathbf{y}) = \begin{pmatrix} \mathbf{x} \otimes I \\ \mathbf{y} \otimes I \end{pmatrix}^{\mathsf{T}} \begin{pmatrix} 0 & I \\ I^{\mathsf{T}} & 0 \end{pmatrix} \begin{pmatrix} \mathbf{x} \otimes I \\ \mathbf{y} \otimes I \end{pmatrix} + \begin{pmatrix} \boldsymbol{\Omega}\_{1} \ \boldsymbol{\Omega}\_{2} \end{pmatrix} \begin{pmatrix} \mathbf{x} \otimes I \\ \mathbf{y} \otimes I \end{pmatrix} + F \tag{13}$$

where <sup>⊗</sup> denotes the Kronecker product: for two matrices <sup>A</sup> <sup>∈</sup> <sup>R</sup><sup>a</sup>×<sup>b</sup> and <sup>B</sup> <sup>∈</sup> <sup>R</sup><sup>c</sup>×<sup>d</sup>, <sup>A</sup> <sup>⊗</sup> <sup>B</sup> = [ <sup>A</sup>(1, 1)B,...,A(1, b)B; ... ; <sup>A</sup>(a, 1)B,...,A(a, b)B] <sup>∈</sup> <sup>R</sup>ac×bd, 0 represents the zero matrices with compatible dimensions, and

$$\Gamma = \frac{1}{2} \begin{pmatrix} F\_{1,1} & \dots & F\_{1,n} \\ \vdots & \ddots & \vdots \\ F\_{m,1} & \dots & F\_{m,n} \end{pmatrix}, \quad \Omega\_1 = \left( H\_1 \; \dots \; H\_m \right), \quad \Omega\_2 = \left( G\_1 \; \dots \; G\_n \right).$$

The form of (13) implies that <sup>B</sup>(**x**, **<sup>y</sup>**) is psd-convex if the matrix <sup>M</sup> <sup>=</sup> - 0 Γ Γ <sup>T</sup> 0 is positive semidefinite. Unfortunately, as [58, Theorem 1] points out, for a non-trivial bilinear function B(**x**, **y**), M may not be positive semidefinite.

<sup>12</sup> Multiple BMI constraints can be joined as a single BMI in a block-diagonal fashion.

Nevertheless, the matrix M can always be decomposed as M = M<sup>1</sup> − M<sup>2</sup> with M1, M<sup>2</sup> 0, i.e., a difference between two psd-matrices. One way to do so is to use the *eigendecomposition* of the (real symmetric<sup>13</sup>) matrix <sup>M</sup> ∈ S(m+n)p. That is, M = V <sup>T</sup>DV , where the orthogonal matrix V contains the eigenvectors of M; D is a diagonal matrix whose diagonal elements are the eigenvalues of M.

Let D<sup>+</sup> be the matrix obtained by setting all negative elements of D to zero and <sup>D</sup><sup>−</sup> <sup>=</sup> <sup>D</sup><sup>+</sup> <sup>−</sup> <sup>D</sup>. We have

$$M = \underbrace{V^\top D^+ V}\_{M\_1} - \underbrace{V^\top D^- V}\_{M\_2}.$$

It follows that M1, M<sup>2</sup> 0 and therefore we find a DC decomposition of B(**x**, **y**):

**Theorem 7 (Difference-of-Convex Decomposition).** *The following form*

$$\mathcal{B}(\mathbf{x}, \mathbf{y}) = \mathcal{B}^{+}(\mathbf{x}, \mathbf{y}) - \mathcal{B}^{-}(\mathbf{x}, \mathbf{y}) \tag{14}$$

*where*

$$\begin{aligned} \mathcal{B}^+(\mathbf{x}, \mathbf{y}) &= \begin{pmatrix} \mathbf{x} \otimes I \\ \mathbf{y} \otimes I \end{pmatrix}^\mathsf{T} M\_1 \begin{pmatrix} \mathbf{x} \otimes I \\ \mathbf{y} \otimes I \end{pmatrix} + \begin{pmatrix} \Omega\_1 \ \Omega\_2 \end{pmatrix} \begin{pmatrix} \mathbf{x} \otimes I \\ \mathbf{y} \otimes I \end{pmatrix} + F \\ \mathcal{B}^-(\mathbf{x}, \mathbf{y}) &= \begin{pmatrix} \mathbf{x} \otimes I \\ \mathbf{y} \otimes I \end{pmatrix}^\mathsf{T} M\_2 \begin{pmatrix} \mathbf{x} \otimes I \\ \mathbf{y} \otimes I \end{pmatrix} \end{aligned}$$

*is a difference-of-convex decomposition of* B(**x**, **y**)*. Namely, the matrix-valued functions* <sup>B</sup><sup>+</sup>(**x**, **<sup>y</sup>**) *and* <sup>B</sup><sup>−</sup>(**x**, **<sup>y</sup>**) *are psd-convex on* <sup>R</sup><sup>m</sup>+<sup>n</sup>*.*

*Remark 5.* In practice, the aforementioned matrices M, M<sup>1</sup> and M<sup>2</sup> induced by eigendecomposition are often highly sparse. One can hence exploit the sparsity to improve the algorithmic performance of the DCP-based synthesis approach.

#### **5.2 Reduction to LMIs**

On top of the DC decomposition (cf. Theorem 7), we can now apply a standard iterative procedure in difference-of-convex programming [10] to solve the BMIs.

The core idea of the procedure is to iteratively solve a series of convex subproblems. More specifically, given a feasible solution **z**<sup>k</sup> = (**x**<sup>k</sup>, **y**<sup>k</sup>) to the BMI optimization problem (12), the "concave part" −B<sup>−</sup>(**x**, **y**) in (14) is linearized around **z**<sup>k</sup>, thereby yielding a series of convex programs (k = 0, 1,...):

$$\begin{array}{ll}\text{maximize} & g(\mathbf{z}) + \frac{1}{2}\delta \left\| \mathbf{z} - \mathbf{z}^{k} \right\|^{2} \\ \text{subject to} & \mathcal{B}^{+}(\mathbf{z}) - \mathcal{B}^{-}\left(\mathbf{z}^{k}\right) - \mathcal{D}\mathcal{B}^{-}\left(\mathbf{z}^{k}\right) \left(\mathbf{z} - \mathbf{z}^{k}\right) \preceq 0 \end{array} \tag{15}$$

where DB<sup>−</sup>(**z**): <sup>R</sup><sup>m</sup>+<sup>n</sup> → S<sup>p</sup> is the derivative of the matrix-valued function <sup>B</sup><sup>−</sup> at **<sup>z</sup>**, i.e., a linear mapping from a vector **<sup>u</sup>** <sup>∈</sup> <sup>R</sup><sup>m</sup>+<sup>n</sup> to a matrix in <sup>S</sup><sup>p</sup>:

$$\mathcal{D}\mathcal{B}^-(\mathbf{z})(\mathbf{u}) \hat{=} \sum\_{i=1}^{n+m} u\_i \frac{\partial \mathcal{B}^-}{\partial z\_i}(\mathbf{z}).$$

<sup>13</sup> M thus only has real eigenvalues.

# **Algorithm 1:** BMI-DC: Solving BMIs based on DC decomposition

**input:** A BMI optimization problem (12) with a strictly feasible initial solution **z**0. **output:** A sequence of feasible solutions S = **z**0,..., **z**k to the BMI optimization. **<sup>1</sup>** k ← 0; S ← **z**0 ; **<sup>2</sup>** M ← reformulation of (12) as (13); **<sup>3</sup>** (M1, M2) ← DC decomposition of M as in (14); **<sup>4</sup> repeat <sup>5</sup>** Construct the convex sub-problem (15) out of (M1, M2) linearized around **z**k; **<sup>6</sup> <sup>z</sup>**k+1 <sup>←</sup> optimum of the program (15); **<sup>7</sup>** S ← S ∪ **z**k+1 ; - S keeps track of visited points **<sup>8</sup>** k ← k + 1; **<sup>9</sup> until z**<sup>k</sup> <sup>−</sup> **<sup>z</sup>**k−<sup>1</sup> < ε for a given tolerance <sup>ε</sup> <sup>∈</sup> <sup>R</sup><sup>+</sup> 0 ; **<sup>10</sup> return** S;

An extra regularization term <sup>1</sup> <sup>2</sup> <sup>δ</sup>**<sup>z</sup>** <sup>−</sup> **<sup>z</sup>**<sup>k</sup><sup>2</sup> with δ < 0 is added in (15) to enforce that g(**z**) strictly increases after each iteration until it stabilizes, which can be encoded as a second-order cone constraint and embedded in SDP solving.

Note that the linearized problem (15) is convex and therefore can be solved efficiently<sup>14</sup> via methods including, among others, augmented Lagrangian methods [35] and gradient descent methods [3]. Furthermore, the Schur complement in Theorem 3 implies that (15) can be reformulated as an LMI problem:

**Theorem 8.** *The quadratic matrix inequality (QMI) constraint*

$$\mathcal{B}^+(\mathbf{z}) - \mathcal{B}^-\left(\mathbf{z}^k\right) - \mathcal{D}\mathcal{B}^-\left(\mathbf{z}^k\right)\left(\mathbf{z} - \mathbf{z}^k\right) \preceq 0$$

*in* (15) *is equivalent to the LMI constraint*<sup>15</sup>

$$
\begin{pmatrix}
\left(\mathbf{z}\otimes I\right)^{\mathsf{T}}N^{\mathsf{T}}-\mathcal{B}^{-}\left(\mathbf{z}^{k}\right)-\mathcal{D}\mathcal{B}^{-}\left(\mathbf{z}^{k}\right)\left(\mathbf{z}-\mathbf{z}^{k}\right)+\mathcal{Q}\left(\mathbf{z}\otimes I\right)+F
\end{pmatrix}\preceq 0
$$

*where* N *is the square root matrix of* M1*, i.e.,* M<sup>1</sup> *=* N<sup>T</sup>N*, and* Ω = Ω<sup>1</sup> Ω<sup>2</sup> *.*

Theorem 8 entails that the series of linearized convex sub-problems of the form (15) can be solved alternatively by most off-the-shelf SDP solvers designated for discharging LMIs via polynomial-time algorithms, say the interiorpoint methods. Furthermore, by taking the optimum of the k-th sub-problem to be the next linearization point **z**<sup>k</sup>+1, we obtain an iterative procedure for solving general BMIs, as depicted in Algorithm 1.

Algorithm 1 falls into the DCP framework [10] and thus enjoys useful properties, e.g., soundness, termination and convergence as follows.

<sup>14</sup> The global optimum of (15) is attainable under standard assumptions, e.g., Slater's condition and the second-order sufficient KKT conditions [3].

<sup>15</sup> This transforms a QMI with matrices in <sup>S</sup><sup>p</sup> to an LMI with matrices in <sup>S</sup>(m+n+1)<sup>p</sup>.

**Theorem 9 (Soundness).** *Every solution* **z**<sup>i</sup> = (**x**<sup>i</sup> , **y**<sup>i</sup> ) ∈ S *with* i = 0,...,k *returned by Algorithm 1 is a feasible solution to the original BMI problem* (12)*.*

The result below states termination and convergence of Algorithm 1 in terms of *KKT points* of (12), i.e., solutions fulfilling the KKT conditions [3] of (12)<sup>16</sup>.

**Theorem 10 (Termination and convergence).** *If* (12) *has finitely many KKT points, then (1) for* <sup>ε</sup> <sup>∈</sup> <sup>R</sup><sup>+</sup>*, Algorithm <sup>1</sup> terminates; (2) for* <sup>ε</sup> = 0*, Algorithm 1 visits an infinite sequence of solutions converging to a KKT point.*

We remark that, under some sufficient KKT conditions and regularity conditions [3], a KKT point suffices as a local optimum. In this case, the infinite sequence {**z**<sup>i</sup> }<sup>i</sup>∈<sup>N</sup> of points visited by Algorithm 1 (for ε = 0) converges to a local optimum of (12).

#### **5.3 Finding the Initial Solution**

The iterative procedure in Algorithm 1 starts with a fed-by-oracle strictly feasible initial solution **z**<sup>0</sup> to the BMI problem (12). Finding such an initial solution, however, is non-trivial in general due to the non-convexity of (12). We argue though, that a strictly feasible initial solution can be obtained for the BMI problem of the form (11) induced by the barrier-certificate synthesis problem.

Recall that in the BMI problem (11), bilinearity arises from the multiplication of B(**a**, **x**) with some unknown multiplier polynomials parameterized by **s**. One way to reduce the BMI constraints to LMIs is to fix every multiplier polynomial to be a non-negative constant, thereby yielding a linear program:

$$\begin{array}{ll}\underset{\lambda,\mathbf{a}}{\text{maximize}} & \lambda\\\text{subject to} & \mathcal{F}\_{\iota}(\mathbf{a},\mathbf{s})\big|\_{\mathbf{s}=(c\_{\iota},0,\ldots,0)} + \lambda I \preceq 0, \quad \iota = 1,2,\ldots,l\end{array} \tag{16}$$

where **<sup>s</sup>** in <sup>F</sup><sup>ι</sup>(**a**, **<sup>s</sup>**) is substituted by (cι, <sup>0</sup>,..., 0) with <sup>c</sup><sup>ι</sup> <sup>∈</sup> <sup>R</sup><sup>+</sup> <sup>0</sup> , which encodes a non-negative constant multiplier polynomial. Observe that no **s**-variable is involved in (16) and the constraints therein are linear in **a**.

Apparently, a strictly feasible solution (λ, **a**) to (16) induces a strictly feasible solution (λ, **a**,(cι, 0,..., 0)) to (11) as well. Moreover, we have

**Lemma 1.** *The LMI program* (16) *always has a strictly feasible solution.*

As a consequence, a strictly feasible solution to the BMI problem (11) can be obtained by solving the LMI problem (16). In fact, when considering Lie derivatives only up to the first order, solving (the feasibility counterpart of) (16) is exactly the procedure to synthesize either an *exponential barrier certificate* [29] (with <sup>c</sup><sup>ι</sup> <sup>∈</sup> <sup>R</sup><sup>+</sup>) or a *convex barrier certificate* [41] (with <sup>c</sup><sup>ι</sup> = 0). Algorithm <sup>1</sup> therefore subsumes existing synthesis techniques in the sense that any valid barrier certificate synthesized by methods in [29,41] can also be discovered by Algorithm 1. Moreover, an alternative way to reduce the BMI constraints to LMIs is to fix the multipliers to be some given non-trivial (SOS) polynomials [62].

<sup>16</sup> Addressing the KKT conditions in detail falls outside the scope of this paper.

**Algorithm 2:** Branch-and-Bound: Searching for a valid parameter **<sup>a</sup>**¯ **input:** A BMI optimization problem of the form (11) with <sup>C</sup>**<sup>a</sup>** <sup>=</sup> {**<sup>a</sup>** | **a**<sup>2</sup> <sup>≤</sup> <sup>L</sup>**a**}. **output:** A valid parameter **a**¯, or otherwise ⊥ indicating a failure. **<sup>1</sup> if** L**<sup>a</sup>** < η **then return** ⊥; abort on fine-enough partitions (<sup>η</sup> <sup>∈</sup> <sup>R</sup>+) /\* sample-and-check is not necessary if Theorem 6 is used \*/ **<sup>2</sup> a**¯ ← a randomly-sampled point in C**a**; **<sup>3</sup> if a**¯ is valid **then return a**¯; check validity (inductive invariance) **<sup>4</sup> if** *proj***a**(S*glb*) ∩ C**<sup>a</sup>** <sup>=</sup> <sup>∅</sup> **then** - S*glb* contains a global set of visited points **<sup>5</sup>** S ← apply BMI-DC in Algorithm 1 to (11) with initial solution in (C**a**, C**s**); **<sup>6</sup>** S*glb* ← S*glb* ∪ S; /\* checking validity is not necessary if Theorem 5 is used \*/ **<sup>7</sup> if** a valid parameter **<sup>a</sup>**¯ <sup>∈</sup> *proj***a**(S) is found **then return <sup>a</sup>**¯; **<sup>8</sup>** (C<sup>1</sup> **<sup>a</sup>**, <sup>C</sup><sup>2</sup> **<sup>a</sup>**) <sup>←</sup> *bisect*(C**a**); partition the parameter space **<sup>9</sup> <sup>a</sup>**¯ <sup>←</sup> Branch-and-Bound(C<sup>1</sup> **<sup>a</sup>**); **<sup>10</sup> if a**¯ = ⊥ **then return a**¯; **<sup>11</sup> else return** Branch-and-Bound(C<sup>2</sup> **<sup>a</sup>**);

*Remark 6.* Different choices of the multiplier constants c<sup>ι</sup> in (16) may lead to different initial solutions fed to Algorithm 1, thereby considerably different number of iterations until termination. In practice, techniques like randomization are worth exploring when choosing these multiplier constants.

#### **6 Incorporating in a Branch-and-Bound Framework**

The aforementioned iterative procedure on solving a series of convex optimizations converges only to a local optimum of the BMI problem (11) (or more generally, (12)). This means that, in some cases, it may miss the global optimum that induces a non-negative λ∗. We present in this section a solution to this problem by incorporating the iterative procedure into a branch-and-bound framework that searches for the global optimum in a divide-and-conquer fashion, as is a common technique in non-convex optimizations.

The basic idea is as follows. We first try to solve the BMI problem (11) by Algorithm 1 over the compact parameter space (C**<sup>a</sup>**, C**<sup>s</sup>**). If a valid solution, (i.e., a solution that contains a valid parameter **a**¯ ∈ C**<sup>a</sup>** such that B(**a**¯, **x**) is an invariant barrier certificate) is found, then the corresponding barrier certificate can be obtained. Otherwise, we keep bisecting C**<sup>a</sup>** and apply Algorithm 1 over each bisection<sup>17</sup>. The procedure, as depicted in Algorithm 2 in a recursive manner, terminates when a valid parameter is found or the partition is fine enough.

Algorithm 2 takes as input a BMI problem of the form (11) that encodes either the sufficient condition in Theorem 5 or the necessary condition in Theorem 6 for invariant barrier certificates. In the former case, a sample-and-check process (Line 2–3) is necessary to attain (weak) completeness (see Theorem 11). The conditional statement in Line 4 rules out parameter (sub-)spaces that have

<sup>17</sup> The validity of **<sup>a</sup>**¯ ∈ C**<sup>a</sup>** does not depend on **<sup>s</sup>**, thus we do not partition <sup>C</sup>**s**.

already been explored, which is the case when the projection of some visited point in S*glb* (a global set that keeps track of visited points by Algorithm 1, initialized as ∅) onto **a** is in the current parameter space.

The following theorem claims a weak completeness result: our method guarantees to find a barrier certificate when there exists an inductive invariant (in the form of a given template) that suffices to certify safety of the system.

**Theorem 11 (Weak Completeness).** *Algorithm 2 returns a valid parameter* **<sup>a</sup>**¯ ∈ C**<sup>a</sup>***, if (1) the partition granularity is fine enough (i.e., small enough* <sup>η</sup> <sup>∈</sup> <sup>R</sup><sup>+</sup>*), (2) the degrees of multiplier polynomials and SOS polynomials used to form* (11) *are large enough, and (3) there exists, for the given template* B(**a**, **x**)*, a strictly valid parameter* **a**ˆ ∈ C**<sup>a</sup>** *(i.e., any parameter in some neighborhood of* **a**ˆ *is valid).*

*Remark 7.* The bisection operation in Algorithm 2 induces —in the worst case an exponential blow-up in the number of branches. In practice, one can prune branches inducing only negative objective values, via, e.g., convex relaxation [26].

#### **7 Experimental Results**

We have carried out a prototypical implementation<sup>18</sup> of our synthesis techniques in Wolfram Mathematica, which was selected due to its built-in primitives for SDP, polynomial algebra and matrix operations. Given a safety verification problem as input, our implementation works toward discovering an invariant barrier certificate (in the form of a given template) that witnesses unbounded-time safety of the system. A collection of benchmark examples (detailed in [57, Appendix B]) has been evaluated on a 2.10 GHz Intel Xeon processor with 376 GB RAM running 64-bit CentOS Linux 7.

Table <sup>1</sup> reports the empirical results. BMI-DC concerns our locally-convergent Algorithm 1 for solving BMIs (encoding the sufficient condition in Theorem 5) based on DC decomposition. We compare our approach with PENLAB [14]—an off-the-shelf solver in Matlab for directly discharging the same BMI problems (with no guarantee on convergence)—and SOSTOOLS [39]—for solving LMIs derived from Prajna and Jadbabaie's original barrier-certificate condition [41]. The comparison is performed under the same problem configurations<sup>19</sup>. Due to numerical errors caused by floating-point computations and the fact that reaching the local/global optimum does not necessarily yield a valid barrier certificate, we additionally perform a posterior check, via both the quantifierelimination procedure in Mathematica and the SMT solver Z3 [37], of the synthesized candidate barrier certificate per Definition 4.

Table <sup>1</sup> shows that BMI-DC suffices to synthesize valid barrier certificates in most of the examples within a reasonable number of iterations (i.e., the number of convex sub-problems solved by SDP). This however does not cover all the cases:

<sup>18</sup> Available at https://github.com/Chenms404/BMI-DC.

<sup>19</sup> For PENLAB and SOSTOOLS, we use their optimized, built-in criteria for termination and methods for finding the initial solutions.


**Table 1.** Empirical results on benchmark examples (time in seconds)

nsys: system dimension; dflow: maximal flow-field degree; dBC: degree of the template barrier certificate. #iter.: number of iterations. 0 means that the initial solution (cf. Subsect. 5.3) is valid. verified: the synthesized barrier certificate is valid (✓), invalid (✗) or inconclusive (**?**, beyond the capability of quantifier elimination in Mathematica and nonlinear reasoning in Z3).

time: CPU-time, excluding that for casting the BMIs/LMIs. Boldface marks the winner among ✓'s.

for the focus example, the solution is close enough to a local optimum (after 100 iterations) but yields still an invalid barrier certificate. This problem can be solved (if there exists an invariant barrier certificate as specified) by enforcing the branch-and-bound framework as presented in Sect. 6. The phase portraits of a selected set of examples and the synthesized invariant barrier certificates are depicted in Fig. 2 (see more in [57, Appendix B]).

The comparison in Table 1 suggests that (1) Our invariant barrier-certificate condition recognizes more barrier certificates than the original (more conservative) condition as implemented in SOSTOOLS. In particular, the lie-high-order example does admit an inductive invariant in the form of the given template, but none of the existing barrier-certificate conditions [4,60,63] —concerning Lie derivatives only up to the first order— recognizes it, since we have <sup>L</sup><sup>1</sup> *<sup>f</sup>* B(**x**)=0

**Fig. 2.** Phase portraits of a selected set of examples with the synthesized invariant barrier certificates. The arrows indicate the vector field (hidden in 3D-graphics for a clear presentation) and the solid curves are randomly sampled trajectories.

for some **x** on the boundary of B and hence it requires to exploit the second-order Lie derivative <sup>L</sup><sup>2</sup> *<sup>f</sup>* B; (2) Our DCP-based synthesis algorithm finds more barrier certificates in less time than directly solving the BMI problems via non-convex optimization techniques as implemented in PENLAB.

We remark that symbolic methods based on, e.g., quantifier elimination [36], can hardly deal with any of the examples listed in Table 1 due to the prohibitively high computation complexity. Moreover, it would be desirable to pursue a comparison with the augmented Lagrangian method for solving BMIs as proposed in [4], which unfortunately is not yet possible due to the unavailability of the implementation thereof. We will discuss crucial differences to [4] in Sect. 8.

#### **8 Related Work**

As surveyed in [15], the research community has, over the past three decades, extensively addressed the automatic verification of safety-critical hybrid systems. The almost universal undecidability of the unbounded-time reachability problem [1], however, confines the sound key-press routines to either semi-decision procedures or approximation schemes, most of which address bounded-time verification by, e.g., computing the finite-time image of a set of initial states.

Invariant generation [36,41], amongst others, is a well-established approximation scheme that provides a reliable witness for safety (or equivalently, unreachability) of dynamical systems over the infinite time horizon. Invariants can be constructed in various forms, e.g., barrier certificates [41,51] and differential invariants [36,40]. With a priori specified templates, the invariant synthesis problem can be reduced to numerical optimizations or constraint solving, as in, e.g., [22,25,46,54].

Most pertinently, Prajna and Jadbabaie proposed in their seminal work [41] a concept coined *barrier certificate* to encode invariants. To enable efficient synthesis via semidefinite programming, the barrier-certificate condition in [41] strengthens the general condition encoding inductive invariance. Since then, significant efforts have been investigated in developing more relaxed (i.e., weaker) forms of barrier-certificate condition that still admit efficient synthesis, thereby leading to, e.g., exponential-type barrier certificates [29], Darboux-type barrier certificates [62], general barrier certificates [8] and vector barrier certificates [51]. To attain efficient synthesis, these barrier-certificate conditions share a common property on convexity. That is, if for some **<sup>a</sup>**1, **<sup>a</sup>**<sup>2</sup> <sup>∈</sup> <sup>R</sup>m, <sup>B</sup>(**a**1, **<sup>x</sup>**) and B(**a**2, **x**) both satisfy the barrier-certificate condition, then for any 0 <μ< 1, B(μ**a**<sup>1</sup> + (1 − μ)**a**2, **x**) must also satisfy the barrier-certificate condition.

However, neither the semantic barrier-certificate condition (9) encoding the general principle of barrier certificates [8,51] nor the inductive invariant condition (8) is convex. This means, when resorting to convex barrier-certificate conditions, one may miss some potential barrier certificates that suffice as inductive invariants witnessing safety. Therefore, non-convex conditions were suggested [60], for which the synthesis problem can be reduced to BMI problems solvable via customized schemes, e.g., the augmented Lagrangian method [4] and the alternating minimization algorithm [63]. Our synthesis techniques also exploit a BMI reduction, with three crucial differences: (1) our invariant barriercertificate condition is equivalent to the inductive invariant condition in the sense of Theorem 4, and thus is less conservative than all the aforementioned conditions which consider Lie derivatives only up to the first order; (2) our DCP-based techniques for solving BMIs naturally inherit appealing results on convergence and (weak) completeness, which are not (and can hardly be) provided by the approaches in [4,60,63]; (3) our DCP-based iterative procedure visits only feasible solutions to the original BMI problem, and hence whenever a solution that induces a non-negative objective value is found, we can safely terminate the algorithm and claim a feasible solution to the original BMI problem, which may yield a valid barrier certificate. This is not the case for the approaches in [4,60,63].

Beyond barrier certificates, Wang and Rajamani [58] investigated the feasibility problem of general BMI problems with an application to multi-objective nonlinear observer design. The technique of eigendecomposition was also used therein to conduct the DC decomposition. The decomposed concave part, however, is simply ignored and no iterative procedure that exhibits convergence to a local optimum can be provided.

The idea of augmenting a locally-convergent algorithm with a branch-andbound framework to find the global optimum has been exploited in the realm of optimization [20] and control [56]. In contrast, our method is designed for the specific problem of barrier-certificate synthesis, and hence our branch-andbound algorithm concerns only the parameter space of **a**, i.e., coefficients of the template barrier certificate.

Finally, we refer interested readers to other approaches to solving BMI problems, e.g., rank minimization [23,38,45], sequential SDP [7,12], as well as methods committed to general non-convex optimizations, e.g., interior point trustregion [5,9,34], successive linearization [24] and primal-dual interior point [59].

#### **9 Conclusion**

Barrier certificates are powerful tools to prove time-unbounded safety of hybrid systems. We have presented a new condition on barrier certificates—the invariant barrier-certificate condition. This condition is by far the least conservative one on barrier certificates, and can be shown as the weakest possible one to attain inductive invariance. We showed that our invariant barrier-certificate condition can be reformulated as an optimization problem subject to bilinear matrix inequalities, which can be solved by our locally-convergent algorithm based on difference-ofconvex programming. By incorporating this algorithm into a branch-and-bound framework, we obtained a weak completeness result. Experiments on benchmark examples suggested that our invariant barrier-certificate condition recognizes more barrier certificates than existing conditions, and that our DCP-based algorithm is more efficient than directly solving the BMIs via off-the-shelf solvers.

We stress that our techniques for solving BMIs are of a general nature rather than being confined to barrier-certificate synthesis. Interesting future directions include to extend our method to other synthesis problems, e.g., discovering invariants and/or termination proofs of deterministic/probabilistic programs.

**Acknowledgements.** The authors would like to thank Hengjun Zhao for the fruitful discussion on differential dynamics requiring high-order Lie derivatives.

#### **References**


466 Q. Wang et al.


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **An Iterative Scheme of Safe Reinforcement Learning for Nonlinear Systems via Barrier Certificate Generation**

Zhengfeng Yang<sup>1</sup>, Yidan Zhang<sup>1</sup>, Wang Lin2(B) , Xia Zeng<sup>3</sup>, Xiaochao Tang<sup>1</sup>, Zhenbing Zeng<sup>4</sup>, and Zhiming Liu3,5

<sup>1</sup> Shanghai Key Lab of Trustworthy Computing, East China Normal University, Shanghai, China

zfyang@sei.ecnu.edu.cn,{ydzhang,xctang}@stu.ecnu.edu.cn <sup>2</sup> School of Information Science and Technology, Zhejiang Sci-Tech University, Hangzhou, China

linwang@zstu.edu.cn

<sup>3</sup> School of Computer and Information Science, Southwest University, Chongqing, China

xzeng0712@swu.edu.cn

<sup>4</sup> Department of Mathematics, Shanghai University, Shanghai, China

#### zbzeng@shu.edu.cn

<sup>5</sup> Centre for Intelligent and Embedded Software, Northwestern Polytechnical University, Suzhou, China

zliu@nwpu.edu.cn

**Abstract.** In this paper, we propose a safe reinforcement learning approach to synthesize deep neural network (DNN) controllers for nonlinear systems subject to safety constraints. The proposed approach employs an iterative scheme where a *learner* and a *verifier* interact to synthesize safe DNN controllers. The *learner* trains a DNN controller via deep reinforcement learning, and the *verifier* certifies the learned controller through computing a maximal safe initial region and its corresponding barrier certificate, based on polynomial abstraction and bilinear matrix inequalities solving. Compared with the existing verification-in-the-loop synthesis methods, our iterative framework is a sequential synthesis scheme of controllers and barrier certificates, which can learn safe controllers with adaptive barrier certificates rather than user-defined ones. We implement the tool SRLBC and evaluate its performance over a set of benchmark examples. The experimental results demonstrate that our approach efficiently synthesizes safe DNN controllers even for a nonlinear system with dimension up to 12.

This work was partially supported by the Scientific and Technological Innovation 2030 Major Projects under Grant 2018AAA0100902, the National Natural Science Foundation of China under Grant 61772203, 61902325, 62032019, 61732019, the Zhejiang Provincial Natural Science Foundation of China under Grant LY20F020020, the Capacity Development Grant of Southwest University under Grant SWU116007, the Fundamental Research Funds for the Central Universities under Grant SWU117058.

**Keywords:** Formal verification · Safe reinforcement learning · Barrier certificates · Continuous dynamical systems

#### **1 Introduction**

The design and synthesis of controllers for dynamical systems is a fundamental problem in the field of control. In recent years, with the boom of deep learning, there has been considerable research activities in the use of deep neural networks (DNNs) for control of cyber-physical systems such as unmanned aerial vehicles, self-driving cars, etc. [33]. For these safety-critical systems, one of the most important and challenging problems is safe controller synthesis, that is, to synthesize a controller guaranteeing that the system's trajectory will never intersect with an undesired region.

A number of techniques included under the umbrella of Deep Reinforcement Learning (DRL) have been used to effectively learn controllers from user-defined reward functions encoding desired system behavior [17,36]. A majority of these works lack formal reasoning about the safety of such DNN-controlled dynamical systems from such learning process. To guarantee the safety property of synthesized DNN controllers, considerable works focus on the safety verification of DNN-controlled closed-loop systems, which is a really hard problem because it is tangled with highly nonlinear DNN expressions. The main research on this topic is through reachable set estimation of DNN-controlled systems, which can only deal with time bounded safety property [11,12,18,19,37]. On the other hand, other than formally verifying synthesized DNN controllers, more recent works have been proposed to learn DNN controllers for dynamical systems with safety guarantees [8,39,40]. For example, a verification-in-the-loop DNN controller training algorithm is presented in [8], which integrates RL framework with user-provided control barrier functions (CBFs) for reward function encoding, combined with SMT based formal CBF checking; a correctness-by-design method is proposed in [39] that first learns DNN controllers and barrier certificates simultaneously using supervised learning, and then performs posterior formal verification of barrier certificates via SMT solvers.

In this paper, we propose a safe reinforcement learning approach to synthesize DNN controller for nonlinear systems subject to safety constraints via barrier certificate generation. The proposed approach employs an iterative scheme where a *learner* and a *verifier* interact to synthesize safe DNN controllers. Firstly, the *learner* applies DRL method to train a DNN controller by encoding the safety requirement (and the barrier certificate requirement, if applicable) into reward function. For the learned controller, the *verifier* computes a Maximal Safe Input Region (MSIR) and the corresponding barrier certificate. Once the MSIR is a superset of the prescribed initial set Θ, it is easy to see that the safety of the closed-loop system under the learned controller with Θ is verified. Otherwise, the computed barrier certificate needs to be adjusted and fed to guide the *learner* to retrain a new controller. The above inductive loop repeats until an MSIR enclosing Θ is computed.

Compared with [8], a user-provided barrier certificate is adopted for reward function encoding and the barrier certificate is fixed through the learning process, whereas in this paper the controllers and the barrier certificates are synthesized simultaneously and yielded in a larger state space, which increases the diversity and flexibility of barrier certificates. Meanwhile, the barrier certificates in our approach are computed by numerical optimization method, which is more efficient than the SMT based method in [8]. Compared with [39], our method is based on RL framework and thus has better data sampling efficiency than the meshing-based data set generation in [39] for supervised learning. Besides, our method is iterative so that can utilize intermediate learned results to guide learning in the next iteration, rather than restarting from scratch as in [39] when a learned barrier certificate failed formal checking. Thanks to these advantages, our method has really good performance in efficiency and scalability even for problems with dimension up to 12.

The main contributions of this paper are summarized as follows:


The paper is organized as follows. Section 2 gives a brief introduction to the safe controller synthesis problem. Section 3 describes an iterative scheme of safe reinforcement learning for safe DNN controller synthesis. In Sect. 4, we provide an overall algorithm with a detailed example attached to depict how the algorithm works. In Sect. 5, we present an experimental evaluation of our algorithm over a set of benchmark examples. We compare with related works in Sect. 6 before concluding in Sect. 7.

#### **2 Preliminaries**

**Notations.** Let R and N be the field of real number and natural number, respectively. R[**x**] denotes the ring of polynomials with coefficients in R over variables **x** = [x1, x2,...,xn] <sup>T</sup> , and R[**x**] <sup>n</sup> denotes the n-dimensional polynomial ring vector. Let <sup>R</sup>[**x**]<sup>d</sup> <sup>⊂</sup> <sup>R</sup>[**x**] be the vector space of polynomials of degree at most <sup>d</sup>. Let N<sup>n</sup> <sup>d</sup> := {<sup>α</sup> <sup>∈</sup> <sup>N</sup><sup>n</sup> : - <sup>i</sup> <sup>α</sup><sup>i</sup> <sup>≤</sup> <sup>d</sup>}. Denote by <sup>Σ</sup>[**x**] <sup>⊂</sup> <sup>R</sup>[**x**] (resp. <sup>Σ</sup>[**x**]<sup>d</sup> <sup>⊂</sup> <sup>R</sup>[**x**]2<sup>d</sup>) the space of sums of squares (SOS) polynomials.

Consider a continuous dynamical system of the form

$$
\dot{\mathbf{x}} = \mathbf{f}(\mathbf{x}),
\tag{1}
$$

where **<sup>x</sup>** = (x1,...,xn)<sup>T</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup> and **<sup>f</sup>** = (f1,...,fn)<sup>T</sup> <sup>∈</sup> <sup>R</sup>[**x**] <sup>n</sup> is the vector field defined on the state space <sup>D</sup> <sup>⊂</sup> <sup>R</sup>n. We assume that **<sup>f</sup>** satisfies the local Lipschitz condition, so that (1) has a unique solution **x**(t, **x**0) in D for every initial state **x**<sup>0</sup> ∈ D at time t = 0.

In many contexts, a dynamical system is equipped with a domain Ψ ⊂ D and an initial set <sup>Θ</sup> <sup>⊂</sup> <sup>Ψ</sup>, represented as a triple <sup>C</sup> . = (**f**, Θ, Ψ). Given a prespecified unsafe region X<sup>u</sup> ⊂ D, we say that the system C is *safe* if all system trajectories starting from Θ can not evolve into any state specified by Xu, which has been widely investigated in safety critical applications.

**Definition 1 (Safety).** *For a constrained continuous dynamical system (CCDS)* C = (**f**,Ψ,Θ) *and a given unsafe region* Xu*, the system is safe if for all* **x**<sup>0</sup> ∈ Θ*, there does not exist* t<sup>1</sup> > 0 *such that*

$$
\forall t \in [0, t\_1]. \mathbf{x}(t, \mathbf{x}\_0) \in \Psi \quad \text{and } \mathbf{x}(t\_1, \mathbf{x}\_0) \in X\_u,
$$

*that is, the system's trajectory never reaches* X<sup>u</sup> *from* Θ *as long as it remains in* Ψ*.*

*Remark 1.* If the trajectory **x**(t, **x**0) first leaves Ψ and then enters Ψ again, then by Definition 1, the part of the trajectory from the first exit point is excluded from our concern and is not relevant to the safety of the considered CCDS.

In this paper, we consider a *controlled CCDS* C = (**f**,Ψ,Θ) with continuous dynamics defined by

$$\begin{cases} \dot{\mathbf{x}} = \mathbf{f}(\mathbf{x}, \mathbf{u}) \\ \mathbf{u} = \mathbf{k}(\mathbf{x}), \end{cases} \tag{2}$$

where **<sup>x</sup>** <sup>∈</sup> <sup>Ψ</sup> <sup>⊆</sup> <sup>R</sup><sup>n</sup> are the system states, **<sup>u</sup>** <sup>∈</sup> <sup>U</sup> <sup>⊆</sup> <sup>R</sup><sup>m</sup> are the control inputs, and **<sup>f</sup>** : <sup>Ψ</sup> <sup>×</sup> <sup>U</sup> <sup>→</sup> <sup>R</sup><sup>n</sup> and **<sup>k</sup>** : <sup>Ψ</sup> <sup>→</sup> <sup>U</sup> are the locally Lipschitz continuous vector field and feedback controller function, respectively. The problem we considered in this paper is defined as follows.

**Definition 2 (Safe Controller Synthesis).** *For a controlled CCDS* C = (**f**,Ψ, Θ) *with* **f** *defined by (2) and a given unsafe region* Xu*, design a locally Lipschitz continuous feedback control law* **k** *such that the closed-loop system* C *with* **f** = **f**(**x**, **k**(**x**)) *is safe as per Definition 1.*

The concept of *barrier certificates* plays an important role in safety verification of continuous systems. The essential idea is to use the zero level set of a barrier certificate B(**x**) as a barrier to separate all the reachable states from the unsafe region. The following theorem states the conditions that must be satisfied by a barrier certificate.

**Theorem 1** *[26]***.** *Given a continuous system* C = (**f**,Ψ,Θ)*, and the unsafe region* <sup>X</sup>u*. Suppose there exists a real-valued function* <sup>B</sup> : <sup>Ψ</sup> <sup>→</sup> <sup>R</sup> *satisfying the following conditions:*

**(i)** B(**x**) ≥ 0 ∀**x** ∈ Θ*,*

**(ii)** B(**x**) < 0 ∀**x** ∈ Xu*,* **(iii)** B(**x**)=0 ⇒ LfB(**x**) > 0 ∀**x** ∈ Ψ*,*

*where* LfB(**x**) *denotes the Lie-derivative of* B(**x**) *along the vector field* **f**(**x**)*, i.e.,* LfB(**x**) = n i=1 ∂B ∂x*<sup>i</sup>* · fi(**x**)*, then* B(**x**) *is a barrier certificate, and the safety of system* C *is guaranteed.*

**Corollary 1.** *For a controlled CCDS* C = (**f**,Ψ,Θ) *with* **f** *defined by (2), a feedback control law* u = **k**(**x**) *can be used to ensure the safety control of* C*, if there exists a barrier certificate for the closed-loop system under the control law* **k**(**x**)*.*

Throughout this paper, we assume that the initial set Θ, the domain Ψ and the unsafe set X<sup>u</sup> are compact semi-algebraic sets, defined by polynomial equations and inequalities. Concretely, the semi-algebraic sets Θ, Ψ and X<sup>u</sup> are represented as follows:

$$\begin{cases} \Theta:=\{\mathbf{x}\in\mathbb{R}^{n}\mid g\_{i}(\mathbf{x})\ge 0, i=1,\ldots,m\_{1}\},\\\Psi:=\{\mathbf{x}\in\mathbb{R}^{n}\mid h\_{j}(\mathbf{x})\ge 0, j=1,\ldots,m\_{2}\},\\X\_{u}:=\{\mathbf{x}\in\mathbb{R}^{n}\mid q\_{k}(\mathbf{x})\ge 0, k=1,\ldots,m\_{3}\},\end{cases}$$

for some polynomials <sup>g</sup>i, h<sup>j</sup> , q<sup>k</sup> <sup>∈</sup> <sup>R</sup>[**x**].

#### **3 Synthesis of Safe Controller via Learning and Verification**

In this section, we introduce an iterative framework for synthesizing a deep neural network (DNN) controller for a CCDS subject to safety constraints. As shown in Fig. 1, the procedure is structured as an inductive loop between a *learner* and a *verifier*. The *learner* trains a DNN controller using reinforcement learning. The trained DNN controller is passed to the *verifier*, which checks the safety of the closed-loop system under the trained controller via barrier certificate generation.

Observing Fig. 1, we first apply the reinforcement learning method to train a neural network controller u = k(**x**) in terms of the target of the safety satisfiability, and then try to yield a barrier certificate B(**x**) based on the bilinear matrix inequalities (BMI) solving, to guarantee the safety of the closed-loop system with the controller k(**x**).

However, for the system with the controller k(**x**), such barrier certificate B(**x**) may not exist. The reasons are twofold: (i) the controller k(**x**) is trained through the trajectories starting from finite points in the initial set Θ; (ii) the existence of the barrier certificate is just a sufficient condition of the safety of the given system.

In this situation, for the learned controller k(**x**), one may compute a Maximal Safe Input Region (MSIR) Θ<sup>γ</sup> and the corresponding barrier certificate B(**x**), which can guarantee the safety of the continuous system with respect to the initial set Θγ. Once Θ<sup>γ</sup> is a superset of the prescribed initial set Θ, i.e., Θ ⊆ Θγ,

**Fig. 1.** The framework of safe neural network controller synthesis.

it is easy to see that the safety of the system with Θ is verified. Otherwise, we need adjust the barrier certificate B(**x**) and the controller k(**x**) sequentially. This operation is able to build an iterative framework, wherein each iteration proceeds in two stages:


#### **3.1 Training of Safe Controller**

In the following, we focus on the *learner* component of Fig. 1 and show how to train a safe controller using deep deterministic policy gradient (DDPG) [23], which is a popular reinforcement learning approach suited for continuous control applications. The DDPG combines the value-based and policy-based method, and is made up of two parts: actor and critic. The critic uses the off-policy data to learn the action-value function, which evaluates how good the action k taken is in the given state **x**. The actor can learn the continuous action policy by using the action-value function. In practice, it is difficult to obtain the exact action-value function and policy function. Thus, two deep neural networks are introduced to solve this problem, i.e. the critic network <sup>Q</sup>(**x**, **<sup>u</sup>**|βQ) and actor network <sup>k</sup>(**x**|βk) with weights <sup>β</sup><sup>Q</sup> and <sup>β</sup>k, respectively.

The reward function should be appropriately designed to achieve the goal of safety controller synthesis via reinforcement learning. For safe controller synthesis, the task is to synthesize a DNN controller such that all the trajectories of the closed-loop system starting from Θ can not evolve into the unsafe region Xu. Thus, the reward function is preliminarily defined as

$$
\hat{r}\_t = \beta\_1 \cdot \text{dist}(X\_u, \mathbf{x}\_t)
$$

where β<sup>1</sup> > 0 is the scale factor, and dist(Xu, **x**t) denotes the distance between the state **x**<sup>t</sup> and the unsafe region Xu. In addition, according to the third condition of Theorem 1, once the trajectory hit the zero level set of barrier certificate it must satisfy L<sup>f</sup>B(**x**t) > 0; otherwise, the system behavior should be penalized. For this purpose, the reward function is updated as

$$r\_t = \begin{cases} \hat{r}\_t - \min(\beta\_2 |\mathcal{L}\_f B(\mathbf{x}\_t)|, \Delta r\_{\min}), & |B(\mathbf{x}\_t)| < \delta \text{ and } \mathcal{L}\_f B(\mathbf{x}\_t) \le 0\\ \hat{r}\_t, & \text{otherwise} \end{cases} \tag{3}$$

where L<sup>f</sup>B(**x**t) = n i=1 ∂B(**x***t*) ∂x*<sup>i</sup>* fi(**x**t, u), β<sup>2</sup> > 0 is the scale factor, δ is a small positive value characterizing the zero-level set of B, and Δrmin > 0 is the threshold avoiding too large fluctuations of reward value. In this work, we set β<sup>1</sup> = 1.0, β<sup>2</sup> = 1.0, δ = 0.1, Δrmin denotes the size of Ψ. Since 0 ≤ rˆ<sup>t</sup> ≤ Δrmin, the setting r<sup>t</sup> (3) can be kept within a certain range, making the convergence effect better.

#### **Algorithm 1.** Barrier Certificate Guided Reinforcement Learning

**Input:** CCDS <sup>C</sup>; unsafe region <sup>X</sup>u; barrier certificate <sup>B</sup>(**x**)

**Output:** DNN Controller k

```
1: Initialize critic Q and actor k, corresponding target networks Q-
                                                                    = Q and k-
                                                                                 = k
```

```
2: Initialize barrier certificate B(x) = ⊥ and replay buffer R = ∅
```

To synthesize the safety controller using reinforcement learning, a dataset of sampled trajectories is needed. To sample trajectories, we first generate a set of initial states from <sup>Θ</sup>. Let **<sup>l</sup>**, **<sup>u</sup>** <sup>∈</sup> <sup>R</sup><sup>n</sup> be the vectors of the lower and upper bounds of Θ, i.e., Θ ⊆ [**l**, **u**]. We first sample from each dimension of [**l**, **u**] equidistantly with a fixed mesh size. For a sampled initial state **x**0, its trajectory is generated, and the transition tuples (**x**t, **x**t+1, **u**t, rt) are collected to form a replay buffer to update the action and critic networks. Concretely, the action network receives a state **x**<sup>t</sup> in time step t as input, and directly outputs a continuous action **<sup>u</sup>**<sup>t</sup> <sup>=</sup> <sup>k</sup>(**x**t|βk). The critic network takes the state **<sup>x</sup>**<sup>t</sup> and the action **<sup>u</sup>**<sup>t</sup> as input, and outputs a scalar Q-value <sup>Q</sup>(**x**t, **<sup>u</sup>**t|βQ). For every <sup>m</sup> simulated time steps, we sample a batch of tuples from the buffer as the training data to update the actor and critic networks, until a certain prescribed termination condition is met for the learning process. The resulting actor network is the synthesized controller. All training related parameters, such as smoothing factor, are set as default. Our DDPG implementation is based on an open-source package DDPG [23]. The algorithm is outlined in Algorithm 1.

*Remark 2.* The barrier certificate is initialized to be ⊥, which means that the *learner* initially trains a DNN controller via standard reinforcement learning, without the aid of barrier certificates.

#### **3.2 Safety Verification with Barrier Certificates**

In the following, we focus on the *verifier* component of the proposed safe DNN synthesis framework, as described in Fig. 2, and show how to verify the safety of the closed-loop system under the DNN controller yielded from the *learner*.

**Fig. 2.** The framework of the *verifier*.

Shown in Fig. 2, the *learner* produces a DNN controller ki(**x**). In order to make the problem of generating barrier certificates amenable to polynomial optimization problem, the *verifier* first employs Bernstein polynomial approximation to abstract the learned DNN controller as a polynomial one ki(**x**), with the associated abstract error modeled as a bounded parameter, that is, **<sup>u</sup>** <sup>=</sup> ki(**x**) + .

By doing it, the safety of the closed-loop system under the DNN controller can be guaranteed via the existence of barrier certificates for the closed-loop system under the abstract controller. The *verifier* then performs bilinear matrix inequalities (BMI) solving technique, to obtain a maximal safe initial region (MSIR) Θ<sup>i</sup> and the corresponding barrier certificate Bi(**x**). Once the computed MSIR Θ<sup>i</sup> contains the given initial set Θ, then the safety of the closed-loop system under the DNN controller **u** = ki(**x**) is verified. Otherwise, the *verifier* slightly adjusts the barrier certificate Bi(**x**), based on quadratic programming solving, to gain an updated one <sup>B</sup>i(**x**), which can separate the unsafe region from the initial set. Then, the refined BC is fed to guide the *learner* to retrain a new DNN controller.

**Polynomial Abstraction of DNN Controllers.** In the following, we consider the DNN controller with a single output, and for multiple-output cases, an extension is to approximate each output respectively. Formally, for a DNN controller <sup>k</sup>(**x**), we seek to compute an approximate polynomial <sup>p</sup>(**x**) <sup>∈</sup> <sup>R</sup>[**x**] with a verified bound <sup>μ</sup> <sup>∈</sup> <sup>R</sup>+, such that

$$|k(\mathbf{x}) - p(\mathbf{x})| < \mu, \forall \mathbf{x} \in \Psi,$$

and the bound μ is as small as possible.

Weierstrass approximation theorem [7] asserts that a continuous function on a closed and bounded interval can be uniformly approximated on the interval by polynomials to any degree of accuracy. In this paper, we will compute the approximate polynomial based on the theory of Bernstein polynomials [9]. Let **<sup>d</sup>** = (d1, ··· , dn) <sup>∈</sup> <sup>N</sup><sup>n</sup> and <sup>f</sup> : [0, 1]<sup>n</sup> <sup>→</sup> <sup>R</sup>. The polynomial

$$B\_{f, \mathbf{d}}(\mathbf{x}) = \sum\_{\substack{0 \le c\_j \le d\_j \\ j \in \{1, \dots, n\}}} f(\frac{c\_1}{d\_1}, \dots, \frac{c\_n}{d\_n}) \prod\_{j=1}^n \binom{d\_j}{c\_j} x\_j^{c\_j} (1 - x\_j)^{d\_j - c\_j}$$

is called the multivariate Bernstein polynomial of f. Theoretically, the Bernstein polynomial Bf,**d**(**x**) converges uniformly to f for d1, ··· , d<sup>n</sup> → ∞. In practice, the estimation of the approximation error bound is needed. As stated in [9], assume f is a Lipschitz continuous function over I : [0, 1]<sup>n</sup> with a Lipschitz constant L, then we have

$$\|\|B\_{f,\mathbf{d}}(\mathbf{x}) - f(\mathbf{x})\|\| \le \frac{L}{2} \left(\sum\_{j=1}^{n} (\frac{1}{d\_j})\right)^{\frac{1}{2}}, \forall \mathbf{x} \in I.$$

Now, for the DNN controller k(**x**) over a domain Ψ, we can apply the above method to obtain a Bernstein polynomial with a valid approximate error bound as its abstraction. Concretely, we first construct an interval enclosure for Ψ, and apply a linear transformation to map the interval enclosure onto the unit box I, then utilize Bernstein polynomial approximation to obtain an abstract polynomial controller k(**x**)+ with <sup>∈</sup> [−μ, μ], where k(**x**) is a Bernstein polynomial of k(**x**) and μ is its valid approximate error bound. Note that the fully-connected neural networks with sigmoid and tanh activation functions are Lipschitz continuous, and the estimation of Lipschitz constants for deep neural networks has been studied in [14,31,34].

**Maximal Safe Initial Region Computation.** Since k(**x**)+ enclosures <sup>k</sup>(**x**), the safety of the closed-loop system under the DNN controller k(**x**) can be guaranteed via the existence of barrier certificates for the closed-loop system under the abstract controller k(**x**) + . From this observation, we try to compute an MSIR Θ<sup>γ</sup> and its corresponding barrier certificate Bγ(**x**), which can guarantee the safety of the closed-loop system under the abstract controller k(**x**) + with respect to Θγ.

Firstly, we consider how to predefine a suitable initial state set template Θ<sup>γ</sup> from the given initial set Θ. In what follows, we provide some parametric initial state sets for two typical representations: Boxes and Euclidean ellipsoids (balls).

**Box Template.** Suppose that the box initial set Θ is represented as

$$\Theta = \{ \mathbf{x} \in \mathbb{R}^n | |x\_i - c\_i| \le b\_i \},$$

where **<sup>x</sup>**<sup>c</sup> = (c1, ··· , cn)<sup>T</sup> is the center of the box, and <sup>b</sup><sup>i</sup> <sup>∈</sup> <sup>R</sup><sup>&</sup>gt;0. Then, the parametric initial set can be expressed as

$$\Theta\_{\gamma} = \{ \mathbf{x} \in \mathbb{R}^n || D^{-1}(\mathbf{x} - \mathbf{x}\_c) ||\_{\infty} \le \gamma \},$$

where D = diag(b1, ··· , bn) is a diagonal matrix.

**Ellipsoid Template.** Suppose that the ellipsoid initial set Θ is expressed as a common representation:

$$\Theta = \{ \mathbf{x} \in \mathbb{R}^n | \mathbf{x} = \mathbf{x}\_c + A\mathbf{v}, \ \|\mathbf{v}\|\_2 \le 1 \},$$

where **x**<sup>c</sup> is the center of the ellipsoid, and the matrix A is nonsingular. Then the parametric initial set can be expressed as

$$\begin{aligned} \Theta\_{\gamma} &= \{ \mathbf{x} \in \mathbb{R}^{n} | \mathbf{x} = \mathbf{x}\_{0} + \gamma A | \mathbf{v}, \|\mathbf{v}\|\_{2} \le 1 \} \\ &= \{ \mathbf{x} \in \mathbb{R}^{n} | \|A^{-1} \left(\mathbf{x} - \mathbf{x}\_{0}\right)\|\_{2} \le \gamma \} .\end{aligned}$$

Without loss of generality, we can select the template of the parametric initial sets by taking the form <sup>Θ</sup><sup>γ</sup> := {**<sup>x</sup>** <sup>∈</sup> <sup>R</sup><sup>n</sup>|g(**x**) <sup>≤</sup> γ, i = 1,...,m1} with <sup>γ</sup> <sup>∈</sup> <sup>R</sup><sup>&</sup>gt;0, where g(**x**) is the polynomial used to defined the prescribed initial set Θ.

In order to enlarge the safe initial region by choice of Θγ, we maximize γ while imposing the constraints for the existence of barrier certificates. Assume that the barrier certificate B(**x**) is a polynomial of degree at most d, whose coefficients form a vector space of dimension s(d) = <sup>n</sup>+<sup>d</sup> d with the canonical basis (**x**<sup>α</sup>) of monomials. Suppose the coefficients are unknown, and denote by **<sup>b</sup>** = (bα) <sup>∈</sup> <sup>R</sup><sup>s</sup>(d) the coefficient vector of <sup>B</sup>(**x**), and write

$$B(\mathbf{x}, \mathbf{b}) = \sum\_{\alpha \in \mathbb{N}\_d^n} b\_{\alpha} \mathbf{x}^{\alpha} = \sum\_{\alpha \in \mathbb{N}\_d^n} b\_{\alpha} x\_1^{\alpha\_1} x\_2^{\alpha\_2} \cdots x\_n^{\alpha\_n},$$

in the canonical basis. Thus, the problem of computing an MSIR Θ<sup>γ</sup> of the closed-loop system under the abstract controller k(**x**) + can be represented as an optimization problem

$$\begin{array}{ll}\gamma\_{opt}^{\*} = \max\_{\mathbf{b}, \gamma} \gamma\\ \text{s.t.} \quad B(\mathbf{x}, \mathbf{b}) \ge 0, \ \forall \mathbf{x} \in \Theta\_{\gamma},\\ \mathcal{L}\_{\mathbf{f}} B(\mathbf{x}, \mathbf{b}) > 0, \ \forall \mathbf{x} \in \Psi \text{ and } B(\mathbf{x}, \mathbf{b}) = 0, \\ B(\mathbf{x}, \mathbf{b}) < 0, \ \forall \mathbf{x} \in X\_{u}. \end{array} \tag{4}$$

Then, Sum-of-Squares (SOS) relaxation technique is applied to encode the optimization problem (4) as a SOS program. In fact, given a basic semi-algebraic set K defined by:

$$\mathbb{K} = \{ \mathbf{x} \in \mathbb{R}^n \mid p\_1(\mathbf{x}) \ge 0, \dots, p\_s(\mathbf{x}) \ge 0 \},$$

where <sup>p</sup><sup>j</sup> <sup>∈</sup> <sup>R</sup>[**x**], <sup>1</sup> <sup>≤</sup> <sup>j</sup> <sup>≤</sup> <sup>s</sup>, a sufficient condition for the nonnegativity of the given polynomial f(**x**) on the semi-algebraic set K is provided as

$$f(\mathbf{x}) = \sigma\_0(\mathbf{x}) + \sum\_{i=1}^{s} \sigma\_i(\mathbf{x}) p\_i(\mathbf{x}),\tag{5}$$

where σ<sup>i</sup> ∈ Σ[**x**]d, 1 ≤ i ≤ s. Thus, the representation (5) ensures that the polynomial f(**x**) is nonnegative on the given semi-algebraic set K.

Observing (4), the polynomial L**<sup>f</sup>**B(**x**, **b**) is involved with the uncertain variable in the range [−μ, μ], which can be written as the constraint, <sup>h</sup>ˆ( ) <sup>≥</sup> <sup>0</sup> with

$$
\hat{h}(\epsilon) := (\epsilon + \mu)(\mu - \epsilon).
$$

Thus, the problem (4) can be transformed into the following optimization problem

$$\begin{array}{lcl}\gamma^{\*} = \max\_{\mathbf{b}, \gamma, \gamma} & \gamma\\ \text{s.t.} & B(\mathbf{x}, \mathbf{b}) - \sigma(\mathbf{x})(\gamma - g(\mathbf{x})) \in \Sigma[\mathbf{x}],\\ & \mathcal{L}\_{\mathbf{f}} B(\mathbf{x}, \mathbf{b}) - \lambda(\mathbf{x}) B(\mathbf{x}, \mathbf{b}) - \sum\_{j} \phi\_{j}(\mathbf{x}) h\_{j}(\mathbf{x}) - \nu(\mathbf{x}, \varepsilon) \hat{h}(\varepsilon) - \epsilon\_{1} \in \Sigma[\mathbf{x}],\\ & -B(\mathbf{x}, \mathbf{b}) - \epsilon\_{2} - \sum\_{j} \kappa\_{j}(\mathbf{x}) q\_{j}(\mathbf{x}) \in \Sigma[\mathbf{x}],\end{array} \tag{6}$$

where <sup>1</sup>, <sup>2</sup> > 0, the entries of σ(**x**), φ<sup>j</sup> (**x**) κ(**x**) ∈ Σ[**x**], and ν(**x**, ε) ∈ Σ[**x**, ε], and <sup>λ</sup>(**x**) <sup>∈</sup> <sup>R</sup>[**x**]. Note that <sup>1</sup>, <sup>2</sup> are needed to ensure positivity of polynomials as required in the second and third constraints in (4). Clearly, the feasibility of the constraints in (6) is sufficient to imply the feasibility of the constraints in (4), thus the optimum of (6) is a lower bound of the optimum of (4), i.e., γ<sup>∗</sup> ≤ γ<sup>∗</sup> opt.

The SOS program (6) is bilinear due to the product of the unknown coefficients of (B(**x**, **b**), λ(**x**)) and (σ(**x**), γ), yielding a non-convex bilinear matrix inequalities (BMI) problem. Fortunately, a Matlab package PENBMI solver [22], which combines the (exterior) penalty and (interior) barrier method with the augmented Lagrangian method, can be applied directly to obtain a numerical solution of the problem (6). The solution γ∗, **b**<sup>∗</sup> to problem (6) yields an MSIR Θ<sup>γ</sup><sup>∗</sup> and its corresponding barrier certificate B(**x**, **b**∗). It means that the closedloop system under the abstract controller k(**x**) + is safe, with respect to <sup>Θ</sup><sup>γ</sup><sup>∗</sup> . Moreover, if the given initial set Θ is a subset of Θ<sup>γ</sup><sup>∗</sup> , then the safety of the closed-loop system under the DNN controller k(**x**) with respect to Θ is verified. Otherwise, B(**x**, **b**∗) will be further refined via quadratic programming method. *Remark 3.* The gap between the optima of problems (4) and (6) decreases as increasing of degrees for the multiplier polynomials. The degree bound for the multiplier polynomials is exponential with the number of variables **x** and the degrees of the polynomials appearing in the semi-algebraic sets. In practice, we set up a truncated SOS programming for (6) by fixing a *priori* (much smaller) degree bound of all the unknown multiplier polynomials, to avoid high computational complexity.

**Barrier Certificate Refinement.** Consider the case in which the initial set Θ is not a subset of the MSIR Θ<sup>γ</sup><sup>∗</sup> . In this case, the barrier certificate B(**x**, **b**∗) can succeed to separate the unsafe region X<sup>u</sup> from Θ<sup>γ</sup><sup>∗</sup> , but it may fail to separate from Θ. In other words, B(**x**, **b**∗) can not be regarded as a truly candidate barrier certificate with respect to Θ and Xu. Therefore, we will utilize the information of B(**x**, **b**∗) to refine it, in order to obtain a new candidate barrier certificate that can separate Θ from Xu. Consider the change in B(**x**, **b**∗) is expected as small as possible, the step of the barrier certificate refinement can be represented as

$$\begin{array}{l}\min \|\hat{\mathbf{b}} - \mathbf{b}^\*\|\_2^2\\ \text{s.t. } B(\mathbf{x}, \hat{\mathbf{b}}) \ge 0 \,\forall \mathbf{x} \in \Theta, \\ B(\mathbf{x}, \hat{\mathbf{b}}) < 0 \,\forall \mathbf{x} \in X\_{u}. \end{array} \tag{7}$$

By investigating (7), the constraints are the ones involving universal quantifiers. To avoid eliminating universal quantifiers directly, here we provide a relaxation technique to deal with (7), which is based on selecting sampling points. For Θ and Xu, let us first construct rectangular meshes in Θ and X<sup>u</sup> respectively, with a mesh spacing <sup>r</sup> <sup>∈</sup> <sup>R</sup><sup>+</sup> (say <sup>r</sup> = 0.05). The resulting mesh point sets are denoted as Ω<sup>Θ</sup> and Ω<sup>X</sup>*<sup>u</sup>* , respectively.

It is known that for a continuously differentiable function φ(**x**) over a compact domain D, the mean value theorem yields that

$$|\phi(\mathbf{x} + \Delta \mathbf{x}) - \phi(\mathbf{x})| \le n\eta \|\Delta \mathbf{x}\|\_{\infty},$$

where **x**, **x** + Δ ∈ Ω are chosen randomly, and η = sup**<sup>x</sup>**∈<sup>D</sup> ∇φ(**x**)<sup>∞</sup>. Based on the above observation, the following implications are satisfied:

$$\begin{cases} B(\mathbf{x}\_j, \hat{\mathbf{b}}) - \delta\_1 \ge 0, \ \forall \mathbf{x}\_j \in \Omega\_\Theta \implies B(\mathbf{x}, \hat{\mathbf{b}}) \ge 0 \,\,\forall \mathbf{x} \in \Theta, \\ B(\mathbf{x}\_j, \hat{\mathbf{b}}) + \delta\_2 < 0, \ \forall \mathbf{x}\_j \in \Omega\_{X\_u} \implies B(\mathbf{x}, \hat{\mathbf{b}}) < 0 \,\,\forall \mathbf{x} \in X\_u. \end{cases} \}$$

where <sup>δ</sup><sup>i</sup> <sup>=</sup> nηi<sup>r</sup> <sup>∈</sup> <sup>R</sup><sup>&</sup>gt;0, i = 1, 2 with <sup>η</sup><sup>1</sup> = sup**<sup>x</sup>**∈<sup>Θ</sup> <sup>∇</sup>B(**x**, **<sup>b</sup>**∗)<sup>∞</sup> and <sup>η</sup><sup>2</sup> <sup>=</sup> sup**<sup>x</sup>**∈X*<sup>u</sup>* ∇B(**x**, **b**∗)<sup>∞</sup>.

By using the above relaxation technique based on sampling points, (7) can be relaxed as the following problem

$$\begin{array}{ll}\min \|\hat{\mathbf{b}} - \mathbf{b}^\*\|\_2^2 \\ \text{s.t.} \quad B(\mathbf{x}\_j, \hat{\mathbf{b}}) - \delta \ge 0, \ \forall \mathbf{x}\_j \in \Omega\_{\Theta}, \\\ B(\mathbf{x}\_j, \hat{\mathbf{b}}) + \delta < 0, \ \forall \mathbf{x}\_j \in \Omega\_{X\_u}, \end{array} \tag{8}$$

which is a typical quadratic programming problem and can be solved by stateof-the-art solvers with great efficiency.

Now, the refined <sup>B</sup>(**x**) = <sup>B</sup>(**x**, **<sup>b</sup>**ˆ) can separate <sup>Θ</sup> from <sup>X</sup>u, but may still not satisfy the Lie derivative condition for barrier certificates. According to Theorem 1, <sup>B</sup>(**x**) is not a truly barrier certificate for the closed-loop system under the abstract controller k(**x**) + with respect to <sup>Θ</sup> and <sup>X</sup>u. Next, the refined <sup>B</sup>(**x**) will be further fed to guide the *learner* to retrain a new controller. To do it, we first consider the additional constraint for the Lie derivative of <sup>B</sup>(**x**), and apply barrier certificate guided reinforcement learning to compute a new DNN controller.

#### **4 Algorithm**

In Sect. 3, we have elaborated on the iteration-based safe controller synthesis method that iteratively co-synthesizes a DNN controller within the RL framework and a polynomial barrier certificate via BMI solving. Briefly, we describe the main implementation steps of our approach in the following Algorithm 2.


```
Input: The CCDS C; unsafe region Xu; maximum number of iterations maxIter
Output: Safe DNN Controller k
1: iter ← 0
2: B ← ⊥
3: while iter < maxIter do
4: k ← Learning(f, Θ, Xu, B)
5: -
      k, μ ← PolyInclusion(k)
6: Θ∗
        γ, B(x, b∗) ← MaxSafeSet(f, -
                                  k, μ, Θ, Xu)
7: if Θ ⊆ Θ∗
              γ then
8: return k
9: end if
10: B ← RefineBarrier(B(x, b∗), Θ, Xu)
11: end while
```
Algorithm 2 shows the iteration scheme of our safe controller synthesis, which guides the experiment implementation. The procedure takes as inputs a CCDS C, an unsafe region Xu, a maximum number of iterations maxIter, and returns a safe DNN controller of a given architecture. In a pass of the iteration, the implementation process has four steps as follows.

(i) Apply the RL method to train a DNN controller. The *learner* introduced in Sect. 3.1 is implemented by Line 4 in Algorithm 2, and the barrier certificate is initialized to be ⊥, which means that the *learner* trains a DNN controller via classical reinforcement learning, without the aid of barrier certificates in the initial pass;


This inductive loop repeats until an MSIR enclosing Θ<sup>γ</sup> and its corresponding barrier certificate are computed or until a timeout is reached.

*Remark 4.* Our procedure is sound, i.e. a valid output from the *verifier* is provably correct. However, we cannot claim any completeness, since our procedure might in general not terminate because the existence of the barrier certificate is just a sufficient condition of the safety of the system, and such a barrier certificate may not exist indeed. Once the procedure fails, we may improve the relaxation precision and then increase the possibility to find the barrier certificate by increasing the degree bound for the multiplier polynomials in the SOS program (6).

Furthermore, an example is used to depict how our safe controller synthesis algorithm works.

*Example 1.* Consider the Van der Pol system

$$
\begin{bmatrix}
\dot{x\_1} \\
\dot{x\_2}
\end{bmatrix} = \begin{bmatrix}
x\_2 \\
\end{bmatrix}
$$

with the domain <sup>Ψ</sup> <sup>=</sup> {**<sup>x</sup>** <sup>∈</sup> <sup>R</sup><sup>2</sup> | −<sup>3</sup> <sup>≤</sup> <sup>x</sup>1, x<sup>2</sup> <sup>≤</sup> <sup>3</sup>}. Our goal is to design a control law k such that all trajectories of the system under u = k(x1, x2) starting from the initial set

$$\Theta = \{ \mathbf{x} \in \mathbb{R}^2 \mid (x\_1 - 1.5)^2 + x\_2^2 \le 1.1^2 \}$$

will never enter the unsafe set

$$X\_u = \{ \mathbf{x} \in \mathbb{R}^2 \mid (x\_1 + 1)^2 + (x\_2 + 1)^2 \le 1 \}.$$

We complete our goal by Algorithm 2, and provide the details here. At first, we apply the reinforcement learning method to train the initial neural network controller u = k0(**x**) in terms of the target of safety satisfiability, which is Step (i) and refers to Line 4 in Algorithm 2, and then try to yield the barrier certificate B(**x**). We compute polynomial abstraction of DNN Controller k0(**x**) via Bernstein polynomials which is Step (ii), where

$$\begin{split} \tilde{k}\_0(\mathbf{x}) &= 0.0142x\_1 + 0.0092x\_2 - 0.0205x\_1^2 + 0.0077x\_1x\_2 + 0.0340x\_2^2 \\ &+ 0.0246x\_1^3 + 0.0018x\_1^2x\_2 - 0.0820x\_1x\_2^2 + 0.0435x\_2^3 + \epsilon. \end{split} \tag{9}$$

with ∈ [−0.05, 0.05], which is implemented by Line 5. Thus, the polynomial abstraction technique can yield an abstract polynomial system.

Go on Step (ii) to compute a maximal safety region Θ<sup>γ</sup> and the corresponding barrier certificate B(**x**). In this case, we parameterize the initial set:

$$\Theta\_{\gamma} = \{ \mathbf{x} \in \mathbb{R}^2 \mid (x\_1 - 1.5)^2 + x\_2^2 \le \gamma \}.$$

For the given abstract polynomial system with the parameterized initial set Θγ, our goal is to maximize the radius γ subject to the existence of a barrier certificate. By calling the PENBMI solver [22] we can obtain a barrier certificate B0(**x**) with the maximal safe initial region Θ<sup>0</sup> (Line 6 in our Algorithm 2), i.e.,

$$\begin{aligned} \Theta\_0 &= \{ \mathbf{x} \in \mathbb{R}^2 \, | \, (x\_1 - 1.5)^2 + x\_2^2 \le 0.8132 \}, \\ B\_0(\mathbf{x}) &= 11.716 + 22.8064x\_1 + 21.5368x\_2 - 4.5273x\_1^2 + 13.8084x\_1x\_2 + 3.0453x\_2^2. \end{aligned} \tag{10}$$

Thus, the safety of the system with the controller k0(**x**) with respect to the set Θ<sup>0</sup> is guaranteed. Now the present controller k0(**x**) can not be safe for whole initial set Θ, we continue to update controller and barrier certificate (Line 7–9).

Let k0(**x**) and B0(**x**) be the initial controller and the initial barrier certificate, we perform the iterative framework to synthesize the controller subject to the safety constraint. As shown in Fig. 3(a), the zero level set of B0(**x**) is the blue dashed line. Observing Fig. 3(a), B0(**x**) can succeed to separate the unsafe region X<sup>u</sup> (the red circle) from Θ<sup>0</sup> (the green dashed circle), but not separate from the initial set Θ, which means that B0(**x**) can not be regarded as the truly barrier certificate. Therefore, one may perturb the coefficients of B0(**x**) to obtain Bˆ0(**x**) which can separate Θ and Xu. And this process corresponds to Step (iv) and Line 10 of our Algorithm 2. The perturbed polynomial is represented as

$$\hat{B}\_0(\mathbf{x}) = 10.5590 + 22.9401x\_1 + 18.2448x\_2 - 0.8954x\_1^2 + 14.4971x\_1x\_2 + 1.1060x\_2^2.$$

As shown in Fig. 1(b), the zero level set of the barrier Bˆ0(**x**) (the blue dash) separates X<sup>u</sup> (the red circle) from Θ (the green circle). According to the concept of barrier certificate and Theorem 1, Bˆ0(**x**) is not a truly barrier certificate, since the condition of the Lie derivative of the barrier certificate is not satisfied. Accordingly, by using the Bˆ0(**x**) and the initial controller k0(**x**), we then try to retrain a control law with an additional constraint of the lie derivative for the barrier certificate Bˆ0(**x**). Calling the *learner* module (Line 4), we update a new control law k1(**x**) represented as a two-hidden layer sigmoid-based DNN with 20 neurons per layer by RL approach.

**Fig. 3.** This picture shows the iteration process of barrier certificate updating when we learn the safe controller. The red circles stand for unsafe regions, the blue curves stand for the zero level set of barrier certificates, and the green circles stand for the initial sets and safe initial sets. Subfigure (a) describes the intermediate results of maximal safe initial set Θ<sup>0</sup> (the green dashed circle) with its associate barrier certificate B<sup>0</sup> obtained from Line 6 in Algorithm 2 at the first iteration. We slightly modify the barrier function B<sup>0</sup> to separate Θ and X<sup>u</sup> by Line 10 and obtain Bˆ<sup>0</sup> which is the blue solid curve shown in Subfigure (b). Using Bˆ<sup>0</sup> as a guide, a new controller is learned, from which a barrier certificate B<sup>1</sup> is generated as shown in Subfigure (c). It can be shown that B<sup>1</sup> is the real barrier certificate of the system. (Color figure online)

Repeating the above abstraction technique and solving the BMI problem for finding the maximal safety initial set Θ1, we obtain the barrier certificate B1(**x**) with respect to Θ1, i.e.,

$$\begin{aligned} \Theta\_1 &= \{ \mathbf{x} \in \mathbb{R}^2 \, | \, (x\_1 - 1.5)^2 + x\_2^2 \le 1.2201 \}, \\ B\_1(\mathbf{x}) &= 10.3661 + 22.6569x\_1 + 17.7852x\_2 - 0.9037x\_1^2 + 14.1832x\_1x\_2 + 0.9471x\_2^2. \end{aligned} \tag{11}$$

It is easy to check that the original initial set Θ is now a subset of Θ1, which means that B1(**x**) is a truly barrier certificate.

#### **5 Experiments**

In this section, we first depict an example of three dimension nonlinear continuous system to show our algorithm by synthesizing a safe DNN controller for it, and then present an experimental evaluation of our algorithm over a set of benchmark examples by comparing with a DNN controller learning framework called *nncontroller* in [39].

*Example 2.* Consider the continuous dynamical system

$$
\begin{bmatrix}
\dot{x\_1} \\
\dot{x\_2} \\
\dot{x\_3}
\end{bmatrix} = \begin{bmatrix}
x\_3 + 8x\_2 \\
\end{bmatrix}
$$

with the domain

$$\Psi = \{ \mathbf{x} \in \mathbb{R}^3 \, | \, x\_1^2 + x\_2^2 + x\_3^2 \le 16 \}.$$

Our goal is to design a control law k such that all trajectories of the closed-loop system under u = k(x1, x2, x3) starting from the initial set

$$\Theta = \{ \mathbf{x} \in \mathbb{R}^3 \mid x\_1^2 + x\_2^2 + x\_3^2 \le 1 \}$$

will never enter the unsafe set

$$X\_u = \{ \mathbf{x} \in \mathbb{R}^3 \mid (x\_1 - 2.1)^2 + (x\_2 - 2.1)^2 + (x\_3^2 - 2.1) \le 1.8^2 \}.$$

It suffices to synthesize a control law k and a barrier certificate B(**x**) with the maximal safe initial region Θ<sup>γ</sup> such that Θ ⊆ Θγ. Suppose that the DNN controller k is represented as a five-hidden layer sigmoid activated DNN with 30 neurons per layer. We first call the *learner* to train a DNN controller, and then call the *verifier* to compute the maximal safe initial region Θ<sup>γ</sup> and its corresponding barrier certificate B(**x**). After two iterations, we successfully obtain a safe DNN controller, and the following barrier certificate

$$\begin{aligned} B(\mathbf{x}) &= 220.1981 - 45.7322x\_1 - 40.2831x\_2 - 218.4765x\_3 + 4.9575x\_1^2 \\ &+ 38.7288x\_1x\_2 - 9.8224x\_1x\_3 - 66.8398x\_2^2 + 17.2562x\_2x\_3 + 18.3967x\_3^2. \end{aligned} \tag{12}$$

As shown in Fig. 4, the zero level set of the barrier certificate B(**x**) (the blue surface) separates X<sup>u</sup> (the red ball) from all trajectories starting from Θ (the green ball). Therefore, the safety of the above system is verified.

**Fig. 4.** Phase portrait of the system in Example 2. The zero level set of the barrier certificate B(**x**) (the blue surface) separates X<sup>u</sup> (the red ball) from all trajectories starting from Θ (the green ball). (Color figure online)

We have implemented a safe controller synthesis tool called *SRLBC* based on Algorithm 2, with Tensorflow 1.14 for the DNN controller synthesis and a Matlab package PENBMI [22] for barrier certificate generation. Table 1 shows the performance evaluation of our *SRLBC* and *nncontroller* in [39] on 12 continuous systems. All experiments are conducted on a machine running Windows 10 with 16 GB RAM, a 3.20 GHz AMD Ryzen 7 3700X CPU, and an NVIDIA GeForce GTX 1650 super GPU.

In Table 1, the origins of these 12 examples are provided in the first column; d**<sup>f</sup>** denotes the maximal degree of the polynomials in the vector fields; n**<sup>x</sup>** denotes the number of the state variables; L and N refer to the numbers of hidden layers and the neurons per each hidden layer, respectively; t<sup>1</sup> and t<sup>2</sup> denote the time spent by *SRLBC* and *nncontroller* in seconds, respectively; the symbol − means that *nncontroller* was unable to return a safe DNN controller within 10,000 s.


**Table 1.** Performance evaluation

Table 1 shows that for the 12 examples, our *SRLBC* manages to handle all of them within 3 iterations, while *nncontroller* can only deal with 8 successfully. Especially for the four examples from C9 to C12 whose dimensions exceed 5, *nncontroller* fails to synthesize safe controllers within specified time bound after various attempt. We have tried different network structures with the number of hidden layers varies from 1 to 5 and the number of hidden neurons chosen among {10, 20, 30, 40}, the *nncontroller* fails to train candidate DNN controllers and barrier certificates within the time limit, whereas our *SRLBC* can yield safe controllers, represented as five-layer sigmoid activated neural networks.

Consider the efficiency of our *SRLBC* and *nncontroller* in terms of the time spent in synthesizing safe DNN controllers for shared examples. On average, our *SRLBC* takes 91.58 s to synthesize a safe DNN controller while *nncontroller* needs 323.2 s, which is about 3.53 times slower than our *SRLBC*. Despite the network structures used for *SRLBC* is more complex than that for *nncontroller*, and the number of neural network neurons of *SRLBC* is much more than that of *nncontroller*, we could synthesize more efficiently.

Obviously, our *SRLBC* scales better than *nncontroller* for the considered examples. Although our *SRLBC* consumes a little more time than *nncontroller* for the systems with dimension 2 or 3, our tool shows its advantage on time consuming when handling the systems with dimension higher than 3 (C6-C8) and its ability on examples C9-C12. Comparing with *nncontroller* which is also a data driven approach, *SRLBC* inherits the advantage in learning efficiency of reinforcement learning, whereas the size of the training data for *nncontroller* increases exponentially with the dimension of the considered systems, which greatly limits the scale of the problem to deal with. Beyond Table 1, we have tried an example of nonlinear polynomial system [16] with dimension up to 12, and *SRLBC* yields successfully a result in 54,314 s while *nncontroller* fails. It is clear that our approach is able to attack large-scale problems.

During the experiment, we have observed that *SRLBC* obtains the nearsafe controllers at the first iteration for most examples, and the remaining work is to refine barrier certificates slightly and use them to guide and adjust the controllers. In fact, the numbers of the iterations in our experiments on the benchmarks did not exceed 3 for all cases. These observations show that our iterative scheme of safe reinforcement learning converges well in practice, because the refinement of the controllers could utilize the intermediate learned results before we get the final results. In addition, *SRLBC* could easily generalize to deal with non-polynomial systems and it has successfully solved the classical continuous Cartpole system [3], which would be presented in the future work.

#### **6 Related Work**

Our work on synthesizing DNN controllers for safety control of nonlinear systems is mainly related to two categories of research, i.e. *formal verification of nonlinear systems with DNN controller* and *safe DNN controller synthesis*. There has been considerable research conducted in these areas because of the applications in safety critical systems in recent years.

**Formal Verification of Nonlinear Systems with DNN Controller.** One of the mainstream methodologies is through constructing over-approximations to the reachable sets of the system trajectories under DNN controllers. And the core technique first focuses on output range analysis of the neural network components, then combines the output range with reachability analysis on the dynamical systems. For instance, based on the output range analysis in [13], Dutta et al. verified the feedback control systems with DNN controllers using mixed-integer linear programming [12]. And they implemented the prototype tool for the neural rule generation inside the tool termed as Sherlock, and used it together with Flow\* for computing the reach sets of the systems [10].

The difference of works on this direction lies in what kind of abstract domains is adopted for output range analysis of the neural network components. A recent attempt involves the work of Xiang et al. that computes the output ranges as a union of convex polytopes [37]. For the piecewise linear systems with ReLU neural network as the controller, they compute the output range of ReLU neural network by a layer-by-layer approach. Dutta et al. propose an approach to abstract the DNN by a local polynomial approximation along with rigorous error bound, and then integrate it with a Taylor model-based flow pipe construction scheme for continuous differential equations to derive the over-approximation of the real reachable set [11]. Likely, Huang et al. present an approach to constructing a polynomial approximation for a DNN controller using Bernstein polynomials, and then integrate result with the plant to get the over-approximated reachable set [18]. There is a different route for reachability of systems with neural network components proposed by Ivanov et al. and termed as Verisig [19]. It transforms the problem of verifying neural network controlled system into a hybrid system verification problem by first transforming a sigmoid-based neural network into an equivalent hybrid system and then composing it with the plant.

Instead of computing reachable sets, a different approach for verifying neural network controlled systems is through barrier certificate synthesis. Tuncali et al. synthesize candidate barrier certificates using simulation-guided techniques, and then verify the overall system safety by checking the validity of the barrier certificate conditions for the candidate [35]. The safety property was proofed, or a counterexample was returned to updated candidate barrier certificates.

**Safety Critical Controller Generation.** Research works in this category differ in: (1) the overall learning framework, e.g. reinforcement learning (RL) or supervised learning; (2) the kind of safety certificate, e.g., control Lyapunov function (CLF) or control barrier function (CBF) [2].

For CLFs or CBFs synthesis, a demonstrator-learner-verifier framework was proposed in [29] to learn polynomial CLFs for polynomial nonlinear dynamical systems; a special type of neural network was designed in [30] as candidates for learning Lyapunov functions; a supervised learning approach was proposed in [5] to learn neural network Lyapunov functions and linear control policies; datadriven model predictive control (MPC) exploiting neural Lyapunov function and neural network dynamics model was proposed in [12,25]. For multi-agent systems, barrier function has recently been applied for safe policy synthesis on POMDP models [1]. The computer science community has dealt with the issue of safe controller learning in different ways from above: for example, a logical-proof based approach was proposed in [15] towards safe RL; a synthesis framework capable of synthesizing deterministic programs from neural network policies was proposed in [41] and so formal verification techniques for traditional software systems can be applied. Compared with these works, [39] learn controllers based on neural networks. To certify the safety property they utilize barrier certificates, which are represented by DNNs as well. In this way, they train DNN controllers and DNN barrier certificates simultaneously, achieving a verification-in-the-loop synthesis. Liu et al. proposed a Recurrent Neural Network (RNN) framework to synthesize feedback control policies for a system under STL specifications [24]. The CBF was used to modify the control policies predicted by the RNN to guarantee safety.

### **7 Conclusion**

In this paper, we have developed a novel scheme for synthesizing safe controllers of nonlinear systems with control against safety constraints. It employs an iterative architecture, where a *learner* trains DNN controllers using reinforcement learning and a *verifier* checks them via computation of maximal safe initial regions and the corresponding barrier certificates, based on polynomial abstraction and bilinear matrix inequalities solving. The key idea in this paper is to use an alternating co-synthesis scheme of controllers and barrier certificates to generate safe controllers, which could refine barrier certificates during iteration. On the one hand, this synthesis scheme has inherited the higher learning efficiency from RL technique than other data driven methods. On the other hand, this iterative architecture could modify barrier certificates to obtain an adaptive one along with DNN controller retraining, and other verification-in-the-loop synthesis methods are usually based on user-defined barrier functions. Furthermore, our BMI solving based barrier certificate generation is more efficient than SMT based verification. The experimental results demonstrate that our method is more scalable and effective than the existing DNN controller synthesis method *nncontroller*.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **HYBRIDSYNCHAADL: Modeling and Formal Analysis of Virtually Synchronous CPSs in AADL**

Jaehun Lee<sup>1</sup>, Sharon Kim<sup>1</sup>, Kyungmin Bae1(B) , and Peter Csaba Olveczky ¨ <sup>2</sup>

<sup>1</sup> Pohang University of Science and Technology, Pohang, Korea kmbae@postech.ac.kr

<sup>2</sup> University of Oslo, Oslo, Norway

**Abstract.** We present the HybridSynchAADL modeling language and formal analysis tool for virtually synchronous cyber-physical systems with complex control programs, continuous behaviors, bounded clock skews, network delays, and execution times. We leverage the Hybrid PALS equivalence, so that it is sufficient to model and verify the simpler underlying synchronous designs. We define the HybridSynchAADL language as a sublanguage of the avionics modeling standard AADL for modeling such designs in AADL, and demonstrate the effectiveness of HybridSynchAADL on a number of applications.

#### **1 Introduction**

Many cyber-physical systems (CPSs) are *virtually synchronous* networks of hybrid components with continuous behaviors combined with sophisticated controllers. They should logically behave as if they were synchronous—in each iteration of the system, all components, in lockstep, read inputs and perform transitions which generate outputs for the next iteration—but have to be realized in a distributed setting, with clock skews and message passing communication. Examples of such CPSs include avionics and automotive systems [34,42], networked medical devices [5,30], and other distributed control systems such as the steam-boiler benchmark [1], where the underlying infrastructure often guarantees bounds on clock skews, network delays, and local execution times.

The uptake of automated formal analysis of such CPSs is challenging, since:


To confront these challenges, we present in this paper the HybridSynch-AADL modeling language and analysis tool, which address them as follows:


Providing formal semantics and analysis for HybridSynchAADL, with its expressive control program formalism, continuous behaviors, and clock skews, and having to cover all possible continuous behaviors based on imprecise clocks, is challenging. We combine Maude [19] and the SMT solver Yices [21] to provide such a semantics, as well as symbolic reachability analysis of bounded invariant properties. To make the analysis feasible, our tool also implements a state-space reduction method that merges symbolic states for Maude-with-SMT to significantly improve the performance of symbolic reachability analysis. We illustrate the use of the HybridSynchAADL language and tool—and compare its effectiveness with other state-of-the-art CPS analysis tools—on a number of hybrid CPS applications, including distributed drones that communicate to reach the "same" location, or fly in formation, without crashing into each other.

Our tool extends the SynchAADL tool [7,9,10] for distributed *real-time* systems without continuous behaviors, where the time when an event takes place can be abstracted away, so there is no need to consider clock skews, and any (sufficiently expressive) explicit-state model checker can be applied. In contrast, HybridSynchAADL must model continuous behaviors and clock skews, and must analyze all possible behaviors based on when the continuous components are sampled and actuated, which depend on the imprecise local clocks. The tool is available at https://hybridsynchaadl.github.io.

#### **2 Preliminaries**

*PALS and Hybrid PALS.* When the infrastructure guarantees bounds Γ on clock skews, network delays, and execution times, the PALS pattern [4,36] reduces the problems of designing and verifying virtually synchronous distributed realtime systems to the much simpler problems of designing and verifying their underlying synchronous designs: Given a synchronous system design *SD*, bounds Γ, and a period p of each round, the PALS transformation gives the asynchronous distributed real-time system *PALS*(*SD*, Γ, p), which is stuttering bisimilar to *SD*.

The synchronous design *SD* is formalized as the synchronous composition of state machines with input and output ports [36]. In each iteration, all machines simultaneously perform a transition, which includes reading inputs, changing the local state, and generating outputs (for the next iteration).

*Hybrid PALS* [8] extends PALS to virtually synchronous CPSs with physical environments that exhibit continuous behaviors. The *physical environment* E*<sup>M</sup>* of a machine M has real-valued parameters x = (x1,...,x*l*). The continuous behaviors of x are modeled by ordinary differential equations (ODEs) that specify different *trajectories* on x. E*<sup>M</sup>* also defines *which* trajectory the environment follows, as a function of the last *control command* received by E*M*.

The local clock of a machine <sup>M</sup> can be seen as a function <sup>c</sup>*<sup>M</sup>* : <sup>R</sup>≥<sup>0</sup> <sup>→</sup> <sup>R</sup>≥<sup>0</sup>, where <sup>c</sup>*M*(t) is the value of the local clock at time <sup>t</sup>, with <sup>∀</sup><sup>t</sup> <sup>∈</sup> <sup>R</sup>≥<sup>0</sup>, <sup>|</sup>c*M*(t)−t<sup>|</sup> < for > 0 the maximal clock skew [36]. In its ith iteration, a controller M samples the values of its environment at time <sup>c</sup>*M*(i·p)+t*s*, where <sup>t</sup>*<sup>s</sup>* is the *sampling time*, and then executes a transition. As a result, the new control command is received by the environment at time <sup>c</sup>*M*(<sup>i</sup> · <sup>p</sup>) + <sup>t</sup>*a*, where <sup>t</sup>*<sup>a</sup>* is the *actuating time*.

*AADL.* The *Architecture Analysis & Design Language* (AADL) [22] is an industrial modeling standard used in avionics, aerospace, automotive, medical devices, and robotics to describe an embedded real-time system. In AADL, a component *type* specifies the component's *interface* (e.g., ports) and *properties* (e.g., periods), and a component *implementation* specifies its internal structure as a set of *subcomponents* and a set of *connections* linking their ports. An AADL construct may have *properties* describing its parameters, declared in *property sets*. The OSATE modeling environment provides a set of Eclipse plug-ins for AADL.

An AADL model describes a system of hardware and software components. Software components include *threads* that model the application software and *data* components representing data types. *System* components are the top-level components. A port is a *data* port, an *event* port, or an *event data* port. A component can have different *modes* and mode-specific property values, subcomponents, etc. Mode transitions are triggered by events.

Thread behavior is modeled as a guarded transition system with local variables using AADL's *Behavior Annex* [23]. When a thread is activated, transitions are applied until a *complete* state is reached. The *dispatch protocol* determines when a thread is executed. A *periodic* thread is activated at fixed time intervals.

*Maude with SMT.* Maude [19] is a language and tool for formally specifying and analyzing distributed systems in rewriting logic. System states are specified as elements of algebraic data types, and transitions are specified using rewrite rules. In addition to its explicit-state analysis methods for concrete states, Maude provides SMT solving and *symbolic reachability analysis* for *constrained terms* <sup>φ</sup> <sup>t</sup>, which symbolically represent all instances of the term t(x1,...,x*n*) satisfying the constraint φ(x1,...,x*n*) [40], using connections to Yices2 [21] and CVC4 [14].

#### **3 The HYBRIDSYNCHAADL Modeling Language**

This section presents the HybridSynchAADL language for modeling virtually synchronous CPSs in AADL. HybridSynchAADL can specify environments with continuous dynamics, synchronous designs of distributed controllers, and nontrivial interactions between controllers and environments with respect to imprecise local clocks and sampling and actuation times.

The HybridSynchAADL language is a subset of AADL extended with property set Hybrid SynchAADL. We use a subset of AADL without changing the meaning of AADL constructs or adding a new annex—the subset has the same meaning for synchronous models and distributed implementations—so that AADL experts can easily develop and understand HybridSynchAADL models.

```
property set Hybrid_SynchAADL is
  Synchronous: inherit aadlboolean applies to (system);
  isEnvironment: inherit aadlboolean applies to (system);
  ContinuousDynamics: aadlstring applies to (system);
  Max_Clock_Deviation: inherit Time applies to (system);
  Sampling_Time: inherit Time_Range applies to (system);
  Response_Time: inherit Time_Range applies to (system);
end Hybrid_SynchAADL;
```
*Environment Components.* An *environment component* models real-valued state variables that continuously change over time. State variables are specified using data subcomponents of type Base\_Types::Float. Each environment component declares the property Hybrid\_SynchAADL::isEnvironment => true.

An environment component can have different *modes* to specify different continuous behaviors (trajectories). A controller command may change the mode of the environment or the value of a variable. The continuous dynamics in each mode is specified using either ODEs or continuous real functions as follows:

```
Hybrid_SynchAADL::ContinuousDynamics =>
     "dynamics1" in modes (mode1), ..., "dynamicsn" in modes (moden);
```
In HybridSynchAADL, a set of ODEs over n variables x1,...,x*n*, say, d*x<sup>i</sup>* <sup>d</sup>*<sup>t</sup>* <sup>=</sup> <sup>e</sup>*i*(x1,...,x*n*) for <sup>i</sup> = 1,...,n, is written as a semicolon-separated string: d/dt(x1) = e1(x1,...,x*n*); ... ; d/dt(x*n*) = e*n*(x1,...,x*n*);

If a closed-form solution of ODEs is known, we can directly specify concrete continuous functions, which are parameterized by a time parameter t and the initial values x1(0),...,x*n*(0) of the variables x1,...,x*n*:

x1(t) = e1(t, x1(0),...,x*n*(0)); ... ; x*n*(t) = e*n*(t, x1(0),...,x*n*(0));

An environment component interacts with discrete controllers by sending its state values, and by receiving actuator commands that may update state variables or trigger mode (and hence trajectory) changes. This behavior is specified in HybridSynchAADL using *connections between ports and data subcomponents*. A connection from a data subcomponent d inside the environment to an output data port o declares that the value of d is "sampled" by a controller. A connection from an environment's input port i to d declares that a controller command arrived at i updates the value of the data subcomponent d.

*Controller Components.* Discrete controllers are usual software components in the Synchronous AADL subset [7,9]. A controller component is specified using the behavioral and structural subset of AADL: hierarchical system, process, thread components, and thread behaviors defined by the Behavior Annex [23].

A controller receives the state of the environment at some *sampling time*, and sends a controller command to the environment at some *actuation time*. Sampling and actuation take place according to the local clock of the controller.

```
Hybrid_SynchAADL::Max_Clock_Deviation => time;
Hybrid_SynchAADL::Sampling_Time => lower bound .. upper bound;
Hybrid_SynchAADL::Response_Time => lower bound .. upper bound;
```
The top-level system component declares the following properties to state that the entire model is a synchronous design with a period T:


*Communication.* In HybridSynchAADL, connections are constrained for synchronous behaviors: no connection is allowed between environments, or between environments and the enclosing system components.

All (non-actuator) outputs of controller components generated in an iteration are available to the receiving *controller* components at the beginning of the *next* iteration. As explained in [7,9], *delayed connections between data ports* meet this requirement. Therefore, two controller components can be connected only by data ports with delayed connections: Timing => Delayed.

Interactions between a controller and an environment occur *instantaneously* at the sampling and actuating times of the controller. Because an environment does not "actively" send data for sampling, every output port of an environment must be a *data* port, whereas its input ports could be of any kind.

#### **4 The HYBRIDSYNCHAADL Tool**

This section introduces the HybridSynchAADL tool supporting the modeling and formal analysis of HybridSynchAADL models. The tool is an OSATE plugin which: (i) provides an intuitive language to specify properties of models, (ii) synthesizes a rewriting logic model from a HybridSynchAADL model, and (iii) performs various formal analyses using Maude combined with SMT solving.

*Specifying Properties.* The tool's *property specification language* allows the user to specify time-bounded invariant and reachability properties as propositional formulas whose atomic propositions are AADL Boolean expressions.

A "named" atomic proposition can be declared in HybridSynchAADL as follows, where each identifier is fully qualified with its component path:

**proposition** [*id* ]: *AADL Boolean Expression*

**Fig. 1.** Interface of the HybridSynchAADL tool.

The following *named invariant property* holds if, for every (initial) state satisfying the initial condition ϕ*init*, all states reachable within the time bound τ*bound* satisfy the invariant condition ϕ*inv* .


A *reachability property* (the dual of an invariant) holds if a state satisfying ϕ*goal* is reachable from some state satisfying ϕ*init* within the time bound τ*bound* .

**reachability** [*name* ]: ϕ*init* ==> ϕ*goal* **in time** τ*bound*

*Tool Interface.* The tool first statically checks whether a given model is a valid model that satisfies the syntactic constraints of HybridSynchAADL.

HybridSynchAADL provides two analysis methods. *Symbolic reachability analysis* can verify that all possible behaviors satisfy a given requirement;<sup>1</sup> if not, a counterexample is generated. *Randomized simulation* repeatedly executes the model until a counterexample is found, by randomly choosing concrete sampling and actuating times, nondeterministic transitions, etc.

Our tool also provides *portfolio analysis* that combines symbolic reachability analysis and randomized simulation. HybridSynchAADL runs both methods in parallel using multithreading, and displays the result of the analysis that terminates first. Symbolic analysis can guarantee the absence of a counterexample, whereas randomized simulation is effective for finding "obvious" bugs.

Figure 1 shows the interface of our tool that is fully integrated into OSATE. The left editor shows the code of FourDronesSystem in Sect. 5, the bottom right editor shows its graphical representation, and the top right editor shows two properties in the property specification language. The HybridSynchAADL menu contains three items for constraint checking, code generation, and formal analysis. The Portfolio Analysis item has already been clicked, and the Result view at the bottom displays the analysis results in a readable format.

<sup>1</sup> Symbolic analysis currently only supports polynomial continuous dynamics, since the Yices2 SMT solver does not support general classes of ODEs.

*Tool Implementation.* We have (in our report [33]) developed a Maude-with-SMT semantics for HybridSynchAADL that formalizes our modeling language and implements our tool's analysis commands. Maude is suitable to capture the expressive control program language, the hierarchical structure of systems, and communication. Symbolic rewriting with SMT allows us to analyze infinite states and all possible behaviors caused by sampling and actuation times with imprecise clocks, where continuous dynamics can be encoded in SMT [18,26].

Nontrivial control programs with many conditional branches and guarded transitions typically involve a large number of symbolic states; to reduce the number of results of executing one iteration of the system, we have implemented a *state merging* optimization technique [11] that merges two symbolic states into one using disjunction and generalization. As shown in the report [33], this state merging dramatically improves the performance of symbolic analysis and makes the formal analysis feasible for such distributed hybrid systems.

The HybridSynchAADL tool uses OSATE's code generation facilities to synthesize a Maude model from the HybridSynchAADL model. It then invokes Maude and an SMT solver to check whether the model satisfies given invariant and reachability requirements. Our tool is implemented in around 6,200 lines of Maude code and around 8,600 lines of Java and Xtend code.

#### **5 Case Study: Collaborating Autonomous Drones**

This section shows how virtually synchronous CPSs for controlling distributed drones—which collaborate to achieve common goals, such as rendezvous and formation control—can be modeled and analyzed in HybridSynchAADL.

*Rendezvous of Multiple Drones.* Consider N drones, where vectors x*<sup>i</sup>* and v*i*, for <sup>1</sup> <sup>≤</sup> <sup>i</sup> <sup>≤</sup> <sup>N</sup>, denote the position and velocity of the <sup>i</sup>-th drone. The continuous dynamics of the <sup>i</sup>-th drone is specified by the ordinary differential equation x˙ *<sup>i</sup>* = v*i*. The controller samples the drone's position and velocity, and gives the new velocity value to the environment as a control command. The goal of rendezvous is for all drones to arrive near a common location simultaneously.

Figure 2 shows the AADL architecture of our rendezvous model for four drones. Each drone is connected to two other drones to exchange positions. A drone component consists of an environment (with the drone's position and velocity) and its controller. Figure 3 shows the implementation of the top-level component, a Drone system component, an Environment system component, and a thread component for a drone controller in HybridSynchAADL.

In each round, the controller obtains the position x from its environment at its sampling time. The position of the connected drone was sent in the previous round. The controller determines a new velocity to synchronize its movement with the other drones using a distributed consensus algorithm [39]. The environment changes its position according to the velocity indicated by its controller, where the new velocity v becomes effective at its actuation time.

**Fig. 2.** The AADL architecture of four drones (left), and a drone component (right).

**Fig. 3.** A HybridSynchAADL model for four distributed drones.

*Verification.* We analyze the following properties up to bound 500 ms using HybridSynchAADL portfolio analysis: (i) drones do not collide (safety), and (ii) all drones can eventually gather together (rendezvous).

**invariant** [safety]: ?initial ==> **not** ?collision **in time** 500; **reachability** [rendezvous]: ?initial ==> ?gather **in time** 500;

We define three propositions: initial, defining the range of initial positions of the four drones dr1, dr2, dr3, and dr4; collision, where two drones collide if the (horizontal and vertical) distance between them is less than 0.1; and gather, indicating that the distance between each pair of drones is less than 1. For example, collision and initial are defined as follows.

```
proposition [initial] :
 abs(dr1.env.x - 1.1) < 0.01 and abs(dr1.env.y - 1.5) < 0.01 and
 abs(dr2.env.x + 1.5) < 0.01 and abs(dr2.env.y + 1.1) < 0.01 and
 abs(dr3.env.x - 1.5) < 0.01 and abs(dr3.env.y - 1.1) < 0.01 and
 abs(dr4.env.x + 1.1) < 0.01 and abs(dr4.env.y + 1.5) < 0.01;
proposition [collision] :
 (abs(dr1.env.x - dr2.env.x) < 0.1 and abs(dr1.env.y - dr2.env.y) < 0.1) or
 ...
 (abs(dr3.env.x - dr4.env.x) < 0.1 and abs(dr3.env.y - dr4.env.y) < 0.1);
```
The analysis result is shown in the Result view at the bottom of Fig. 1. There is a witness for rendezvous, obtained by symbolic reachability analysis in 1.7 seconds. A counterexample for safety is found by randomized simulation in 1.5 seconds, since initial does not constrain the speed of the drones. In [33], we add initial velocity constraints, and verify that safety holds up to the time bound by symbolic reachability analysis in 15 minutes.

#### **6 Experimental Evaluation**

We compare the performance of HybridSynchAADL's symbolic analysis with four reachability analysis tools for hybrid automata, HyComp [18], SpaceEx [24], Flow\* [17], and dReach [31], on models of rendezvous and formation control for distributed drones, and on networked thermostats (adapted from [6,29]). We use *simplified* models with less complex control; otherwise, most of the other tools time out (see [33] for results on more complex models). We use two invariant properties for each model: *Inv* , which holds, and *Inv* <sup>⊥</sup>, which does not hold.

To use the other tools, we have "encoded" the *synchronous designs* of the HybridSynchAADL models as networks of hybrid automata. Each component is modeled as a hybrid automaton with three modes: starting a new round, sampling, and controller transition/actuation. The behavior of a controller is encoded as single jumps. We use flat hybrid automata (obtained by HYST [12]) for Flow\* and dReach, which do not support networks of hybrid automata.

We measure the execution times for analyzing the invariant properties *up to* bound 500 ms, with a timeout of 60 minutes. For HybridSynchAADL, we use


**Table 1.** HybridSynchAADL vs. HyComp, SpaceEx, dReach, and Flow\*.

a specialized version of Maude with Yices 2.6 for polynomial arithmetic [44]. For SpaceEx, we use PHAVer for linear dynamics, and STC for nonlinear polynomial dynamics. For Flow\*, we use adaptive steps, and TM orders 1 (for single) and 2 (for double). We use the default precision for dReach, and BMC for HyComp. We have run all experiments on Intel Xeon 2.8GHz with 256 GB memory.

The results are summarized in Table 1, as execution times (seconds) over time bounds (<sup>B</sup> · 100 ms), with <sup>N</sup> the number of components. The results for "Rend (double)" (rendezvous with double-integrator dynamics, where control input is given by acceleration instead of velocity) do not include HyComp, which does not support nonlinear polynomial dynamics. For *Inv* , Table 1 shows the largest time bound for which the tool could prove the absence of counterexamples. Often, *tools timed out when trying to verify that Inv holds up to time bound 500.*<sup>2</sup> For *Inv* <sup>⊥</sup>, the table shows the smallest bound for which the tool found counterexamples.<sup>3</sup> As seen, HybridSynchAADL outperforms the other tools in most cases, in particular for complex models with larger N.

<sup>2</sup> E.g., for "Rend (single)" with N = 4, HybridSynchAADL needs 5.8 seconds for B = 5, whereas SpaceEx needs 4.5 seconds for B = 1 and timed out for B > 1.

<sup>3</sup> Flow\* occasionally found (spurious) counterexamples at smaller bounds, because of over-approximation by the Taylor model flowpipe construction.

#### **7 Related Work**

Our tool can model check virtually synchronous CPSs with both complex control programs and continuous behaviors (and imprecise local clocks, etc.), whereas most formal tools are strong at analyzing either discrete or continuous behaviors. The latter includes analysis tools for hybrid automata [13,18,24], which do not deal well with the "discrete complexity" (e.g., control programs) of CPSs.

The *Hybrid Annex* [2,3] allows specifying continuous behaviors in AADL, but message delays, clock skews, etc., are not taken into account. Controllers are defined in Hybrid CSP instead of in AADL's convenient Behavioral Annex. Another hybrid annex is proposed in [38], and an AADL sublanguage, AADL+, in [35]. None of these languages support automated formal correctness analysis.

Hybrid PALS models with simple controllers are encoded as logical formulas and analyzed by dReal in [8]. However, there is no tool support, and so CPSs must be *manually* modeled as SMT formulas in [8]. In contrast, we provide a tool for modeling Hybrid PALS models using a well-known modeling standard.

Our work is also related to *almost-synchronous* systems, including approximate synchrony [20], quasi-synchrony [15,16,28,32], GALS [27,37], time-triggered architectures [41,43], etc. Our method makes it possible to verify such systems *with continuous behaviors*, which are typically not considered in related work.

#### **8 Concluding Remarks**

We have presented the HybridSynchAADL modeling language and formal analysis tool for modeling and analyzing the *synchronous designs*—and, by the Hybrid PALS equivalence, also the corresponding *asynchronous distributed realtime system* with bounded clock skews, network delays, and execution times of virtually synchronous networks of hybrid systems with potentially complex control programs in the modeling standard AADL. Our tool provides randomized simulation and symbolic reachability analysis, and is fully integrated into the OSATE modeling environment for AADL. We have shown that in most cases, HybridSynchAADL's symbolic analysis outperforms state-of-the-art hybrid systems reachability analysis tools on a number of distributed hybrid systems.

Currently, HybridSynchAADL's symbolic analysis is restricted to systems with (nonlinear) polynomial continuous dynamics, because the underlying SMT solver, Yices2, cannot deal with general classes of ODEs. We should therefore integrate Maude with ODE solvers such as dReal [25] and Flow\* [17].

**Acknowledgments.** This work was supported in part by the National Research Foundation of Korea (NRF) grants funded by the Korea government (MSIT) (No. 2019R1C1C1002386 and No. 2017M3C4A7068175).

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Computing Bottom SCCs Symbolically Using Transition Guided Reduction**

Nikola Beneˇs, Luboˇs Brim, Samuel Pastva(B) , and David Safr´ ˇ anek

> Faculty of Informatics, Masaryk University, Brno, Czech Republic {xbenes3,brim,xpastva,safranek}@fi.muni.cz

**Abstract.** Detection of bottom strongly connected components (BSCC) in state-transition graphs is an important problem with many applications, such as detecting recurrent states in Markov chains or attractors in dynamical systems. However, these graphs' size is often entirely out of reach for algorithms using explicit state-space exploration, necessitating alternative approaches such as the symbolic one.

Symbolic methods for BSCC detection often show impressive performance, but can sometimes take a long time to converge in large graphs. In this paper, we provide a symbolic state-space reduction method for labelled transition systems, called *interleaved transition guided reduction* (ITGR), which aims to alleviate current problems of BSCC detection by efficiently identifying large portions of the non-BSCC states.

We evaluate the suggested heuristic on an extensive collection of 125 real-world biologically motivated systems. We show that ITGR can easily handle all these models while being either the only method to finish, or providing at least an order-of-magnitude speedup over existing state-of-the-art methods. We then use a set of synthetic benchmarks to demonstrate that the technique also consistently scales to graphs with more than 2<sup>1000</sup> vertices, which was not possible using previous methods.

**Keywords:** Bottom SCC · Symbolic algorithm · Boolean network

#### **1 Introduction**

Finding strongly connected components (SCCs) is a basic problem in graph theory. It is impractical or even impossible for large graphs to find SCCs using explicit depth-first search, motivating the study of symbolic SCCs computation. The structure of SCCs in a graph is captured by its quotient graph, obtained by collapsing each SCC into a single node. This graph is acyclic, thus defines a partial order on the SCCs. *Bottom SCCs* (BSCCs) are SCCs corresponding to leaf nodes in the quotient graph (alternatively referred to as *Terminal SCCs*).

Detection of BSCCs is an important problem with many applications. For example, in Markov chains and Markov decision processes, the recurrent states c The Author(s) 2021

belong to terminal SCCs [1,11,38]. In LTL model checking, the detection of bottom SCCs is used during the decomposition of the property automaton to speed up the model checking procedure [52]. Another example of an application where detection of BSCCs is crucial is detecting non-terminating sections of parallel programs written in C or C++ [55]. In models of dynamical systems, which are of our primary interest, BSCCs correspond to so-called *attractors* that determine the long-term behaviour of the system [43]. Identification of attractors has many important applications, including communication protocols [4,47], systems biology [31,40], mathematical physics [26], ecology [54], epidemiology [42], etc. In biology, the possibility of reaching a particular phenotype of a living cell is indicated by the presence of a specific attractor [40]. The knowledge of attractors then unlocks the path towards cell control [33], reprogramming [49] and even regenerative medicine [17]. Consequently, detection of BSCCs is a fundamental task important not only in computer-aided verification but also many other disciplines.

Our motivation to develop a new symbolic approach to find BSCCs comes from the need to handle extremely large graphs representing *labelled transition systems* that encode the behaviour of complex real-world concurrent processes. In particular, assuming we deal with finite-state systems, such large transition systems are typically generated from models encoded in a compact formalism such as process calculi, Petri nets, Boolean networks [32,57], their combinations [6] or other higher-level modelling languages. For such transition systems, the limits of general symbolic SCC algorithms also define the limits of realistic applications.

In most cases, the size of a transition system generated from a model is exponential in the number of concurrently interacting entities. For example, in the case of biological systems, the number of entities is typically ranging from several hundred to hundreds of thousands. Despite strong simplifications employed at the side of models, the size of respective transition systems rarely falls below 10<sup>6</sup> states and is usually much bigger [23,27,44]. Thus, the need to tackle large transition systems gives us a solid motivation to revisit the algorithmics for BSCC detection.

In general, it is possible to find all BSCCs as a part of a general SCC decomposition algorithm. There is a rich history of research on computing SCCs symbolically. An algorithm based on forward and backward reachability performing <sup>O</sup>(*n*<sup>2</sup>) symbolic steps was presented by Xie and Beerel in [59]. Bloem et al. present an improved O(*n* · log *n*) algorithm in [7]. Finally, an O(*n*) algorithm was presented by Gentilini et al. in [25]. This bound has been proved to be tight in [16]. In [16], the authors argue that the algorithm from [25] is optimal even when considering more fine-grained complexity criteria, like the diameter of the graph and the diameter of the individual components. Ciardo et al. [62] use the idea of saturation [20] to speed up state exploration when computing each SCC in the Xie-Beerel algorithm and compute the transitive closure of the transition relation using a novel algorithm based on saturation.

Our approach is motivated by the fact that techniques working very well for full SCC decomposition do not help to sufficiently accelerate BSCC detection. At the same time, some heuristics, such as saturation, can provide a meaningful impact even when using simpler algorithms [62]. The key novelty of our method lies in a heuristic called *transition guided reduction* that filters the state space by reflecting the possibility of transitions to appear in BSCCs. This step allows to remove some states not belonging to any BSCC, and that way reduce the transition system under analysis to be tractable by the modified Xie-Beerel algorithm with saturation [62].

To target specific characteristics of transition systems representing dynamical systems, e.g., those generated by Boolean networks (BNs) [32,57], several specialised symbolic SCC decomposition methods have been developed. Since systems for our evaluation come primarily from BNs, we also discuss these specialised methods here. A BN consists of Boolean variables, each having a Boolean update function. Update functions change the state of the variables. The semantics of a BN is a transition system where the states are the possible valuations of the variables, and the transitions are induced by the execution of the update functions. Some of the existing algorithms utilise the synchronous update semantics (updates of all variables executed synchronously) that significantly simplifies the problem [24]. However, it is known that synchronous update can produce unrealistic behaviour [37,53]. Models with asynchronous update (concurrently executed updates of variables) are closer to reproducing the real behaviour [15]. For the evaluation of our method, we consider asynchronous BNs. Various specialised techniques of BSCC detection have been developed for asynchronous BNs, including BDDs [24,46,56,60], optimisation [34,35], algebraic-based methods [29], SAT [28], answer set programming [45], concurrency theory [14], sampling [61], or network structure decomposition [18,21]. Moreover, detection of BSCCs is also present as a necessary step in cell reprogramming [41,49] and cell control [2,33] based on BNs. To the best of our knowledge, existing methods specialised for asynchronous BNs do not satisfactorily handle huge models (hundreds of variables and beyond). The best state-of-the-art tools [21,56] are not yet able to robustly work with BNs of such size. We believe that the generally applicable heuristic we propose in this paper can significantly shift the present technology towards massive real-world applications (thousands of variables and beyond).

The main contribution of this paper is a *novel symbolic method for BSCCs detection in state-transition graphs of huge labelled transition systems* for which the problem cannot be handled by existing algorithms. We introduce a novel reduction technique, called *interleaved transition guided reduction (ITGR)*, which aims to enable the use of existing methods by removing large portions of the irrelevant non-BSCC states. The method relies on the observation that BSCCs in real-world systems rarely employ all transition labels available. Therefore, if a state *s* can fire a transition with a label that is not employed by some BSCC reachable from *s*, after applying ITGR, *s* is eliminated. As a result, all paths in the remaining state space only perform transitions with labels employed by their reachable BSCCs. What makes the method truly competitive is the interleaving of multiple processes, each of which performs the reduction for a different transition label. The completion of faster processes speeds up the remaining parts of the computation, which would be otherwise intractable.

To show the real-world benefits of our method, we use a wide collection of models and compare the prototype implementation of ITGR to the state-ofthe-art tool CABEAN [56] as well as to our own implementation of the Xie-Beerel BSCC detection algorithm [62]. In particular, we consider a set of 125 real-world asynchronous BNs selected from publicly available Boolean network repositories, and show that ITGR can easily handle all these models, while either being the only method to finish the computation or providing at least an orderof-magnitude speedup over existing methods. Additionally, we analyse a set of 200 even larger but structurally similar synthetic BNs, which generate transition systems with approximately 2<sup>1000</sup> states. We show that ITGR is the only method that can consistently handle systems of this magnitude.

#### **2 Preliminaries**

To represent a wide variety of large discrete systems, we consider the abstraction generally known as *labelled transition systems*:

**Definition 1.** *Let* L *be a non-empty set of labels. A* labelled transition system *over* <sup>L</sup> *is a pair <sup>T</sup>* = (*S,* { *<sup>a</sup>* −→ | *<sup>a</sup>* ∈ L})*, where <sup>S</sup> is a finite non-empty set of* states*, and for each <sup>a</sup>* ∈ L*, <sup>a</sup>*−→ ⊆ *<sup>S</sup>* <sup>×</sup> *<sup>S</sup> is a* transition relation*.*

When (*s, s* ) <sup>∈</sup> *<sup>a</sup>*−→, we write *<sup>s</sup> <sup>a</sup>*−→ *<sup>s</sup>* , and when (*s, s* ) <sup>∈</sup> *<sup>a</sup>*−→ for *some a* ∈ L, we simply write *s* → *s* . When there is a path *s*<sup>1</sup> → *s*<sup>2</sup> → *...* → *sn*, we write *s*<sup>1</sup> →<sup>∗</sup> *sn*. Each labelled transition system *T* can be seen as a directed *state-transition graph G<sup>T</sup>* = (*S, E*), whose vertices are the states of *T* and whose edges are given by the transition relations, i.e. (*s, s* ) ∈ *E* ⇐⇒ *s* → *s* . This formalism can naturally describe a wide variety of modelling frameworks with built-in nondeterminism, such as Petri nets, Boolean networks, or multi-valued regulatory networks.

We assume to have a symbolic representation of a labelled transition system that allows us to perform symbolic set operations on the subsets of *S* (union ∪, intersection <sup>∩</sup>, difference \, subset test <sup>⊆</sup>, pick an element Pick, etc.) as well as apply the following operations using the associated transition relations:

$$\text{Post}(a, X) = \{ s' \in S \mid \exists s \in X. s \stackrel{a}{\longrightarrow} s' \}$$

$$\text{PRE}(a, X) = \{ s \in S \mid \exists s' \in X. s \stackrel{a}{\longrightarrow} s' \}$$

$$\text{CANPost}(a, X) = \{ s \in X \mid \exists s' \in S. s \stackrel{a}{\longrightarrow} s' \}$$

We further assume a symbolic complexity model where the complexity of each such operation is in <sup>O</sup>(1). Additionally, we use the notation AllPost(*X*) = - *<sup>a</sup>*∈L Post(*a, X*) for *all successors* and AllPre(*X*) = - *<sup>a</sup>*∈L Pre(*a, X*) for *all* *predecessors*. However, the symbolic complexity of these operations is in O(|L|). Finally, we assume that the labels in L are sorted based on the order in which they influence the variables in the symbolic representation (as in, for example, an ordered binary decision diagram [10]). As a consequence, we index the labels and write L = {*a*1*,...,a*|L|}.

Now let us recall a few basic definitions from graph theory in order to define the BSCC detection problem for labelled transition systems:

**Definition 2.** *Let G* = (*V,E*) *be a directed graph. A subset C* ⊆ *V is a* strongly connected component (SCC) *of G iff it is a maximal subset such that for all pairs v, v* ∈ *C, there is a path from v to v .*

*A strongly connected component C is called* bottom *(or* terminal*, BSCC in the following) when there is no edge going from any v* ∈ *C to any v* ∈ *V* \ *C.*

For a given *s* ∈ *V* , we write *SCC*(*s*) to denote the strongly connected component that contains *s*. Furthermore, we say that a set *X* is SCC-closed when *X* = - *<sup>x</sup>*∈*<sup>X</sup> SCC*(*x*). This means that every SCC of *<sup>G</sup>* is either included in *<sup>X</sup>*, or is completely disjoint with *X*. As an example, a set of all reachable vertices from any given initial set is SCC-closed. When |*SCC*(*x*)| = 1 and (*x, x*) ∈ *E*, the SCC is called *trivial*.

For a set *X* ⊆ *V* , we sometimes use the term *the basin of X* to denote the set of all the vertices that have a path to a state in *X*, formally *basin*(*X*) = {*u* | ∃*v* ∈ *X* : *u* →<sup>∗</sup> *v*}. Note that although the name is motivated by the notion of attractor basins in dynamical systems, we use it in a more generalised form here, i.e. we do not require *X* to be a BSCC.

**Problem 1.** *Let T be a labelled transition system and G<sup>T</sup> its corresponding state-transition graph. The problem of* bottom strongly connected component detection (BSCC detection) *is to identify all subsets of S that correspond to the bottom strongly connected components of G<sup>T</sup> .*

A detailed analysis of optimal symbolic asymptotic complexity of a full SCC decomposition can be found in [16]. However, the authors in [16] use a slightly different complexity model, where operations like AllPost and AllPre also assume O(1) complexity. However, their observations about the relationship between problem complexity and graph (or component) diameter are very relevant.

#### **3 Basic Symbolic BSCC Detection**

First, we discuss a BSCC detection algorithm from [62], which will form our baseline going forward. In [62], the authors discuss several symbolic approaches to SCC and BSCC computation, as well as fair cycle detection using various symbolic algorithms, and compare them on large systems from computer science.

In particular, the paper points out that more complex approaches, like the lock-step method [7], work well for full SCC decomposition but do not bring much benefit to the detection of BSCCs. However, the authors do highlight the benefits **Algorithm 1:** Basic BSCC detection algorithm with saturation.

```
1 Function BSCC(universe ⊆ S)
2 while universe = ∅ do
3 pivot ← Pick(universe);
4 basin ← Bwd∗({pivot}, universe);
5 forward ← {pivot};
6 repeat
7 (fixpoint, forward) ← Fwd(forward, universe);
8 until fixpoint or forward ⊆ basin;
9 if forward ⊆ basin then
10 Output(forward);
11 universe ← universe \ basin;
12 Function Bwd∗(reachable, universe)
13 repeat
14 (fixpoint, reachable) ← Bwd(reachable, universe);
15 until fixpoint;
16 return reachable;
17 Function Bwd(reachable, universe)
18 for i ∈ |L| ... 1 do
19 pre ← universe ∩ Pre(ai, reachable);
20 if pre ⊆ reachable then
21 return (false, reachable ∪ pre);
22 return (true, reachable);
```
of basin saturation [19] as a heuristic to speed up the state space search. What we present here is therefore the Xie-Beerel algorithm [59], adapted to BSCC detection with saturation, based on the notes from [62] (we have rewritten the pseudocode to better match the presentation style and background of this paper, though).

The method is summarised in Algorithm 1, which shows the main procedure (BSCC) as well as the reachability procedures Bwd and Bwd∗, which we also use in the later sections. We omit the pseudocode for Fwd and Fwd∗, as they are identical to the Bwd case, only swapping Pre for Post.

**Reachability and Saturation.** The forward and backward reachability procedures are divided into two methods each, Fwd, Bwd, Fwd<sup>∗</sup> and Bwd∗. Since they are functionally symmetrical, we only explicitly discuss backward reachability, with everything directly translating to forward reachability as well.

Bwd performs a single backward reachability step and returns the new set of states together with an indication of whether a fixed point has been reached (i.e. whether no new states have been discovered). Note that in classical saturation, once Bwd selects some *ai*, it is typically applied repeatedly. However, for our primary application domain (Boolean networks), multiple subsequent applications of a single transition would not yield any benefit; we thus use this observation to simplify the pseudocode. In other cases, the recommended approach is to follow [19].

Bwd<sup>∗</sup> then simply wraps Bwd into a cycle that actually computes the full fixed point of the reachable set. This separation into two sub-procedures allows us to perform reachability step-by-step or even interleave multiple reachability procedures (which will come into play later). Remember that for saturation to work well, the ordering of labels needs to follow the ordering of variables in the symbolic representation.

**Xie-Beerel Algorithm.** The main algorithm relies on the well-known observation that for a fixed pivot vertex, the SCC of this vertex can be computed as the intersection of vertices forward and backward reachable from pivot. When searching for BSCCs, we can easily extend this with two extra observations: First, pivot is in a BSCC when only the SCC itself is forward-reachable from pivot. Second, a vertex backward-reachable from pivot is either in the same SCC as pivot (in which case it is in a BSCC iff pivot is in a BSCC), or it is not in a BSCC.

Based on these two extra observations, the original algorithm is modified in two ways: First, not just the SCC around pivot, but all backward-reachable vertices are eliminated at the end of each iteration. Second, the backward reachability from pivot is computed in full, as these are the vertices we can eliminate. However, the forward reachability is terminated early if it leaves the backwardreachable set, since this implies that pivot does not belong to a BSCC.

The asymptotic complexity of this algorithm (in terms of symbolic operations) is O(|L| · |*S*|), which follows from the fact that every vertex will appear in basin exactly once but may need O(|L|) operations to be discovered. Note that optimal symbolic algorithms for BSCC detection are expected to have linear asymptotic complexity. That is, however, assuming a model where AllPost is an O(1) operation, not O(|L|). This may be reasonable for some (in particular synchronous) systems, but as demonstrated in [19], saturation is typically more effective in practice, even though it is not asymptotically optimal in this model.

In [62], the authors show very impressive performance numbers for this simple algorithm. However, there are two drawbacks, which we believe can be improved significantly. And as we demonstrate in the evaluation, while powerful, this algorithm certainly has limits on some real-world models.

First, the performance of this method is directly tied to the selection of the pivot vertex. If the BSCCs of the graph are relatively small, the probability of picking a right pivot is also tiny (remember, even an SCC with 2<sup>100</sup> vertices is only a minuscule fraction of a graph with 2<sup>1000</sup> vertices). As a consequence, the algorithm may require a lot of pivots to explore the entire graph. Second, the overall complexity is limited by the diameter of the whole graph instead of the diameter of the BSCCs. Even if the pivot is picked perfectly, the algorithm still has to explore each BSCC's whole basin sequentially. To some extent, this is inevitable; however, as we hope to demonstrate in the next section, it is not always necessary.

**Algorithm 2:** Core reduction principle


To sum up, Algorithm 1 is a powerful tool for the detection of BSCCs. However, it performs best in graphs where the BSCCs either form a large portion of the state space or have basins of small diameter, allowing the algorithm to converge quickly.

#### **4 Transition Guided Reduction**

**Fig. 1.** Example of transition guided reduction. Square nodes show the pivots set used for this reduction (in this case, the states that can fire transition *a*). Double-drawn states are the BSCCs of the graph. The green area then shows the extendedComponent induced by the two *a* transitions, and the blue area is the bottom set. The striped states are the basins of the two sets, which are eliminated in this reduction. (Color figure online)

In this section, we introduce a technique that we call *transition guided reduction* (TGR) to eliminate a large portion of non-BSCC states. Algorithm 1 can then perform much better on this reduced state space.

We present the technique in two steps: First, in Algorithm 2, we present the core principle of the reduction procedure and prove its correctness. This approach is generally applicable to any directed graph. Then in Algorithm 3 we show how to apply Algorithm 2 in the context of a labelled transition system. Here, we can exploit the knowledge of the transition labels to guide the reduction.

The reduction principle is described in Algorithm 2 and illustrated in Fig. 1. Given a set of pivot states and the current universe of all considered states, the method starts by computing forward—the set of all states reachable from the pivot states. Using this forward set, we then compute the extendedComponent of the given pivot states. Formally, an extended component of set *X* is a subset *X* ⊆ *S* that contains all states from *X*, as well as all paths between the states in *X*. It is a superset of the union - *<sup>x</sup>*∈*<sup>X</sup> SCC*(*x*) but also contains all paths (and SCCs on these paths) that lead between the elements of this union.

We can observe the following properties:


The algorithm then computes the two sets of states that definitely do not contain a BSCC according to these observations and discards these sets from the state space. This is done on lines 5–7 and 8–10, respectively.

Now we can formulate the following theorem:

**Theorem 1.** *If state s* ∈ universe *is discarded by Algorithm 2, then it is not part of any BSCC.*

*Proof.* The proof follows from the two previous observations. If the state *s* is removed on line 7, it means *s* can reach a state *s* ∈ forward. Since the set forward is SCC-closed, we get *SCC*(*s*) = *SCC*(*s* ). State *s* therefore does not belong to a BSCC.

Similarly, if the state *s* is removed on line 10, then it means *s* can reach a state *s* ∈ bottom, and again due to the fact that *bottom* is SCC-closed, the state *s* does not belong to a BSCC. 

However, this does not provide any guidance as to which pivots should we select for the reduction or why. This is addressed in Algorithm 3. Here, we go through all the available transitions *a* ∈ L and select as the pivots the set of all the states that can fire *a* (notice that pivots in Fig. 1 also correspond to such states). As a result, all BSCCs that use *a* are contained in extendedComponent and all BSCCs that do not use *a*, but *a* is performed in their basin are contained in the bottom set. This effectively separates the BSCCs based on the transitions that they use.



Finally, notice that if all transitions of a certain type are eliminated, we remove *a* from L completely. In large systems, this can significantly reduce the overhead of the Fwd*/*Bwd procedures that have to iterate through <sup>L</sup>.

To better describe the cases in which this reduction works well, let us first formally define the following:

**Definition 3.** *Given a labelled transition system T and a state s, the* fire set *F*(*s*) *is the subset of transition labels F*(*s*) ⊆ L *that can be fired in state s, i.e. <sup>a</sup>* <sup>∈</sup> *<sup>F</sup>*(*s*) ⇔ ∃*s* <sup>∈</sup> *<sup>S</sup>* : *<sup>s</sup> <sup>a</sup>*−→ *<sup>s</sup> . A* transitive fire set *F*∗(*s*) *is the union of all the fire sets F*(*s* ) *of all the states s reachable from s (i.e. s* →<sup>∗</sup> *s ).*

Notice that for any two states such that *s* →<sup>∗</sup> *s* , it holds that *F*∗(*s* ) ⊆ *F*∗(*s*). This also means that in any SCC, the transitive fire set of all states is the same.

**Theorem 2.** *Let s be an arbitrary state and s be a state of a BSCC such that s* →<sup>∗</sup> *s . If F*∗(*s*) = *F*∗(*s* )*, Algorithm 3 discards the state s.*

*Proof.* Since *s* →<sup>∗</sup> *s* and *F*∗(*s*) = *F*∗(*s* ), it follows that *F*∗(*s* ) ⊂ *F*∗(*s*). Let *a* ∈ *F*∗(*s*) \ *F*∗(*s* ) be arbitrary and let us consider the iteration of the main loop when *a* is selected. Assume that *s* has not been discarded in any of the previous iterations (otherwise, the proof is already finished). Let *E* be the extendedComponent computed in the current iteration. Then *s* is either *in E*, or *s* can *reach E*, because *a* ∈ *F*∗(*s*).

If *s* ∈ *E*, but *s* can reach *E*, then *s* is eliminated on line 7 of Algorithm 3 as part of the forward basin. When *s* ∈ *E*, it holds that *s* ∈ bottom, since *a* ∈ *F*∗(*s* ). However, since *s* →<sup>∗</sup> *s* , we know that *s* is removed on line 10 because it belongs to the basin of the bottom set. 

However, note that the other implication does not hold. That is, these are not the *only* states that Algorithm 3 eliminates (this can be also seen in Fig. 1).

Based on this theorem, we can derive two extra observations which help to explain the effectiveness of the reduction:

**Corollary 1.** *If a transition system T has a* trivial *BSCC, then the whole basin of this SCC is discarded by Algorithm 3.*

**Corollary 2.** *If a state s is not discarded by Algorithm 3, then all paths starting in s in the reduced state space only use the same transitions as contained in BSCCs reachable from s.*

The first corollary follows from the fact that *F*∗(*s*) = ∅ iff *s* is a trivial BSCC. The second corollary is essentially a rephrasing of Theorem 2, but it highlights an important property of the reduction: if some transition label is not used by a BSCC, all states in its basin that use it will be eliminated. In our experience, real-world systems rarely use *all* available labels in *all* BSCCs (unless most of the state space is just a single large BSCC). Thus by using this pre-processing step, we can greatly simplify the work of Algorithm 1 by pruning "easily identifiable" non-BSCC states.

There is one more point to be made here: Algorithm 1 has to walk the entire depth of the BSCC basins, which can be substantial. Meanwhile, our approach can often "skip ahead" because it is not starting from a single pivot but rather a larger subset of states. However, this may not always be sufficient. In practice, some transitions may be much harder to reduce than others. We address this problem in the next section.

#### **5 Interleaved Transition Guided Reduction**

While TGR can significantly reduce the number of states Algorithm 1 needs to consider, TGR itself iterates transitions in an arbitrary order which can significantly influence the speed and number of steps the reduction needs to perform. Removing a transition potentially reduces the number of states which subsequent reductions need to consider. It is thus beneficial to perform the "easiest" reductions first, as this can greatly simplify the following "harder" cases.

However, determining which reductions are "easy" and which are "hard" is not a simple problem. We could try to use additional structural information about the system to determine this, but that would limit us to a specific subclass of models. Instead, we let the algorithm determine this dynamically on the fly.

Our approach is summarised in Algorithm 4. Instead of reducing one transition relation at a time, we interleave all reductions in one procedure. This is done by creating a number of *processes*, one per each *a* ∈ L, that we run in an interleaving fashion. The processes work in two phases: Forward and ExtendedComponent. The goal of a process in the Forward phase is to compute the value of the forward set starting from the states that enable an *a*transition, and then switch to the ExtendedComponent phase, in which the goal of the process is to compute the corresponding extendedComponent set. The computation proceeds using the one-reachability-step functions Fwd and Bwd, which we defined in Algorithm 1. Every process has its process variables that are local to each process, but their values are kept between steps: The set reach represents the part of forward that has already been discovered; the set component represents the part of extendedComponent that has already been discovered; the variable weight is explained below; and the variable phase holds the current phase of the process (Forward or ExtendedComponent).

The process selected for execution in each iteration (line 31) is the one with the smallest *weight*. The weight of a process is determined by the size of the symbolic representation of the set it is currently expanding (reach in the first **Algorithm 4:** Interleaved Transition Guided Reduction

```
1 Process ItgrWorker(a ∈ L)
     process variables: reach, component, weight, phase
     shared variables : L, universe, processes
2 initialisation:
3 reach ← CanPost(a, universe);
4 weight ← NodeCount(reach);
5 phase ← Forward;
6 step if phase is Forward:
7 (fixpoint, reach) ← Fwd(reach ∩ universe, universe);
8 weight ← NodeCount(reach);
9 if fixpoint then
10 if universe = reach then
11 basin ← Bwd∗(reach);
12 universe ← universe \ (basin \ reach);
13 component ← CanPost(a, universe);
14 weight ← NodeCount(component);
15 phase ← ExtendedComponent;
16 step if phase is ExtendedComponent:
17 (fixpoint, component) ← Bwd(component, reach ∩ universe);
18 weight ← NodeCount(component);
19 if fixpoint then
20 bottom ← (reach ∩ universe) \ component;
21 if bottom = ∅ then
22 basin ← Bwd∗(bottom, universe);
23 universe ← universe \ (basin \ bottom);
24 if CanPost(a, universe) = ∅ then
25 L←L\{a};
26 stop current process (remove it from processes);
27 Function ITGR(universe ⊆ S)
28 processes ← {ItgrWorker(a) | a ∈ L};
29 initialise all processes;
30 while processes = ∅ do
31 p ← MinByKey(processes, weight);
32 run one step of p;
33 return universe;
```
phase, component in the second). For BDDs (or MDDs), this is the number of nodes in the decision diagram (NodeCount). The algorithm thus prioritises processes that have the potential to advance quickly because they will use fast symbolic operations.

Notice that the universe variable is shared by all processes and needs to be now taken into account in multiple places. This means that whenever one process discards some states from universe, all processes benefit from this change. This update can be performed safely because whenever Algorithm 4 discards some states, the discarded set is SCC-closed.

Because both Algorithm 3 and Algorithm 4 compute the same states for forward and extendedComponent (modulo the states eliminated by other reductions), Theorem 1 and Theorem 2 remain valid for Algorithm 4 as well. The only difference is that this approach should be much more resilient to a bad ordering of transition relations.

Finally, let us note that this approach should be quite simple to parallelise to some extent. If *w* parallel workers are available, the algorithm can advance *w* processes at a time instead of picking a single process. However, we do not pursue this approach in this paper as the other methods we use as a reference are also not parallelised.

#### **6 Evaluation**

To see how ITGR affects the performance of attractor detection for real-life systems, we implemented the method for asynchronous Boolean networks (BN), a common logical modelling framework used predominantly in systems biology.

Using this implementation, we aim to support the following claims:


To evaluate the first claim, we compare our implementation with the tool CABEAN [56] on a set of 125 real-world Boolean networks with up to 350 variables. We then generate a pseudo-random set of 200 networks with similar structural characteristics to our real-world benchmarks, but with up to 1100 variables. We show that ITGR can successfully deal with models of this magnitude as well. Finally, we compare the performance of ITGR with the basic attractor detection algorithm as well as with "sequential" TGR on both benchmark sets, showing that ITGR is overall faster and is the only method able to handle the large benchmarks efficiently.

The whole set of benchmarks, as well as the implementation of all the algorithms in Rust, is available as a paper artefact at Zenodo<sup>1</sup>. Additionally, the method is successfully employed by our tool AEON that facilitates long-term analysis of Boolean networks [3].

Before we present the actual benchmark results, let us also first briefly comment on the modelling paradigm chosen (Boolean networks) and the actual setup used to perform the measurements.

<sup>1</sup> https://doi.org/10.5281/zenodo.4709882.

#### **6.1 Boolean Networks**

A Boolean network, as the name suggests, consists of *n* Boolean variables, each variable having an associated update function *bi*. The state space of the network consists of 2*<sup>n</sup>* Boolean variable valuation vectors, {0*,* <sup>1</sup>}*n*. Each update function takes the current state of the network and produces a new value that is assigned to the associated variable, *<sup>b</sup><sup>i</sup>* : {0*,* <sup>1</sup>}*<sup>n</sup>* → {0*,* <sup>1</sup>}. We assume the update functions can be applied non-deterministically, resulting in an asynchronous updating scheme. This is not the only updating scheme used in practice (e.g. synchronous or generalised asynchronous can be used as well) but is generally considered to cover the possible behaviour of the biological system well.

Typically, an update function of a particular variable *x* only depends on a smaller subset of the system variables. In such a case, we say that these variables *regulate x* (specifically, *y* regulates *x* if the update function of *x* depends on the value of *y*). The number of such *regulations* in a Boolean network can be viewed to represent the connectedness or structural complexity of the network in general. In short, the more regulations the network has, the more complex update functions it contains, possibly resulting in more complex behaviour. Variables and regulations together form a directed *regulatory graph*.

A Boolean network with an asynchronous updating scheme fits naturally into our definition of a labelled transition system. The state space of the network variables corresponds with *<sup>S</sup>*, i.e. *<sup>S</sup>* <sup>=</sup> {0*,* <sup>1</sup>}*<sup>n</sup>*. Each transition *<sup>a</sup><sup>i</sup>* of <sup>L</sup> corresponds to the application of the *i*-th Boolean update function *b<sup>i</sup>* to the *i*-th Boolean variable, i.e. (*s, s* ) ∈ *a<sup>i</sup>* ⇔ *s* = *s*[*i* ← *bi*(*s*)] ∧ *s* = *s*.

When dealing with Boolean networks, BSCCs are typically referred to as *attractors*. The rationale behind this term is that the BSCCs are the states where the fair runs of any system eventually converge to—the behaviour is thus *attracted* towards these states. In the following, we use these two terms interchangeably.

As a symbolic representation, the most natural choice for Boolean networks are Reduced Ordered Binary Decision Diagrams (ROBDD, or BDD) [10], as these can be easily used to represent sets of Boolean vectors. We do not make any specific optimisations with regards to variable ordering, but to enable saturationlike reachability, we assume that the ordering of transitions *a*1*,...,a<sup>n</sup>* follows that of the variables in the ROBDD that they update.

Since a Boolean network consists of *n* Boolean variables, a set of states of such a network can be seen as a Boolean formula (represented as a BDD) over the network variables. Here, a state belongs to such a set iff it represents a satisfying assignment of this formula – a fairly standard approach to state-space encoding using BDDs. To apply a particular update function, we must first construct a BDD describing all states where the update function should change the value of its associated variable (note that this BDD can be reused in subsequent steps). By computing an intersection of a state set with this BDD (yielding the result of the CanPost operation) and then performing a "bit flip" of the updated variable in the result BDD, we obtain a set of successor states with respect to this one update function (i.e. Post). Similarly, we can obtain CanPre and Pre.

#### **6.2 Benchmark Set-Up**

**Real-world Models.** To provide the best possible real-world evaluation of our method, we have collected all the models from publicly available Boolean network repositories that we were aware of, and that support the universally accepted SBML-qual format [12] for model transfer. Specifically, our benchmark set includes models available through GINsim [13], Cellcollective [30], Biomodels [39] and the COVID-19 disease map project [48]. Together, the benchmark set consists of 125 models, peaking at 351 variables and 1100 regulations, respectively.

Note that some of these models contain Boolean constants (also called inputs or parameters) that can be specified by the user. For such models, we performed a simple parameter sampling to determine if some of the valuations result in nontrivial attractors, as these are the main focus of this paper. If such valuation was found, we have used it in our benchmark set. However, for the vast majority of models (approximately 90%), there were either no tunable parameters or the sampling did not find any significant changes in the structure of attractors.

**Environment.** We ran all the benchmarks on a machine with a modest 4-core i7 4790 and 32 GB of RAM. However, none of the benchmarks used more than one core at a time, and typical memory consumption was significantly below 1 GB. Hence our evaluation should be reproducible even using a much slower machine.

We have measured the runtime for each model using the standard Unix time utility, with a one-hour timeout per benchmark model. We have run a large portion of the benchmarks repeatedly but have not observed any significant variance in runtime; we thus only report average runtime values.

**CABEAN.** In the real-world performance test, we compare our method to the tool CABEAN [56]. To the best of our knowledge, CABEAN is both the most recent and the most advanced tool that targets the detection of non-trivial attractors in asynchronous Boolean networks. Other tools that we know of (such as [14,21,36]) are not built for systems of the size we are dealing with (for example, due to explicit state-space representation). CABEAN focuses on Boolean network reprogramming, but as a necessary component of this process, it also provides state-of-the-art methods for attractor detection. Specifically, CABEAN uses symbolic manipulation using BDDs, just as our method, but implements advanced decomposition techniques [50] to reduce the state space of the network.

#### **6.3 Real-World Networks**

The core of our results is summarised in Fig. 2. On the left, we see a comparison of total successfully completed benchmarks by both CABEAN and ITGR, and on the right, we have relative speedup for each individual benchmark. On the right, we only show benchmarks that took CABEAN more than one second to complete (remaining models would be normally easy to compute even without any special techniques).

**Fig. 2.** The left plot shows the total number of benchmarks that each tool has completed before a certain time limit. The dashed line represents CABEAN, whereas the solid line shows our ITGR implementation. On the right, the graph displays the relative runtime between CABEAN and ITGR. The dotted lines represent 10x and 100x speedup compared to CABEAN. The solid circles are the benchmarks where CABEAN successfully computed the attractors. The crosses represent the benchmarks where CABEAN was able to finish the decomposition but failed to extract the actual attractors. Notice that we use logarithmic scaling for the time in both graphs.

In this test, ITGR completed all but one benchmark in less than 1 min. The one remaining case took almost 15 min to complete. However, the reduction process for this model was also quite fast at roughly 100 s. The rest of the computation was spent on identifying the 352 non-trivial attractors in the remaining state space (together, the attractors account for almost 2<sup>85</sup> states – by far the largest we have seen in any model). Out of the 125 benchmarks, we uncovered non-trivial attractors in exactly 40 models (however, this also includes 6 models with only small 2- or 4-state attractors).

On the other hand, CABEAN failed to compute attractors for 19 of these 125 models (15*.*2%). Upon closer inspection, all but one of these 19 benchmarks contained non-trivial attractors, which means CABEAN failed for 45% of models with non-trivial attractors.

However, we note that on some models, CABEAN did not simply timeout but actually terminated early due to a segfault. While this behaviour does not seem to be directly linked to the total size of the attractors, it certainly appears to be more common in such networks. We have seen this happen in networks with relatively small attractors, while other networks (even one with a 2<sup>30</sup> state attractor) were completed successfully. We hypothesise that this occurs when the decomposition is (at least partially) successful but does not reduce the complexity of the network enough to continue with attractor search<sup>2</sup>.

These failed attempts are visualised in the right plot using crosses, as they represent an interesting lower-bound approximation on the performance of this decomposition technique. Overall, the plot shows that for the vast majority of models, ITGR provides an order of magnitude (10x) speedup compared to CABEAN, with some (especially larger) models attacking or exceeding the 100x speedup threshold.

Overall, we have shown that ITGR is capable of solving all publicly available problem instances (that we know of), and it outperforms current state-of-the-art decomposition methods with the median of 16x speedup (77x average). Naturally, we also compared the actual attractors found by CABEAN and ITGR, and we are happy to report that we found no inconsistencies between the two methods.

#### **6.4 Pseudo-random Networks**

Next, we set out to test the limits of ITGR on larger models than the ones available in the public repositories today. Specifically, we wanted to test networks with 1000 or more Boolean variables. While such a number is arguably not possible to achieve in a single hand-made model, fully or semi-automated machine learning techniques [5,8,22,51] are making models of this magnitude much more approachable.

**Pseudo-Random BNs.** To create a benchmark set of larger models, we have decided to generate pseudo-random networks structurally similar to our realworld benchmark set. Biological systems, specifically protein and gene regulatory networks, are known to follow certain properties of small-world networks [9, 58]. However, aside from other differences, they are directed and typically quite sparse (our real-world benchmark set has the average node degree of 4.3). This makes most common random network models unsuitable for this specific task. For example, the famous Watts–Strogatz model would, in this case, assume that the average degree is significantly larger than *ln*(1000) ∼ 7.

We have thus first measured the relative in- and out-degree distributions in the regulatory graphs of our real-world networks and then generated random networks by sampling from this distribution to approximate the real-world dataset. Additionally, regulatory graphs of Boolean networks are essentially always weakly connected. In each model, we have thus filtered out all variables except the largest weakly connected component. Note that this makes the dataset slightly skewed towards more connected networks (i.e. more challenging), as these have a higher chance of being weakly connected when randomly generated. However, it is still well within the connectivity limits expected based on the real-world dataset.

<sup>2</sup> Developers of CABEAN have confirmed that this assumption is essentially accurate, i.e. for some structurally complex networks with large non-trivial attractors, the tool can segfault instead of timing-out.

To generate the boolean functions, we have measured that 80.7% of the regulations in the real-world dataset are positively monotonous, with the remaining being negatively monotonous (monotonicity is typically expected in biological networks). Each regulation was thus assigned monotonicity based on this distribution, and a function was generated by randomly choosing between ∧ and ∨ when connecting the positive/negative literals. Note that this does not cover the full spectrum of possible Boolean functions, but it is well within reason for biological networks, where some techniques tend to even implicitly assume the function is just a simple conjunction/disjunction of literals.

**Fig. 3.** Runtime comparison of ITGR, "sequential" TGR and the basic symbolic BSCC detection. The left plot shows the real-world benchmark set with up to 350 variables per model. On the right, we see the *medium* synthetic benchmark ranging from 50 to 1100 variables. Note that the *large* synthetic benchmark (∼ 1000 variables only) is not shown, as ITGR was the only method capable of actually completing these models. All the time axes have a logarithmic scale.

**Performance.** In the end, we have obtained two benchmark sets: A *medium* set with 100 networks ranging from 50 to 1100 variables, and one *large* benchmark set, also with 100 models, but all with ∼ 1000 variables and ranging from 2471 to 5099 regulations. Out of these 200 models, we discovered non-trivial attractors in 61 of them.

The runtime for the *medium* benchmarks is summarised in Fig. 3 (right). Here, we see that ITGR successfully completed all instances within 10 min. For the *large* benchmark set, ITGR consistently finished 98% of the models within 5 to 10 min. The remaining two outliers took 28 and 55 min to complete. Similar to what we saw in the real-world benchmark, these models contained very large non-trivial attractors (the largest having again more than 2<sup>80</sup> states) and were thus not limited by the speed of the reduction but by the diameter of the actual attractors.

Additionally, for each reduction procedure, we kept track of the actual number of reachability iterations that needed to be performed (specifically, how many times we applied line 21 of Bwd as shown in Algorithm 1, or the same line in Fwd). For all models, this number was well below 10 000 iterations, which is quite low considering the procedure needed to evaluate up to 1100 transition relations. In particular, this supports our hypothesis that ITGR works well due to typically short distances between states that can fire individual transitions.

#### **6.5 Interleaving Performance Impact**

Finally, we would like to evaluate the influence of smart interleaving on the performance of ITGR. For this purpose, we consider three algorithms:


Keep in mind that all three approaches use the same implementation of symbolic representation and differ only in the actual attractor detection. Also, TGR/ITGR use the basic algorithm to actually identify attractors once the reduction is completed. Any speedup between TGR and the basic algorithm can be thus directly attributed to the state-space reduction, while any speedup between ITGR and TGR is due to the introduction of interleaving.

For the real-world benchmarks (up to 350 variables) and medium synthetic benchmarks (50 to 1100 variables), the comparison is presented in Fig. 3. Here, we see that the basic algorithm is indeed not generally sufficient for large networks, finishing only 62 of the 125 real-world models and only 5 of the medium synthetic benchmarks (the main reason was typically poor pivot selection; however, some instances also timed out due to long reachability procedures).

The difference between TGR and ITGR is not as drastic for real-world models. TGR finished in 122/125 instances but was consistently slower than ITGR, especially on larger models (on one instance, we have even seen a 55 min vs 2.9 s speedup, i.e. more than 1000x). However, as we look into even larger graphs with the medium synthetic benchmark set, ITGR easily outperforms TGR, which completed only 26/100 instances.

Finally, for the large benchmarks, all with ∼ 1000 variables, we have ITGR completing all benchmarks within the 1-h timeout (with 98% finishing within 10 min); no other method has finished any of the 100 models within this limit. This leaves ITGR as the only implementation in this comparison capable of successfully analysing networks with 1000 or more Boolean variables.

#### **7 Conclusions**

In this paper, we present a novel symbolic method for BSCC detection in statetransition graphs of labelled transition systems, called *interleaved transition* *guided reduction* (ITGR). The method relies on the observation that BSCCs in real-world systems rarely employ all transition labels available. Therefore, if a state *s* can fire a transition with a label that is not employed by some BSCC reachable from *s*, after applying ITGR, *s* is eliminated. As a result, all paths in the remaining state space only perform transitions with labels employed by their reachable BSCCs. If the system has only trivial BSCCs, this solves the problem completely. For non-trivial BSCCs, this may make the problem tractable using previously known techniques.

ITGR relies on smart interleaving to prioritise the elimination of "symbolically easier" transitions. Completing the reduction in this order allows ITGR to subsequently simplify the analysis of transitions which would initially be too complex to handle.

We tested the method on a large benchmark set of real-world Boolean networks (up to 350 variables) as well as randomly generated benchmarks (up to 1100 variables) with similar structural properties. Our experiments show that ITGR significantly outperforms the state-of-the-art tool CABEAN and can easily handle all models from both benchmark sets, pushing the boundary of what was previously possible in this field.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Implicit Semi-Algebraic Abstraction for Polynomial Dynamical Systems**

Sergio Mover1(B) , Alessandro Cimatti<sup>2</sup>, Alberto Griggio<sup>2</sup>, Ahmed Irfan<sup>3</sup>, and Stefano Tonetta<sup>2</sup>

<sup>1</sup> LIX, CNRS, Ecole Polytechnique, Institut Polytechnique de Paris, ´ Palaiseau, France sergio.mover@lix.polytechnique.fr

<sup>2</sup> Fondazione Bruno Kessler, Trento, Italy

<sup>3</sup> Stanford University, Stanford, USA

**Abstract.** Semi-algebraic abstraction is an approach to the safety verification problem for polynomial dynamical systems where the state space is partitioned according to the sign of a set of polynomials. Similarly to predicate abstraction for discrete systems, the number of abstract states is exponential in the number of polynomials. Hence, semi-algebraic abstraction is expensive to explicitly compute and then analyze (e.g., to prove a safety property or extract invariants).

In this paper, we propose an implicit encoding of the semi-algebraic abstraction, which avoids the explicit enumeration of the abstract states: the safety verification problem for dynamical systems is reduced to a corresponding problem for infinite-state transition systems, allowing us to reuse existing model-checking tools based on Satisfiability Modulo Theory (SMT). The main challenge we solve is to express the semi-algebraic abstraction as a first-order logic formula that is linear in the number of predicates, instead of exponential, thus letting the model checker lazily explore the exponential number of abstract states with symbolic techniques. We implemented the approach and validated experimentally its potential to prove safety for polynomial dynamical systems.

### **1 Introduction**

Non-linear dynamical systems are characterized by continuous evolution resulting from ordinary differential equations containing non-linear polynomials. Proving safety properties for non-linear dynamical systems is extremely challenging, and several approaches have been proposed. Semi-automatic deductive verification techniques based on theorem proving include proving hybrid programs using differential dynamic logic [27] or hybrid Cyber Physical System (CPS) using Hybrid Hoare Logic (HHL) [21]). Among various automatic techniques (e.g., [30]), an important line of work applies symbolic model checking to abstractions of hybrid systems, both with, using qualitative predicate abstraction [34]. Unfortunately, the problem with the above techniques is twofold. On one side, the abstractions are often unable to precisely lift important information, thus resulting in an abstract system that is not strong enough to prove the property. On the other side, the abstraction computation may be too expensive to compute, especially in the non-linear case.

To tackle the first problem, we consider the semi-algebraic decomposition for dynamical systems of [32]. The idea is to build an abstraction from a given set of polynomials, partitioning the concrete state space according to the sign of each polynomial. The abstraction is *exact*: there is a transition from an abstract state to another abstract state if and only if there is (at least) a concrete transition from the two concretizations of the abstract states. Semi-algebraic decomposition is also appealing because it can be made *more precise* adding new polynomials.

The abstraction can be computed by means of logical operations (by repeatedly checking the satisfiability of quantifier-free formulas interpreted over the reals). However, the second problem remains: the explicit computation of the abstraction is extremely costly, since it requires the enumeration of all possible transitions between abstract states, which are exponential in the number of considered polynomials.

Interestingly, an effective use of abstraction is at the core of the most successful verification techniques for discrete infinite-state transition systems. The technique of predicate abstraction [16] was originally adapted for symbolic verification in [9] and then optimized in [19]. This idea has been further developed in *implicit predicate abstraction* [35], which eliminates the burden of an up-front exponential blowup in the computation of the abstract states by embedding the abstraction in the symbolic encoding of the transitions. This approach has been used also in combination with IC3 [1,5,6].

In this paper, we propose a new approach to the verification of dynamical systems with non-linear polynomial dynamics based on the use of semi-algebraic decomposition. The contributions of the paper are the following:


*Outline:* The rest of the paper is structured as follows: Sect. 2 gives an overview of the approach with a motivating example; Sect. 3 provides the background definitions; Sect. 4 shows the naive encoding of the abstraction, while in Sect. 5 we derive the linear encoding and define the related implicit semi-algebraic abstraction; in Sect. 6, we present the experimental results; in Sect. 7, we discuss the related work and, finally, in Sect. 8, we draw some conclusions and directions for future work.

#### **2 Overview of the Approach**

Consider a verification problem (adapted from [22]) on the non-linear dynamical system with two variables <sup>x</sup> and <sup>y</sup>, and differential equations ˙<sup>x</sup> <sup>=</sup> <sup>−</sup>2y, <sup>y</sup>˙ <sup>=</sup> <sup>x</sup><sup>2</sup>. We want to prove that the system cannot reach the set of bad states (x + 2)<sup>2</sup> + <sup>y</sup><sup>2</sup>−<sup>1</sup> <sup>≤</sup> 0 (i.e., it never leaves the safe region (x+2)<sup>2</sup>+y<sup>2</sup>−<sup>1</sup> <sup>&</sup>gt; 0) when starting from the initial set of states <sup>x</sup> <sup>−</sup> <sup>y</sup> <sup>−</sup> <sup>1</sup> <sup>2</sup> <sup>≥</sup> <sup>0</sup> <sup>∧</sup> <sup>x</sup> + 2 <sup>&</sup>gt; 0. Note that although in this example the evolution of the system is not restricted, our approach can deal with the more general case in which the evolution is constrained by an invariant condition that must always hold. The system is safe and avoids the set of bad states (see system's dynamic in Fig. 1a).

We can prove that the system is safe by first constructing and then model checking a discrete semi-algebraic abstraction [32]: given the set of polynomials <sup>A</sup> := {<sup>x</sup> <sup>−</sup> <sup>y</sup> <sup>−</sup> <sup>1</sup> <sup>2</sup> , x <sup>+</sup> <sup>y</sup> <sup>+</sup> <sup>1</sup> <sup>2</sup> , x + 2}, the semi-algebraic abstraction partitions the state space according to the sign ({>, <, <sup>=</sup>}) of the polynomials in <sup>A</sup> (an example of abstract state is the state <sup>x</sup> + 2 <sup>&</sup>gt; <sup>0</sup> <sup>∧</sup> <sup>x</sup> <sup>−</sup> <sup>y</sup> <sup>−</sup> <sup>1</sup> <sup>2</sup> <sup>&</sup>lt; <sup>0</sup> <sup>∧</sup> <sup>x</sup> <sup>+</sup> <sup>y</sup> <sup>+</sup> <sup>1</sup> <sup>2</sup> <sup>&</sup>lt; <sup>0</sup> represented as 1 in Fig. 1b). There exists a transition from an abstract state to another one if the two states are neighbors and there exists at least one trajectory of the dynamical system going from one state to the other. The existence of such condition can be checked using the *LZZ* algorithm [22], which checks if a semialgebraic set ψ is a differential invariant for a polynomial dynamical system *f* when its execution is restricted to the domain H (another semi-algebraic set). The algorithm reduces the invariant check to the satisfiability of the Non-Linear Real Arithmetic Theory formula *LZZ*ψ,*<sup>f</sup>* ,H(Z), where Z is a set of realvalued variables. We can systematically check if there exists a transition from an abstract state s<sup>1</sup> to the abstract state s<sup>2</sup> proving that s<sup>1</sup> *is not invariant* when restricted to the domain <sup>s</sup><sup>1</sup> <sup>∨</sup> <sup>s</sup><sup>2</sup> (i.e., checking that *LZZ*<sup>s</sup>1,*<sup>f</sup>* ,s1∨s<sup>2</sup> (Z) is false).

Furthermore, we can use an algorithm, called *LazyReach* [32], to compute the forward set of reachable abstract states starting from the initial states. As usual, if no abstract states intersect the set of bad states then the system is safe, and the reachable set of abstract states is a continuous invariant for the system. Figure 1b shows the state space of the dynamical system: the initial and bad states of the verification problem (represented with the green and red region respectively), the solution of the polynomials from A (represented as blue lines), and further superimpose the set of reachable abstract states and transitions (represented as numbered circles and arrows between the circles). The abstraction shown in Fig. 1b is the result after applying *LazyReach* to the verification problem.

**Fig. 1.** Safety verification problem and reachable states of the abstraction for the nonlinear dynamical system ˙*<sup>x</sup>* <sup>=</sup> <sup>−</sup>2*y, <sup>y</sup>*˙ <sup>=</sup> *<sup>x</sup>*<sup>2</sup>, bad states (*<sup>x</sup>* + 2)<sup>2</sup> <sup>+</sup> *<sup>y</sup>*<sup>2</sup> <sup>−</sup> <sup>1</sup> <sup>≤</sup> 0 (red circle), and initial set of states *<sup>x</sup>* <sup>−</sup> *<sup>y</sup>* <sup>−</sup> <sup>1</sup> <sup>2</sup> ≥ 0 ∧ *x* + 2 *>* 0 (green region). Figure **(a)** shows the verification problem and the system's vector field. Figure **(b)** shows the reachable abstract states and the transitions of the algebraic abstraction (numbered circles and arrows) computed using *LazyReach* and the differential invariant (green and gray regions) obtained from the set of polynomials <sup>A</sup> <sup>=</sup> {*<sup>x</sup>* <sup>−</sup> *<sup>y</sup>* <sup>−</sup> <sup>1</sup> <sup>2</sup> *, x* + *y* + <sup>1</sup> <sup>2</sup> *, x* + 2} (blue lines), computed using *Implicit Abstraction*. Abstract states represent different combinations of signs for the abstraction's polynomials. Examples of abstract states are <sup>1</sup> *<sup>x</sup>*+2 *<sup>&</sup>gt;* <sup>0</sup>∧*x*−*y*<sup>−</sup> <sup>1</sup> <sup>2</sup> *<sup>&</sup>lt;* <sup>0</sup>∧*x*+*y*<sup>+</sup> <sup>1</sup> <sup>2</sup> *<sup>&</sup>lt;* 0, <sup>2</sup> *<sup>x</sup>*+2 *<sup>&</sup>gt;* <sup>0</sup>∧*x*−*y*<sup>−</sup> <sup>1</sup> <sup>2</sup> = 0∧*x*+*y*<sup>+</sup> <sup>1</sup> <sup>2</sup> *<* 0, and <sup>3</sup> *<sup>x</sup>* + 2 *<sup>&</sup>gt;* <sup>0</sup> <sup>∧</sup> *<sup>x</sup>* <sup>−</sup> *<sup>y</sup>* <sup>−</sup> <sup>1</sup> <sup>2</sup> = 0 <sup>∧</sup> *<sup>x</sup>* <sup>+</sup> *<sup>y</sup>* <sup>+</sup> <sup>1</sup> <sup>2</sup> = 0. (Color figure online)

A main challenge for the *LazyReach* algorithm is to explicitly enumerate the reachable states and transitions among them, since their number is exponential in the number of polynomials A (i.e., the number of total states is already 3|A<sup>|</sup> ). For the example above, where we have 3 polynomials, the maximum number of states would be 27, with an even bigger number of transitions (e.g., one must consider the transition between each pair of neighbouring abstract states). Even if *LazyReach* enumerates the reachable abstract states on-the-fly, the explosion in the number of states and transitions is still a bottleneck. Our implementation of *LazyReach* applied to the above example explores a total of 9 states and checks the existence of 27 transitions, taking about 12 s to complete.

A possible solution to tackle the state explosion problem is the *DWCL* algorithm, proposed in [32]. The *DWCL* algorithm<sup>1</sup> tries to reduce the number of abstract states by checking if the sign of a polynomial <sup>a</sup> <sup>∈</sup> <sup>A</sup> is invariant, that is if:

<sup>1</sup> We provide the main intuition behind the *DWCL* algorithm and we refer the reader to [32] for a detailed exposition.


When a predicate a 0 is a continuous invariant, the algorithm strengthens the invariant of the dynamical system (by adding a 0 to the invariants), allowing to remove a from the set of polynomials A. While the *DWCL* algorithm may already find a strong-enough invariant to prove the safety property, the algorithm falls back to the *LazyReach* algorithm in the general case to explore the abstract state space, hopefully with a strengthened invariant domain and a smaller set of polynomials. In practice, the state-space explosion problem of *LazyReach* still exists in the case "not enough" polynomials are sign-invariant, as it happens in our motivating example. In the example, no polynomials are sign-invariant<sup>2</sup>: this means that the *DWCL* algorithm will not remove any polynomials from the set A and *LazyReach* will still suffer from the state-space explosion problem.

The semi-algebraic abstraction is a specific instance of predicate abstraction [16] of the dynamical system *f*. For discrete-state systems, there exist efficient algorithms to either *explicitly* compute the abstraction using Satisfiability Modulo Theory (SMT) solvers [19,20] or to *implicitly* represent the abstraction and directly verify a safety property (e.g., implicit predicate abstraction [35]). Since these algorithms work on a fully symbolic representation of the abstract state space, they can cope with the state-space explosion due to the number of predicates of the abstraction. However, applying the same symbolic-state techniques to compute or verify the semi-algebraic abstraction is still challenging, mainly because it requires to express the transition relation T(X, X ) of the semi-algebraic abstraction in a first-order logic formula. We notice that such transition relation T can be directly obtained from the abstraction's definition<sup>3</sup>:

$$\exists Z. \left(\bigvee\_{(s\_1, s\_2) \in 3^\lambda} s\_1(X) \land s\_2(X') \land \left(\neg LZZ\_{s\_1, \mathfrak{f}, s\_1 \vee s\_2}(Z)\right)\right).$$

The above transition relation enumerates all the possible pairs of abstract states and its size is exponential in the number of polynomials in A. The additional variables Z are copies of the state variables of the system and are used to encode the LZZ condition. Clearly, even creating such formula is not scalable and hinders the application of the standard abstraction and verification techniques used for discrete systems. Note that, while the LZZ algorithm works for semi-algebraic sets (i.e., the candidate invariant ψ and the invariant states H are both arbitrary Boolean combinations of non-linear arithmetic terms), here we apply LZZ to

<sup>2</sup> The differential-cut (DC) and the differential divide-and-conquer (DDC) proof rules used in *DWCL* fail for all the polynomials from A, so *DWCL* would not remove any polynomial.

<sup>3</sup> For clarity, here we do not include additional constraints in the transition relation, such as the neighborhood relation, which instead we consider later in Sect. 4.

check the existence of a transition between two abstract states, hence we still have to explicitly enumerate the abstract transitions.

Our main contribution, presented in Sect. 5, is a compact formulation of the above transition relation that has a size *linear* in the number of the polynomials A. The steps to obtain such exponentially smaller transition are:


$$\exists Z. (InsExpl\_f(X, X', Z) \lor OutExpl\_f(X, X', Z)),$$

where *InsExpl<sup>f</sup>* (X, X , Z) encodes the "inside condition" for all the pairs of transitions (and similarly for the "outside condition" *OutExpl<sup>f</sup>* (X, X , Z)).

3. The formula *InsExpl<sup>f</sup>* (X, X , Z) still contains an explicit enumeration on the pairs of abstract states. We show how we obtain an equivalent formula, *InsSymb<sup>f</sup>* (X, X , Z), that encodes the same condition for each polynomial <sup>a</sup> <sup>∈</sup> <sup>A</sup> in the abstraction, obtaining a linear, instead of exponential, encoding. We apply the same reasoning on *OutExpl<sup>f</sup>* (X, X , Z).

We then use the concise transition relation of T to obtain a symbolic transition system S*Impl*,<sup>P</sup> that implicitly encodes the semi-algebraic abstraction for the dynamical system *<sup>f</sup>* with the polynomials <sup>A</sup> and predicates <sup>P</sup> <sup>=</sup> {a <sup>0</sup> <sup>|</sup> <sup>a</sup> <sup>∈</sup> <sup>A</sup> <sup>∧</sup> ∈ {>, <, <sup>=</sup>}}. Technically, instead of computing the predicate abstraction, we encode an implicit abstraction [35]. Consequently, we avoid the expensive quantifier elimination step. We can then verify the safety property on the transition system S*Impl*,<sup>P</sup> using an SMT-based model checking algorithm. We use the algorithm from [4], since S*Impl*,<sup>P</sup> contains non-linear arithmetic formulas. Our approach verifies the example of Fig. 1 and finds the continuous invariant:

$$(x - y < \frac{1}{2} \lor x \ge -2) \land (x - y \ge \frac{1}{2} \lor x + y \ge -\frac{1}{2}) \land (x - y \ge \frac{1}{2} \lor x + y > -\frac{1}{2}),$$

which is shown in the union of the green and gray regions in Fig. 1b.

#### **3 Preliminaries**

In this work, we consider first order logic formulas in the theory of non-linear arithmetic over the reals (NRA). We denote with φ(X) the formula φ containing free variables from the set <sup>X</sup> <sup>=</sup> {x1,...,x<sup>n</sup>}. We simplify the notation of the formula φ(X) to φ when the set X is clear from the context.

#### **Invariant Verification for Polynomial Dynamical Systems**

*Safety Verification of Dynamical Systems.* Given a set of variables X we write *X* = [x1,...,xn] <sup>T</sup> to specify a vector containing all the variables in X ordered lexicographically. We use the subscript *X*<sup>i</sup> to access to the i-th element of the vector. We focus on *polynomial dynamical systems* of ordinary differential equations (ODEs) *X***˙** = *f*(*X*), where *X***˙** is the vector of first-order derivatives of the variables *X* and *f*(*X*) is a vector of polynomials (i.e., *f*i(*X*) is a polynomial). The *safety verification problem* consists of proving that every trajectory of the dynamical system *X***˙** = *f*(*X*) starting inside the initial set of states ψ and while being inside the evolution domain constraints H remains inside the safe set of states φ. We write the problem using the *differential dynamic logic* [27] formula:

$$
\psi \to [\dot{\mathbf{X}} = \mathbf{f}(\mathbf{X}) \& \ H] \phi,\tag{1}
$$

asserting that if the system is in a state satisfying the pre-condition φ (the initial states) this implies (→ operator) that all the trajectories evolving according to the ODE *X***˙** = *f*(*X*) and evolution domain H (box modality []) will satisfy the post-condition φ (the safe states). Formally, the system is safe if:

$$
\forall \mathbf{x}\_0 \in \psi. \forall \tau \ge 0. \forall t \in [0, \tau]. ((\varphi(\mathbf{x}\_0, t) \in H) \to \varphi(\mathbf{x}\_0, t) \in \phi),
$$

were the differentiable function <sup>ϕ</sup> : <sup>R</sup><sup>n</sup>+1 <sup>→</sup> <sup>R</sup><sup>n</sup>, such that <sup>d</sup> dt (ϕ(*x*0, t)) = *<sup>f</sup>*(ϕ(*x*0, t)), is the solution to the initial value problem *<sup>x</sup>*<sup>0</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup> (i.e., <sup>ϕ</sup>(*x*0, t) describes the state the dynamical system *<sup>f</sup>* reaches after <sup>t</sup> <sup>∈</sup> <sup>R</sup> time when starting in the initial state *x*0).

The problem of proving the system is safe can be reduced to find a formula <sup>θ</sup>(X) such that: i) <sup>H</sup> <sup>∧</sup> <sup>ψ</sup> <sup>→</sup> <sup>θ</sup>, ii) <sup>θ</sup> <sup>→</sup> [*X***˙** <sup>=</sup> *<sup>f</sup>*(*X*) & <sup>H</sup>] <sup>θ</sup>, and iii) <sup>θ</sup> <sup>→</sup> <sup>φ</sup>. θ(X) is a *continuous invariant* [28] that contains the initial states and that is contained in the safe states.

*LZZ Algorithm* [22]. The LZZ algorithm reduces the problem of checking if θ is a continuous invariant to checking the validity of the following formula:

$$\text{LZZ}\_{\theta, f, H}(X) \dot{=} ( (\theta(X) \land H(X) \land In\_{f,H}(X)) \to In\_{f,\theta}(X) )\land \tag{2}$$

$$( (\neg \theta(X) \land H(X) \land In\_{-f,H}(X)) \to \neg In\_{-f,\theta}(X) ),$$

where the formula *In<sup>f</sup>* ,γ(X) for the ODEs *f* and the formula γ represents the set of states which will evolve inside the set γ for some non-zero time in the future. Respectively, the formula *In*−*<sup>f</sup>* ,γ(X) represents the set of states evolved inside the set <sup>γ</sup> for some non-zero time in the past, and <sup>−</sup>*<sup>f</sup>* represents the dynamical system evolving in "reverse". Note that the construction of the formula *In<sup>f</sup>* ,γ(X) assumes γ to be in disjunctive normal form (DNF):

$$\gamma = \bigvee\_{d \in disj(\gamma)} \bigwedge\_{a \bowtie a \bowtie pred(d)} a(X) \bowtie 0,$$

where disj(γ) are the disjuncts of a formula γ, pred(d) are the predicates in the disjunct <sup>d</sup>, and ∈ {>, ≥}<sup>4</sup>. The formula *In<sup>f</sup>* ,γ(X) is defined as:

$$In\_{f, \gamma}(X) = \bigvee\_{d \in disj(\gamma)} \bigwedge\_{a \bowtie 0 \in pred(d)} In\_{f, a \bowtie 0}(X). \tag{3}$$

The formula *In<sup>f</sup>* ,a0(X) encodes the set for a single predicate a 0 using the Lie derivatives of the polynomial a(X). The i-th *Lie derivative* L<sup>i</sup> *<sup>f</sup>* <sup>a</sup> of a polynomial a(X) with respect to the ODEs *f* is defined recursively as:

$$L\_f^{(0)}a \doteq a, \qquad L\_f^{(i)}a \doteq \frac{\partial}{\partial \mathbf{X}} L\_f^{(i-1)}af.$$

*In<sup>f</sup>* ,a>0(X) encodes that the first non-zero Lie derivative of a must be positive in order for the trajectories of the system to enter the set a > 0 and stay inside the set for a positive time<sup>5</sup>(see [22] and [12] for a thorough explanation):

$$\operatorname{Im}\_{f,a>0}(X)\doteq\bigvee\_{0\le i\le N\_{a,f}}\left(\bigwedge\_{0\le j0\right),\tag{4}$$

$$\operatorname{In}\_{f,a\geq 0}(X) \doteq \operatorname{In}\_{f,a>0}(X) \vee \bigvee\_{0\leq i\leq N\_{a,f}} L\_f^{(i)}a = 0,\tag{5}$$

where Na,*<sup>f</sup>* is an integer constant and is an upper bound on the minimum integer number <sup>r</sup> (called *rank*) such that <sup>L</sup>(r) *<sup>f</sup>* <sup>a</sup> = 0 (for all <sup>x</sup> <sup>∈</sup> <sup>R</sup><sup>n</sup>). <sup>N</sup>a,*<sup>f</sup>* can be computed using Gr¨obner basis as explained in [22].

In the following, we will only use the fact that the formula *In<sup>f</sup>* ,γ(X) for the DNF formula γ is the DNF formula where In<sup>f</sup> is applied to the predicates (as shown in Formula (3)).

*Semi-Algebraic Abstraction* [32]. The *semi-algebraic abstraction* of the dynamical system *X***˙** = *f*(*X*) partitions its state space with respect to a set of polynomials <sup>A</sup> =˙ {a1,...,a<sup>m</sup>}. The abstraction is the (explicit state) transition system <sup>S</sup><sup>A</sup> =˙ <sup>3</sup><sup>A</sup>, If,A, Tf,<sup>A</sup> where:

	- <sup>s</sup><sup>1</sup> is an abstract state *adjacent* to <sup>s</sup>2. The abstraction exploits the continuity assumption on *f* and does not allow the system to transition directly from a state where a predicate is greater than 0 (e.g., a > 0) to a state where the same predicate is less than 0 (e.g., a < 0), and vice-versa.

<sup>4</sup> Later we also consider equalities (i.e., predicates of the form *a* = 0). The construction of *In<sup>f</sup>* ,a=0(*X*) can be found in [12].

<sup>5</sup> In our implementation we encode *In<sup>f</sup>* ,a>0(*X*) using the remainders of the Lie derivative, as in [12].

The abstraction does not visit two abstract states containing predicates with opposite signs, forcing instead to visit the intermediate state where the predicate is equal to 0.

• There exists a continuous trajectory from <sup>s</sup><sup>1</sup> to <sup>s</sup>2. This condition corresponds to checking that the following differential dynamic logic formula is *not valid* (i.e. s<sup>1</sup> is not a differential invariant when restricting the evolution domain to <sup>s</sup><sup>1</sup> <sup>∨</sup> <sup>s</sup>2):

$$s\_1 \to [\dot{\mathbf{X}} = \mathbf{f}(\mathbf{X}) \&\ s\_1 \lor s\_2] s\_1,$$

which can be checked using the sound and complete LZZ algorithm, i.e. checking the satisfiability of the first-order formula <sup>¬</sup>*LZZ*<sup>s</sup>1,*<sup>f</sup>* ,s1∨s<sup>2</sup> (Z)).

Since the number of states 3<sup>A</sup> is finite we can compute the set of reachable states. The concretization of this set, θ contains the initial states and is a differential invariant. If θ further implies the safe states ψ, then we prove the safety verification problem 1. However, the computation of the abstract transition relation is exponential in the number of polynomials in A because we would need to enumerate all the possible pairs of transitions (s1, s2) <sup>∈</sup> <sup>3</sup><sup>A</sup> <sup>×</sup> <sup>3</sup><sup>A</sup>.

#### **Predicate Abstraction**

<sup>A</sup> *symbolic transition system* <sup>S</sup> is a tuple <sup>S</sup> =˙ V, I,T , where <sup>V</sup> is a set of (state) variables, I(V ) is a formula representing the initial states, and T(V,V ) is a formula representing the transition relation. A *state* s of S is an interpretation of the state variables V . A (finite) *path* π of S is a finite sequence π =˙ s0, s1,...,s<sup>k</sup> of states with the same domain and interpretation of symbols in the signature Σ such that <sup>s</sup><sup>0</sup> <sup>|</sup><sup>=</sup> <sup>I</sup> and for all <sup>i</sup>, 0 <sup>≤</sup> i<k, <sup>s</sup>i, s <sup>i</sup>+1 <sup>|</sup><sup>=</sup> <sup>T</sup>. We say that a state <sup>s</sup> is *reachable* in S iff there exists a path of S ending in s. Given a formula P(V ) and a transition system <sup>S</sup>, the *invariant verification problem*, denoted with <sup>S</sup> <sup>|</sup><sup>=</sup> <sup>P</sup>, checks if for all the finite paths <sup>s</sup>0, s1,...,s<sup>k</sup> of <sup>S</sup>, for all <sup>i</sup>, 0 <sup>≤</sup> <sup>i</sup> <sup>≤</sup> <sup>k</sup>, <sup>s</sup><sup>i</sup> <sup>|</sup><sup>=</sup> <sup>P</sup>.

Predicate Abstraction [16] partitions the concrete system <sup>S</sup> <sup>=</sup> V, I,T according to a finite set of predicates <sup>P</sup> =˙ {p1,...,p<sup>k</sup>} in a finite symbolic transition system:

$$\widehat{S}\_{\mathbb{P}} = \langle V\_{\mathbb{P}}, \widehat{I}\_{\mathbb{P}}(V\_{\mathbb{P}}), \widehat{T}\_{\mathbb{P}}(V\_{\mathbb{P}}, V'\_{\mathbb{P}}) \rangle$$

using a new abstract Boolean variable v<sup>p</sup> for each predicate p (V<sup>P</sup> = {v<sup>p</sup> <sup>|</sup> <sup>v</sup> <sup>∈</sup> <sup>V</sup> } is the set of those new variables). The abstraction relation *H*P(V,VP) ˙= <sup>p</sup>∈<sup>P</sup> <sup>v</sup><sup>p</sup> <sup>↔</sup> <sup>p</sup>(<sup>V</sup> ) defines how a set of concrete states is abstracted to the abstract states. We compute the abstraction of a formula ψ(V ) by existentially quantifying the concrete variables V :

$$
\widehat{\psi}\_{\mathbb{P}}(V\_{\mathbb{P}}) \dotplus \exists V. (\psi(V) \land H\_{\mathbb{P}}(V, V\_{\mathbb{P}})) .
$$

Similarly, we compute the abstract transition relation for T(V,V ):

$$
\widehat{T}\_{\mathbb{P}}(V\_{\mathbb{P}}, V\_{\mathbb{P}}') \doteq \exists V, V'. (T(V, V') \land H\_{\mathbb{P}}(V, V\_{\mathbb{P}}) \land H\_{\mathbb{P}}(V', V\_{\mathbb{P}}')).
$$

The above formulation is sufficient to compute the predicate abstraction for an infinite-state transition system <sup>S</sup> <sup>=</sup> V, I,T and a set of predicates <sup>P</sup>. However, the main challenge in computing the abstraction is to eliminate the quantifiers, since quantifier elimination is expensive to compute.

*Implicit Predicate Abstraction.* Implicit Predicate Abstraction [35] is a model checking algorithm that avoids computing the abstract version of the initial states, safety property, and transition relation, instead it encodes the existence of a path in the abstract system. It exploits the fact that the abstraction induces an equivalence relation among concrete states of the system (i.e., two concrete states are equivalent if they belong to the same abstract state) and that this relation can be expressed as a quantifier free formula:

$$EQ\_{\mathbb{P}}(V,\overline{V}) \doteq \bigwedge\_{p \in \mathbb{P}} p(V) \leftrightarrow p(\overline{V}).\tag{6}$$

We use the equivalence *EQ*P(V, V ) to relate two sets of concrete states and we encode the problem of reaching a set of target states <sup>¬</sup><sup>P</sup> in <sup>k</sup> steps of the transition system S as follows:

$$\begin{aligned} BMC\_{\mathbb{P}}^{k} & \doteq I(V^{0}) \wedge EQ\_{\mathbb{P}}(V^{0}, \overline{V}^{0}) \wedge \\ & \bigwedge\_{1 \le h < k} \left( T(\overline{V}^{h-1}, V^{h}) \wedge EQ\_{\mathbb{P}}(V^{h}, \overline{V}^{h}) \right) \wedge T(\overline{V}^{k-1}, V^{k}) \wedge \\ & EQ\_{\mathbb{P}}(V^{k}, \overline{V}^{k}) \wedge (\neg P(\overline{V}^{k})) .\end{aligned}$$

The formula *BMC*<sup>k</sup> <sup>P</sup> is satisfiable iff there exists a path in the abstract transition system <sup>S</sup><sup>P</sup> of length <sup>k</sup> starting from the (abstracted) initial states <sup>I</sup> <sup>P</sup>(VP) and reaching the (abstracted) bad states ¬ PP(VP).

#### **4 Explicit Computation of the Semi-Algebraic Abstraction**

We frame the problem of computing the semi-algebraic abstraction as a predicate abstraction problem. This formulation allows us to use the standard techniques to compute or analyze the predicate abstraction for discrete systems.

We consider the invariant verification problem <sup>ψ</sup> <sup>→</sup> [*X***˙** <sup>=</sup> *<sup>f</sup>*(*X*) & <sup>H</sup>]<sup>φ</sup> as in Eq. (1) and a set of polynomials <sup>A</sup> <sup>=</sup> {a1,...,a<sup>m</sup>} for the abstraction. We construct a symbolic transition system of the semi-algebraic abstraction:

$$
\widehat{S}\_{\mathbb{P}} \doteq \langle V\_{\mathbb{P}}, \widehat{I}\_{\mathbb{P}}(V\_{\mathbb{P}}), \widehat{T}\_{\mathbb{P}}(V\_{\mathbb{P}}, V'\_{\mathbb{P}}) \rangle,
$$

where the set of predicates of the abstraction is <sup>P</sup> <sup>=</sup> {a <sup>0</sup> <sup>|</sup> <sup>a</sup> <sup>∈</sup> <sup>A</sup> <sup>∧</sup> ∈ {>, < , <sup>=</sup>}}, and the set of abstract variables <sup>V</sup><sup>P</sup> is defined as in Sect. <sup>3</sup> (i.e., the abstraction contains a Boolean variable <sup>v</sup><sup>p</sup> for each predicates <sup>p</sup> <sup>∈</sup> <sup>P</sup>). We similarly use the formula HP(X, VP) to describe the equivalence relation of the concrete states. The formulas I <sup>P</sup>(VP) and <sup>¬</sup> PP(VP) are the semi-algebraic abstraction of the initial states <sup>ψ</sup> and of the unsafe states <sup>¬</sup>φ:

$$
\widehat{I\_{\mathbb{P}}}(V\_{\mathbb{P}}) \dot{=} \exists X. (\psi(X) \land H\_{\mathbb{P}}(X, V\_{\mathbb{P}})), \qquad \widehat{\neg P\_{\mathbb{P}}}(V\_{\mathbb{P}}) \dot{=} \exists X. (\neg \phi(X) \land H\_{\mathbb{P}}(X, V\_{\mathbb{P}})),
$$

and we obtain the abstraction by existentially quantifying the concrete variables <sup>X</sup>. The definition of the abstract transition relation <sup>T</sup>P(VP, V <sup>P</sup>), which differs from the encoding of the semi-algebraic decomposition, is:

$$
\widehat{T}\_{\mathbb{P}}(V\_{\mathbb{P}}, V\_{\mathbb{P}}') \doteq \exists X, X'. \left( N(X, X') \land H(X) \land H(X') \land \tag{7}
$$

$$
H\_{\mathbb{P}}(X, V\_{\mathbb{P}}) \land H\_{\mathbb{P}}(X', V\_{\mathbb{P}}') \land \exists Z. T\_h(X, X', Z) \right),
$$

where N(X, X ) encodes the adjacent relation between abstract states:

$$N(X, X') = \bigwedge\_{a \in \mathbb{N}} \left( (a(X) < 0 \to a(X') \le 0) \land (a(X) > 0 \to a(X') \ge 0) \right),$$

and TA(X, X , Z) encodes the existence of a transition in the dynamical system *<sup>f</sup>* for each pair of abstract states (s1, s2) <sup>∈</sup> <sup>3</sup><sup>A</sup>:

$$T\_{\mathbb{A}}(X, X', Z) \doteq \bigvee\_{(s\_1, s\_2) \in 3^{\mathbb{A}}} \left( s\_1(X) \wedge s\_2(X') \wedge \neg LZZ\_{s\_1, f, s\_1 \vee s\_2}(Z) \right). \tag{8}$$

**Theorem 1.** *The transition systems* <sup>S</sup><sup>A</sup> *and* <sup>S</sup><sup>P</sup> *are bisimilar.*

$$\text{Corollary 1. } \hat{S}\_{\mathbb{P}} \vdash \widehat{\neg\neg P}\_{\mathbb{P}}(V\_{\mathbb{P}}) \text{ implies } \psi \to [\dot{\mathbf{X}} = \mathbf{f}(\mathbf{X}) \&\ H] \phi.$$

*Proof* (*sketch).* The proof follows directly from Theorem 1.

While the encoding of the transition relation <sup>T</sup><sup>P</sup>(VP, V <sup>P</sup>) is symbolic, it (and in particular the sub-formula TA(X, X , Z)) explicitly enumerates an exponential number of abstract pairs of states. Clearly, this encoding is not practical and defeats the purpose of using symbolic techniques to compute the abstraction.

#### **5 Linear Encoding of the Semi-Algebraic Abstraction**

#### **Specializing the LZZ Formula for Checking Abstract Transitions**

The construction of the semi-algebraic abstraction uses the formula <sup>¬</sup>*LZZ*<sup>s</sup>1,*<sup>f</sup>* ,s1∨s<sup>2</sup> (Z) to encode the existence of a transition from the abstract state s<sup>1</sup> to the abstract state s2. We observe that here the LZZ algorithm is applied to formulas with a specific structure – the abstract states s1(Z) and s2(Z), in contrast to arbitrary semi-algebraic sets as in the general case of *LZZ*θ,*<sup>f</sup>* ,H(X) where the formulas <sup>θ</sup> and <sup>H</sup> are in DNF. Instead, in the case of *LZZ*<sup>s</sup>1,*<sup>f</sup>* ,s1∨s<sup>2</sup> (Z),

each abstract state <sup>s</sup>i(X) assigns a sign to each polynomial <sup>a</sup> <sup>∈</sup> <sup>A</sup> and is represented as conjunctions of predicates <sup>s</sup><sup>i</sup> <sup>=</sup> <sup>a</sup><sup>1</sup> <sup>1</sup> <sup>0</sup> <sup>∧</sup> <sup>a</sup><sup>2</sup> <sup>2</sup> <sup>0</sup> <sup>∧</sup> ...a<sup>m</sup> <sup>m</sup> 0, where j∈ {>, <, <sup>=</sup>}. We will write the conjunction representing a state <sup>s</sup>i(X) as a0∈s*<sup>i</sup>* <sup>a</sup>(X) 0. Also note that the evolution domain constraints are also a disjunction of two abstract states <sup>s</sup><sup>1</sup> <sup>∨</sup> <sup>s</sup>2.

We specialize Eq. (2) to the specific case of *LZZ*s1,*<sup>f</sup>* ,s1∨s<sup>2</sup> (Z). We will use such specialization to obtain a compact (linear in the number of polynomials) encoding later in the section. Instantiating the formula (2) to the case of *LZZ*<sup>s</sup>1,*<sup>f</sup>* ,s1∨s<sup>2</sup> (Z), we get:

$$\text{LZZ}\_{s\_1, f, s\_1 \vee s\_2}(Z) \doteq $$

$$\{(s\_1(Z) \land (s\_1(Z) \lor s\_2(Z)) \land In\_{f, s\_1 \lor s\_2}(Z)) \to In\_{f, s\_1}(Z)\} \land \tag{9}$$

$$\{(\neg s\_1(Z) \land (s\_1(Z) \lor s\_2(Z)) \land In\_{-f, s\_1 \lor s\_2}(Z)) \to In\_{-f, s\_1}(Z)\}$$

$$\{(\neg s\_1 \lor \neg s\_2 \lor \neg s\_1 \lor \neg s\_2 \lor \neg s\_1 \lor \neg s\_2 \lor \neg s\_1)\} \land \} \to \{\neg s\_1 \lor \neg s\_2 \lor \neg s\_1\}$$

Applying the Boolean identities: (α ∧ (α ∨ β)) ↔ α, (¬α ∧ (α ∨ β)) ↔ ¬α ∧ β

$$\begin{aligned} \iff ((s\_1(Z) \land \operatorname{In\_{f,s\_1} \lor s\_2}(Z)) \to \operatorname{In\_{f,s\_1}}(Z)) \land \\ \qquad ((\neg s\_1(Z) \land s\_2(Z) \land \operatorname{In\_{-f,s\_1} \lor s\_2(Z)) \to \neg \operatorname{In\_{-f,s\_1}}(Z)) \end{aligned} \tag{10}$$

Rewriting the implication and applying De Morgan's laws:

$$\Longleftrightarrow (\neg s\_1(Z) \lor \neg \operatorname{In}\_{f, s\_1 \lor s\_2}(Z) \lor \operatorname{In}\_{f, s\_1}(Z)) \land \tag{11}$$
 
$$(s\_1(Z) \lor \neg s\_2(Z) \lor \neg \operatorname{In}\_{f, s\_1 \lor s\_2}(Z) \lor \neg \operatorname{In}\_{-f, s\_1}(Z))$$
 
$$\cdots \quad \cdot \quad \cdot \quad \cdot \quad \cdot \quad \cdot \quad \cdot \quad \cdot \quad \cdot \quad \cdot \quad \cdot \quad \cdot \quad \cdot$$

Expanding the definition of In(Eq. (3)) :*In<sup>f</sup>* ,α∨<sup>β</sup> = ( ˙ *In*−*<sup>f</sup>* ,α <sup>∨</sup> *In<sup>f</sup>* ,β)

$$\begin{aligned} \operatorname{In}\_{-f,\alpha\vee\beta} & \doteq (\operatorname{In}\_{-f,\alpha} \vee \operatorname{In}\_{-f,\beta}) \\ \Longleftrightarrow (\neg s\_1(Z) \vee \neg(\operatorname{In}\_{f,s\_1}(Z) \vee \operatorname{In}\_{f,s\_2}(Z)) \vee \operatorname{In}\_{f,s\_1}(Z)) \wedge \\ (s\_1(Z) \vee \neg s\_2(Z) \vee \neg(\operatorname{In}\_{-f,s\_1}(Z) \vee \operatorname{In}\_{-f,s\_2}(Z)) \vee \neg \operatorname{In}\_{-f,s\_1}(Z)) \end{aligned} \tag{12}$$

Applying the Boolean identities: (¬(α ∨ β) ∨ α) ↔ (¬β ∨ α), (¬(α ∨ β) ∨ ¬α) ↔ ¬α

$$\begin{aligned} \iff (\neg s\_1(Z) \lor \neg \operatorname{In\_{f,s\_2}}(Z) \lor \operatorname{In\_{f,s\_1}}(Z)) \land\\ \land (s\_1(Z) \lor \neg s\_2(Z) \lor \neg \operatorname{In\_{-f,s\_1}}(Z)) .\end{aligned} \tag{13}$$

Note that, while In does not distribute over arbitrary Boolean formulas (see [12]), when we expand the definition of In*<sup>f</sup>* ,s1∨s<sup>2</sup> (Eq. (12)), the formula <sup>s</sup><sup>1</sup> <sup>∨</sup> <sup>s</sup><sup>2</sup> is in DNF. Thus, Formula (13) is equivalent to the initial Formula (9) of *LZZ*<sup>s</sup>1,*<sup>f</sup>* ,s1∨s<sup>2</sup> (Z). We then write the negation of the Formula <sup>13</sup> as:

$$\begin{aligned} \neg \text{LZZ}\_{s\_1, f, s\_1 \lor s\_2}(Z) &\doteq (s\_1(Z) \land In\_{f, s\_2}(Z) \land \neg In\_{f, s\_1}(Z)) \lor \\ &\qquad (\neg s\_1(Z) \land s\_2(Z) \land In\_{-f, s\_1}(Z)). \end{aligned} \tag{14}$$

#### **Linear Encoding of the Semi-Algebraic Transition Relation**

In the following steps, we revise the formula TA(X, X , Z) that encodes the existence of the transitions in the abstraction, still enumerating all possible pairs of states, using the specialized LZZ encoding from Eq. (14). We substitute the subformula <sup>¬</sup>*LZZ*<sup>s</sup>1,*<sup>f</sup>* ,s1∨s<sup>2</sup> (Z) with the specialized LZZ encoding (Eq. (16)); we then distribute the conjunction <sup>s</sup>1(X) <sup>∧</sup> <sup>s</sup>2(X ) over the disjunction present in the definition of <sup>¬</sup>*LZZ*<sup>s</sup>1,*<sup>f</sup>* ,s1∨s<sup>2</sup> (Z) (Eq. (17)), and then over possible pairs of states (Eq. (18)). We rename the two disjuncts in Eq. (18) as *InsExpl<sup>f</sup>* (X, X , Z) and *OutExpl<sup>f</sup>* (X, X , Z)) (Eq. (19)). The formulas *InsExpl<sup>f</sup>* (X, X , Z) and *OutExpl<sup>f</sup>* (X, X , Z)) still enumerate explicitly the abstract states. However, each of these formulas is a conjunction of predicates, application of the In*<sup>f</sup>* operator to a conjunction of predicates, and negations of the application of In*<sup>f</sup>* .

$$\begin{aligned} T\_h(X, X', Z) &\doteq \\ \exists Z. \quad \bigvee\_{(s\_1, s\_2) \in 3^h} (s\_1(X) \land s\_2(X') \land \neg LZZ\_{s\_1, f, s\_1 \lor s\_2}(Z)) \\ \Longleftrightarrow \exists Z. \quad \bigvee\_{(s\_1, s\_2) \in 3^h} \left( \begin{array}{c} s\_1(X) \land s\_2(X') \land \left( (s\_1(Z) \land In\_{f, s\_2}(Z) \land \neg In\_{f, s\_1}(Z)) \lor \\ & (\neg s\_1(Z) \land s\_2(Z) \land In\_{-f, s\_1}(Z)) \right) \end{array} \right) \end{aligned}$$

$$\iff \exists Z. \quad \bigvee\_{\{s\_1, s\_2\} \in 3^{\mathbb{A}}} \left( \begin{pmatrix} (s\_1(X) \land s\_2(X') \land s\_1(Z) \land In\_{f, s\_2}(Z) \land \neg In\_{f, s\_1}(Z)) \lor \\ ((s\_1(X) \land s\_2(X') \land \neg s\_1(Z) \land s\_2(Z) \land In\_{-f, s\_1}(Z)) \end{pmatrix} \right) \tag{17}$$

(16)

$$\iff \exists Z. \begin{pmatrix} \mathsf{V}\_{\left(\mathsf{s}\_{1},\mathsf{s}\_{2}\right) \in \mathsf{s}^{\mathsf{h}}} \left( \mathsf{s}\_{1}(X) \wedge \mathsf{s}\_{2}(X') \wedge \mathsf{s}\_{1} \wedge \mathsf{In}\_{f,\mathsf{s}\_{2}}(Z) \wedge \neg \mathsf{In}\_{f,\mathsf{s}\_{1}}(Z) \right) \vee \\\ V\_{\left(\mathsf{s}\_{1},\mathsf{s}\_{2}\right) \in \mathsf{g}^{\mathsf{h}}} \left( \mathsf{s}\_{1}(X) \wedge \mathsf{s}\_{2}(X') \wedge \neg \mathsf{s}\_{1} \wedge \mathsf{s}\_{2} \wedge \mathsf{In}\_{-f,\mathsf{s}\_{1}}(Z) \right) \end{pmatrix} \tag{18}$$

$$\iff \exists Z. (InsExpl\_f(X, X', Z) \lor OutExpl\_f(X, X', Z)).\tag{19}$$

We now show how we obtain a formula *InsExpl<sup>f</sup>* (X, X , Z) with a linear size. We expand the definition of the formula *InsExpl<sup>f</sup>* (X, X , Z) with respect to the predicates in s<sup>1</sup> and s2. Recall that each abstract state is a conjunction of predicates obtained from the set of polynomial A (i.e., s =˙ <sup>a</sup>∈<sup>A</sup> a <sup>a</sup> 0, <sup>a</sup><sup>∈</sup> {>, <, <sup>=</sup>}) and that we use a <sup>0</sup> <sup>∈</sup> <sup>s</sup> to enumerate the predicates in <sup>s</sup>.

$$\operatorname{Ind} \operatorname{Expl}\_f(X, X', Z) \doteq \bigvee\_{s\_1, s\_2 \in \mathbb{3}^\lambda} \left( \bigwedge\_{a \diamond 0 \in s\_1} a(X) \bowtie 0 \land \bigwedge\_{a \diamond 0 \in s\_2} a(X') \bowtie 0 \land \newline (20) \right)$$

$$\bigwedge\_{a \diamond 0 \in s\_1} a(Z) \bowtie 0 \land \bigwedge\_{a \diamond 0 \in s\_2} \operatorname{In}\_{f, a \diamond 0}(Z) \land$$

$$\bigvee\_{a \diamond 0 \in s\_1} \neg \operatorname{In}\_{f, a \diamond 0}(Z) \Big).$$

In the above formula, we used De Morgan rules to rewrite the formula a0∈s<sup>1</sup> *In<sup>f</sup>* ,a0(Z) as the formula a0∈s<sup>1</sup> <sup>¬</sup>*In<sup>f</sup>* ,a0(Z). We express the formula *InsExpl<sup>f</sup>* (X, X , Z) as an enumeration of the predicates, over the variables X and X , determining the abstract states s<sup>1</sup> and s2, instead of the pairs of abstract states:

$$\operatorname{Ins} \operatorname{Symb}\_f(X, X', Z) \doteq \bigwedge\_{a \in \mathbb{A}, \mathbb{M} \bowtie \{>, <, =\}} \left( a(X) \bowtie 0 \to a(Z) \bowtie 0 \right) \wedge \tag{21}$$
 
$$\bigwedge\_{a \in \mathbb{A}, \mathbb{M} \bowtie \{>, <, =\}} \left( a(X') \bowtie 0 \to \operatorname{In}\_{f, a \lhd 0}(Z) \right) \wedge$$
 
$$\bigvee\_{a \in \mathbb{A}, \mathbb{M} \not\in \{>, <, =\}} \left( a(X) \bowtie 0 \land \left( \neg \operatorname{In}\_{f, a \lhd 0}(Z) \right) \right).$$

**Lemma 1.** *InsExpl<sup>f</sup>* (X, X , Z) *and InsSymb<sup>f</sup>* (X, X , Z) *are equivalent.*

*Proof (sketch).*

<sup>⇒</sup>) We show that <sup>μ</sup> <sup>|</sup><sup>=</sup> *InsExpl<sup>f</sup>* (X, X , Z) implies <sup>μ</sup> <sup>|</sup><sup>=</sup> *InsSymb<sup>f</sup>* (X, X , Z). Since <sup>μ</sup> <sup>|</sup><sup>=</sup> *InsExpl<sup>f</sup>* (X, X , Z) we have that μ is an interpretation for one of the disjuncts on the possible pairs of states of *InsExpl<sup>f</sup>* (X, X , Z):

$$\bigwedge\_{a\bowtie 0 \in s\_1} a(X) \bowtie 0 \land \bigwedge\_{a\bowtie s\_2} a(X') \bowtie 0 \land \bigwedge\_{a\bowtie s\_1} a(Z) \bowtie 0 \land \bigwedge\_{a\bowtie s\_1} a(Z) \bowtie 0 \land \bigwedge\_{a\bowtie s\_1} a(Z) \bowtie 0 \land \bigwedge\_{a\bowtie s\_1} \neg In\_{f,a\bowtie 0}(Z).$$

$$\bigwedge\_{a\bowtie s\_2} In\_{f,a\bowtie s\_2}(Z) \land \bigvee\_{a\bowtie s\_1} \neg In\_{f,a\bowtie 0}(Z).$$

Hence, there exist two (and exactly two) abstract states s1, s2, such that <sup>μ</sup> <sup>|</sup><sup>=</sup> <sup>s</sup>1(X) and <sup>μ</sup> <sup>|</sup><sup>=</sup> <sup>s</sup>2(X ). This means that any predicate a <sup>0</sup> <sup>∈</sup> <sup>s</sup><sup>1</sup> is such that <sup>μ</sup> <sup>|</sup><sup>=</sup> a (X) and similarly for predicates not in the state <sup>s</sup><sup>2</sup> for the variables <sup>X</sup> (recall that, given a polynomial <sup>a</sup> <sup>∈</sup> <sup>A</sup>, the possible abstraction predicates a > 0, a < 0, and a = 0 are mutually exclusive). We show that μ is an interpretation for all the conjuncts in *InsSymb<sup>f</sup>* (X, X , Z). We have that <sup>μ</sup> <sup>|</sup><sup>=</sup> <sup>a</sup>∈A,∈{>,<,=} <sup>a</sup>(X) <sup>0</sup> <sup>→</sup> <sup>a</sup>(Z) <sup>0</sup> since for all <sup>a</sup> <sup>∈</sup> <sup>A</sup>, <sup>μ</sup> <sup>|</sup><sup>=</sup> <sup>a</sup>(X) <sup>0</sup> <sup>→</sup> <sup>a</sup>(Z) 0 (when <sup>a</sup> <sup>∈</sup> <sup>s</sup><sup>1</sup> we have <sup>μ</sup> <sup>|</sup><sup>=</sup> <sup>a</sup>(Z) 0, while when <sup>a</sup> <sup>∈</sup> <sup>s</sup><sup>1</sup> the implication trivially holds). Similarly, this happens for <sup>a</sup>∈A,∈{>,<,=} <sup>a</sup>(X) <sup>0</sup> <sup>→</sup> <sup>a</sup>(Z) <sup>0</sup> . We can see the disjunction: <sup>a</sup>∈A,∈{>,<,=} <sup>a</sup>(X) <sup>0</sup> <sup>∧</sup> (¬*In<sup>f</sup>* ,a0(Z)) as: a0∈s<sup>1</sup> <sup>a</sup>(X) <sup>0</sup> <sup>∧</sup> (¬*In<sup>f</sup>* ,a0(Z)) <sup>∨</sup> a0 ∈s<sup>1</sup> <sup>a</sup>(X) <sup>0</sup> <sup>∧</sup> (¬*In<sup>f</sup>* ,a0(Z)) .

We have that μ satisfies the first disjunct (and hence the whole disjunction) because when a <sup>0</sup> <sup>∈</sup> <sup>s</sup><sup>1</sup> we have that <sup>μ</sup> <sup>|</sup><sup>=</sup> a0∈s<sup>1</sup> <sup>¬</sup>*In<sup>f</sup>* ,a0(Z).

⇐) We show that <sup>μ</sup> <sup>|</sup><sup>=</sup> *InsSymb<sup>f</sup>* (X, X , Z) implies <sup>μ</sup> <sup>|</sup><sup>=</sup> *InsExpl<sup>f</sup>* (X, X , Z). As before, we notice that are only two predicates <sup>s</sup>1, s<sup>2</sup> such that <sup>μ</sup> <sup>|</sup><sup>=</sup> <sup>s</sup>1(X) and <sup>μ</sup> <sup>|</sup><sup>=</sup> <sup>s</sup>2(X ) and that all the predicates not in s<sup>1</sup> and not in s<sup>2</sup> do not hold in μ. Thus, from <sup>μ</sup> <sup>|</sup><sup>=</sup> *InsSymb<sup>f</sup>* (X, X , Z) we have that

<sup>μ</sup> <sup>|</sup><sup>=</sup> <sup>a</sup>∈s<sup>1</sup> <sup>a</sup>(Z) <sup>0</sup> <sup>∧</sup> <sup>a</sup>∈s<sup>2</sup> *In<sup>f</sup>* ,a0(Z) <sup>∧</sup> <sup>a</sup>∈s<sup>1</sup> <sup>¬</sup>*In<sup>f</sup>* ,a0(Z).

Hence, μ is a model for at least one of the disjuncts in *InsExpl<sup>f</sup>* (X, X , Z). We similarly define the compact encoding of *OutExpl<sup>f</sup>* (X, X , Z):

$$OutSymb\_f(X, X', Z) \doteq \bigwedge\_{a \in \mathbb{A}, \mathbb{A} \models \{>, <, =\}} \left( a(X) \bowtie 0 \to \operatorname{In}\_{-f, a \bowtie 0}(Z) \right) \wedge \qquad (22)$$

$$\bigwedge\_{a \in \mathbb{A}, \mathbb{A} \models \{>, <, =\}} \left( a(X') \bowtie 0 \to a(Z) \bowtie 0 \right) \wedge$$

$$\bigvee\_{a \in \mathbb{A}, \mathbb{A} \not\in \{>, <, =\}} \left( a(X) \bowtie 0 \land \neg a(Z) \bowtie 0 \right) \cdot$$

**Lemma 2.** *OutExpl<sup>f</sup>* (X, X , Z) *and OutSymb<sup>f</sup>* (X, X , Z) *are equivalent. Proof.* The proof of Lemma 2 is similar to the proof of Lemma 1.

We now express the transition relation from Eq. (7) in a compact form:

T -*Symb*P(VP, V <sup>P</sup>) ˙= <sup>∃</sup>X, X . - N(X, X ) <sup>∧</sup> <sup>H</sup>(X) <sup>∧</sup> <sup>H</sup>(X )∧ (23) <sup>H</sup>A(X, VP) <sup>∧</sup> <sup>H</sup>A(X , V <sup>P</sup>)∧ <sup>∃</sup>Z.(*InsSymb<sup>f</sup>* (X, X , Z) <sup>∨</sup> *OutSymb<sup>f</sup>* (X, X , Z)) .

**Theorem 2.** <sup>T</sup><sup>P</sup>(VP, V <sup>P</sup>) *and* T -*Symb*P(VP, V <sup>P</sup>) *are equivalent. Proof.* Follows directly from Lemma 2 and Lemma 1.

#### **Implicit Semi-Algebraic Abstraction**

The formula T -*Symb*P(VP, V A) represents the transition relation of the semialgebraic abstraction. Computing the finite-state transition system representing the semi-algebraic abstraction requires to eliminate the existential quantifiers from the initial states, transition relation, and safety property formulas. However, the above formula T -*Symb*P(VP, V <sup>A</sup>) contains non-linear real arithmetic terms from the polynomials and the Lie derivatives we compute in In*<sup>f</sup>* , so removing the quantifiers from the formula requires to apply a quantifier elimination algorithm (e.g., Cylindrical Algebraic Decomposition [8]) that does not scale, even when the number of polynomials is small. Instead, we construct a symbolic transition system that *implicitly* encodes the abstraction:

$$S\_{Impl, \mathbb{P}} \doteq \langle X \cup \overline{X} \cup Z, \psi(X) \wedge EQ\_{\mathbb{P}}(X, \overline{X}), T\_{Impl, \mathbb{P}}(\overline{X}, X', Z) \wedge EQ\_{\mathbb{P}}(X', \overline{X'}) \rangle,$$

where

$$\begin{aligned} T\_{\operatorname{Impl},\mathbb{P}}(\overline{X},X',Z) &\doteq N(\overline{X},X') \wedge H(\overline{X}) \wedge H(X') \wedge \\ &(InsSymb\_f(\overline{X},X',Z) \vee OutSymb\_f(\overline{X},X',Z)) .\end{aligned}$$

The above encoding is a an implicit predicate abstraction [35] that preserves reachability properties and is such that:

**Theorem 3.** <sup>S</sup>*Impl*,<sup>P</sup> <sup>|</sup><sup>=</sup> <sup>P</sup>(X) *if and only if* <sup>S</sup><sup>P</sup> <sup>|</sup><sup>=</sup> ¬¬ PP(VP)*.*

Thus, we can model check the transition system <sup>S</sup>*Impl*,<sup>P</sup> <sup>|</sup><sup>=</sup> <sup>P</sup>(X) to prove a property P holds on the dynamical system. Note that, to this purpose, we can apply standard SMT-based model checking algorithms.

The transition system S*Impl*,<sup>P</sup> doubles the state space introducing a copy of the state variables X and encodes the equivalence relation between pairs of concrete states in X and in X with the formula *EQ*P(X, X) (c.f. Formula 6). The transition relation T*Impl*,P(X,X , Z) then encodes a transition in the semi-algebraic abstraction with the linear encoding *InsSymb<sup>f</sup>* (X,X , Z) and *OutSymb<sup>f</sup>* (X,X , Z). In this way, a transition in the transition system S*Impl*,<sup>P</sup> corresponds to a transition in the semi-algebraic abstraction, and vice-versa.

#### **6 Experimental Evaluation**

#### **Research Questions**

We evaluate the performance of our approach (*Implicit Abstraction*) for the verification of invariant properties on the semi-algebraic abstraction of dynamical systems. *Implicit Abstraction* first encodes the semi-algebraic abstraction in a transition system (as we show in Sect. 5), and then model checks the invariant on the transition system with an off-the-shelf model checker. Our experiments aim to answer the following research questions:

**RQ 1**: How does *Implicit Abstraction* compare with the *LazyReach* algorithm [32], which explicitly enumerates the reachable states of the abstraction? **RQ 2**: How does *Implicit Abstraction* compare with the *DWCL* algorithm [32], which applies a divide-and-conquer strategy to reduce the number of polynomials in the abstraction?

#### **Experimental Setup**

We implemented the construction of the implicit abstraction transition system in Python using *PySMT* [11] to manipulate formulas, and *SymPy* [23] for polynomial manipulation and Gr¨obner bases computation (i.e., to compute the Lie derivatives' ranks). We verify the implicit abstraction transition system with the model checking algorithm for symbolic transition systems with NRA constraints from [4]. The algorithm abstracts the non-linear transition system into a linear transition system, which is checked by the algorithm in [6] and is implemented using the *MathSAT* [7] SMT solver. We implemented both the *LazyReach* and the *DWCL* algorithms in the same Python tool. Our implementation of *DWCL* can use different backends to decide the satisfiability of NRA formulas, namely *MathSAT*<sup>6</sup>, the *z3* SMT solver [25], or *Mathematica* [17].

We consider 90 invariant verification problems for dynamical systems from the *KeyMaera X* theorem prover [10]. These problems are a superset of the

<sup>6</sup> *MathSAT* uses a different decision procedure [4] than *z3* and *Mathematica* based on incremental linearization rather than cylindrical algebraic decomposition.

ones used in [32] and are used in the Applied Verification of Continuous and Hybrid Systems (ARCH) competition [24]. We obtain a total of 180 benchmark instances using, for each problem, two sets of polynomials for the semi-algebraic abstraction<sup>7</sup>. The first set contains all the factors of the right-hand side of the ODEs; the second set extends the first one by including also the Lie derivatives of the polynomials. The latter set induces an abstraction that is more precise but also has a larger state-space.

We evaluate the performance of the algorithms *Implicit Abstraction*, *LazyReach*, and *DWCL* to solve the above verification problems. The underlying problem requires to decide the satisfiability of NRA formulas, and the decision procedures for this problem are efficient for different subsets of problems. For this reason we further evaluate different configurations of the *LazyReach* and *DWCL* algorithms using three different solvers for NRA formulas (*MathSAT*, *z3* , and *Mathematica*). Note that, while in principle, we could use multiple SMT backends also in the model checking algorithm [4] and replace the *MathSAT* SMT solver with another SMT solver (e.g., *z3* ), this change would not significantly impact the overall performance, because the algorithm abstracts the non-linear formulas with linear ones where both *MathSAT* and *z3* have comparable performance.

We run the *Implicit Abstraction*, *LazyReach*, and *DWCL* algorithm on all the 180 benchmark instances with a time out of 100 seconds, and we measure the execution times to either prove (*safe* result) or find an abstract counterexample (*unknown* result) for each instance. An archive containing the necessary to reproduce the experiments is available online at http://www.sergiomover.eu/ cav2021.html.

#### **Results**

*RQ 1* - Implicit Abstraction vs. LazyReach. From the cumulative plot in Fig. 2, we see that *Implicit Abstraction* almost always outperforms *LazyReach*.

From the cumulative plot in Fig. 2a we see that *Implicit Abstraction* significantly outperforms *LazyReach* on *safe* instances. For better readability, in the plot we only show the (virtual) portfolio algorithm running each configuration of *LazyReach*, *Virtual Best LazyReach*, obtained by considering the best run time among the different configurations of *LazyReach* using different backend solvers. *Virtual Best LazyReach* solves a total of 42 safe instances, while *Implicit Abstraction* solves 100 safe instances. The scatter plots shown in the first row of Fig. 3 confirms the same intuition (note that the safe instances represented as blue circles are mostly in the lower-right triangle of the plot).

Figure 2b shows the cumulative plot when verifying *unknown* instances. Note that the total number of unknown instances in the benchmarks are much smaller than the safe ones (combining the results of all the algorithms we have 123 safe instances, 19 unknown instances, and 38 still unsolved instances). From

<sup>7</sup> The benchmarks have 321 sign-invariant polynomials (c.f. Sect. 2) over a total of 1089 polynomials that *DWCL* will use to split the state space.

Fig. 2b, we see that the performance of *Implicit Abstraction* is comparable with *LazyReach*, solving a total of 8 instances and 11 instances respectively.

*RQ 2* - Implicit Abstraction vs. DWCL. From the cumulative plots in Fig. 2, the *Virtual Best DWCL* solves 37 more instances than *Implicit Abstraction*. However, we also see from Fig. 2 that the global *Virtual Best* solves more instances and is faster than *Virtual Best DWCL*. In fact, *Implicit Abstraction* is *orthogonal* to *DWCL* and is comparable to *DWCL* when fixing either *Mathematica* or *z3* (*Implicit Abstraction* solves 108 instances, *DWCL Mathematica* solves 109, and *DWCL z3* solves 114).

The scatter plots in the second row of Fig. 3 compare *Implicit Abstraction* with *DWCL MathSAT*, *DWCL Mathematica*, and *DWCL z3* . From these plots, we see that there are several instances that are solved by only one of the two algorithms compared in each plot. While we see similar data when comparing *Implicit Abstraction* with *Virtual Best DWCL* (always in the scatter plots of Fig. 3), the number of instances solved uniquely by *Implicit Abstraction* seems smaller. We get a more precise picture of the complementarity of *Implicit Abstraction*, *DWCL Mathematica*, and *DWCL z3* from the diagrams in Fig. 4, where we can clearly see that *Implicit Abstraction* is orthogonal to both *DWCL Mathematica* and *DWCL z3* . From the diagram, we see that when using a different backend (i.e., *Mathematica* or *z3* ) *DWCL* solves a different set of instances. This difference in performance using *Mathematica* and *z3* is not surprising since *Mathematica* and *z3* uses different algorithms to solve formulas in NRA.

We further notice that *Implicit Abstraction* uses the *MathSAT* SMT solver in the backend, and from our experiments (see again Fig. 3) *DWCL MathSAT* performs quite poorly compared to both *DWCL Mathematica* and *DWCL z3* . While naively replacing *MathSAT* in the model checking algorithm we use [4] would not provide a significant performance improvement, it is reasonable to think that investigating a tighter integration with either *z3* or *Mathematica* could improve the model checking performance. However, we believe this integration to be beyond the scope for this paper, where we enable the use of symbolic model checking techniques to analyze the semi-algebraic decomposition.

#### **7 Related Work**

In this work, we focus on the (unbounded time) safety verification problem for polynomial dynamical systems. Such problem is relevant when proving safety for hybrid programs [27] with Keymaera X [10] or for hybrid CPS with the HHL Prover [36]. Our reduction to transition systems may be used as sub-procedure in both theorem provers to automate the search of a continuous invariant.

There exist different techniques to prove safety properties for polynomial dynamical systems (see e.g., [13]): barrier certificates [18,29], first integrals [14], and Darboux Polynomials [15]. All these techniques are orthogonal to semialgebraic abstraction, and can be used to find invariant polynomials to restrict the abstract state space. Pegasus [33] implements all the above techniques, the *LazyReach*, and *DWCL* algorithms. Our algorithm can be integrated in Pegasus.

**Fig. 2.** Plots the total number of instances (on the y axis) as a function of the cumulative time (in seconds, on the x axis) took by *Implicit Abstraction*, *LazyReach*, and *DWCL* to solve (a) *safe* and (b) *unknown* instances. The comparison includes the results of *LazyReach* and *DWCL* using different (*MathSAT*, *z3* , and *Mathematica*), as well as virtual portfolios combining the best results obtained by a given algorithm when run with multiple backends. We omit some configurations in (b) to improve readability.

The LZZ [22] procedure has been originally proposed to synthesize a continuous invariant. Instead, we use the LZZ procedure to encode the abstract transition relation, and then we prove a safety property in the abstraction. We also provide a specialized encoding of LZZ to check the existence of abstract transitions.

The semi-algebraic abstraction [32] is a qualitative abstraction [34,37]. In this work, we propose a different algorithm to verify semi-algebraic abstractions that allows us to explore the abstract state-space symbolically, in contrast to the *LazyReach* algorithm [32]. In principle, our technique is orthogonal to the *DWCL* algorithm [32], since we could replace *LazyReach*, which is used in *DWCL* as a sub-routine, with our approach (i.e., model check the implicit abstraction).

Relational abstraction [31] abstracts the dynamical system's trajectories with a discrete transition relation, reducing the verification problem on the continuous system to a verification problem on the discrete system. The implicit encoding of the semi-algebraic abstraction can be seen as an instance of relational abstraction, where a trajectory of the dynamical system is mapped to a sequence of abstract transitions (similarly to what happen with relational abstractions for time-sampled systems in [2,38]). Since relational abstractions can be composed with each other (e.g., see [26]), we can strengthen the implicit semi-algebraic abstraction encoding with a relational abstraction. This composition is useful in the case the semi-algebraic abstraction cannot easily capture the system's behavior (e.g., a precise relation of the time elapsed in a transition [26]).

**Fig. 3.** Scatter plots comparing the run time (in seconds) of *Implicit Abstraction* (on the y axis) with *LazyReach* (first row, on the x axis) and *DWCL* (second row, on the x axis). Blue circles represent safe verification problems. Red crosses are instances where the algorithm found an abstract counterexample. When *Implicit Abstraction* runs for more than the 100 s time out, we plot the instance on the vertical line marked as *to*, and similarly for *LazyReach* and *DWCL* on the horizontal line.

**Fig. 4.** Diagrams representing the distribution of unique instances solved combining different algorithms (*DWCL Mathematica*, *DWCL z3* , and *Implicit Abstraction*). Each set, displayed as a dotted circle enclosed by a dotted line, represents the set of instances solved with one algorithm. The number shown in each partition is the number of instances solved uniquely by the sets forming the partition. For example, the central partition (i.e., the intersection of all the sets) of the diagram (a) shows that *DWCL Mathematica*, *DWCL z3* , and *Implicit Abstraction* solved the same set of 141 instances.

Predicate abstraction [16] is a commonly used abstraction techniques to verify infinite-state systems. Several symbolic techniques [3,19,20] focus on the efficient computation of the predicate abstraction. In principle, we can also use those technique to explicitly compute the semi-algebraic abstraction. However, the up-front, explicit computation of the abstraction is a bottleneck and can be avoided with implicit predicate abstraction [35] when the goal is to verify a safety property on the abstract system. We use implicit abstraction to obtain an implicit encoding of the semi-algebraic abstraction. The transition system of the semi-algebraic abstraction contains NRA formulas (the polynomials can be non-linear or the Lie derivative of the polynomials are non-linear). While there are few algorithms and tool that can verify such transition systems (e.g., [4]), our technique is agnostic to the underlying model checking algorithm.

#### **8 Conclusions and Future Work**

In this paper, we addressed the safety problem of polynomial dynamical systems. We built on the LZZ algorithm to define a symbolic encoding of the abstraction based on a set of polynomials. The encoding is linear in the number of polynomials and can be used to implicitly represent the abstraction without the need of enumerating the abstract states, enabling the use of SMT-based model checking techniques. The experimental evaluation showed that the approach is promising and complementary to existing techniques solving a number of new instances.

The main directions for future works are, on one side, refining the abstraction discovering new polynomials that are able to remove spurious abstract counterexamples, and, on the other side, the application of the approach to hybrid systems where the continuous dynamics depends on the discrete state of the system.

**Acknowledgements.** S. Mover was partially supported by the academic chair "Complex Systems Engineering", Ecole Polytechnique-ENSTA Paris-T´ ´ el´ecom Paris-Thal´es-Dassault Aviation-Naval Group-DGA-Foundation ParisTech-FX and the AID project "Validation of Autonomous Drones and Swarms of Drones". A. Cimatti, A. Griggio, and S. Tonetta were partially supported by ECSEL JU under grant agreement No 876852. The JU receives support from EU's H2020 programme, Austria, Czech Republic, Germany, Ireland, Italy, Portugal, Spain, Sweden, Turkey.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **IMITATOR 3: Synthesis of Timing Parameters Beyond Decidability**

Etienne Andr´ ´ <sup>e</sup>(B)

Universit´e de Lorraine, CNRS, Inria, LORIA, 54000 Nancy, France Andre.Etienne@lipn13.fr

**Abstract.** Real-time systems are notoriously hard to verify due to nondeterminism, concurrency and timing constraints. When timing constants are uncertain (in early the design phase, or due to slight variations of the timing bounds), timed model checking techniques may not be satisfactory. In contrast, parametric timed model checking synthesizes timing values ensuring correctness. IMITATOR takes as input an extension of parametric timed automata (PTAs), a powerful formalism to formally verify critical real-time systems. IMITATOR extends PTAs with multi-rate clocks, global rational-valued variables and a set of additional useful features. We describe here the new features and algorithms offered by IMITATOR 3, that moved along the years from a simple prototype dedicated to robustness analysis to a standalone parametric model checker for timed systems.

**Keywords:** Parametric timed automata · Parameter synthesis · Real-time systems

#### **1 Introduction**

Real-time systems are often used in critical environments, and may be verified using formal methods. Such systems are notoriously hard to verify due to nondeterminism, concurrency and timing constraints. Timed model checking provides designers with techniques to formally verify a real-time system. However, timed model checking may not always be fully satisfactory: First, in the early design phase, timing constants may not be known and, without them, model checking is not possible; Second, at runtime, timing constants may vary (due to uncertain bounds, or to processor clock drifts), in which case the model checking result may not hold anymore. In contrast, parametric timed model checking *synthesizes* timing values ensuring the system correctness.

Parametric timed automata (PTAs) are a powerful formalism to reason about, and formally verify critical real-time systems [5]. PTAs are finite state

This work is partially supported by the ANR-NRF French-Singaporean research program ProMiS (ANR-19-CE25-0015).

c The Author(s) 2021 A. Silva and K. R. M. Leino (Eds.) CAV 2021, LNCS 12759, pp. 552–565, 2021. https://doi.org/10.1007/978-3-030-81685-8\_26

**Fig. 1.** Examples of graphical outputs

automata extended with clocks, i.e., real-valued variables evolving linearly, that can be compared with either integer constants or parameters in guards (constraints to take a transition) and invariants (constraints to remain in a location).

IMITATOR takes as input networks of "IMITATOR PTAs" (IPTAs) extending PTAs with several convenient features such as stopwatches, multi-rate clocks or global shared rational-valued variables.

IMITATOR answers variants of the following problem:


IMITATOR answers this problem by synthesizing sets of parameter valuations in the form of a finite disjunction of linear constraints over the parameters.

IMITATOR is a command-line only tool; its input is text-based (partially inspired by HyTech syntax [41]) and is "human-readable", different from, e.g., XML. IMITATOR produces standardized result files (that can be possibly parsed from external tools), and can produce graphical outputs, such as in Fig. 1.

The expressive power (i.e., ease to write a complicated model in a compact manner) of the tool has been largely improved since IMITATOR 2.5 [17], and IMITATOR is now a parametric timed model checker taking as inputs a model and a property, implementing various synthesis algorithms.

#### **2 An Expressive Input Language**

*Parametric Timed Automata (PTAs).* Timed automata (TAs) [3] extend finitestate automata with *clocks*, i.e., real-valued variables evolving at the same rate 1, that can be compared to integers along edges ("guards") or within locations ("invariants"). Clocks can be reset (to 0) along transitions. PTAs extend TAs with (timing) parameters, i.e., unknown rational-valued constants [5].

*Example 1.* In the model in Fig. 2 (that goes beyond the syntax of PTAs, see Example 2), there are four locations, depicted as rounded rectangles. Invariants are depicted using dotted rectangles. In the invariant of location working, clock x is compared to parameter ptotal . The guard of the transition from coffee to working compares clock t to pcoffee ; this clock t is reset to 0 along this transition.

IMITATOR *Parametric Timed Automata (IPTAs).* IMITATOR takes as input models described as networks of IMITATOR parametric timed automata (IPTAs). IPTAs extend PTAs with a set of useful features, described in the following.

*Global Rational-Valued Variables.* Global variables (called "discrete") can be defined, and are part of the discrete part of a state, together with locations (and different from clocks and parameters that are part of the *continuous* part). Global variables in IMITATOR are exact rationals, following exact arithmetics (as opposed to, e.g., floating-point arithmetic that can accumulate errors and lead to faulty assertions). Exact rationals are encoded in IMITATOR using the GNU MP library. Such discrete variables can be updated along transitions, and can also be part of the clock guards and invariants; in fact, virtually any linear expression over clocks, parameters and discrete variables can be used in guards, invariants and updates. Non-linear arithmetic expressions over sole discrete variables are allowed too.

*Automata Synchronization.* IPTAs can be synchronized together on shared actions, or by reading shared variables. All variables (clocks, parameters, discrete) are potentially global in IMITATOR. This allows users to define models component by component.

*Arbitrary Flows.* Since version 3.0, IMITATOR supports arbitrary (constant) flows for clocks; this way, clocks do not necessarily evolve at the same time, and can encode different concepts from only time: temperature, amount of completion, continuous cost... Their value can increase or decrease at any predefined rate in each location, and can become negative. In that sense, IMITATOR's clocks are closer to *continuous variables* (as in hybrid automata) rather than TAs' clocks; nevertheless, we keep the name *clock* for sake of backward-compatibility. This makes IMITATOR support a parametric extension of *multi-rate automata* [2]. This notably includes stopwatches, where clocks can have a 1- or 0-rate [36].

*Additional Syntax Improvements.* Beyond the aforementioned increase of the syntactic expressive power, the syntax was enhanced with accepting locations (that can be used in properties), global constants, "if... then... else" conditions in updates, and with the ability to include model fragments from different

**Fig. 2.** An IPTA example: writing papers and drinking coffee

files (new syntax #include(modelpart.imi)). Several simplifications were made to the syntax to keep it "human-readable". For example, location workingFast of Fig. 2 is written in IMITATOR syntax as follows:

$$\mathbf{\color{red}{1}\mathbf{\color{red}{o}}}\quad\mathbf{\color{red}{w}\mathbf{\color{red}{k}\mathbf{\color{red}{S}}}\mathbf{\color{red}{Fast}}}\mathbf{\color{red}{x}}\mathbf{\color{red}{i}}\quad\mathbf{\color{red}{n}\mathbf{\color{red}{x}\mathbf{\color{red}{i}}}\mathbf{\color{red}{x}}\mathbf{\color{red}{e}}}\mathbf{\color{red}{x}} \leq \mathbf{\color{red}{p}\mathbf{\color{red}{T}}\mathbf{\color{red}{a}}\mathbf{1}}\mathbf{\color{red}{f}\mathbf{\color{red}{o}}\mathbf{\color{red}{x}}\mathbf{\color{red}{x}}\mathbf{\color{red}{f}} = \mathbf{\color{red}{2}}\mathbf{\color{red}{f}}$$

*Translations.* Finally, translations of the model are available to other model checkers such as HyTech [41] and Uppaal [42] (in both cases, not all features can be translated since some of the features of IMITATOR do not exist in the target tool, e.g., Uppaal does not support parameters nor complex linear constraints over clocks (only "diagonal")). Graphical translations of the model are also available to JPEG, PDF and LATEX formats.

*Example 2.* Consider the IPTA in Fig. 2, modeling a researcher writing papers. The model features two clocks t (measuring the time when needing a coffee) and x (measuring the amount of work done on a given paper), both initially 0. Their rate is always 1, unless otherwise specified (e.g., in workingFast). Initially, the researcher is working (location working) on a paper, requiring an amount of work ptotal . When the paper is completed (guard x = ptotal), the IPTA moves to location finished. From there, at any time, the researcher can start working on a new paper (transition back to working, updating x and t).

Alternatively, after at least a certain time (guard <sup>t</sup> <sup>≥</sup> <sup>p</sup>need ), the researcher may need a coffee; this action can only be taken until a maximum number of coffees have been drunk for this paper (*nb* ≤ *max* − 1), where *nb* is a discrete global variable recording the number of coffees drunk while working on the current paper. When drinking a coffee (location coffee), the work is obviously not progressing ( ˙x = 0). Drinking a coffee takes exactly pcoffee time units (guard t = pcoffee back to location working). Observe that, from the second paper onwards (transition labeled with *restart*), the researcher is already half-way of her/his need for a coffee (parametric update <sup>t</sup> <sup>←</sup> <sup>0</sup>.<sup>5</sup> <sup>×</sup> <sup>p</sup>need [22]).

Also, whenever 80% of the work is done (guard <sup>x</sup> <sup>≥</sup> <sup>0</sup>.8×ptotal), the researcher may work twice as fast (location workingFast, with a rate 2 for clock x). In that case, (s)he needs a coffee faster too (0.<sup>6</sup> <sup>×</sup> <sup>p</sup>need ).

All three durations pcoffee , pneed and ptotal are timing parameters. We fix their parameter domains as follows: <sup>p</sup>coffee , <sup>p</sup>total <sup>∈</sup> [0,∞) and <sup>p</sup>need <sup>∈</sup> [1,∞). The maximum number of coffees *max* <sup>∈</sup> [0,∞) is also a parameter; observe that it is (only) compared to the discrete variable *nb*, and therefore can be seen as a "discrete parameter"—which is allowed by the liberal syntax of IMITATOR.

The example in Fig. 2 could not be modeled with Uppaal due to the presence of timing parameters, stopwatches, multi-rate clocks and non-0 update. It may be modeled using HyTech; however, most algorithms implemented in IMITATOR (even the most basic ones, such as liveness synthesis) do not exist in HyTech, as HyTech mainly focuses on basic state space computation.

#### **3 A Variety of Synthesis Algorithms**

The formalism of networks of IPTAs is "highly undecidable" for most problems. Indeed, while several problems are decidable for timed automata (notably the reachability [3]), most interesting problems become undecidable in the presence of timing parameters [5,8] , notably when such parameters are unbounded [35]. On top of this, multi-rate automata together with linear constraints over the clocks also yield undecidability [2]. Finally, the mere use of stopwatches, even without the aforementioned extensions, brings undecidability [36]. Also note that, in contrast to several existing model checkers, IMITATOR offers the use of unbounded rational variables, therefore with an infinite domain. For all these reasons, it is always possible to find examples of IPTAs for which the algorithms implemented in IMITATOR would not terminate with an exact (sound and complete) result. The rational behind IMITATOR is therefore to follow a "best-effort" approach, by:


IMITATOR outputs a standardized result (in a text file), that contains the synthesized constraint with a set of information, and notably the *validity* of the constraint, i.e., whether the set of valuations is *exact* (sound and complete), *possibly over-approximated*, *possibly under-approximated*, or *potentially invalid* i.e., when both under-approximating and over-approximating heuristics were used. By default, IMITATOR attempts to synthesize an exact result; only when some specific options are used (e.g., a limit on the number of states explored, or on the computation time), approximations may be used. These approximations are conservative for most algorithms; for example, if an approximation is used for safety synthesis, then the result will be under-approximated (i.e., the system is safe for all synthesized valuations—even though some more safe valuations may exist).

IMITATOR offers two main classes of synthesis: *i) Witness* (or counterexample), which attempts to exhibit at least one parameter valuation satisfying the property; often, IMITATOR still outputs a *symbolic* set of valuations (i.e., a linear constraint over the parameters), but stops the analysis as soon as one such set is found. *ii)* Normal *synthesis*, where IMITATOR attempts to synthesize *all* parameter valuations satisfying the property.

Properties include reachability (denoted by "EF", following the TCTL syntax), safety (denoted by "AGnot"), liveness, deadlock-freeness, robustness, and some others.

Throughout this section, we exemplify the main synthesis algorithms of IMI-TATOR on Example 2. <sup>1</sup> All the results synthesized in the following are exact (sound and complete), unless otherwise specified.

*Safety.* A first algorithm of IMITATOR is safety synthesis, i.e., synthesizing parameter valuations for which a discrete state (location and/or valuation of the discrete variables) is unreachable for all runs. For example, one synthesize the valuations for which it is impossible to drink any coffee, i.e., it is impossible to reach the coffee location of the "researcher" automaton of Fig. 2.

# synth AGnot( loc[researcher] = coffee)

The result is: *max* <sup>∈</sup> [0, 1) <sup>∨</sup> - *max* <sup>≥</sup> <sup>1</sup> <sup>∧</sup> <sup>p</sup>total <sup>&</sup>lt; <sup>p</sup>need 10 

Let us explain this result. The first disjunct is trivial: if the researcher is not allowed to drink any coffee (*max* < 1), the transition to coffee (guarded by "*nb* ≤ *max* −1") can never be taken. The second disjunct is, despite the relative simplicity of this model, less trivial: assume for illustration that pneed = 10 and ptotal = 1, and let us show that the researcher is still able to start drinking a coffee in this situation. After the first paper completion (action restart), we have <sup>x</sup> <sup>←</sup> 0 and <sup>t</sup> <sup>←</sup> 5. After one time unit in location working (<sup>x</sup> = 1 and t = 6), the researcher moves to workingFast, and can immediately move to coffee (guard <sup>t</sup> <sup>≥</sup> <sup>0</sup>.<sup>6</sup> <sup>×</sup> <sup>p</sup>need is now satisfied). This scenario, that can be seen on the parametric state space output by IMITATOR (see Fig. 1a), is also possible for larger values of ptotal . This explains the strict inequality ptotal < *<sup>p</sup>*need <sup>10</sup> .

<sup>1</sup> All finishing executions for our example using IMITATOR 3.0 "Cheese" ea560fd on a Dell XPS 13 7390 Intel- CoreTM i7-10510U CPU @ 1.80 GHz running Linux Mint 20 Ulyana terminate within *<* 1 *s*. All examples and results can be found at [9].

*Reachability.* Reachability can be seen as the opposite of safety, i.e., the goal is to synthesize parameter valuations for which a discrete state is reachable for at least one run. For example, one can ask for the valuations for which it is possible to drink at least one coffee:

# synth EF( loc[researcher] = coffee)

The result is *max* <sup>≥</sup> <sup>1</sup> <sup>∧</sup> <sup>p</sup>total <sup>≥</sup> *<sup>p</sup>*need <sup>10</sup> , which is obviously the complement of the result synthesized for the aforementioned safety property.

One can also synthesize valuations for which it is possible to drink at least five coffees while working on some article (i.e., *nb* ≥ 5).

# synth EF( loc[researcher] = coffee & nb >= 5)

The result is *max* <sup>≥</sup> <sup>5</sup> <sup>∧</sup> <sup>p</sup>total <sup>≥</sup> <sup>37</sup> <sup>10</sup> <sup>×</sup> <sup>p</sup>need .

*Minimum-Time Reachability.* Minimal-time synthesis [12] aims at synthesizing parameter valuations minimizing the time needed to reach a discrete state. Here, we can ask for the valuations for which it is possible to finish an article after drinking at least 2 coffees:

# synth EFtmin( loc[researcher] = finished & nb >= 2)

The result is *<sup>p</sup>*total <sup>2</sup> <sup>+</sup> <sup>p</sup>need + 2 <sup>×</sup> <sup>p</sup>coffee <sup>≤</sup> <sup>2</sup> <sup>∧</sup> *max* <sup>≥</sup> 2 and the minimal time is 2. That is, any of these valuations guarantee the reachability of a state where the researcher has drunk 2 coffees, and the minimum time is 2 (recall that <sup>p</sup>need <sup>∈</sup> [1,∞)).

*Optimal Parameter Reachability.* One can ask here for the valuations for which the value of a given parameter is minimized or maximized when reaching a given state. Let us ask for the valuations minimizing the value of ptotal when finishing a paper after drinking (at least) 3 coffees.

# synth EFpmin( loc[researcher] = finished & nb >= 3, pTotal)

The result is *max* <sup>≥</sup> <sup>3</sup> <sup>∧</sup> <sup>p</sup>total = 2.<sup>1</sup> <sup>∧</sup> <sup>p</sup>need = 1. Observe that <sup>p</sup>coffee is not involved in this constraint (contrarily to minimum-time synthesis); indeed, the time spent in drinking coffee does not impact the total duration of the *work* (ptotal), as the progress of clock x is stopped in coffee.

*Parametric Deadlock Freeness.* Deadlocks are states in which no discrete action can be taken, and time cannot elapse ("timelock"). Such situations may denote ill-formed models. IMITATOR offers an algorithm [7] synthesizing parameter valuations for which the model is deadlock-free. In case of "early termination" (predefined bound on the depth of the state space or on the computation time), a backward procedure synthesizes a subset of correct (deadlock-free) valuations.

```
# synth DeadlockFree
```
For this property, the analysis does not terminate, as the state space is infinite (unbounded rational-valued parameters, unbounded variable *nb*) and IMITATOR needs to explore it as a whole to deduce deadlock-freeness for our example.

Adding a bound on the depth of the state space (option -depth-limit 40) yields termination, with a *pair* of constraints: an under-approximated positive constraint (i.e., valuations that are guaranteed to be deadlock-free) *max* <sup>&</sup>lt; <sup>16</sup> <sup>∨</sup> (*max* <sup>≥</sup> <sup>16</sup> <sup>∧</sup> <sup>p</sup>total <sup>&</sup>lt; <sup>27</sup> <sup>2</sup> <sup>p</sup>need ), and an over-approximated negative constraint (i.e., valuations that *might* be deadlocked) *max* <sup>≥</sup> <sup>16</sup>∧ptotal <sup>≥</sup> <sup>27</sup> <sup>2</sup> <sup>p</sup>need . Observe that both constraints are complementary, i.e., IMITATOR is sure that the former set is deadlock-free, and is not sure that the latter set contains deadlocks. (Note that, in fact, the model is very likely to be deadlock-free for all valuations, even though IMITATOR is not able to show it.)

*Liveness Synthesis.* A new feature of IMITATOR 3 is cycle synthesis, i.e., parameter valuations for which there exists an infinite run, possibly passing infinitely often by a given discrete state (B¨uchi condition). IMITATOR uses by default an original algorithm by Laure Petrucci and Jaco van de Pol based on NDFS extended with parametric subsumption and pruning [45] (other algorithms, such as BFS, are also available [11]). In our running example , one can ask for the valuations for which the researcher infinitely often writes papers after drinking (at least) 3 coffees for each of them.

```
# synth CycleThrough( loc[researcher] = finished & nb >=3)
```
The result is *max* <sup>≥</sup> <sup>3</sup> <sup>∧</sup> <sup>p</sup>total <sup>≥</sup> <sup>2</sup>.<sup>1</sup> <sup>×</sup> <sup>p</sup>need .

*Robustness.* Inherited from earlier versions of IMITATOR, one can apply the *inverse method* [29] (also called trace preservation [21]) that, given a reference parameter valuation, synthesizes the set of parameter valuations for which the set of "traces" (discrete behaviors, i.e., abstracting time information away) is the same as for this reference valuation.


The result is: - <sup>3</sup> <sup>×</sup> <sup>p</sup>need <sup>&</sup>gt; <sup>p</sup>total <sup>≥</sup> <sup>2</sup> <sup>×</sup> <sup>p</sup>need <sup>∧</sup> *max* <sup>∈</sup> [2, 3) ∨ - <sup>2</sup>.<sup>1</sup> <sup>×</sup> <sup>p</sup>need <sup>&</sup>gt; <sup>p</sup>total <sup>≥</sup> <sup>2</sup> <sup>×</sup> <sup>p</sup>need <sup>∧</sup> *max* <sup>≥</sup> <sup>3</sup> . The synthesized constraint can be seen as a characterization of the *robustness* of the original parameter valuation.

*Synthesis Using Patterns.* Another way to specify properties is to use a set of predefined observer patterns [6,28]. Observer patterns are translated into observer automata (called reachability testing in [1]), and their correctness reduces to reachability. This procedure is transparent to the user, i.e., (s)he only needs to specify the pattern and IMITATOR takes care of the translation and synthesis. IMITATOR patterns specify the order between actions, extended with (possibly parametric) timing information. The syntax is detailed in the user manual, and the semantics is given in [6].

For example, one can synthesize the set of valuations such that, every time the researcher restarts a new article, (s)he completes it within 5 time units. That is, every occurrence of the restart action must be followed within (at most) 5 time units by the done action.

# synth pattern( everytime restart then eventually done within 5)

A part of the valuations set is: *max* <sup>≥</sup> <sup>6</sup> <sup>∧</sup> <sup>5</sup> <sup>−</sup> <sup>6</sup> <sup>×</sup> <sup>p</sup>coffee <sup>≥</sup> <sup>p</sup>total <sup>≥</sup> <sup>4</sup>.<sup>7</sup> <sup>×</sup> <sup>p</sup>need . A graphical 2D representation projected onto ptotal and pcoffee (setting pneed = 2 and *max* = 3) is given in Fig. 1b.

*Other Algorithms.* IMITATOR features a number of additional algorithms, including *i) non-Zeno infinite run synthesis* [27], *ii) behavioral cartography* [16] that partitions the parameter space into *tiles* where the discrete behavior is uniform, or *iii) parametric reachability preservation*, that takes as input a discrete state and a reference valuation, and synthesizes valuations for which this discrete state is reachable iff it is reachable for the reference valuation [25]. The two latter algorithms can be distributed over a cluster, showing interesting results, and can be used to perform reachability synthesis while being faster than the normal reachability synthesis algorithm for some benchmarks [14,15]. Finally, compositional verification for a subclass of IPTAs (a parametric extension of event-recording automata [4]) was proposed in [24].

#### **4 Distribution**

IMITATOR is distributed under the terms of the GNU General Public License. Its source code is therefore publicly available, and benefited from several contributors' additions. IMITATOR is available online<sup>2</sup>, together with its documentation, and a benchmarks library [26].

IMITATOR depends on several libraries. Notably, the core engine relies on the Parma Polyhedra Library (PPL) [32] for the computation of symbolic states. As a consequence, IMITATOR can be cumbersome to compile. For this reason, standalone binaries are available for all Linux-like systems. A Docker version<sup>3</sup> (made by Jaime Arias) and a prototype Web service<sup>4</sup> are available too.

An extensive user manual, explaining all algorithms and providing users with a full description of the input syntax for models and properties, is available [10].

#### **5 A Selection of Applications**

IMITATOR was applied to a variety of both academic and industrial case studies over the last few years. These applications range within several domains, including real-time systems, testing and monitoring, cybersecurity, or hardware verification. One can cite:

<sup>2</sup> https://www.imitator.fr.

<sup>3</sup> https://hub.docker.com/r/imitator/imitator/.

<sup>4</sup> https://imitator.lipn.univ-paris13.fr/.


#### **6 Related Tools**

HyTech [41] was the first model checker for hybrid systems (a class of formalisms beyond PTAs), including parameters; it is not maintained anymore.

Uppaal [42] is a state-of-the-art tool for modeling and verifying systems modeled as networks of timed automata and extended with variables and data structures; while Uppaal became a major tool for model checking timed automata, it does not support parametric verification, and the use of clocks is restricted to comparing one clock with one constant or with another clock, while IMITATOR allows a liberal syntax based on polyhedra.

RomEo´ [43] performs parameter synthesis for parametric time Petri nets with inhibitor arcs [47].

While RomEo´ shares similarities with IMITATOR, it does not support (extensions of) timed automata, and notably not multi-rate clocks.

SpaceEx [39] is a tool for verifying hybrid systems. It is not specifically dedicated to parameter synthesis, and mainly targets safety and reachability, in contrast to IMITATOR that proposes multiple synthesis algorithms.

IMITATOR's input syntax also shares some similarities with that of PHAVer-Lite [33] (a fork of PHAVer and predecessor of SpaceEx, that uses PPLite [34] instead of PPL [32]), coming from the fact that both IMITATOR and PHAVerLite originate from the HyTech syntax.

#### **7 Perspectives**

To gain some further speed for models that require less expressiveness (notably no strict inequality nor rational-valued variables), offering to replace PPL [32] with PPLite [34], or using standard 32-bit integers instead of GNU MP rationals is on our agenda.

**Acknowledgement.** While the author has been the main developer of IMITATOR since 2008, several colleagues brought very valuable enhancements, notably Camille Coti and Sami Evangelista [14] (on distributed algorithms), Nguy Ho`ang Gia [27] (on non-Zeno algorithms), Vincent Bloemen [12] (on minimal-time synthesis), Laure Petrucci and Jaco van de Pol [11] (on NDFS-based cycle synthesis), and Jaime Arias for multiple practical enhancements. Many thanks to Dylan Marinho for a careful rereading of this paper, and to Stephan Merz for useful suggestions.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Formally Verified Switching Logic for Recoverability of Aircraft Controller**

Ratan Lal1(B) , Aaron McKinnis<sup>2</sup>, Dustin Hauptman<sup>2</sup>, Shawn Keshmiri<sup>2</sup>, and Pavithra Prabhakar<sup>1</sup>

<sup>1</sup> Kansas State University, Manhattan, KS, USA *{*ratan,pprabhakar*}*@ksu.edu <sup>2</sup> Department of Aerospace, University of Kansas, Lawrence, KS, USA *{*amckinnis,dhauptman,keshmiri*}*@ku.edu

**Abstract.** In this paper, we investigate the design of a safe hybrid controller for an aircraft that switches between a classical linear quadratic regulator (LQR) controller and a more intelligent artificial neural network (ANN) controller. Our objective is to switch safely between the controllers, such that the aircraft is always recoverable within a fixed amount of time while allowing the maximum time of operation for the ANN controller. There is a *priori* known safety zone for the LQR controller operation in which the aircraft never stalls, over accelerates, or exceeds maximum structural loading, and hence, by switching to the LQR controller just before exiting this zone, one can guarantee safety. However, this *priori* known safety zone is conservative, and therefore, limits the time of operation for the ANN controller. We apply reachability analysis to expand the known safety zone, such that the LQR controller will always be able to drive the aircraft back to the safe zone from the expanded zone ("recoverable zone") within a fixed duration. The "recoverable zone" extends the time of operation of the ANN controller. We perform simulations using the hybrid controller corresponding to the recoverable zone and observe that the design is indeed safe.

#### **1 Introduction**

Different types of controller designs have been investigated for aircraft control, such as Linear Quadratic Regulators [28], Fuzzy Logic (FL) [8], and Artificial Neural Networks [26]. The LQR controllers provide an optimal controller for linear time invariant (LTI) systems that minimizes a quadratic cost function and guarantees stability and robustness. Though the LQR design is not directly applicable to non-linear systems, often non-linear systems are approximated by linear systems via linearization around the equilibrium point, thus enabling the application of the LQR based design. Although the LQR controller provides good performance for LTI systems [28], studies have shown that the ANN controllers have better performance in the presence of uncertain environments [26]. The ANN controller is especially suitable for adaptive flight control applications, where system dynamics are dominated by unknown nonlinearities [19]. An aircraft can experience a number of issues that may cause failures in the system. Things like over-acceleration can cause the aircraft to gain too much energy and enter into unstable modes, while rapid de-acceleration and hard maneuvers will cause increased structural loading, leading to broken lifting platforms. Another issue is that of stall, in which the airflow over the lifting section crosses a "critical angle of attack", compromising the lift generation. All of these problems can occur as a function of the control input or as external disturbances, such as high wind gust, further complicating the problem. Though ANN-based adaptive controllers are capable of handling these situations, guaranteeing safe functionality of these systems remains a challenge due to the complexity of these controllers. So, we have LQR-based controllers on one hand, that are efficient in nominal conditions, and are simple enough to be amenable to analysis, and sophisticated ANN-based controllers on the other hand that can handle difficult environmental conditions, but are, at the same time, too complex to be amenable to analysis. Our solution is a "hybrid controller" consisting of a simplex like architecture [7], wherein, we switch between the ANN and LQR controller in such a way that safety is guaranteed by the switching logic, that is, the aircraft is always recoverable from a stall within a fixed amount of time if it occurs.

Our broad objective is to find an ANN-based controller that can improve performance in uncertain environments. To achieve this goal, we need to train the ANN-controller, however, it is risky to train an ANN controller during a real flight test as it poses a safety risk. Hence, the solution we propose is to switch between a traditional LQR controller and the ANN controller in such a way that safety is guaranteed. More precisely, we allow the ANN controller to operate while the aircraft remains within a "safe zone" from which the LQR controller can guarantee that the aircraft never stalls. When the ANN controller is on the verge of leaving the safe zone, we switch to the LQR controller. However, these expert determined safe zones are often too conservative (small), thereby not providing sufficient time of operation for the ANN controller. A longer duration of operation for the ANN controller is desirable for the learning process, so we provide a method to extend the safe zone to a larger set ("recoverable zone"), which guarantees that the aircraft recovers within a fixed amount of time if a stall occurs. The recoverable zone computation is performed using formal methods based reachable set computation, thereby providing a formally verified switching component decision procedure that guarantees the safe operation of the aircraft.

We consider a dynamic model of a fixed-wing aircraft, with six-degrees-offreedom (6-DOF), which is used as an experimental platform to employ a hybrid controller that consists of an intelligent and automatic switching between an LQR and an ANN based controller. The aircraft dynamics consists of a decoupled longitudinal and lateral linear time invariant dynamics, with a decoupled statefeedback LQR controller for each component. For our simulations, we consider an ANN controller that combines aircraft guidance and control systems and performs end-to-end mapping from error states to control surface values, in order to fly along a straight line with steady state wings-level and altitude hold.

We have performed Hardware in The Loop (HITL) simulation of the hybrid controller in conjunction with the the 6-DOF differential equations, on the aircraft avionics using the open source software, QGroundControl. Our simulations exhibit that the number of sample iterations for which ANN controller actions are performed while ensuring safe flying, increases as the learning space (recoverable zone) is expanded.

#### **2 Related Work**

Artificial Neural Networks have been widely used in many control applications, such as automatic generation control of interconnected power systems [41], irrigation scheduling [37], micro-turbine power plant [36], solar binding [4], robotics [1,6], and aircraft control [17]. ANN is popularly used in flight control [19], robot control [25] as well as for non-linear systems [42].

Verification has been extensively applied to dynamical systems, and focus on over-approximation based methods including predicate abstraction [3,22], state-space exploration based fix-point computation [14], Hamilton-Jacobi based methods [2], symbolic state space exploration based methods [16], Satisfiability Modulo Theory (SMT) based methods [20,21,23,38], and counter-example guided abstraction-refinement based methods [24,31,32].

Recent studies [40] compare several neural network verification algorithms. Formal verification of feedforward neural networks with different activation functions, such as ReLU [18] and Lipschitz-continuous functions [33], have been studied. Different verification problems have been considered including output range analysis [10], and robustness analysis [15]. Verification methods include those based on reduction to satisfiability solving [18], optimization solving [12], abstract interpretation [35], abstraction-refinement [30], and linearization [13]. Verification of ANN with feedback controllers has been explored [11].

In this paper, one of the problems we study is stall. The stall could occur due to many reasons. Researchers have developed different techniques to avoid or recover from the stall. Deep stall has been studied [27], which is an uncontrollable state at which the angle of attack (AOA) increases automatically and will be locked at a certain AOA which is far beyond the critical angle of attack. A stall due to wing has been studied [39]. The stall avoidance/recovery have been studied [9]. Here, we present a hybrid controller consisting of ANN and LQR controller similar to simplex design [7], which will not only recover, but also provide more learning space for the ANN controller to explore. Our hybrid controller is different from the simplex design [7] in many perspectives. Our hybrid controller makes the decision between ANN and LQR control input via safety checking performed based on an under-approximation reach set, which is computed off-line. However, in the work [7], the analysis is performed based on an over-approximation reach set. Also, in the work [7], an initial set is known; however, in our work, a target set ("safe zone") is known and the initial set is unknown.

#### **3 Hybrid Controller Architecture**

In this section, we provide details of the hybrid controller architecture which is shown in Fig. 1. It has mainly four components: (a) Aircraft dynamics, (b) LQR controller (c) ANN controller, and (d) Switching logic. For the aircraft dynamics, we consider a 6-DOF model of the fixed wing aircraft. The hybrid controller consists of the LQR and the ANN controller, and the switching logic; the LQR and the ANN controller each receive the state of the aircraft periodically (which is obtained from the aircraft dynamics model in the simulations) and compute the inputs to the aircraft. The switching logic decides which input is fed back to the aircraft (dynamics) at each sample time, based on the current state of the system. The state of the system (dynamics) is updated according to the input selected. We note that the details of the ANN controller is not important for the correctness of this work, since the safety is guaranteed even when the ANN control is considered as a black box. However, we adapt the ANN controller from the work [34] for the ANN component of the hybrid controller. We briefly describe the important aspects of the aircraft dynamics, LQR controller and the switching logic.

**Fig. 1.** Hybrid controller architecture **Fig. 2.** Switching Logic for LQR and ANN controller

#### **3.1 Aircraft Dynamics**

We start with a brief description of the aircraft states and motion. The aircraft has 3 axes, the roll axis (I), pitch axis (J) and yaw axis (K) as shown in Fig. 3. Motion occurs in two planes, the longitudinal, axes (I) and (K), and lateral, axes (I) and (J), which are often considered to be decoupled.

In the longitudinal plane, the states are, velocity (V ), angle of attack (α), pitch angle (θ) and pitch rate (q), and control inputs are thrust (δ*t*) and elevator

**Fig. 3.** Overview of aircraft

deflection δ*e*. All the states and control inputs are shown in Fig. 3. The angle of attack (α) is the angle between the roll axis (I) and the direction of velocity (V ). The pitch angle (θ) is the angle between the roll axis (I) and the horizontal axis. The pitch rate (q) is the rate of change in the pitch angle θ. When the pitch angle (θ) changes, the lateral plane rotates and the roll and yaw axes will change to I<sup>1</sup> and K1, respectively. The thrust (δ*t*) generates a force that is used to move the aircraft forward along the roll axis, and the elevator deflection (δ*e*) is a control surface located at the rear of the aircraft which primarily controls the pitch angle (θ). The longitudinal dynamics is a linear dynamics of the form **x**˙ *lon* = A*lon***x***lon* + B*lon***u***lon*, where **x***lon* = [V, α, θ, q] , **u***lon* = [δ*t*, δ*e*] , and A*lon* and B*lon* are specific matrices.

In the lateral plane, the states are, side-slip angle (β), roll angle (φ), roll rate (p) and yaw rate (r), and control inputs are aileron deflection (δ*a*) and rudder deflection (δ*r*). The states and control inputs are shown Fig. 3. The angle of side-slip (β) is the angle between the roll axis (I) and the direction of incoming airflow. When the roll axis I rotates, the pitch axis (J) and the yaw axis (K) will change to J<sup>2</sup> and K2, respectively. The roll angle (φ) is the angle between J and J2. The roll rate (p) is the rate of change in the roll angle (φ). The yaw rate (r) is the rotational rate of change in the yaw axis (K). The aileron deflection (δ*a*) is the control surface which is used to control the rotation of the roll axis (I). The rudder deflection (δ*r*) is the control surface which is used to control the rotation of the yaw axis (K). The lateral dynamics is a linear dynamics of the form **x**˙ *lat* = A*lat***x***lat* + B*lat***u***lat*, where **x***lat* = [β, φ, p, r] , **u***lat* = [δ*a*, δ*r*] , and A*lat* and B*lat* are specific matrices.

#### **3.2 LQR Controller**

Linear Quadratic Regulator (LQR) controller for a linear dynamics **x**˙ = A**x**+B**u** is an optimal controller that minimizes a quadratic cost function (J). It is a linear state feedback controller of the form −Kx, where K is referred to as the gain matrix. The closed loop dynamics is given by **x**˙ = (A − BK)**x**; which is the system behavior when controller by the LQR controller. Since the longitudinal and lateral dynamics of the aircraft are decoupled, we have an LQR controller for each component with gains K*lon* and K*lat*, resulting in corresponding closed loop systems, **x**˙ *lon* = (A*lon* − B*lon*K*lon*)**x***lon* and **x**˙ *lat* = (A*lat* − B*lat*K*lat*)**x***lat*.

#### **3.3 Switching Algorithm for the Safety of ANN Controller**

Stall is one of the important issues for any aircraft. Stall is a condition in which the angle of attack surpasses a critical bound and greatly decreases lift generation. Consequently, the aircraft will start rapidly descending. Additional problems occur when the aircraft encounters large accelerations, primarily about the roll and yaw axes, which can lead the aircraft into an unstable spiral mode, a dangerous and usually unrecoverable event. Finally, rapid maneuvers can lead to large loads on the aircraft structure, causing permanent deformation or breaking the structure altogether. Generally, exact constraints for these problems cannot be found due to the complexity of aircraft motion. However, a set of safe constraints has been generated for the testbed aircraft by examining previous flight test data in which problems did not occur.

The objective of the switching logic is to arbitrate the switching between the LQR and ANN based controllers, while maintaining safety and at the same time providing ANN controller the maximum opportunity to operate, and thereby learn. Our premise is that we have some known safe zone S give by an expert in which LQR controller actions are safe, that is, if we apply control input **u** = −K**x**, when **x** ∈ S, to the LTI dynamics of the aircraft, then the aircraft never stalls. However, if we apply control input **u** obtained by the ANN controller at a state **x** ∈ S, we cannot ensure that the system never stalls. Computing such a safe zone for an ANN controller would be computationally hard. Hence, the switching algorithm computes the effect of applying **u** computed by the ANN controller, and decides to pass it on to the system, if it infers that the system will be safe in the next step. Otherwise, it outputs the input suggested by the LQR controller. In either case, it ensures that the system is in the S region at all times during the operation of the flight. The details of the switching algorithm are provided in Fig. 2.

The performance of the hybrid controller depends on the safe zone. The safe zone obtained by expert advice is often conservative. Hence, we provide a method to extend the safe zone ("recoverable zone") for which the switching algorithm guarantees that the system is always recoverable within the fixed duration if it occurs. Next, we provide the details of computing the recoverable zone.

#### **4 Computation of Recoverable Zone**

In this section, we provide the details of computing a recoverable zone for the fixed time T > 0. Our broad goal is to compute all those states from which the given safe zone S can be reached within the time T > 0 for an LTI dynamics of aircraft which is in the form of **x**˙ = (A − BK)**x**, where K is an LQR control gain matrix. This is the problem of computing the backward reach set of a linear system

$$
\dot{\mathbf{x}} = C\mathbf{x} \tag{1}
$$

where C = A − BK. The solution of a linear system **x**˙ = C**x** is given by x(t) = e*Ct*x(0), where x(t) is the state of the system at time t. Hence, we define the backward reach set for a given linear closed loop system as follows:

**Definition 1.** *[Backward Reach Set] Given a linear closed loop system x*˙ = A*x, a time horizon* T > 0*, and a final set of states* X*<sup>f</sup> , the backward reach set ReachB*(X*<sup>f</sup>* , A, [0, T]) *is defined as follows:*

$$\operatorname{Reach}\_B(\mathcal{X}\_f, A, [0, T]) = \{ \mathbf{x} \mid \exists \ t \in [0, T], e^{At} \mathbf{z} \in \mathcal{X}\_f \}.$$

Next, we formally define the recoverable zone in terms of backward reach set.

**Definition 2.** *[Recoverable Zone] Given system in Eq. (1), a time horizon* T > 0*, and a safe zone* S*, a recoverable zone* S *is defined as follows:* S = *ReachB*(S, C, [0, T]).

The computation of the recoverable zone S can be alternatively tackled using a forward reachability analysis on the following transformed equation.

$$
\dot{\mathbf{x}} = -C\mathbf{x} \tag{2}
$$

We define forward reach set for a given linear closed loop system as follows:

**Definition 3.** *[Forward Reach Set] Given a linear closed loop system x*˙ = A*x, a time horizon* T > 0*, and an initial set of states* X0*, forward reach set Reach<sup>F</sup>* (X0, A, [0, T]) *is defined as follows:*

$$\operatorname{Reach}\_F(\mathcal{X}\_0, A, [0, T]) = \{ e^{At} \mathfrak{a}\_0 \mid \exists \ t \in [0, T], \exists \ \mathfrak{a}\_0 \in \mathcal{X}\_0 \}.$$

Equation (2) is obtained from Eq. (1) by negating the right hand side. The effect of the transformation is that the system now evolves backward in time. We notice that the set of states that can reach S within time T from Equation (1) (*ReachB*(S, C, [0, T])) is equal to the set of states reached using Equation (2) from S in a given time horizon T > 0 (*Reach<sup>F</sup>* (S, C, [0, T])). Next, we formulate this equivalence of forward and backward reach sets of the two systems, namely Equations (1), (2) in Theorem 1.

**Theorem 1.** *Given systems in Equation (1) and Equation (2), a time horizon* T > 0*, a safe zone* S*, we have Reach<sup>F</sup>* (S, −C, [0, T]) = *ReachB*(S, C, [0, T]).

The computation of the exact recoverable zone is complex because the solution of Equation (2) consists of exponential function, and there are no known algorithms for solving constraints with exponential functions, unlike solvers for linear and polynomial functions. Hence, several over-approximation methods have been investigated [5,16,20,29,31,32]. An over-approximated recoverable zone violates the property of the recoverable zone, that is, it contains point that are not guaranteed to reach the safe zone within the time bound. In this situation, the stall may not be recoverable if it occurs. Therefore, we compute an under-approximation of the exact recoverable zone S which is conservative, nevertheless, ensures the safety of the switching algorithm.

#### **4.1 Under-Approximation of Recoverable Zone**

In this section, we provide a method to compute an under-approximation of the exact recoverable zone S . While computing under-approximations are in general hard, we use a simple idea that provides a practically viable under-approximation for our purposes. Our broad approach is based on sampling, and consists of an under-approximate reach set which is the union of the reach set at certain time points, as opposed to all the points in the given interval. We sample the time interval [0, T] at sample times that are multiples of r. Then, we compute forward reach set from safe zone S under Equation (2) at sample times r, 2r, . . . , kr = T and take their union, that is, the under-approximation of the recoverable zone denoted *Approx*(S) is - *k i*=0 *Reach<sup>F</sup>* (S, −C, ir), where *Reach*(S, −C, ir) denotes the forward reach set from S at time ir. Next, we show that *Approx*(S) is an underapproximation of the recoverable zone S . We formulate this in Theorem 2.

**Theorem 2.** *Given system in Equation (2), a time horizon* T > 0*, a safe zone* S*, we have Approx*(S) ⊆ *Reach<sup>F</sup>* (S, −C, [0, T]).

Note that *Approx*(S) converges to the exact recoverable zone S as r → 0.

#### **5 Experimental Analysis**

In this section, we provide the details of our implementation of hybrid controller architecture. Then, we present the experimental results.

#### **5.1 Experimental Setup**

The experimentation method for preliminary concept testing is a Hardware in The Loop (HITL) simulation. The HITL runs the 6-DOF differential equations, on the aircraft avionics, which are then propagated using a Runge-Kutta fourth order integration method.

**Fig. 4.** AFS 6.0 **Fig. 5.** HITL aggressive trajectory

This technique generates all aircraft states and control inputs that are necessary to the operation of the switch. The main advantage of conducting these simulations as an HITL rather than software simulations is that all the codes will be tested on the actual hardware used for flight, showcasing any shortcomings in computation power or integration missteps, which may impact flight test success.

The current avionics, Autopilot Flight System (AFS) 6.0, consists of three main components. Sensor data and outputs are handled by the Pixhawk 2.1 cube. The onboard computer which runs the in-house designed guidance, navigation and control (GNC) algorithms, as well as handles the state emulation is the Nvidia Tegra Nano. The Tegra Nano is a low cost system, with a quad-core CPU and a 128 core GPU. The final component is a 900 MHz telemetry unit which serves as the communication between the aircraft and the ground station, where the ground station provides a visual representation of the current aircraft state as well as relevant GNC information. The ground station used for these simulations is a modified version of the open source software, QGroundControl, which is also used to generate way-points for the given area of operation. Figure 4 shows both the front and back sides of the custom avionics boards.

While in HITL, the ANN controllers are very stable due to being trained with similar dynamic models to those that are used to propagate the simulation. This makes it unlikely to see the switching logic in action as no control inputs would be deemed unsafe, especially in grid or racetrack patterns that make up the majority of flight test operations. To circumvent this, an oddly shaped trajectory, shown in Fig. 5, with multiple sharp turns is used to ensure previously un-visited states are achieved. The simulation is run for approximately one lap of the given trajectory for each value of the time horizon shown in the following section.

#### **5.2 Experimental Results**

In this section, we present the simulation results for the performance of hybrid controller. For the simulation, we consider the safe zone provided by experts, which are given in Table 1. We run the simulation for different recoverable zones, which are computed for different values of time horizon T, namely, T = 0.05, T = 0.15, T = 0.25, and T = 0.35 with time step τ = 0.05 unit. The simulation results are shown in Figs. 6 and 7. The simulation results are plotted in Fig. 6 and Fig. 7 for longitudinal velocity and lateral angle of side-slip, respectively.

**Fig. 6.** Switching between ANN and LQR controller for the longitudinal velocity



In both Figs. 6 and 7, we observe that the recoverable zone expands when the time horizon T increases.

**Fig. 7.** Switching between ANN and LQR controller for the lateral angle of side-slip

Also, we observe that the number of sample iterations in which ANN controller actions are performed, increases when the recoverable zone is expanded. For instance, in Figs. 6 and 7, for T = 0.35, ANN controller actions have been performed from the sample iteration 1500 to 3000, which was not the case for T = 0.25. For clarity, in Table 2, we present the number of sample iterations N for both ANN and LQR controller in which their actions have been performed, for different values of time horizon T.


**Table 2.** Number of sample iterations for ANN and LQR controller

In Table 2, we observe that N grows for ANN controller when the time horizon T increases, that is, the recoverable zone is expanded. However, N decreases for LQR controller when T increases. This validate the fact that hybrid controller framework provides ample time for the ANN controller to learn while ensuring a safe flight.

#### **5.3 Practical Challenges**

The implementation of the hybrid controller proved to be complex in two ways. First, the timing of the switching logic was important to the overall safety of the project. When delays are introduced into the system, the current state of the aircraft and the information the switch is making the decision on can become out of sync. If the switching logic is behind the aircraft states it can make incorrect calls on whether or not the aircraft is still safe, and cause the ANN to overextend its operation, leading to a loss of control. This is made worse as aircraft have large inertias and relatively slow time constants on control inputs meaning they can become uncontrollable much quicker than most dynamic systems. This need for extreme low latency operation caused many changes in the code structure including a rewrite from Python to C++ and parallelization of applicable code. The second practical problem is that the lack of full state feedback and lowquality sensor data. Two of the aircraft states, angle of attack and sideslip angle, cannot be directly measured by low cost systems. The easiest solution is to employ a Kalman filtering technique to estimate these two states. However, if the aircraft is experiencing a large perturbation away from the trim point, the Kalman Filter can diverge very rapidly and feed incorrect information to the switch about the relevant states. On top of this, many of the measured states are taken using low-cost, off the shelf components. In a similar way, the use of these components may introduce noise or a bias which could allow the aircraft to go into the uncontrollable region without alerting the switch or the aircraft operator. Low pass filtering is applied to attempt to deal with the noise, but the imparted delay to the sensor data must also be taken into consideration.

#### **6 Conclusions**

We have developed a hybrid controller for an aircraft dynamics which provides considerable amount of time to the ANN controller to operate and learn, while at the same time guarantees the safe operation of the flight at all times. In future, we will consider more sophisticated ANN controllers and investigate methods for computing larger recoverable zones that allow for further increase of the ANN operation time. Additionally, experimentation will be done with real flight tests, moving past HITL simulations.

**Acknowledgements.** Pavithra Prabhakar was partially supported by NSF CAREER Grant No. 1552668, NSF Grant No. 2008957, ONR YIP Grant No. N000141712577 and USDA Grant No. 2017-67007-26153. Also, this work was partially supported with funding from National Aeronautics and Space Administration (NASA) Grant No. NNX15AJ97H and Grant No. 80NSSC19C0102.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# SceneChecker**: Boosting Scenario Verification Using Symmetry Abstractions**

Hussein Sibai(B) , Yangge Li , and Sayan Mitra

Coordinated Science Laboratory, University of Illinois at Urbana-Champaign, Urbana, USA {sibai2,li213,mitras}@illinois.edu

**Abstract.** We present SceneChecker, a tool for verifying scenarios involving vehicles executing complex plans in large cluttered workspaces. SceneChecker converts the scenario verification problem to a standard hybrid system verification problem, and solves it effectively by exploiting structural properties in the plan and the vehicle dynamics. SceneChecker uses symmetry abstractions, a novel refinement algorithm, and importantly, is built to boost the performance of any existing reachability analysis tool as a plug-in subroutine. We evaluated SceneChecker on several scenarios involving ground and aerial vehicles with nonlinear dynamics and neural network controllers, employing different kinds of symmetries, using different reachability subroutines, and following plans with hundreds of waypoints in complex workspaces. Compared to two leading tools, DryVR and Flow\*, SceneChecker shows 14<sup>×</sup> average speedup in verification time, even while using those very tools as reachability subroutines.

**Keywords:** Hybrid systems · Safety verification · Symmetry

#### **1 Introduction**

Remarkable progress has been made in safety verification of hybrid and cyberphysical systems in the last decade [2–9]. The methods and tools developed have been applied to check safety of aerospace, medical, and autonomous vehicle control systems [4,5,10,11]. The next barrier in making these techniques usable for more complex applications is to deal with what is colloquially called the *scenario verification problem*. A key part of the scenario verification problem is to check that a vehicle or an agent can execute a plan through a complex environment. A planning algorithm (e.g., probabilistic roadmaps [12] and rapidly-exploring random trees (RRTs) [13]) generates a set of possible paths avoiding obstacles, but only considering the geometry of the scenario, not the dynamics. The verification task has to ensure that the plan can indeed

The authors are supported by a research grant from The Boeing Company and a research grant from NSF (FMITF: 1918531). We would like to thank John L. Olson, Aaron A. Mayne, and Michael R. Abraham from The Boeing Company for valuable technical discussions.

be safely executed by the vehicle with all the dynamic constraints and the state estimation uncertainties. Indeed, one can view a scenario as a hybrid automaton with the modes defined by the segments of the planner, but this leads to massive models. Encoding such automata in existing tools presents some practical hurdles. More importantly, analyzing such models is challenging as the over-approximation errors and the analysis times grow rapidly with the number of transitions. At the same time, such large hybrid verification problems also have lots of repetitions and symmetries, which suggest new opportunities.

We present SceneChecker, a tool that implements a symmetry abstractionrefinement algorithm for efficient scenario verification. Symmetry abstractions significantly reduce the number of modes and edges of an automaton *H* by grouping all modes that share symmetric continuous dynamics [14]. SceneChecker implements a novel refinement algorithm for symmetry abstractions and is able to use any existing reachability analysis tool as a subroutine. Our current implementation comes with plug-ins for using Flow<sup>∗</sup> [4] and DryVR [6]. SceneChecker's verification algorithm is sound, i.e., if it returns *safe*, then the reachset of *H* indeed does not intersect the unsafe set. The algorithm is lossless in the sense that if one can prove safety without using abstraction, then SceneChecker can also prove safety via abstraction-refinement, and typically a lot faster. SceneChecker can be found on figshare: https://figshare.com/ articles/software/CAV2021\_reduce\_v6\_ova/14504352 and its website: https://publish. illinois.edu/scenechecker/. An extended version of this paper is available online [1].

SceneChecker offers an easy interface to specify plans, agent dynamics, obstacles, initial uncertainty, and symmetry maps. SceneChecker checks if a fixed point has been reached after each call to the reachability subroutine, avoiding repeating computations. First, SceneChecker represents the input scenario as a hybrid automaton *<sup>H</sup>* where modes are defined by the plan's segments. It uses the symmetry maps provided by the user to construct an abstract automaton *Hv*. Automaton *Hv* represents another scenario with fewer segments, each representing an equivalence class of symmetric segments in *H*. A side effect of the abstraction is that upon reaching waypoints in *Hv*, the agent's state resets non-deterministically to a set of possible states. For example, in the case of rotation and translation invariance, the abstract scenario would have a single segment for any set of segments with a unique length in the original scenario. SceneChecker refines *Hv* by splitting one of its modes to two modes. That corresponds to representing a set of symmetric segments with one more segment in the abstract scenario, capturing more accurately the original scenario1.

We evaluated SceneChecker on several scenarios where car and quadrotor agents with nonlinear dynamics follow plans to reach several destinations in 2D and 3D workspaces with hundreds of waypoints and polytopic obstacles. We considered different symmetries (translation and rotation invariance) and controllers (Proportional-Derivative (PD) and Neural Networks (NN)). We compared the verification time of SceneChecker with DryVR and Flow\* as reachability subroutines against Flow\* and DryVR as standalone tools. SceneChecker is faster than both tools in all scenarios considered, achieving an average of 14× speedup in verification time (Table 1). In certain scenarios where Flow\* timed out (executing for more than 120 min), SceneChecker

<sup>1</sup> A figure showing the architecture of SceneChecker can be found in the extended version [1].

is able to complete verification in as fast as 12 min using Flow\* as a subroutine. SceneChecker when using abstraction-refinement achieved 13<sup>×</sup> speedup in verification time over not using abstraction-refinement in scenarios with the NN-controlled quadrotor (Sect. 7).

*Related Work.* The idea of using symmetries to accelerate verification has been exploited in a number of contexts such as probabilistic models [15,16], automata [17,18], distributed architectures [19], and hardware [20,21]. Some symmetry utilization algorithms are implemented in Murφ[22] and Uppaal [23].

In our context of cyber-physical systems, Bak et al. [24] suggested using symmetry maps, called *reachability reduction transformations*, to transform reachsets to symmetric reachsets for continuous dynamical systems modeling non-interacting vehicles. Maidens et al. [25] proposed a symmetry-based dimensionality reduction method for backward reachable set computations for discrete dynamical systems. Majumdar et al. [26] proposed a safe motion planning algorithm that computes a family of reachsets offline and composes them online using symmetry. Bujorianu et al. [27] presented a symmetry-based theory to reduce stochastic hybrid systems for faster reachability analysis and discussed the challenges of designing symmetry reduction techniques across mode transitions.

In a more closely related research, we presented a modified version of DryVR that utilizes symmetry to cache reachsets aiming to accelerate simulation-based safety verification of continuous dynamical systems [28]. We developed the related tool CacheReach that implements a hybrid system verification algorithm that uses symmetry to accelerate reachability analysis [29]. CacheReach caches and shares computed reachsets between different modes of non-interacting agents using symmetry. SceneChecker is based on the theory of symmetry abstractions of hybrid automata we presented in [14]. We suggested computing the reachset of the abstract automaton instead of the concrete one then transform it to the concrete reachset using symmetry maps to accelerate verification. SceneChecker is built based on this line of work with significant algorithmic and engineering improvements. In addition to the abstraction of [14], SceneChecker 1) maps the unsafe set to an abstract unsafe set and verifies the abstract automaton instead of the concrete one and 2) decreases the over-approximation error of the abstraction through refinement. SceneChecker does not cache reachsets and thus saves cache-access and reachset-transformation times and does not incur overapproximation errors due to caching that CacheReach suffers from [29]. At the implementation level, SceneChecker accepts plans that are general directed graphs and polytopic unsafe sets while CacheReach accepts only single-path plans and hyperrectangle unsafe sets. We show more than 30× speedup in verification time while having more accurate verification results when comparing SceneChecker against CacheReach (Table 1 in Sect. 7).

#### **2 Specifying Scenarios in** SceneChecker

A scenario verification problem is specified by a set of fixed obstacles, a plan, and an agent that is supposed to execute the plan without running into the obstacles (e.g., see Fig. 1B). For ground and air vehicles, for example, the agent moves in a subset of the 2D or the 3D Euclidean space called the *workspace*. A *plan* is a directed graph *G* = *V,S* with vertices *V* in the workspace called *waypoints* and edges *S* called *segments*2. A general graph allows for nondeterministic and contingency planning.

An *agent* is a control system that can follow waypoints. Let the state space of the agent be *X* and Θ ⊆ *X* be the uncertain initial set. Let *sinit* be the initial segment in *G* that the agent has to follow. From any state *x* ∈ *X*, the agent follows a segment *s* ∈ *S* by moving along a *trajectory*. A trajectory is a function ξ : *<sup>X</sup>* <sup>×</sup> *<sup>S</sup>* <sup>×</sup> <sup>R</sup>≥<sup>0</sup> <sup>→</sup> *<sup>X</sup>* that meets certain dynamical constraints of the vehicle. Dynamics are either specified by ordinary differential equations (ODE) or by a black-box simulator. For ODE models, ξ is a solution of an equation of the form: *<sup>d</sup>*ξ *dt* (*x,s,t*) = *f*(ξ (*x,s,t*)*,s*), for any *<sup>t</sup>* <sup>∈</sup> <sup>R</sup>≥<sup>0</sup> and ξ (*x,s,*0) = *x*, where *f* : *X* ×*S* → *X* is Lipschitz continuous in the first argument. Note that the trajectories only depend on the segment the agent is following (and not on the full plan *G*). We denote by ξ *.fstate*, ξ *.lstate*, and ξ *.dom* the initial and last states and the time domain of the time bounded trajectory ξ, respectively.

We can view the obstacles near each segment as sets of unsafe states, *O* : *S* → <sup>2</sup>*<sup>X</sup>* . The map *tbound* : *<sup>S</sup>* <sup>→</sup> <sup>R</sup>≥<sup>0</sup> determines the maximum time the agent should spend in following any segment. For any pair of consecutive segments (*s,s* ), i.e. sharing a common waypoint in *G*, *guard*((*s,s* )) defines the set of states (a hyperrectangle around a waypoint) at which the agent is allowed to transition from following *s* to following *s* .

Scenario JSON file is the first of the two user inputs. It specifies the scenario: Θ as a hyperrectangle; *S* as a list of lists each representing two waypoints; *guard* as a list of hyperrectangles; *tbound* as a list of floats; and *O* as a list of polytopes. Output of SceneChecker is the scenario verification result (*safe* or *unknown*) and a number of useful performance metrics, such as the number of modesplits, number of reachability calls, reachsets computation time, and total time. SceneChecker can also visualize the various computed reachsets.

#### **3 Transforming Scenarios to Hybrid Automata**

The input scenario is first represented as a hybrid automaton by a Hybrid constructor. This constructor is a Python function that parses the Scenario file and constructs the data structures to store the scenario's hybrid automaton components. In what follows, we describe the constructed automaton informally. In our current implementation, sets are represented either as hyper-rectangles or as polytopes using the Tulip Polytope Library3.

<sup>2</sup> We introduce this redundant nomenclature because later we will reserve the term edges to talk about mode transitions in hybrid automata. We use waypoints instead of vertices as a more natural term for points that vehicles have to follow.

<sup>3</sup> https://pypi.org/project/polytope/.

*Scenario as a Hybrid Automaton.* A hybrid automaton has a set of *modes* (or discrete states) and a set of continuous states. The evolution of the continuous states in each mode is specified by a set of trajectories and the transition across the modes are specified by *guard* and *reset* maps. The agent following a plan in a workspace can be naturally modeled as a hybrid automaton *H*, where *sinit* and Θ are its initial mode and set of states.

Each segment *s* ∈ *S* of the plan *G* defines a *mode* of *H* (e.g. see Fig. 1A). The set of edges *E* ⊆ *S* ×*S* of *H* is defined as pairs of consecutive segments in *G*. For an edge *e* ∈ *E*, *guard*(*e*) is the same as that of *G*. The *reset* map of *H* is the identity map. We will see in Sect. 5 that abstract automata will have nontrivial reset maps.

*Verification Problem.* An *execution* of length *k* is a sequence σ := (ξ0*,s*0)*,...,*(ξ*<sup>k</sup>,sk*). It models the behavior of the agent following a particular path in the plan *G*. An execution σ must satisfy: 1) ξ<sup>0</sup>*.fstate* ∈ Θ and *s*<sup>0</sup> = *sinit*, for each *i* ∈ {0*,...,k* − 1}, 2) (*si,si*<sup>+</sup>1) ∈ *E*, 3) ξ*<sup>i</sup>.lstate* ∈ *guard*((*si,si*<sup>+</sup>1)), and 4) ξ*<sup>i</sup>.lstate* = ξ*<sup>i</sup>*+1*.fstate*, and 5) for each *i* ∈ {0*,...,k*}, ξ*<sup>i</sup>.dom* ≤ *tbound*(*si*)*.* The set of *reachable states* is *ReachH* := {σ*.lstate* | σ is an execution}. The restriction of *ReachH* to states with mode *s* ∈ *S* (i.e., agent following segment *s*) is denoted by *ReachH*(*s*). Thus, the hybrid system verification problem requires us to check whether ∀*s* ∈ *S*, *ReachH*(*s*)∩*O*(*s*) = 0./

### **4 Specifying Symmetry Maps in** SceneChecker

The hybrid automaton representing a scenario, as constructed by the Hybrid constructor, is transformed into an abstract automaton. SceneChecker uses symmetry abstractions [14]. The abstraction is constructed by the abstract function (line 1 of Algorithm 1) which uses a collection of pairs of maps Φ = {(γ*<sup>s</sup>* : *X* → *X,*ρ*<sup>s</sup>* : *S* → *S*)}*s*∈*<sup>S</sup>* that is provided by the user. We describe below how these maps are specified by the user in the Dynamics file. These maps should satisfy:

$$\forall t \ge 0, \mathbf{x}\_0 \in X, \mathbf{s} \in \mathcal{S}, \mathbf{y}\_s(\xi(\mathbf{x}\_0, \mathbf{s}, t)) = \xi(\chi\_s(\mathbf{x}\_0), \rho\_s(\mathbf{s}), t). \tag{1}$$

where ∀*s* ∈ *S*, the map γ*<sup>s</sup>* is differentiable and invertible. Such maps are called *symmetries* for the agent's dynamics. They transform the agent's trajectories to other symmetric ones of its trajectories starting from symmetric initial states and following symmetric modes (or segments in our scenario verification setting). It is worth noting that (1) does not depend on whether the trajectories ξ are defined by ODEs or black-box simulators. Currently, condition (1) is not checked by SceneChecker for the maps specified by the user. However, in the following discussion, we present some ways for the user to check (1) on their own. For ODE models, a sufficient condition for (1) to be satisfied is if: ∀ *x* ∈ *X,s* ∈ *S,* ∂ γ*s* ∂ *<sup>x</sup> f*(*x,s*) = *f*(γ*s*(*x*)*,*ρ*<sup>s</sup>*(*s*)), where *f* is the right-hand-side of the ODE [30]. For black-box models, (1) can be checked using sampling methods. In realistic settings, dynamics might not be exactly symmetric due to unmodeled uncertainties. In the future, we plan to account for such uncertainties as part of the reachability analysis.

In scenario verification, a given workspace would have a coordinate system according to which the plan (waypoints) and the agent's state (position, velocity, heading angle, etc.) are represented. In a 2D workspace, for any segment *s* ∈ *S*, an example symmetry ρ*<sup>s</sup>* would transform the two waypoints of *s* to a new coordinate system where the second waypoint is the origin and *s* is aligned with the negative side of the horizontal axis (see Fig. 1D). The corresponding γ*<sup>s</sup>* would transform the agent's state to this new coordinate system (e.g. by rotating its position and velocity vectors and shifting the heading angle). For such a pair (γ*s,*ρ*<sup>s</sup>*) to satisfy (1), the agent's dynamics have to be invariant to such a coordinate transformation and (1) merely formalizes this requirement. Such an invariance property is expected from vehicles' dynamics–rotating or translating the lane should not change how an autonomous car behaves.

Dynamics file is the second input provided by the user in addition to the Scenario file and it contains the following:

polyVir(*<sup>X</sup> ,s*): returns γ*<sup>s</sup>*(*X* ) for any polytope *X* ⊂ *X* and segment *s* ∈ *S*. modeVir(*s*): returns ρ*<sup>s</sup>*(*s*) for any given segment *s* ∈ *S*.

virPoly(*<sup>X</sup> ,s*): returns γ<sup>−</sup><sup>1</sup> *<sup>s</sup>* (*X* ), implementing the inverse of polyVir. computeReachset(*initset,s,T*): returns a list of hyperrectangles overapproximating the agent's reachset starting from *initset* following segment *s* for *T* time units, for any set of states *initset* ⊂ *X*, segment *s* ∈ *S*, and *T* ≥ 0.

#### **5 Symmetry Abstraction of the Scenario's Automaton**

In this section, we describe how the abstract function in Algorithm <sup>1</sup> uses the functions in the Dynamics file to construct an abstraction of the scenario's hybrid automaton provided by the Hybrid constructor. Given the symmetry maps of Φ, the symmetry abstraction of *H* is another hybrid automaton *Hv* that aggregates many symmetric modes (segments) of *H* into a single mode of *Hv*.

*Modes and Transitions.* Any segment *s* ∈ *S* of *H* is mapped to the segment ρ*<sup>s</sup>*(*s*) in *Hv* using modeVir. The set of modes *Sv* of *Hv* is the set of segments {ρ*<sup>s</sup>*(*s*)}*s*∈*S*. For any *sv*, *tboundv*(*sv*) = max*s*∈*S,sv*<sup>=</sup>ρ*<sup>s</sup>*(*s*) *tbound*(*s*). In the example of Sect. 4 (Fig. 1D), the segments in *Hv* are aligned with the horizontal axis and ending at the origin. The number of segments in *Hv* would be the number of segments in *G* with unique lengths. The agent would always be moving towards the origin of the workspace in the abstract scenario. Any edge *e* = (*s,s* ) ∈ *E* of *H* is mapped to the edge *ev* = (ρ*s*(*s*)*,*ρ*<sup>s</sup>* (*s* )) in *Hv*. The *guard*(*e*) is mapped to γ*<sup>s</sup>*(*guard*(*e*)) using polyVir which becomes part of *guardv*(*ev*) in *Hv*. For any *x* ∈ *X*, *reset*(*x,e*), which is equal to *x*, is mapped to γ*s* (γ<sup>−</sup><sup>1</sup> *<sup>s</sup>* (*x*)) and becomes part of *resetv*(*x,ev*) in *Hv*. In our example in Sect. 4, the γ<sup>−</sup><sup>1</sup> *<sup>s</sup>* (*x*) would represent *x* in the absolute coordinate system assuming it was represented in the coordinate system defined by segment *s*. The γ*s* (γ<sup>−</sup><sup>1</sup> *<sup>s</sup>* (*x*)) would represent γ<sup>−</sup><sup>1</sup> *<sup>s</sup>* (*x*) in the new coordinate system defined by segment *s* . The *guardv*(*ev*) would be the union of rotated hyperrectangles centered at the origin that result from translating and rotating the guards of the edges represented by *ev*. The initial set Θ of *H* is mapped to Θ*<sup>v</sup>* = γ*sinit*(Θ), the initial set of *Hv*. A formal definition of symmetry abstractions can be found in [1] (or [14]).

The unsafe map *O* is mapped to *Ov*, where ∀*sv* ∈ *Sv,Ov*(*sv*) = ∪*s*∈*S,*ρ*s*(*s*)=*sv* γ*<sup>s</sup>*(*O*(*s*)). That means that the obstacles near any segment *s* ∈ *S* in the environment will be mapped to be near its representative segment ρ*<sup>s</sup>*(*s*) in *Hv*.

A forward simulation relation between *H* and *Hv* can show that if *Hv* is safe with respect to *Ov*, then *H* is safe with respect to *O*. More formally, if ∀*sv* ∈ *Sv,ReachHv* (*sv*)∩ *Ov*(*sv*) = 0, then / ∀*s* ∈ *S,ReachH*(*s*)∩*O*(*s*) = 0 [ / 14].

#### **6** SceneChecker **Algorithm Overview**

A sketch of the core abstraction-refinement algorithm is shown in Algorithm 1. It constructs a symmetry abstraction *Hv* of the concrete automaton *H* resulting from the Hybrid constructor. SceneChecker attempts to verify the safety of *Hv* using traditional reachability analysis. SceneChecker uses a *cache* to store per-mode initial sets from which reachsets have been computed and thus avoids repeating computations. An example run is shown in Fig. 1.

**Fig. 1.** A simple scenario with a car following a plan with six segments is shown in B. Set of initial positions (green square), unsafe set (grey), and the segments (black lines). The automaton (A) has one mode per segment. Translation and rotation symmetries are used to abstract A to the automaton C. The abstraction translates and rotates each segment of the original scenario to a segment aligned with the *x*-axis and ends at the origin resulting in the segments (i.e. modes) *s*<sup>0</sup> *v* and *s*<sup>1</sup> *<sup>v</sup>* . The unsafe set is transformed accordingly for each mode as shown in D. SceneChecker computes the reachset of C which turns out to be unsafe; to illustrate the process this abstract reachset transformed to the original scenario is shown in E. The colors refer to a different abstract modes. The algorithm refines C to F by adding *s*<sup>2</sup> *<sup>v</sup>* (same segment as *s*<sup>1</sup> *<sup>v</sup>* but different guard). The reachset of F is safe and the algorithm terminates (H). (The colored figure is available in the online version of this paper)

The core algorithm verify (Algorithm 2) is called iteratively. If verify returns (*safe,*⊥) or (*unknown,*⊥), SceneChecker returns the same result. If verify instead results in (*refine,s*∗ *<sup>v</sup>* ), splitMode (check the extended version of this paper [1] for the formal definition) is called to refine *Hv* by splitting *s*<sup>∗</sup> *<sup>v</sup>* into two modes *s*<sup>1</sup> *<sup>v</sup>* and *s*<sup>2</sup> *<sup>v</sup>* . Each of **Algorithm 1.** SceneChecker(Φ = {(γ*s,*ρ*<sup>s</sup>*)}*s*∈*S,H,O*)

1: *Hv,Ov* <sup>←</sup> abstract(*H,O,*Φ) 2: ∀*s* ∈ *S,rv*[*s*] ← ρ*<sup>s</sup>*(*s*) 3: **while** *True* **do** 4: *cache* ← {*sv* → 0/ | *sv* ∈ *Sv*} 5: *result,s*∗ *<sup>v</sup>* <sup>←</sup> verify(*rv*[*sinit*]*,*Θ*<sup>v</sup>, cache,rv,Hv,Ov*) 6: **if** *result* = *safe* or *unknown* **then return:** *result*

7: **else** *rv,Hv,Ov* <sup>←</sup> splitMode(*s*<sup>∗</sup> *<sup>v</sup> ,rv,Hv,Ov,H,O*)

the two modes would represent part of the set of the segments of *S* that were originally mapped to *sv* in *rv*. Then the edges, guards, resets, and the unsafe sets related to *sv* are split according to their definitions.

The function verify executes a *depth first search* (DFS) over the mode graph of *Hv*. For any mode *sv* being visited, computeReachset computes *Rv*, an over-approximation of the agent's reachset starting from *initset* following segment *sv* for time *tboundv*(*sv*). If *Rv* <sup>∩</sup> *Ov*(*sv*) = 0,/ verify recursively calls *sv*'s children continuing the DFS in line 6. Before calling each child, its initial set is computed and the part for which a reachset has already been computed and stored in *cache* is subtracted. If all calls return *safe*, then *initset* is added to the other initial sets in *cache*[*sv*] (line 12) and verify returns *safe*. Most importantly, if verify returns (*refine,s*<sup>∗</sup> *<sup>v</sup>* ) for any of *sv*'s children, it directly returns (*refine,s*∗ *<sup>v</sup>* ) for *sv* as well (line 7). If any child returns *unknown* or *Rv* intersects *Ov*(*sv*), verify will need to split *sv*. In that case, it checks if *rv*−1[*sv*] is not a singleton set and thus amenable to splitting (line 10). If *sv* can be split, verify returns (*refine,sv*). Otherwise, verify returns (*unknown,*⊥) implicitly asking one of *sv*'s ancestors to be split instead.

*Correctness.* SceneChecker ensures that all the refined automata *Hv*'s are abstractions of the original hybrid automaton *H* (a proof is given in the extended version of this paper [1]). For any mode with a reachset intersecting the unsafe set, SceneChecker keeps refining that mode and its ancestors until safety can be proven or *Hv* becomes *H*.

# **Theorem 1 (Soundness).** *If* SceneChecker *returns safe, then H is safe.*

If verify is provided with the concrete automaton *<sup>H</sup>* and unsafe set *<sup>O</sup>*, it will be the traditional safety verification algorithm having no over-approximation error due to abstraction. If such a call to verify returns *safe*, then SceneChecker is guaranteed to return *safe*. That means that the refinement ensures that the over-approximation error of the reachset caused by the abstraction is reduced to not alter the verification result.

*Counter-examples.* SceneChecker currently does not find counter-examples to show that the scenario is *unsafe*. There are several sources of over-approximation errors, namely, computeReachset and guard intersections. Even after all the overapproximation errors from symmetry abstractions are eliminated, as refinement does, it still cannot infer unsafe executions or counter-examples because of the other errors. We plan to address this in the future by combining the current algorithm with systematic simulations.

# **Algorithm 2.** verify(*sv,initset,cache,rv,Hv,Ov*)

1: *Rv* <sup>←</sup> computeReachset(*initset,sv*) 2: **if** *Rv* ∩*Ov*(*sv*) = 0/ **then** 3: **for** *s <sup>v</sup>* ∈ *children*(*sv*) **do** 4: *initset* ← *resetv*(*guardv*((*sv,s v*))∩*Rv*)\*cache*[*s v*] 5: **if** *initset* = 0/ **then** 6: *result,s*∗ *<sup>v</sup>* <sup>←</sup> verify(*<sup>s</sup> v,initset , cache,rv,Hv,Ov*) 7: **if** *result* = *refine* **then return:** *refine,s*∗ *v* 8: **else if** *result* = *unknown* **then break** 9: **if** *Rv* ∩*Ov*(*sv*) = 0 or / *result* is *unknown* **then** 10: **if** <sup>|</sup>*rv*−1[*sv*]<sup>|</sup> *<sup>&</sup>gt;* <sup>1</sup> **then return:** *refine,sv* 11: **else return:** *unknown,*⊥ 12: *cache*[*sv*] ← *cache*[*sv*]∪*initset* 13: **return:** *safe,*⊥

#### **7 Experimental Evaluation**

*Agents and Controllers.* In our experiments, we consider two types of nonlinear agent models: a standard 3-dimensional car (C) with bicycle dynamics and 2 inputs, and a 6-dimensional quadrotor (Q) with 3 inputs. For each of these agents, we developed a PD controller and a NN controller for tracking segments. The NN controller for the quadrotor is from Verisig's paper [9] but modified to be rotation symmetric (check the extended version of this paper [1] for more details). Similarly, the NN controller for the car is made rotation symmetric. Both NN controllers are translation symmetric as they take as input the difference between the agent's state and the segment being followed. The PD controllers are translation and rotation symmetric by design.

*Symmetries.* We experimented with two different collections of symmetry maps Φs: 1) translation symmetry (T), where for any segment *s* in *G*, γ*<sup>s</sup>* maps the states so that the coordinate system is translated by a vector that makes its origin at the end waypoint of *s*, and 2) rotation and translation symmetry (TR), where instead of just translating the origin, Φ rotates the *xy*-plane so that *s* is aligned with the *x*-axis, which we described in Sect. 4. For each agent and one of its controllers, we manually verified that condition (1) is satisfied for each of the two Φs using the sufficient condition for ODEs in Sect. 4.

*Scenarios.* We created four scenarios with 2D workspaces (S1-4) and one scenario with a 3D workspace (S5) with corresponding plans. We generated the plans using an RRT planner [31] after specifying a number of goal sets that should be reached. We modified S4 to have more obstacles but still have the same plan and named the new version S4.b and the original one S4.a. When the quadrotor was considered, the waypoints of the 2D scenarios (S1-4) were converted to 3D representation by setting the altitude for each waypoint to 0. Scenario S5 is the same as S2 but S5's waypoints have varying altitudes. The scenarios have different complexities ranging from few segments and obstacles to hundreds of them. All scenarios are safe when traversed by any of the two agents.

We verify these scenarios using SceneChecker and CacheReach, each with two instances, one with DryVR and the other with Flow\*, implementing computeReachset. We also use DryVR and Flow\* as independent tools to verify the same scenarios. The results of experiments with tools that involve DryVR (i.e., SceneChecker+DryVR, CacheReach+DryVR, and DryVR) are stochastic and change between runs. The reason is that each time DryVR is called, it randomly samples traces of the system from which it computes the requested reachset. We fix the random seed for repeatable results in this section. We show close averaging-based results on SceneChecker's website.

SceneChecker is able to verify all scenarios with PD controllers. The results are shown in Table <sup>14</sup> and plotted for C-S1 using SceneChecker+Flow\* in Fig. 1.

*Observation 1:* SceneChecker *offers fast scenario verification and boosts existing reachability tools* Looking at the two total time (Tt) columns for the two instances of SceneChecker with the corresponding columns for Flow\* and DryVR, it becomes clear that symmetry abstractions can boost the verification performance of reachability engines. For example, in C-S4.a, SceneChecker+DR was around 20<sup>×</sup> faster than DryVR. In C-S3, SceneChecker with Flow\* was around 16<sup>×</sup> faster than Flow\*. In scenario Q-S5, SceneChecker timed out at least in part because a computeReachset call to Flow\* timed out. Even when many refinements are required and thus causing several repetitions of the verification process in Algorithm 1, SceneChecker is still faster than DryVR and Flow\* (C-S4.b). All three tools resulted in *safe* for all scenarios when completed executions.

*Observation 2:* SceneChecker *is faster and more accurate than* CacheReach Since CacheReach only handles single-path plans, we only verify the longest path in the plans of the scenarios in its experiments. CacheReach's instance with Flow\* resulted in unsafe reachsets in C-S1 and C-S4.b scenarios likely because of the caching overapproximation error. In all scenarios where CacheReach completed verification besides C-S4.b, it has more Rc and longer Tt (more than 30× in C-S2) while verifying simpler plans than SceneChecker using the same reachability subroutine. In all Q scenarios, both instances of CacheReach, with Flow\* and DryVR, timed out.

*Observation 3: More symmetric dynamics result in faster verification time* SceneChecker usually runs slower in 3D scenarios compared to 2D ones (Q-S2 vs. Q-S5) in part because there is no rotational symmetry in the *z*-dimension to exploit. That leads to larger abstract automata. Therefore, many more calls to computeReachset are required.

We only used SceneChecker's instance with DryVR for agents with NNcontrollers5. We tried different Φs. The results are shown in Table 2. When not using abstraction-refinement, SceneChecker took 10.5, 130.95, and 74.15 min for the QNN-S2, QNN-S3, and QNN-S4 scenarios, while DryVR took 5.22, 52.56, and 61.31 min for the same scenarios, respectively. Comparing these results with those in Table 2 shows

<sup>4</sup> Figures presenting the reachsets of the concrete and abstract automata for different scenarios can be found in the extended version of this paper [1] as well as the machine specifications.

<sup>5</sup> Check the extended version [1] for a discussion about our attempts for using other verification tools for NN-controlled systems as reachability subroutines.


**Table 1.** Comparison between SceneChecker, DryVR (DR), Flow\* (F\*), and CacheReach (CacheR). Both SceneChecker and CacheReach use reachability tools as subroutines. The subroutines used are specified after the '+' sign. Φ is TR. The table shows the number of mode-splits performed (Nrefs), the total number of calls to computeReachset (Rc), the total time spent in reachset computations (Rt), and the total computation time in minutes (Tt). In scenarios where a tool ran over 120 min, we marked the Tt column as 'Timed out' (TO) and the other ones as 'Not Available' (NA).

that the speedup in verification time of SceneChecker is caused by the abstractionrefinement algorithm, achieving more than 13× in certain scenarios (QNN-S4 using Φ <sup>=</sup> T). SceneChecker+DR was more than 10<sup>×</sup> faster than DryVR in the same scenario.

**Table 2.** Comparison between Φs. In addition to the statisitics of Table 1, this table reports the number of modes and edges in the initial and final (after refinement) abstractions (|*Sv*| *i* , |*Ev*| *i* ; <sup>|</sup>*Sv*|*<sup>f</sup>* , and <sup>|</sup>*Ev*|*<sup>f</sup>* , respectively)


*Observation 4: Choice of* Φ *is a trade-off between over-approximation error and number of refinements* The choice of Φ affects the number of refinements performed and the total running times (e.g. QNN-S2, QNN-S3, and QNN-S4). Using TR leads to a more succinct *Hv* but larger over-approximation error causing more mode splits. On the other hand, using T leads to a larger *Hv* but less over-approximation error and thus fewer refinements. This trade-off can be seen in Table 2. For example, QNN-S4 with Φ = T resulted in zero mode splits leading to |*Sv*| *<sup>i</sup>* <sup>=</sup> <sup>|</sup>*Sv*|*<sup>f</sup>* <sup>=</sup> 7, while Φ = TR resulted in 4 mode splits, starting with |*Sv*| *<sup>i</sup>* <sup>=</sup> 1 modes and ending with <sup>|</sup>*Sv*|*<sup>f</sup>* <sup>=</sup> 5, and longer verification time because of refinements. On the other hand, in QNN-S3, Φ = TR resulted in Nref<sup>=</sup> 5, <sup>|</sup>*Sv*|*<sup>f</sup>* <sup>=</sup> 6, and Tt<sup>=</sup> <sup>12</sup>*.*7 min while Φ <sup>=</sup>T resulted in Nref<sup>=</sup> 4, <sup>|</sup>*Sv*|*<sup>f</sup>* <sup>=</sup> 11, and Tt= 16*.*88 min.

*Observation 5: Complicated dynamics require more verification time* Different vehicle dynamics affect the number of refinements performed and consequently the verification time (e.g. QNN-S2, QNN-S4, CNN-S2, and CNN-S4). The car appears to be less stable than the quadrotor leading to longer verification time for the same scenarios. This can also be seen by comparing the results of Tables 1 and 2. The PD controllers lead to more stable dynamics than the NN controllers requiring less total computation time for both agents. More stable dynamics lead to tighter reachsets and fewer refinements.

#### **8 Limitations and Discussions**

SceneChecker allows the choice of modes to be changed from segments to waypoints or sequences of segments as well. The waypoint-defined modes eliminate the need for segments of *G* to have few unique lengths, but only allow Φ <sup>=</sup> T. SceneChecker splits only one mode per refinement and then repeats the computation from scratch. It has to refine many times in unsafe scenarios until reaching the result *unknown*. We plan to investigate other strategies for eliminating spurious counter-examples and returning valid ones in unsafe cases. In the future, it will be important to address other sources of uncertainty in scene verification such as moving obstacles, interactive agents, and other types of symmetries such as permutation and time scaling. Finally, it will be useful to connect a translator to generate scene files from common road simulation frameworks such as CARLA [32], commonroad [33], and Scenic [34].

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Effective Hybrid System Falsification Using Monte Carlo Tree Search Guided by QB-Robustness**

Zhenya Zhang1(B) , Deyun Lyu<sup>1</sup> , Paolo Arcaini<sup>2</sup> , Lei Ma1,3,4 , Ichiro Hasuo<sup>2</sup> , and Jianjun Zhao<sup>1</sup>

 Kyushu University, Fukuoka, Japan zhang.zhenya.623@m.kyushu-u.ac.jp <sup>2</sup> National Institute of Informatics, Tokyo, Japan University of Alberta, Edmonton, Canada Alberta Machine Intelligence Institute, Edmonton, Canada

**Abstract.** Hybrid system falsification is an important quality assurance method for cyber-physical systems with the advantage of scalability and feasibility in practice than exhaustive verification. Falsification, given a desired temporal specification, tries to find an input of violation instead of a proof guarantee. The state-of-the-art falsification approaches often employ stochastic hill-climbing optimization that minimizes the degree of satisfaction of the temporal specification, given by its quantitative *robust semantics*. However, it has been shown that the performance of falsification could be severely affected by the so-called *scale problem*, related to the different scales of the signals used in the specification (e.g., rpm and speed): in the robustness computation, the contribution of a signal could be *masked* by another one. In this paper, we propose a novel approach to tackle this problem. We first introduce a new robustness definition, called *QB-Robustness*, which combines classical Boolean satisfaction and quantitative robustness. We prove that QB-Robustness can be used to judge the satisfaction of the specification and avoid the scale problem in its computation. QB-Robustness is exploited by a falsification approach based on Monte Carlo Tree Search over the structure of the formal specification. First, tree traversal identifies the sub-formulas for which it is needed to compute the quantitative robustness. Then, on the leaves, numerical hill-climbing optimization is performed, aiming to falsify such sub-formulas. Our in-depth evaluation on multiple benchmarks demonstrates that our approach achieves better falsification results than the state-of-the-art falsification approaches guided by the classical quantitative robustness, and it is largely not affected by the scale problem.

**Keywords:** Falsification · Signal temporal logic · Scale problem · Monte carlo tree search · Robust semantics · QB-Robustness

This work is supported in part by JSPS KAKENHI Grant No. 20H04168, 19K24348, 19H04086, JST-Mirai Program Grant No. JPMJMI20B8, Japan. Lei Ma is also supported by Canada CIFAR AI Program and Natural Sciences and Engineering Research Council of Canada. Paolo Arcaini and Ichiro Hasuo are supported by ERATO HASUO Metamathematics for Systems Design Project (No. JPMJER1603), JST.

c The Author(s) 2021

A. Silva and K. R. M. Leino (Eds.) CAV 2021, LNCS 12759, pp. 595–618, 2021. https://doi.org/10.1007/978-3-030-81685-8\_29

#### **1 Introduction**

Cyber-Physical Systems (CPS) are *hybrid systems* that combine physical systems (with continuous dynamics) and digital controllers (that are inherently discrete). Being often safety-critical, their quality assurance is of great importance and widely investigated by both academia and industry. The continuous dynamics of hybrid systems leads to infinite search spaces, making their verification often extremely difficult.

*Falsification* has been proposed as a more practically feasible approach that tackles the dual problem of verification: instead of exhaustively proving a property, falsification intends to uncover the existence of its violation with counterexamples. Formally, the problem is defined as follows. Given a *model* M taking an input signal **<sup>u</sup>** and outputting a signal <sup>M</sup>(**u**), and a *specification* <sup>ϕ</sup> (a temporal formula), the falsification problem consists in finding a *falsifying input*, i.e., an input signal **<sup>u</sup>** such that the corresponding output <sup>M</sup>(**u**) violates <sup>ϕ</sup>.

The most pursued and successful approach to the falsification problem consists in turning it into an optimization problem; we call it *optimization-based falsification*. This is possible thanks to the quantitative *robust semantics* of temporal formulas [14,19]. Robust semantics extends the classical Boolean satisfaction relation **<sup>v</sup>** <sup>|</sup><sup>=</sup> <sup>ϕ</sup> in the following way: it assigns a value **<sup>w</sup>**, ϕ <sup>∈</sup> <sup>R</sup>∪{∞, −∞} (i.e., *robustness*) that tells not only whether ϕ is satisfied or violated (by the sign), but also *how robustly* the formula is satisfied or violated.

Optimization-based falsification approaches adopt *hill-climbing* stochastic optimization strategies to generate inputs to decrease robustness, which terminate when they find an input with negative robustness, i.e., a *falsifying input* that triggers the violation of the specification ϕ. Different optimization-based falsification algorithms have been proposed (see [26] for a survey), and mature tools (e.g., Breach [13] and S-TaLiRo [4]) have also been developed.

The *scale problem* is a recognized issue in optimization-based falsification [21, 40], which could arise when multiple signals with different scales are present in the specification. Namely, it is due to the computation of robust semantics of Boolean connectives, i.e., the way in which the robustness values of different sub-formulas are compared and aggregated: such computation is problematic in the presence of signals that take values having different order of magnitudes.

**Example 1.** As very simple example, let us consider the formula <sup>ϕ</sup> <sup>≡</sup> -[0,30](ϕ<sup>1</sup> <sup>∧</sup> <sup>ϕ</sup>2), with <sup>ϕ</sup><sup>1</sup> <sup>≡</sup> *gear* <sup>&</sup>lt; 6 and <sup>ϕ</sup><sup>2</sup> <sup>≡</sup> *speed* <sup>&</sup>lt; 130. It is apparent that ϕ<sup>1</sup> is always satisfied (in any car model with 5 gears), and it has been added in the specification as redundant check.<sup>1</sup> According to robust semantics, the Boolean connective ∧ is interpreted by minimum , and the "always" operator -[0,30] is interpreted by infimum ; the robustness of an atomic formula <sup>f</sup>(*x*) < c is given by the margin <sup>c</sup> <sup>−</sup> <sup>f</sup>(*x*). Therefore, the robustness of <sup>ϕ</sup> under the signal (*gear* , *speed*), where *gear* , *speed* : [0, 30] <sup>→</sup> <sup>R</sup>, is

<sup>1</sup> Note that we built such a trivial example just to make the scale problem very easy to understand. However, in general, the scale problem frequently occurs on much less trivial specifications, as we will see in the experiments.


In this paper, we propose a novel approach to tackle the scale problem in optimization-based falsification. Our intuition and insights are that we should try to avoid the comparison of robustness values of different sub-formulas, so that one sub-formula does not mask the contribution of another one.

To achieve this, we first propose a new way of computing the satisfaction of a formula that combines *quantitative* robust semantics and *Boolean* semantics. We name the new semantics as QB-Robustness. QB-Robustness, for each type of formula ϕ, requires selecting a sub-formula ϕ<sup>k</sup> among its sub-formulas {ϕ1,...,ϕK}. For <sup>ϕ</sup>k, the quantitative robust semantics is computed, while for the other sub-formulas the Boolean semantics is computed. Therefore, the computation of QB-Robustness requires identifying a path Σ along the parse tree of the formula ϕ, where visited sub-formulas are those for which the quantitative robustness is computed. We prove that QB-Robustness, independently of the selected Σ, is equivalent (in terms of sign and satisfaction) to the quantitative robust semantics (and also to the Boolean one).

In general QB-Robustness is a useful tool for avoiding the scale problem of falsification. By definition, the quantitative robustness of different sub-formulas is never compared, so removing the main cause of the scale problem. It would then make sense to use it for guiding the optimization-based falsification process. However, QB-Robustness requires to choose a particular sequence Σ of subformulas for which to compute the quantitative robustness. It is relatively easy to show that some of them provide a better guidance than others to the falsification search. Considering the previous example, if Σ contains ϕ1, we can encounter the problem that the quantitative robustness of ϕ<sup>1</sup> would not provide any guidance (i.e., no big variations in the robustness values would be observed). On the other hand, if Σ contains ϕ2, the quantitative robustness would have larger variations, providing more effective guidance to the search.

Then, the key problem is how to select the *best* Σ, that enables the hillclimbing optimization used in falsification to be more effective. In general, although it is often difficult to know the best Σ in advance, it is still possible to learn it by observing sampling results using different Σ. Based on this intuition, we propose a novel falsification approach that identifies the sequences Σ that is more likely to be efficient, and uses them in the new falsification trials. Our approach could be seen as an instantiation of the classical Monte Carlo Tree Search (MCTS) method [8,28], which is able to efficiently tackle the *exploration-exploitation* tradeoff. In our context, exploration consists in incrementally constructing the tree that represents all the possible sequences, and exploitation consists in selecting the best Σ and running optimization-based falsification in which QB-Robustness with Σ is used.

Overall, the major *Contributions* of this paper are summarized as follows:


*Paper Structure.* In Sect. 2, we introduce the preliminaries of the optimizationbased falsification. In Sect. 3, we introduce the novel STL semantics QB-Robustness, and, in Sect. 4, we describe the MCTS-based falsification approach that uses QB-Robustness. In Sect. 5, we describe the experiments and evaluation results. Finally, we discuss most relevant work to ours in Sect. 6, and conclude the paper in Sect. 7.

#### **2 Preliminaries**

In this section, we briefly review the falsification framework based on *robust semantics* of temporal logic [14].

Let <sup>T</sup> <sup>∈</sup> <sup>R</sup><sup>+</sup> be a positive real. An <sup>M</sup>*-dimensional signal* with a time horizon <sup>T</sup> is a function **<sup>w</sup>**: [0, T] <sup>→</sup> <sup>R</sup><sup>M</sup>. We treat the system model as a black box, i.e., its behaviors are only observed from inputs and their corresponding outputs. Formally, a *system model*, with M-dimensional input and N-dimensional output, is a function <sup>M</sup> that takes an input signal **<sup>u</sup>**: [0, T] <sup>→</sup> <sup>R</sup><sup>M</sup> and returns a signal <sup>M</sup>(**u**): [0, T] <sup>→</sup> <sup>R</sup><sup>N</sup> . Here the common time horizon <sup>T</sup> <sup>∈</sup> <sup>R</sup><sup>+</sup> is arbitrary.

**Definition 1 (STL Syntax).** We fix a set **Var** of variables. In Signal Temporal Logic (STL), *atomic propositions* and *formulas* are defined as follows, respectively: <sup>α</sup> ::<sup>≡</sup> <sup>f</sup>(x1,...,x<sup>N</sup> ) <sup>&</sup>gt; 0, and <sup>ϕ</sup> ::<sup>≡</sup> <sup>α</sup> |⊥|¬<sup>ϕ</sup> <sup>|</sup> <sup>ϕ</sup> <sup>|</sup> <sup>ϕ</sup> <sup>|</sup> -<sup>I</sup><sup>ϕ</sup> <sup>|</sup> ♦I<sup>ϕ</sup> <sup>|</sup> <sup>ϕ</sup> <sup>U</sup><sup>I</sup> <sup>ϕ</sup> Here <sup>f</sup> is an <sup>N</sup>-ary function <sup>f</sup> : <sup>R</sup><sup>N</sup> <sup>→</sup> <sup>R</sup>, <sup>x</sup>1,...,x<sup>N</sup> <sup>∈</sup> **Var**, and <sup>I</sup> is a closed non-singular interval in <sup>R</sup>≥<sup>0</sup>, i.e., <sup>I</sup> = [a, b] or [a,∞), where a, b <sup>∈</sup> <sup>R</sup> and a<b. -,♦ and <sup>U</sup> are temporal operators, which are usually known as *always*, *eventually* and *until* respectively. The always operator and eventually operator ♦ can also be considered as special cases of the until operator U, where ♦I<sup>ϕ</sup> ≡U<sup>I</sup> <sup>ϕ</sup> and -<sup>I</sup><sup>ϕ</sup> ≡ ¬♦I¬ϕ. Other common connectives such as <sup>→</sup>, are introduced as syntactic sugar: ≡ ¬⊥, <sup>ϕ</sup><sup>1</sup> <sup>→</sup> <sup>ϕ</sup><sup>2</sup> ≡ ¬ϕ<sup>1</sup> <sup>∨</sup> <sup>ϕ</sup>2.

**Definition 2 (Quantitative Robust Semantics).** Let **<sup>w</sup>**: [0, T] <sup>→</sup> <sup>R</sup><sup>N</sup> be an <sup>N</sup>-dimensional signal, and <sup>t</sup> <sup>∈</sup> [0, T). The <sup>t</sup>*-shift* **<sup>w</sup>**<sup>t</sup> of **<sup>w</sup>** is the signal **<sup>w</sup>**<sup>t</sup> : [0, T <sup>−</sup> <sup>t</sup>] <sup>→</sup> <sup>R</sup><sup>N</sup> defined by **<sup>w</sup>**<sup>t</sup> (t ) := **w**(t + t ). Let ϕ be an STL formula. We define the *robustness* **<sup>w</sup>**, ϕ <sup>∈</sup> <sup>R</sup> ∪ {∞, −∞} as follows, by induction on the construction of formulas. and denote infimums and supremums of real numbers, respectively. , the binary version of , denotes minimum.

$$\begin{array}{lcl} \left[\mathbf{w}, f(x\_{1}, \cdots, x\_{N}) > 0\right] &:= f\left(\mathbf{w}(0)(x\_{1}), \cdots, \mathbf{w}(0)(x\_{N})\right) \\ \left[\mathbf{w}, \bot\right] &:= -\infty & \left[\mathbf{w}, \neg\varphi\right] &:= -\left[\mathbf{w}, \varphi\right] \\ \left[\mathbf{w}, \bigwedge\_{i} \varphi\_{i}\right] &:= \bigcap\_{i} \left[\mathbf{w}, \varphi\_{i}\right] & \left[\mathbf{w}, \bigvee\_{i} \varphi\_{i}\right] &:= \bigcup\_{i} \left[\mathbf{w}, \varphi\_{i}\right] \\ \left[\mathbf{w}, \bigsquare\_{I} \varphi\right] &:= \bigcap\_{t \in I \cap [0, T]} \left[\mathbf{w}^{t}, \varphi\right] & \left[\mathbf{w}, \lozenge\_{I} \varphi\right] &:= \bigcup\_{t \in I \cap [0, T]} \left[\mathbf{w}^{t}, \varphi\right] \\ \left[\mathbf{w}, \varphi\_{1} \bigsquare\_{I} \varphi\_{2}\right] &:= \bigcup\_{t \in I \cap [0, T]} \left(\left[\mathbf{w}^{t}, \varphi\_{2}\right] \sqcap \prod\_{t' \in [0, t)} \left[\mathbf{w}^{t'}, \varphi\_{1}\right]\right) \end{array}$$

The original STL semantics is Boolean, given by a binary relation |= between signals and formulas. The robust semantics refines the Boolean one in the following sense: **<sup>w</sup>**, ϕ <sup>&</sup>gt; 0 implies **<sup>w</sup>** <sup>|</sup><sup>=</sup> <sup>ϕ</sup>, and **<sup>w</sup>**, ϕ <sup>&</sup>lt; 0 implies **<sup>w</sup>** <sup>|</sup><sup>=</sup> <sup>ϕ</sup>, see [19, Prop. 16].

#### **2.1 Hill Climbing-Guided Falsification**

So far, the falsification problem has received extensive industrial and academic attention. One possible approach direction by hill-climbing optimization is an established field, too: see [2–4,10,13–15,17,26,29,36–39,42] and the tools Breach [13] and S-TaLiRo [4]. We formulate the problem and the methodology, for later use in describing our falsification approach.

**Definition 3 (Falsifying Input).** Let <sup>M</sup> be a system model, and <sup>ϕ</sup> be an STL formula. A signal **<sup>u</sup>**: [0, T] <sup>→</sup> <sup>R</sup>|**Var**<sup>|</sup> is a *falsifying input* if -<sup>M</sup>(**u**), ϕ <sup>&</sup>lt; 0; the latter implies <sup>M</sup>(**u**) <sup>|</sup><sup>=</sup> <sup>ϕ</sup>.

The use of quantitative robust semantics -<sup>M</sup>(**u**), ϕ <sup>∈</sup> <sup>R</sup>∪{∞, −∞} in the above problem enables the use of hill-climbing optimization.

**Definition 4 (Hill Climbing-Guided Falsification).** Assume the setting in Definition 3, for finding a falsifying input, the methodology of *hill climbing-guided falsification* is presented in Algorithm 1. Here, the function Hill-Climb makes a guess of an input signal **u** , aiming at minimizing the robustness -M(**u** ), ϕ. It does so, learning from the sampling history H that contains the previous observations of input signals and their corresponding robustness values.

The Hill-Climb function can be designed based on various stochastic optimization algorithms. Typically, at the early phase of the optimization, the proposal of new input is usually based on random sampling; as the set of sampling history grows larger, the algorithm takes various metaheuristic-based strategies to achieve the optimization goal efficiently. Examples of such algorithms include Covariance Matrix Adaption Evolution Strategy (CMA-ES) [7] (used in our experiments), Simulated Annealing, Global Nelder Mead [32], etc.


#### **Algorithm 1.** Hill climbing-guided falsification

#### **3 QB-Robustness**

The scale problem is a known important issue that negatively affects the performance of falsification, which arises when connective operators (i.e., conjunction and disjunction) with operands that predicate on different signals appear in the STL formula under falsification. According to the classic quantitative robust semantics (see Definition 2), the robustness of those formulas is calculated based on the comparison (minimum for conjunction, and maximum for disjunction) between robustness values coming from the different operand sub-formulas. However, since different signals may differ in magnitude, the comparison may be biased, such that one signal **w** may always (or often) *mask* the contribution of the others, and, therefore, the final robustness may be dominated by this signal **w**. Note that, although the scale problem affects connective operators, it is not only local to the place of their application, but it is always propagated to the robustness of the whole formula. The scale problem has been shown as a root cause of the failure of many falsification problems [21,40].

In this work, we propose a novel approach for solving the *scale problem* in falsification. Our approach consists in introducing a new semantics for STL that does not suffer from the scale problem. Such new semantics will be used in a falsification approach based on Monte Carlo Tree Search. We describe details of the new semantics in this section, and the new falsification approach in Sect. 4.

The new proposed semantics, called QB-Robustness, combines quantitative robustness and Boolean satisfaction. By construction, it never compares quantitative robustness values that come from different sub-formulas, thus avoiding the scale problem. QB-Robustness is defined for the whole STL formulas, except for the "until" operator <sup>ϕ</sup><sup>1</sup> <sup>U</sup><sup>I</sup> <sup>ϕ</sup>2, when <sup>ϕ</sup><sup>1</sup> is an arbitrary formula. We still support it as "eventually" and "always" operators<sup>2</sup>, i.e., when <sup>ϕ</sup><sup>1</sup> <sup>=</sup> . Note that this is not a major limitation, as QB-Robustness still supports the majority of specifications that are used in industry: indeed, in the experiments, we were able to

<sup>2</sup> Recall from Definition 1 that the "eventually" and "always" operators are defined in terms of the "until" operator.

handle all the specifications used in falsification competitions [18], which collect benchmarks from industrial case studies.

To better explain the computation of QB-Robustness, we introduce some definitions. Let us first define the notion of immediate sub-formula for STL.

**Definition 5 (Immediate Sub-Formulas).** Let ϕ be an STL formula (see Definition 1). We define the set ISForm(ϕ) of *immediate sub-formulas* of <sup>ϕ</sup> as follows:

ISForm(α) := <sup>∅</sup> ISForm(⊥) := <sup>∅</sup> ISForm(¬ϕ) := ISForm(ϕ) ISForm( - i∈{1,...,K} <sup>ϕ</sup>i) := {ϕ1,...,ϕK} ISForm( i∈{1,...,K} ϕi) := {ϕ1,...,ϕK} ISForm(-<sup>I</sup>ϕ) := ISForm(ϕ) ISForm(♦Iϕ) := ISForm(ϕ)

Intuitively, the immediate sub-formula set of a connective (conjunction or disjunction) contains all its operands. For the other unary operators (temporal operators, negation, etc.), its immediate sub-formula set is given by the immediate sub-formula set of its argument.

The computation of QB-Robustness requires to select some nested immediate sub-formulas. To this aim, we introduce the notion of sub-formula sequence.

**Definition 6 (Sub-Formula Sequence).** Let ϕ be an STL formula. A subformula sequence <sup>Σ</sup> <sup>=</sup> <sup>σ</sup><sup>1</sup> · ... · <sup>σ</sup><sup>L</sup> w.r.t. <sup>ϕ</sup> is defined as follows:

$$\sigma\_1 \in \mathfrak{IsFormal}(\varphi) \qquad \qquad \sigma\_{l+1} \in \mathfrak{IsFormal}(\sigma\_l) \quad \text{with } l = 1, \dots, L - 1$$

where the · is the concatenation operator in the sequence. We use <sup>Σ</sup><sup>k</sup> to denote the kth element of Σ. Moreover, we denote the first element by Σhead, and the last element by <sup>Σ</sup>rear. We use <sup>Σ</sup>head to denote <sup>Σ</sup> without <sup>Σ</sup>head. We identify with <sup>ε</sup> the empty sequence; when ISForm(ϕ) = <sup>∅</sup>, we use <sup>ε</sup> as its sub-formula sequence. We identify with *Σ*<sup>ϕ</sup> the set of all the sub-formula sequences rooted in ϕ.

To be specific, in a sub-formula sequence Σ, each element is one of the subformulas of the previous element. This means that, for Boolean connectives, only one of the operands is selected. Moreover, an atomic sub-formula predicating over a single signal can only appear as the final element of a sequence. We exploit these characteristics of Σ to define QB-Robustness, which combines the quantitative robustness of the sub-formulas related to a given signal with the Boolean satisfaction of the other sub-formulas. QB-Robustness, given a sequence Σ, decides whether to compute the quantitative robust semantics or the Boolean semantics of a sub-formula, by considering whether the sub-formula belongs to Σ or not. This implies that, in the case of conjunction and disjunction, we evaluate the quantitative robustness of the sub-formula in Σ and the Boolean satisfaction of the other sub-formulas. Based on such intuition, we define the semantics of our proposed QB-Robustness in Definition 7, and demonstrate its usefulness in Theorem 1.

**Definition 7 (Semantics of QB-Robustness).** Let ϕ be an STL formula as defined in Definition 1, and <sup>Σ</sup> be a sub-formula sequence w.r.t. <sup>ϕ</sup>. For <sup>ϕ</sup> <sup>≡</sup> <sup>ϕ</sup><sup>i</sup> <sup>|</sup> <sup>ϕ</sup>i, let <sup>ϕ</sup><sup>k</sup> <sup>∈</sup> ISForm(ϕ) be the first element <sup>Σ</sup>head of <sup>Σ</sup>, then we can represent these two cases as <sup>ϕ</sup> <sup>≡</sup> <sup>ϕ</sup><sup>k</sup> <sup>∧</sup> <sup>ϕ</sup><sup>k</sup> <sup>|</sup> <sup>ϕ</sup><sup>k</sup> <sup>∨</sup> <sup>ϕ</sup>k, where <sup>ϕ</sup><sup>k</sup> is the conjunction (or disjunction, respectively) of the other formulas in ISForm(ϕ) \ {ϕk}. The QB-Robustness QBRob(**w**, ϕ, Σ) of <sup>ϕ</sup> w.r.t. <sup>Σ</sup> is defined as follows:

$$\begin{array}{lcl} \mathsf{QBRob}(\mathbf{w},\alpha,\varepsilon) & \coloneqq & [\mathbf{w},\alpha] & \mathsf{QBRob}(\mathbf{w},\bot,\varepsilon) & \coloneqq & -\infty \\ \mathsf{QBRob}(\mathbf{w},\neg\varphi,\Sigma) & \coloneqq & -\mathsf{QBRob}(\mathbf{w},\varphi,\Sigma) \\ \mathsf{QBRob}(\mathbf{w},\varphi\_{k}\wedge\varphi\_{\overline{k}},\Sigma) & \coloneqq & \begin{cases} \mathsf{QBRob}(\mathbf{w},\varphi\_{k},\Sigma\_{\overline{\mathtt{head}}}) & \text{if } \mathbf{w}\nmid\varphi\_{\overline{k}} \\ -\infty & \text{otherwise} \end{cases} \\ \mathsf{QBRob}(\mathbf{w},\varphi\_{k}\vee\varphi\_{\overline{k}},\Sigma) & \coloneqq & \begin{cases} \mathsf{QBRob}(\mathbf{w},\varphi\_{k},\Sigma\_{\overline{\mathtt{head}}}) & \text{if } \mathbf{w}\nmid\varphi\_{\overline{k}} \\ \infty & \text{otherwise} \end{cases} \\ \mathsf{QBRob}(\mathbf{w},\square\_{I}\varphi,\Sigma) & \coloneqq & \bigcap\_{t\in I}\mathsf{QBRob}(\mathbf{w}^{t},\varphi,\Sigma) \\ \mathsf{QBRob}(\mathbf{w},\diamondsuit\_{I}\varphi,\Sigma) & \coloneqq & \bigcup\_{t\in I}\mathsf{QBRob}(\mathbf{w}^{t},\varphi,\Sigma) \end{array}$$

We now prove that the semantics of QB-Robustness is equivalent (in the sense of satisfaction) to the Boolean semantics, and so it can be used to show violation of a specification in a falsification algorithm, as we do in this paper.

**Theorem 1.** *Let* <sup>ϕ</sup> *be an STL formula. Given a signal* **<sup>w</sup>***, for any* <sup>Σ</sup> <sup>∈</sup> *<sup>Σ</sup>*ϕ*, it holds that* QBRob(**w**, ϕ, Σ) <sup>&</sup>gt; <sup>0</sup> *implies* **<sup>w</sup>** <sup>|</sup><sup>=</sup> <sup>ϕ</sup>*. Similarly, for any* <sup>Σ</sup> <sup>∈</sup> *<sup>Σ</sup>*ϕ*, it holds that* QBRob(**w**, ϕ, Σ) <sup>&</sup>lt; <sup>0</sup> *implies* **<sup>w</sup>** <sup>|</sup><sup>=</sup> <sup>ϕ</sup>*.*

*Proof.* We first recall from [19, Prop. 16] that **<sup>w</sup>**, ϕ <sup>&</sup>lt; 0 implies **<sup>w</sup>** <sup>|</sup><sup>=</sup> <sup>ϕ</sup>, and that **<sup>w</sup>**, ϕ <sup>&</sup>gt; 0 implies **<sup>w</sup>** <sup>|</sup><sup>=</sup> <sup>ϕ</sup>. We prove Theorem <sup>1</sup> by induction on the structure of the formula.

	- Case <sup>ϕ</sup> <sup>=</sup> <sup>ϕ</sup> <sup>∧</sup> <sup>ψ</sup>, where <sup>ψ</sup> is an arbitrary formula. Let <sup>Σ</sup> <sup>=</sup> <sup>ϕ</sup> · <sup>Σ</sup> , and let us consider the two cases in which QBRob(**w**, ϕ, Σ) is negative and positive separately:
		- <sup>∗</sup> If QBRob(**w**, ϕ, Σ) <sup>&</sup>lt; 0, there are two sub-cases:

· if QBRob(**w**, ϕ , Σ ) < 0, then **w**, ϕ < 0 (by assumption). Then, by the robust semantics of conjunction, also **w**, ϕ < 0 holds, and so it does **<sup>w</sup>** <sup>|</sup><sup>=</sup> <sup>ϕ</sup>.

· if QBRob(**w**, ϕ , Σ ) > 0, then **w**, ϕ > 0 (by assumption). Then, it holds **<sup>w</sup>** <sup>|</sup><sup>=</sup> <sup>ψ</sup> by Definition 7, and, therefore, it holds **<sup>w</sup>** <sup>|</sup><sup>=</sup> <sup>ϕ</sup>.

<sup>∗</sup> If QBRob(**w**, ϕ, Σ) <sup>&</sup>gt; 0, it means that QBRob(**w**, ϕ , Σ ) > 0 and **<sup>w</sup>** <sup>|</sup><sup>=</sup> <sup>ψ</sup> (by Definition 7). By assumption, if QBRob(**w**, ϕ , Σ ) > 0, then **w**, ϕ <sup>&</sup>gt; 0. Therefore, **<sup>w</sup>** <sup>|</sup><sup>=</sup> <sup>ϕ</sup>.

• Case <sup>ϕ</sup> <sup>=</sup> -<sup>I</sup>ϕ . Let Σ = Σ , and let us consider the two cases in which QBRob(**w**, ϕ, Σ) is negative and positive separately:

<sup>∗</sup> By Definition <sup>7</sup>, QBRob(**w**, ϕ, Σ) <sup>&</sup>lt; 0 indicates that there exists a <sup>t</sup> <sup>∈</sup> <sup>I</sup> such that QBRob(**w**<sup>t</sup> , ϕ , Σ) < 0. By assumption, it holds that **<sup>w</sup>**<sup>t</sup> <sup>|</sup><sup>=</sup> <sup>ϕ</sup> . Then, by the semantics of the always operator -, it holds that **<sup>w</sup>** <sup>|</sup><sup>=</sup> <sup>ϕ</sup>.

<sup>∗</sup> By Definition <sup>7</sup>, QBRob(**w**, ϕ, Σ) <sup>&</sup>gt; 0 indicates that for all <sup>t</sup> <sup>∈</sup> <sup>I</sup> it holds that QBRob(**w**<sup>t</sup> , ϕ , Σ) > 0. Then, by assumption, it holds that for all <sup>t</sup> <sup>∈</sup> <sup>I</sup>, **<sup>w</sup>**<sup>t</sup> <sup>|</sup><sup>=</sup> <sup>ϕ</sup> . So, by the semantics of the always operator -, it holds that **<sup>w</sup>** <sup>|</sup><sup>=</sup> <sup>ϕ</sup>.

• Case <sup>ϕ</sup> <sup>=</sup> <sup>¬</sup>ϕ . Let Σ = Σ , and let us consider the two cases in which QBRob(**w**, ϕ, Σ) is negative and positive separately:

<sup>∗</sup> By Definition 7, QBRob(**w**, ϕ, Σ) <sup>&</sup>lt; 0 indicates that QBRob(**w**, ϕ , Σ ) <sup>&</sup>gt; 0. By assumption, it holds that **<sup>w</sup>** <sup>|</sup><sup>=</sup> <sup>ϕ</sup> , and therefore, **<sup>w</sup>** <sup>|</sup><sup>=</sup> <sup>ϕ</sup>.

<sup>∗</sup> By Definition 7, QBRob(**w**, ϕ, Σ) <sup>&</sup>gt; 0 indicates that QBRob(**w**, ϕ , Σ ) <sup>&</sup>lt; 0. By assumption, it holds that **<sup>w</sup>** <sup>|</sup><sup>=</sup> <sup>ϕ</sup> , and, therefore, **<sup>w</sup>** <sup>|</sup><sup>=</sup> <sup>ϕ</sup>.

• Proofs for the cases of <sup>ϕ</sup> <sup>=</sup> <sup>ϕ</sup> <sup>∨</sup> <sup>ψ</sup> and <sup>ϕ</sup> <sup>=</sup> ♦Iϕ follow similar proof patterns, and so are left to the readers.

We use an example to illustrate how QB-Robustness is used for checking the satisfiability of an STL formula.

**Example 2.** Let **<sup>w</sup>**: [0, T] <sup>→</sup> <sup>R</sup><sup>2</sup> be a 2-dimensional signal and <sup>ϕ</sup> <sup>=</sup> -<sup>I</sup> (ϕ1∨ϕ2) be an STL formula where ϕ<sup>1</sup> and ϕ<sup>2</sup> are two atomic formulas. Intuitively, to make <sup>ϕ</sup> falsified, there must exist <sup>t</sup> <sup>∈</sup> <sup>I</sup> such that **<sup>w</sup>**<sup>t</sup> <sup>|</sup><sup>=</sup> <sup>ϕ</sup><sup>1</sup> and **<sup>w</sup>**<sup>t</sup> <sup>|</sup><sup>=</sup> <sup>ϕ</sup>2. Let us consider a non-trivial falsification problem in which, for most of the signals **w**, sets {<sup>t</sup> <sup>∈</sup> <sup>I</sup> <sup>|</sup> **<sup>w</sup>**<sup>t</sup> <sup>|</sup><sup>=</sup> <sup>ϕ</sup>1} and {<sup>t</sup> <sup>∈</sup> <sup>I</sup> <sup>|</sup> **<sup>w</sup>**<sup>t</sup> <sup>|</sup><sup>=</sup> <sup>ϕ</sup>2} are non-empty and disjoint.

By Definition 7, given the sub-formula sequence Σ = ϕ<sup>1</sup> of ϕ, the corresponding QB-Robustness is QBRob(**w**, ϕ, Σ) = <sup>t</sup>∈<sup>I</sup>QBRob(**w**<sup>t</sup> , ϕ<sup>1</sup> <sup>∨</sup> <sup>ϕ</sup>2, ϕ1), i.e., it takes the infimum of QBRob(**w**<sup>t</sup> , ϕ<sup>1</sup> <sup>∨</sup> <sup>ϕ</sup>2, ϕ1) over <sup>t</sup> <sup>∈</sup> <sup>I</sup>. Again, by Definition 7, for any t <sup>∈</sup> <sup>I</sup>, QBRob(**w**<sup>t</sup>- , ϕ<sup>1</sup> <sup>∨</sup> <sup>ϕ</sup>2, ϕ1) is computed as follows:


Therefore, when <sup>Σ</sup> <sup>=</sup> <sup>ϕ</sup>1, it holds that QBRob(**w**, ϕ, Σ) = <sup>t</sup>∈S**w**<sup>t</sup> , ϕ1, where <sup>S</sup> <sup>=</sup> {<sup>t</sup> <sup>∈</sup> <sup>I</sup> <sup>|</sup> **<sup>w</sup>**<sup>t</sup> <sup>|</sup><sup>=</sup> <sup>ϕ</sup>2}, i.e., the infimum of the quantitative robustness of <sup>ϕ</sup><sup>1</sup> on the interval when ϕ<sup>2</sup> is violated. Indeed, once this value is negative, it means that there exists a point <sup>t</sup> <sup>∈</sup> <sup>I</sup> when both <sup>ϕ</sup><sup>1</sup> and <sup>ϕ</sup><sup>2</sup> are violated; by the Boolean semantics of *always* and *disjunction*, ϕ is violated.

#### **4 MCTS-Based Falsification Guided by QB-Robustness**

QB-Robustness never compares robustness values coming from signals with different magnitudes, and, therefore, it does not suffer from the scale problem. As such, it could be used in falsification approaches instead of the classical pure quantitative robustness.

However, a sub-formula sequence Σ is required when calculating QB-Robustness, and such sequence is not unique (see Definition 7). Note that the selection of the sequence can affect the performance of the numerical optimization algorithms used in falsification. Let us consider <sup>ϕ</sup> <sup>≡</sup> -((*gear* <sup>&</sup>lt; 6)∧(*speed* <sup>&</sup>lt; 130)) as an example. As explained in Sect.1, numerical optimization will perform better if guided by the robustness values coming from *speed* rather than by those coming from *gear* . Therefore, in a falsification approach using QB-Robustness, it is important to select an appropriate sub-formula sequence Σ.

By using the QB-Robustness, the problem of falsifying an STL formula ϕ consists in finding *both* a signal **w** and a sub-formula sequence Σ such that QBRob(**w**, ϕ, Σ) <sup>&</sup>lt; 0. The selection of <sup>Σ</sup> is discrete, while the search for **<sup>w</sup>** is numerical. In order to combine these processes that are different in nature, we propose to adapt Monte Carlo Tree Search (MCTS) [8,28]. In the following, we firstly give a brief introduction to MCTS in Subsect. 4.1, and then present the application of MCTS to our falsification problem in Subsect. 4.2.

#### **4.1 MCTS Background**

MCTS exemplifies the "trial and error" philosophy, and has achieved a great success over the past decade, most notably in fields such as the computer Go game [35]. MCTS explores the action space given by the possible actions of the system; for example, in the Go game, these are the positions where to put the next stone. The approach builds a tree of sequences of actions, and assigns rewards to the different branches. MCTS performs the search by iteratively taking the following four steps. See Fig. 1, where the general scheme is adapted to our current setting, for illustration.


**Fig. 1.** The workflow of MCTS-based falsification guided by QB-Robustness. Let us consider the falsification of an STL formula ϕ = -<sup>I</sup> (ϕ<sup>1</sup> ∨ ϕ2), where ϕ<sup>1</sup> = ϕ<sup>11</sup> ∧ ϕ12. Initially, there is only the root in the tree, so the algorithm selects it for expansion. Then, the algorithm keeps on randomly selecting a child of a non-fully expanded node, until a leaf node is reached. By reaching a leaf, a sub-formula sequence Σ has been constructed; the algorithm performs playout using Σ, by running hill-climbing optimization guided by the QB-Robustness with Σ, to estimate the reward of the path. After playout, the algorithm backpropagates the reward and the number of visits from the leaf to the root. When all the children of a node are expanded, selection is done based on the UCB1 algorithm. After many loops, the algorithm has explored all the possible sub-formula sequences in *Σ*ϕ, and it starts allocating more resources to those branches where hill-climbing optimization progresses more smoothly. The algorithm terminates either when a falsifying input is found, or when the budget is exhausted.

is a real number in [0, 1]. Reward can be interpreted differently in different contexts. For example, in the Go game, the reward of a position is measured by the winning rate when a stone is positioned there; this is estimated by randomly playing the game until the end for n times, and then taking the ratio <sup>n</sup>*<sup>w</sup>* <sup>n</sup> of the number of winning as the winning rate.

– *Backpropagation.* Backpropagation updates the number of visits and the reward of the nodes along the path from the node of playout to the root. These data are used in subsequent loops to decide the branches to explore.

At the end, the action space will be sufficiently explored in an unbalanced manner, by focusing on the most promising sub-spaces giving the highest rewards.

#### **4.2 Proposed QB-Robustness-Guided Falsification Approach**

We here propose a falsification framework based on MCTS in which, during tree construction, we synthesize and select a sub-formula sequence that facilitates the falsification progress the most, and, at the bottom layer of the tree, we run numerical optimization to search for a falsifying input and provide feedback (i.e., backpropagation) to guide the sequence selection.

We formalize our algorithm in Algorithm 2 and visualize its execution in Fig. 1. In the following, we elaborate on our approach.

#### **Algorithm 2.** MCTS-based falsification guided by QB-Robustness

**Require:** a system model M, an STL formula ϕ, and the following tunable parameters: a scalar c for UCB1, an MCTS budget BM, and a playout budget B<sup>P</sup> .


We construct the tree in this way: each node represents a sequence of formulas, and each edge of a node is a sub-formula of the last element of the sequence represented by the node. The root is initialized with a sequence holding ϕ only (Lines 2–3) and some other properties including the number of visits to the different nodes (Line 4), the reward (Line 5), and the history of hill-climbing sampling (Line 6). The main process of MCTS consists in calling the MCTSSearch function iteratively with the root as argument (Line 8), until the exhaustion of the MCTS budget or a falsifying input is found (Line 7). The MCTSSearch function (Line 9) goes through the four phases, namely *selection*, *expansion*, *playout* and *backpropagation*, of the original MCTS algorithm.

**Selection.** Selection happens when a node has children (Line 10) and these have all been expanded (Line 11). It selects a child according to the UCB1 [6] algorithm (Line 12) to take a balance between exploration and exploitation. The exploitation is embodied by the reward <sup>R</sup>(<sup>Σ</sup> · <sup>ϕ</sup>i)—the higher the reward is, the more likely a falsifying input is found following that branch. Exploration, instead, is considered via 2 ln <sup>N</sup>(Σ) <sup>N</sup>(Σ·ϕ*i*) that is negatively correlated to the number of visits to a child—the more the child was visited before, the less chance it will be visited again. The scalar c is a tunable parameter that balances the trade-off between exploration and exploitation. After a child <sup>Σ</sup> · <sup>ϕ</sup><sup>k</sup> is selected, it will be taken as the argument of the next MCTSSearch loop (Line 18).

**Expansion.** If not all the children of a node have been expanded (Line 13), a child will be expanded. Expansion consists in randomly selecting a child from the unexpanded child list (Line 14), adding it to the tree (Line 15), initilizing properties including the number of visits and history (Lines 16–17). After expansion, the newly expanded child will be taken as the argument of the recursive call to MCTSSearch (Line 18).

**Playout.** If a leaf node that has no children to expand is reached, the playout phase will start to devise a reward for evaluating the visited path. In our context, we define the reward based on the best robustness value that can be obtained with the path; specifically, playout consists in running hill-climbing guided falsification to search for a minimal robustness value (Line 22). Note that the sequence Σ represented by a leaf node is actually the concatenation between ϕ and a subformula sequence of ϕ. We extract the suffix of Σ, i.e., the sub-formula sequence, to compute the QB-Robustness as a guidance to the hill-climbing optimization (Line 23). If a negative QB-Robustness is found (Line 24), then the whole algorithm can be terminated and the input signal **u** that triggers the negative QB-Robustness can be returned as the falsifying input (Line 25); otherwise, the sampling history of hill climbing will be saved (Line 26) so that the future playout at the same leaf can be restored from that context. After playout, the reward of the leaf node will be updated based on the definition of the reward, which will be introduced below. *Reward* Since our goal is to find a sequence Σ with which hill-climbing optimization can minimize QBRob(**w**, ϕ, Σ) smoothly, we connect the reward with the hill-climbing progress. Formally, given a sampling history <sup>H</sup>, our reward (Line 27) is defined as Rwd(rb , H) := max rb*h*−min ({rb- }∪rb*h*) max rb*<sup>h</sup>* , where rb<sup>h</sup> is the history of robustness values in <sup>H</sup>.

**Backpropagation.** In MCTS, the playout result of a leaf is backpropagated to the higher layer nodes along the path, so that the future selection on the high layer is referred. Backpropagation updates two properties of each ancestor of the leaf till the root, the reward (Line 19) and the number of visits (Line 28).

**Remark 1 (Approach Complexity).** With respect to classical falsification, our approach introduces an exploration phase for searching the "best" subformula sequence to instantiate QB-Robustness. The number of these sequences corresponds to the number of atomic sub-formulas (and so the leaves of the


**Table 1.** Benchmarks – STL specifications

tree). Considering that most of the time is spent on playout, the complexity of our approach grows linearly with the number of atomic sub-formulas.

#### **5 Experimental Evaluation**

In this section, we present the experiments we conducted to evaluate the effectiveness of the proposed approach. We first introduce the experiment setup in Subsect. 5.1, and then we present the experimental evaluation results by answering three research questions in Subsect. 5.2.

#### **5.1 Experiment Setup**

*Simulink Models and Specifications* As our benchmarks, we selected three Simulink models frequently used in the falsification community (i.e., in the falsification competitions [18]), and 30 specifications defined for them. All these models are complicated hybrid systems with multiple input and output signals. The specifications are STL formulas that formalize system requirements regarding safety, performance, etc. Since we are interested in assessing the influence of the scale problem to the performance of the compared falsification approaches, all the considered specifications predicate over, at least, two signals. Table 1 reports the 30 specifications under test. The IDs of the specifications identify which models they belong to. A description of the three models and of their specifications is as follows.


*Baseline Approach and Our Proposed Approach.* In our experiments, we compare the performances of our proposed approach with the baseline Breach approach. We implemented our approach in the tool ForeSee, which stands for **FOR**mula **E**xploitation by **S**equence tr**EE** for falsification.

Breach is a state-of-the-art falsification tool that implements the classic falsification workflow we introduced in Sect. 2. The quantitative robustness calculation in Breach is based on the robust semantics given in Definition 2. Breach also encapsulates several stochastic optimization algorithms, such as CMA-ES, Simulated Annealing, etc. The implementation of our ForeSee approach uses Breach only for interfacing with the Simulink model and for the calculation of quantitative robustness; instead, the calculation of QB-Robustness, and the implementation of the MCTS algorithm are novel. Since CMA-ES has proved to be the state-of-the-art stochastic algorithm [39], we select CMA-ES as our backend optimizer for the playout phase.<sup>3</sup>

<sup>3</sup> ForeSee is available at https://github.com/choshina/ForeSee.

We apply the two approaches, ForeSee and Breach, to each benchmark specification reported in Table 1. Since both approaches are based on stochastic optimization, we repeat each experiment for 30 times, as suggested by a guideline for conducting experiments with randomized algorithms [5]. For each experiment, both approaches have been given a total timeout B<sup>M</sup> of 900 s (see Algorithm 2).

*Evaluation Metrics.* As first evaluation metric, we compute the *falsification rate (FR)* as the number of runs (out of 30) in which the approach returns a falsifying input. Therefore, FR is an indicator of the *effectiveness* of an approach, i.e., it reflects the ability of an algorithm to falsify the specification. As second evaluation metric, we compute the *average time* (seconds), as average execution time of the successful falsification runs. Therefore, the average time is an indicator of the *efficiency* of the approach. We do not report the number of simulations because these are consistent with the execution time.

*Experiment Platform.* In our experiments, we use Breach [13] (ver 1.2.13) with CMA-ES (the state of the art). Breach accepts piece-wise constant signals as input for the Simulink models; we use the same settings used in falsification competitions [18]: we use piece-wise constant signals with five control points for AT and AFC, and with four control points for FFR. As configuration of MCTS (see Algorithm 2), we set the UCB1 scalar c to 0.2, and the playout budget B<sup>P</sup> to 10 generations. The experiments have been executed on an Amazon EC2 c4.2xlarge instance (2.9 GHz Intel Xeon E5-2666 v3, 15 GB RAM).

#### **5.2 Evaluation**

We here analyze the experimental results using three research questions (RQs).

**RQ1.** *Does the proposed approach perform better than state-of-the-art falsification approaches?*

In this RQ, we aim at assessing whether the proposed approach is indeed able to tackle the scale problem in falsification and performs better than state-of-theart approaches. Table 2 reports, for each specification benchmark, the falsification rate FR and the average execution time of our proposed approach ForeSee and of the baseline Breach. The table further reports the difference of the two metrics between the two approaches. We highlight in gray the best results in which ForeSee has an FR of 15 units higher than Breach. We observe that for 25 benchmarks out of 30, ForeSee has a better FR, and in 15 of these the improvement is significant (selected in gray). Note that there are notable cases, such as AT3, AT13, AT16, and AT17, in which Breach only finds at most two falsifying inputs, while ForeSee finds always at least 29 falsifying inputs. In four cases, Breach has a better FR: while for AT8, AFC6, and FFR2 the difference is minimal, it is quite large for AT14. We further inspected such specification and its corresponding model (see Table 1); we noticed that all the sub-formulas in AT14 must be falsified to falsify the whole specification<sup>4</sup>, and they are all

<sup>4</sup> Note that all binary connectives of AT14 are disjunctions; indeed, <sup>A</sup> <sup>→</sup> <sup>B</sup> is the syntactic sugar for ¬A B.


**Table 2.** Falsification performance comparison between Breach and ForeSee on benchmarks. Timeout: 900 s. FR in (/30), time in secs.

difficult to be falsified. In such a case, there is no best sub-formula sequence Σ: therefore, the time spent by ForeSee in exploring different Σ does not provide any improvement.

Regarding the time execution, there is no clear trend among the different results: sometimes ForeSee is faster, other times Breach is. However, even in the cases in which ForeSee is slower, it is still below the timeout by which it manages to find a falsifying input (so, leading to better falsification rates).

#### **RQ2.** *Does the proposed approach solve the scale problem effectively?*

The benchmarks reported in Table 1 and experimented in RQ1, predicate over signals having different scales and so they suffer from the scale problem. RQ1 showed that ForeSee is very efficient in falsifying them. In this RQ, we want to make a more systematic study of the effects of the scale problem; indeed, the scale problem could manifest itself in *different ways*, depending on the difference of the order of magnitudes of the different signals (e.g., speed [km/h] vs. rpm, or speed [km/h] vs. rph). To assess this, we take six specifications from Table 1 and we artificially modify their outputs: namely, we multiply by 10<sup>k</sup> (with different k values depending on the specification) the *speed* of AT1, AT3, AT4,


**Table 3.** Falsification performance under different scales. Each rescaled signal is rescaled by 10<sup>k</sup>.

and AT9; the *rpm* of AT15; and the *EngineSpeed* of AFC3. For each artificial rescaling, both the Simulink model and the specification have been changed.<sup>5</sup> We run ForeSee and Breach on these rescaled benchmarks. Table 3 reports the experimental results for each k. The table also reports the minimum, maximum, and mean results for FR and execution time. We observe that the performance of Breach, in terms of FR, is very sensitive to the scale problem. Indeed, for all the specifications, FR decreases with increasing or decreasing k; notable examples are AT3 and AT4 in which Breach can (almost) always falsify with the minimum k, but never falsifies with the maximum two k. This is the demonstration of the effects of the scale problem on falsification approaches that only rely on quantitative robust semantics where the robustness values of different signals are compared. By looking at the results of ForeSee, instead, we observe that it is much more robust and its FR performance is independent of the applied rescaling. This clearly shows that our falsification approach guided by QB-Robustness is successful in avoiding the scale problem.

<sup>5</sup> Note that k = 0 corresponds to the experimental result in Table 2, and we report it again for reference.


**Table 4.** Falsification performance under different MCTS hyperparameters.

These results also allow us to show that the naive approach based on normalization for solving the scale problem does not work, as also reported in [41]. Indeed, one may think that a solution for tackling the scale problem could be to rescale the signals in a way to make them have the same order of magnitude. This is not a good approach. Let us consider the results in Table 3c for AT4 (-[0,30] (*speed* <sup>&</sup>lt; <sup>135</sup> <sup>∧</sup> *rpm* <sup>&</sup>lt; 4780)). In this case, *speed* is multiplied by 10<sup>k</sup>. We may think that the best falsification result should occur when *speed* is multiplied by 10<sup>2</sup>, because this would make the two signals both in the order of thousands. However, this rescaling is the one giving the worst result. The best result is actually given by the rescaling making *speed* even smaller (i.e., <sup>k</sup> <sup>=</sup> <sup>−</sup><sup>2</sup> and <sup>k</sup> <sup>=</sup> <sup>−</sup>1). This means that the correct way for handling the scale problem cannot be identified in advance, but we need an approach as ours that *learns* during falsification the best strategy.

#### **RQ3.** *How do the hyperparameters of MCTS influence the performance of the proposed approach?*

Our proposed approach is an instantiation of the Monte Carlo Tree Search (MCTS) method [8,28] that can be configured with some hyperparameters, namely the scalar c used by UCB1 (Line 12 in Algorithm 2), and the playout budget B<sup>P</sup> (Line 21 in Algorithm 2), both used for balancing between exploration and exploitation. Therefore, the performance of MCTS could be affected by the values used for these hyperparameters. In this RQ, we try to assess this. We selected three benchmarks specifications (AT17, AT19, and AT21) and varied one hyperparameter while keeping the other fixed. Namely, we experimented with <sup>c</sup> ∈ {0, <sup>0</sup>.02, <sup>0</sup>.2, <sup>0</sup>.5, <sup>1</sup>} and budget <sup>B</sup><sup>P</sup> = 10 (see Table 4a), and with <sup>B</sup><sup>P</sup> ∈ {2, <sup>5</sup>, <sup>10</sup>, <sup>15</sup>, <sup>20</sup>} and budget <sup>c</sup> = 0.2 (see Table 4b). Looking at the results of Table 4a for AT17 and AT21, there seems to be some influence by the scalar c. In AT17, the worst result in terms of FR is obtained when c is 0, meaning that MCTS only focuses on *exploitation*. AT17 is a specification that suffers from the scale problem, as shown by the very bad performance of Breach in Table 2; for such a specification, we need to perform some *exploration* to find the best Σ: this explain the low performance of MCTS with c = 0. On the other hand, the worst FR performance of AT21 is given by the highest value c = 1 that requires MCTS to spend a lot of time in exploration. Since AT21 is not an extremely difficult specification (indeed Breach has FR of 10 in Table 2), such very conservative approach does not pay off, while more greedy approaches (i.e., with lower c) have better performance.

Looking at the results of Table 4b related to B<sup>P</sup> , it seems that there is no too much influence. The only difference is given in AT17 with B<sup>P</sup> = 2 where the FR is slightly lower than the other cases. This means that, provided that a sufficiently large value for B<sup>P</sup> is given, ForeSee is not too sensitive to it.

#### **6 Related Work**

Quality assurance of CPS has been actively studied, due to its great significance. Different approaches, including but not limited to model checking, theorem proving, rigorous numerics, and nonstandard analysis [9,16,20,22,23,31,33], have been proposed to solve the problem. However, due to the scalability issue and existence of black-box components, those approaches are not widely applied in the real-world systems.

The optimization-based falsification approach inherits the search-based testing methodology, and is much more scalable than pure verification-based approaches. The key issue of search-based testing is the *exploration-exploitation trade-off*. This issue has been discussed for the verification of quantitative properties (e.g., [34]). In the falsification community, there have also been a lot of works focusing on that, and these works tackle the problem from different perspectives. *Metaheuristics* refers to high-level heuristic strategies that utilize heuristics to improve the search efficiency. Several metaheuristic strategies have been applied in falsification, such as *Simulated Annealing* [1], *tabu search* [10], and so on. *Coverage-guided falsification* [2,10,15,29] aims to guide the search using some coverage metrics, so that the search space is sufficiently explored. Recently, machine learning techniques have also been applied to falsification to enhance the search ability. For instance, *Bayesian optimization* [3,11,36] utilizes an *acquisition function* to balance exploration and exploitation; *Reinforcement learning* [27,37] naturally emphasizes on exploration.

The scale problem is a recognized issue [12,21,40] that is known to severely affect the performance of falsification. In [40], we proposed a multi-armed bandit approach to solve the problem in a specific setting, that is, safety properties with Boolean connectives: -<sup>I</sup> (ϕ<sup>1</sup> <sup>∧</sup>ϕ2) and -<sup>I</sup> (ϕ<sup>1</sup> <sup>∨</sup>ϕ2). The approach is not applicable to formulas having more nested sub-formulas, or even connectives having more operands; therefore many of the benchmarks we used in Subsect.5.2 fall out of the scope of [40]. The techniques introduced in [12,21] rely on explicit declaration of *input vacuity* and *output robustness*. Compared to their approaches, our method does not need that, but we learn the significance of each signal through tree exploration and reward computation.

MCTS, as an effective search framework, has been applied in testing hybrid systems. In [30], the authors applied an adaption of MCTS in testing, namely, adaptive press testing, to detect the potential dangerous cases of airborne collision. A recent study of MCTS on hybrid system falsification is [39]. There, the authors discretized the search space to construct the search tree, and then applied MCTS to explore different sub-spaces. Compared to their approach, our work aims to tackle the scale problem and so we exploit the structure of specification formulas to construct the tree search framework.

#### **7 Conclusion and Future Work**

Optimization-based falsification is a widely used approach for quality assurance of CPS, that tries to find an input violating a Signal Temporal Logic (STL) specification. It does this by exploiting the quantitative robust semantics of the specification, trying to minimize its robustness. The performance of falsification is affected by the *scale problem* in the presence of the comparison of robustness values of sub-formulas predicating over signals having different scales. In this paper, we propose QB-Robustness, a new STL semantics that does not suffer from the scale problem, because it avoids such comparison. The computation of QB-Robustness requires to specify a sub-formula sequence telling for which sub-formulas the quantitative robustness must be computed. We then propose a Monte Carlo Tree Search (MCTS)-based falsification approach that synthesizes a sub-formula sequence for QB-Robustness, and uses this for guiding numerical optimization. Experimental results show that the proposed approach achieves better falsification results than a state-of-the-art falsification tool that uses standard quantitative robust semantics.

In the analysis of RQ1, we observed that, when the specifications have a particular structure, our approach has no advantage and, actually, it could decrease the performance by trying to find a best sub-formula sequence that does not exist for the current initial sampling. As future work, we plan to devise some heuristics that could handle these cases: for example, we could perform a better initial sampling (see Subsect. 2.1) that could provide a better initial guidance.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Fast Zone-Based Algorithms for Reachability in Pushdown Timed Automata**

S. Akshay1(B) , Paul Gastin<sup>2</sup> , and Karthik R. Prakash<sup>1</sup>

<sup>1</sup> Department of CSE, Indian Institute of Technology Bombay, Mumbai, India {akshayss,karthikrprakash}@cse.iitb.ac.in <sup>2</sup> Universit´e Paris-Saclay, ENS Paris-Saclay, CNRS, LMF, 91190 Gif-sur-Yvette, France paul.gastin@lsv.fr

**Abstract.** Given the versatility of timed automata a huge body of work has evolved that considers extensions of timed automata. One extension that has received a lot of interest is timed automata with a, possibly unbounded, stack, also called pushdown timed automata (PDTA). While different algorithms have been given for reachability in different variants of this model, most of these results are purely theoretical and do not give rise to efficient implementations. One main reason for this is that none of these algorithms (and the implementations that exist) use the so-called zone-based abstraction, but rely either on the region-abstraction or other approaches, which are significantly harder to implement.

In this paper, we show that a naive extension, using simulations, of the zone based reachability algorithm for the control state reachability problem of timed automata is not sound in the presence of a stack. To understand this better we give an inductive rule based view of the zone reachability algorithm for timed automata. This alternate view allows us to analyze and adapt the rules to also work for pushdown timed automata. We obtain the first zone-based algorithm for PDTA which is terminating, sound and complete. We implement our algorithm in the tool TChecker and perform experiments to show its efficacy, thus leading the way for more practical approaches to the verification of timed pushdown systems.

**Keywords:** Timed automata · Zone-based abstractions · Pushdown automata · Simulations · Reachability

#### **1 Introduction**

Timed automata [7] are a popular formalism for capturing real-time systems, and of use for instance, in model checking of cyber-physical systems. They extend

This work was partly supported by ReLaX CNRS IRL 2000, DST/CEFIPRA/INRIA project EQuaVE and SERB Matrices grant MTR/2018/00074.

finite automata with real variables called clocks whose values increase over time; transitions are guarded by constraints over these variables. The main problem of interest is the reachability problem, which asks whether a given state can be reached while satisfying the constraints imposed by the guards. This problem is known to be PSPACE-complete (already shown in [7]). The PSPACE algorithm, uses the so-called region-automaton construction, which essentially abstracts the timed automaton into an exponentially larger finite automaton of regions (collections of clock valuations), which is sound and complete for reachability.

Despite this complexity-theoretic hardness, the model of timed automata has proved to be extremely influential and versatile, resulting in an enormous body of work on its theory, variants and extensions over the past 25 years. Almost since its inception, researchers also began to develop tools to extend from theoretical algorithms to solve practical problems. Such tools range from the classical and richly featured tool UPPAAL [9,23] to the more recent open-source tool TChecker [19], which have been used on industry strength benchmarks and perform rather well on many of them. These tools use a different algorithm for reachability, where reachable sets of valuations are represented as zones and explored in a graph. While a naive exploration of zones does not terminate, the algorithms used identify different strategies [8,18,21], e.g., subsumption or simulations, extrapolations, for pruning the zone-based exploration graphs, while preserving soundness and completeness of reachability. While this does not change the worst case complexity, in practice, the zone exploration results in much better *practical* performance as it allows on-the-fly computation of reachable zones. One could even argue that the wider adoption of timed automata paradigm in the verification community has been a result of scalable implementations and tools built on this zone-based approach.

In light of this, zone-based algorithms are often looked for to improve practical performance of extensions of timed automata as well. For instance, for timed automata with diagonal constraints, classical zone-based approaches were shown to be unsound [11,12], but recently, an approach has been developed which adapts the existing construction and obtains fast zone-based algorithms [17]. In the present paper, we are interesting in adding a different feature to timed automata, namely an unbounded lifo-stack. This results in a powerful model of *pushdown timed automata (PDTA for short)*, in which the source of "infinity" is both from real-time and the unbounded stack. Unsurprisingly, this model and its variants have been widely studied over the last 20 years with several old and recent results on decidability of reachability, related problems and their complexity, including [1–5,10,13–16]. A wide variety of techniques have been employed to solve these problems, from region-based abstractions, to using atoms and systems of constraints, to encoding into different logics etc. However, except for [4,5], to the best of our knowledge, none of the others carry an implementation. In [5], the implementation uses a tree-automaton implicitly based on regions and the focus in [4] is towards multi-pushdown systems. A common factor of all these works is that none of them consider zone-based abstractions.

In this paper, we ask whether zone-based abstractions can be used to decide efficiently reachability questions in PDTA. We focus on the problem of wellnested control-state reachability of PDTA, i.e., given a PDTA, an initial and a target state, does there exist a run of the PDTA that starts at the initial state with empty stack and reaches the target state with an empty stack (in between, i.e., during the run, the stack can indeed be non-empty). As with timed automata, our goal here is towards its applicability to build powerful tools which could lead to wider adoption of the PDTA model and showcase its utility to model-checking timed recursive systems. As the first step, we examine the difficulties involved in mixing zones with stacks and point out that a naive adaptation of the zone-based algorithm would not be sound. Then we propose a new algorithm that modifies the zone-based algorithm to work for pushdown timed automata. This is done in three steps.


We implement our approach to build the first zone-based tool that efficiently solves well-nested control state reachability for PDTA. Our tool is built on top of existing infrastructure of TChecker [19], an open source tool and benefits from many existing optimizations. We perform experiments to show the practical performance of multiple variants of our algorithm and show how our most optimized version is vastly better in performance than other variants and of course the earlier region-based approach on a suite of example benchmarks.

We note that our PDTA model differs slightly from the model considered in [1,3], as there is no age on stack and time spent on stack cannot be compared with clocks. Hence our model is closer to [10,16]. However, in [13], it was shown that these two models are equivalent, more specifically, the stack can be untimed without loss of expressivity (albeit with an exponential blowup). Thus our approach can be applied to the other model as well by just untiming the stack. There are other more powerful extensions [14,15] studied especially in the context of binary reachability, where only theoretical results are known. We also remark that the idea of combining the subsumption relation between zones with an equivalence relation also occurs while tackling liveness, or Buchi acceptance, in timed automata. This has been studied in depth [20,22,24], where the naive zone-based algorithm does not work, forcing the authors to strengthen the simulation relation in different ways. Though these problems are quite different, there are surprising similarities in the issues faced, as explained in Sect. 3.

The structure of the paper is as follows: we start with preliminaries and move on to the difficulty in using zones and simulation relations in solving reachability in PDTA. Then, we introduce in Sect. 4 our inductive rules for timed automata and PDTA and show their correctness. In Sect. 5, we present our algorithm and helpful data-structural advancements. We show the experimental performance in Sect. 6 and end with a brief conclusion. Proofs that are missing and more experimental results can be found in the long version of the paper available at [6].

#### **2 Preliminaries**

#### **2.1 Timed Automata**

Timed automata extend finite-state automata with a set X of (non-negative) real-valued variables called *clocks*. We let Φ(X) denote the set of constraints ϕ that can be formed using the grammar: ϕ :: = x <sup>∼</sup> c <sup>|</sup> x <sup>−</sup> y <sup>∼</sup> c <sup>|</sup> ϕ <sup>∧</sup> ϕ, where x, y <sup>∈</sup> X, c <sup>∈</sup> <sup>N</sup>, ∼ ∈ {≤, <sup>≥</sup>, <, >}, where each x <sup>∼</sup> c is called an atomic constraint. A clock valuation is a map <sup>v</sup> : <sup>X</sup> <sup>→</sup> <sup>R</sup>≥<sup>0</sup> and is said to satisfy <sup>ϕ</sup>, denoted v <sup>|</sup><sup>=</sup> ϕ, if ϕ evaluates to true when each clock x <sup>∈</sup> X is replaced with v(x). For δ <sup>∈</sup> <sup>R</sup>≥<sup>0</sup>, we write <sup>v</sup> <sup>+</sup> <sup>δ</sup> to denote the valuation defined as (v <sup>+</sup> δ)(x) = v(x) + δ for all clocks x. For R <sup>⊆</sup> X, we write [R]v to denote the valuation obtained by resetting clocks in R, i.e., ([R]v)(x) = 0 if x <sup>∈</sup> R, and ([R]v)(x) = <sup>v</sup>(x) otherwise. Finally, <sup>v</sup><sup>0</sup> is the valuation that sets all clocks to 0.

A timed automaton <sup>A</sup> is a tuple (Q, X, q<sup>0</sup>, Δ, F), where <sup>Q</sup> is a finite set of states, <sup>X</sup> is a finite set of clocks, <sup>q</sup><sup>0</sup> <sup>∈</sup> <sup>Q</sup> is an initial state, <sup>F</sup> <sup>⊆</sup> <sup>Q</sup> is the set of final states and Δ <sup>⊆</sup> Q <sup>×</sup> Φ(X) <sup>×</sup> <sup>2</sup><sup>X</sup> <sup>×</sup> <sup>Q</sup> is a set of transitions. A transition t <sup>∈</sup> Δ is of the form (q, g, R, q ), where q, q are states, g <sup>∈</sup> Φ(X) is the guard of the transition and R <sup>⊆</sup> X is the set of clocks that are reset at the transition. The semantics of a timed automaton A is given as a transition system *TS*(A) over configurations. A configuration is a pair (q, v) where q <sup>∈</sup> Q is a state and v is a valuation, with the initial configuration being (q<sup>0</sup>, v<sup>0</sup>). The transitions are of two types. First, for a configuration (q, v) and <sup>δ</sup> <sup>∈</sup> <sup>R</sup>≥<sup>0</sup>, (q, v) <sup>δ</sup> −→ (q, v <sup>+</sup> δ) is a delay transition. Second, for t = (q, g, R, q ) <sup>∈</sup> Δ, (q, v) <sup>t</sup> −→ (q , v ) is a discrete transition if v <sup>|</sup><sup>=</sup> g and v = [R](v). A run is an alternating sequence of delays and discrete transitions starting from the initial configuration, and is said to be accepting if the last state in the sequence is a final state.

#### **2.2 Reachability, Zones and Simulations**

The problem of control-state reachability asks whether a given timed automaton has an accepting run. This problem is known to be PSPACE-complete [7], originally shown via the so-called region abstraction. Note that, since *TS*(A) is infinite, some abstraction is needed to get an algorithm. In practice however, the abstraction used to solve reachability, e.g., in tools such as UPPAAL [23] or TChecker [19] is the *zone abstraction*. A zone Z is defined as a set of valuations defined by a conjunction of atomic clock constraints. Given a guard g and reset R, we define the following operations on zones: time elapse −→Z <sup>=</sup> {v <sup>+</sup> δ <sup>|</sup> v <sup>∈</sup> Z, δ <sup>∈</sup> <sup>R</sup>≥<sup>0</sup>}, guard intersection <sup>g</sup> <sup>∩</sup> <sup>Z</sup> <sup>=</sup> {<sup>v</sup> <sup>∈</sup> <sup>Z</sup> <sup>|</sup> <sup>v</sup> <sup>|</sup><sup>=</sup> <sup>g</sup>} and reset [R]Z <sup>=</sup> {[R]v <sup>|</sup> v <sup>∈</sup> Z}. The resulting sets are also zones. With this, we can define the zone graph *ZG*(A) as a transition system obtained as follows: the nodes are (state, zone) pairs and (q,Z) <sup>t</sup> −→ (q , Z ), if t = (q, g, R, q ) is a transition of A and <sup>Z</sup> <sup>=</sup> −−−−−−−→ [R](<sup>g</sup> <sup>∩</sup> <sup>Z</sup>). The initial node is (q0, Z<sup>0</sup> <sup>=</sup> −−→{v<sup>0</sup>}) and a path in the zone graph is said to be accepting if it ends at an accepting state. The zone graph is known to be sound and complete for reachability, but as the graph may still be infinite, this does not give an algorithm for solving reachability yet.

To obtain an algorithm, one resorts to different techniques such as extrapolation or simulation. Here we focus on *simulation relations* which will lead to finite abstractions. Given a timed automaton A, a binary relation on configurations is called a *simulation* if whenever (q, v) (q , v ), we have q <sup>=</sup> q and


We often simply write v q <sup>v</sup> instead of (q, v) (q, v ). We can now lift this to sets Z, Z of valuations as <sup>Z</sup> q <sup>Z</sup> if for all <sup>v</sup> <sup>∈</sup> <sup>Z</sup> there exists <sup>v</sup> <sup>∈</sup> <sup>Z</sup> such that v q v . We say that node (q,Z) is subsumbed by node (q,Z ) when Z q <sup>Z</sup> . As a consequence we obtain the following lemma.

**Lemma 1.** *If* (q,Z) <sup>t</sup> −→ (q<sup>1</sup>, Z<sup>1</sup>) *in ZG*(A) *and* Z q <sup>Z</sup> *, then* (q,Z ) <sup>t</sup> −→ (q<sup>1</sup>, Z 1) *and* <sup>Z</sup><sup>1</sup> <sup>q</sup><sup>1</sup> <sup>Z</sup> 1*.*

*Proof.* Indeed, let <sup>v</sup><sup>1</sup> <sup>∈</sup> <sup>Z</sup><sup>1</sup> <sup>=</sup> −−−−−−−→ [R](<sup>g</sup> <sup>∩</sup> <sup>Z</sup>). We find <sup>v</sup> <sup>∈</sup> <sup>Z</sup> and <sup>δ</sup> <sup>≥</sup> 0 such that <sup>v</sup> <sup>|</sup><sup>=</sup> <sup>g</sup> and <sup>v</sup><sup>1</sup> = [R]v+δ. Since <sup>Z</sup> q <sup>Z</sup> , we find v <sup>∈</sup> Z with v q v . We deduce that v <sup>|</sup><sup>=</sup> <sup>g</sup> and [R]<sup>v</sup> <sup>q</sup><sup>1</sup> [R]v , which implies <sup>v</sup><sup>1</sup> <sup>q</sup><sup>1</sup> <sup>v</sup> <sup>1</sup> with <sup>v</sup> <sup>1</sup> = [R]v <sup>+</sup> <sup>δ</sup> <sup>∈</sup> Z <sup>1</sup> <sup>=</sup> −−−−−−−→ [R](<sup>g</sup> <sup>∩</sup> <sup>Z</sup> ). 

A simulation is said to be *finite* if for every sequence of nodes (q<sup>1</sup>, Z<sup>1</sup>), (q<sup>2</sup>, Z<sup>2</sup>),... there exist two nodes (qi, Zi) and (qj , Zj ) with i<j such that <sup>q</sup>i <sup>=</sup> <sup>q</sup>j and <sup>Z</sup>j <sup>q</sup>*<sup>i</sup>* <sup>Z</sup>i. The importance of the finiteness is that it allows us to stop exploration of zones along a branch of the zone graph: when a node (qj , Zj ) is reached which is subsumed by an earlier node (qi, Zi), we may cut the exploration since all control states reachable from the latter are already reachable from the former. For a timed automaton A, we call this pruned graph as ZG(A). Thus, if the simulation relation is finite, then ZG(A) is finite, sound and complete for control state reachability. We formalize this algorithm in Sect. 4, using inductive rules.

Various finite simulations have been shown to exist in the literature, including the famous LU-abstractions [8], and more recent G-abstractions based on sets of guards [17]. Hence this theory indeed has resulted in better implementations and is used in standard tools in this domain.

We will see that using simulation in the context of pushdown timed automata is not always sound, in some cases we need a stronger condition to stop the exploration. Towards this, we consider the equivalence relation on nodes induced by the simulation relation: <sup>Z</sup> <sup>∼</sup>q <sup>Z</sup> if <sup>Z</sup> q <sup>Z</sup> and <sup>Z</sup> q <sup>Z</sup>. We say that the simulation is strongly finite if the induced equivalence relation ∼ has finite index. Notice that strongly finite implies finite but the converse does not necessarily hold. Fortunately, the usual simulations for timed automata, in particular the LU-simulation and the G-simulation, are strongly finite.

#### **2.3 Pushdown Timed Automata (PDTA)**

A Pushdown Timed Automaton <sup>A</sup> is a tuple (Q, X, q0, Γ, Δ, F), where <sup>Q</sup> is a finite set of states, <sup>X</sup> is a finite set of clocks, <sup>q</sup><sup>0</sup> <sup>∈</sup> <sup>Q</sup> is an initial state, <sup>Γ</sup> is the stack alphabet, F <sup>⊆</sup> Q is the set of final states and Δ is a set of transitions. A transition t <sup>∈</sup> Δ is of the form (q, g, op, R, q ), where q, q are states, g <sup>∈</sup> Φ(X) is the guard of the transition and R <sup>⊆</sup> X is the set of clocks that are reset at the transition, op is one of three stack operations: nop or pusha or popa for some a <sup>∈</sup> Γ.

The semantics of a PDTA A is given as a transition system *TS*(A) over configurations. A configuration here is a tuple (q, v, χ) where q <sup>∈</sup> Q is a state, v is a valuation, χ <sup>∈</sup> Γ<sup>∗</sup> is the stack content, with the initial configuration being (q<sup>0</sup>, v<sup>0</sup>, ε). The transitions are of two types. First, for a configuration (q, v, χ) and <sup>δ</sup> <sup>∈</sup> <sup>R</sup>≥<sup>0</sup>, (q, v, χ) <sup>δ</sup> −→ (q, v <sup>+</sup> δ, χ) is a delay transition. Second, for t <sup>=</sup> (q, g, op, R, q ) <sup>∈</sup> Δ, (q, v, χ) <sup>t</sup> −→ (q , v , χ ) is a discrete transition if v <sup>|</sup><sup>=</sup> g, v = [R](v) and

– if op = nop, then χ <sup>=</sup> <sup>χ</sup>, – if op = pusha then <sup>χ</sup> <sup>=</sup> <sup>χ</sup> · <sup>a</sup>,

– if op = popa, then <sup>χ</sup> <sup>=</sup> <sup>χ</sup> · <sup>a</sup>.

A run is an alternating sequence of delays and discrete actions starting from the initial configuration. It is accepting if the last state in the sequence is final.

Our main focus is the *well-nested control state reachability* problem for PDTA, which asks whether a configuration (q, v, ε) with q <sup>∈</sup> F is reachable, where the stack is empty. Later, in Sect. 7, we remark how our solution can be extended to solve general control state reachability, i.e., asking whether a configuration (q, v, χ) with q <sup>∈</sup> F is reachable, possibly with a nonempty stack χ.

**Fig. 1.** A simple PDTA with 2 clocks {x, y}. Note that if we ignore the push/pop actions we get a TA, say A.

**Fig. 2.** Zone graph with simulation edges for finiteness. Again ignoring push/pop actions gives us a zone graph for the TA. <sup>Z</sup><sup>0</sup> is the initial zone.

#### **3 Zones in PDTA and the Problem with Simulations**

As mentioned earlier, zones are collections of clock valuations defined by conjunctions of timing constraints, and exploring zones reached by a timed automaton gives a sound and complete abstraction for state reachability. To make sure that the exploration is finite we need to prune the graph and one way this is done by simulation, i.e., not exploring paths from some nodes if they are "subsumed" by earlier nodes visited in the graph. Consider Fig. 1, in which we ignore the pusha and popa or we can think of them as internal actions. Then the usual zonegraph construction with simulation would give the graph depicted in Fig. 2. In this section, just for illustration we instantiate the simulation relation to be the well-known LU-simulation (we do not give the definition here as it is not relevant to what comes later, instead we refer to earlier work [8]). Using this, we obtain that the rightmost node is subsumed by the previous one, and hence the dotted simulation edge. If we did not do this we immediately observe that we get an infinite graph with increasing sets of zones.

Now, our first question is whether this zone exploration with simulation can be lifted to PDTA. In this example, if we were to add back the push/pop edges, we get exactly the same Zone graph with annotations, and further, the final state is indeed reachable. Hence, for this particular example we do obtain a finite, sound and complete graph exploration. However, in general it turns out that the procedure is not sound.

Consider the example in Fig. 3. In this example, again considering it as a TA (ignoring the push/pops), we would get the zone graph below, which would be finite, sound and complete for reachability in that TA. But if we consider it as a PDTA, now doing the same does not preserve soundness. In other words, in the PDTA, <sup>q</sup><sup>3</sup> is no longer reachable. However, in the zone graph we would conclude that it is reachable due to the simulation edge. If, to fix this, we remove the dotted simulation edge, then we will lose finiteness.

Thus, it seems that we have a difficult situation where zones with the simulation relation, needed for termination, do not preserve soundness. This situation resembles the situation studied in [20,22,24], where the authors study liveness or Buchi-acceptance conditions in timed automata. Again in that situation, the

**Fig. 3.** A PDTA and its zone graph with simulation. With the simulation (dotted) edges, <sup>q</sup><sup>3</sup> is reachable in the zone graph, but its not reachable in the PDTA.

naive algorithm with zone simulation does not work and the authors are forced to strengthen the simulation relation in different ways.

Surprisingly, it turns out, that even in our very different problem setting of reachability in PDTA, a similar solution works. That is, we replace simulation by equivalence (defined in the previous section) as the pruning criterion. However, there are two issues (i) it is not easy to prove its correctness and (ii) this is far from efficient as shown in the experimental section. Our goal to use zones in the first place was efficiency and hence we would like to prune the zone graph as much as possible, i.e., we would like to use simulation edges as much as possible. In the next two sections, we describe our fix. We first show a different view of the exploration algorithm as a fixed point rule based approach. This allows us to then describe our fix in the same language, which is much easier to understand conceptually. Also as a corollary we will be able to show that using equivalence everywhere also gives a correct algorithm. After proving the correctness of our rule-based algorithm, we then tackle the challenges in implementing it.

#### **4 Viewing Reachability Algorithms Using Rewrite Rules**

In this section, our goal is to compute a set S of nodes of the zone graph of a PDTA, as a least fixed point of a small set of inductive rules, such that a control state q occurs in S, i.e., (q,Z) <sup>∈</sup> S for some Z iff q is reachable in the PDTA from its initial state. To understand the rules and their correctness it is easier to first visualize this on plain timed automata without any push-pop edges.

#### **4.1 Rewrite Rules for Timed Automata.**

Given a TA <sup>A</sup> = (Q, X, q<sup>0</sup>, Δ, F), the set <sup>S</sup> containing all reachable nodes of the zone graph, can be obtained as the least fixed point of the following inductive rules, with a natural deduction style of presentation.

$$\overline{S := \{ (q\_0, Z\_0) \}}^{\text{start}}$$

Fast Zone-Based Algorithms for Reachability in Pushdown Timed Automata 627

$$\frac{(q,Z)\in S}{} \quad q \xrightarrow{g,R} q' \quad \quad Z' = \overrightarrow{R(g\cap Z)} \neq \emptyset \quad \text{Trans}$$

Let S<sup>∗</sup> denote the set at the fixed point by starting with the start rule and repeatedly applying the trans rule. It is easy to see that this computes the set of all reachable nodes of the zone graph: the start rule starts with the initial node and each application of trans rule takes a reachable node and applies a transition of the automaton and includes the resulting node reached. However, this set S<sup>∗</sup> is a priori infinite since number of zones is infinite.

To make it finite we add a condition under which we will apply the transition rule based on a finite simulation relation (let us denote it ) for A.

$$\frac{(q,Z)\in S\qquad q\xrightarrow{g,R}q'\qquad Z'=\overrightarrow{R(g\cap Z)}\neq\emptyset}{S\to\{(q',Z')\},\text{ unless }\exists(q',Z'')\in S,\;Z'\preceq\_{q'}Z''}\text{ trans-}\leq$$

Thus to obtain an algorithm, we would explore all nodes in the Zone graph using a search algorithm (say DFS/BFS) and we would add a node only if it is not subsumed by an already visited node, according to the simulation relation. We explained in Sect. 2.2 that doing this preserves soundness and completeness and gives a finite exploration.

**Lemma 2.** *Let* S<sup>∗</sup> *denote any set obtained from the start rule and by repeatedly applying Trans till a fixed point is reached. Note that depending on the order of applications we may have different sets. Then we have:*


We do not give the proof here as (i) it is only a reformulation of known results and (ii) it will be subsumed by the much stronger theorem we prove next.

#### **4.2 Rewrite Rules for PDTA**

Let <sup>A</sup> = (Q, X, q<sup>0</sup>, Γ, Δ, F) be a PDTA, we will need not just a set but a tuple of sets. More precisely, we maintain a set of nodes S called *root* nodes. For each root node (q,Z) <sup>∈</sup> <sup>S</sup>, we also maintain a set of nodes, denoted <sup>S</sup>(q,Z). The intuition is that root nodes are those that can be reached after pushing a symbol to the stack, whereas <sup>S</sup>(q,Z) will be the set of nodes that can be reached from (q,Z) with a well-nested run, i.e., starting with an empty stack and ending in an empty stack. This is to avoid storing the stack contents in our algorithm, which would be another source of infinity. Again, we use simulations to make the computation finite. So we fix a strongly finite simulation relation for A.

Our inductive rules for the control state reachability of pushdown timed automata are given in Table 1. Note that the internal rule is just the same as for timed automata above. The start rule not only starts the set of nodes computation but also the set of roots computation as described above. So the only **Table 1.** Inductive rules for control state reachability of PDTA

$$\widetilde{\mathfrak{S}} := \{ (q\_0, Z\_0) \}, \\ S\_{\{q\_0, Z\_0\}} := \{ (q\_0, Z\_0) \} $$

$$\begin{array}{llll} (q,Z)\in\mathfrak{S} & \quad (q',Z')\in S\_{(q,Z)} & \quad q' \xrightarrow{g,\text{no},R} q'' & \quad Z'' = \overline{R(g\cap Z')} \neq \emptyset \\ \hline S\_{(q,Z)} := S\_{(q,Z)} \cup \{(q'',Z'')\}, \text{ unless } \exists (q'',Z'') \in S\_{(q,Z)}, \; Z'' \leq\_{q''} Z'' \\\\ \cline{2-4} (q,Z)\in\mathfrak{S} & \quad (q',Z')\in S\_{(q,Z)} & \quad q' \xrightarrow{g,\text{pusha}\_{a},R} q'' & \quad Z'' = \overline{R(g\cap Z')} \neq \emptyset \\ \cline{2-4} \mathfrak{S} & \coloneqq\mathfrak{S} \cup \{(q',Z'')\}, \; S\_{(q',Z'')} = \{(q',Z'')\}, \text{ unless } \exists (q'',Z'') \in \mathfrak{S}, \; Z'' \sim\_{q''} Z'' \\\\ & \quad \square & \quad \square & \quad \square & \quad \square \end{array} \text{Push}$$

(*q,Z*) <sup>∈</sup> <sup>S</sup> (*q*--*, Z*1) <sup>∈</sup> <sup>S</sup> (*q*- *, Z*- ) <sup>∈</sup> *<sup>S</sup>*(*q,Z*) (*q*- 1*, Z*- <sup>1</sup>) <sup>∈</sup> *<sup>S</sup>*(*q*--*,Z*1) *q g,*push*a,R* −−−−−−→ *<sup>q</sup>*-- *q*- 1 *<sup>g</sup>*1*,*pop*a,R*<sup>1</sup> −−−−−−−→ *<sup>q</sup>*<sup>2</sup> *Z*-- <sup>=</sup> −−−−−−→ *<sup>R</sup>*(*<sup>g</sup>* <sup>∩</sup> *<sup>Z</sup>*- ) ∼*<sup>q</sup>*-- *Z*<sup>1</sup> *<sup>Z</sup>*<sup>2</sup> <sup>=</sup> −−−−−−−−→ *<sup>R</sup>*1(*g*<sup>1</sup> <sup>∩</sup> *<sup>Z</sup>*- <sup>1</sup>) = ∅ Pop *<sup>S</sup>*(*q,Z*) := *<sup>S</sup>*(*q,Z*) ∪ {(*q*2*, Z*2)}, unless <sup>∃</sup>(*q*2*, Z*- <sup>2</sup>) <sup>∈</sup> *<sup>S</sup>*(*q,Z*), *<sup>Z</sup>*<sup>2</sup> *<sup>q</sup>*<sup>2</sup> *<sup>Z</sup>*- 2

interesting rules are the Push and Pop rules. The push rule says that when a push is encountered, then we must start exploring from a new root (i.e., context). So the only complicated rule is the Pop rule. Here the intuition is that if we see a push at a node and from a root equivalent to the root created from it, (i.e., its context) we see a matching pop reaching a new node, then this push-pop context is complete, and we can add this new node to the set of reachable nodes. This is precisely the point where we *need* equivalence rather than simulation and this will be made clear in the proof of the theorem below.

**Theorem 1.** *Let* <sup>S</sup><sup>∗</sup> *and* (S(q,Z))(q,Z)∈S<sup>∗</sup> *denote any tuple of sets obtained from the start rule and by repeatedly applying the rules in Table 1 till a fixed point is reached*<sup>1</sup>*. Note that we always have* (q<sup>0</sup>, Z<sup>0</sup>) <sup>∈</sup> <sup>S</sup>∗*. The following statements hold:*


*Proof.* **1.** Note that only the Push rule creates new root nodes and the red condition states that a new root node is added only if there isn't already an equivalent node in S∗. Since the simulation relation is strongly finite, the set of roots <sup>S</sup><sup>∗</sup> must be finite. Also, before adding a node to some <sup>S</sup>(q,Z) with the internal rule or the pop rule, we check that the node is not subsumed by an existing one. Since the simulation relation is finite, this ensures that each set <sup>S</sup>(q,Z) is finite.

<sup>1</sup> As before, there could be several such sets depending on the order in which the rules are applied.

$$\begin{array}{ccccccccc}(q,v,\varepsilon) & \xrightarrow{\scriptstyle{\bf{z}}} & (q\_{i},v\_{i},\varepsilon) & \xrightarrow{\scriptstyle{\bf{z}}} & (q\_{i+1},v\_{i+1},a) & \xrightarrow{\scriptstyle{\bf{z}}} & (q\_{n-1},v\_{n-1},a) & \xrightarrow{\scriptstyle{\bf{z}}\_{1}} & (q\_{n},v\_{n},\varepsilon) \\ v\preceq\_{q}Z & v\_{i}\preceq\_{q\_{i}}Z\_{i} & v\_{i+1}\preceq\_{q\_{i+1}}Z\_{i+1} & & v\_{n-1}\preceq\_{q\_{n-1}}Z\_{n-1} & & v\_{n}\preceq\_{q\_{n}}Z\_{n} \\ (q,Z)\in\mathfrak{S} & (q\_{i},Z\_{i})\in S\_{(q\_{i},Z)} & Z\_{i+1}=\overline{R(g\cap Z\_{i})} & (q\_{n-1},Z\_{n-1})\in S\_{(q\_{i+1},Z\_{i+1}')} & & Z\_{n}=\overline{R\_{1}(g\_{1}\cap Z\_{n-1})} \\ & & & Z\_{i+1}\wedge\_{q\_{i+1}}Z\_{i+1}' & & & Z\_{n}\preceq\_{q\_{n}}Z\_{n}' \\ & & & & (q\_{i+1},Z\_{i+1}')\in\mathfrak{S} & & (q\_{n},Z\_{n}')\in S\_{(q\_{i},Z)} \\ \end{array}$$

**2.** Let (q,Z) <sup>∈</sup> <sup>S</sup><sup>∗</sup> and assume that (q , v , ε) is reachable from some (q, v, ε) with v q <sup>Z</sup>, i.e., there exists a run (q, v, ε)=(q1, v1, χ<sup>1</sup>) →···→ (qn, vn, χn) = (q , v , ε). We will then show that <sup>v</sup>n <sup>q</sup>*<sup>n</sup>* <sup>Z</sup> for some (qn, Z ) <sup>∈</sup> <sup>S</sup>(q,Z). The proof is by induction on n. Base case: For n = 1 we have q <sup>=</sup> <sup>q</sup> and <sup>v</sup> <sup>=</sup> <sup>v</sup>. The result is obtained by taking <sup>Z</sup> <sup>=</sup> <sup>Z</sup>. Notice that (q,Z) <sup>∈</sup> <sup>S</sup>(q,Z) follows immediately from the start rule if <sup>q</sup> <sup>=</sup> <sup>q</sup><sup>0</sup>, <sup>Z</sup> <sup>=</sup> <sup>Z</sup><sup>0</sup> or from the push-create rule.

Let us then assume that the statement holds for runs of length at most n−1. Consider any run of the form (q, v, ε)=(q1, v1, χ<sup>1</sup>) → ··· → (qn, vn, χn <sup>=</sup> <sup>ε</sup>) with v q <sup>Z</sup>. Notice that its last transition (qn−1, vn−1, χn−<sup>1</sup>) <sup>→</sup> (qn, vn, χn) cannot be a push transition (in the PDTA) since <sup>χ</sup>n <sup>=</sup> <sup>ε</sup>. Hence, we have three subcases, depending on the last transition.


$$v\_i \preceq\_{q\_i} Z\_i \text{ for some } (q\_i, Z\_i) \in S\_{(q, Z)}\,. \tag{1}$$

From the push transition we have

$$\exists t = q\_i \xrightarrow{g, \text{push}\_a, R} q\_{i+1} \in \Delta \text{ with } v\_i \left| = g \text{ and } v\_{i+1} = [R] v\_i \right. \tag{2}$$

Let <sup>Z</sup>i+1 <sup>=</sup> −−−−−−→ <sup>R</sup>(<sup>g</sup> <sup>∩</sup> <sup>Z</sup>i). By definition of the simulation relation, we deduce from <sup>v</sup>i <sup>q</sup>*<sup>i</sup>* <sup>Z</sup><sup>i</sup> that <sup>v</sup>i+1 <sup>q</sup>*i*+1 <sup>Z</sup>i+1. We can apply the Push rule to obtain

$$(q\_{i+1}, Z'\_{i+1}) \in \mathfrak{S}^\* \text{ for some } Z'\_{i+1} \sim\_{q\_{i+1}} Z\_{i+1} \tag{3}$$

possibly with Z i+1 <sup>=</sup> <sup>Z</sup>i+1 as a special case. Further the segment of run (qi+1, vi+1, a) <sup>→</sup> ...(qn−<sup>1</sup>, vn−<sup>1</sup>, a) in the PDTA never pops the symbol a (by choice, since otherwise the push and pop would not be matching). Hence we will also have the same sequence of transitions forming a run (qi+1, vi+1, ε) <sup>→</sup> ...(qn−<sup>1</sup>, vn−<sup>1</sup>, ε). Using <sup>v</sup>i+1 q*i*+1 <sup>Z</sup>i+1 <sup>∼</sup>q*i*+1 <sup>Z</sup> i+1, we deduce that <sup>v</sup>i+1 <sup>q</sup>*i*+1 <sup>Z</sup> i+1. By induction hypothesis,

$$v\_{n-1} \preceq\_{q\_{n-1}} Z\_{n-1} \text{ for some } (q\_{n-1}, Z\_{n-1}) \in S\_{(q\_{i+1}, Z'\_{i+1})} \cdot \tag{4}$$

Finally, we have the pop transition

$$t\_1 = q\_{n-1} \xrightarrow{g\_1, \text{pop}\_a, R\_1} q\_n \in \Delta \text{ with } v\_{n-1} \left| = g\_1 \text{ and } v\_n = [R\_1] v\_{n-1} \right. \tag{5}$$

We let <sup>Z</sup>n <sup>=</sup> −−−−−−−−−−→ <sup>R</sup><sup>1</sup>(g<sup>1</sup> <sup>∩</sup> <sup>Z</sup>n−<sup>1</sup>). From <sup>v</sup>n−<sup>1</sup> <sup>q</sup>*n*−<sup>1</sup> <sup>Z</sup>n−<sup>1</sup> and the definition of the simulation relation we obtain <sup>v</sup>n <sup>q</sup>*<sup>n</sup>* <sup>Z</sup>n. Then, combining all the above equations (1–5), and applying the Pop-rule we obtain some (qn, Z n) <sup>∈</sup> <sup>S</sup>(q,Z) with <sup>Z</sup>n <sup>q</sup>*<sup>n</sup>* <sup>Z</sup> n (possibly <sup>Z</sup> n <sup>=</sup> <sup>Z</sup>n). Finally we get <sup>v</sup><sup>n</sup> <sup>q</sup>*<sup>n</sup>* <sup>Z</sup><sup>n</sup> <sup>q</sup>*<sup>n</sup>* <sup>Z</sup> n. This completes the proof.

**3.** We will show that the following property is invariant by rule applications:

$$\forall (q, Z) \in \mathfrak{S}, \,\forall (q', Z') \in S\_{(q, Z)}, \forall v' \in Z', \text{ there is a run }\tag{\text{Inv}}$$

$$(q, v, \varepsilon) \xrightarrow{\ast} (q', v'', \varepsilon) \text{ with } v \in Z \text{ and } v' \preceq\_{q'} v''$$

The invariant holds initially, i.e., after application of the start rule. Indeed, in this case we have <sup>S</sup> <sup>=</sup> {(q0, Z<sup>0</sup>)} and <sup>S</sup>(q0,Z0) <sup>=</sup> {(q0, Z<sup>0</sup>)}. Hence (q , Z )=(q,Z) = (q0, Z<sup>0</sup>) and for all <sup>v</sup> <sup>∈</sup> <sup>Z</sup><sup>0</sup> we can choose the empty run (q0, v, ε) <sup>0</sup> −→ (q0, v, ε).

We show below that (Inv) is preserved by application of an internal/push/pop rule. Therefore, the invariant still holds when reaching the fixed point, which proves the soundness. Let us write <sup>S</sup><sup>−</sup> and <sup>S</sup><sup>−</sup> (q,Z) for the sets before the application of the rule and <sup>S</sup> and <sup>S</sup>(q,Z) for the sets after the application of the rule.

**Internal Rule.** Let (q,Z) <sup>∈</sup> <sup>S</sup> <sup>=</sup> <sup>S</sup>−, (q , Z ) <sup>∈</sup> <sup>S</sup>(q,Z) and <sup>v</sup> <sup>∈</sup> <sup>Z</sup> . If (q , Z ) ∈ S− (q,Z) then we get the result since (Inv) holds before applying the internal rule. Otherwise, there is some (q<sup>1</sup>, Z<sup>1</sup>) <sup>∈</sup> S<sup>−</sup> (q,Z) and a transition <sup>t</sup> <sup>=</sup> <sup>q</sup><sup>1</sup> g,nop,R −−−−−→ q with Z <sup>=</sup> −−−−−−→ R(g <sup>∩</sup> Z<sup>1</sup>).

By definition, there exists <sup>v</sup><sup>1</sup> <sup>∈</sup> <sup>Z</sup><sup>1</sup> such that <sup>v</sup><sup>1</sup> <sup>|</sup><sup>=</sup> <sup>g</sup> and <sup>v</sup> = [R]v<sup>1</sup> <sup>+</sup> <sup>δ</sup> for some <sup>δ</sup> <sup>≥</sup> 0. Hence we have a run (q<sup>1</sup>, v<sup>1</sup>, ε) <sup>t</sup> −→ <sup>δ</sup> −→ (q , v , ε). Since the invariant holds before the internal rule, there is a run (q, v, ε) <sup>∗</sup> −→ (q<sup>1</sup>, v <sup>1</sup>, ε) with v <sup>∈</sup> Z and <sup>v</sup><sup>1</sup> <sup>q</sup><sup>1</sup> <sup>v</sup> <sup>1</sup>. Now since is a simulation we obtain that (q<sup>1</sup>, v <sup>1</sup>, ε) <sup>t</sup> −→ <sup>δ</sup> −→ (q , v, ε) with v q- v and we are done.

**Push Rule.** Let (q,Z) <sup>∈</sup> <sup>S</sup>, (q , Z ) <sup>∈</sup> <sup>S</sup>(q,Z) and <sup>v</sup> <sup>∈</sup> <sup>Z</sup> . If (q,Z) <sup>∈</sup> <sup>S</sup><sup>−</sup> then we get the result since (Inv) holds before applying the Push rule. Otherwise, we must have (q , Z )=(q,Z) and we can choose the empty run (q, v , ε) <sup>0</sup> −→ (q, v , ε).

$$\begin{array}{c} (q\_1', v\_4, a) \xrightarrow{t\_1} \xrightarrow{\delta'} (q\_2, v', \epsilon) \\ \mid \lambda \\ (q'', v\_3, a) \xrightarrow{\*} (q\_1', v\_4', a) \\ \mid \lambda \\ (q', v\_2, \varepsilon) \xrightarrow{t} \xrightarrow{\delta} (q''', v\_3', a) \\ \mid \lambda \\ (q, v, \varepsilon) \xrightarrow{\*} (q', v\_2', \varepsilon) \xrightarrow{t} \xrightarrow{\delta} (q'', v\_3'', a) \xrightarrow{\*} (q\_1', v\_4'', a) \xrightarrow{t\_1} \xrightarrow{\delta'} (q\_2, v'', \epsilon) \end{array}$$

#### **Fig. 5.** Construction for the soundness.

**Pop Rule.** Let (q,Z) <sup>∈</sup> <sup>S</sup> <sup>=</sup> <sup>S</sup>−, (q2, Z<sup>2</sup>) <sup>∈</sup> <sup>S</sup>(q,Z) and <sup>v</sup> <sup>∈</sup> <sup>Z</sup><sup>2</sup>. Again, if (q2, Z<sup>2</sup>) <sup>∈</sup> S<sup>−</sup> (q,Z) then we get the result since (Inv) holds before applying the Pop rule. Otherwise, by definition of the pop rule we have:


5. some pop transition <sup>t</sup><sup>1</sup> <sup>=</sup> <sup>q</sup> 1 <sup>g</sup>1,pop*a*,R<sup>1</sup> −−−−−−−→ q<sup>2</sup>,

with <sup>Z</sup><sup>2</sup> <sup>=</sup> −−−−−−−−→ <sup>R</sup><sup>1</sup>(g<sup>1</sup> <sup>∩</sup> <sup>Z</sup> <sup>1</sup>). The construction below is illustrated in Fig. 5.

Since <sup>v</sup> <sup>∈</sup> <sup>Z</sup><sup>2</sup>, we get some <sup>v</sup><sup>4</sup> <sup>∈</sup> <sup>Z</sup> <sup>1</sup> such that <sup>v</sup><sup>4</sup> <sup>|</sup><sup>=</sup> <sup>g</sup><sup>1</sup> and <sup>v</sup> = [R<sup>1</sup>]v<sup>4</sup> <sup>+</sup> <sup>δ</sup> for some δ <sup>≥</sup> 0. Hence we have a run (q <sup>1</sup>, v4, a) <sup>t</sup><sup>1</sup> −→ <sup>δ</sup>- −→ (q2, v , ε).

Now, applying the invariant to (q, Z<sup>1</sup>) <sup>∈</sup> <sup>S</sup>, (q <sup>1</sup>, Z <sup>1</sup>) <sup>∈</sup> <sup>S</sup>(q--,Z1) and <sup>v</sup><sup>4</sup> <sup>∈</sup> <sup>Z</sup> 1, we get a run (q, v3, ε) <sup>∗</sup> −→ (q 1, v <sup>4</sup>, ε) with <sup>v</sup><sup>3</sup> <sup>∈</sup> <sup>Z</sup><sup>1</sup> and <sup>v</sup><sup>4</sup> q- 1 v <sup>4</sup>. Hence, we also have a run (q, v3, a) <sup>∗</sup> −→ (q, v <sup>4</sup>, a).

Let v <sup>3</sup> <sup>∈</sup> <sup>Z</sup> <sup>=</sup> −−−−−−→ <sup>R</sup>(<sup>g</sup> <sup>∩</sup> <sup>Z</sup> ) <sup>∼</sup>q-- <sup>Z</sup><sup>1</sup> with <sup>v</sup><sup>3</sup> q-- v <sup>3</sup>. we get some <sup>v</sup><sup>2</sup> <sup>∈</sup> <sup>Z</sup> such that <sup>v</sup><sup>2</sup> <sup>|</sup><sup>=</sup> <sup>g</sup> and <sup>v</sup> <sup>3</sup> = [R]v<sup>2</sup> <sup>+</sup> <sup>δ</sup> for some <sup>δ</sup> <sup>≥</sup> 0. Hence we have a run (q , v<sup>2</sup>, ε) <sup>t</sup> −→ <sup>δ</sup> −→ (q, v <sup>3</sup>, a).

Finally, we apply the invariant to (q,Z) <sup>∈</sup> <sup>S</sup>, (q , Z ) <sup>∈</sup> <sup>S</sup>(q,Z) and <sup>v</sup><sup>2</sup> <sup>∈</sup> <sup>Z</sup> , we get a run (q, v, ε) <sup>∗</sup> −→ (q , v <sup>2</sup>, ε) with <sup>v</sup> <sup>∈</sup> <sup>Z</sup> and <sup>v</sup><sup>2</sup> q- v 2.

By repeatedly applying the property of simulation , we may extend the run from (q , v <sup>2</sup>, ε) with (q , v <sup>2</sup>, ε) <sup>t</sup> −→ <sup>δ</sup> −→ (q, v <sup>3</sup> , a) <sup>∗</sup> −→ (q <sup>1</sup>, v <sup>4</sup> , a) <sup>t</sup><sup>1</sup> −→ <sup>δ</sup>- −→ (q<sup>2</sup>, v, ε) where <sup>v</sup><sup>3</sup> q-- v <sup>3</sup> q-- v <sup>3</sup> and <sup>v</sup><sup>4</sup> q- 1 v <sup>4</sup> q- <sup>1</sup> <sup>v</sup> <sup>4</sup> . Finally <sup>v</sup> <sup>q</sup><sup>2</sup> <sup>v</sup>. Therefore, the invariant holds after the pop rule. 

#### **5 Algorithm for PDTA Reachability via Zones**

In this section, we describe Algorithm 1 implementing the fixed point computation defined by the inductive rules in Table 1. We describe the structure of the algorithm and its main data-structures.

Notice first that the sets <sup>S</sup> and <sup>S</sup>(q,Z) for (q,Z) <sup>∈</sup> <sup>S</sup> can be alternatively represented as a single set of pairs of nodes:

$$\mathcal{S} = \{ [(q, Z), (q', Z')] \mid (q, Z) \in \mathfrak{S} \text{ and } (q', Z') \in S\_{(q, Z)} \} \dots$$

We can recover <sup>S</sup> as the first projection of <sup>S</sup> and <sup>S</sup>(q,Z) as the second projection of <sup>S</sup> filtered by the first component being (q,Z). We use both notations below depending on which is more convenient. The start rule initializes S to {[(q0, Z<sup>0</sup>),(q0, Z<sup>0</sup>)]}.

Let us consider first the rule for internal transitions. For each already discovered pair of nodes [(q,Z),(q , Z )] ∈ S (or (q , Z ) <sup>∈</sup> <sup>S</sup>(q,Z) with (q,Z) <sup>∈</sup> <sup>S</sup>), we have to consider each possible internal transition q g,nop,R −−−−−→ <sup>q</sup> and check whether the node (q, Z) with Z <sup>=</sup> −−−−−−→ R(g <sup>∩</sup> Z ) should be added to <sup>S</sup>(q,Z) or is subsumed by an existing node. This is like a graph traversal. The set S stores the already discovered pairs of nodes, and we will use a ToDo (unordered) list to store the newly discovered nodes from which outgoing transitions should be considered. The ToDo list should also consist of pairs [(q,Z),(q , Z )] so that when a new node (q, Z) is discovered by an internal transition from (q , Z ) we know to which set <sup>S</sup>(q,Z) it should be added.

As we can see from Theorem 1-soundness, given (q,Z) <sup>∈</sup> <sup>S</sup>, the set <sup>S</sup>(q,Z) should consist of nodes reachable from (q,Z) via a well-nested run. Hence, when dealing with a pair [(q,Z),(q , Z )] ∈ S and we see a push transition q g,push*a*,R −−−−−−→ q with Z <sup>=</sup> −−−−−−→ R(g <sup>∩</sup> Z ), we should not try to add the pair (q, Z) to <sup>S</sup>(q,Z) since the corresponding run would not be well-nested. Instead, we should search for a matching pop transition which could be taken after a well-nested run starting from (q, Z). This is why the push rule adds the new root (q, Z) to <sup>S</sup> (unless it is equivalent to an existing root). The pair of nodes [(q, Z),(q, Z)] is newly discovered and added to the ToDo list for further exploration.

The push transition may be matched with several pop transitions (which could be already discovered or yet to be discovered by the algorithm). To avoid revisiting the push transition many times, it will be stored by the algorithm in an additional set Spush. More precisely, we will store in Spush the tuple [(q,Z), a,(q, Z)] meaning that the root node (q, Z) may be reached from the root node (q,Z) via a well-nested run reaching some (q , Z ) followed by a transition pushing a onto the stack.

Finally, assume that, when dealing with a pair [(q<sup>1</sup>, Z<sup>1</sup>),(q <sup>1</sup>, Z <sup>1</sup>)] ∈ S, we see a pop transition q 1 <sup>g</sup>1,pop*a*,R<sup>1</sup> −−−−−−−→ <sup>q</sup><sup>2</sup> with <sup>Z</sup><sup>2</sup> <sup>=</sup> −−−−−−−−→ <sup>R</sup><sup>1</sup>(g<sup>1</sup> <sup>∩</sup> <sup>Z</sup> <sup>1</sup>). We will check whether it can be matched with an already visited push transition, stored in the set Spush as a pair [(q,Z), a,(q, Z)] with (q, Z)=(q<sup>1</sup>, Z<sup>1</sup>). If this is the case, the pop rule may be applied and the node (q<sup>2</sup>, Z<sup>2</sup>) added to <sup>S</sup>(q,Z) (unless it is subsumed by an existing node). The newly discovered pair of nodes [(q,Z),(q<sup>2</sup>, Z<sup>2</sup>)] is also added to the ToDo list for further exploration. Once again, the pop transition may also be matched with push transitions that will be discovered later by the algorithm. To avoid revisiting the pop transition many times, we store the tuple [(q<sup>1</sup>, Z<sup>1</sup>), a,(q<sup>2</sup>, Z<sup>2</sup>)] in a new set <sup>S</sup>pop.

**Data Structures.** We use a data structure TLM to store the triple of sets (S, <sup>S</sup>push, <sup>S</sup>pop) and which is accessed with the following methods.


Concretely, the data structure should store sets of nodes (q,Z) and be able to search or iterate through such sets. In order to make the algorithm slightly faster, we will segregate our sets of nodes, with the name of the state. We will use a hashmap in order to accomplish this task. See Fig. 6 where the concrete data structure is depicted.

We will use a first level hashmap to store the set of roots S. To implement TLM.isNewNode(q, Z, q , Z ), we first search for (q,Z) in the first level map, then a pointer TLM[q][Z][0] will lead to a second level hashmap for the set of nodes <sup>S</sup>(q,Z) and we search for (q , Z ) in this second level map. See Fig. 6(b).

To implement TLM.isNewPop(q, Z, a, q , Z ) and TLM.iterPop(q, Z, a), we first search the root node (q,Z) in the first level map, then a pointer TLM[q][Z][2] will lead to a second level hashmap storing the set of triples (a, q , Z ) such that [(q,Z), a,(q , Z )] ∈ Spop. To speed up the access, this second level pop map is segregated first on the key a, then on the key q to get the list of corresponding zones Z . See Fig. 6(c,d).

Finally, we also store the set <sup>S</sup>push to implement TLM. isNewPush(q, Z, a, q , Z ) and TLM.iterPush(a, q , Z ). Notice that Spush consists of triples [(q,Z), a,(q , Z )] where both (q,Z) and (q , Z ) are root nodes from <sup>S</sup>. Notice also that for the iteration we fix the second node (q , Z ). To get an efficient implementation, we first search the root node (q , Z ) in the first level map, then a pointer TLM[q ][Z ][1] will lead to a second level hashmap storing the set of triples (a, q, Z) such that [(q,Z), a,(q , Z )] ∈ Spush. To speed up the access, this second level push map is segregated first on the key a, then on the key q to get the list of corresponding zones Z. See Fig. 6(c,d).



We now show correctness of Algorithm 1. Note that TLM encodes a triple of sets (S, <sup>S</sup>push, <sup>S</sup>pop) defined by:

$$\begin{aligned} \mathcal{S} &= \{ [(q, Z), (q', Z')] \mid (q', Z') \in \mathsf{TLM}[q][Z][0] \} \\ \mathcal{S}\_{\text{push}} &= \{ [(q, Z), a, (q', Z')] \mid (a, q, Z) \in \mathsf{TLM}[q'][Z'][1] \} \\ \mathcal{S}\_{\text{pop}} &= \{ [(q, Z), a, (q', Z')] \mid (a, q', Z') \in \mathsf{TLM}[q][Z][2] \} \end{aligned}$$

**Fig. 6.** Two level map implementing the data structure TLM storing the sets <sup>S</sup>, <sup>S</sup>push, Spop.

Recall also the correspondence explained at beginning of Sect. 5 between a set <sup>S</sup> of pairs of nodes, and the set of roots <sup>S</sup> together with the sets of nodes <sup>S</sup>(q,Z) for (q,Z) <sup>∈</sup> <sup>S</sup>.

**Theorem 2.** *The set* <sup>S</sup> *encoded by* TLM *computed by Algorithm <sup>1</sup> is a fixed point obtained starting from the empty set by applying the inductive rules in Table 1. Therefore, Algorithm 1 terminates and is sound and complete for wellnested control state reachability of pushdown timed automata.*

*Proof (sketch).*


For the full proof details, we refer the reader to the long version [6]. 

### **6 Experiments and Results**

*Implementation* We build on the existing architecture of an open-source tool for analysis of timed automata, TChecker [19]. Our tool along with the benchmarks we used is available at https://github.com/karthik-314/PDTA reachability and more details can be found [6]. The input for our implementation are PDTA, rather than TA so we modify TChecker in order to run our experiments. While most of the TChecker file format will remain the same, the only place where we make a change to the syntax of the input, will be the edges. TChecker uses the following format, for its transitions,

```
edge:<Process>:<src>:<tgt>:<label>{
    do:<Reset1(x=0)> ; <Reset2(y=0)> :
    provided: <guard1(x==0)> && <guard2(y>=1)>}
```
The new format in order to incorporate the pushes and pops will be,

```
edge:<Process>:<src>:<tgt>:<label>{
    do:<Reset1(x=0)> ; <Reset2(y=0)> :
    provided: <guard1(x==0)> && <guard2(y>=1)>}
    [<push/pop>:<symbol>]
```
In case the operation is nop, the square brackets are left empty.

We have implemented two variants of Algorithm 1 for PDTA and we will compare these between each other and also with a region-based approach. More precisely, we consider the following 3 algorithms:

	- TLM.isNewNode(q, Z, q , Z ): Returns false if <sup>∃</sup>[(q,Z),(q , Z)] ∈ S with <sup>Z</sup> <sup>∼</sup>q- Z, and true otherwise.
	- TLM.isNewPop(q, Z, a, q , Z ): Returns false if <sup>∃</sup>[(q,Z), a,(q , Z)] ∈ Spop with <sup>Z</sup> <sup>∼</sup>q- Z, and true otherwise.

As mentioned in Sect. 4, if instead of simulation, we just use equivalence everywhere, we do obtain a correct algorithm for reachability in PDTA. Hence it is interesting to compare it with the above approach.

– **Region Based Implementation** (RB): A previous implementation [5], uses a region based approach in order to solve the non-emptiness problem in PDTA. We note two features of the algorithm. First, it uses a tree-automaton based approach for efficiency and correctness, but underlying it is the region (rather than zone) construction. Second, it works only with closed guards, while our approach works with closed and open guards.

We note the following important points regarding our implementation:

1. The used in our implementation will be LU [8], without extrapolation and with global clock bounds.


All experiments are run on Intel-i5 10th Generation processor, with an 8GB RAM, with a timeout of 120 seconds.

*Benchmarks.* We used a total of 10 benchmarks in our experiments, but parameterized several of them in order to test the scalability and to give us more insight into performance comparisons. The benchmark and their parameterizations are explained in [6]. We highlight only some salient points here. The benchmark <sup>B</sup><sup>1</sup> is the PDTA from Fig. 1. B<sup>2</sup>(k) is directly adapted from Fig. <sup>3</sup> with the constant <sup>y</sup> <sup>≤</sup> 1 parametrized to <sup>y</sup> <sup>≤</sup> <sup>k</sup>, and <sup>k</sup> + 1 pops between <sup>q</sup><sup>0</sup> and <sup>q</sup><sup>2</sup>. Note that <sup>q</sup><sup>3</sup> is unreachable regardless of the value of <sup>k</sup>. Benchmarks <sup>B</sup>3, B<sup>4</sup> are adapted from [5] with <sup>B</sup><sup>3</sup> involving untiming of a stack age into normal clocks. <sup>B</sup>5, B<sup>6</sup> involve significant interplay of push/pop edges and clocks and <sup>B</sup>6, B<sup>7</sup> also have open guards. More details can be found in [6]. We also note that automata B<sup>1</sup>, B<sup>3</sup>(3, 4), B<sup>5</sup>(k1, k<sup>2</sup>), B8, B<sup>9</sup>(k1, k<sup>2</sup>) accept a nonempty language, while the rest are empty. As described earlier this does not change the performance of the simulation and equivalence based approaches, but may significantly change the performance of the Region Based Approach.

*Results* Table 2 contains a selection of our experimental results; more can be found in [6]. From the table, we conclude first that the zone based approach is indeed faster than the Region Based Approach for all examples. Second, the simulation based approach runs faster than the equivalence based approach for all examples if the ToDo priority for removal remains the same. In fact, the performance of the simulation based approach depends mostly on the size of the PDTA, but the equivalence based approach is dependant on the constants used in guards as well, which is even more the case for the region based approach. Finally, our approach can easily handle closed and open guards.

Most of the timeouts that occurred during the experiments are due to Out of Memory (OoM) kills, especially for larger sized PDTAs. For smaller sized PDTA such as B<sup>2</sup>(100), the recorded number of nodes before timeout was 154700.

Regarding the performance, we would like to emphasize that B1, B2, B3, B4, B7 were designed to compare the Zone approach to the region (RB) approach.

**Table 2.** Results on the Benchmarks. Time recorded in ms, and timeout (T.O.) used is 120 s. OoM stands for Out of Memory kill. Results rounded up to 1 decimal. # nodes refers to the number of nodes in the zone/region graph explored. In case of timeout <sup>≥</sup> n, refers to recorded number of nodes n before timeout occurred. NA in RB columns represents that the region based approach does not handle open guards in transitions (B<sup>6</sup>, <sup>B</sup><sup>7</sup> have open guards.)


As a consequence these models are very simple and the number of explored nodes remains almost the same regardless of whether we use ∼ or to prune, which reflects in the times/sizes not being too different. However, the other examples B5, B6 are more complex and have nodes that get pruned during exploration (both using ∼ and ). Here we can see the clear improvement of over ∼ both in terms of time taken and also of number of explored nodes.

#### **7 Discussion and Future Work**

In this paper, we examined how an unbounded stack can be integrated seamlessly with zone-abstractions in timed automata. We would like to point out that two easy extensions of our work are possible. First, as remarked earlier, our algorithm checks for well-nested reachability, i.e., it requires to reach a final state with empty stack for acceptance. But we can generalize this to general control-state reachability by showing that a control state q is reachable in the PDTA (with possibly a non-empty stack) iff some node (q,Z) is discovered by our algorithm and added to some <sup>S</sup>(q-,Z-) (and not just to <sup>S</sup>(q0,Z0) as in the well-nested case). While this idea is simple and requires only minor edits to the existing algorithm, the proof of correctness requires more work and we leave this for future work.

Secondly, we can handle the model with ages in stack as in [1,3] with an exponential blowup (thanks to [13]). However, an open question is whether this blowup can be avoided in practice. As noted earlier, there exist extensions [14,15] studied especially in the context of binary reachability, which are expressively strictly more powerful, for which decidability results are known. It would be interesting to see how we can extend the zone-based approach to those models.

Finally, it seems interesting to examine further the link to the liveness problem, possibly allowing us to transfer ideas and obtain faster implementations. Another possibility would be to use the extrapolation operator (rather than, or in addition to, simulation), which we have not considered in this work.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Security**

### **Verified Cryptographic Code for Everybody**

Brett Boston<sup>1</sup>, Samuel Breese<sup>1</sup>, Joey Dodds1(B) , Mike Dodds<sup>1</sup>, Brian Huffman<sup>1</sup>, Adam Petcher<sup>2</sup>, and Andrei Stefanescu<sup>1</sup>

> <sup>1</sup> Galois, Inc., Portland, USA jdodds@galois.com <sup>2</sup> Amazon Web Services, Seattle, USA

**Abstract.** We have completed machine-assisted proofs of two highlyoptimized cryptographic primitives, AES-256-GCM and SHA-384. We have verified that the implementations of these primitives, written in a mix of C and x86 assembly, are memory safe and functionally correct, by which we mean input-output equivalent to their algorithmic specifications. Our proofs were completed using SAW, a bounded cryptographic verification tool which we have extended to handle embedded x86. The code we have verified comes from AWS LibCrypto. This code is identical to BoringSSL and very similar to OpenSSL, from which it ultimately derives. We believe we are the first to formally verify these implementations, which protect the security of nearly everybody on the internet.

**Keywords:** Cryptography · Automated reasoning · Verification

#### **1 Introduction**

Widely-used cryptographic libraries such as OpenSSL [20], BoringSSL [16], and AWS LibCrypto [2] are an enticing target for formal verification. These libraries are used, to a first approximation, by everybody—or at least the four billion or so worldwide users of the internet. Each primitive in these libraries typically consists of a modest amount of code, but these primitives loom large in both their security and performance impact. Cryptographic primitives are also unusual in that they have clearly defined specifications and very few dependencies, which removes some major challenges from general-purpose verification. As a result, in recent years many efforts have been made to verify cryptographic library code.

However, despite significant progress, widely-used cryptographic libraries have resisted verification, at least for the versions of the primitives that are used in practice. This is because these primitives are some of the most heavily optimized pieces of code in existence. For a cloud service, every packet involves a call to at least one cryptographic primitive, so even small optimizations will have large performance and cost impacts. As a result, for AES and SHA there is an enormous gap in complexity between simple and easily verified high-level reference implementations, and the highly optimized implementations used in production.

Optimizations create several difficulties when verifying cryptographic primitives. First, primitives are typically written in a mix of C and assembly. This means that a verification tool must model both of these languages and the manner in which they can interact. Furthermore, each optimization step inherently increases the difficulty of verification, because each requires one or more theorems showing that the optimization is sound. To add to this, many of these optimizations break the abstractions used in algorithm specifications. For example, the SHA-384 specification is defined using a function called Sigma0 that is unfolded and rearranged during the optimisation process (see Subsect. 6.1). Solver-based automation typically struggles to recover these abstractions.

The verification of cryptographic code has seen huge advances in recent years. Purpose-built libraries such as EverCrypt [21] can now match the performance of hand-tuned OpenSSL. These correct-by-construction libraries may be the future, but as of 2021 they have not yet seen wide mainstream adoption. Our aim as formal methods practitioners is to verify the cryptographic code on which users depend. What has been missing until now is the ability to verify the legacy cryptographic code that runs in production for hundreds of millions of users. This is the problem we solve.

*Approach and Results.* We have formally verified the memory safety and functional correctness of two key cryptographic primitives, AES-256-GCM and SHA-384 as they currently appear in the new AWS LibCrypto library (AWS-LC) [2]. AWS-LC is a general-purpose library maintained by Amazon Web Services for use with AWS applications. We targeted these algorithms in particular because they are used within AWS and included in the Commercial National Security Algorithms Suite [18]. We chose a block cipher and a hashing algorithm in order to cover multiple algorithm types and to be representative of other algorithms in AWS-LC.

Cryptographic algorithms have fixed specifications which permit a narrow range of designs, and as a result, implementations change slowly. The AES-256-GCM and SHA-384 implementations in AWS-LC are identical to those in Google's BoringSSL library, and as a result, our proofs apply to it as well. For these primitives, there are only small differences between BoringSSL and OpenSSL, and we are confident our proofs would also apply to OpenSSL with minor modifications.

Our proofs show that the implementations of AES-256-GCM and SHA-384 are input-output equivalent to formal specifications of their expected behaviour. We write our specifications in Cryptol [11], a pre-existing high-level language designed for use by cryptographic experts. Cryptol specifications are executable, so our proofs establish that for any input, the implementation and specification produce exactly the same result. To boot, our proofs guarantee that the code is free of undefined behaviour such as memory safety errors, meaning that any remaining correctness errors are local to the code being proved and cannot affect the calling context. We do not verify side-channel properties, nor do we analyse cryptographic security properties of the AES-256-GCM and SHA-384 algorithms.

We performed these proofs using the Software Analysis Workbench (SAW) [14]. SAW is an industrial verification tool designed to prove equivalence properties between abstract specifications and lower-level, more optimized implementations. SAW is a bounded verifier: loops must be verified under preconditions that guarantee termination, and data-structures must be statically allocated with bounded sizes.

We have run our proofs on fixed sizes of input data, i.e., fixed numbers of bytes to be hashed/encrypted/decrypted. The number of loop iterations in these algorithms are strictly fixed by the input size so this also implicitly bounds the execution length. We chose these sizes so as to exercise all branches and boundary conditions in the code and specification (in this, we follow Galois and AWS's previous work: see Chudnov *et al.* [7]). We discuss the scope and limitations of our proof in Sect. 7.

Each proof of a cryptographic primitive in SAW has two stages. In the first, the imperative input code is converted to a functional term using bounded symbolic execution. This depends on a high-fidelity model of the input languages. SAW already had an LLVM model used for C and C++ verification. For AES-256-GCM and SHA-384 we developed a new SAW model of x86 assembly, along with an interface with SAW's existing LLVM model. As well as modeling core x86, this also included modeling special-purpose instructions used to achieve high performance. A successful conversion only occurs for well-defined programs, and implies that the program is free of undefined behavior under the given preconditions.

In the second stage of a SAW proof, the symbolic term is compared to a specification term written in Cryptol. For many applications, SAW can discharge these equivalences automatically, but this is where the optimizations in AES-256- GCM and SHA-384 made verification much more challenging. The proof steps involved cannot be discharged automatically by current solvers, so instead, our proofs make careful use of rewriting logic to massage the terms into a form that can be discharged. Some of these proof steps may be amenable to automated solving in future.

Our proofs were developed collaboratively between a team of expert verification engineers. As well as technical innovation, these proofs also required careful proof engineering. By this, we mean the analog of software engineering—a combination of proof design, tool design, and team working practices which makes it possible to execute effectively on a verification goal. We found that to a degree, proof engineering *is* software engineering; that is, successful proof engineering has similarities to the practices needed when developing a challenging software project.

Aside from proofs and tool capabilities, there is something else notable about our project: we verify code that was never intended for formal verification. This is in contrast to many other efforts, which target systems that were designed with assurance in mind. For example, Galois and AWS previously verified an Amazon TLS library that was purpose-built as a high-assurance alternative to OpenSSL's TLS support [7], while in the EverCrypt library, code and proof were developed in parallel, and even the API was designed to simplify specifications [21]. We verify legacy code because this is the code that is actually used in AWS-LC and its predecessors.

*Contributions.* The key contributions of this paper are as follows:


All proof scripts are available online<sup>1</sup>.

#### **1.1 Related Work**

There is a considerable amount of recent work in cryptographic verification, representing a large space of application domains and design requirements. While our work is widely applicable, we do not consider it a one-size-fits-all solution. We discuss how a developer might choose between the many verified cryptography efforts in Subsect. 7.2. Here we give an overview of projects that target C or x86, or that are closely related technically. We do not review work on verifying cryptographic security properties, which is orthogonal to the problem of verifying that code matches algorithm.

The closest work to ours in terms of technical approach is Galois and AWS's previous work verifying the HMAC and DRBG primitives in the AWS s2n TLS library [7]. Just as we do, they use SAW to verify production cryptographic code. The main difference from our current project is the complexity of the primitives verified. The HMAC and DRBG primitives are inherently simpler algorithms, and are written in C, rather than x86. Furthermore, this code was designed for verification, unlike the OpenSSL-derived code we target. In earlier work, Ye *et al.* also verified C versions of HMAC and DRBG from OpenSSL using the foundational Verified Software Toolchain (VST) [22].

The Everest project has developed verified C/x86 cryptographic library called EverCrypt [5,10,21,23]. Recent results are extremely impressive, with performance comparable to highly optimised OpenSSL code. However, EverCrypt

<sup>1</sup> https://github.com/awslabs/aws-lc-verification.

represent a different philosophy from ours, where the library and proof are codesigned, and in some cases code is synthesized. This approach looks towards a future where such libraries replace hand-written libraries like AWS-LC, BoringSSL, and OpenSSL. Our philosophy is complementary: we verify code as it currently exists while we wait for the future to arrive.

EverCrypt also differs in that they use a proof-assistant style of reasoning more similar to Coq or Isabelle. The advantage of this is that proofs are very flexible—for example, they work for unbounded input sizes. However, the cost is that proofs are relatively more verbose. Proof size is hard to estimate in Ever-Crypt, because the proof and implementation are mixed, but the earlier Vale paper [10] suggests that EverCrypt's proof of AES-GCM uses 2000 lines of proof library plus additional proof mixed in. In comparison, SAW is designed to automate reasoning where possible, and the proof of AES-256-GCM implementation takes us less than 1000 lines of proof (including white-space and comments, for attempted apples-to-apples comparison).

The CASM [17] project verifies x86-based cryptography taken from OpenSSL, including SHA-256 (we verify SHA-384). CASM's toolchain is similar to ours, based on symbolic execution and SMT solvers. However, CASM only examines functions over message blocks, rather than the whole SHA-256 algorithm. CASM also does not verify the most highly optimised versions of this algorithm. For example, it omits x86 EVP and vector operations, two of the main challenges.

Fiat Crypto [9] is a related approach, although it does not apply to the algorithms proved in this paper. It foundationally generates portable C field arithmetic implementations from a high level specification. Code synthesized by Fiat Crypto has already been added to OpenSSL. Jasmin [1] is another foundational synthesis approach. It generates high-performance vectorized x86 implementations. The Jasmin implementation of ChaCha20-Poly1305 outperforms similar hand-optimized implementations. We have not seen Jasmin implementations of SHA-2 or AES-GCM.

SAW's approach has some similarities to model checking, in that it is a bounded verification technique. However, proofs are based on symbolic execution, that is, construction of logical terms representing the program denotation, and proofs are bounded on input buffer size, not program execution length perse.

#### **2 Project Design Constraints**

Our objective in this project was to verify the cryptographic code which is actually deployed, and to ensure it stays verified as it changes over time<sup>2</sup>. To do this, we used *continuous reasoning*, a term due to Peter O'Hearn [19]. In continuous reasoning, there is a tight connection between code, software engineering process, and verification tools. Several recent industry projects have successfully used continuous reasoning practices. It was also important that our tools

<sup>2</sup> In fact, we do not expect AES-256-GCM and SHA-384 to change often in AWS-LC, but this work takes place in the context of a larger AWS-LC assurance project.

maintain the existing institutional trust in the original codebase—this ruled out whole-code replacements such as EverCrypt. This resulted in the following design constraints:


These constraints led us to use the SAW tool as our basis for verification [14]. Our project can be seen as a follow on to Galois and AWS's prior verification of AWS s2n which had many of the same design objectives [7]. Chudnov *et al.* showed that SAW can be used for continuous reasoning for a relatively simple piece of C cryptography. The difference in our current project is the inherent difficulty of verifying the code.

### **3 AES-256-GCM and SHA-384 Proof Structure**

Conceptually, SAW's approach to proof works as follows. The tool symbolically executes C and x86 code, resulting in a collection of functional terms. A term describes every program output mathematically as a function of program inputs. Once side conditions have been discharged, completion of symbolic execution also implies that the program is safe: that is, memory safety errors cannot occur. In the final step of the proof, these functional terms are compared to specifications using a solver to determine whether they are equivalent.

*Interfaces.* At the top level of our proof, we verify the AWS LC primitives against OpenSSL's EVP interface<sup>3</sup>. OpenSSL and its descendants use this interface to make it easy to swap out algorithms without exposing their implementations. This complicates the verification task by hiding functions behind pointers and union types. It has also attempted to remain largely backwards compatible for years, resulting in an API that is not as clean as it might be otherwise. Perhaps for these reasons, previous cryptographic verification projects have not verified the EVP interface.

<sup>3</sup> https://wiki.openssl.org/index.php/EVP.

**Fig. 1.** Part of the EVP interface for AES-256-GCM.

*SAW-Script Specifications.* The top-level EVP specifications are defined in SAWscript, the high-level control language for SAW. Figure 1 shows part of the SAWscript EVP interface for AES-256-GCM. In its form, this interface consists of a series of instructions in SAW-script, but in its effect, it is a Hoare-style pre/post specification. The interface sets up symbolic memory (the pre-condition), symbolically executes the function (crucible\_execute\_func), and then checks that the resulting symbolic memory contains the correct values (the post-condition).

For AES, the main purpose of the pre-condition is to define the layout of memory that results from the AES initialization function. Because we define post-condition for the initialization function that match the specification given here, we can end-to-end verify the common use case of initializing memory, encrypting some input, and returning the result.

The script defines the memory pre- and post-conditions for the function using points-to assertions. In SAW-script, we allocate symbolic memory at specific sizes using the crucible\_alloc commands. We can then use the points\_to command to specify that a pointer points to symbolic memory. The ptr\_to\_fresh command is a convenience function that allocates a pointer, and then initializes it with symbolic memory.

SAW's logic is less expressive than a full separation logic, but specifications can naturally be interpreted in terms of separation, including the property that memory cells do not overlap. To make the memory layout easier to understand, consider the following separation logic triple, which roughly corresponds to the layout defined in the SAW-script:

```
{cipher data ptr -
                     → ctx... ∗ in ptr -
                                       → in ∗ out ptr -
                                                       → ( : [len])}
EVP_CipherUpdate(ctx_ptr, out_ptr, out_len_ptr, in_ptr, len)
          -
           cipher data ptr -
                               → cipher update(ctx...) ∗
            in ptr -
                   → in ∗ out ptr -
                                   → ctr32 encrypt(ctx, in)
```
Rather than syntactically divide the pre-condition and post-condition, as in a Hoare triple, the two are divided by the call to crucible\_execute\_func, which indicates symbolic execution of the target C or x86 function. Crucible is the intermediate language for symbolic execution used by SAW. Internally, the semantics of LLVM, x86, and other SAW input languages are defined by translation to Crucible.

One reason for the complexity of these specification is that SAW differentiates between data that is allocated and initialized and data that is just initialized. Other verification tools tend to treat all allocated data as initialized (for example, this is true of CBMC [8]). This is generally a sound approximation because C compilers tend to behave predictably, but our approach is more accurate to the specification of C.

*Functional Specifications.* The other role of SAW-script is to verify the connection between the implementation and algorithmic specification. In SAW, specification are written in Cryptol, a domain-specific language designed for cryptographic specifications [11]. In the postcondition of the script, we use references to Cryptol functions to map the outputs of running the program to the outputs of our specification programs, ctr32\_encrypt and cipher\_update. The final lines of the specification assert that the memory cells resulting from the program must match the required values, i.e., those that would result from executing the Cryptol specification.

We show ctr32encrypt in Fig. 2. This function defines the top-level behavior of the CTR mode of encryption, which repeatedly increments an initialization vector, encrypts the incremented value with the secret key, and performs an XOR of that encryption with the plaintext.

The first line of the specification defines the type of the function, parameterized by type variable n. AES\_GCM\_Ctx is a structure used to maintain state for the incremental interface to AES, which allows for data to be encrypted and decrypted as it becomes available, rather than all at once. The [n][8] arguments are sequences of bytes with length n.

The function body consists of a sequence comprehension. This takes input bytes one at a time, and labels them with i, which draws from the sequence counting up from ctx.len. The separate function EKij performs the encryption step using the initialization vector and the key contained in the context. The take and drop functions are used to convert the 64-bit length contained in the context to a 32-bit number required by the EKij function.

**Fig. 2.** Top-level Cryptol specification for AES update.

Another example of a functional specification is the following line describing the Sigma0 function:

S0 x = (x >>> 28) ^ (x >>> 34) ^ (x >>> 39)

In the SHA-384 code, this function is implemented by the Perl code given in Fig. 3. This does not execute directly, but rather generates assembly code, which is what we verify. The instructions ror and xor correspond to the cryptol operations >>> and ^ respectively.

In order to include the implementation here, some constants have been substituted, and we have extracted the relevant lines from around 20 other lines calculating other parts of SHA. Those lines are mixed in with even more lines of non-interfering SHA calculations, presumably in order to keep the processor saturated. Symbolic execution allows us to reason just about these lines of code, because interleaved instructions that don't change the result of the computation in a relevant way will not be included when reasoning about the results of individual computations.


**Fig. 3.** Perl implementation of internal SHA computation.

Notice also that the shift amounts are different between the functional specification and the code. In Cryptol, the shift amounts are 28, 34, and 39, but in the implementation, we see shifts by 39 − 34, 34 − 28, and 28. This is a performance optimisation, but it makes the proof effort more difficult. To close this gap, we use a system of verified rewrites (see Sect. 6).

*Verification Process.* Once a specification has been defined, it must be verified. SAW divides verification into two phases: symbolic execution, and verification of equivalence. Symbolic execution converts an imperative operation into a functional term suitable for automated reasoning. Even without specifying the expected high-level behaviour of the AES function, the memory layout defined in the pre-condition is enough for symbolic execution to complete, which has the effect of proving the imperative code memory safe. We typically verify memory safety in this way before developing a specification. This lets us separate concerns between functional and safety properties.

The final task once symbolic term has been generated is to compare it to a specification term. SAW uses SMT solving to discharge these proofs, and in most use cases, these can be completed automatically. However, the complexity of the optimization stages in AES-256-GCM and SHA-384 makes the gap between specification and implementation too large to be completely automated. SAW solves this with a small tactic language embedded into SAW-script that supports term rewriting. Each of these rewrites Sect. 6.

*Modular Reasoning.* Symbolic execution is a precise technique with hard limits on its scalability. The AES-256-GCM and SHA-384 functions are too large to be symbolically executed in their entirety. SAW solves this problem through using a modular reasoning system called *overrides*. SAW treats specifications as executable code that can be freely substituted for implementation functions. When a function is verified equivalent to a Cryptol specification, calls to that function can be overridden (i.e., replaced) during symbolic execution. As Cryptol specifications are typically much less complex than implementations, this massively increases the tractability of the verification task.

As a result, a typical SAW proof consists of a hierarchy of equivalence proofs. The proof begins at the leaf functions, which are verified by symbolic execution. The functions at the next level are then symbolically executed with the leaf functions replaced by their specifications. These are then also added to the library of verified functions. This proceeds until the top-level function is verified. One of the main tasks when developing a SAW proof is defining these internal specifications (our proof is unusual in that we also needed a significant number of rewrite rules).

We also use the override mechanism at the interface between C and x86 code. Functions in x86 are proved equivalent to Cryptol specifications, and these specifications can then be used as overrides in the surrounding C context. This approach works because we have defined a compatible memory model that works for both C and x86 code—see Sect. 5 for more.

Finally, the override functionality can be used to assume specifications for functionality that has been assumed, not verified. This is useful for library calls that might be out of scope for a particular project, but that might be verified in the future. For example, Chudnov *et al.*'s SAW proofs for s2n [7] use this approach to parameterize the proofs of HMAC and DRBG over different primitives<sup>4</sup>. In our proofs, the only assumptions we make are that OPENSSL\_malloc and OPENSSL\_free behave correctly.

#### **4 SAW's Verification Pipeline**

SAW is structured as a pipeline of linked verification stages. The inputs to the pipeline are, firstly, executable mathematical specifications for the top-level function, and selected sub-functions; secondly, the compiled code, made up of LLVM and embedded x86 binary code; and thirdly, a proof script which sets up memory, identifies the mapping between Cryptol specifications and function interfaces, and contains the rewrites that are applied to the resulting logical terms. The verification pipeline then works as follows:


Verification proceeds with functions progressively higher on the call-graph, until the top-level equivalence is proved between code and specification.

While the structure of this pipeline is simple, making it work for real code requires a significant amount of tool sophistication. SAW is the product of many years of refinement and development, and we used many of the components in this pipeline without modification.

Our C support is based on SAW's LLVM support, which is mature, and has been used in many other industry and government verification projects—for example, Chudnov *et al.* [7]. While we do not claim complete coverage of the standard, in practice we rarely need to add new C language features to SAW. Likewise, Cryptol support is built into SAW and is designed to be symbolically executed, so this part of the tool required no modifications. The Macaw and What4 tools similarly functioned without modification.

<sup>4</sup> In fact, we have now verified some of the primitives that were only assumed in this previous work, meaning it should be possible to stitch these proofs together end-to-end.

Therefore, in this paper we focus on the new capabilities of the tool: our symbolic execution of x86 instructions, and verified rewrites. For a more detailed treatment of the SAW suite as a whole, readers should look at the SAW documentation and tutorial [13,14].

#### **5 New Capability: x86 Semantics**

The first SAW capability we developed for this project was symbolic execution for x86 assembly code, including support for mixed C/x86 code. Doing this required us to solve two problems. First, decompiling the binary into a series of x86 instructions, and second, defining the semantics of instructions, which mainly involves defining the model of memory.

To decompile we use Macaw, a SAW sibling project which is able to parse Elf binaries and output a control-flow graph complete with the representation of the x86 instructions [12]. We treat Macaw as a black box, and in fact any decompiler with similar capabilities could serve in its place.

Once the CFG has been constructed, we apply our x86 semantics. For the behaviour of individual instructions, we consulted the Intel manual. We note that processor manuals contain errors, and hand-encoding the semantics could also introduce errors. However, we have reasonable confidence in this encoding because, in practice, most conceivable errors would immediately cause the proof to fail. This is because cryptographic functions are very sensitive to small changes: most small value errors would result in a dramatically different output.

Much more important and subtle is the memory model, which describes under what conditions reads and writes to memory can occur, as well as describing how reads and writes can be combined to store and retrieve values. Unlike C, there are almost no accepted memory usage rules for assembly programming, aside from the conventions used in a particular program and the Application Binary Interface (ABI) for functions that can be called externally. Fortunately, AES and SHA implementations are designed to be called by C programs. They therefore must follow C-like conventions and respect the ABI. Memory is used to get inputs and define outputs, read global constants, and maintain a stack for storing temporary results. Functions always respect the boundaries of data as provided. Because of this, we were able to adapt SAW's well-tested model used for LLVM support.

In SAW's memory model, addresses are represented by a pair of integers: the first integer is a base address, identifying an allocated memory region, while the second is an offset into the region. Memory operations, such as pointer arithmetic and pointer comparisons, are only well-defined for addresses in the same region.

Even after defining this model, we had to decide how to apply it within the proof. There were two options: (i) modeling the entire memory as a single region, and (ii) representing different objects as separate regions. The former is the more flexible because it does not enforce any invariants on the way that memory is used. Any read or write within the entire memory region is valid at any time. This comes at an increased cost of manually specifying necessary invariants. For example, each function would have to manually encode the memory region it might write to so that its calling function can predict all of the side effects of calling it.

Instead, we take the second approach: automatically specifying such memory invariants as part of the way that memory can be used. This means that some valid assembly will be impossible to verify. It could be completely safe and correct, but because it violates the strict memory model we've chosen, our tool will be unable to reason about it. On the other hand, the memory model we chose works for all of the cryptographic assembly code we've run into, and implementing the memory model in this way saves us a substantial amount of specification and proof work.

It is not surprising that this approach works; The models and abstractions C uses for memory are useful in assembly as well. Furthermore, the ABI and the C memory model have heavily co-evolved, making the C memory model a natural fit for assembly functions that match the ABI.

The memory model is applied by symbolic execution of the CFG that results from Macaw. This symbolic execution has two main functions: efficiently update a symbolic representation of memory, and discharge side conditions that must hold in order for symbolic execution to continue. The result is a SAW-core term representing the input-output behaviour of the x86 binary code.

#### **6 New Capability: Verified Rewrites**

The second SAW capability we developed was a simple language of term rewrites for use in proofs.

After symbolic terms have been constructed from C, x86, or Cryptol, we must prove equivalences between these terms. The design goal with SAW is that these proofs are completed mostly automatically using SMT solvers. While this has worked well in previous, less-complicated proofs, the functional terms that result from AES-256-GCM and SHA-384 often proved to be intractable for the solvers without preprocessing. This is exactly because these algorithms are so heavily optimised, as we have discussed above.

In order to solve this, we introduce a language of equivalences between terms that are themselves verified by the solver. By applying these rewrites, we can close the gap between the more abstract Cryptol term and the optimized C/x86 term. These rewrites serve as a small tactic language for controlling the proof, while preserving the principle that SAW proofs are mostly automatic.

To illustrate how this works, we consider an example rewrite from our SHA-384 proof. In the Cryptol portion of our proof, we define the following function, S0 (shortened for convenience from Sigma0):

S0 x = (x >>> 28) ^ (x >>> 34) ^ (x >>> 39)

In Cryptol, >>> and <<< are right and left rotation respectively, while ^ is XOR. In order to complete the proof, we need to be able to rewrite occurrences of this function. To do this, we define the following rewrite, Sigma0\_thm:

#### Sigma0\_thm <- prove\_folding\_theorem {{ \x -> (x ^ ((x ^ (x <<< 59)) <<< 58)) <<< 36 == S0 x }};

The left hand side of this equation is how symbolic execution interprets the code in Fig. 3. The rotate-rights have been swapped to rotate-lefts, which allows our semantics to model both types of instruction by rotate-left. In order to swap the rotates, we subtract the rotate amount from 64, which is why we have a different set of constants than we see in either the specification or the implementation.

The solver verifies this equivalence for all possible values of x and saves it with the name Sigma0\_thm. In this case, we verify the equality using the ABC solver [6] through What4, but different solvers can be applied as needed to provide different equivalences.

Consider the following SAWscript command, which verifies an x86 function matches its specification:

```
sha512_block_data_order_spec <-
```

```
crucible_llvm_verify_x86 m "<filename>" "sha512_block_data_order"
 [ ("K512", 5120) ] // Initialize global for round constants
 true
 sha512_block_data_order_spec
 (do {
   simplify (cryptol_ss ()); // std simplifications
   simplify (addsimps thms empty_ss); // folding theorems
   simplify (addsimp concat_assoc_thm empty_ss); // final theorem
   w4_unint_yices ["S0", "S1", "s0", "s1", "Ch"]; // uninterpreted fns
  });
```
Here, the do-block defines the order in which the simplification rewrite rules are applied. The folding theorems thms contains 30 rewrite rules, including the Sigma0\_thm presented above. The concat\_assoc\_thm theorem normalizes the concatenations that result from other proof rules. The final line of this script instructs the Yices solver to treat certain functions as uninterpreted, including the S0 function. This illustrates the usefulness of the rewriting support. Rather than reasoning about the S0 function directly, we rely on the verified rewrites. This allows us to abstract away from complexity that previously made the proof infeasible for the solver.

Overall, the tactics we use for SAW proofs constitute a very simple decision procedure, made up almost exclusively of user-supplied rewrites. The other main mechanism we have for guiding proofs is the modular override system described in Sect. 3, which allows us to decompose proof tasks into lemmas, at least at the granularity of functions. In practice, we have found that these capabilities are sufficient to meet the needs of proving cryptographic implementations correct with respect to specifications.

Ultimately, we may find ourselves limited by the tools available in SAW for controlling the proof process, particularly if we attempt to prove higher degrees of abstraction between specification and implementation. These more manual proofs largely fall outside of the scope of what SAW aims to do well. An ideal solution would be to export proof goals to Coq, Lean, or F\*, all of which already have highly-usable better tools for manual proof. Chudnov *et al.* have previously demonstrated that SAW proofs about code and more abstract Coq proofs can be connected in this way.

#### **6.1 Role of Rewrites in AES-256-GCM and SHA-384 Proofs**

Rewrites in SAW can be seen as a small tactic language, serving a similar purpose to proof tactics in Coq or Isabelle. However, SAW occupies a very different point in design space, because it is designed to maximize proof automation. Heavy use of SMT-backed automation is the reason our proofs were feasible, but if the automation makes poor choices, it can also obstruct the proofs. We use rewrites along with appropriate choices of abstraction boundaries, to recover abstractions that automation would not discover itself.

For example, consider the Sigma0\_thm rewrite defined above. The solver can verify the rewrite when supplied in isolation. However in the context of SHA, the solver fails to identify this as a valuable fact. One reason is that S0 is a function that is present in the Cryptol specification, but this abstraction is lost when we symbolically execute an x86 function. The rewrites replace occurrences of S0 with an uninterpreted function, pruning the proof space dramatically.

However, there is the trade-off in reintroducing such an abstraction. Even if the abstraction holds locally, the functionality that calls that abstraction might depend on the internal functionality. In that case, swapping out the code for an uninterpreted function could actually turn a solvable goal into an unsolvable one. The answer is to choose these rewrites carefully: this is one of the main intellectual challenges in completing a proof. In general, this problem is undecidable. For example the rewrite rules inferred may not terminate, this means that at best it might be a guided special-purpose mode of solvers, rather than a general purpose approach.

Our rewrites plug into SAW late in the pipeline, after many of SAW's optimizations. This means that rewrites sometimes have to compensate for earlier optimizations. SAW is designed to aggressively optimize terms into a form suitable for the solver, and in some cases, this means breaking up abstractions that would be useful in completing the proof. In these cases, our rewrites must operate on the post-optimization proof term.

For example, in one case, it would have been desirable for our proof to use the term:

```
{{ \x -> (slice_59_5_0 x) # (slice_0_59_5 x) == x <<< 59 }};
```
However, SAW discovered that it could drop off the operation on the final byte of x, but to do so, it had to break up x into its constituent bytes. This is a desirable optimization if the term is passed to the solver directly, because the solver itself will reason at the level of bytes. However, this made writing an appropriate rewrite for our proof much more challenging. We include the eventual rewrite rule in Fig. 4. Again, a large amount of the intellectual challenge with our proof rested in finding appropriate rewrites that integrated with SAW's existing automation.

**Fig. 4.** Example rewrite rule. This rule is made much more complex by the fact it happens after SAW's existing term optimization phases.

#### **7 Results and Lessons Learned**

Our proofs run on the current version of AWS-LC [2] as of January 2021, built using the default compiler flags. We verify the AVX implementation of SHA-384, which is the current fastest implementation. Our AES-256-GCM proof uses the code path for AESNI, CLMUL, and AVX instructions.

*Proof Size and Composition.* Our code can be broken down into top-level functional specifications, top-level interface specifications, and proof scripts. The top-level specifications are what must be understood in order to understand the results of our proofs.

We have 168 lines of top-level interface specifications, which define the 8 interface functions that we've proved correct. Those functions specify memory layouts for the interface functions and link them to the top-level functional specification. We have 435 lines of top-level functional specifications, which were only slightly modified from specifications that we and others have used in previous cryptographic verification projects. These are almost completely free of implementation details, and live in a specifications only repository, separate from the code. If the functional specifications were made any shorter, they would likely also be less readable, so we believe they are close to optimal for their purpose.

The proof scripts consist of 1286 lines of intermediate function specifications, rewrite rules, tactics, and proof running logic. These intermediate functions are both proved and checked each time they're called. As a result, they do not need to be understood or trusted in order to believe the top level results.

Following continuous reasoning practice [19], our proofs are integrated into the CI process for AWS-LC. We do not expect this code to change, but we have adopted this practice as part of a larger AWS-LC assurance effort, including code which does change more often. Quickcheck versions of our proofs run as GitHub actions that take around 25 min<sup>5</sup>. The complete version runs on private systems in 30 min, but using more cores and memory. A significant part of our proof and tool development effort was dedicated to making sure proofs could run within a time budget acceptable for CI (typically 1 h). For example, this sometimes required introducing overrides to break the proof into smaller segments.

*Achieving Trust in the Proof.* SAW is designed to increase confidence in software, but it cannot supply total certainty. A key question is therefore what parts of the toolchain and proof must be trusted. For our proofs, the Trusted Code Base (TCB) consists of:


Although this TCB is significant, it is comparable in scale to similar verification projects like EverCrypt [21]. The highest impact improvement would likely proving the algorithms at arbitrary sizes. While we could throw computation at running the proofs at a wide range of fixed input sizes, this would spend computation and developer wait time with fairly little benefit. We believe an inductive approach is achievable in future work and would allow us to verify the algorithms once and for all.

In the mean time, we have covered some of the most-used block sizes, as well as all code paths. Given we have verified the algorithm at fixed sizes, and the code does not branch on input size, the only place that bugs could remain is the looping behaviour at other sizes. We have inspected the dynamically bounded loops in the code carefully to mitigate this possibility.

<sup>5</sup> https://github.com/awslabs/aws-lc-verification/actions.

We could also shrink the TCB by applying foundational techniques such as used in the Verified Software Toolchain project [3]. This would remove much of the need to trust the tool itself. However, we believe doing this would make it infeasibly expensive to develop a tool as complex as SAW, at least given current foundational techniques.

Another important question is whether a correctness failure in the toolchain could result in a proof that does not establish the result we expect. We believe the probability of this is quite low. Our current best defense against this failure of TCB is thorough testing and code review. SAW itself is a well-tested tool that has been used for many projects. The language models have also been tested, and it is unlikely that a behavioural bug could cause an incorrect specification to be verified. For this to occur, several failures would need to occur at once.

As an aside, the intent of the people doing the proof, and the precise nature of any external review should be considered when answering this question. A tool bug is unlikely to result in a proof that falsely appears correct, *assuming that the proof effort is done in good faith*. On the other hand, all tools have bugs, and in most logic-based tools, bugs can allow the construction of false proofs that appears superficially correct. In other words, for most current tools, trust in verified code requires trust in the team and process producing the proof.

We believe the highest risk of accidental error lies in the specifications. It is quite common for draft specifications to contain subtle discrepancies between what users intends and the specification's formal meaning. We mitigate this with extensive manual audit. Every line of code we write is reviewed at least once within the verification team, and once by AWS-LC domain experts. The internal review ensures that our specifications are correct and that our style is consistent with our guidelines. The external review allows us to ensure that we have explained our proofs correctly, and that we have correctly specified the functions in the context that they are being used.

*Proof Engineering Process.* The proofs were completed over six calendar months, using approximately nine person-months engineering effort total. We consider this to be an upper bound estimate as the proof effort was mixed in with tool improvements, in particular for the less-mature x86 tooling. The core team consisted of four engineers, with additional contributions from verification tooling experts and AWS-LC domain experts. This project represented a significant engineering effort, but for our project, this represented a good use of resources to achieve a high level of confidence in the AWS-LC code. Proofs were completed alongside more traditional assurance approaches, e.g., testing, fuzzing, and code audits.

New proof techniques and tooling were a factor in our success, but there is no single technical breakthrough that made these proofs possible. While combined x86 and C verification is challenging, it would likely be possible (although not easy) to add such a capability to a number of existing tools. Rather, a series of tool extensions, design choices, and engineering working practices combined to make the project feasible.

Using SAW, we automated most of the trivial reasoning, which meant that a majority of proof engineering was spent on legitimately difficult verification problems. These mainly involved understanding the code being verified and using rewrites to manually rearrange verification terms to make them amenable to automated proving. Many of these steps could in principle be automated, but in practice engineers sometimes needed to resort to clunky debugging measures. We find it unsurprising that highly specialized code such as AWS-LC would generate edge cases that challenge generic proof automation. For proofs of this type, for now we believe completely automated proving is out of reach.

We take several steps to try to minimise engineer effort when building proofs. The most important of these is to lean on automation wherever possible. One example is that we try to avoid internal specifications, which are often the most challenging part of the proof. Because SAW is a bounded verifier, internal specifications are just a performance optimization—given sufficient compute resources, we could in principle symbolically execute the entire code-base. Of course, in practice, internal specifications are needed to make the proof tractable. Our practice is to prove functions at the largest scope which fits within our time budget. By doing this, we are sometimes able to avoid specifying internal functions that do relatively little computationally.

Another important strategy for us is to separate memory-safety proofs from functional correctness proofs. We have found that much of the technical risk in a verification project can be eliminated at the memory-safety stage. This is where the verification tools are most likely to run into show-stopping bugs that will put success of the project in jeopardy. Separating these concerns results in proof terms that are smaller and easier to understand, so bugs are easier to diagnose. Then, if we run into challenges during the correctness proving phase, we can limit the cause to correctness properties, eliminating a large fraction of the proof from consideration.

An important factor that enabled us to carry out these proofs is a team of expert proof engineers who have built their skills over years. This project was undertaken by a team which has worked continuously on verification projects for four years. This expertise has given us a better understanding of what we can attempt, and a far wider toolkit to dip into when things go wrong. We have seen, anecdotally, similar evidence of improved verification capabilities from other long-standing teams—for example, for the Project Everest, SeL4, and CompCert projects. Long-standing teams of proof experts are still unusual, but we believe they will be necessary to achieve the most ambitious proof engineering tasks, just as they are in software and tool development.

A significant lesson that we have learned about proof engineering is that a tool's behaviour when it fails is more important than success. This is a critical aspect of verification tools often overlooked in research papers. Many tools show a demo where everything works, but in a proof engineering effort, the vast majority of time is spent with a proof that does not work. In that sense, one of the most critical aspects of a verification tool is what it does when the proofs are not working. SAW provides some support for diagnosing errors, but there is a lot of room for improvement. It lacks tooling to allow proof engineers to easily inspect and modify proof terms that are not successfully proving. Furthermore, it has inefficiencies that can make repeatedly running and modifying proofs slow and painful, increasing the pain of developing proofs and reducing the time that engineers can spend on the real challenges of verification.

#### **7.1 Trade-Offs When Building on Existing Verification Tools**

As we saw in Sect. 6, rewriting is an example where SAW's existing tooling made some parts of our proof more awkward. It is reasonable to wonder whether we could have modified SAW to allow more control of the rewriting pipeline. This highlights an interesting trade-off that exists when developing proofs using a more mature tool like SAW.

SAW has existed for a decade, and has been developed and improved iteratively over this time. Design decisions such as the order in which optimizations occur can sometimes be baked deeply into the tool. This stands in contrast to more experimental tools which often have short histories and a relatively clean design that can be torn down and refactored easily. SAW also has an active user community which relies on it for different verification and assurance tasks. The main users are at Galois, Amazon Web Services, and in the US government. This means that tool changes need wider approval from a community. Again, this stands in contrast to research tools which often have a single designer who is also the main user. The effect of this is that changes such as the introduction of rewriting must be carefully designed to fit with SAW's existing architecture.

The pay-off for these restrictions is an enormous increase in the power and scope of what we can achieve with the tool. In the large, we have benefited from many features that were developed by independent research teams. For example, we rely on the Macaw decompiler, which we used off-the-shelf without modification. The SAW LLVM semantics is likewise a product of many years of research, which did not require any further work from us. In the small, SAW embodies many, many clever tricks and pieces of good design that together make verification of challenging problems more feasible. Sometimes working with a mature tool imposes costs, but overall we believe it raises the bar for our work in a way that easily justifies the cost.

An open question for us is how we can make such collaboration possible across the verification community. Boogie [4] is a good example of a verification technology that has seen use across different teams and institutions. Proof assistants and SMT solvers are also widely used as a basis for new tools. However, there are still very few software verification tools that have seen significant adoption. We believe such tools will be necessary in the future if we collectively are to tackle larger and more complex verification problems.

#### **7.2 Verified Code Generation Versus Verifying Existing Code**

While the approach in this paper results in an artifact that may appear externally similar to other state-of-the art verified cryptography efforts, there are some engineering factors that might influence which approach is most appropriate for a particular cryptographic use-case. The approach of EverCrypt, Jasmin, Fiat, and similar efforts require the user to produce code or a model in a language that is specific to the verification system. While these systems have demonstrated ability to produce efficient verified implementations, they cannot directly verify existing code. Our approach verifies existing code without modification, and there are several engineering benefits to this.

The most significant reason to verify existing code is that producing new code or modifying existing code introduces risk. Modifying optimized cryptographic code is particularly risky because it is complex, and because an error could have a devastating impact on the security of the system. A software project may be unwilling to accept the risk of modifying mature code, even if the new code is formally verified. For example, OpenSSL and its variants have been tested and audited over more than a decade, and this maturity is appealing to many software projects. In our approach, the code is verified without any modification. Zero new risk is introduced, and the verification process only increases trust in the system. A related benefit is that the verified code maintains any existing certifications, such as FIPS 140-2.

Another benefit of verifying existing code is that the verification works on the programming languages that are already used in the project, and the build pipeline does not require additional compilers or other tooling to support the language of the verification system. Having the build depend on this tooling can be risky because it is less familiar and less mature compared to the compilers and build systems that are typically utilized. There is a risk that a build pipeline could break or produce incorrect machine code due to a bug or lack of understanding of the verification system. In contrast, our approach produces a verification pipeline that is parallel to the build pipeline, and a failure in this pipeline does not have any impact on the main build pipeline.

Many cryptographic applications do not have any legacy concerns and never plan on maintaining or improving cryptographic code by hand. In those cases, EverCrypt, Jasmin, and Fiat all produce trustworthy, high-performance implementations that might prove easier to use and understand than what is provided by OpenSSL and its variants. Long-term support might be a concern, given these are research tools. However the slow-moving nature of cryptographic code makes it less likely that the implementations would need modification in the future.

#### **8 Conclusion and Future Work**

The purpose of formal verification is to allow users to be confident in the software on which they depend. This is the reason that AWS-LC, BoringSSL, and OpenSSL are excellent targets for formal verification. Nearly everyone who uses the internet relies on this code for security, either through end-user software, or through a cloud provider's infrastructure. Our proofs show for the first time that this kind of highly optimised, hand-written code matches its mathematical specification. More importantly, we show that such code can be verified for a reasonable amount of proof engineering effort.

We do not consider our proofs the last word on this code—there are several ways in which our work can be improved. Most importantly, we have not yet verified the OpenSSL version of AES-256-GCM and SHA-384. Based on inspection of the code, we believe the proofs would only need small changes to the term rewrites, but this is currently not a high priority in comparison to further AWS-LC assurance work.

There are also several ways we could improve the proofs themselves. We have verified this code at fixed input sizes. We believe we have covered all edge cases, so the probability that bugs remain is low, but a size-agnostic proof would be more complete. Our proofs also rely on term rewriting tactics to close the gap between implementation and specification. These rewrites are specialized to our application and are therefore the most fragile part of the proof. We believe that, with further research, automated solvers could solve many of these logical queries without the need for manual tactics (this would also make our proofs less fragile against code change). Finally, our proofs say nothing about nonfunctional security properties, such as timing or architectural side channels, nor do they connect to cryptographic security proofs.

We are at an exciting moment for cryptographic verification. It is now possible to deploy verified cryptography without compromising on performance. We are tantalisingly close to a world where most cryptographic traffic originates from verified code, and where new cryptographic primitives are verified as a matter of course. For our part, we consider AES-256-GCM and SHA-384 a stepping stone to the real prize: a fully verified library of production-grade cryptographic primitives. Stay tuned!

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Not All Bugs Are Created Equal, But Robust Reachability Can Tell the Difference**

Guillaume Girol1(B), Benjamin Farinier<sup>2</sup>, and S´ebastien Bardin<sup>1</sup>

<sup>1</sup> Universit´e Paris-Saclay, CEA, List, Gif-sur-Yvette, France {guillaume.girol,sebastien.bardin}@cea.fr <sup>2</sup> TU Wien, Vienna, Austria benjamin.farinier@tuwien.ac.at

**Abstract.** This paper introduces a new property called *robust reachability* which refines the standard notion of reachability in order to take replicability into account. A bug is robustly reachable if a *controlled input* can make it so the bug is reached whatever the value of *uncontrolled input*. Robust reachability is better suited than standard reachability in many realistic situations related to security (e.g., criticality assessment or bug prioritization) or software engineering (e.g., replicable test suites and flakiness). We propose a formal treatment of the concept, and we revisit existing symbolic bug finding methods through this new lens. Remarkably, robust reachability allows differentiating bounded model checking from symbolic execution while they have the same deductive power in the standard case. Finally, we propose the first symbolic verifier dedicated to robust reachability: we use it for criticality assessment of 4 existing vulnerabilities, and compare it with standard symbolic execution.

#### **1 Introduction**

**Context.** Many problems in software verification are encoded as *reachability* queries of some undesired condition—a bug, the exploitation of a vulnerability, *etc.* When a verification engine establishes that a certain buggy location in the program is reachable, an input triggering the bug is reported to the developer so that it can be fixed. In the case of techniques based on an under-approximation of program behaviors, like Symbolic Execution (SE) [9] or Bounded Model Checking (BMC) [13], we even have *in principle* the guarantee that the reported issue is real (*correctness*): there are no false positives.

**Problem.** Yet, things are more subtle in practice, as some bugs can be triggered reliably whereas others only happen in very specific and highly improbable initial

This work has been partially supported by ANR (grant ANR-20-CE25-0009-TAVA) and ERC (grant agreement 771527-BROWSEC).

conditions. While standard reachability cannot tell the difference, this distinction is crucial in many real-life scenarios related to security (bug triage, bug prioritization, criticality assessment) or software engineering (test suite replicability and the problem of flaky tests [42]). For example, fuzzers are able to detect so many bugs [38] that they can lead to "bug triage issues" [30]. If each *replicable* (reliably-triggered) bug is hidden by dozens of more *fragile* ones in the reports of a verification engine, it is hard to focus development effort efficiently. Also, if one is only interested in vulnerability reports, bugs which cannot be reliably triggered may even be dismissed as "not exploitable" altogether.

**Goal and Challenges.** *Our goal is to develop a formal framework able to distinguish replicable bugs from fragile bugs, and amenable to automatic software verification—precisely, we want to be able in practice to find such replicable bugs.* This is challenging as we need to avoid any quantitative [37] or probabilistic reasoning [2,34], insofar as they would hinder automation on real examples—these techniques are often either restricted to finite-state systems [2,34] or rely on highly expensive model counting solvers [11,39].

**Proposal.** Our approach consists in partitioning inputs of the program into *controlled inputs* and *uncontrolled inputs*. This lets us refine the concept of reachability into *robust reachability*: a (buggy) location of a program is robustly reachable if there exist controlled inputs, such that for all uncontrolled inputs, this location is reached. In other words, with adequate input we do not need luck.

We typically focus on *security* scenarios where an *attacker* provides controlled input in one go, without knowledge of uncontrolled input – typically sending a malicious crafted file to obtain remote code execution or privilege escalation. We *deliberately* exclude interactive attack scenarios and weaker interpretations like "bugs replicable *most of the time*" in order to keep proof methods *tractable*.

Proving robust reachability is harder than standard reachability. While we show that robust reachability is expressible in formalisms like branching temporal logics [14], hyperproperties [16] or hyper temporal logic [15], there exist no efficient automated analysis methods for these formalisms at the software level (for Turing-complete languages). Therefore, we investigate dedicated verification techniques, revisiting standard methods (SE, BMC) for standard reachability as well as some of their standard companion optimizations.

Our prototype of Robust Symbolic Execution (RSE) relies on the ability of state of the art Satisfiability Modulo Theory (SMT) solvers [4] to generate models for *universally* quantified formulas [25,27,44], which comes with a performance and completeness cost—yet we report promising results.

**Contributions.** We claim the following contributions.

– We formally introduce the concept of robust reachability (Sect. 4) and motivate its use (Sect. 2), giving practical examples where standard reachability leads to false positives in practice (whatever the underlying verification technology). We also characterize robust reachability in terms of temporal logic and hyperproperties, and compare it with *non-interference* (Sect. 4);


We believe robust reachability is an important sweet spot in terms of expressiveness and tractability, allowing to highlight serious bugs in practical situations. We hope this first step will pave the way to more refinements and applications of robust reachability.

#### **2 Motivation**

In this section we show why standard reachability is not always a good fit for bug finding, as it cannot distinguish between *replicable* bugs and *fragile* bugs.

**Fig. 1.** Simple stack buffer overflow

**Stack Canaries.** Consider the program presented in Fig. 1. It suffers from a stack buffer overflow: if variable n is greater than 8 (the size of buffer), then 0x61 will be written to stack memory above buffer. For high enough n, this will overwrite the return address (Fig. 1b, line 3) of function victim and make the program jump to an unexpected program location when victim returns.

<sup>1</sup> The tool, benchmark and data are available at https://github.com/binsec/cav2021 artifacts and https://zenodo.org/record/4721753.

Mitigations for such programming errors exist, like Stack Smashing Protection (SSP) [18]. This technique consists in pushing a randomly-chosen constant value called a *canary* at the top of the stack in the prologue of each function, and checking that this value is intact before returning. If the canary has been tampered with, the program exits to prevent exploitation (Fig. 1b, line 11). Here, SSP prevents the attacker from overwriting the return address of victim, as doing so also overwrites the canary with 0x61616161. This will be detected at line 10 of Fig. 1b with probability 1−2−<sup>32</sup> on a 32-bit architecture: the only way to pass through it is to have the canary value equal to 0x61616161. *Hence, the buffer overflow in this program is not exploitable anymore.*

**Table 1.** Standard reachability is not a good criterion to measure the protection of SSP on the program of Fig. 1.


**The Problem with Standard Reachability.** Can the attacker hijack the control flow without triggering SSP? We can model this security question as a *standard reachability query* over inputs controlled input and global random value. The attacker succeeds if line 12 is reachable with the additional condition that the return address of victim is overwritten with an unexpected address.

Unfortunately, this standard reachability query is satisfiable with the canary global random value equal to 0x61616161 and controlled input equal to *e.g.*, 42. And indeed, binary-level SE tools Angr [46] or Binsec [23] do report the bug as reachable (cf. Table 1). Yet, this answer is unsatisfying as this only happens with a very low probability: it may not be considered a plausible attack. *Hence, it turns out that SE can yield false positives in practice—especially in a security context.*

**Proposal: Robust Reachability.** We label controlled input as a *controlled input* and global random value as an *uncontrolled input*. There exists no value of controlled input such that victim returns to an address tampered with independently of the value of global random value. We thus say that our exploitation condition (line 12) is not *robustly reachable*. We can automatically verify this intuition. We adapted the SE engine of Binsec to robust reachability: our tool finds the vulnerability when we disable the protection (by labelling the canary as *controlled input*) and does not find it anymore when the protection is present. This shows that robust reachability can model the protection provided by SSP, while standard reachability cannot.

This phenomenon is not restricted to stack protectors. We identify in Table 2 several situations where standard reachability may lead to false positives, unlike robust reachability. Note that some cases (randomisation based protections, uninitialized reads) concern binary-level issues, and cannot be observed from a source-level analysis.

**Discussion.** Consider the slightly different problem of reaching line 11 in Fig. 1b. It is reachable for all values of the canary *except* 0x61616161, hence it is not considered robustly reachable – *all* values of uncontrolled input should lead to line 11. This restriction is *deliberate*. A more quantitative approach would hinder automation. For similar reasons, we limit ourselves to non-interactive scenarios, where the attacker input is chosen before uncontrolled input are known. We will further motivate these choices in Sects. 4.1 and 6.4.

Despite these deliberate restrictions, our case studies (Sect. 6.2) show the versatility of robust reachability. In the example above, we distinguish inputs controlled by an attacker (a bad guy) from inputs which he cannot influence (see also *e.g.*libvncserver in Sect. 6.2). But with doas (Sect. 6.2), we distinguish inputs controlled by the system administrator (the good guy) from those which vary on each execution. Other situations are possible, for instance deterministic inputs versus non-deterministic ones like in the case of flaky tests [42]—where there are neither good nor bad guys. Robust reachability can help in all these situations either assessing the "quality" of a given trigger or test suite (criticality, replicability), generating "good" triggers or test suites, or proving their absence.


**Table 2.** Program constructs for which standard reachability yields fragile input

#### **3 Background**

Consider a program P and S the set of its possible states. Each state s ∈ S is labeled by a program location λ(s) ∈ L. Execution of the program is represented by a (one-step) successor relation →∈ S × S; its transitive reflexive closure is denoted by <sup>→</sup>-. For a finite trace <sup>t</sup> ∈ S and s, s ∈ S two states, we write <sup>s</sup> <sup>→</sup>- <sup>t</sup> s if t starts with s, ends with s and follows →. The initial state s0(y) depends on the program input y. For a location ∈ L and input y we write y if <sup>s</sup>0(y) <sup>→</sup> <sup>s</sup> where <sup>λ</sup>(s) = . Additionally, for a trace <sup>t</sup> ∈ S-, we write y <sup>t</sup> if <sup>s</sup>0(y) <sup>→</sup>- <sup>t</sup> s where λ(s) = . We use *trace* for successions of states and *path* for successions of locations. By abuse of notation, the path corresponding to a trace <sup>t</sup> ∈ S is <sup>λ</sup>(t) ∈ L-. For a path π, we denote its length |π| and we write y π if <sup>∃</sup><sup>t</sup> <sup>∈</sup> <sup>S</sup>-. λ(t) = π ∧ y <sup>t</sup> where is the final location of π.

**Definition 1 (standard reachability).** *Given a program* P*, a location* ∈ L *is reachable if* ∃y. y *.*

It is often useful to consider the case of reaching a location with a state s satisfying some predicate φ. This can be reduced to standard reachability by adding if (φ) */\*new target\*/* at the target location.

**Definition 2 (correctness, completeness).** *Let* V : (P,l) → {**1**, **0**} *be a verifier taking as input a program* P *and a location :*


In general, verifying reachability is undecidable, so verifiers cannot be both correct and complete. Correct verifiers can still be k-complete as k-completeness can be thought of as completeness for finite-path systems.


**Fig. 2.** Reachability of with SE and BMC

**Symbolic Execution (SE) and Bounded Model Checking (BMC).** SE [9] incrementally explores all paths in the program (up to, say, a bound k) and when an explored path reaches the target location , checks that this path is indeed executable. This is performed by converting a path π to an SMT formula pcπ, called *path constraint*, which has input y as its only free variable and is equivalent to y π, i.e., a path is executable if and only if its path constraint is satisfiable. Conversely, BMC [13] considers the program as a whole and builds a SMT formula expressing that one of the paths of length at most k leads to . It is equivalent to the disjunction of the path constraints of these paths. The target is reachable in k steps at most if and only if this formula is satisfiable.

These algorithms are detailed in Fig. 2, where GetPredicate turns a path into its path constraint and GetPaths(k) yields all paths below size bound k.

**Proposition 1.** *SE and BMC have the same expressive power: both are correct and* k*-complete.*

Interestingly, we show in Sect. 5 this is not true anymore with robust reachability.

**Solvers.** SE and BMC commonly discharge their satisfiability queries to SMT solvers [4] which take formulas as input, and output whether they are satisfiable (along with a model) or not. Typical queries are expressed in the quantifier-free fragments of well known theories (linear integer arithmetic, bitvectors, arrays, *etc.*) where SMT solvers perform well in practice. In case of an undecidable theory, we can use incomplete solvers (possibly answering unknown), at the price of k-completeness.

### **4 Robust Reachability**

#### **4.1 Definition**

We introduce the new notion of *robust reachability*. We partition the input y into the *controlled input* a and the *uncontrolled input* x—we denote y - (a, x). Let A and X be the sets of possible controlled and uncontrolled inputs respectively. A location is *robustly reachable* when the attacker can choose controlled input a ∈ A without having to rely on specific values of the uncontrolled input x ∈ X to reach his target. Input a is then called a *robust trigger*—otherwise it is a *fragile trigger*.

**Definition 3 (Robust reachability).** *A location* ∈ L *is robustly reachable if* ∃a. ∀x.(a, x) *. This definition depends on the partition of inputs.*

**Proposition 2.** *Robust reachability implies standard reachability. The converse implication does not hold.*

**Discussion.** As already mentioned at the end of Sect. 2, our definition of robust reachability specifically targets a threat model where the attacker speaks first, unaware of uncontrolled inputs. It deliberately excludes interactive systems where the attacker can choose some input, then receive some program output possibly leaking uncontrolled input, and then choose some more input *depending on what was received*. Modeling such situations requires additional quantifier alternations, which deeply impact the performance of proof methods and cripple automation, as shown in Sect. 6.4.

Likewise, a bug triggered for all uncontrolled inputs but one is not robustly reachable according to Definition 3. A quantitative definition of robust reachability could take into account the *proportion of uncontrolled inputs* triggering a bug. This hints at works about model counting [11,39], but the problem at hand is actually harder. Consider the following alternative definition: *(i)* find amax ∈ A such that a maximal proportion of uncontrolled inputs x lead to : (amax, x) ; *(ii)* measure how robustly can be reached by computing the proportion of uncontrolled inputs x such that (amax, x) . Current model counting algorithms can only tackle problem *(ii)* along one path, and we argue in Sect. 6.4 that even *(ii)* alone is considerably more expensive than our SMT-based approach.

*In other words, Definition* 3 *is a tradeoff to keep robust reachability amenable to automated verification. This does not prevent it from meeting its main goal: drawing the attention on more serious bugs. Some may of course be missed, but, as our case studies will show (Sect.* 6*), a good number will be found.*

In the rest of this section, we review a few related properties and see how much they overlap with, but do not remove the need of, robust reachability.

#### **4.2 Relation with Non-interference**

We partition inputs and outputs of a system into either *high* (highly classified) or *low* (public, e.g. observable). A system satisfies *non-interference* [31] when low outputs do not depend on high inputs, implying that secrets cannot leak. Robust reachability can be reformulated in a very non-interference-sounding phrasing: uncontrolled inputs (call them high) must not interfere with the attacker reaching the target location (the low output). Let us clarify this link.

Formally, let high input be uncontrolled input x, and low input be controlled input a. Let low output be whether control flow reached location . Non interference of the resulting system means that ∀a, x, x . ((a, x) ⇐⇒ (a, x ) ).

**Proposition 3.** *If is (standardly) reachable and the system satisfies noninterference with the high/low partition described above, then is robustly reachable. The converse is false.*

Robust reachability requires a single value of the controlled input a for which reachability of is guaranteed but says nothing for other values of a, whereas non-interference constrains the system to behave much more independently of uncontrolled input than robust reachability but says nothing of reachability.

#### **4.3 Interpretation in Terms of Hyperproperty**

Robust reachability and its negation are not trace properties: the observation of a single trace is never enough to prove or disprove them. For example, observing a single trace reaching target with input (a, x) is both compatible with being robustly reachable (if all other inputs (a, x ), x ∈ X also reach ), and with not being robustly reachable (if some other x is such that (a, x ) does not reach ). Robust reachability and its negation thus belong to the more general class of *hyperproperties* [16], i.e. statements relating several traces.

More specifically, Clarkson et al. [16] show that any hyperproperty is the intersection of a hypersafety hyperproperty (*i.e.*something bad cannot happen) and a hyperliveness hyperproperty (something good will eventually happen). Hypersafety is generally thought as easier to prove, notably with selfcomposition [6]. Unfortunately, robust reachability and its negation are pure hyperliveness in the general case: no finite set of finite traces can falsify them. However, in some conditions, they degenerate partly into hypersafety:

**Proposition 4.** *If the domain* X *of uncontrolled inputs is finite, then the negation of robust reachability is not pure hyperliveness (* i.e.*, it has a non-trivial hypersafety component).*

*Proof.* Robust reachability of can be proved by finding controlled input a ∈ A such that for all uncontrolled input x ∈ X one observes a trace starting with input (a, x) and reaching . When X is finite, this means that a finite observation can disprove non-(robust reachability). This is the definition of hypersafety.

This idea—trying to observe a hopefully small set of traces which together prove robust reachability—is crucial for algorithms and leads to our use of path merging in Sect. 5.3.

#### **4.4 Interpretation in Terms of Temporal Logic**

**Computational Tree Logic (CTL).** CTL [14] is a temporal logic over the tree of possible traces. Let L be a labeling which maps states to the set of (atomic) predicates they satisfy. If is a predicate, the CTL formula is satisfied by all systems whose initial state s<sup>0</sup> verifies ∈ L(s0). If φ is a CTL formula and s a state, then **EX**φ expresses that φ holds in at least one (direct) successor of s, and **AF**φ that all traces arising from s eventually reach a state from which φ holds. CTL introduces other operators, not needed here.

**Proposition 5.** *It is possible to express robust reachability with CTL.*

*Proof.* Let S - S∪A∪{si} where s<sup>i</sup> is a new state, let → -→ ∪{(si, a) | a ∈ A} ∪ {(a, s0(a, x)) | a ∈ A, x ∈ X}, and let L (s) be equal to L(s) if s ∈ S and ∅ otherwise. Then is robustly reachable if, and only if **EXAF** is true in the new extended system (S ,→ , L ) with s<sup>i</sup> as initial state.

**HyperLTL.** It is also possible to express robust reachability in the temporal logic HyperLTL [15], which allows to reason over sets of traces π, assuming we have an atomic predicate ≡<sup>v</sup> stating that the first states of two traces have the same value for variable v. Robust reachability of can then be expressed as ∃π. ∀π . **F**<sup>π</sup> ∧(π ≡<sup>a</sup> π → **F**<sup>π</sup>- ), where **F**<sup>π</sup> denotes that trace π goes through . In other words, there exists a trace π reaching s.t. all traces sharing the same controlled input also reach .

#### **4.5 Robust Reachability and Automatic Verification**

The previous classification does not help us find an efficient *software verification method* for robust reachability. Indeed, while efficient CTL model checkers exists for the finite case [12] or very specific formalisms such as pushdown systems [47], most efforts in (general) software verification have been directed towards the verification of safety temporal formulas or simple termination [17] (formulas of the form **AF**ϕ). Moreover, temporal logics like HyperLTL [15] suffer the same limitations, and checking for both reachability and non-interference is probably too strong a requirement in practice. Finally, one can prove the *absence* of robust reachability by proving the absence of standard reachability. It is thus possible to use existing algorithms for unreachability, based *e.g.*on invariant computation, at the price of even larger over-approximation than when they are used for their original purpose. This kind of approach is not our focus. In this paper we look for *correct verifiers* able to prove robust reachability (and report robust triggers) rather than to disprove it.

#### **5 Automatically Proving Robust Reachability**

We now extend SE and BMC to the robust case.

#### **5.1 Robust Bounded Model Checking**

As mentioned in Sect. 3, BMC determines the reachability of a location by building a family of SMT formulas <sup>ϕ</sup>k(a, x) equivalent to <sup>∃</sup><sup>t</sup> ∈ S-. |t| ≤ k ∧ (a, x) <sup>t</sup> . ϕ<sup>k</sup> expresses that is reachable in less that k steps. Then one proves that is reachable if and only if ∃k. ∃a. ∃x. ϕk(a, x). This extends to robust reachability:

**Proposition 6.** *If the domain of uncontrolled input* X *is finite or the system has finitely many paths, then is robustly reachable if and only if* ∃k. ∃a. ∀x. ϕk(a, x)*.*

*Proof.* (⇐= ) comes directly from the definition of ϕk.(=⇒ ). If is robustly reachable, let a<sup>0</sup> be a robust trigger. The set of paths P arising from inputs in {a0}×X is finite (bounded either by X or the number of paths in the system), and ∀x. - <sup>π</sup>∈<sup>P</sup> pcπ(a0, x) holds. Let <sup>k</sup> = 1 + max<sup>π</sup>∈<sup>P</sup> <sup>|</sup>π|. All paths in <sup>P</sup> are unrolled in ϕ<sup>k</sup> so - <sup>π</sup>∈<sup>P</sup> pcπ(a0, x) =<sup>⇒</sup> <sup>ϕ</sup>k(a0, x) and thus <sup>∀</sup>x. ϕk(a0, x).

As a result, it is enough to replace the condition "∃y. φ is satisfiable" by "∃a. ∀x. φ is satisfiable" in Fig. 2b.

**Corollary 1.** *The resulting algorithm, robust BMC, is correct w.r.t.robust reachability. If the domain of uncontrolled input* X *is finite or the system has finitely many paths, then robust BMC is also* k*-complete.*

The finiteness hypothesis is required: if a program reaches a location after having executed a loop an unbounded, uncontrolled number of times, then robust BMC has to unroll an unbounded number of paths to prove robust reachability.

#### **5.2 Robust Symbolic Execution**

Similarly to BMC, we check that a path π robustly reaches the target by checking the satisfiability of ∃a. ∀x. pcπ(a, x), instead of ∃a. ∃x. pcπ(a, x). This means replacing "∃y. φ is satisfiable" by "∃a. ∀x. φ is satisfiable" in Fig. 2a. Unfortunately the resulting algorithm, robust SE, is not exactly what we want, as it proves a stronger property.

**Definition 4 (Single-path robust reachability).** *A location* ∈ L *is singlepath robustly reachable if* <sup>∃</sup><sup>π</sup> ∈ L-. <sup>∃</sup>a. <sup>∀</sup>x. <sup>∃</sup><sup>t</sup> ∈ S-. λ(t) = π∧(a, x) <sup>t</sup> *. In other words, the path used to reach is the same regardless of the uncontrolled input.*

**Proposition 7.** *Single-path robust reachability implies robust reachability. The converse implication does not hold.*

**Proposition 8.** *Robust SE is correct and* k*-complete w.r.t.single-path robust reachability.*

*Proof.* By construction, pcπ(a, x) is equivalent to (a, x) π so ∃π. ∃a. ∀x. pcπ(a, x) is equivalent to single-path robust reachability of the last location of π.

**Corollary 2.** *Robust SE is correct but incomplete for robust reachability.*

*Interestingly, the expressive powers of SE and BMC, which are the same for standard reachability, diverge when extended to robust reachability.*

#### **5.3 Path Merging**

Path merging [33] (a.k.a. state joining) consists in identifying "close" paths leading to the same location and replacing them by a *merged* path (summary). With original path constraints pc<sup>π</sup><sup>1</sup> and pc<sup>π</sup><sup>2</sup> , the merged path constraint is pc<sup>π</sup><sup>1</sup> ∨pc<sup>π</sup><sup>2</sup> . This is only an optimization in the standard setting, with no impact on k-completeness. The situation is different in the robust setting.

```
Data: bound k, target 
1 φ := ⊥
2 for path π in GetPaths (k) do
3 if π goes through  then
4 φ := φ ∨ GetPredicate(π)
5 if ∃a. ∀x. φ is satisfiable then
6 return true
7 end
8 return false
```
**Algorithm 1:** RSE+: Robust SE with systematic path merging

```
1 void main(a, x) {
2 if (x) x++; // π1
3 else x--; // π2
4
5 if (!a) bug();
6 }
```
**Fig. 3.** An example where path merging is required

Consider the program in Fig. 3: the bug is robustly reachable with controlled input a = 0, but the control flow takes one of two paths π<sup>1</sup> and π<sup>2</sup> depending on the value x of uncontrolled input. This bug will not be found by robust SE as defined previously, as neither π<sup>1</sup> nor π<sup>2</sup> fulfills the satisfiability criterion ∃a. ∀x. pcπ*<sup>i</sup>* (a, x). However, if π<sup>1</sup> and π<sup>2</sup> are merged, then the bug is found because ∃a. ∀x. pcπ<sup>1</sup> (a, x)∨pcπ<sup>2</sup> (a, x) is satisfiable. This leads us to robust SE with systematic path merging (RSE+, Algorithm 1), better fit to robust reachability.

**Proposition 9.** *Robust SE with systematic path merging (RSE+) is correct for robust reachability. If the domain of uncontrolled input* X *is finite or the system has finitely many paths, then it is also* k*-complete.*

*Proof.* For k-completeness: If is robustly reachable, let a<sup>0</sup> be a robust trigger. The set of paths P arising from inputs in {a0}×X is finite (bounded either by X or the number of paths in the system). Let k = 1 + max<sup>π</sup>∈<sup>P</sup> |π|. For bound k, when GetPaths has output all paths in P, - <sup>π</sup>∈<sup>P</sup> pc<sup>π</sup> <sup>=</sup><sup>⇒</sup> <sup>φ</sup> so <sup>∃</sup>a. <sup>∀</sup>x. φ is satisfiable.

In conclusion, *path merging improves the completeness of robust SE*. This is surprising because path merging is merely optional in standard SE.

#### **5.4 Revisiting Standard Optimizations and Constructs**

Some optimizations commonly used in SE are not correct nor complete anymore in a robust setting. We show here how to adapt them.


```
uncontrolled int x ;
if (x<10) { /* a */ }
else { /* b */ }
/* c */
if (x>20) {
 /* d */
 if (x>30) { /* e */ }
 else { /* f */ }
}
```
**Algorithm 2:** Implementation of GetPaths with path pruning

```
Fig. 4. Failure case for universal path
pruning
```
**Incremental Path Pruning** [3,48]**.** When a path has an unsatisfiable path constraint, all its descendent paths are also infeasible. For example, the path acd in Fig. 4 has path constraint x < 10 ∧ x > 20, which is unsatisfiable. One can prune this path, *i.e.*stop exploring it and its children acde and acdf.

```
Data: entrypoint 0, bound k
P := {0}
while P = ∅ do
   Take a path π out of P
   if |π| > k then continue
   if ∃a. ∀x. pcπ unsat then
       /* Skip MaybeMerge to
          disable path
          merging */
       P := MaybeMerge(π, P)
       continue
   end
   yield π
   P :=
    P ∪ {children paths of π}
end
```
**1 Function** MaybeMerge(π, P) **2** Choose u a transitive child of the last location of π *(ideally, a strict postdominator of the second to last location of* π*)* **3** Let π the longest strict prefix of π. **4** Let U the set of paths from π to u **<sup>5</sup> if** ∃a. ∀x. - π--<sup>∈</sup><sup>U</sup> π- is SAT **then 6** Merge paths in U and add the result to P **7 end 8 return** P

**Algorithm 4:** Incomplete path merging for universal path pruning

**Algorithm 3:** GetPaths with universal path pruning

In Fig. 2a this would be an optimization of GetPaths: as shown in Algorithm 2, one checks that the path constraint of currently explored paths are satisfiable, and if not, the paths at fault are *pruned*, and their children paths are not explored. As a result, we now issue satisfiability queries in two occasions: during GetPaths to *prune* paths (Algorithm 2, line 5), and when *validating* a candidate reaching path (Fig. 2a, line 5). Pruning queries and validation queries must be treated differently.

Robust SE is obtained from SE by adding a universal quantifier to *validation queries* but not *pruning queries*. The path constraint for path a in Fig. 4 is pc<sup>a</sup> = x < 10 but ∃a. ∀x. pc<sup>a</sup> is false. Same applies for b. If we added a universal quantifier to pruning queries—which we call *universal path pruning*, see Algorithm 3—we would prune a and b, and incorrectly conclude that c is not robustly reachable. In other words, Symbolic Execution with universal path pruning (denoted RSE∀) is correct but not complete.

Universal path pruning, however, conveys an interesting intuition: the full if branch below acd in Fig. 4 is not robustly reachable, because ∀x. x > 20 is false. With normal path pruning and RSE+, we would needlessly explore these paths. To take advantage of this, we keep RSE<sup>∀</sup> but improve its completeness with path merging, as depicted in Algorithm 4.

The main idea is that when a set of paths are to be pruned, they may pass the universal pruning test ∃a. ∀x. pc when merged together. One way to find such sets of paths is the use the Control Flow Graph (CFG) of the program. For example when trying to prune π = a in Fig. 4, we know by invariant of the set P of paths to be explored that π = the empty path passes the universal test. We compute the strict postdominator u = c of π : when the paths from π to c join again, they pass the pruning test again. We then replace π by this merged path in the set P of paths to be explored.

Note that computing a postdominator is not required for correction. In our implementation, we cannot compute the exact CFG at the binary level so the chosen u may be wrong. In line 5 of Algorithm 4 we check that we picked correctly, and otherwise, merging failed and we prune π. Despite the heuristic approach, the technique proves useful, as we will see in Sect. 6.

We denote Robust SE with universal path pruning and path merging as RSE∀+. It is correct and less incomplete than RSE∀.

**Assumptions.** It is common to model complex parts of the system by introducing their result as a symbolic input z and then *assume* that z satisfies the required properties. For example, Address Space Layout Randomisation (ASLR) for the stack pointer could be modeled by adding an *assumption* that esp ∈ [m, M] where m and M are in-

**Fig. 5.** Unsound assumption, in pseudo-C.

lined constant values. In standard SE this would be translated to an *assertion* esp<sup>0</sup> ∈ [m, M] conjoined to the path constraint pcπ, where esp<sup>0</sup> is the initial value of esp. *Actually, in standard SE and BMC, assertions and assumptions are dealt with identically.*

In a robust setting, to the contrary, adding an *assumption* ψ to a path constraint yields ψ =⇒ pcπ, while adding an *assertion* φ yields pc<sup>π</sup> ∧ φ. Additionally, assumptions which mix controlled and uncontrolled inputs can make the algorithms above unsound without adaptation: in Fig. 5, reachability of bug maps to the SMT query ∃a. ∀x. x < a =⇒ ⊥. It is satisfiable, with a = 0, which makes the premise false. However, this does not correspond to an executable path. Actually, formalizing robust reachability assuming ψ(a, x) naively by ∃a. ∀x.(ψ(a, x) =⇒ a, x ) does not imply standard reachability anymore. A slight adaptation is needed:

**Definition 5 (Robust reachability under assumption).** *A location is robustly reachable under the assumption of* ψ *when*

$$(\exists a. \left( (\exists x. \,\psi(a,x)) \land (\forall x. \left( \psi(a,x) \implies (a,x) \vdash \ell) \right) \right))$$

This definition preserves the implication from robust to standard reachability. The algorithms we presented are easily adapted to take it into account.

*Interestingly, in the robust case, SE and BMC cannot handle assertions and assumptions in the same way anymore.*

**Concretisation and Other Optimizations.** When path constraints along a path become too complex, some variables can be *concretized*: their symbolic value can be replaced by a concrete one [21,29,45]. Formally, concretizing a variable u to value 42 corresponds to adding an *assertion* u = 42. This sacrifices k-completeness for tractability. Actually, any additional constraint can be added, and several common optimizations (e.g., domain shrinking, path filtering) can be seen through this lens. These optimizations must be taken with care in the robust setting. First, considering them as assumptions instead of assertions would be incorrect. Second, if the value of the concretized variable ultimately depends semantically on uncontrolled input, the path does not pass universal validation anymore: for example, when concretizing x to 42, ∃a. ∀x. pc(a, x) ∧ x = 42 is unsatisfiable because ∀x. x = 42 is false. As a result, locations visited further on this path become robustly unreachable. *In other words, concretisation only works on controlled or constant values.*

#### **5.5 About Constraint Solving**

Adaptations to robust reachability require solvers to deal with one alternation of quantifiers. Most theories become undecidable with quantifiers. Dedicated algorithms exist for a few decidable quantified theories, *e.g.*the array property fragment [7] or Presburger arithmetic [8]. For other theories, generic methods like E-matching [40] and MBQI [27] have proven rather efficient, although not complete. Sound approximations [25] also have been proposed to reduce quantified formulas to quantifier-free ones. In our experiments, the newly introduced quantifier associates to an increase in the frequency of time-outs and memoryouts, as seen in Sect. 6.3 and specifically Table 4.

#### **6 Proof-of-Concept of a Robust Symbolic Execution Engine**

#### **6.1 Implementation**

We propose Binsec/RSE, the *first* symbolic execution engine dedicated to robust reachability. We base our proof-of-concept on Binsec [23], a binary executable formal analysis engine written in OCaml and already used in several significant case studies [19,20,43]. For the sake of experimental evaluation (Sect. 6.3) we actually implement five variants of robust reachability: **RSE** (basic approach in Sect. 5.2 with existential path pruning Sect. 5.4), **RSE+** (the same plus systematic path merging, Sect. 5.3), **RSE**<sup>∀</sup> (RSE with universal path pruning, Algorithm 3), **RSE**∀**+** (same, with path merging during path pruning, Algorithm 4), and **RBMC** (Sect. 5.1). Binsec/RSE emits quantified formulas in the theory of bitvectors and arrays (arrays are used to model memory) which are then solved by the quantified solver Z3 [22]. We reuse the recent ROW simplification [26] to reduces the number of array indexations. The source code of Binsec/RSE, the test suite and the case studies of this section are available for reproduction at https://github.com/binsec/cav2021-artifacts and https:// zenodo.org/record/4721753.

#### **6.2 Case Studies: Exploitability Assessment for Vulnerabilities**

We show here how Binsec/RSE (unless otherwise specified, the RSE+ variant) can help in vulnerability assessment. Especially, we demonstrate that robust reachability allows deeper insights into a bug than standard reachability, by replaying 4 existing vulnerabilities.

**CVE-2019-15900 in doas.** doas is a utility granting higher privileges to users specified in a configuration file. User IDs are sometimes parsed incorrectly and left uninitialized. We look for a *vulnerable configuration file* denying root access to the attacker such that the (flawed) executable reliably grants root access to the attacker. For simplicity we assume that the system has no named users and groups and the configuration file has two lines.

Binsec/RSE with standard reachability reports that root access is granted with a configuration file containing permit :("@@@@@ when the initial memory address 0xffefffff contains the group ID of the attacker and the stack starts at 0xfff0001f. *This is a typical "false positive in practice": these conditions may vary unpredictably across executions so we cannot conclude regarding the exploitability of the flaw.*

With robust reachability where the configuration file is controlled but the initial state of memory is not, Binsec/RSE reports in less than 10 s that root access is granted reliably to the attacker when the configuration file contains deny :4 and permit b%@)@@(. This is more useful, but b%@)@@( We test therefore if any other given user name is also affected by running the analysis with this user name concretized in the initial state. By this method, we proved that the flaw is also robustly reachable for wwww, a possible typo of a usual user name, as well as all two-letter lowercase user names.

In other words, if the system administrator grants privileges to a non existing user by mistake, he may unknowingly grant them to the attacker instead. *Here, robust reachability provides us with invaluable insight about the severity of a bug where standard reachability fails.*

**CVE-2019-20839 in libvncserver.** An attacker-chosen null-terminated string is copied by an unbounded strcpy into a 108-bytes buffer, leading to a stack buffer overflow. Exploitability is not guaranteed: null bytes cannot be copied, the executable is protected by SSP, *etc.*. Starting from the vulnerable function, we ask whether it is possible to return to the address 0xdeadbeef, chosen arbitrarily.

Binsec/RSE reports that for standard reachability, the bug can be reached when: *(1)* the stack starts at 0xfff00000; *(2)* the initial value of the return address of the function is 0; *(3)* the gs segment starts at 0xf7f00000; *(4)* the stack canary is 0x01010180; *(5)* neither system call in the function fails; *(6)* file descriptor 0 is free; *(7)* the input path has a specific value. *The attacker cannot prepare such a state, so this is another false positive in practice.*

With robust reachability, when only the input buffer is controlled and not the stack canary, Binsec/RSE fails to prove or disprove exploitability in 24 h. However, if we mark the canary as controlled, Binsec/RSE finds an exploit in about 15 min. This suggests the canary brings a real protection against exploitation.

**CVE-2019-14192 in U-boot.** U-boot is an open-source boot-loader, popular for embedded boards. When booting over Network File System (NFS), U-boot does not validate the length field of some network packets. This length is subtracted 16 and used as a size to be copied. If a malicious packet declares a length of less than 16, computation underflows and leads to a buffer overflow.

We encode the situation as follows: the input network packet is controlled, the IP address of the victim is constant, the NFS state machine is initialized to expect the appropriate packet type and all other values are uncontrolled. Binsec/RSE with the RSE∀+ variant (RSE+ times out here) proves in about 2 min that a memory copy of more than 4GB is robustly reachable, which is a strong indication of the criticality of this denial-of-service vulnerability.

**CVE-2019-19307 in Mongoose.** Mongoose is an embedded networking library. When receiving large MQTT packets, the length of the parsed packet can be computed as 0. The parsing loop does not advance and is thus infinite. We look for network packets whose length is parsed as 0 but are accepted as valid. Binsec/RSE proves in less than a second that such situations are robustly reachable when only the network packet is controlled, confirming exploitability.

#### **6.3 Experimental Evaluation**

**Research Questions.** We now seek to investigate in a more systematic way the following research questions:


**Table 3.** The 46 reachability problems selected for our evaluation


**Protocol.** We base our analysis on a set of 46 reachability problems on binary executables from various architectures (i686-windows-pc, i686-linux-gnu and armv7-linux-gnu) presented in Table 3. The average trace length for reachable problem instances is 809 instruction-long, with a maximum of 18k instructions. The problems fall into two categories: real code and synthetic examples (*e.g.*code designed to be analysed). For each executable, Binsec/RSE determines if a certain location is robustly reachable from a certain initial state. If this is the case a model is output by Binsec/RSE, and compared to a ground truth obtained by manual analysis. Tests were run on Intel Xeon E-2176M(12)@4.4 GHz and we use Z3 4.8.7. Results are classified as follows:

**Correct** Binsec/RSE proves the expected result, i.e. it either reports a robust trigger or rightfully proves the absence of such a trigger;

**False positive** a fragile trigger is reported;



**Table 4.** Comparison of standard and robust algorithms over our 46 test cases

**Precision (RQ1).** As expected, robust variants do not report any false positives, and path merging increases completeness. RSE variants with universal path pruning (RSE∀, RSE∀+) are less complete than those with existential path pruning, but they are less prone to timeouts. This is the case of CVE-2019-14192 in U-boot (Sect. 6.2), for example. RBMC suffers from path explosion (time out) much more often than RSE variants. *Overall, Robust SE with path merging and* *existential path pruning is the most promising method among those presented here, with 44/46 correct answers.* RSE∀*+ is less complete but terminates more often.*

Note that two interesting test cases in the "real" category of Table 3 need path merging to prove robust reachability: one where a pointer with uncontrolled alignment is passed to memcpy, and one where a branch depends on the result of IO. These situations are common programming idioms, demonstrating the importance of path merging.

**Gain Associated to Robustness (RQ2).** We compare standard SE with RSE+, the most precise algorithm of RQ1. *Standard reachability has about 30% false positives while robust reachability has none, at the cost of slightly more timeouts.*

There are no false positives in code in the "real" category, except in CVE replays. Our interpretation is that well-functioning programs are designed to behave the same regardless of the uncontrolled environment: concrete memory layout, stack canaries, *etc.*. Robust reachability becomes decisive on buggy code, notably with undefined behavior. This is also illustrated by case studies (Sect. 6.2).

**Path Pruning (RQ3).** We compare RSE∀, which features universal path pruning, to RSE, which features usual path pruning. Comparison is limited to test runs of more than a second which succeed with both methods. This is to prevent comparing a run where Binsec/RSE proves that the target is reachable and stops, to a run where Binsec/RSE does not find the target and explores the whole program. *RSE*<sup>∀</sup> *explores 17% less paths and interprets 21% less instructions than RSE.* This comes at the price of more universally quantified SMT queries: the average time per SMT query goes up by 25%. Overall the run time of both methods is very close.

With path merging, the difference in paths explored disappears: RSE∀+ explores 1% less paths and instructions than RSE+. This is due to the fact that for some tests, path merging "unlocks" some new paths. Overall, RSE∀+ is 6% slower than RSE+ on successful, terminating tests.

**Performance (RQ4).** In this question, we compare the run time of robust algorithms to SE. Comparison is done on the same basis as before, except that we count timeouts. RSE+ is 74% slower than standard SE on geometric average. This is mostly due to newly introduced time-outs (up to 260× slower) since median slowdown is only 15%. RSE<sup>∀</sup> is more consistently slower with about 30% slowdown in both geomean and median. This is mainly explain by increased solver time (universal path pruning queries). RSE∀+ is close in median slowdown, but path merging introduces new timeouts and drives the average slowdown up to 62%. *RSE+ has a low overhead compared to standard SE, except for a few time-outs (2/46).*

#### **6.4 Additional Considerations**

We excluded interactive systems and quantitative approaches from our definition of robustness (Definition 3, Sect. 4.1) to keep automated proof methods tractable. We motivate this choice by experimentally showing that these alternatives yield significant overhead. Technical details are provided in Appendix A.

**Quantitative Reasoning and Model Counting.** We could imagine refining our definition of robust reachability, looking for some controlled input for which the number of uncontrolled inputs allowing to reach the intended target is maximal (or, above a certain threshold). Although we have already observed that model counters do not directly solve this problem (Sect. 4.1), we can lower bound its runtime cost by the cost of determining the number of uncontrolled x satisfying a path constraint for some given controlled input a0. We experimentally measured it with SearchMC [39] and SMTApproxMC [11], two of the few model counters supporting the SMTlib2 format and the QF BV theory. We compare this to our "all-or-nothing" qualitative approach on our 4 CVE case-studies: *the quantitative approach is here several orders of magnitude slower than our qualitative method*—SMTApproxMC always times out while SearchMC is at least 400× slower.

**Interactive Systems and Quantifier Alternations.** We estimate the cost of adding more quantifier alternations in order to deal with interactive systems (Sect. 4.1), by modifying queries on the two of our case studies where interactive input makes sense (libvncserver and doas, *cf.* Sect. 6.2). *RSE+ in this setting does not terminate within 24 h*, highlighting the fact that current SMT solvers have a very hard time generating models for quantified formulas beyond ∃∀. It seems to be a fundamental issue as none of Z3 [22], Boolector [41] and CVC4 [5] is able to prove in less than 1 h that ∀z. ∃a. a XOR 1 = z holds over 32-bit bitvectors.

### **7 Related Work**

Broadly speaking, we are interested in defining a subclass of *comparatively more interesting bugs* amenable to automation. We review related prior attempts.

**Automatic Exploit Generation (AEG).** These approaches seek to demonstrate the *impact* of a bug by automatically generating an exploit from it [1,10,36]. *This is complementary to robustness*, which focuses on replicability. Actually, both techniques could be advantageously combined, as a replicable exploit is clearly more threatening than a fragile one. Current AEG methods being based on symbolic methods, adapting them for robustness looks feasible.

**Quantitative Reasoning and Model Counting.** Several approaches rely on probabilities or counting to distinguish important issues from minor ones—for example (quantitative) *probabilistic model checking* [2,34] or *quantitative information flow analysis* [37]. Robust reachability could be refined in such a way. Yet, current quantitative approaches do not scale on software, as they often rely either on the finite-state hypothesis, or on *model counting* solvers [32], which are only at their beginning (see Sects. 4.1 and 6.4).

**Flakiness.** The opposition between *flaky* tests and *sturdy* tests [42, section 6.3] is close to that between robustly reachable bugs and normally reachable bugs. A test is flaky when it is reachable, but not robustly reachable under the partition of inputs where controlled inputs are deterministic inputs and uncontrolled inputs are non-deterministic inputs. *Flakiness is thus a particular case of (non-) robustness*. Especially, our tool can help find non-flaky tests.

**Fairness.** *Fairness assumptions* in model checking [35] aim at discarding traces considered as unrealistic and avoiding false alarms from the user point of view. While the goal is rather similar to ours, the two techniques are very different: fairness assumptions typically require certain sets of states to be visited infinitely often along a trace, while robust reachability requires that a trace cannot be influenced by uncontrolled input w.r.t.a given reachability property.

**Symbolic Execution and Quantifiers.** Finally, while symbolic execution is commonly performed with quantifier-free constraints, a notable exception is *higher-order test generation* [28], where Godefroid proposes to rely on universally quantified uninterpreted functions (∀∃ queries) in order to soundly approximate opaque code constructs. Higher-order test generation and robust reachability are complementary as they serve two different purposes: robust reachability can only be used in a modest way for opaque code constructs (finding controlled inputs for which their value does not matter), while higher-order test generation is inadequate for robust reachability, as it would be as if the attacker could choose the controlled inputs knowing the uncontrolled ones.

#### **8 Conclusion**

We introduce the novel concept of robust reachability, that we argue is better suited than standard reachability in several important scenarios for both security (e.g., criticality assessment, bug prioritization) and software engineering (e.g., replicable test suites). We formally define and study robust reachability, discuss how standard symbolic methods to prove reachability can be revisited to deal with the robust case, design and implement the first robust symbolic execution engine and demonstrate its abilities in criticality assessment over 4 CVEs. We believe robust reachability is an important sweet spot in terms of expressiveness and tractability. We hope this first step will pave the way to more refinements and applications of robust reachability.

#### **A Details on the Experiments Supporting Sect. 6.4**

We reuse the notations of the discussion in Sect. 4.1.

**Model Counting.** For simplicity, consider single-path robust reachability of along a path with path constraint pc(a, x). It is equivalent to ∃a. ∀x. pc(a, x). A more quantitative approach would be to consider amax s.t.the ratio r(amax) of x satisfying pc(amax, x) is maximal. The larger r(amax), the more robustly reachable . We try to experimentally get an idea of the cost of computing this. Determining amax is an open problem, but we can lower bound the full computation time by the time to compute r(amax) from amax. As the algorithms below are randomized, we can measure the time to compute r(a0) for any a0.

We collect the path constraint of the first path standardly reaching the target in our 4 case studies of Sect. 6.2. We arbitrarily choose a<sup>0</sup> satisfying ∃x. pc(a0, x), and compare the time to (dis)prove ∀x. pc(a0, x) with Z3 to the time to approximate r(a0) with two of the few model counters supporting SMTlib2 input in the QF BV theory: SearchMC [39] (with tolerance ε = 0.8 and confidence 1 − δ = 0.95) and SMTApproxMC [11] (with tolerance ε = 0.8 and 1 iteration). We found no tool supporting arrays, so arrays were blasted. As shown in Table 5, the quantitative approach is orders of magnitude slower in all cases, and especially in the one case where it is indeed significantly more precise than our qualitative approach (u-boot).

**Table 5.** All-or-nothing (Z3) *vs* quantitative (SearchMC, SMTApproxMC) approaches: runtime and lower bound on r(a0). Timeout (TO) is 2,400 s.


**Quantifier Alternations.** We want to model a leak in ASLR in libvncserver (Sect. 6.2): the attacker knows about an address z and wants to use the bug to jump to z. The corresponding property is: for all values<sup>2</sup> of z, there exists an attacker input a such that for all other uncontrolled inputs x, control flow is diverted to z. This uses another universal quantifier, which we exclude in our definition of robust reachability to keep satisfiability queries tractable. We implemented this for libvncserver (additional quantification on the target jump address) and doas (additional quantification on the user and group ID of the attacker, and the typoed user name): RSE+ does not terminate within 24 h.

#### **References**


<sup>2</sup> Without a null byte, but we ignore this detail for the sake of simplicity.


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **A Temporal Logic for Asynchronous Hyperproperties**

Jan Baumeister<sup>1</sup> , Norine Coenen<sup>1</sup> , Borzoo Bonakdarpour<sup>2</sup> , Bernd Finkbeiner<sup>1</sup> , and C´esar S´anchez3(B)

> <sup>1</sup> CISPA Helmholtz Center for Information Security, Saarbr¨ucken, Germany <sup>2</sup> Michigan State University, East Lansing, USA <sup>3</sup> IMDEA Software Institute, Madrid, Spain cesar.sanchez@imdea.org

**Abstract.** *Hyperproperties* are properties of computational systems that require more than one trace to evaluate, e.g., many information-flow security and concurrency requirements. Where a trace property defines a set of traces, a hyperproperty defines a set of sets of traces. The temporal logics HyperLTL and HyperCTL\* have been proposed to express hyperproperties. However, their semantics are *synchronous* in the sense that all traces proceed at the same speed and are evaluated at the same position. This precludes the use of these logics to analyze systems whose traces can proceed at different speeds and allow that different traces take stuttering steps independently. To solve this problem in this paper, we propose an *asynchronous* variant of HyperLTL. On the negative side, we show that the model-checking problem for this variant is undecidable. On the positive side, we identify a decidable fragment which covers a rich set of formulas with practical applications. We also propose two model-checking algorithms that reduce our problem to the HyperLTL model-checking problem in the synchronous semantics.

#### **1 Introduction**

Hyperproperties [8] extend the conventional notion of trace properties [1] from a set of traces to a set of sets of traces. In other words, a hyperproperty stipulates a system property and not the property of just individual traces. Many interesting requirements in computing systems are hyperproperties and cannot be expressed by trace properties. Examples include (1) a wide range of information-flow security policies such as *noninterference* [14] and *observational determinism* [28],

This work was funded in part by Madrid Regional Government under project "S2018/TCS-4339 (BLOQUES-CM)", by Spanish National Project "BOSCO (PGC2018-102210-B-100)", by the German Research Foundation (DFG) as part of the Collaborative Research Center "Foundations of Perspicuous Software Systems" (TRR 248, 389792660), by the European Research Council (ERC) Grant OSARES (No. 683300), and by the United Stated NSF SaTC Award 2100989.

A. Silva and K. R. M. Leino (Eds.) CAV 2021, LNCS 12759, pp. 694–717, 2021. https://doi.org/10.1007/978-3-030-81685-8\_33

**Fig. 1.** Program *<sup>P</sup>*<sup>1</sup> **Fig. 2.** Program *<sup>P</sup>*<sup>2</sup> **Fig. 3.** <sup>K</sup> with a selfloop

(2) sensitivity and robustness requirements in cyber-physical systems [27], and (3) consistency conditions such as *linearizability* in concurrent data structures [5].

HyperLTL [7] is a temporal logic for hyperproperties that enriches LTL with quantifiers allowing explicit and simultaneous quantification over multiple execution traces. For example, the observational determinism security policy [28] stipulates that any two executions that start in two *low-equivalent* states (i.e., states whose value of publicly observable variables are the same), should remain in low-equivalent states. This property can be expressed in HyperLTL as the following formula, called ϕOD,∀π.∀π .(l<sup>π</sup> ↔ l<sup>π</sup>- ) → -(l<sup>π</sup> ↔ l<sup>π</sup>- ). However, the semantics of HyperLTL (and other formal languages for hyperproperties) is *synchronous*, meaning that they completely abstract away the notion of time passage. In HyperLTL, all traces proceed at the same speed, as all temporal operators move the position on all traces simultaneously. Consider the program *P*<sup>1</sup> in Fig. 1, where input values 0 and 1 are possible for *high-secret* variable h. This renders two possible traces shown in Fig. 4a that satisfy ϕOD.

The synchronous semantics of HyperLTL has a shortcoming which has practical implications as well: formulas are not invariant under *stuttering*. Note that, contrary to LTL, disallowing the use of ◯ does not make the formula invariant under stuttering, as traces can still stutter independently. This limits the scope of application of HyperLTL to only those settings where different traces can be perfectly aligned. For example, consider program *P*<sup>2</sup> in Fig. 2, where line <sup>4</sup> in *P*<sup>1</sup> is refined to its intermediate code using a register that stores the value l + 1 and then stores this value in memory location l in lines <sup>4</sup> and 5, respectively. Applying the synchronous semantics of HyperLTL results in declaring a violation of ϕOD in the second position. This, however, is not an accurate interpretation of ϕOD (assuming that an attacker only has access to the memory footprint and not the CPU registers or a timing channel), as the two traces are stutter equivalent with respect to the state of variable l. In fact, the synchronous semantics of HyperLTL may incorrectly identify good programs as bad because it ignores the notion of relative time between traces. This problem is generally amplified in Kripke structures where self-loops correspond to non-deterministic choices that model that the system may remain in a state for some arbitrary time. For instance, consider K in Fig. 3 and HyperLTL formula ∀π.∀π .((b<sup>π</sup> ↔ b<sup>π</sup>- ) U -(a<sup>π</sup> ↔ a<sup>π</sup>- )). Only pairs of traces that take the self-loop the same number of times satisfy this formula. However, since the goal of employing a self-loop is typically to make the duration of staying in a state irrelevant, this semantics is too restrictive.

**Fig. 4.** Synchronous vs. asynchronous semantics for HyperLTL.

Besides HyperLTL, other logics have been proposed that allow trace quantification, for example, H<sup>μ</sup> [15], which extends the linear time μ-calculus [3] with path quantifiers and indexed next operators. For Hμ, the model-checking problem is in general undecidable, but two fragments, the k-synchronous, k-context bounded fragments, have been identified for which model checking remains decidable [15].

In this paper, we propose an asynchronous temporal logic for hyperproperties. Our main motivation is to be able to reason about execution traces according to the relative order of the sequences of actions in each trace but not about the duration of each action. Software is inherently asynchronous, and so is hardware in many cases if one abstracts the execution platform or many features of the execution platform like pipelines, caches, memory contention, etc. We call our temporal logic *Asynchronous HyperLTL* or in short, A-HLTL. The key addition is the notion of *trajectory* that controls the relative speed at which traces progress by chosing at each instant which traces move and which traces stutter. For example, the trajectory shown in Fig. 4c for the two traces of the program in Fig. 2 allows the lower trace to stutter in the first position while the upper trace advances. On the contrary, in the third position, the upper trace stutters while the lower trace moves from the second to the third position. This trajectory enables identification of stutter equivalence of the two traces with respect to state variable l and, hence, successful verification of observational determinism. In order to reflect the notion of trajectories in our logic, we lift the syntax of HyperLTL by allowing a trajectory modality. This way, the corresponding formula for observational determinism in A-HLTL is the following:

$$(\varphi\_{\textsf{OD}} \overset{\text{def}}{=} \forall \pi. \forall \pi'. \mathsf{E}. (li\_{\pi} \leftrightarrow li\_{\pi'}) \rightarrow \Box(lo\_{\pi} \leftrightarrow lo\_{\pi'}).$$

where E denotes the *existence* of a trajectory for temporal operator -. The A-HLTL formula for the Kripke structure in Fig. 3 is ∀π.∀π .E.((b<sup>π</sup> <sup>↔</sup> <sup>b</sup><sup>π</sup>- ) U -(a<sup>π</sup> ↔ a<sup>π</sup>- )). A-HLTL allows us to reason about relational properties between two different systems that differ on timing, like for example, translation validation [22], which relates executions of the target code with the source code with respect to a (trace or hyper) property.

We show an encoding of the PCP problem into model-checking a formula of the shape ∀π.∀π .E.(ψ1(π, π )∧✸ψ2(π, π )), which implies that model-checking A-HLTL is undecidable, even for the universal fragment. On the positive side, we show two decidable fragments of A-HLTL. The first algorithm is based on a *stuttering construction* in which we modify the Kripke structure to accept all stuttering expansions of the original paths. This algorithm can handle fragment <sup>∀</sup>π<sup>1</sup> ...πn.E.ψ, where the <sup>ψ</sup> is a *phase formula*, a class of safety formulas that appear in many hyperproperties and are the building block of expressing trace equivalence. Our second algorithm uses an *acceleration construction* to convert a finite sequence of transitions that do not change phase, into a single transition. This algorithm is able to handle formulas with arbitrary quantification but a simpler kind of phase formulas. A-HLTL is, thus, the first logic for hyperproperties that can express the major asynchronous hyperproperties of interest within decidable fragments. Moreover, A-HLTL is the first logic for asynchronous hyperproperties with a practical model checking algorithm. Both algorithms use internally HyperLTL model-checking as a building block. However, the reduction from A-HLTL model-checking into HyperLTL requires modifying both the formula and the model in a highly non-trivial way, to encode the exitence of trajectories. The choice of using HyperLTL model-checking as a building block is based on the existence of tools, but it does not imply that asynchronous properties of interest can be expressed in HyperLTL directly.

We have evaluated the stuttering construction on two sets of cases studies: a range of compiler optimizations and an SPI bus protocol. In both case studies, we were able to prove system correctness using our reduction from A-HLTL to synchronous HyperLTL.

*Organization.* The rest of the paper is structured as follows. Section 2 contains the preliminaries, and Sect. 3 introduces A-HLTL and presents examples of properties expressible in A-HLTL. Section 4 describes the decidable fragments and present procedures for the model-checking problem. Section 5 shows that the model-checking problem for general A-HLTL formulas is undecidable and present the lower-bound complexity. Experimental results are presented in Sect. 6. Finally, Sect. 7 discusses the related work, while Sect. 8 concludes. Detailed proofs appear in the longer version of this paper in [4].

#### **2 Preliminaries**

Let AP be a set of *atomic propositions* and Σ = 2AP be the *alphabet*, where we call each element of Σ a *letter*. A *trace* is an infinite sequence σ = a0a<sup>1</sup> ··· of letters from Σ. We denote the set of all infinite traces by Σ<sup>ω</sup>. We use σ(i) for <sup>a</sup><sup>i</sup> and <sup>σ</sup><sup>i</sup> for the suffix <sup>a</sup>iai+1 ··· . A *pointed trace* is a pair (σ, p), where <sup>p</sup> <sup>∈</sup> <sup>N</sup><sup>0</sup> is a natural number (called the *pointer*). Pointed traces allow to traverse a trace by moving the pointer. Given a pointed trace (σ, p) and n > 0, we use (σ, p) + n as a short for (σ, p + n). We denote the set of all pointed traces by PTR <sup>=</sup> {(σ, p) <sup>|</sup> <sup>σ</sup> <sup>∈</sup> <sup>Σ</sup><sup>ω</sup> and <sup>p</sup> <sup>∈</sup> <sup>N</sup>0}.

Two pointed traces (σ, p) and (σ , p ) are *stuttering equivalent* if there are two infinite sequences of indices p = i<sup>0</sup> < i<sup>1</sup> ... and p = j<sup>0</sup> < j<sup>1</sup> ... such that for all k ≥ 0 and for all l ∈ [ik, ik+1) and l ∈ [jk, jk+1), σ(l) = σ (l ). A pointed trace (σ , p ) is a *stuttering expansion* of (σ, p) if there is a sequence p = j<sup>0</sup> < j<sup>1</sup> <... such that for all k ≥ 0 and for all l ∈ [jk, jk+1), σ(p + k) = σ (l). We say that σ is stuttering equivalent to σ if (σ, 0) is stuttering equivalent to (σ , 0), and that σ is a stuttering expansion of σ if (σ , 0) is a stuttering expansion of (σ, 0).

A *Kripke structure* is a tuple K = S, S*init*, δ, L , where S is a set of states, S*init* ⊆ S is the set of initial states, δ ⊆ S × S is a transition relation, and L : S → Σ is a labeling function on the states of K. We require that for each s ∈ S, there exists s ∈ S, such that (s, s ) ∈ δ.

A *path* of a Kripke structure is an infinite sequence of states s(0)s(1)··· ∈ <sup>S</sup><sup>ω</sup>, such that <sup>s</sup>(0) <sup>∈</sup> <sup>S</sup>*init* and (s(i), s(<sup>i</sup> + 1)) <sup>∈</sup> <sup>δ</sup>, for all <sup>i</sup> <sup>≥</sup> 0. A *trace* of a Kripke structure is a trace <sup>σ</sup>(0)σ(1)σ(2)···∈ <sup>Σ</sup><sup>ω</sup>, such that there exists a path <sup>s</sup>(0)s(1)··· ∈ <sup>S</sup><sup>ω</sup> with <sup>σ</sup>(i) = <sup>L</sup>(s(i)) for all <sup>i</sup> <sup>≥</sup> 0. Abusing notation we use σ = L(ρ) to denote that σ is the trace corresponding to path ρ. We denote by Traces(K, s) the set of all traces of <sup>K</sup> with paths that start in state <sup>s</sup> <sup>∈</sup> <sup>S</sup>, We denote by Traces(K, A) the set of all traces that start from some state in <sup>A</sup> <sup>⊆</sup> <sup>S</sup> and Traces(K) as a short for Traces(K, S*init*).

**HyperLTL.** HyperLTL [7] is a temporal logic that extends LTL [19,21] for hyperproperties, which allows reasoning about multiple execution traces simultaneously. The syntax of HyperLTL is:

$$\begin{aligned} \varphi &::= \exists \pi.\varphi \quad \mid \,\forall \pi.\varphi \quad \mid \,\psi\\ \psi &::= a\_{\pi} \quad \mid \,\psi \lor \psi \mid \,\neg\psi \quad \mid \,\bigcirc \psi \quad \mid \,\psi \,\mathcal{U} \,\psi \end{aligned}$$

where π is a *trace variable* from an infinite supply of trace variables. The intended meaning of a<sup>π</sup> is that proposition a ∈ Σ holds in the current time in trace π. Trace quantifiers ∃π and ∀π allow reasoning simultaneously about different traces of the computation. Atomic predicates a<sup>π</sup> refer to a single trace π. Given a HyperLTL formula <sup>ϕ</sup>, we use Vars(ϕ) for the set of trace variables quantified in ϕ. A formula ϕ is well-formed if for all atoms a<sup>π</sup> in ϕ, π is quantified in ϕ (i.e., <sup>π</sup> <sup>∈</sup> Vars(ϕ)) and if no trace variable is quantified twice in <sup>ϕ</sup>. Given a set of traces T, the semantics of a HyperLTL formula ϕ is defined in terms of trace assignments, which is a (partial) map from trace variables to indexed traces <sup>Π</sup> : Vars(ϕ) PTR. The trace assignment with empty domain is denoted by <sup>Π</sup>∅. We use *Dom*(Π) for the subset of Vars(ϕ) for which <sup>Π</sup> is defined. Given a trace assignment Π, a trace variable π, a trace σ and a pointer p, we denote by Π[π → (σ, p)] the assignment that coincides with Π for every trace variable except for π, which is mapped to (σ, p). Also, we use Π + n to denote the trace assignment Π such that Π (π) = Π(π) + n for all π ∈ *Dom*(Π) = *Dom*(Π ). The semantics of HyperLTL is:

$$\begin{aligned} \Pi &\vdash\_T \exists \pi. \varphi & \quad \text{iff} \qquad \text{for some } \sigma \in T, \; \Pi[\pi \mapsto (\sigma, 0)] \vdash\_T \varphi\\ \Pi &\vdash\_T \forall \pi. \varphi & \quad \text{iff} \qquad \text{for all } \sigma \in T, \; \Pi[\pi \mapsto (\sigma, 0)] \vdash\_T \varphi\\ \Pi &\vdash\_T \psi & \quad \text{iff} \qquad \Pi \vdash\_T \psi\\ \Pi &\vdash\_\pi & \quad \text{iff} \qquad a \in \sigma(p), \; \text{where } (\sigma, p) = \Pi(\pi) \end{aligned}$$

$$\begin{aligned} \Pi &=\_T \psi\_1 \vee \psi\_2 & \text{iff} & \quad \Pi &=\_T \psi\_1 \text{ or } \Pi &=\_T \psi\_2\\ \Pi &= \neg \psi & \text{iff} & \quad \Pi \not\models \psi\\ \Pi &= \bigcirc \psi & \text{iff} & \quad (\varPi + 1) \vdash \psi\\ \Pi &= \psi\_1 \amalg \psi\_2 & \text{iff} & \quad \text{for some } j \ge 0 \ (\varPi + j) \vdash \psi\_2\\ & & \text{and for all } 0 \le i < j, (\varPi + i) \vdash \psi\_1 \end{aligned}$$

Note that quantifiers assign traces to trace variables and set the pointer to the initial position 0. We say that a set of traces T is a model of a HyperLTL formula ϕ, denoted T |= ϕ whenever Π<sup>∅</sup> |=<sup>T</sup> ϕ. A Kripke structure K is a model of a HyperLTL formula <sup>ϕ</sup>, denoted by K |<sup>=</sup> <sup>ϕ</sup>, whenever Traces(K) <sup>|</sup><sup>=</sup> <sup>ϕ</sup>.

#### **3 Asynchronous HyperLTL**

We introduce a temporal logic A-HLTL as an extension of HyperLTL to express asynchronous hyperproperties.

**Trajectories.** To model the asynchronous passage of time, we now introduce the notion of a trajectory, which chooses when traces move and when they stutter. Let V be a set of trace variables and let I ⊆ V. The I-successor of a trace assignment Π, denoted by Π + I, is the trace assignment Π such that Π (π) = Π(π) + 1 if π ∈ I and Π (π) = Π(π) otherwise. That is, the pointers of indices in I advance by one step, while the others remain the same. A *trajectory* t : t(0)t(1)t(2)··· for a formula ϕ is an infinite sequence of non-empty subsets of Vars(ϕ). Essentially, in each step of the trajectory one or more of the traces make progress. A trajectory is fair for a trace variable <sup>π</sup> <sup>∈</sup> Vars(ϕ) if there are infinitely many positions j such that π ∈ t(j). A trajectory is fair if it is fair for all trace variables in Vars(ϕ). Given a trajectory <sup>t</sup>, by <sup>t</sup> i , we mean the suffix <sup>t</sup>(i)t(<sup>i</sup> + 1)··· . Furthermore, for a set of trace variables <sup>V</sup>, we use TRJ<sup>V</sup> for set of all trajectories for indices from V.

#### **3.1 Syntax and Semantics of Asynchronous HyperLTL**

The syntax of Asynchornous HyperLTL is:

$$\begin{aligned} \varphi &::= \exists \pi.\varphi \mid \forall \pi.\varphi \mid \mathsf{E}\psi \mid \mathsf{A}\psi\\ \psi &::= a\_{\pi} \quad \mid \neg\psi \quad \mid \psi\_1 \lor \psi\_2 \mid \psi\_1 \; \mathcal{U} \; \psi\_2 \mid \mathsf{C}\psi \end{aligned}$$

where <sup>a</sup> <sup>∈</sup> AP, <sup>π</sup> is a trace variable from an infinite supply <sup>V</sup> of trace variables, E is the existential trajectory modality and A is the universal trajectory modality. The intended meaning of E is that there is a trajectory that gives an interpretation of the relative passage of time between the traces for which the temporal formula that relates the traces is satisfied. Dualy, A means that for all trajectories, the resulting alignment makes the inner formula true. It is important to note that there is no nesting of trajectory modalities and that all temporal operators in a formula are interpreted with respect to a single modality.

We use the usual syntactic sugar for Boolean operators *true* def = a<sup>π</sup> ∨ ¬aπ, *false* def = ¬*true*, ϕ<sup>1</sup> ∧ ϕ<sup>2</sup> def = ¬(¬ϕ<sup>1</sup> ∨ ¬ϕ2), and the syntactic sugar for temporal operators ✸ϕ def = *true* U ϕ, ϕ<sup>1</sup> → ϕ<sup>2</sup> def = ¬ϕ<sup>1</sup> ∨ ϕ2, and ϕ def = ¬✸¬ϕ, etc.

As before, we use trace assignments for the semantics of A-HLTL. Given (Π, t) where Π is a trace assignment and t a trajectory, we use (Π, t) + 1 for the successor of (Π, t) defined as (Π , t ) where t = t <sup>1</sup>, and Π (π) = Π(π) + 1 if π ∈ t(0) and Π (π) = Π(π) otherwise. We use (Π, t) + k as the k-th successor of (Π, t).

The satisfaction of an asynchronous HyperLTL formula ϕ over a trace assignment Π and a set of traces T, denoted by Π |=<sup>T</sup> ϕ is defined as follows:


We say that a set T of traces satisfies a closed sentence ϕ, denoted by T |= ϕ, if Π<sup>∅</sup> |=<sup>T</sup> ϕ. We say that a Kripke structure K satisfies an A-HLTL formula ϕ (and write K |<sup>=</sup> <sup>ϕ</sup>) if and only if we have Traces(K, S*init*) <sup>|</sup><sup>=</sup> <sup>ϕ</sup>.

#### **3.2 Examples of A-HLTL**

We illustrate the expressive power of A-HLTL by introducing the asynchronous version of well-known properties.

*Linearizability.* [16] requires that any history of execution of a concurrent data structure (i.e., sequence of invocation and response by different threads) matches some sequential order of invocations and responses:

$$
\varphi\_{\mathsf{LNRZ}} \stackrel{\text{def}}{=} \forall \pi. \exists \pi'. \mathsf{E}. \Box(\mathsf{histor}\_{\pi} \leftrightarrow \mathsf{histor}\_{\pi'}).
$$

where history denotes method invocations (and not the actual execution of the internal instructions of the concurrent library) by the different threads and the response observed, trace π ranges over the concurrent data structure and π ranges over its sequential counterpart.

*Goguen and Meseguer's Noninterference (GMNI).* [14] stipulates that, for all traces, the low-observable output must not change when all high inputs are removed:

$$
\varphi\_{\mathsf{GMM}} = \stackrel{\text{def}}{=} \forall \pi. \exists \pi'. \mathsf{E}. (\Box \lambda\_{\pi'}) \land \Box (lo\_{\pi} \leftrightarrow lo\_{\pi'}).
$$

where λπ expresses that all of the high inputs in the current state of π have dummy value λ, and denotes low-observable output proposition.

*Not never Terminates.* [18] requires that for every initial state, there is a terminating trace and a non-terminating trace:

$$\varphi\_{\mathsf{NNT}} \stackrel{\text{def}}{=} \forall \pi. \exists \pi'. \exists \pi''. \mathsf{E}. (\pi[0] = \pi'[0] = \pi''[0]) \to \ (\lozenge \mathsf{term}\_{\pi'} \land \sqsupset \mathsf{term}\_{\pi''})$$

*Termination-Insensitive Noninterference.* [25] requires that for two executions that start from a low-observable states, information leaks are permitted if they are transmitted purely by the program's termination behavior. That is, the program may diverge on some high inputs and terminate on others:

$$\varphi\_{\mathsf{TIN}} \stackrel{\text{def}}{=} \forall \pi. \forall \pi'. \mathsf{E}. \left(l\_{\pi} \leftrightarrow l\_{\pi'}\right) \quad \rightarrow \left(\begin{array}{c} (\Box \neg \mathsf{term}\_{\pi} \vee \Box \neg \mathsf{term}\_{\pi'}) \vee \\ \Diamond (\mathsf{term}\_{\pi} \wedge \mathsf{term}\_{\pi'} \wedge l\_{\pi} \leftrightarrow l\_{\pi'}) \end{array}\right).$$

*Termination-Sensitive Noninterference.* [2] Termination-sensitive noninterference is the same as termination insensitive, except that it forbids one trace to diverge and the other to terminate:

$$\varphi\_{\mathsf{T}\mathsf{S}\mathsf{N}} \stackrel{\text{def}}{=} \forall \pi. \forall \pi'. \mathsf{E}. \left(l\_{\pi} \leftrightarrow l\_{\pi'}\right) \to \left(\begin{subarray}{c} \left(\Box \neg \mathsf{term}\_{\pi} \wedge \Box \neg \mathsf{term}\_{\pi'}\right) \vee \\ \Diamond \left(\mathsf{term}\_{\pi} \wedge \mathsf{term}\_{\pi'} \wedge l\_{\pi} \leftrightarrow l\_{\pi'}\right) \end{subarray}\right)$$

#### **4 Model-Checking A-HLTL**

In this section, we show the decidability of the model-checking problem for two classes of A-HLTL formulas using two different algorithms:


In both cases the problem is reduced to model-checking HyperLTL formulas, which is known to be decidable [7,12]. We describe each construction separately.

#### **4.1 The Stuttering Construction**

We consider first A-HLTL formulas of the form <sup>∀</sup>π<sup>1</sup> ...πn.E.ψ. We will then extend our results to the <sup>∃</sup><sup>∗</sup> fragment, to handle the <sup>A</sup> trajectory modality and to a larger collection of predicates. The class of temporal formulas ψ that we handle are called *admissible* formulas, and are defined as the Boolean combination of:


Given an admissible formula ψ, we use ψ*ph* for its phase formula, and we use ψ[ψ*ph*  ξ] for the formula that results from ψ by replacing ψ*ph* with ξ. Since ψ*ph* occurs only once in ψ, we use the fact that ψ*ph* appears with a single polarity. We present here the construction for positive polarity which is the case in all practical formulas (the case for negative polarity is analogous).

The algorithm has two parts. First, we generate the *stuttering* Kripke structure <sup>K</sup>*st* whose paths are the stuttering expansions of paths in the original Kripke structure K. Then, we modify the admissible formula ψ into ψ*sync* such that K |<sup>=</sup> <sup>∀</sup>π<sup>1</sup> ...πn.E.ψ if and only if <sup>K</sup>*st* <sup>|</sup><sup>=</sup> <sup>∀</sup>π<sup>1</sup> ...πn.ψ*sync*. We describe each of the concepts separately.

*Phase Formulas.* We first define *atomic phase formulas* ( <sup>p</sup>∈<sup>P</sup> <sup>p</sup><sup>π</sup>*<sup>i</sup>* <sup>↔</sup> <sup>p</sup><sup>π</sup>*<sup>j</sup>* ) which are characterized by (πi, π<sup>j</sup> , P), where <sup>P</sup> <sup>⊆</sup> AP and <sup>π</sup><sup>i</sup> and <sup>π</sup><sup>j</sup> are two different trace variables. We use *color* to refer to a valuation of the variables in P. Essentially, an atomic phase formula asserts that all propositions in P coincide in both traces at all points in time, that is, both traces exhibit the same sequence of colors. Since the passage of time proceeds at different speeds in the different traces—according to the trajectory—atomic phase formulas state the traces for π<sup>i</sup> and π<sup>j</sup> are sequences of phases of the same color, where corresponding phases may have different lengths. A phase formula is formed from atomic formulas as follows:

$$\Box(\bigwedge\_{p \in P^1} p\_{\pi\_i^1} \leftrightarrow p\_{\pi\_j^1} \wedge \dots \wedge \bigwedge\_{p \in P^k} p\_{\pi\_i^k} \leftrightarrow p\_{\pi\_j^k})$$

We use <sup>P</sup> : {(π<sup>1</sup> <sup>i</sup> , π<sup>1</sup> <sup>j</sup> , P<sup>1</sup>),...,(π<sup>k</sup> <sup>i</sup> , π<sup>k</sup> <sup>j</sup> , P<sup>k</sup>)} for the collection of predicates and trace variables that characterize a phase formula.

*Stuttering Kripke Structure.* We start from <sup>K</sup> and create <sup>K</sup>*st* that accepts the stuttering expansions of traces in K. First, the alphabet of atomic propositions is enriched with a fresh proposition *st*, that is AP*st* <sup>=</sup> AP∪{*st*}, to encode whether the state represents a real move or a stuttering move. Given K = S, S*init*, δ, L , the stuttering Kripke structure is <sup>K</sup>*st* <sup>=</sup> S*st*, S*init*, δ*st*, L*st* where:


The construction generates a Kripke structure <sup>K</sup>*st* which is linear in the size of the original Kripke structure K. It is easy to see that every stuttering expansion of a path of <sup>K</sup> has a corresponding path in <sup>K</sup>*st*, where the repeated version of state <sup>s</sup> is captured by state <sup>s</sup>*st*. Conversely every path <sup>ρ</sup> in <sup>K</sup>*st* whose trace satisfies -✸¬*st* can be turned into its "stuttering compression" by removing all stuttering states, which is a path of K. Note that the constraint -✸¬*st* guarantees that there are infinitely many non-stuttering positions in ρ , so ρ is well-defined. Hence, this constructions provides a one-to-one correspondence between a trajectory toguether with a tuple of traces of K, and the corresponding tuple of traces of <sup>K</sup>*st*.

*State and Monadic Formulas are not Affected by Trajectories.* State formulas are relational formulas that are evaluated at the beginning of the computation. Temporal monadic formulas only refer to one trace variable and are stuttering invariant by definition. Therefore, none of these formulas are affected by the stuttering induced by a trajectory, as the relative stuttering among traces does not affect their truth valuation. We first note that given a trace assigned for each of the trace variables in Vars(ϕ) the truth value of state formulas and monadic formulas does not depend on the trajectory chosen.

*Phase Alignment of Asynchronous Sequences.* We use the stuttering in <sup>K</sup>*st* to encode the relative progress of traces as dictated by a trajectory. We will now introduce synchronous HyperLTL formulas to reason in <sup>K</sup>*st* about the corresponding states during the asynchronous evaluation in K. The important concept is that of "phase changes", which are the points in a trace σ at which the valuation of the predicates P in an atomic phase formula (πi, π<sup>j</sup> , P) change. Let Π be a trace assignment for traces in K that maps π<sup>i</sup> to a pointed trace (σ, l). We say that in assignment Π, trace variable π<sup>i</sup> is *about to change phase* with respect to (πi, π<sup>j</sup> , P) if for some p ∈ P either p ∈ σ(l) but p /∈ σ(l+ 1) or p /∈ σ(l) but <sup>p</sup> <sup>∈</sup> <sup>σ</sup>(l+ 1). Note that in <sup>K</sup>*st* the next relevant letter (the one corresponding to σ(l + 1) is the first letter that is not a stuttering letter). Formula *change*<sup>P</sup> (πi) captures that the next non-stuttering step of π<sup>i</sup> is a phase change (with respect to predicates in P and therefore with respect to atomic phase formula α):

$$change\_P(\pi\_i) \stackrel{\text{def}}{=} \bigvee\_{p \in P} p\_{\pi\_i} \not\hookrightarrow \bigodot(st\_{\pi\_i} \mathcal{U} \, p\_{\pi\_i})$$

A phase change for π<sup>i</sup> in atomic phase formula (πi, π<sup>j</sup> , P) implies that π<sup>j</sup> must also proceed to change phase. The second observation is that when π<sup>i</sup> and π<sup>j</sup> are not changing phases, any choice that the trajectory makes will preserve the valuation of the atomic phase formula.

We now capture formally this intuition as formulas. Predicate *move*(πi) def = ◯(¬*st*π*<sup>i</sup>* ) indicates whether trace variable π<sup>i</sup> will move (and not stutter) at a given instant of the computation. The following temporal formula captures the consistency criteria of phase changes as a synchronized decision for moving traces π<sup>i</sup> and π<sup>j</sup> related by an atomic phase formula (πi, π<sup>j</sup> , P):

$$\begin{array}{c} \mathit{align{\tiny{\tiny{\tiny{\tiny{\tiny{\pi}\_{i}}}}}}(\pi\_{i}) \stackrel{\scriptstyle{\mbox{def}}}{=} \\ \begin{pmatrix} (\mathit{move}(\pi\_{i}) \land & \mathit{move}(\pi\_{j})) \to (\mathit{change}\_{P}(\pi\_{i}) \leftrightarrow \mathit{change}\_{P}(\pi\_{j})) \land \\ (\mathit{move}(\pi\_{i}) \land \neg \mathit{move}(\pi\_{j})) \to \neg \mathit{change}\_{P}(\pi\_{i}) & \land \\ (\neg \mathit{move}(\pi\_{i}) \land & \mathit{move}(\pi\_{j})) \to \neg \mathit{change}\_{P}(\pi\_{j}) \end{array} \end{array}$$

We will reduce the model-checking problem in A-HLTL to checking in <sup>K</sup>*st* that tuples of traces that align phase changes—for all atomic phase formulas— satisfy all sub-formulas of the specification ψ. The following two formulas express that all atomic phase formulas align, and that all traces are fair (all traces eventually move):

$$phase\stackrel{\text{def}}{=} \bigwedge\_{(\pi\_i,\pi\_j,P)\in \mathcal{P}} \operatorname{align}\_{\left(\pi\_i,\pi\_j,P\right)}\\\\ \qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\square\odot\neg st\_i$$

We will then check in <sup>K</sup>*st* that all stuttering traces that align phases and are fair satisfy the desired formula ψ, that is (*phase* ∧ *fair*) → ψ. Note that all those tuples of traces that do not align phases are ruled out in the antecedent.

A final technical detail in the construction is that we must guarantee that for all tuples of paths of K there are stuttering expansions that are fair and align phases, and that they have the same number of phases. Otherwise, there are paths of K that cannot be aligned, which inevitably leads to a violation of ψ*ph*. It could be the case that some tuple of traces of K cannot possibly align the phase changes corresponding to all atomic phase formulas. This can happen in two cases: (1) when two traces have different number of phases, and (2) when there is a circular dependency between the atomic formulas that force the trajectory to synchronize the traces in incompatible orders. The first case is captured by:

$$
gamma \stackrel{\text{def}}{=} \bigvee\_{(\pi\_i, \pi\_j, P)} \left( \Box \neg change\_P(\pi\_i) \right) \not\leftrightarrow \left( \Box \neg change\_P(\pi\_j) \right),
$$

The second case is captured by the following formula, where *cycles*(ψ*ph*) are the sequences of atomic formulas that form a simple cycle, that is [(π<sup>0</sup>, π<sup>1</sup>, P<sup>0</sup>),(π<sup>1</sup>, π<sup>2</sup>, P<sup>1</sup>)...(π<sup>k</sup>, π<sup>0</sup>, P<sup>k</sup>)] such that the second trace variable is the first trace variable of the next atomic phase formula, circularly (see Ex. 1 below):

$$block{array}{c} \mathit{block} \stackrel{\mathit{def}}{=} \bigvee\_{C \in \mathit{cycles}(\psi\_{ph})} \left( \bigwedge\_{\{\pi\_i, \pi\_j, P\} \in C} change\_P(\pi\_i) \wedge \neg change\_P(\pi\_j) \right),$$

Essentially, *block* encodes whether the set of traces involved cannot proceed without violating *phase*, because *align* forbids all traces involved to move. Hence, the formula *phase* <sup>U</sup> (*missalign* <sup>∨</sup> *block*) captures to those traces of <sup>K</sup>*st* that contain an aligned prefix of computation that lead to a miss-alignment or a block. The proof of correctness shows that given a tuple of traces of K, if there is a trajectory that aligns the phase changes (which must exist if there is a trajectory that makes ψ*ph* true), then all trajectories that respect *phase* will also align the phase changes (and also satisfy ψ*ph*).

We are finally ready to describe the synchronous phase formula ψ*sync*. First, this formula is only evaluated against tuples of fair traces, which correspond to the stuttering extensions of paths of K. Then, the phase formula ψ*ph* is translated into a formula that captures (1) that following a phase alignment cannot lead to a block or to two traces changing phases a different number of times, and (2) that if phases are aligned then ψ*ph* holds. Formally,

$$\psi\_{sync} \stackrel{\text{def}}{=} fair \to \psi[\psi\_{ph} \lhd \psi'], \quad \text{where } \psi' = \begin{pmatrix} \neg(phase \ \mathcal{U} \ (missalign \ \lor \text{block})) & \land \\ \Box phase \to \psi\_{ph} \end{pmatrix}$$

*Example 1.* We illustrate the previous definitions with the Kripke structures <sup>K</sup>1, <sup>K</sup><sup>2</sup> and <sup>K</sup><sup>3</sup> in Fig. <sup>5</sup> and their stuttering variants <sup>K</sup>*st* <sup>1</sup> , <sup>K</sup>*st* <sup>2</sup> and <sup>K</sup>*st* <sup>3</sup> Consider formula <sup>∀</sup>π1.∀π2.E.-(a<sup>π</sup><sup>1</sup> ↔ a<sup>π</sup><sup>2</sup> ). Consider the following trace assignments:


Consider the trace assignment <sup>Π</sup><sup>1</sup> on the left, where <sup>π</sup><sup>1</sup> is a trace of <sup>K</sup>*st* 1 corresponding to the path of K<sup>1</sup> that visits s1, and π<sup>2</sup> corresponds to the path that visits s2. This trace assignment aligns the atomic phase formula (π1, π2, {a}) at all positions. In particular, at position 0, we have *change*{a}(π1), but ¬*change*{a}(π2), and ¬*move*(π1) and *move*(π2), as *align*{a} requires.

**Fig. 5.** Kripke structure K<sup>1</sup> (left), K<sup>2</sup> (middle) and K<sup>3</sup> (right).


Consider now the trace assignment Π<sup>2</sup> in the middle, where again π<sup>1</sup> corresponds to the path in <sup>K</sup>*st* <sup>1</sup> that visits s<sup>1</sup> and π<sup>2</sup> the path that visits s2. In this case, we have ¬*align*{a} at position 0 because *change*{a}(π1) and ¬*change*{a}(π2) hold, and both *move*(π1) and *move*(π2). Consider

now <sup>Π</sup><sup>3</sup> on the right, where <sup>π</sup><sup>1</sup> corresponds to the path of <sup>K</sup>*st* <sup>2</sup> that visits <sup>s</sup><sup>3</sup> and <sup>π</sup><sup>2</sup> to the path of <sup>K</sup>*st* <sup>2</sup> that visits s4. In this case *align*{a} holds at 0 and *missalign* holds at 1 because at 1, -¬*change*{a}(π1) holds, but not -<sup>¬</sup>*change*{a}(π2). Therefore, *phase* <sup>U</sup> (*missalign* <sup>∨</sup> *block*) holds for <sup>Π</sup><sup>3</sup>. Finally, consider <sup>∀</sup>π1.∀π2.∀π3.E.-(a<sup>π</sup><sup>1</sup> ↔ a<sup>π</sup><sup>2</sup> ∧ b<sup>π</sup><sup>2</sup> ↔ b<sup>π</sup><sup>3</sup> ∧ c<sup>π</sup><sup>3</sup> ↔ c<sup>π</sup><sup>2</sup> ) and the trace assignment <sup>Π</sup> of <sup>K</sup>*st* <sup>3</sup> shown below on the left. In this case *phase* holds at position 0 and *block* holds at position 1. This is because *change*{a}(π1) and ¬*change*{a}(π2), *change*{b}(π2) and ¬*change*{b}(π3), and *change*{c}(π3) and also ¬*change*{c}(π1). This illustrates that it will not be possible to align all three atomic phase formulas.

We are now ready to state the main result of this section.

**Theorem 1.** *Let* K *be a Kripke structure and* ψ *an admissible formula. Then,* K |<sup>=</sup> <sup>∀</sup>π<sup>1</sup> ...πn.E.ψ *if and only if* <sup>K</sup>*st* <sup>|</sup><sup>=</sup> <sup>∀</sup>π<sup>1</sup> ...πn.ψ*sync.*

Dually, to show that the ∃<sup>∗</sup> fragment is decidable, we consider replacing ψ*ph* by the formula

$$
\psi\_{esymc} \stackrel{\text{def}}{=} fair \land \psi[\psi\_{ph} \lhd (\Box phase \land \psi\_{ph})].
$$

**Theorem 2.** *Let* K *be a Kripke structure and* ψ *an admissible formula. Then* K |<sup>=</sup> <sup>∃</sup>π<sup>1</sup> ...πn.E.ψ *if and only if* <sup>K</sup>*st* <sup>|</sup><sup>=</sup> <sup>∃</sup>π<sup>1</sup> ...πn.ψ*esync.*

The proof of Theorem 2 takes a witness tuple and trajectory in K and shows that the induced tuple in <sup>K</sup>*st* is *fair*, satisfies *phase* and that the valuation of <sup>ψ</sup>*ph* is preserved. Similarly, as before, tuples of traces of <sup>K</sup>*st* that are fair and follow phase alignments induce a trajectory on their stuttering compression that also preserve ψ*ph*.

**Corollary 1.** *The problems of model-checking* ∀<sup>∗</sup> *admissible* A-HLTL *formulas and* ∃<sup>∗</sup> *admissible* A-HLTL *formulas is decidable.*

We finally consider the negation of phase formulas, called *co-phase formulas*, which are formulas of the form ✸¬R where R a conjunction of atomic phase formulas. Interestingly, deciding co-admissible formulas (consisting of Boolean combinations of state-formulas, monadic temporal formulas and one co-phase formula in positive polarity) is easier than before, as one can turn the co-phase formula into a monadic formula enumerating all the violations of the atomic phase formulas (p ∈ P such that p<sup>π</sup>*<sup>i</sup>* ↔ p<sup>π</sup>*<sup>j</sup>* ) turns the atomic phase formula into (✸p<sup>π</sup>*<sup>i</sup>* ∧ ✸¬p<sup>π</sup>*<sup>j</sup>* ∨ (✸¬p<sup>π</sup>*<sup>i</sup>* ∧ ✸p<sup>π</sup>*<sup>j</sup>* . It follows that model-checking co-admissible formulas is also decidable (for both ∀<sup>∗</sup> and ∃<sup>∗</sup>). Note that an admissible formula in negative polarity is a co-admissible formula in positive polarity (and vice versa). Finally, since K |<sup>=</sup> <sup>∀</sup>π<sup>1</sup> ... <sup>∀</sup>πn.A.ψ if and only if <sup>K</sup> |<sup>=</sup> <sup>∃</sup>π<sup>1</sup> ... <sup>∃</sup>πn.E.¬ψ, it follows that model-checking is also decidable for the A modality for both admissible and co-admissible formulas (in both polarities), and for both the ∀<sup>∗</sup> and ∃<sup>∗</sup> fragments.

**Theorem 3.** *Model-checking* ∀<sup>∗</sup> *or* ∃<sup>∗</sup> *admissible and co-admissible formulas is decidable both for formulas with* E *and formulas with* A*.*

#### **4.2 The Accelerating Construction**

The admissible formula in the stuttering construction can express many formulas of interest, but the quantifier structure admits no quantifier alternation. We now consider a second decidable fragment for A-HLTL formulas consisting of formulas with arbitrary quantification <sup>Q</sup>1π1.Q2π2.....Qnπ<sup>n</sup>E.ψ such that <sup>Q</sup><sup>i</sup> ∈ {∀, ∃}, but where ψ is an admissible formula where all atomic phase formulas use the same atomic predicates <sup>P</sup> <sup>⊆</sup> AP. We call these admissible formulas *simple admissible formulas*. The proof of decidability proceeds this time by creating the *accelerated* Kripke structure <sup>K</sup>*acc*, where paths jump in one step to the next phase change, and reducing to a HyperLTL model-checking problem on <sup>K</sup>*acc*.

*Accelerated Kripke Structure.* The main idea of the acceleration construction is to convert a finite sequence of transitions in K that only change phase in the last transition into a single transition in <sup>K</sup>*acc*. Also, an infinite sequence of transitions with no phase change is transformed into a self-loop around a sink state. The alphabet remains the same, AP. Given <sup>K</sup> <sup>=</sup> S, S*init*, δ, L , the accelerated Kripke structure is <sup>K</sup>*acc* <sup>=</sup> S*acc*, S*init*, δ*acc*, L*acc* where:


This construction can, with standard techniques, be enriched to encode the satisfaction of the temporal monadic formulas along paths of K, and then also accelerate the fairness conditions (annotating the accepting states reached along the accelerated paths) into <sup>K</sup>*acc*.

*Relating Paths to Accelerated Paths.* We now define two auxiliary functions to aid in the proof.


Given a trace assignment <sup>Π</sup> for formula <sup>Q</sup>1π1.... <sup>Q</sup>nπn.E.ψ that assigns Π(πi)=(σi, 0) for every i and a path assignment Π for formula Q1π1.....Qnπn.ψ that assigns Π (π)i = (σ <sup>i</sup>, 0), we write *acc*(Π) = Π if the paths that generate the corresponding traces are related by *acc*. Similarly we defined *dec*(Π ) = Π. It is easy to show from the construction above that if <sup>Π</sup> <sup>|</sup><sup>=</sup> E<sup>ψ</sup> then *acc*(Π) <sup>|</sup><sup>=</sup> <sup>ψ</sup>, and if <sup>Π</sup> <sup>|</sup><sup>=</sup> <sup>ψ</sup> then *dec*(Π ) <sup>|</sup><sup>=</sup> Eψ.

The main result for the accelerating construction follows immediately from this observation and allows to reduce the model-checking problem to HyperLTL.

**Theorem 4.** *Let* <sup>K</sup> *be an arbitrary Kripke structure,* <sup>Q</sup>1π1.....Qnπn.E.ψ *such that* <sup>ψ</sup> *is a simple admissible formula. Then* K |<sup>=</sup> <sup>Q</sup>1π1.... <sup>Q</sup>nπ<sup>n</sup>E.ψ *if and only if* <sup>K</sup>*acc* <sup>|</sup><sup>=</sup> <sup>Q</sup>1π1.... <sup>Q</sup>nπn.ψ*.*

#### **4.3 Decidable Practical A-HLTL Formulas**

We revisit the properties expressed in Sect. 3.2.


– *Termination-insensitive noninterference*. To handle ϕTIN we rewrite the formula as follows

$$\varphi\_{\mathsf{TIN}} \stackrel{\text{def}}{=} \forall \pi. \forall \pi'. \mathsf{E}. \left(l\_{\pi} \leftrightarrow l\_{\pi'}\right) \quad \rightarrow \begin{pmatrix} (\Box \neg \mathsf{term}\_{\pi} \vee \Box \neg \mathsf{term}\_{\pi'}) \vee\\ \Box ((l\_{\pi} \wedge \mathsf{term}\_{\pi}) \leftrightarrow (l\_{\pi'} \wedge \mathsf{term}\_{\pi'})) \end{pmatrix}.$$

Note that (l<sup>π</sup> <sup>∧</sup> termπ) can be turned into a state predicate of <sup>π</sup>. This formula is equivalent because the last case is evaluates precisely to l<sup>π</sup> ↔ lπ when both traces terminate. This formula can be handled by the stuttering construction.

– *Termination-sensitive noninterference*. Similarly, to handle ϕTSN we rewrite the formula as

$$\varphi\_{\mathsf{TSN}} \stackrel{\text{def}}{=} \forall \pi. \forall \pi'. \mathsf{E}. \left(l\_{\pi} \leftrightarrow l\_{\pi'}\right) \quad \rightarrow \begin{pmatrix} (\Box \neg \mathsf{term}\_{\pi} \wedge \Box \neg \mathsf{term}\_{\pi'}) \vee\\ \Box ((l\_{\pi} \wedge \mathsf{term}\_{\pi}) \leftrightarrow (l\_{\pi'} \wedge \mathsf{term}\_{\pi'})) \end{pmatrix}.$$

This is again equivalent because the last case again is the only relevant case when both paths terminate. Again, this case is covered by the stuttering construction.

#### **5 Undecidability and Lower-Bound Complexity**

In this section, we show that the general problem of model-checking A-HLTL is undecidable. Then, we show a polynomial reduction from the synchronous HyperLTL model-checking into A-HLTL model-checking, which shows that even for those A-HLTL formulas for which the model-checking is decidable, this problem is no easier than the corresponding problem for HyperLTL, which is known to be PSPACE-hard in the size of the Kripke structure.

**Theorem 5.** *Let* K *be a Kripke structure and* ϕ *be an asynchronous HyperLTL formula. The problem of determining whether or not* K |= ϕ *is undecidable.*

*Proof (sketch).* We reduce the complement of the *post correspondence problem (PCP)* [23,26] to the A-HLTL model checking problem. PCP consists of a set of dominos, for example, of the form [ <sup>w</sup> <sup>v</sup> ] = {[ <sup>b</sup> ca ], [ <sup>a</sup> ab ], [ ca <sup>a</sup> ], [ abc <sup>c</sup> ]} and the problem is to decide whether there is a sequence of dominos (with possible repetitions), such that the upper and lower finite strings of the dominos are equal. A solution to the above set of dominos is the sequence [ <sup>a</sup> ab ][ <sup>b</sup> ca ][ ca <sup>a</sup> ][ <sup>a</sup> ab ][ abc <sup>c</sup> ]. We map a given set of dominos to a Kripke structure that allows arranging the dominos in a sequence (see Fig. 6 for an example), where v and w indicate lower and upper words, respectively, *dom*<sup>i</sup> is for each domino [ <sup>w</sup>*<sup>i</sup>* <sup>v</sup>*<sup>i</sup>* ], and proposition *lc* marks whether or not a new letter is processed. The A-HLTL formula in our reduction is the following such that *dom*<sup>π</sup>*<sup>w</sup>* def = <sup>i</sup>∈[1..k] *dom*<sup>i</sup> <sup>π</sup>*<sup>w</sup>* :

$$
\varphi\_{\overline{p:p}} \stackrel{\text{def}}{=} \forall \pi\_w \forall \pi\_v. \mathsf{E}. \left(\varphi\_{type} \to \left(\varphi\_{domino} \lor \varphi\_{word}\right)\right).
$$

**Fig. 6.** Mapping from PCP to model checking A-HLTL (only construction for dominos [ *w*1 *<sup>v</sup>*<sup>1</sup> ]=[ *<sup>b</sup> ca* ] and [ *<sup>w</sup>*<sup>4</sup> *<sup>v</sup>*<sup>4</sup> ]=[ *abc <sup>c</sup>* ] are shown).

$$\begin{split} \text{where} \qquad \varphi\_{type} & \stackrel{\text{def}}{=} \Big( (w\_{\pi\_{w}} \wedge \neg v\_{\pi\_{w}}) \mathcal{U} \, end\_{\pi\_{w}} \Big) \wedge \Big( (\neg w\_{\pi\_{v}} \wedge v\_{\pi\_{v}}) \mathcal{U} \, end\_{\pi\_{v}} \Big) \\ \varphi\_{domino} & \stackrel{\text{def}}{=} \big( dom\_{\pi\_{w}} \leftrightarrow dom\_{\pi\_{v}} \Big) \wedge \bigotimes\_{i=1}^{k} dom\_{\pi\_{w}}^{i} \not\models dom\_{\pi\_{v}}^{i} \\ \varphi\_{word} & \stackrel{\text{def}}{=} \big( lc\_{\pi\_{w}} \leftrightarrow lc\_{\pi\_{v}} \Big) \ \wedge \bigotimes\_{l \in \Sigma\_{pc}} (l\_{\pi\_{w}} \not\models l\_{\pi\_{v}}) \end{split}$$

The intention of formula ϕ*pcp* is that the Kripke structure is a model of the formula if and only if the original PCP problem has *no* solution. Intuitively, formula ϕ*type* forces trace π<sup>w</sup> (respectively, πv) to traverse only the traces labeled by w (respectively, v) to build a w-word (respectively, v-word). Formula ϕ*domino* establishes that the trajectory aligns the positions at which the domino indices are checked and at last once the index is different. Finally, formula ϕ*word* captures if π<sup>w</sup> and π<sup>v</sup> are aligned to compare the letters, at least one pair of the letters prescribed by the existential trajectory are different. In the detailed proof in [4], we show that the constructed Kripke structure satisfies formula ϕ*pcp* if and only if the answer to deciding PCP is negative.

Theorem 5 above implies that there is no algorithm to decide the modelchecking problem correctly for every formula and every system. However, as we saw in Sect. 4 for some formulas the model-checking problem is decidable. We now show that in these cases the problem is at least as hard as model-checking HyperLTL, which is known to be PSPACE-hard [7,24].

**Theorem 6.** *Given a HyperLTL formula* ϕ *and a Kripke structure* K *there is a* A-HLTL *formula* ϕ *and a Kripke structure* K *such that* K *is linear in the size of* K*,* ϕ *is polynomial on the size of* ϕ *and* K |= ϕ *if and only if* K |= ϕ *.*

The proof proceeds as follow. Giving K we build a Kripke structure K that alternates between real states in K and synchronization states. Then the formula is transformed to force alternations at every other step, therefore forcing the trajectory to synchronize (see [4] for details). Since the model-checking problem for HyperLTL is PSPACE-hard on the size of the Kripke structure, the same follows for A-HLTL.

**Corollary 2.** *For asynchronous HyperLTL formulas, the model checking problem is PSPACE-hard in the size of the system.*

#### **6 Case Studies and Evaluation**

We applied our algorithm for the ∀<sup>∗</sup> <sup>π</sup>E A-HLTL fragment to several examples. After manually reducing the asynchronous model checking problem to a synchronous one, we use MCHyper [10,11] to check our property. MCHyper is a model checker for synchronous HyperLTL that can handle formulas with up to one quantifier alternation. It computes the self composition of the system and composes it with the formula automaton. ABC [6] is then used as the backend tool checking the reachability of a violation.

Our reduction from the asynchronous to the synchronous semantics follows the stuttering construction described in Sect. 4.1. To model check a system against an A-HLTL formula, we first add a stuttering input to the system that forces the system to stutter in the current state. The transformed formula ensures that the stuttering guarantees synchronous phase changes. In future work, we will fully automate our reduction resulting in a verification tool for asynchronous hyperproperties from the decidable fragment. We now describe the various case studies<sup>1</sup>. All our experiments were performed on a MacBook Pro with a 3.3 GHz processor and 16 GB of RAM running MacOS 11.1.

#### **6.1 Compiler Optimizations**

We modeled the source and target programs of different compiler optimization techniques (from [20]) as finite state machines encoded as circuits, and used asynchronous hyperproperties to prove the correspondence between both programs. We analyzed the following optimizations:


<sup>1</sup> The experimental data is publicly available at https://github.com/reactive-systems/ MCHyper in case-studies/asynchronous-hyperltl\_2021.


**Table 1.** Verification times of MCHyper and system sizes in number of latches (#ls) and and-gates (#ands) for the case studies.

Besides evaluating each optimization individually, we also examined several combinations of these optimizations. Each optimization affects the alignment between source and target program, so synchronous hyperproperties fail to recognize the correspondence between both programs. Using asynchronous hyperproperties instead allows us to compensate for this misalignment by stuttering the programs accordingly. Essentially, each optimization is checked against the following A-HLTL formula in which π represents traces from the source program and π traces from the target program:

$$\forall \pi. \forall \pi'. \mathsf{E}. (\bigwedge\_{i \in I} i\_{\pi} \leftrightarrow i\_{\pi'}) \to (\Box \bigwedge\_{o \in O} o\_{\pi} \leftrightarrow o\_{\pi'})$$

This formula states that for all pairs of traces that initially agree on the inputs from the set I there exists a trajectory that aligns the phase changes of the outputs in set O. We use the stuttering construction and MCHyper to verify that in all cases the source and target programs go through the same phases of possibly different length. The results of this case study are summarized in Table 1(a). We note that A-HLTL model-checking subsumes the approach in [20] based on construction of a *buffer automaton* to reason about the alignment of executions.

#### **6.2 SPI Bus Protocol**

The Serial Peripheral Interface (SPI) is a bus protocol that supports a single main component's communication with multiple secondary components. Each secondary can be selected individually by the main via the secondary's own *ss* ("secondary select") input signal. If a secondary is enabled (that is, if ¬*ss* holds as the secondary select is "active low"), it reads the *mosi* (main out, secondary in) signal and writes to the *miso* (main in, secondary out) wire.

We verify the behavior of a single SPI secondary component that receives an input which it sends to the main component upon request. This behavior should always be the same, independent of when the secondary is enabled or how fast the bus protocol's "serial clock" (*sclk*) set by the main component ticks compared to the secondary's internal clock. The A-HLTL formula we check is the following (see observational determinism in Sect. 1):

$$\forall \pi. \forall \pi'. \mathsf{E}. \left( \begin{array}{c} \bigwedge \quad i\_{\pi} \leftrightarrow i\_{\pi'} \\ \wedge \\ \quad SPI \text{ input assumptions} \end{array} \right) \to \square \left( \begin{array}{c} (miso\_{\pi} \wedge \neg sclk\_{\pi} \wedge \neg ss\_{\pi}) \\ \leftrightarrow \\ (miso\_{\pi'} \wedge \neg sclk\_{\pi'} \wedge \neg ss\_{\pi'}) \end{array} \right)$$

This formula (called SPI-correct in Table 1(b)) ensures that for all pairs of traces π and π that agree on the initial configuration, on the input, and additional *SPI input assumptions*, there is a trajectory that aligns their relevant behavior. We consider it relevant that both secondaries agree on their *miso* output whenever they are enabled and the *sclk* is low. Checking *miso* only when the *sclk* is low is sufficient as changes on *miso* only occur at falling edges of the *sclk*. The *SPI input assumptions* are required to guarantee the implicit assumptions of the protocol, for example, that the *sclk* behaves as an infinitely ticking clock. By introducing additional variables and applying logical transformations, we obtain an equivalent formula that syntactically lies in the fragment of the stuttering construction. Again, we reduce this model checking problem to the synchronous semantics and use MCHyper to perform the verification.

In a second experiment, we modified the system to send the value only once and checked it for termination insensitive noninterference SPI-term (see Sects. 3.2 and 4.3). In our setup, we use the variable *term* to flag that the secondary has sent the full value. In the premise of the formula, we require that the input value is equal on both traces and again assume that the inputs conform to the SPI protocol. The conclusion checks if both secondaries have sent the same values by using additional variables that are set together with *term*. The results of this case study are summarized in Table 1(b).

#### **7 Related Work**

The study of specific hyperproperties, such as noninterference, dates back to the seminal work by Goguen and Meseguer [14] in the 1980s. The first systematic study of hyperproperties is due to Clarkson and Schneider [8].

It is well-known that classic specification languages like LTL cannot express hyperproperties. There are two principal methods with which the standard logics have been extended to express hyperproperties:

– The first method is the quantification over variables that identify specific paths or traces. The temporal logics LTL, CTL<sup>∗</sup> have been extended with quantification over traces and paths, resulting in the temporal logics Hyper-LTL and HyperCTL<sup>∗</sup> [7]. There are also extensions of the μ-calculus, most recently, the temporal fixpoint calculus H<sup>μ</sup> [15], which extends the linear time μ-calculus [3] with path quantifiers and indexed next operators.

– The second method is the addition of the equal-level predicate E to first-order and second-order logics, like MPL, MSO, FOL, and S1S, which results in the logics FOL[E], S1S[E], MPL[E], MSO[E] [9,13].

HyperCTL∗, MPL[E], and MSO[E] are branching-time logics, we therefore focus in the following on the linear-time logics HyperLTL, Hμ, FOL[E], and S1S[E]. Among these logics, HyperLTL is the only logic for which practical model-checking algorithms are known [10,11,17]. For HyperLTL, the algorithms have been implemented in the model checkers MCHyper and bounded model checker HyperQube. As discussed in this paper, HyperLTL is limited to synchronous hyperproperties.

FOL[E] can express a limited form of asynchronous hyperproperties. As shown in [9], FOL[E] is subsumed by HyperLTL with additional quantification over predicates. Using such predicates as "markers," one can relate different positions in different traces. However, only a finite number of such predicates is available in each formula. S1S[E] is known to be strictly more expressive than FOL[E] [9], and conjectured to subsume H<sup>μ</sup> [15]. For S1S[E] and Hμ, the model checking problem is in general undecidable; for Hμ, two fragments, the ksynchronous, k-context bounded fragments, have been identified for which model checking remains decidable [15]. Even though some asynchronous properties can be expressed in these decidable fragments of Hμ, there is no systematic study to characterize practical properties that can be encoded. Like S1S[E] and Hμ, asynchronous HyperLTL has an (in general) undecidable model checking problem. However, in this paper we have identified decidable fragments of asynchronous HyperLTL that can express observational determinism, noninterference, and linearizability. A-HLTL is thus the first logic for hyperproperties that can express the major asynchronous hyperproperties of interest within decidable fragments. Furthermore, asynchronous HyperLTL is the first logic for asynchronous hyperproperties with a practical model checking algorithm.

#### **8 Conclusion**

We have introduced A-HLTL, a temporal logic to describe asynchronous hyperproperties. This logic extends HyperLTL with *trajectory* modalities, which control when a trace proceeds and when it stutters. Synchronous HyperLTL corresponds to a trajectory that always moves all paths in a lock-step manner. This notion of trajectory allows to define formulas that are invariant under stuttering, paving the way for relevant model-checking optimizations such a partial order reduction and abstraction-refinement techniques in the context of hyperproperties. We show that model-checking A-HLTL formulas is in general undecidable, and identify two fragments of A-HLTL formulas, which cover a rich set of security requirements and can be decided by a reduction to HyperLTL model-checking. This in turn has allowed us to the reuse the existing model-checker MCHyper.

Future work includes the study of larger decidable fragments (that encompass both fragments studied here), extending the logic allowing several trajectory modalities, as well as their implementation in practical tools. Extending bounded model-checking [17] to A-HLTL is another interesting research problem. Asynchronous hyperproperties are important for applying a logic-based verification approach to verify hyperproperties for *software* programs, because the relative speed of the execution of programs depends on many factors like the compiler, hardware, execution platform and concurrent running programs, that the analysis must tolerate. Therefore, future work includes adapting techniques for infinite-state software model-checking, like deductive methods, abstraction, etc. to verify A-HLTL properties of software systems.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Product Programs in the Wild: Retrofitting Program Verifiers to Check Information Flow Security**

Marco Eilers(B) , Severin Meier, and Peter M¨uller

Department of Computer Science, ETH Zurich, Zurich, Switzerland *{*marco.eilers,peter.mueller*}*@inf.ethz.ch

**Abstract.** Most existing program verifiers check trace properties such as functional correctness, but do not support the verification of hyperproperties, in particular, information flow security. In principle, product programs allow one to reduce the verification of hyperproperties to trace properties and, thus, apply standard verifiers to check them; in practice, product constructions are usually defined only for simple programming languages without features like dynamic method binding or concurrency and, consequently, cannot be directly applied to verify information flow security in a full-fledged language. However, many existing verifiers encode programs from source languages into simple intermediate verification languages, which opens up the possibility of constructing a product program on the intermediate language level, reusing the existing encoding and drastically reducing the effort required to develop new verification tools for information flow security.

In this paper, we explore the potential of this approach along three dimensions: (1) Soundness: We show that the combination of an encoding and a product construction that are individually sound can still be unsound, and identify a novel condition on the encoding that ensures overall soundness. (2) Concurrency: We show how sequential product programs on the intermediate language level can be used to verify information flow security of concurrent source programs. (3) Performance: We implement a product construction in Nagini, a Python verifier built upon the Viper intermediate language, and evaluate it on a number of challenging examples. We show that the resulting tool offers acceptable performance, while matching or surpassing existing tools in its combination of language feature support and expressiveness.

#### **1 Introduction**

Since computer programs increasingly handle sensitive user data and communicate using encryption, it is vital that programs do not leak secret data such as private keys to attackers, that is, that they are *information flow secure*. One way of formalizing information flow security is *noninterference*, a so-called *2 hyperproperty*, i.e., a property of pairs of executions of the program.

Noninterference can be checked by type systems [45] and static analyses [22]. However, complex language features (such as concurrency) and noninterference properties (such as termination sensitivity) generally require the expressiveness of deductive verification. In recent years, many automated and expressive verification tools have been developed for a wide range of programming languages, but most of these tools are limited to trace properties (properties of single program traces) and cannot prove hyperproperties such as noninterference.

The problem we address in this paper is how to retrofit existing program verifiers to check noninterference. Compared to building noninterference verifiers from scratch, which can take years when targeting substantial subsets of real-world programming languages, this approach would allow us to reuse most aspects of existing verifiers, such as the semantic representation of language features and proof search algorithms. Moreover, it naturally allows one to verify combinations of correctness and noninterference properties.

In principle, existing program verifiers can be used to verify hyperproperties by reducing them to trace properties via self-composition [6] or product programs [4,5]. However, self-composition does not allow modular verification [48], and product programs have generally been defined only for simple languages without features like dynamic method binding or concurrency [4,18]. Applying product constructions to programs written in complex languages would therefore require defining and implementing new and complex product constructions for every new verifier.

We explore a more efficient approach here: We leverage the fact that most automatic deductive verifiers are organized into a custom frontend, which encodes a source program into an intermediate verification language (IVL), and a reusable backend, which verifies the IVL program using generic proof search engines. Boogie [3], Viper [35], and Why3 [21] are examples of such IVLs, which power a large number of program verifiers; for instance Boogie is used by Dafny [29], VCC [13], Spec# [30], and GPUVerify [8], Why3 [21] by Frama-C [14] and Krakatoa [20], and Viper [35] by Vercors [10], Prusti [2], and Nagini [17]. The ubiquitiy of this architecture offers a chance to retrofit existing verifiers to check noninterference by performing the product construction on the level of the IVL (an approach that is already used by SymDiff [27] for the related problem of program equivalence). The resulting architecture, which allows one to reuse both the frontend and the backend of the existing verifier, is shown in Fig. 1.

Performing the product construction on the IVL-level has three major advantages over a product construction on the source program: (1) It cleanly separates the encoding of the source language (which tends to be complex for full-fledged languages) from the product construction. (2) The product construction is much simpler since IVLs are small, sequential languages. (3) The product construction can be reused across all verifiers built on the same IVL. Overall, this architecture therefore has the potential to make existing verifiers information flow aware with substantially less effort than building a new tool from scratch.

Even though this approach has strong advantages, there are several open questions that must be addressed to make it useful and widely applicable:

**Fig. 1.** Proposed architecture for information flow verifiers. The existing encoding from source to IVL (frontend) as well as the proof search (backend) can be reused. The product construction needs to support only the (relatively small) IVL and can be reused across different verifiers.


In this paper, we answer these three questions. We focus our investigation on modular product programs [18], a kind of product program that allows modular verification and is well-suited for precise specification and verification of information flow security. We make the following contributions:


These results demonstrate that the proposed approach can indeed be used to retrofit an existing verifier to *soundly* check information flow security, even for concurrent programs. The resulting tool, made with only a fraction of the effort required for the development of a new verifier, can compete with custom-made tools in its expressiveness at an acceptable performance cost.

#### **2 Preliminaries**

In this section, we introduce the necessary background about noninterference and product programs.

#### **2.1 Noninterference**

A common way of formalizing information flow security is *noninterference* [23]. Informally, noninterference specifies that the secret (or *high*) inputs of a program do not influence the values of its public (or *low*) outputs. We will not define a formal semantics here, but just assume that there is a steps-to relation s, σ→s , σ that relates program configurations consisting of a store σ and a statement s.

We formalize noninterference as a property of pairs of program executions (that is, a *2-hyperproperty* [12]) as follows:

**Definition 1.** *A program* s *with a set of input variables* I *and output variables* O*, of which some subsets* I*<sup>l</sup>* ⊆ I *and* O*<sup>l</sup>* ⊆ O *are low, satisfies noninterference iff for all* σ1, σ<sup>2</sup> *and* σ 1, σ <sup>2</sup>*, if* ∀x ∈ I*l*. σ1(x) = σ2(x) *and* s, σ1 →<sup>∗</sup> skip, σ 1 *and* s, σ2 →<sup>∗</sup> skip, σ <sup>2</sup> *then* ∀x ∈ O*l*.σ <sup>1</sup>(x) = σ <sup>2</sup>(x)*.*

Note that in this definition (and throughout this paper unless stated otherwise), we do not consider non-terminating executions, i.e., we focus on verifying termination-insensitive noninterference.

#### **2.2 Modular Product Programs**

Proving hyperproperties requires reasoning about multiple (here, two) executions of a program. However, hyperproperties can be reduced to properties of a single execution by using *self-composition* [6] or *product programs* [4]. The idea is to duplicate a program's state space by creating two renamed copies of all variables, one for each execution (we write x(*i*) for the ith renaming of variable x, and lift this notation to expressions), and to transform each statement so that it has the effect of the original statement on both copies of the state. Unlike selfcomposition, which achieves this effect by simply duplicating every statement, *modular product programs* [18] do not duplicate loops and method calls, and instead encode differing control flow through *activation variables*, which represent, for each execution, whether or not it is active (i.e., it executes the code) at the current point in the program. This approach results in a structural alignment


**Fig. 2.** A modular product program (on the right) of the program on the left.

of both program executions, which allows one to use method specifications and loop invariants that relate both executions, as we discuss below. We denote the product of statement s under activation variables p1 and p2 as s˚*<sup>p</sup>*.

Figure 2 shows an example program and the respective product program. For both functions, the product program duplicates the parameters of the original function and adds boolean activation variables p1 and p2. Control structures like conditionals are encoded by creating a set of new activation variables (lines 4–7). For example, p1t represents whether the first execution is active in the thenbranch of the conditional, which is the case if it was active at the beginning of the function and the if-condition is true for the first execution. Conversely, p2e represents whether the second execution is active in the else-branch. Primitive statements like assignments are then executed under the condition that their execution is active at the current point in the execution (lines 8–11). Crucially, the method call to bar is *not* duplicated; it is executed if at least one execution is active at the call site, and the values of the current activation variables are passed to the function, meaning that if an execution is inactive at the call site, no state changes will be performed for that execution in the called method.

Because a single method call in the product represents the calls from both executions, one can reason about method calls modularly in terms of *relational* specifications, i.e., specifications that relate behavior of two executions of the method, as opposed to *unary* specifications that describe only a single execution. Relational specifications are encoded as ordinary specifications in the product program that relate parameters from the two different executions.

As an example, assume that bar prints the value of its input z, which must therefore be low. We can express this as a (relational) precondition *low*(z), which can be encoded as the precondition p1 <sup>∧</sup> p2 <sup>⇒</sup> z1 <sup>=</sup> z2 in the product of bar.

Events the attacker can observe (such as I/O) must not happen depending on a secret, to avoid leaking secret data. It is, thus, useful to express in specifications that the control flow at the current program point is low, i.e., whether the current statement is executed does not depend on secret data. This property is denoted in specifications as *lowEvent*. We generally write <sup>P</sup>˚*<sup>p</sup>* for the encoding of assertion P under activation variables p1 and p2; *lowEvent*˚*<sup>p</sup>* is then defined as p1 <sup>=</sup> p2.

A unary (that is, non-relational) predicate Q, such as a standard method pre- or postcondition, is encoded in the product program as applying to each active execution, i.e., <sup>Q</sup>˚*<sup>p</sup>* is defined as p1 <sup>⇒</sup> <sup>Q</sup>(1) <sup>∧</sup> p2 <sup>⇒</sup> <sup>Q</sup>(2).

Compared to type systems and taint analyses, verification based on product programs allows for much more precise reasoning. Assume for example that foo's parameter x is high. Nonetheless, we can show that the example does not leak information, since the precondition of bar, *low*(z), will always be fulfilled (y <sup>&</sup>gt; 0 is true independently of the value of x). In contrast, security type systems would flag y as high, since it is assigned to under a high guard, leading to imprecision.

In addition to ordinary noninterference, modular product programs can also be used to encode more advanced security properties, including terminationsensitive noninterference, value-dependent sensitivity [36], and a form of declassification [18].

#### **3 Sound Products of IVL Encodings**

In this section, we address the first question from the introduction, namely, whether we can soundly combine an existing encoding into an IVL with a product construction. We first describe the proposed architecture in greater detail. Then we show a potential soundness issue and define a sufficient criterion on the IVL encoding for the entire approach to be sound. Finally, we discuss an example of a common encoding pattern that violates the criterion, show that it is indeed unsound, and propose an alternative sound encoding.

#### **3.1 Proposed Architecture**

The architecture proposed in the introduction (Fig. 1) enables the construction of information flow aware verifiers with relatively little effort, by reusing most of the frontend encoding of the source language to an IVL as well as the entire backend proof search. The only major change that is necessary is that the frontend and potentially the IVL have to be extended to allow for the use of information flow assertions in specifications. Crucially, the frontend does not have to know their meaning; it can treat relational source-level assertions like *low*(e) like ordinary unary predicates and simply translate them to their counterparts on the IVL level. IVL-level relational assertions will then be translated to ordinary assertions during the product transformation.

In the remainder of this paper, we will generally assume that the existing IVL encoding is used unchanged, and point out when changes need to be made.

#### **3.2 Soundness Issue**

Surprisingly, combining a sound encoding from source language to IVL with a sound IVL-level product construction may result in a verification technique that is *unsound* in the presence of relational specifications. Consider the source program in Fig. <sup>3</sup> (left), where P is some predicate.

**Fig. 3.** Example of an encoding that is unsound in our setting. The program on the left can be encoded into a conditional statement (identical to the source program, modulo language syntax) or to the program on the right; the latter leads to unsoundness if P is a relational predicate.

A frontend could encode the body of foo into an identical (modulo syntax) conditional statement on the IVL level (assuming the IVL provides conditionals, assignments, and assert statements). Alternatively, it could produce the encoding shown in Fig. 3 (right), which directly asserts a sufficient precondition of the source program. If P is a unary predicate, both encodings are sound: If they verify, the original program is correct. However, if P(y) is a relational predicate, for instance, *low*(y), then the encoding on the right is *unsound*: *low*(5) and *low*(7) are trivially true (since 5 = 5 and 7 = 7), so the assertion in the encoded program trivially passes, yet the original program is clearly incorrect: If x is greater than 7 in one execution but less in the other, y will have different values in both executions, and will therefore not be low.

The underlying reason is that the encoding on the right does not encode the exact behavior of the source program; it encodes a verification condition computed by the frontend that is sound if assertions are unary, but may not be sound for relational assertions.

We will now (1) formalize this intuition and derive a sufficient condition for the soundness of an encoding in this approach, and (2) show an example of this problem occurring in real frontends, and describe how it can be solved.

#### **3.3 Soundness Criterion**

We write Σ and S for states and statements of the source language, and σ and s for states and statements of the IVL. States may contain, for example, a mutable heap and a variable store. For simplicity, we assume that both source and IVL statements contain a statement skip that represents a finished computation. We also assume that there is a small-step transition relation → for both languages, and that the standard notion of Hoare triple validity - {P}s{Q} is defined for the IVL. We let P and Q range over (source and IVL level) assertions from a standard assertion language extended with *low*(e) and *lowEvent*, and assume a standard definition of assertion validity for pairs of states.

We define an encoding to be a triple α, ∼=, β, where α : S → s is an encoding from source statements to statements of the target language (i.e., the IVL), β similarly encodes assertions to the target language, and ∼= relates source language states to corresponding target language states.

We first define the desired relational soundness property, which expresses that if an encoded Hoare triple holds for the encoded program, then the original property holds for all pairs of executions of the source program:

**Definition 2.** α, ∼=, β *is* relationally sound *iff, for all* S, Σ1, Σ2, Σ 1, Σ <sup>2</sup>, P, Q*, if* - { <sup>β</sup>(P)˚*p*}<sup>α</sup>(S)˚*p*{ <sup>β</sup>(Q)˚*p*} *and* <sup>Σ</sup>1, Σ<sup>2</sup> - P *and* -S, Σ1 →<sup>∗</sup> skip, Σ 1 *and* -S, Σ2 →<sup>∗</sup> skip, Σ <sup>2</sup>*, then* Σ 1, Σ <sup>2</sup> -Q*.*

Product programs represent the operational behavior of two program executions by the operational behavior of a single product program execution. The unsoundness shown before is caused by the fact that the encoding into the IVL does not reflect the operational behavior of the conditional statement (replacing it by an assertion of a sufficient precondition) and, thus, the resulting product does not soundly reflect two executions of the source program.

We call an encoding that preserves the operational behavior of the source program *operational*: It encodes every step of the source program into some number of steps of the target program so that c initial states result in matching end states. Similarly, it encodes specifications from the source level into targetlevel specifications that hold in matching states. We can formalize this intuition by requiring that the source and target programs are connected by the simulation relation ∼=:

**Definition 3.** α, ∼=, β *is an* operational *encoding if: (1) for all* Σ,Σ , σ, S, S *, if* -S, Σ→-S , Σ *and* Σ ∼= σ*, then* α(S), σ →<sup>∗</sup> α(S ), σ *for some* σ *s.t.* Σ ∼= σ *, and (2) if* Σ ∼= σ *then* Σ - P *iff* σ β(P)*.*

Note that this notion allows the encoding to overapproximate the behaviors of the source program, i.e., admit steps that are not possible on the source level, but not vice versa.

For the example in Fig. 3, it is easy to see that this criterion is fulfilled by the left encoding: the source and IVL programs are identical (modulo syntax), matching states are identical states (modulo state encodings), and the behavior of both programs is identical. The encoding on the right, however, is *not* operational: While the left program modifies the state, the right program never performs any state modification.

We now show that operationality is sufficient for relational soundness:

#### **Theorem 1.** *If* α, ∼=, β *is operational then it is relationally sound.*

Note that operationality is a sufficient but not necessary condition; encodings of verification conditions may be sound for relational verification as well. The main advantage of applying the operationality criterion instead of directly reasoning about relational soundness is that, since operationality represents the simple notion that the IVL program performs equivalent steps and equivalent state changes to the source program, it is intuitive and easy to check whether a given encoding is operational. Additionally, some encodings (like the one Vercors uses for parallel blocks) are not operational, but can be seen as simplified versions of a possible operational encoding that generate the same proof obligations; these can also be quickly identified as relationally sound.

#### **3.4 Practical Relevance**

In most existing frontends, the encoding of virtually all source language constructs is operational; the main appeal of IVLs is, after all, that frontends *do not* have to compute verification conditions, but can instead "compile" input programs into an IVL without worrying about the verification process itself. However, many frontends still use non-operational encodings at least for *some* language constructs. Examples for this are VCC's encoding of local blocks, Dafny's encoding of calls on traits, Prusti's encoding for loops, and Nagini's encoding of dynamically-bound calls, which we will discuss in detail in the next subsection. Additionally, as we will discuss in Sect. 4, all encodings of concurrent source languages into sequential IVLs necessarily have some non-operational elements.

Where non-operational encodings are used, this is often intentional to enable modular verification, since operational encodings for some language constructs are inherently non-modular (see the example in the next subsection). In practice, one can therefore use the operationality criterion to quickly check that the existing encoding is sound for the vast majority of source language statements, and subsequently check the few remaining ones for relational soundness in detail.

#### **3.5 Example: Dynamically-Bound Calls**

In this section, we show a real example of an unsound encoding of dynamicallybound calls that violates the operationality criterion, and show how to derive a sound alternative.

Statically-bound method calls, i.e., calls whose exact target is fixed at compile time, can be encoded as procedure calls on the IVL level, which yields an operational encoding if the operational semantics of the IVL treats calls analogously to the source semantics. The IVL verifier might later reason about calls in terms of pre- and postconditions instead of actually performing a call, but this transformation is not relevant here as long as the product program is constructed *before* such a desugaring step.

However, the same approach does not work for dynamically-bound calls, i.e., calls whose target is chosen at runtime based on the type of the call's receiver. Since the implementation to be executed is generally not known during modular verification, it is not possible to encode dynamically-bound calls as procedure calls with the usual operational semantics (and existing IVLs do not offer dynamically-bound calls). Therefore, dynamically-bound calls are typically (e.g., in Dafny and Nagini) directly encoded using the method specification. Additional, separate proof obligations enforce that all overrides of a method respect *behavioral subtyping* [33], i.e., live up to the specification of the overridden method.

Consider method A.foo in Fig. <sup>4</sup> (left), which returns a constant integer and guarantees in its postcondition that the result is low. A dynamically-bound call a. foo(), where a has the static type A, will be encoded as an assertion of the (here, trivial) precondition of A.foo, followed by an assumption of the postcondition (we ignore side effects here for simplicity).


**Fig. 4.** Example of a problematic method override. B.foo overrides A.foo and has a compatible specification, but the implementations return different values.

This encoding is sound if foo has a purely unary specification, without any relational parts. However, it does *not* fulfill our operationality criterion: The semantics of the source program performs a call to an implementation of foo (selected based on the dynamic type of a), whereas the IVL encoding directly encodes the proof obligations (similarly to the example from Fig. 3).

Since the encoding is not operational, we have to check whether it is still relationally sound. Method B.foo in Fig. <sup>4</sup> (right), which overrides A.foo, shows that it is not. B.foo's contract is identical with that of A.foo, so behavioral subtyping holds trivially. B.foo's implementation satisfies the contract because it also returns a constant (but, importantly, a different one). Now, if a client calls a. foo() and, depending on a secret, the dynamic type of a is either A or B, then, depending on the secret, the result will be either 0 or 1. With the standard encoding of dynamically-bound calls outlined above, however, the client will assume the postcondition of A.foo and will therefore incorrectly conclude that the returned result is low.

To avoid this unsoundness while retaining the ability to use relational specifications<sup>1</sup>, the problematic encoding must be replaced, either with an operational one, or with a different non-operational encoding that is sound for relational specifications. The former option is not applicable here: An operational encoding for dynamically-bound calls would essentially have to case split on the dynamic type of the receiver and invoke the appropriate override. Since such an encoding is inherently non-modular (all possible overrides need to be known), we follow the alternative option: we give an example of a non-operational, but sound encoding.

For our new encoding we exploit the fact that the standard encoding is unsound only if the two executions of the program resolve the dynamicallybound call to two different implementations, that is, if the dynamic types of the receiver differ in the two executions. We reflect this observation by adjusting the encoding of pre- and postconditions as follows: (1) If the postcondition of a method guarantees that an expression is low, we assume this at the call site *only if* the dynamic type of the receiver is also low, that is, the calls in the two program executions are resolved to the same implementation. (2) Similarly, if a precondition requires that the call is a low event, we enforce that the receiver type is low in addition to the usual criterion for low events. Low events typically perform observable behavior such as I/O; it is therefore important that the *same* observable behavior is produced, independent of the receiver type. The meaning of low-assertions in preconditions remains unchanged, because the requirement of

<sup>1</sup> One could, of course, forbid the use of relational specifications in some places to trivially avoid the unsoundness; this, however, is typically not useful in practice.

a method to receive low arguments is independent of the invoked implementation and must, thus, not be weakened. *lowEvent*-assertions are generally not allowed in postconditions, where they add no expressiveness.

We encode this adjustment as follows:

$$\begin{aligned} \lceil low(e) \rceil\_{post\_r}^{\dot{p}} &= (p^{(1)} = p^{(2)} \land type(r^{(1)}) = type(r^{(2)})) \Rightarrow e^{(1)} = e^{(2)} \\ \lceil lowEvent \rceil\_{pre\_r}^{\dot{p}} &= p^{(1)} = p^{(2)} \land type(r^{(1)}) = type(r^{(2)}) \end{aligned}$$

where *type*(e) represents the dynamic type of expression e, P ˚*p post<sup>r</sup>* is the encoding of P in the postcondition of a call with receiver r, and <sup>P</sup>˚*<sup>p</sup> pre<sup>r</sup>* represents the same for the precondition. We leave the remaining encoding untouched, meaning that we can summarize the resulting encoding as follows:


In the example above, this encoding lets the caller assume that the result is low only if it can prove that the dynamic type of a is low.

The adjusted encoding is indeed sound:

**Theorem 2.** *Let* S*<sup>c</sup> be of the form* x:=r.m()*, where* r *has static type* A*, and let* pre*A.m and* post*A.m be the pre- and postcondition of* A.m*. Assume that the implementation of A.m and its overrides fulfill their specifications and satisfy behavioral subtyping. Then the described encoding of* S*<sup>c</sup> is relationally sound.*

Note that this encoding is incomplete, since it is not aware that two different receiver types can lead to the same implementation being called (e.g., if one type inherits from the second and does not override the called method). Alternative encodings could explicitly represent this possibility. Conversely, one could approximate further (while remaining sound) by requiring the receiver *values* to be low, not just their types, in encodings that do not model dynamic types.

#### **4 Product Programs and Concurrency**

Automated verification of information flow security for concurrent programs is challenging because one needs to reason about a pair of executions that may have different thread interleavings. In fact, we are aware of only one tool that currently allows this, SecC, which automates SecCSL, a concurrent separation logic for information flow security proofs [19]. A product construction applied directly to concurrent programs would have to faithfully represent all combinations of potential thread interleavings, which makes verification infeasible. Consequently, to the best of our knowledge, no such product construction exists.

For trace properties, many existing verifiers avoid reasoning about all possible thread interleaving by employing a program logic (such as concurrent separation logic [38]) that essentially reduces verification to sequential reasoning and allows concurrent verification problems to be encoded into sequential IVLs. Examples for such verifiers include Vercors and Nagini (using the Viper IVL), as well as Chalice [32], VCC, and Spec# (using the Boogie IVL).

In this section, we show how to use IVL-level product programs to extend such verifiers to handle information flow. We first describe how existing IVL encodings for concurrent languages work, and subsequently show how we can use similar principles to apply an IVL-based product construction, and which additional proof obligations we must fulfill to ensure that no flows exist as a result of concurrency. We will do this for two different notions of information flow security for concurrent programs, *possibilistic* and *probabilistic* noninterference; however, the principles behind the approach may also extend to alternative notions of information flow security such as observational determinism [49].

Our goal is to describe a technique that applies to a wide range of source languages, IVLs, proof techniques, and encodings. Therefore, we focus on the high-level concepts, instead of formalizing them for one specific setting.

#### **4.1 Concurrent IVL Encodings**

Since all IVLs we are aware of are sequential languages, encodings from concurrent source languages to IVLs do not model the exact behavior of the original language, in particular, the aforementioned thread interleavings (i.e., these encodings are non-operational). Instead, they encode a verification condition that ensures that the original program is correct for *every possible* thread interleaving.

While the exact proof techniques differ between frontends, and can be based for example on Concurrent Separation Logic (CSL) [38] or ownership [13,24,26], they generally follow a common pattern [31]: They prove that the source program is data race free, which ensures that thread interactions need to be considered *only* at well-defined synchronization points, for instance, upon acquiring or releasing a lock. The code between such interaction points can be considered to execute without interference from other threads, and thus can be reasoned about as if it were sequential.

We focus on locks here, but other synchronization primitives are handled analogously. Program logics based on CSL or ownership systems formally connect a lock and the heap locations it protects, such that these locations may be accessed only while holding the respective lock. In addition, they associate locks with an invariant that constrains the values of the heap locations it protects. When acquiring a lock, a thread may assume that this lock invariant holds, and when releasing a lock, it has to prove that the invariant is reestablished. A frontend can encode this into an IVL as depicted in Fig. 5.

Our solution for information flow verification in concurrent programs follows the same basic approach: We exploit that code between lock operations can be considered to execute without interference, and that we can therefore use

**Fig. 5.** Standard IVL encoding of lock operations. *Inv*( l ) denotes the invariant constraining the memory protected by lock l.

ordinary sequential product programs to reason about this code. To capture the thread interactions at synchronization points, we extend lock invariants to contain relational assertions (which can prescribe that some values protected by the lock are low), and add additional checks around lock operations to ensure that they do not give rise to unwanted information flow.

#### **4.2 Possibilistic Noninterference**

For concurrent programs, standard noninterference is too strict a property because concurrent programs are usually non-deterministic. One way of approaching this problem is to instead verify *possibilistic noninterference*, which enforces that high information does not influence the *possible* values of low outputs, i.e., if some combination of low output values is reachable from an initial state, then the same combination of low output values must still be reachable using *some* possible thread schedule after arbitrarily changing the high inputs. Possibilistic noninterference can be defined as follows:

**Definition 4.** *A program* s *with a set of input variables* I *and output variables* O*, of which some subsets* I*<sup>l</sup>* ⊆ I *and* O*<sup>l</sup>* ⊆ O *are low, satisfies possibilistic noninterference iff for all* σ1, σ<sup>2</sup> *and* σ <sup>1</sup>*, if* ∀x ∈ I*l*. σ1(x) = σ2(x) *and* s, σ1 →<sup>∗</sup> skip, σ <sup>1</sup> *then* s, σ2 →<sup>∗</sup> skip, σ <sup>2</sup> *for some* σ <sup>2</sup> *s.t.* ∀x ∈ O*l*.σ <sup>1</sup>(x) = σ <sup>2</sup>(x)*.*

Note that this property allows high inputs to influence the *probability* of different outputs and may therefore not be desirable in all scenarios; we discuss a stronger notion of noninterference in the next subsection.

Since we build on a proof technique that ensures data race freedom, we can see each program trace as a sequence of local operations and lock operations by specific threads, where (1) every local operation depends only on previous (local or lock) operations of the same thread, and (2) every lock operation depends only on the previous local operations of the same thread and all previous lock operations (of arbitrary threads). As a result, we can (akin to partial order reduction) rearrange segments freely as long as we retain the overall order of lock operations and the order of operations of every specific thread; in particular, we can rearrange a trace so that it consists of a number of *segments*, such that in each segment, one thread executes any number of local operations and then one lock operation.

**Fig. 6.** Statement encoding for possibilistic information flow security. For loops, we check that the loop guard is low, ensuring that termination is also low.

Based on this observation, we impose proof obligations that ensure the following property: For every program trace with some schedule and some low and high inputs, and for arbitrary different high inputs, there exists a second trace such that: (1) Both traces include the same lock operations performed by the same threads, in the same order, and (2) at each lock operation, the lock's invariant holds; in particular, the relational assertions of the lock invariant correctly relate the state protected by the lock in both traces.

To enforce this property, we devise four proof obligations that can be checked thread-locally:


Note that, even though the lock operations of both traces are closely aligned, their local operations may differ. For instance, a thread may branch on a high guard as long as no lock operation is performed before the control flow re-joins.

The above checks are sufficient to satisfy Definition 4. The proof goes by induction on the number of segments of the traces and leverages the soundness of sequential verification within each segment.

**Encoding.** The aforementioned checks can simply be checked as part of the encoding of lock operations. We adjust the encoding from Fig. 5 for possibilistic noninterference as shown in Fig. 6. For thread acquire and release, the assertions of *lowEvent* and *low*(l) directly ensure properties (1) and (3). Assuming and asserting the lock invariant works as in the standard IVL encoding for concurrent programs, but now this invariant can be relational, ensuring property (4). The condition on while loops is used to ensure property (2), which can be done simply

**Fig. 7.** Possibilistic information flow violation via a secret-dependent lock release. The Cell state 4 is visible to other threads only if secret is True.

by asserting that the loop condition is low for every loop in the program (we assume, for simplicity, that there is no infinite recursion).

**Discussion.** With our verification technique, the product construction on the IVL level does not need to be aware of concurrency in *any* way; applying the standard sequential product construction to the updated encoding is sufficient to ensure possibilistic noninterference in concurrent programs.

To the best of our knowledge, we are the first to consider possibilistic information flow in a setting with locks, and therefore the first to propose that the order of lock operations must be constrained. The example in Fig. 7 demonstrates that this requirement is indeed necessary to prevent unwanted information flow: The CellLock protects the val field of a Cell object, which is intended to be low. The code unconditionally sets the field to two constants (first to 4, then to 5), which should be allowed since the constants are low. However, whether the lock is *released* while the cell has value 4 depends on a secret. As a result, when a different thread acquires the lock and sees that the value is 4, this leaks that the secret *must* have been true.

Another example that illustrates the requirement to ensure that high data does not influence *which* lock a lock operation accesses can be found in Fig. 8. Here, two locks are created, and thread 1 acquires the first one. Thread 2 acquires, depending on the secret, either the same lock or a different one. This influences the possible results of the program: If both threads acquire the same lock, then the **print** statements of one thread cannot be interleaved with those of the other, otherwise they can. As a result, if the attacker observes the pattern 1212 (or any other interleaving of 1s and 2s), they know with certainty that the two threads acquired different locks and secret must therefore be False.

The necessity to prevent termination differences in a concurrent setting has been recognized before in work on security type systems [45].

**Fig. 8.** Possibilistic information flow violation through locks. If secret is true, both threads acquire the same lock, and their critical sections cannot be interleaved.

**Fig. 9.** Example of probabilistic information flow. With a non-deterministic scheduler, secret does not influence the set of possible outputs, but a greater secret leads to higher probability of seeing a final cell value of 2.

#### **4.3 Probabilistic Noninterference**

Possibilistic noninterference is too imprecise for many applications. Figure 9 illustrates the problem: The final value of c. val can be either 1 or 2, that is, possibilistic noninterference holds. However, with most schedulers, a final value of 2 is much more likely for greater secret values than for lower values because the assignment of 1 is more likely to happen before the assignment of 2.

A stronger notion of noninterference that forbids such leaks is *probabilistic* noninterference, which requires that two executions from low-equivalent initial states will produce the same low outputs with the same probabilities.

**Definition 5.** *A program* s *with a set of input variables* I *and output variables* O*, of which some subsets* I*<sup>l</sup>* ⊆ I *and* O*<sup>l</sup>* ⊆ O *are low, satisfies probabilistic noninterference iff for all* σ1, σ<sup>2</sup> *and* σ <sup>1</sup>*, if* ∀x ∈ I*l*. σ1(x) = σ2(x) *and* s, σ1 →<sup>∗</sup> skip, σ <sup>1</sup> *with probability* p *then* s, σ2 →<sup>∗</sup> skip, σ <sup>2</sup> *with probability* p *for some* σ <sup>2</sup> *s.t.* ∀x ∈ O*l*.σ <sup>1</sup>(x) = σ <sup>2</sup>(x)*.*

The information flow in Fig. 9 is caused by secret data influencing the timing of thread 2, which in turn may affect the relative order of modifications of shared variables. To prevent secrets from influencing the timing of operations, we additionally assert that every branch condition in the program is low, meaning that the two executions will always follow the same code path, which leads to the adjusted encoding in Fig. 10. Note that the check that branch conditions are low must also be performed for any implicit branches; e.g., with the encoding of

**Fig. 10.** Statement encoding for probabilistic information flow security.

dynamically-bound calls shown before, we must now assert that the type of the receiver of every such call is low. Also note that since we enforce that branches are low, the *lowEvent* conditions we showed in the possibilistic encoding will be trivially fulfilled and can be omitted here. However, we still need to assert that acquired and released lock references are low. This last requirement has not been discussed in previous work (whereas forbidding high branches is standard practice in type systems and program logics [36]).

With this adjusted encoding, probabilistic noninterference can be verified using simple assertions in the IVL encoding and subsequently performing a standard product construction on the IVL level. So, in summary, this approach lets us extend existing verifiers for concurrent programs to verify both possibilistic and probabilistic noninterference with very small changes in the frontend, and without requiring any changes on the level of the IVL (except the ability to write relational specifications) and the product construction.

#### **5 Implementation and Evaluation**

In this section, we evaluate the performance of the proposed architecture, by extending the previously information flow unaware Nagini verifier for Python [17] according to our design. We will first briefly describe Nagini and the adaptations we needed to make, then evaluate the performance overhead generated by the product transformation, and subsequently evaluate the implementation on a number of information flow examples, comparing it to SecC [19] in the process.

#### **5.1 Nagini**

Nagini is an automated verifier for statically-typed Python 3 programs. It supports a large subset of the Python language, comprising features like exception handling, polymorphism, dynamic field creation, and concurrency. Reasoning about some of these features is quite intricate even without the overhead of a product construction, so we believe that Nagini is a good target to evaluate the performance of the proposed architecture for verifiers for complex languages.

Nagini encodes Python programs and their specifications into the Viper IVL [35], and then uses Viper's backend verifiers to automatically verify those programs using the Z3 SMT solver [34]. For concurrent programs, Nagini uses an encoding similar to the one described in Sect. 4, using implicit dynamic frames [44] (a flavor of separation logic [38,40]) to prove data race freedom; as a result, we could modify its existing encoding as shown in Sect. 4 to prove both possibilistic and probabilistic noninterference for concurrent programs. Nagini's existing encoding from Python to Viper is almost entirely operational, we only adapted the encoding of dynamically-bound calls as shown in Sect. 3.5.

We extended Nagini's existing specification language to include information flow specifications and implemented the modular product program transformation for 2-hyperproperties for the existing Viper AST (enriched, again, with new AST nodes for information flow specifications). For convenience, we also slightly extended the Viper-based product transformation to directly transform statements that Nagini previously encoded using gotos, such as **break** and **continue** statements. The Viper extension for product programs<sup>2</sup> and the extended version of Nagini<sup>3</sup> are open source and available online.

#### **5.2 Performance Overhead of the Product Construction**

Our first goal is to evaluate the performance overhead generated by the product construction. We compared the verification times of Nagini's entire functional test suite with and without the product transformation enabled. The test cases range from small programs targeting specific language or specification constructs, to realistic code examples taken from programming tutorials. We ran each test five times on a warmed up JVM with the information flow extension enabled and disabled, without adding any information flow specifications. Our test system was a 12 core AMD Ryzen 3900X with 32 GB of RAM running Ubuntu 20.04.1.

All tests report the same results with and without the product transformation, meaning that completeness is not impacted by the extension, and that we can indeed still reason about the entire language subset supported by Nagini. Without the product transformation, each test case takes between 3 and 9 s, with the majority taking between 3 and 5. For most cases, enabling the product construction leads to an increase in verification time that is clearly acceptable (less than 11% for half the tests, less than 30% for three quarters, and less than 100% for 90% of the tests). For five test cases, the slowdown is a factor between 5 and 12, and a single outlier (a quicksort implementation) has a slowdown factor of 17.5 and a resulting verification time of two minutes. We believe that the main reason for the large slowdown for these particular test cases is the use of quantifiers in their specifications (e.g., to specify properties of all elements in a list). Quantifier handling is difficult for automated verification in general, because unbounded chains of quantifier instantiations can occur during the proof search [15], and this problem seems to be exacerbated when using the product encoding.

<sup>2</sup> https://github.com/viperproject/silver-sif-extension.

<sup>3</sup> https://github.com/marcoeilers/nagini.

**Table 1.** Programs evaluated for proving information flow security. We show the total lines of code (LOC) including implementation and specification but excluding whitespace, lines of specification and proof annotation (Ann.), the property we proved (Prop., where NI = noninterference, TNI = termination sensitive noninterference, PS = possibilistic noninterference) and the verification time in seconds (*T*), averaged over five runs.


We conclude that the performance impact of the product transformation is acceptable for most examples, but can be significant for programs with complex functional specifications.

#### **5.3 Expressiveness and Comparison with SecC**

In a second step, we evaluated the expressiveness and performance of our implementation on a number of challenging examples from the literature. In particular, we use the examples from the original paper about modular product programs [18] (sequential examples collected from various previous papers, translated to Python) and from this paper, both shown in Table 1, as well as examples taken from SecC [19], the only other automated verification tool for concurrent programs we are aware of, shown in Table 2. The latter table includes the CDDC case study [36], which models an embedded device that interacts simultaneously with multiple users and classified networks. Our examples represent the state of the art in automated information flow verification, requiring semantic reasoning that would not be possible in a type system, and using complex information flow specifications including declassification, termination-sensitive noninterference, and value-dependent sensitivity [36]. As mentioned before, these features can be easily encoded into modular product programs using existing techniques [18].

Nagini was able to verify all examples, which demonstrates that our approach can handle concurrent implementations and express complex noninterference properties. For the examples from Table 1, Nagini takes only between 3 and 12 s each. As for the tests from SecC, Nagini takes around five seconds for three

**Table 2.** Comparison with SecC. We show the total lines of code and lines of specification for Nagini (LOC*<sup>N</sup>* , Ann*<sup>N</sup>* ) and SecC (LOC*S*, Ann*S*), the property we proved (Prop., where NI = noninterference, PR = probabilistic noninterference) and the verification time in seconds in both tools (*T<sup>N</sup>* and *TS*) and in Nagini *without* the product construction (*TNP* ), averaged over five runs.


of them, 52 s for the CDDC case study, and 183 s for an example involving a large number of quantifiers. We believe that 52 s for a complex case study is still acceptable, whereas the slowest example demonstrates that extensive use of quantifiers will lead to problematic performance in practice.

Table 2 shows that SecC is much faster than our implementation. However, SecC was designed and implemented for information flow verification from scratch, without being able to reuse code from an existing verifier, whereas our extended Nagini implementation could be implemented with minimal effort. Besides this crucial difference, Nagini and SecC differ in many other ways, e.g., in their supported language features, automation (see the table for required annotations), and specification styles. As a result, direct performance comparisons between the two are difficult; in fact, the unmodified version of Nagini *without* the product construction already takes more time than SecC on four out of five examples, likely as a result of the overhead required for modeling more complex language features.

#### **6 Related Work**

There are various existing type systems (e.g., [37,45]) and static analyses (e.g., [11,22]) for proving information flow security. Compared to verification based on product programs, these are more automated, but less precise. Moreover, there are dedicated program logics for information flow verification, such as SecCSL [19], Covern [36], and Veronica [42], all of which allow proving probabilistic noninterference for concurrent programs based on different reasoning techniques. The implementation of the former in SecC is the only existing tool that automates information flow verification for concurrent programs, see Sect. 5.

Relational logics, such as Relational Hoare Logic [7] and Cartesian Hoare Logic [46], allow proving general relational program properties, which includes noninterference. However, while they tackle a more general problem, they generally work only for sequential programs. Some tools automate information flow verification using self-composition, e.g., for C [9] and for Java [41]. Compared to modular product programs, this approach generally does not allow for modular proofs of information flow security [18,48].

Modular product programs were presented by Eilers et al. [18]. Other forms of product programs differ in the way executions are interleaved. While some keep executions in lock step [4], like modular product programs, others do not describe a deterministic product construction and allow for arbitrary interleavings [5]. In particular, Shemer et al. [43] propose property-directed self-composition, which dynamically determines how to compose and interleave different executions based on the property to be verified. Similarly, Strichman and Veitsman [47] propose a product-like construction that interleaves recursive functions whose executions are not in lock step. Recently, Pick et al. [39] showed how to automatically infer information flow specifications on modular product programs, which can likely be combined with the approach examined in this paper.

To the best of our knowledge, SymDiff [27] for the Boogie IVL is the only existing tool that constructs product programs on an IVL-level. SymDiff is a tool for differential program verification, which requires reasoning about pairs of executions of two different (but related) programs and is thus similar to hyperproperty verification; in fact, SymDiff has also been used to verify noninterference in the past [1]. The authors of SymDiff have proposed different techniques for modularly proving mutual function summaries, similar to relational specifications, one of which uses a kind of product construction [25,28]. However, they do not examine potential soundness problems arising from this approach, nor do they discuss if it can be applied to concurrent source programs.

#### **7 Conclusion**

We presented an approach for retrofitting existing IVL-based program verifiers to check information flow security using product programs. This approach allows reusing existing frontends to reduce the required implementation effort. We have shown when this technique is sound, that it can incorporate concurrency, and that it can be implemented in an existing verifier with acceptable performance.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Constraint-Based Relational Verification**

Hiroshi Unno1,2(B) , Tachio Terauchi<sup>3</sup>, and Eric Koskinen<sup>4</sup>

 University of Tsukuba, Ibaraki, Japan uhiro@cs.tsukuba.ac.jp RIKEN AIP, Tokyo, Japan Waseda University, Tokyo, Japan Stevens Institute of Technology, New Jersey, USA

**Abstract.** In recent years they have been numerous works that aim to automate relational verification. Meanwhile, although Constrained Horn Clauses (CHCs) empower a wide range of verification techniques and tools, they lack the ability to express hyperproperties beyond *k*-safety such as generalized non-interference and co-termination.

This paper describes a novel and fully automated constraint-based approach to relational verification. We first introduce a new class of predicate Constraint Satisfaction Problems called pfwCSP where constraints are represented as clauses modulo first-order theories over predicate variables of three kinds: ordinary, well-founded, or functional. This generalization over CHCs permits arbitrary (i.e., possibly non-Horn) clauses, well-foundedness constraints, functionality constraints, and is capable of expressing these relational verification problems. Our approach enables us to express and automatically verify problem instances that require non-trivial (i.e., non-sequential and non-lock-step) self-composition by automatically inferring appropriate *schedulers* (or *alignment*) that dictate when and which program copies move. To solve problems in this new language, we present a constraint solving method for pfwCSP based on *stratified* CounterExample-Guided Inductive Synthesis (CEGIS) of ordinary, well-founded, and functional predicates.

We have implemented the proposed framework and obtained promising results on diverse relational verification problems that are beyond the scope of the previous verification frameworks.

**Keywords:** Relational verification · Constraint solving · CEGIS

#### **1 Introduction**

We describe a novel constraint-based approach to automatically solving a wide range of relational verification problems including k-safety, co-termination [6, 10], termination-sensitive non-interference (TS-NI) [63], and generalized noninterference (GNI) [40] for infinite-state programs.

A key challenge in relational property verification is the discovery of *relational invariants* which relate the states of multiple program executions. However, whereas most prior approaches must fix the execution *schedule*<sup>1</sup> (e.g., lockstep or sequential) [8,20,21,42,54,57], a recent work by Shemer et al. [50] has proposed a method to automatically infer sufficient *fair* schedulers to prove the goal relational property. Importantly, the schedulers in their approach can be *semantic* in which the choice of which program to execute can depend on the *states* of the programs as opposed to the classic *syntactic* schedulers such as lock-step and sequential that can only depend on the control locations. However, their approach requires the user to provide appropriate atomic predicates and is not fully automatic. Moreover, they only support k-safety properties. A recent work has proposed a method for automatically verifying non-hypersafety relational properties but only for *finite* state systems [19].

Meanwhile, today's constraint-based frameworks are also insufficient at automating relational verification. The class of predicate constraints called Constrained Horn Clauses (CHCs) [13] has been widely adopted as a "common intermediate language" for uniformly expressing verification problems for various programming paradigms, such as functional and object-oriented languages. Example uses of the CHCs framework include safety property verification [29,30,35] and refinement type inference [33,36,53,56,66]. The separation of constraint generation and solving has facilitated the rapid development of constraint generation tools such as RCaml [56], SeaHorn [30], and JayHorn [35] as well as efficient constraint solving tools such as SPACER [37], Eldarica [32], and HoIce [14]. Unfortunately, CHCs lack the ingredients to sufficiently express these relational verification problems.

In this paper we introduce automated support for relational verification by generalizing CHCs and introducing a new class of predicate Constraint Satisfaction Problems called pfwCSP. This language allows constraints that are *arbitrary (i.e., possibly non-Horn)* clauses modulo first-order theories over predicate variables that can be *functional predicates*, *well-founded predicates* or ordinary predicates. We then show that, thanks to the enhanced predicate variables, pfwCSP can express *non-hypersafety* relational properties such as cotermination [11], termination-sensitive non-interference (TS-NI) [63], and generalized non-interference (GNI) [40]. In addition, our approach effectively quantifies over the schedule, expressing *arbitrary fair semantic scheduling* thanks to non-Horn clauses and functional predicates (functional predicates are needed to express fairness in the presence of non-termination which is needed for properties like co-termination and TS-GNI). The flexibility allows our approach to automatically discover a fair semantic schedule and verify difficult relational problem instances that require non-trivial schedules. We prove that our encodings are *sound* and *complete*. Expressing relational invariants with such flexible scheduling is not possible with CHCs. However, pfwCSP retains a key benefit of CHCs: the idea of separating constraint generation from solving.

<sup>1</sup> The notion of *schedule* is also often called an *alignment* in literature.

We next present a novel constraint solving method for pfwCSP based on *stratified* CounterExample-Guided Inductive Synthesis (CEGIS) of ordinary, wellfounded, and functional predicates. In our method, ordinary predicates represent relational inductive invariants, well-founded predicates witness synchronous termination, and functional predicates represent Skolem functions witnessing existential quantifiers that encode angelic non-determinism. These witnesses for a relational property are often mutually dependent and involve many variables in a complicated way (see the extended report [58] for examples). The synthesis thus needs to use expressive templates without compromising the efficiency. Stratified CEGIS combines CEGIS [51] with stratified families of templates [55] (*i.e.*, decomposing templates into a series of increasingly expressive templates) to achieve completeness in the sense of [34,55], a theoretical guarantee of convergence, and a faster and stable convergence by avoiding the overfitting problem of expressive templates to counterexamples [44]. The constraint solving method naturally generalizes a number of previous techniques developed for CHCs solving and invariant/ranking function synthesis, addressing the challenges due to the generality of pfwCSP that is essential for relational verification.

We have implemented the above framework and have applied our tool PCSat to a diverse collection of 20 relational verification problems and obtained promising results. The benchmark problems go beyond the capabilities of the existing related tools (such as CHCs solvers and program verification tools). PCSat has solved 15 problems fully automatically by synthesizing complex witnesses for relational properties, and for the 5 problems that could not be solved fully automatically within the time limit, PCSat was able to solve them semiautomatically provided that a part of an invariant is manually given as a hint.

### **2 Overview**

#### **2.1 Relational Verification Problems**

*k***-safety**. Consider the following program taken from [50] that uses a summation to calculate the square of x, and then doubles it.

```
doubleSquare(bool h, int x) {
   int z, y=0;
   if (h) { z = 2*x; } else { z = x; }
   while (z>0) { z--; y = y+x; }
   if (!h) { y = 2*y; }
   return y;
}
```
This program also takes another input h and, if the value of h is true, calculates the result differently. The classical relational property *termination-insensitive non-interference* (TI-NI) says that, roughly, an observer cannot infer the value of high security variables (h in this case) by observing the outputs (y). This is a *2-safety property* [17,54]: it relates two executions of the same program. In this example, we ask whether two executions that initially agree on x (*i.e.*, x<sup>1</sup> = x2) will agree on the resulting y (*i.e.*, y<sup>1</sup> = y2). The subscripts in these relations indicate copies of the program: x<sup>1</sup> is variable x in the first copy of the program and x<sup>2</sup> is variable x in the second copy. More generally, k-safety means that if the initial states of a k-tuple of programs satisfy a pre-relation *Pre*, then when they all terminate the k-tuple of post states will satisfy post-relation *Post*.

The literature proposes many ways to reason about k-safety including methods of reducing a multi-program problem to a single-program problem, such as through self-composition [8,54,57], product programs [7], and their variants [21,46,50,52,59]. Their key challenge is that of *scheduling*: how to interleave the programs' executions so that invariants in the combined program are able to effectively describe cross-program relationships. Indeed, as proved by [50], verifying this example with the na¨ıve lock-step scheduling is impossible with only linear arithmetic invariants while linear arithmetic invariants suffice with a more "semantic" scheduling that schedules the copy with h<sup>1</sup> = false to iterate the loop twice per each iteration of the loop in the copy with h<sup>2</sup> = true.

In this paper, we will describe a way to pose the scheduling problem as a part of a series of constraints so that the search for an effective scheduler is relegated to the solver level. In our approach, a k-safety verification problem is encoded as a set of constraints containing (ordinary) predicate variables that represent the scheduler to be discovered and a relational invariant preserved by the scheduler. Specially, we introduce a predicate variable inv that represents a relational invariant and for each <sup>A</sup> ⊆ {1,...,k}, a predicate variable sch<sup>A</sup>(V-1,..., Vk) where Vi are the variables of the ith program, and add constraints that say that if the predicate is true, then the programs whose index are in A will step forward while the rest remain still and also inv is preserved by the step. For soundness, it is important to constrain the scheduler to be *fair*, *i.e.*, at least one program that can progress must be scheduled to progress if there is a program that can progress. As we shall show in Sect. 4, non-Horn clauses are essential to expressing the fairness constraint. Roughly, the idea is to use a clause with multiple positive predicate variables (*i.e.*, *head disjunction*) to say "*if the relational invariant holds, then at least one of the unfinished programs must be scheduled to progress*."

Our approach is similar to and is inspired by the approach of [50] that also infers a fair semantic scheduler. However, their approach requires the user to provide sufficient atomic predicates manually and is not fully automated. By contrast, our approach soundly-and-completely encodes the k-safety verification problem together with scheduler inference as a set of constraints thanks to the expressiveness of pfwCSP, and automatically solves those constraints by the stratified CEGIS algorithm (cf. Sect. 7 for further comparison).

**Co-termination.** Now consider the following pair of programs.

Pcot <sup>1</sup> : while (x>0) { x = x - y; } Pcot <sup>2</sup> : while (x>0) { x=x-2 × y; } A (non-safety) relational question is whether these programs Pcot <sup>1</sup> and Pcot <sup>2</sup> agree on termination [6,10]. In general they do not: if, for example, Pcot <sup>1</sup> is executed with x < 0 and Pcot <sup>2</sup> with <sup>x</sup> <sup>&</sup>gt; <sup>0</sup>∧<sup>y</sup> = 0, the first will terminate while the second will diverge. However, under the pre-relation *Pre* ≡ x<sup>1</sup> = x<sup>2</sup> ∧ y<sup>1</sup> = y2, they will *agree* on termination: the first program terminates iff the second one does. The property falls outside of the k-safety fragment as it cannot be refuted by finite execution traces. It is worth noting that *termination-sensitive non-interference* (TS-NI) is the conjunction of TI-NI and co-termination of two copies of the same target program with *Pre* equating the copies' low inputs.

Proving co-termination, like k-safety, can be aided by scheduler and we can again use our constraints over predicate variables. But this is not enough. We need additional constraints to ensure that whenever one of the two has terminated, the other is also guaranteed to terminate. To address this, we next introduce *well-founded predicate variables*. These predicate variables will appear in our generalized language of constraints as terms of the form wfr(Vi, V- <sup>i</sup> ), where the relation wfr must be *discovered* by the constraint solving method. (In Sect. <sup>5</sup> we describe how to achieve this through our stratified CEGIS algorithm.) For the above example, our stratified CEGIS algorithm and our tool PCSat automatically discovers (1) a schedule where the two programs step together when x<sup>1</sup> > 0 and x<sup>2</sup> > 0, (2) a relational invariant that implies that if the first program is terminated, then either the second program is terminated or <sup>y</sup><sup>2</sup> <sup>≥</sup> <sup>1</sup> (and vice-versa), and (3) well-founded relations that (combined with the relational invariant) witness that if the loop has terminated in the second program (x<sup>2</sup> <sup>≤</sup> 0) but not in the first (x<sup>1</sup> <sup>&</sup>gt; 0), then a transition in the first is wellfounded (and vice-versa). In Sect. 4, we show how co-termination problems can be soundly-and-completely encoded in pfwCSP.

**Generalized Non-interference.** Now consider the following program.

```
gniEx(bool high, int low) {
  if (high) {
    int x = *$^int$; if (x >= low) { return x; } else { while (true) {} }
  } else {
    int x = low; while ($*^bool$) { x++; } return x;
  }
}
```
The <sup>∗</sup>int (resp. <sup>∗</sup>bool) above indicates an integer (resp. a binary) nondeterministic choice. *Termination-insensitive generalized non-interference* (TI-GNI) [40] is an extension of non-interference to non-deterministic programs, and it says that for any two copies of the program with possibly different values for the high security input (high in this example) and with the same value for the low security input (low in this example), if one copy has a terminating execution that ends in some output (the final value of x in this example), then the other copy has either a terminating execution ending in the same output or a nonterminating execution. The *termination-sensitive* variant (TS-GNI) strengthens the condition by asserting that if one copy has a terminating execution then the other copy has a terminating execution that ends in the same output. Both GNI variants are ∀∃ hyperproperties and fall outside of the <sup>k</sup>-safety fragment.

Verifying GNI requires handling non-determinism. Note that nondeterminism occurs both *demonically* (*i.e.*, as ∀) and *angelically* (*i.e.*, as ∃) in GNI. While handling demonic non-determinism is straightforward in a constraint-based verification since the term variables are implicitly universally quantified, handling angelic non-determinism is less straightforward.

**Fig. 1.** Overview of the contributions and how they achieve a constraint-based strategy for relational verification.

Our approach handles finitary angelic non-determinism like <sup>∗</sup>bool by adding non-Horn clauses with head disjunctions that roughly express the condition "*the relational invariant remains true in one of the finitely many next step choices*". To handle infinitary non-determinism like <sup>∗</sup>int, we introduce *functional predicate variables* denoted f(V,r - ). In these terms, f is a predicate variable to be discovered but with a new wrinkle: this predicate involves a return value r and the interpretation of f is a *total function* over <sup>V</sup>-. For this example, we introduce the term f(V,r - ) where r represents the value chosen non-deterministically at ∗int and V are program variables and *prophecy variables* that represent the final return values of the demonic copy. For this example, PCSat automatically discovers the predicate r = ret<sup>1</sup> where ret<sup>1</sup> is the prophecy variable for the return value of the demonic copy. With it, PCSat is able to verify TS-GNI and TI-GNI of this example. We remark that functional predicates are also used to encode scheduler fairness in the presence of non-termination and is needed to ensure soundness for properties like co-termination and TS-GNI. In Sect. 4.3, we show how TI-GNI and TS-GNI can be soundly-and-completely encoded in pfwCSP.

#### **2.2 Challenges and Contributions**

There are several challenges that we face in supporting relational verification problems with a constraint-based approach. The subsequent sections of this paper are organized around addressing those challenges as follows:

– We first ask how to generalize the constraint language to go beyond CHCs to express a more general class of relational verification problems. To this end, in Sect. 3, we present a new language called *predicated constraint satisfaction problems* (pfwCSP), which incorporate non-Horn clauses, (ordinary) predicate variables, well-founded predicate variables, and functional predicate variables.


In sum, Fig. 1 depicts each of these sections and how, together, they enable relational verification. For space, extra materials are deferred to the extended report [58].

#### **3 Predicate Constraint Satisfaction Problems pfwCSP**

As discussed in Sect. 2, CHCs are insufficient to express important relational verification problems. In the section we introduced a generalized language of constraints called pfwCSP. The language of constraint satisfaction problems (CSP) permits non-Horn clauses, **p**redicate variable terms, including those for **f**unctional predicates and **w**ell-founded relations (pfw). We now define pfwCSP.

Let <sup>T</sup> be a (possibly many-sorted) first-order theory with the signature <sup>Σ</sup>. The syntax of T -formulas and T -terms is: (formulas) <sup>φ</sup>:: = <sup>X</sup>(t) | p(-

$$\begin{array}{l} \text{means and even-roundueu relations (\mu w). we now use} \\ \text{ (possibly many-sorted) first-order theory with the} \\ T\text{-forms and } T\text{-terms is:} \\\\ \text{(formalas) } \phi::=X(\widehat{t}) \mid p(\widehat{t}) \mid \neg \phi \mid \phi\_1 \lor \phi\_2 \mid \phi\_1 \land \phi\_2) \\ \text{(terms) } t::=x \mid f(\widetilde{t}) \\ \end{array}$$

Here, the meta-variables x and X respectively range over term and predicate variables. The meta-variables p and f respectively denote predicate and function symbols of Σ. We use s as a meta-variable ranging over sorts of the signature <sup>Σ</sup>. We write for the sort of propositions and <sup>s</sup><sup>1</sup> <sup>→</sup> <sup>s</sup><sup>2</sup> for the sort of functions from s<sup>1</sup> to s2. We write ar(o) and sort(o) respectively for the arity and the sort of a syntactic element o. A function f represents a constant if ar(f) = 0. We write *ftv*(φ) and *fpv*(φ) respectively for the set of free term and predicate variables that occur in <sup>φ</sup>. We write <sup>x</sup> for a sequence of term variables, <sup>|</sup>x-| for the length of x-, and for the empty sequence. We often abbreviate <sup>¬</sup>φ<sup>1</sup> <sup>∨</sup> <sup>φ</sup><sup>2</sup> as <sup>φ</sup><sup>1</sup> <sup>⇒</sup> <sup>φ</sup>2. We henceforth consider only well-sorted formulas and terms. We use <sup>ϕ</sup> as a meta-variable ranging over <sup>T</sup> -formulas without predicate variables.

We now define a pCSP C (with ordinary but without well-founded and functional predicate variables) to be a finite set of clauses of the form i=1 Xi(-∨ m +1 ¬Xi(-

$$
\varphi \lor \left( \bigvee\_{i=1}^{\ell} X\_i(\tilde{t}\_i) \right) \lor \left( \bigvee\_{i=\ell+1}^{m} \neg X\_i(\tilde{t}\_i) \right) \tag{1}
$$

where 0 <sup>≤</sup> <sup>≤</sup> <sup>m</sup>. We write *ftv*(c) for the set of free term variables of a clause <sup>c</sup>. The set of free term variables of <sup>C</sup> is defined by *ftv*(C) = <sup>c</sup>∈C *ftv*(c). We regard the variables in *ftv*(c) as implicitly universally quantified. We write *fpv*(C) for the set of free predicate variables that occur in C. A *predicate substitution* σ is a finite map from predicate variables X to closed predicates of the form λx1,...,xar(X).ϕ. We write <sup>σ</sup>(C) for the application of <sup>σ</sup> to <sup>C</sup> and dom(σ) for the domain of <sup>σ</sup>. We call <sup>σ</sup> <sup>a</sup> *syntactic solution* for <sup>C</sup> if *fpv*(C) <sup>⊆</sup> dom(σ) and <sup>|</sup><sup>=</sup> <sup>σ</sup>(C). Similarly, we call a predicate interpretation <sup>ρ</sup> <sup>a</sup> *semantic solution* for <sup>C</sup> if *fpv*(C) <sup>⊆</sup> dom(ρ) and <sup>ρ</sup> <sup>|</sup><sup>=</sup> <sup>C</sup>.

*Remark 1.* The language pCSP generalizes over existing languages of constraints. CHCs can be obtained as a restriction of pCSP where <sup>≤</sup> 1 in (1) for all clauses. We can also define coCHCs as pCSP but with the restriction that <sup>m</sup> <sup>≤</sup> + 1 for all clauses. A linear CHCs is a pCSP that is both CHCs and coCHCs.

We next extend pCSP to pfwCSP by adding well-foundedness and functionness constraints. A pfwCSP (C, <sup>K</sup>) consists of


We write <sup>ρ</sup> <sup>|</sup><sup>=</sup> *WF*(X) if the interpretation <sup>ρ</sup>(X) of the predicate variable <sup>X</sup> is and there is no infinite sequence v-1, v-<sup>2</sup>,... of sequences <sup>v</sup><sup>i</sup> of values of the sorts <sup>s</sup> such that (vi, v<sup>i</sup>+1) ∈ <sup>ρ</sup>(X) for all <sup>i</sup> <sup>≥</sup> 1. We write <sup>ρ</sup> <sup>|</sup><sup>=</sup> *FN* (X) if <sup>X</sup> is *functional*, that is, sort(X) = (s, s - ) <sup>→</sup> for some <sup>s</sup> and <sup>s</sup>, and <sup>ρ</sup> <sup>|</sup><sup>=</sup> <sup>∀</sup>x-: s. - (∃y : s.X(x, y - ))∧∀y1, y<sup>2</sup> : s.(X(x, y - <sup>1</sup>)∧ X(x, y - <sup>2</sup>) <sup>⇒</sup> <sup>y</sup><sup>1</sup> <sup>=</sup> <sup>y</sup>2) holds. We call a predicate interpretation <sup>ρ</sup> <sup>a</sup> *semantic solution* for (C, <sup>K</sup>) if <sup>ρ</sup> is a semantic solution of <sup>C</sup>, <sup>ρ</sup> <sup>|</sup><sup>=</sup> *WF*(X) for all <sup>X</sup> such that <sup>K</sup>(X) =⇓, and <sup>ρ</sup> <sup>|</sup><sup>=</sup> *FN* (X) for all <sup>X</sup> such that <sup>K</sup>(X) = <sup>λ</sup>. The notion of syntactic solution can be similarly generalized to pfwCSP.

**Definition 1 (Satisfiability of** pfwCSP**).** The predicate satisfiability problem of a pfwCSP (C, <sup>K</sup>) is that of deciding whether it has a semantic solution.

*Remark 2.* Recall that we assume that the <sup>T</sup> -formulas <sup>ϕ</sup> in pCSP clauses do not contain quantifiers. The assumption, however, is not a restriction for pfwCSP because we can Skolemize quantifiers using functional predicates.

### **4 Relational Verification with Constraints**

We now present reductions from relational verification problems to pfwCSP, thus enabling a new route to automation of these problems. We begin with k-safety, and then move toward liveness and non-determinism, which are thorny problems in the relational setting. We first provide some basic definitions and notations. <sup>A</sup> *state* of the program <sup>P</sup><sup>i</sup> is a valuation of the variables <sup>V</sup>-

*Programs.* We consider programs P1,. . . ,P<sup>k</sup> on variables V <sup>1</sup>,. . . ,V <sup>k</sup>, respectively. <sup>i</sup>. We represent such a valuation by a sequence of values <sup>v</sup> such that |v-| = |V<sup>i</sup>|. We assume that each <sup>P</sup><sup>i</sup> is defined by the predicate <sup>T</sup>i(Vi, Vi ) denoting its one-step transition relation i.e., Ti(v, v- ) implies that evaluating <sup>P</sup><sup>i</sup> one step from the state <sup>v</sup> reaches the state v- . We also assume that there is a predicate <sup>F</sup>i(V<sup>i</sup>) that represents the final states of the program such that <sup>F</sup>i(v-) and Ti(v, v- ) implies v- = v- , i.e., the program self-loops when it reaches a final state. We say that a state <sup>v</sup>- (multistep) reaches a final state <sup>v</sup> in the evaluation of <sup>P</sup>i, written <sup>v</sup>- i v- , if there exists a non-empty finite sequence of states <sup>π</sup> such that <sup>π</sup>[1] = <sup>v</sup>-, π[|π|] = v- , <sup>T</sup>i(π[<sup>j</sup> <sup>−</sup>1], π[j]) for all 1 < j ≤ |π|, and <sup>F</sup>i(v- ). We write v- <sup>i</sup> ⊥ if there exists a non-terminating evaluation from <sup>v</sup> in Pi, *i.e.*, if there exists an infinite sequence of states  such that [1] = <sup>v</sup>-, <sup>T</sup>i([<sup>j</sup> <sup>−</sup>1], [j]) for all 1 < j, and <sup>¬</sup>Fi([j]) for all 0 < j. We note that a program may be non-deterministic, that is, <sup>T</sup>i(v, v- ) and Ti(v, v-) may both be true for some <sup>v</sup>- = v-. <sup>A</sup> <sup>k</sup>*-safety property* is given by predicates *Pre*(V-) and *Post*(V-

#### **4.1** *k***-Safety**

) that respectively denote the pre and the post relations across the k-tuple.

**Definition 2 (**k**-safety).** The k*-safety property verification problem* is to decide if the following holds: ∀v- = v-1,..., vk.∀v- = v-,..., v-.*Pre*(v-) ∧ i∈[k] vi v-⇒ *Post*(v-

$$\forall \tilde{v} = \tilde{v}\_1, \dots, \tilde{v}\_k. \forall \tilde{v}' = \tilde{v}\_1', \dots, \tilde{v}\_k'. Pre(\tilde{v}) \land \bigwedge\_{i \in [k]} \tilde{v}\_i \leadsto\_i \tilde{v}\_i' \Rightarrow Post(\tilde{v}')$$

That is, any k-tuple of final states reachable from a k-tuple of states satisfying the precondition satisfies the post condition. For instance, the TI-NI verification from Sect. 2.1 is a 2-safety property where P<sup>1</sup> and P<sup>2</sup> are copies of the same program, *Pre* states that the low inputs of the two programs are equal (*i.e.*, x<sup>1</sup> = x<sup>2</sup> in the example), and *Post* states that the low outputs of the two programs are equal (*i.e.*, y<sup>1</sup> = y<sup>2</sup> in the example). 

We now describe a new way to pose the k-safety relational verification problem via constraints written in pfwCSP. We write [k] for the set {1,...,k}. We define <sup>P</sup><sup>+</sup>[k] = {<sup>S</sup> <sup>⊆</sup> [k] <sup>|</sup> <sup>S</sup> <sup>=</sup> ∅}. Let <sup>V</sup>- = V <sup>1</sup>,. . . ,V <sup>k</sup> be a k-tuple of vectors, corresponding to the variables of the k programs.

**Definition 3 (**k**-safety through constraints).** We define pfwCSP constraints C<sup>S</sup> be the set of following clauses:


Here, inv and sch<sup>A</sup> (for each <sup>A</sup> ∈ P<sup>+</sup>[k]) are ordinary predicate variables. Roughly, the predicate variables sch<sup>A</sup> describe a *scheduler*. The scheduler stipulates that when sch<sup>A</sup>(v-1,..., v<sup>k</sup>) is true, each <sup>P</sup><sup>i</sup> such that <sup>i</sup> <sup>∈</sup> <sup>A</sup> takes a step from the state v<sup>i</sup> while the others remain still. Note that the scheduler is *semantic* in the sense that which programs are scheduled to be executed next can depend on the current states of the programs. Clauses (1)–(3) assert that inv is an invariant sufficient to prove the given safety property with the scheduler defined by sch<sup>A</sup>'s. Clauses (4) say that if an inv-satisfying state is such that the processes in A are allowed to move and some program has not yet terminated, then at least one process in A has not yet terminated. Clause (5) says that any state satisfying inv has to satisfy some sch<sup>A</sup>. Clauses (4) and (5) ensure the *fairness* of the scheduler, that is, at least one unfinished program is scheduled to make progress if there is an unfinished program.

**Theorem 1 (Soundness and Completeness of** <sup>C</sup>S**).** *The given* <sup>k</sup>*-tuple of programs satisfies the given* <sup>k</sup>*-safety property iff* <sup>C</sup><sup>S</sup> *is satisfiable.*

We note that the soundness direction crucially relies on scheduler fairness. The completeness is with respect to semantic solutions (cf. Definition 1) and it is only "relative" with respect to syntactic solutions: a syntactic solution only exists when the predicates of the background theory are able to express sufficient invariants and schedulers (impossible in general for any decidable theory when the class of programs is Turing-powerful as in our case when the background theory of predicates is QFLIA).

It is important to note that C<sup>S</sup> is *not* CHCs because clause (5) has a head disjunction. C<sup>S</sup> may be seen as a constraint-based formulation of the approach proposed in [50]. However, their approach requires the user to provide sufficient predicates manually and is not fully automated, while our approach can fully automatically solve the problems by constraint solving (cf. Sect. 5).

*Example 1.* The formalization allows flexible scheduling. For instance, for the TI-NI example from Sect. 2.1, our approach is able to infer the predicate substitution that maps sch{1}, sch{2}, and sch{1,2} to λV . <sup>h</sup><sup>1</sup> ∧ ¬h<sup>2</sup> <sup>∧</sup> <sup>z</sup><sup>1</sup> = 2z2, λV . - ¬h<sup>1</sup> ∧ h<sup>2</sup> ∧ <sup>z</sup><sup>2</sup> = 2z1, and λV . - (h<sup>1</sup> ∧ ¬h<sup>2</sup> ⇒ z<sup>1</sup> +1 = 2z2) ∧ (¬h<sup>1</sup> ∧ h<sup>2</sup> ∧ z<sup>2</sup> +1 = 2z1) respectively, where <sup>V</sup> is the list of the variables in the two program copies. The inferred predicates stipulate that the copy with h = true is scheduled to execute the loop two times per every loop iteration of the copy with h = false. The extended report [58] shows the pfwCSP encoding of the example. A solution generated by PCSat is also shown in [58].

#### **4.2 Co-termination**

Intuitively, co-termination means that if one program terminates, then a second program must terminate [6,10]. This can also be thought of as a form of relational *termination problem*. 2 to decide if for all <sup>v</sup>-1, v-<sup>2</sup> such that *Pre*(v-1, v-2), if v-1 -1 then v-

**Definition 4 (Co-termination).** The *co-termination verification problem* is <sup>1</sup> v <sup>2</sup> -<sup>2</sup> ⊥.

Roughly, the property says that from any pair of states related by *Pre*, if P<sup>1</sup> terminates, then P<sup>2</sup> must also terminate. Note that this is an *asymmetric* property. A symmetric version can be obtained by also asserting the property with the positions of the two programs exchanged. The symmetric version implies, assuming that there is at least one execution from any *Pre*-related state, that from any pair of *Pre*-related states, all executions from one state terminates iff all executions from the other one do as well. We now present an encoding of conditional co-termination in pfwCSP. **Definition 5 (Co-termination through constraints).** Let <sup>V</sup>- 

 = V <sup>1</sup>, V <sup>2</sup>. We define pfwCSP constraints CCoT be the set of following clauses: (1) *Pre*(V-) ∧ fnb(V,b - ) <sup>⇒</sup> inv(0, b, <sup>V</sup>-(2) inv(d, b, V-(3a) inv(d, b, V-) ∧ schFT(d, b, V- 


Here, schTT, schFT, and schTF are 2-specialization of the <sup>k</sup>-safety scheduler of Definition 3. Clauses (3x)'s are similar to (3) of Definition <sup>3</sup> and assert that inv is an invariant under the scheduler. Clauses (4x)'s and (5), like (4) and (5) of Definition 3, are used to ensure the scheduler fairness. However, they are insufficient for co-termination as a non-terminating copy can be scheduled indefinitely leaving the other copy unscheduled. Clauses (1) and (2) are added to amend the issue. In (1), fnb is a functional predicate variable that is used to select a *bound* b, and (2) asserts that the *difference* d between the numbers of steps taken by the two copies is within <sup>b</sup> in any state in inv when neither copy has terminated. Note that d is initialized to 0 by (1) and properly updated in (3x)'s. Finally, by using the well-founded predicate variable wfr, (6) asserts that if <sup>P</sup><sup>1</sup> has terminated, then so must eventually P2.

<sup>2</sup> The property has also been called *relative termination* [31].

**Theorem 2 (Soundness and Completeness of** CCoT**).** *The given pair of programs co-terminate iff* CCoT *is satisfiable.*

As with Theorem 1, the soundness direction relies on scheduler fairness.

*Example 2.* Via the encoding, our PCSat tool is able to verify the symmetric co-termination example from Sect. 2.1 by automatically inferring the solution described there. For space, the concrete constraint set and solution are given in the extended report [58].

#### **4.3 Generalized Non-interference**

We now turn to another relational property that cannot simply be captured by k-safety or co-termination. So-called *termination-insensitive (resp. -sensitive) generalized non-interference* (resp. TI-GNI, TS-GNI) are ∀∃ hyperproperties: from any pre-related pair of states whenever one side can take a move to a post state, there must be a way for the other side to also move to a post state such that the post-relation holds. As remarked in Sect. 2, verifying GNI requires reasoning about both *demonic* (*i.e.*, for all) and *angelic* (*i.e.*, exists) *non-determinism*. following holds. If *Pre*(v-1, v-2) and v-1 v then **(TI-GNI)** (∃v-.v-2 v-

**Definition 6 (TI/TS-GNI).** The *GNI verification problem* is to decide if the <sup>1</sup> -1 2 <sup>2</sup> -2 ∧ *Post*(v-1 , v-2 )) ∨ v-<sup>2</sup> -<sup>2</sup> <sup>⊥</sup>; or **(TS-GNI)** <sup>∃</sup>v-2 .v-<sup>2</sup> -2 v-2 ∧ *Post*(v-1 , v-2 ).

Note that our definition is parameterized by *Pre* and *Post*. The standard GNI definitions [40] can be obtained by letting P<sup>1</sup> and P<sup>2</sup> be copies of the same target program and letting *Pre* be the predicate equating the low inputs of the copies and *Post* be the predicate equating the low outputs of the copies. we define a relation <sup>U</sup><sup>2</sup> to be one such that <sup>T</sup>2(v, v-) ⇔ ∃r.U2(r, v, v-

To formalize the pfwCSP encodings of the GNI verification problems, ) and U2(r, v, v- ) ∧ U2(r, v, v-) ⇒ v- = v-. Roughly, U<sup>2</sup> is a function version of the transition relation T<sup>2</sup> with the extra parameter r to make the non-deterministic choices explicit.

We now show the pfwCSP encodings of TI-GNI and TS-GNI. The key idea is to augment the encodings for k-safety and/or co-termination with *functional predicate variables* and *prophecy variables* that respectively represent the nondeterministic choices of the angelic side (*i.e.*, P2) and the final outputs of the demonic side (*i.e.*, P1).

**Definition 7 (TI-GNI through constraints).** We define pfwCSP constraints <sup>C</sup>TIGNI as <sup>C</sup><sup>S</sup> in Definition <sup>3</sup> for <sup>k</sup> = 2 but with the following modifications: prophecy variables <sup>p</sup> where |p- 



(m6) Each occurrence of T2(V <sup>2</sup>, V 2 V <sup>2</sup>, r) <sup>∧</sup> <sup>U</sup>2(r, <sup>V</sup> <sup>2</sup>, V 2 ) where fnr is a functional predicate variable.

Modifications (m1)–(m5) concern prophecy variables. They are initialized arbitrarily as shown in (m2), propagated unmodified through the transitions as shown in (m4), and finally checked if they match P1's outputs in (m5). Modification (m6) adds functional predicate variables to express the angelic nondeterministic choices of P2. The functional predicate variables shift the onus of making the right choices to the solver's task of discovering sufficient assignments to them. Importantly, the functional predicate takes the prophecy variables as parameters, thus allowing dependence on the final outputs of the demonic side. 

**Definition 8 (TS-GNI through constraints).** We define pfwCSP constraints CTSGNI as CCoT in Definition 5 but with modifications of Definition 7 except (m3) and (m5), and with the following modifications: 1(p, -1) ∧ p-(m5') The clause inv(p, - 1(p, - 

(m3') F<sup>1</sup> is replaced by F <sup>1</sup> defined by F V <sup>1</sup>) <sup>⇔</sup> <sup>F</sup>1(<sup>V</sup> = V 1. V <sup>1</sup>, V <sup>2</sup>) <sup>∧</sup> <sup>F</sup> V <sup>1</sup>) <sup>∧</sup> <sup>F</sup>2(<sup>V</sup> <sup>2</sup>) <sup>⇒</sup> *Post*(<sup>V</sup> <sup>1</sup>, V <sup>2</sup>) is added.

CTSGNI is similar to CTIGNI except that it contains the difference bound and wellfoundedness constraints to handle the "co-termination" aspect of TS-GNI, *i.e.*, if P<sup>1</sup> terminates and makes an output then P<sup>2</sup> must also be able terminate and make a matching output. One subtle aspect of the encoding is that (m3') modifies the final state predicate for P<sup>1</sup> to enforce co-termination only when the prophecy is correct. However, it is worth noting that TS-GNI is *not* a conjunction of TI-GNI and co-termination. For instance, the GNI example from Sect. 2.1 satisfies TS-GNI but does not satisfy co-termination.

**Theorem 3 (Soundess and Completeness of TI-GNI).** *The given pair of programs satisfy TI-GNI iff* CTIGNI *is satisfiable.*

**Theorem 4 (Soundess and Completeness of TS-GNI).** *The given pair of programs satisfy TS-GNI iff* CTSGNI *is satisfiable.*

The soundness directions are proven by "determinizing" the angelic choices by solutions to the functional predicate variables and reducing the argument to those of k-safety and co-termination. The completeness directions are proven by "synthesizing" sufficient angelic choice functions from program executions.

*Example 3.* Via the encoding, our PCSat tool is able to verify the TS-GNI example from Sect. 2.1 by automatically inferring not only the functional predicate described there but also relational invariants and well-founded relations given in the extended report [58]. For space, the concrete constraint set is also given in [58].

*Remark 3.* The angelic non-determinism encoding can be optimized by using head disjunctions when the non-determinism is finitary (*i.e.*, maxv-|{v- | T2(v, v- )}| is finite) instead of using functional predicate variables. For this, we modify clauses (3) and (3x)'s of Definition 7 and 8 to contain multiple positive occurrences of inv where each occurrence represents one of the finitely many possible choices.

*Remark 4.* Recall that we allow any program to be non-deterministic. The ksafety and co-termination encodings treat non-determinism in all programs as demonic, whereas the GNI encodings treat those in one program (*i.e.*, P1) as demonic and those in the other program (*i.e.*, P2) as angelic. In general, an arbitrary program can be made angelic by applying the transformation done in the angelic side of GNI encodings (to factor out non-determinism).

#### **5 Constraint Solving Method for pfwCSP**

We describe a CEGIS-based method for finding a (syntactic) solution of the given pfwCSP (C, <sup>K</sup>). Our method iterates the following phases until convergence. The iteration maintains and builds a sequence σ of *candidate solutions* and a sequence <sup>E</sup> of *example instances* where <sup>E</sup>(i) are ground clauses obtained from <sup>C</sup> by instantiating the term variables and serve as a counterexample to the candidate solution <sup>σ</sup>(i−1), for each <sup>i</sup>-th iteration. The iteration starts from <sup>E</sup>(1) <sup>=</sup> <sup>∅</sup>.

**Synthesis Phase:** We check if (E(i), <sup>K</sup>) is unsatisfiable. If so, we stop by returning <sup>E</sup>(i) as a genuine counterexample to the input problem (C, <sup>K</sup>). Otherwise, we use the synthesizer <sup>S</sup>*<sup>T</sup>*<sup>B</sup> (cf. Sect. 5.1) to find a solution <sup>σ</sup>(i) of (E(i), <sup>K</sup>), which will be used as the next candidate solution.

**Validation Phase:** We check if <sup>σ</sup>(i) is a genuine solution to (C, <sup>K</sup>) by using an SMT solver. If so, we stop by returning σ(i) as a solution. Otherwise, for each clause <sup>c</sup> ∈ C not satisfied by <sup>σ</sup>(i) , we obtain a term substitution θ<sup>c</sup> such that dom(θc) = *ftv*(c) and |<sup>=</sup> <sup>θ</sup>c(σ(i) (c)). We then update the example set by adding a new example instance for each unsatisfied clause (i.e., <sup>E</sup>(i+1) <sup>=</sup> <sup>E</sup>(i) ∪ { <sup>θ</sup>c(c) <sup>|</sup> <sup>c</sup> ∈ C∧ |<sup>=</sup> <sup>σ</sup>(i)(c) }), and proceed to the next iteration.

The above procedure satisfies the usual *progress property* of CEGIS: discovered counterexamples and candidate solutions are not discovered again in succeeding iterations. Furthermore, as discussed in Sect. 5.1, by carefully designing the synthesizer S*<sup>T</sup>*<sup>B</sup> by incorporating *stratified* CEGIS, we achieve *completeness* in the sense of [34,55]: if the given pfwCSP (C, <sup>K</sup>) has a syntactic solution expressible in the stratified families of templates, a solution of the pfwCSP is eventually found by the procedure. In Sect. 5.1, we discuss the details of the synthesis phase. There, for space, we focus on the theory of quantifier-free linear integer arithmetic (QFLIA). For space, we defer the details of the unsatisfiability checking process to the extended report [58].

*Remark 5.* The implementation described in Sect. 6 contains an additional phase called *resolution phase* for accelerating the convergence. There, we first apply unit propagation repeatedly to the given <sup>E</sup>(i) to obtain positive examples <sup>E</sup>(i)+ of the form <sup>X</sup>(v-) and negative examples <sup>E</sup>(i)<sup>−</sup> of the form <sup>¬</sup>X(v-). We then repeatedly apply resolution principle to the clauses in the input clauses C and the clauses <sup>E</sup>(i)+ ∪ E(i)<sup>−</sup> to obtain additional positive and negative examples.

#### **5.1 Predicate Synthesis with Stratified Families of Templates**

We describe our candidate solution synthesizer S*<sup>T</sup>*B. S*<sup>T</sup>*<sup>B</sup> performs a templatebased search for a solution to the given example instances. As we shall show, our approach allows searching for assignments to all predicate variables (of all three kinds) in the given instance which is important because satisfying assignments to different predicate variables often inter-dependent. There, however, is a trade-off between expressiveness and generalizability. With less expressive templates like intervals, we may miss actual solutions. But with very expressive templates like polyhedra, there could be many solutions, and a solution thus returned is liable to overfitting, *i.e.*, the solution to the example instances becomes too specific to be an actual solution to the original input clauses. [44] discusses a similar overfitting issue in the context of grammar-based synthesis.

$$\begin{array}{c} DEC\_{i,j}(\widetilde{x},\widetilde{y}) \triangleq \bigvee\_{k=1}^{n} \big(r\_{i,k}(\widetilde{x}) > r\_{j,k}(\widetilde{y}) \wedge \bigwedge\_{\ell=1}^{k-1} r\_{i,\ell}(\widetilde{x}) \ge r\_{j,k}(\widetilde{y}) \wedge \bigwedge\_{\ell=1}^{k-1} r\_{i,\ell}(\widetilde{x}) \ge r\_{j,\ell}(\widetilde{y}) \wedge \bigvee\_{\ell=1}^{n} \\\ r\_{i,k}(\widetilde{x}) \triangleq c\_{i,k,0} + \sum\_{\ell=1}^{\mathrm{ar}(X)/2} c\_{i,k,\ell} \cdot x\_{\ell} & D\_{i}(\widetilde{x}) \triangleq \bigwedge\_{k=1}^{nc} c'\_{i,k,0} + \sum\_{\ell=1}^{\mathrm{ar}(X)/2} c'\_{i,k,\ell} \cdot x\_{\ell} \ge 0 \end{array}$$

**Fig. 2.** Stratified families of templates

Our remedy to the issue is *stratified families of predicate templates*, inspired by a similar approach proposed in the context of predicate abstraction with CEGAR [34,55]. Initially, we assign each predicate variable a less expressive template and gradually refine it in a counterexample-guided manner: if no solution exists in the current template, we generate and analyze an unsat core to identify the *parameters of the families of templates* that should be updated. The stratification of templates thus automatically pushes the template to an expressive one (e.g., polyhedra) when it needs to. Importantly, with our approach, expressive templates are not always used but only when they should be used.

**Stratified Families of Templates.** We have designed three stratified families of templates shown in Fig. 2, respectively for ordinary (•), well-founded (⇓), and functional (λ) predicate variables. First, for each ordinary predicate variable X, we prepare the stratified family of templates T • <sup>X</sup>(*nd*, *nc*, *ac*, *ad*) with unknowns ci,j,k's to be inferred and its accompanying constraint φ• <sup>X</sup>(*nd*, *nc*, *ac*, *ad*). The body of T • <sup>X</sup> is a DNF with affine inequalities as atoms. The parameter *nd* (resp. *nc*) is the number of disjuncts (resp. conjuncts). The parameter *ac* is the upper bound of the sum of the absolute values of coefficients ci,j,k (k > 0), and *ad* is the upper bound of the absolute value of ci,j,<sup>0</sup>.

Secondly, for each functional predicate variable X, we prepare the stratified family of templates T ⇓ <sup>X</sup>(*np*, *nl*, *nc*, *rc*, *rd*, *dc*, *dd*) with unknowns <sup>c</sup>i,j,k's and c i,j,k's and its accompanying constraint <sup>φ</sup>⇓ <sup>X</sup>(*np*, *nl*, *nc*, *rc*, *rd*, *dc*, *dd*). <sup>T</sup> ⇓ <sup>X</sup> represents the well-founded relation induced by a *piecewise-defined lexicographic affine ranking function* [2,39,39,60,61] where ri,j is the affine ranking function template for the j-th lexicographic component of the i-th region specified by the discriminator Di. The parameter *np* (resp. *nl*) is the number of regions (resp. lexicographic components). The parameters *rc*, *rd*, *dc*, *dd* are the upper bounds of (the sums of) the absolute values of unknowns, similar to *ac* and *ad* of T • X. The first conjunct of T ⇓ <sup>X</sup> asserts that the return value of each ranking functions is non-negative. The second and the third conjuncts assert that the discriminators cover all relevant states. Note that discriminators may overlap, and for such overlapping regions, the maximum return value of the ranking functions is used. The fourth conjunct asserts that the return value of the piecewise-defined ranking function strictly decreases from <sup>x</sup> to y-. Here, *DEC* i,j (x, y-) asserts that the return value of the lexicographic ranking function for the <sup>i</sup>-th region at <sup>x</sup>- is greater than that for the <sup>j</sup>-th region at <sup>y</sup>-. It follows that for any substitution θ for the unknowns in T ⇓ <sup>X</sup>, <sup>θ</sup>(<sup>T</sup> ⇓ <sup>X</sup>) represents a well-founded relation. Our implementation PCSat uses a refined version of T ⇓ <sup>X</sup> shown in the extended report [58].

Finally, for each functional predicate variable X, we prepare the stratified family of templates T <sup>λ</sup> <sup>X</sup>(*nd*, *nc*, *dc*, *dd*, *ec*, *ed*) with unknowns <sup>c</sup>i,j 's and <sup>c</sup> i,j,k's and its accompanying constraint φ<sup>λ</sup> <sup>F</sup> (*nd*, *nc*, *dc*, *dd*, *ec*, *ed*). <sup>T</sup> <sup>λ</sup> <sup>X</sup> characterizes a piecewise-defined affine function with discriminators <sup>D</sup>1,...,D*nd*−<sup>1</sup> and branch expressions e1,...,e*nd* . The parameter *nc* is the number of conjuncts in each discriminator. The parameters *dc*, *dd*, *ec*, *ed* are the upper bounds of (the sums of) the absolute values of unknown, similar to *ac* and *ad* of T • <sup>X</sup>. Note that for any substitution θ for the unknowns in T <sup>λ</sup> <sup>X</sup>, <sup>θ</sup>(<sup>T</sup> <sup>λ</sup> X)(x, r - ) expresses a total function that maps x to r. p-

Next, we give the details of the candidate solution synthesis process. Let <sup>∈</sup> <sup>Z</sup><sup>n</sup> where <sup>n</sup> is the number of parameters summed across all templates, and let T <sup>α</sup> X(p-) and φ<sup>α</sup> X(p-) (for <sup>α</sup> ∈ {•, ⇓, λ}) project the corresponding parameters. Each p- <sup>∈</sup> <sup>Z</sup><sup>n</sup> induces a *solution space* p- {T(p-)[θ] <sup>|</sup> <sup>θ</sup> <sup>|</sup><sup>=</sup> *Con*(p-)} where T(p-)[θ] {<sup>X</sup> → <sup>θ</sup>(<sup>T</sup> <sup>K</sup>(X) X (p-)) <sup>|</sup> <sup>X</sup> <sup>∈</sup> *fpv*(C)} and *Con*(p-) <sup>X</sup>∈*fpv*(C) <sup>φ</sup>K(X) X (p-). Let p-1 ≤ ppany p- ∈ Zn, and p-1 ≤ ppp-

<sup>2</sup> be the point-wise ordering. Note that is a finite set for <sup>2</sup> implies -<sup>1</sup> ⊆ -<sup>2</sup>. We start the CEGIS process with some small initial parameters <sup>p</sup>-(0) (the parameters will be maintained as a state of the CEGIS process). The synthesis phase of each iteration tries to find a solution <sup>θ</sup> <sup>∈</sup> p-(i) to the given example instances (E, <sup>K</sup>) where <sup>p</sup>-(i) are the current parameters. This is done by using an SMT solver for QFLIA to find <sup>θ</sup> satisfying <sup>T</sup>(p-(i) )[θ](E) <sup>∧</sup> <sup>θ</sup>(*Con*(p-(i) )). If such θ is found, we return T(p-(i) )[θ] as the candidate solution for the next validation phase of the CEGIS process. Note that, by construction of the templates, the solution is guaranteed to assign each well-founded (resp. functional) predicate variable a well-founded relation (resp. total function). Otherwise, no solutions to the given example instances (E, <sup>K</sup>) can be found in p-(i), and we update the parameters to some p-(i+1) > p-(i) such that p-(i+1) contains a solution for (E, <sup>K</sup>). Here, it is important to do the update in a *fair manner* [34,55], that is, in any infinite series of updates p-(0), p-(1),... , every parameter is updated infinitely often (the details are deferred to below). By the progress property and the fact that every p is finite, this ensures that every parameter is updated infinitely often in an infinite series of CEGIS iterations. We thus obtain the following property. *complete in the sense of [34,55]: if there is* <sup>p</sup>p-

**Theorem 5.** *Our CEGIS-procedure based on stratified families of templates is and* <sup>σ</sup> <sup>∈</sup> *such that* σ *is a syntactic solution to the given* pfwCSP (C, <sup>K</sup>)*, a syntactic solution to* (C, <sup>K</sup>) *is eventually found by the procedure.* p-

Note that, while the solution space of each stratum (*i.e.*, -(i) ) is finite, our procedure searches the infinite solution space obtained by taking the infinite union of the solution spaces of the template family strata (*i.e.*, <sup>i</sup>∈<sup>ω</sup> p-(i) ).

*Remark 6.* Our template-based synthesis simultaneously finds ordinary, wellfounded, and functional predicates that are mutually dependent through the given (E, <sup>K</sup>). This means that templates for different kinds of predicate variables are updated in a synchronized and balanced manner, which benefits the synthesis of mutually dependent witnesses for a relational property (see the extended report [58] for examples).

*Updating Parameters of Template Families via Unsat Cores.* We now describe the parameter update process. We first obtain the unsat core of the unsatisfiability of T(p-(i))[θ](E) <sup>∧</sup> <sup>θ</sup>(*Con*(p-(i))) from the SMT solver. We then analyze the core to obtain the parameters of template families, such as the number of conjuncts and disjuncts, that have caused the unsatisfiability. Here, there could be a dependency between predicate variables and in such a case our unsat core analysis enumerates all the involved predicate variables from which we obtain the parameters of template families to be updated. We then increment these parameters in some fair manner, by limiting the maximum differences between different parameters to some bounded threshold, and repeatedly solve the resulting constraint until a solution is found. Thus, the parameters of stratified families of templates are grown on-the-fly guided by the reasons for unsatisfiability. We found that a careful design of parameter update strategies important for scaling the stratified CEGIS to hard relational verification problems. The manual tuning, however, is tiresome and suboptimal. We leave as future work to investigate methods for automatic tuning of parameter update strategies.

#### **6 Evaluation**

To evaluate the presented verification framework, we have implemented PCSat, a satisfiability checking tool for pfwCSP based on stratified CEGIS. PCSat supports the theory of Booleans and the quantifier-free theory of linear inequalities over integers and rationals. The tool is implemented in OCaml, using Z3 [41] as the backend SMT solver. We ran the tool on a diverse collection of 20 relational verification problems, consisting of k-safety, co-termination, and GNI problems. Though we have manually reduced them to pfwCSP using the presented method in Sect. 4, this process can be easily automated. The full benchmark set is provided in the extended report [58]. All experiments have been conducted on 3.1 GHz Intel Xeon Platinum 8000 CPU and 32 GB RAM with the time limit of 600 s.

The experimental results are summarized in Table 1. The columns "Time (s)" and "#Iters" respectively show elapsed wall clock time in seconds and numbers of CEGIS iterations. PCSat solved 15 verification problems fully automatically and 5 problems labeled with the symbol † and/or ‡ semi-automatically. For the 4 problems labeled with †, we manually provided small hints for invariant synthesis (interested readers are referred to [58]). The provided hints for all but SquareSum are non-relational invariants that can be inferred prior to relational verification by using a CHCs solver or an invariant synthesizer. For the 2 problems labeled with ‡, we manually chose the initial value for the parameters of the template family for ordinary predicate variables to reduce the elapsed time. This can be automated by running PCSat with different initial values in parallel.

The problems DoubleSquareNI h\*\*, HalfSquareNI, ArrayInsert, and SquareSum are k-safety verification problems obtained from [50] that require nonlock-step scheduling.<sup>3</sup> The problems DoubleSquareNI h\*\* are generated from Example 1 by a case analysis of the valuation for the boolean variables h<sup>1</sup> and h2. PCSat solved all the k-safety problems but SquareSum *fully automatically*. The tool Pdsc proposed in [50] can verify them but requires the user to provide the atomic predicates for expressing relational invariants and schedulers. The problems CotermIntro1 and CotermIntro2 are asymmetric co-termination problems obtained from the symmetric problem in Example 2 and are verified by PCSat fully automatically. The problems TS GNI h\*\* are generated from Example 3 by

<sup>3</sup> We omitted ArrayIntMod from [50] because its verification requires the theory of arrays which the current version of PCSat does not fully support.

a case analysis and are verified by PCSat with small non-relational hints. We have also tested PCSat on various TS-GNI (SimpleTS GNI1, SimpleTS GNI2, InfBranchTS GNI) and TI-GNI problems (TI GNI h\*\*) and obtained promising results. As far as we know, no existing tools can verify these non-k-safety relational problems.

Furthermore, manual inspection of the PCSat's output logs for the GNI problems that required hints revealed that the functional predicate synthesis appears to be the main bottleneck of the current version. In fact, we confirmed that the problems can be solved in less than 10 s if appropriate functional predicates for angelic non-determinism are manually provided. As future work, we plan to investigate methods for improved functional predicate synthesis.

#### **7 Related Work**

#### **7.1 Relational Verification**

There has been substantial work on verifying relational properties. They include program logics, type systems, or program analysis frameworks such as abstract interpretation and model checking [1,5,9,19,25,52,62], program transformation approaches such as self-composition or product programs [4,7,15,20,21,42,47, 54,57,64], and various other approaches [3,18,23,46,59]. We refer to [43] for an excellent survey. However, most prior automatic approaches address only the ksafety fragment [17,54] and cannot verify non-k-safety (actually, not even hypersafety) properties such as co-termination, TS-NI, TI-GNI, and TS-GNI [6,11,40]. The only exception that we are aware is the recent work by Coenen et al. [19] that proposes a sound method for automatically verifying ∀∃ hyperproperties such as GNI for finite state systems. To our knowledge, we are the *first* to propose a sound-and-complete approach to automatically verifying these non-hypersafety properties for infinite state programs.<sup>4</sup>

A key task in many relational verification methods, including ours, is the discovery of *relational invariants* which relate the states of multiple program executions. While most prior approaches are limited to fixed execution schedule (or *alignment*) such as lock-step and sequential [7,8,20,21,42,54,57], a recent work by Shemer et al. [50] has proposed a k-safety property verification method that automatically infers fair schedulers sufficient to prove the goal property. Importantly, the schedulers in their approach can be *semantic* in which the choice of which program to execute can depend on the *states* of the programs as opposed to the classic *syntactic* schedulers such as lock-step and sequential that can only depend on the control locations. Our approach also infers such fair semantic schedulers, and as remarked before, they enable solving instances like doubleSquare that are difficult for previous approaches. However, [50] requires

<sup>4</sup> However, [19] can verify (relational) temporal properties, whereas we only support functional properties that are given by pre and post conditions of whole program runs. We leave as future work to investigate methods for verifying relational temporal properties of infinite state programs.


**Table 1.** Experimental results of PCSat on the relational verification benchmarks

the user to provide appropriate atomic predicates and is not fully automatic. By contrast, our approach soundly and completely encodes the problem as a constraint satisfaction problem and fully automatically verifies hard instances like doubleSquare by constraint solving.

Furthermore, our work extends the fair semantic scheduler synthesis to beyond k-safety problems like co-termination, TI-GNI and TS-GNI, in a sound and complete manner. We note that the extensions are non-trivial and involves delicate uses of functional predicate variables and well-founded predicate variables to ensure scheduler fairness in the presence of non-termination as well as uses of prophecy variables and functional predicate variables to handle angelic non-determinism. The higher-degree of automation and the extension to nonk-safety properties are thanks to the expressive power of our novel constraint framework pfwCSP.

#### **7.2 Predicate Constraint Solving**

Our pfwCSP solving technique builds on and generalizes a number of techniques developed for CHCs solving as well as invariant and ranking function discovery. Most closely related to our constraint solving method are CEGIS-based [51] and data-driven approaches to solving CHCs [14,22,24,26,27,38,44,45,48,49,65]. As remarked before, the new pfwCSP framework is strictly more expressive than CHCs and extending the prior techniques to the new framework is non-trivial.

Our stratified CEGIS is inspired by the idea of stratified languages of predicates proposed in the context of predicate abstraction with CEGAR [34,55]. It is also similar in spirit to the work by Padhi et al. [44], but they use a stratified family of grammars. Also none of these prior works use unsat cores for updating the language/grammar stratum, synthesize well-founded relations and functional predicates, or support non-Horn clauses.

Our class of pfwCSP constraints is related to *existentially-quantified Horn clauses* (E-CHCs) introduced by Beyene et al. [12]. E-CHCs does not have non-Horn clauses or functional predicate variables. However, it has disjunctive well-foundedness constraints which are similar to our well-founded predicate variables. Also, existential quantifiers can be used to encode head disjunctions and functional predicates. We conjecture that pfwCSP and E-CHCs are interreducible, but it is not trivial to fill the gap. Also, inter-reducibility is a desirable feature: different formats may have different benefits. For relational verification, as we have shown, pfwCSP enables direct sound-and-complete encodings of the problems. For instance, head disjunctions allow direct encoding of scheduler fairness and finitary angelic non-determinism (cf. Remark 3). And, functional predicate variables can be explicitly given necessary-and-sufficient parameters to encode angelic non-determinism and difference bounds for ensuring scheduler fairness in the presence of non-termination. The tight encodings also lead to reduction in search space and benefited the constraint solving.

#### **8 Conclusion**

We have introduced the class pfwCSP of predicate constraint satisfaction problems that generalizes CHCs with arbitrary clauses, well-foundedness constraints, and functionality constraints. We have then established a program verification framework based on pfwCSP by showing that (1) pfwCSP can soundly-andcompletely encode various classes of relational problems of infinite-state nondeterministic programs, including hard instances of k-safety, co-termination, and termination-sensitive generalized non-interference that benefit from statedependent scheduling/alignment (Theorems 1–4), and (2) existing CHCs solving and invariants/ranking function synthesis techniques can be adopted to pfwCSP solving and further improved with the idea of stratified CEGIS for simultaneously achieving completeness (Theorem 5) and practical effectiveness.

In future work we plan to investigate ways to improve functional predicate synthesis, automatic tuning of parameter update strategies for constraint solving, and whether a constraint-based approach (and the techniques presented in the present paper) can be extended to reason about relational temporal properties such as the ones expressed in hyper temporal logics [16,25].

**Acknowledgments.** We thank the anonymous reviewers for their suggestions. This work was supported by ONR grant # N00014-17-1-2787, JST ERATO HASUO Metamathematics for Systems Design Project (No. JPMJER1603), and JSPS KAKENHI Grant Numbers 17H01720, 18K19787, 19H04084, 20H04162, 20H05703, and 20K20625.

#### **References**

1. Aguirre, A., Barthe, G., Gaboardi, M., Garg, D., Strub, P.: A relational logic for higher-order programs. J. Funct. Program. **29**, E16 (2019)


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Pre-deployment Security Assessment for Cloud Services Through Semantic Reasoning**

Claudia Cauli1(B) , Meng Li<sup>2</sup>, Nir Piterman<sup>1</sup>, and Oksana Tkachuk<sup>2</sup>

> <sup>1</sup> University of Gothenburg, Gothenburg, Sweden <sup>2</sup> Amazon Web Services, Seattle, U.S.A.

**Abstract.** Over the past ten years, the adoption of cloud services has grown rapidly, leading to the introduction of automated deployment tools to address the scale and complexity of the infrastructure companies and users deploy. Without the aid of automation, ensuring the security of an ever-increasing number of deployments becomes more and more challenging. To the best of our knowledge, no formal automated technique currently exists to verify cloud deployments during the design phase. In this case study, we show that Description Logic modeling and inference capabilities can be used to improve the safety of cloud configurations. We focus on the Amazon Web Services (AWS) proprietary declarative language, CloudFormation, and develop a tool to encode template files into logic. We query the resulting models with properties related to security posture and report on our findings. By extending the models with dataflow-specific knowledge, we use more comprehensive semantic reasoning to further support security reviews. When applying the developed toolchain to publicly available deployment files, we find numerous violations of widely-recognized security best practices, which suggests that streamlining the methodologies developed for this case study would be beneficial.

#### **1 Introduction**

The term *Infrastructure as Code* (*IaC* ) refers to the practice of configuring, provisioning, and updating systems resources from source code files, which are compiled into atomic instructions and then executed to deploy the desired architecture [29]. The advantage of handling code, instead of manually provisioning resources, lies in the capability to use version control systems, orchestration frameworks, and automated testing tools as part of the deployment process. In addition to instructions relevant for resource creation, dependencies, and updates, IaC configuration files contain information about settings, dataflow, and access control. In a time when cloud companies provide customers with simple-to-launch, albeit extremely powerful infrastructure, it is crucial to automatically and provably verify the security of such systems.

In this study, we investigate IaC deployment frameworks and how these are formally modeled and reasoned upon. We explore the usage of description c The Author(s) 2021

logics (DLs) as a conceptual-modeling formalism that is expressive, decidable, and equipped with mature tooling. We argue that formal reasoning techniques applied to deployment templates are an immensely valuable tool for developers and security engineers by substantially aiding the automation of time-consuming security reviews; helping them to detect complex logical errors at earlier stages; and, containing the costs that finding and fixing security issues at later stages would cause. As the prevalence of cloud infrastructure increases, in addition to experts, automated reasoning tools could benefit inexperienced users as well.

**System Studied.** We focus on the Amazon Web Services proprietary IaC tool, CloudFormation, the first to be introduced at a large scale, over ten years ago. AWS, cloud provider within Amazon, serves millions of customers worldwide. These include private businesses as well as government, education, nonprofit, and healthcare organizations. While the cloud provider is responsible for the faithful deployment of the customers' desired configurations, it is the customer's duty to make sure that these comply with the security requirements of their business context. Few management tools of this scale exist. Notable mentions are Terraform [37], Microsoft Azure's *Resource Manager* [28], Google Cloud's *Deployment Manager* [19], and the recently introduced OASIS standard TOSCA [6].

**Goal of Study.** Our goal is to improve the quality of the security analyses that are performed over IaC configurations pre-deployment; and by doing so, their overall security. With this study, we investigate the application of description logics to the formalization and reasoning over IaC deployments. In particular, we are interested in three aspects: *(i)* whether proposed cloud configurations comply with security best practices, *(ii)* how to aid customers in building more secure infrastructure *before* deploying it, and *(iii)* to what extent formal automated techniques can support manual pre-deployment security reviews.

**Challenges.** Little research has been done so far on the possibility to formalize IaC languages, and no research has been done to devise a logic that is well-suited to reason about cloud infrastructure. By nature, cloud infrastructure interacts with an open environment that is, at best, only partially known. In particular, external-facing APIs and users participate in these interactions. By design, cloud services allow for the composition of smaller components into large infrastructure, the complexity of which creates a challenge with respect to security. Our models should capture the connectivity of resources, the flow of information that spans across multiple paths, and the rich security-related data available in IaC configuration files. This is further complicated by the need for a query language for verification and falsification, able to express that mitigations must be present (vs. may be absent), and security issues must be absent (vs. may be present). Importantly, we need practical tools that support the implementation of all these parts and that can scale to real-world IaC configurations.

**Our Contribution.** We provide a framework to encode IaC into description logic, and investigate its effectiveness in answering configuration queries and reasoning about dataflow, trust boundaries, and potential issues within the system. Specifically, we test DLs reasoning capabilities to infer new facts about underspecified resources (such as those not included in a given deployment but used by it) and leverage DLs *open-world assumption* to perform verification and refutation, depending on the property being checked. We formalize additional security knowledge that allows for checking system-level semantic properties; i.e., properties that consider the nature of the cloud environment and more complex reachability over an *inferred* graph representation of the infrastructure.

Throughout the study, we make four novel contributions: (*i*) the formalization and logical encoding of AWS CloudFormation (Sect. 3); (*ii*) a technique to express security properties (Sect. 4); (*iii*) the experimental evaluation of encoding and query times, accounting for the most common security issues that we found over publicly available IaC templates (Sect. 5); and (*iv*) an extension that enables semantic dataflow reasoning (Sect. 6). Our tool is implemented in Scala and available online [14]. We include preliminaries in Sect. 2; discuss related work in Sect. 7; and conclude in Sect. 8.

#### **2 Preliminaries**

**Description Logics.** DLs are a family of logics well suited to model *relationships between entities*. They provide the logical foundation of the well-known *Web Ontology Language* [20,23,32], for which extensive tool support exists (e.g., the Prot´eg´e editor and off-the-shelf reasoners such as FaCT, HermiT, and Pellet [18,30,36,39]). We introduce the description logic ALC [1,24,34], *Attributive Logic with Complement*, and two additional features that are relevant for our study. ALC formulae are built from symbols from the alphabets N<sup>C</sup> , of atomic concept names; NR, of role names; and N<sup>I</sup> , of individual names. These are the DL equivalents of FOL unary predicates, binary predicates, and constants, respectively. ALC concept expressions are built according to the grammar:

$$C, D ::= \bot \mid \top \mid \mathsf{A} \mid \neg C \mid C \sqcup D \mid C \sqcap D \mid \exists \mathsf{r}. C \mid \forall \mathsf{r}. C$$

where <sup>A</sup> is an atomic concept from the set <sup>N</sup><sup>C</sup> ; C, D are possibly complex concepts; and r is a role from the alphabet <sup>N</sup>R. Terminological knowledge is represented via general concept inclusion axioms C D. As an example, in the remainder of this paper we will refer to two standard axioms that enforce the domain and range of binary relations: dom(r, C) ≡ ∃r.<sup>C</sup> and ran(r, C) ≡ ∃r−.C. Assertional knowledge is represented via concept assertions C(a) and role assertions r(a, b). In this paper, we will use three additional operators: *inverse roles*, *functionality constraints*, and *complex role inclusions*. The first, denoted r−, encodes the converse of the binary relationship r. The second enforces binary relationships to be functional. The third, written r ◦ s t, establishes that the chaining of the two relationships r and s implies the relationship t, and can be used to implement transitivity (when r = s = t). A model of a DL knowledge base is an interpretation I, over a domain Δ, that satisfies all the axioms and assertions contained and implied by the knowledge base. For the purpose of our application, we leverage two classical inference problems: *satisfiability* and *instance retrieval*, whose full definitions are found in standard textbooks [2,3].

**AWS CloudFormation.** AWS CloudFormation, cfn, provides users with a declarative programming language and a framework to provision and manage over 500 *resources* spread across 70 *services* [15].<sup>1</sup> Services are products such as storage, databases, and processors, and their interface is implemented through resources, which are the actual modules that users declare and deploy. Their declaration is done by writing one or more so-called *CloudFormation Templates* (JSON-formatted configuration files). Within a template, users configure settings and communication of the desired resource instances. As an example, let us consider one of the most widely known storage products within AWS: the Simple Storage Service S3 (also illustrated in Listings 1.1 and 1.2). The CloudFormation interface for S3 consists of two resources: S3::Bucket and S3::BucketPolicy. A Bucket is a single unit of storage whose properties include encryption, replication, and logging settings, which can be viewed as the bucket's own configuration parameters. They could also be *references* to other resources that are connected to the current one, e.g., the unique ID of another bucket where logs are stored. <sup>A</sup> BucketPolicy is a resource that links an access control policy to a bucket. All the properties that can be instantiated and the structure of resource-types such as S3::Bucket and S3::BucketPolicy are given in the *CloudFormation Resource Specification* [15]. The *resource specification* is a collection of files that prescribe resource properties and their allowed values. Provided that a *configuration file* is valid with respect to the specifications, an IaC deployment environment compiles it into instructions that are then executed to provision the requested resources in the correct dependency order and with the desired settings.

#### **3 Formalization and Encoding of IaC Deployments**

While setting up this case study, we found it convenient to come up with a formalization, of both IaC resource specifications and IaC configuration files, to use as an intermediate representation during the encoding process. This was also needed since we could not find suitable research in the area (although some preliminary research on IaC formalization does exist: e.g., the PhD thesis in [12]). As mentioned in Sect. 2, users consult the *resource specifications* to find out what fields and values are allowed when declaring a resource. Intuitively, these provide a sort of type-

```
"ResourceType":
"S3::Bucket": {
  "Properties":{
    "BucketName" : "String",
    "LoggingConfiguration": {
       "Type": "LoggingConfiguration",
       "Required": false } ... }} ,
"PropertyTypes": ...,
"S3::Bucket .LoggingConfiguration":{
  "Properties": {
    "DestinationBucketName":{
       "Type": "String",
       "Required": false },
    "LogFilePrefix":{
       "Type": "String",
       "Required": false }}}
```

```
Listing 1.1. S3::Bucket specification
```
system, or JSON schema, against which *configuration files* must validate. Configuration files contain the resource declarations of the instances that the user wishes to deploy. Let us illustrate this with some examples. Listing 1.1 shows a snippet of the S3::Bucket resource-type specification. In addition to the main

<sup>1</sup> As of August 2020, exact number is Region-dependent.

resource type, the specification includes definitions for its subproperties, their types, and whether these are required. Although the example only shows string properties, in general, allowed properties values range over objects, arrays, and primitive types such as integers, doubles, longs, strings, and booleans. Listing 1.2, on the other hand, shows a common usage scenario of the S3 storage service, where a bucket with basic configuration is used to store the desired data. The instance has logical ID ConfigS3Bucket, is of type S3::Bucket, and specifies two top-level properties, BucketName and LoggingConfiguration. It is easy to see that this instance declaration validates against the resource specification of Listing 1.1. This snippet is taken from one of the benchmark deployments evaluated in Sect. 5 (StackSet 15) and, incidentally, it violates a security best practice: "no bucket should store its own logs." Such formalization has been instrumental to capture infrastructure configurations, resources settings and inter-connections, and to precisely and automatically encode it into DL.

*Encoding.* We translate IaC specifications into DL terminological knowledge, and IaC configurations into assertional knowledge. The conceptual modeling features needed to model the former include axioms to define *domain* and *range* of properties, *requiredness*, and *functionality*. These give us enough expressivity to infer qualities of nodes that are under-

specified, such as those that are referenced by a template but not declared in it (e.g., already deployed and running elsewhere), whose configuration is unknown. To give readers an intuition of the encoding procedure, let us look at the equation below, which contains some of the axioms and assertions generated by the translation of the code in Listings 1.1 and 1.2.

```
SpecS3::Bucket = { dom(bucketName,BUCKET), ran(bucketName, String),
                (Funct bucketName), ..., dom(destinationBucket, LOGCONFIG),
                ran(destinationBucket,BUCKET), ... }
```

```
Config = { BUCKET(ConfigS3Bucket), bucketName(ConfigS3Bucket, "ConfigStore"),
           loggingConfig(ConfigS3Bucket, x), destinationBucket(x, ConfigS3Bucket),
           logFilePrefix(x, "config-bucket-logs") }
```
#### **4 Security Properties Specification**

We group properties into three categories that reflect their high-level meaning: *security issues*, *mitigations*, and *global protections* to security concerns. We view these in analogy to *must* and *may* specifications, which one would use to express that an issue may be present (vs. must be absent) or that a protection must be in place (vs. may be missing). Each property type is matched to a corresponding query structure, which aids the translation of security requirements into formal specifications and implements different fail/pass logics. Queries are written as description logic expressions whose outcome can be one of UNSAT, SAT with no instance found (SAT/0), and SAT with instances (SAT/+). These are achieved by running a satisfiability check, possibly followed by an instance retrieval call.

*Mitigations* are configurations of single resources that reduce the likelihood of a security event. In order to pass, these checks must be verified. Examples are:


*Security Issues* are configurations that potentially increase exposure to security concerns. In order to pass, these checks must be falsified. Examples are:


*Global Protections* are more general mitigations, applied on single resources or as configuration patterns, whose presence and proper configuration ensures protection over the system as a whole. Examples are:

P1 *"There is an alarm configured to perform an action when triggered,"* and P2 *"There is a configuration recorder logging changes to the infrastructure."*

We refer the reader to the repository in [14] for the properties specification files.<sup>2</sup>

### **5 Application to Existing Infrastructure**

We now discuss the application of our approach to real-world IaC deployments. We analyze AWS CloudFormation specification and configuration files, showing that the approach is practical, scalable, and identifies potential security issues.

*Operation of the Tool.* We develop a tool that performs three main tasks. First, the encoding of the cfn resource specifications into formal models (*Resource Terminologies*).<sup>3</sup> Second, the encoding of the actual cfn configuration files, also called *StackSet*, into formal models (*Infrastructure Model*). Third, inference and query answering for a set of predefined queries. We use the OWLApi [22] for the encoding phase, and JFact [39] as the inference engine.

<sup>2</sup> https://tiny.cc/PropertiesSpecifications.

<sup>3</sup> Available here: https://tiny.cc/ResourceTerminologies.


**Table 1.** Evaluation results (mean times in millisec).

*Experimental Setup.* We run our tool on 15 CloudFormation StackSets openly available on GitHub. Regarding metrics, we define the infrastructure size as the numbers of both declared resources (N) and their types (NRT ). The latter determines which *resource terminologies* are imported into the final encoded model and thus influences its size, measured in number of logical axioms (Nα). The smallest StackSet has 6 resources and 6 resource types, the largest has 508 resources and 21 resource types. We implement 50 properties from the ScoutSuite collection [35] that are applicable at design time and, thus, over IaC deployment files. Of the 50 properties, 29 are *mitigations*, 18 are *security issues*, and 3 are *global protections*. We conduct our evaluation on an Intel Core i5 with 16 GB RAM and perform warmup runs and clear the heap before each measurement. This tuning helps to minimize the impact of just-in-time compilation and to reduce the likelihood of garbage collection during the measured benchmark runs.

*Results Evaluation.* The average compilation time of the entire cfn resource specifications (542 files) was 940 ms. Table 1 reports the results of our experimental evaluation. StackSets are sorted by number of resources. For each, we measure the time taken by the stackset encoding (ENC), inference (INF), and query answering task (grouped by outcome: UNSAT, SAT with no instances, and SAT with instances). As we can see from the table, the encoding time increases with the infrastructure's size, producing larger models that require longer inference times. Average query answering times increase accordingly. UNSAT queries have shorter average answering times than those evaluating to SAT/0 or SAT/+ (UNSAT proofs are found before a SAT outcome can be deduced). In addition, once a query is proved SAT, we invoke a procedure for *instances retrieval* to determine whether satisfying instances are present or not. The specific infrastructure configuration and its size are the main influencing factors of query answering times. Considering that the average template has about 50–100 resources, and templates having 100–500 resources are rare, the results suggest that our approach scales to real-world IaC templates. For example, StackSet **04** has 132 resources, is encoded in 363 ms, classified in 2.1 s, and has a max average perquery time of 162 ms. Assuming a pool of 100 checks to be run, the automated modeling and verification of such an infrastructure would take, in the worst-case, around 18 s.

#### **5.1 Found Security Issues**

Across all 15 deployments, we run 15 × 50 = 750 checks: 608 pass and 142 fail. Of the 142 failing checks, 73 do not return any instance and 69 return one or more instances (i.e., they fail with a SAT/+ outcome). Such a difference is due to the nature of the single check and its definition of failure. A *global protection* check fails when no instance implementing the protection is found; a *security issue* check fails whenever is possible (SAT/0 or SAT/+); and a *mitigation* check fails when no instance is found. We consider SAT/+ findings particularly important, as they do not only witness a potential security issue but also an actual misconfiguration. In particular, the 69 SAT/+-failing checks fail on 239 resource instances, with the most found issues being:

	- *Missing or misconfigured logging* 46
	- *Missing User password reset requirement* 12
		- *Misconfigured authorization* 3
		- *Misconfigured networking configuration* 3

The 73 findings returning no instances fall into two groups: the absence of any monitoring or alarming system is very frequent, as is the dependency on external resources whose security posture cannot be assessed.

> *Absent global monitoring/alarming/logging protection* 41 *Usage of external resources with unknown configuration* 32

#### **6 Semantic Reasoning About Dataflows**

To conclude our study, we manually craft two proof-of-concept models of terms related to cloud security (ontologies). We use these to extend the formalization of the CloudFormation IaC specification that was automatically generated by our tool. Such domain-specific ontologies formalize several common cloud terms,

```
"CustomerData": {
  "Type": "AWS::S3::Bucket",
  "Properties": {
    "LoggingConfig": {
      "DestinationBucket": "
           AccessLog" }}},
"TopicSubscription":{
  "Type": "AWS::SNS::Subscription",
  "Properties": {
    "Endpoint": "devs@mail",
    "Protocol": "email",
    "TopicArn": "AccessTopic" }}
                                        "TestData": {
                                          "Type": "AWS::S3::Bucket",
                                          "Properties": {
                                             "LoggingConfig": {
                                               "DestinationBucket": "
                                                   AccessLog" }}},
                                        "AccessLog": {
                                          "Type": "AWS::S3::Bucket",
                                          "Properties": {
                                             "NotificationConfig" : {
                                               "TopicConfig" : {
                                                 "Topic":"AccessTopic" }}}},
                                        "AccessTopic": {
                                          "Type": "AWS::SNS::Topic" ... }
```
**Fig. 1.** Sample template: accounts *prod* (left) and *test* (right).

such as account, deployment, authenticated and unauthenticated users; generic dataflow terms, such as storage, process, nodes, and flows of different kind; and service-specific dataflow terms. By adding these on top of the underlying IaC formal specification, we can reason about the higher-level business logic and reachability of the infrastructure, and we can abstract it and visualize it in a more convenient way. This is where the full inference power of description logics comes into play. Such an inference power would be hard to achieve with an alternative encoding (e.g., using a modal logic). Let us illustrate how this technique is applied to system-level analyses of interest for a security review: *dataflow* and *trust boundary* analyses. A trust boundary is a portion of a system whose components trust each other and where data can securely flow. Multiple trust boundaries may exist within one system. Dataflows that travel across boundaries may introduce security issues and should be carefully reviewed. In Fig. 1, we see an example of such a situation, where the infrastructure is deployed across two accounts, *prod* and *test*, sharing resources AccessLog and AccessTopic. In our encoding, we use the so-called DLs *inclusion* axioms to rewrite properties that (when chained) imply the existence of a more general relation and to infer additional characteristics of nodes. For example, in the following list axioms 2– 7 formalize the relationships of *"logging to"* and *"sending notifications to"* a resource, which imply the existence of a *transitive dataflow* between nodes; and axioms 8–9 allow to infer that the node *devs@mail* is an external node.


This encoding enables us to compute a succinct dataflow diagram from the reasoned IaC configuration (see Fig. 2), and to formally verify properties that usually require a manual analysis of the infrastructure and its underlying graph representation. E.g., the question, *"can data flow from the customer-data bucket to the outside?"* can now be formalized as a DL formula and, using a

reasoning engine, the existence of a dataflow that starts on the *customerdata* bucket and reaches the *devs@mail* node can now be inferred. We note that, due to the structure of the TopicSubscription resource, this dataflow could not have been detected with simple reachability analysis on a graph built without the aid of semantic reasoning. Moreover, the dataflow diagram highlights another potential

**Fig. 2.** Dataflow extracted from Fig. 1

source of information leakage: testers being exposed to customer access information. This needs to be mitigated by enforcing the proper trust boundaries, in particular, by adding a dedicated access log storage for *customer-data* bucket in the *prod* account.

#### **7 Related Work**

To the best of our knowledge, the problem of formally verifying the design of a cloud infrastructure in its entirety has not been addressed before. Formal reasoning techniques have been successfully applied to different aspects of the cloud, e.g. networks and access policies [4,5,7,16]. Non-formal tools exist that recommend and run checks against *already deployed* resources [13,35], or scan IaC templates [10,11,38] for syntactical patterns violating security best practices. These checks overlap considerably and can be expressed in our framework as well. The disadvantages of such tools are that checks are local to single components, can be performed only post-deployment, need complex configurations, access permissions, or even manual interaction. The CFn-Linter [10] has a rule-based component that users can extend with custom syntax checks, but none of the rules currently available focus on security. The CFn-nag linting tool [11] checks compliance to best practices only locally to the single resources; e.g., it cannot detect issues such as *"there is an events queue, receiving from a bucket with critical functionality, that may not be encrypted"* or *"there might be a user that is shared by multiple policies"* (which would go against the *least privilege* principle); as well as including in its analysis external resources that are referenced by the template being linted.

Regarding our choice of logic, large-scale configuration problems have been tackled with description logic before [26,27]. Simpler first-order logic formulas with operators to represent object-oriented interface relationships could be used to model IaC specifications. However, such an encoding would only partially solve our problem, which is more complex because our overall goal is to do formal semantic analyses (e.g., dataflow and threat modeling). Semantic-based approaches, even DL-based, are being used to do conceptual modeling of security engineers' expertise with the provable and explainable inference capabilities of logics. As an example, we refer the reader to the OWASP *"Ontology-driven Threat Modeling"* project [31] that aims at the formalization of security-related knowledge in the context of different types of computer systems by means of description logic ontologies. In contrast to logic programming languages, such as Datalog, DLs inherently support functionality axioms and the existence of anonymous individuals within a domain that is assumed to be open. These are supported out-of-the-box without the need for an additional, more complex, axiomatization or encoding. In particular, we took advantage of DL's openworld assumption to implement, in our properties encoding, verification and falsification. Another alternative to DLs as a modeling language would be to use 3-valued models with labels on states and transitions and apply model checking [8,9]. However, expressive branching-time logics [25,33] have not been studied in the context of 3-valued models and we are also not aware of tool support at the level available for DLs (cf. [17,21]).

#### **8 Conclusion and Future Work**

Throughout this case study, we investigated the usage of description logicsbased semantic reasoning to evaluate the security of cloud infrastructure predeployment. We encoded Amazon Web Services' Infrastructure as Code specifications and configurations into description logic models and verified the presence and absence of potential security issues. We showed how this approach enables deeper system-level analyses such as dataflow analysis. All results can be generalized to other existing IaC tools. While working on this project, we interacted with developers on two occasions. First, for the benchmark templates used in our experimental evaluation, we contacted the owners, told them about the misconfigurations, and discussed potential security implications. Second, within AWS, security engineers use a technique based on this paper for security reviews of AWS products before they are launched, helping developers fix real issues pre-deployment. In the process, we received valuable feedback that we used for improving precision and reducing the number of false-positive results. We plan to continue researching for an even better-fitting description logic formalism, query language, three-valued semantics, and decision procedures for verification and falsification of properties relevant to security analyses, such as dataflows, trust boundaries, and threat modeling.

**Acknowledgements.** This research is supported by the ERC consolidator grant D-SynMA under the European Union's Horizon 2020 research and innovation programme (grant agreement No. 772459) and by Amazon Web Services.

### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

# **Synthesis**

### **Synthesis with Asymptotic Resource Bounds**

Qinheping Hu(B) , John Cyphert, Loris D'Antoni, and Thomas Reps

University of Wisconsin-Madison, Madison, USA *{*qhu28,jcyphert,ldantoni,treps*}*@wisc.edu

**Abstract.** We present a method for synthesizing recursive functions that satisfy both a functional specification and an asymptotic resource bound. Prior methods for synthesis with a resource metric require the user to specify a *concrete* expression exactly describing resource usage, whereas our method uses big-*O* notation to specify the *asymptotic* resource usage. Our method can synthesize programs with complex resource bounds, such as a sort function that has complexity *O*(*n* log(*n*)).

Our synthesis procedure uses a type system that is able to assign an asymptotic complexity to terms, and can track recurrence relations of functions. These typing rules are justified by theorems used in analysis of algorithms, such as the Master Theorem and the Akra-Bazzi method. We implemented our method as an extension of prior type-based synthesis work. Our tool, SynPlexity, was able to synthesize complex divideand-conquer programs that cannot be synthesized by prior solvers.

#### **1 Introduction**

Program synthesis is the task of automatically finding programs that meet a given behavioral specification, such as input-output examples or complete formal specifications. Most of the work on program synthesis has been devoted to qualitative synthesis, i.e., finding *some* correct solution. However, programmers often want more than just a correct solution—they may want the program that is smallest, most likely, or most efficient. While there are some techniques for adding a quantitative *syntactic* objective in program synthesis [12]—e.g., finding a smallest solution, or a most likely solution with respect to some distribution little attention has been devoted to quantitative *semantic* objectives—e.g., synthesizing a program that has a certain asymptotic complexity.

Recently, Knoth et al. [16] studied the problem of resource-guided program synthesis, where the goal is to synthesize programs with limited resource usage. Their approach, which combines refinement-type-directed synthesis [18] and automatic amortized resource analysis (AARA) [9], is restricted to *concrete* resource bounds, where the user must specify the *exact* resource usage of the synthesized program as a *linear* expression. This limitation has two drawbacks: (i) the user must have insights about the coefficients to put in the supplied bound—which means that the user has to provide details about the complexity of code that does not yet exist; (ii) the limitation to linear bounds means that the user cannot specify resource bounds that involve logarithms, such as *O*(log *n*) and *O*(*n* log *n*), common in problems based on divide and conquer.

In this paper, we introduce SynPlexity, a type-system paired with a typedirected synthesis technique that addresses these issues. In SynPlexity, the user provides as input a refinement type that describes both the functionality and the *asymptotic* (big-*O*) resource usage of a program. For example, a user might ask SynPlexity to synthesize an implementation of a sorting function with resource usage *O*(*n* log *n*), where *n* is the length of the input list. As in prior work, SynPlexity also takes as input a set of auxiliary functions that the synthesized program can use. SynPlexity then uses a type-directed synthesis algorithm to search for a program that has the desired functionality, and satisfies the asymptotic resource bound. SynPlexity's synthesis algorithm uses a new type system that can reason about the asymptotic complexity of functions. To achieve this goal, this type system uses two ideas.


Gu´eneau et al. observed that reasoning with *O*-notation can be tricky, and exhibited a collection of plausible-sounding, but flawed, inductive proofs [8, §2]. We avoid this pitfall via SynPlexity's type system, which establishes whether a term satisfies a given recurrence relation. SynPlexity uses theorems that connect the form of a recurrence relation—e.g., the number of recursive calls, and the argument sizes in the subproblems—to its asymptotic complexity. In particular, the SynPlexity type system does not encode inductive proofs of the kind that Gu´eneau et al. show can go astray.

SynPlexity can synthesize functions with complexities that cannot be handled by existing type-directed tools [16,18], and compares favorably with existing tools on their benchmarks. Furthermore, for some domains, SynPlexity's type system allows us to discover auxiliary functions automatically (e.g., the split function of a merge sort), instead of requiring the user to provide them.

<sup>1</sup> The recurrence relation above is one possible instantiation of the Master Theorem [5, §4.5 and §4.6]; it can also be instantiated as *<sup>T</sup>*(*u*) <sup>≤</sup> <sup>2</sup>*T*( *<sup>u</sup>* <sup>2</sup> ) + *<sup>O</sup>*(*u*). The type system makes use of certain templates for instantiating the algorithm-analysis theorems that we use. The use of templates means that the type system does not use all possible instantiations, but all instantiations used in the type system are valid ones.

*Contributions.* The contributions of our work are as follows:


Complete proofs and details of the type system can be found in the technical report [11].

#### **2 Overview**

In this section, we illustrate the main components of our algorithm through an example. Consider the problem of synthesizing a function prod that implements the multiplication of two natural numbers, *x* and *y*. We want an efficient solution whose time complexity is *O*(log *x*) with respect to the value of the first argument *x*. In Subsect. 2.1, we show how existing type-directed synthesizers solve this problem in the absence of a complexity-bound constraint. In Subsect. 2.2, we illustrate how to specify asymptotic bounds in type-directed synthesis problems. In Subsect. 2.3, we show how the tracking of recurrence relations can be used to establish complexity bounds as well as guide the synthesis search.

#### **2.1 Type-Directed Synthesis**

We first review one of the state-of-the-art type-directed synthesizers, Synquid, through the aforementioned example—i.e., synthesizing a program prod that computes the product of two natural numbers. In Synquid, the specification is given as a refinement type that describes the desired behavior of the synthesized function. We specify the behavior of prod using the following refinement-type:

$$\{\text{prod} :: \mathbf{x} : \{\mathbf{1}\mathbf{nt} \mid v \ge 0\} \to \mathbf{y} : \{\mathbf{1}\mathbf{nt} \mid v \ge 0\} \to \{\mathbf{1}\mathbf{nt} \mid v = \mathbf{x} \* \mathbf{y}\} \dots\}$$

Here the types of the inputs x and y, as well as the return type of prod are refined with predicates. The refinement {Int | *v* ≥ 0} declares x and y to be non-negative, and the refinement {Int | *v* = x∗y} of the return type declares the output value to be an integer that is equal to the product of the inputs x and y. In addition to the specification, the synthesizer receives as input some signatures of auxiliary functions it can use. The specifications of auxiliary functions are also given as refinement types. In our example, we have the following functions:

$$\begin{array}{ll} \mathsf{even} :: \mathtt{x} : \mathtt{Int} \to \{ \mathtt{Bool} \mid \mathtt{x} \ \mathtt{mod} \, 2 = 0 \} & \mathsf{dec} :: \mathtt{x} : \mathtt{Int} \to \{ \mathtt{Int} \mid v = \mathtt{x} - 1 \} \\\\ \mathsf{double} :: \mathtt{x} : \mathtt{Int} \to \{ \mathtt{Int} \mid v = \mathtt{x} + \mathtt{x} \} & \mathsf{div} 2 :: \mathtt{x} : \mathtt{Int} \to \{ \mathtt{Int} \mid v = \lfloor \frac{\mathtt{x}}{2} \rfloor \} \\\\ \mathsf{push} :: \mathtt{x} : \mathtt{Int} \to \mathtt{y} : \mathtt{Int} \to \{ \mathtt{Int} \mid v = \mathtt{x} + \mathtt{y} \} \end{array}$$

With the above specification and auxiliary functions, Synquid will output the implementation of prod shown in Eq. (1).

#### prod = *λ*x.*λy.* **if** x==0 **then** x **else** plus y (prod (dec x) y) (1)

Synquid uses a sophisticated type system to guarantee that the synthesized term has the desired type. Furthermore, Synquid uses its type system to prune the search space by only enumerating terms that can possibly be typed, and thus meet the specification. Terms are enumerated in a top-down fashion, and appropriate specifications are propagated to sub-terms. As an example, let us see how Synquid synthesizes the function body—an if-then-else term—in Eq. (1), which is of refinement type {Int <sup>|</sup> *<sup>v</sup>* <sup>=</sup> <sup>x</sup>∗y}. Synquid will first enumerate an integer term for the then branch—a variable term x. Then, with the then branch fixed, the condition guard must be refined by some predicate *ϕ* under which the then branch (the term x refined by *v* = x) fulfills the goal type {Int | *v* = x ∗ y}, i.e., <sup>∀</sup>x*,* <sup>y</sup> <sup>≥</sup> <sup>0</sup>*.ϕ* <sup>∧</sup> *<sup>v</sup>* <sup>=</sup> <sup>x</sup> <sup>=</sup><sup>⇒</sup> *<sup>v</sup>* <sup>=</sup> <sup>x</sup> <sup>∗</sup> <sup>y</sup>. With this constraint, Synquid identifies the term x == 0 as the condition. Finally, Synquid propagates the negation of the condition to the else branch—the else branch should be a term of type {Int | *v* = x ∗ y} with the path condition x = 0—and enumerates the term plus y (prod (dec x) y) as the else branch, which has the desired type.

The program in Eq. (1) is correct, but inefficient. Let us count each call to an auxiliary function as one step; and let *T*(*x*) denote the number of steps in which the program runs with input *x*. The implementation in Eq. (1) runs in *Θ*(*x*) steps because *T*(*x*) satisfies the recurrence *T*(*x*) = *T*(*x*−1)+2, implying *T*(*x*) ∈ *Θ*(*x*). Because, Synquid does not provide a way to specify resource bounds, such as *O*(log *x*); one cannot ask Synquid to find a more efficient implementation.

#### **2.2 Adding Resource Bounds**

In our tool, SynPlexity, one can specify a synthesis problem with an asymptotic resource bound, and can ask SynPlexity to find an *O*(log *x*) implementation of prod. To express this intent, the user needs to specify (1) the asymptotic resource-usage bound the synthesized program should satisfy, (2) the cost of each provided auxiliary function, and (3) the size of the input to the program.

*Asymptotic Resource Bound.* We extend refinement types with resource annotations. The annotated refinement types are of the form *τ* ; *α* where *τ* is a regular refinement type, and *α* is a resource annotation. The following example asks the synthesizer to find a solution with the resource-usage bound *O*(log *u*):

$$\mathtt{prod} :: \langle \mathtt{x} ; \{\mathtt{Int} \mid v \ge 0\} \to \mathtt{y} ; \{\mathtt{Int} \mid v \ge 0\} \to \{\mathtt{Int} \mid v = \mathtt{x} \* \mathtt{y}\}, O(\log u) \rangle$$

*Cost of Auxiliary Functions.* The auxiliary functions supplied by the user serve as the API in terms of which the synthesized program is programmed. Thus, the resource usage of the synthesized program is the sum of the costs of all auxiliary calls made during execution. We allow users to assign a polynomial cost *O*(*u<sup>a</sup>*), for some constant *a*, or a constant cost *O*(1) to each auxiliary function. Here, *u* is a free variable that represents the size of the problem on which the auxiliary function is called.

In the prod example, all auxiliary functions are assigned constant cost, e.g., we give even the signature even :: x:Int → {Bool | x mod 2=0}*, O*(1).

*Size of Problems.* The user needs to specify a size function, size:*τ* → Int, that maps inputs to their sizes, e.g., when synthesizing the sorting function for an input of type list, the size function can be *λ*l*.*|*l*|—the length of the input list. In the prod example, the size function is size = *λ*x*.λ*y*.*x.

#### **2.3 Checking Recurrence Relations**

We extend Synquid's refinement-type system with resource annotations, so that the extended type system enforces the resource usage of terms. The idea of the type system is to check if the given function satisfies some recurrence relation. If so, it can infer that the function also satisfies the corresponding resource bound. For example, according to the Master Theorem [3], if a function *f* satisfies the recurrence relation *<sup>T</sup>*(*u*) <sup>≤</sup> *<sup>T</sup>*( *<sup>u</sup>* <sup>2</sup> ) + *O*(1) where *u* is the size of the input, then the resource usage of *f* is bounded by *O*(log *u*). Checking if a function satisfies a given recurrence relation can be performed by checking if the function contains appropriate recursive calls—e.g., if a function contains one recursive call to a sub-problem of half size, and consumes only a constant amount of resources in its body, then it satisfies *<sup>T</sup>*(*u*) <sup>≤</sup> *<sup>T</sup>*( *<sup>u</sup>* <sup>2</sup> ) + *O*(1).

The following rule is an example of how we connect recurrence annotations and resource bounds.

*x* : *τ*1*, f* : *τ*<sup>1</sup> → *τ*2*, Γ t* :: *τ*2; ([1*, u* 2 ]*, O*(1)) *Γ* (fix *f. λx.t*) :: *τ*<sup>1</sup> → *τ*2; *O*(log *u*)

The rule instantiates the Master Theorem example above. Note that, the annotation ([1*, <sup>u</sup>* <sup>2</sup> ]*, O*(1)) states that the function body contains up to one recursive call to a problem of size *<sup>u</sup>* <sup>2</sup> , and the resource usage in the body of *t* (aside from calls to *f* itself) is bounded by *O*(1). The rule states that if the function body *<sup>t</sup>* of type *<sup>τ</sup>*<sup>2</sup> contains one recursive call to a sub-problem of size *<sup>u</sup>* <sup>2</sup> , then the function will be bounded by *O*(log *u*).

The implementation of prod shown in Eq. (2) runs in *O*(log *x*) steps.

$$\texttt{prod} = \lambda \texttt{x}. \ \lambda \texttt{y}. \ \texttt{if} \ \texttt{x} == \ \texttt{0} \ \texttt{then} \ \texttt{ x} \ \texttt{else} \tag{2}$$
 
$$\texttt{if} \ \texttt{even} \ \texttt{x} \ \texttt{then} \ \texttt{double (prod} \ \texttt{(div2 x) y} \ \texttt{y}$$
 
$$\texttt{else} \ \texttt{ plus} \ y \ \texttt{(double (prod} \ \texttt{(div2 x) y)} \ \texttt{)}$$

To check that, SynPlexity's type system counts the number of recursive calls along any path of the function. There are three paths (two nested if-then-else terms) in the program, and at most one recursive call along each path. Also, one can check that the problem size of each recursive call is no more than <sup>x</sup> 2 .

**Fig. 1.** SynPlexity syntax.

$$\begin{array}{lcll} \text{Logical expr.} & \varphi, \phi, \psi ::= \boldsymbol{x} \mid \mathfrak{m}(\psi) \mid \top \mid \perp \mid c \mid \psi \bmod \psi \mid \psi \wedge \psi \mid \psi \vee \psi\\ & & \mid \neg \psi \mid \psi = \psi \mid \psi \* \psi \mid \psi \vee \psi \mid \psi + \psi \mid \psi - \psi\\ \text{Ordinary type} & & B ::= \textbf{Bool} \mid \mathsf{Int} \mid D\\ \text{Refinement type} & & \tau ::= \{B \mid \varphi \} \mid x\_1 : \tau\_1 \to \dots \to x\_n : \tau\_n \to y : \tau\\ \text{Annotated type} & & \gamma ::= \langle \tau; \alpha \rangle\\ \text{Recurrence ann.} & & \alpha ::= \langle \ [c\_1, \phi\_1]\_{\mathfrak{f}}, \dots, [c\_n, \phi\_n]\_{\mathfrak{f}}; O(\psi) \rangle\\ \text{Enviromment} & & \Gamma ::= \cdot \mid x : \gamma; \Gamma \mid \varphi; \Gamma \mid \mathsf{recFun} :: x; \Gamma \mid \mathsf{args} ::= x\_1 \dots x\_n; \Gamma \mid \mathsf{c} \rangle\\ \end{array}$$

**Fig. 2.** SynPlexity types.

For example, the recursive call prod (div2 x) y calls to a problem with size div2 x, which is consistent with [1*, <sup>u</sup>* <sup>2</sup> ], and *u* is x because size x y = x. In addition, the condition that the resource usage of the body is bounded by *O*(1) is satisfied because only auxiliary functions with constant cost are called.

#### **3 The SYNPLEXITY Type System**

In this section, we present our type system. First, we give the surface language and the types, which extend the Synquid liquid-types framework with resource annotations (Subsect. 3.1). Then, we show the semantics of our language (Subsect. 3.2). Finally, we present SynPlexity's type system (Subsect. 3.3), which our synthesis algorithm uses to synthesize programs with desired resource bounds.

#### **3.1 Syntax and Types**

*Syntax.* Consider the language shown in Fig. 1. In the language, we distinguish between two kinds of terms: *elimination terms* (E-terms) and *introduction terms* (I-terms). E-terms consist of variable terms, constant values *c*, and application terms. Condition guards and match scrutinies can only be E-terms. I-terms are branching terms and function terms. The key property of I-terms is that if the type of any I-term is known, the types of its sub-terms are also known (which is not the case for E-terms).

*Types.* Our language of types, presented in Fig. 2, extends the one of Synquid [18] with *recurrence annotations*, which are used to track recurrence relations on functions. To simplify the presentation, we ignore some of the features of the type system of Synquid [18] that do not affect our algorithm. In particular, we do not discuss polymorphic types and the enumerating strategy that ensures that only terminating programs are synthesized. However, our implementation is built on top of Synquid, and supports both of those features.

Logical expressions are built from variables, constants, arithmetic operators, and other user-defined logical functions. Logical expressions in our type system can be used as refinements *ϕ*, size expressions *φ*, or bound expressions *ψ*. Refinements *ϕ* are logical predicates used to refine ordinary types in refinement types {*B* | *ϕ*}. We usually use a reserved symbol *v* as the free variable in *ϕ*, and let *v* represents the inhabitants, i.e., inhabitants of the type {*B* | *ϕ*} are valuations of *v* that satisfy *ϕ*. For example, the type {Int | *v* mod 2=0} represents the even integers. Size expressions and bound expressions are used in recurrence annotations, and are explained later.

Ordinary types includes primitive types and user-defined algebraic datatypes *D*. Datatype constructors C are functions of type *τ*1→ *...* →*τ<sup>n</sup>* → *D*. For example, the datatype List(Int) has two constructors: Cons : Int → List(Int) → List(Int), and Nil : List(Int). Refinement types are ordinary types refined with some predicates *ψ*, or arrow types. Note that, unlike Synquid's type system, SynPlexity's type system does not support higher-order functions<sup>2</sup>—i.e., arguments of functions have to be non-arrow types. All occurrences of *τ<sup>i</sup>* and *τ* in arrow types *x*<sup>1</sup> : *τ*1→ *...* →*x<sup>n</sup>* : *τn*→*y* : *τ* have to be ordinary types or refined ordinary types. We will discuss this limitation in Sect. 7.

We use recFun to denote the name of the function for which we are performing type-checking, and args to denote the tuple of arguments to recFun. For example, in the function prod shown in Eq. (1), recFun=prod and args=x y. An environment *Γ* is a sequence of variable bindings *x* : *γ*, path conditions *ϕ*, and assignments for variables recFun and args.

*Recurrence Annotations.* Annotated types are refinement types annotated with recurrence annotations. A recurrence annotation is a pair ([*c*1*, φ*1]f*,...,* [*cn, φn*]f; *O*(*ψ*)) consisting of (1) a set of recursive-call costs of the form [*ci, φi*]f, and (2) a resource-usage bound of the form *O*(*ψ*). Intuitively, a recurrence annotation tracks the number *c<sup>i</sup>* of recursive calls to f of size *φ<sup>i</sup>* in the first element [*c*1*, φ*1]f*,...,* [*cn, φn*]<sup>f</sup> of the pair, as well as the asymptotic resource usage of the *body* of the function (the second element *O*(*ψ*)). Using these quantities, we can compute a recurrence relation describing the resource usage of the function recFun. For example, the recurrence annotation ([1*, u* − 1]f*,* [1*, u* − 2]f; *O*(1)) corresponds to the recurrence relation *T*f(*u*) ≤ *T*f(*u* − 1) + *T*f(*u* − 2) + *O*(1).

A *recursive-call cost* [*c, φ*]<sup>f</sup> associated with a function f denotes that the body of f can contain up to *c* recursive calls to subproblems that have sizes up to the one specified by size expression *φ*. A size expression, *φ*, is a polynomial over a reserved variable symbol *u* that represents the size of the top-level problem. In our paper, a *problem* with respect to a function *g* :: *x*<sup>1</sup> : *τ*<sup>1</sup> → *...* →*x<sup>n</sup>* : *τ<sup>n</sup>* →*y* : *τ* is a tuple of terms *e*<sup>1</sup> *...en*, to which *g* can be applied—i.e., *e<sup>i</sup>* has type *τ<sup>i</sup>* for all

<sup>2</sup> However, the type system can be extended to support restricted higher-order functions (Sect. 5).

*i* from 1 to *n*. For the problems of function *g*, the size of each problem is defined by a *size function* size*g*—a user-defined logical function that has type *τ*<sup>1</sup> → *...* →*τn*→Int; i.e., it takes a problem of *g* as input and outputs a non-negative integer. In the body of *g*, we say that a recursive-call term *g e*<sup>1</sup> *...e<sup>n</sup> satisfies* a size expression *φ* if for all *x*1, *...*, *xn*, size*<sup>g</sup> e*1 *... en* ≤ [(size*<sup>g</sup> x*<sup>1</sup> *...xn*)*/u*]*φ*, where the *xi*'s are the arguments of *g* and the *ei*'s are the evaluations of *e<sup>i</sup>* on input *x*<sup>1</sup> *...xn*. (See Sect. 3.2 for the formal definition of -·.) Note that one annotation can contain multiple recursive-call costs, which allows the function to make recursive calls to sub-problems with different sizes. We often abbreviate *τ,*(*O*(1)) as *τ* and omit f in recursive-call costs if it is clear from context.

A resource bound *O*(*ψ*) of a non-arrow type specifies the bound of the resource usage strictly within the top-level-function body. A resource bound in a signature of an auxiliary function *f* specifies the resource usage of *f*. *Bound expressions ψ* in *O*(*ψ*) are of the form *u<sup>a</sup>* log*<sup>b</sup> u* + *c* where *a*, *b*, and *c* are all non-negative constants, and *u* represents the size of the top-level problem.

*Example 1.* In the function prod (Eq. (2)), the recursive-call term prod (div2 x) y satisfies the recursive-call cost [1*, <sup>u</sup>* <sup>2</sup> ], because sizeprod = *λz.λw.z*, and

$$\texttt{size}\_{\texttt{prod}} \,\, [(\texttt{div2 x})] \,\, [\texttt{y}] = \,\, [\texttt{div2 x}] = \lfloor \frac{\texttt{x}}{2} \rfloor = [(\texttt{size}\_{\texttt{prod}} \,\, \texttt{x y})/u] \lfloor \frac{u}{2} \rfloor.$$

#### **3.2 Semantics and Cost Model**

We introduce the *concrete*-cost semantics of our language here. The semantics serves two goals: (1) it defines the evaluation of terms (i.e., how to obtain values), which can be used to compute the sizes of problems in application expressions, and (2) it defines the resource usages of terms.

Besides the syntax shown in Fig. 1, implementations of auxiliary functions can contain calls to a tick function tick(*c, t*), which specifies that *c* units of a resource are used, and the overall value is the value of *t*. Note that in our synthesis language, we are not actually synthesizing programs with tick functions. We assume that tick functions are only called in the implementations of auxiliary functions. In the concrete-cost semantics, a configuration *t, C* consists of a term *t* and a nonnegative integer *C* denoting the resource usage so far. The evaluation judgment *t, C* → *t , C* +*CΔ* states that a term *t* can be evaluated in one step to a term (or a value) *t* , with resource usage *CΔ*. We write *t, C* →<sup>∗</sup> *t , C*+*CΔ* to indicate the reduction from *t* to *t* in zero or more steps. All of the evaluation judgments are standard. Here we show the judgment of the tick function, where resource usage happens.

$$\overline{\langle \mathtt{tick}(c,t),C\rangle \hookrightarrow \langle t,C+c\rangle}^{\text{SEM-TICK}}$$

For a term *t*, *t* denotes the evaluation result of *t*, i.e., *t,* · →<sup>∗</sup> *t,* ·.

*Example 2.* Consider the following function that doubles its input.

fix double*.λ*x*.***if** x=0 **then** 0 **else** tick(1,2 + double(x-1))*.*

Let *t*body denote the function body **if** x=0 **then** 0 **else** tick(1,2+double(x-1)). The result of evaluating double on input 5 is 10, with resource usage 5.

```

(fix double.λx.tbody)5, 0
```
→ **if** 5=0 **then** 0 **else** tick(1,2+double(4))*,* 0

→ **if** false **then** 0 **else** tick(1,2+ double(4))*,* 0


With the standard concrete semantics, the complexity of a function *f* is characterized by its resource usage when the function is evaluated on inputs of a given size.

**Definition 1 (Complexity).** *Given a function fix f.λy.t of type* : *τ*<sup>1</sup> → *τ*2*, with size function size<sup>f</sup>* : *<sup>τ</sup>*<sup>1</sup> <sup>→</sup> <sup>N</sup>*, and suppose that for any possible input <sup>x</sup>, the configuration* (*fix f.λy.t*)*x,* 0 *can be reduced to v, Cx for some value <sup>v</sup>. Then, if <sup>T</sup><sup>f</sup>* : <sup>N</sup> <sup>→</sup> <sup>N</sup> *is a function such that, for all,* <sup>u</sup> <sup>≥</sup> <sup>0</sup>*, T<sup>f</sup>* (u) = sup*<sup>x</sup> s.t. size<sup>f</sup>* (*x*)=u *Cx, we say that T<sup>f</sup> is* **the complexity function of** *f.*

Note that Definition 1 assumes that the top-level term (fix *f.λy.t*)*x* can be reduced to some value. Thus, Definition 1 only applies to terminating programs.

**Definition 2 (Big-O notation).** *Given two integer functions f and g, we say that f dominates g, i.e., g* ∈ *O*(*f*)*, if* ∃*c, M* ≥ 0*.* ∀*x* ≥ *c. g*(*x*) ≤ *Mf*(*x*)*.*

In the rest of the paper, we use *T<sup>f</sup>* to denote the complexity function of the function *f*, and we say the complexity of *f* is *bounded* by a function *g* if *T<sup>f</sup>* ∈ *O*(*g*). As an example, the complexity of the double function shown in Example 2 is *T*double(*u*) := *u*, and hence *T*double(*u*) ∈ *O*(*u*).

*Auxiliary functions.* We allow users to supply signatures for auxiliary functions, instead of implementations. It is an obligation on users that such signatures be sensible; in particular, when the user gives the signature *τ*<sup>1</sup> → {*B* | *ϕ*(*v, y*)}*, O*(*ψ*(*u*)) for auxiliary function *f*, the user asserts that there exists some implementation fix *f.λy.t* of *f*, such that: 1) for any input *x*, the output of *f* on *x* satisfies *ϕ*, i.e., *ϕ*(-(fix *f.λy.t*)*x, x*) is valid; and 2) for any input *x*, the complexity of *f* is bounded by *ψ*(*u*), i.e., *T<sup>f</sup>* (*u*) ∈ *O*(*ψ*(*u*)). Signatures always over-approximate their implementations, as illustrated by the following example.

*Example 3.* The signature doubleRelaxed :: x:Int → {Int <sup>|</sup> *<sup>v</sup>* <sup>≤</sup> <sup>3</sup> <sup>∗</sup> *<sup>x</sup>*}*, <sup>O</sup>*(*u*<sup>2</sup>) describes an auxiliary function that computes *no more* than the input times 3, and has quadratic resource usage. Note that the function double shown in Example 2 can be an implementation of this signature because double(*x*) = <sup>2</sup> <sup>∗</sup> *<sup>x</sup>* <sup>≤</sup> <sup>3</sup> <sup>∗</sup> *<sup>x</sup>*, and the complexity function *<sup>T</sup>*double(*u*) = *<sup>u</sup>* is in *<sup>O</sup>*(*u*<sup>2</sup>).

#### **3.3 Typing Rules**

The typing rules of SynPlexity are inspired by bidirectional type checking [17] and type checking with cost sharing [16]. Recall that we use recFun to denote the name of the function for which we are performing type-checking, and args to denote the tuple of arguments to recFun.

An environment *Γ* is a sequence of variable bindings of the form *x* : *γ*, path conditions *ϕ*, and assignments of the form *x* = *ϕ* for recFun and the components of args. SynPlexity's typing rules use three judgments: 1) *<sup>Γ</sup> t* :: *γ* states that *t* has type *γ*, 2) *Γ γ*<sup>1</sup> *<*: *γ*<sup>2</sup> states that *γ*<sup>2</sup> is a subtype of *γ*1, and 3) *Γ γ γ*1|*γ*<sup>2</sup> states that *γ*<sup>1</sup> and *γ*<sup>2</sup> share the costs in *γ*

*Subtyping.* The subtyping relations between refinement types are relatively standard and can be found in the technical report [11]. The subtyping relations between annotated types allow us to compare resource consumption of recurrence annotations. The following is the rule for comparing recursive-call costs.

$$\frac{c' > c \quad \Gamma \vdash \forall u. \ \phi' \ge \phi}{\Gamma \vdash [c, \phi] <: \ [c', \phi']} <: \text{-Rec} \ . $$

For example, if one branch of some branching term has type *τ,*([1*, <sup>u</sup>* <sup>3</sup> ]*, O*(*ψ*)), it can be over-approximated by a super type *τ,*([1*, <sup>u</sup>* <sup>2</sup> ]*, O*(*ψ*)). The idea is that the resource usage of an application calling to a problem of size *<sup>u</sup>* <sup>2</sup> will be larger than the resource usage of the application calling to a smaller problem of size *<sup>u</sup>* <sup>3</sup> (assuming all resource usages are monotonic).

Subtyping rules also allow the type system to compare branches with a different number of recursive calls. For example, base cases of recursive procedures have no recursive calls, and thus have types of the form *τ,*([]*, O*(*ψ*)). With subtyping, these types can be over-approximated by types of the form *τ,*([*c, φ*]*, O*(*ψ*)).

*Cost Sharing.* When a term has more than one sub-term in the same path, e.g., the condition guard and the then branch are in the same path in an ite term, the recursive-call costs of the term will be shared among its sub-terms. The sharing operator *α α*1|*α*<sup>2</sup> partitions the recursive-call costs of *α* into *α*<sup>1</sup> and *α*2—i.e., the sum of the costs in *α*<sup>1</sup> and *α*<sup>2</sup> equals the cost in *α*. The following is the sharing rule for a single recursive-call cost:

$$\frac{c\_1, c\_2 \ge 0 \quad \quad c\_1 + c\_2 \le c}{\Gamma \vdash [c, \phi] \lor [c\_1, \phi] \mid [c\_2, \phi]} \text{ S-Por}$$

Other sharing rules can be found in the technical report [11]. The idea is that a single cost *c* can be shared as two costs *c*<sup>1</sup> and *c*<sup>2</sup> such that their sum is no more than *c*. An annotation can be shared as two parts if every recursive cost [*ci, φi*] in it can be shared as two parts [*c*<sup>1</sup> *<sup>i</sup> , φ*1] and [*c*<sup>2</sup> *<sup>i</sup> , φ*2]. Finally, annotations can also be shared as more than two parts.


**Table 1.** Annotations that can be used to instantiate the rule T-Abs.

*Example 4.* There are multiple ways to share the recurrence annotation ([1*, <sup>u</sup>* <sup>2</sup> ]*,* [1*, <sup>u</sup>* <sup>2</sup> ]; *O*(*u*)):

$$F \vdash ([1, \lfloor \frac{u}{2} \rfloor], [1, \lceil \frac{u}{2} \rceil]; O(u)) \lor ([1, \lfloor \frac{u}{2} \rfloor], [1, \lceil \frac{u}{2} \rceil]; O(u)) \mid ([\ \lfloor, O(u) \rangle, \lfloor$$

where one annotation contains both recursive-call costs [1*, <sup>u</sup>* <sup>2</sup> ]*,* [1*, <sup>u</sup>* <sup>2</sup> ]; and the other contains no recursive-call cost. And

$$F \vdash ([1, \lfloor \frac{u}{2} \rfloor], [1, \lceil \frac{u}{2} \rceil]; O(u)) \lor ([1, \lfloor \frac{u}{2} \rfloor]; O(u)) \mid ([1, \lceil \frac{u}{2} \rceil]; O(u)),$$

where each annotation contains one recursive-call cost.

*Function Terms.* The rule T-Abs shown below is really a rule-schema that is parameterized in terms of an annotation (A) for a function body *t*, and a resource bound (*B*) for the function term. If the function body *t* has some recurrence relation described by the annotation *A*, then the function f will satisfy the resource-usage bound *B*. Some example patterns are shown in Table 1. 3

$$\begin{aligned} \varGamma' &= [\mathbf{recFun} \leftarrow \mathbf{f}][\mathbf{args} \leftarrow x\_1 \ldots x\_n] \varGamma \\ \gamma\_f &= \langle x\_1 : \tau\_1 \to \ldots \to x\_n : \tau\_n \to \tau, (B) \rangle \\ \varGamma' &\colon x\_1 : \langle \tau\_1, O(1) \rangle; \ldots ; x\_n : \langle \tau\_n, O(1) \rangle; \mathbf{f} : \gamma\_\mathbf{f} \vdash t : \langle \tau, (A) \rangle \\ \varGamma &\vdash \mathbf{fix} \ \mathbf{f}. \lambda x\_1 \ldots \lambda x\_n \boldsymbol t : \langle x\_1 : \tau\_1 \to \ldots \to x\_n : \tau\_n \to \tau, (B) \rangle \end{aligned} \qquad \text{ $\Gamma$ -ABS}$$

For example, if the annotation of the function body is ([1*, <sup>u</sup>* <sup>2</sup> ]; *O*(1)), then the resource bound in the function type will be *O*(log *u*), i.e., the resource usage of f is bounded by *O*(log(size<sup>f</sup> *x*<sup>1</sup> *...xn*)).

At the same time, the rule stores the name f of the recursive function into recFun, and its arguments as a tuple into args.

*Example 5.* We use a function fix bar*.λx.*if *x* = 1 then 1 else 1+bar(div2 *x*) to illustrate the first pattern in Table 1. The body of bar has the annotated type ([1*, <sup>u</sup>* <sup>2</sup> ]; *O*(1)) because (i) there exists only one recursive call to a sub-problem whose size is half of the top-level problem size *u*, and (ii) the resource usage inside the body is constant (with the assumption that all auxiliary functions

<sup>3</sup> The patterns shown in Table 1 are those we used in the implementation. Patterns capturing other recurrence relations can be added to the type system if needed.

have constant resource usage). This type appears in row 1, column 4 of Table 1. Consequently, the recurrence relation of bar is *<sup>T</sup>*(*u*) <sup>≤</sup> *<sup>T</sup>*( *<sup>u</sup>* <sup>2</sup> ) + *O*(1) (row 1, column 3), where *T*(*u*) is the resource usage of bar on problems with size *u*. Finally, according to the Master Theorem, the resource usage of bar is bounded by *O*(log *u*) (row 1, column 2).

*Branching Terms.* In rule T-If, the condition has type Bool with refinement *ϕe*. Two branches have different types—the then branch follows the path condition *ϕe*, and the refinement *ϕ* of the branch term, while the else branch follows the path condition ¬*ϕe*. By having both branches share the same recurrence annotation, T-If can introduce some imprecision. In particular, if the branches belong to different complexity classes, the annotation of the conditional term will be the upper bound of both branches.

$$\begin{aligned} \Gamma \vdash \alpha \lor \alpha\_1 | \alpha\_2 & \quad \Gamma \vdash e :: \langle \{\mathsf{Bool} \, 1 \mid \varphi\_e \}, \alpha\_1 \rangle \\ \hline \Gamma, \varphi\_e \vdash t\_1 :: \langle \{B \mid \varphi \}, \alpha\_2 \rangle & \quad \Gamma, \neg \varphi\_e \vdash t\_2 :: \langle \{B \mid \varphi \}, \alpha\_2 \rangle \\ \hline \Gamma \vdash \mathsf{if} \, e \, \mathsf{then} \, t\_1 \, \mathsf{else} \, t\_2 :: \langle \{B \mid \varphi \}, \alpha \rangle \end{aligned} \text{T-IF}$$

The rule T-Match is slightly different: (1) there can be more than two branches, (2) all branches have the same type *τ,α*2, and (3) variables in each case C*<sup>i</sup>* (*x*<sup>1</sup> *<sup>i</sup> ...x<sup>n</sup> <sup>i</sup>* ) are introduced in the corresponding branch.

$$\Gamma \vdash \alpha \Downarrow \alpha\_1 | \alpha\_2 \quad \Gamma \vdash e :: \langle \tau\_s, \alpha\_1 \rangle$$

$$\frac{\mathsf{C}\_i = \tau\_1 \to \dots \to \tau\_n \to \tau\_s \quad \Gamma; x\_i^1: \tau\_1; \dots; x\_i^n: \tau\_n \vdash t\_i :: \langle \tau, \alpha\_2 \rangle}{\Gamma \vdash \mathtt{match}\ e \ \mathtt{with}\ |\_i \ \mathsf{C}\_i \ \langle x\_i^1 \dots x\_i^n \rangle \mapsto t\_i :: \langle \tau, \alpha \rangle} \operatorname{\mathsf{T}-Mat\mathsf{CH}}$$

*E-terms.* The typing rules for E-terms are shown in Fig. 3. The two rules for application terms are the key rules of our type system. Let us first look at the E-RecApp rule for recursive-call terms. Recall that the recursive-call annotation tracks the number of recursive calls and the sizes of sub-problems. If the term f *e*<sup>1</sup> *...e<sup>n</sup>* is a recursive call—i.e., *Γ*(recFun) = f—the number of recursive calls in one of the recursive-call costs will increase by one—i.e., [*ck, φk*] in the premise becomes [*c<sup>k</sup>* + 1*, φk*] in the conclusion. Also, we want to make sure that the size of the subproblem this application term is called on satisfies the size expression *φk*. If each callee term is refined by the predicate *ϕi*, i.e., *Γ e<sup>i</sup>* :: {*B<sup>i</sup>* | *ϕi*}*, αi* , then the fact that the size of the problem *e*<sup>1</sup> *...e<sup>n</sup>* satisfies *φ<sup>k</sup>* can be implied by the validity of the predicate *m <sup>i</sup>*=1[*yi/v*]*ϕ<sup>i</sup>* ⇒ (size *y*<sup>1</sup> *...y<sup>m</sup>* ≤ [size *Γ*(args)*/u*]*φk*). We introduce validity checking, written *Γ* |= *ϕ* , to state that a predicate expression *ϕ* is always true under any instance of the environment *Γ*.

*Example 6.* Recall Eq. (2). According to the rule T-RecApp, the recursive call prod (div2 x) y has type {Int <sup>|</sup> *<sup>v</sup>* <sup>=</sup> <sup>x</sup> <sup>2</sup> ∗ <sup>y</sup>}*,*([1*, <sup>u</sup>* <sup>2</sup> ]); *O*(1). Note that the first argument (div2 x) has type {Int <sup>|</sup> *<sup>v</sup>* <sup>=</sup> <sup>x</sup> <sup>2</sup> }, the second argument y has

$$\begin{array}{c} \begin{array}{c} \Gamma \vdash e :: \gamma' \qquad \Gamma \vdash \gamma' << \gamma\\ \Gamma \vdash e :: \gamma \end{array} \begin{array}{c} \Gamma \vdash \neg \gamma' << \gamma\\ \text{E-SUBTPE} \end{array} \begin{array}{c} \Gamma(x) = \gamma\\ \Gamma \vdash x :: \gamma \end{array} \begin{array}{c} \text{E-VAR} \end{array} \end{array}$$

$$\begin{array}{c} \Gamma \vdash \mathsf{g} : \langle x\_{1} \mathrel{\mathop{:}} \tau\_{1} \to \ldots \to x\_{m} \mathrel{\mathop{:}} \tau\_{m} \to \{B \mid \varphi\}, \langle O(\psi\_{\mathsf{k}}) \rangle \rangle \\ \Gamma (\mathsf{recFun}) \neq \mathsf{g} \qquad \Gamma \vdash \langle [c\_{1}, \phi\_{1}] , \ldots, \ldots, [c\_{n}, \phi\_{n}]; O(\psi) \rangle \vee \alpha\_{1} \mid \ldots \mid \alpha\_{m} \\ \forall 1 \leq i \leq m \qquad \Gamma \vdash e\_{i} : \langle \{B\_{i} \mid \varphi\_{i}\}, \alpha\_{i} \rangle \qquad \Gamma \vdash \{B\_{i} \mid \varphi\_{i}\} < \circ \, \tau\_{i} \\ \Gamma \vdash \bigwedge\_{i=1}^{m} [y\_{i}/v] \varphi\_{i} \Rightarrow \left( [\mathtt{skip} \, \mathtt{e}\_{\mathtt{e}} \, y\_{1} \, \ldots \, y\_{m}/u] \psi\_{\mathtt{k}} \in O([\mathtt{size} \, \mathtt{e} \, \Gamma (\mathtt{args})/u] \psi \right) \\ \tau = \{B \mid \, [z\_{i}/x\_{i}] \varphi \land \bigwedge\_{i=1}^{n} [z\_{i}/v] \varphi\_{i} \, \, z\_{i} \notin FV(\varphi), z\_{i} \notin FV(\varphi\_{i}) \\ \Gamma \vdash \mathtt{ge} \, e\_{1} \ldots e\_{m} \coloneqq \langle \tau, ([c\_{1}, \phi\_{1}], \ldots, [c\_{n}, \phi\_{n}]; O(\psi)) \rangle \end{array}$$

$$\begin{split} & \Gamma \vdash \mathbf{f} : \langle x\_{1} : \tau\_{1} \to \ldots \to x\_{m} ; \tau\_{m} \to \{B \mid \varphi\}, \alpha \rangle \qquad \Gamma (\mathtt{recFun}) = \mathtt{f} \\ & \Gamma \vdash \langle [c\_{1}, \phi\_{1}], \ldots, [c\_{k}, \phi\_{k}], \ldots, [c\_{n}, \phi\_{n}]; O(\psi) \rangle \,\,\!/\alpha\_{1} \} \dots \boldsymbol{\Box} \alpha\_{m} \\ & \forall 1 \le i \le m \qquad \Gamma \vdash e\_{i} : \langle [B\_{i} \mid \varphi\_{i}], \alpha\_{i} \rangle \qquad \Gamma \vdash \{B\_{i} \mid \varphi\_{i}\} < \boldsymbol{\frown} \,\, \tau\_{i} \\ & \Gamma \vdash \bigwedge\_{i=1}^{m} [y\_{i}/v] \varphi\_{i} \Rightarrow \langle \mathtt{size} \, \mathtt{e}\_{1} \rangle \ldots y\_{m} \leq [\mathtt{size} \, \, \Gamma \langle \mathtt{args} \rangle/u] \phi\_{k} \rangle \\ & \tau = \{B \mid [z\_{i}/x\_{i}] \varphi \land \bigwedge[z\_{i}/v] \varphi\_{i} \} \, z\_{i} \notin FV(\varphi), z\_{i} \notin FV(\varphi\_{i}) \\ & \Gamma \vdash \mathtt{f} e\_{1} \ldots e\_{m} \coloneqq \langle \tau, \{[c\_{1}, \phi\_{1}], \ldots \{c\_{k} + 1, \phi\_{k}], \ldots, [c\_{n}, \phi\_{n}]; O(\psi) \} \rangle \quad \textbf{E-RECAPP} \end{split}$$

**Fig. 3.** Typing rules of E-terms

type {Int | *v* = y}, the size function is sizeprod = *λz.λw.z*, and the arguments in the context are *Γ*(args) = x y. Therefore, the following predicate is valid:

$$\begin{aligned} [y\_1/v](v = \lfloor \frac{x}{2} \rfloor) \land [y\_2/v](v = y) &\Rightarrow \mathtt{size}\_{\mathtt{prod}} \ y\_1 \ y\_2 = \lfloor \mathtt{size}\_{\mathtt{prod}} \ \Gamma(\mathtt{args}/u) \rfloor \lfloor \frac{u}{2} \rfloor \\ \Leftrightarrow \ (y\_1 = \lfloor \frac{x}{2} \rfloor) \land (y\_2 = y) &\Rightarrow y\_1 = \lfloor \frac{x}{2} \rfloor. \end{aligned}$$

The rule E-App states that callees have types *τi*, and the resource usage does not exceed the bound *O*(*ψ*) in the annotation. Similar to the E-RecApp rule, the size of the problem g calls to is [size<sup>g</sup> *y*<sup>1</sup> *...ym/u*] with the premise *m <sup>i</sup>*=1[*yi/v*]*ϕi*. The validation checking *m i*=1 [*yi/v*]*ϕ<sup>i</sup>* ⇒ [size<sup>g</sup> *y*<sup>1</sup> *...ym/u*]*ψ*<sup>g</sup> ∈ *O*([size *Γ*(args)*/u*]*ψ*) in the rule states that for any instance of *Γ*, the size of the problem in the application term is in the big-O class *O*([size *Γ*(args)*/u*]*ψ*). Note that the membership of big-O classes can be encoded as an ∃∀ query. The query is non-linear, and hence undecidable in general. However, we observed in our experiments that for many benchmarks the query stays linear. Furthermore, even when the query is non-linear, existing SMT solvers are capable of handling many such checks in practice.

#### **3.4 Soundness**

We assume that the resource-usage function *ψ* and the complexities *T* of each function are all nonnegative and monotonic integer functions—both the input and the output are integers. We show soundness of the type system with respect to the resource model. The soundness theorem states that if we derive a bound *O*(*ψ*) for a function f, then the complexity of f is bounded by *ψ*.

**Theorem 1 (Soundness of type checking).** *Given a function fix f.λx*<sup>1</sup> *. . . λxn.t and an environment Γ, if Γ fix f.λx*<sup>1</sup> *. . . λxn.t* :: *τ, O*(*ψ*)*, then the complexity of f is bounded by ψ.*

Our type system is incomplete with respect to resource usage. That is, there are functions in our programming language that are actually in a complexity class *O*(*p*(*x*)), but cannot be typed in our type system. The main reason why our type system is incomplete is that it ignores condition guards when building recurrence relations, and over-approximates if-then-else terms by choosing the largest complexity among all the paths including even unreachable ones.

#### **4 The SynPlexitySynthesis Algorithm**

In this section, we present the SynPlexity synthesis algorithm, which uses annotated types to guide the search of terms of given types.

#### **4.1 Overview of the Synthesis Algorithm**

The algorithm takes as input a goal type f : *τ, O*(*ψ*), an environment *Γ* that includes type information of auxiliary functions, and the size functions for f and all auxiliary functions. The goal is to find a function term of type *τ, O*(*ψ*).

The algorithm uses the rules of the SynPlexity type system to decompose goal types into sub-goals, and then applies itself recursively on the sub-goals to synthesize sub-terms. Concretely, given a goal *γ*, the algorithm tries all the typing rules, where the type in the conclusion matches *γ*, to construct sub-goals: for each sub-term *t* in the conclusion, there must be a judgment *Γ t* :: *γ* in the premise; thus, we construct the sub-goal *γ* —the desired type of *t*. For each I-term rule, the type of each sub-term is always known, and thus a fixed set of sub-goals is generated. For each E-term rule, the algorithm enumerates E-terms up to a certain depth (the depth can be given as a parameter or it can automatically increase throughout the search). If the algorithm fails to solve some sub-goal using some E-term rule, it backtracks to an earlier choice point, and tries another rule.

Because the top-level goal is always a function type, the algorithm always starts by applying the rule T-Abs, which matches the resource bound *O*(*ψ*) using Table 1 to infer a possible recurrence annotation for the type of the function body. Also T-Abs constructs a sub-goal type for the function body. In the rest of this section, we assume that goals are not function types.

**Algorithm 1:** GenerateE(*Γ, γ, d*) **Input :** Context *<sup>Γ</sup>*, goal type *<sup>γ</sup>* <sup>=</sup> {*<sup>B</sup>* <sup>|</sup> *<sup>ϕ</sup>*}*, <sup>α</sup>*, depth bound *<sup>d</sup>* **<sup>1</sup> for** *<sup>t</sup>* <sup>←</sup> EnumerateE(*Γ, d, B*) **do <sup>2</sup> if** CheckE(*t, Γ, γ*) **then return** *t* **<sup>3</sup> return** ⊥

*Synthesizing E-Terms.* The algorithm for synthesizing E-terms is shown in Algorithm 1. It enumerates each E-term *t*—with depth up to *d*—that satisfies the base type *B* in the goal *γ* := {*B* | *ϕ*}*,*([*c*1*, φ*1]*..*[*cn, φn*]; *O*(*ψ*)) from the context *Γ*. For each such E-term *t*, the algorithm checks whether *t* satisfies the goal type with a subroutine CheckE, which operates as follows.

When *t* is a variable term, CheckE checks the refined type of *t* against the goal. When *t* is an application term, CheckE first checks if the total number of recursive calls in the term *t* exceeds the bound *<sup>i</sup> ci*, and if it does, the term *t* is rejected. Otherwise, CheckE checks the sizes of sub-problems of recursive calls in *t*. Formally, to check if a recursive application term *f*(*t*1*, .., tm*) is consistent with some [*ck, φk*], the algorithm queries the validity of the following predicate

$$(\bigwedge\_{i=1}^{m} [y\_i/v] \varphi\_i \Rightarrow (\mathtt{size}\_f(y\_1 \dots y\_m) = [\mathtt{size}\_f(\varGamma(\mathtt{args}))/v] \phi\_k)),$$

where the *yi*'s are fresh variables, and the *ϕi*'s are the refinements of terms *ti*'s. If the sizes of sub-problems are not consistent with the recursive-call costs [*c*1*, φ*1]*..*[*cn, φn*], the term *t* is rejected. Note that one recursive call can possibly satisfy more than one [*ck, φk*]. The algorithm enumerates all possible matches. Finally, CheckE checks the refined type of *t* against the goal.

Checking the validity of auxiliary application terms is similar. CheckE needs to establish that the following predicate holds, which asserts that the resource usage of an auxiliary function does not exceed the bound *O*(*ψ*).

$$\bigwedge\_{i=1}^{m} [y\_i/v] \varphi\_i \Rightarrow \left( [\mathtt{size}\_{\mathsf{g}} \ y\_1 \dots y\_m/v] \psi\_{\mathsf{g}} \in O([\mathtt{size} \ \varGamma(\mathtt{args})/v] \psi) \right) \dots$$

Recall that the above query is undecidable in general, and is checked with best effort by an SMT solver in SynPlexity.

*Synthesizing I-Terms.* Algorithm 2 shows the algorithm for synthesizing I-Terms. GenerateI first tries to synthesize an E-term for the goal *γ* (line (1)).

If there is no E-term that satisfies the goal, and the match bound *m* is greater than 0, GenerateI chooses to apply the rule T-Match lines (2)–(8). First, it enumerates candidate scrutinees *s*, which are E-terms of some data type. Then it generates match *patterns* according to the type of *s* (line (3)), updates the goal with a new recursive-call cost (line (4)), and generates case terms *t<sup>i</sup>* for each pattern *pattern*[*i*] (lines (5)–(7)). The subroutine UpdateCost is used to subtract the recursive-call cost usage from the cost in *γ*. Finally, if all case terms are found, the algorithm constructs the corresponding match-term and returns it.

If there is no match-term satisfying the goal, GenerateI applies the rule T-If to synthesize a term of the form **if** *cond* **then** *t<sup>T</sup>* **else** *t<sup>F</sup>* , and performs three steps to construct sub-goals for sub-terms *cond*, *t<sup>T</sup>* , and *t<sup>F</sup>* : (1) it enumerates the condition guard *cond* (line (10)) of type bool; (2) it updates the cost in the goal *γ* (line (11)); and (3) it propagates sub-goals to the two branches *t<sup>T</sup>* and *t<sup>F</sup>* with *cond* and ¬*cond* as the path condition (lines (12) and (13)), respectively. Finally, if both *t<sup>T</sup>* and *t<sup>F</sup>* are found, the algorithm constructs the corresponding if-term and returns it as a solution (line (14)).

*Optimization.* Algorithm 2 discussed above is based on bidirectional typeguided synthesis with liquid types (Synquid [18]). Therefore, *liquid abduction* and *match abduction*, two optimizations used in Synquid, can also be used in SynPlexity. These two techniques allow one to synthesize the branches of ifand match-terms, and then use logical abduction to infer the weakest assumption under which the branch fulfills the goal type.


*Example 7.* We illustrate in Fig. 4 how the algorithm synthesizes the *O*(log *x*) implementation of prod presented in Eq. (2). We omit the type contexts in the example. We will use "??" to denote intermediate terms being synthesized (i.e., holes in the program). At the beginning, the type of ??<sup>1</sup> (i.e., the term we are synthesizing) is an arrow type with resource bound *O*(log *u*) specified by the input goal. In this example, SynPlexity applies to the arrow type the rule

**Fig. 4.** Trace of the synthesis of an *O*(log *x*) implementation of prod.

T-Abs, parameterized according to the first rule in Table 1. This step produces the sub-problem of synthesizing the function body ??2, whose annotation is ([1*, <sup>u</sup>* <sup>2</sup> ]; *O*(1))—which means that ??<sup>2</sup> should contain at most one recursive call to sub-problems with size *<sup>u</sup>* 2 .

Next, SynPlexity chooses to fill ??<sup>2</sup> with an if-then-else term (by applying the T-If rules) with three sub-problems: the condition guard ??3, the then branch ??<sup>4</sup> and the else branch ??5. Note that here we share the number of recursive calls [1*, <sup>u</sup>* <sup>2</sup> ] as follows: 0 recursive calls in the condition guard, and 1 in the then branch and the else branch. The left arrow E-App shows how SynPlexity enumerates terms and checks them against the goal types of sub-problems. For example, to fill ??4, SynPlexity enumerates terms of type {Int <sup>|</sup> *<sup>v</sup>* <sup>=</sup> <sup>x</sup> <sup>∗</sup> <sup>y</sup> <sup>∧</sup> <sup>x</sup> = 0*,*([1*, <sup>u</sup>* <sup>2</sup> ]; *O*(1)) , which are restricted to contain at most one recursive call to prod. In Fig. 4, SynPlexity has picked the term x to fill ??4. The refinement type of the variable term x is {Int | *v* = x ∧ x = 0} where x = 0 is the path condition. To check that x also satisfies the type of ??4, the algorithm needs to apply rule E-SubType, and check that, for any *v* and x, *<sup>v</sup>* <sup>=</sup> <sup>x</sup> <sup>∧</sup> <sup>x</sup> = 0 implies *<sup>v</sup>* <sup>=</sup> <sup>x</sup> <sup>∗</sup> <sup>y</sup> <sup>∧</sup> <sup>x</sup> = 0, and [0*, <sup>u</sup>* <sup>2</sup> ] is approximated by [1*, <sup>u</sup>* <sup>2</sup> ].

After applying another T-If rule for ??5, SynPlexity produces three new sub-problems ??6, ??7, and ??8. When enumerating terms to fill ??7, SynPlexity finds an application term double (prod (div2 x) y) that satisfies the goal {Int <sup>|</sup> *<sup>v</sup>* <sup>=</sup> <sup>x</sup> <sup>∗</sup> <sup>y</sup> <sup>∧</sup> x mod 2=0*,*([1*, <sup>u</sup>* <sup>2</sup> ]; *O*(1)) . To check that the size of the problem in the recursive call prod (div2 x) <sup>y</sup> satisfies the recursive-call cost [1*, <sup>u</sup>* <sup>2</sup> ], the type system first checks the refinement of the callee. The refinement of the first argument (div2 x) is *<sup>ϕ</sup>*<sup>1</sup> := *<sup>v</sup>* <sup>=</sup> <sup>x</sup> <sup>2</sup> . The refinement of the second argument y is *ϕ*<sup>2</sup> := *v* = y. Consequently, the size of the sub-problem prod (div2 x) y satisfies [1*, <sup>u</sup>* <sup>2</sup> ] because [*z/v*]*ϕ*<sup>1</sup> <sup>∧</sup> [*w/v*]*ϕ*<sup>2</sup> <sup>=</sup><sup>⇒</sup> size *z w* = [(size x y)*/v*] *<sup>u</sup>* <sup>2</sup> , which can be simplified to *<sup>z</sup>* <sup>=</sup> <sup>x</sup> <sup>2</sup> ∧ *<sup>w</sup>* <sup>=</sup> <sup>y</sup> <sup>=</sup><sup>⇒</sup> *<sup>z</sup>* <sup>=</sup> <sup>x</sup> <sup>2</sup> . (Recall that the size function for prod is size := *λz.λw.z*.)

The algorithm is sound because it only enumerates well-typed terms.

**Theorem 2 (Soundness of the synthesis algorithm).** *Given a goal type τ, O*(*ψ*) *and an environment Γ, if a term fix f.λx*1*..λxn.t is synthesized by* SynPlexity*, then the complexity of f is bounded by ψ.*

#### **5 Extensions to the SynPlexityType System**

In this section, we introduce two extensions to the SynPlexity type system.

**Recurrence Relations with Correlated Sizes.** The type system shown in Sect. 3 only tracks sub-problems with independent sizes. For example, consider the recurrence relation *T*(*u*) = *T*(*l*) + *T*(*r*) + *O*(1), where the variables *l* and *r* are correlated by the constraint *l* + *r < u*. This relation is needed to reason about programs that manipulate binary trees or binary heaps, where *l* and *r* represent the sizes of the two children. To support such a recurrence relation, we extend SynPlexity's type system with recursive-call costs of the form [1*, l*]*,* [1*, u* − 1 − *l*], where *l* is a free variable. When correlated recurrence relations are present, the synthesis algorithm will: (1) match the first enumerated recursive-call term to [1*, l*], and instantiate the size *l* with *s*, where *s* is the size of the recursive-call term (*s* should be smaller than the size *u* of the top-level function); and (2) use the size *s* of the recursive-call term computed in step 1 to constrain the algorithm to enumerate only recursive-call terms of sizes at most *u* − 1 − *s*.

**Synthesis of Auxiliary Functions.** Most of the existing type-directed approaches require the input to the problem to contain all needed auxiliary functions. With SynPlexity, some of the auxiliary functions needed to solve synthesis problems with resource annotations can be synthesized automatically.

For example, consider the problem prod described in Sect. 2. In this problem, we observe that one of the provided auxiliary functions, div2, strongly resembles one of the elements of the recurrence relation, *<sup>T</sup>*(*u*) <sup>≤</sup> *<sup>T</sup>*( *<sup>u</sup>* <sup>2</sup> ) + *O*(1), needed to synthesize a program with the desired resource usage. In particular, we know that one needs an auxiliary function that can take an input of size *u* and produce an output of size *<sup>u</sup>* <sup>2</sup> . In this example, the required auxiliary function div2 merely needs to divide the input by 2 (and round down), but in certain cases it might need a more precise refinement type than merely changing the size of the input. For example, the auxiliary function split used by merge sort needs to split the input list xs into two lists v1 and v2 that are half the length of the input *and* such that elems(v1) elems(v2) = elems(xs). However, all we know from the refinement is that the output lists must be half the length of the original list.

Although we do not know what this auxiliary function should do exactly, we can use the size constraint appearing in the recurrence relation to define part of the refinement type we want the auxiliary function to satisfy. SynPlexity builds on this idea and incorporates an (optionally enabled) algorithm, SynAuxRef, that while trying to synthesize a solution to the top-level synthesis problem also tries in parallel to synthesize auxiliary functions that can create sub-problems with the size constraints needed in the recurrence relation. To address the problem mentioned above—i.e., that we do not know the exact refinement type the auxiliary function should satisfy—SynAuxRef enumerates *auxiliary refinements*, which are possible specifications that the auxiliary function aux we are trying to synthesize might satisfy.

**Synthesis with Higher-Order Functions.** Although SynPlexity does not support higher-order functions in general, it can solve restricted but practical problems with higher-order functions. The restriction supported introduces four assumptions on the synthesis problems. First, we assume that the resource usage of any function argument *g* is constant, i.e., *g* : *τ, O*(1). Second, arrow-type arguments in recursive calls in the synthesized program are the same as the arrow-type arguments of the top-level function. For example, in the body of a higher-order function fix *f.λgλxλy.t*, all recursive application terms must be of the form *f*(*g, ,* ) where can be any well-typed term. Third, we assume that the sizes of outputs of functions as arguments do not affect the asymptotic resource usage of the synthesized programs. Finally, arrow-type arguments cannot appear in size functions.

We extend the syntax and the type system of SynPlexity to support the restricted problems (the detail of this extension can be found in the technical report [11]). We also modify the synthesis algorithm to prune E-terms that break the second or third restriction mentioned above.

To support the second restriction (i.e., that we need to call the same function arguments in recursive calls), the synthesis algorithm first stores the function arguments of the top-level functions. Later, when a recursive call is enumerated, the synthesizer checks whether the recursive call has the same function arguments, and rejects the candidate if it does not.

To support the third restriction (i.e., that the behavior of function arguments should not affect the resource usage), the synthesis algorithm avoids enumerating nested application terms where the resource usage of the outer application depends on the value of an inner application term that calls a function argument.

#### **6 Evaluation**

In this section, we evaluate the effectiveness and performance of SynPlexity, and compare it to existing tools.<sup>4</sup> We implemented SynPlexity in Haskell on top of Synquid by extending its type system with recurrence annotations as presented in Sect. 3. The detailed results can be found in the technical report [11].

<sup>4</sup> All the experiments were performed on an Intel Core i7 4.00 GHz CPU, with 8 GB of RAM. We used version 4.8.9 of Z3. The timeout for each benchmark was 10 min.

#### **6.1 Comparison to Prior Tools**

We compared SynPlexity against two related tools: Synquid [18] and ReSyn [16], which are also based on refinement types.

*Benchmarks.* We considered a total of 77 synthesis problems: 56 synthesis problems from ReSyn (each benchmark specifies a concrete linear-time resource annotation), 16 synthesis problems from Synquid (which do not include resource annotations) that are not included in ReSyn, and 5 new synthesis problems involving non-linear resource annotations. In these synthesis problems, synthesis specifications and auxiliary functions are all given as refinement types. For 3 of the new benchmarks, the auxiliary function required to split the input into smaller ones is not given—i.e., the synthesizer needs to identify it automatically.

The three solvers (SynPlexity, Synquid, and ReSyn) have different features, and hence not all synthesis problems can be encoded as synthesis benchmarks for a single solver. In the rest of this section, we describe what benchmarks we considered for each tool, and how we modified the benchmarks when needed.

Synquid: Synquid does not support resource bounds, so we encoded 77 synthesis problems as Synquid benchmarks by dropping the resource annotations. Synquid returns the first program that meet the synthesis specification, and cannot provide any guarantees about the resource usage of the returned program. Synquid can solve 75 benchmarks, and takes on average 3.3s. For 10 benchmarks Synquid synthesizes a non-optimal program—i.e., there exists another program with better concrete resource usage. For example, on the ReSyn-triple-2 benchmark (where the input is a list *xs*), Synquid found a solution with resource usage *O*(|*xs*| <sup>2</sup>), while both SynPlexity and ReSyn can synthesize a more efficient implementation with resource usage *<sup>O</sup>*(|*xs*|). The two benchmarks that Synquid failed to solve include the new benchmark SynPlexity-merge-sort'. In this benchmark, the auxiliary function required to break the input into smaller inputs is not given, without which the sizes of solutions become much larger. Therefore Synquid times out.

ReSyn: We ran ReSyn on the 56 ReSyn benchmarks with the corresponding concrete resource bounds. We could not encode 16 problems because ReSyn does not support non-linear resources bounds—e.g., the bound log |*y*| in the AVLinsert Synquid benchmark. ReSyn solved all 56 benchmarks with an average running time of 18.3 s.

SynPlexity: We manually added resource usages and resource bounds to existing problems to encode them for SynPlexity. For Synquid benchmarks without concrete resource bounds, we chose well-known time complexities as the bounds, e.g., we added the resource bound *O*(*u* log *u*) to the Sort-merge-sort problem. For the ReSyn benchmarks, we translated the concrete resource usage and resource bounds to the corresponding asymptotic ones—e.g., for the ReSyncommon' benchmark with the concrete resource bound |*ys*|+|*zs*|, we constructed a SynPlexity variant with the asymptotic bound *O*(*u*) and a size function *λys.λzs.*|*ys*<sup>|</sup> <sup>+</sup> <sup>|</sup>*zs*|. We could not encode 3 synthesis problems as SynPlexity benchmarks: two of them involved higher-order functions that do not satisfy the assumptions introduced in Sect. 5, and the other one has an exponential resource-usage bound *O*(2*<sup>u</sup>*) (the Tree-create-balanced problem from Synquid).

SynPlexity solved 73 benchmarks with an average running time of 8.1s. Unlike Synquid, SynPlexity guarantees that the synthesized program satisfies the given resource bounds. After extending the implementation to support the restrictions discussed in Sect. 5, SynPlexity solved 5/6 benchmarks with higher-order functions. For 10 benchmarks, SynPlexity found programs that had better resource usage than those synthesized by Synquid. Furthermore, SynPlexity can encode and solve 9 problems that ReSyn could not solve because the resource bounds involve logarithms. However, SynPlexity cannot encode and solve 2 benchmarks that involve higher-order functions and do not satisfy the restrictions introduced in Sect. 5. SynPlexity could solve 3 problems that required synthesizing both the main function (e.g., SynPlexitymerge-sort) and its auxiliary function (e.g., a function splitting a given list into two balanced partitions). No other tool could solve the SynPlexity-merge-sort' benchmark.

*Finding.* SynPlexity can express and solve 73/77 benchmarks. SynPlexity has comparable performance to existing tools, and can synthesize programs with resource bounds that are not supported by prior tools.

#### **6.2 Pruning the Search Space with Annotated Types**

SynPlexity uses recurrence annotations to guide the search and avoids enumerating terms that are guaranteed to not match the specified complexity. We compared the numbers of E-terms enumerated by SynPlexity and Synquid for 56 benchmark on which both tool produced same solutions. Synquid always enumerated at least as many E-terms as SynPlexity, and SynPlexity enumerated strictly fewer E-terms for 26/56 benchmarks. For these 26 benchmarks, SynPlexity can on average prune the search space by 6.2%. For example, in one case (BST-delete) SynPlexity enumerated 2,059 E-terms, while Synquid enumerated 2,202.

*Finding.* On average, SynPlexity reduces the size of the search space by 6.2% for approximately half of the benchmarks.

### **7 Related Work**

*Resource-Bound Analysis.* Rather than determining whether a given program satisfies a specification, a synthesizer determines whether there exists a program that inhabits a given specification. The branch of verification that we draw upon for resource-based synthesis is resource-bound analysis [20].

Within the literature on automated resource-bound analysis, there are methods that extract and solve recurrence relations for imperative code [2,4,7,15]. However, these methods are unlike the type system presented in this work because they extract concrete complexity bounds as recurrence relations, and then solve the recurrences to find a concrete upper bound on resource usage. The dominant terms of the resulting concrete bounds can then be used to state a big-*O* complexity bound. In contrast, we want to synthesize programs with respect to a big-*O* complexity directly, which is more similar to the manual reasoning of [6,8]. Thus, if we were to use these techniques for our problem, the first step in our synthesis algorithm would be to pick a concrete complexity function given a big-*O* complexity, and then reverse the verification problem with regards to that concrete complexity. However, for any big-*O* complexity, there are an infinite number of functions that satisfy that complexity, which presents a significant challenge at the outset. Our design choice also has some drawbacks. As noted in [8], reasoning compositionally with big-*O* complexity is challenging due to the hidden quantifier structure of big-*O* notation. Thus, to maintain soundness our type system has to sacrifice precision and generality in some places. For example, when a function has multiple paths, our type system over-approximates by choosing the largest complexity among all the paths.

Another set of methods to generate resource bounds are type-based [9,10,14, 19]. As we discussed throughout the paper, the complexities generated by these methods are concrete functions and not expressed with big-*O* notation, although [19] is sometimes able to pattern match a case of the Master Theorem. These type systems differ from ours in a few ways. The AARA line of research [9,10,14] is able to assign amortized complexity to programs, but is not able to generate logarithmic bounds. [19] is also able to perform amortized analysis; however, the technique is not fully automated, and instead requires the user to provide type annotations on terms, which are then checked by the type system.

*Type- and Resource-Aware Synthesis.* The SynPlexity implementation is built on top of Synquid [18] a type-directed synthesis tool based on refinement types and polymorphism. The work that most closely resembles ours is ReSyn [16]. As in our work, they combine the type-directed synthesizer Synquid with a type system that is able to assign complexity bounds to functional programs. The type system used in ReSyn is based on one originally used in the context of verification [10]. That work uses a sophisticated type system to assign amortized resource-usage bounds to a given program. The type system of ReSyn differs from the one presented in Sect. 3 in a few significant ways.

As highlighted earlier, ReSyn automatically infers bounds on recursive functions using amortized analysis and is restricted to linear bounds, whereas our system is able to synthesize complexities of the form *O*(*n<sup>a</sup>* log*<sup>b</sup> n* + *c*).

Another difference is that ReSyn synthesizes programs with a concrete complexity bound. This approach has advantages and disadvantages. For instance, it places an extra burden on the human to provide the correct bound with precise coefficient. On the other hand, the user might want an implementation that has a complexity with a small coefficient, whereas our system provides no guarantee that the complexity of an implementation will have a small coefficient in the dominant term: SynPlexity only guarantees asymptotic behavior.

ReSyn can synthesize programs with higher-order functions, which are supported only in a restricted manner by SynPlexity. To handle higher-order functions, ReSyn attaches resource units to types, which gives it *resource polymorphism*. Moreover, costs of inputs with function types can be written generally as polymorphic types (i.e., costs can be polymorphic with respect to the size of the specific input types). SynPlexity does not have *asymptotic resource polymorphism* because it cannot directly compose unknown big-*O* functions (i.e., the complexity of higher-order inputs). We envision that with carefully crafted restrictions on the resource annotations of higher-order functions, SynPlexity could handle synthesis problems involving such functions, e.g., assuming that the complexity of input functions is known and the refinements of input functions are precise enough. Detailed discussion about these restrictions can be found in Sect. 5 and the technical report [11]. Because big-*O* functions cannot be directly composed, developing a more general extension to SynPlexity that supports higher-order functions is a challenging direction for future work.

**Acknowledgments.** Supported, in part, by a gift from Rajiv and Ritu Batra; by multiple Facebook Research Awards; by a Microsoft Faculty Fellowship; by NSF under grants 1420866, 1763871, and 1750965; and by ONR under grants N00014-17-1-2889 and N00014-19-1-2318. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the authors, and do not necessarily reflect the views of the sponsoring entities.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### Program Sketching by Automatically Generating Mocks from Tests

Nate F. F. Bragg<sup>1</sup> , Jeffrey S. Foster<sup>1</sup> , Cody Roux<sup>2</sup> , and Armando Solar-Lezama<sup>3</sup>

<sup>1</sup> Tufts University, Medford, MA 02155, USA {nate,jfoster}@cs.tufts.edu <sup>2</sup> Draper Laboratory, Cambridge, MA 02140, USA croux@draper.com <sup>3</sup> Massachusetts Institute of Technology, Cambridge, MA 02139, USA asolar@csail.mit.edu

Abstract. Sketch is a popular program synthesis tool that solves for unknowns in a sketch or partial program. However, while Sketch is powerful, it does not directly support modular synthesis of dependencies, potentially limiting scalability. In this paper, we introduce Sketcham, a new technique that modularizes a regular sketch by automatically generating mocks—functions that approximate the behavior of complete implementations—from the sketch's test suite. For example, if the function f originally calls g, Sketcham creates a mock g<sup>m</sup> from g's tests and augments the sketch with a version of f that calls gm. This change allows the unknowns in f and g to be solved separately, enabling modular synthesis with no extra work from the Sketch user. We evaluated Sketcham on ten benchmarks, performing enough runs to show at a 95% confidence level that Sketcham improves median synthesis performance on six of our ten benchmarks by a factor of up to 5× compared to plain Sketch, including one benchmark that times out on Sketch, while exhibiting similar performance on the remaining four. Our results show that Sketcham can achieve modular synthesis by automatically generating mocks from tests.

Keywords: Program synthesis, mocks, Sketch

#### 1 Introduction

Program synthesis by sketching, as embodied by the Sketch synthesis tool [30], is a popular technique that has been applied to a wide variety of problems [5,7,13,14,15,16,18,22,29]. A Sketch input (henceforth a sketch) is a program written in a C-like language augmented with holes, unknown constants, and generators, unknown expressions. The solution for a sketch is specified using test cases called harnesses, also written in the Sketch language, that make assertions about the results of to-be-synthesized code. Sketch searches for a solution using counterexample-guided inductive synthesis (CEGIS), which alternately synthesizes a candidate solution and then uses a verifier to check the assertions; any counterexamples from verification feed into the next round of synthesis [27].

c The Author(s) 2021

A. Silva and K. R. M. Leino (Eds.): CAV 2021, LNCS 12759, pp. 808–831, 2021. https://doi.org/10.1007/978-3-030-81685-8\_38

One key challenge of using Sketch is that it does not specifically support modular synthesis. More precisely, even if an input sketch is divided into a number of functions that call each other, Sketch solves them all together. This approach potentially limits scalability, as SAT formulas created by Sketch can grow quite quickly as function calls are inlined. A Sketch user could potentially work around this by manually replacing calls to to-be-synthesized functions with calls to Sketch models [24], which are mocks, i.e., functions that, in place of full implementations, approximate the desired behavior with a specification in the form of assertions about individual cases. However, writing additional specifications is both time consuming and redundant with developing the original sketch.

In this paper, we introduce Sketcham (short for Sketch and Mocks), a novel technique that converts a regular sketch problem into a modular sketch problem by automatically generating mocks from harnesses. More specifically, suppose Sketcham is given a sketch in which function f calls g and g is tested by harness h. Sketcham first converts h into a mock g<sup>m</sup> that has the same function signature as g but whose body encodes the assertions from h. Then, Sketcham augments the original sketch with new code in which f calls g<sup>m</sup> instead of g, thereby allowing f to be synthesized separately from g. Thus, by converting tests (harnesses) to mocks (specs), Sketcham enables modular synthesis without extra work from the user. Section 2 gives an overview of Sketcham.

Sketcham generates the new, modular sketch problem using a sequence of three algorithms. First, Sketcham traverses the original sketch to build a mapping A from function names to a set of assertions in which each function is called. Note that we place some limitations of the assertions—e.g., they can contain at most one function call—to guarantee we can always translate them from harness assertions to mock assertions. Next, Sketcham traverses A, generating a mock f<sup>m</sup> for each function f ∈ dom(A), where f<sup>m</sup> encodes the assertions in A(f). Finally, Sketcham generates new mock harnesses that are the same as the original harnesses, except they call mocks instead of the underlying functions. Section 3 presents Sketcham's core algorithms.

We implemented Sketcham as an additional pass to Sketch, which we evaluated on ten benchmarks. We found a high variance in running time, both under Sketch and under Sketcham. To account for this difference, we used the Clopper-Pearson method [6], running each configuration (synthesis tool–benchmark combination) up to 1,487 times, reaching 95% confidence that the true median running time lies within 20% of the experimental median, excluding failures and runs exceeding a 60 minute timeout. We found that, for six of ten benchmarks, Sketcham runs up to 5× faster than Sketch; for one benchmark Sketcham is up to a factor of 0.98× slower; for the remaining three benchmarks, performance is indistinguishable. We examined one benchmark, deduplication of elements in an array, in detail. We found that the performance improvement is largely due to a mock that does a thorough job representing the function it mocks, and that the performance improvement occurs during the CEGIS synthesis phase rather than the CEGIS verification phase. Section 4 presents our evaluation.

(a) dedup and sort (simplified).

```
1 harness void h_sort(int n,
2 int[n] vs) {
3 int[n] svs = sort(vs);
4 for(int i=0; i<n-1; ++i)
5 assert svs[i] <= svs[i+1];
6 /* also elts(vs)=elts(svs) */
7
8 }
                            ➪
                              model int[n] sort_mock(int n, 1
                                             int[n] vs) { 2
                               int[n] svs = sort_uf(vs); 3
                               for(int i=0; i<n-1; ++i) 4
                                assume svs[i] <= svs[i+1]; 5
                               /* also elts(vs)=elts(svs) */ 6
                               return svs; 7
                              } 8
```
(d) Translating sort's test harness into a mock.

Fig. 1: Sketcham applied to deduplication via sorting.

In summary, Sketcham demonstrates that modular synthesis can be achieved by automatically generating mocks from tests (specs from harnesses) without additional user effort.

#### 2 Overview

To illustrate Sketcham, consider Figure 1a, which shows a simplified sketch whose solution deduplicates an array of integers. This sketch makes use of Sketch holes ??, which are unknown constants, and generators such as expr(vars, ops), which is an unknown expression composed of variables vars combined with

operands ops, including PL for addition and MI for subtraction. The correct solutions for the holes and generators are shown in end-of-line comments.

At the top of Figure 1a, function dedup takes a length n and array vs, and it returns the deduplicated array and, by reference, the deduplicated array's length sz (in Sketch, functions can only have at most one return value, hence the returnby-reference sz). The dedup function begins by calling another function, sort, to sort the array (line 3). Then it initializes sz to a hole and loops through the array (lines 4-5). In each iteration, it computes an expression j of sz and i (line 6) used in a conditional guard (line 7; details of guard not shown). If the condition holds, the element at position i is copied into res and sz is updated; otherwise the element is ignored. Finally, dedup returns the result array res.

The sort function (line 13) takes the length and array and returns a sorted array. This particular sketch is for merge sort. Here the programmer knows that merge sort involves sorting two sub-arrays but isn't sure about the details. After some initialization (not shown), it makes two recursive calls to sort subarrays (lines 15 and 16). Then it loops over the sorted sub-arrays, merging the elements into array vs, which is returned. The loop guard (line 17) uses a different generator, exprBool(vars, ops), that generates arithmetic comparisons (<, <=, etc) among expressions generated by calling expr(vars, ops).

Harnesses and Mocks. To test the expected behavior of dedup and sort, the sketch also includes two harnesses, h\_dedup and h\_sort. Figure 1b shows the call graph of the sketch with the harnesses, and the left side of Figure 1d shows a portion of h\_sort (we omit h\_dedup for brevity). This harness calls sort and then makes assertions about the results, e.g., that the output array is sorted. Harnesses are distinguished from regular functions by the keyword harness, and their arguments are treated as universally quantified. Thus, h\_sort tests that for all n and arrays vs of length n, the sort function is correct.

To solve this synthesis problem, Sketch converts dedup, sort, and a harness into a single SAT formula and then uses CEGIS to find a solution. This approach works, but the formula passed to the solver is large, because it contains both functions' worth of code, and complex, because reasoning about the code in dedup requires simultaneously reasoning about the code in sort. Thus, mashing together both functions into a single SAT formula potentially limits the scalability of Sketch.

The key idea of Sketcham is to observe that this sketch is actually modular it has been divided into two functions, each with their own tests. Sketcham takes advantage of this modularity by creating a new synthesis problem that includes mock versions of functions in the sketch, which can then be used to enable separate reasoning about each function.

The right side of Figure 1d shows sort\_mock, the mock version of sort. The mock has the same signature as sort, but instead of containing the actual sorting code, it contains assertions from h\_sort about sort's expected behavior. In detail, in place of calling sort, the mock calls a fresh uninterpreted function sort\_uf on line 3. Then it makes assumptions (rather than assertions) about the result array svs (line 5), and finally returns svs (line 7). The mock itself is a

```
int doub(int m) {
  return m * 2;
}
harness void h(int n) {
  int out = doub(n * 10);
  assert out == (n + n) * ??;
}
```

```
model int doub_mock(int m) {
  int out = doub_uf(m);
  assume (0 == m%10) =⇒
     out == (m/10 + m/10) * ??;
  return out;
}
```

```
(a) Double.
```
(b) Mock double.

Fig. 2: The double function and its mock.

Sketch model (indicated by the model keyword), and where the mock is called, Sketch will replace the call with the assumptions in the model's body [24].

Next, Sketcham creates new code that uses the mock, as shown in Figure 1c. (Here the dashed, greyed boxes are for functions and harnesses that are generated but do not improve solving time; see Section 4.2.) In particular, dedup' is the same as dedup, except it calls sort\_mock instead of sort, and h\_dedup'' is the same as h\_dedup but it calls dedup' instead of dedup.

The final sketch includes h\_dedup'', h\_dedup' (a trivial harness that calls a mocked dedup), and h\_dedup—in that order—as well as the harnesses for sort. Sketcham searches for a solution for each harness in order, i.e., it tries to solve h\_dedup'' first. Notice that, critically, when Sketcham solves h\_dedup'', it need not consider the code of sort, but rather only its specification as encoded in the mock. In practice, this means that Sketcham can solve h\_dedup'' up to 18.1× faster than Sketch solves h\_dedup, a significant speedup.

Moreover, sort\_mock encodes the specification of sort, so once Sketcham solves h\_dedup'', it has found a solution for h\_dedup as well. To preserve correctness, Sketcham keeps the original harnesses such as h\_dedup, because mocks with partial specifications can lead to partially incorrect solutions to the harnesses using them. However, even in these cases, the counterexamples they generate can still help more quickly narrow the synthesis search space for the original harness, and lead to an ultimately valid solution.

Quantifier Elimination. In Figure 1d, the translation from harness to mock was straightforward: the call to the mocked function becomes a call to an uninterpreted function, and asserts become assumes. Sometimes, however, the translation is more complex. Consider the sketch in Figure 2a, which includes a function doub that doubles its input and a harness h that calls doub(n\*10) and asserts the result is (n+n)\*?? for some hole.

Notice this assertion only describes arguments of the form n\*10 for some n, i.e., implicitly there must exist some m such that m = n\*10 for the assertion to hold. Sketcham performs quantifier elimination [1,4] on such nested existentials, following the approach of Kuncak et al. [17]. Figure 2b shows the resulting mock.

Fig. 3: Sketcham architecture

Here, in the assumption, n is replaced by witness candidate m/10. Because m is an integer, we also add a precondition that m is evenly divisible by 10.

We note that Sketcham includes quantifier elimination for completeness, and in our evaluation we consider the sketch in Figure 2a. However, we did not find quantifier elimination necessary for our other benchmarks.

#### 3 The Sketcham Algorithm

Next we more formally describe Sketcham, which is implemented as a pass within Sketch as shown in Figure 3. The presentation that follows reflects this Sketch implementation without loss of generality of the core algorithm for converting tests to mocks. The Sketch frontend consumes the input sketch and transforms it into the Sketch intermediate representation (Sketch IR), which is passed to the Sketch backend. Sketch IR encodes first-order logic augmented with theories of arithmetic, arrays, functions, and more, as discussed below. When the backend loads the IR, it performs loop unrolling, function inlining, and other transformations that are needed by the solver [26], yielding a program p. Standard Sketch then uses CEGIS to solve the synthesis problem, outputting a hole assignment that the frontend uses to produce the solved sketch. Sketcham modifies this process by inserting, after optimization, a mock rewriting phase, described below, that transforms p into the augmented program p<sup>m</sup> for CEGIS.

We formalize Sketcham on the fragment of Sketch IR shown in Figure 4. Here types are omitted, and we assume the sketch is type-correct. A program sketch p is a sequence of harness and function definitions. A harness definition h tags a function definition as a test harness. A function definition d is given named parameters<sup>1</sup> and a body, which is a sequence of statements. Statements s are

<sup>1</sup> For simplicity, we assume parameter names are unique across the whole program.

```
p ::= (h | d)*
   h ::= harness d
   d ::= def f ( x , . . . , x ) { s* }
   s ::= x := e | return e | assert φ | assume φ
   e, φ, ψ ::= f ( e , . . . , e ) | uop e | e bop e | n | x | ?? x
   uop ::= ¬ | -
   bop ::= ∧ | ∨ | ⊕ | =⇒ | = | + | - | * | / | %
x, y ∈ variable names G ∈ graphs A : f → Φ
f, g ∈ function names Φ ∈ set of φ F : f → f
```
Fig. 4: Sketcham's fragment of Sketch IR

assignments, returns, assertions, and assumptions. The most critical expressions e in our algorithm are function calls f ( e , . . . , e ) with their arguments. The detailed grammar for the remaining expressions is unimportant in the remainder of this section, but for completeness we show expressions for unary and binary logical and arithmetic operations uop e and e bop e; constants n; variables x ; and named holes ?? x . Below, we sometimes use the metavariables φ and ψ in place of e to indicate an expression used for Boolean-valued formulas.

Given the input Sketch IR program p as shown in Figure 3, Sketcham creates the output sketch by first calling BuildAssertMap (Algorithm 1) to build mapping A from function names to assertions from tests of those functions. Next, GenerateMocks (Algorithm 2) uses A to construct mocks for functions in the domain of A, yielding program p 0 , which includes the original sketch p plus those mocks. This step also returns a mapping F from the original function names to the corresponding mock names. Finally, MockHarnesses (Algorithm 3) creates the output sketch pm, which augments p <sup>0</sup> with copies of the original sketch's harnesses, except the copies call the mocks instead of the original functions.

Critically, during this last step, holes are not renamed when the harnesses are copied. Moreover, the newly generated harnesses are prepended to the sketch. Thus, when CEGIS tries solving each harness in p<sup>m</sup> in order, it will first find solutions that are consistent with the mocks. Then when it reaches the original harnesses (which must remain in case there is information in them not captured by the mocks—see discussion of GenerateMocks below), CEGIS can use the information it already derived from the mocks to find the ultimate solution to the original problem.

In the remainder of this section, we describe each step of the algorithm in detail. Below, we capitalize the names of sets of a given metavariable (e.g., Φ is a set of formulas φ, etc.), and we use vector notation to indicate arrays (e.g., ~s is an array of statements s).

Building the assertion mapping. Each mock expresses the specification of an original function as it is encoded by that function's tests. To start, Sketcham collects assertions from those tests into an assertion mapping. Algorithm 1 builds



the assertion mapping A from the input sketch p. The algorithm begins by initializing A to empty and Φ to the set of all assertions from all tests in p. It then selects two subsets of Φ. The set Φ<sup>0</sup> contains all assertions that do not include calls to any functions, and the set Φ<sup>1</sup> contains all assertions that include exactly one function call. We exclude assertions with multiple function calls so that mocks are standalone, to conform to the technical requirements Sketch imposes on models. As a consequence, we exclude some terms that present no such concerns (e.g., conjunctions of otherwise unrelated terms), as translating them to assumptions may be much more complex or even impossible. We leave extending BuildAssertMap to more assertion patterns to future work.

For each function f called in an assertion in Φ1, on line 7 we next compute the set Φ<sup>f</sup> from Φ<sup>0</sup> (the assertions that hold throughout each test, including at calls to f) and the subset of Φ<sup>1</sup> that refers to f. For example, consider the assertion in h\_sort in Figure 1d. This code refers to the result of calling sort(n, vs), so Φ<sup>1</sup> = {φi(sort(n,vs))}, where the φis capture the assertions in h\_sort. Additionally, if we picked, say, a loop unrolling bound of 4, then Sketch would implicitly assert n<4, resulting in Φ<sup>0</sup> = {n<4}. In general, Φ<sup>0</sup> might contain additional assertions that are irrelevant to the calls in Φ1. For example, loop unrolling for harness h\_dedup (not shown) might add another bound m<4 to Φ<sup>0</sup> for sort. However, such irrelevant assertions will not change the resulting mock.

In some cases, we cannot add assertions in Φ<sup>f</sup> to A because other assertions on the same variables interfere. For example, suppose the sketch includes assert f(x) and assert g(x). Then Φ<sup>f</sup> might not completely characterize f—the assertion in Φ<sup>f</sup> is valid only if assert g(x) also holds, which puts an unknown (until the full sketch is solved) constraint on x. Thus, in this case, our algorithm discards the assertions in Φ<sup>f</sup> . More specifically, on line 9, the loop

Algorithm 2 Mock rewriting: generate mocks

Input: p - the sketch Input: A - output of Algorithm 1 Output: p 0 - the sketch augmented with mock definitions Output: F - finite mapping from an original function name to its mock 1: function GenerateMocks(p, A) 2: F ← ∅, p<sup>0</sup> ← p 3: for all f 7→ Φ ∈ A do 4: def f(~x){. . .} ← the definition of f in p 5: f<sup>u</sup> ← FreshName(f) 6: ~s ← [ ] 7: Φ<sup>0</sup> ← {φ ∈ Φ | 0 = |f(. . .) ∈ φ|} 8: Φ<sup>1</sup> ← {φ ∈ Φ | 1 = |f(. . .) ∈ φ|} 9: for all φ ∈ Φ<sup>1</sup> do . convert asserts into assumes 10: f(~e) ← the lone function call in φ 11: φ<sup>u</sup> ← φ[f(~e) := fu(~x)] . substitute uninterpreted function 12: Ψ ← {x<sup>i</sup> = e<sup>i</sup> | 0 ≤ i < |~x|} . equate parameters to arguments 13: φ <sup>0</sup> ← ( V Φ0) ∧ ( V Ψ) =⇒ φ<sup>u</sup> . the condition where φ holds 14: φ <sup>00</sup> ← JFV(φ); Ψ ` φ 0 K 15: ~s.append(assume φ 00) 16: end for 17: f<sup>m</sup> ← FreshName(f) 18: F[f] ← f<sup>m</sup> 19: d<sup>m</sup> ← def fm(~x){ ~s return fu(~x) } . create the mock definition 20: p 0 .insert(dm) 21: end for 22: end function

removes any φ ∈ Φ<sup>f</sup> whose free variables overlap with free variables outside of Φ<sup>f</sup> . The process iterates in case free variable dependencies cascade. For example, the existence of assert g(x) would eliminate assert f(x-y), which would in turn eliminate assert f(y). The result is the transitive closure of the allowable assertions about each function.

Generate mocks. Next, Algorithm 2 iterates through each function in the domain of A, generating a corresponding mock to add to the augmented sketch p 0 . As it does so, it also builds a map F from function names to the names of the generated mocks.

For each f 7→ Φ ∈ A, GenerateMocks begins by finding the definition of f and creating a corresponding freshly named uninterpreted function fu. It then initializes ~s, the assumptions to be inserted into the new mock body, to empty. Then, from each asserted formula φ ∈ Φ, the algorithm creates a formula φ<sup>u</sup> by substituting the single function call f(~e) in φ with a call fu(~x), where ~x are the formal parameters of f (line 11). Notice this call to f<sup>u</sup> is the same no matter the


original call to f, which ensures the generated mock conforms to the technical requirements Sketch imposes on models. To encode the actual information at the call site, we next add a precondition. The algorithm constructs φ 0 (line 13), which is an implication denoting that φ<sup>u</sup> holds if the ancillary asserts Φ0, and the equalities x<sup>i</sup> = e<sup>i</sup> from the call to f hold. One nuance we elide here is that Sketch augments all function calls with an additional explicit path condition parameter that captures conditional branches taken up to the point of the call, which makes it easier for Sketch to translate the IR into a SAT formula. For soundness, we include this path condition as a premise of φ <sup>0</sup> and assign f<sup>u</sup> the path condition >. Note that our implementation trims Φ<sup>0</sup> before adding it to φ 0 to the subset containing only the variables in ~e.

Next, the algorithm performs quantifier elimination on φ 0 , yielding φ <sup>00</sup> (line 14). More precisely, <sup>J</sup>FV(φ); <sup>Ψ</sup> ` <sup>φ</sup> 0 <sup>K</sup> eliminates variables in FV(φ) from <sup>φ</sup> 0 , searching for witnesses in Ψ. Then, φ <sup>00</sup> is added to ~s as an assume, and the loop continues until all mappings for f have been handled.

Finally, on lines 17-19 the algorithm computes a fresh Sketch name f<sup>m</sup> for f, adds a mapping to F, and creates function definition d<sup>m</sup> for fm. The function f<sup>m</sup> takes the same arguments as f, assumes all formulas in ~s, and returns f<sup>u</sup> on fm's arguments. Thus, when f<sup>m</sup> is called, the assertions about f from its original test suite in p are assumed on fm's arguments, as we saw in Section 2. The definition d<sup>m</sup> is added to p 0 , and mock generation continues until all mappings in A have been traversed.

New mock harnesses. The last step of Sketcham adds calls to the mocks generated by GenerateMocks. One na¨ıve approach would be to simply replace each call to f with a call to f<sup>m</sup> for all f 7→ f<sup>m</sup> ∈ F. However, this will not work for two reasons. First, we need a full solution for the holes in all functions, including

those that are mocked. Replacing calls to f with calls to f<sup>m</sup> would remove many constraints on the holes in f, underconstraining their solutions. Second, as we saw earlier the template for f might contain additional information excluded by BuildAssertMap, so replacing f by f<sup>m</sup> might underconstrain f's callers.

Our solution is to create an output sketch that includes both the original sketch—including all calls to f in their original form—and duplicate sketch code that calls f<sup>m</sup> in place of f. The duplicated code refers to the same holes as the original sketch. Hence, information derived from the duplicated code can potentially greatly speed up solving of the original code.

Algorithm 3 shows MockHarnesses, which creates this duplicate code. The algorithm begins by constructing a call graph G from the sketch p 0 from the previous step. Note that none of the mocks in p <sup>0</sup> are called yet, so the call graph is the same as for the original sketch. Next, the algorithm duplicates the sketch one level of the call stack at a time, starting at the mocks and working up toward the harnesses. To limit duplication, e.g., for mocks called by recursive functions whose duplication would loop infinitely, the algorithm bounds the duplication depth. For each level i, it iterates through all functions g ∈ Callers(G, dom Fi), meaning functions g that call a function in the domain of F<sup>i</sup> . It duplicates each such g, replacing calls to functions f ∈ dom F<sup>i</sup> with calls to F<sup>i</sup> [f], and then adds the duplicated function to the sketch. Since g has now been renamed, g 7→ g 0 is added to a new mapping Fi+1, and calls to it are duplicated in the next iteration, repeating until reaching the root of the call graph or the maximum duplication depth. Note the process is the same for both regular function definitions and for functions that are harnesses.

For example, suppose harness h calls function g, which in turn calls function f, and assume GenerateMocks created f<sup>m</sup> and gm. Then in the first iteration, MockHarnesses creates a duplicate h 0 that calls g<sup>m</sup> and a duplicate g 0 that calls fm. In the next and final iteration, it creates a duplicate h <sup>00</sup> that calls g 0 .

When we insert the duplicate functions, we insert them before the original functions. This ensures that when we insert the duplicate harnesses that call the mocks, Sketch will solve those harnesses before solving the original ones.

#### 4 Evaluation

We evaluated Sketcham on ten benchmarks, running each from 11 to 1487 times until reaching statistically significant results. We found that, for six of ten benchmarks, Sketcham performs up to 5× faster than Sketch, for one benchmark Sketcham is slower by a factor of up to 0.9×, and for the remaining three benchmarks performance is indistinguishable. We examined the benchmark dedup (Figure 1) in depth and found that, as suspected, overall performance improvement is due to improved synthesis time when using sort\_mock.

Implementation. Sketcham comprises approximately 1075 lines of C++ code within the Sketch backend. The user enables Sketcham with -mock and specifies the max mock duplication depth via --bnd-mock-depth, which defaults to 3.

Because they clone and then rearrange the input Sketch IR program, the run time of Algorithms 1-3 is approximately linear in the number of functions and the number of asserts in the sketch. Our implementation covers the features given as part of the Sketch IR fragment in Figure 4, with the modification that we explicitly depict assignment, which Sketch IR does not require because it structurally hashes expressions to yield a compact in-memory representation [26]. We also note that Sketch includes additional features that we leave to future work, such as complex harness types, and that quantifier elimination is currently restricted to arithmetic expressions.

Benchmarks. We used the following benchmarks:


Sketch has a multitude of configuration options that can have a large effect on performance. The middle portion of Table 1 gives values for the four options that differ across the benchmarks: int type, whether Sketch uses symbolic integers (in either a bit-vector encoding or a sparse encoding [26]) or native integers [28]; int bits, the number of bits per integer; loop unroll, the maximum loop unrolling depth; and func inline, the maximum depth of function call inlining.

We selected values for these options that reflected each benchmark's design and demonstrated pronounced run time differences from Sketch to Sketcham, as follows. double and absval use Sketch's defaults. fib tests recursively computing the Fibonacci sequence up to the tenth entry, so function call inlining is set accordingly. regex is required to reject bad matches, which requires higher unrolling and inlining. datetime, boyerMoore, spellcheck, and minpair need higher loop unrolling to iterate over long strings. These last three and both dedups also do much better using native integers. The dedups also run unreasonably slowly with more bits or higher unroll, so we reduced the amount of unrolling. In all our benchmarks, any configuration options not discussed here were left as their defaults, including the mock duplication depth, with the default of 3.

Methodology. All measurements were taken on a 3.2 GHz AMD Ryzen 5 1600 system with 32GB of RAM. We found that while most benchmarks consistently


Table 1: Benchmark config options and characteristics.

perform within half an order of magnitude under both Sketch and Sketcham, in a few cases synthesis time varies by as much as two and a half orders of magnitude. To account for this variance during our evaluation, we repeatedly ran each benchmark until achieving statistical significance, between 11 and 1487 times, as listed in the rightmost portion of Table 1. Each run was executed with the system otherwise almost totally idle to minimize interference. While most runs completed successfully, we exclude those that exceed a 60 minute timeout or fail to synthesize due to exhausting system memory or a crash within Sketch. To give an idea of the problem size, the leftmost portion of Table 1 lists the numbers of lines and holes per benchmark.

As other work has observed [9], performance evaluation methodologies that lack rigor can lead to misleading and incorrect conclusions. To avoid this problem, we collect enough data to calculate a percentile's confidence interval (CI) at a given confidence level (CL). We employ the classic Clopper-Pearson [6] (or "exact") method using the probabilities of the Binomial distribution to iteratively calculate confidence intervals for a given dataset. While other methods are often used, many of these assume an underlying Gaussian distribution. The underlying distributions for our measurements are not known and do not appear to be Gaussian, a case the exact method handles correctly.

Run time variance is not correlated across configurations, so the number of runs needed for significance can differ from Sketch to Sketcham, as reflected in the "total" columns of Table 1. We ran each configuration repeatedly until measurements met two statistical significance conditions. First, that they reach a 95% CL that the population median lies within at most a 20% CI around the sample median. For example, for a sample median of 100 s, the population median might lie between 90 s and 110 s, or between 98 s and 118 s, depending on the underlying distribution. Second, the CI must range entirely between the first and third quartiles to increase the confidence that the median measurements adequately reflect the underlying distribution. In seven out of ten benchmarks these two conditions were sufficient to yield CIs that did not overlap across Sketch Sketch Sketcham

Sketch Sketcham

Sketch Sketcham

datetime

boyerMoore

Sketch Sketcham

Sketch Sketcham

Sketch Sketcham

Sketch Sketcham

Sketch Sketcham

Sketch Sketcham

Sketch Sketcham

Sketch Sketcham

Sketch Sketcham

spellcheck

minpair

double

dedup\_i

boyerMoore

0.0

0

200

400

600

800

0.1

0.2

0.3

0.4

0.5

0.2 0.3 0.4 0.5 0.6 0.7

200

0.1

20 30

> 120 140

800

100

0

spellcheck

minpair

regex

Sketch Sketcham

Sketch Sketcham

fib

Sketch Sketcham

spellcheck

Sketch Sketcham

dedup\_i

boyerMoore

0.1

0.1 0.2 0.3 0.4 0.5 0.6 0.7

0.2

0.3

0.4

0.5

absval

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

Sketch Sketcham

Sketch Sketcham

dedup\_i

Sketch Sketcham

dedup\_i

Sketch Sketcham

absval

boyerMoore

Sketch Sketcham

boyerMoore

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

0.1

0.1

0.2

1.0 1.5 2.0 2.5 3.0

0.3

0.4

0.5

0.2

0.3

0.4

0.5

0.0 0.5 1.0 1.5 2.0 2.5 3.0

40 60 80 100 600 800 1000 1200 0.2 0.3 0.4 30 40 50 60 400 600 600 800 1000 1200 0.2 0.3 0.4 40 60 80 60 80 100 120 600 800 1000 1200 0.2 0.3 0.4 40 50 60 70 Fig. 5: Total time (s). Times are drawn as notched box plots, which give the distribution's median inside a notch indicating its confidence interval. As usual, the box extends to the first and third quartiles, and whiskers extend to the full distribution. To better focus on the data, we truncate some whiskers. Note differing y-axis scales both here and below.

Sketch Sketcham 0 spellcheck Sketch Sketcham 0 dedup\_m Sketch Sketcham 0.0 double Sketch Sketcham 0 fib Sketch Sketcham 0 minpair Sketch Sketcham 0 dedup\_m Sketch Sketcham 0.0 double 0 0 20 0 200 0.0 0 10 and Sketcham, which allows for statistically significant performance claims about these benchmarks.

Sketch Sketcham

spellcheck

Sketch Sketcham

dedup\_m

10 20

0.1

200 400

80

800

regex

minpair

Sketch Sketcham

Sketch Sketcham

double

fib

Sketch Sketcham

Sketch Sketcham

spellcheck

regex

100

70 80

70

0.5

0.1

40

400

70 80

1400

800

#### 70 4.1 Performance

200 400

20

100

80

Sketch Sketcham

regex

fib

Sketch Sketcham

regex

0

20

40

60

80

100

Sketch Sketcham

Sketch Sketcham

Sketch Sketcham

Sketch Sketcham

Sketch Sketcham

20

regex

fib

dedup\_m

datetime

Sketch Sketcham

Sketch Sketcham

absval

Sketch Sketcham

absval

Sketch Sketcham

boyerMoore

datetime

Sketch Sketcham

datetime

Sketch Sketcham

0.0 0.5 1.0 1.5 2.0 2.5 3.0

0.0 0.5 1.0 1.5 2.0 2.5 3.0

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

absval

0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7

0.0 0.5 1.0 1.5 2.0 2.5 3.0

> > 0

20

40

60

80

100

20 30 40 50 60 400 600 40 60 40 60 80 100 20 30 40 50 60 400 600 30 40 50 60 400 600 40 60 80 Figure 5 shows the running times of Sketch and Sketcham on our benchmarks. The distribution of times is shown as notched box plots. The boxes extend from the first to the third quartile, with the median shown as a mid-line. The CI is indicated by the notch. The whiskers extended to the minimum and maximum values (some whiskers are truncated to allow for a closer view of the median).

Sketch Sketcham 0 10 Sketch Sketcham 0 200 Sketch Sketcham 0 20 Sketch Sketcham 0 20 Sketch Sketcham 0 10 Sketch Sketcham 0 200 10 20 200 20 Following standard practice, we conclude that two configurations have a statistically significant difference in performance if their CIs do not overlap, as there is then high probability that the median times of the distributions are different.

0

Sketch Sketcham

fib

minpair

Sketch Sketcham

0

20

40

60

80

100

0

spellcheck

Sketch Sketcham

regex

0

20

40

60

80

100

We see that for six of the ten benchmarks, Sketcham is faster than Sketch, while one is marginally slower and three display no significant performance change. We investigated each benchmark's performance in detail, discussed next. The performance differences we report are ratios of the run time of Sketch to Sketcham for a given benchmark. Due to uncertainty we report speedup ranges for the median, comparing the opposite extents of each CI. This ranges from, at minimum, the ratio of the faster end of Sketch's CI to the slower end of Sketcham's CI, up to, at maximum, the ratio of the slower end of Sketch's CI to the faster end of Sketcham's CI.

The times shown are total run time, which can be broken down into synthesis, verification, and overhead time. For Sketcham, overhead can further be broken down into mock construction and normal Sketch overhead. The total runtime overhead of mock construction is less than 0.4% for all benchmarks except regex (3%) and both dedups (∼20%). In most cases, this time was dominated by the GenerateMocks and BuildAssertMap phases.

The double benchmark's performance is approximately the same in both cases. In fact, the CIs overlap almost completely, suggesting the performance may be dominated by constant factors in Sketch.

The absval benchmark is also approximately the same. It is another simple program that Sketch solves very quickly, and as such the mocks only add to the verification time.

The fib benchmark asserts that, on integers 0 to 9, the to-be-synthesized linear-time Fibonacci implementation returns the same result as an exponentialtime implementation. In Sketch, the calls to the exponential-time algorithm cause a slowdown. But since Sketcham replaces calls to the exponential-time algorithm with calls to a (constant-time) mock, Sketcham achieves a speedup of 3.8–4.5×. While it is difficult to make out in the plot, the median and CI lie immediately above the first quartile for Sketcham.

The datetime benchmark fails to synthesize in Sketch due to memory exhaustion, but it consistently synthesizes in just a few seconds using Sketcham. Investigating further, we found the bottleneck is a function that parses strings into integers in a loop that converts digits and adds them to a running total. For example, the digit sequence abc is converted to the integer 100\*a+10\*b+c. This conversion loop is unrolled to the maximum bound by Sketch, and the input strings are of varying sizes, which is encoded as a separate formula for each possible length. The SAT conversion algorithm translates symbolic arithmetic formulas according to combinations of possible values of their subformulas, which results in very large SAT formulas in this case. Later in the conversion, these are merged back together in another quadratic operation. Due to the number of formulas and overall formula size, this eventually exhausts memory. While Sketcham technically faces the same issue, it does so after decomposing the sketch into smaller formulas, and thus these limits are never approached.

The boyerMoore benchmark runs 4–5× faster under Sketcham than Sketch. The reason is similar to the previous case. boyerMoore includes a generator that constructs arithmetic expressions that add and subtract a small set of values including a hole. Sketch constructs these expressions recursively so they grow quickly, with the total number of terms determined exponentially by the degree of function inlining, and the resulting expressions have high symmetry, both factors that slow down solving, further compounded by the location of this expression deep within the sketch. Because Sketcham breaks the problem's dependencies, this expression can be synthesized separately from the rest of the program, which proceeds much more quickly.

The regex benchmark's overall performance using Sketcham is statistically significantly slower by a factor of 0.98×, which is a minimal difference in practice. The main mocked function here performs compilation of a regular expression into instructions for a virtual machine. Because compilation is recursive, it is difficult to give a specification that Sketcham can use. It is instead given by example with an exhaustive set of subproblems, which greatly increases the number of harnesses to solve. While most harnesses keep similar performance and the slowest harness is 8% faster in Sketcham, this is not enough margin to improve overall solve time.

The spellcheck benchmark using Sketcham sees a speedup of 1.5×, while minpair performs roughly the same (0.89–1.04×). Both rely on the same Levenshtein edit distance algorithm. The harness for this algorithm, which is the most time-consuming in either sketch, runs last in both settings, which reveals the source of the performance difference between the two benchmarks. minpair is dominated by synthesis time and spellcheck by verification time, which means that harnesses for the minimum pair function are more difficult to synthesize than for the spellcheck function, and so the former accumulates more state within the solver that is compounded when solving the Levenshtein harness. This slows it down enough to decrease the overall performance. On the other hand, the improvement of spellcheck is distributed across all individual harnesses, and across both synthesis and verification time, more than making up for the time it takes to construct and solve the mock harnesses.

Finally, the dedups show a notable performance improvement with Sketcham. In both dedup<sup>i</sup> and dedupm, the problem is large and complex enough that plain Sketch struggles with it. Sketcham eliminates the interactions of holes across the deduplication and sorting functions, which speeds up synthesis by a factor of 1.3–1.9× for dedup<sup>i</sup> and 1.003–1.5× for dedupm.

#### 4.2 Case Study: Deduplication

Next, we examine the performance of dedup<sup>i</sup> and dedup<sup>m</sup> in detail, as they illustrate the strengths and weaknesses of Sketcham. We break our discussion into comparisons of solving time across harnesses and comparisons of CEGIS synthesis time to CEGIS verification time.

Time to Solve Each Harness. Both dedup<sup>i</sup> and dedup<sup>m</sup> are structured the same way, and Sketcham creates the harnesses and mocks shown in Figure 1c for both. Figure 6 breaks down the total times for dedup<sup>i</sup> and dedupm, grouped by the 400 500

Sketch h\_sort

> Sketch h\_dedup

Sketcham h\_sort' h\_sort

Sketcham h\_dedup' h\_dedup'' h\_dedup

Fig. 6: Harness time (s)

300 harnesses for sort and for dedup. We exclude overheads such as time spent in mock construction, parsing the input, and reassembling the output.

Sketch h\_sort Sketch h\_dedup Sketcham h\_sort' h\_sort Sketcham h\_dedup' h\_dedup'' h\_dedup 0 100 200 dedup\_m We make several observations. First, comparing the first and third columns within each subfigure, we see the time for solving h\_sort is the same for Sketch and Sketcham. This makes sense because h\_sort' adds no information—it calls mocked sort and then immediately asserts the same specification as in the mock. Note that, while the trivial h\_sort' harness could be elided here, creating an analogous harness would be useful if the harness accidentally contained a contradiction. In such a case, Sketcham would almost instantly decide the harness is unsatisfiable, whereas Sketch could spend an arbitrary amount of time reasoning about the computation in the actual called function before detecting the contradiction.

Second, comparing the second and fourth columns within each subfigure, we see that the CI of h\_dedup using Sketcham lies well below the CI using Sketch. The speed improves by a factor of 3.2–4.7× for dedup<sup>i</sup> and 2.2–4.9× for dedupm. Examining this result in detail, we find that Sketcham works exactly as intended: h\_dedup'' calls the mocked sort, enabling it to synthesize quickly and assign holes correctly, which are then simply verified when checking h\_dedup (and h\_dedup' is trivial, similarly to h\_sort').

Third, also comparing the second and fourth columns, we see the variance in performance for Sketch is much greater than for Sketcham. Investigating further, we found this occurs for two reasons. First, the specification in h\_sort is weak enough<sup>2</sup> that sometimes an incorrect hole assignment for sort satisfies the verifier and is only discovered while synthesizing h\_dedup, forcing the solver to backtrack at great cost and simultaneously consider the holes in both functions. Second, even when the solver finds a correct assignment for sort, it includes the

<sup>2</sup> In addition to the specification we have supplied, a complete specification of sort relies on the existence of a permutation function over the array's indices.

0.2

0.4

0.6

0.8

1.0

1.2

Sketch Sketcham

Sketch Sketcham

Sketch Sketcham

0.2

0.4

0.6

0.8

1.0

1.2

Sketch Sketcham

1200 Fig. 7: Synthesis and verification time (s)

1000 1200

600 800 20 30 600 800 20 30 entire formula again while solving h\_dedup, resulting in a much larger problem and corresponding variability. In contrast, with Sketcham, h\_dedup'' is decoupled from sort, eliminating these issues.

40 50 60

40 50 60

1000

Sketch Sketcham 0 200 400 dedup\_m synthesis Sketch Sketcham 0 10 dedup\_m verification Sketch Sketcham 0 200 400 dedup\_m synthesis Sketch Sketcham 0 10 dedup\_m verification Fourth, we observe that both Sketch and Sketcham can solve h\_sort about 10× faster for dedup<sup>i</sup> than for dedupm. Overall, merge sort is more challenging for Sketch than insertion sort (note that since Sketch finitizes the problem by, e.g., unrolling loops, asymptotic complexity does not play a role). More surprisingly, synthesizing h\_dedup is also faster for dedup<sup>i</sup> compared to dedupm. We believe this occurs because synthesis of h\_dedup must sometimes recover from a bad hole assignment from h\_sort, which will be quicker for dedup<sup>i</sup> , and because the easier synthesis of dedup<sup>i</sup> means the solver accumulates less state, such as conflict clauses, that would otherwise slow down solving subsequent harnesses.

Finally, we begin to get a clearer picture of the divergence between dedup<sup>i</sup> and dedupm. In dedup<sup>i</sup> , h\_dedup synthesis is the performance driver, and the improvement using Sketcham has a significant impact on total performance improvement. In dedup<sup>m</sup> it is overshadowed by h\_sort, which dominates to the point that improvement elsewhere is not as significant a contributor. Combined with the overhead of mock construction, this leads to a less pronounced improvement in total performance.

Synthesis and Verification Time. Figure 7 shows the times for the CEGIS synthesis phase and verification phase for each benchmark under Sketch and Sketcham. Not shown are the overheads of mock construction, parsing, etc., which for dedup<sup>i</sup> we found took 3–4s in Sketch versus 17–19s in Sketcham, and for dedup<sup>m</sup> took 90–96s in Sketch versus 201–207s using Sketcham. We believe much of the difference between these could be eliminated with additional engineering effort.

Looking at verification times in Figures 7c and 7d, we see that while the verification times for Sketch and Sketcham are different, they are still relatively close: Sketcham is 0.81–0.86× slower for dedup<sup>i</sup> and 1.12–1.16× faster for dedupm. In contrast, comparing synthesis times in Figures 7a and 7b, we see a more significant speedup for Sketcham over Sketch: 1.59–2.55× for dedup<sup>i</sup> and 1.28–2.28× for dedupm. Moreover, if we compare synthesis and verification time, we see that the overall solving time for both benchmarks is dominated by synthesis time. Indeed, we observed even greater synthesis speedups on other benchmarks including fib (4.2–5.1×) and boyerMoore 5.2–6.9×, but the most extreme of which was spellcheck, which saw synthesis speed up by 308.4–345.7× using Sketcham. Thus, we find that Sketcham's performance improvements come from reducing synthesis time by introducing mocks that decrease the number of holes that need to be considered at once.

#### 4.3 Discussion

In general, we found that Sketch's performance is unpredictable in practice, which is influenced by factors such as the solver's random seed. For example, in terms of overall solving time, our experimental runs included several outliers (not shown in Figure 5) near the 60 minute timeout. In these cases, Sketch essentially makes a very poor initial guess for the holes, and verification produces counterexamples that do not add much information. Both Sketch and Sketcham exhibit this issue.

Moreover, often what seem like minor changes in the program sketch or configuration options can result in totally different solver behavior, and hence performance. One example of this was boyerMoore, which turned out to be non-linearly sensitive to the loop unrolling parameter. This benchmark was also extremely fickle about the problem formulation—holes in what seemed to be innocuous locations would lead to timeouts in both Sketch and Sketcham. Another example is dedup, which initially had a specification that omitted a requirement that the output array did not have a negative length. Without this constraint, the performance benefit of Sketcham was overwhelmed by the variability of the solver exploring ultimately impossible scenarios.

Overall, our results suggest that while Sketcham can't always outperform plain Sketch, it performs best on problems split into functions whose tests cover the behavior the sketch actually relies on while being easier to compute than the functions' actual implementations. While Sketcham affected the performance of both CEGIS phases, the best improvements were observed when the solving time of dependencies was dominated by the synthesis phase. For programs with these properties, Sketcham can exhibit a performance improvement of as much as 5× overall, with synthesis time improvements alone of up to 345.7×. Moreover, in some cases, such as datetime, Sketcham can solve problems that are out of reach of plain Sketch. For programs where these properties do not hold, Sketcham performance is typically similar to plain Sketch.

#### 5 Related Work

There are several threads of related work.

Program Synthesis with models. As discussed earlier, our work builds on work by Singh et al. [24], who propose manually created models for Sketch. While Sketcham relies on the core algorithm of that work, Sketcham frees the Sketch user from needing to write models, because we create mocks automatically from normal sketches. Mariano et al. [18] use algebraic specifications to model libraries. In contrast, our approach derives specifications from the input program's assertions, without requiring the programmer to add annotations.

Deriving mocks and specs from tests. Saff et al. [21] use the capture and replay of actual test executions to automatically generate mock dependencies with the goal of speeding up test execution. Fazzini et al. [8] further generalize this capture-and-replay technique to consistently model the environment of a mobile app under test, allowing for testing apps that use an inconsistent resource like a database or network device. Both of these target normal testing rather than synthesis. Nguyen et al. [19] leverage symbolic execution over input-output test pairs to perform program repair. However, they use these tests to model individual expressions instead of modeling entire functions. The insight underlying these approaches is similar to ours, however Sketcham is capable of both input-output pairs and general properties, and does not rely on either concrete or symbolic execution of tests.

Component-based synthesis. Gulwani et al. [10] model programs using logical input-output relations to synthesize loop-free bit-vector programs. Shi et al. [23] combine many solutions that each only partially meet a specification into one that meets the entire specification. Both approaches limit the synthesis search space by building their solutions from the bottom up, from a selection of base components. Smith and Albarghouthi [25] prune the search space using bottom up algebraic rewriting of the program into an equivalent normal form. In contrast to these, Sketcham derives its benefits from breaking apart input sketches from the top down, at function level granularity.

Modular synthesis using symbolic or actual execution. Samak et al. [22] derive specifications of class methods using symbolic execution and use them to synthesize a replacement shim class one method at a time. Van Geffen et al. [31] use symbolic execution to model abstract virtual machines to modularly synthesize a compiler one instruction at a time. In contrast, because our approach derives mocks directly from the input's assertions, we need not consider the code itself when modeling it. Hua et al. [11] modularize the synthesis of library calls through execution of actual partial programs. In contrast, we attempt to avoid called functions entirely by relying on their inferred specifications.

Other approaches. Bod´ık et al. [2] finalize incomplete programs using angelic nondeterminism. In contrast, Sketcham does not introduce arbitrary angelic values, but instead constrains any angelic-like behavior using a function's inferred specification. Huang et al. [12] use a divide-and-conquer strategy to iteratively split synthesis problems according to heuristics. In contrast, Sketcham splits problems structurally in a single pass. Polikarpova et al. [20] speed up synthesis through modular verification using refinement types. In contrast, our approach achieves a similar kind of modularity without being type-directed.

#### 6 Conclusion

This paper presents Sketcham, a new technique for decomposing program sketches during synthesis by turning a function's test suite into a mock that a caller can invoke in place of that function, thereby allowing separate reasoning about callers and callees. Sketcham gathers asserts from tests into a specification for each function which it embodies as a Sketch model. We implemented Sketcham as an additional pass with Sketch and evaluated it on a set of ten benchmarks. Our rigorous evaluation strategy ensured at a confidence level of 95% that our measurements demonstrate performance gains of as much as 5×, including one benchmark that otherwise timed out on Sketch. Based on these results, we believe that automatically generating mocks from tests with Sketcham is a promising new approach for achieving modular synthesis.

### Acknowledgments

We would like to thank Norman Ramsey, Milod Kazerounian, and the anonymous reviewers for their helpful comments. This research was supported in part by a Draper Fellowship.

### References


Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Counterexample-Guided Partial Bounding for Recursive Function Synthesis**

Azadeh Farzan and Victor Nicolet(B)

University of Toronto, Toronto, Canada {azadeh,victorn}@cs.toronto.edu

**Abstract.** Quantifier bounding is a standard approach in inductive program synthesis in dealing with unbounded domains. In this paper, we propose one such bounding method for the synthesis of recursive functions over recursive input data types. The synthesis problem is specified by an input reference (recursive) function and a *recursion skeleton*. The goal is to synthesize a recursive function equivalent to the input function whose recursion strategy is specified by the recursion skeleton. In this context, we illustrate that it is possible to *selectively* bound a *subset* of the (recursively typed) parameters, each by a suitable bound. The choices are guided by counterexamples. The evaluation of our strategy on a broad set of benchmarks shows that it succeeds in efficiently synthesizing non-trivial recursive functions where standard across-the-board bounding would fail.

#### **1 Introduction**

Most computational tasks can be broken into logical units, many of which involve evaluating a function over a data collection. Recursively defined data types are broadly used to implement these collections. In functional languages, recursive functions implement computations over these recursive data types. Consider a typical scenario where a programmer has implemented a function f over a collection C by defining a recursive data type A and implementing f as a recursive function fooA. Later, the programmer may need a different implementation foo<sup>B</sup> of f over a different data type B; perhaps B is better suited for an optimized implementation of f, or the programmer now needs an implementation of a new function g (in addition to f) over the collection C and the data type B is a much better choice than A for implementing g efficiently. Ideally, the programmer should not have to start from scratch implementing fooB.

In this paper, we propose a generic and efficient algorithm for synthesizing recursive functions in such contexts. Our synthesis problem is specified by the following three components: (1) a recursive *reference implementation* that precisely defines the functionality, (2) a high level *recursion skeleton* that specifies a recursion strategy (i.e. a traversal plan over the new recursive data type) for the target code, and (3) a mapping, called *representation function*, that converts an instance of the new data type to one of the old data type (of the reference implementation), and establishes that the two are different implementations of the same concept.

Let us illustrate our problem setup with the aid of an example. Consider the standard A-labelled binary trees, recursively defined as T → N il | Node(A, T, T) for an arbitrary type A, and the maximum in-order prefix sum (mips)

function depicted on the right. mips maintains a pair of values: sum, which keeps track of the sum of the elements it has traversed so far, and mps, which maintains the maximum value over all such sums. This reference implementation pre-

**Fig. 1.** Maximum in-order prefix sum

cisely defines the functional specification for a function f.

Suppose that the programmer needs an alternative implementation that can

be efficiently parallelized, and therefore, opts for the divideand-conquer *recursion skeleton* depicted on the right. The *par-*


*tially defined* code specifies that the tree should be traversed in a manner that each subtree is processed separately, and then the results should be combined by a function join. It does not, however, specify what computation is performed; the implementation of join and the initial value for s0 are *unknown*. In this example, labeled binary trees are the recursive data type for both the reference implementation and the target of synthesis. In cases like this, the representation function simply becomes the identity function.

Our algorithm reduces the problem to a set of recursion-free synthesis problems, which are solved using existing synthesis tools. It synthesizes the unknown computations for join and s0, and therefore produces the divide-and-conquer implementation of mips on binary trees:


At the high level, the problem of synthesizing a new recursive function can be framed as checking the validity of formulas of the type ∃f∀x : θ.φ(f, x, . . .) where θ is a recursive data type (i.e. x ranges over a set of inductively defined terms), f is the target recursive function, and the ellipses stand in for all the relevant components of our specific problem statement as outlined before. Elements of type θ are unbounded in two different dimensions: the recursive structure can be of arbitrary size and each element of it belongs to an unbounded (data) domain. A straightforward way of under-approximating the unbounded specification is to bound the universal quantifier ∀x : θ in both dimensions. The synthesis problem is reformulated to synthesize the function from a bounded set of examples which are concrete bounded elements of the data type with concrete elements in them. This can be done by applying a counterexample-guided inductive synthesis (CEGIS) [34] algorithm in the straightforward way.

Alternatively, one can attempt to tackle the two dimensions independently. The quantifier ∀x : θ can be bounded in one dimension, i.e. recursive structures of bounded size can be considered, and yet the elements of these bounded structures can range over unbounded domains. More formally, the universal quantification is instantiated over a finite set of bounded-depth terms, denoted by set T, and the resulting specification becomes ∃f.∀*a* ∈ D. - <sup>t</sup>∈<sup>T</sup> <sup>φ</sup>(f, t) where *<sup>a</sup>* are the free variables of the terms in T and of non-recursive type D. This bounding reduces the original problem to a standard functional synthesis problem (over unbounded data domains) that can be discharged to one of the many known solvers, employing a variety of techniques for it. The set of terms in T can still be discovered in a counterexample guided loop in the spirit of CEGIS, and therefore this algorithm can be viewed as a *symbolic* CEGIS variant.

The thesis of this paper is that forcing bounds on all recursively typed variables is unnecessary and can be avoided algorithmically. A subset of variables can retain their unbounded quantification and yet the problem can be reduced to a recursion-free functional synthesis instance. Recall the mips example. The join function takes two trees, l and r, and a value a as an input. The recursion-free specification for join can retain a universal quantifier on all trees for l and limit its bounded exploration to r. In other words, one can successfully synthesize the join function from examples enumerating a few small candidate trees for r and treating h(l) (i.e. the result of the computation on l) and not l itself for the inductive enumeration of examples for synthesis. We discuss in the paper how this information can be algorithmically derived from the specific components of our synthesis problem: the reference implementation, the recursion skeleton, and the representation function.

Beyond the decision on what quantifiers should be bounded, the synthesis algorithm also needs to determine a set of terms that are used to bound these quantifiers. We propose an algorithm that discovers these bounds guided by counterexamples in a refinement-style loop. We show that this algorithm is sound, satisfies the expected weak-progress property that other CEGIS instances have, and is *parsimonious* in a precise sense. We have implemented this algorithm as a prototype synthesis tool Synduce and demonstrate that Synduce can efficiently synthesize recursive functions from specifications.

#### **2 Background and Notation**

The notation introduced in this section is used for formalizing the result of applying recursive functions to symbolic inputs.

*Terms.* We make use of a set of *symbols* that are partitioned into *terminal symbols* Σ, *non-terminal symbols* N , and an infinite set of typed *variables* V. There is a unique symbol ◦? that stands for a *hole*. Terms are defined by the grammar T → x | T T where x is a symbol, and T T is a function application. These are the relevant classes of terms:


Two terms are equal, denoted by t =<sup>α</sup> t (standard alpha conversion), iff there exists two injective substitutions σ : F V (t) →V\ (F V (t) ∪ F V (t )) and σ : F V (t ) →V\ (F V (t) ∪ F V (t )) such that σt = σ t (i.e. syntactically equal).

A symbolic term t can be **expanded** into a term t iff there exists a substitution σ : F V (t) → T(F V (t ) ∪ Σ) that substitutes the free variables of t for symbolic terms with the free variables of t such that t = σt. The relation over symbolic terms, is a partial order defined as, t t iff t can be expanded into t . A single variable is the maximal element according to this partial order and concrete terms (of any depth) are minimal elements.

*Recursive Functions.* This paper focuses on recursive functions f : τ → D with terms of a recursive type (τ or θ) as input, and an output of type D. These functions can be executed on concrete or symbolic input terms of type τ . We assume all functions can be translated to *recursion schemes* as defined below:

**Definition 1 (**[26]**).** *A* recursion scheme *is a tuple* P = Σ, N , R, Λ *where:*


$$\begin{array}{ccccc} & (pure) & F \ x\_1 & \dots & x\_m \to t \\ \neg (pattern \text{ } matching) & F \ x\_1 & \dots & x\_m \ p \to t \\ \neg \cdot \cdot & \cdot & \cdot & \cdot \end{array}$$

*where the* x<sup>i</sup> *are variables,* p *is a symbolic term,* t *is an applicative term in* T(Σ ∪N ∪{x1,...,xn})*, and* F *is a non-terminal.*

*–* Λ : τ → D *is a distinguished non-terminal symbol whose defining rules are always pattern-matching rules.*

We associate with each recursion scheme P a notion of reduction. A *redex* is an applicative term of the form F σx<sup>1</sup> ... σx<sup>m</sup> σp for a substitution σ : V → T(Σ, N , V) and rule F x<sup>1</sup> ... x<sup>m</sup> p → t in R. The *contractum* of the redex is σt. The one-step reduction relation →⊆ T(Σ, N , V) × T(Σ, N , V) is defined by C[s] → C[t] whenever s is a redex, t is a contractum and C[ ] is a one-hole context. A recursion scheme is *deterministic* iff for any redex F s<sup>1</sup> ... s<sup>m</sup> there is exactly one rule l → r (in R) which *matches* that redex, i.e. there exists a substitution θ such that F s<sup>1</sup> ... s<sup>m</sup> = θ l.

Given a recursion scheme P = Σ, N , R, Λ and a term s ∈ T(Σ, N , V), L(P, s) denotes the language of (Σ ∪N ∪F V (s))-labelled trees resulted from the maximal rewriting of the term s with the one-step reduction relation associated to P. If P is deterministic, then L(P, s) is a singleton (the term s reduces to only one possible term), and s<sup>P</sup> denotes the unique resulting term. This notion of reduction is slightly different from the one used in [26], in that we do not require the substitution to be closed.

*Symbolic Evaluation.* For any function f that can be defined as a recursion scheme, the symbolic evaluation of f on input s is simply s<sup>f</sup> . In other words, f(s) = s<sup>f</sup> . In this view, recursive functions and the corresponding recursion schemes are interchangeable. For a recursion scheme Σ, N , R, Λ representing a function f and a variable x, f(x) and Λ x become two different ways of referencing the same concept. In this paper, we assume that all recursion schemes to be deterministic total functions. Specifically, they terminate on all inputs; symbolic evaluation (or the equivalent reduction) of a symbolic term always terminates.

*Types Notation.* We use capital letters A, B, C, and D to refer to base types, which are scalar types (int, bool, char,...) or unlabeled products of scalar types (e.g. int×int). Our focus is on functions that take as input elements of recursive variant (or sum) types denoted by τ, θ,.... We denote by κ1,...,κ<sup>n</sup> the constructors of a variant type τ with n variants. Each constructor is assimilated to a terminal symbol τ<sup>1</sup> × ... × τ<sup>k</sup> → τ , where k ≥ 0. We assume that all recursive types define finite structures, that is, one can always construct a term of type τ with a finite number of constructors and elements of base type. x : τ denotes the judgement x is of type τ , and ∀x : τ denotes the universal quantification of all variables x of type τ .

In this setting, where we distinguish base types and recursive types, we differentiate **bounded terms**, which are symbolic terms where all free variables are of base type (in VB), and **unbounded terms** where some variables can be of recursive type. An unbounded term t is a symbolic term of finite size, but there are infinitely many bounded terms that are expansions of t.

#### **3 Formal Definition of the Synthesis Problem**

The synthesis problem solved in this paper is defined by three components: a reference recursive function f : τ → D, a representation function r : θ → τ that maps inputs of the target function to those of f, and a recursion skeleton for the target function. All three components are formally modelled by recursion schemes (Definition 1). f and r are standard recursive functions representable by deterministic recursion schemes. The recursion scheme for the recursion skeleton S[Ξ] : θ → D includes a special set Ξ of symbols as a subset of its terminal symbols, which correspond to the unknown components for synthesis. These unknowns stand for constants or functions that have to be synthesized.

At the high level, the solution to the synthesis problem is the definition of a new recursive function. At the low level, each of the unknowns in Ξ need to be given a definition. In each problem instance, it is assumed that f and S[Ξ] use a common set of terminal symbols Σ that belong to a background theory T (e.g. linear integer arithmetic). Formally, the solution is identified by a mapping Z from the unknowns Ξ to function definitions λx1. . . . λxn.t where n ≥ 0 and t is a symbolic term in T(Σ, {x1,...,xn}) (a concrete term if n = 0). Let S[Ξ/Z] be the recursion scheme obtained by replacing the unknowns Ξ by their definition in Z. Any solution Z that satisfies the following specification is a valid solution:

$$\Psi \equiv \forall x:\theta, \mathcal{S}[\Xi/Z](x) = (f \circ r)(x)$$

*Example 1.* We use a problem instance with the goal of synthesizing a recursive function on *tree paths* as a running example of this paper. Recall the mips function given in Fig. 1. Suppose that we want to transform it to a function on *tree paths*<sup>1</sup> as an alternative data type to labelled binary trees. For an A-labelled tree (of type T ree), *Path* is a datatype defined by the following grammar:

$$Path \rightarrow Top \mid Zip((\top \mid \bot), A, Tree, Path)$$

Intuitively, a path decomposes a tree as shown on the right. The path *Zip*(, a, ta, *Zip*(⊥, b, tb, *Zip*(, c, tc, x))), from the root to a leaf decomposes the tree into the subtrees ta, tb, and tc.

The synthesis problem is specified by three recursion schemes. The recursion scheme f, on the right, models the function mips from Fig. 1. Λ<sup>f</sup> is

f : ⎧ ⎪⎪⎨ ⎪⎪⎩ <sup>Λ</sup>*<sup>f</sup>* <sup>t</sup> <sup>→</sup> <sup>G</sup> (0, 0) <sup>t</sup> G s *Nil* <sup>→</sup> s G s *Node*(a, l, r) <sup>→</sup> G (L a (Gsl)) r L a (s, m) <sup>→</sup> (s <sup>+</sup> a, *max* (s <sup>+</sup> a, m))

the non-terminal corresponding to the main function mips and G is an auxiliary function. An additional non-terminal L is used to mirror the tuple decomposition done by the let-binding in the code of mips.

The second recursion scheme is the representation function r from paths to trees. The input path is recursively decomposed by the

<sup>S</sup>[s<sup>0</sup>, g*<sup>l</sup>*, g*<sup>r</sup>*] : The last recursion scheme specifies the recursion skeleton of the target function with un-

$$r: \begin{cases} A\_r \text{ } Top & \rightarrow Nil \\ A\_r \text{ } Lip(\top, a, t, z) \rightarrow Node(a, t, A\_r, z) \\ A\_r \text{ } Dip(\bot, a, t, z) \rightarrow Node(a, A\_r, z, t) \end{cases}$$

rewrite rules, and *Node* is constructed recursively on the right or on the left depending on the first value contained in the *Zip* constructor.

$$\mathcal{S}[s\_0, g\_l, g\_r] : \begin{cases} A\_S \text{ Top} & \to s\_0 \\ A\_S \text{ Dip}(\top, a, t, z) \to g\_l \text{ a } (A\_f \text{ } t) \text{ (}A\_S \text{ } z \text{)} \\ A\_S \text{ Dip}(\bot, a, t, z) \to g\_r \text{ a } (A\_f \text{ } t) \text{ (}A\_S \text{ } z \text{)} \end{cases}$$

knowns s0, g<sup>l</sup> and gr. It traverses the input path, making recursive calls (Λ<sup>S</sup> z) on paths, and calling the reference function on subtrees (Λ<sup>f</sup> t). The goal is then to synthesize implementations of s0, g<sup>l</sup> and g<sup>r</sup> such that S[s0, gl, gr] is equivalent to f ◦ r.

<sup>1</sup> This example is from [24], which calls this data type *zipper*.

#### **4 Recursion-Free Approximations**

A *system of recursion-free equations* models an approximation of the full functional specification Ψ for a recursive synthesis problem instance.

**Definition 2.** *Given two sets of terminals* Σ *and* Ξ*, a* system of recursion-free equations *is a finite set of constraints* {e<sup>i</sup> = e <sup>i</sup>} *where* e, e ∈ T(Σ ∪ Ξ, VB)*.*

We denote by {e<sup>i</sup> = e <sup>i</sup>}<sup>i</sup>∈<sup>I</sup> the set of constraints of the system, and {xj}<sup>1</sup>≤j≤<sup>n</sup> <sup>≡</sup> <sup>i</sup>∈<sup>I</sup> F V (ei) <sup>∪</sup> F V (e <sup>i</sup>) are the free variables in the system. The above system defines a synthesis problem where Σ is the signature of some theory T and Ξ is the set of unknowns to be synthesized. A solution Z to this synthesis problem is a mapping from Ξ to function definitions. Z is valid iff the following formula is *valid*:

$$\forall x\_1: D\_1 \ldots \forall x\_n: D\_n \cdot \bigwedge\_{i \in I} (e\_i = e'\_i) [\Xi/Z]^{-1}$$

where (e<sup>i</sup> = e <sup>i</sup>)[Ξ/Z] denotes the term in which the unknowns Ξ have been replaced by their definition in Z. In the rest of the paper, we consider systems of recursion-free equations where the set of terminals Σ and the set of unknowns Ξ are fixed and the same as in the main synthesis problem of Sect. 3. We say that a system E is a sound approximation of a system E (E - E) (or the synthesis problem Ψ) when any solution of E (or Ψ) is also a solution of E .

#### **4.1 Partially Bounded Quantification**

Consider the formal definition of the synthesis problem in Sect. 3. Bounding the quantifiers consists in expressing the problem on a finite set of bounded terms. This bounding effectively eliminates recursion; recursive calls can be inlined a bounded number of times. Yet, since the free variables of the bounded term are universally quantified over an infinite base domain, a bounded term t of type θ represents an infinite set of concrete inputs (of bounded size).

We propose a different strategy for bounding the quantifiers: we aim to instantiate the quantifier on a finite set of bounded and *unbounded* terms such that the resulting specification is not recursive. To start, we instantiate the universal quantifier by a finite set of arbitrary symbolic terms T. Our first approximation then becomes the set of constraints:

$$E(T) = \{ \mathcal{S}[\Xi](t) = (f \circ r)(t) \mid t \in T \} \tag{1}$$

The set of constraints E(T) can be seen as a synthesis problem where free variables are universally quantified and the unknowns in Ξ are to be synthesized. E(T) is not guaranteed to be a system of recursion-free equations for all choices of T. For an arbitrary symbolic term t, calls to recursive functions may appear in subterms of S[Ξ](t) and (f ◦ r)(t). Restricting T to bounded terms would yield a recursion-free system after symbolic evaluation of both sides of the equation.

This, however, is too restrictive. There may exist unbounded terms t where the equation S[Ξ](t)=(f ◦ r)(t) can be *rewritten* to an equivalent recursionfree equation. Intuitively, in an applicative term (resulting from the symbolic evaluation of a recursive function f) the simple subterms of the form f(x) where x is a variable can be eliminated by replacing f(x) with a single variable a of type D which now stands for the result of the invocation of f on any x.

**Definition 3.** *A symbolic term* t *is maximally reducible (*t *is a MR-term) by a recursion scheme* P = (Σ, N , R, Λ) *iff* t<sup>P</sup> *is an applicative term in* T(Σ, N , V) *such that replacing all subterms of the form* (Λ x) *(where* x ∈ V*) by a fresh variable* x ∈/ F V (t) *yields a symbolic term.*

*Example 2.* The term z = *Zip*(, a, t, T op) where a is an integer and t is of type T ree is maximally reducible by f ◦ r and S[s0, gl, gr] (cf. Example 1). First we have r(z) = z<sup>r</sup> = *Node*(a, t, N il) and (f ◦ r)(z) = G (L a (Λ<sup>f</sup> t)) *Nil*. If Λ<sup>f</sup> t is replaced by (a1, a2) (of type int × int), then the term can be reduced further to (a<sup>1</sup> + a, max(a<sup>1</sup> + a, a2)). For the other function, we have S[s0, gl, gr](z) = g<sup>l</sup> a (Λ<sup>f</sup> t) s0. If Λ<sup>f</sup> t is also replaced by (a1, a2), then the term reduces to the symbolic term g<sup>l</sup> a (a1, a2) s0. Note that z is an unbounded term, since t is a variable representing a tree of arbitrary depth.

If every term in T is maximally reducible by both (f ◦ r) and S[Ξ], then every call to a recursive function can be eliminated in E(T). Note that this new *sufficient* condition for E(T) to be recursion free is strictly weaker than the condition of having the terms in T to be bounded; a maximally reducible term need not be a bounded term.

**Definition 4.** *A set of constraints* E(T) = {S[Ξ](t)=(f ◦ r)(t) |t ∈ T} *is well-formed iff every* t ∈ T *is maximally reducible by* f ◦ r *and* S[Ξ]*.*

A well-formed set of constraints E(T) can be transformed to a system of recursion-free equations. For each free variable x : θ in E(T), a fresh variable a : D is added and the subterms (f ◦ r)(x) and S[Ξ](x) are replaced by a in every constraint. We call this rewriting step *recursion elimination* over D. Note that the calls to f ◦ r and S[Ξ] are both replaced by the same variable, since their equivalence is part of the specification of the synthesis problem.

The transformation described above produces a recursion-free system of equations, but it does not always yield a *sound abstraction*, specifically when f ◦ r is *not onto* D. There may exist a solution of Ψ that is not a solution of the resulting system of equations. This can be fixed by having additional constraints (invariants) on the fresh variables. Let *Im<sup>f</sup>* : D → bool a predicate such that f ◦ r is onto {c | c : D ∧ *Im<sup>f</sup>* (c)}. Then, the abstraction is sound if the choices for a : D are limited to when *Im<sup>f</sup>* (a) holds.

*Example 3.* Recall Example 1. The maximum in-order prefix sum is not onto int × int, since the second element of the pair is always a positive integer. The constraint *Im<sup>f</sup>* (x, y) = y ≥ 0 is required to make the function onto. In Example 2, a<sup>2</sup> must be a positive integer.

**Definition 5.** *Let* T *be a set of maximally reducible terms by* f ◦ r *and* S[Ξ]*, and Im<sup>f</sup> a predicate such that* f ◦ r *is* onto {c | c : D ∧ *Im<sup>f</sup>* (c)}*. We denote by* E(T) *the equation system obtained by rewriting each constraint in* E(T) *to a recursion free equation, through recursion elimination over* {c | c : D ∧ *Im<sup>f</sup>* (c)}*.*

In the synthesis problem defined by E(T), the variables introduced by recursion elimination are universally quantified over their restricted range. The exact encoding of the range restriction by *Im<sup>f</sup>* depends on the implementation of a synthesis oracle.

**Proposition 1.** Z *is a solution of* E(T) *iff* Z *is a solution of* E(T)*.*

The proof follows from the construction of E(T) based on E(T). Combining this with the fact that E(T) results from bounding the universal quantifications in Ψ, we can conclude that E(T) approximates Ψ.

**Theorem 1 (Sound approximation).** *If* T *is a set of maximally reducible terms by* f ◦ r *and* S[Ξ]*,* E(T) *is a sound approximation of* Ψ*.*

By construction, any solution of the functional specification Ψ is a solution of the system of equations E(T).

*Example 4.* Let T = {T op, Zip(, a, t, T op), Zip(⊥, a, N il, z)} be a set of terms, where a : int, t : T ree and z : P ath. T op is a concrete term, therefore maximally reducible. We saw in Example 2 that Zip(, a, t, T op) is a MR-term. With a similar reasoning, one can conclude that Zip(⊥, a, N il, z) is a MR-term; note how the term differs in which subterm is unbounded depending on the first component of the Zip. Therefore, E(T) is a well-formed set of constraints and by substituting Λ<sup>f</sup> t and Λ<sup>S</sup> z for (a1, a2) (where a<sup>1</sup> : int and a<sup>2</sup> ∈ {v : int|v ≥ 0}), we obtain the following recursion-free system of equations:

$$\mathcal{E}(T) = \begin{cases} 0, 0 = s\_0, \\ a\_1 + a, \max(a\_1 + a, a\_2) = g\_l \ a \ (a\_1, a\_2) \ s\_0, \\ a\_1 + a, \max(a\_1 + a, a\_2) = g\_r \ a \ s\_0 \ (a\_1, a\_2) \end{cases}$$

with free variables a : int, a<sup>1</sup> : int and a<sup>2</sup> ∈ {v : int|v ≥ 0}.

In contrast to a canonical CEGIS setting, where the approximation is the specification projected over a finite set of concrete terms, our abstraction is over an infinite set of concrete terms represented by a finite set of symbolic terms. In the original functional specification, the equational constraint (f ◦ r)(x) = S[Ξ](x) ranges over all possible terms x of type θ. In the abstraction E(T), the universally quantified variables are the free variables of the terms in the equations, which correspond to the variable symbols of scalar type used in the symbolic terms of T, modulo the introduction of fresh variables during the rewriting of the set of constraints E(T) to the system of equations E(T).

#### **4.2 Refining Systems of Equations**

Our approximation, the system of equations E(T), is parametric on a set of maximally reducible terms T. This approximation can be refined by adding terms to T, since for any two set of terms R and T such that R ⊆ T, E(R) -E(T).

The convergence of the refinement process depends on the terms added at each step. We present our refinement algorithm in the next section, but the main insights behind it, not tied to specific algorithmic choices, are captured by Propositions 2 and 3.

**Proposition 2.** *Let* T *be a set of MR-terms and* Z *be a solution of* E(T)*. Then for any term* t *such that there exists* t ∈ T *s.t* t t *,* Z *is a solution of* E(T∪{t })*.*

This proposition implies that if Z is a spurious solution, then a counterexample term showing that it is not a solution of Ψ is necessarily not expanded from a term in T. We also learn that T should ideally be an antichain of at every refinement round, since adding expanded terms does not strengthen the approximation.

**Proposition 3.** *Given two terms* t *and* t *such that* t t *(i.e.* t *is an expansion of* t*) and a set of MR-terms* T *such that* ∀x ∈ T,¬(x t ∧ t x)*, we have* E(T ∪ {t}) E(T ∪ {t })*.*

Adding the less expanded term (i.e. t) yields both a more general approximation and a more compact one. In other words, given a choice, always choose the least expanded term as the counterexample for refinement.

#### **5 Synthesis Algorithm**

Our synthesis algorithm computes a sequence of approximations of the functional specification Ψ from Sect. 3. Each approximation is a system of equations of the form E(T) (Definition 5). The approximations are incrementally refined until the synthesis solution for one is also a valid solution for the synthesis problem specified by Ψ.

Figure 2 illustrates the work flow of our algorithm. At the beginning of each iteration, a solution of the system of recursion-free equations E(T) is synthesized. If no solution is found, then there is no solution for the original synthesis problem, since the E(T) is guaranteed to be a sound approximation (Theorem 1). If a solution Z is found, then Z is verified against Ψ and if it passes, then it is returned as a solution. Otherwise, the

**Fig. 2.** Approximation refinement algorithm.

verifier returns a counterexample term x<sup>C</sup> . By Proposition 2, x<sup>C</sup> cannot be an expansion of any term in T, and new terms related to x<sup>C</sup> have to be added to T in the spirit of refinement.

The algorithm additionally keeps track of a set U of non-maximally reducible terms, which intuitively represents the set of inputs not *covered* by the current approximation. The sets T and U are *complementary* in a precise sense: T ∪U is always a *boundary of* . A boundary (of a partial order) is an antichain C such that for any bounded term t, there is some c in C such that c t.

The counterexample x<sup>C</sup> is necessarily an expansion of some term u<sup>C</sup> ∈ U. But since u<sup>C</sup> is by definition not maximally reducible, one cannot just remove it from <sup>U</sup> and add it to <sup>T</sup>. The Expand step takes <sup>u</sup><sup>C</sup> as an input and produces two sets T and U to update the current sets T and U and repair the boundary before the loop restarts.

The figure on the right is a graphical representation of the boundary repair. The sets T (in blue) and U (in red) initially form a boundary. This boundary is updated by removing the term <sup>u</sup><sup>C</sup> and adding <sup>U</sup> and <sup>T</sup> (the results of the Expand step) to form a new boundary. The fact that T ∪ U always forms a boundary is a required invariant of this refinement loop: (i) T, as a parameter of E(T), is required to be an antichain (as discussed

in Sect. 4.2), and (ii) the Generalize step relies on the assumption that <sup>U</sup> is an antichain containing all the terms not yet sufficiently expanded to be in T.

We rely on existing tools/techniques for the steps Synthesize and Verify of Fig. 2. In the following, we describe the Initialize, Expand, and Generalize steps of the algorithm.

*Initialization.* There is a straightforward way to initialize T and U: apply the Expand component to a single variable <sup>x</sup> of type <sup>θ</sup> and take the resulting sets T of maximally reducible terms and U of non-maximally reducible terms. The Expand step is described in the next section. For Example 1, a variable <sup>x</sup> of type *Path* is expanded to produce T = {T op} and U = {*Zip*(⊥, a, t, z), *Zip*(, a, t, z)} with variables a, t, and z of the appropriate types.

# **5.1** Expand **: Producing Maximally Reducible Terms**

Given an input term <sup>u</sup><sup>C</sup> , Expand generates two sets <sup>T</sup> and <sup>U</sup> such that the terms

in T are maximally reducible by both f ◦ r and S[Ξ]. The computation of these terms is done by expanding the input term u<sup>C</sup> until a set of maximally reducible terms is found. The algorithm on the right illustrates the process. At each step, a term u<sup>0</sup> is picked from the set of non-maximally reducible terms U . This term is expanded once, by a call to ExpandOnce (which is described later). The resulting set of terms is then partitioned into a set of maximally reducible terms T


and a set of non-maximally reducible terms U; the latter is used to update U . The choice of u<sup>0</sup> at the first line of the loop is important for the termination of the algorithm. There may be an infinite sequence of expansions if the u0's are adversarially chosen. There always exists a finite sequence of expansions yielding bounded terms which are by definition maximally reducible. A breadth-first exploration of all expansions is one such strategy that ensures the termination of the algorithm.

**ExpandOnce**. The input of ExpandOnce is a term <sup>u</sup><sup>0</sup> that is not maximally reducible. The following proposition characterizes u<sup>0</sup> and the reason for its nonreducibility:

**Proposition 4.** *Let* u<sup>0</sup> ∈ T(Σ, V) *and* g = (Σ, N , R, Λ) *a recursion scheme.* u<sup>0</sup> *is not maximally reducible by* g *iff there exists a subterm of* u0<sup>g</sup> *of the form* s = F t<sup>1</sup> ...t<sup>n</sup> x*, where* F ∈ N *and* F = Λ*, the terms* t<sup>1</sup> ...t<sup>n</sup> *are applicative terms, and* x ∈ F V (u0)*.*

The proof by cases on the subterms of u<sup>0</sup> is given in the extended version of this paper [7]. In order to take a step towards making u<sup>0</sup> maximally reducible, the variable x needs to be expanded. Expanding x into a term guarantees some rule F x<sup>1</sup> ...x<sup>n</sup> p → t ∈ R can be used to reduce u<sup>0</sup> further. Such a rule is guaranteed to exist for a recursion scheme representing a total function.

Next, we define how u<sup>0</sup> is expanded at a variable x identified by Proposition 4. u<sup>0</sup> can be written as C[x] for some one-hole context C[ ]. Assume the type β of x has constructors κ1,...κ<sup>n</sup> where each κ<sup>i</sup> has type γ<sup>i</sup> → β. The *pointwise expansion* of u<sup>0</sup> at x is the set of terms {C[κ1(x1)],...,C[κn(xn)]} where each x<sup>i</sup> is a variable (or a tuple) of variables of type γi.

In summary, ExpandOnce first identifies a variable <sup>x</sup> in <sup>u</sup><sup>0</sup> (Proposition 4) that needs to be expanded and then performs the *pointwise expansion* of u<sup>0</sup> at x and returns the resulting set of terms.

One important feature of ExpandOnce is that terms are expanded only where needed. Proposition 4 identifies the precise location (i.e. x) where expanding is necessary and ignores locations where it is not.

*Example 5.* Recall Example 1. Suppose u<sup>0</sup> = *Zip*(, a, t, z) is a (symbolic) path and an input to ExpandOnce, where <sup>a</sup> is an integer, <sup>t</sup> is of type T ree, and <sup>z</sup> is of type *Path*. u<sup>0</sup> is not maximally reducible and has to be expanded. Note that r(u0) = *Node*(a, t, Λ<sup>r</sup> z) and therefore (f ◦ r)(u0) = G (L a (Λ<sup>f</sup> t)) (Λ<sup>r</sup> z). The subterm (Λ<sup>r</sup> z) blocks the reduction of the term starting with G, because z blocks the reduction of Λ<sup>r</sup> z and therefore, u<sup>0</sup> has to be expanded at z. The pointwise expansion of u<sup>0</sup> at z yields the terms u<sup>1</sup> = Zip(, a, t, T op), u<sup>2</sup> = *Zip*(, a, t, *Zip*(, a , t , z )), and u<sup>3</sup> = Zip(, a, t, *Zip*(⊥, a , t , z ))}. Note that the tree element t need not be expanded; we showed in Example 2 that u<sup>1</sup> is maximally reducible and therefore, the expansion loop stops and returns T = {u1} and U = {u2, u3}.

Consider the symmetric term *Zip*(⊥, a, t, z) acquired by replacing the in u<sup>0</sup> with ⊥. The expansion of this term yields T = *Zip*(⊥, a, *Nil*, z) and U = {*Zi*(⊥, a, *Node*(a ,l, r), z)}. Note that unlike the case for u0, the tree element of the path has to be expanded and the path element need not be expanded.

#### **5.2 Counterexample Generalization**

The generalization of the counterexample x<sup>C</sup> is the unique term u<sup>C</sup> ∈ U such that u<sup>C</sup> x<sup>C</sup> . The term u<sup>C</sup> is guaranteed to exist because the algorithm maintains the invariant that T ∪ U is a *boundary*, and it is unique since U is always an antichain.

*Example 6.* After initialization, the synthesis solver attempts to find a solution for the system of equations given in Example 4. One possible solution is

$$s\_0 = (0,0) \qquad \qquad g\_l(a,(s\_1,m\_1),(s\_2,m\_2)) = a+s\_1, \max(m\_1,a+s\_1)$$

together with a similar solution for gr. But the solution for g<sup>l</sup> is incorrect; the first component should be a + s<sup>1</sup> + s<sup>2</sup> (i.e. the sum of both partial sums and the label of the node). The verifier returns a counterexample of the form x<sup>c</sup> = *Zip*(, 1, *Node*(?), *Zip*(, −2, *Node*(?), ?)) where the question marks stand for concrete subterms of the appropriate type. These subterms are ignored. The counterexample is generalized by selecting u<sup>2</sup> = Zip(, a, t, Zip(, a , t , z )) (where u<sup>2</sup> x<sup>C</sup> ), the term that was stored in U after the expansion described in Example 5. This determines where the algorithm must unfold the path one more time to build a stronger approximation.

We report in Sect. 7 that Synduce succeeds in finding a solution for this example with 3 refinement rounds in 1.57 s, whereas the symbolic CEGIS (described in Sect. 1) times out after 10 min over 6 refinement rounds.

#### **5.3 Algorithm Properties**

*Soundness.* Under the assumption that the steps Synthesize and Verify are soundly implemented, the overall algorithm is sound. By construction, T is always a set of maximally reducible terms. Therefore, E(T) is a guaranteed to be a sound approximation of Ψ by Theorem 1. The soundness of the verification oracle guarantees that any returned solution is in fact a solution of the synthesis problem specified by Ψ.

*Weak Progress.* Consider the naive algorithm that would expand T by simply adding the counterexample x<sup>C</sup> to it; x<sup>C</sup> is a maximally reducible term after all. This naive algorithm satisfies a weak progress property, namely that, the spurious solution Z from any round will not be a solution in any future round. Our algorithm does something more sophisticated and therefore it has to be argued that the same weak progress property holds. First, Expand satisfies the following property that guarantees T ∪ U to always be a boundary:

**Proposition 5.** *Let* t *be some symbolic term and* T , U *be the results of the call to* Expand(t)*. Then* T ∪ U *is a boundary of the set* {t |t t }*.*

Let <sup>u</sup><sup>C</sup> be the generalization of <sup>x</sup><sup>C</sup> . Proposition <sup>5</sup> guarantees that Expand computes and adds all possible expansions of u<sup>C</sup> to T. This in turn implies that there always exists a term t x<sup>C</sup> in the updated set T (after the call to Expand), which rules <sup>x</sup><sup>C</sup> out as a spurious solution in all future rounds. Note that the algorithm relies on the existence of u<sup>C</sup> in U. For this, it requires T ∪ U to be a boundary.

*Parsimony.* Finally, we can show that our algorithm is parsimonious with the selection of the terms for T in the following precise way:

**Theorem 2.** *[Parsimony] Let us assume* (T,U) *is a boundary that our algorithm reaches in some round, then* (T,U) *is optimal in the following two senses:*


Intuitively, all the terms in T are expanded to the extent necessary and no proper subset of T can form a boundary that maintains the same precise approximation that T ∪ U induces. The full proof appears in [7].

#### **6 Implementation**

Our approach is implemented in Synduce [36], a tool written in OCaml [22], and the inputs are recursive functions and datatypes written in Caml.

#### **6.1 Verification and Synthesis Oracles**

Synduce uses bounded model checking to implement Verify from Fig. 2. A bounded check for the validity of a synthesis solution Z is encoded as the validity of the formula ∧t∈<sup>T</sup> ∀a ∈ F V (t).S[Ξ/Z](t)=(f ◦ r)(t) for a set of *bounded* terms T. Z3 [25] is used as the backend SMT solver, which produces a counterexample in the form of a term for which at least one equality constraint is invalid.

Synduce spends most of its time in the Synthesize box of Fig. 2. Since the input to Synthesize is guaranteed to be a recursion-free synthesis specification, any off-the-shelf syntax-guided synthesis (SyGuS) [4] solver that supports the standard language [29] can be used to implement Synthesize. We use CVC4 [5] for the results presented in this section.

A SyGuS problem is specified by a grammar describing the space of programs to be synthesized and a set of constraints. In this case, the grammar is generated from the type of the functions to be synthesized (the unknowns in Ξ), which can be inferred from the constraints where they appear. Instances of generic grammars for integers and booleans can be found in the SyGuS language specification [29], and these grammars for base types can be combined into tuples in a straightforward manner. The constraints are the equations of the system, with the addition of the predicates constraining the domain of the variables, i.e. *Im<sup>f</sup>* from Definition 5. Each recursion-free equation e = e is translated to a constraint of the form ¬( - v∈F V (e)∪F V (e-) *Im<sup>f</sup>* (v)) ∨ e = e where Im<sup>f</sup> (v) is the predicate associated to the variable v.

#### **6.2 Baseline Method**

The goal of our experimentation is to evaluate the efficiency and efficacy of the proposed partial quantifier bounding approach for synthesis of recursive programs. Since there is no available (automated) tool that solves the specific problem posed in this paper, we implemented the symbolic CEGIS technique (as outlined in Sect. 1) to serve as a *baseline*. To be precise, the algorithm of Fig. 2 is modified by removing the Generalize and Expand steps; the symbolic counterexample returned by the verification at each step is added directly to the set of terms instead of being generalized. The set T is also initialized as a set of bounded terms of some minimal depth, depending on the particular definition of the data type. Note that since the baseline method is counterexample-guided, it is better than the more straightforward finitization techniques, for example, manual finitization by a preset bound.

We also implemented the concrete CEGIS method (outlined in Sect. 1) to confirm that the symbolic CEGIS is the better choice. Symbolic CEGIS solves 6 more benchmarks than concrete CEGIS, and does better time-wise in the vast majority of the rest. Detailed results are given in the extended version of this paper [7].

#### **6.3 Optimizations**

We implemented a few simple, straightforward and generic (i.e. they can be incorporated in any SyGuS solver) optimizations. These aim to compensate for the brittleness of the SyGuS solvers, which can fail for very simple constraints for no good reason. Here is a brief overview of these optimizations, which are applicable to any system of equations (baseline's and ours):


These optimizations are applied to both the baseline method and our algorithm for the purpose of evaluation. The extended version of this paper [7] includes more detailed evaluation of them and experimental results illustrating their precise impact on each algorithm.

### **7 Evaluation**

We evaluate Synduce on a broad set of benchmarks. Our benchmarks are grouped into six categories. Table 1 lists all the benchmarks, grouped accordingly. Each category, shares the same representation function and *polymorphic* recursion skeleton, but a different reference implementation is used to specify the synthesis problem. The recursion skeletons (and the representation functions) are polymorphic and therefore reusable. Only 9 different skeletons and 4 different representation functions were used across our 43 benchmarks. More details about the benchmarks, including the simple 9 utilized skeletons, appear in the extended version of this paper [7].

#### **7.1 Case Studies**

*Changing Tree Traversals.* An example of this category is the mips example used in the introduction. The reference function is a natural implementation of a function with a post- or in-order traversal of a binary tree. The target is an equivalent implementation corresponding to the divide-and-conquer tree homomorphism style recursion.

*From Trees to Paths.* A tree path (zipper in [24]) is a data structure used to represent a tree together with a subtree that is the focus of attention. Our running example belongs in this category. The other benchmarks in this category are from [24].

*Enforcing Tail Recursion.* In this category, the reference implementation is a direct-style recursion on the data structure, while the recursion skeleton specifies that an accumulator should be used to make the function tail-recursive. Tail recursive functions generally compile to more efficient code.

*Combining Traversals.* Suppose a collection of existing implementations computes different functions with different traversals of the same data structure. If in some larger context all of these functions need to be computed, *combining* them can lower the amortized cost. In this set of benchmarks, we synthesize automatically the implementation that corresponds to traversing the data structure with a single recursion strategy, combining the computations into one.

*Tree Flattening.* These benchmarks target the synthesis of an implementation on the more complex *plane tree* data structure from a reference implementation on the simpler binary tree data structure.

*Parallelizing Functions on Lists.* Parallelizing a function on lists can be seen as the translation of a recursive function on cons-lists to a homomorphic function on lists built with the concatenation operator. These benchmarks are from [8,9,23].

#### **7.2 Experimental Results**

To best of our knowledge, there are no available tools that can be directly compared against Synduce. We can transform our specification to a format that can be accepted by Leon [18]. However, the latter does not succeed in solving even the simplest of our benchmarks (e.g. *sum* in the list function parallelization category), likely due to the fact that the required deductive rules are missing. We comment on the rest of the available tools in Sect. 8.

Table 1 presents the results of comparing Synduce against the baseline method. Both techniques use symbolic counterexamples, and therefore, the comparison can highlight the performance impact of our partial bounding algorithm. The most important point of comparison is the overall synthesis time. In 9 out of 43 benchmarks, the baseline method times out. In another 5 cases, it outperforms the baseline by two orders of magnitude. In the easiest of the benchmarks, i.e. when the overall synthesis time of the baseline is in tens of milliseconds, the two methods are equally good within a small margin of error. The bold number in each row highlights the fastest synthesis time.

Amongst the 9 benchmarks for which the baseline algorithm times out, 7 are cases where Synduce takes advantage of partial bounding by leaving some quantifiers unbounded. The baseline algorithm in these cases requires more terms and terms of higher complexity in the finite approximations. Two of the 9 benchmarks (*post-order mps* and *sum + mts + mps* ) are cases where the set of maximally

**Table 1.** Experimental Results. Benchmarks are grouped by categories introduced in Sect. 7.1. # steps indicates the number of refinement rounds. <sup>T</sup>*last* is the elapsed time before the last call to the SyGuS solver in the last refinement step before timeout. All times are in seconds. The best time is highlighted in bold font. A '-' indicates timeout (> 10 min). The "Inv" column indicates if codomain constraints were required. Experiments are run on a laptop with 16G memory and an i7-8750H 6-core CPU at 2.20 GHz running Ubuntu 19.10.


reducible terms is exactly the set of bounded terms (i.e. one cannot take advantage of partial bounding), but Synduce still outperforms the baseline because it adds smaller terms to the abstraction through generalization and produces less complex problems for the backend synthesis oracle. In summary, both counterexample generalization and the partial bounding yield big practical advantages in comparison with the baseline symbolic CEGIS algorithm.

It is noteworthy that whenever an instance is hard, the majority of the time is spent in the Synthesize step. This becomes nearly 100% of the time for the baseline algorithm whenever it times out. The weakness of the baseline method lies in the fact that the recursion-free instances generated by it are too difficult to solve by the backend solver. The timeout occurs within a few refinement rounds (at most 7) when the baseline algorithm gets stuck in the Synthesize step attempting to solve a prohibitively difficult recursion-free synthesis instance.

Across all benchmarks, our algorithm generally requires fewer refinement rounds than the baseline method. The few exceptions are the cases where the synthesis oracle gets lucky in producing a good solution when the target programs are very simple, for example in the case of the *sum* and *product* benchmarks of the flat tree category.

Finally, to isolate the precise contribution of the partial bounding idea, we evaluated the effect of each optimization on each algorithm. The applicability of a particular optimization highly depends on the particular set of constraints, which in turn depends on the specific benchmark and the algorithm (ours vs baseline) producing the constraints. Our synthesis algorithm yields more general and more succinct constraints, to which the optimizations are more often applicable. Of the 9 cases where Synduce succeeds and the baseline method times out, 7 are due to the inapplicability of these (simple) optimizations. Synduce outperforms the baseline algorithm with all optimizations turned off for both. The detailed results are given in the extended version of this paper [7].

#### **8 Related Work**

Synthesizing recursive programs is a challenging task, and several automated techniques have tackled the problem with different specifications of the problem and different approaches to the solution.

*Finitization*, for example by bounding the depth of unbounded inputs or the number of recursive calls or loop iterations, is a straightforward way of dealing with unboundedness in synthesis [4,37] and verification [10]. In [32,33], high-level synthesis techniques use domain specific knowledge to finitize input programs. Quantifier instantiation, i.e. replacing quantified terms with ground terms, is commonly used in theorem proving and verification, and has also been useful in synthesis [31]. Our proposed algorithm can be viewed in the spirit of quantifier instantiation, with the major difference that (universally) quantified terms are replaced with other (universally) quantified terms which are still over an unbounded domain, yet with fewer degrees of freedom in unboundedness.

*Synthesis through Program Transformation.* Our precise problem statement is inspired by the transformation system developed by Burstall and Darlington [6]. They set to automate the task of transforming an initial program specified as a set of first-order recursion equations into a more efficient program, by altering the recursive structure. Their approach is based on transformation rules and semi-automatic. They use specific rules, e.g. *associativity* of a data operation, to perform the transformations and such rules do not generalize well. We defer the reasoning about the operations on the data to an SMT solver, and therefore need not rely on such rules. Techniques based on program transformation have been applied to the synthesis of special classes of recursive programs before [13,15]. For example, the work in [1] focuses on tail recursion and a lot of attention has been given to producing divide-and-conquer recursions in the way of automated parallelization [2,8,23].

*Synthesizing Recursive Functional Programs.* Inductive techniques were developed to construct recursive programs from input/output examples [35], and this approach has been extended in more recent work [16,17]. The latter two are examples of an analytical approach to program synthesis in which programs are constructed from the analysis of examples. Other recent approaches are searchbased methods. Escher [3] synthesizes recursive functions from user-provided components by interactively asking for more examples from the user. λ**<sup>2</sup>** [11] synthesizes data structure transformations from input/output examples using higher-order functions.

Tools like λ**<sup>2</sup>** and Escher can be complementary to Synduce in a more general context of recursion synthesis. The user can try to synthesize an implementation of a recursive function over a *simple* data type using λ**<sup>2</sup>** or Escher using input/output examples with a higher chance of success. This then serves as the reference implementation input to Synduce which can aim for a more sophisticated implementation over a more complex recursive datatype.

Myth [27], Myth2 [12] and SynQuid [28] use type information to direct the search for a program satisfying a specification. In Myth, this specification is a set of input/output examples. Myth2 generalizes this approach by treating examples as limited types. The specification for Synquid is a polymorphic refinement type, and the tool synthesizes an implementation of the given type using components provided by the user. Type-based approaches work well within the expressivity of refinement-types as specifications, but refinement types cannot express constraints for all desired synthesis tasks. Our specification is strictly stronger than both input/output examples and refinement types.

In SyntRec [14], reusable templates are used to facilitate the synthesis of algebraic data type (ADT) transformations. The reusable templates are meant to lessen the burden of the user in specifying the search space of the programs to be synthesized every time. The recursion skeletons in our framework are effectively (reusable) polymorphic recursion templates. The user can be provided with a library of common recursive datatypes with representation functions mapping between these types, and useful recursion skeletons on these datatypes. SyntRec [14] synthesizes ADT transformations from a functional specification. In contrast, our tool takes this transformation as input (the representation function) and synthesizes a function from ADT to a base type.

Leon [18], a deductive verification and synthesis framework, can synthesize recursive functions from first-order specifications with recursive predicates. In Sect. 7, we commented on a comparison of Leon against Synduce.

*Higher-Order Recursion Schemes.* We use recursion schemes as a model for our programs, but our contribution has very little to do with the original work introducing this model. Higher-order recursion schemes have been introduced for model checking functional programs [19–21,30]. Pattern matching recursion schemes, introduced in [26], provide a model for functional programs that manipulate ADTs. We use them as an accurate description of a class of functions on ADTs and the notion of reduction associated with them as a crisp way of formulating symbolic evaluation.

#### **9 Discussion and Future Work**

We have demonstrated that partial bounding of quantifiers can be a powerful tool for the synthesis of recursive programs. Circumventing the unnecessary bounding of some quantifiers leads to simpler instances of recursion-free synthesis subtasks that can be handled by the current tools. Moreover, our counterexample generalization also yields simpler terms for bounding the quantifiers that have to be bounded. This is the result of our focus being on a class of recursive functions that perform structural recursion (i.e. recursion that deconstructs its inputs). This, together with our specific problem setup, takes the guesswork out of counterexample generalization and provides the means for a *constructive* counterexample generalization scheme which is demonstrably effective.

The reliance on structural recursion, therefore, limits the class of reference implementations and recursion skeletons that can define an acceptable synthesis instance in our framework. Another limitation tied to the input model is that the output of the recursive functions has to belong to the base (non-recursive) types to accommodate the reduction of the problem to one that can be solved by a backend solver. Consequently, the unknowns in a target recursion scheme have to all be functions from base types to base types.

In our problem setup, the recursion strategy (given by the recursion skeleton) is an integral part of the specification since it is used to communicate programmer intent. Expecting a *complete* recursion skeleton may be viewed as another limitation of our technique. For example, the mts (maximal tail sum) function can be computed as function on a list maintaining only one integer value (i.e. the current value of the maximum tail sum), yet, to implement mts in a divideand-conquer strategy, another computation, the sum of the elements of the list, has to be performed alongside this one. It would be great if the user can ask for a divide-and-conquer recursion strategy without having to know that the additional computation of sum is required as well.

Ideally, the user should be permitted to provide an *incomplete* recursion skeleton which sufficiently communicates the intent and leave the recursion skeleton to be completed automatically by the synthesis procedure. This is a tricky problem. There are not only many recursion strategies to choose from, but each choice also leads to unboundedly many ways to organize the computation on data. This adds yet another dimension of unboundedness to the synthesis problem beyond the two already tackled in this paper. Note that in other recursion synthesis work such as [3,12,14,28], new operations on data are not synthesized, and in contrast drawn from an existing pool of operations. Therefore, this particular problem does not apply in those contexts.

Finally, our method currently does not take into account invariants over recursive data types, e.g. an invariant that specifies that a tree is a binary search tree. Some properties of the datatypes can be encoded through the representation function, e.g. the associativity of the concatenation operator in the category of list parallelization benchmarks. Incorporating the more general invariants in future work will broaden the expressivity of the framework in handling more interesting problems.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **PAYNT: A Tool for Inductive Synthesis of Probabilistic Programs**

Roman Andriushchenko<sup>1</sup> , Milan Ceˇ <sup>ˇ</sup> ska1(B) , Sebastian Junges<sup>2</sup> , Joost-Pieter Katoen<sup>3</sup> , and Simon Stupinsk´ ˇ y<sup>1</sup>

> Brno University of Technology, Brno, Czech Republic ceskam@fit.vutbr.cz University of California, Berkeley, USA RWTH Aachen University, Aachen, Germany

**Abstract.** This paper presents PAYNT, a tool to automatically synthesise probabilistic programs. PAYNT enables the synthesis of finite-state probabilistic programs from a program sketch representing a finite family of program candidates. A tight interaction between inductive oracleguided methods with state-of-the-art probabilistic model checking is at the heart of PAYNT. These oracle-guided methods effectively reason about all possible candidates and synthesise programs that meet a given specification formulated as a conjunction of temporal logic constraints and possibly including an optimising objective. We demonstrate the performance and usefulness of PAYNT using several case studies from different application domains; e.g., we find the optimal randomized protocol for network stabilisation among 3M potential programs within minutes, whereas alternative approaches would need days to do so.

#### **1 Introduction**

Probabilistic programs are a powerful modelling language to describe systems containing probabilistic uncertainty. Their correctness and efficiency can be described as a set of declarative temporal constraints. Various verification tools cater for automating their a posterior verification: does a program satisfy a specification? Here, we focus on finite-state programs and consider specifications given as (conjunction of) temporal logic constraints. The automated verification of such constraints is supported by probabilistic model checkers such as Storm [19], Prism [35] or Modest [27].

These model checkers typically require a fixed program or a fixed model. This is not always in line with their intended usage: To keep development costs manageable and development cycles fast, system designs are preferably verified as

c The Author(s) 2021

This work has been partially supported by the Czech Science Foundation grant GJ20-02328Y and the ERC AdG Grant 787914 FRAPPANT, NSF grants 1545126, 1646208 and 1837132, DARPA contracts FA8750-18-C-0101 (AA), FA8750-20-C-0156 (SDCPS), by Berkeley Deep Drive, and by Toyota under the iCyPhy center.

A. Silva and K. R. M. Leino (Eds.) CAV 2021, LNCS 12759, pp. 856–869, 2021. https://doi.org/10.1007/978-3-030-81685-8\_40

**Fig. 1.** The workflow of the synthesis process.

early as possible. However, at early design stages not all system details are known or they are deliberately left out, and systems or their models are incomplete they contain holes. A hole may e.g., reflect a partially implemented controller for a complex system or an unspecified component for wireless communication.

A key aspect of the design cycle is to explore these designs, i.e., to do *design space exploration*. The verification challenge now is to analyze all combinations of fixing the hole with a concrete behavior/subsystem and reveal (Pareto-)optimal designs. Alternatively, designs should be robust for engineering choices made downstream, e.g., a system should ideally not depend on the specific characteristics of a single communication interface. Verifying that every combination of options satisfies the specification ensures that changes in available components do not need to trigger a redesign.

The application areas above require to reason about the presence and absence of designs (aka: realizations) satisfying a specification in a family of designs. To allow for efficient reasoning it is crucial that this family is concisely represented. A convenient way to describe such a family is to use *sketching* [2,45]. A sketch can be thought of as a program (or model) with holes, naturally fitting the use case outlined above.

Clearly, enumerating single realizations is unfeasible in the light of the combinatorial *design space explosion*. Instead, the prevalent approach connected with sketching is based on inductive synthesis. The idea is to analyze a single realization and generalize the analysis results to a set of realizations, often using the notion of counterexamples. In probabilistic programs, such a notion is challenging, as counterexamples are typically complex objects [1].

Driven by a range of applications, there has been significant algorithmic progress in the analysis of probabilistic program sketches and temporal logic constraints over the last years. Baier *et al.* [14] explored the use of symbolic model-checking methods so as to consider sets of realizations at once. Ceˇ ˇ ska *et al.* [12] used abstraction-refinement on sets of realizations and complemented this with a counterexample-guided inductive synthesis approach [11]. The latter two approaches have recently been integrated [3] and yield a speed up of multiple orders of magnitude over a baseline that enumerates all realizations.

This paper presents PAYNT<sup>1</sup> (Probabilistic progrAm sYNThesizer) that takes a program sketch, concisely describing a finite family of finite Markov

<sup>1</sup> Available at https://github.com/gargantophob/synthesis.

chains (MCs), and a specification, and finds a family member (aka: realization) that (potentially optimally) satisfies the specification, see Fig. 1. The design of PAYNT is rooted in oracle-guided synthesis and enables the flexible combination of a variety of state-of-the-art algorithms. For efficiency purposes, key algorithms are implemented within the Storm [19] model checker that dominated recent tool comparisons [24]. To deliver flexibility, the tool is built in a modular fashion on top of a python API. To ease the learning curve, the tool takes a conservative extension to the widespread Prism language as input.

PAYNT aims at two user groups: First, it provides a development platform for alternative algorithmic approaches, e.g. exploiting recent neurosymbolic approaches to find good designs. The tool provides the interface to define sketches and all baseline algorithms under one roof. Secondly, the analysis of sets of realizations is a valuable backend for automatic engines, e.g., when synthesizing finite-state controllers for partially observable MDPs (POMDPs) [33].

*Related work.* The synthesis problems for parametric probabilistic systems can be divided into two categories.

*Topology synthesis,* akin to aim of PAYNT, assumes a finite set of parameters affecting the MC topology. Finding a realization satisfying a given reachability property is NP-complete in the number of parameters [13], and can be naively solved by analysing all individual family members. An alternative [14] is to model the MC family by a Markov decision process (MDP) and use off-the-shelf MDP model-checking algorithms. The ProFeat [14] and QFLan [47] tool take this approach to quantitatively analyze alternative designs of software product lines [23,36]. These tools are limited to small families. To improve the scalability, inductive methods based on *abstraction-refinement* over the MDP representation [12], and *counter-example guided inductive synthesis* (CEGIS) for MCs [11] have been proposed. As shown by the *Maze* model in Sect. 5, the topology synthesis is closely linked to controller synthesis for POMDPs, a popular model for planning in AI under uncertainty. Other recent approaches to POMDP controller synthesis include the use of neural network oracles (obtained by reinforcement learning) to guide the search [48] and adaptive learning schemes based on imitation learning [30]. Note that the problem of sketching probabilistic programs that fit given data as studied, e.g., in [39,44], is different.

*Parameter synthesis* considers models with a fixed topology but with uncertain parameters associated to transition probabilities (or rates). It aims to analyze how the MC (or MDP) behaviour depends on the parameter values. Scalable approximate parameter synthesis techniques treat identical parameters in different transitions independently [10,42] and have been implemented in Storm [19] and Prism [35]. Exact approaches construct rational functions for symbolic reachability probabilities [16] and were improved in [18,25,29]. This approach has been also applied to problems such as model repair [4,40].

Both synthesis problems can be attacked by *search-based techniques* that do not ensure an exhaustive exploration of the parameter space. These include evolutionary techniques [26,38] and genetic algorithms [22]. Their combination

**Fig. 2.** The server for request processing.

with parameter synthesis has been pursued in [8] and is implemented in the tool RODES [9] to synthesize robust systems.

#### **2 Using** PAYNT

We exemplify the usage of PAYNT by the following synthesis problem.

Consider a server for request processing depicted in Fig. 2. Requests are generated (externally) in random intervals and upon arrival stored in a request queue of capacity Qmax. When the queue is full, the request is lost. The server has three profiles – *sleeping*, *idle* and *active* – that differ in their power consumption. The requests are processed by the server only when it is in the *active* state. Switching from a low-energy state into the active state requires additional energy as well as an additional random latency before the request can be processed. We further assume that the power consumption of request processing depends on the current queue size. The operation time of the server is random but finite.

The server is controlled by a power manager (PM) that observes the current queue size and then sets the desired power profile. More precisely, the PM distinguishes between four queue occupancy levels determined by the threshold levels T1, T2, and T3. These values are controllable parameters that denote *which fraction of the queue capacity is occupied*. In other words, the PM observes the queue occupancy of the intervals: [0, T1] ,(T1, T2] etc. For each occupancy level, the PM changes to the associated power profile P1,...,P<sup>4</sup> ∈ {0, 1, 2}, where numbers 0 through 2 encode the profiles *sleeping*, *idle* and *active*, respectively.

PAYNT takes as an input a *sketch* – a program description in the PRISM (or JANI) language containing some undefined parameters (holes) with associated options from domains. A PRISM program consists of one or more reactive modules that may interact with each other using synchronization. A module has a set of (bounded) variables that span its state space. Possible transitions between states of a module are described by a set of guarded commands of the form:

```
[action] guard → p1 : update1 ...... + pn : updaten
```
If the guard evaluates to true, an update of the variables is chosen according to the probability distribution given by expressions p<sup>1</sup> through pn. The actions are used to force two or more modules to make the command simultaneously (i.e. to synchronize). The holes can appear in guards and updates. Replacing each hole with one of its options yields a complete program with the semantics given by a finite-state Markov chain. The following sketch describes the PM (the modules implementing the other components of the server are omitted for brevity).

```
module PM
```

```
pm : [0..2] init 0; // 0 - sleep, 1 - idle, 2 - active
    [sync0] q <= T1*QMAX -> (pm'=P1);
    [sync0] q > T1*QMAX & q <= T2*QMAX -> (pm'=P2);
    [sync0] q > T2*QMAX & q <= T3*QMAX -> (pm'=P3);
    [sync0] q > T3*QMAX -> (pm'=P4);
endmodule
```
In our example, we consider the following holes and domains describing: the thresholds <sup>T</sup><sup>1</sup> ∈ {0, <sup>0</sup>.1, <sup>0</sup>.2, <sup>0</sup>.3, <sup>0</sup>.4}, T<sup>2</sup> ∈ {0.5}, T<sup>3</sup> ∈ {0.6, <sup>0</sup>.7, <sup>0</sup>.8, <sup>0</sup>.9}<sup>2</sup>, the corresponding power profiles P1,...,P<sup>4</sup> ∈ {0, 1, 2}, and the queue capacity <sup>Q</sup>max ∈ {1,..., <sup>10</sup>}. The resulting sketch describes a *design space* of 10·5·4·3<sup>4</sup> <sup>=</sup> 16, 200 different power managers where the average size of the underlying MC (of the complete system) is around 900 states.

The goal is to find the concrete power manager, i.e., the instantiation of the holes, that minimizes power consumption while the expected number of lost requests during the operation time of the server is below 1. Such specification Φ is formalized as a list of temporal logic formulae in the PRISM syntax:

```
R{"lost"}<= 1 [ F "finished" ] R{"power"}min=? [ F "finished" ]
```
Using the sketch and the specification Φ, PAYNT effectively explores the design space and finds a hole assignment inducing a program that satisfies Φ, provided such assignment exists. Otherwise, it reports that such design does not exist. For the example, PAYNT produces the following output containing the hole assignment and the quality wrt. Φ of the corresponding program:

```
hole assignment: QMAX=5,T1=0,T2=0.5,T3=0.7,P1=1,P2=2,P3=2,P4=2
R[exp]{"lost"}=0.6822759696 [F "finished"]
R[exp]{"power"}min=9100.064246 [F "finished"]
```
The obtained optimal power manager has queue capacity 5 with thresholds (after rounding) at 0, 2 = 5 · 0.5 and 3 = 5 · 0.7. In addition, the power manager always maintains an active profile unless the request queue is empty, in which case the device is put into an idle state. This solution leads to the expected number of lost requests of ≈ 0.68 < 1 and the power consumption of 9,100 units. PAYNT computes this *optimal* solution in one minute. This is three times faster than a naive enumeration of all solutions.

Let us consider a more complex variant of the synthesis problem inspired by the well-studied model of a dynamical power manger for complex electronic systems [5,21]. The corresponding sketch describes around 43M available solutions with an the average MC size of 3.6k states. While enumeration needs more than 1 month to find the optimal power manager, PAYNT solves it within 10 h.

<sup>2</sup> Note that this simply ensures that *T*<sup>1</sup> *< T*<sup>2</sup> *< T*3. PAYNT further supports *restrictions*—additional constraints on parameter combinations.

#### **3 Synthesis of Probabilistic Programs**

We formalize the synthesis problems supported by PAYNT and briefly present state-of-the-art synthesis algorithms; more details can be found in [3,11,12].

#### **Problem Statement**

*Sketch.* PAYNT uses sketches to define the set of designs. Let P be a sketch containing holes from the set H = {Hk}<sup>k</sup> with R<sup>k</sup> being the set of options available for hole Hk. Let R = - <sup>k</sup> R<sup>k</sup> denote the set of all hole assignments (realizations), P[r] denote the program induced by a substitution r ∈ R and D<sup>r</sup> denote the underlying MC. Note that the size of the set R is exponential in |H|.

*Specification.* PAYNT supports conjunctions of specifications with reachability and expected rewards. For a set T of target states, *reachability* properties ϕ ≡ Pλ[F T] with λ ∈ [0, 1] and ∈ {<, ≤, >, ≥} express that the probability to reach T relates to λ ∈ [0, 1] according to . *Expected reward* properties <sup>ϕ</sup> <sup>≡</sup> <sup>E</sup>λ[F T] express that the expected reward accumulated before T is reached relates to <sup>λ</sup> <sup>∈</sup> <sup>R</sup><sup>+</sup> according to ∈ {<, ≤}. Let <sup>P</sup>[r] <sup>|</sup><sup>=</sup> <sup>ϕ</sup> denote that the program P[r] induced by the realisation r satisfies ϕ. For a specification Φ = {ϕi}<sup>i</sup>∈<sup>I</sup> given by a finite set of properties, we write P[r] |= Φ to denote ∀i ∈ I : P[r] |= ϕi.

*Synthesis problems.* PAYNT is able to answer two types of synthesis questions for a PRISM sketch P with a set R of realizations and a specification Φ:

**Feasibility**: Find a realization r ∈ R such that P[r] |= Φ.

**Maximality**: For property ϕmax, find a realization r<sup>∗</sup> ∈ R such that r<sup>∗</sup> ∈ arg max r∈R {P[P[r] <sup>|</sup><sup>=</sup> <sup>ϕ</sup>max] | P[r] <sup>|</sup><sup>=</sup> <sup>Φ</sup>} .

Variants of the maximal synthesis problem for expected rewards and minimization are defined analogously. PAYNT also supports a relaxed variant of maximal synthesis, ε*-maximal synthesis*: find a realization r<sup>∗</sup> such that P[r∗] |= Φ and <sup>P</sup>[P[r∗] <sup>|</sup><sup>=</sup> <sup>ϕ</sup>max] <sup>≥</sup> (1−ε) · max<sup>r</sup>∈R {P[P[r] <sup>|</sup><sup>=</sup> <sup>ϕ</sup>max] | P[r] <sup>|</sup><sup>=</sup> <sup>Φ</sup>} for a given ε ∈ (0, 1].

#### **Existing Synthesis Methods**

Synthesis methods can be classified into two orthogonal groups: i) *complete* methods allowing to prove non-existence or optimally of the given problem, and ii) *incomplete* methods leveraging various smart search strategies and evolutionary algorithms [22,26,38]. While its architecture is flexible, the current

**Fig. 3.** Oracle-guided synthesis (adapted from [3]).

release of PAYNT is built around state-of-the-art *complete* methods. As a baseline and reference algorithm, the tool implements the so-called *one-by-one approach* [15] which simply enumerates through each realization r ∈ R. The designspace explosion renders this approach unusable for large problems, necessitating the usage of advanced techniques that exploit any structure of the family of MCs.

*Oracle-guided synthesis.* At the heart of PAYNT is an oracle-guided inductive synthesis approach [31,32,46]. A *learner* selects a realization r and passes it to an *oracle*. The oracle answers whether r satisfies Φ and, crucially, gives additional information, usually a counter-example (CE), whenever this is not the case. PAYNT implements two orthogonal different oracles: (a) an *inductive* oracle CE examines single realizations to infer statements about other realizations [11]. (b) a *deductive* oracle AR (Abstraction Refinement) argues about sets of realizations by considering (an aggregation of) these realizations at once [12]. PAYNT supports the combined use of these two oracles as a hybrid synthesis method [3].

Figure 3 [3] illustrates the communication between the learner and the two oracles. The Abstr-Oracle analyzes a sub-family <sup>R</sup> with 3 possible outcomes: 1) it proves that all its realizations satisfy Φ, i.e., that the synthesis problem is feasible, or 2) it proves that all its realizations violate Φ, i.e., the learner can safely discard R, or 3) the analysis is inconclusive and it returns safe bounds on the best- and worst-case behavior of all realizations in <sup>R</sup> wrt. <sup>Φ</sup>. The CE-Oracle analyzes a realization r and either proves that r satisfies Φ or it generalizes r into a subfamily R . The learner can discard R since it is guaranteed that all its realizations violate Φ. In the hybrid approach, the CE-Oracle exploits the bounds in order to compute smaller CEs allowing a better generalization. The learner maintains a queue of subfamilies R ⊆ R that has to be further processed and also controls which oracle is used based on their previous performance.

#### **4 Tool Architecture of** PAYNT

PAYNT is implemented on top of the probabilistic model checker Storm [19]. While the high-performance parts were implemented in C++, we use a python API to flexibly construct the overall synthesis loop. For SMT-solving, we use

**Fig. 4.** The tool architecture (Color figure online)

Z3 [37]. PAYNT takes a PRISM [35] or JANI [7] sketch and a set of temporal properties, and returns a satisfying realization, if such exists. Otherwise, it reports that no such realization exists.

Figure 4 depicts a high-level view on the tool architecture, which primarily consists of components for family handling (**purple**), chain building (**green**) and model checkers (**red**).

The *family handlers* are used to store information about the previously covered design space: Member enumeration simply iterates over all realizations. The SAT representation stores a SAT-formula describing unexplored realizations and uses the SMT solver Z3 for linear (bounded) integer arithmetic to retrieve the next candidate realization. The subfamily queue stores a collection of unexplored subfamilies and refines these subfamilies as hyper-rectangles. The *chain builders* take as input a single assignment r ∈ R or a set R ⊆ R of realizations, and produce an representation of the MC or a quotient MDP, respectively in the internal memory model of the model checkers. The *model checkers* are then used to verify these chains. They either output yes/no or, in the case of MDPs, provide lower and upper bounds on satisfiability probabilities. PAYNT includes a module for counterexample generation by using either a MaxSat [17,49] or a greedy state-expansion [3] approach.

Figure 4 also illustrates three analysis loops that mirror the behaviour of 1-by-1 enumeration (the baseline), CEGIS and AR. The 1-by-1 approach simply iterates over all possible realizations until a satisfying one is obtained. The CEGIS loop additionally constructs counterexamples to each unsatisfying realization r ∈ R, yielding a whole subset R ⊆ R of realizations that are pruned from the family. In contrast to this enumeration, the AR loop constructs and model checks MDPs from the subfamily queue and subsequently refines these subfamilies if the obtained bounds on satisfiability yield inconclusive results.

**Table 1.** Case study statistics and PAYNT synthesis times versus the naive 1-by-1 enumeration. Two problems per model are considered: an optimal synthesis problem (hard) and a feasibility problem (easy). In both cases, all realizations need to be explored to prove optimality and unsatisfiability, resp. Values indicated with <sup>∗</sup> are estimates.


The hybrid approach combines both AR and CEGIS approaches and switches between the two loops mid-execution. In particular, the integrated method executes the abstraction-refinement loop and, whenever it encounters an undecidable family that needs to be split, CEGIS takes a chance at analyzing it for a limited time period. If some family members are excluded based on a counterexample, the CEGIS engine updates the corresponding SAT representation to ensure it does not analyze the same member twice. There are two additional links that couple the AR and CEGIS loops and enable efficient integrated analysis. First is the use of bounds from MDP model checking during the greedy CE generation to allow the construction of larger family-aware conflicts. Since these bounds are associated with the states of the quotient MDP M<sup>R</sup> for the (sub-)family and counterexamples are constructed as sub-MCs of the MC Dr, r ∈ R, in the integrated setting we construct D<sup>r</sup> directly from MR, to save time on converting bound values between the two chains.

The implementation of PAYNT is composed of *30* Python modules containing *7k* source lines of code. These metrics consider only our implementation and do not include the extensions contributed to Storm and its Python API, invoked by PAYNT. All modules adhere to coding conventions for the Python code *PEP 8* [41,43] and are documented with *Sphinx* for automatic generation of documentation. The specific logic components are tested with unit tests to maintain their correct functionality. Regression tests verify the accuracy and correctness of the synthesis results. Our tests currently cover more than *90%* of the source code lines.

#### **5 Performance Evaluation and Applicability**

Table 1 lists the results of PAYNT on two variants (hard and easy) of five different case studies from various domains taken from [11,12]. Further on, we demonstrate the applicability of PAYNT and interpret the synthesis results for two of these case studies. All experiments are run on an Ubuntu 19.04 machine with Intel i5-8300H (4 cores at 2.3 GHz) and using up to 8 GB RAM, with all the algorithms being executed single-threaded. The artefact allowing to reproduce the experiments is avaiable at https://doi.org/10.5281/zenodo.4726056.

*Maze.* This synthesis problem can be seen as an instance of POMDP controller synthesis. A robot is deployed at a random location inside a known maze, see Fig. 5. The robot is only equipped with a simple wall sensor, and cannot distinguish maze cells with identical sets of surrounding walls such as cells 1 and 3, and cells 11 through 13. Observation-equivalent cells are indicated by the same color in Fig. 5. Possible actions are movements in the four cardinal directions. Movements are subject to a random error: e.g., upon moving east, with a small probability the robot actually moves west. We sketch a robot controller that helps it to reach the exit of the maze (cell 12). The controller may use a single bit of memory initially having the value 0. The holes in this sketch are taken actions (where to steer, how to change the memory bit) based on the current observation (detected walls, current memory state). This sketch describes a family of 9.4M candidate programs. Our goal is to find a realization that minimizes the expected number of steps to reach the exit.

Using the inductive synthesis techniques, PAYNT explores the set of candidate realizations in an hour (1 by-1 enumeration takes more than one day) and synthesizes the controller depicted in Fig. 5. Here arrows represent the steering direction based on the current memory value (number at the base of an arrow), as well as the corresponding memory update (number at the tip of an arrow). For instance, a robot in cell 1 goes west if the memory value is 0 and goes east otherwise, without changing the memory in either case. A robot at cell 0 always goes east and sets its memory bit to 1. The synthesized controller is

**Fig. 5.** The spatial structure of *Maze*. Cells with identical sets of surrounding walls are depicted with similar colors. The arrows depict the synthesized controller. (Color figure online)

optimal. If a robot reaches a cell with a unique set of enclosing walls (cells 0, 2 and 4), then it knows its precise position within the maze and can navigate to the exit. Similarly, navigating north from cells 11 or 13 ensures to eventually reach cells 0 or 4. If the robot is deployed in an orange or purple cell, then it has to 'try' one possible direction in order to recognize its position within the maze. For example, a robot deployed at cells 5–10 will first go north (recall that the initial memory value is 0), from where it can determine its cell. Note that in this observation group it is indeed more beneficial to first explore north since the robot is twice as likely to be initially deployed at locations 5/7/8/10, as compared to locations 6 and 9. The expected time to reach the exit for this policy is ≈9.8 steps. Note that this cannot be improved by adding more memory to the controller.

*Herman.* This case study considers a token ring with an odd number of stations that are connected by a unidirectional ring. Each station has a Boolean flag, observable by itself and by its successor in the ring. A station has a token when the two flags it observes are identical. A good configuration is a situation in which only one station has a token. All other configurations are faulty. A token protocol is self-stabilizing, if the ring gets from a faulty configuration into a good configuration. The performance can be measured as stabilization time, i.e., the expected number of rounds to reach a good configuration.

We sketch a variant of *Herman's randomized self-stabilization protocol* [6,28, 34]. In this protocol, all stations behave the same<sup>3</sup>. The protocol is synchronized, and in every round a station without token flips its flag. Every station that has a token must *choose* whether to pass a token (by setting its flag accordingly). In the original protocol this choice is the resolved on a single (biased) coin flip. We are interested in the synthesis of alternatives. We give each station an additional single bit of memory and the choice between 25 different coin biases. The parameters in the sketch are the choice of a coin based on the memory value as well as the memory updates. By resolving the choices, we obtain the same protocol for each station. The parameter combinations yield a family of 3.1M programs and the goal of the synthesizer is to identify the one that minimizes stabilization time from an initial configuration (all flags true). For a sketch describing a system with 5 stations, PAYNT finds the optimal protocol in around 18 min, while the 1-by-1 enumeration takes more than a day. The obtained optimal strategy relies on initially using the most fair coins available (bias ≈ 0.25) and keeping the memory bit at 1. Whenever a process eventually decides to keep the token, the memory is reset to 0 and the process starts using highly unfair coins (bias ≈ 0.07), implying that the process is more likely to keep its token for a long time until it is eventually passed further. Using this strategy, the system can on average stabilize in four rounds.

#### **References**


<sup>3</sup> In such anonymous networks, stabilization cannot be solved in a deterministic way [20].


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Adapting Behaviors via Reactive Synthesis**

Gal Amram<sup>1</sup>, Suguman Bansal<sup>2</sup>, Dror Fried3(B), Lucas Martinelli Tabajara<sup>4</sup>, Moshe Y. Vardi<sup>4</sup>, and Gera Weiss<sup>5</sup>

> <sup>1</sup> Tel-Aviv University, Tel Aviv-Yafo, Israel <sup>2</sup> University of Pennsylvania, Philadelphia, USA suguman@seas.upenn.edu <sup>3</sup> The Open University of Israel, Ra'anana, Israel dfried@openu.ac.il <sup>4</sup> Rice University, Houston, USA *{*lucasmt,vardi*}*@rice.edu <sup>5</sup> Ben-Gurion University of the Negev, Beersheba, Israel geraw@bgu.ac.il

**Abstract.** In the *Adapter Design Pattern*, a programmer implements a *Target* interface by constructing an *Adapter* that accesses an existing *Adaptee* code. In this work, we present a reactive synthesis interpretation to the adapter design pattern, wherein an algorithm takes an *Adaptee* and a *Target* transducers, and the aim is to synthesize an *Adapter* transducer that, when composed with the *Adaptee*, generates a behavior that is equivalent to the behavior of the *Target*. One use of such an algorithm is to synthesize controllers that achieve similar goals on different hardware platforms. While this problem can be solved with existing synthesis algorithms, current state-of-the-art tools fail to scale. To cope with the computational complexity of the problem, we introduce a special form of specification format, called *Separated GR(k)*, which can be solved with a scalable synthesis algorithm but still allows for a large set of realistic specifications. We solve the realizability and the synthesis problems for Separated GR(*k*), and show how to exploit the separated nature of our specification to construct better algorithms, in terms of time complexity, than known algorithms for GR(*k*) synthesis. We then describe a tool, called SGR(*k*), that we have implemented based on the above approach and show, by experimental evaluation, how our tool outperforms current state-of-the-art tools on various benchmarks and test-cases.

#### **1 Introduction**

Inspired by the well known adapter design pattern [18], we study the use of reactive synthesis for generating adapters that translate inputs meant for a target transducer to inputs of an adaptee transducer. Consider, as one motivating example, the practice of adding code to an operating system that mitigates the risk posed by newly discovered hardware vulnerabilities like Spectre and Meltdown [23,26]. While the discovery of such vulnerabilities puts constraints on how

A. Silva and K. R. M. Leino (Eds.) CAV 2021, LNCS 12759, pp. 870–893, 2021. https://doi.org/10.1007/978-3-030-81685-8\_41

the hardware can be used, the patch of the operating system (called adapter in this paper) takes upon itself to take care of running all applications without change [25]. It does so by allowing applications of the existing interface, while adapting their operation in way that ensures that the system is not exposed to the new threat.

Formally, we propose the following synthesis problem: given two finite-state transducers called *Target* and *Adaptee*, synthesize a finite-state transducer called Adapter such that

#### *Adaptee* ◦ *Adapter* -*Target*.

The symbol ◦ stands for standard transducer composition and the symbol - stands for an equivalence relation, a generalization of sequential equality, which we explain below. In words, we want an *Adapter* that stands between an *Adaptee* and its inputs and guarantees, such that the composition *Adaptee* ◦ *Adapter* is equivalent to *Target*. In the vulnerability patching example, *Adaptee* is a model of the constrained hardware and *Target* is a model of the hardware as used before the discovery of the vulnerability, without the new constraints. The *Adapter* that we generate models the patch that mediates between the vulnerable hardware and applications that are not aware of the vulnerability.

In our setting, an input to the synthesis algorithm is the equivalence relation along with the specification of the adaptee and of the target. While the problem of synthesizing an adapter such that *Adaptee* ◦ *Adapter* is sequentially equal to *Target* may be useful in some cases [32], we study here a more general problem. This is called for by applications such as the vulnerability covering patches described above. Specifically, we allow our users to specify an equivalence relation between *Adaptee* ◦ *Adapter* and *Target* that is not necessarily sequential equality. In this paper, we propose to use ω-regular properties [20] for specifying this equivalence relation, as follows. We assume, without loss of generality, that the outputs of both the *Target* and the *Adaptee* are assignments to disjoint sets of atomic propositions. We then consider sequences of pairs of such assignments that correspond to zipped runs of *Adaptee* ◦*Adapter* and of *Target* over the same input. Having this set of sequences in mind, the user specifies a set of temporal properties using an ω-regular formalism such as LTL or B¨uchi automata. The transducer *Adaptee* ◦ *Adapter* is considered equivalent to *Target* if all the properties that the user specified are satisfied for each sequence in the set [19]. Note that the equivalence relation can be very different than sequential equality, it can, for example, say that *Adaptee* ◦*Adapter* must be, in a way, a "mirror image" of *Target*, as demonstrated by the cleaning robots example in Sect. 4.1, where *Target* is a robot that cleans some rooms and *Adaptee* ◦ *Adapter* is a robot that clean all the rooms that *Target* did not clean.

The solution that we propose in this paper consists of two phases: we first transform the transducers to transition systems and arrive at a game structure that is more amenable for game-based techniques. Then we make use of the specific form of the resulting game and some simplifying assumptions about the form of the equivalence properties to solve the game efficiently. The game structures that we analyze consist of pairs of transition systems called *Input* and *Output*, accompanied by a set of ω-regular properties that specify equivalence relation between the two, as described above. The game that we solve is, then, to find a controller that reads the assignments to the variables of the *Input* and produces a valid sequence of assignments to the variables of the *Output* such that all the properties are satisfied. The translation of the transducers to this game structure is rather direct, as elaborated in Sect. 4. The *Input* transition system is generated from the *Target* transducer and the *Output* transition system is generated from the *Adaptee* transducer. This is because we want the *Adapter* , which we generate from the controller as described below, to consider the behavior of the *Target* and to translate it to a command that generates an equivalent behaviour of *Adaptee*. Once we find a controller that solves the game, we can transform it to an *Adapter* as we detail in Sect. 4.

The synthesis problem that we defined so far is as hard computationally as general LTL synthesis and is thus double exponential in the worst case [37]. To cope with this difficulty, we propose to use a well known fragment of LTL called GR(k). GR(k) generalizes the GR(1) subset of LTL [9], a practical fragment of LTL for which a feasible reactive synthesis algorithm exists (see, e.g., [8,28, 33]). Furthermore, GR(k) formulas are known to be highly expressive, as they can encode most commonly appearing LTL industrial patterns [15,29,30] and DBA properties (see related works for details). In addition to using GR(k), since the *Input* and *Output* in our model are separated transition systems, with separated sets of atomic propositions, we focus on properties that separate input and output variables. That is, our specification has the form k <sup>i</sup>=1(φ<sup>i</sup> → ψi), where the <sup>φ</sup><sup>i</sup> and <sup>ψ</sup><sup>i</sup> are conjunctions of LTL GF (Globally in the Future) formulas over *Input* variables only and *Output* variables only respectively. We call this model *Separated GR(k)*. We show through several case-studies that this fragment of LTL suffices to specify a range of useful equivalence relations.

We study the problems of realizability and synthesis on Separated GR(k) game. For that, we first consider a sub-problem of solving a *weak B¨uchi* game. Then we identify and make use of a property of separated games that we call *delay property*: the system can delay its response to the environment indefinitely as long as it remains in the same connected component of the game graph. This allows us to decide the realizability of Separated GR(k) in O(|ϕ| + N) symbolic operations, and to synthesize a controller for a realizable specification in O(|ϕ|N) symbolic operations, where ϕ is the Separated GR(k) specification, and N is the size of the state-space. Thus, Separated GR(k) games are easier to solve that solving GR(k) games which require O(N<sup>k</sup>+1k!) operations [35]. This demonstrates the efficiency of our framework, since |ϕ| tends to be smaller than N and in most practical cases, |ϕ| ∈ O(log(N)).

The benefits of the complexity-theoretic improvement are reflected in empirical evaluations on our case studies of separated GR(k) formulas. We demonstrate that while separated GR(k) formulas are challenging for state-of-the-art synthesis tools, a symbolic BDD-based implementation of our algorithm solves them scalably and efficiently.

The rest of the paper is organized as follows: Sect. 2 introduces necessary preliminaries. Separated GR(k) games are introduced and formulated in Sect. 3. In Sect. 4 we describe how to use Separated GR(k) games synthesis to generate the adapter transducer, and introduce several use-cases. Next, we turn to solving separated GR(k) games. An overview of our solution approach and a necessary property for correctness of algorithm, called the delay property, is given in Sect. 5. A complete symbolic algorithm is presented in Sect. 6. An empirical evaluation on case-studies is presented in Sect. 7. Finally, in Sects. 8 and 9 respectively, we give related work and conclude. Detailed proofs appear in the full version of the paper [3].

#### **2 Preliminaries**

**General Definitions.** Given a set of Boolean variables V, a *state over* V is an assignment s to the variables in V. We describe s as the subset of V that is assigned True in <sup>s</sup>. The set of *primed variables of* <sup>V</sup> is <sup>V</sup> <sup>=</sup> {v <sup>|</sup> <sup>v</sup> ∈ V}. Then s = {v | v ∈ s} is the primed state s over V . An *assertion over* V is a Boolean formula over variables V. A state s satisfies an assertion ρ over the same variables, denoted <sup>s</sup> <sup>|</sup><sup>=</sup> <sup>ρ</sup>, if <sup>ρ</sup> evaluates to True by assigning *true* to the elements of s. We define the *projection* of a state s on a subset U⊆V as denoted by s|<sup>U</sup> = s ∩ U. We extend the notion of projection to a set of states S ⊆ 2<sup>V</sup> by defining S|<sup>U</sup> = {s|<sup>U</sup> | s ∈ S}.

Our specification is a special form of *Linear Temporal Logic (* LTL*)*. LTL [36] extends propositional logic with infinite-horizon temporal operators. The syntax of an LTL formula over a finite set of Boolean variables <sup>V</sup> is defined as follows: <sup>ϕ</sup>:: = <sup>v</sup> ∈V|¬<sup>ϕ</sup> <sup>|</sup> <sup>ϕ</sup> <sup>∧</sup> <sup>ϕ</sup> <sup>|</sup> <sup>ϕ</sup> <sup>∨</sup> <sup>ϕ</sup> <sup>|</sup> X<sup>ϕ</sup> <sup>|</sup> <sup>ϕ</sup>U<sup>ϕ</sup> <sup>|</sup> F<sup>ϕ</sup> <sup>|</sup> Gϕ. Here X (Next), U (Until), F (Eventually), G (Always) are temporal operators. The semantics of LTL can be found in [5, Chapter 5].

We model the adapters as transducers. A *transducer* is a deterministic finitestate machine with no accepting states, but with additional output alphabet and an additional function from the set of states to the output alphabet. A formal definition of a transducer is not required for this paper.

The algorithms developed in this paper are symbolic, i.e. manipulate implicit representations of sets of states. To this end, we use *Binary Decision Diagrams (BDDs)* [10] to represent assertions. For a BDD B and sets of variables <sup>V</sup>1, ···V<sup>n</sup>, we write B(V1,..., <sup>V</sup><sup>n</sup>) to denote that <sup>B</sup> represents an assertion over <sup>V</sup>1∪···∪V<sup>n</sup>. For a state <sup>s</sup> over <sup>V</sup>, we write <sup>s</sup> <sup>|</sup><sup>=</sup> <sup>B</sup>(V) to denote that the assertion that B represents is satisfied by the state s. BDDs support several *symbolic operations*: conjunction (∨), disjunction (∧), negation (¬), and extraction of variables using the ∃ and ∀ operators. We measure time complexity of a symbolic algorithm by a worst case #symbolic-operations it performs. A discussion on a rigorous treatment of BDD operations can be found in the paper's full version [3].

**Game Structures and Games.** We follow the notations of [9]. A game structure *GS* = (I, O, θI, θO, ρI, ρO) defines a turn-based interaction between an *environment* and a *system* players. The input variables I and output variables O are two disjoint sets of Boolean variables that are controlled by the environment and system, respectively. The environment's *initial assumption* θ<sup>I</sup> is an assertion over I, and the system's *initial guarantee* θ<sup>O</sup> is an assertion over I∪O. The environment's *safety assumption* ρ<sup>I</sup> is an assertion over I∪O∪I , where the interpretation of (*i*0, *o*0, *i* <sup>1</sup>) |= ρ<sup>I</sup> is that from state (*i*0, *o*0) the environment can assign *i*<sup>1</sup> to the input variables. W.l.o.g, we assume that ρ<sup>I</sup> is deadlock free, i.e., for all (*i*0, *o*0) there exists an *i*<sup>1</sup> s.t. (*i*0, *o*0, *i* <sup>1</sup>) |= ρI. Similarly, the system's *safety guarantee* ρ<sup>O</sup> is an assertion over I∪O∪I ∪O , where the interpretation of (*i*0, *o*0, *i* <sup>1</sup>, *o* <sup>1</sup>) |= ρ<sup>O</sup> is that from state (*i*0, *o*0) when the environment assigns *i*<sup>1</sup> to the input variables, the system can assign *o*<sup>1</sup> to the output variables. Again, w.l.o.g, we assume that ρ<sup>O</sup> is deadlock free, i.e., for all (*i*0, *o*0, *i* <sup>1</sup>) there exists an *o*<sup>1</sup> s.t. (*i*0, *o*0, *i* <sup>1</sup>, *o* <sup>1</sup>) |= ρO.

A play over *GS* progresses by the players taking turns to assign values to their own variables ad infinitum, where the players must satisfy the initial conditions at the start and the safety conditions thereafter. Formally, a *play* π = s0, s1,... is an infinite sequence of states over I∪O such that s<sup>0</sup> |= θ<sup>I</sup> ∧θ<sup>O</sup> and (s<sup>j</sup> , s <sup>j</sup>+1) |= ρI∧ρ<sup>O</sup> for all j ≥ 0. A *play prefix* is either a play or a finite sequence of states that can be extended to a play. Then a *strategy* is a function <sup>f</sup> : (2I∪O)<sup>+</sup> <sup>×</sup> <sup>2</sup><sup>I</sup> <sup>→</sup> <sup>2</sup><sup>O</sup> such that if s0,...,s<sup>m</sup> is a play prefix, (sm, *i* ) |= ρ<sup>I</sup> and f(s0,...,sm, *i*) = *o*, then (sm, *i* , *o* ) |= ρO. Intuitively, a strategy directs the system on what to assign to the output variables, depending on the history of a play and the most recent assignment by the environment to the input variables. A play prefix is said to be *consistent with a strategy* f if for all states s<sup>j</sup> = (*i*<sup>j</sup> , *o*<sup>j</sup> ) in that prefix, f(s0,...,s<sup>j</sup>−1, *i*<sup>j</sup> ) = *o*<sup>j</sup> for all j ≥ 0. A strategy is memoryless if it only depends on the last state and the most recent assignment to the input variables. Formally, a *memoryless strategy* is a function f : (2I∪O) × 2<sup>I</sup> → 2<sup>O</sup> such that if (sm, *i* ) |= ρ<sup>I</sup> and f(sm, *i* ) = *o*, then (sm, *i* , *o* ) |= ρO.

A *game* is a tuple (*GS*, ϕ) where *GS* is a game structure over inputs I and outputs O and ϕ is an LTL formula over I∪O called a *winning condition*. A play π is *winning* for the system if π |= ϕ. A strategy f *wins from state* s if every play π from s that is consistent with f is winning for the system. A strategy f *wins from* S, where S is an assertion over I∪O, if it wins from every state s |= S. The *winning region* of the system is the set of states from which it has a winning strategy. A strategy f is *winning* if for every state *i* |= θ<sup>I</sup> there exists a state *o* ∈ 2<sup>O</sup> such that (*i*, *o*) |= θ<sup>O</sup> and f wins from (*i*, *o*). In this paper, we have the following games that are defined over the following winning conditions.


Given a game (*GS*, ϕ), *realizability* is the problem of deciding whether a winning strategy for the system exists, and *synthesis* is the problem of constructing a winning strategy if one exists. We note that a realizability check can be reduced to the identification of the winning region, W: A winning strategy exists iff for all *i* |= θ<sup>I</sup> there exists *o* ∈ 2<sup>O</sup> such that (*i*, *o*) |= θ<sup>O</sup> and (*i*, *o*) ∈ W. Hence, the synthesis problem can be solved by constructing a strategy that wins from W.

**Game Graphs and Weak B¨uchi Games.** The *game graph* for a game structure *GS* is the directed graph (V,E) with vertices V = 2I∪O and edges E = {(s, t) | (s, t ) |= ρ<sup>I</sup> ∧ ρO}. Intuitively, vertices are states over I and O, and edges represent valid transitions between states according to the safety conditions. The game graph can be useful for analyzing the structural properties of a game structure via graph-theoretical properties.

<sup>A</sup> *finite path* in a directed graph (V,E) is a sequence <sup>v</sup>0,...,v<sup>n</sup> <sup>∈</sup> <sup>V</sup> <sup>+</sup> such that (v<sup>j</sup> , vj+1) <sup>∈</sup> <sup>E</sup> for all 0 <sup>≤</sup> j<n. An *infinite path* <sup>v</sup>0, v1,... <sup>∈</sup> <sup>V</sup> <sup>ω</sup> is similarly defined. A vertex u is said to be *reachable* from another vertex v if there is a finite path from v to u. A *strongly connected component* (SCC) of a directed graph (V,E) is a maximal set of vertices within which every vertex is reachable from every other vertex. It is well known that SCCs partition the set of vertices of a directed graph, and that the set of SCCs is partially ordered with respect to reachability. Also note that every infinite path ultimately stays in an SCC.

Let (*GS*,GFϕ) be a game with a B¨uchi winning condition, and let *<sup>S</sup>*<sup>0</sup> ..., *<sup>S</sup>*<sup>m</sup> be the set of SCCs that partition the game graph of *GS*. We say that (*GS*,GFϕ) is a *weak B¨uchi game* if, given the set F of states that satisfy the assertion ϕ, for every SCC *S*i, either *S*<sup>i</sup> ⊆ F or *S*<sup>i</sup> ∩ F = ∅. Thus, the SCCs of a weak B¨uchi game are either *accepting components*, meaning all of its states are contained in F, or *non-accepting components*, meaning none of its states is present in F. As a consequence, a play in a weak B¨uchi game is winning for the system if the play ultimately never exits an accepting component. Similarly, a strategy is winning for the system if it can guarantee that every play will ultimately remain inside an accepting component.

#### **3 Separated GR(***k***) Games**

Our framework relies on the core idea of reducing the problem of adapter generation to synthesizing a *Separated GR(*k*) game*, which we define in this section. At a high-level, a separated GR(k) differentiates from a regular GR(k) game in a separation between input and output variables in both the game structure and winning condition. We show in later sections that the separation of variables leads to algorithmic benefits to the synthesis problem. Formally we have the following.

**Definition 1.** *A game structure GS* = (I, O, θI, θO, ρI, ρO) separates variables *over input variables* I *and output variables* O *if:*


The interpretation of a game structure which separates variables is that the underlying game graph (V,E) is the product of two distinct directed graphs over disjoint sets of variables: G<sup>I</sup> over the variables I∪I , and G<sup>O</sup> over the variables O∪O . For J ∈ {I, O}, the vertices of G<sup>J</sup> correspond to states over J and there is an edge between states s and t if (s, t ) |= ρ<sup>J</sup> .

Next, the notion of separation of variables extends to games with GR(k) winning conditions as follows:

**Definition 2.** *A GR(k) winning condition* ϕ *over* I∪O *separates variables w.r.t.* I *and* O *if* ϕ = k <sup>l</sup>=1( n*l* <sup>i</sup>=1 GF(ϕl,i) <sup>→</sup> m*<sup>l</sup>* <sup>j</sup>=1 GF(ψl,j )) *such that each* <sup>ϕ</sup>l,i *is an assertion over* I *and each* ψl,j *is an assertion over* O*.*

A *Separated GR(k) game* is a GR(k) game (*GS*, ϕ) over I∪O in which both *GS* and ϕ separate variables w.r.t. I and O.

A major observation is that in a game played over a separated game structure, the actions of the two players are independent: the environment's actions do no limit the system's actions, and vice versa. In later sections we see how this observation leads to algorithmic improvements in solving separated GR(k) games over a regular GR(k) game. Specifically, in Sect. 4 we see how to use Separated GR(k) games to generate the adapter transducer. In Sects. 5 and 6 we discuss algorithms for realizability and synthesis of Separated GR(k) games.

#### **4 From Transducers to Separated GR(***k***)**

We describe, using an end-to-end-example, how adapter transducer generation can be reduced to synthesis of Separated GR(k) games.

We begin with user-provided *Target* and *Adaptee* transducers. These transducers model the behavior of a system that we want to use (*Adaptee*) and the behavior of a system that we want to emulate (*Target*). For example, the transition systems in Fig. 1 formulates the following scenario. (1) *Target* is an hardware interface that we want to support, such that the U (up) and the D (down) commands send the hardware from mode s<sup>0</sup> to modes s<sup>1</sup> and s2, respectively, from which the S (stay) command keeps the system looping at the chosen mode. (2) *Adaptee* that is a hardware that we can use that also has three modes, but which does not allow the command S after U. Instead, it allows a D command that switches the mode back to s0.

**Fig. 1.** An example of *Target* and *Adaptee* transducers. In this example, the *t<sup>i</sup>* and *a<sup>i</sup>* variables encode the binary representation of the mode being moved to.

The second step is a formulation of the equivalence relation, where we define the type of emulation that we require. In our example we want to maintain the following property: if *Target* visits a mode s<sup>i</sup> infinitely often for a certain input sequence, then so does *Adaptee* ◦ *Adapter* . This can be expressed in LTL as:

$$\bigwedge\_{i=0}^{2} \mathsf{GF}(\mathit{bin\_t}(s\_i)) \to \mathsf{GF}(\mathit{bin\_a}(s\_i))$$

where bint(si) denotes the binary representation of mode s<sup>i</sup> using variables t1, t0, and similarly for bina(si) using variables a1, a0. Note that in this example we cannot just synthesize an adapter that cycles through all modes in *Adaptee* ◦ *Adapter* infinitely often, since the *Adaptee* transducer does not allow that.

As a third step, to generate a separated GR(k) game, we translate the *Target* and *Adaptee* transducers to *Input* and *Output* transition systems as depicted, for example, in Fig. 2. Since *Adaptee* and *Target* are two separate transducers, each with its own structure, it is natural to model these as two separate transition systems on distinct variables. Thus, the transition systems are produced by the well known projection construction that turns an FST into a FSA that accepts the output language of the transducers [32]. Note that in our setting *Target* is translated to *Input* and *Adaptee* is translated to *Output*. This may appear as a role inversion to readers. We propose it because the role of the controller in our setting is to translate the behavior of *Target* to an equivalent behavior of the *Adaptee*.

**Fig. 2.** A direct translation of the *Target* transducer to an *Input* transition system and of the *Adaptee* transducer to an *Output* transition system.

These separate transition systems, together with the specification described above, form a Separated GR(k) that, as a fourth step, we can feed to the Separated GR(k) synthesis algorithm. The output of the algorithm is a transducer called Controller, that maps runs of *Input* to runs of *Output*, as shown, in our example, in Fig. 3. This, in fact, connects the output of the *Target* to the output of the *Adaptee*.

As a final step, from the controller we can construct the *Adapter* using the formula *Adapter* <sup>=</sup> *Adaptee*−<sup>1</sup> ◦ *Controller* ◦ *Target*. This means that *Adapter* contains an internal model of the *Target* and of the *Adaptee*. These internal

$$\begin{array}{c} \begin{array}{c} \neg t\_1 \land \neg t\_0 \mid \neg a\_1 \land \neg a\_0\\ \neg t\_1 \land \neg t\_0 \mid a\_1 \land \neg a\_0\\ \hline \Updownarrow \\ \neg t\_1 \land \neg t\_0 \mid \neg a\_1 \land \neg a\_0 \end{array} \end{array} \xrightarrow{\begin{array}{c} \neg t\_1 \land \neg t\_0 \mid \neg a\_1 \land \neg a\_0\\ \neg t\_1 \land \neg t\_0 \mid \neg a\_1 \land \neg a\_0 \end{array}} \begin{array}{c} \neg t\_1 \land \neg t\_0 \mid \neg a\_1 \land \neg a\_0\\ \hline t\_1 \land \neg t\_0 \mid \neg a\_1 \land \neg a\_0\\ \hline \neg t\_1 \land \neg t\_0 \mid \neg a\_1 \land a\_0\\ \neg t\_1 \land \neg t\_0 \mid \neg a\_1 \land a\_0 \end{array} \qquad \begin{array}{c} \neg t\_1 \; \neg t\_0\\ \hline \neg a\_0\\ \hline \neg a\_1 \land \neg a\_0 \end{array} \end{array}$$

**Fig. 3.** A controller that reads runs of the *Input* transition system and generates runs of the *Output* transition system such that the specified Separated GR(2) formula is guaranteed to be true.

models are used to translate inputs to expected outputs of the adapter, then feed them to the controller, and then feed the output of the controller to the reverse of *Adaptee* to generate an input to *Adaptee* that emulates the behaviour of *Target*. Note that it is possible to invert transducers symbolically [21].

#### **4.1 Additional Usages of Our Technique**

We give two more examples to demonstrate uses of Separated GR(k).

**Cleaning Robots.** This example demonstrates how one can use our technique to fulfill tasks that have not been covered by an execution of an existing transducer. Consider a cleaning robot (the *Target* transducer) that moves along a corridor-shaped house, from room 1 to room n. The robot follows some plan and accordingly cleans some of the rooms. Our goal is to synthesize a controller that activates a second cleaning robot (the *Adaptee* transducer) that follows the first robot and cleans exactly those rooms left uncleaned. Each robot controls a set of variables indicating which room they are in and which rooms they have cleaned, and additionally the original robot controls a variable indicating whether it is done with its cleaning. Our controller is required to fulfill requirements of the form: GF(*done*) <sup>∧</sup> GF(!*in*:*clean*i) <sup>→</sup> GF(*out*:*clean*i), GF(*done*) <sup>∧</sup> GF(*in*:*clean*i) <sup>→</sup> GF(!*out*:*clean*i).

**Railway Signalling.** This example demonstrates how one can use our technique to improve the quality of an existing transducer. We consider a junction of n railways, each equipped with a signal that can be turned on (light in green) or off (light in red). Some railways overlap and thus their signals cannot be turned on simultaneously. We consider an overlapping pattern where railways 1–4 overlap, and similarly 3–6, 5–8, and so on.

An existing system (the *Target* transducer) was programmed to be strictly safe in order to avoid accidents, so it never raises two signals simultaneously. We want to improve the system's performance by synthesizing a controller that reads the assignments that the existing transducer produces and accordingly assign values to the signals in such a way as to produce both safe and *maximal* valuations: the ith signal is turned on if and only if the signal of every rail that overlaps with the ith rail is off. Furthermore, we want to maintain liveness properties of the *Target* system: (1) every signal that is turned on infinitely often by the existing system must be turned on infinitely often by the new system as well, and (2) if a signal is turned on at least once every m steps (where m is a parameter of the specification) by the existing system, then the same holds for the new system.

Note that, in terms of the GR(k) formula, this example is similar to the "hardware" example that we gave; we want to emulate the *Target*'s execution. The crux of the example lies in its *Adaptee*. Here, unlike in the explanatory example, the *Adaptee* is not a given hardware, but rather a virtual component that the user introduced to improve the *Target* performance. In this case the *Adaptee* produces safe and maximal signals.

#### **5 Overview for Solving Separated GR(***k***) Games**

The adapter generation framework described in Sect. 4 relies on synthesizing a controller from a separated GR(k) game. In this section and the next, we describe how to solve separated GR(k) games. This section gives an overview of the algorithm in Sect. 5.1 and describes a necessary property, called the delay property, in Sect. 5.2. The delay property is necessary to prove correctness of our synthesis algorithm. Later, Sect. 6 gives the complete algorithm and proves its correctness.

#### **5.1 Algorithm Overview and Intuition**

Following Sect. 3, we are given a Separated GR(k) game that consists of a game structure *GS* and a winning condition in a GR(k) form ϕ = k <sup>l</sup>=1 ϕl, where ϕ<sup>l</sup> = n*l* <sup>i</sup>=1 GF(al,i) <sup>→</sup> m*<sup>l</sup>* <sup>j</sup>=1 GF(gl,j ). Let <sup>G</sup> be the game graph of *GS*. Consider an infinite play π in *GS*. Like every infinite path on a finite graph, π eventually stabilizes in an SCC S. Due to separation of variables, the game graph G can be decomposed into an input graph G<sup>I</sup> and an output graph GO. Then the projection of S on the inputs is an SCC S<sup>I</sup> in GI, and the projection of S on the outputs is an SCC S<sup>O</sup> in GO. The input side of π converges to S<sup>I</sup> whereas the output side π converges to SO.

Now, let S be an SCC with projections S<sup>I</sup> on G<sup>I</sup> and S<sup>O</sup> on GO. Then we call S *accepting* if for *every* constraint ϕl, where l ∈ {1,...,k}, one of the following holds:

**All guarantees hold in** *S***.** For every j ∈ {1,...,m<sup>l</sup>}, there exists *o* ∈ S<sup>O</sup> such that *o* |= gl,j .

**Some assumption cannot hold in** *S***.** There exists j ∈ {1,...,n<sup>l</sup>} such that for all *i* ∈ SI, *i* |= al,j .

Then from the definition of an accepting SCC we have the following: a strategy that makes sure that every play converges to an accepting SCC, in which all the relevant guarantee states are visited, is a winning strategy for the system in (*GS*, ϕ). To synthesize such a strategy, we do the following: (i) synthesize a strategy f<sup>B</sup> for which every play converges to an accepting SCC; (ii) synthesize a strategy f*travel* that travels within every accepting SCC, satisfying as many of the gl,j guarantees as possible. (iii) construct an overall winning strategy f that works as follows: the system plays f<sup>B</sup> until reaching an accepting SCC S, then the system switches to f*travel* to satisfy as many of the gl,j guarantees in S as possible; if the environment moves the play to a non-accepting SCC, the system can start playing f<sup>B</sup> again to reach a different accepting SCC.

The strategy f<sup>B</sup> can be found by synthesizing the weak B¨uchi game (*GS*,GF(*acc*)), where *acc* is the assertion that accepts exactly those states that belong to accepting SCCs (note that (*GS*,GF(*acc*)) is a well defined weak B¨uchi game). f*travel* can be constructed by simply finding a path in S<sup>O</sup> that satisfies the maximum number of guarantees.

A complication arises however when switching between f*travel* and fB, since it is conceivable that while the system is following f*travel* , the environment could move to a different SCC that is outside of the winning region of fB. Thus, it is not clear that we can combine these strategies to make an overall winning strategy for the system. To show that we can indeed combine both strategies, we need the following property that we call the *delay property*: if (*i*1, *o*1) is a state in the winning region of fB, and (*i*2, *o*0) is a state for which there is a path in G<sup>I</sup> from *i*<sup>1</sup> to *i*<sup>2</sup> and a path in G<sup>O</sup> from *o*<sup>0</sup> to *o*1, then (*i*2, *o*0) is also in the winning region of fB. We formally state and prove the delay property in Sect. 5.2. In Sect. 6 we give details of the construction of fB, f*travel* and the use of the delay property to prove correctness of the overall winning strategy f.

#### **5.2 The Delay Property**

The delay property essentially says that if an SCC S is contained in the winning region, and the environment moves from S unilaterally to a different SCC S , then S is also in the winning region of the system. In this section, we prove that the B¨uchi game (*GS*,GF(*acc*)) where *GS* = (I, <sup>O</sup>, θI, θO, ρI, ρO), as defined in Sect. 5.1, satisfies the delay property. Throughout this section, we write G<sup>I</sup> and G<sup>O</sup> to denote the graphs over 2<sup>I</sup> and 2O, respectively, as in Sect. 5.1. We start with the following lemma that states that the system can still win in spite of a single step delay.

**Lemma 1.** *Let i*0, *i*<sup>1</sup> ∈ 2<sup>I</sup> *such that* (*i*0, *i* <sup>1</sup>) |= ρI*, and assume that the system can win from* (*i*0, *o*0)*. Then the system can also win from* (*i*1, *o*0)*.*

*Proof.* Let f be a winning strategy for the system from (*i*0, *o*0). We construct a winning strategy f<sup>d</sup> from (*i*1, *o*0). Intuitively, f<sup>d</sup> acts from state (*i*1, *o*0) as if it were following f from state (*i*0, *o*0), with a delay of a single step: the input in the current step is used to choose the output in the next step.

We use f to define f<sup>d</sup> inductively over play prefixes of length m ≥ 1, by setting fd((*i*1, *o*0),...,(*i*m, *o*<sup>m</sup>−<sup>1</sup>), *i*<sup>m</sup>+1) = f((*i*0, *o*0),...,(*i*<sup>m</sup>−<sup>1</sup>, *o*<sup>m</sup>−<sup>1</sup>), *i*m). Note that f<sup>d</sup> is well defined since *GS* separates variables: from state (*i*, *o*), the outputs that can be chosen for the successor state depend only on *o*, and not on *i*. Note that by this definition, for every play (*i*1, *o*0),(*i*2, *o*1),...,(*i*m+1, *o*m),... consistent with fd, the play (*i*0, *o*0),(*i*1, *o*1),...,(*i*m, *o*m),... is consistent with f. We remark that we define f<sup>d</sup> only for proving the lemma, and it is *not* part of our solution.

Next, we show that f<sup>d</sup> is winning from (*i*1, *o*0). Take a play (*i*1, *o*0),(*i*2, *o*1),... , consistent with fd. By the construction, (*i*0, *o*0),(*i*1, *o*1),... is consistent with f. Since this is a play on a weak B¨uchi game, after some point it must remain in a single SCC S, say from state (*i*<sup>j</sup> , *o*<sup>j</sup> ). Since f is a winning strategy, the SCC S must be accepting. Then *o*<sup>j</sup> , *o*<sup>j</sup>+1,... is an infinite path in the SCC S|O, and *i*<sup>j</sup> , *i*<sup>j</sup>+1,... is an infinite path in the SCC S|I. Consequently, (*i*1, *<sup>o</sup>*0),(*i*2, *<sup>o</sup>*1),... converges to an SCC <sup>S</sup><sup>ˆ</sup> in which <sup>S</sup>ˆ|<sup>I</sup> <sup>=</sup> <sup>S</sup>|<sup>I</sup> and <sup>S</sup>ˆ|<sup>O</sup> <sup>=</sup> <sup>S</sup>|O. Since the conditions for an SCC D to be accepting depend only on the relation between <sup>D</sup>|<sup>I</sup> and <sup>D</sup>|O, we have that <sup>S</sup><sup>ˆ</sup> is accepting since <sup>S</sup> is accepting as well. 

We can now prove the delay property, following by straightforward induction from Lemma 1.

**Theorem 1 (Delay Property Theorem).** *Let i*0,..., *<sup>i</sup>*<sup>n</sup> <sup>∈</sup> (2I)<sup>+</sup> *be a path in* <sup>G</sup>I*, and for* <sup>m</sup> <sup>≥</sup> <sup>0</sup>*, let o*−<sup>m</sup>,..., *<sup>o</sup>*<sup>0</sup> <sup>∈</sup> (2O)<sup>+</sup> *be a path in* <sup>G</sup>O*. Assume that the system can win from* (*i*0, *o*0)*. Then the system can also win from* (*i*n, *o*−<sup>m</sup>)*.*

*Proof.* From (*i*n, *o*−<sup>m</sup>), the system can simply ignore the inputs and follow the path in G<sup>O</sup> to *o*0. Let (*i*<sup>n</sup>+<sup>m</sup>, *o*0) be the state at that point in some play. Note that there is a path between *i*<sup>n</sup> and *i*<sup>n</sup>+<sup>m</sup>, and therefore there is a path between *i*<sup>0</sup> and *i*<sup>n</sup>+<sup>m</sup>. If the system can win from (*i*0, *o*0) then by using Lemma 1 in the induction steps, the system can win by induction from (*i*, *o*0) for all *i* such that there is a path in between *i*<sup>0</sup> and *i*. Therefore, the system can win from (*i*<sup>n</sup>+<sup>m</sup>, *o*0), and by consequence from (*i*n, *o*−<sup>m</sup>).

A corollary of Theorem 1 is the following statement about the structure of the winning region of the weak B¨uchi game <sup>B</sup> = (*GS*,GF(*acc*)) as defined in Sect. 5.1.

#### **Corollary 1.** *The winning region of* B *is a union of SCCs.*

*Proof.* Let (*i*, *o*) be a state in the winning region of B, let (ˆ*i*, *o*ˆ) be a state in the same SCC S of (*i*, *o*), and let S|<sup>I</sup> and S|<sup>O</sup> be the projections of S on G<sup>I</sup> and GO, respectively. Then there is a path *i*0,..., *i*<sup>n</sup> for some n ≥ 0 in S|<sup>I</sup> such that *<sup>i</sup>*<sup>0</sup> <sup>=</sup> *<sup>i</sup>* and <sup>ˆ</sup>*<sup>i</sup>* <sup>=</sup> *<sup>i</sup>*n. Similarly, there is a path *<sup>o</sup>*−<sup>m</sup>,..., *<sup>o</sup>*<sup>0</sup> for some <sup>m</sup> <sup>≥</sup> 0 in S|<sup>O</sup> such that ˆ*o*<sup>0</sup> = *o* and ˆ*o* = *o*−<sup>m</sup>. Then by the delay property of Theorem 1, the vertex (ˆ*i*, *<sup>o</sup>*ˆ)=(*i*n, *<sup>o</sup>*−<sup>m</sup>) is also in the winning region of <sup>B</sup>.

We use Theorem 1 and Corollary 1 in the proof of correctness of the overall winning strategy f, as described in Sect. 6.2.

### **6 Algorithms for Solving Separated GR(***k***) Games**

In this section we provide the exact details of our synthesis algorithm for Separated GR(k) games, as described in Sect. 5.1. Since constructing f<sup>B</sup> involves defining and solving a weak B¨uchi game, we first describe these in Sect. 6.1. We remark that our weak B¨uchi game synthesis algorithm works for all weak B¨uchi games, and not just for the special weak B¨uchi game defined in Sect. 5.1. Specifically, it works even when the underlying game structure does not separates variables. Next, in Sect. 6.2, we complete the algorithm construction and describe the correctness of our overall synthesis algorithm.

#### **6.1 Realizability and Synthesis for Weak B¨uchi Games**

We present a symbolic algorithm to solve synthesis of a weak B¨uchi game. When represented in explicit state-representation, weak B¨uchi games are known to be solved in linear-time in the size of the game [12,27]. In this section, we adapt the algorithm from [12,27] to symbolic state-space representation. For sake of exposition, we give an overview of the algorithm and then present our symbolic modification.

**Overview** Given a weak B¨uchi game, recall that each SCC in its game graph G is either an accepting SCC or a non-accepting SCC. The goal is to find the winning regions in the weak B¨uchi game. This can be done by backward induction on the topological ordering of the SCCs as follows. Let (*S*0,... *S*m) be a topological sort of the SCCs in G.

**Base Case:** Consider all *terminal partitions*, say *S*<sup>j</sup> ,..., *S*m; that is, every SCC from which no other SCC is reachable. In this case, plays beginning in a terminal SCC will never leave it. Therefore, all states of terminal SCCs that are accepting are in the winning region of the system and all states of terminal SCCs that are non-accepting are not in the winning region of the environment.

**Induction Step:** Let −→<sup>S</sup> = (S<sup>i</sup>+1,...,Sm), and suppose that the set −→<sup>S</sup> has been classified into winning regions for the system W<sup>s</sup> <sup>i</sup>+1 and the environment W<sup>e</sup> <sup>i</sup>+1, respectively. Let −→<sup>S</sup> *new* = (*S*<sup>j</sup> , *<sup>S</sup>*<sup>j</sup>+1,..., *<sup>S</sup>*i) be the SCCs from which all edges leaving the SCC lead to an SCC in −→S . Further, let A and N be the unions of all accepting SCCs and all non-accepting SCCs in −→S new, respectively. Then the basic idea is as follows: The system can win from s ∈ N if and only if it can force F(W<sup>s</sup> <sup>i</sup>+1) from s. Analogously, the system can win from s ∈ A if and only if it can force G(<sup>A</sup> <sup>∪</sup> <sup>W</sup><sup>s</sup> <sup>i</sup>+1) from s. Hence, by solving these reachability and safety games, we can update W<sup>s</sup> <sup>i</sup>+1 and <sup>W</sup><sup>e</sup> <sup>i</sup>+1 into <sup>W</sup><sup>s</sup> <sup>j</sup> and W<sup>s</sup> <sup>j</sup> that partition the larger set (S<sup>j</sup> ,...,Sm) into winning regions for the system and the environment. The winning strategy can be constructed in a standard way as a side-product of the reachability and safety games in each step, see for example [40,41].

**Symbolic Algorithm for Weak B¨uchi Games.** Given a weak B¨uchi game <sup>B</sup> = ((I, <sup>O</sup>, θI, θO, ρI, ρO),GF(*acc*)) with BDDs representing <sup>θ</sup>I, <sup>θ</sup>O, <sup>ρ</sup>I, <sup>ρ</sup><sup>O</sup> and *acc*, our goal is to compute a BDD for the winning region and to synthesize a memoryless winning strategy for the system. The construction follows a fixed-point computation that adapts the inductive procedure described in the overview: In the basis of the fixed point computation, the winning region is the set of accepting terminal SCCs; in the inductive step, the winning region includes winning states by examining SCCs that are higher in the topological ordering on SCCs. In what follows we describe a sequence of BDDs that we construct towards constructing the overall BDD for the winning region. We use the notation *X* to denote a set of variables over I∪O. For the sake of the current construction, memoryless strategies are given in the form of BDDs over *X* , *X* , for further details on the BDDs constructions see the full version for details [3].

*BDD constructions.* We start by constructing a BDD for a predicate that indicates whether two states in a game structure are present in the same SCC. Let predicate Reach(s, t ) hold if there is a path from state s over I∪O to state t over I∪O in the game structure *GS*. Similarly, a predicate Reach<sup>−</sup><sup>1</sup>(s, t ) holds if and only if Reach(t, s ) holds. BDDs for Reach and Reach<sup>−</sup><sup>1</sup> can be computed in O(N) symbolic operations using the transition relation of the game structure. Then, a BDD indicating if two states share the same SCC, is constructed in <sup>O</sup>(N) symbolic operations by SCC(*<sup>X</sup>* , *<sup>X</sup>* ) := Reach(*<sup>X</sup>* , *<sup>X</sup>* ) <sup>∧</sup> Reach<sup>−</sup><sup>1</sup>(*<sup>X</sup>* , *<sup>X</sup>* ).

Next, we construct a BDD for the union of the terminal SCCs, required by the basis of induction for the construction of the winning region. Let predicate Terminal(s) hold if state <sup>s</sup> over I∪O is present in a terminal SCC. Then Terminal(*<sup>X</sup>* ) := <sup>∀</sup>*<sup>X</sup>* : Reach(*<sup>X</sup>* , *<sup>X</sup>* ) <sup>→</sup> SCC(*<sup>X</sup>* , *<sup>X</sup>* ). Therefore, given BDDs for Reach and SCC, the construction of Terminal requires <sup>O</sup>(1) symbolic operations.

*Computing the Winning Region.* We now describe the fixed-point computation to construct a BDD for the winning region in a weak B¨uchi game. Let Reachability(M,N)(*<sup>X</sup>* ) denote a BDD generated by solving a reachability game that takes as input a set of source states M and target states N and outputs those states in M from which the system can guarantee to move into N. Similarly, let Safety(M,N)(*<sup>X</sup>* ) denote a BDD generated by solving a safety game that takes as input a set of source states M and target states N and outputs those states in M from which the system can guarantee that all plays remain inside the set N. These constructions are standard, details can be found in [20, Chapter 2].

Now, let Win(s) denote that state <sup>s</sup> over I∪O is in the winning region. Then, Win(*<sup>X</sup>* ) is the fixed point of the BDD Win Aux defined below, where the construction essentially follows the high-level algorithm description. The BDD Acc(*<sup>X</sup>* ) represents the formula acc encoding the set of accepting states. In addition, DC<sup>i</sup> (*X* ) is the union −→S of the Downward-Closed set of SCCs, i.e. the SCCs that have already been classified into winning or not-winning, and DC<sup>i</sup> new(*<sup>X</sup>* ) is the union −→<sup>S</sup> new of the SCCs in DC<sup>i</sup> (*X* ) that were not in DC<sup>i</sup>−<sup>1</sup>(*<sup>X</sup>* ). Finally, N<sup>i</sup> (*<sup>X</sup>* ) is the subset <sup>N</sup> of non-accepting states in DC<sup>i</sup> new(*X* ), and A<sup>i</sup> (*<sup>X</sup>* ) is the subset <sup>A</sup> of accepting states in DC<sup>i</sup> new(*X* ). We then define Win Aux as follows.

**Base Case.**


#### **Inductive Step.**


$$\begin{array}{llll} \texttt{4:} & \mathsf{A}^{i+1}(\check{X}) := \mathsf{DC}\_{new}^{i+1}(\check{X}) \wedge \mathsf{Acc}(\check{X})\\ \texttt{5:} & \mathsf{Win.Aux}^{i+1}(\check{X}) := \mathsf{Win.Aux}^{i}(\check{X}) \vee \mathsf{Recachability}\_{(\mathsf{N}^{i+1}(X), \mathsf{Win.Aux}^{i}(\check{X}))}(X) \\ & & \mathsf{\vee} \mathsf{Safety}\_{(\mathsf{A}^{i+1}(X), \mathsf{A}^{i+1}(X)) \vee \mathsf{Win.Aux}^{i}(\check{X})}(X) \\ & & \mathsf{\vee} \mathsf{A}^{i} : \mathsf{A}^{i} : \mathsf{A}^{i} : \mathsf{A}^{i} : \mathsf{A}^{i} : \mathsf{A}^{i} : \mathsf{A}^{i} : \mathsf{C}^{i+1} \,\mathsf{A}^{i} \,\mathsf{X} \end{array}$$

To explain the construction of Win, note that a state <sup>s</sup> in DC<sup>i</sup>+1(*<sup>X</sup>* ) is winning in one of these cases: (i) <sup>s</sup> is a winning state in DC<sup>i</sup> (*X* ). (ii) s is a non-accepting state in DC<sup>i</sup>+1(*<sup>X</sup>* ) from which the system can force the play into a winning state in DC<sup>i</sup> (*X* ). This set of states can be obtained from Reachability(N*i*+1(X),Win Aux*i*(*<sup>X</sup>* ))(X). (iii) <sup>s</sup> is an accepting state in DC<sup>i</sup>+1(*<sup>X</sup>* ) from which the system can guarantee that every play that leaves the accepting SCC moves into a winning state in DC<sup>i</sup> (*X* ). This set of states can be obtained from Safety(A*i*+1(X),A*i*+1(X)∨Win Aux*i*(*<sup>X</sup>* ))(X).

Finally, to check realizability, construct the BDD ∀I(InitIn(I) <sup>→</sup> ∃O(InitOut(O) <sup>∧</sup> Win(I∪O))), where InitIn(I) and InitOut(O) are BDDs representing θ<sup>I</sup> and θO, respectively. This BDD is equal to true iff B is realizable.

The fixed-point computation can be extended in a standard way to also compute a BDD representation Fb(X, X ) of the winning strategy fB, such that (s,(i , o )) <sup>|</sup><sup>=</sup> Fb(X, X ) iff fB(s, i) = o, as we elaborate in the full version [3]. We then have the following theorem that follows our construction.

**Theorem 2.** *Realizability and synthesis for weak B¨uchi games can be done in* O(N) *symbolic steps.*

*Proof Outline.* The proposed construction symbolically implements the inductive procedure of the explicit algorithm. Hence, it correctly identifies the system's winning region. It remains to show that the algorithm performs O(N) symbolic operations. First of all, the constructions of SCC and Terminal take <sup>O</sup>(N) symbolic operations collectively. It suffices to show that in the i-th induction step, solving the reachability and safety games performs <sup>O</sup>(|DC<sup>i</sup>+1 \ DC<sup>i</sup> |) operations. This can be proven by a careful analysis of the operations and the sizes of resulting BDDs using standard results on safety and reachability games.

#### **6.2 Realizability and Synthesis for Separated GR(***k***) Games**

We finally make use of the elements obtained so far towards solving synthesis for Separated GR(k) games. Our construction follows the overview from Sect. 5.1. To recall, we describe and construct two auxiliary strategies f<sup>B</sup> and f*travel* and combine them to generate the final strategy f. We use the delay property theorem from Sect. 5.2 to prove the correctness of our algorithm.

We are given a Separated GR(k) game structure *GS* = (I, O, θI, θO, ρI, ρO) and a winning condition ϕ = k <sup>l</sup>=1 ϕl, where ϕ<sup>l</sup> = n*l* - <sup>i</sup>=1 GF(al,i) <sup>→</sup> <sup>m</sup>*<sup>l</sup>* <sup>j</sup>=1 GF(gl,j )). We first represent *GS* and <sup>ϕ</sup> as BDDs by standard means. We then define and construct the following.

*Constructing* fB. Auxiliary strategy f<sup>B</sup> is the winning strategy of the system player in a weak B¨uchi game constructed form the separated GR(k) game. To construct a weak B¨uchi game, we first construct, in O(|ϕ| + N) symbolic operations, a BDD Acc(I∪O) that describes the set of accepting states. The construction is standard. Next, let acc be the assertion represented by Acc (the assertion defined in Sect. 5.1). Then the weak B¨uchi game is <sup>B</sup> = (*GS*,GF(*acc*)). Finally, we construct f<sup>B</sup> as the winning strategy of B, following Sect. 6.1.

*Constructing* f*travel* . For the construction of f*travel* , we arbitrarily order all guarantees that appear in our GR(k) formula: *gar* <sup>0</sup>,..., *gar*<sup>m</sup>−<sup>1</sup>. For each guarantee *gar* <sup>j</sup> , we construct a reachability strategy f<sup>r</sup>(j) that, when applied inside an SCC S<sup>O</sup> in the output game graph GO, moves towards a state that satisfies *gar* <sup>j</sup> without ever leaving SO. In case no such state exists in SO, f<sup>r</sup>(j) returns a distinguished value ⊥. Note that this strategy can entirely ignore the inputs. We equip f*travel* with a memory variable *mem* that stores values from {0,...,m−1}. Then f*travel*(s, i) is operated as follows: for *mem*, *mem* +1,... we find the first *mem* +j (mod m) such that the SCC of s includes a *gar* <sup>j</sup> -state, and activate f<sup>r</sup>(*mem*+j) to reach such state. If no guarantees can be satisfied in S, we just return an arbitrary output to stay in SO. The construction of f*travel* requires O(|ϕ|N) symbolic BDD-operations as we need to construct m reachability strategies (clearly, m ≤ |ϕ|).

*Constructing the overall strategy f.* Finally, we interleave the strategies f<sup>B</sup> and f*travel* into a single strategy f as follows: given a state s and an input i, if <sup>s</sup> <sup>|</sup><sup>=</sup> Acc(X) (that is, if <sup>s</sup> is an accepting state), then set <sup>f</sup>(s, i) = <sup>f</sup>*travel*(s, i); otherwise set f(s, i) = fB(s, i). Whenever f switches from f<sup>B</sup> to f*travel* , the memory variable *mem* is reset to 0. The next lemma proves that if f<sup>B</sup> is winning then so is f.

**Lemma 2.** *If* f<sup>B</sup> *is a winning strategy for the weak B¨uchi game* B = (*GS*,GF(*acc*))*, then* <sup>f</sup> *is a winning strategy for the Separated GR(*k*) game* (*GS*, ϕ)*.*

*Proof.* Since f<sup>B</sup> is a winning strategy, then for every initial input *i* |= θ<sup>I</sup> there is an initial output *o* |= θ<sup>O</sup> such that (i, o) is in the winning region of GS. We show that playing f always keeps the play in the winning region of GS, and therefore the play eventually converges to an accepting SCC. Once this happens, following f*travel* guarantees that ϕ is satisfied. We know that as long as the play is in the winning region of B, following f<sup>B</sup> will keep it inside the winning region. Therefore, when we switch from f<sup>B</sup> to f*travel* we must be inside the winning region and, by definition of f, in some accepting SCC S. Then f*travel* makes sure that as long as the environment remains in S|I, the projection of S over the inputs, the system remains in S|O, the projection of S over the output. Thus all in all the play remains in the winning region of S.

Therefore, the only way that the play can leave the winning region is if, when the system is in a state (*i*0, *o*0) and chooses some output *o*−<sup>m</sup> according to f*travel* , the environment chooses input *i*<sup>n</sup> such that the play leaves S and moves to a state (*i*n, *o*−m) in a different SCC of G. Note, however, that in this case there is a path from *i*<sup>0</sup> to *i*<sup>n</sup> and a path from *o*−<sup>m</sup> to *o*<sup>0</sup> (since by construction f*travel* remains in the same SCC in GO). Since (*i*0, *o*0) is in the winning region, by Theorem 1 we have that (*i*n, *o*−<sup>m</sup>) is in the winning region as well.

**Final Results.** Given Lemma 2, we can obtain our final results on synthesis and realizability of Separated GR(k) games, as follows. Given a Separated GR(k) game (*GS*, ϕ), construct *acc* and solve the weak B¨uchi game (*GS*,GF(*acc*)). Then construct fB, ftravel and f as described above. If realizable, then f<sup>B</sup> is a winning strategy and from Lemma 2 we have that f is a winning strategy for (*GS*, ϕ). If (*GS*,GF(*acc*)) is unrealizable, then the environment can force every play to converge to a non-accepting SCC. Since the GR(k) winning condition cannot be satisfied from a non-accepting SCC, (*GS*, ϕ) is also not realizable. Thus we have the following theorem, see [3] for full details.

**Theorem 3.** *Realizability for separated GR(*k*) games can be reduced to realizability of weak B¨uchi games.*

The final result on solving Separated GR(k) games is then as follows, see [3] for full details.

**Theorem 4.** *Let* (*GS*, ϕ) *be a separated GR(*k*) game over the input/output variables* I *and* O*, respectively. Then, the realizability and synthesis problems for* (*GS*, ϕ) *are solved in* O(|ϕ| + N) *and* O(|ϕ|N) *symbolic operations, respectively, where* N = |2I∪O|*.*

*Proof Outline.* Realizability and synthesis follow Lemma 2 and Theorem 3. It is left to analyze the number of symbolic operations for constructing f<sup>B</sup> and then f. In symbolic operations, constructing *acc* takes O(|ϕ| + N), and computing the winning region <sup>W</sup> for (*GS*,GF(*acc*)) takes <sup>O</sup>(N). Checking realizability can be done by checking if for every initial input i there is an initial output o such that (i, o) ∈ W, which takes O(1). The winning strategy f<sup>B</sup> can be computed in the process of computing W, taking the same number of operations (see [3] for details). Finally, constructing f*travel* takes O((#*gars*)N) ≤ O(|ϕ|N), where *gars* are all guarantees GF(gi,) that appear in <sup>ϕ</sup>. Therefore, constructing <sup>f</sup> takes O(|ϕ|N) symbolic operations in total.

Note that this result is an improvement over the complexity of synthesizing GR(k) games in general [35].

#### **7 Implementation and Evaluation**

We have implemented our Separated GR(k) framework for realizability and synthesis in a prototype tool SGR(k). The tool implements our symbolic algorithm using the CUDD [39] package for BDD manipulation. Our tool is evaluated on a suite of benchmarks created from the examples described in Sect. 4.

**Benchmark Suite.** We have created a suite of parametric benchmarks from the three examples described in Sect. 4. Our suite consists of 38 realizable specifications. The parametric versions of the examples are described here.

The *multi-mode hardware* example is a generalization of the example presented at the beginning of Sect. 4. It is parameterized by the number of bits n and has 2<sup>n</sup> modes. The *Target* can move from mode 0 to any mode and stay there, while the *Adaptee* can only move from mode 0 to odd-numbered modes, and up and down between modes 2i and 2i + 1. The specification consists of 2n variables. We generate 10 such benchmarks with n ∈ {1,..., 10}.

The *cleaning robots* example is parameterized in the number of rooms. For a scenario with n rooms, the specification is written over 4n + 1 variables. We create 10 such benchmarks with n ∈ {1 ..., 10}.

The *railways signalling* example consists of two parameters: a junction of n railways and the frequency parameter m. With parameters n and m, the specification consists of (2 + 2log m)n variables. We generate 18 benchmarks with n ∈ {2,..., 10} and m ∈ {2, 3}.

**Experimental Setup and Methodology.** We evaluate our tool against Strix [1,31], the current state-of-the-art tool for LTL synthesis and SYNTCOMP 2020 winner of 3 out of 4 tracks [2]. In order to run our benchmarks on Strix, we transform the benchmarks (a game structure and a winning condition) into an LTL formula that characterizes the same winning plays using the strict semantics from [22]. To the best of our knowledge, there is no other synthesis/realizability tool that operates on GR(k) specifications.

We compare the running time for checking realizability. For this, we compare the running time of realizability checks of each benchmark on both tools. Every benchmark is tested 10 times on both tools. We do this to account for the randomness introduced during BDD construction due to the automatic variable ordering by CUDD. For each benchmark we evaluate (a) the number of executions on which the tools terminate and (b) the mean running time over 10 executions.

All experiments were executed on a single node of a high-performance computer cluster consisting of an Intel Xeon processor running at 2.6 GHz with 32 GB of memory with a timeout of 10 mins.

**Observations and Inferences.** Our experiments clearly demonstrate the scalability and efficiency of our tool in solving Separated GR(k) formulas.

Figure 4 plots the mean running time for the three benchmarks. We further report the mean values in Table 1. The table rows refer to the benchmarks we

**Fig. 4.** Mean running time for different classes of benchmarks.

examine, and the columns refer to the value of the parameter n. As an example, for the specification Cleaning(3), SGR(k)'s mean running time is 0.07 s. (row titled Cleaning(n); SGR(k), column titled 3) and Strix's mean realizability check running time is 58.3 s. (row titled Cleaning(n);Strix), column titled 4). Cells reading 'TO' indicate experiments reached a timeout.

The results show that our tool solves a significantly larger number of benchmarks than Strix. On the few benchmarks which Strix solves, our tool outperforms


**Table 1.** Mean realizability check running times (sec.)

it by several orders of magnitude. Although the running time may vary depending on the automatic variable ordering chosen by CUDD, we do not believe it would vary enough to significantly change the results. Specifically, we calculated the 99% confidence interval for our results, and validated that for all data points our tool's entire interval lies below the entire interval for Strix.

Only three benchmarks were unsolvable by our tool (in the sense that the majority of the 10 executions timed out). The three benchmarks are the railway signal examples with (n = 10, m = 2), (n = 9, m = 3), and (n = 10, m = 3). These benchmarks consist of a large number of variables (54, 40, and 60, respectively), making them particularly challenging. All executions of the remaining benchmarks were solved in less than 4 mins by our tool.

We also examined the number of solved executions per benchmark. Our tool solved all 10 executions for 35 out of 38 benchmarks. These are the 35 benchmarks that appear as solved in Fig. 4. For the railway signalling benchmark with (<sup>n</sup> = 10, m = 2), our tool solved 2 out of 10 executions. In contrast, Strix was not able to solve even one execution for 31 out of 38 benchmarks. Even increasing the timeout to 8hrs only allowed Strix to solve a single additional benchmark. In total, Strix and our tool verified realizability of 7 benchmarks and 36 out of 38 benchmarks, respectively. In summary, our experiments demonstrate that our tool is able to solve specifications which are challenging for existing tools.

#### **8 Related Work**

The Adapter design pattern was introduced in [18], and has been used in many software contexts since. Our interpretation of the pattern is inspired by automata based description of the pattern proposed by Pedrazzini [34]. We reformulated the problem as synthesis of reactive controllers that compose with existing systems to achieve a temporal specification, e.g. [7,13,17]. Note that our work differs from such frameworks in its variables separation feature. A work with a concept similar to adapting behaviors is the *Shield synthesis* that studies the problem in which a synthesized controller corrects safety violations of an existing controller [24]. Note that in contrast, our problem is mostly concerned about liveness adaptation.

Reactive synthesis of LTL winning conditions is 2EXPTIME complete in the size of the formula [37], making it difficult to scale for applications. An approach to overcome the computational barrier has been to investigate fragments and variants of LTL with lower complexity for synthesis [4,14,16]. One such fragment is GR(k) [9], that offers a balance between efficiency and expressiveness. Specifically, GR(k) games are known to be efficient as they are solved in exponential time in the number of conjunctions k rather than exponential in the state-space [35]. Several studies have also shown that GR(k) specifications are highly expressive. As evidence, all properties expressed by deterministic B¨uchi automata (DBA) can be expressed in GR(k) [16], where a study of commonly appearing LTL patterns has shown that 52 of 55 patterns are DBA properties [15,29]. DBA properties have also been identified as common patterns in robotics applications [30].

Finally, Separated GR(k) games exhibit the *delay property*, which intuitively means that the system can win even after delaying its action for a finite amount of time while ignoring the environment before "catching up" with the environment. While this is reminiscent of asynchrony in reactive systems [6,38], a further exploration of relations between asynchrony and the delay property is required.

#### **9 Conclusion**

This paper presents a reactive systems-based model of the adapter design pattern. We model the adapters as transducers and reduce the problem of finding an *Adapter* transducer for a given *Adaptee* and *Target* systems, to the problem of synthesizing strategies for Separated GR(k) games. Through an analysis of theoretical complexity and algorithmic performance, we show that realizability and synthesis of Separated GR(k) games is efficient and scalable. Furthermore, by outperforming Strix, an existing state-of-the-art synthesis tool, we show that algorithms for the Separated GR(k) class of specifications add value to the portfolio of reactive synthesis tools.

The benefits of separation of input and output variables were previously shown in the context of Boolean Functional Synthesis [11]. Through this work, we showed that separation also leads to practically viable solutions in temporal reactive synthesis, specifically when encoding the types of equivalence relations that appear in reactive adaptation (where properties of runs of the first system are compared to properties of runs of the other). Since the systems may be loosely coupled, i.e., they may not run on the same clock, specifications that impose joint temporal constraints on the two systems may not be realizable. Thus, our proposition to use the type of equivalence that separated GR(k) formulas allow, gives users the power needed for comparing the *overall behaviors* of the systems while allowing realizability and efficient synthesis.

The results presented in this paper encourage future studies on the separation of variables in a broader context. For instance, reason about variants of the adapter design pattern that do not separate variables all the way through. That is to say, variants that translate to more general GR(k) specifications in which the separation appears in the input and output systems but not in the specification itself. One could further study the notion of separation of variables in more the general LTL specifications. Another direction is to consider systems that gets two types of input: from the input system (i.e. the *Target*) as well as from an environment. We believe that these future directions would enable the development of tools for synthesis from temporal specifications with a focus on expressing practical applications as well as ensuring scalability and efficiency.

**Acknowledgements.** We thank Supratik Chakraborty and Dana Fisman for useful comments. Work is supported in part by NSF grant 2030859 (CRA's CIFellows Project), NSF grants IIS-1527668, CCF-1704883, IIS-1830549, an award from the Maryland Procurement Office, ISF grant 2714/19, and by the Lynn and William Frankel Center for Computer Science.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Causality-Based Game Solving**

Christel Baier<sup>1</sup> , Norine Coenen<sup>2</sup> , Bernd Finkbeiner<sup>2</sup> , Florian Funke<sup>1</sup> , Simon Jantsch<sup>1</sup> , and Julian Siber2(B)

> <sup>1</sup> Technische Universit¨at Dresden, Dresden, Germany {christel.baier,florian.funke,

simon.jantsch}@tu-dresden.de <sup>2</sup> CISPA Helmholtz Center for Information Security, Saarbr¨ucken, Germany {norine.coenen,finkbeiner,julian.siber}@cispa.de

**Abstract.** We present a causality-based algorithm for solving twoplayer reachability games represented by logical constraints. These games are a useful formalism to model a wide array of problems arising, e.g., in program synthesis. Our technique for solving these games is based on the notion of *subgoals*, which are slices of the game that the reachability player necessarily needs to pass through in order to reach the goal. We use Craig interpolation to identify these necessary sets of moves and recursively slice the game along these subgoals. Our approach allows us to infer winning strategies that are structured along the subgoals. If the game is won by the reachability player, this is a strategy that progresses through the subgoals towards the final goal; if the game is won by the safety player, it is a permissive strategy that completely avoids a single subgoal. We evaluate our prototype implementation on a range of different games. On multiple benchmark families, our prototype scales dramatically better than previously available tools.

#### **1 Introduction**

Two-player games are a fundamental model in logic and verification due to their connection to a wide range of topics such as decision procedures, synthesis and control [1,2,6,7,11,21]. Algorithmic techniques for *finite-state* two-player games have been studied extensively for many acceptance conditions [20]. For *infinitestate* games most problems are directly undecidable. However, infinite state spaces occur naturally in domains like software synthesis [34] and cyber-physical systems [23], and hence handling such games is of great interest. An elegant classification of infinite-state games that can be algorithmically handled, depending

This work was partially supported by DFG grant 389792660 as part of TRR 248 – CPEC, see https://perspicuous-computing.science, the Cluster of Excellence EXC 2050/1 (CeTI, project ID 390696704, as part of Germany's Excellence Strategy), DFGprojects BA-1679/11-1 and BA-1679/12-1, the Research Training Group QuantLA (GRK 1763), and by the European Research Council (ERC) Grant OSARES (No. 683300).

on the acceptance condition of the game, was given in [14]. The authors assume a symbolic encoding of the game in a very general form. More recently, incomplete procedures for solving infinite-state two-player games specified using logical constraints were studied [4,18]. While [4] is based on automated theorem-proving for Horn formulas and handles a wide class of acceptance conditions, the work in [18] focusses on reachability games specified in the theory of linear arithmetic, and uses sophisticated decision procedures for that theory.

In this paper, we present a novel technique for solving logically represented reachability games based on the notion of *subgoals*. A *necessary* subgoal is a transition predicate that is satisfied at least once on every play that reaches the overall goal. It represents an intermediate target that the reachability player must reach in order to win. Subgoals open up game solving to the study of causeeffect relationships in the form of counterfactual reasoning [28]: If a cause (the subgoal) had not occurred, then the effect (reaching the goal) would not have happened. Thus for the safety player, a necessary subgoal provides a chance to win the game based on local information: If they control all states satisfying the pre-condition of the subgoal, then any strategy that in these states picks a transition outside of the subgoal is winning. Finding such a necessary subgoal may let us conclude that the safety player wins without ever having to unroll the transition relation.

On the other hand, passing through a necessary subgoal is in general not enough for the reachability player to win. We call a subgoal *sufficient* if indeed the reachability player has a winning strategy from every state satisfying the post-condition of the subgoal. Dual to the description in the preceding paragraph, sufficient subgoals provide a chance for the reachability player to win the global game as they must merely reach this intermediate target. The two properties differ in one key aspect: While necessity of a subgoal only considers the paths of the game arena, for sufficiency the game structure is crucial.

We show how Craig interpolants can be used to compute necessary subgoals, making our methods applicable to games represented by any logic that supports interpolation. In contrast, determining whether a subgoal is sufficient requires a partial solution of the given game. This motivates the following recursive approach. We slice the game along a necessary subgoal into two parts, the pre-game and the post-game. In order to guarantee these games to be smaller, we solve the post-game under the assumption that the considered subgoal was bridged *for the last time*. We conclude that the safety player wins the overall game if they can avoid all initial states of the post-game that are winning for the reachability player. Otherwise, the pre-game is solved subject to the winning condition given by the sufficient subgoal consisting of these states. This approach does not only determine which player wins from each initial state, but also computes symbolically represented winning strategies with a causal structure. Winning safety player strategies induce necessary subgoals that the reachability player cannot pass, which constitutes a cause for their loss. Winning reachability player strategies represent a sequence of sufficient subgoals that will be passed, providing an explanation for the win. All missing proofs for our theoretical results can be found in the full version of this paper [3].

The Python-based implementation CabPy of our approach was used to compare its performance to SimSynth [18], which is, to the best of our knowledge, the only other available tool for solving linear arithmetic reachability games. Our experiments demonstrate that our algorithm is competitive in many case studies. We can also confirm the expectation that our approach heavily benefits from qualitatively expressive Craig interpolants. It is noteworthy that like Sim-Synth our approach is fully automated and does not require any input in the form of hints or templates. Our contributions are summarized as follows:


**Related Work.** The problem of solving linear arithmetic games is addressed in [18] using an approach that relies on a dedicated decision procedure for quantified linear arithmetic formulas, together with a method to generalize safety strategies from truncated versions of the game that end after a prescribed number of rounds. Other approaches for solving infinite-state games include deductive methods that compute the winning regions of both players using proof rules [4], predicate abstraction where an abstract controlled predecessor operation is used on the abstract game representation [38], and symbolic BDD-based exploration of the state space [15]. Additional techniques are available for finite-state games, e.g., generalizing winning runs into a winning strategy for one of the players [31].

Our notion of subgoal is related to the concept of landmarks as used in planning [22]. Landmarks are milestones that must be true on every successful plan, and they can be used to decompose a planning task into smaller sub-tasks. Landmarks have also been used in a game setting to prevent the opponent from reaching their goal using counter-planning [32]. Whenever a planning task is unsolvable, one method to find out why is checking hierarchical abstractions for solvability and finding the components causing the problem [36].

Causality-based approaches have also been used for model checking of multithreaded concurrent programs [24,25]. In our approach, we use Craig interpolation to compute the subgoals. Interpolation has already been used in similar contexts before, for example to extract winning strategies from game trees [16] or to compute new predicates to refine the game abstractions [10]. In [18], interpolation is used to synthesize concrete winning strategies from so called *winning strategy skeletons*, which describe a set of strategies of which at least one is winning.

#### **2 Motivating Example**

Consider the scenario that an expensive painting is displayed in a large exhibition room of a museum. It is secured with an alarm system that is controlled via a control panel on the opposite side of the room. A security guard is sleeping at the control panel and occasionally wakes up to check whether the alarm is still armed. To steal the painting, a thief first needs to disable the alarm and then reach the painting before the alarm has been reactivated. We model this scenario as a two-player game between a safety player (the guard) and a reachability player (the thief) in the theory of linear arithmetic. The moves of both players, their initial positions, and the goal condition are described by the formulas:

$$\begin{aligned} Int &\equiv \quad \neg \mathbf{r} \land x = 0 \land y = 0 \land p = 0 \land a = 1 \land t = 0, \\ Guard &\equiv \quad \neg \mathbf{r} \land \mathbf{r'} \land x' = x \land y' = y \land p' = p \\ &\land ((t' = t - 1 \land a' = a) \lor (t \le 0 \land t' = 2)), \\ Thef &\equiv \quad \mathbf{r} \land \neg \mathbf{r'} \land t' = t \\ &\land x + 1 \ge x' \ge x - 1 \land y + 1 \ge y' \ge y - 1 \\ &\land (x' \ne 0 \lor y' \ne 10 \implies a' = a) \\ &\land (x' \ne 10 \lor y' \ne 5 \lor a = 1 \implies p' = p), \\ Ch^{-1} &= \neg \mathbf{r} \land x = 1 \end{aligned} \tag{14.11}$$

*Goal* ≡ ¬**r** ∧ p = 1.

The thief's position in the room is modeled by two coordinates x, y <sup>∈</sup> <sup>R</sup> with initial value (0, 0), and with every transition the thief can move some bounded distance. Note that we use primed variables to represent the value of variables after taking a transition. The control panel is located at (0, 10) and the painting at (10, 5). The status of the alarm and the painting are described by two boolean variables a, p ∈ {0, 1}. The guard wakes up every two time units, modeled by the variable <sup>t</sup> <sup>∈</sup> <sup>R</sup>. The variables x, y are bounded to the interval [0, 10] and <sup>t</sup> to [0, 2]. The boolean variable **r** encodes who makes the next move. In the presented configuration, the thief needs more time to move from the control panel to the painting than the guard will sleep. It follows that there is a winning strategy for the guard, namely, to always reactivate the alarm upon waking up.

Although it is intuitively fairly easy to come up with this strategy for the guard, it is surprisingly hard for game solving tools to find it. The main obstacle is the infinite state space of this game. Our approach for solving games represented in this logical way imitates *causal reasoning*: Humans observe that in order for the thief to steal the painting (i.e., the effect p = 1), a transition must have been taken whose source state does not satisfy the pre-condition of (steal) while the target state does. Part of this cause is the condition a = 0, i.e., the alarm is off. Recursively, in order for the effect a = 0 to happen, a transition setting a from 1 to 0 must have occurred, and so on.

Our approach captures these cause-effect relationships through the notion of *necessary subgoals*, which are essential milestones that the reachability player has to transition through in order to achieve their goal. The first necessary subgoal corresponding to the intuitive description above is

$$C\_1 = (Guard \lor Thief) \land p \neq 1 \land p' = 1.$$

In this case, it easy to see that C<sup>1</sup> is also a *sufficient subgoal*, meaning that all successor states of C<sup>1</sup> are winning for the thief. Therefore, it is enough to solve the game with the modified objective to reach those predecessor states of C<sup>1</sup> from which the thief can *enforce* C<sup>1</sup> being the next move (even if it is not their turn). Doing so recursively produces the necessary subgoal

$$C\_2 = (Guard \lor Thief) \land a \neq 0 \land a' = 0,$$

meaning that some transition must have caused the effect that the alarm is disabled. However, C<sup>2</sup> is *not* sufficient which can be seen by recursively solving the game spanning from successor states of C<sup>2</sup> to C1. This computation has an important caveat: After passing through C2, it may happen that a is reset to 1 at a later point (in this particular case, this constitutes precisely the winning strategy of the safety player), which means that there is no canonical way to slice the game along this subgoal into smaller parts. Hence the recursive call solves the game from C<sup>2</sup> to C<sup>1</sup> *subject to* the bold assumption that any move from a = 0 to a = 1 is winning for the guard. This generally underapproximates the winning states of the thief. Remarkably, we show that this approximation is enough to build winning strategies for *both* players from their respective winning regions. In this case, it allows us to infer that moving through C<sup>2</sup> is always a losing move for the thief. However, at the same time, any play reaching *Goal* has to move through C2. It follows that the thief loses the global game.

We evaluated our method on several configurations of this game, which we call *Mona Lisa*. The results in Sect. 6 support our conjecture that the room size has little influence on the time our technique needs to solve the game.

#### **3 Preliminaries**

We consider two-player reachability games defined by formulas in a given logic L. We let L(V) be the L-formulas over a finite set of variables V, also called *state predicates* in the following. We call V = {*v* | *v* ∈ V} the set of *primed variables*, which are used to represent the value of variables after taking a transition. Transitions are expressed by formulas in the set L(V∪V ), called *transition predicates*. For some formula ϕ ∈ L(V), we denote the substitution of all variables by their primed variant by ϕ[V/V ]. Similarly, we define ϕ[V /V].

For our algorithm we will require the satisfiability problem of L-formulas to be decidable and *Craig interpolants* [13] to exist for any two mutually unsatisfiable formulas. Formally, we assume there is a function Sat : <sup>L</sup>(V) <sup>→</sup> <sup>B</sup> that checks the satisfiability of some formula ϕ ∈ L(V) and an unsatisfiability check Unsat : <sup>L</sup>(V) <sup>→</sup> <sup>B</sup>. For interpolation, we assume that there is a function Interpolate : L(V) × L(V) → L(V) computing a *Craig interpolant* for mutually unsatisfiable formulas: If ϕ, ψ ∈ L(V) are such that Unsat(ϕ ∧ ψ) holds, then ψ =⇒ Interpolate(ϕ, ψ) is valid, Interpolate(ϕ, ψ) ∧ ϕ is unsatisfiable, and Interpolate(ϕ, ψ) only contains variables shared by ϕ and ψ.

These functions are provided by many modern *Satisfiability Modulo Theories* (SMT) solvers, in particular for the theories of linear integer arithmetic and linear real arithmetic, which we will use for all our examples. Note that interpolation is usually only supported for the quantifier-free fragments of these logics, while our algorithm will introduce existential quantifiers. Therefore, we resort to quantifier elimination wherever necessary, for which there are known procedures for both linear integer arithmetic and linear real arithmetic formulas [29,33].

In order to distinguish the two players, we will assume that a Boolean variable called **r** ∈ V exists, which holds exactly in the states controlled by the reachability player. For all other variables v ∈ V, we let D(v) be the domain of v, and we define D = -{D(v) | v ∈ V}. In the remainder of the paper, we consider the variables V and their domains to be fixed.

**Definition 1 (Reachability Game).** *A reachability game is defined by a tuple* G = *Init*, *Safe*, *Reach*, *Goal , where Init* ∈ L(V) *is the* initial condition*, Safe* ∈ L(V∪V ) *defines the transitions of player SAFE, Reach* ∈ L(V∪V ) *defines the transitions of player REACH and Goal* ∈ L(V) *is the* goal condition*.*

*We require the formulas* (*Safe* =⇒ ¬**r**) *and* (*Reach* =⇒ **r**) *to be valid.*

A *state* s of G is a valuation of the variables V, i.e., a function s: V→D that satisfies s(v) ∈ D(v) for all v ∈ V. We denote the set of states by S, and we let SSAFE be the states s such that s(**r**) = false, and SREACH be the states s such that s(**r**) = true. The variable **r** determines whether REACH or SAFE makes the move out of the current state, and in particular *Safe* ∧ *Reach* is unsatisfiable.

Given a state predicate ϕ ∈ L(V), we denote by ϕ(s) the closed formula we get by replacing each occurrence of variable v ∈ V in ϕ by s(v). Similarly, given a transition predicate τ ∈ L(V∪V ) and states s, s , we let τ (s, s ) be the formula we obtain by replacing all occurrences of v ∈ V in τ by s(v), and all occurrences of v ∈ V in τ by s (v). For replacing only v ∈ V by s(v), we define τ (s) ∈ L(V ). A *trap state* of G is a state s such that (*Safe* ∨*Reach*)(s) ∈ L(V ) is unsatisfiable (i.e., s has no outgoing transitions).

A *play* of G starting in state s<sup>0</sup> is a finite or infinite sequence of states <sup>ρ</sup> <sup>=</sup> <sup>s</sup>0s1s<sup>2</sup> ... <sup>∈</sup> *<sup>S</sup>* <sup>+</sup> <sup>∪</sup> *<sup>S</sup>* <sup>ω</sup> such that for all i < len(ρ) either *Safe*(si, si+1) or *Reach*(si, si+1) is valid, and if ρ is a finite play, then slen(ρ) is required to be a trap state. Here, len(s<sup>0</sup> ...sn) = n for finite plays, and len(ρ) = ∞ if ρ is an infinite play. The set of plays of some game G = *Init*, *Safe*, *Reach*, *Goal* is defined as Plays(G) = {ρ = s0s1s<sup>2</sup> ... | ρ is a play in G s.t. *Init*(s0) holds}. REACH *wins* some play ρ = s0s<sup>1</sup> ... if the play reaches a goal state, i.e., if there exists some integer 0 ≤ k ≤ len(ρ) such that *Goal*(sk) is valid. Otherwise, SAFE wins play ρ. A *reachability strategy* σ*<sup>R</sup>* is a function σ*<sup>R</sup>* : *S* <sup>∗</sup>SREACH → *S* such that if σ*R*(ωs) = s and s is not a trap state, then *Reach*(s, s ) is valid. We say that a play ρ = s0s1s<sup>2</sup> ... is *consistent* with σ*<sup>R</sup>* if for all i such that si(**r**) = true we have si+1 = σ*R*(s<sup>0</sup> ...si). A reachability strategy σ*<sup>R</sup>* is *winning* from some state s if REACH wins every play consistent with σ*<sup>R</sup>* starting in s. We define *safety strategies* σ*<sup>S</sup>* for SAFE analogously. We say that a player *wins in or from a state* s if they have a winning strategy from s. Lastly, REACH *wins the game* G if they win from some initial state. Otherwise, SAFE wins.

We often project a transition predicate T onto the source or target states of transitions satisfying T, which is taken care of by the formulas Pre(*T*) = ∃V . *T* and Post(*T*) = ∃V. *T*. The notation ∃V (resp. ∃V ) represents the existential quantification over all variables in the corresponding set. Given ϕ ∈ L(V), we call the set of transitions in G that move from states not satisfying ϕ, to states satisfying ϕ, the *instantiation* of ϕ, formally:

$$\text{Instantitate}(\varphi, \mathcal{G}) = (Safe \lor Recall) \land \neg \varphi \land \varphi'.$$

#### **4 Subgoals**

We formally define the notion of subgoals. Let G = *Init*, *Safe*, *Reach*, *Goal* be a fixed reachability game throughout this section, where we assume that *Init*∧*Goal* is unsatisfiable. Whenever this assumption is not satisfied in our algorithm, we will instead consider the game G = *Init* ∧¬*Goal*, *Safe*, *Reach*, *Goal* which does satisfy it. As states in *Init* ∧*Goal* are immediately winning for REACH, this is not a real restriction.

**Definition 2 (Enforceable transitions).** *The set of* enforceable transitions *relative to a transition predicate* T ∈ L(V∪V ) *is defined by the formula*

$$\operatorname{Enf}(T,\mathcal{G}) = \{ Safe \lor Recall \} \land T \land \neg \exists \mathcal{V}'. \{ Safe \land \neg T \}.$$

The enforceable transitions operator serves a purpose similar to the *controlled predecessors* operator commonly known in the literature, which is often used in a backwards fixed point computation, called *attractor construction* [37]. For both operations, the idea is to determine controllability by REACH. The main difference is that we do not consider the whole transition relation, but only a predetermined set of transitions and check from which predecessor states the post-condition of the set can be enforced by REACH. These include all transitions in T controlled by REACH and additionally transitions in T controlled by SAFE such that *all other transitions* in the origin state of the transition also satisfy T. The similarity with the controlled predecessor is exemplified by the following lemma:

**Lemma 3.** *Let* T *be a transition predicate, and suppose that all states satisfying* Post(T)[V /V] *are winning for REACH in* G*. Then all states in* Pre(Enf(T, G)) *are winning for REACH in* G*.*

*Proof.* Clearly, all states in Pre(Enf(T, G)) that are under the control of REACH are winning for REACH, as in any such state they have a transition satisfying T (observe that Enf(T, G) =⇒ T is valid), which leads to a winning state by assumption.

So let s be a state satisfying Pre(Enf(T, G)) that is under the control of SAFE. As Pre(Enf(T, G))(s) is valid, s has a transition that satisfies T (in particular, s is not a trap state). Furthermore, we know that there is no s ∈ *S* such that *Safe*(s, s )∧ ¬*T*(s, s ) holds, and hence there is no transition satisfying ¬*T* from s. Since Post(T)[V /V] is winning for REACH, it follows that from s player SAFE cannot avoid playing into a winning state of REACH.

We now turn to a formal definition of *necessary subgoals*, which intuitively are sets of transitions that appear on every play that is winning for REACH.

**Definition 4 (Necessary subgoal).** *A* necessary subgoal C ∈ L(V∪V ) *for* G *is a transition predicate such that for every play* <sup>ρ</sup> <sup>=</sup> <sup>s</sup>0s<sup>1</sup> ... *of* <sup>G</sup> *and* <sup>n</sup> <sup>∈</sup> <sup>N</sup> *such that Goal*(sn) *is valid, there exists some* k<n *such that* C(sk, sk+1) *is valid.*

Necessary subgoals provide a means by which winning safety player strategies can be identified, as formalized in the following lemma.

**Lemma 5.** *A safety strategy* σ*<sup>S</sup> is winning in* G *if and only if there exists a necessary subgoal C for* G *such that for all plays* ρ = s0s<sup>1</sup> ... *of* G *consistent with* <sup>σ</sup>*<sup>S</sup> there is no* <sup>n</sup> <sup>∈</sup> <sup>N</sup> *such that* <sup>C</sup>(sn, sn+1) *holds.*

*Proof.* " =⇒ ". The transition predicate *Goal*[V/V ] (i.e., transitions with endpoints satisfying *Goal*) is clearly a necessary subgoal. If σ*<sup>S</sup>* is winning for SAFE, then no play consistent with σ*<sup>S</sup>* contains a transition in this necessary subgoal. "⇐=". Let C be a necessary subgoal such that no play consistent with σ*<sup>S</sup>* contains a transition of C. Then by Definition 4 no play consistent with σ*<sup>S</sup>* contains a state satisfying *Goal*. Hence σ*<sup>S</sup>* is a winning strategy for SAFE.

Of course, the question remains how to compute non-trivial subgoals. Indeed, using *Goal* as outlined in the proof above provides no further benefit over a simple backwards exploration (see Remark 15 in the following section).

Ideally, a subgoal should represent an interesting key decision to focus the strategy search. As we show next, Craig interpolation allows to extract partial causes for the mutual unsatisfiability of *Init* and *Goal* and can in this way provide necessary subgoals. Recall that a Craig interpolant ϕ between *Init* and *Goal* is a state predicate that is implied by *Goal*, and unsatisfiable in conjunction with *Init*. In this sense, ϕ describes an observable *effect* that must occur if REACH wins, and the concrete transition that instantiates the interpolant *causes* this effect.

**Proposition 6.** *Let* ϕ *be a Craig interpolant for Init and Goal . Then the transition predicate* Instantiate(ϕ, G) *is a necessary subgoal.*

*Proof.* As ϕ is an interpolant, it holds that *Goal* =⇒ ϕ is valid and *Init* ∧ ϕ is unsatisfiable. Consider any play ρ = s0s<sup>1</sup> ... of G such that *Goal*(sn) is valid for some <sup>n</sup> <sup>∈</sup> <sup>N</sup>. It follows that <sup>¬</sup>ϕ(s0) and <sup>ϕ</sup>(sn) are both valid. Consequently, there is some 0 ≤ i<n such that ¬ϕ(si) and ϕ(si+1) are both valid. As all pairs (sk, sk+1) satisfy either *Safe* or *Reach*, it follows that Instantiate(ϕ, G) (si, si+1) is valid. Hence, Instantiate(ϕ, G) is a necessary subgoal.

While avoiding a necessary subgoal is a winning strategy for SAFE, reaching a necessary subgoal is in general not sufficient to guarantee a win for REACH. This is because there might be some transitions in the necessary subgoal that produce the desired effect described by the Craig interpolant, but that trap REACH in a region of the state space where they cannot enforce some other necessary effect to reach goal. For the purpose of describing a set of transitions that is guaranteed to be winning for the reachability player, we introduce *sufficient subgoals*.

**Definition 7 (Sufficient subgoal).** *A transition predicate F* ∈ L(V ∪ V ) *is called a* sufficient subgoal *if REACH wins from every state satisfying* Post(*F*)[V /V]*.*

*Example 8.* Consider the Mona Lisa game G described in Sect. 2.

$$C\_1 = (Guard \lor Thief) \land p \neq 1 \land p' = 1$$

qualifies as sufficient subgoal, because REACH wins from every successor state as all those states satisfy *Goal*. Also, every play reaching *Goal* eventually passes C1, and hence C<sup>1</sup> is also necessary. On the other hand,

$$C\_2 = (Guard \lor Thief) \land a \neq 0 \land a' = 0$$

is only a necessary subgoal in G, because SAFE wins from some (in fact all) states satisfying Post(C2).

If the set of transitions in the necessary subgoal C that lead to winning states of REACH is definable in L then we call the transition predicate F that defines it the *largest sufficient subgoal* included in C. It is characterized by the properties (1) F =⇒ C is valid, and (2) if F is such that F =⇒ F is valid, then either F ≡ F , or F is not a sufficient subgoal. Since C is a necessary subgoal and F is maximal with the properties above, REACH needs to see a transition in F eventually in order to win. This balance of necessity and sufficiency allows us to partition the game along F into a game that happens after the subgoal and one that happens before.

**Proposition 9.** *Let* C *be a necessary subgoal, and* F *be the largest sufficient subgoal included in* C*. Then REACH wins from an initial state* s *in* G *if and only if REACH wins from* s *in the pre-game*

$$\mathcal{G}\_{pre} = \langle Int, Safe \wedge \neg F, Recall \wedge \neg F, \text{Pre}(\text{Enf}(F, \mathcal{G})) \rangle.$$

*Proof.* " =⇒ ". Suppose that REACH wins in G from s using strategy σR. Assume for a contradiction that SAFE wins in Gpre from s using strategy σS. Consider strategy σ <sup>S</sup> such that σ *<sup>S</sup>* (ωs ) = σ*<sup>S</sup>* (ωs ) if (*Safe* ∧ ¬F)(s ) is satisfiable, and else σ *<sup>S</sup>* (ωs ) = σ *<sup>S</sup>* (ωs ), where σ *<sup>S</sup>* is an arbitrary safety player strategy in G. Let ρ = s0s<sup>1</sup> ... be the (unique) play of G consistent with both σ<sup>R</sup> and σ <sup>S</sup>, where s<sup>0</sup> = s. Since σ<sup>R</sup> is winning in G and C is a necessary subgoal in G, there must exist some <sup>m</sup> <sup>∈</sup> <sup>N</sup> such that <sup>C</sup>(sm, sm+1) is valid. Let <sup>m</sup> be the smallest such index. Since F =⇒ C, we know for all 0 ≤ k<m that ¬F(sk, sk+1) holds. Hence, there is the play ρ = s0s<sup>1</sup> ...s<sup>m</sup> ... in Gpre consistent with σS. The state sm+1 is winning for REACH in G, as it is reached on a play consistent with the winning strategy σR. Hence, we know that F(sm, sm+1) holds, because F is the largest sufficient subgoal included in C. If (*Reach* ∧F)(sm, sm+1) held, we would have that Pre(Enf(F, G)(sm) holds: a contradiction with ρ being consistent with σS, which we assumed to be winning in Gpre. It follows that (*Safe*∧F)(sm, sm+1) holds. We can conclude that (*Safe* ∧ ¬F)(sm) is unsatisfiable (i.e., s<sup>m</sup> is a trap state in Gpre), because in all other cases SAFE plays according to σ*<sup>S</sup>* , which cannot choose a transition satisfying F. However, this implies that Pre(Enf(F, G)(sm) holds, again a contradiction with ρ being consistent with winning strategy σS. "⇐=". If REACH wins in Gpre they have a strategy σ*<sup>R</sup>* such that every play consistent with σ*<sup>R</sup>* reaches the set Pre(Enf(F, G)). As F is a sufficient subgoal, the states Post(F) are winning for REACH by definition. It follows by Lemma 3 that all states satisfying Pre(Enf(F, G)) are winning in G. Combining σ<sup>R</sup> with a strategy that wins in all these states yields a winning strategy for REACH in G.

#### **5 Causality-Based Game Solving**

Lemma 9 in the preceding section foreshadows how subgoals can be employed in building a recursive approach for the solution of reachability games. Before turning to our actual algorithm, we describe a way to symbolically represent nondeterministic memoryless strategies. As discussed in [18], there is no ideal strategy description language for the class of games we consider. Our approach allows us to describe sets of concrete strategies as defined in Sect. 3 with linear arithmetic formulas. This framework will prove convenient for *strategy synthesis*, i.e., the computation of winning strategies instead of simply determining the winner of the game.

#### **5.1 Symbolically Represented Strategies**

We will represent strategies for both players using transition predicates S ∈ L(V∪V ), henceforth called *symbolic strategies*, where we only require that (<sup>S</sup> <sup>=</sup><sup>⇒</sup> (*Safe* <sup>∨</sup> *Reach*)) is valid. A sequence <sup>s</sup><sup>0</sup> ...s<sup>n</sup> <sup>∈</sup> <sup>S</sup><sup>+</sup> is called a *play prefix* if it is a prefix of some play in G, (¬*Goal*)(s<sup>j</sup> ) holds for all 0 ≤ j ≤ n, and s<sup>n</sup> is not a trap state. We say that a play prefix ρ = s<sup>0</sup> ...s<sup>n</sup> *conforms to* a symbolic reachability strategy S if for all j<n we have that S(s<sup>j</sup> , sj+1) holds whenever s<sup>j</sup> ∈ SREACH (and analogously for safety strategies). A play conforms to S if all its play prefixes conform to S. We say that S is winning for REACH in s if all plays from s that conform to S are winning for REACH and all play prefixes s<sup>0</sup> ...s<sup>n</sup> ∈ S∗SREACH from s that conform to S are such that (S ∧ *Reach*)(sn) is satisfiable (and analogously for SAFE). The second condition ensures that the player cannot be forced to play a transition outside of S by their opponent while the play has not reached a trap state or *Goal*, and in particular guarantees the existence of a concrete strategy (as defined in Sect. 3) conforming to S.

**Lemma 10.** *If REACH (SAFE) has a winning symbolic strategy in* s*, then REACH (SAFE) has a concrete winning strategy in* s*.*

*Proof.* Let S by a symbolic winning strategy for REACH. Let σ<sup>R</sup> be any reachability strategy such that for all play prefixes ωs ∈ S∗SREACH that conform to S the formula S(s, σR(ωs)) is valid. Such a function is guaranteed to exist, as (S∧*Reach*)(s) is satisfiable for all such play prefixes by definition. Furthermore, σ<sup>R</sup> is winning as all play prefixes of plays consistent with σ<sup>R</sup> conform to S, and hence all these plays are winning by assumption. The proof for SAFE is analogous. 

This representation allows us to specify nondeterministic strategies, but classical memoryless strategies on finite arenas (specified as a function σ : SREACH → S or *S*SAFE → S) can also be represented in this form using a disjunction over formulas <sup>v</sup>∈V (<sup>v</sup> <sup>=</sup> <sup>s</sup>(v) <sup>∧</sup> <sup>v</sup> <sup>=</sup> <sup>σ</sup>(s)(v)) for varying <sup>s</sup> <sup>∈</sup> <sup>S</sup>.

The following lemma shows that a necessary subgoal directly yields a symbolic strategy for SAFE if the subgoal is, in a certain sense, *locally avoidable* by SAFE. It will be our main tool for synthesizing safety player strategies.

**Lemma 11.** *Let* C *be a necessary subgoal for* G *and suppose that* Unsat(Enf(C, G)) *holds. Then, Safe* ∧ ¬C *is a winning symbolic strategy for SAFE in* G*.*

#### **5.2 A Recursive Algorithm**

We now describe our algorithm which utilizes necessary subgoals to decompose and solve two-player reachability games (Algorithm 1). It is incomplete in the sense that it does not return on every input (Sect. 5.3 discusses special cases with guaranteed termination). If the algorithm returns on input G, it returns a triple (R, SREACH, SSAFE), where (1) R is a state predicate characterizing the initial states that are winning for REACH in G, (2) SREACH is a symbolic strategy for REACH that wins in all initial states satisfying R, and (3) SSAFE is a symbolic strategy for SAFE that wins in all initial states satisfying *Init* ∧¬R. The returned safety strategy SSAFE is such that ¬SSAFE is a necessary subgoal that SAFE can avoid locally in the game G restricted to intial states *Init* ∧ ¬R (see Lemma 11).

Algorithm 1 works as follows. States satisfying *Init* and *Goal* are immediately winning for REACH and thus always part of the returned formula R. Following the discussion at the beginning of Sect. 4, further analysis considers the game starting in the remaining initial states I = *Init* ∧ ¬*Goal*. If there is no such state, we may return that all initial states are winning (line 5). Here, REACH wins from R without playing any move, and hence SREACH = false is a valid winning symbolic strategy (winning symbolic strategies are only required to provide moves in prefixes that have not seen *Goal* so far). We may choose SSAFE arbitrarily as there is no initial state winning for SAFE.

If the algorithm does not return in line 5, a necessary subgoal C between I and *Goal* is computed by instantiating a Craig interpolant ϕ for the two predicates (lines 6 and 7, see also Proposition 6). We break up the remaining description of the algorithm into three parts, which correspond to the main cases that occur when splitting the game along the subgoal C.

**Algorithm 1:** Reach(G) **In :** reachability game G = -*Init*, *Safe*, *Reach*, *Goal* **Out:** triple (R, SREACH, SSAFE) s.t. – *R* ∈ L(V) represents the set of initial states winning for REACH; – SREACH is a winning symbolic reachability strategy for states in R; – SSAFE is a winning symbolic safety strategy for states in *Init* ∧ ¬R. **1 begin <sup>2</sup>** *R* ← *Init* ∧ *Goal* **<sup>3</sup>** *I* ← *Init* ∧ ¬*Goal* **4 if** Unsat(*I* ) **then 5 return** *R*, false, false **<sup>6</sup>** ϕ ← Interpolate(*I*, *Goal* ) **<sup>7</sup>** *C* ← Instantiate(ϕ, G) **<sup>8</sup> if** Unsat(Enf(*C*, G)) **then <sup>9</sup> return** *R*, false, *Safe* ∧ ¬C **<sup>10</sup>** G*post* ← -Post(C)[V- /V], *Safe* ∧ ϕ, *Reach* ∧ ϕ, *Goal* **<sup>11</sup>** *Rpost*, S*post* REACH, S*post* SAFE ← Reach - G*post* **<sup>12</sup>** *F* ← C ∧ *Rpost*[V/V- ] **<sup>13</sup> if** Unsat(Enf(*F*, G)) **then <sup>14</sup> return** *<sup>R</sup>*, false, *Safe* ∧ ¬<sup>F</sup> <sup>∧</sup> (<sup>ϕ</sup> <sup>=</sup><sup>⇒</sup> <sup>S</sup>*post SAFE* ) **<sup>15</sup> if** Sat((*Reach* ∨ *Safe*) ∧ ϕ ∧ ¬ϕ- ∧ ¬*Goal* ) **then <sup>16</sup>** *F* ← *F* ∨ *Goal*[V/V- ] **<sup>17</sup>** ϕ ← false **<sup>18</sup>** G*pre* ← -I, *Safe* ∧ ¬*F*, *Reach* ∧ ¬*F*,Pre(Enf(*F*, G)) **<sup>19</sup>** *Rpre*, S*pre* REACH, S*pre* SAFE ← Reach - G*pre* **<sup>20</sup> return** *R* ∨ *Rpre,* **<sup>21</sup>** combine(S*pre REACH*, F, S*post REACH*)*,* **<sup>22</sup>** (¬<sup>ϕ</sup> <sup>=</sup><sup>⇒</sup> <sup>S</sup>*pre SAFE*) <sup>∧</sup> (<sup>ϕ</sup> <sup>=</sup><sup>⇒</sup> <sup>S</sup>*post SAFE* )

**Case 1:** SAFE **can avoid the subgoal** C**.** If the necessary subgoal C qualifies for Lemma 11, we can immediately conclude that SAFE is winning for all states statisfying I (lines 8 and 9). An instance of this case occurs if the interpolant describes a bottleneck in the game which is fully controlled by SAFE. The winning symbolic reachability strategy is *Safe* ∧ ¬C in this case (line 9), and we will assume that safety strategies returned by recursive calls of the algorithm are essentially negations of necessary subgoals that can be avoided by SAFE.

If Lemma 11 is not applicable, we next find those transitions in C that move into a winning state for the safety player. This is achieved by analyzing the *post-game* (line 10):

$$\mathcal{G}\_{post} = \langle \text{Post}(C)[\mathcal{V}'/\mathcal{V}], Safe \wedge \varphi, Reach \wedge \varphi, Goal \rangle.$$

Its initial states are exactly the states one sees after bridging the subgoal C. In order to make sure that Gpost is, in some sense, easier to solve than G, we restrict both *Safe* and *Reach* to ϕ, which is the interpolant used to compute the subgoal C. This has the effect of removing all transitions in states *not* satisfying ϕ, making them trap states. For the safety player this makes Gpost easier to win than G as all plays ending in such a trap state without seeing *Goal* before are winning for SAFE in Gpost. Hence we formally have:

**Lemma 12.** *If* S *is a winning symbolic reachability strategy from* s *in* Gpost*, then* S *is also winning from* s *in* G*.*

Due to the restriction to ϕ, intuitively REACH wins from a state s in Gpost if they can win from s in G *while staying inside the interpolant* ϕ. In other words, REACH must guarantee that the necessary subgoal C is not visited again in the play. Still, the set Rpost, as returned in line 11 by the recursive call to Algorithm 1 on Gpost, is a sufficient subgoal in G, by the above lemma. Furthermore, if SAFE can avoid all states satisfying Rpost (see line 13), then this also implies a winning strategy from all initial states in I. The reason is that REACH can only win by eventually visiting a state from which they can win without leaving ϕ again, as (*Goal* =⇒ ϕ) is valid. This is not possible if SAFE can avoid all states in Rpost.

In this case we construct <sup>S</sup>SAFE as follows. We assume that <sup>¬</sup>Spost SAFE is a necessary subgoal that can be locally avoided in Gpost from all states satisfying Post(C)[V /V] ∧ ¬Rpost, and furthermore, we know that F := C ∧ Rpost[V/V ] can be locally avoided in <sup>G</sup> (line 13). Intuitively, playing according to <sup>S</sup>post SAFE in Gpost yields a strategy for SAFE which avoids *Goal* and may move back into a state satisfying ¬ϕ, which forces REACH to bridge the subgoal C again in order to win. It follows that <sup>F</sup> <sup>∨</sup> (<sup>ϕ</sup> ∧ ¬Spost SAFE) is a necessary subgoal from I that can be locally avoided by SAFE in G, and the corresponding symbolic strategy is *Safe* ∧ ¬<sup>F</sup> <sup>∧</sup> (<sup>ϕ</sup> <sup>=</sup><sup>⇒</sup> <sup>S</sup>post SAFE) (we additionally intersect the negated necessary subgoal with *Safe* to ensure that the symbolic strategy only includes legal transitions).

So far, the subgoal was such that SAFE could avoid it entirely, or at least avoid all states from which REACH would win when forced to remain inside the post-game. If this is not the case, then we also need to consider the *pre-game* (line 18):

$$\mathcal{G}\_{pre} = \langle I, Safe \wedge \neg F, Reach \wedge \neg F, \text{Pre}(\text{Enf}(F, \mathcal{G})) \rangle.$$

which intuitively describes the game before bridging the interpolant C for the last time. The exact definition of F will depend on whether C *perfectly partitions* the game or not. In both cases F will be the largest sufficient subgoal contained in a necessary subgoal, which lets us apply Proposition 9 to conclude that the initial winning regions of G and Gpre coincide.

**Case 2: The Subgoal Perfectly Partitions** G**.** We say that ϕ *perfectly partitions* G if (*Reach*∨*Safe*)∧ϕ∧¬ϕ ∧¬*Goal* is unsatisfiable (cf. line 15). Intuitively, this means that there is no transition that "undoes" the effect of the subgoal C. If this holds, then the restriction of Gpost to states satisfying ϕ is *de facto* no longer a restriction, as no play can reach such a state anyway after passing through the subgoal. This intuition is formalized by the following lemma.

**Lemma 13.** *Assume that* ϕ perfectly partitions G*, and let* s *be a state satisfying* Post(C)[V /V]*. Then REACH wins from* s *in* Gpost *if and only if REACH wins from* s *in* G*.*

It follows that F = C ∧ Rpost[V/V ] is the largest sufficient subgoal included in C. By Proposition 9, the same initial states are winning for REACH in Gpre and in G. In this case, we construct the desired safety strategy (line 22) as

$$
\mathfrak{S}\_{\mathsf{SAFE}} = (\neg \varphi \implies \mathfrak{S}\_{\mathsf{SAFE}}^{pre}) \land (\varphi \implies \mathfrak{S}\_{\mathsf{SAFE}}^{post}),
$$

where <sup>¬</sup>Spre/post SAFE are assumed to be necessary subgoals avoidable by SAFE in the corresponding subgames. Intuitively, the combined strategy consists of following Spre SAFE as long as one remains in the pre-game, which, by induction hypothesis, allows SAFE to avoid all transitions from F if starting in Rpre. If the play crosses C ∧ ¬F, the strategy is to play according to the winning strategy of the postgame.

A symbolic strategy for REACH can be given by combining pre- and poststrategies as follows (line 21):

$$\begin{split} \text{combine}(\mathfrak{S}^{pre}\_{\text{REACH}}, F, \mathfrak{S}^{post}\_{\text{REACH}}) &:= (\text{Pre}(\mathfrak{S}^{post}\_{\text{REACH}}) \implies \mathfrak{S}^{post}\_{\text{REACH}}) \\ &\quad \land \quad ((\neg \text{Pre}(\mathfrak{S}^{post}\_{\text{REACH}}) \land \text{Pre}(F)) \implies F) \\ &\quad \land \quad ((\neg \text{Pre}(\mathfrak{S}^{post}\_{\text{REACH}}) \land \neg \text{Pre}(F)) \implies \mathfrak{S}^{pre}\_{\text{REACH}}) \\ &\quad \land \quad (\text{Pre}(\mathfrak{S}^{post}\_{\text{REACH}}) \lor \text{Pre}(F) \lor \text{Pre}(\mathfrak{S}^{pre}\_{\text{REACH}})) . \end{split}$$

This represents a nested conditional strategy that prefers the strategies of the subgames in the priority order Spost REACH, F, and finally Spre REACH. The reason for this order is that the winning condition in the post-game coincides with the global winning objective (to reach *Goal*), while in the pre-game REACH tries to reach a winning state in the post-game. The set F is exactly the bridge between these two. The last condition makes sure that the strategy only includes transitions of states in which it is winning.

**Case 3: The subgoal does not perfectly partition** G**.** If Sat((*Reach* ∨ *Safe*) ∧ ϕ ∧ ¬ϕ ∧ ¬*Goal*) is true in line 15, we can no longer assume that F is the largest sufficient subgoal in C. The reason is that SAFE may win in Gpost by moving out of the subgame, but if this move leads to a winning state for REACH in G, then such a strategy is winning in Gpost, but not in G. So we can only assume that F is sufficient (this follows by Lemma 12). In order to apply Proposition 9 we extend F by all transitions that move directly into *Goal* (line 16). This immediately yields a necessary and sufficient subgoal, and so again Proposition 9 applies to Gpre (line 18). We could have also added *Goal*-states to F in Case 2, but we have observed that not doing so improves the performance of our procedure considerably.

The reachability strategy is composed of Spre REACH, F, and Spost REACH exactly as in Case 2 (line 21). As all transitions in F are losing for SAFE, and these are the only ones that are removed in Gpre, essentially SAFE can play using the same strategies in G and Gpre. We implement this by setting ϕ to false (line 17), in which case <sup>S</sup>SAFE (line 22) equals (true <sup>=</sup><sup>⇒</sup> <sup>S</sup>pre SAFE) <sup>∧</sup> (false <sup>=</sup><sup>⇒</sup> <sup>S</sup>post SAFE) <sup>≡</sup> <sup>S</sup>pre SAFE.

Finally, we formally state the partial correctness of the algorithm, using the ideas from above.

**Theorem 14 (Partial correctness).** *If* Reach(G) *returns* (R, S*REACH*, S*SAFE*)*, then*


*Remark 15 (Simulating the attractor).* Note that Craig interpolants are by no means unique. If we choose the interpolation function so that Interpolate(I, *Goal*) always returns *Goal* (this is a valid interpolant), Algorithm 1 essentially simulates the attractor. In this case the subgoal C (line 7) contains exactly the transitions that move directly into *Goal*. All states in Post(C)[V /V] are then winning for REACH and hence Rpost would be equivalent to Post(C)[V /V], which implies that C ≡ F holds in this case. The new goal states in Gpre are set to Pre(Enf(F, G)), which are exactly the states in Pre(C) that either are controlled by REACH, or such that all their transitions are included in F. Hence the set Pre(Enf(F, G)) is exactly the classical controlled predecessor.

One effect of slicing the game along general subgoals is that the initial predicate of the post-game (which describes all states satisfying the post-condition of the subgoal) may be satisfied by many states that do not necessarily need to be considered in order to decide who wins from the initial states of G (for example, because they are not reachable from any initial state, or cannot reach *Goal*). This can be a drawback if the (superfluous) size of the subgames makes them hard to solve. Notably, this is in general less of an issue for approaches based on unrolling of the transition relation: The method of solving increasingly large step-bounded games [18] will only consider states that are reachable from *Init*, while backwards fixpoint computations will not explore states that do not reach *Goal*. A way of coping with this is to provide additional information on the domains of variables, whenever this is available (we discuss the effect of bounding variable domains in Sect. 6). Indeed, in the case where all variable domains are finite, Algorithm 1 is guaranteed to terminate, as shown in the next subsection.

#### **5.3 Special Cases with Guaranteed Termination**

Deciding the winner in the types of games we consider is generally undecidable (see [18] for the case that L is linear real arithmetic). Since Algorithm 1 returns a correct result whenever it terminates, this implies that it cannot always terminate. In this section, we give two important cases in which we can prove termination.

**Theorem 16.** *If the domains of all variables in* G *are finite, then* Reach(G) *terminates.*

*Remark 17 (Time complexity).* The termination argument in the proof yields a single-exponential upper bound on the runtime of the algorithm, where the input size is measured in the number of concrete transitions of the game. This is because in both recursive calls the subgames may be "almost" as large as the input – they are only guaranteed to have at least one concrete transition less.

We now show that, under certain assumptions, our algorithm also terminates for games that have a finite bisimulation quotient. To this end, we first clarify what bisimilarity means in our setting. A relation R ⊆ S × S over the states of G is called a *bisimulation* on G, if


We say that s<sup>1</sup> and s<sup>2</sup> are *bisimilar* (denoted by s<sup>1</sup> ∼ s2) if there exists a bisimulation R such that (s1, s2) ∈ R. Bisimilarity is an equivalence relation, and it is the coarsest bisimulation on G. The equivalence classes are called *bisimulation classes*. As the winning region of any player can be expressed in the μ-calculus [39] and the μ-calculus is invariant under bisimulation [9], it follows that bisimilar states are won by the same player.

**Lemma 18.** *Let* R *be a bisimulation on* G*. If* (s1, s2) ∈ R*, then REACH wins from* s<sup>1</sup> *in* G *if and only if REACH wins from* s<sup>2</sup> *in* G*.*

We will assume that for each bisimulation class S<sup>i</sup> there exists a formula ψ<sup>i</sup> ∈ L(V) that *defines* Si, formally: For all s ∈ S, ψi(s) holds if and only if s ∈ Si. Furthermore, we will assume that the interpolation procedure *respects* ∼, formally: Interpolate(ϕ, ψ) is equivalent to a disjunction of formulas ψi. Such an interpolant exists if ψ or ϕ already satisfy this assumption.

**Theorem 19.** *Let* G *be a reachability game with finite bisimulation quotient under* ∼ *and assume that all bisimulation classes of* G *are definable in* L*. Furthermore, assume that* Interpolate *respects* ∼*. Then,* Reach(G) *terminates.*

#### **6 Case Studies**

In this section we evaluate our approach on a number of case studies. Our prototype CabPy<sup>2</sup> is written in Python and implements the game solving part of the presented algorithm. Extending it to returning a symbolic strategy using the ideas outlined above is straightforward. We compared our prototype with Sim-Synth [18], the only other readily available tool for solving linear arithmetic games. The evaluation was carried out with Ubuntu 20.04, a 4-core Intel® Core™ i5 2.30 GHz processor, as well as 8 GB of memory. CabPy uses the PySMT [19] library as an interface to the MathSAT5 [12] and Z3 [30] SMT solvers. On all benchmarks, the timeout was set to 10 min. In addition to the winner, we report the runtime and the number of subgames our algorithm visits. Both may vary with different SMT solvers or in different environments.

#### **6.1 Game of Nim**

Game of Nim is a classic game from the literature [8] and played on a number of heaps of stones. Both players take turns of choosing a single heap and removing at least one stone from it. We consider the version where the player that removes the last stone wins. Our results are shown in Fig. 1. In instances with three heaps or more we bounded the domains of the variables in the instance description, by specifying that no heap exceeds its initial size and does not go below zero.

Following the discussion in Sect. 5.3, we need to bound the domains to ensure the termination of our tool on these instances. Remarkably, bounding the variables was not necessary for instances with only two heaps, where our tool CabPy scales to considerably larger instances than SimSynth. We did not add the same constraints to the input of SimSynth, as for SimSynth this resulted in longer runtimes rather than shorter. In Game of Nim, there are no natural necessary subgoals that the safety player can locally control.

The results (see Fig. 1) demonstrate that our approach is not completely dependent on finding the right interpolants and is in particular also competitive when the reachability player wins the game. We suspect that SimSynth performs worse in these cases because the safety player has a large range of possible moves in most states, and inferring the win of the reachability player requires the tool to backtrack and try our all of them.

#### **6.2 Corridor**

We now consider an example that demonstrates the potential of our method in case the game structure contains natural bottlenecks. Consider a corridor of 100 rooms arranged in sequence, i.e., each room i with 0 ≤ i < 100 is connected to room i + 1 with a door. The objective of the reachability player is to reach

<sup>2</sup> The source code of CabPy and our experimental data are both available at https://github.com/reactive-systems/cabpy. We provide a virtual machine image with CabPy already installed for reproducing our evaluation [35].


**Fig. 1.** Experimental results for the Game of Nim. The notation (h1,...,h*n*) denotes the instance played on n heaps, each of which consists of h*<sup>i</sup>* stones. Instances marked with b indicate that the variable domains were explicitly bounded in the input for CabPy.


**Fig. 2.** Experimental results for the Corridor game. The safety player controls the door between rooms r − 1 and r.

room 100 and they are free to choose valid values from R<sup>2</sup> for the position in each room at every other turn. The safety player controls some door to a room r ≤ 100. Naturally, a winning strategy is to prevent the reachability player from passing that door, which is a natural bottleneck and necessary subgoal on the way to the last room.

The experimental results are summarized in Fig. 2. We evaluated several versions of this game, increasing the length from the start to the controlled door. The results confirm that our causal synthesis algorithm finds the trivial strategy of closing the door quickly. This is because Craig interpolation focuses the subgoals on the room number variable while ignoring the movement in the rooms in between, as can be seen by the number of considered subgames. SimSynth, which tries to generalize a strategy obtained from a step-bounded game, struggles because the tool solves the games that happen between each of the doors before reaching the controlled one.

#### **6.3 Mona Lisa**

The game described in Sect. 2 between a thief and a security guard is very well suited to further assess the strength and limitations of both our approach as well as of SimSynth. We ran several experiments with this scenario, scaling the size of the room and the sleep time of the guard, as well as trying a scenario where the guard does not sleep at all. Scaling the size of the room makes it harder for SimSynth to solve this game with a forward unrolling approach, while our approach extracts the necessary subgoals irrespective of the room size. However, scaling the guard's sleep time makes it harder to solve the subgame between the two necessary subgoals, while it only has a minor effect on the length of the unrolling needed to stabilize the play in a safe region, as done by SimSynth.

The results in Fig. 3 support this conjecture. The size of the room has *almost no effect at all* on both the runtime of CabPy and the number of considered subgames. However, as the results for a sleep value of 4 show, the employed combination of quantifier elimination and interpolation introduces some instability in the produced formulas. This means we may get different Craig interpolants and slice the game with more or less subgoals. Therefore, we see a lot of potential in optimizing the interplay between the employed tools for quantifier elimination and interpolation. The phenomenon of the runtime being sensitive to these small changes in values is also seen with SimSynth, where a longer sleep time sometimes means a faster execution.

#### **6.4 Program Synthesis**

Lastly, we study two benchmarks that are directly related to program synthesis. The first problem is to synthesize a controller for a thermostat by filling out an incomplete program, as described in [4]. A range of possible initial values of the room temperature c is given, e.g., 20.8 ≤ c ≤ 23.5, together with the temperature dynamics which depend on whether the heater is on (variable <sup>o</sup> <sup>∈</sup> <sup>B</sup>). The objective for SAFE is to control the value of o in every round such that c stays between 20 and 25. This is a common benchmark for program synthesis tools and both CabPy and SimSynth solve it quickly (see Fig. 4). The other problem relates to Lamport's bakery algorithm [26]. We consider two processes using this protocol to ensure mutually exclusive access to a shared resource. The game describes the task of synthesizing a scheduler that violates the mutual exclusion. This essentially is a model checking problem, and we study it to see how well the tools can infer a safety invariant that is out of control of the safety player.


**Fig. 3.** Experimental results for the Mona Lisa game.


**Fig. 4.** Experimental results for program synthesis problems.

For our approach, this makes no difference, as both players may play through a subgoal and the framework is well suited to find a safety invariant. The forward unrolling approach of SimSynth, however, seems to explore the whole state space before inferring safety, and fails to find an invariant before a timeout.

#### **7 Conclusion**

Our work is a step towards the fully automated synthesis of software. It targets symbolically represented reachability games which are expressive enough to model a variety of problems, from common game benchmarks to program synthesis problems. The presented approach exploits causal information in the form of *subgoals*, which are parts of the game that the reachability player needs to pass through in order to win. Having computed a subgoal, which can be done using Craig interpolation, the game is split along the subgoal and solved recursively. At the same time, the algorithm infers a structured symbolic strategy for the winning player. The evaluation of our prototype implementation CabPy shows that our approach is practically applicable and scales much better than previously available tools on several benchmarks. While termination is only guaranteed for games with finite bisimulation quotient, the experiments demonstrate that several infinite games can be solved as well.

This work opens up several interesting questions for further research. One concerns the quality of the returned strategies. Due to its compositional nature, at first sight it seems that our approach is not well-suited to handle global optimization criteria, such as reaching the goal in fewest possible steps. On the other hand, the returned strategies often involve only a few key decisions and we believe that therefore the strategies are often very sparse, although this has to be further investigated. We also plan to automatically extract deterministic strategies from the symbolic ones [5,17] we currently consider.

Another question regards the computation of subgoals. The performance of our algorithm is highly influenced by which interpolant is returned by the solver. In particular this affects the number of subgames that have to be solved, and how complex they are. We believe that template-based interpolation [27] is a promising candidate to explore for computing good interpolants. This could be combined with the possibility for the user to provide templates or expressive interpolants directly, thereby benefiting from the user's domain knowledge.

#### **References**


**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

### **Author Index**

Abate, Alessandro II-3 Agarwal, Pratyush I-341 Akshay, S. I-619 Albert, Elvira II-863 Alur, Rajeev I-249 Amram, Gal I-870 André, Étienne I-552 Andriushchenko, Roman I-856 Arcaini, Paolo I-595 Armstrong, Alasdair I-303 Arquint, Linard I-367 Ayoun, Sacha-Élie II-827 Backes, J. II-851 Bae, Kyungmin I-491 Baier, Christel I-894 Baier, Daniel II-195 Bak, Stanley I-263 Balunovic, Mislav I-225 Bansal, Suguman I-870 Bardin, Sébastien I-669 Barrett, Clark II-461 Batz, Kevin II-524 Baumeister, Jan I-694 Bayless, S. II-851 Bendík, Jaroslav II-313 Beneš, Nikola I-505 Berzish, Murphy II-289 Beyer, Dirk II-195 Biere, Armin II-363 Bodeveix, Jean-Paul II-337 Bonakdarpour, Borzoo I-694 Boston, Brett I-645 Bozzano, Marco II-209 Bragg, Nate F. F. I-808 Breese, Samuel I-645 Brim, Luboš I-505 Brown, Kristopher II-461 Brunel, Julien II-337

Campbell, Brian I-303 Carpenter, Taylor I-249 Cauli, Claudia I-767

Ceška, Milan ˇ I-856 Chakraborty, Supratik II-911 Chalupa, Marek II-887 Chatterjee, Krishnendu I-341 Chemouil, David II-337 Chen, Guangke I-175 Chen, Jiayu I-225 Chen, Mingshuai I-443, II-524 Chen, Taolue I-175 Chen, Xiaohong II-477 Chiari, Michele II-387 Christakis, Maria I-201, II-777 Cimatti, Alessandro I-529, II-209 Clochard, Martin I-367 Coenen, Norine I-694, I-894 Cogumbreiro, Tiago I-403 Constantinides, George II-626 Cyphert, John I-46, I-783 D'Antoni, Loris I-84, I-783 DaCosta, D. II-851 Dahlqvist, Fredrik II-626 Dan, Andrei I-225 Day, Joel D. II-289 Dodds, Joey I-645 Dodds, Mike I-645 Dutertre, Bruno II-266 Dwyer, Matthew B. I-137

Eilers, Marco I-718 Eisenhut, Jan II-411 Elad, Neta I-317 Elbaum, Sebastian I-137 Eniser, Hasan Ferit I-201

Fang, Wang I-151 Farinier, Benjamin I-669 Farzan, Azadeh I-832 Ferlez, James I-287 Fernandes Pires, Anthony II-209 Finkbeiner, Bernd I-694, I-894 Foster, Jeffrey S. I-808 Fried, Dror I-870 Friedberger, Karlheinz II-195

Fu, Yu-Fu II-149 Funke, Florian I-894 Ganesh, Vijay II-289 Gardner, Philippa II-827 Gastin, Paul I-619 Genaim, Samir II-863 Giacobbe, Mirco II-3 Girol, Guillaume I-669 Gnad, Daniel II-411 Goel, Shilpi I-26 Gopinath, Divya I-3 Griggio, Alberto I-529, II-209 Guan, Ji I-151 Gupta, Aarti II-461 Gupta, Ashutosh II-911 Hahn, Ernst Moritz II-651 Hallé, Sylvain II-500 Hamilton, Nathaniel I-263 Hasuo, Ichiro I-595, II-75 Hauptman, Dustin I-566 Heljanko, Keijo II-363 Hermanns, Holger I-201 Hobor, Aquinas II-801 Hoffmann, Jörg I-201, II-411 Holtzen, Steven II-577 Hu, Qinheping I-84, I-783 Huffman, Brian I-645 Hur, Chung-Kil II-752 Immerman, Neil I-317 Irfan, Ahmed I-529, II-461 Itzhaky, Shachar I-110, II-125 Ivanov, Radoslav I-249 Jacobs, Bart II-27 Jacobs, Swen II-435 Jansen, Nils II-602 Jantsch, Simon I-894 Jewell, K. II-851 Johnson, Andrew I-380 Johnson, Taylor T. I-263 Jonáš, Martin II-209 Jones, B. F. II-851 Joshi, S. II-851 Jovanovi´c, Dejan II-266 Junges, Sebastian I-856, II-553, II-577, II-602

Kaminski, Benjamin Lucien II-524 Katoen, Joost-Pieter I-443, I-856, II-524 Keshmiri, Shawn I-566 Khedr, Haitham I-287 Kim, Dongjoo II-752 Kim, Jinwoo I-84 Kim, Sharon I-491 Kimberly, Greg II-209 Kincaid, Zachary I-46, II-51 Kla˘ska, David II-887 Kokologiannakis, Michalis I-427 Koskinen, Eric I-742 Kothari, Yugesh I-201 Kovács, Laura I-317 Kremer, Gereon II-231 Kulczynski, Mitja II-289 Kura, Satoshi II-75

Lal, Ratan I-566 Lange, Julien I-403 Launchbury, N. II-851 Lee, Insup I-249 Lee, Jaehun I-491 Lee, Juneyoung II-752 Lefaucheux, Engel II-172 Leow, Wei Xiang II-801 Leutgeb, Lorenz II-99 Li, Jianlin I-201 Li, Meng I-767 Li, Pengfei II-728 Li, Yangge I-580 Lin, Anthony W. II-243 Lin, Wang I-467 Lin, Zhengyao II-477 Liu, Jiaxiang II-149 Liu, Zhiming I-467 Lluch Lafuente, Alberto II-411 Lonsing, Florian II-461 Lopes, Nuno P. II-752 Lopez, Diego Manzanas I-263 Lyu, Deyun I-595

Ma, Lei I-595 Maksimovi´c, Petar II-827 Mandrioli, Dino II-387 Manea, Florin II-289 Mann, Makai II-461 Mansur, Muhammad Numair II-777 Mariano, Benjamin II-777 Markgraf, Oliver II-243


Ölveczky, Peter Csaba I-491 Oortwijn, Wytse I-367 Osama, Muhammad II-447 Ouaknine, Joël II-172

Pal, Neelanjana I-263 Pappas, George I-249 Parthasarathy, Gaurav II-704 P˘as˘areanu, Corina S. I-3 Pastva, Samuel I-505 Pathak, Shreya I-341 Pavlogiannis, Andreas I-341 Peleg, Hila I-110 Pereira, João C. I-367 Pereira, Mário II-677 Perez, Mateo II-651 Petcher, Adam I-645 Peyras, Quentin II-337 Piterman, Nir I-767 Polikarpova, Nadia I-110 Prabhakar, Pavithra I-566 Pradella, Matteo II-387 Prakash, Karthik R. I-619 Preiner, Mathias II-231 Pulte, Christopher I-303 Purser, David II-172

Rain, Sophie I-317 Rakamari´c, Zvonimir II-626 Ravara, António II-677 Reinhard, Tobias II-27 Reps, Thomas I-46, I-84, I-783 Rong, Dennis Liew Zhen I-403 Ro¸su, Grigore II-477 Roux, Cody I-808 Rowe, Reuben N. S. I-110 Roy, Diptarko II-3 Rubio, Albert II-863 Ryou, Wonryong I-225 Šafránek, David I-505 Sagiv, Mooly I-317 Sakr, Mouhammad II-435 Salvia, Rocco II-626 Sánchez, César I-694 Santos, José Fragoso II-827 Schewe, Sven II-651 Schröer, Philipp II-524 Sergey, Ilya I-110 Seshia, Sanjit A. II-553, II-577, II-602 Sewell, Peter I-303 Shi, Xiaomu II-149 Shoukry, Yasser I-287 Shriver, David I-137 Sibai, Hussein I-580 Siber, Julian I-894 Simner, Ben I-303 Singh, Gagandeep I-225 Singher, Eytan II-125 Slobodova, Anna I-26 Solar-Lezama, Armando I-808 Somenzi, Fabio II-651 Song, Fu I-175 Stan, Daniel II-243 Stefanescu, Andrei I-645 Strejˇcek, Jan II-887 Stupinský, Šimon I-856 Summers, Alexander J. II-704 Sumners, Rob I-26 Sun, Youcheng I-3 Swords, Sol I-26 Tabajara, Lucas Martinelli I-870

Tang, Xiaochao I-467 Terauchi, Tachio I-742 Tkachuk, Oksana I-767 Toman, Viktor I-341

Tomovi˘c, Luká˘s II-887 Tonetta, Stefano I-529 Torfah, Hazem II-553 Tran, Hoang-Dung I-263 Tremblay, Hugo II-500 Trentin, P. II-851 Trinh, Minh-Thai II-477 Trivedi, Ashutosh II-651 Tsai, Ming-Hsien II-149

Unadkat, Divyesh II-911 Unno, Hiroshi I-742, II-75 Usman, Muhammad I-3

Vafeiadis, Viktor I-427 Van den Broeck, Guy II-577 van der Berg, Freark I. II-690 Vardi, Moshe Y. I-870 Vazquez-Chanlatte, Marcell II-577 Vechev, Martin I-225

Wahl, Thomas I-380 Wang, Bow-Yaw II-149 Wang, Qiuye I-443 Wang, Yuting II-728 Weimer, James I-249 Weiss, Gera I-870 Wijs, Anton II-447 Wojtczak, Dominik II-651 Wolf, Felix A. I-367 Worrell, James II-172 Wu, Jinhua II-728 Wüstholz, Valentin I-201, II-777

Xu, Xiangzhe II-728 Xue, Bai I-443

Yang, Bo-Yin II-149 Yang, Xiaodong I-263 Yang, Yahan II-461 Yang, Zhengfeng I-467 Yin, Zhenguo II-728 Ying, Mingsheng I-151 Yu, Emily II-363

Zeng, M. Q. II-851 Zeng, Xia I-467 Zeng, Zhenbing I-467 Zhan, Naijun I-443 Zhang, Hongce II-461 Zhang, Yedi I-175 Zhang, Yidan I-467 Zhang, Zhenya I-595 Zhao, Jianjun I-595 Zhao, Zhe I-175 Zhu, Shaowei II-51 Zicarelli, Hannah I-403 Zuleger, Florian II-99