Skip to content Skip to navigation

CYBER-PHYSICAL SYSTEMS AND REINFORCEMENT LEARNING (479SM)

A.Y. 2020 / 2021

Professor 
Period 
Second semester
Credits 
6
Duration/Length 
48
Type of Learning Activity 
Related/additional subjects
Study Path 
[PDS0-2018 - Ord. 2018] common
Mutuazione 
Mutuato: SM35 - 575SM - CYBER-PHYSICAL SYSTEMS
Syllabus 
Teaching language 

English

Learning objectives 

In this course you will learn fundamental concepts of Cyber-Physical
Systems, with particular attention on modelling, monitoring their
behaviour, and learning.
Knowledge and understanding: basic and some advanced topics in hybrid
modelling, switching systems, real-time verification, Markov Decision
processes.
Applying knowledge and understanding: being capable of identifying the
best formalism to describe a CPS, of monitoring CPS behaviour, described
in a formal language, and of learning models of the environment and
optimal policies from observations, using reinforcement learning.
Communication skills: being able to explain the basic ideas and
communicate the results to experts and to non-experts.
Learning skills: being capable of exploring literature and find alternative
approaches and combine them to solve complex problems

Prerequisites 

Knowledge of Python and scientific Python and/or Matlab, Extended knowledge of
statistics, and machine learning. Basic knowledge of stochastic modeling.

Contents 

1. Introduction to CPS and application domains, with examples (e.g.,
medical CPS, transportation CPS, energy CPS)
2. Modeling formalisms: hybrid and switching systems, stochastic hybrid systems, Markov Decision Processes.
3. Control strategy: PID, Model Predictive Control, Reinforcement Learning
4. Monitoring CPS: formal requirements for CPS models, verification, monitoring
5. Simulation-based testing, falsification and parameter mining

Teaching format 

Frontal lectures and hands on sessions, both individual and in groups.
Ideally, each lecture will have a part of frontal teaching and a part of
hands-on training. Exercises and group tasks may be left for further
elaboration.

Extended Programme 

1. Introduction to CPS and application domains, with examples (e.g.,
medical CPS, transportation CPS, energy CPS)
2. Modeling formalisms: hybrid and switching systems, stochastic hybrid systems, Markov Decision Processes.
3. Control strategy: PID, Model Predictive Control, Reinforcement Learning
4. Monitoring CPS: formal requirements for CPS models, verification, monitoring
5. Simulation-based testing, falsification and parameter mining

End-of-course test 

The exam will be a project work.
Each project will consist of one or more tasks and will be concluded with a report on achieved results. During the presentation, extra questions can be asked to assess the individual preparation on the
topics of the course

Other information 

.

Texts/Books 

R. Alur. Principles of Cyber-Physical Systems, MIT press, 2015

Introduction to Embedded Systems: A CPS approach, MIT Press, 2017, (https://ptolemy.berkeley.edu/books/leeseshia/)

Principles of Model Checking. Baier, Katoen, MIT press, 2008

Reinforcement learning: An introduction, RS Sutton, AG Barto -Cambridge, 2011.
(http://incompleteideas.net/book/bookdraft2017nov5.pdf)


Back to list of courses