Available online at www.sciencedirect.

com

ScienceDirect
Procedia Manufacturing 11 (2017) 292 – 299

27th International Conference on Flexible Automation and Intelligent Manufacturing, FAIM2017,

27-30 June 2017, Modena, Italy

Semantic modelling of hybrid controllers for robotic cells

Mathias Haagea*, Jacek Maleca, Anders Nilssonb, Maj Stenmarka, Elin Anna Toppa
a
Department of Computer Science, Lund University, Box 118, 221 00 Lund, Sweden
b
Department of Automatic Control, Lund University, Box 118, 221 00 Lund, Sweden

Abstract

Programmable Logic Controllers (PLCs) play an important role for integration of hardware and software in industrial robot cells.
In this paper we propose a semantic grounding of the Sequential Function Charts (SFC) notation for specification of PLC
programs. Our semantic grounding is expressed in the OWL semantic language and forms part of our semantic robot framework,
called KIF (Knowledge Integration Framework). KIF is a set of ontologies and associated tools to ensure interoperability
between heterogeneous equipment making up a robot cell. We also present a tool set for manipulating SFC instances stored in
RDF triple stores reachable through the RDF4J framework. The semantic grounding and tool set are evaluated in a teaching-by-
demonstration experiment in a small parts assembly setup featuring a collaborative industrial robot, ABB YuMi, where the tool
set is used to create and execute SFC descriptions on-the-fly based on data from human demonstrations.

©
© 2017
2017Published by Elsevier
The Authors. B.V. by
Published This is an open
Elsevier B.V.access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of the scientific committee
Peer-review under responsibility of the scientific committee of theof27th
the International
27th International Conference
Conference on Flexible
on Flexible Automation
Automation and and
IntelligentManufacturing
Intelligent Manufacturing.

Keywords: IEC 61131-3; Sequential Function Chart; SFC; Programmable Logic Controller; PLC; Ontology; Knowledge Integration Framework;

1. Introduction

Programmable Logic Controllers (PLCs) play an important role in integration and control of equipment in
automation. PLCs exist on a variety of platforms such as industrial personal computers (PCs), embedded controllers,
and pure software PLCs executing on ordinary PCs running consumer operating systems. In industrial robot cells,

* Corresponding author.
E-mail address: mathias.haage@cs.lth.se

2351-9789 © 2017 Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license
(http://creativecommons.org/licenses/by-nc-nd/4.0/).
Peer-review under responsibility of the scientific committee of the 27th International Conference on Flexible Automation and Intelligent Manufacturing
doi:10.1016/j.promfg.2017.07.108
 Mathias Haage et al. / Procedia Manufacturing 11 (2017) 292 – 299 293

the robot controller typically co-exists with a PLC that controls cell equipment. The PLC is defined by the IEC
61131 standard and a part of the standard (IEC 61131-3 [1]) defines five programming languages for writing PLC
programs: Instruction Lists (IL), Structured Text (ST), Ladder Diagrams (LD), Function Block Diagrams (FBD), and
Sequential Function Charts (SFC). The first four languages implement code blocks, so called program organization
units (POU). The last language fills a different role, as it allows explicit structuring and description of control flow
between several POUs, aiming to simplify understanding of complex PLC programs. SFC is simply a state machine
description with steps and transitions.
Semantic grounding links data items to concepts with known meaning. This paper proposes a grounding of the
structuring concepts in a SFC program description. This allows capture of a machine-understandable view of PLC
program structure that can be further used for analysis, fault handling, synthesis, etc. Semantic state machine
descriptions have been around for a while. A review shows attempts within several non-PLC notations such as SDL
(Specification and Description Language) [2], UML (Universal Modelling Language) [3], and generic OWL (Web
Ontology Language) [4]. For PLC, semantic descriptions have been utilized for transformation from IEC 61131-3 to
IEC 61499 [5]. [6] presents lessons learned while constructing a 61131-compliant state machine ontology and
implementing PLCs supporting OPC Unified Architecture based on said ontology. Compared to [6], our ontology
focuses on capturing the structuring parts of the SFC, specifying configuration and coordination, and less on
capturing the execution model, though we can transform our representation into an executable description.
The SFC ontology presented is part of the knowledge integration framework (KIF [7, 8, 9, 10]). KIF has been
applied in several EU projects: SIARAS1, ROSETTA2, PRACE3, SMErobotics4, and SARAFun5, and aims to
increase robustness of data exchanges within a robot system by ensuring semantic compatibility through grounding.
Secondary uses include utilization of grounded concepts to aid specific phases of working with robots, such as
configuration management and programming. KIF aims to provide grounding for all information available in a
modern robot controller and is a continuously ongoing work. Knowledge available in KIF grounds its concepts in a
number of ontologies. These ontologies are stored in a server, called the KIF server, which is based on the RDF4J
semantic database [11]. Besides providing data grounding, the KIF server is also used as a database for semantic
instance information. The KIF server allows storing of tasks as instantiated SFC ontology concepts, where SFC
instances are created and manipulated through a domain-specific command line toolset, provided together with the
server [12, 13, 14].
The outline of this paper is as follows. Section 2 presents the 61131-3 sequential function chart notation in a
tutorial manner. Next, section 3 presents the SFC ontology concepts that are part of the paper contribution. Section 4
proceeds to present the KIF server and the basic tool-set used to manipulate SFC instances. Section 5 presents a
recent application for the SFC representation in the KIF framework. Finally, section 6 draws conclusions from the
presented work.

2. Sequential Function Charts (SFC) in IEC 61131-3

This section provides a brief introduction and summary of structuring concepts in sequential function charts as
they appear in IEC 61131-3. The authors recommend [15] as a good source of further information. For the 61131
family of languages, programs are organized in program organization units (POUs). A POU corresponds roughly to
a procedure or method in a regular programming language like C or Java. A POU holds declarations and instructions
programmed in one of the five defined programming languages. The SFC language is a graphical state machine
notation consisting of steps and transitions that is used in the instruction part of a POU. The SFC notation allows

1
SIARAS, Skill-based Inspection and Assembly for Reconfigurable Automation Systems. http://cordis.europa.eu/project/rcn/101641_en.html.
2
ROSETTA, RObot control for Skilled ExecuTion of Tasks in natural interaction with humans; based on Autonomy, cumulative knowledge and
learning. http://fp7rosetta.org.
3
PRACE, A Programmable Robot ApprentiCE. http://cordis.europa.eu/project/rcn/101641_en.html.
4
SMErobotics, The European Robotics Initiative for Strengthening the Competitiveness of SMEs in Manufacturing by integrating aspects of
cognitive systems. http://www.smerobotics.org.
5
SARAFun, Smart Assembly Robots with Advanced Functionalities. http://h2020sarafun.eu/.
294 Mathias Haage et al. / Procedia Manufacturing 11 (2017) 292 – 299

specification of parallel and sequential execution paths. Table 1 shows available transitions. Divergent paths
correspond to selection of one execution path. Simultaneous sequences correspond to parallel execution of several
execution paths.
SFC allow actions to be associated with steps, for instance calling other POUs when specific conditions are
fulfilled, expressed as action qualifiers. Table 2 shows what action qualifiers are available. Actions are encapsulated
in action blocks. An action block describes an action together with an action name, an action qualifier, possibly a
Boolean indicator variable and possibly an instruction list. The action name is assumed to be declared as a Boolean
variable within the POU. The qualifier defines action execution semantics. Within Table 2, Stored and Reset refers
to setting and resetting the Boolean value of the action name. The indicator variable, if present, is handled manually
in the action instruction list. Instruction list associated with an action block is executed continuously while the step is
active unless the qualifier says otherwise.

Table 1 SFC transition types, from [15].

Transition type Description

Single sequence Only one transition is available from a step.
Divergent path Several transitions are available from a step. The first transition to
become active, evaluated from left to right, selects the next step to
become active.
Divergent path with user-defined priority A number is associated with each transition. Transitions are
evaluated in priority order, lowest number first.
Divergent path under user control All transitions are assumed to be mutually exclusive, therefore
transition conditions are evaluated without specified order.
Convergence of sequences Several transitions are entering a step. When one of the entering
transitions are evaluates to true and the corresponding from step is
active, execution continues into the to step.
Simultaneous sequences A transition may connect to several steps and when triggered all
steps are executed in parallel.
Convergence of simultaneous sequences Several steps are connected to a transition. To continue execution all
from steps need to be active as well as the transition itself.
Sequence loop A transition refers back to a previous step.
Sequence skip A transition bypasses a step.

The SFC execution model marks steps and transitions as active or inactive. Steps are active if they contain an
execution token triggering actions according to qualifiers. Transitions are active when their condition is fulfilled.
The token wanders between steps along active transitions. Transitions are always directed, so there is a from step and
a to step. The active status is determined by evaluating a Boolean condition associated with the transition. Execution
always starts in an initial step, which receives a token upon program start. This concludes the, by necessity compact,
summary of the SFC language.

Table 2 Examples of action qualifiers, from [15].

Abbreviation Name Description

N Non-stored
R overriding Reset
S Stored
L time Limited
D time Delayed
P Pulse
The action is active while the step is active
The action is deactivated
The action is activated until Reset
The action is activated for a limited time
The action is activated after a delay if and only
if the step is still active
execute when step becomes active and inactive
 Mathias Haage et al. / Procedia Manufacturing 11 (2017) 292 – 299 295

3. SFC Ontology

This section presents the concepts of our SFC ontology. The aim is to present a small set of concepts that capture
the structuring properties of SFCs and provide a simple usage model that makes working with the concepts straight-
forward. The concepts should be SFC-based so that it is possible to refer to the standard as a final resource
determining semantic meaning. The presented concepts are based on the 2nd edition of the 61131-3 standard. There is
a 3rd edition available as of 2013, but to our knowledge there should be no significant impact on the presented
concepts.
The usage model is associative and open-ended. That is, arbitrary data instances, called entities below, can be
given SFC properties by establishing a relation to a chosen SFC concept, such as Step, Transition, etc. The model is
open-ended since we do not provide limits on, for instance, what relations can be used to associate an entity with a
Step. Left-out concepts, such as what is considered a transition condition, can thus be easily incorporated from other
ontologies. In RDF triplet terminology (<subject, predicate, object>), for some tuples we allow any predicate and for
others any object. We believe this approach will allow us to apply the ontology in a broader context, e.g., by being
able to bring in other ontologies to further specify open-ended “placeholders”, depending on context and situation.
Five concept categories are used in our coverage of SFC: steps, transitions, actions, declarations and, finally,
execution state. Table 3 shows an overview of the presented concepts. Each of the concepts has a corresponding URI
allowing it to be located in the ontology. Most of the SFC concepts belong to the coordination.owl ontology,
created for expressing behavior, while some (like e.g., Parameter) belong to the more general skill ontology
rosetta.owl. Both are available from our knowledge server KIF (http://kif.cs.lth.se/ontologies/XXXX.owl, where
XXXX stands for the ontology name) and are described in [9] and [10].
Steps, transitions and actions are all associated with entities through open-ended relations. Important SFC
relations, such as transition condition, action name, priority, and parameter are open-ended in their targeting. Action
blocks are created by association to an entity. The ontology provides relations to action name and indicator. The
qualifier is modelled as several relations, one per condition. It is required but not enforced, that the action block
entity is decorated with name and qualifier relations. The qualifier relation points to an open-ended action. Step
entities are associated with one or several action blocks through an open-ended relation. It is possible to enforce an
ordering of action blocks through the next action relation.
Divergent paths and simultaneous sequences are modelled as transitions augmented with divergence relations to
subsequent steps. For instance, an entity marked as simultaneous sequence, should have one entering transition and
one or several leaving steps. The convergence of the simultaneous sequence is modelled the opposite way. Divergent
path and convergence of divergent path is modelled the same way as a simultaneous sequence. However, since it
might be necessary to keep track of an ordering between leaving transition relations, a next relation is available.
Leaving out the next relation implies a divergent path under user control. User-defined priorities use a priority
relation. The ontology also allows capturing snapshots of execution state. Steps, transitions and action block entities
can all be decorated with one or several execution state relations.
296 Mathias Haage et al. / Procedia Manufacturing 11 (2017) 292 – 299

Table 3 Main SFC ontology concepts

Concept Kind Ontology reference

Step Object <http://kif.cs.lth.se/ontologies/coordination.owl#Step>
Initial step Object <http://kif.cs.lth.se/ontologies/coordination.owl#InitStep>

Transition Object <http://kif.cs.lth.se/ontologies/coordination.owl#Transition>

Entering transition Predicate <http://kif.cs.lth.se/ontologies/coordination.owl#hasPreviousStep>
Leaving transition Predicate <http://kif.cs.lth.se/ontologies/coordination.owl#hasNextStep>
Transition condition Predicate <http://kif.cs.lth.se/ontologies/coordination.owl#hasTransitionGuard>

Divergent path Object <http://kif.cs.lth.se/ontologies/coordination.owl#SelectionDivergence>

Simultaneous sequence Object <http://kif.cs.lth.se/ontologies/coordination.owl#SimultaneousDivergence>
Convergence of Object <http://kif.cs.lth.se/ontologies/coordination.owl#SelectionConvergence>
divergent path
Convergence of Object <http://kif.cs.lth.se/ontologies/coordination.owl#SimultaneousConvergence>
simultaneous sequence
Next step/transition Predicate <http://kif.cs.lth.se/ontologies/coordination.owl#hasNextTransition>
Priority Predicate <http://kif.cs.lth.se/ontologies/coordination.owl#hasPriorityOver>

Action block Object <http://kif.cs.lth.se/ontologies/coordination.owl#Action>

Action name
Action indicator
Action with qualifier
(N, R, L, D, …)
Next action
Predicate <http://kif.cs.lth.se/ontologies/coordination.owl#hasActionName>
Predicate <http://kif.cs.lth.se/ontologies/coordination.owl#hasActionIndicator>
Predicates <http://kif.cs.lth.se/ontologies/coordination.owl#hasActionQualifier>
Predicate <http://kif.cs.lth.se/ontologies/coordination.owl#hasNextAction>

Input parameter Object <http://kif.cs.lth.se/ontologies/rosetta.owl#InputParameter>

Output parameter Object <http://kif.cs.lth.se/ontologies/rosetta.owl#OutputParameter>
Input and output Object <http://kif.cs.lth.se/ontologies/rosetta.owl#Parameter>
parameter

Execution state Predicate <http://kif.cs.lth.se/ontologies/coordination.owl#ExecutionState>

Active / inactive step
Active / inactive
transition
Active / inactive action
Token
Objects Restriction on ExecutionState
Objects Restriction on ConditionOccurrence
Objects Restriction on ExecutionState
Object <http://kif.cs.lth.se/ontologies/coordination.owl#hasActivity>

4. KIF and SFC Instances

This section presents a tool set of command-line commands for creating and working with instances of the above
ontology to support task formation. The ontology is small but contains sufficient detail so that stored SFC instances
can be transformed into executable entities to allow for both analysis of existing SFCs and synthesis of new SFCs.
Table 4 shows a selection of available commands. The kifX commands are generic utilities for managing instance
data. sfcX commands target creation of SFC instances. Table 5 shows a script creating a small SFC.
 Mathias Haage et al. / Procedia Manufacturing 11 (2017) 292 – 299 297

Table 4 Sample commands for working with SFC instances [13, 14].

Command Description
kifrepository list and create new repositories
kifcontext list and create contexts within a repository. A context is a named graph within a repository. It
allows retrieval of data that belongs only to the context
kifstatement list and create new data within a context within a repository. Data is stated as RDF triples
organized as subject, predicate, object
kifupload upload a file in RDFXML format to KIF server
kifnode create nodes (simple entities) unique to a repository. Nodes are used as an aid when
constructing data structures such as SFC graphs in KIF
sfcstart create a start state for the state machine
sfcinput create an input variable for the state machine
sfcoutput create an output variable for the state machine
sfcstate create a state/step
sfcaction associate an action with a state
sfctransition create a transition between two states
sfclist print all data associated with a state machine as a comma-separated list of triples, one triple
per line

sfc2dot create an input file to the graphviz dot program to perform graphical layout of the state
machine
dot2jgrafchart create a xml file suitable as input to JGrafchart

Table 5 Creation of a simple SFC with one start step and one additional step through a bash script

REPO = kifrepository –n
START = sfcstart $REPO
STEP = sfcstate $REPO “My_step”
TRANSITION = sfctransition $REPO “My_transition” “true” $START $STEP

The KIF server itself has no means of executing the SFC instances. Therefore, the two commands at the bottom
of Table 6 transform the SFC instance into grafchart notation (a specific SFC dialect) suitable for execution in the
JGrafchart tool [16]. The reason to select grafchart for execution was the availability of KIF services that allow easy
manipulation (composition, modification, instantiation) of the representation.

Table 6 Transform a SFC into an executable and execute in jgrafchart as a bash script. Dot is used to layout the graph.

sfclist $REPO >statemachine.txt

sfc2dot statemachine.txt >statemachine.gv
dot -Tplain statemachine.gv >statemachine.plain
dot2jgrafchart statemachine.txt statemachine.plain >statemachine.xml
jgrafchart -s statemachine.xml

5. Application

The KIF server with presented SFC ontology and associated toolset was applied in a teaching by demonstration
scenario within the SARAFun EU-project. Only a brief description is provided here, since the testbed is extensively
described in [17] (in this volume). The left image in Figure 1 shows a small parts assembly problem being
demonstrated by a human. An RGBD camera sensor acquires visual information including both color and depth
298 Mathias Haage et al. / Procedia Manufacturing 11 (2017) 292 – 299

images. This information is used for detecting and tracking assembly parts and hands of a human teacher during
demonstration. The tracking data is utilized by a key-frame extractor, which constructs semantic graph and
automatically extracts important stages of the assembly, called key-frames. The extracted key-frames are visualized
to the teacher through an HRI interface for confirmation. Here, the teacher may add or remove key-frames and insert
additional semantic information, while an ongoing animation of the demonstrated assembly is shown in a simulation
environment based on the position of the assembly parts in the selected key-frames. Once the selection is finalized
based on the user’s feedback, the information stored in the key-frames is used for constructing a robot task. The task
consists of key-frames mapped to action sequences. Figure 2 shows the SFC resulting from this process. Note that
the text in the states is just automatically generated uuid, not intended to be read by humans. The right image of
Figure 1 shows the robot testbed for execution of the SFC. In the teaching by demonstration architecture, the
captured demonstration is transformed into a SFC task representation stored in KIF server and from there executed
using JGrafchart and ROS. Since the JGrafchart tool is not running as part of ROS, a software bridge allows the KIF
and ROS services to communicate. In ROS, a dispatcher service interprets issued SFC actions and calls
corresponding services with appropriate parameters for discrete- and continuous-time motion control. Keeping an
open grounded representation of the task is essential for the next phase of the prototype, which will include several
approaches for the operator to modify and interact on-the-fly with the demonstrated task for error correction,
adjustments, etc., requiring several modules in the testbed to interact, understand and manipulate the task
dynamically.

Figure 1 Testbed for teaching by human-only demonstration. Left: human demonstration. Right: robotic testbed.

Figure 2 SFC generated from human demonstration, with automatically generated uuid-s as state names.
 Mathias Haage et al. / Procedia Manufacturing 11 (2017) 292 – 299 299

6. Conclusion

The paper has presented an approach to extend an SFC ontology with a toolset for manipulation of SFC
instances. The ontology is a part of a larger knowledge integration framework for robotics, encompassing a set of
industrially-relevant robotic ontologies, and has been applied to capture task formation in a teaching-by-
demonstration scenario for small-parts assembly. SFC instances reflecting the task can be transformed into an
executable specification and then executed on the robot controller. A specially created ROS bridge allows ROS
services and actions to be called from the executed SFC. Altogether, KIF with its ontology server, JGrafchart as an
SFC execution environment, and ROS as a vast, well-organized library of robot-related utilities, form a development
environment enabling research on and prototyping of advanced knowledge-based solutions for industrial
applications.

Acknowledgment

The research leading to these results has received funding from the European Community’s Framework
Programme Horizon 2020 – under grant agreement No 644938 – SARAFun.

References

[1] IEC61131-3 Edition 2.0 (2003-01), TC/SC 65B, Programmable controllers - Part 3: Programming languages, http://www.iec.ch.
[2] M.B. Babac, M. Kunštić. SDL Ontology for Specifying Systems Based on Finite State Machines. In: Demazeau Y. et al. (eds) Trends in
Practical Applications of Agents and Multiagent Systems. Advances in Intelligent and Soft Computing, vol 71. Springer, Berlin, Heidelberg.
2010.
[3] Youcef Belgueliel, Mustapha Bourahla, and Mourad Brik, "Towards an Ontology for UML State Machines," Lecture Notes on Software
Engineering vol. 2, no. 1, pp. 116-120, 2014.
[4] Peter Dolog. The Ontology for State Machines. http://people.cs.aau.dk/~dolog/fsm/. Accessed 16 February 2017.
[5] D. Wenbin, V. Dubinin, & V. Vyatkin. Migration from PLC to IEC 61499 Using Semantic Web Technologies. IEEE Transactions on Systems,
Man, And Cybernetics: Systems, 44(3), 277-291. doi:10.1109/TSMCC.2013.2264671, 2014.
[6] Wim Pessemier, Geert Deconinck, Gert Raskin, Philippe Saey, Hans Van Winckel. Developing a PLC-friendly state machine model: lessons
learned. Proc. SPIE 9152, Software and Cyberinfrastructure for Astronomy III, 915208 (July 18, 2014); doi:10.1117/12.2054881.
[7] Jacek Malec. KIF ontologies. https://git.cs.lth.se/jacek/skill-ontologies. Accessed 16 February 2017.
[8] Ludwig Jacobsson. A module-based skill ontology for industrial robots. Master’s thesis, Department of Computer Science, Faculty of
Engineering LTH, Lund University, 2015.
[9] Ludwig Jacobsson, Jacek Malec, and Klas Nilsson. Modularization of skill ontologies for industrial robots. In 47th International Symposium
on Robotics, ISR 2016, pages 181– 186. VDE Verlag gmbh Berlin, 2016.
[10] Maj Stenmark and Jacek Malec. Knowledge-Based Instruction of Manipulation Tasks for Industrial Robotics. Robotics and Computer
Integrated Manufacturing, 33:56–67, 2015.
[11] RDF4J homepage. http://rdf4j.org/. Accessed 16 February 2017.
[12] Mathias Haage. Installation instructions for creating a running KIF server instance. https://git.cs.lth.se/mathias/KIF_installation. Accessed 16
February 2017.
[13] Mathias Haage. KIF utilities for execution of SFC instances in JGrafchart tool. https: //git.cs.lth.se/mathias/grafchart_utils. Accessed 16
February 2017.
[14] Mathias Haage. KIF utilities for manipulation of SFC instances. https://git.cs. lth.se/mathias/kif_utils. Accessed 16 February 2017.
[15] Karl-Heinz John and Michael Tiegelkamp. IEC 61131-3: Programming Industrial Automation Systems: Concepts and Programming
Languages, Requirements for Programming Systems, Decision-Making Aids. Springer Verlag, 2014.
[16] A. Theorin, Sequential Control Language for Industrial Automation. PhD Thesis. Department of Automatic Control, Lund University, 2014.
[17] Mathias Haage, Grigoris Piperagkas, Christos Papadopoulos, Ioannis Mariolis, Jacek Malec, Yasemin Bekiroglu, Mikael Hedelind, and
Dimitrios Tzovaras. Teaching assembly by demonstration using advanced human robot interaction and a knowledge integration framework.
In 27th International Conference on Flexible Automation and Intelligent Manufacturing, 2017.