White Paper On Hybrid Approach For ADAS Data Fusion Algorithm Development

A Hybrid Approach for ADAS Data Fusion Algorithm
Development - From High-Level Prototypes to ECUs

Abstract
Current development of Advanced Driver Assistance Systems (ADASs) and automated
vehicles aims to address more and more complex scenarios. Not only the variety of
situations, also the area covered by sensors increases. This in turn is dealt with by
more and more complex sensors. In addition to the increased number of sensors even
the algorithms used for the signal processing are getting more complex to deal with
more challenging tasks given by innovative ADAS functions. This puts special focus
on the tool chain for ADAS development, as the overall time budget from an
algorithmic prototype to the series electronic control unit (ECU) remains the same, if
not even decreases. The paper at hand outlines design considerations for a data
fusion system of an ADAS and describes a novel methodology to support the
development process until series deployment. Furthermore, the novel hybrid
development approach is classified with respect to state-of-the-art methodologies.
The table below summarizes the conclusions.
Development
Transition to
Development
approach
series system
time for data
Flexibility
Algorithm
Sensor
support
interface
fusion
Direct ECU
implementation
Coding-based
Model-based
Hybrid
Not necessary
Manual reimplementation
Template-based
code generation
Novel code
generation
via
long
very low
none
ECU; direct
long
low
none
PC; direct
medium
low
limited
short
high
full
additional
hardware
PC; direct
Keywords: Data Fusion, Data Logging, ADAS, Perception Model, Bayesian Filtering,
Code Generation, Rapid Prototyping
Author: Norman Mattern, Head of Products & Services, BASELABS GmbH

Introduction
Current developments in the domain of Advanced Driver Assistance Systems (ADASs)
handle more and more complex functions. This increased complexity leads to more
sensors being mounted on vehicles and puts a focus on the fusion of data of these
numerous sensors. This data fusion offers different degrees of freedom but also
algorithmic and implementation challenges. For that reason, the next section lists
typical approaches how the algorithmic processing of a system using multiple sensors
can be partitioned. Independent from the actual partitioning, the algorithms need to
be designed as well. They finally need to be available for an automotive electronic
control unit (ECU) for the later use in series production. Therefore, the paper at hand
additionally describes different scenarios including a novel approach how the
transition from a prototype to an ECU can be supported by tools in the context of
signal processing for ADAS sensors like radars, laser, and cameras. In the last section,
tools for the design of data fusion algorithms are compared.
Partitioning Approaches for Sensor Data Fusion System

The presence of various sensors, the capability to perform processing of data either in
the sensors or a separate ECU, and the interaction of OEMs and suppliers create some
degrees of freedom for the design of a data fusion system for ADAS. At the same time,
the algorithms to handle the increased amount of sensor data become more complex.
Additionally to that increasing complexity, different aspects for a data fusion system
need to be considered as well:
1. Fusion of more than one sensor, e.g. several radar and camera sensors
2. Ability to exchange sensor models and sensors suppliers
3. Ability to configure even the number and positions of sensors
4. Different applications and thus different models shall be implemented
The following sections describe typical partitioning approaches for such sensor data
fusion systems for ADAS.
www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

State of the Art Monolithic System

In the past, two tendencies could be observed:
1. Implementation of tracking and fusion functionality shifted to tier 1 supplier
2. As a result, tracking was also handled by those systems
However, that contradicts the aim to be flexible in choosing the supplier. While a
single sensor system such as a radar-only ACC can be treated as a monolithic
component as shown in Figure 1, a system using data of a multitude of sensors
cannot. For such systems, a more modular solution is desirable.
Radar
sensor data
Algorithm
(Tracking)
tracked objects
Algorithm
(Function)
Monolithic system of function supplier
Figure 1: Monolithic system architecture
Combining Track Lists - Track-to-Track Fusion

Depending on the requirements of an OEM, different architectures are possible. A
typical example is that different sensors are delivered by different suppliers. The
sensors may be stand-alone such as cameras or multi-sensor systems, for example
radars observing the vehicles surrounding. For the combination of the information of
the different sensors and sensor systems, their raw data (that has not been subject to
pre-processing in the sensor) is not used. Instead, the data that is provided by the
sensors is usually already pre-processed and tracking algorithms are applied already
in the sensor, resulting in object lists. As a result, putting together the information of
the different sensors means actually combining lists of tracked objects. This is
commonly referred to as track-to-track fusion. Figure 2 depicts an example of such
an architecture.
An advantage of this architecture is that the amount of data transferred between the
stages is limited. Furthermore, the combination of separate sensors requires less
sensor domain specific knowledge, as data is provided ready-to-use. At the same
time this pre-processing in terms of tracking always means reduction of information,
as only tracks considered as valid are provided to the track-to-track fusion. However,
in todays more and more complex scenarios every information is valuable and

therefore centralized processing of all sensor information incl. raw data can be an
advantage. The central data fusion architecture offers this benefit and is addressed in
the next paragraph.
Stereo Cam
sensor data
Algorithm
(Tracking)
tracked
objects
System of camera supplier 1
Mono Cam
sensor data
Algorithm
(Tracking)
tracked
objects
Algorithm
(Track-to-track
Fusion)
System of camera supplier 2
Short Range
Radar 1
sensor data
Short Range
Radar 2
sensor data
Long Range
Radar
sensor data
Algorithm
(Data Fusion /
Tracking)
Algorithm
(Function)
System of OEM
or supplier
tracked
objects
Multi sensor system of radar supplier
tracked
objects
System of OEM
or supplier
Figure 2: Track-to-track fusion architecture
Combining Sensor Raw Data Central Data Fusion

Another approach of partitioning is to conduct the fusion of the sensor raw data in
one central data fusion instance. Clearly, that instance directly takes the
measurements from the sensors (such as vehicle positions in the image of a camera or
azimuth, range and range rate from a radar) instead of pre-processed data. Figure 3
illustrates the partitioning in that architecture.
An advantage of this configuration is that all information delivered by the sensors is
also available to the data fusion, which is the optimal basis for such a component.
However, the development and processing effort is higher.

Stereo Cam
sensor data
Supplier 1
Mono Cam
sensor data
Algorithm
(Data Fusion /
Tracking)
Supplier 2
Radar 1
tracked
objects
Algorithm
(Function)
System of OEM
or supplier 6
sensor data
Supplier 3
Radar N
sensor data
Supplier 4
System of OEM
or supplier 5
Figure 3: Central data fusion architecture
Development and implementation of tracking and sensor data fusion

algorithms for ADAS
While the last chapter outlined different approaches to partition the processing of
data, i.e. one or multiple algorithms, the next chapter deals with the algorithms
implementation. That includes the design of the algorithm itself as well as the
transition to an automotive ECU as used in series systems.
Commonly, the development of systems for tracking and sensor data fusion for ADAS
starts with the implementation of an algorithmic prototype. After the functionality is
proven, that system is transformed to a representation which can be run on an ECU.
This representation finally needs to meet requirements specific to the automotive
domain, e.g. MISRA conformity.
General prototype development approaches

High level prototypes are used to get fast first results of the behaviour of the data
fusion system which shall be developed. Typical approaches are:
1. Model-based development
2. Coding-based development

3. Hybrid development
The following sections will describe the different approaches in more detail.
Furthermore, the involved system development set-ups will be presented on a
simplified example. This example assumes that a certain algorithm, be it tracking or
other data fusion applications, may be designed and finally brought to an automotive
ECU. The hypothetical system uses a sensor which is connected via CAN bus.
Model-based development
As the name indicates, model-based development uses models, i.e. abstract
representation of knowledge. Simple examples for such models are PID controller or
state machines, which are configured to solve a certain problem. Pure model based
development utilizes tools like Simulink [1]. However, algorithms for sensor data
fusion for ADASs often include the perception of the environment. Prominent example
are multi object trackers (MOT), usually implemented by Bayes filters. One example
for that would be a bank of Kalman filters with existence estimation and measurement
association. Such systems offer many degrees of freedom in their design, as system
and measurement state and models may be chosen. They are therefore very complex
to handle in a model-based approach.
Figure 4 illustrates typical set-ups involved in the model-based development process.
Usually, logging tools (e.g. Vector CANalyzer [2]) are used to provide sensor data from
a test vehicle (Set-up M1). This data is used for offline playback algorithm design in a
model-based
development
environment
like
Simulink
(Set-up
M2).
If
the
model-based algorithm shall be tested in real-world environments, so-called rapid

control prototyping (RCP) hardware like the dSpace MicroAutoBox [3] offer a way to
connect the algorithms to the sensors, as it provides for example CAN bus access
(Set-up M3). The configured model is then transferred from the model-based
development to the RCP platform where it is executed. Any adoptions to the models
are usually made in the model-based development environment, which generates an
alternating process between the set-ups M2 and M3. Once the algorithm is fixed,
so-called series-coder products can generate code to be run on an automotive ECU.
However, series-coder may support not all models for code-generation, but only a
defined sub-set.

Set-up M1
Recording of sensor data
Data
recorder
CAN reader
CAN data logging tool

CAN bus
Vehicle-under-test
Sensor
data
Offline data
player
Set-up M2
Playback of sensor data
Development of algorithms
Algorithm
Model-based development environment on PC

Code generation for rapid control
prototyping platform
CAN Reader
Algorithm
Feedback from
the real-world
Set-up M3
Test of algorithms during
their development
CAN writer
Rapid control prototyping hardware platform

CAN bus
CAN bus
Vehicle-under-test
Code generation for

automotive ECU platform
CAN Reader
i/f
Algorithm
i/f
(C-code)
Set-up M4
Validation of the system
CAN writer
ECU with automotive operating system
CAN bus
CAN bus
Vehicle-under-test
Figure 4: Model-based algorithm development approach
Coding-based development
The opposite of that is a pure coding-based approach, which is illustrated in Figure 5.
In that case the system is implemented in a high-level framework, usually given by
C++, C#, Java, or MATLAB [4] libraries (Set-up C1). However, even if supporting
software libraries or other code may exist in that case, in general the whole
complexity of the models needs to be handled by the programmer.

CAN reader
Algorithm (highlevel language)
CAN data
CAN data
CAN writer
Data player/
recorder
PC-based high-level software framework
CAN bus
CAN bus
Vehicle-under-test
Manual re-implementation of
algorithm and interface (i/f)
CAN reader
CAN data
i/f
Algorithm
CAN data
i/f
(C-code)
CAN writer
Data player/
recorder
CAN bus
Set-up C1
Recording of
sensor data
Playback of
sensor data
Development of
algorithms
Test of
algorithms
during their
development
Set-up C2
Playback of
sensor data
Test of
algorithms
code
CAN bus
Vehicle-under-test
Manual transfer of
algorithms code
CAN Reader
i/f
Algorithm
i/f
(C-code)
Set-up C3
Validation of the
system
CAN writer
ECU with automotive operating system
CAN bus
CAN bus
Vehicle-under-test
Figure 5: Coding-based development approach
An advantage is that C, C++ and C# offer efficient ways to access hardware of sensors.
Examples for such frameworks are ADTF [5], BASELABS Connect [6], or RTMaps [7].
Aim of the algorithm development process is (C-)code running on an automotive ECU.
For that reason, the code from set-up C1, yielding efficiency of development time
using high-level languages, needs to be transformed to that representation, as
features like using runtime-polymorphism of object oriented programming or
dynamic memory allocation are commonly not permitted by the MISRA standard used
for automotive ECU. Additionally to the manual re-implementation of the actual code
of the algorithm, even the environment-specific interfaces need to be implemented.
Those interfaces transform the data from a level of separate scalar signals (e.g.
radar-target 1, azimuth value in rads) on the CAN bus (for example given by a
dbc-file) to a data structure representation which can be used by the algorithm code.

ECU implementation approaches

Systems described by a model-based approach can be transformed to an
ECU-capable representation by so-called Coder products, which generate C-code.
Even though this transformation is convenient and fast, the disadvantages of the
model-based approach remain.
For the coding-based approach, the transition from prototypical code to C-code is
usually performed manually. In case of C++ this usually means complete re-write. In
case of C the code still needs to meet requirements like no dynamic memory
allocation or no function pointers, which finally comes close to a re-write. In any case
this high-level to low-level transformation is very time consuming, which make the
requirements to both the hand-coded high-level prototype and low-level code even
higher.
Hybrid development approach

Hybrid development offers a way to combine the advantages of both model-based
and coding-based development. An SDK like BASELABS Create [6] provides
functionality to configure a data fusion system for the needs of the particular ADAS
algorithm architecture, be it monolithic, track-to-track fusion, or central. By means of
that, only valid filters can be created, still giving developers and researchers the yield
from the modelling capabilities and design efficiency of high-level languages like C#.
Already existing code (e.g. algorithms implemented in C++) can be used as well. At
the same time, the product BASELABS Code [6] offers a way to instantly generate the
necessary C-code for automotive ECUs from the algorithms implemented before.
Figure 6 shows the set-ups involved in the hybrid-development approach. The
Set-ups M1 and C1 (of the coding-based approach) are comparable, as both take care
of the sensor access, data record and replay, and algorithm development. However, in
the hybrid-development approach, the developed algorithm components can be
based on the powerful C# language as well as the BASELABS Create libraries for ADAS
specific tracking algorithms [6]. This enables the developer to directly generate
C-code as well as necessary interfaces. The actual code with attached interfaces is
automatically integrated in the high-level configuration of the software framework
BASELABS Connect. By means of that, the generated C-code can be stimulated by both
recorded and real-world data instantly, without the need to change the system design
or the development environment. Besides the effect of a consolidated tool chain,

development and test time can be reduced significantly. In addition, the interface
generation is not limited to BASELABS Connect. The code and configuration
generation can be adopted not only to other high-level rapid prototyping frameworks,
but also to such frameworks targeting automotive ECUs such as AUTOSAR.
Sensor data
Algorithm (C#-Code)
CAN reader
Data player/
recorder
In-House-Framework
Set-up H1
Recording of
sensor data
Playback of sensor
data
Development of
algorithms
Test of algorithms
during their
development
Result
CAN writer
Automatic generation of
C-code und interfaces
Display
Algorithm (C-Code)
ADTF

CAN bus
CAN bus
Vehicle-under-test
Direct re-use of the

algorithm s code
CAN reader
Set-up H2
Validation of the
system
CAN writer
Algorithm (C-Code)
RTOS, AUTOSAR, etc.
ECU
CAN bus
CAN bus
Vehicle-under-test
Figure 6: Novel hybrid development approach
By using the hybrid-development approach not only the given advantages for a
high-level prototype are exploited, the approach also enables:
1. The use of the same code in high-level prototype and ECU system
2. Quick changes of ECU code even in complex ADAS algorithm architectures
3. Reduced system development time
Comparison of the approaches

While
both
model-based
development
and
coding-based
development
are
successfully used in the practical development of automotive software, they both have
specific advantages and disadvantages. In the following, the flexibility in design and
10

the development velocity will be compared. Furthermore, the tool-support of

signal-processing algorithms specific for the ADAS domain will be listed.
Flexibility and Velocity

Table 1 summarizes different aspects of the basic algorithms design and
development process. For example, while template-based code generation of
model-based approaches offer less error prone results in the series systems, the
flexibility in changing even complex models and architectures in data fusion
algorithms for ADAS may be limited. At the same time, real-time access to sensors
may require additional hardware. On the other side, coding-based development of
algorithms offers these possibilities. However, it generally lacks model-checks and
the availability of code generators. The hybrid development approach provided by the
BASELABS tools [6] unifies the advantages of both worlds: high flexibility, direct
sensor interfaces without additional hardware, strong algorithm support and a short
development time. In addition, it reduces the number of required set-ups in the
design and implementation process.
Development
Transition to
Development
approach
series system
time for data
Flexibility
Algorithm
Sensor
support
interface
fusion
Direct ECU
implementation
Coding-based
Model-based
Hybrid
Not necessary
Manual reimplementation
Template-based
code generation
Novel code
generation
via
long
very low
none
ECU; direct
long
low
none
PC; direct
medium
low
limited
short
high
full
additional
hardware
PC; direct
Table 1: Comparison of different approaches.
11

Tool-Support for data fusion algorithm development

In the following, different tools for the development of ADAS algorithms shall be
compared. Additionally to the general user support in signal processing, special focus
is put on typical ADAS-domain specific challenges.
The basis of the algorithm blocks in outlined in Figure 1 - Figure 3 is state estimation,
typically using Bayes filters [8]. Those filters are used to estimate the state of one
single instance of an object. In that case, one filter consists of a system and at least
one measurement model, where the internal state of the filter can be represented by
different means, i.e. Bayes filter implementations. Table 2 summarizes the
tool-support for Bayesian state estimation in terms of ready-to-use algorithms and
models.
System/State
MATLAB R2013a
BASELABS Create
Linear
Linear
transition model
Space
Representation
Non-linear
Vector of floating point
numbers
Dedicated type for spaces

o Dimension names
o Dimension units ([m], [m/s], etc)
Bayes filter
Kalman filter
implementations
Kalman filter
Extended Kalman filter
Unscented Kalman filter
Particle filter
Build-in models
Linear
Linear
Non-linear and automotive specific
Table 2: Available tool-support on Bayesian state estimation
While one single filter is of course suitable to track only one target at the first stage,
additionally functionality is added to the algorithmic system to estimate the states of
multiple objects. That contains basically the association of particular measurements
from the sensors to certain tracks as well as the evaluation of the existence of tracks.
Furthermore, as real-world sensors are not perfect, clutter needs to be modelled as
12

well. Table 3 summarizes how those challenges of multiple-object tracking are

handled by tools providing support on that topic.
Measurement
Association Algorithms
MATLAB R2013a
BASELABS Create
Hungarian
Single local nearest neighbor
assignment
(Complexity O(n))
Existence Estimation/
(Complexity O(n))
Multiple local nearest neighbor (O(n))
Sequential Probability ratio Testing
Clutter Handling
(SPRT)
-
Integrated Probabilistic Data

Association (IPDA)
generalized probabilistic data
association (GPDA)
Supporting
functionality
Gating
-
Detection models
Persistence models
Table 3: Available tool-support on multiple-object tracking
Finally, the application of multiple-object tracking to ADAS brings some typical

additional tasks, as different sensors with potentially overlapping field-of-views may
be used for more-and-more complex systems. Furthermore, different levels of
abstraction might be used for the sensor interfaces. Table 4 summarizes the
tool-support on ADAS specific multiple-object tracking using multiple sensors like
outlined in Figure 2 and Figure 3.
13

MATLAB R2013a
Number of sensors/
measurement models
Track-to-Track fusion
1
-
Sensor field-of-view
BASELABS Create
Arbitrary
Covariance intersection
Different field-of-views of multiple
support
sensors including
-
o Overlapping regions
o Non-overlapping regions
o Region dependent detection and
persistence models
Table 4: Available tool-support on ADAS-specific data fusion systems using multiple sensors
Conclusion
Although the development of ADAS systems is specific in terms of the system
components (e.g. the sensors and ECU used in the system), the signal processing
algorithms for ADAS are system-independent on a mathematical level. To benefit
from this inherent independency, the underlying architecture needs to be sufficiently
generalized. Then, the algorithm development can be supported by the right tools.
The BASELABS tools focus on the automotive sector and provide domain specific
models and functionality. That is, data logging and handling on the one side as well as
comprehensive support by BASELABS Create in the development of the actual
algorithms for the data fusion, as it is based on a generalized architecture.
Each BASELABS tool can be used on its own to support engineers and developers
during particular process stages. The combination of the tools yields a unique
end-to-end chain from high-level algorithm and perception model design down to an
automotive ECU. In the hybrid approach, less development and implementation
set-ups are required compared to the other approaches. This in turn may significantly
decrease development time and error-proneness and at the same time increase the
users flexibility as well as the overall system quality.
14

References
1. The Mathworks (2014), Simulink Simulation and Model-based Design,
http://www.mathworks.com/products/simulink/, last accessed 2014/01/14
2. Vector Informatik GmbH (2014), CANalyzer,
http://vector.com/vi_canalyzer_en.html, accessed 2014/01/14
3. dSpace GmbH (2014), dSpace MicroAutoBox,
http://www.dspace.com/en/inc/home/products/hw/micautob.cfm, accessed
2014/01/14
4. The Mathworks (2014) , MATLAB The Language of Technical Computing,
http://www.mathworks.com/products/matlab/, last accessed 2014/01/14
5. Elektrobit Cooperation (2014), Elektrobit: driver-assistance : EB Assist ADTF,
http://automotive.elektrobit.com/home/driver-assistance-software/eb-assist-a
dtf.html, accessed 2014/01/14
6. BASELABS GmbH (2014), Products BASELABS, http://www.baselabs.de, accessed
2014/01/14
7. Intempora S.A. (2014), RTMaps Overview,
http://www.intempora.com/rtmaps4/rtmaps-software/overview.html, accessed
2014/01/14
8. Sebastian Thrun, Wolfram Burgard, Dieter Fox, Probabilistic Robotics, MIT Press
2005
15

White Paper On Hybrid Approach For ADAS Data Fusion Algorithm Development

Caricato da

Informazioni sul documento

Titolo originale

Copyright

Formati disponibili

Condividi questo documento

Condividi o incorpora il documento

Opzioni di condivisione

Hai trovato utile questo documento?

Questo contenuto è inappropriato?

Copyright:

Formati disponibili

White Paper On Hybrid Approach For ADAS Data Fusion Algorithm Development

Caricato da

Copyright:

Formati disponibili

A Hybrid Approach for ADAS Data Fusion Algorithm

Development - From High-Level Prototypes to ECUs

time for data

A Hybrid Approach for ADAS Data Fusion Algorithm

Partitioning Approaches for Sensor Data Fusion System

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

A Hybrid Approach for ADAS Data Fusion Algorithm

State of the Art Monolithic System

Monolithic system of function supplier

Figure 1: Monolithic system architecture

Combining Track Lists - Track-to-Track Fusion

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

A Hybrid Approach for ADAS Data Fusion Algorithm

System of camera supplier 1

System of camera supplier 2

Multi sensor system of radar supplier

Figure 2: Track-to-track fusion architecture

Combining Sensor Raw Data Central Data Fusion

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

A Hybrid Approach for ADAS Data Fusion Algorithm

Figure 3: Central data fusion architecture

Development and implementation of tracking and sensor data fusion

General prototype development approaches

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

A Hybrid Approach for ADAS Data Fusion Algorithm

model-based algorithm shall be tested in real-world environments, so-called rapid

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

A Hybrid Approach for ADAS Data Fusion Algorithm

CAN data logging tool

Model-based development environment on PC

Rapid control prototyping hardware platform

Code generation for

ECU with automotive operating system

Figure 4: Model-based algorithm development approach

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

A Hybrid Approach for ADAS Data Fusion Algorithm

Algorithm (highlevel language)

ECU with automotive operating system

Figure 5: Coding-based development approach

A Hybrid Approach for ADAS Data Fusion Algorithm

ECU implementation approaches

Hybrid development approach

A Hybrid Approach for ADAS Data Fusion Algorithm

PC-based high-level software framework

Direct re-use of the

RTOS, AUTOSAR, etc.

Figure 6: Novel hybrid development approach

Comparison of the approaches

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

A Hybrid Approach for ADAS Data Fusion Algorithm

the development velocity will be compared. Furthermore, the tool-support of

Flexibility and Velocity

time for data

Table 1: Comparison of different approaches.

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

A Hybrid Approach for ADAS Data Fusion Algorithm

Tool-Support for data fusion algorithm development

Dedicated type for spaces

Table 2: Available tool-support on Bayesian state estimation

www.baselabs.de | info@baselabs.de | +49 371 3371 51 10

A Hybrid Approach for ADAS Data Fusion Algorithm

well. Table 3 summarizes how those challenges of multiple-object tracking are

Single local nearest neighbor