Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
m
pl
im
en
ts
of
Reactive
Microservices
Architecture
Jonas Bonr
Reactive Microservices
Architecture
Design Principles
for Distributed Systems
Jonas Bonr
Beijing
Tokyo
First Edition
978-1-491-95934-3
[LSI]
Table of Contents
1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Services to the Rescue
Slicing the Monolith
SOA Dressed in New Clothes?
3
3
5
8
11
12
13
17
22
28
30
32
34
35
38
41
42
47
iii
CHAPTER 1
Introduction
Traditional enterprise systems are designed as monolithsall-inone, all-or-nothing, difficult to scale, difficult to understand and dif
ficult to maintain. Monoliths can quickly turn into nightmares that
stifle innovation, progress, and joy. The negative side effects caused
by monoliths can be catastrophic for a companyeverything from
low morale to high employee turnover, from preventing a company
from hiring top engineering talent to lost market opportunities, and
in extreme cases, even the failure of a company.
The war stories often sound like this: We finally made the decision
to make changes to our Java EE application, after seeking approval
from management. Then we went through months of big up-front
design before we eventually started to build something. But most of
our time during construction was spent trying to figure out what the
monolith actually did. We became paralyzed by fear, worried that
one small mistake would cause unintended and unknown side
effects. Finally, after months of worry, fear, and hard work, the
changes were implementedand hell broke loose. The collateral
damage kept us awake for nights on end while we were firefighting
and trying to put the pieces back together.
Does this sound familiar?
Experiences like this enforce fear, which paralyzes us even further.
This is how systems, and companies, stagnate. What if there was a
better way?
Youve got to start with the customer experience and work back
towards the technology.
Steve Jobs
Chapter 1: Introduction
1 I am using the word Microservice and service interchangeably throughout this docu
Chapter 1: Introduction
The world of the software architect looks very different today than it
did 10-15 years ago when SOA emerged. Today, multi-core process
ors, cloud computing, mobile devices and the Internet of Things
2 SOA is Dead; Long Live Services by Anne Thomas, VP and Distinguished Analyst at
Gartner, Inc.
(IoT) are emerging rapidly, which means that all systems are dis
tributed systems from day onea vastly different and more chal
lenging world to operate in.
As always, new challenges demand a new way of thinking and we
have seen new systems emerge that are designed to deal with these
new challengessystems built on the Reactive principles, as defined
by the Reactive Manifesto.3
The Reactive principles are in no way new. They have been proven
and hardened for more than 40 years, going back to the seminal
work by Carl Hewitt and his invention of the Actor Model, Jim Gray
and Pat Helland at Tandem Systems, and Joe Armstrong and Robert
Virding and their work on Erlang. These people were ahead of their
time, but now the world has caught up with their innovative think
ing and we depend on their discoveries and work more than ever.
What makes Microservices interesting is that this architecture has
learned from the failures and successes of SOA, kept the good ideas,
and re-architected them from the ground up using Reactive princi
ples and modern infrastructure. In sum, Microservices are one of
the most interesting applications of the Reactive principles in recent
years.
done so already, I recommend that you read it now because this book rests on the foun
dation of the Reactive principles.
Chapter 1: Introduction
CHAPTER 2
1 For an insightful discussion on the problems caused by a mutable state, see John
Backus classic Turing Award Lecture Can Programming Be Liberated from the von
Neumann Style?
2 Neil Gunters Universal Scalability Law is an essential tool in understanding the effects
https://en.wikipedia.org/wiki/Bulkhead_(partition).
Some people might come to think of the Titanic as a counterexample. It is actually an interesting study4 in what happens when
you dont have proper isolation between the compartments and how
that can lead to cascading failures, eventually taking down the whole
system. The Titanic did use bulkheads, but the walls that were sup
pose to isolate the compartments did not reach all the way up to the
ceiling. So when 6 out of its 16 compartments were ripped open by
the iceberg, the ship started to tilt and water spilled over from one
compartment to the next, until all of the compartments were filled
with water and the Titanic sank, killing 1500 people.
Resiliencethe ability to heal from failuredepends on compart
mentalization and containment of failure, and can only be achieved
by breaking free from the strong coupling of synchronous commu
nication. Microservices communicating over a process boundary
using asynchronous message-passing enable the level of indirection
and decoupling necessary to capture and manage failure, orthogo
nally to the regular workflow, using service supervision.5
Isolation between services makes it natural to adopt Continuous
Delivery. This allows you to safely deploy applications and roll out
and revert changes incrementallyservice by service.
Isolation also makes it easier to scale each service, as well as allowing
them to be monitored, debugged and tested independentlysome
thing that is very hard if the services are all tangled up in the big
bulky mess of a monolith.
4 For an in-depth analysis of what made Titanic sink see the article Causes and Effects of
5 Process (service) supervision is a construct for managing failure used in Actor lan
guages (like Erlang) and libraries (like Akka). Supervisor hierarchies is a pattern where
the processes (or actors/services) are organized in a hierarchical fashion where the par
ent process is supervising its subordinates. For a detailed discussion on this pattern see
Supervision and Monitoring.
10
Act Autonomously
Insofar as any agent acts on reason alone, that agent adopts and acts
only on self-consistent maxims that will not conflict with other
maxims any such agent could adopt. Such maxims can also be
adopted by and acted on by all other agents acting on reason alone.
Law of Autonomy by Immanuel Kant
6 Our definition of a promise is taken from the chapter Promise Theory from Thinking
in Promises by Mark Burgess (OReilly), which is a very helpful tool in modeling and
understanding reality in decentralized and collaborative systems. It shows us that by
letting go and embracing uncertainty we get on the path towards greater certainty.
Act Autonomously
11
The Unix philosophy7 and design has been highly successful and still
stands strong decades after its inception. One of its core principles is
that developers should write programs that have a single purpose, a
small well-defined responsibility and compose well with other small
programs.
This idea was later brought into the Object-Oriented Programming
community by Robert C. Martin and named the Single Responsibil
ity Principle (SRP),8 which states a class or component should only
have one reason to change.
There has been a lot of discussion around the true size of a Micro
service. What can be considered micro? How many lines of code
can it be and still be a Microservice? These are the wrong questions.
Instead, micro should refer to scope of responsibility, and the
guiding principle here is the the Unix philosophy of SRP: let it do
one thing, and do it well.
If a service only has one single reason to exist, providing a single
composable piece of functionality, then business domains and
responsibilities are not tangled. Each service can be made more gen
erally useful, and the system as a whole is easier to scale, make resil
ient, understand, extend and maintain.
7 The Unix philosophy is captured really well in the classic book The Art of Unix Pro
8 For an in-depth discussion on the Single Responsibility Principle see Robert C. Martins
12
13
10 Domain-Driven Design (DDD) was introduced by Eric Evans in his book Domain-
11 See Jay Kreps epic article The Log: What every software engineer should know about
12 Martin Fowler has done a couple of good write-ups on Event Sourcing and CQRS.
15
13 The quote is taken from Pat Hellands insightful paper Immutability Changes Every
thing.
16
persistence model for Microservices due to its natural fit with Asyn
chronous Message-Passing (see Figure 2-4).
17
18
14 As brilliantly explained by Joel Spolsky in his classic piece The Law of Leaky Abstrac
tions.
15 The fallacies of RPC has not been better explained than in Steve Vinoskis Conve
19
17 See the Integration section in Chapter 3 for a discussion on how to interface with clients
18 We are using Tyler Akidaus definition of streaming, A type of data processing engine
that is designed with infinite data sets in mind from his article The world beyond
batch: Streaming 101.
21
19 Location Transparency is an extremely important but very often ignored and under-
appreciated principle. The best definition of it can be found in the glossary of the Reac
tive Manifestowhich also puts it in context: http://www.reactivemanifesto.org/
glossary#Location-Transparency.
22
cation.
23
There are two types of reference locality: temporal, reuse of specific data; and spatial,
keeping data relatively close in space. It is important to understand and optimize for
both.
25
CHAPTER 3
27
1 As Albert Einstein proved in his 1905 paper On the Electrodynamics of Moving Bod
ies.
2 That fact that information has latency and that the speed of light represents a hard (and
3 For a discussion of the relativity of time, the present and the past in distributed sys
tems, and how we need change the way we model systems so that we can break free
from the illusion that we live in a globally consistent now, see my talk Life Beyond the
Illusion of Present YouTube video, 53:54, from a presentation at Voxxed Days, posted
March 6, 2016. You should also take time to read Justin Sheehys great article on the
topic There is No Now.
28
live happily under the illusion that time and the present is absolute
(see Figure 3-1).
4 If you have not experienced this first-hand then I suggest that you spend some time
5 Pat Hellands paper is essential reading for anyone building Microservices-based sys
29
Service Discovery
The greatest obstacle to discovery is not ignoranceit is the illu
sion of knowledge.
Daniel J. Boorstin
6 CP refers to Consistency and Partition Tolerance in the CAP Theorem, and means that
changed. So what was the gain in having strong consistency for that information?
9 AP refers to Availability and Partition Tolerance in the CAP Theorem, and means that
10 CRDTs is one of the most interesting ideas coming out of distributed systems research
Service Discovery
31
API Management
Be conservative in what you do, be liberal in what you
accept from others.
Jon Postel
12 For example session state, credentials for authentication, cached data, and so on.
13 Originally stated by Jon Postel in RFC 761 on TCP in 1980.
14 It has among other things influenced the Tolerant Reader Pattern.
15 There is a semantic difference between a service that is truly new, compared to a new
32
16 The API Gateway pattern has been used successfully by Netflix and Amazon API Gate
API Management
33
34
Integration
Nature laughs at the difficulties of integration.
Pierre-Simon Laplace
35
36
Figure 3-4. Managing faulty services with the Circuit Breaker pattern
Integration
37
Security Management
The users going to pick dancing pigs over security every time.
Security is a not a product, but a process.
Bruce Schneier
If someone asks us to make sure that not every service can call the
Billing service, what do we do then?
It is important to distinguish between authentication and authoriza
tion. Authentication is the process of making sure that a client
(human or service) is who she says she is (typically using a user
name and a password), while authorization is the process of allow
ing or denying the user to access a specific resource.
18 Defining the procedure for assessing use-cases is outside of the scope of this paper.
19 A good discussion on different client-side semantics of eventual consistencyinclud
38
Both are important to get right and need to work in concert. There
are many ways to make this work, each way with their own benefits
and drawbacks.
TLS Client Certificates, also called Mutual Authentication or Two
Way Authentication, can provide a very robust security solution for
inter-service authentication in which each service is given a unique
private key and certificate on deployment. In this strategy, it is not
only the server that is verifying the identity of the client, but the cli
ent verifying the identity of the server. This means its safe not only
from eavesdropping, but from a completely hostile network where
an attacker could potentially intercept and redirect requestssuch
as the Internet itself (see Figure 3-5). Communication over SSL is
safe from eavesdropping and on an open, well understood standard.
It is, however, complicated to manage, and benefits from support by
the underlying platform.
If the services are HTTP-based, they can make use of HTTPS Basic
Authentication. It is well understood and straightforward, but it can
be complicated to manage SSL certificates on all the machines and
the requests can no longer be cached by a reverse proxy.
One advantage is that it provides Two Way Authentication similar to
the Client Certificate solution, where client verifies the identity of
the server using the servers certificate before it sends the creden
tials, and the server verifies the identify of the client using the cre
dentials it sends.
Another approach is to use Asymmetric Request Signing. In this solu
tion, each service is given its own private key to sign requests with,
while the public keys for each service are made known the Service
Discovery service. The drawback is that as a proprietary solution, it
can be vulnerable to eavesdropping or request replay attacks if your
network has been compromised.
Security Management
39
20 As the character Mia Wallace stated in Quentin Tarantinos movie Pulp Fiction, and
Peter Bailis later used in his excellent talk Silence is Golden: Coordination-Avoiding
System Design.
41
You have to start by looking at the data and work with a domain
expert to understand its relationships, guarantees and integrity con
straints from a business perspective, exploiting reality.
This often includes denormalizing the data. Continue by defining
the consistency (transactional) boundaries in the system, within
which you can rely on strong consistency. Then you should let these
boundaries drive the design and scoping of the Microservices. If you
design your services with data dependencies and relationships in
mind it is possible to reduce, and sometimes completely eliminate,
the coupling of datawhich means that you do not have to coordi
nate the changes to it.
general idea behind it in his blog post Memories, Guesses, and Apologies.
42
22 Another excellent paper by Pat Helland, where he introduced the idea of ACID 2.0, is
Building on Quicksand.
43
One tool that embraces these ideas is CRDTs, as they are eventually
consistent, rich data-structures (including counters, sets, maps and
even graphs) that compose, and that converge without coordination.
The ordering of the updates does not matter, and can always be
automatically merged safely. CRDTs are fairly recent, but have been
hardened in production for quite some years, and there are
production-grade libraries that you can leverage directly (for exam
ple in Akka and Riak).
However, relying on eventual consistency is sometimes not permis
sible, since it can force us to give up too much of the high-level busi
ness semantics. If that is the case then using causal consistency can
be a good trade-off. Semantics based on causality is what humans
expect and find intuitive. The good news is that causal consistency
can be made both scalable and available (and is even proven23 to be
the best we can do in an always available system). Causal consis
tency is usually implemented using logical time24 and is available in
many NoSQL databases, Event Logging and Distributed Event
Streaming products (products allowing use of logical time to imple
ment causal consistency include Riak and Red Bulls Eventuate).
23 That Causal Consistency is the strongest consistency that we can achieve in an always
available system was proved by Mahajan et al. in their influential paper Consistency,
Availability, and Convergence.
24 The use of wall clock time (timestamps) for state coordination is something that should
most often be avoided in distributed system design due to the problems of coordinating
clocks across nodes, clock skew etc. Instead, rely on logical time, which gives you a sta
ble notion of time that you can trust, even if nodes fail, messages drop etc. There are
several good options available, one is Vector Clock.
44
45
But what about RDBMSs? You can actually get pretty far using SQL
as well. In one of his papers,25 Peter Bailis talks about coordinationavoidance in RDBMSs and shows how many standard SQL opera
tions can be made without coordinating the changesi.e., without
transactions. The list of operations includes: Equality, Unique ID
Generation, Greater than Increment, Less than Decrement, Foreign
Key Insert and Delete, Secondary Indexing and Materialized Views.
What about transactions? Dont I need transactions?
In general, application developers simply do not implement large
scalable applications assuming distributed transactions.26
Pat Helland
25 Peter Bailis is an assistant professor at Stanford and one of the leading experts on dis
tributed and database systems in the world. The paper referenced is Coordination
Avoidance in Database Systems.
26 This quote is from Pat Hellands excellent paper Life Beyond Distributed Transac
tions.
referred to as XA.
28 Originally defined in the 1987 paper SAGAS by Hector Garcia-Molina and Kenneth
Salem.
46
Summary
When designing individual Reactive Microservices, it is important
to adhere to the core traits of Isolation, Single Responsibility,
Autonomy, Exclusive State, Asynchronous Message-Passing and
Mobility. Whats more, Microservices are collaborative in nature and
only make sense as systems. It is in between the Microservices that
the most interesting, rewarding, and challenging things take place,
and learning from past failures30 and successes31 in distributed sys
tems and collaborative services-based architectures is paramount.
What we need is comprehensive Microservices platforms that pro
vide the heavy lifting for distributed systems, and offer essential
services and patterns built on a solid foundation of the Reactive
principles.
29 For more information see the paper Highly Available Transactions: Virtues and Limi
30 The failures of SOA, CORBA, EJB and synchronous RPC are well worth studying and
understanding.
31 Successful platforms with tons of great design ideas and architectural patterns have so
much to teach usfor example, Tandem Computers NonStop platform, the Erlang
platform and the BitTorrent protocol.
Summary
47