Red Hat JBoss AMQ and Apache Kafka :

Which to Use ?

Christian Posta (@christianposta)

Principal Architect

Paolo Patierno (@ppatierno)

Senior Software Engineer
Christian Posta
Principal Architect at Red Hat

christian@redhat.com
@christianposta

Author Microservices for Java Developers

Committer Apache Camel, ActiveMQ, ActiveMQ PMC,
Fabric8.io
Contributor to Apache Kafka in spare time
Worked with large Microservices, web-scale companies
Blogger, speaker about microservices, DevOps, cloud,
distributed systems, messaging
Paolo Patierno
Senior Software Engineer at Red Hat

ppatierno@redhat.com
@ppatierno

Messaging and IoT team

Lead/Committer on Eclipse Hono
Committer on Eclipse Vert.x and Paho
Hacking low constrained devices in spare time
Technologies and protocols globetrotter
Blogger and speaker about distributed systems, messaging, IoT
and embedded world
Looking closely at JBoss AMQ
and Apache Kafka
Seriously? We are still talking about messaging?
How hard can it be!

Send a message
Receive a message
Get my messages in order
Get them once and only once
Why dont we just pick the fastest
one and be done?
Safety guarantees Scalability
Recency Availability
Reliability Performance
JBoss AMQ
JBoss AMQ

Standards based (JMS 1.x/2.x)

Multi protocol (AMQP 1.0, MQTT, STOMP, OpenWire)
Based on open community, Apache ActiveMQ
Highly available, fault tolerant, performant
Trusted and used in many highly critical use cases around the world
Uses high-level queueing constructs such as queue, topic durable topic etc
Producing messages
Durability
Dispatching messages
Consuming messages
Recap:
For persistent messages, block waiting for acknowledgement
Broker persists messages before continuing
Broker responsible for dispatching to consumers
Broker can handle redelivery on failover
Consumers acknowledge each message
Broker has transactional capabilities including XA transaction
Apache Kafka
Apache Kafka

Developed originally at LinkedIn

Built to support high throughput, high ingest of log data
Kafka is (at its core) a distributed log
Built for data pipeline between services and analytic clusters
Highly parallelizable
Persistent/replicated
Key/Value structure (not message in the traditional sense)
Uses concept of publish-subscribe topics
Producing messages
Durability
Producing messages
Producing messages
Dispatching messages
Dispatching messages
Consuming messages
Recap:
Send data as fast as possible
Make efficient usage of transmission via batching
Brokers write messages but dont sync
Rely on batching/write back to actual disk when the OS flushes
Not responsible for dispatching, tracking acks etc
Consumers track position in the log
Consumers coordinate to balance consumption of messages
Two different brokers with different tradeoffs for
different use cases. They co-exist in your
architecture just fine.
Use JBoss AMQ for:
You care about individual messages
You want clients to use standard APIs (e.g., JMS) or wire protocols (e.g., AMQP)
You need transactional sends and receives
Youre doing request-reply messaging
Using selectors
Heterogeneous client/protocol messaging (ie, AMQP, MQTT, STOMP, etc)
You send metadata/headers/properties with your messages
You need Time-To-Live semantics (TTL)
Need Dead Letter Queue semantics (DLQ)
You dont want to implement broker functionality in your clients (ie, partitioning,
dispatching, coordination)
Use Kafka for:

You care about messages in volume

You care about raw throughput, high performance
You need sliding-window replay abilities
Large numbers of subscribers for published events
You need to finely control the parallelism/scalability of consumers
You want to leverage application-level replication vs HA storage
You need total order guarantees at the partition level
 Shhh. Sneak peak...

Find it from OReilly week of May 22nd!

 but they can live together
Bridging AMQP to Kafka

Why ?
AMQP 1.0
is supported by AMQ 7 Broker
is THE protocol for AMQ 7 Interconnect
is supported by all AMQ 7 Clients
Maybe you have an application
with JMS/AMQP senders
but now you need a different way to consume
or a different feature (i.e. re-read stream)
Avoiding changes on the senders side
Bridging AMQP to Kafka

What does bridging mean ?

Mapping AMQP semantics on Kafka
From/to an AMQP address to/from a Kafka topic
Addressing topic partitions through AMQP
Handling QoS and flow control
Dealing with messages :
AMQP full featured packets
Kafka simple raw bytes
Bridging AMQP to Kafka

Senders

Source endpoints Producers

Receivers

Kafka
Cluster
Sink endpoints Consumers
What about semantics
attach ([topic]) AMQP - Kafka [topic]
sende transfer () / disposition() / flow (credits) P0
r
attach ([topic]/group.id/[group.id])
P1

receiver transfer () / disposition() / flow (credits)

P2

Attach a link on the Kafka [topic] address, specifying [group.id] for consumer only
QoS : settlement implies
waiting for Kafka acks from leaders/followers
and Kafka offsets commit
Flow : credits implies
sender can send on Kafka acks
paused/resumed Kafka consumer
 and message structure ?

AMQP full featured messages vs Kafka raw bytes

Conversion
Default : AMQP body to/from Kafka bytes (losing metadata)
JSON : mapping AMQP body, properties, annotations
Raw : raw AMQP message bytes become Kafka message (only for AMQP
enabled clients on both sides)
Clients : AMQP to Kafka

A JMS/AMQP client doesnt need changes

Interacting with an A-MQ queue/topic is the same as Kafka topic
Just change the address
A simplified API
Kafka protocol needs to open multiple connections to partition leaders
One connection, links multiplexing for interacting with Kafka topics (partitions)
The bridge does the magic
 Demo : the deployment
Messaging-as-a-Service (EnMasse)

AMQ 7 Interconnect Apache Kafka

JMS/AMQP clients
AMQP - Kafka

AMQ 7 Broker

Kafka Web UI App

Demo : use cases

Apache Kafka
Getting messages published by AMQP/JMS clients
Re-read the stream
from beginning
From a specific offset
AMQ
Filtering on messages
Using a selector on a message property
Request/reply :
Using store and forward
Using direct messaging
 Demo : AMQP - Kafka

One Kafka topic / one partition

JMS/AMQP sender
Native Kafka and JMS/AMQP receivers (belonging to different consumer groups)

JMS/AMQP clients
AMQ 7 Interconnect
Apache Kafka
AMQP - Kafka Kafka Web UI App
Demo : filtering

Asking for a filter/selector when receiver connects

Its up to the broker delivering only messages which meet the filter
No filtering at application level

JMS/AMQP receiver
AMQ 7 Interconnect
Filter : count % 2 = 0
AMQ 7 Broker

JMS/AMQP sender

count : <val>
 Demo : request/reply

Store and forward is used Direct messaging is used

Queue for request and temporary queue Requests and replies are just routed
for response hosted on broker through the router
Address for request, a dynamic one for
response
JMS client
AMQ 7 Interconnect AMQ 7 Interconnect
AMQP client AMQP server
AMQ 7 Broker

JMS server
Demo : material

Message as a Service - EnMasse : https://github.com/EnMasseProject/enmasse

AMQP - Kafka bridge : https://github.com/EnMasseProject/amqp-kafka-bridge
Kafka deployment : https://github.com/EnMasseProject/barnabas
Demo code : https://github.com/ppatierno/amqp-kafka-demo
THANK YOU
@christianposta
@ppatierno
THANK YOU
plus.google.com/+RedHat facebook.com/redhatinc

linkedin.com/company/red-hat twitter.com/RedHatNews

youtube.com/user/RedHatVideos