Webinar

Log Processing and Analysis with Elasticsearch, Logstash

and Kibana
Date: 21 Mar 2018

Speakers:
Yash Badiani - Practice Lead, Big Data Analytics & AI/ML, CIGNEX Datamatics
Bhavin Shah – Technical Architect, CIGNEX Datamatics

CIGNEX Datamatics Confidential www.cignex.com

Before We Begin
• This webinar is being recorded.
• Use the chat window for
– Technical assistance
– Q&A
• We will answer quick questions at the end of the webinar.
• For more information, please email us at info@cignex.com

2 CIGNEX Datamatics Confidential www.cignex.com

About our Presenters

Yash Badiani, Bhavin Shah

Practice Lead, Big Data Analytics & Technical Architect at
AI/ML at CIGNEX Datamatics CIGNEX Datamatics

Yash has a DW/BI, Big Data background and has
experience in architecting, designing & implementing
large end to end Big Data Analytics & Machine Learning
Platforms. Yash has extensive experience on proprietary,
Open Source & Cloud based data management
technologies and works towards building the right
solution for the Customers to derive business insights
from their data for competitive advantage & better
customer service. Yash has delivered several webinars on
areas such as NoSQL Databases(MongoDB &
Elasticsearch), Machine Learning & succeeding on Big
Data Platform builds
3 CIGNEX Datamatics Confidential www.cignex.com
Bhavin has 11+ years of global experience in architecting,
designing & implementing scalable enterprise
applications using Web, Data Management & Big Data
Technologies.
Bhavin has exposure to the full stack application &
product development life cycle involving UI, Services &
Backend development using RDBMS & NoSQL databases
such as Marklogic & Elastic Search with Web
Development frameworks such as
Java/J2EE/Spring/Hibernate.
Agenda
• Use Case
• Key Requirements
• Architectural Considerations
• Our Approach
– Platform Evaluation
– Extensibility
– Security
– Design
– Performance Testing & Tuning
– Deployment & Monitoring
• Key Takeaways

4 CIGNEX Datamatics Confidential www.cignex.com

Use Case

Monitoring

J2EE Microservice

Acquire Analyze/
Parse Store
Angular App Visualize

.Net App
Raw Log Archive
Backup
Application
Frameworks

5 CIGNEX Datamatics Confidential www.cignex.com

Functional and Non Functional Requirements

Acquire data from disparate application frameworks Better control on customization through Open
(Docker, .NET, Java, etc.) Source Solution

Ability to visualize and analyze application Scalable to acquire high volumes of log
access patterns / failures in REAL TIME events(~3K-5K events/sec)

Non Functional
Support 75+ Concurrent Users

Functional
User Management

Integration with Enterprise Data Warehouse Ensure compliance to Architectural

(Hadoop) Principles of Source Systems
Requirements

Automation on Installation, Deployment,

Provide the Developers with self serve analysis of
Performance Test & Operations
log events for quick resolution

High Availability & Scalability with ZERO Loss of

Security & Encryption – Data – At rest / In motion Log Events

6 CIGNEX Datamatics Confidential www.cignex.com

Our Approach
Platform Evaluation

Off-the-shelf Open Source Off-the-shelf Proprietary Combination of open source tools

Design & Architecture Considerations

Extensibility Security Auditing User Management Performance Deployment

Log Shipper/Collections Performance Test

Independent Generic Event Measurement
Platform Support Approach Automation Tuning
Component Transporters Procedure

Deployments

Secure Transmission Open Source Tools Custom Scripts

Platform
3rd Party Plugin Custom Scripts
Supported
Key Takeaways

7 CIGNEX Datamatics Confidential www.cignex.com

Platform Evaluation
Platform Evaluation
Off-the-shelf Open Source Off-the-shelf Proprietary Combination of open source tools
(Graylog) (Splunk) (ELK & Search Guard)

Design & Architecture Considerations

Extensibility Security Auditing User Management Performance Deployment

Log Shipper/Collections Performance Test

Independent Generic Event Measurement
Platform Support Approach Automation Tuning
Component Transporters Procedure

Deployments

Secure Transmission Open Source Tools Custom Scripts

Platform
3rd Party Plugin Custom Scripts
Supported
Key Takeaways

8 CIGNEX Datamatics Confidential www.cignex.com

Platform Evaluation
Evaluation Criteria

Yes Yes Yes

Authentication/User Management
• Proprietary • Open Source • Available with 3rd Party tool

Integration with disparate data sources Yes Yes Yes

(Docker based micro service) • Splunk logging driver • Limited to Gelf • TCP, Syslog, FileBeat etc.

Yes Yes
TLS support (Docker) Yes • Supports UDP. Security • TCP+TLS support is available
not off the shelf with TCP driver.
Yes Yes Yes
Ease of querying data
• Its own query language • Similar to Lucene • Lucene query syntax
No
Open Source • /GB of log data and Yes Yes
retention

Customization No No Yes

Conclusion: ELK due to Flexibility, fully integrated Log Management solution to support Log
Shipping, Log Indexing, storage and visualization.

9 CIGNEX Datamatics Confidential www.cignex.com

Log Shipper Evaluation
Platform Evaluation
Off-the-shelf Open Source Off-the-shelf Proprietary Combination of open source tools
(Graylog) (Splunk) (ELK & Search Guard)

Design & Architecture Considerations

Extensibility Security Auditing User Management Performance Deployment

Log Shipper/Collections Performance Test

Independent Generic Event
Platform Support Measurement
Component Transporters Approach Automation Tuning
(File Beat) Procedure
(Fluent Beat) (Syslog, TCP)

Deployments

Secure Transmission Open Source Tools Custom Scripts

Platform
3rd Party Plugin Custom Scripts
Supported
Key Takeaways

10 CIGNEX Datamatics Confidential www.cignex.com

Log Shipper Evaluation
Evaluation Criteria GELF Syslog TCPSocketAppender
Compatible with
Yes Yes Yes Yes
Source & Logstash

Minimum Higher Minimum

Footprint on source Minimum
• GELF logging • Need file beat as an agent • LogstashTCPSocketAppender
system • Syslog logging driver
driver on source system as part of logback
No
Yes
Secure transmission No • Docker v17.09 is not
• Based on backpressure- Yes
guarantee • UDP protocol providing TCP+TLS
sensitive protocol
support

Multiline support No Yes No Yes

Conclusion: TCPSocketAppender due to secure and minimum footprint on source system (Java
based applications)

11 CIGNEX Datamatics Confidential www.cignex.com

Security Evaluation
Log Aggregation Platform Evaluation
Off-the-shelf Open Source Off-the-shelf Proprietary Combination of open source tools
(Graylog) (Splunk) (ELK & Search Guard)

Design & Architecture Considerations

Extensibility Security Auditing User Management Performance Deployment

Log Shipper/Collections Performance Test

Independent Generic Event
Platform Support Measurement
Component Transporters Approach Automation Tuning
(File Beat) Procedure
(Fluent Beat) (Syslog, TCP)

Deployments

Secure Transmission Open Source Tools Custom Scripts

Platform Supported 3rd Party Plugin Custom Scripts

(X-Pack) (Search Guard) (Shell Scripts)
Key Takeaways

12 CIGNEX Datamatics Confidential www.cignex.com

Security Evaluation

Evaluation Criteria

No Yes
Open Source Yes
• Need X-Pack for security • Free license

Security at rest Yes Yes Yes

Security in motion Yes Yes Yes
User management Yes No Yes
No - due to no User
Recommend No - due to enterprise license Yes
management

Conclusion: Search Guard due to security feature with user management as part of free license.

13 CIGNEX Datamatics Confidential www.cignex.com

Design
Log Aggregation Platform Evaluation
Off-the-shelf Open Source Off-the-shelf Proprietary Combination of open source tools
(Graylog) (Splunk) (ELK & Search Guard)

Design & Architecture Considerations

Extensibility Security Auditing User Management Performance Deployment

Log Shipper/Collections Performance Test

Independent Generic Event
Platform Support Measurement
Component Transporters Approach Automation Tuning
(File Beat) Procedure
(Fluent Beat) (Syslog, TCP)

Deployments

Secure Transmission Open Source Tools Custom Scripts

Platform
3rd Party Plugin Custom Scripts
Supported
(Search Guard) (Shell Scripts)
(X-Pack)
Key Takeaways

14 CIGNEX Datamatics Confidential www.cignex.com

Key Decisions
Architecture
• ELK 6.1.0 Basic license as Framework
• TCPSocketAppender as log shipper
• Search Guard for security between ELK
nodes 6.1.0-21.0 for elastic and 6.1.0-
10.0 for Kibana
• Load balancer (HAProxy) before Logstash
and Kibana for High availability
• Monitoring via X-Pack free license
15 CIGNEX Datamatics Confidential
Design
• Raw log file back and ingestion to Hadoop
• Day-wise index for easy backup and restore
• Curator 5.4 for Elastic Index snapshot and
restore
• OpenSSL for secure connection between
source systems and Logstash
• Log rotate of ELK log files rotation
www.cignex.com
Design and Architecture
Monitoring
X-Pack (Free Basic License)
JSON
Format

OpenSSL
Docker Container on VM

OpenSSL
ELK 6.0 X-Pack Free License – On Premise
TCP Logstash 1

FileBeat
HTTPS
Elasticsearch Nodes ES Client
Kibana 1

TCP
Load Balancer Logstash
Micro Console Socket Node
service Appender Appender Curator
Search
Data Node 1 Guard
Search
Angular based UI Guard Load Balancer
Search Guard for Kibana
NFS for UM
HTTPS
Data Node 2 Search
Guard

FileBeat
HTTPS
UI App Browser logs

TCP
via POST API
ES Client
Logstash 2 Kibana 2
Data Node 3 Node

OpenSSL
.Net Application

FileBeat
OpenSSL

.Net User
Log File
App JSON Management
Format Data Warehouse via Search
HDFS Gaurd

16 CIGNEX Datamatics Confidential www.cignex.com

Performance Test
Platform Evaluation
Off-the-shelf Open Source Off-the-shelf Proprietary Combination of open source tools
(Graylog) (Splunk) (ELK & Search Guard)

Design & Architecture Considerations

Extensibility Security Auditing User Management Performance Deployment

Log Shipper/Collections Performance Test

Independent Generic Event
Platform Support Measurement
Component Transporters Approach Automation Tuning
(File Beat) Procedure
(Fluent Beat) (Syslog, TCP)

Deployments

Secure Transmission Open Source Tools Custom Scripts

Platform
3rd Party Plugin Custom Scripts
Supported
(Search Guard) (Shell Scripts)
(X-Pack)
Key Takeaways

17 CIGNEX Datamatics Confidential www.cignex.com

Performance Test Approach

18 CIGNEX Datamatics Confidential www.cignex.com

Performance Test Automation – Log Ingestion
Java based simulator for generating required log events. Configurable via CLI:
Total Total Avg
No of No of Delay between Avg Message Time taken to Publish Avg Time to Message
Iterations Published Persisted Messages
Simulator Messages Iterations (ms) Size (ms) Publish(~min) Lost
Messages Messages per Second

1 4000 25 1000 3k 16415 0.27 100000 83042 16958 6091.989

1 4000 25 1000 3k 25823 0.43 100000 88410 11590 3872.517
1 4000 1000 2000 3k 2000922 33.34 4000000 4000000 0 1999.078

2 4000 500 2000 3k 1000811 , 1000927 16.68 4000000 3798548 201452 3996.527
2 4000 500 2000 3k 1000771 , 1000940 16.68 4000000 3954304 45696 3996.581
2 4000 500 2000 3k 1000743 , 1001010 16.68 4000000 4000000 0 3996.497

12002839 ,
2 4000 6000 2000 3k 12004548 200.06 48000000 48000000 0 3998.769
15003381 ,
2 4000 7500 2000 3k 15006113 250.07 60000000 60000000 0 3998.735

Threshold
Tuned Parameters:
Queue Size of TCPSocketAppender – 32768 |Flush Interval of File Output Plugin Logstash – 1 sec

19 CIGNEX Datamatics Confidential www.cignex.com

Performance Test Automation – Concurrent Search
JMeter test suites for firing search requests with concurrent users

No of Keyword Search Avg FullTextSearch Avg Multiple Query

Total Loop JMeter Delay between each Total Run
Users Queries Time in ms Time in ms Request Average
Requests Count Instances Requests Time ( mins)
per Loop (Avg. bytes xfer) (Avg bytes xfer) Time

304 ms (1560852
250 5 10 5 1 67 ms (5188 bytes) 267 ms (1353651 bytes) bytes) 3 seconds ~7
354 ms (1560852
3750 25 10 5 3 81 ms (5188 bytes) 298 ms (1353651 bytes) bytes) 3 seconds ~30

40000 200 50 4 1 382 ms (802 bytes) 593 ms (1180774 bytes) 417 ms (37414 bytes) 3 seconds ~48

100000 500 50 4 1 386 ms (802 bytes) 608 ms (1180770 bytes) 423 ms (37413 bytes) 3 seconds ~120

200000 1000 75 4 1 500 ms(803 bytes) 593 ms (1180768 bytes) 410 ms (37414 bytes) 3 seconds ~239

Throughput

Tuned Parameters:
Elastic Shard Configuration 3:1 | Kibana Default Search Page Size - 200

20 CIGNEX Datamatics Confidential www.cignex.com

Performance Stats

Elasticsearch
4000 events/sec ingestion throughput
800 millions log events
Data size: 750 GB

Logstash

Kibana
75 concurrent users search request
Search on 30 days/750GB of data
Avg. 500 milliseconds response time

21 CIGNEX Datamatics Confidential www.cignex.com

Deployment Evaluation
Log Aggregation Platform Evaluation
Off-the-shelf Open Source Off-the-shelf Proprietary Combination of open source tools
(Graylog) (Splunk) (ELK & Search Guard)

Design & Architecture Considerations

Extensibility Security Auditing User Management Performance Deployment

Design

Log Shipper/Collections Performance Test

Independent Generic Event
Platform Support Measurement
Component Transporters Approach Automation Tuning
(File Beat) Procedure
(Fluent Beat) (Syslog, TCP)

Deployments
Open Source Tools Custom Scripts
Secure Transmission (Ansible, Chef, Puppet) (Shell Scripts)
Platform
3rd Party Plugin Custom Scripts
Supported
(Search Guard) (Shell Scripts)
(X-Pack)
Key Takeaways

22 CIGNEX Datamatics Confidential www.cignex.com

Deployment Evaluation
Evaluation Criteria Shell Script

Ease of deployment tool installation Easy Moderate Difficult Easy

Able to pull deployment artifacts from local system Yes Yes Yes Yes
Able to pull deployment artifacts from central repository (Git) Yes Yes Yes Yes
One click installation facility on local host Yes Yes Yes Yes
One click installation facility on remote host Yes Yes Yes Yes
Ease of adding new node to existing cluster (install & configure) Yes Yes Yes Yes
Ease of upgrading existing node Yes Yes Yes Yes
Start/stop/restart of any ELK node Yes Yes Yes Yes
Security of functional user credentials (elastic/kibana etc) Yes Yes Yes Yes
Recommended by ELK Yes Yes Yes -
Ease of deployment tool maintenance Yes Moderate Complex -
Client agent installation required No Yes Yes -

Open source/licensed Open Source Open Source Open Source Open Source

Conclusion: Ansible (ELK suggests Ansible/Chef/Puppet)

23 CIGNEX Datamatics Confidential www.cignex.com

Benefits Delivered

Real time monitoring of Log events from diverse application frameworks

Real time analysis of failures leading to quicker resolution & higher availability

High Availability & Scalability to handle a high volume of log events(4000 events/sec)

End to End Automation on Deployment, Performance Test, Operations

Secure Open Source based solution resulting in lower TCO

Ability to create customized log event processing pipelines for each source

Real time ELK cluster monitoring via Kibana dashboard

24 CIGNEX Datamatics Confidential www.cignex.com

Key Takeaways

Evaluate ALL aspects Design in detail for Gather volume details &
(Platform, Component, Consider all technology
Performance, Deployment, Security, Extensibility, Availability, plan ahead for sizing,
options – Cloud, Open
Operations) in Depth Performance, Security, performance.
Source, Proprietary before
(Requirements to Evaluation Auditing, Deployment & Benchmarking truly
Criteria to Tool Options) finalizing the stack
Operations helps

Identify the right Performance Test &

Evaluate / Implement
versions & deployment Tune with every release Be Agile, Fail Fast to
automation at every
approach for PROD and make it an ongoing succeed fast
stage
early activity

25 CIGNEX Datamatics Confidential www.cignex.com

Check Out Our Previous Webinars

Faster Big Data Analytics with Building Scalable Big Data Text (Document) Classification
MongoDB Analytics Platform driving using Machine Learning
business ROI

Download Presentation - Link Download Presentation - Link Download Presentation - Link

26 CIGNEX Datamatics Confidential www.cignex.com

 Thank you

Contact Us

Sales: sales@cignex.com | Jobs – jobs@cignex.com | Others – info@cignex.com

www.cignex.com facebook.com/CIGNEXTechnologies twitter.com/cignex youtube.com/cignexglobal

CIGNEX Datamatics Confidential www.cignex.com