ADMIN Network & Security

SPONSORED BY

ANOTHER

10 Tools
More marvelous utilities
for configuring and
managing your
Terrific
FOR THE BUSY ADMIN
2017 EDITION

network
Find the perfect tool to
Rescue lost data
Roll back config changes
Correct Bash input
Visualize Ping statistics
Bonus articles
5 Log Monitoring Tools
Web Analytics without Google
www . admin - magazine . com US$ 7.95

ADMIN
Network & Security
Dear Readers:
Every system administrator needs a toolkit of simple,
useful tools to help with practical tasks. ADMINs 10
Terrific Tools series shines the spotlight on the best free
tools for admins. This latest list, from the toolkit of Linux
ADMIN Special
Editor in Chief Joe Casad
Managing Editor R
 ita L Sooby
Copy Editor A
 my Pettle
Layout / Graphic Design Dena Friesen, Lori White
Advertising
Ann Jesse, ajesse@linuxnewmedia.com
Phone: +1-785-841-8834
Publisher Brian Osborn
Customer Service / Subscription
For USA and Canada:
Email: cs@linuxnewmedia.com
Phone: 1-866-247-2802
(toll-free from the US and Canada)
www.admin-magazine.com
While every care has been taken in the content of
the m
agazine, the publishers cannot be held
responsible for the accuracy of the information
contained within it or any c onsequences arising
from the use of it.
Copyright & Trademarks 2017 Linux New Media
USA, LLC
Cover Illustration Nataliia Natykach, 123RF.com
No material may be reproduced in any form
whatsoever in whole or in part without the written
permission of the p ublishers. It is assumed that all
correspondence sent, for example, letters, email,
faxes, photographs, articles, drawings, are s upplied
for publication or license to third parties on a non-
exclusive worldwide basis by Linux New Media
unless otherwise stated in writing.
All brand or product names are trademarks oftheir
respective owners. Contact us if we havent
credited your copyright; we will always correct any
oversight.
ADMIN is published by Linux New Media USA,
LLC, 616 Kentucky St, Lawrence, KS 66044, USA.
Published in Europe by: Sparkhaus Media GmbH,
Zieblandstr. 1, 80799 Munich, Germany
W W W. A D M I N - M AGA Z I N E .CO M
Welcome ANOTHER 10 TERRIFIC TOOLS
ANOTHER
10 Tools Terrific
FOR THE BUSY ADMIN
2017 EDITION
Magazine columnist Charly Khnast, includes useful
utilities for security, performance monitoring, network
troubleshooting, and more. Read on! We hope you find
some tools for your own toolkit in this years collection.
Table of Contents
Etckeeper. . . . . . . 4 Socket Statistics. . . 9
Track /etc changes in Git so you can A fast and easy tool for monitoring
restore previous configurations. network stats.
Prettyping and ddrescue. . . . . . . . . . 10
Read errors wont deter this helpful
Asciiflow. . . . . . . . 5
data recovery tool.
Transform boring ping data into
colorful statistics.
Keepalived . . . . . . . . . 11
Configure a floating IP address for
Dnstop. . . . . . . . . . 6 failover.
Monitor DNS traffic on your network.
paping. . . . . . . . . . . . 12
The Fuck. . . . . . . . 7 Use this connectivity checker when
ping cant connect.
Correct your command-line
misspellings.
httpstat . . . . . . . . . . 13
This simple tool wil help you discover
testssl.sh . . . . . . . 8 how long a web server takes to serve
A handy script for testing SSL up an HTML page.
certificates.
As a special bonus, were also including two more articles
with great tools for your personal admin toolkit:
Five Log Monitoring Tools. . . . . . . . . . . . . . . . . . . . . . . 14
Anyone who wants to watch logfiles can use a full-featured monitoring tool like Nagios;
however, a lightweight alternative also sniffs out threats and takes much less time to set up.
Web Analytics Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
If you are looking for an alternative to Google Analytics, try a free alternative such as Piwik,
Open Web Analytics, or eAnalytics.
1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I 3
ANOTHER 10 TERRIFIC TOOLS Etckeeper

The Sys Admins Daily Grind: Etckeeper

Fountain
of Youth
Track /etc changes in Git so you can restore previous configurations. By Charly Khnast

Etckeeper is part of many distribu-

tions and is otherwise available from
GitHub [1]. Of course, Git also needs
to be installed on the computer.
Anyone who hasnt used Git on their
system should configure a few funda-
mental settings:

git config global user.name "Charly"

git config global user.email U Figure 1: If someone has configured something in /etc, Git notices the difference when comparing with the
"charly@example.com"
git config global core.editor "vim"
Because I execute the etckeeper and
git commands with sudo, these set-
tings prevent the root user from being
entered as a committer.
Preserving Youth
To begin, I need to initialize the new
repository, then I save all /etc files in
it for the first time:
cd /etc
sudo etckeeper init
sudo etckeeper commit "Initial etc commit"
Some distributions, such as Ubuntu,
execute this step automatically when
installing Etckeeper. In this case, I see
output like:
> On branch master
> nothing to commit, working directory clean
Sometimes I dont want all the data
in /etc/ in the repository, so I tell Git
which directories Etckeeper should
ignore in the .gitignore file right after
repository.
the # end section [...] etckeeper
line:
# end section managed by etckeeper
ghostscript/*
Now, however, all the data from /
etc/ since the first commit is already
in the repository, so I have to remove
the undesired files manually:
sudo git rm r cached ghostscript/* Info
To demonstrate, I change a little
something in /etc/postfix/main.cf;
in fact, sudo git status shows that
Git notices the difference when com-
paring the data with the repository
(Figure 1). I can now check in the
new version:
sudo git commit a m "main.cf changed"
An overview of all actions is provided
with sudo git log command:
commit9695e06a8175bd5cf485316f20d8fb 6d6fcc1e49
Author: Charly <charly@example.com>
Date: Wed May 11 14:18:50 2016 +0200
main.cf changed
Its important to realize that this pro-
cess implements versioning theres
no backup! Etckeeper and Git help you
comprehend changes in configuration
files and roll them back if necessary,
but thats a lot for anyone whos con-
figured something incorrectly. n
[1] Etckeeper:
[https://github.com/joeyh/etckeeper]
Charly Khnast
Charly Khnast is a Unix operating system
administrator at the Data Center in Moers,
Germany. His tasks include firewall and DMZ
security and avail-
ability. He divides his
leisure time into hot,
wet, and eastern sec-
tors, where he enjoys
cooking, freshwater
aquariums, and
learning Japanese,
respectively.

4 1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I W W W. A D M I N - M AGA Z I N E .CO M

The sys admins daily grind:
Prettyping and Asciiflow
Block
Heroes
Transform boring ping data into colorful statistics. By Charly Khnast
One weapon for command-line
warriors is Prettyping [1], a shell
script that wraps around the
ping command. It reads
its tasks, keeps a re-
cord of run times and
packet losses, and
shows at the command
line in block graphics the aver-
age values since starting the
tool and for the past 60 seconds
(Figure 1).
The script runs on any system with
Bash and Awk (i.e., also on OS X
and probably also in the new Linux
environment on Windows 10). Pret-
typing detects whether it is running
in a terminal and how wide the ter-
minal is, then scales the output ac-
cordingly. If you think the output is
a little too clownish, you can switch
to a more staid monochrome display
Figure 1: Its a colorful world folks: Prettyping visualizing ping statistics in the terminal.
W W W. A D M I N - M AGA Z I N E .CO M
Prettyping and Asciiflow
using nocolor. Prettyping passes
on to ping any parameters that it
isnt familiar with.
Everything <--|__ASCII__|
If you read RFCs, you will occasion-
ally see small ASCII graphics that
show connections more compactly
than is possible with sentences. Au-
thors typically painstakingly create
such charts with boxes and arrows
with ASCII symbols, such as plus
and minus signs, (back-)slashes, and
pipes. Naturally, at some point, vari-
ous ASCII graphic victims have writ-
ten tools but none are as easy and
intuitive to use as Asciiflow [2].
Asciiflow is a website that at first
looks like a blank sheet of graph pa-
per. In a toolbar at the top, you can
1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I
ANOTHER 10 TERRIFIC TOOLS
select boxes, lines, arrows, text, and
so on, and then simply draw on the
blank sheet using the mouse (Figure
2). Once you are happy with your
work, you just press the export sym-
bol and hey, presto the finished
ASCII graphic appears in your clip-
board. Asciiflow also has an import
function. My verdict on it: \o/. n
Info
[1] Prettyping:
[http://denilson.sa.nom.br/prettyping/]
[2] Asciiflow: [http://asciiflow.com]
Figure 2: A few boxes drawn in Asciiflow often say
more than a thousand words.
5
ANOTHER 10 TERRIFIC TOOLS Dnstop

The sys admins daily grind: Dnstop

Save the Day

Monitor DNS traffic on your network. By Charly Khnast

Most distributions include dnstop. asking for .xyz domains but hey ho, In my short observation period, this
If you prefer to build it yourself, you if I press 2 or 3, I can extend the view was all of them, thankfully:
will find the source code online [1], to include the second and third levels.
but make sure you download and code Count %
build the matching Libpcap [2] first. Frequent and Rare
I launched the tool on the computer
hosting my DNS cache with the fol-
Resource Records Noerror 23987 100.0

lowing command:
dnstop l 3 eth0
The l 3 parameter tells dnstop to
explore name requests up to the third
level. For a request like www.linux-
magazine.com, com is the first or top-
level domain, linux-magazine is thus
the second-level domain, and www is
the third level.
When I press the 1 button on a com-
puter running the command listed
above, I can see which top-level
domains are most frequently queried
(Figure 1). What I am interested in
here is which device on my network is
Pressing T takes you to another
practical statistic. It shows you what
resource record types are most fre-
quently requested. It is unsurpris-
ing to see requests for A (IPv4) and
AAAA records (IPv6) topping the
list (Figure 2). Well back in the field,
is the A6 record, which comes from
the early days of IPv6 and is about as
widespread as gas streetlamps today.
Other fairly sparsely represented re-
cords include DNSKEY, which come
from DNSSEC (DNS Security). In con-
trast to A6 IPv6, DNSSEC is increasing
steadily but still not well established.
Pressing R (for Result) shows you
how many requests were successful.
If I use dnstop for evaluations at
work and then save them some-
where I need to think about data
protection. To avoid problems from
the outset, I tend to launch the tool
with a for anonymize. Then, dn-
stop replaces the client IP addresses
with consecutive numbers, while
all the other evaluations work as
expected. n
Info
[1] Dnstop: [http://dns.measurementfactory.
com/tools/dnstop/]
[2] Libpcap: [https://github.com/
thetcpdumpgroup/libpcap]

Figure 1: Pressing the 1 key displays statistics with the requested top-level domains. Figure 2: Pressing T shows you the Resource Record overview. The A records
It comes as little surprise that .com tops the list, but who is looking for .xyz? typical of IPv4 have a two-thirds majority.

6 1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I W W W. A D M I N - M AGA Z I N E .CO M
 The Fuck ANOTHER 10 TERRIFIC TOOLS

The sys admins daily grind: The Fuck

Expletives
Not Deleted
Correct your command-line misspellings. By Charly Kuhnast

As early as 1998, Sean Dreil- actually or of course (slap

inger created the first statistics
on swear words in Linux [1].
It showed evidence of a sharp
increase of the word fuck in
denied even to prayer.
kernel 2.1.5. Vidar Holen [2]
delved even deeper into the
murky depths (Figure 1). He
found evidence of almost 50 inci-
dences of fuck, many incidences
of shit, and even 180 incidences of
bastard.
How often Linus has raised a warn-
ing middle finger against hardware
vendors such as Nvidia is unfortu-
nately unknown; however, contem-
porary historians have probably only
investigated the kernel mailing list
quantitatively.
Damned Shell!
forehead). This alias saves me
profanity provides a relief much suppressed cursing after
forgetting to sudo.
However, The Fuck can do far
Mark Twain more; it can even prevent you
from shooting yourself in the
foot with more complex Git
cd .. [enter/^/V/ctrl+c] commands. Ergo, the world needs
much more of The Fuck to make it a
and you just need to confirm by more polite place. n
pressing Enter. Of course, cd.. is a
very simple example. Many users
will already have created aliases for Info
these typical errors. Incidentally, my [1] Linux kernel fuck count:
favorite alias is [http://durak.org/sean/pubs/kfc/]
[2] Linux kernel swear counts:
alias doch='sudo $(history p !1)' [https://www.vidarholen.net/contents/
wordcount/]
where doch, in this case, roughly [3] The Fuck:
translated into English, could mean [https://github.com/nvbn/thefuck]

Cursing has made its way out of the

kernel to other system components; a
good example of this in many ways is
The Fuck [3], a semi-automatic cor-
rection function for shell input. The
tool, written in Python 3 is installed
as follows:

sudo apt install python3 python3dev

git clone https://github.com/nvbn/thefuck
cd thefuck
sudo ./setup.py install

After installing, if you make a typing

error, such as forgetting the space
in cd .., you first see the standard
-bash: cd..: command not found.
However, if you then type fuck, The Figure 1: Kernel versions are shown on the x axis, and incidence of words on the y axis. Top to bottom: crap
Fuck suggests the correct command, (blue line), shit (green), penguin (teal), fuck (red), bastard (violet).

W W W. A D M I N - M AGA Z I N E .CO M 1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I 7
 ANOTHER 10 TERRIFIC TOOLS testssl.sh

The sys admins daily grind: testssl.sh

A Key Role
A handy script for testing SSL certificates. By Charly Khnast

Web servers with SSL certificates will testssl P <address> the whole arsenal of test routines
likely be the rule. The administrators (Figure 1).
responsibility for keeping track of which Listing 2 shows the strongest en-
SSL and TLS settings are up to date on cryption method the server was able Aha
which servers tends to grow proportion- to negotiate. The script additionally
ally. I have found a trusty helper for this helps me discover whether the server Finally, a tip from the developer of
task, testssl.sh [1], a shell script with supports HTTP/2 with: testssl.sh: If you additionally install
many capabilities. For example, typing the aha [2] tool, you can easily con-
testssl.sh Y <address> vert the console output to HTML:
testssl U https://<server>
It might then respond with Ser- testssl.sh <address> | aha > U

runs scan tests for popular exploits
like Heartbleed, Poodle, and Crime
(see output in Listing 1), and I can
trigger each test option individually.
To discover whether a server is vul-
nerable for Logjam, I would just type:
vice detected: HTTP HTTP2/ALPN
http/1.1 (offered).
The tests are not restricted to HTTPS.
I can test a mail server like this:
testssl.sh starttls smtp <address>
/<Path>/servertest.html
After wrapping this up in a small
shell script, I can then use cron to
check the TLS and SSL status of my
servers on a weekly basis. n

testssl.sh logjam <address> which gives me an equally good

quality of analysis as for the web Info
To test whether the server offers protocol and it is very exhaustive [1] Testssl.sh:
its ciphers in a defined order (from in this case. If you dont specify any [https://github.com/drwetter/testssl.sh]
strong to weak), I type: parameters testssl.sh runs through [2] Aha: [https://github.com/theZiz/aha]

Listing 1: testssl U https://< server>

01 Service detected: HTTP
02 Testing vulnerabilities
03
04 Heartbleed (CVE20140160) not vulnerable (OK) (timed out)
05 CCS (CVE20140224) not vulnerable (OK)
06 Secure Renegotiation (CVE20093555) not vulnerable (OK)
07 Secure ClientInitiated Renegotiation not vulnerable (OK)
08 CRIME, TLS (CVE20124929) not vulnerable (OK)
09 [...]

Listing 2: testssl P <address>

01 Testing server preferences
02
03 Has server cipher order? yes (OK)
04 Negotiated protocol TLSv1.2
05 Negotiated cipher ECDHERSAAES256GCMSHA384, 256 bit ECDH
06 Cipher order
07 [A long list of ciphers offered]
Figure 1: Testssl.sh even checks mail server certificates.

8 1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I W W W. A D M I N - M AGA Z I N E .CO M
 Socket Statistics ANOTHER 10 TERRIFIC TOOLS

The sys admins daily grind: socket statistics

Short Cut
A fast and easy tool for monitoring network stats. By Charly Khnast

A bunch of parameters control the socket. I need to use root privileges for
behavior of netstat. Administrators this; otherwise, p has no effect.
can also happily combine options so process is
that some netstat calls look like an Numbers, Please! on port 25. Logical
armadillo has rolled across the key- links are also okay:
board. With ss (like socket statistics), A further default is that ss translates
there is an even more specific tool the port numbers from the /etc/ser # ss t4 dport = :443 or dport = :80
for this purpose. It comes from the vices file into names. I can switch ESTAB 0 0 10.0.0.201:53389 10.0.0.118:http
iproute2 package [1] just like its sib- this off using n (for numeric). If I
lings bridge, rtacct, rtmon, tc, ctstat, want the tool to resolve the IP ad- ccze [2] sorted the coloration in
lnstat, nstat, routef, routel, rtstat, dresses concerned into names, I just Figure 1, but grc [3] or rainbow [4]
and ip. add an r. Using 4 and 6, I can limit would have managed it, too. n
Because a running Linux uses several it to one of the two TCP/IP versions.
hundred or more ports, ss comes with I find it useful to filter by source
plenty of filters. I mostly need the t (sport) and destination port (dport). Info
and u parameters, which restrict the For example, [1] Iproute2: [http://www.linuxfoundation.
results to TCP or UDP sockets. Other org/collaborate/workgroups/networking/
parameters limit the list to raw, Da- # ss natp6 sport = :25 iproute2]
tagram Congestion Control Protocol LISTEN 0 100 :::25 :::* U [2] Ccze: [http://freecode.com/projects/ccze/]
(DCCP), package, and Unix domain users:(("master",27452,13)) [3] Grc: [http://kassiopeia.juls.savba.sk/
sockets. ~garabik/software/grc.html]
Caution: The tool only displays TCP shows whether the mail server is [4] Rainbow:
sockets for existing connections by listening via IPv6 (yes) and which [https://github.com/nicoulaj/rainbow]
default (es
tablished or
close_wait). If I
also want to see
sockets in the
lists status as
I usually do I
have to type
ss ta. If I only
want the tool
to pay attention
to listening TCP
sockets and sup-
press all others,
I use ss lt. If
I add p here,
I can also see
which process
opened the Figure 1: The socket collector ss, color-supported here, too, provides a good overview.

W W W. A D M I N - M AGA Z I N E .CO M 1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I 9
 ANOTHER 10 TERRIFIC TOOLS ddrescue

The sys admins daily grind:

ddrescue and DDRescue-GUI

Recovery
Needed
Read errors wont deter this helpful data recovery tool. By Charly Khnast

Krrr, krrr ! At least things are creating an image with all the data that on my test Ubuntu using these three
clear-cut when a hard disk gives up can be accurately read. In the following steps:
the ghost: You toss the offending disk, example, /dev/sdd1 is a partition with
get a new one, and put the backup on read errors on a USB flash drive: sudo addaptrepository ppa:hamishmb/myppa
it. However, those undead data me- sudo aptget update
dia that trick people into continuing sudo ddrescue n /dev/sdd1 U sudo aptget install fym ddrescuegui
working on them with no idea of the /home/charly/stick.img logfile.log
potential impact are a real pain. As Figure 1 shows, the interface is
I recently determined that an SDHC The second, more time-consuming businesslike and functional. The
card in my camera saves one out of phase involves using the tool to sort GUI sets the important parameters,
20 images (on average) as a colorful through the faulty blocks and save as but not all of them by far. Although
mess of pixels. I do know that mem- much data from them as possible. The I hope no one will need to use
ory cards give up the ghost sooner or command is just the same as before, ddrescue permanently, the GUI is
later. However, I didnt realize that except you leave out the n parameter. nevertheless a real help. n
my camera could save to two cards In the wake of ddrescue, there is still
simultaneously a feature I stupidly an armada of other parameters that Info
didnt use. But, Im all the wiser now. control the tools behavior. [1] ddrescue:
What if really important data is There is also a GUI [2] that you can [http://www.gnu.org/software/ddrescue/]
stored on a haywire device that you use to make some quick, useful de- [2] DDRescue-GUI:
just cant get rid of? This is where fault settings. I installed it quickly [https://launchpad.net/ddrescuegui]
ddrescue [1] comes in. The tool is al-
ready quite ancient, but its developers
look after it untiringly and adapt it to
new types of data media. (It should
not be confused with the even older
dd_rescue.) Ddrescue is officially
named GNU ddrescue; the packages
on Debian and derivatives are there-
fore dubbed gddrescue.
The tool is included with many popu-
lar distributions. The first two letters
subtly indicate a relationship with dd,
and ddrescue actually generates a data
mediums or partitions image. Unlike
dd, however, it cant be stopped by read
errors; instead, it stubbornly saves ev-
erything that it can get its teeth into.

Two-Speed Transmission
Administrators usually use ddrescue in Figure 1: The genuinely helpful front end for ddrescue is DDRescue-GUI, which graphically implements the
two phases. The first phase involves important parameters.

10 1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I W W W. A D M I N - M AGA Z I N E .CO M
 Keepalived ANOTHER 10 TERRIFIC TOOLS

The sys admins daily grind: Keepalived

Fit Without a
Pacemaker
Configure a floating IP address for failover. By Charly Khnast

Services without which nothing whereas the other only gets the IP if [2] Corosync:
works are clear candidates for dop- the master is down. The VRRP (Vir- [http://corosync.github.io/corosync/]
pelgngers on my network. If the tual Router Redundancy Protocol) is [3] Keepalived: [http://www.keepalived.org
master fails, or if I just need to shut used to swap states.
down the server for maintenance, To be able to bind services to an IP Listing 1: keepalived.conf
I want the service automatically to address that does not (yet) exist on 01 ! Configuration File for keepalived
start on the second server and to be the system, I need to make some 02
available on the same IP address as changes to /etc/sysctl.conf: 03 global_defs {
before, if possible. 04 notification_email {
To do this, the IP address needs to sudo echo "net.ipv4.ip_nonlocal_bind = U 05 charly@kuehnast.com
migrate quickly and without much 1" >> /etc/sysctl.conf 06 }
overhead to the backup machine. sudo sysctl p 07 notification_email_from ha_test@kuehnast.com
08 smtp_server localhost
Addresses like this are known as
09 smtp_connect_timeout 30
floating IPs. The migration helpers Now I can launch Keepalived by
10 }
here are Pacemaker [1] and Coro- typing:
11
sync [2]. These tools cant do much 12 vrrp_instance VI_1 {
more than facilitate the move, how- sudo service keepalived start 13 state MASTER
ever, so the solution seems a little 14 interface eth0
over the top for a simple failover The floating IP appears on the server 15 virtual_router_id 51
scenario. with the higher priority value. If I 16 priority 100
A more streamlined solution called shut down the master, I can see from 17 advert_int 1
Keepalived [3] is part of the default the Syslog on the secondary machine 18 authentication {
toolset in most distributions. I just that it quickly assumes the master 19 auth_type PASS
installed Keepalived and jumped in role (Figure 1). 20 auth_pass meinpasswort
feet first. What I found were two 21 }
servers with IP addresses 10.0.0.1 and 22 virtual_ipaddress {
23 10.0.0.100
10.0.0.2. My floating IP of choice is Info
24 }
10.0.0.100. [1] Pacemaker:
25 }
[http://clusterlabs.org/wiki/Pacemaker]
Minor Difference
The /etc/keepalived/keepalived.conf
configuration file looks like Listing 1.
It differs on the two machines only
in one aspect: the priority (line 16).
I need to configure a smaller num-
ber on one of the machines than on
the other to define the master. This Figure 1: A glance at the Syslog on the slave machine shows that it has become the master after a failure of
server gets the floating IP by default, the production machine.

W W W. A D M I N - M AGA Z I N E .CO M 1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I 11
 ANOTHER 10 TERRIFIC TOOLS paping

The sys admins daily grind paping

A Better ping
Use this connectivity checker when ping cant connect. By Charly Kuhnast

Hi there! Anybody home? Quite of- <no_of_checks>, I can define how of- apps? No way: The open source
ten, admins need to make sure that ten Paping checks out the target. tool has apparently reached the
at least the physical connection be- end of its design cycle no-
tween two computers is still up or Give Him a Break! body has touched the C++
back up. Obviously, ping is the tool code for two years, but that is
of choice when it comes to proving If you know that your target re- not surprising for such
or refuting accessibility. However, sides on a particularly slow or fast a simple tool.
there are cases in which a firewall line, you might want to tune the t If your Linux
blocks the ICMP ping. Anyone with a <timeout_in_msec> parameter; that is, distribution
modicum of knowledge can save the the time in which Paping abandons does not include
energy they would otherwise waste its task. Without the option, the tool Paping in its reposi-
cussing and instead type: throws in the towel after 1 second tories, you can grab the
(1,000ms). source or Linux version
nmap p <port> Purists who find the attractive built for 32- and 64-bit
output shown in Figure 1 too psy- systems [1].
Dont cheer too soon, though, be- chedelic, or cave dwellers sitting Firewall in the way during
cause often an intrusion prevention in front of terminals or dot matrix ping testing? I dont care! n
system blocks the Nmap signature of printers, can use the nocolor op-
the workaround. tion. This kicks Paping back into the
Paping [1] establishes a connection monochrome era. Info
to a configurable port and immedi- So, will Papings programmer Mike [1] Paping:
ately terminates it again. This works Lovell be building an API for Android [https://code.google.com/p/paping/]
almost anywhere. It measures the
round-trip time in milliseconds, just
like ping. The simplest form of the
call is very intuitive:

paping <target>

The tool can handle optional param-

eters, too (Figure 1); a typical call
looks like:

paping www.google.com p 80 c 4

The p parameter specifies theTCP

port to target 80 in this case
whichwill return the expected Figure 1: Paping pinging port 80 on Googles web server. No matter what lies in between, the web server will
results for a web server. By using c respond after an average of 48ms in this case.

12 1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I W W W. A D M I N - M AGA Z I N E .CO M

The sys admins daily grind: httpstat
My Point of View
This simple tool wil help you discover how long a web server takes to serve up
an HTML page. By Charly Khnast
Httpstat is a Python script that wraps
itself around cURL. Apart from Py-
thon 2 or 3 and cURL, it has no other
dependencies. You can retrieve it from
the GitHub repository and call it using:
wget https://raw.githubusercontent.com/
reorx/httpstat/master/httpstat.py
python httpstat.py <URL>
Figure 1: The page that httpstat requests via HTTP, and receives quickly, is only an error message in reality.
Figure 2: When requested using HTTPS, the HTML page obviously takes longer to appear.
W W W. A D M I N - M AGA Z I N E .CO M
httpstat ANOTHER 10 TERRIFIC TOOLS
https:// for web pages secured with
TLS.
Figure 1 shows httpstat measuring an
unencrypted call. Four milliseconds for
a DNS reply is a really good value, but I
If the Python installer pip is present cheated: The name of the site is cached
on your system, you also can pick up on my local Dnsmasq. As soon as my
the script and call it with: computer has to turn to my providers
DNS, the value rises to 80-200ms. The
pip install httpstat TCP handshake is 22ms, which is about
httpstat <URL> par for the course.
U The time the server needs to create the
Although you can leave an http:// page (Server Processing) shows whether
out of the URL, you cannot omit the web server has some tuning poten-
tial that I have not tapped. My example
is not representative, because instead
of HTML, the server simply outputs
301 Moved Permanently, which means
I should have called the page using
HTTPS. A browser would do that inde-
pendently, but not cURL.
Figure 2 requests the same page using
HTTPS. The lookup and TCP values re-
main the same, but the TLS Handshake
takes forever for this static page. The
value can go up to several seconds for a
big site with a large volume of dynamic
content and advertising banners.
Httpstat is not controllable using
command-line parameters because
they would be fielded by cURL; how-
ever, you can influence the tool with
environment variables. The line
export HTTPSTAT_SHOW_SPEED=true
tells httpstat to show how quickly the
web page is delivered (e.g., speed_
download: 219.6 KiB/s, speed_upload:
0.0 KiB/s). The httpstat website [1]
explains all of the variables and has
links to the httpstat implementation
in Go, Bash, and PHP. n
Info
[1] httpstat:
[https://github.com/reorx/httpstat]
1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I 13
 ANOTHER 10 TERRIFIC TOOLS Five Log Monitoring Tools

Five lean tools for monitoring logfiles

Small Supervisor
Anyone who wants to watch logfiles can use a
full-featured monitoring tool like Nagios; however,
a lightweight alternative also sniffs out threats
and takes much less time to set up. By Tim Schrmann

as LOGalyze [1], Logcheck [2], the Raspberry Pi, as well as on servers

Logwatch [3], MultiTail [4], and
A systems SwatchDog [5] can help you here.
logfiles not Unlike with large monitoring solutions
only record failed like Nagios and Icinga, the minor vari-
login attempts by users, ants focus on analyzing logfiles. They
but they also log program er- use fewer resources and can be set up
rors and information about attacks. much more quickly. They are there-
Admins therefore should keep a fore ideally suited for use on weak
continuous eye on them. Tools such hardware and embedded devices like
with few selected services.
All candidates use one or more log-
files and filter out important messages
according to predefined rules. As an
option, they can send the result by
email to the administrator or output
it on the command line. Admins can
also add their own filter rules, usually
in the form of regular expressions.

Table 1: Tools for Logfile Monitoring

Name LOGalyze Logcheck Logwatch MultiTail SwatchDog
URL http://www.logalyze.com http://logcheck. https://sourceforge. https://www.vanheus- https://sourceforge.net/
alioth.debian.org net/projects/logwatch/ den.com/multitail/ projects/swatch/
Tested version 4.1.4 1.3.18 7.4.3 6.4.2 3.2.4
License GNU GPLv2 GNU GPLv2 MIT license GNU GPLv2 GNU GPLv2
Filtering / with yes / no yes / yes yes / no yes / yes yes / yes
regular expressions
Notification by email yes yes yes no (via external pro- yes
gram)
Permanent yes yes yes yes yes
monitoring of a log
Lead Image nlshop1, 123RF.com

Unique processing of no no yes no yes

a complete log
Information about no yes (limited) yes (limited) no no
security problems
Summary / statistics yes no yes no no
GUI yes no no no no

14 1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I W W W. A D M I N - M AGA Z I N E .CO M

Figure 1: LOGalyze displays statistics directly in the web interface as a pie, bar, or line chart. Here it is clear
that the number of events dropped in recent times.
Sometimes the developer provides
a set of oft-needed rules. Powerful
tools may also put together a report
about the state of the system and in-
dicate security problems. However, a
comparison of the above candidates
shows that these functions are not a
matter of course (Table 1).
LOGalyze
LOGalyze [1] comes from the Hungar-
ian company Zuriel Ltd. The formerly
proprietary tool may now be available
under GPLv2, but the developers are
still keeping the source code under
lock and key. The latest version 4.1.4
was released in December 2016, but it
only fixes minor bugs from the almost
four-year-old previous version. LOGa-
lyze therefore still relies on Oracles
Java runtime environment in the
completely outdated version 1.6.
A short and concise guide in PDF
format provides a description of the
installation. Administrators can con-
figure LOGalyze using a supplied
web application that requires one of
the application servers Apache Tom-
cat, Jetty, GlassFish, or JBoss. The
hopelessly outdated Tomcat 6.0.35
from 2011 is included with the in-
stallation package; however, it can
W W W. A D M I N - M AGA Z I N E .CO M
Five Log Monitoring Tools
be quickly booted using a prepared
script.
Initially, administrators can create
one or more collectors in the user
interface. These collectors retrieve the
log data via the network or from a
file. Admins can switch each collector
on or off individually. LOGalyze then
generates statistics and reports from
the imported data and summarizes all
Figure 2: LOGalyze has found three events for the ubuntu computer.
1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I
ANOTHER 10 TERRIFIC TOOLS
critical errors, for example, in a con-
cise report (Figure 1).
Admins can also create their own
statistics and reports by clicking the
corresponding criteria in the user
interface. LOGalyze then generates
either a PDF or CSV file on this basis.
Admins can either download these
files or have them sent by email.
They can also search the logfiles
for terms. LOGalyze may not allow
regular expressions, but it does link
several search terms using operators
like AND and OR. It stores frequently
required search queries to allow
quick retrieval later via mouse click
(Figure 2). LOGalyze provides plenty
of predefined searches which, among
other things, quickly list all the errors
from the syslog.
Logcheck
The Debian project currently looks
after Logcheck [2], which is available
under GPLv2. It independently as-
sesses logfiles for problems, security
vulnerabilities, and possible intrusion
attempts. After its started, Logcheck
accesses the syslog and the auth.log
by default. However, you can use the
tool on other logfiles.
Logcheck compares all records added
since the last test with a load of pre-
15
 ANOTHER 10 TERRIFIC TOOLS Five Log Monitoring Tools
Figure 3: Ubuntu users can start Logcheck with the help of the logcheck user. The o parameter outputs
the events in the terminal. The outputs can be quite confusing.
set filtering rules. Depending on the
result, the tool either moves directly
to the next entry or classifies it as an
important system event, security is-
sue, or warning. Logcheck then sends
all the events from the last three
categories in an email to the adminis-
trator or writes them in the standard
output (Figure 3).
Administrators can choose between
three filter levels to maintain an over-
view: The highest, called paranoid,
is intended for particularly safety-
relevant systems with a few selected
services. In this filter level, Logcheck
provides an especially large number
of detailed messages that it would
discard in the other levels. The Server
level is the default, and there are the
least messages in the Workstation
level. Logcheck sends warnings about
security issues and attacks in each
filter level. To ensure that the tool
only reports each system event once,
it remembers the last position in the
logfiles to be assessed with the help
of the Perl script Logtail.
All filter rules are available as regular
expressions, so that admins can add
their own, as desired (Figure 4). To
provide a better overview, all of the
expressions for a service, such as the
Apache web server, are moved to
a separate configuration file. When
started, Logcheck automatically
16 1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I
imports all configuration files. The
developers kindly provide a package
with several of these configuration
files. However, the rules contained in
this Logcheck database only cover a
few basic errors and particularly im-
portant attack patterns.
Most distributions have Logcheck in
their repositories. On Debian systems,
a cron job initiates Logcheck every
hour, and the attentive tool is auto-
matically activated at every system
startup. Any suitable command-line
program assumes the responsibility
for sending emails the task is as-
signed to Sendmail by default.
Logwatch
Like Logcheck, Logwatch [3] is wait-
ing to be installed in the repositories
of most major distributions. The tool
is available under the MIT license
and requires Perl 5.8. Once Logwatch
Figure 4: Among other things, in the Server filter level, Logcheck discards all the events that correspond to
these regular expressions in the /etc/logcheck/ignore.d.server/sudo file.
starts, it accesses all the logs known
to it and checks all the events from
the last 24 hours in them. Admins can
extend or shorten this observation pe-
riod at their discretion.
Unlike with the competitor Logcheck,
Logwatch generates a concise sum-
mary (Figures 5 and 6) from the read
events. A separate section is devoted
to each of the services running on the
system, and the information displayed
there is based on the respective ser-
vice. For example, Logwatch lists all
the packages installed in the past 24
hours for the package manager dpjk.
If Logwatch was unable to meaning-
fully interpret an event, the tool sim-
ply attaches this event to the report as
an attachment.
The administrator informs Logwatch
about numerous configuration files,
about which services are running
on the system, and about in which
logfiles the services usually store
their information. In turn, the con-
figuration files are distributed across
several subdirectories. Logwatch
provides finished configuration files
for many important and well-known
system services, and some distribu-
tors supplement them with additional
services. Thanks to these specifica-
tions, Logwatch scours more logfiles
immediately after the installation
than its competitor Logcheck.
A specialized Perl script analyzes a
services logfiles. For example, the /
usr/share/logwatch/scripts/services/
dpkg script processes the logfiles of the
package manager dpkg. If Logwatch
is to analyze an individually compiled
service for the administrator, it needs
to be able to write an evaluation script
in Perl. A detailed how-to included
with Logwatch helps here.
W W W. A D M I N - M AGA Z I N E .CO M

Figure 5: Logwatch can also generate the summary in text form
The analysis scripts import the events
from the logfiles, generate a sum-
mary, and pass it on to Logwatch. Ul-
timately, the tool sends the collected
results in an email, writes them to
a file, or delivers them via stdout.
In the past, Logwatch presented all
information on a simple HTML page
upon request (Figure 6). The final
report also contains some system in-
formation, for example, the available
disk space.
Admins can also dictate the detail
of Logwatchs report. There are a
total of 10 detail levels available. The
individual evaluation scripts deter-
mine which (additional) information
each level of detail produces. The
distributions generally start Logwatch
via cron job once at night and send
the generated report via email. The
tool delegates the actual sending to
Sendmail or another, freely selectable
command-line program.
W W W. A D M I N - M AGA Z I N E .CO M
Five Log Monitoring Tools
tool does not have any pre-made reg-
MultiTail
MultiTail [4] presents the ends of
several text files in only one (termi-
nal) window (Figure 7). Additional
Figure 6: or as a HTML file with rather small font.
1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I
ANOTHER 10 TERRIFIC TOOLS
features were added over the course
of time, in particular filter and moni-
toring functions for logfiles.
The admins can thus filter the flood
of information using regular expres-
sions. If a regular expression applies,
MultiTail launches an external pro-
gram upon request. You can be sent
email notifications, for example. Al-
ternatively, MultiTail works like a vi-
sual pipe in that it writes the filtered
information in files or forwards it to
other processes. MultiTail can even
act as a syslog server itself and accept
outputs from other programs upon
request such as netstat.
The tool also highlights the row con-
cerned and attracts attention via a
beeper. An admin can highlight indi-
vidual events in specific colors using
regular expressions. For example, you
can highlight in red all rows starting
with Error. MultiTail also automati-
cally converts inputs. It converts IP
addresses into the appropriate do-
main name, converts signal numbers
into names, and supplies each date in
the local format.
However, administrators need to set
up MultiTail completely on their own.
Unlike Logcheck or Logwatch, the
ular expressions. Admins also need
to manually configure the forwarding
and sending of emails. MultiTail is
included with all major distributions
and is available under GPLv2. The
17
 ANOTHER 10 TERRIFIC TOOLS Five Log Monitoring Tools
Figure 7: Admins can determine the order of the logfiles themselves in MultiTail. They just need to depict two
logfiles one above the other.
Listing 1: Configuration File .swatchrc
01 watchfor /warning|error/
02 echo
03 mail addresses=tim\@example.com, subject=error occurred
tool also comes with a detailed man-
ual in HTML format.
SwatchDog
The Simple Log Watcher, Swatch for
short [5], began as a small watchdog
tool that monitored a syslog for activi-
ties. The program, which is available
under GPLv2, now digests any logfiles.
Formally, the command-line tool is
called SwatchDog to avoid any confu-
sion with a well-known Swiss watch
Figure 8: SwatchDog checks the whole /var/logs/auth.log file once here based on the examine parameter.
18 1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I
manufacturer. However,
in most distributions, it
is in the swatch package,
and the man page is the
only documentation.
The tool itself consists of a small Perl
script that assesses the logfiles stated
via parameter (Figure 8). Swatch-
Dog either goes through all the rows
contained in the files or continu-
ously monitors the file. In the latter
case, SwatchDog can be started as a
daemon and thus move to the back-
ground. Upon request, the tool also
accepts log data via a pipe.
In any case, you can specify in a con-
figuration file for which events the
tool needs to perform which actions.
added and customized filtering rules.
This configuration file uses Swatch-
Dogs own syntax Listing 1 shows
a simple example. According to the
instructions shown there, SwatchDog
needs to search for the keywords
warning and error. Administrators can
use a regular expression for specify-
ing the search pattern.
SwatchDog then performs all the ac-
tions that follow in the indented list.
For example, echo outputs the corre-
sponding line from the logfile on the
console, while mail sends the mes-
sage by email with the subject line
Error occurred to tim@example.com.
SwatchDog also calls any programs
(exec) and forwards the affected event
via pipe (pipe command). Perl experts
can store Perl code, which the tool
executes.
Users start SwatchDog on the com-
mand line by default. Administra-
tors need to create a suitable cron
job or systemd units themselves.
Unlike with Logwatch, SwatchDog
does not provide an example con-
figuration. Users should therefore
initially plan a bit of time to write a
suitable configuration file.
Learn to Love the Dog
The choice of the appropriate tool
massively depends on the specific
requirements and your personal pro-
gramming skills. None of the five
candidates can replace a full-scale
monitoring system, let alone an intru-
sion detection system. In any case,
administrators need to interpret the
sent system events themselves. See
the Old Comrades box for some
other alternatives.
LOGalyze provides a GUI and can
also be remotely operated via your
browser. However, anyone who wants
to use the tool should remember
the tools age. The supplied Tomcat
version also needs to be replaced as
quickly as possible. Admins also need
to be able to figure out for themselves
how to use LOGalyze.
Logcheck can be put into operation
particularly quickly. Anyone who
masters regular expressions can re-
duce the flood of data using quickly
W W W. A D M I N - M AGA Z I N E .CO M

While Logcheck only sends the na-
ked events to the admin, Logwatch
provides the admin with a summary.
If admins want to monitor their own
Figure 9: As this diagram of Petit shows, a lot of events were entered in the first five minutes of the log recording.
Old Comrades
Anyone searching the Internet for other
alternatives to the featured tools will auto-
matically stumble upon a few representatives
that are now obsolete. The Logdigest [6]
tool works like Logcheck, but has been on
ice since 2009. LogSurfer [7] is pretty
similar to SwatchDog, but can also group
similar events. In addition, LogSurfer is
written in C and should therefore work
much more quickly. However, the most
recent version of the tool was released in
September 2011.
Five Log Monitoring Tools
services using Logwatch, they need
Perl scripts.
MultiTail is worthwhile for admin-
istrators who literally want to keep
Petit [8] is about the same age, but it is still
in the repositories of Ubuntu. The tool uses
language analysis methods to curb the flood
of data, especially in system logs. This allows
administrators to, for example, list all words
that occur particularly frequently in a logfile. In
addition, the tool draws a graph that presents
the number of messages in a given period of
time (Figure 9). The hash function, which keeps
track of similar messages in the log, is also
interesting. It allows the viewer to immediately
see which errors occur most frequently.
ANOTHER 10 TERRIFIC TOOLS
an eye on several logfiles and only
want to trigger actions in certain
cases. Sending emails and forward-
ing filtered events may be possible;
however, to do so admins need to
write suitable regular expressions
and manually configure MultiTail.
The tool is therefore useful as a very
good supplement to Logcheck and
Logwatch.
Finally, SwatchDog is comparable to
Logwatch: It can be set up quickly
but requires knowledge of regular
expressions. Additionally, it only re-
ports individual events specified by
the administrator. n
Info
[1] LOGalyze: [http://www.logalyze.com]
[2] Logcheck:
[http://logcheck.alioth.debian.org]
[3] Logwatch: [https://sourceforge.net/
projects/logwatch/]
[4] MultiTail:
[https://www.vanheusden.com/multitail/]
[5] SwatchDog:
[https://sourceforge.net/projects/swatch/]
[6] Logdigest: [https://sourceforge.net/
projects/logdigest/]
[7] LogSurfer:
[http://www.crypt.gen.nz/logsurfer/]
[8] Petit:
[http://crunchtools.com/software/petit/]
ANOTHER 10 TERRIFIC TOOLS Web Analytics Tools
Traffic analysis tools for websites
Data for Breakfast
If you are looking for an alternative to Google Analytics, try a free
alternative such as Piwik, Open Web Analytics, or eAnalytics.
By Ferdinand Thommes
Admins who wanted details of the
visitors to their websites in the early
years of the Internet had to labori-
ously read the web servers logs. The
first log file analysis applications ap-
peared 20 years ago. Analog [1], We-
balizer [2] and AWStats [3], which
date from this period, are still occa-
sionally in use (see the Simple Web
Analytics Tools box).
In 2005, Google launched Google
Analytics (GA) [4], a website analy-
sis service that is hugely popular
today. Open source tools such as
Piwik [5] picked up on this trend
towards graphical web analytics, but
moved its focus to the customers
own server.
With the help of web analytics, site
operators collect and evaluate data
on the surfing habits of their visi-
tors. The access data are of interest
not only for commercial reasons;
the companies behind the sites also
often seek to better understand their
customers and their interests. The
following applies: The closer an op-
erator knows the visitors and their
preferences, the better the operator
can optimize its offerings to suit the
target group.
20 1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I
Good to Know
Site operators are often interested in
where the visitors come from, what
they are looking for, what items they
click on, and how long they remain
on the site. It can also be useful to
know when they leave the site. Ad-
mins want to know what browsers
and operating systems visitors to the
site use, which files and documents
they download and with what band-
width, and how many visitors sub-
scribe to newsletters or RSS feeds.
Web shop operators are interested
in how many visitors add goods to
their shopping carts, to then purchase
them, or possibly not. If a website
hosts advertising for third parties,
web analysis is essential, because ac-
cess figures and similar factors deter-
mine the prices for advertisers.
Open Access
The market offers many different web
analytics tools. They include around
150 commercial, typically proprietary
applications, aimed at larger corporate
websites. There are also some free and
partly also open source tools. This ar-
ticle looks at Piwik, Open Web Analyt-
ics [7], and eAnalytics [8] (Table 1).
From a technical point of view, the
web analytics tools either prepare
web server logfiles, or special tags
integrated into the HTML web pages
giving the admin statistics and graph-
ics for a quick overview and access to
all necessary key indicators. Although
the server-based method analyzes the
logfiles of the web server, developers
of the client-based variant add tracking
pixels into the source code of the web
page to determine the key indicators.
Although none of the two methods fully
represents the actual traffic of a website,
the client-based system of counting pix-
els, combined with the controversial use
of cookies, is currently just about win-
ning the accuracy stakes.
Privacy Issues
Because they evaluate cookies and
store the visitors IP addresses, web
analytics tools always face a difficult
legal situation. For example, Germa-
nys Telemedia Act (TMG) [9] allows
you to create user profiles if the user
does not object to the purposes of
Lead Image bowie15, 123RF.com
advertising and market research. Such
a profile is only allowed to contain an
anonymized IP address in addition to
the data on the use of the website. IP
addresses are typically automatically
truncated to this end.
W W W. A D M I N - M AGA Z I N E .CO M

Simple Web Analytics Tools
Many system administrators are quite happy
with the simpler, resource-friendly log evalua-
tions provided by statistics tools.
The oldest open source tools include Analog
developed in 1995 and Webalizer first released
in 1997. Both applications are still regularly
updated today. The tools evaluate the logs
several times a day, when run by the admin or
a cron job. AWStats is also a simple analysis
program. It has generated statistics about
web page visits since 2000 and is still under
Figure 1: GoAccess demo application in the browser.
The TMG also requires the service
provider to inform the user in a pri-
vacy statement on the website of
whether, to what extent, and for what
purpose it processes the IP address.
And, the TMG stipulates that users
Table 1: Three Web Statistics Tools at a Glance
Piwik
Platforms Cross-Platform
License GPLv3 and others
Under development since 2009
Language PHP
Methods JavaScript tags, log analysis, tracking pixels JavaScript tags, log analysis, tracking pixels eAnalytics tag, tracking pixels
Functions Visitors (visitors, unique visitors),
operating system, browser version,
downloads, IP address (pseudonymization
capable), geolocation by city, page
impressions, referrer, plugins
W W W. A D M I N - M AGA Z I N E .CO M
Web Analytics Tools
active development. The script, implemented
entirely in Perl, uses logfile analysis on web,
mail, and FTP servers to produce its reports
as HTML pages. Simple bar charts graphically
enhance the results.
GoAccess [6] (Figure 1) gives the admin the
ability to output and continuously update anal-
yses in real time in a terminal or in a browser.
GoAccess can handle virtually any log format
used by Apache, Nginx, Amazon S3, Elastic
Load Balancing, CloudFront, and others.
must have an option to object to the
creation of user profiles.
Probably the most controversial and
at the same time most successful tool
for website traffic analysis pages is
the Google Analytics online service,
Open Web Analytics (OWA)
Cross-Platform
GPLv2
2009
PHP
Visitors (visitors, unique visitors), operating
system, downloads, browser version, IP
address, geo location by country, page
impressions, referrer, heat maps
1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I
ANOTHER 10 TERRIFIC TOOLS
which was launched in 2005, and
which 50 percent of all websites
employ. It is clearly the top dog. In
contrast to the applications covered in
this article, the data collected by GA
leaves users computers and heads to
the United States, where data protec-
tion provisions are not as stringent as
in Germany and the rest of Europe.
For example, GA delivers the un-
abridged IP address to the parent com-
pany. Also, the website visitor may not
necessarily be informed of the fact that
Google is collecting its data. Browser
add-ons like Ghostery or NoScript can
disable GA [10] to provide protection
against unwanted data collection. GA
doesnt cost anything up to a traffic
volume of 10 million hits a month, but
it only delivers certain data following
a 24-hour delay. Also, the user has to
agree to Googles using the data for its
own purposes.
Piwik
Piwik (Figure 2) is growing in popu-
larity around the world. Users have
downloaded the cross-platform inde-
pendent, open source program, which
is written in PHP, almost three million
times since 2008. Fans of GA will
most likely find the functions they are
familiar with from Google in Piwik,
Site visitors are offered an opt-out
in an IFrame, and Piwik respects the
browsers Do Not Track feature.
Piwik collects data with tracking pix-
els, JavaScript, log analysis, and cook-
ies, and stores these in a MySQL da-
tabase. Access is via a web interface.
The latest version, 3.0.1, introduces a
eAnalytics
Debian/Ubuntu
AGPLv3
2011
Java and others
Visitors (visitors, unique visitors),
operating system, downloads,
browser version, IP address
(pseudonymization capable, can be
switched off), geolocation by city,
page impressions, referrer, plugins
21
ANOTHER 10 TERRIFIC TOOLS Web Analytics Tools

new user interface on the basis of Ma-
terial Design and Angular 1.4. Piwik is
available under the GPLv3, but partly
also under the BSD license.
Data Collection
Piwik Web Analytics collects relevant
data such as visitor counts, keywords,
referrers, and much more. This data
tool prepares the data in a graphically
appealing way and delivers the results
in the form of reports. These include
statistics on page views and unique
visits. The visitor analysis also provides
information on the countries of origin
and the browsers and operating systems
used. Referrers tell the operator which
website sent a visitor to their offering.
The tool relies on plugins to implement
its functions; you can add or remove
these as needed. Piwik supports real-
time updates, shows developments
and trends, offers campaign and target
tracking for online stores, and is multi-
client capable for multiple websites.
Thanks to the configurable dashboard,
admins can manage all their sites at
a glance. A tool for aliasing the IP ad-
dresses is also part of the package and
thus ensures data protection. There
are also corresponding apps for the
iOS and Android platform. Last but
not least, users will find a detailed
list of features with in-depth expla-
nations [11] on the project website.
Admins can use plugins to extend the
already abundant wealth of features
that Piwik comes with out the box.
A recent addition is the premium pl-
ugins, of which the project recently
presented three [12]. Admins need
to store Piwik software on the server
and then install the system in a
browser. If needed, you can integrate

Figure 2: Piwik is the best known and functionally the most similar open source alternative to Google Analytics.

Figure 3: Piwik can be combined with various web applications, for example, WordPress.

22 1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I W W W. A D M I N - M AGA Z I N E .CO M

Figure 4: You can try out Open Web Analytics with the aid of an online demo.
Piwik via enhancements in WordPress
(Figure 3), MediaWiki, Magento,
Joomla, vBulletin, and more than 60
other applications.
A demo version is available [13] on
the project website. Piwik Pro sees
the application run in the cloud [14].
Piwik is under active development
and has a fairly lively community.
Web Analytics Tools
Open Web Analytics
Under development since 2009, Open
Web Analytics (OWA) is not as well
known as Piwik, but it keeps pace
with Piwik and GA in terms of its
feature set, even outpacing them in
places (Figure 4). For example, it
offers integrated heat maps, which
ANOTHER 10 TERRIFIC TOOLS
competitors need to load as exten-
sions. They help admins analyze
mouse movements on web pages.
OWA uses a PHP front end with a
MySQL back end and collects statis-
tics by integrating a JavaScript into
the target site (Figure 5). Admins
can evaluate the results using Ja-
vaScript, but also directly via PHP
ANOTHER 10 TERRIFIC TOOLS Web Analytics Tools
or REST-based APIs. OWA supports
WordPress or MediaWiki pages, and a
third-party extension exists for Drupal.
The OWA framework is released un-
der the GPLv2 license and is also suit-
able for campaign and e-commerce
tracking. Users can define reports and
dashboards that go beyond the stan-
dard selection to suit their needs. The
application lets you integrate vari-
ous web pages, which the OWA user
can aggregate or view individually.
It handles many administrative tasks
directly at the command line of the
server instead of in the browser.
The integrated event queuing is a
unique feature among the applica-
tions presented here. If the database
cannot process peak visiting times
quickly enough, it first writes the data
to a simple logfile (Flat File Based
Event Queuing, [15]) and then parses
it via a PHP call:
/path/to/php5 cli.php cmd=processEventQueue
Piwik offers a similar function in the
form of Queued Tracking [16], which
was added in the form of a plugin in
2015. But, in this case, the software
writes its data to a Redis instance rather
than directly to a classic database.
Figure 5: OWA collects statistics on referrers as shown here by integrating JavaScript into the landing page.
24 1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I
eAnalytics
eAnalytics (Figure 6) is not well
known internationally, but is popular
in Germany. The analysis tool, re-
leased by Integrated Analytics GmbH
five years ago as open source, is de-
signed for medium-sized enterprises.
It seeks to make the technologies
used in large-scale companies afford-
able for firms with fewer resources.
The focus is on merging data. eAna-
lytics seeks to meaningfully link data
from web analytics with enterprise
data from CRM systems, data min-
ing and warehousing, and marketing
systems.
The company offers support and man-
aged services for eAnalytics and will
build enterprise-specific extensions if
necessary. eAnalytics provides a simple
user opt-out like Piwik. At the same
time, it honors Do Not Track requests
from the browser. IP addresses can be
pseudonymized in the configuration or
not collected in the first place.
The software is released under the
AGPLv3 license. Data from the tags of
the page, external data from Google
AdWords, a proprietary Twitter exten-
sion, and company-specific data serve
as the data sources.
eAnalytics preconfigures 10 dash-
boards that can be extended using
widgets. At the same time, it offers
55 interactive reports. You can install
version 1.1.3 directly on the server as
a Debian package; the current version
is optimized for Ubuntu 14.04. Source
code is available on SourceForge, but
only for the older version 0.9.2 [17].
eAnalytics collects data with a JavaS-
cript tool named eAnalytics Tag [18],
which the server operator needs to
additionally integrate [19]. One ad-
vantage of the web analytics software
is that admins can distribute it to sev-
eral machines in environments with
many servers. For example, you can
set up a separate server for the Tags
component. You can explore eAnalyt-
ics upfront via a VMware image [20].
Conclusions
Fundamentally, the tools examined
here Piwik, OWA, and eAnalytics
do approximately the same thing as
GA. Given appropriate hardware, the
candidates can be used for websites
with several million page views a
day. The clear advantage is that all
three keep the data on your own serv-
ers. This makes it easy for admins to
W W W. A D M I N - M AGA Z I N E .CO M

comply with European data protection
regulations.
Whether hosting locally is an ad-
vantage or a disadvantage for you
is something you have to decide for
yourself. But if you want to install,
update, and maintain the applica-
tions yourself, you will need powerful
hardware in contrast to a scenario
with GA. Piwik and eAnalytics, at
least, offer supervised hosting.
If you completely reject GA, but do not
have sufficiently powerful hardware
for the mainline programs featured
here, you can turn as an alternative to
lean logfile analysis programs such as
AWStats, Webalizer, or Analog. These
may not provide the same wealth of
information as the analytic applica-
tions, but they do still prepare the data
in a clear-cut and graphically appeal-
ing way. They are fine if you only need
an approximate overview.
Piwik has the edge in terms of popu-
larity, dissemination, and developer
community, which improves reliabil-
ity and allows for long-term planning.
The situation is not so clear with the
Figure 6: eAnalytics is open source software from Germany. The figure shows the default view; the latest version of the software is available as a package for Ubuntu 14.04.
W W W. A D M I N - M AGA Z I N E .CO M
two other candidates. Although both
published their latest versions in the
last twelve months, growth is far
more restrained all told, and there are
fewer developers on board. If you are
interested in the integration of the ac-
quired data with your business data,
eAnalytics is the obvious choice;
however, this unfortunately means
doing without access to the source
code for the current versions. n
Info
[1] Analog: [https://en.wikipedia.org/wiki/
Analog_(program)]
[2] Webalizer: [http://www.webalizer.org]
[3] AWStats: [http://www.awstats.org]
[4] Google Analytics:
[https://analytics.google.com]
[5] Piwik: [https://piwik.org]
[6] GoAccess: [https://goaccess.io]
[7] Open Web Analytics:
[http://openwebanalytics.com]
[8] eAnalytics: [http://eanalytics.de]
[9] Telemedia Act (in German): [http://www.
gesetzeiminternet.de/tmg/__15.html]
[10] Google Analytics opt-out: [https://tools.
google.com/dlpage/gaoptout]
1 0 T E R R I F I C TO O L S FO R T H E B U SY A D M I N : 20 1 7 E D I T I O N S P O N S O R E D BY L P I
Web Analytics Tools ANOTHER 10 TERRIFIC TOOLS
[11] Piwik features:
[http://piwik.org/features/]
[12] Premium plugins:
[https://piwik.org/blog/2016/11/premium
pluginsnowavailablemarketplace/]
[13] Piwik demo: [http://demo.piwik.org]
[14] Piwik hosting: [http://piwik.org/hosting/]
[15] Event queuing: [http://www.
openwebanalytics.com/?cat=9&paged=3]
[16] Queued tracking: [https://plugins.piwik.
org/QueuedTracking]
[17] eAnalytics on SourceForge:
[https://sourceforge.net/projects/
eanalytics/]
[18] eAnalytics Tag:
[http://eanalytics.de/resources/
download/eanalyticsdownload.html]
[19] Tagging Guide: [http://eanalytics.de/
uploads/media/eAnalytics_Page_Tag-
ging_Guide_english_V1_7.pdf]
[20] VMware image:
[http://eanalytics.de/resources/
download/eanalyticsdownload.html]
Author
Ferdinand Thommes lives and works as a
Linux developer, freelance writer, and tour
guide in Berlin.
25