Remote System Monitoring (without using a

Forwarder from Splunk) ,but works with the *nix

App

Version: 1.0
Date: 22.04.2010

SPP, Lsungen im Team

Seite 1/6

Project
Project Leader
Responsible
Created
Last Change
Revision
Reference

Remote System Monitoring (without using a Forwarder from Splunk)

Alexander Sznyi
Alexander Sznyi
22.04.2010

Change log
No.
1

Date
22.04.2010

SPP, Lsungen im Team

Version
1.0

Author
Sznyi

Comment
Create Document

Seite 2/6

Table of Contents
1
2
3
4
5
6

Why ........................................................................................................................................................................................................4
Readme.................................................................................................................................................................................................4
Create new Scripts (Example: rtop.sh) ......................................................................................................................................5
Add Script Input at Splunk .............................................................................................................................................................5
Control your new Input...................................................................................................................................................................6
Summary ..............................................................................................................................................................................................6

SPP, Lsungen im Team

Seite 3/6

1 Why
Because it could be that the System Administrators do not want to install the Splunk Forwarder on their
Systems.

2 Readme
Requirements:

Install ssh on your *nix Systems

Example Ubuntu
# sudo apt-get install openssh-server openssh-client

Add on the system, that you want to Monitor, an new user

Example Ubuntu
Login at the Remotehost
# sudo adduser rmonitor

Copy the scripts you want to use from the *nix App from Splunk
Example with top.sh
Login at the Splunk Host (Indexer or Forwarder)
#
#
#
#

sudo su rmonitor
ssh rmonitor@remotehost mkdir p monitoring/scripts
scp /opt/splunk/etc/apps/unix/bin/top.sh rmonitor@remotehost:/usr/rmonitor/monitoring/scripts
scp /opt/splunk/etc/apps/unix/bin/common.sh rmonitor@remotehost:/usr/rmonitor/monitoring/scripts

Test the script , Login at the remotehost with the user rmonitor
# cd /usr/rmonitor/monitoring/scripts
# ./top.sh
Example Output:
PID USER
PR
NI
VIRT
1388 rmonitor
20
0
19148
1 root
20
0
19296
2 root
15
-5
0
3 root
RT
-5
0
4 root
15
-5
0

RES
1168
1544
0
0
0

SHR
S
876
1128
0
0
0

pctCPU
R
S
S
S
S

pctMEM
2
0.2
0
0.3
0
0.0
0
0.0
0
0.0

Check your systems for automatic login for ssh

Example Ubuntu
Login at the Splunk Host (Indexer or Forwarder)
#
#
#
#

sudo su xxx (user where splunkd is running)

ssh-keygen -t rsa
ssh rmonitor@remotehost mkdir p .ssh
cat .ssh/id_rsa.pub | ssh rmonitor@remotehost 'cat >> .ssh/authorized_keys'

!!! TEST !!!

#ssh rmonitor@remotehost ls

SPP, Lsungen im Team

Seite 4/6

cpuTIME COMMAND
0:00.02 top
0:01.98 init
0:00.03 kthreadd
0:00.03 migration/0
0:00.01 ksoftirqd/0

3 Create new Scripts (Example: rtop.sh)

Login at your Splunk Host
#sudo su splunk
#cd /opt/splunk/etc/apps/unix/bin
#touch rtop.sh
#chmod +x rtop.sh
#vi rtop.sh
script rtop.sh
#!/bin/sh
# Create by Alexander Szoenyi from Company SPP
# Example:
# ssh user@remotehost /home/user/monitoring/scripts/top.sh
#
ssh rmonitor@10.1.1.120 /home/rmonitor/monitoring/scripts/top.sh
!!! Save the script !!!

4 Add Script Input at Splunk

Login to Splunk WebGui
Launch to the *nix App
Go to the Manager -> Data inputs -> Scripts -> New
Source
Command
/opt/splunk/etc/apps/unix/bin/rtop.sh
Interval
60
Source name override (optional)
top

Host
Host field value (optional)
remotehost

Source type
Set sourcetype field for all events from this source.
Set sourcetype
Manual
Source type (optional)
top

Index
Set the destination index for this source.
Index
os

!!! SAVE and Enable your new Script Input !!!

SPP, Lsungen im Team

Seite 5/6

5 Control your new Input

Login to Splunk WebGui
Launch to the *nix App
Search
# index=os sourcetype=top
You can see a new host with the top output.

6 Summary
If you have test this with the top.sh script you can make the same with all other scripts at
/opt/splunk/etc/apps/unix/bin .
It is also an example for your own scripts that you want to index remotely.

SPP, Lsungen im Team

Seite 6/6