Splunk Architecture Workshop

We Got Splunk
Its Installed
Cool
Now What?

Who Am I?

Sean Wilkerson, Partner/Consultant, Aplura

Who Am I?

Sean Wilkerson, Partner/Consultant, Aplura

~15 Years of Network ! S"stems ! InfoSec

A #e$a%e& of 'e%eral (o)*ana)ement

+alf Spent #eplo"/*ana)e ',SS/SI*/SI-*

+alf Spent #eplo"/*ana)e ',SS/Splunk

. /e$ent SANS (o) *)mt Summits

Splunk Pro Ser0 Partner Sin$e .112

Splunk makes me happ"

Who Are You?

You 3now Splunk is /i)ht for You

You 3now 3e" Splunk Con$epts

You /e4uire Purpose5uilt Ar$hite$ture

You +a0e S$ale

5i) #ata

(ots of 6sers

(ots of #atat"pes

+i)hl" #istri7ute%

Splunk PS Work
*a8or A$ti0ities

App Inte)ration

App #e0elopment

-nhan$ements

/epairs
Ar$hite$ture

*i)rations

*er)in)/5reakout

6p)ra%es

#//+A

5uil% for Spee%/Si9e

5ut Isn:t Splunk -;?

5ut Isn:t Splunk -;?

Yes<<<an% no, err, let me e=plain

5ut Isn:t Splunk -;?

Yes<<<an% no, err, let me e=plain

Sin)le7o= $an 7e >ran%ma-as"

5ut Isn:t Splunk -;?

Yes<<<an% no, err, let me e=plain

Sin)le7o= $an 7e >ran%ma-as"

P?@ -nterprise Nee%s A -nterprise Plan

5ut Isn:t Splunk -;?

Yes<<<an% no, err, let me e=plain

Sin)le7o= $an 7e >ran%ma-as"

P?@ -nterprise Nee%s A -nterprise Plan

Compare% with SI*/SI-* Competitors@

5ut Isn:t Splunk -;?

Yes<<<an% no, err, let me e=plain

Sin)le7o= $an 7e >ran%ma-as"

P?@ -nterprise Nee%s A -nterprise Plan

Compare% with SI*/SI-* Competitors@

Splunk is a walk in the park

In$re%i7l" more fle=i7le

*ore transparen$"

5ut Isn:t Splunk -;?

Yes<<<an% no, err, let me e=plain

Sin)le7o= $an 7e >ran%ma-as"

P?@ -nterprise Nee%s A -nterprise Plan

Compare% with SI*/SI-* Competitors@

Splunk is a walk in the park

In$re%i7l" more fle=i7le

*ore transparen$"
Note@ P? B Pro?ip C ?hese are keepers

Content A0aila7le NowD

?alk@ aplura<$om/splunkli0e%$

5P@ aplura<$om/splunk7p


Si9in) Your #eplo"ment

See Pre0ious ?alkEsF

3e" -lements

Ar$hite$ture Supports Golume E#ata an% 6sa)eF

Carefull" Plan@ S"stems, Stora)e, an% Pla$ement

A%ministration

/eferen$e Ar$hite$ture in H5 *in



#eplo"ment Ser0er
E#SF@

*ana)es All
Splunk Confi)s
0ia Apps

Platform
In%epen%ent

-nsures
6niformit"

Allows for /api%

Chan)es


?he Colle$tion ?ier
$oul% in$lu%e@

S"slo) /e$eipt

'ile *onitor

#5 *onitor

S$ript -=e$ution
Consi%er@

/elia7ilit"

*ana)ement

-n$r"ption


Parsin) ,perations@

Sear raw %ata

Prep for In%e=

?imestampin)

-0ent 5reakin)

In%e=e% 'iel%s
I+'@

#eplo" !B . for +A

(ow Stora)e Nee%s

G*s Work >reat

Consi%er /e)ional
or ;onespe$ifi$


Splunk In%e=ers@

I+' ?asks

Write #ata

/etrie0e #ata
Plan as a #5
/e4uire 211& I,PS
G*s are tri$k"


Sear$h +ea% is
where the ma)i$
happens
?he S+ 4ueries the
In%e=ers
A$$ounts,
Permissions,
#ash7oar%s,
Sear$hes, an%
'iel%s, all li0e in
Sear$h

*i)rations, *i)rations, *i)rationsD

*i)rations, *i)rations, *i)rationsD
?his is m" fa0orite part of EIw&F

JWarnin)J

Splunk has man" options an% is 0er" fle=i7le

Sli%es here are to )i0e "ou i%eas

?his ma" not 7e 7est for "our use$ase

When in %ou7t@

/ea% #o$s

+it Splunk7ase

Call Support

Call PS

Prep Yourself

Splunk mi)rations are easil" %one

You $an %o almost an"thin) without %ata loss

C an% minimal %owntime

5ut, this is "our %ata< ?his is 0alua7le

6n%erstan% how %ata flows throu)h Splunk

6n%erstan% the states of %ata

6n%erstan% how Splunk $ommuni$ates

P?@ Plan out "our steps in a%0an$e

,l% a%a)e applies well to mi)rations
You $an ha0e it )oo%
You $an ha0e it fast
You $an ha0e it $heap

,l% a%a)e applies well to mi)rations
You $an ha0e it )oo%
You $an ha0e it fast
You $an ha0e it $heap
Choose an" two

*i)rate Solo to #istri7ute%

*i)rate Solo to #istri7ute%

*i)rate Solo to #istri7ute%

*i)rate Solo to #istri7ute%

No %ata was harme%

Steps@

P?@ Consi%er Splunk

#S

P?@ Separate Parsin)

*o0e Sear$h ?ier

A%% another in%e=er

/inse, repeat<

*i)rate S.#@ Separate Parsin)

Controllin) the parsin) is an important first step

Consi%er an Interme%iate +ea0" 'orwar%er

Allows $ontrol of the %ataflow

#is$rete apps to mana)e parsetime rules

5rokers ?CP sessions to 6ni0ersal 'orwar%ers

/e%u$es loa% on in%e=ers

It is like hittin) the Ktur7o 7uttonL

,ften %one with no %owntime an% no %ata loss

*i)rate S.#@ Separate Parsin)

Setup Splunk on new I+' host

Sparse $onfi), no in%e=, forwar%s all

%ata to Solo1

S"n$ all parsetime $onfi)s to I+'

from Solo1

+a0e 6's sen% %ata to new I+'

If possi7le, up%ate sen%ers with new

re$ei0er

Now 6'!I+'!Solo1

*i)rate S.#@ #ist Sear$h

Separate Sear$h, allows for hori9ontal s$alin)

?his affe$ts the user en0ironment

Shoul% plan for li)ht user %owntime

*i)rate S.#@ #ist Sear$h

Setup Splunk on Sear$h

No lo$al in%e=in)

S"n$ all sear$h $onfi)s from

Solo1 to S+1

#on:t for)et 6ser $onfi)s

6se opportunit" to simplif" with

%is$rete apps

P?@ Wat$h for app name

pre$e%en$e

*ake #NS $han)e of

CNA*- use% for Splunk

P?@ #isa7le/remo0e sear$hes

from Solo1

*i)rate In%e=er

In%e=ers $an 7e mo0e%, with proper plannin)

You must $onsi%er some important thin)s

When on, the" are like %ata7ases Eso, 7e $arefulF

When off, it is all flatfiles Eeasil" mo0e% or rs"n$F

#epen%in) on spe$ifi$ o78e$ti0e, $onsi%er

options Ewhat is Splunk $apa7le ofF

*i)rate In%e=er Steps

5uil% new In%e=er

/ea%" to *i)rate@

S"n$ $onfi)s from ,l%1 to New1

Stop %ata from )oin) to ,l%1

,l%1@ /estart Splunk, then stop Erolls 7u$ketsF

/s"n$ hot/warm, $ol% 7u$kets from ,l%1 to New1

/estart Splunk on New1

A%% New1 to S+ Etest to see Minternal on New1F

/e%ire$t I+' to sen% to New1

*er)e In%e=

Splunk %ata is in%e=e% in K7u$ketsL

?o)ether, the 7u$kets make up an in%e=

What if "ou want to $onsoli%ate in%e=es?

*er)e In%e= N 5u$ket 5ri)a%e

+ot 7u$kets are li0e E$urrentl" 7ein) written toF

Ne0er tou$h these<<<e0erD

Warm 7u$kets are rea%onl"

P?@ /s"n$ ina%0an$e of mi)ration %ate to prep

some warm 7u$kets an% shorten outa)e win%ow

?he tri$k with 7u$kets is not to $onfuse Splunk

/ename the 7u$ket I#s appropriatel"

*er)e In%e= N 5u$ket 5ri)a%e

-=amine a 7u$ket@ OSP(6N3M+,*-/0ar/li7/splunk/IN#-P/%7

rw 1 root root Q1R *a" 1S ..@5R <7u$ket*anifest
rw 1 root root 11 *a" 1S ..@QS Creation?ime
%rw=== Q root root S1RT *a" 1S ..@5Q db_1337!"#$1_1337!#%$&_
%rw=== Q root root S1RT *a" 1S ..@5U db_1337$3"#_1337!"7""_1
%rw=== Q root root S1RT *a" 1S ..@52 db_1337$%17_1337$1#_&
%rw=== . root root S1RT *a" 1S ..@QS >lo7al*eta#ata
rw 1 root root 111 *a" 1S ..@5R +osts<%ata
%rw=== Q root root S1RT *a" 1S .Q@11 hotM01MQ
rw 1 root root Q1. *a" 1S ..@52 <meta*anifest
rw 1 root root .UT *a" 1S ..@5R Sour$es<%ata
rw 1 root root 11R *a" 1S ..@5R Sour$e?"pes<%ata

*er)e In%e= N 5u$ket 5ri)a%e

Steps to mer)e in%e=es

5a$kup e0er"thin) first

Stop %ataflow into In%e=erEsF

/estart Splunk on In%e=erEsF Efor$es 7u$ket rollF

S"n$ 7u$kets from ol% into new

/ename 7u$kets su$h that 7u$ketI#s in$rement

P?@ 6se (arr" Wall:s rename<pl

Pur)e K<7u$ketmanifestL for$in) splunk to re7uil%

/estart Splunk on In%e=er new

#isa7le In%e= on ol%

Start %ataflow

*er)e In%e= /ename S$ript

I use (arr" Wall:s VrenameV perl s$ript E$omes with 67untu 7ut N,? Cent,S//+-(F

(arr" Wall:s VrenameV takes se%st"le mat$hes, so the followin) will work@
/7in/sh
$% OSP(6N3M#5/'A(G)'IN*)+/E%7W$ol%%7F
S?A/?B111
for i in ls rt% %7MJX %o S?A/?BOEEOS?A/? & 1FFX rename n0 Vs/%&O/OS?A/?/V Oi X%one

?he ls with rt% ar)s, sorts re0erse 7" time an% onl" shows the %ire$tories, so the ol%est 7u$ket
will 7e liste% first an% therefore will 7e the first 7u$ket in "our list to keep the or%er somewhat
the wa" it woul% ha0e 7een ori)inall"<

?he 7usiness with the S?A/? thin) helps to maintain a $ounter so "ou $an mo0e throu)h
them one at a time< ?his is important when "ou are %oin) 7oth the +,? then C,(# #5s, an%
therefore "ou nee% to start on a spe$ifi$ num7er<

(i$ense *aster *i)ration

(i$ense master was a%%e% with S<.

It uses a splunk% ser0i$e Et$p/212RF

In rare $ases "ou mi)ht want to mo0e the (*

Splunk %oesn:t like when the (* mo0es

Your s"stems ma" flipout temporaril"

Steps 0ar" 7ase% on sour$e an% %est of (*

+a0in) a #eplo"ment Ser0er helps a )reat %eal

+i)hCapa$it" Colle$tion

#ire$tion %epen%s on use$ase

S$ripte% Input

>oo% for li)ht output, not hea0"/fast %ata

If hea0"/fast, ha0e same s$ript lo) to file then Splunk monitor that file

S"slo)

,k for li)ht output, 7ut $onsi%er a s"slo) ser0i$e

A s"slo) ser0i$e is li)hter wei)ht with fewer $onfi)s an% up%ates

S"slo) writes to files, then Splunk monitors< 5uiltin relia7ilit"<

See 5P )ui%e a7out kernel/ser0i$e tunin)< ?his makes a 7i) %ifferen$e<

+A/#/

Splunk %oes +A 0er" well

Can 7e 7uilt in to e0er" element of the Ar$hite$ture

#/ is not intuiti0e

No e0ent repli$ation

#efine what +A/#/ mean an% mean to "ou

-nsure "our terms mat$h the features

+A Ar$hite$ture

'orwar%ers nati0el" $an loa%7alan$e outputs

Sear$h +ea%s $an 7e pooled to)ether

A sin)le outa)e of In%e=er1

/esults in that store% %ata not a0aila7le for

4ueries

No loss of new %ata sin$e in$omin) will )o to

In%e=er.

#/ Ar$hite$ture

No 7uiltin an% index-aware e0ent repli$ation

You ha0e se0eral options

+a0e Splunk forwar% %ata to alternate site

6se ,S tools to s"n$ %ata to alternate site

6se Stora)e to repli$ate %ata to alternate site

None of these are seamless

?his topi$ is an entire presentation in itself

#/ Ar$hite$ture N Splunk 'orwar%s

?he followin) are $ommon 4uestions
an% pro7lems that fre4uentl" arise
%urin) Pro Ser0 en)a)ements

What %oes %eplo"ment ser0er %o?

/emotel" $ontrols Splunk $onfi)s for -nt<

6ses Splunk apps

/uns with whate0er ri)hts Splunk has

6ni0ersal Esupports whate0er Splunk %oesF

Can 7e 0er" powerful an% fle=i7le

-soteri$ an% %o$s are )eneri$

P?@ We in$lu%e it in almost e0er" en)a)ement

Can We 6se #S 'or All Confi)s?
Yes "ou $an, 7ut is what what "ou want?
Its a WYSIWY> Eor WYSIWY-F worl% for users<
Wikis, C*S, >oo)le <<<
?he" want to e%it what the" see in a 7rowser<
P?@ 6se #S for all non0iewa7le $onfi)s
P?@ 5eware of lo$al user$han)es
P?@ 5eware of hostspe$ifi$ items
P?@ 5e Patient Ean% $he$k "our lo)sF

#o We Nee% 6', we ha0e <<<

#o We Nee% 6', we ha0e <<<
Splunk 6'

-n$r"ption, Compression, Che$ksum, (oa%

5alan$es, Au%itin), an% StoreY'orwar%

(i)htwei)ht to run an% mana)e

#oesn:t parse %ata Ethat is left for the ne=t tierF

6' free to install an% mana)e for -nterprise

P?@ /e%u$e Ser0i$e Impa$t
P?@ Chan)e "our passwor%

Sen% /aw #ata to In%e=ers?
You $an< In%e=ers are %esi)ne% to parse an% in%e= the raw %ata<
It %oesn:t mean "ou alwa"s shoul%<
P?@ In man" situations, use an Interme%iate +ea0" 'orwar%er

5rokers ?CP sessions from 6ni0ersal 'orwar%ers

Parses All /aw

/e%u$es In%e=er (oa%

In$reases Performan$e

See 5P )ui%e for man" other a%0anta)es<<<

P?@ -nsure Minternal is forwar%e%

I+' outputs<$onf
Zt$pout[
forwar%e%in%e=<filter<%isa7le B true
forwardedinde,-1-blacklist .
in%e=An%'orwar% B false
ma=\ueueSi9e B 11.S*5
%efault>roup B in%e=M$luster
Zt$pout@in%e=M$luster[
auto(5're4uen$"BT1
auto(5Btrue
$ompresse%Bfalse
ser0erBspin%e=er1@RRRU,spin%e=er.@RRRU

We Are #e$entrali9e%, Where #o
We Pla$e In%e=ers

Customers often pla$e In%e=ers near %ata

?he" 7elie0e this will sa0e on 7an%wi%th

?his assumption is usuall" false

>enerall", position in%e=ers near S+

-=$eption

If users are at 7oth lo$ations, then "ou ha0e to

%e$i%e

#eCentrali9e% -=ample 1@ 3C/P+P

All main operations were in 3ansas Cit"

Satellite offi$e in Phoeni= with slow link

Customer pla$e% In%e=er in P+P thinkin) the $entral

S+ woul% 8ust 4uer" on o$$asion

/esult is, all %ata is 4uerie% se0eral times per %a"<

/e4uires more 7an%wi%th than if S+/IN#P were lo$al

See ne=t 4uestion a7out %ata plannin)

#eCentrali9e% -=ample .@ NY/63

Central I? is in NY, ,ffi$es in NY an% (on%on

63 users nee% onl" their own %ata

NY users nee% onl" their own %ata

With an In%e=er in ea$h lo$ation, $entral I?

$oul% sear$h 7oth< ?his woul% re4uire
maintainin) E.F sear$h en0ironments

'rin)e $ase

We ha0e a lot of %ata< ?oo mu$h?

Capa$it" plannin) for s"stems/NICs/7an%wi%th

(oa%7alan$er, NICteamin), an% 11> oh m"

6se num7ers, not hun$hes Ewhen possi7leF

P?@ Compute perse$on% in$omin) a0era)e

Cal$ulate 0ia Splunk li$ense

Cal$ulate 0ia in$omin) %ata

\ui$k a0era)es %on:t a$$ount for peak/0alle"

Cal$ulate from Splunk (i$ense

?ake a 51>5 li$ense

#i0i%e 7" %ail" se$on%s E2T,S11F

51111111/2TS11B5U2 5/sX = 2 to )et 7its

5U2J2BST.S 7/s B S<Tk7/s

H<5] of a 1117ase? or <1115] of a 1> NIC

Cal$ulate from #ata

-=1@ #ail" s"slo) 0olume B 11>

11111111111 7"tesX multipl" 7" 2 an% )et

21111111111 7itsX %i0i%e 7" 2TS11 an% )et

R.5R.5 7its/s or R.5 k7/s

H1] of a 1117ase? or <111] of a 1> NIC

-=.@ -PS are 1,.11, a0) e0ent is .51 7"tes

.51J2 B .k7sX multipl" 7" -PS

.S11J.k7s B .S11k7s

HQ] of a 1117ase? or <11Q] of a 1> NIC

+ow %o I sear$h another S+?

You %on:t reall" sear$h the S+ %ire$tl"

Confi)ure the outputs.conf on "our S+ to sen%

results to the in%e=ers

/eferre% to as KSpra"in) #ataL

Impro0e sear$h performan$e

P?@ Plan for this e=tra %ata 0olume on in%e=ers

P?@ Create all summar" in%e=es on In%e=ers

+ow %o we test apps?

*an" $ustomers %eplo" test apps to pro%< S+

New apps ma" $olli%e with $urrent $onfi)

6nwante% apps ma" $onsume resour$es

P?@ Consi%er a test/%e0 Sear$h +ea% EG* worksF

?ies in to production %ata without $ausin) harm

?his is a )reat pla$e to tr" out new $onfi)s

You $an also test "our own apps here

?est up)ra%es here first

Is it ok to 8ust ha0e one sour$et"pe?

>enerall", no<

?he fiel% sourcetype is spe$ial to Splunk

If "ou are on7oar%in) all %ata 0ia s"slo) Ean%

one sour$et"peF $onsi%er sour$et"peMroutin)
'rom transforms.conf@
Zfor$eMsour$et"peMforM====[
*)S'_/)0 . 1eta*ata2Sourcet3pe
()G)+ . writeM"ourMre)e=
45(1A' . sourcet3pe22newMsour$et"pe

Is it ok to 8ust ha0e one in%e=?

Sure, its up to "ou<

In%e= separation allows AC(, /etention, Spee%

If "ou are on7oar%in) all %ata into main 7ut want

to separate some out, $onsi%er in%e=Mroutin)
'rom transforms.conf@
Zfor$eMin%e=MrouteMforM====[
*)S'_/)0 . _1eta*ata2Inde,
()G)+ . writeM"ourMre)e=
45(1A' . newMin%e=

In%e=/Sour$et"pe /outin)

In%e=/Sour$et"pe routin) happens at parsin)

If I+' are in use, then appl" the $onfi) there

If no I+', then usuall" appl" on the in%e=ers

When %o we $reate a new in%e=?
*" .<5 \uestions /ule for Creatin) a new In%e=@

When %o we $reate a new in%e=?
*" .<5 \uestions /ule for Creatin) a new In%e=@
1<#ata has a %ifferent retention poli$"? EY/NF

When %o we $reate a new in%e=?
*" .<5 \uestions /ule for Creatin) a new In%e=@
1<#ata has a %ifferent retention poli$"? EY/NF
.<#ata has %ifferent a$$ess restri$tions? EY/NF

When %o we $reate a new in%e=?
*" .<5 \uestions /ule for Creatin) a new In%e=@
1<#ata has a %ifferent retention poli$"? EY/NF
.<#ata has %ifferent a$$ess restri$tions? EY/NF
.<5 You want to spe$ifi$all" in$lu%e/e=$lu%e %ata
e<)< in%e=Bfoo or N,? in%e=Bfoo? EY/NF

When %o we $reate a new in%e=?
*" .<5 \uestions /ule for Creatin) a new In%e=@
1<#ata has a %ifferent retention poli$"? EY/NF
.<#ata has %ifferent a$$ess restri$tions? EY/NF
.<5 You want to spe$ifi$all" in$lu%e/e=$lu%e %ata
e<)< in%e=Bfoo or N,? in%e=Bfoo? EY/NF
If "ou answere% KYL at all, then $reate a new in%e=

+ow %o we know the %ata is ri)ht?
Gali%ate

+ow %o we know the %ata is ri)ht?
Gali%ate
GA(I#A?-



+ow %o we know the %ata is ri)ht?
Gali%ate
GA(I#A?-
G N A N ( N I N # N A N ? N -


+ow %o we know the %ata is ri)ht?
Gali%ate
GA(I#A?-
G N A N ( N I N # N A N ? N -
GA(I#A?-

Gali%ate N 11 *in< ?ime Au%it

?ime is 7roken Ke0er"whereL

5ut we ha0e N?P<<<

I*/I/ nee%s relia7le time

P?@ Perform a monthl" time

au%it Es$he%ule itF

Gali%ate N 11 *in< ?ime Au%it

?ime is 7roken Ke0er"whereL

5ut we ha0e N?P<<<

I*/I/ nee%s relia7le time

P?@ Perform a monthl" time

au%it Es$he%ule itF

Splunk Sear$h A sour$et"pe7"

sour$et"pe

'rom timepi$ker sele$t

K/ealtime/All?imeL

Wat$h K?imelineL for 0isual

4ueues

Gali%ate N #upli$ate #ata

#ou7le& %ata e=ists at almost e0er" $ustomer

Consumes li$ense, $onsumes stora)e

Ina$$urate count reportin)

#on:t monitor 7oth the raw s"slo) an% ar$hi0es

#on:t use s"slo) an% a 6' on the same host

P?@ \uer" for %ou7le%ata on o$$asion

Gali%ate N *issin) #ata

?oo often $ustomers are missin) 0alua7le %ata

You %is$o0er when "ou nee% it<<<too late

#eplo"ment *onitor helps 7ut isn:t perfe$t

P?@ A%% alert after on7oar%in) a new %ata t"pe

Can stora)e 7e a%%e% later?

Yes, "ou $an $han)e almost an"thin) "ou want

(e0era)e tiers effe$ti0el" Ehot, warm, $ol%F

A%%in) %ata mi)ht mean %ata mi)ration

#ata mi)ration $an 7e tri$k"

P?@ >et Stora)e 'olks In0ol0e% -arl"D

P?@ Confi)ure %ata retention 7efore it 7e$omes

an emer)en$"

?AuasMparser

6sera)ents are
%iffi$ult

?here are thousan%s

of them

Parsin) them is a
ni)htmare

People are alrea%"

%oin) this@ user
a)entstrin)<info

6ses their li7rar" to

enri$h 6sera)ent
information with
easier to use fiel%s

#omain Cate)ories

6ses %ata from

url7la$klist<$om

Not 8ust 7la$klistin),

7ut also $ate)ories
for %omains

Cate)or" information
for more than Q<5
million %omains

#ata at OT per month

Can 7e applie% to an"

e0ents with %omain
name information

We7 Pro=ies

-mail

#NS

Content A0aila7le NowD

?alk@ www<aplura<$om/splunkli0e%$

5P@ www<aplura<$om/splunk7p