FlexConnector deep dive

and best practices

Sunny Suen
Managing Principal, Asia
ESP Solutions Consulting

Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
FlexConnector deep dive and best practices
Introduction
Agenda Target audience
FlexConnector deep dive Go through FlexConnector training and
Customized event feeding options documentation
Advanced topics in FlexConnector development Have created FlexConnector in practical environment
FlexConnector best practices
Essential steps on FlexConnector configuration &
development
Reference
Best Practice of FlexConnector submission to improve Documentation
Maintainability FlexConnector Development Guide
Readability ArcSight Categorization Technical Note
Efficiency Other SmartConnector Configuration Guide
Accuracy Training
FlexConnector Training

2 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
FlexConnector deep dive

Customized event feeding options

Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
SmartConnector architecture
IDS
Event data Event data
SmartAgent
(ArcSight message)

Firewall
Event data Event data
SmartAgent
(ArcSight message)

Event data Event data

Unix Syslog SmartAgent
(ArcSight message) ArcSight manager

ArcSight message
Event ID
Event name
Base event count
Event data
Detect time
Target address
Source address
Device address

4 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Consideration for event feeding
SmartConnector vs. FlexConnector
SmartConnector FlexConnector
An application that collects raw events from security devices, The FlexConnector framework is a software development
processes them into HP ArcSight security events, and kit (SDK) that lets you create a SmartConnector tailored to
transports them to destination devices. the devices on your network and their specific event data.
SmartConnectors are generally one of the following types The available FlexConnectors are:
File Connectors Logfile FlexConnector (fixed-format)
Database Connectors Regex FlexConnector (variable-format)
API Connectors Database FlexConnectors
SNMP Connectors SNMP FlexConnector
Microsoft Windows Event Log Connectors Syslog FlexConnector
SyslogConnectors XML FlexConnector
ScannerConnectors Scanner FlexConnector
FlexConnectors REST FlexConnector
ModelConnectors Key-value FlexConnector (via Logfile/Regex
FlexConnector)

5 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Consideration for event feeding
Do we need FlexConnector?
First, capture the requirement
Capture all device log details
Vendor Name (e.g. ABCDE Technologies)
Product Name (e.g. ABCDE Web Server)
Software/Firmware version (e.g. version X.X)
Log Type (e.g. flat file)
Log format / transport (Free format text log)
Log rotation scheme (Daily; filename containing event log date: AccessLog_yyyyMMdd.log)
Event Type (e.g. access audit log)
Match supported device list of SmartConnector
If not in the list, seek for following options

6 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Consideration for event feeding
Do we need FlexConnector?
Options for customized event feeding
FlexConnector Development
For a complete development of parser/categorization on all required events
Identify FlexConnector format / transport
Reuse of SmartConnector parser/categorization
Partial development for unparsed/uncategorized events
Identify the similar type of parser/categorization
Map files/External Mapper
For further interpretation some event values
E.g. elaborating department name HR to Human Resource Department
Common Event Format (CEF)
Add/modify application log format as CEF output

7 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
FlexConnector development
Type of FlexConnectors
Logfile FlexConnector (fixed-format)
Regex FlexConnector (variable-format)
Database FlexConnectors
SNMP FlexConnector
Syslog FlexConnector
XML FlexConnector
Scanner FlexConnector
REST FlexConnector
Key-value FlexConnector (via Logfile/Regex FlexConnector)

8 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Reuse of SmartConnector parser

When I need this?

To reuse (convert) standard SmartConnectors parser of the required log transport type, such as
Convert a file-based FlexConnector to folder-based FlexConnector
A file-based FlexConnector is wrapped by syslog transport
How to use it?
Identify the folder/file name of standard parser file (from aup)
unzip -l {$Connector}/current/system/agent/arcsightagents.aup
e.g. apache/apache_access_file
Scenario 1: Converting standard file reader Connector to multi-folder Connector
Configure Multi-Folder FlexConnector to assign the configfile as the standard parser path
agents[0].foldertable[0].configfile=apache/apache_access_file
Scenario 2: Converting standard file reader Connector to syslog transport
Create Syslog FlexConnector with following extraprocessor statement
extraprocessor[0].type=regex
extraprocessor[0].filename=apache/apache_access_file
9 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Reuse of SmartConnector categorization

When I need this?

To reuse the standard categorization file of one supported device to the FlexConnector
How to use it?
Check in the agent.log on the categorization file being used
Identify the folder/file name of standard categorization file (from aup)
unzip -l {$Connector}/current/system/agent/arcsightagents_{date-version}.aup
e.g. apache/apache.csv

Create additional categorization file, e.g. newvendor/newproduct.csv
Include the folder/file name of standard categorization file into the link file, e.g.
Create a file newvendor/newproduct.link.csv in the acp/categorizer/current folder
Add following lines into newproduct.link.csv
/apache/apache.csv
/newvendor/newproduct.csv
10 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Map file revisit

When I need this?

To map additional values from event field(s) to assign new values
How to use it?
create map.n.properties (where is 1,2,3,etc in sequence)
put getter and setter mapping on the first line of the map.n.properties file, such as
Line 1: Getters and setters
event.deviceHostName,set.event.deviceCustomString1
Line 2 onwards: value mapping
Host1,HR Dept
Host2,FIN Dept
Type of getters
Exact match: Header Row=event.deviceHostName, data=Host1
Range: Header Row=range.event.destinationPort, data=10000-19999
Regex: Header Row=regex.event.deviceHostName, data=HR.*

11 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
External mapper

When I need this?

Perform external database query on event field for additional mapping information
How to use it?
Check the agent URI (with 2 equal sign (==) suffices)
Create the folder: user/agent/extmap/{agent URI}
Create the external mapper file extmap.n.properties (where n=1,2,3 in sequence) into this folder

12 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
External mapper
(Continued)
Add following lines into the extmap.n.properties
type=sql
field.getter=deviceAddress
field.setter.count=1
field.setter[0]=deviceCustomString1
#field.addrs.as.numbers=true
jdbc.class=org.gjt.mm.mysql.Driver
jdbc.url=jdbc:mysql://localhost:3306/threatIntel
jdbc.username=intel
jdbc.password=OBFUSCATE.4.8.1:LOX6GXaJ+5imr6M1wmwkNg==
jdbc.query=select address, threatVector from watchList WHERE address in (?\u0000?)
The password has can be created by this command
./arcsight agent runjava com.arcsight.agent.loadable._ExternalMapperComponent -p 'arcsight'

13 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
CEF format
Type of FlexConnectors
When I need this?
To adapt the application log to ArcSight CEF format. This can eliminate the FlexConnector parser
maintenance afterwards
How to use it?
Obtain the Common Event Format Documentation
Adapt the application log output to CEF format HP ArcSight
Create event categorization file based for all possible event types Common
Event
Format

14 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
FlexConnector deep dive

Advanced FlexConnector topics

Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Advanced topics in FlexConnector development

Scope
Advanced techniques commonly encountered in practical environment
Not currently discussed in documentation in details
Topics
Fragmented event lines
Character encoding

16 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Fragmented event lines

When I need this?

Some devices send single event in multiple log lines
To merge the information of all the related events into a single one
Handling options
multiline.regex
Regex (?s)
Line.ignore.regex
Event merging

17 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Event merging

When I need this?

In some instances the events sent by the device will not necessarily be close together, there
could be events that will be sent in between other events, such as

18 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Event merging
(Continued)
How to use it?
Use merge parameters:
merge.count=1 merge[0].ends.count=3
merge[0].pattern.count=2 merge[0].ends[0].token=NAME4
merge[0].pattern[0].token=NAME1 merge[0].ends[0].regex=RESULT
merge[0].pattern[0].regex=(BIND|UNBIND|MOD|R merge[0].ends[1].token=NAME5
ESULT) merge[0].ends[1].regex=RESULT2
merge[0].pattern[1].token=NAME2 merge[0].ends[2].token=NAME6
merge[0].pattern[1].regex=(BIND|UNBIND|MOD|R merge[0].ends[2].regex=RESULT3
ESULT)2 merge[0].timeout=60000
merge[0].starts.count=1 merge[0].id.tokens=conn|msgId
merge[0].starts[0].token=NAME3 merge[0].id.tokens.delimiter=|
merge[0].starts[0].regex=(BIND|UNBIND|MOD) merge[0].sendpartialevents=true
merge[0].capacity=100

19 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Character encoding

When I need this?

The raw log data contains non-ASCII characters
How to use it?
Configuration: agent.properties
The Multi Folder Follower FlexConnector
agents[0].foldertable[0].encoding=UTF-16LE
SNMP FlexConnector
snmp.charset={Your character set for the foreign language} (such as snmp.charset=big5)
Configuration: JVM option
For other FlexConnector (such as Syslog FlexConnector)
In ${CONNECTOR}/current/bin/scripts/connectors.sh (or connectors.bat), append following options in the
ARCSIGHT_JVM_OPTIONS:
-Dfile.encoding=(character encoding) (such as -Dfile.encoding=gb2312)

20 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Character encoding
(Continued)
Development: regex
main / submessage regex statement
convert non-ascii/multi-byte characters into unicode notation, such as \u7528\u6236 (Chinese
characters of User)
Conversion tool: native2ascii (included in JDK distribution)
$ native2ascii

\u7528\u6236

put unicode notation in regex statement

regex=\\u7528\\u6236: (\\S+)

21 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
FlexConnector best practices

Files to be created/configured in
FlexConnector development

Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Folder structure

FlexConnector Configuration
${ArcSight Connector Home}/current/user/agent
User Development Folders
flexagent: parser
acp: event categorization
fcp: standard parser / parser override
map: map file
extmap: external mapper
lib: jdbc driver

23 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Configuration
agent.properties
Location under {$Connector_Home}/current
user/agent
When I need it?
To tune the advanced configuration parameters, such as:
File Rotation
Filename extractor
Subagent list
Refer to agent.default.properties for more options
How to use it?
For Software Connector, modify the file directly in user/agent/agent.properties
For Connector Appliance, use Diagnostic Wizard and select agent.properties

24 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Development
parser (*.properties)
Location under {$Connector_Home}/current
user/agent/flexagent
Sections in properties file
Parser configuration
Token Declaration
ArcSight event field assignment
Severity mapping
Conditional mapping
Extraprocessor
Point to Note
Proper comment should be required
Sample messages
Message group
Section heading

25 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Development
File extension and Location
flexagent/
Log-file: <product_name>.sdkfilereader.properties
Regex Log-file: <product_name>.sdkrfilereader.properties
Regex Folder Log-file: <product_name>.sdkrfilereader.properties
XML Folder Log-file: <product_name>.xqueryparser.properties
Time-based Database: <product_name>/<table_name>.sdktbdatabase.properties
ID-based Database: <product_name>/<table_name>.sdkibdatabase.properties
Multi-Database: <product_name>/table <n>/<table_name>.sdktbdatabase.properties
Syslog: syslog/<product_name>.subagent.sdkrfilereader.properties
SNMP: <vendor>/sdksnmp.#.snmptrap.properties (where # = trap type)
Scanner (Normal Text): <vendor>.scanner.sdkrfilereader.properties
Scanner (XML): <vendor>.scanner. xqueryparser.properties
Scanner (Database): <vendor>. sdkdatabase.properties
REST: <product_name>.jsonparser.properties

26 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Development
Categorization (*.csv)
Location under {$Connector_Home}/current
user/agent/acp/categorizer/current/<device_vendor>/<device_product>.csv
File format
Getters and setters
Exact match, range, regex getters are all supported
Single categorization file: vendor1/product1.csv
Multiple categorization file: vendor1/product1.link.csv, vendor1/product.csv and vendor1/product.0.csv
Point to Note
File name capitalization
All folder and file names are in small letters
Space or non-alphanumeric letters are converted into underscore -, such as
AS/400 -> as_400
Microsoft Windows -> microsoft_windows
27 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Development
Categorization (*.csv) (Continued)
Point to Note (continued)
Try using Regex getter and put some catch all to enhance the categorization result, such as
Logon.* (For all undefined event type with Logon prefix)
.* (For all other undefined event type)
Required categorization fields
regex.event.deviceEventClassId,set.event.categoryObject,set.event.category
Behavior,set.event.categoryTechnique,set.event.categoryDeviceGroup,set.eve
nt.categoryDeviceType,set.event.categoryOutcome,set.event.categorySignific
ance,set.event.agentSeverity
Obtain the ArcSight Categorization Technical Note

28 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Key field assignment

Topics
Time fields:
deviceReceiptTime, startTime, endTime
Device name for categorization link:
deviceVendor / deviceProduct
Event type:
name/deviceSeverity/deviceAction/deviceEventClassId
Custom fields
Use deviceCustom* and avoid flex*
flex* fields are reserved for console mapping to additional data
Source-destination vs attacker-target
Only use source/destination pair (e.g. sourceUserName/destinationUserName)
attacker/target pair: result of correlation, not to be assigned in FlexConnector
Assignable fields (list of assignable fields in FlexConnector Developers Guide)
29 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Local testing (CSV/CEF output)

When I need this?

To test and debug the parser/categorization in the development phase, it will be more efficient
to verify locally rather than submitting to logger and ESM
How to use it?
For connector destination, choose CEF log file or CSV log file
Select header row to help readablility
For CSV log file, you can choose the required fields as output

30 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Packaging

For submission of FlexConnector for production usage, we recommend:

Single Package file
Folder structure following Connector installation standard
Put device vendor/device product/FlexConnector version number in the package name
Example
Package: $deviceProduct_flexconnector_vx.x_yyyy-mm-dd.zip
Content folder structure:
acp/
fcp/
flexagent/

31 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Documentation

Highly recommend to provide following documentation in the FlexConnector files

for ongoing maintenance:
Parser comments
Configuration Documentation

32 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Please give me your feedback
Session TB3033 Speaker Sunny Suen

Please fill out a survey.

Hand it to the door monitor on your way out.
Thank you for providing your feedback, which
helps us enhance content for future events.

33 Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Thank you

Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.