Category

Architecture
Architecture
Architecture
Architecture
Architecture
Architecture
Architecture
Authentication and Access Control
Authentication and Access Control
Certification
Certification
Certification
Collection
Collection
Collection
Collection
Collection
Collection
Collection
Collection
Collection
Collection
Collection
Collection
Collection
Collection
Collection
Collection
Collection
Collection
Collection
Company
Company
Company
Company
Company
Configuration
Configuration
Configuration
Configuration
Configuration
Configuration
Correlation
Correlation
Correlation
Correlation
Correlation
Correlation
Event Sources
Event Sources
Event Sources
Event Sources
Expert Usage
Expert Usage
Expert Usage
Expert Usage
Expert Usage
nstallation
nstallation
nstallation
nstallation
nstallation
nstallation
nstallation
nstallation
ntegration
Licensing
Maturity
Professional Services
Professional Services
Professional Services
Professional Services
Professional Services
Reporting
Reporting
Reporting
Reporting
Security
Security
Security
Storage
Storage
Support
Support
Support
Support
Support
System Administration and
Management
System Administration and
Management
System Administration and
Management
System Administration and
Management
System Administration and
Management
System Administration and
Management
Thought Leadership
Thought Leadership
Threat Monitoring
Training
Training
Training
Training
Upgrade
Upgrade
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
User Experience
"Cloud"
"Cloud"
"Cloud"
"Cloud"
"Cloud"
Ease of use
Sales Process
Sales Process
Sales Process
Visible Risk
Criteria
Can the architecture be deployed in a way so there is there a Local data store per business unit/location
Does the solution support Windows, Unix and Linux Platforms for Agent (collection) Systems?
Flexible Deployment Options (Appliance, Software, Virtual Appliance/Machine, SaaS)
s the "Collection" portion of the solution Scalable? (What is the max input per collection endpoint and aggregation point)
Overall Scalability - Not simply in EPS but in terms of usability, Number of concurrent query/reports/users at peak load (published max EPS)
Windows, Unix and Linux Platforms for Server and Database Systems?
Can the system normalize time stamps across event logs?
Does the solution support LDAP? Further does the solution support AD inheritance of security controls?
Does the systems support Role based user access and strong and granular ACL's
s the current version of the product Common Criteria Validated? To what degree? s the current shipping code certified or an older release?
Does the product have a FPS 140-2 Certified configuration?
s the product on a DoD entity approved product list? (Army, Air Force, Navy, DSA?)
Can the Collection and Log Forwarding mechanism "intelligently" store and forward information?
Can each agent or collection mechanism scale to support single event source rates for extremely high volume devices (proxy, netflow, dns, firewall) in excess of 1
Can the Agent (End Point or Management Log Collection Mechanism) cache event locally in case of network or system outage?
Can the Agent (End Point or Management Log Collection Mechanism) Batch information before forwarding?
Can the Agent (End Point or Management Log Collection Mechanism) Add Contextual nformation to the events being forwarded (Log Source, Date/Time Stamp,
Can the Agent (End Point or Management Log Collection Mechanism) function in bandwidth limited situations by A. Prioritize forwarding of events based on defined
Can the Agent (End Point or Management Log Collection Mechanism) aggregate similar information based on custom defined grouping values?
Can the Agent (End Point or Management Log Collection Mechanism) aggregation be tuned to customer defined time groupings (# seconds, minutes, hours) for
each event source? For example can count exact matches in a per minute increment for firewall connections and on per hour increments for syslog messages
Does the Agent (End Point or Management Log Collection Mechanism) compress and encrypt data being forwarded to the Log Management and/or SEM Solution?
Can the Agent (End Point or Management Log Collection Mechanism) be installed without physical access to the machine if deployed as a server resident software
Can the Agent (End Point or Management Log Collection Mechanism) Software be updated from a centralized management system?
Can the Agent (End Point or Management Log Collection Mechanism) Configuration Settings be updated from a centralized management system?
Does the Agent support standard inputs from all of the following logging mechanisms: Syslog, syslog-ng, SNMP v2, SNMP v3, ODBC/JDBC, CSV, XML, structured
Can the Microsoft Windows Agent (collection mechanism) map GUD/SUD to local registry/names/references for each event D in the SYSTEM, SECURTY and
Are Agents that rely on Event Source Vendor AP's to connect and collect information approved and/or certified by that event source Vendor?
Can the Agent follow dynamically changing folders and file names? For example in order to support event sources like S Web Logs or custom applications that
Does the Agent support Database Administrator Logging (From both SQL and System / File Based Sources) for Oracle, MSSQL, MySql and DB2?
Agent parsing and mapping customization. Can the agent's parsing be modified to assist with custom log messages? Can the normalization or categorization
Can the Agent act as a NTP source for Source Event Logs or otherwise help in time synchronization for source event logs?
Alliances and Supported ntegration Partners. Syslog is not integration. Full Certification of interoperability with which solutions?
Financial Performance - Subjective Measurements
Marketplace opinions and reviews.
Executive Leadership of the Vendor. Strong Team? Trustworthy?
$ spent on R&D for new products vs Development on incremental growth of core product.
Data export feature (extract logs (cvs, txt, etc.)
Data Workflow ntegration (Bidirectional access to information via external workflow tools?)
Email interface for report distribution
Email (SMTP) interface for alerting
nterface to 3rd party applications (ticketing/workflow application, Clearpoint dashboard, existing business logging solutions etc.)
Time difference adjustment feature (to allow the logging system to cope with devices with inconsistent times)
Advanced Correlation (Across reporting devices, mmediate and Over Time, Referencing Host, Application, Vuln, Asset information, exclusion/inclusion lists,
Aggregation Capabilities (Counting and streamlining) s that timeframe adjustable, are the fields aggregated upon flexible?
Basic Correlation (dentify a known bad using if / then statements)
Cross Device Correlation (Events from System x and y can be matched no matter where they were collected)
Does the vendor provide automatic Hotlist and Correlation Rules updates
Filtering Capabilities (Exclude and nclude)
Does the Event Collector/Agent/Connector keep a local cache of events so that events are not lost in case of application or connectivity issues?
Does the product support all anticipated Event Source ntegration Methods (File, Syslog, SNMP, ODBC/JDBC, AP)
Does the product support Common Security, Network, Operating System and Applications?
Will the solution support Custom Applications? How does it accomplish this?
Advanced Analysis - Statistical
Advanced Analysis - Visualization (meaningful use cases defined?)
Does the product support the most difficult use-cases for correlation? (Multi-Vendor, Multi-Event, Custom Application and Custom field correlation)
Will the product function to support the needs of the Tier 1 SOC Analyst?
Will the product support the needs of Tier2/3 ncident Response Team?
Can the nstallation and nitial Configuration be handled by technical staff with minimal training?
Does the solution track all logging devices
Ease of setup, Maturity of Documentation and Support to facilitate this effort?
How difficult will installation be for new or custom requirements?
How does the solution facilitate asset tracking?
Overall ease of installation and maintenance across Company
Rate level of complexity required integrating log sources.
Supported Vendors, Products and Versions
Can external analysis tools or products be launched in a contextually sensitive manner from the product? (Example: Can the user right click on a cell and pass one
Describe your Licensing Model and Pricing (List)
Has the product has the benefit of evolving with large complex, customer requirements? Not departmental deployments, but enterprise deployments?
Are tuning services provided as part of support or through Paid Professional Services
Can the vendor or partner provide an engineer to support their operations?
Does the vendor provide content (parsing, reports, correlation rule) packages?
Professional Services Scalability or does Partner Delivery Maturity exist?
s there expertise across the vendors partners (VAR, MSSP, Consulting Orgs, etc) to support both basic and advanced consulting needs?
Business graphics available
Does the solution support reporting for business dashboards?
Matches or exceeds standard reporting capabilities
Scheduled HTML reports
Change protection of stored logs (i.e. digital signatures, hashes)
Does the system have standard Hardened configurations?
Self-Signed or Major/nternal CA SSL Certificates Supported for event transport between Log Management and SEM solutions?
Compressed Cold storage (i.e. in an archive)
Hot storage (i.e. in local cache)
Does the vendor provide 24x7x365 support by phone?
How does the vendor charge for new versions? Upgrade requires additional funds or included in the support contract?
How does the vendor ensure the privacy and security of customer,s information when trouble shooting or records management?
Does the vendor have support globally?
Support Team expertise - Survey other customers for % of solutions solved at Level 1 versus escalation.
Are automated "Back-up" routines available?
Are user and/or admin actions logged and monitored
Does the vendor allow for Customer specific OS and DB "Hardening"
Does the Vendor provide an easy method for Patching the underlying Database Structure?
Does the vendor provide an easy method to install application "Patches"
Does the vendor provide and easy method for Patching the underlying Operating System?
Does the product seem to be forward-looking in its capabilities?
nnovation (anticipating of customer needs, threat trends, and new technologies)
Can the system support "live", "custom", or "dynamic" threat feeds for live correlation and alerting? Threat ntelligence Feeds such as P's, Subnets, Domain, Files,
Certification or record of completion available
Classroom Available? Easy to set up on my site (if necessary)?
Does the vendor provide product training?
Does the Vendor have Web based Training Available? Please Describe the offerings (Feel free to use attachment or reference URL if necessary)
How difficult is an Major Upgrade of Product (1.x) to (2.x) (is it centralized and does it minimize outages)
How difficult is an Minor Upgrade (x.1 to x.2) of Product (is it centralized and does it minimize outages)
Accessibility to internal, centralized log sources, in normalized and raw form
Automated Hotlist Trigger
Can access to data be restricted according to access rights (i.e. only business unit A can see business unit A,s data)?
Can analyst easily change correlation actions?
Can analyst understand correlation actions?
Can the analyst drive deeper analysis or via tools with a single action (right click and select)?
Can the console present just the events a particular analyst is assigned to handle?
Capability to share self-developed queries across businesses as template
Compliance User Case - Basic User
Could the product act as an incident management tool, accepting case notes, and related information on an incident?
Customizable queries
Customizable real-time alerting based on specified criteria
Distributed search across multiple data stores
Does the analyst interface allow easy access to actionable data?
Does the analyst interface require software to be installed on their workstations? f so, what are the pre-requisites?
Does the concept of a "Hotlist" or comparison list exist?
Does the customer Web interface allow easy access to data of interest to customers?
Does the product allow access to raw logs? f yes, how easily and quickly?
Functionality to initiate certain actions based on real-time alert (i.e. sending email/text message, executing script etc.)
How easy was it to extract a subset of the data from the log sources?
How likely is the solution to miss log/events from log sources?
How quickly can various forms of data be retrieved?
s the data presented in a manner that makes sense to the analyst?
s the interface user friendly or technical?
s there a history audit of the queries?
Manual Hotlist Update
Performance when searching for various data elements (P addresses, usernames, event types, etc.)
Pre-defined standard queries
Security Analyst Use Case - Advanced User
Security Analyst Use Case - Basic User
Security Analyst-friendliness (spend time investigating, not clicking),
SEM to Log Management Search Capabilities (Launch tool, api or contextually sensitive browser search)
Simple query (one conditional argument)
Transparency and integration of reporting mechanisms (e.g., can analyst understand why SM brought item to console?),
Was the data useful, or only a starting point?
Web Based Access or Front-end Available?
What data in the console can be used to search (e.g., if an SSOD is found in a VPN log, can all log data be quickly search for that SSOD?)
What format that the results of a report be exported out in (Excel, CSV, PDF?)
Can the solution be deployed off premise (hosted) and still segment my organization's data to meet my stringent (segmentation, retention, availability)
aaS, PaaS or SaaS (What control do you have over the "Cloud" environment; what control do we have over the environment?
What is the pricing model for network, storage and processing of my information in your (or partner) "cloud"?
What service will you provide in the cloud, will you automatically identify unparsed events and fix them? Will you re-establish connectivity to "down" sensors?
s your offering Hosted Appliances or virtual appliances or truly elastic in nature and transportable/extendable?
s your software designed to track user input to undertand how users interact with the system? Objective measures of feature use (misuse)? How is this information
What are your thoughts on a live "bake-off" between competing solutions in our environment?
What concessions do you have to use my company as a reference? (Services/Traning Credits, xx% off software list, xx% off of maintenance?)
Does the vendor "jump" the main POC to get to exec's or work through the process? What internal battles do need to prepare for when dealing with this vendor?
Copyright (C) VisibleRisk, LLC - You may use this document simply ask that you retain source attribution to VisibleRisk in any derivatives or publications.