Intelligent Mapping: Design Document

Version 1.0
26/Sep/06

Prepared by

Sachin Dharmapurikar
Akanksha Technologies, Hyderabad
 1. Introduction
Vigilantmonitor is consisting of two major components which participate in
gathering and processing the log data. These components are viz. vEngine & vSpider. In a
typical scenario both the components are residing in network on two different machines.
It is observed that log files contain repetitive strings on large extent. Repetitive
transmission of same log strings is redundant usage of the local area network. Intelligent
mapping is a solution which is aimed to reduce the redundant usage of network
bandwidth and also provide large amount of substitution based compression at database
storage end.
Intelligent mapping should be a separate component from vSpider & vEngine
infrastructure and helps them as a library. The conceptual diagram of Intelligent mapping
component is shown below:

Spider
Intelligent Mapper
Map
Manager

Reader Queue Dispatcher

Log

To Engine

Fig. 1.1: Spider architecture displaying Intelligent Mapper

The similar implementation is at the engine side too.
 DB
Engine
Map Manager &
Reverse Mapper
To Spider

TLR Processor Queue TLR Service

DB
From Spider

Fig. 1.1: Engine architecture displaying Map manager & Reverse Mapper
Intelligent mapping achieves large amount of data storage compression as well
utilizes minimum network bandwidth for data transfer. As a result of data storage
compression, the queries working on log data become faster by 200% and more.
Functions and roles of each component are defined in subsequent sections.
 2. Intelligent Mapping in vSpider
All the log files are read and interpreted by vSpider. The interpreted data is
converted into TLR (Transferred Log Records) and those are sent to vEngine. In a typical
log file, most of TLR contains repetitive strings which causes not only redundancy in
database storage but consume lot of bandwidth in transmission of records. To avoid this
vSpider uses mapping mechanism to avoid