Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
Motivation:
The act of manually going through plain log files, grepping all over the
place, severely limits the value you can extract from them.
Thus, we felt the need to have a Log analytics tool. Which can help with
quick searches, and also draw valuable conclusions from the same.
Index Level:
ElasticSearch Query capabilities can be vastly optimized by proper
organization of Documents in Indexes.
A few possible solutions would be:
1. Keep all logs in 1 index (Cons: Very large Search space)
2. Organize logs on the basis of HostIDs generating them (Con: Still a
very large search space, Pro: if Host to be examined is previously
known, searches can be more target specific)
3
Conclusion:
As per the given use cases, HostID related Queries are rare. Hence, logs
maintained with respect to time(Option 3) was selected.
The Diagram shows flow for a single host. This applies to all hosts running the Appender.
Using only one extra thread, can defeat the idea of Asynchronous logging,
when the number of logging requests generated parallel increase, and the
main thread would need to wait for the single thread, which would be
making http calls to finish sending the previous lot of logs before it can
take up the new queue of logs. Thus a thread pool adds more parallelism
and later for scale, only the number of threads can be adjusted to fit the
need.