The goal of the EFS Log Report project is to gather, aggregate, analyze and report on EFS usage based on EFS log information. Currently, the only usage logs available for EFS are those for EFS over HTTP; however, the architecture will need to be scalable so that the usage information from NFS servers themselves can be aggregated and analyzed in the future.
There are two basic components of the initial implementation:
- an aggregation component to get all of the logs into a single store
- an extension to [AWStats][AWStats] for the graphical viewing component
Since the logfiles (for EFS over HTTP) are lighttpd logfiles, scattered across multiple servers around the world, we need to aggregate those. Since the logs are (presumably) very large, we need to minify them.
For aggregation, we use an extension to [lighttpd][lighttpd] that allows specification of facility and priority so that we can then route logs based on policy.
We need to be able to specify the facility and priority that lighttpd logs use. That also requires registration of those values (and sane defaults) in the lighttpd configuration file parsing.
Logs go into an [HBase][Hbase] store per region, and are then further aggregated globally.
[lighttpd] http://www.lighttpd.net/ [Hbase] http://hbase.apache.org/ [AWStats] http://awstats.sourceforge.net/