@inproceedings{Mao_WIDM_2001,
  author    = {Yun Mao and Kang Chen and Dongsheng Wang and Weimin Zheng},
  title     = {Cluster-based online monitoring system of web traffic},
  booktitle = {3rd International Workshop on Web Information and Data Management
               ({WIDM} 2001)},
  address   = {Atlanta, GA, USA},
  month     = {November},
  year      = {2001},
  pages     = {47-53}
}

Abstract:

Web traffic has been increasing and evolving rapidly in recent years. It is important to measure the volume and characteristic of such dominant traffic to understand large-scale user access pattern and analyze performance of Web applications. Among the common methods of Web measurements, the passive way using packet monitoring is more advantageous since it provides comprehensive information and is transparent to end-users. However, the throughput of current packet monitoring system is limited by the bandwidth of network adapters. Computational capacity and buffer size are also potential performance bottlenecks for monitoring high-speed links. This paper proposes a cluster-based online monitoring system, which generates rich logs by packet sniffing for Web analysis in a cluster environment. Powered by cluster computing technologies, the system achieves high performance and availability. Other techniques, such as online reconstruction in memory and kernel packet filter, are also adopted to improve the system performance. The experimental results indicate that it is feasible to trace in high-speed links with the cluster-based system.