Pre-Processing of University Webserver Log Files for Intrusion Detection

Web Server log files can reveal lots of interesting patterns when analyzed. The results obtained can be used in various applications, one of which is detecting intrusions on the web. For good quality of data and usable results, there is the need for data preprocessing. In this research, different stages of data preprocessing were carried out on web server log files obtained over a period of five months. The stages are Data Conversion, Session Identification, Data Cleaning and Data Discretization. Data Discretization was carried out in two phases to take care of data with continuous attributes. Some comparisons were carried out on the discretized data. The paper shows that with each preprocessing step, the data becomes clearer and more usable. At the final stage, the data presented offers a wide range of opportunities for further research. Therefore, preprocessing web server log files provides a standard processing platform for adequate research using web server logs. This method is also useful in monitoring and studying web usage pattern in a particular domain. Though the research covers webserver log obtained from a University domain, and thus, reveals the pattern of web access within a university environment, it can also be applied in e-commerce and any other terrain.