How to remove cruft from your apache log

Conditional logging is a feature in apache web server which help you exclude some requests getting logged.

Keeping an eye on apache logs is an essential part of a webmaster’s job. Checking the logs help you detect performance problems and security issues. There are many log analyser software available to do this job effectively and create graphs, charts and other visualisations. Still, there are times when you need to check data directly from raw logs. But not all entries are useful. Access to a web page creates log entries not just for that page but for the JavaScript files, CSS files and images included in that page. This makes difficult to find information easily from raw logs. Separating such entries from main access log would help you to analyse data easily.

There are two ways to remove cruft from main log.

  1. Conditional logging.
  2. Separate (sub)domains for media and feeds.

Conditional logging

You can exclude some requests, by setting environment variables using SetEnvIf and adding env=!variable clause to the CustomLog directive in your apache configuration.

For example, if you have your images located in /images directory, you can exclude all requests to that directory from main log and if you want, log them in a separate log.


SetEnvIf Request_URI "^/images/" images
CustomLog /logs/example.com/access.log combined env=!images
CustomLog /logs/example.com/images_access.log combined env=images

See Conditional Logs in apache documentation.

This is what I do in my apache configuration:


SetEnvIf Request_URI "^/wordpress/" cruft
SetEnvIf Request_URI "^/robots\.txt$" cruft
SetEnvIf Request_URI "^/favicon\.ico$" cruft
CustomLog /logs/example.com/access.log combined env=!cruft

All my javascript, css, images and feeds are located in separate domains. See how to setup a separate sub-domain for media files in WordPress

Or get RSS feed

Author:

Syam Kumar is the Owner and Editor of WebMaster View. He is based in Cochin, Kerala, India. His interests include information design concepts, standards compliant web development and Linux.

This entry was posted on Thursday, November 26th, 2009 at 8:33 pm and modified by WebMaster View on Saturday, November 28th, 2009 at 1:13 am. You can follow any responses to this entry through the RSS 2.0 feed. Responses are currently closed, but you can trackback from your own site.

Comments : 1

  1. Thanks! Nice read! Apache logs can be a pain to sort through!

    Web Hosting

Pings/Trackbacks: 4

  1. How to Move Your WordPress Feeds to a Sub-domain - WebMaster View
  2. Tweets that mention How to remove cruft from your apache log - WebMaster View -- Topsy.com
  3. Setup a separate sub-domain for media files in WordPress - WebMaster View
  4. How to remove cruft from your apache log - WebMaster View Webmaster World