papyri/md/tomcat_logging.md

# Tomcat Logging

## Contents

  - [Default Logging in Tomcat 5.5](#default-logging-in-tomcat-55)
      - [Problem with growing log size](#problem-with-growing-log-size)
      - [Solving with Apache's rotatelogs](#solving-with-apaches-rotatelogs)
      - [Log rotation script example for rotatelogs](#log-rotation-script-example-for-rotatelogs)
      - [Solving with logrotate.d](#solving-with-logrotated)
  - [Using log4j with Tomcat 5.5](#using-log4j-with-tomcat-55)
      - [Install needed libraries](#install-needed-libraries)
      - [Configure logging.properties](#configure-loggingproperties)
  - [References](#references)

Articles in this series

  - [Tomcat Mechanics](tomcat_mechanics.md)
  - [Tomcat Configuration](tomcat_configuration.md)
  - [Tomcat Packaging](tomcat_packaging.md)
  - **Tomcat Logging**


## Default Logging in Tomcat 5.5

### Problem with growing log size

The core logging file catalina.out with Tomcat grows and has no builtin log rotation without switching all of Tomcat to log4j. This file can grow to enormous sizes and use up disk space when it could be rotated automatically. The core startup script bin/catalina.sh does a very simple redirect of STDOUT and STDERR to the file catalina.out. This causes the file descriptor to be held open and makes it difficult to get this large file rotated without restarting Tomcat as a whole.

### Solving with Apache's rotatelogs

There are a few variations of a solution; at its core, the idea is to pipe the logging to a program which can rotate the file on it's own. In most cases the Apache software is also installed on the server which includes the 'rotatelogs' app: /usr/bin/rotatelogs by default on RHEL/CentOS RPM installs, otherwise in \<apache\>/bin/rotatelogs with a make install, etc.

Once you have rotatelogs on the server, a very minor change is made to the \<tomcat\>/bin/catalina.sh to pipe the logging instead; this is identical to Apache use. In Tomcat 5/6, the change looks like this:

```
org.apache.catalina.startup.Bootstrap "$@" start \
>> "$CATALINA_BASE"/logs/catalina.out 2>&1 &

org.apache.catalina.startup.Bootstrap "$@" start \
|/usr/bin/rotatelogs "$CATALINA_BASE"/logs/catalina.out.%Y%m%d-%H%M 200M >> /dev/null 2>&1 &
```

The implementation in each version of Tomcat will differ slightly as the script changes from release to release; it's important to note there are two places to change the code as the 'start' block has two ways that Tomcat can be started based on the security model. The '200M' is completely customizable, it can be time based or size based with most piped logging apps like rotatelogs.

### Log rotation script example for rotatelogs

You still need separate logic to remove old logs. The rotatelogs format keeps the catalina log files small but does nothing to purge or compress old ones so you'll need to set up a bash script in cron that does a 'find ... -exec rm ...' or similar, based on desire.

```
#!/bin/bash
# archivecatalina.sh
# v1.0

## SETTINGS MUST BE CHANGED TO USE THIS SCRIPT ##

# This script will operate on the already rotated catalina.out logfiles
# generated by the changing catalina.sh to use a piped logging method
# such as:
#
#  org.apache.catalina.startup.Bootstrap "$@" start \
#  |/usr/bin/rotatelogs "$CATALINA_BASE"/logs/catalina.out.%Y%m%d-%H%M 200M >> /dev/null 2>&1 &

# Run this from crontab nightly at an off hour,
# as gzip can suck up some serious CPU on huge logs:
# 0 2 * * * /opt/sbin/archivecatalina.sh

########## SETTINGS BEGIN ##########

# ARCHIVE_METHOD settings
# 0 = unset (default)
# 1 = compress in place
# 2 = move to another location
# 3 = compress and move
ARCHIVE_METHOD=0

# DURATION_KEEP setting
# <n> = number of days prior to keep around
DURATION_KEEP=28

# CATALINA_LOGS setting
# path to source catalina.out.* files
CATALINA_LOGS=/usr/local/tomcat/logs

# CATALINA_ARCH setting
# path to store older logs based on ARCHIVE_METHOD=2 or 3
# NOTE: this directory is NOT autocleaned on purpose!!
CATALINA_ARCH=/mnt/server/arclogs

# DURATION_WORK setting
# <n> = number of days prior to start working (default 1 day)
DURATION_WORK=1

########## SETTINGS END ##########

# What's our log name
LOG_BASE="catalina.out"

# Errorlevels
ERR_ERROR=33
ERR_PANIC=66

# Default RedHat program locations
MV=/bin/mv
DT=/bin/date
RM=/bin/rm
ID=/usr/bin/id
GZ=/usr/bin/gzip

# Are we root?
USER=`$ID -u`
if [ "X$USER" != "X0" ]; then
  echo "PANIC: Only root can run this script!"
  exit $ERR_PANIC
fi

# Sanity checks
if [ $ARCHIVE_METHOD -eq 0 ]; then
  echo "PANIC: ARCHIVE_METHOD is set to 0, please configure the script."
  exit $ERR_PANIC
fi
if [ $DURATION_WORK -lt 1 ]; then
  echo "PANIC: DURATION_WORK is set to less than one day."
  exit $ERR_PANIC
fi
if [ $DURATION_KEEP -le $DURATION_WORK ]; then
  echo "PANIC: DURATION_KEEP is less than or equal to DURATION_WORK."
  exit $ERR_PANIC
fi

# Does the source path exist?
if [ ! -x $CATALINA_LOGS ]; then
  echo "ERROR: $CATALINA_LOGS doesn't exist!"
  exit $ERR_ERROR
fi

# Backup dir exists/writable?
if [ $ARCHIVE_METHOD -eq 2 ] || [ $ARCHIVE_METHOD -eq 3 ]; then
  if [ ! -x $CATALINA_ARCH ]; then
    echo "ERROR: $CATALINA_ARCH doesn't exist or isn't writable!"
    exit $ERR_ERROR
  fi
fi

# generate fromat stamps
DATE_WORK=`$DT --date "now - $DURATION_WORK days" +"%Y%m%d"`
DATE_KEEP=`$DT --date "now - $DURATION_KEEP days" +"%Y%m%d"`

# log actions below depends on formatting looking like:
# /some/path/catalina.out.20100520-1345 (YYYYMMDD-HHMM)
# (.gz is accounted for)

# ARCHIVE_METHOD=1
compress_keep (){
  for logfile in $CATALINA_LOGS/$LOG_BASE.*; do
    # if it ends in gz it's already compressed
    if [ ${logfile:(-3):3} == ".gz" ]; then
      continue
    else
      # grab the %Y%m%d out of the name
      DTS=${logfile:(-13):8}
      if [ $DTS -ge $DATE_KEEP ] && [ $DTS -le $DATE_WORK ]; then
        # compress in place
        $GZ $logfile
      fi
    fi
  done
}

# ARCHIVE_METHOD=2
move_only (){
  for logfile in $CATALINA_LOGS/$LOG_BASE.*; do
    # strip off .gz, we don't care if it's compressed
    logtemp = ${logfile%.gz}
    DTS=${logtmp:(-13):8}
    if [ $DTS -ge $DATE_KEEP ] && [ $DTS -le $DATE_WORK ]; then
      $MV -f $logfile $CATALINA_ARCH/
    fi
  done
}

# ARCHIVE_METHOD=3
compress_move (){
  for logfile in $CATALINA_LOGS/$LOG_BASE.*; do
    logtemp = ${logfile%.gz}
    DTS=${logtmp:(-13):8}
    if [ $DTS -ge $DATE_KEEP ] && [ $DTS -le $DATE_WORK ]; then
      # already compressed
      if [ ${logfile:(-3):3} == ".gz" ]; then
        $MV -f $logfile $CATALINA_ARCH/
      else
        $GZ -c $logfile >> $CATALINA_ARCH/$logfile.gz
        $RM -f $logfile
      fi
    fi
  done
}

# cleanup older than DURATION_KEEP
cleanup_older (){
  for logfile in $CATALINA_LOGS/$LOG_BASE.*; do
    logtemp = ${logfile%.gz}
    DTS=${logtmp:(-13):8}
    if [ $DTS -lt $DATE_KEEP ]; then
      $RM -f $logfile
    fi
  done
}

# the main case
case $ARCHIVE_METHOD in
  1)
    compress_keep
    ;;
  2)
    move_only
    ;;
  3)
    compress_move
    ;;
  *)
    echo "PANIC: Invalid ARCHIVE_METHOD settting: $ARCHIVE_METHOD"
    exit $ERR_PANIC
esac

# do a final cleanup
cleanup_older

exit 0
```


### Solving with logrotate.d

Another way that works just as well:

  - <http://java.dzone.com/articles/how-rotate-tomcat-catalinaout>

It's important to note that "copytruncate" verb used, it causes the original file to be truncated in place so it's FD; the only real downside to this method is that the copy process can chew up serious I/O if it's a really chatty Tomcat creating huge files. Using a rotatelogs method can save a bit of strain in this situation.


## Using log4j with Tomcat 5.5

Often when debugging a strange problem the default logging in Tomcat 5 does not go into enough detail to help pinpoint the problem. Adding log4j to the Tomcat instance can help track down problems, as well as provide proper log rotation to null out the growing catalina.out issue.

### Install needed libraries

  - Download [Log4J](http://logging.apache.org/log4j/1.2/download.html) (v1.2 or later), unpack the tarball and place the log4j-\<version\>.jar in $CATALINA\_HOME/common/lib directory. In a RHEL based RPM install, this is usually /usr/share/tomcat/common/lib.
  - Download [Commons Logging](http://commons.apache.org/logging/download_logging.cgi) and place the commons-logging-x.y.z.jar (not commons-logging-api-x.y.z.jar) in $CATALINA\_HOME/common/lib with the log4j jar added above.

### Configure logging.properties

Create a file called log4j.properties with the following content and save it into common/classes.

```
log4j.rootLogger=DEBUG, R
log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=${catalina.home}/logs/tomcat.log
log4j.appender.R.MaxFileSize=10MB
log4j.appender.R.MaxBackupIndex=10
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%p %t %c - %m%n
```

Start (or restart) Tomcat and log4j will now be creating a debug logfile for examination. This is just a basic example of a log4j setup, there are a lot of options that can be configured for use.


## References

**Log4J API Reference (FileAppender)**

  - <http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/FileAppender.html>
  - <http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/RollingFileAppender.html>
  - <http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html>
  - <http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/varia/ExternallyRolledFileAppender.html>

**Tomcat Logging**

  - <http://tomcat.apache.org/tomcat-5.5-doc/logging.html>
  - <http://tomcat.apache.org/tomcat-6.0-doc/logging.html>
  - <http://tomcat.apache.org/tomcat-7.0-doc/logging.html>
  - <http://tomcat.apache.org/tomcat-8.0-doc/logging.html>

**General Resources**

  - <http://wiki.apache.org/tomcat/FAQ/Logging>
  - <http://wiki.apache.org/logging-log4j/>
  - <http://commons.apache.org/logging/>