Setting up your application log with Logrotate

First story

Recently, our server was down, the main reason because the log files burned out our storage. We must find a way to restart the server, the temporary solution at that time was zip the logs and create new logs and then restart the server.

After that, I spend the time to research about Logrotate and init the Logrotate configurations for our servers. By the way, Logrotate is a system utility that manages the automatic rotation and compression of log files. This means by using the Logrotate, your log files can be compressed after a specific time and Logrotate will init an empty log file for you and you don’t need to get your server to switch over to the new log file.

Getting Started

Logrotate is installed by default on Ubuntu 16.04, and is set up to handle the log rotation needs of all installed packages, including rsyslog, the default system log processor.

Step 1: Enable the compress feature

Open the /etc/logrotate.conf file and remove the comment sign for compress feature

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# see "man logrotate" for details
# rotate log files weekly
weekly

# use the syslog group by default, since this is the owning group
# of /var/log/syslog.
su root syslog

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
    missingok
    monthly
    create 0664 root utmp
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0660 root utmp
    rotate 1
}

# system-specific logs may be configured here

Step 2: Init your Logrotate configuration

Create a new file called logrotate.conf from your server.

1
2
3
4
5
6
7
8
9
/var/www/myapp/shared/log/production.log {
  daily
  missingok
  rotate 90
  notifempty
  compress
  delaycompress
  copytruncate
}

There are few options for the logrotate:

  • daily: rotate daily. You can use weekly, monthly
  • missingok: ignore if the log file doesn’t exist.
  • rotate 90: only keep 90 days of logs around.
  • notifempty: don’t rotate if the log file is empty.
  • compress: gzip the log rotation. This uses gzip by default and results in files ending in .gz.
  • delaycompress: When delaycompress is active, an archived log is compressed the next time that the log is rotated. This can be important when you have a program that might still write to its old log file for a time after a fresh one is rotated in. Note that delaycompress works only if you have compress in your configuration.
  • copytruncate: copy the log files and then empties it. This makes sure the log files always exists so you don’t need to reload the server.

Step 3: Set the cronjob to run Logrotate

My Logrotate configuration will be run daily at 3:00 AM so my cron:

1
0 3 * * * logrotate /home/myapp/logrotate.conf --state /home/myapp/logrotate-state --verbose

Conclusion

The Logrotate is cool, it will help us save storage from the server. After the rotate run, you can trigger another cron (maybe 4:00 AM) to upload the gzip file to clould (S3 for example).