How to tidy Magento 2 log files using logrotate

Magento 2 Video Tutorials Beginner Tutorials
,
1

As you’d expect with most applications, specific events get logged for a variety of reasons like Security, Maintenance and Development. However, if these log files go unchecked then they can grow so large that your disk drive completely fills up.

Watch

Before using logrotate

As you can see from the example below, these are the sizes of my logs in my Sandbox Server (the server I use for testing and creating videos). Notice how the cron and debug logs are over 700MB each after a few months of use - And this isn’t even a Live Server with any traffic.

-rw-rw-r--  1 magento www-data      2209 Jul  1 00:00 connector.log
-rw-rw-r--  1 magento www-data 726124470 Jul  2 15:37 cron.log
-rw-rw-r--  1 magento www-data 748220390 Jul  2 15:37 debug.log
-rw-rw-r--  1 magento www-data     10966 Jun 15 15:52 exception.log
-rw-rw-r--  1 magento www-data     46269 Jun 16 10:38 magento.cron.log
-rw-rw-r--  1 magento www-data    217632 Jun 27 14:51 payment.log
-rw-rw-r--  1 magento www-data      6230 May 13 10:27 setup.cron.log
-rw-rw-r--  1 magento www-data  19742732 Jul  2 15:37 system.log
-rw-rw-r--  1 magento www-data      7938 Jun 15 15:52 temando.log
-rw-rw-r--  1 magento www-data       680 Apr  8 10:31 update.cron.log
-rw-rw-r--  1 magento www-data    160731 Apr  8 10:31 update.log

After using logrotate

Now take a look after I use a tool called “logrotate”. The tool has compressed the most recent logs and archived them for later use. See how the compressed versions are significantly smaller. Those 700MB logs are now only 56MB. I’ll go over the other features later in the video.

-rw-rw-r-- 1 magento www-data      274 Jun 27 14:51 connector.log-2020-07-04-1593880904.gz
-rw-rw-r-- 1 magento www-data 53832871 Jul  4 16:41 cron.log-2020-07-04-1593880904.gz
-rw-rw-r-- 1 magento www-data 54994385 Jul  4 16:41 debug.log-2020-07-04-1593880904.gz
-rw-rw-r-- 1 magento www-data      629 Jun 15 15:52 exception.log-2020-07-04-1593880904.gz
-rw-rw-r-- 1 magento www-data     2668 Jun 16 10:38 magento.cron.log-2020-07-04-1593880904.gz
-rw-rw-r-- 1 magento www-data     9935 Jun 27 14:51 payment.log-2020-07-04-1593880904.gz
-rw-rw-r-- 1 magento www-data      724 May 13 10:27 setup.cron.log-2020-07-04-1593880904.gz
-rw-rw-r-- 1 magento www-data   447188 Jul  4 16:41 system.log-2020-07-04-1593880904.gz
-rw-rw-r-- 1 magento www-data      383 Jun 15 15:52 temando.log-2020-07-04-1593880904.gz
-rw-rw-r-- 1 magento www-data       92 Apr  8 10:31 update.cron.log-2020-07-04-1593880904.gz
-rw-rw-r-- 1 magento www-data    11381 Apr  8 10:31 update.log-2020-07-04-1593880904.gz

Important: Be sure to reference the documentation when following along, as sometimes I miss-speak in the video.

1. Introduction

Before we begin, I want to highlight some key points about this tutorial:

  • It’s written for Ubuntu
  • The commands and variables assume that you’ve installed Magento 2 using my previous tutorials

So, right now I’m signed into my server as the “superuser”. If you’ve been following along for a while now, you’ll know I don’t use “root” user for security purposes.

2. Install logrotate

Ubuntu typically comes pre-installed with a tool called “logrotate”. The tool automatically compresses, moves, renames or deletes log files base on a number of variables such as size, time or date.

It’s great for ensuring that you keep you logs tidy so that they don’t end up overwhelming your disk space. Plus, the smaller files make it much easier to navigate when searching for logs.

2.1 Check logrotate is installed

You can verify if it’s install by running the following command, which will out the version you have installed: [02:48]

logrotate --version

If you see an error, then you’ll need to install it now. This is done by completing 2 simple steps.

Skip steps 2.2 and 2.3 if you already have logrotate installed.

2.2 Update Repositories

So, the first step is to update our existing repositories by entering:

sudo apt-get update

2.3 Install logrotate

Then to install the logrotate tool using:

sudo apt-get install logrotate

3. Prepare a logrotate directory

The default behaviour of logrotate is to output the rotated logs inside the same directory as the original logs. However, in my opinion this can look a little cluttered. So, what I like to do is keep the rotated logs inside a sub-directory to keep things a little tidier.

4.1 Create a logrotate directory

So, first we need to make a directory to keep our rotated logs in. Do this by running: [03:50]

sudo mkdir /var/www/html/var/log/logrotate

Which will create a directory called “logrotate” inside out Magento log directory.

4.1 Set logrotate directory permissions

Because we created that directory as “craig” (the superuser), it means the directory will have the incorrect Owner and Group Permissions. Therefore, we need to change Owner to “magento” and the Group to “www-data” with the following: [04:30]

sudo chown magento:www-data /var/www/html/var/log/logrotate

5. Setup a logrotate configuration

Now we can move on to creating a configuration file for the logrotate tool, that contains instructions for what we want it to do.

5.1 Create a new configuration file

Let’s start by using the nano text editor to create a configuration file called “magento2” inside of the logrotate directory. [05:25]

sudo nano /etc/logrotate.d/magento2

5.2 Compose the configuration variables

With the nano text editor open, let’s add the following instructions. To save time, you can easily copy and paste these from the supporting article. A link to that article can be found in the video description below.

Edit: Removed seconds from date format as it seemed unnecessary.

/var/www/html/var/log/*.log {
    su magento www-data
    weekly
    missingok
    rotate 4
    compress
    notifempty
    dateext
    dateformat -%Y-%m-%d
    create 664 magento www-data
    olddir logrotate
}

Configuration variables explained

Variable Description
/var/www/html/var/log/*.log Location of Magento logs to rotate
su magento www-data Tasks are ran with correct Owner and Group
weekly Rotate files weekly
missingok Don’t produce error if log file missing
rotate 4 Keep last 4 rotate logs
compress Compress rotated logs
notifempty Don’t rotate empty log
dateext Add date stamp to rotated log
dateformat -%Y-%m-%d-%s Set date format when dateext enabled
create 664 magento www-data Set rotated logs to the correct Permission, Owner and Group
olddir logrotate Sub directory to move rotated logs

Now save your changes and exit the nano editor by pressing Ctrl+X to exit, Y to save and Enter to confirm.

5.3 Testing

To ensure that both the logrotate directory and configuration file were setup correctly, we can force the logrotate configuration to run now the typing: [07:55]

Note: I miss-spoke this next line in the video. This is the correct command.

sudo logrotate -f /etc/logrotate.d/magento2

Note: This commands can take 1-2 minutes to complete depending on the log file sizes

If no errors appear, check that the rotated logs are in the logrotate directory by entering: [08:10]

ls -l /var/www/html/var/log/logrotate

And as you can see, these rotated logs have been archived inside our logrotate sub-directory.

Summary

So, there you have it. You’ve successfully setup logrotate to help smartly organise your Magento 2 log files.

References

1 Like