How to Send Linux Logs to a Remote Server

The main reason to apply remote logging is the same reason because of which a dedicated /var partition is recommended: a matter of space, but not only. By sending logs to a dedicated storage device you can prevent your logs from taking all the space while keeping a huge historical database to afford bugs.

Uploading logs to a remote host allows us to centralize reports for more than one device and to keep a report backup to research in case something fails preventing us from accessing logs locally.

This tutorial shows how to setup a remote server to host logs and how to send these logs from client devices and how to classify or divide logs in directories by client host.

To follow instructions you can use a virtual device, I took a free tier VPS from Amazon (if you need help setting up an Amazon device they have great dedicated content on it on LinuxHint at https://linuxhint.com/category/aws/). Note the server public IP is different than its internal IP.

Prior to starting:

The software used to send logs remotely is rsyslog, it comes by default on Debian and derived Linux distributions, in case you don't have it run:

# sudo apt install rsyslog

You can always check the rsyslog state by running:

# sudo service rsyslog status

As you can see the status on the screenshot is active, if your rsyslog isn't active you can always start it by running:

# sudo service rsyslog start

# systemctl start rsyslog

Note: For additional information on all options to manage Debian services check Stop, start and restart services on Debian.

Starting rsyslog isn't relevant right now because we will need to restart it after making some changes.

How to Send Linux Logs to a Remote Server: The Server Side

First of all, on the server edit the file /etc/resyslog.conf using nano or vi:

# nano /etc/rsyslog.conf

Within the file, uncomment or add the following lines:

module(load="imudp")
input(type="imudp" port="514")
module(load="imtcp")
input(type="imtcp" port="514")

Above we uncommented or added logs receptions through UDP and TCP, you can allow only one of them or both them, once uncommented or added you'll need to edit your firewall rules to allow incoming logs, to allow logs reception through TCP run:

# ufw allow 514/tcp

To allow incoming logs through UDP protocol run:

# ufw allow 514/udp

To allow through both TCP and UDP run the two commands above.

Note: for more information on UFW you can read Working with Debian Firewalls (UFW).

Restart rsyslog service by running:

# sudo service rsyslog restart

Now continue on the client to configure sending logs, then we'll get back to the server to improve the format.

How to Send Linux Logs to a Remote Server: The Client Side

On the client sending logs add the following line, replacing the IP 18.223.3.241 for your server IP.

*.* @@18.223.3.241:514

Exit and save changes by pressing CTRL +X.

Once edited restart the rsyslog service by running:

# sudo service rsyslog restart

On the server side:

Now you can check logs inside /var/log, when opening them you'll notices mixed sources for your log, the following example shows logs from Amazon's internal interface and from the Rsyslog client (Montsegur):

A zoom shows it clear:

Having mixed files isn't comfortable, below we will edit rsyslog configuration to separate logs according to the source.

To discriminate logs inside a directory with the name of the client host add the following lines to the server /etc/rsyslog.conf to instruct rsyslog how to save remote logs, to do it within the rsyslog.conf add the lines:

$template RemoteLogs,"/var/log/%HOSTNAME%/.log"
*.* ?RemoteLogs
& ~

Exit saving changes by pressing CTRL +X and restart rsyslog on the server again:

# sudo service rsyslog restart

Now you can see new directories, one called ip-172.31.47.212 which is AWS internal interface and other called “montsegur” like the rsyslog client.

Within the directories you can find the logs:

Conclusion:

Remote logging offers a great solution to a problem which can bring services down if the server storage becomes full of logs, as said in the beginning, it is also a must in some cases in which the system may be seriously damaged without allowing access to logs, in such cases a remote log server guarantees sysadmin access to the server history.

Implementing this solution is technically pretty easy and even free considering high resources aren't need and free servers like AWS free tiers are good for this task, should you increase log transference speed you can allow UDP protocol only (despite losing reliability). There are some alternatives to Rsyslog such as: Flume or Sentry, yet rsyslog remains the most popular tool among Linux users and sysadmins.

I hope you found this article on How to Send Linux Logs to A remote server useful.