Lampros - Weird Bricks

logtail2 - get only new lines from logfiles

24 January, 2016 | CentOS

Some time ago I was working on a project where we only wanted to process the new files from a logfile - the requirement was that we should never process the same line more than once.

So we started counting the lines of the logfile and adding a bunch of checks and before you knew it we were looking into the log's inodes to deal with rotation.... and then someone told me that there was already a tool that did all of the above - logtail2.

Here's how I set it up on CentOS and some examples:

First of all you'll need the EPEL repository:

yum install epel-release -y

Now you can install the logcheck package which contains logtail2.

yum install logcheck -y

Let's create a fake logfile:

for i in {1..20}; do echo "fake line $i" >> fakelog.log; done

Let's check how many lines this log has:

[root@tester ~]# wc -l fakelog.log 
20 fakelog.log

20 lines - excellent. Now let's use logtail2 on it:

[root@tester ~]# logtail2 fakelog.log 
fake line 1
fake line 2
fake line 3
fake line 4
fake line 5
fake line 6
fake line 7
fake line 8
fake line 9
fake line 10
fake line 11
fake line 12
fake line 13
fake line 14
fake line 15
fake line 16
fake line 17
fake line 18
fake line 19
fake line 20

Now let's try the exact same command:

[root@tester ~]# logtail2 fakelog.log 
[root@tester ~]#

No output! The reason we got no output the second time is because there were no new lines :)

Let's add some more lines to our fake logfile:

for i in {21..30}; do echo "fake line $i" >> fakelog.log; done

Now run logtail2 again:

[root@tester ~]# logtail2 fakelog.log 
fake line 21
fake line 22
fake line 23
fake line 24
fake line 25
fake line 26
fake line 27
fake line 28
fake line 29
fake line 30

What happens if we delete the file and then add some new lines?

rm fakelog.log
rm: remove regular file `fakelog.log'? y

OK - add the new lines:

for i in {1..5}; do echo "fake line - round 2 $i" >> fakelog.log; done

Now run logtail2 on it again:

[root@tester ~]# logtail2 fakelog.log 
*** WARNING ***: Log file fakelog.log is smaller than last time checked!
*************** This could indicate tampering.
fake line - round 2 1
fake line - round 2 2
fake line - round 2 3
fake line - round 2 4
fake line - round 2 5

Sweet :) we only got the new lines - logtail2 is a great tool to have in your sys admin arsenal.