Logs

What is it? #

Logs are the record of what happened on a machine. When something breaks, they are usually the only evidence available.

There are two places to look. Traditional log files under /var/log, written by services such as Nginx, and the systemd journal, which captures the output of anything managed as a service.

Finding things quickly is the skill worth building: filter by service, by time, by severity, then search the text.

The operational concern is size. Logs grow continuously, and a full disk caused by unrotated logs is one of the most common server incidents.

Think of it like this #

A building's security cameras and visitor book. When something goes wrong, you look at the recording around that time.

Recordings take space, so old footage is overwritten. Nobody keeps every frame forever, and a system that never deletes anything eventually fills the store.

Simple example #

A user reports an error at 14:32. You need to see what the application logged then, whether Nginx returned a 500, and whether the system did anything unusual around the same time.

Code #

# The journal: anything run as a systemd service
journalctl -u myapp -f                       # follow live
journalctl -u myapp --since "14:25" --until "14:40"
journalctl -u myapp -p err                   # errors and worse
journalctl -u myapp -n 200 --no-pager
journalctl --since today | grep -i "timeout"
journalctl -k                                # kernel messages
journalctl -u myapp -o json-pretty | head    # structured output

# Traditional log files
tail -f /var/log/nginx/error.log
less /var/log/nginx/access.log
grep " 500 " /var/log/nginx/access.log | tail -20

# Useful locations
# /var/log/nginx/         web server access and error logs
# /var/log/auth.log       SSH logins and sudo (secure on RHEL)
# /var/log/syslog         general system messages (messages on RHEL)
# /var/log/postgresql/    database logs

# Quick analysis of an access log
awk '{print $9}' access.log | sort | uniq -c | sort -rn    # status code counts
grep " 500 " access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head
awk '<span class="katex-error" title="ParseError: KaTeX parse error: Expected &#x27;}&#x27;, got &#x27;EOF&#x27; at end of input: 9 == 500 {print" style="color:#cc0000">9 == 500 {print</span>4, $7}' access.log | tail -20        # when and what failed

# Rotation: /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 14              # keep two weeks
    compress
    delaycompress
    missingok
    notifempty
    create 0640 appuser appgroup
    sharedscripts
    postrotate
        systemctl reload myapp > /dev/null 2>&1 || true
    endscript
}

# Keep the journal bounded too
sudo journalctl --disk-usage
sudo journalctl --vacuum-size=500M
# permanent: SystemMaxUse=500M in /etc/systemd/journald.conf

How it works #

journalctl -u filters by unit, which immediately removes everything unrelated. Combining it with --since and --until narrows to the window that matters, which is far faster than scrolling.

-p err filters by priority. During an incident, starting with errors only and widening if necessary is the quickest path to the cause.

journalctl -k shows kernel messages, which is where you find the OOM killer, disk errors and network problems that the application never saw.

For file-based logs, tail -f during a reproduction and grep for analysis afterwards cover most needs.

The awk examples treat the access log as columns. Field 9 is the status code and field 7 the path in the default Nginx format, so counting status codes or finding which paths return 500s is one line.

Rotation replaces the current file periodically, compresses the old one and deletes anything beyond the retention count. Without it, a busy log grows until the disk is full and every service fails at once.

postrotate tells the service to reopen its log file. Without that step, the service keeps writing to the renamed file, and the new one stays empty — a confusing failure that looks like logging has stopped.

The journal needs its own size limit, since it is stored separately from /var/log files.

Real-world use #

The first five minutes of most incidents are spent in logs: the application journal for the error, Nginx for the status codes, and the kernel log if a process disappeared.

/var/log/auth.log is worth knowing for security work. Failed SSH attempts, successful logins and sudo usage all appear there, and a sudden volume of failures indicates a brute-force attempt.

Centralised logging replaces this on multi-server systems, because logs on one machine are of limited use when twenty are serving traffic. The logging lesson in the system design track covers that side.

Disk full from logs is a genuinely frequent incident, and it usually happens during another incident — when error logging is at its highest. Rotation configured in advance is the fix.

Log retention is also a policy question. Logs contain personal data, so keeping everything forever creates a compliance problem as well as a storage one.

Common mistakes #

• Reading whole log files instead of filtering by unit, time and severity.
• No rotation, so logs fill the disk during the incident you most need them.
• Forgetting the postrotate reload, so the service writes to a deleted file.
• Leaving the journal unbounded, consuming gigabytes over time.
• Only checking application logs when the kernel log holds the answer.

Practice #

On a machine you control, use journalctl to show only errors from one service in the last hour. Then analyse an access log to count status codes and find the paths returning the most errors. Finally, write a logrotate configuration with 14-day retention and compression.