What is it? #
On a production VPS there are three log sources that matter: your application, the web server, and the system.
During an incident the sequence is usually the same: check the application journal for errors, the Nginx logs for status codes and timing, and the system log if a process disappeared.
Keeping logs useful means two things. Structure, so they can be searched, and rotation, so they do not fill the disk.
Retention is a decision, not a default. Long enough to investigate, short enough to be affordable and compliant.
Think of it like this #
Three cameras pointed at different parts of a shop: the till, the door and the fuse box.
The useful skill is knowing which one to review first for a given problem, and having them still recording because nobody let the storage fill up.
Simple example #
Users report errors at 14:32. You check the application journal around that time, the Nginx access log for the status codes, and the kernel log to rule out a memory kill.
Code #
# 1. Application — anything run as a systemd service
journalctl -u myapp --since "14:25" --until "14:40"
journalctl -u myapp -p err -n 100 --no-pager
journalctl -u myapp -f # follow live
# 2. Nginx — what the client actually received
sudo tail -f /var/log/nginx/app.error.log
awk '<span class="katex-error" title="ParseError: KaTeX parse error: Expected '}', got 'EOF' at end of input: 9 ~ /^5/ {print" style="color:#cc0000">9 ~ /^5/ {print</span>4, $7, $9}' /var/log/nginx/app.access.log | tail -20
awk '{print $9}' /var/log/nginx/app.access.log | sort | uniq -c | sort -rn
# 3. System — did the kernel intervene?
journalctl -k --since "14:25"
dmesg -T | grep -i "killed process"# A log format that includes timing — worth setting before you need it
log_format timed '<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mi>e</mi><mi>m</mi><mi>o</mi><mi>t</mi><msub><mi>e</mi><mi>a</mi></msub><mi>d</mi><mi>d</mi><mi>r</mi><mo>−</mo></mrow><annotation encoding="application/x-tex">remote_addr -</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:0.8444em;vertical-align:-0.15em;"></span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal">e</span><span class="mord mathnormal">m</span><span class="mord mathnormal">o</span><span class="mord mathnormal">t</span><span class="mord"><span class="mord mathnormal">e</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="katex-sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">a</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal">dd</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord">−</span></span></span></span>remote_user [$time_local] '
'"<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mi>e</mi><mi>q</mi><mi>u</mi><mi>e</mi><mi>s</mi><mi>t</mi><mi mathvariant="normal">"</mi></mrow><annotation encoding="application/x-tex">request"</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.0359em;">q</span><span class="mord mathnormal">u</span><span class="mord mathnormal">es</span><span class="mord mathnormal">t</span><span class="mord">"</span></span></span></span>status $body_bytes_sent '
'"<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>h</mi><mi>t</mi><mi>t</mi><msub><mi>p</mi><mi>r</mi></msub><mi>e</mi><mi>f</mi><mi>e</mi><mi>r</mi><mi>e</mi><mi>r</mi><mi mathvariant="normal">"</mi><mi mathvariant="normal">"</mi></mrow><annotation encoding="application/x-tex">http_referer" "</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:0.8889em;vertical-align:-0.1944em;"></span><span class="mord mathnormal">h</span><span class="mord mathnormal">tt</span><span class="mord"><span class="mord mathnormal">p</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.1514em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="katex-sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight" style="margin-right:0.0278em;">r</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.1076em;">f</span><span class="mord mathnormal" style="margin-right:0.0278em;">er</span><span class="mord mathnormal" style="margin-right:0.0278em;">er</span><span class="mord">""</span></span></span></span>http_user_agent" '
'rt=<span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>r</mi><mi>e</mi><mi>q</mi><mi>u</mi><mi>e</mi><mi>s</mi><msub><mi>t</mi><mi>t</mi></msub><mi>i</mi><mi>m</mi><mi>e</mi><mi>u</mi><mi>r</mi><mi>t</mi><mo>=</mo></mrow><annotation encoding="application/x-tex">request_time urt=</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:0.854em;vertical-align:-0.1944em;"></span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal">e</span><span class="mord mathnormal" style="margin-right:0.0359em;">q</span><span class="mord mathnormal">u</span><span class="mord mathnormal">es</span><span class="mord"><span class="mord mathnormal">t</span><span class="msupsub"><span class="vlist-t vlist-t2"><span class="vlist-r"><span class="vlist" style="height:0.2806em;"><span style="top:-2.55em;margin-left:0em;margin-right:0.05em;"><span class="pstrut" style="height:2.7em;"></span><span class="katex-sizing reset-size6 size3 mtight"><span class="mord mathnormal mtight">t</span></span></span></span><span class="vlist-s"></span></span><span class="vlist-r"><span class="vlist" style="height:0.15em;"><span></span></span></span></span></span></span><span class="mord mathnormal">im</span><span class="mord mathnormal">e</span><span class="mord mathnormal">u</span><span class="mord mathnormal" style="margin-right:0.0278em;">r</span><span class="mord mathnormal">t</span><span class="mspace" style="margin-right:0.2778em;"></span><span class="mrel">=</span></span></span></span>upstream_response_time';
access_log /var/log/nginx/app.access.log timed;# Find the slowest requests, using the timing field
awk '{print <span class="katex"><span class="katex-mathml"><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>N</mi><mi>F</mi><mo separator="true">,</mo></mrow><annotation encoding="application/x-tex">NF,</annotation></semantics></math></span><span class="katex-html" aria-hidden="true"><span class="katex-base"><span class="katex-strut" style="height:0.8778em;vertical-align:-0.1944em;"></span><span class="mord mathnormal" style="margin-right:0.109em;">N</span><span class="mord mathnormal" style="margin-right:0.1389em;">F</span><span class="mpunct">,</span></span></span></span>7}' /var/log/nginx/app.access.log | sort -rn | head -20# Rotation: /etc/logrotate.d/myapp
/var/log/myapp/*.log {
daily
rotate 14
compress
delaycompress
missingok
notifempty
create 0640 appuser adm
postrotate
systemctl reload myapp > /dev/null 2>&1 || true
endscript
}
# Bound the journal too
sudo journalctl --disk-usage
# /etc/systemd/journald.conf: SystemMaxUse=500M
sudo systemctl restart systemd-journaldRetention, decided deliberately
application logs 14-30 days locally is usually enough to investigate
access logs 30-90 days; useful for traffic analysis and abuse
auth logs 90+ days; needed for security investigation
anything with personal data the shortest period that meets your needs,
because retention creates legal obligationsHow it works #
Filtering the journal by unit, time and priority narrows thousands of lines to the handful that matter. Doing all three at once is far faster than scrolling.
The Nginx access log columns are positional: field 9 is the status code and field 7 the path in the default format. That is why the awk one-liners work, and why a custom format changes the field numbers.
Adding and upstream_response_time to the log format is worth doing in advance. The difference between them tells you whether time was spent in your application or elsewhere — a slow client upload shows as a long request time with a short upstream time.
Rotation replaces the current file, compresses the old one and deletes beyond the retention count. The postrotate reload tells the service to reopen its log file; without it the service keeps writing to the renamed file and the new one stays empty.
The journal is stored separately from /var/log files and needs its own size limit, or it grows until the disk is full.
Retention is a genuine decision. Longer is better for investigation and worse for cost and privacy obligations, and logs containing personal data fall under data protection rules.
Real-world use #
Disk full caused by logs is one of the most common server incidents, and it tends to happen during another incident when error logging is at its peak.
The timing fields in the access log are the fastest way to find slow endpoints without any instrumentation, which makes them valuable on a server with no metrics stack.
Once there is more than one server, centralised logging becomes necessary. Logs on one machine are of limited use when three are serving traffic.
Errors should also go to an error tracking service, which groups them, deduplicates and alerts. Logs tell you what happened; error tracking tells you it is happening now.
Never log secrets. Tokens and credentials in logs are readable by everyone with log access and are retained for the full retention period.
Common mistakes #
- Reading whole log files instead of filtering by unit, time and severity.
- No rotation, so logs fill the disk at the worst moment.
- Missing the postrotate reload, leaving the service writing to a deleted file.
- Leaving the journal unbounded.
- Logging tokens or personal data, creating a security and compliance problem.
Practice #
Set up an Nginx log format with request timing, then find the twenty slowest requests and the distribution of status codes. Configure rotation with 14-day retention and a postrotate reload, bound the journal size, and write down your retention decision for each log type.