Why Server Hardening Matters
In 2026, automated scanning bots find newly provisioned servers within minutes. Without proper hardening, your server will be compromised before you finish reading this article. Here is a comprehensive checklist based on years of production experience.
1. Keep Everything Updated
The most basic yet most overlooked security measure. Enable automatic security updates:
# Ubuntu/Debian
sudo apt install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades
# RHEL/Rocky
sudo dnf install dnf-automatic
sudo systemctl enable --now dnf-automatic-install.timer
2. SSH Hardening
SSH is the primary attack vector. Configure these in /etc/ssh/sshd_config:
PermitRootLogin no
PasswordAuthentication no
PubkeyAuthentication yes
MaxAuthTries 3
AllowUsers admin deployer
ClientAliveInterval 300
ClientAliveCountMax 2
3. Firewall Configuration
Use nftables (successor to iptables) to implement a default-deny policy. Only open ports you absolutely need.
4. Kernel Hardening with sysctl
# /etc/sysctl.d/99-hardening.conf
net.ipv4.ip_forward = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
kernel.randomize_va_space = 2
5. File System Security
- Mount
/tmp with noexec,nosuid
- Use
AIDE or OSSEC for file integrity monitoring
- Set proper file permissions โ follow the principle of least privilege
- Use disk encryption (LUKS) for sensitive data
Security is not a product, but a process. โ Bruce Schneier
6. Audit and Logging
Enable auditd for comprehensive system auditing. Forward logs to a centralized SIEM. Monitor failed login attempts, privilege escalation, and file changes.
Conclusion
Hardening is an ongoing process, not a one-time task. Regularly review your configurations, stay updated on CVEs, and practice incident response before you need it.
Enjoyed this article?
Subscribe to get more sysadmin tips and tutorials delivered to your inbox.
Get in Touch โ
The Container Security Landscape
Containers share the host kernel, making security configuration critical. A misconfigured container can lead to full host compromise.
1. Use Minimal Base Images
# Bad
FROM ubuntu:latest
# Good
FROM alpine:3.19
# or even better
FROM scratch
Smaller images have fewer vulnerabilities. Use distroless images when possible.
2. Run as Non-Root
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser
3. Image Scanning
Integrate Trivy or Grype into your CI/CD pipeline. Never deploy images with critical vulnerabilities.
4. Docker Security Options
docker run \
--read-only \
--tmpfs /tmp \
--security-opt no-new-privileges:true \
--cap-drop ALL \
--cap-add NET_BIND_SERVICE \
myapp:latest
5. Network Segmentation
Create custom Docker networks. Never expose containers directly to the internet without a reverse proxy.
6. Secrets Management
Never hardcode secrets in Dockerfiles or images. Use Docker secrets, Vault, or environment-specific secret injection.
A container is only as secure as its configuration.
Enjoyed this article?
Subscribe to get more sysadmin tips and tutorials delivered to your inbox.
Get in Touch โ
Why SysAdmins Need Kubernetes
Kubernetes has become the de facto standard for container orchestration. As a sysadmin, understanding K8s is no longer optional โ it is essential.
1. Cluster Architecture
Understand the control plane (API server, etcd, scheduler, controller manager) and worker nodes (kubelet, kube-proxy, container runtime).
2. Essential kubectl Commands
kubectl get nodes -o wide
kubectl get pods --all-namespaces
kubectl describe pod <pod-name>
kubectl logs -f <pod-name>
kubectl exec -it <pod-name> -- /bin/sh
3. Resource Management
Always set resource requests and limits. Without them, a single pod can starve the entire node.
4. RBAC
Implement Role-Based Access Control from day one. Follow least privilege โ no one needs cluster-admin except break-glass accounts.
5. Monitoring with Prometheus Operator
Deploy the kube-prometheus-stack for out-of-the-box monitoring of your cluster metrics, pod health, and node utilization.
Enjoyed this article?
Subscribe to get more sysadmin tips and tutorials delivered to your inbox.
Get in Touch โ
Why Ansible?
Ansible is agentless, uses YAML for playbooks, and has a massive module ecosystem. It is the perfect starting point for infrastructure automation.
1. Installation and Setup
pip install ansible
mkdir ~/ansible && cd ~/ansible
echo "[webservers]\nweb1.example.com\nweb2.example.com" > inventory
2. Your First Playbook
---
- name: Configure web servers
hosts: webservers
become: yes
tasks:
- name: Install nginx
apt:
name: nginx
state: present
update_cache: yes
- name: Start nginx
service:
name: nginx
state: started
enabled: yes
3. Roles and Directory Structure
Organize complex automation with Ansible roles. Each role has tasks, handlers, templates, and variables.
4. Ansible Vault for Secrets
ansible-vault create secrets.yml
ansible-vault edit secrets.yml
ansible-playbook site.yml --ask-vault-pass
5. Best Practices
- Use idempotent tasks โ running a playbook twice should have the same result
- Version control everything in Git
- Test with Molecule before deploying
- Use tags for selective execution
Enjoyed this article?
Subscribe to get more sysadmin tips and tutorials delivered to your inbox.
Get in Touch โ
The Monitoring Trinity
Effective monitoring requires metrics collection (Prometheus), visualization (Grafana), and alerting (Alertmanager). Together, they form the foundation of observability.
1. Installing Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.53.0/prometheus-2.53.0.linux-amd64.tar.gz
tar xvfz prometheus-*.tar.gz
cd prometheus-*/
./prometheus --config.file=prometheus.yml
2. Node Exporter for System Metrics
Deploy node_exporter on every server to collect CPU, memory, disk, and network metrics automatically.
3. Grafana Dashboards
Import community dashboards (ID 1860 for Node Exporter) and customize them. Set up data sources and create custom panels.
4. Alerting Rules
groups:
- name: infrastructure
rules:
- alert: HighCPU
expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU on {{ $labels.instance }}"
5. Best Practices
- Monitor the monitoring system itself
- Use recording rules for expensive queries
- Set up PagerDuty/OpsGenie integration for on-call
Enjoyed this article?
Subscribe to get more sysadmin tips and tutorials delivered to your inbox.
Get in Touch โ
The 3-2-1 Rule
Keep 3 copies of your data, on 2 different media types, with 1 copy offsite. This is the gold standard of backup strategy.
1. rsync for File Backups
rsync -avz --delete \
--exclude=".cache" \
/var/www/ \
backup@remote:/backups/www/
2. Database Backups
# PostgreSQL
pg_dump -Fc mydb > /backups/mydb_$(date +%Y%m%d).dump
# MySQL
mysqldump --single-transaction mydb | gzip > /backups/mydb.sql.gz
3. Automated Backup Scripts
Create cron jobs with proper error handling, notification, and rotation. Always test your restore process โ an untested backup is not a backup.
4. Offsite Storage
Use AWS S3 with lifecycle policies, Backblaze B2, or a secondary data center. Encrypt all offsite backups with GPG or age.
There are two types of people: those who have lost data and those who will.
Enjoyed this article?
Subscribe to get more sysadmin tips and tutorials delivered to your inbox.
Get in Touch โ
Default Deny Philosophy
Start with a default deny policy and only open what is necessary. This is the foundation of firewall security.
1. nftables Basics
#!/usr/sbin/nft -f
flush ruleset
table inet filter {
chain input {
type filter hook input priority 0; policy drop;
ct state established,related accept
iif lo accept
tcp dport 22 accept
tcp dport {80, 443} accept
}
chain forward {
type filter hook forward priority 0; policy drop;
}
chain output {
type filter hook output priority 0; policy accept;
}
}
2. Rate Limiting
Protect against brute force attacks by rate-limiting SSH connections. Combine with fail2ban for IP-based blocking.
3. Logging Dropped Packets
Add logging rules before your drop rules to troubleshoot connectivity issues without compromising security.
Enjoyed this article?
Subscribe to get more sysadmin tips and tutorials delivered to your inbox.
Get in Touch โ
DNS is the Backbone
When DNS breaks, everything breaks. Understanding DNS management is a core sysadmin skill.
1. DNS Record Types
A, AAAA, CNAME, MX, TXT, SRV, NS, SOA โ know them all and when to use each one.
2. Running Your Own DNS
# BIND9 zone file example
$TTL 86400
@ IN SOA ns1.example.com. admin.example.com. (
2026062001 ; Serial
3600 ; Refresh
900 ; Retry
604800 ; Expire
86400 ) ; Minimum
IN NS ns1.example.com.
IN NS ns2.example.com.
IN A 93.184.216.34
3. DNSSEC
Enable DNSSEC to prevent DNS spoofing attacks. Most cloud DNS providers support it natively.
4. Monitoring DNS
Monitor resolution times, check for propagation issues, and set up alerts for record changes.
Enjoyed this article?
Subscribe to get more sysadmin tips and tutorials delivered to your inbox.
Get in Touch โ
The CI/CD Imperative
Manual deployments are a liability. Automated pipelines reduce errors, increase velocity, and provide audit trails.
1. GitLab CI Example
stages:
- test
- build
- deploy
test:
stage: test
script:
- npm test
- npm run lint
build:
stage: build
script:
- docker build -t myapp:$CI_COMMIT_SHA .
- docker push registry.example.com/myapp:$CI_COMMIT_SHA
deploy_prod:
stage: deploy
script:
- kubectl set image deployment/myapp myapp=myapp:$CI_COMMIT_SHA
only:
- main
2. Jenkins vs GitLab CI
GitLab CI is more integrated and uses YAML configuration. Jenkins is more flexible with plugins but requires more maintenance. Choose based on your team size and needs.
3. Pipeline Security
- Never store secrets in pipeline code
- Use OIDC for cloud authentication
- Scan artifacts for vulnerabilities
- Require approval for production deployments
Enjoyed this article?
Subscribe to get more sysadmin tips and tutorials delivered to your inbox.
Get in Touch โ
Migration is Not Just Copy-Paste
A poorly planned migration can cause hours of downtime and data loss. This playbook covers the systematic approach I use for every migration.
Phase 1: Assessment
- Inventory all services, dependencies, and configurations
- Document network topology and firewall rules
- Identify peak usage patterns for scheduling
Phase 2: Preparation
Set up the target environment. Mirror configurations using Ansible playbooks. Test with non-production data first.
Phase 3: Data Sync
# Initial sync
rsync -avz /data/ target:/data/
# Final sync (during maintenance window)
rsync -avz --delete /data/ target:/data/
Phase 4: Cutover
DNS TTL reduction, final data sync, service switchover, and verification. Have a rollback plan ready.
The best migration is one that users never notice.
Enjoyed this article?
Subscribe to get more sysadmin tips and tutorials delivered to your inbox.
Get in Touch โ
Cloud is Not Just Someone Else Computer
Cloud infrastructure requires different thinking than traditional data centers. Embrace elasticity, automation, and managed services.
1. Multi-AZ Architecture
Deploy across multiple Availability Zones for high availability. Use load balancers to distribute traffic and handle zone failures gracefully.
2. Infrastructure as Code
# Terraform example
resource "aws_instance" "web" {
ami = "ami-0c55b159cbfafe1f0"
instance_type = "t3.micro"
tags = {
Name = "web-server"
Environment = "production"
ManagedBy = "terraform"
}
}
3. Cost Optimization
- Use spot instances for non-critical workloads
- Right-size instances based on actual usage
- Implement auto-scaling to match demand
- Use reserved instances for predictable workloads
Enjoyed this article?
Subscribe to get more sysadmin tips and tutorials delivered to your inbox.
Get in Touch โ
SSH is Your Front Door
SSH is the most targeted service on any server. Going beyond basic key authentication is essential for production environments.
1. Certificate-Based Authentication
Instead of managing individual public keys, use an SSH CA to sign certificates. This simplifies key management at scale.
2. SSH Bastion Hosts
Never expose SSH directly to the internet. Use a bastion/jump host as the single entry point with MFA.
3. SSH Configuration Hardening
# /etc/ssh/sshd_config
KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group16-sha512
Ciphers chacha20-poly1305@openssh.com,aes256-gcm@openssh.com
MACs hmac-sha2-512-etm@openssh.com
HostKeyAlgorithms ssh-ed25519,rsa-sha2-512
4. SSH Agent Forwarding Alternatives
Avoid agent forwarding (it is a security risk). Use ProxyJump instead for multi-hop connections.
Enjoyed this article?
Subscribe to get more sysadmin tips and tutorials delivered to your inbox.
Get in Touch โ
Databases Are Not Just for DBAs
As a sysadmin, you need to handle database backups, replication, performance tuning, and basic troubleshooting.
1. PostgreSQL Performance Tuning
# postgresql.conf key parameters
shared_buffers = 25% of RAM
effective_cache_size = 75% of RAM
work_mem = 64MB
maintenance_work_mem = 512MB
max_connections = 200
wal_level = replica
2. Automated Backups
Use pg_basebackup for physical backups and pg_dump for logical backups. Always test restores.
3. Replication
Set up streaming replication for high availability. Monitor replication lag and automate failover with Patroni.
4. Connection Pooling
Use PgBouncer to manage database connections efficiently and prevent connection exhaustion.
Enjoyed this article?
Subscribe to get more sysadmin tips and tutorials delivered to your inbox.
Get in Touch โ
Why Centralized Logging?
When troubleshooting across 100 servers, SSH-ing into each one to grep logs is not an option. Centralized logging is essential.
1. ELK Stack Components
- Elasticsearch: Search and analytics engine
- Logstash: Log processing pipeline
- Kibana: Visualization dashboard
2. Lightweight Alternative: Loki + Promtail
# promtail config
scrape_configs:
- job_name: system
static_configs:
- targets: [localhost]
labels:
job: varlogs
__path__: /var/log/*.log
3. Log Retention Policies
Define retention based on compliance requirements. GDPR requires you to justify how long you keep logs containing personal data.
4. Alerting on Log Patterns
Create alerts for error spikes, security events, and application anomalies using Kibana alerts or ElastAlert.
Enjoyed this article?
Subscribe to get more sysadmin tips and tutorials delivered to your inbox.
Get in Touch โ
Incidents Will Happen
The question is not if, but when. Having a structured incident response process reduces MTTR (Mean Time to Resolution) and minimizes damage.
1. The Incident Response Lifecycle
- Detection: Monitoring alerts, user reports, anomaly detection
- Triage: Assess severity, impact, and urgency
- Containment: Isolate affected systems to prevent spread
- Eradication: Remove the root cause
- Recovery: Restore services with verification
- Post-mortem: Document, learn, and improve
2. Communication During Incidents
Establish clear communication channels. Use a war room (Slack channel, Zoom call). Assign roles: Incident Commander, Communications Lead, Technical Lead.
3. Post-Mortem Template
# Incident Post-Mortem
## Summary
What happened in 2-3 sentences.
## Timeline
- HH:MM - Alert triggered
- HH:MM - Investigation started
- HH:MM - Root cause identified
- HH:MM - Fix deployed
- HH:MM - Services restored
## Root Cause
Technical explanation.
## Action Items
- [ ] Fix the bug
- [ ] Add monitoring
- [ ] Update runbook
Every incident is a learning opportunity. Blameless post-mortems build better systems.
Enjoyed this article?
Subscribe to get more sysadmin tips and tutorials delivered to your inbox.
Get in Touch โ