Complete production-ready setup guide for Ubuntu Server running Node.js applications with PM2
Includes advanced security hardening, comprehensive monitoring, intrusion detection, and enterprise-grade deployment practices.
Table of Contents
- Initial Server Setup & User Configuration
- SSH Hardening & Key Management
- Firewall & Network Security
- Kernel Hardening & System Security
- Mandatory Access Control (MAC)
- System Auditing
- Intrusion Detection & File Integrity
- System Monitoring & Performance Tools
- Node.js & PM2 Installation
- Advanced PM2 Configuration
- Custom PM2 & Node.js Monitoring
- Log Management & Analysis
- Nginx Reverse Proxy (Advanced)
- SSL/TLS & Certificate Management
- Process Monitoring & Auto-Recovery
- Database Security Hardening
- Automated Backup Strategy
- Application Performance Monitoring (APM)
- Deployment Automation
- High Availability Setup
- Performance Optimization
- Security Scanning & Hardening
- Compliance & Logging
- Maintenance & Operations
- Advanced Troubleshooting & Debugging
- Quick Reference & Cheatsheets
1. Initial Server Setup & User Configuration
Update System Packages
# Update package lists
sudo apt update
# Upgrade all installed packages
sudo apt upgrade -y
# Full distribution upgrade
sudo apt dist-upgrade -y
# Remove unused packages
sudo apt autoremove -y
Create Non-Root User
# Create new user (replace 'username' with your desired username)
sudo adduser username
# Add user to sudo group
sudo usermod -aG sudo username
# Verify user was added to sudo group
groups username
Switch to Non-Root User
# Switch to the new user
su - username
# Or login via SSH with the new user
Configure Timezone
# List available timezones
timedatectl list-timezones
# Set timezone (example: UTC)
sudo timedatectl set-timezone UTC
# Verify timezone
timedatectl
Set Hostname
# Set hostname
sudo hostnamectl set-hostname your-server-name
# Update /etc/hosts
sudo nano /etc/hosts
# Add the following line (replace with your hostname and IP):
# 127.0.0.1 your-server-name
# your_server_ip your-server-name
Secure Shared Memory
Add to /etc/fstab to prevent execution from shared memory:
sudo nano /etc/fstab
Add this line at the end:
tmpfs /run/shm tmpfs defaults,noexec,nosuid 0 0
Remount shared memory:
sudo mount -o remount /run/shm
2. SSH Hardening & Key Management
Generate SSH Key (On Your Local Machine)
Option 1: RSA 4096-bit
ssh-keygen -t rsa -b 4096 -C "[email protected]"
Option 2: ED25519 (Recommended - More Secure)
ssh-keygen -t ed25519 -C "[email protected]"
Copy SSH Key to Server
# Copy key to server (replace username and server_ip)
ssh-copy-id -i ~/.ssh/id_rsa.pub username@server_ip
# Or for ED25519:
ssh-copy-id -i ~/.ssh/id_ed25519.pub username@server_ip
SSH Server Configuration
Edit SSH configuration:
sudo nano /etc/ssh/sshd_config
Apply these security settings:
# Disable root login
PermitRootLogin no
# Disable password authentication (only after SSH key is set up!)
PasswordAuthentication no
PubkeyAuthentication yes
# Change default SSH port (optional but recommended)
Port 2222
# Limit authentication attempts
MaxAuthTries 3
# Set login grace time
LoginGraceTime 30
# Disable empty passwords
PermitEmptyPasswords no
# Protocol version
Protocol 2
# Client alive interval (prevent timeout)
ClientAliveInterval 300
ClientAliveCountMax 2
# Limit users who can SSH (replace with your username)
AllowUsers username
# Disable X11 forwarding if not needed
X11Forwarding no
# Disable TCP forwarding if not needed
AllowTcpForwarding no
# Use strong ciphers only
Ciphers [email protected],[email protected],[email protected],aes256-ctr,aes192-ctr,aes128-ctr
MACs [email protected],[email protected],hmac-sha2-512,hmac-sha2-256
KexAlgorithms curve25519-sha256,[email protected],diffie-hellman-group16-sha512,diffie-hellman-group18-sha512
Restart SSH service:
sudo systemctl restart sshd
Important: Before closing your current SSH session, test the new configuration in a new terminal:
# If you changed the port:
ssh -p 2222 username@server_ip
SSH Key Rotation Strategy
# Generate new key periodically (every 90-180 days)
ssh-keygen -t ed25519 -C "[email protected]" -f ~/.ssh/id_ed25519_new
# Add new key to server
ssh-copy-id -i ~/.ssh/id_ed25519_new.pub username@server_ip
# Update ~/.ssh/authorized_keys on server to remove old keys
3. Firewall & Network Security
Install and Configure UFW
# Install UFW
sudo apt install ufw -y
# Set default policies
sudo ufw default deny incoming
sudo ufw default allow outgoing
# Allow SSH (use your custom port if changed)
sudo ufw allow 2222/tcp # or 'sudo ufw allow 22/tcp' for default
# Allow HTTP
sudo ufw allow 80/tcp
# Allow HTTPS
sudo ufw allow 443/tcp
# Allow specific IP to SSH (more secure)
sudo ufw allow from YOUR_IP_ADDRESS to any port 2222
# Enable UFW
sudo ufw enable
# Check status
sudo ufw status verbose
Rate Limiting with UFW
# Limit SSH connections to prevent brute force
sudo ufw limit 2222/tcp
# View rules with numbers
sudo ufw status numbered
# Delete a rule by number
sudo ufw delete [number]
Install and Configure Fail2ban
# Install Fail2ban
sudo apt install fail2ban -y
# Copy default configuration
sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
# Edit configuration
sudo nano /etc/fail2ban/jail.local
Basic Fail2ban configuration:
[DEFAULT]
# Ban IP for 1 hour
bantime = 3600
# IP is banned if it has generated "maxretry" during the last "findtime"
findtime = 600
maxretry = 5
# Email notifications (configure your email)
destemail = [email protected]
sendername = Fail2Ban
action = %(action_mwl)s
[sshd]
enabled = true
port = 2222
filter = sshd
logpath = /var/log/auth.log
maxretry = 3
bantime = 7200
[nginx-http-auth]
enabled = true
filter = nginx-http-auth
port = http,https
logpath = /var/log/nginx/error.log
[nginx-limit-req]
enabled = true
filter = nginx-limit-req
port = http,https
logpath = /var/log/nginx/error.log
[nginx-botsearch]
enabled = true
port = http,https
filter = nginx-botsearch
logpath = /var/log/nginx/access.log
maxretry = 2
Start and enable Fail2ban:
sudo systemctl start fail2ban
sudo systemctl enable fail2ban
# Check status
sudo fail2ban-client status
# Check specific jail
sudo fail2ban-client status sshd
# Unban an IP
sudo fail2ban-client set sshd unbanip IP_ADDRESS
Advanced: Port Knocking (Optional)
Install knockd:
sudo apt install knockd -y
# Edit configuration
sudo nano /etc/knockd.conf
Example configuration:
[options]
UseSyslog
[openSSH]
sequence = 7000,8000,9000
seq_timeout = 5
command = /sbin/iptables -I INPUT -s %IP% -p tcp --dport 2222 -j ACCEPT
tcpflags = syn
[closeSSH]
sequence = 9000,8000,7000
seq_timeout = 5
command = /sbin/iptables -D INPUT -s %IP% -p tcp --dport 2222 -j ACCEPT
tcpflags = syn
Enable knockd:
sudo systemctl enable knockd
sudo systemctl start knockd
Usage from client:
# Install knock on your local machine
sudo apt install knockd
# Knock to open port
knock server_ip 7000 8000 9000
# Connect via SSH
ssh -p 2222 username@server_ip
# Knock to close port
knock server_ip 9000 8000 7000
DDoS Protection with iptables
# Create DDoS protection script
sudo nano /etc/iptables/ddos-protection.sh
Add the following:
#!/bin/bash
# Drop invalid packets
iptables -t mangle -A PREROUTING -m conntrack --ctstate INVALID -j DROP
# Drop new packets that are not SYN
iptables -t mangle -A PREROUTING -p tcp ! --syn -m conntrack --ctstate NEW -j DROP
# Drop packets with bogus TCP flags
iptables -t mangle -A PREROUTING -p tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags FIN,SYN FIN,SYN -j DROP
iptables -t mangle -A PREROUTING -p tcp --tcp-flags SYN,RST SYN,RST -j DROP
# Limit connections per source IP
iptables -A INPUT -p tcp -m connlimit --connlimit-above 80 -j REJECT --reject-with tcp-reset
# Limit RST packets
iptables -A INPUT -p tcp --tcp-flags RST RST -m limit --limit 2/s --limit-burst 2 -j ACCEPT
iptables -A INPUT -p tcp --tcp-flags RST RST -j DROP
# Limit new TCP connections per second per source IP
iptables -A INPUT -p tcp -m conntrack --ctstate NEW -m limit --limit 60/s --limit-burst 20 -j ACCEPT
iptables -A INPUT -p tcp -m conntrack --ctstate NEW -j DROP
Make it executable and run:
sudo chmod +x /etc/iptables/ddos-protection.sh
sudo /etc/iptables/ddos-protection.sh
4. Kernel Hardening & System Security
sysctl Configuration
Edit sysctl configuration:
sudo nano /etc/sysctl.conf
Add these security settings:
# IP Forwarding (disable unless needed for routing)
net.ipv4.ip_forward = 0
net.ipv6.conf.all.forwarding = 0
# IP Spoofing protection
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
# Ignore ICMP redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0
# Ignore send redirects
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
# Disable source packet routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0
# Log Martians (packets with impossible addresses)
net.ipv4.conf.all.log_martians = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
# Ignore ICMP ping requests (optional - prevents ping)
net.ipv4.icmp_echo_ignore_all = 0
# Ignore Broadcast ICMP
net.ipv4.icmp_echo_ignore_broadcasts = 1
# Enable TCP SYN Cookie Protection (against SYN flood attacks)
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 5
# TCP/IP stack hardening
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15
# Increase TCP buffer sizes for better network performance
net.core.rmem_default = 31457280
net.core.rmem_max = 67108864
net.core.wmem_default = 31457280
net.core.wmem_max = 67108864
net.core.somaxconn = 4096
net.core.netdev_max_backlog = 65536
net.core.optmem_max = 25165824
net.ipv4.tcp_mem = 65536 131072 262144
net.ipv4.tcp_rmem = 8192 87380 67108864
net.ipv4.tcp_wmem = 8192 65536 67108864
# Enable IP conntrack to protect against DoS attacks
net.netfilter.nf_conntrack_max = 1000000
net.netfilter.nf_conntrack_tcp_loose = 0
# Protect against tcp time-wait assassination hazards
net.ipv4.tcp_rfc1337 = 1
# Decrease the time default value for connections to keep alive
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15
# Kernel self-protection
kernel.dmesg_restrict = 1
kernel.kptr_restrict = 2
kernel.yama.ptrace_scope = 2
kernel.kexec_load_disabled = 1
# Restrict core dumps
fs.suid_dumpable = 0
Apply sysctl settings:
sudo sysctl -p
Disable IPv6 (If Not Needed)
# Add to /etc/sysctl.conf
echo "net.ipv6.conf.all.disable_ipv6 = 1" | sudo tee -a /etc/sysctl.conf
echo "net.ipv6.conf.default.disable_ipv6 = 1" | sudo tee -a /etc/sysctl.conf
echo "net.ipv6.conf.lo.disable_ipv6 = 1" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
Secure /tmp and /var/tmp
Edit /etc/fstab:
sudo nano /etc/fstab
Add these lines:
tmpfs /tmp tmpfs defaults,noexec,nosuid,nodev 0 0
tmpfs /var/tmp tmpfs defaults,noexec,nosuid,nodev 0 0
Remount:
sudo mount -o remount /tmp
sudo mount -o remount /var/tmp
Enable ASLR (Address Space Layout Randomization)
# Check current ASLR status (should be 2)
cat /proc/sys/kernel/randomize_va_space
# Enable ASLR (if not already enabled)
echo "kernel.randomize_va_space = 2" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
Restrict Core Dumps
# Edit limits.conf
sudo nano /etc/security/limits.conf
Add:
* hard core 0
Also add to sysctl.conf:
echo "fs.suid_dumpable = 0" | sudo tee -a /etc/sysctl.conf
sudo sysctl -p
Kernel Module Blacklisting
Create blacklist file:
sudo nano /etc/modprobe.d/blacklist.conf
Add modules you want to disable (example):
# Disable uncommon network protocols
install dccp /bin/true
install sctp /bin/true
install rds /bin/true
install tipc /bin/true
# Disable uncommon filesystems
install cramfs /bin/true
install freevxfs /bin/true
install jffs2 /bin/true
install hfs /bin/true
install hfsplus /bin/true
install udf /bin/true
# Disable USB storage (if not needed)
# install usb-storage /bin/true
5. Mandatory Access Control (MAC)
Install AppArmor
# AppArmor is usually pre-installed on Ubuntu
sudo apt install apparmor apparmor-utils apparmor-profiles apparmor-profiles-extra -y
# Check AppArmor status
sudo aa-status
Enable AppArmor
sudo systemctl enable apparmor
sudo systemctl start apparmor
# Check if AppArmor is enabled
sudo apparmor_status
Create Custom AppArmor Profile for Node.js/PM2
# Create profile directory if it doesn't exist
sudo mkdir -p /etc/apparmor.d
# Create Node.js profile
sudo nano /etc/apparmor.d/usr.bin.node
Add the following profile:
#include <tunables/global>
/usr/bin/node {
#include <abstractions/base>
#include <abstractions/nameservice>
#include <abstractions/openssl>
# Node.js binary
/usr/bin/node mr,
/usr/lib/node_modules/** r,
# Application directories (adjust to your app location)
/home/*/app/** r,
/var/www/** r,
/opt/app/** r,
# Allow execution of application scripts
/home/*/app/**/*.js r,
/var/www/**/*.js r,
# Temporary files
/tmp/** rw,
/var/tmp/** rw,
# Logging
/var/log/** rw,
# PM2 directories
/home/*/.pm2/** rw,
# Network access
network inet stream,
network inet6 stream,
network inet dgram,
network inet6 dgram,
# Process capabilities
capability setuid,
capability setgid,
capability net_bind_service,
capability chown,
capability fowner,
capability fsetid,
capability kill,
capability dac_override,
# Allow reading system information
/proc/*/stat r,
/proc/*/status r,
/proc/sys/kernel/random/uuid r,
# Node modules with native code
/usr/lib/x86_64-linux-gnu/** rm,
/lib/x86_64-linux-gnu/** rm,
}
Load the profile:
# Parse and load the profile in complain mode first
sudo aa-complain /usr/bin/node
# Check for syntax errors
sudo apparmor_parser -r /etc/apparmor.d/usr.bin.node
# View status
sudo aa-status
Switch Between Enforce and Complain Modes
# Set to complain mode (logs violations but doesn't block)
sudo aa-complain /usr/bin/node
# Set to enforce mode (blocks violations)
sudo aa-enforce /usr/bin/node
# Disable a profile
sudo aa-disable /usr/bin/node
Troubleshoot AppArmor Denials
# View AppArmor denials
sudo dmesg | grep apparmor
sudo journalctl | grep apparmor
sudo grep DENIED /var/log/syslog
# Use aa-logprof to adjust profiles based on denials
sudo aa-logprof
6. System Auditing
Install Auditd
sudo apt install auditd audispd-plugins -y
# Start and enable auditd
sudo systemctl start auditd
sudo systemctl enable auditd
# Check status
sudo systemctl status auditd
Configure Audit Rules
Edit audit rules:
sudo nano /etc/audit/rules.d/audit.rules
Add comprehensive audit rules:
# Delete all existing rules
-D
# Buffer size
-b 8192
# Failure mode (0=silent 1=printk 2=panic)
-f 1
# Audit system calls
-a always,exit -F arch=b64 -S execve -k exec
-a always,exit -F arch=b32 -S execve -k exec
# Monitor file access - Critical system files
-w /etc/passwd -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/gshadow -p wa -k identity
-w /etc/sudoers -p wa -k sudoers
-w /etc/sudoers.d/ -p wa -k sudoers
# Monitor SSH configuration
-w /etc/ssh/sshd_config -p wa -k sshd_config
# Monitor privileged commands
-w /usr/bin/sudo -p x -k sudo_execute
-w /usr/bin/su -p x -k su_execute
# Monitor user/group modifications
-w /usr/sbin/useradd -p x -k user_modification
-w /usr/sbin/userdel -p x -k user_modification
-w /usr/sbin/usermod -p x -k user_modification
-w /usr/sbin/groupadd -p x -k group_modification
-w /usr/sbin/groupdel -p x -k group_modification
-w /usr/sbin/groupmod -p x -k group_modification
# Monitor network configuration
-w /etc/network/ -p wa -k network_modifications
-w /etc/hosts -p wa -k network_modifications
-w /etc/hostname -p wa -k network_modifications
-w /etc/issue -p wa -k system_info_modifications
# Monitor system startup scripts
-w /etc/init.d/ -p wa -k init
-w /etc/systemd/ -p wa -k systemd
# Monitor cron jobs
-w /etc/cron.allow -p wa -k cron
-w /etc/cron.deny -p wa -k cron
-w /etc/cron.d/ -p wa -k cron
-w /etc/cron.daily/ -p wa -k cron
-w /etc/cron.hourly/ -p wa -k cron
-w /etc/cron.monthly/ -p wa -k cron
-w /etc/cron.weekly/ -p wa -k cron
-w /etc/crontab -p wa -k cron
-w /var/spool/cron/ -p wa -k cron
# Monitor login/logout events
-w /var/log/faillog -p wa -k login
-w /var/log/lastlog -p wa -k login
-w /var/log/tallylog -p wa -k login
# Monitor process and session initiation
-w /var/run/utmp -p wa -k session
-w /var/log/wtmp -p wa -k logins
-w /var/log/btmp -p wa -k logins
# Monitor kernel modules
-w /sbin/insmod -p x -k modules
-w /sbin/rmmod -p x -k modules
-w /sbin/modprobe -p x -k modules
-a always,exit -F arch=b64 -S init_module,delete_module -k modules
# Monitor mounts
-a always,exit -F arch=b64 -S mount -S umount2 -k mounts
-a always,exit -F arch=b32 -S mount -S umount -S umount2 -k mounts
# Monitor file deletion by users
-a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat -k delete
-a always,exit -F arch=b32 -S unlink -S unlinkat -S rename -S renameat -k delete
# Monitor changes to system time
-a always,exit -F arch=b64 -S adjtimex -S settimeofday -k time_change
-a always,exit -F arch=b32 -S adjtimex -S settimeofday -S stime -k time_change
-w /etc/localtime -p wa -k time_change
# Monitor use of privileged commands
-a always,exit -F path=/usr/bin/passwd -F perm=x -F auid>=1000 -F auid!=4294967295 -k privileged_passwd_command
# Monitor permission modifications
-a always,exit -F arch=b64 -S chmod -S fchmod -S fchmodat -F auid>=1000 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b32 -S chmod -S fchmod -S fchmodat -F auid>=1000 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S chown -S fchown -S fchownat -S lchown -F auid>=1000 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b32 -S chown -S fchown -S fchownat -S lchown -F auid>=1000 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b64 -S setxattr -S lsetxattr -S fsetxattr -S removexattr -S lremovexattr -S fremovexattr -F auid>=1000 -F auid!=4294967295 -k perm_mod
-a always,exit -F arch=b32 -S setxattr -S lsetxattr -S fsetxattr -S removexattr -S lremovexattr -S fremovexattr -F auid>=1000 -F auid!=4294967295 -k perm_mod
# Make configuration immutable (requires reboot to change)
-e 2
Restart auditd to apply rules:
sudo systemctl restart auditd
# Or reload rules without restart
sudo augenrules --load
View Audit Logs
# View all audit logs
sudo ausearch -ts today
# Search for specific events
sudo ausearch -k identity
sudo ausearch -k sudo_execute
sudo ausearch -k user_modification
# View failed login attempts
sudo ausearch -m USER_LOGIN -sv no
# View successful logins
sudo ausearch -m USER_LOGIN -sv yes
# Search by user
sudo ausearch -ua username
# Generate audit report
sudo aureport
# Generate authentication report
sudo aureport -au
# Generate failed authentication report
sudo aureport -au --failed
# Generate summary of executable commands
sudo aureport -x
Automated Audit Reports
Create daily audit report script:
sudo nano /usr/local/bin/audit-report.sh
Add:
#!/bin/bash
REPORT_FILE="/var/log/audit/daily-report-$(date +%Y%m%d).txt"
{
echo "=== Daily Audit Report ==="
echo "Date: $(date)"
echo ""
echo "=== Authentication Summary ==="
sudo aureport -au --summary
echo ""
echo "=== Failed Authentications ==="
sudo aureport -au --failed
echo ""
echo "=== User Modifications ==="
sudo ausearch -k user_modification -ts today
echo ""
echo "=== Sudo Commands ==="
sudo ausearch -k sudo_execute -ts today
echo ""
echo "=== File Deletions ==="
sudo ausearch -k delete -ts today
echo ""
} > "$REPORT_FILE"
echo "Audit report generated: $REPORT_FILE"
Make it executable:
sudo chmod +x /usr/local/bin/audit-report.sh
Add to cron:
sudo crontab -e
# Add this line for daily report at 11:59 PM
59 23 * * * /usr/local/bin/audit-report.sh
7. Intrusion Detection & File Integrity
Install AIDE (Advanced Intrusion Detection Environment)
sudo apt install aide aide-common -y
# Initialize AIDE database (this takes several minutes)
sudo aideinit
# Move the new database to active location
sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
Configure AIDE
Edit AIDE configuration:
sudo nano /etc/aide/aide.conf
Key configuration sections:
# Define what to monitor
/etc PERMS
/bin PERMS
/sbin PERMS
/usr/bin PERMS
/usr/sbin PERMS
/var/www PERMS
/home PERMS
# Exclude directories that change frequently
!/var/log
!/var/cache
!/var/tmp
!/tmp
!/proc
!/sys
!/dev
!/run
# Custom rules for your application
/opt/app PERMS
/home/username/app PERMS
Run AIDE Checks
# Manual check
sudo aide --check
# Update database after legitimate changes
sudo aide --update
sudo mv /var/lib/aide/aide.db.new /var/lib/aide/aide.db
Automate AIDE Checks
Create daily check script:
sudo nano /etc/cron.daily/aide-check
Add:
#!/bin/bash
AIDE_LOG="/var/log/aide/aide-$(date +%Y%m%d).log"
# Create log directory
mkdir -p /var/log/aide
# Run AIDE check
/usr/bin/aide --check > "$AIDE_LOG" 2>&1
# Check exit code
if [ $? -ne 0 ]; then
# Send alert (configure email)
echo "AIDE detected changes on $(hostname)" | mail -s "AIDE Alert" [email protected]
fi
Make it executable:
sudo chmod 755 /etc/cron.daily/aide-check
Install and Configure rkhunter
sudo apt install rkhunter -y
# Update rkhunter database
sudo rkhunter --update
# Update file properties database
sudo rkhunter --propupd
Configure rkhunter:
sudo nano /etc/rkhunter.conf
Key settings:
# Update mirrors
UPDATE_MIRRORS=1
MIRRORS_MODE=0
# Email reports
[email protected]
MAIL_CMD=mail -s "[rkhunter] Warnings found for ${HOST_NAME}"
# Allow SSH root login setting to match your configuration
ALLOW_SSH_ROOT_USER=no
# Script whitelist (add as needed)
SCRIPTWHITELIST=/usr/bin/lwp-request
SCRIPTWHITELIST=/usr/bin/GET
SCRIPTWHITELIST=/usr/bin/whatis
# Allow hidden directories/files
ALLOWHIDDENDIR=/dev/.udev
ALLOWHIDDENDIR=/dev/.static
ALLOWHIDDENDIR=/dev/.initramfs
# Package manager
PKGMGR=DPKG
Run rkhunter check:
sudo rkhunter --check --skip-keypress
Automate rkhunter:
sudo nano /etc/cron.daily/rkhunter-check
Add:
#!/bin/bash
/usr/bin/rkhunter --update
/usr/bin/rkhunter --cronjob --report-warnings-only
Make executable:
sudo chmod 755 /etc/cron.daily/rkhunter-check
Install and Configure ClamAV
sudo apt install clamav clamav-daemon -y
# Stop the service to update virus definitions
sudo systemctl stop clamav-freshclam
# Update virus definitions
sudo freshclam
# Start the service
sudo systemctl start clamav-freshclam
sudo systemctl enable clamav-freshclam
Create scan script:
sudo nano /usr/local/bin/clamav-scan.sh
Add:
#!/bin/bash
LOGFILE="/var/log/clamav/clamav-scan-$(date +'%Y-%m-%d').log"
DIRTOSCAN="/home /var/www /opt"
# Create log directory
mkdir -p /var/log/clamav
echo "ClamAV scan started at $(date)" > "$LOGFILE"
# Scan and log
clamscan -r -i --exclude-dir=/proc --exclude-dir=/sys --exclude-dir=/dev $DIRTOSCAN >> "$LOGFILE"
echo "ClamAV scan completed at $(date)" >> "$LOGFILE"
# Check for infections
if grep -q "Infected files: 0" "$LOGFILE"; then
echo "No infections found"
else
# Send alert
echo "ClamAV found infections on $(hostname)" | mail -s "ClamAV Alert" [email protected]
fi
Make executable:
sudo chmod +x /usr/local/bin/clamav-scan.sh
Schedule weekly scans:
sudo crontab -e
# Add: Run every Sunday at 3 AM
0 3 * * 0 /usr/local/bin/clamav-scan.sh
8. System Monitoring & Performance Tools
Install Interactive Monitoring Tools
# htop - Interactive process viewer
sudo apt install htop -y
# iotop - Disk I/O monitoring
sudo apt install iotop -y
# nethogs - Network bandwidth per process
sudo apt install nethogs -y
# net-tools - Network utilities
sudo apt install net-tools -y
Install sysstat (sar, iostat, mpstat)
sudo apt install sysstat -y
# Enable data collection
sudo systemctl enable sysstat
sudo systemctl start sysstat
Configure sysstat:
sudo nano /etc/default/sysstat
Set:
ENABLED="true"
Edit collection interval:
sudo nano /etc/cron.d/sysstat
Usage examples:
# CPU usage statistics
sar -u 1 10 # 10 samples, 1 second apart
# Memory usage
sar -r 1 10
# I/O statistics
sar -b 1 10
# Network statistics
sar -n DEV 1 10
# View historical data (today)
sar -f /var/log/sysstat/sa$(date +%d)
# Display all statistics for specific time
sar -A -s 08:00:00 -e 12:00:00
# iostat - I/O statistics
iostat -x 1 10
# mpstat - Processor statistics
mpstat -P ALL 1 10
Install and Configure netdata
netdata provides real-time performance monitoring with a web dashboard.
# Install netdata
bash <(curl -Ss https://my-netdata.io/kickstart.sh)
# Or manual installation
sudo apt install netdata -y
# Start and enable netdata
sudo systemctl start netdata
sudo systemctl enable netdata
Configure netdata:
sudo nano /etc/netdata/netdata.conf
Key settings:
[global]
# Bind to localhost only (access via reverse proxy)
bind to = 127.0.0.1
[web]
web files owner = root
web files group = netdata
# Enable authentication
# Generate password: echo -n 'password' | md5sum
admin password = your_md5_hashed_password
Access netdata:
http://your-server:19999
For production, proxy it through Nginx (covered in Section 13).
Install Prometheus Node Exporter
# Create user for node_exporter
sudo useradd --no-create-home --shell /bin/false node_exporter
# Download and install
cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvf node_exporter-1.7.0.linux-amd64.tar.gz
sudo cp node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
Create systemd service:
sudo nano /etc/systemd/system/node_exporter.service
Add:
[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
Start node_exporter:
sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter
# Check status
sudo systemctl status node_exporter
# Test metrics endpoint
curl http://localhost:9100/metrics
Create Performance Baseline Script
sudo nano /usr/local/bin/performance-baseline.sh
Add:
#!/bin/bash
BASELINE_FILE="/var/log/performance-baseline-$(date +%Y%m%d-%H%M).txt"
{
echo "=== Performance Baseline ==="
echo "Date: $(date)"
echo ""
echo "=== System Information ==="
uname -a
echo ""
echo "=== CPU Information ==="
lscpu
echo ""
echo "=== Memory Information ==="
free -h
echo ""
echo "=== Disk Information ==="
df -h
echo ""
echo "=== Network Configuration ==="
ip addr
echo ""
echo "=== Top Processes by CPU ==="
ps aux --sort=-%cpu | head -20
echo ""
echo "=== Top Processes by Memory ==="
ps aux --sort=-%mem | head -20
echo ""
echo "=== Network Connections ==="
netstat -tuln
echo ""
echo "=== Load Average ==="
uptime
echo ""
echo "=== I/O Statistics ==="
iostat -x
echo ""
} > "$BASELINE_FILE"
echo "Performance baseline saved to $BASELINE_FILE"
Make executable:
sudo chmod +x /usr/local/bin/performance-baseline.sh
# Run to create baseline
sudo /usr/local/bin/performance-baseline.sh
9. Node.js & PM2 Installation
Install Build Essential and Compilation Tools
# Install build tools for native Node.js modules
sudo apt install build-essential libssl-dev -y
# Install additional tools
sudo apt install git curl wget -y
Option 1: Install Node.js via NVM (Recommended)
NVM allows you to manage multiple Node.js versions.
# Download and install NVM
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
# Load NVM
export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"
# Verify NVM installation
command -v nvm
# Install Node.js LTS
nvm install --lts
# Install specific version
nvm install 20
# List installed versions
nvm ls
# Use specific version
nvm use 20
# Set default version
nvm alias default 20
# Verify installation
node --version
npm --version
Option 2: Install Node.js via NodeSource
# Install Node.js 20.x (LTS)
curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt install -y nodejs
# Verify installation
node --version
npm --version
Install PM2 Globally
# Install PM2
sudo npm install -g pm2
# Verify installation
pm2 --version
# Update PM2
sudo npm install -g pm2@latest
Configure PM2 Startup Script
# Generate startup script (as your non-root user)
pm2 startup
# This will output a command to run with sudo, execute it
# Example: sudo env PATH=$PATH:/usr/bin /usr/lib/node_modules/pm2/bin/pm2 startup systemd -u username --hp /home/username
# Verify PM2 is configured to start on boot
sudo systemctl status pm2-username
Install PM2 Log Rotate Module
# Install pm2-logrotate
pm2 install pm2-logrotate
# Configure log rotation
pm2 set pm2-logrotate:max_size 10M
pm2 set pm2-logrotate:retain 30
pm2 set pm2-logrotate:compress true
pm2 set pm2-logrotate:dateFormat YYYY-MM-DD_HH-mm-ss
pm2 set pm2-logrotate:workerInterval 30
pm2 set pm2-logrotate:rotateInterval '0 0 * * *'
pm2 set pm2-logrotate:rotateModule true
# View current settings
pm2 conf pm2-logrotate
Optional: Install PM2 Plus/Keymetrics
# Link to PM2 Plus (commercial monitoring)
pm2 plus
# Or specify key
pm2 link YOUR_SECRET_KEY YOUR_PUBLIC_KEY
Configure npm for Global Packages (Optional)
If not using NVM, configure npm to install global packages without sudo:
# Create directory for global packages
mkdir ~/.npm-global
# Configure npm to use new directory
npm config set prefix '~/.npm-global'
# Add to PATH
echo 'export PATH=~/.npm-global/bin:$PATH' >> ~/.profile
# Source profile
source ~/.profile
10. Advanced PM2 Configuration
Basic PM2 Commands
# Start an application
pm2 start app.js
# Start with name
pm2 start app.js --name "my-app"
# Start with cluster mode
pm2 start app.js -i max
# List all processes
pm2 list
# Show detailed info
pm2 show app-name
# Monitor processes
pm2 monit
# Stop application
pm2 stop app-name
# Restart application
pm2 restart app-name
# Reload (zero-downtime reload)
pm2 reload app-name
# Delete from PM2
pm2 delete app-name
# Save process list
pm2 save
# Resurrect saved processes
pm2 resurrect
# Flush logs
pm2 flush
# View logs
pm2 logs
pm2 logs app-name
pm2 logs --lines 200
Create PM2 Ecosystem File
Create ecosystem.config.js in your project root:
module.exports = {
apps: [
{
name: "my-app",
script: "./src/index.js",
instances: "max", // or number like 4
exec_mode: "cluster",
watch: false,
max_memory_restart: "500M",
env: {
NODE_ENV: "production",
PORT: 3000,
},
env_development: {
NODE_ENV: "development",
PORT: 3000,
},
env_production: {
NODE_ENV: "production",
PORT: 3000,
},
error_file: "./logs/err.log",
out_file: "./logs/out.log",
log_date_format: "YYYY-MM-DD HH:mm:ss Z",
merge_logs: true,
autorestart: true,
max_restarts: 10,
min_uptime: "10s",
listen_timeout: 3000,
kill_timeout: 5000,
wait_ready: true,
shutdown_with_message: false,
},
],
}
Multi-App Ecosystem Configuration
module.exports = {
apps: [
{
name: "api-server",
script: "./apps/api/index.js",
instances: 4,
exec_mode: "cluster",
max_memory_restart: "500M",
env: {
NODE_ENV: "production",
PORT: 3000,
},
},
{
name: "worker",
script: "./apps/worker/index.js",
instances: 2,
exec_mode: "cluster",
max_memory_restart: "300M",
cron_restart: "0 0 * * *", // Restart daily at midnight
env: {
NODE_ENV: "production",
},
},
{
name: "scheduler",
script: "./apps/scheduler/index.js",
instances: 1,
exec_mode: "fork",
max_memory_restart: "200M",
env: {
NODE_ENV: "production",
},
},
],
}
Use Ecosystem File
# Start all apps from ecosystem file
pm2 start ecosystem.config.js
# Start with specific environment
pm2 start ecosystem.config.js --env production
# Start only specific app
pm2 start ecosystem.config.js --only api-server
# Update running apps from ecosystem file
pm2 restart ecosystem.config.js
# Reload with zero-downtime
pm2 reload ecosystem.config.js
# Delete all apps
pm2 delete ecosystem.config.js
Cluster Mode Optimization
module.exports = {
apps: [
{
name: "app",
script: "./app.js",
instances: "max", // Use all CPU cores
exec_mode: "cluster",
// CPU-based scaling
// instances: require('os').cpus().length - 1,
// Instance-specific configuration
instance_var: "INSTANCE_ID",
// Listen for 'ready' event for graceful reload
wait_ready: true,
// Graceful shutdown timeout
kill_timeout: 5000,
// Listen timeout
listen_timeout: 3000,
},
],
}
Example app code for graceful reload:
const express = require("express")
const app = express()
// Your routes here
app.get("/", (req, res) => {
res.send("Hello World")
})
const server = app.listen(3000, () => {
console.log("Server started on port 3000")
// Signal PM2 that app is ready
if (process.send) {
process.send("ready")
}
})
// Graceful shutdown
process.on("SIGINT", () => {
console.log("Received SIGINT, starting graceful shutdown")
server.close(() => {
console.log("Server closed")
process.exit(0)
})
// Force close after 10 seconds
setTimeout(() => {
console.error("Forced shutdown after timeout")
process.exit(1)
}, 10000)
})
Memory Leak Detection
module.exports = {
apps: [
{
name: "app",
script: "./app.js",
// Restart if memory exceeds 500MB
max_memory_restart: "500M",
// Minimum uptime before considering "online"
min_uptime: "10s",
// Maximum restarts within 1 min
max_restarts: 10,
// Delay between restarts
restart_delay: 4000,
// Autorestart
autorestart: true,
// Watch for file changes (development only)
watch: false,
// Ignore watch
ignore_watch: ["node_modules", "logs"],
},
],
}
PM2 with TypeScript
module.exports = {
apps: [
{
name: "ts-app",
script: "./dist/index.js", // Compiled JavaScript
// Or use ts-node directly (not recommended for production)
// interpreter: './node_modules/.bin/ts-node',
// script: './src/index.ts',
instances: "max",
exec_mode: "cluster",
env: {
NODE_ENV: "production",
TS_NODE_PROJECT: "./tsconfig.json",
},
},
],
}
Health Checks
Implement health check endpoint in your app:
// Express example
app.get("/health", (req, res) => {
res.status(200).json({
status: "ok",
uptime: process.uptime(),
timestamp: Date.now(),
})
})
Configure external monitoring to check this endpoint.
PM2 Deploy System
Configure deployment in ecosystem file:
module.exports = {
apps: [
{
name: "app",
script: "./app.js",
},
],
deploy: {
production: {
user: "node",
host: ["192.168.1.100"],
ref: "origin/main",
repo: "[email protected]:username/repo.git",
path: "/var/www/production",
"pre-deploy-local": "",
"post-deploy":
"npm install && pm2 reload ecosystem.config.js --env production",
"pre-setup": "",
ssh_options: "StrictHostKeyChecking=no",
},
},
}
Deploy commands:
# Setup remote server (first time)
pm2 deploy production setup
# Deploy code
pm2 deploy production
# Deploy and show logs
pm2 deploy production --force
# Revert to previous deployment
pm2 deploy production revert 1
# Execute command on remote
pm2 deploy production exec "pm2 reload all"
# List deployments
pm2 deploy production list
11. Custom PM2 & Node.js Monitoring
Create Enhanced PM2 Monitoring Script
sudo nano /usr/local/bin/pm2_monitor.sh
Add comprehensive monitoring script:
#!/bin/bash
# Configuration
LOG_FILE="/var/log/pm2_monitor.log"
MAX_LOG_SIZE="100M"
DATE=$(date '+%Y-%m-%d %H:%M:%S')
ALERT_EMAIL="[email protected]"
CPU_THRESHOLD=80
MEMORY_THRESHOLD=85
# Create log entry
{
echo "=== PM2 SYSTEM MONITOR: $DATE ==="
# System Load and Uptime
echo "SYSTEM UPTIME & LOAD:"
uptime
# Overall CPU Usage
echo ""
echo "CPU USAGE:"
top -bn1 | grep "Cpu(s)" | awk '{print "CPU: " 100 - $8 "%"}'
# Overall Memory Usage
echo ""
echo "MEMORY USAGE:"
free -h | grep -E "(Mem:|Swap:)"
# Disk Usage
echo ""
echo "DISK USAGE:"
df -h / | tail -1 | awk '{print "Root: " $5 " used (" $3 "/" $2 ")"}'
# PM2 Status
echo ""
echo "PM2 PROCESSES STATUS:"
pm2 jlist | jq -r '.[] | "\(.name): Status=\(.pm2_env.status) CPU=\(.monit.cpu)% Memory=\(.monit.memory/1024/1024|floor)MB Restarts=\(.pm2_env.restart_time) Uptime=\(.pm2_env.pm_uptime)"' 2>/dev/null || pm2 list
# Node.js Process Details
echo ""
echo "NODE.JS PROCESS DETAILS:"
ps aux | grep -E "(PM2|node)" | grep -v grep | awk '{printf "%-10s CPU:%s%% MEM:%s%% PID:%s CMD:%s\n", $1, $3, $4, $2, $11}'
# PM2 Logs Errors (last 10)
echo ""
echo "RECENT PM2 ERRORS:"
pm2 logs --err --lines 10 --nostream 2>/dev/null || echo "No recent errors"
# Network Connections
echo ""
echo "NETWORK CONNECTIONS (Port 3000-3010):"
netstat -tuln 2>/dev/null | grep -E ":(300[0-9]|301[0-9])" | wc -l | xargs -I {} echo "Active connections: {}"
# Check for memory leaks (processes using >1GB)
echo ""
echo "HIGH MEMORY PROCESSES (>1GB):"
ps aux --sort=-%mem | awk '$6 > 1048576 {printf "%-10s %s MB %s\n", $1, $6/1024, $11}' | head -5
# Event Loop Lag (if available from PM2)
echo ""
echo "PM2 METRICS:"
pm2 describe 0 2>/dev/null | grep -E "(restarts|uptime|memory|cpu)" || echo "PM2 detailed metrics not available"
echo "========================================="
echo ""
} >> "$LOG_FILE"
# Check for high CPU usage
CPU_USAGE=$(top -bn1 | grep "Cpu(s)" | awk '{print 100 - $8}' | cut -d. -f1)
if [ "$CPU_USAGE" -gt "$CPU_THRESHOLD" ]; then
echo "ALERT: High CPU usage detected: ${CPU_USAGE}%" >> "$LOG_FILE"
echo "High CPU usage on $(hostname): ${CPU_USAGE}%" | mail -s "CPU Alert" "$ALERT_EMAIL" 2>/dev/null
fi
# Check for high memory usage
MEM_USAGE=$(free | grep Mem | awk '{print int($3/$2 * 100)}')
if [ "$MEM_USAGE" -gt "$MEMORY_THRESHOLD" ]; then
echo "ALERT: High memory usage detected: ${MEM_USAGE}%" >> "$LOG_FILE"
echo "High memory usage on $(hostname): ${MEM_USAGE}%" | mail -s "Memory Alert" "$ALERT_EMAIL" 2>/dev/null
fi
# Check for crashed PM2 processes
STOPPED_PROCESSES=$(pm2 jlist | jq '[.[] | select(.pm2_env.status != "online")] | length' 2>/dev/null)
if [ "$STOPPED_PROCESSES" -gt 0 ]; then
echo "ALERT: $STOPPED_PROCESSES PM2 processes are not online" >> "$LOG_FILE"
echo "PM2 process alert on $(hostname): $STOPPED_PROCESSES processes down" | mail -s "PM2 Process Alert" "$ALERT_EMAIL" 2>/dev/null
# Auto-restart stopped processes
pm2 resurrect
fi
# Rotate log if it gets too large
if [ -f "$LOG_FILE" ]; then
LOG_SIZE=$(stat -c%s "$LOG_FILE" 2>/dev/null || stat -f%z "$LOG_FILE" 2>/dev/null)
MAX_SIZE_BYTES=104857600 # 100MB in bytes
if [ "$LOG_SIZE" -gt "$MAX_SIZE_BYTES" ]; then
mv "$LOG_FILE" "${LOG_FILE}.old"
gzip "${LOG_FILE}.old"
echo "Log rotated at $(date)" > "$LOG_FILE"
fi
fi
Make it executable:
sudo chmod +x /usr/local/bin/pm2_monitor.sh
Schedule Automated Monitoring
crontab -e
Add monitoring schedule:
# Monitor PM2 every 5 minutes
*/5 * * * * /usr/local/bin/pm2_monitor.sh
# Or monitor hourly
# 0 * * * * /usr/local/bin/pm2_monitor.sh
Node.js Heap Memory Monitoring Script
sudo nano /usr/local/bin/node_heap_monitor.sh
Add:
#!/bin/bash
LOG_FILE="/var/log/node_heap_monitor.log"
DATE=$(date '+%Y-%m-%d %H:%M:%S')
{
echo "=== Node.js Heap Monitor: $DATE ==="
# Get all Node.js processes
for PID in $(pgrep -f "node"); do
echo ""
echo "Process PID: $PID"
# Get heap statistics if process exposes them
if [ -f "/proc/$PID/status" ]; then
echo "Memory Status:"
grep -E "(VmRSS|VmSize|VmPeak)" /proc/$PID/status
fi
# Command line
echo "Command:"
ps -p $PID -o command --no-headers
done
echo "==========================================="
echo ""
} >> "$LOG_FILE"
Make it executable and schedule:
sudo chmod +x /usr/local/bin/node_heap_monitor.sh
# Add to crontab for hourly monitoring
crontab -e
# Add: 0 * * * * /usr/local/bin/node_heap_monitor.sh
Event Loop Lag Detection
Install and use loopbench in your Node.js application:
npm install loopbench
In your application:
const loopbench = require("loopbench")()
// Check event loop lag every minute
setInterval(() => {
const lag = loopbench.delay
const overLimit = loopbench.overlimit
if (overLimit) {
console.warn(`Event loop lag: ${lag}ms - OVER LIMIT`)
// Send alert or log to monitoring system
}
}, 60000)
// Also log on Express endpoint
app.get("/metrics", (req, res) => {
res.json({
eventLoopDelay: loopbench.delay,
eventLoopOverLimit: loopbench.overlimit,
uptime: process.uptime(),
memory: process.memoryUsage(),
})
})
Integration with External Alerting
Slack Webhook Integration
sudo nano /usr/local/bin/alert_slack.sh
Add:
#!/bin/bash
SLACK_WEBHOOK_URL="https://hooks.slack.com/services/YOUR/WEBHOOK/URL"
MESSAGE=$1
curl -X POST -H 'Content-type: application/json' \
--data "{\"text\":\"$MESSAGE\"}" \
$SLACK_WEBHOOK_URL
Make executable:
sudo chmod +x /usr/local/bin/alert_slack.sh
Use in monitoring scripts:
# Replace mail command with Slack alert
/usr/local/bin/alert_slack.sh "High CPU usage on $(hostname): ${CPU_USAGE}%"
PagerDuty Integration
#!/bin/bash
PAGERDUTY_KEY="your_integration_key"
MESSAGE=$1
curl -X POST https://events.pagerduty.com/v2/enqueue \
-H 'Content-Type: application/json' \
-d "{
\"routing_key\": \"$PAGERDUTY_KEY\",
\"event_action\": \"trigger\",
\"payload\": {
\"summary\": \"$MESSAGE\",
\"severity\": \"critical\",
\"source\": \"$(hostname)\"
}
}"
PM2 Metrics Dashboard
Create simple metrics endpoint:
sudo nano /usr/local/bin/pm2_metrics_server.js
Add:
const http = require("http")
const { exec } = require("child_process")
const PORT = 9209
const server = http.createServer((req, res) => {
if (req.url === "/metrics") {
exec("pm2 jlist", (error, stdout, stderr) => {
if (error) {
res.writeHead(500)
res.end("Error getting PM2 metrics")
return
}
try {
const processes = JSON.parse(stdout)
const metrics = processes.map((proc) => ({
name: proc.name,
status: proc.pm2_env.status,
cpu: proc.monit.cpu,
memory: proc.monit.memory,
restarts: proc.pm2_env.restart_time,
uptime: Date.now() - proc.pm2_env.pm_uptime,
}))
res.writeHead(200, { "Content-Type": "application/json" })
res.end(JSON.stringify(metrics, null, 2))
} catch (e) {
res.writeHead(500)
res.end("Error parsing PM2 data")
}
})
} else {
res.writeHead(404)
res.end("Not found")
}
})
server.listen(PORT, "127.0.0.1", () => {
console.log(`PM2 Metrics server running on port ${PORT}`)
})
Run as PM2 process:
pm2 start /usr/local/bin/pm2_metrics_server.js --name pm2-metrics
pm2 save
Access metrics:
curl http://localhost:9209/metrics
12. Log Management & Analysis
Install and Configure Logwatch
# Install Logwatch
sudo apt install logwatch -y
# Create configuration directory
sudo mkdir -p /var/cache/logwatch
# Edit configuration
sudo nano /etc/logwatch/conf/logwatch.conf
Configure Logwatch:
LogDir = /var/log
TmpDir = /var/cache/logwatch
Output = mail
Format = html
MailTo = [email protected]
MailFrom = [email protected]
Range = yesterday
Detail = Med
Service = All
Run Logwatch manually:
# Generate report for today
sudo logwatch --output stdout --format text --range today --detail high
# Save to file
sudo logwatch --output file --filename /var/log/logwatch.log --detail High --range today
# Email report
sudo logwatch --mailto [email protected] --range yesterday
Automated daily reports are usually configured by default in /etc/cron.daily/00logwatch.
PM2 Log Management
# View PM2 logs
pm2 logs
# View logs for specific app
pm2 logs app-name
# View last 200 lines
pm2 logs --lines 200
# View logs with JSON format
pm2 logs --json
# View only error logs
pm2 logs --err
# View only output logs
pm2 logs --out
# Flush all logs
pm2 flush
# Flush logs for specific app
pm2 flush app-name
Configure PM2 Log Rotation
Already configured in Section 9, but here are additional options:
# View current configuration
pm2 conf pm2-logrotate
# Set maximum file size before rotation
pm2 set pm2-logrotate:max_size 10M
# Number of rotated logs to keep
pm2 set pm2-logrotate:retain 30
# Compress rotated logs
pm2 set pm2-logrotate:compress true
# Date format for rotated logs
pm2 set pm2-logrotate:dateFormat YYYY-MM-DD_HH-mm-ss
# Rotation schedule (cron format)
pm2 set pm2-logrotate:rotateInterval '0 0 * * *'
# Enable/disable module rotation
pm2 set pm2-logrotate:rotateModule true
Centralized Logging with rsyslog
Configure rsyslog to forward logs to a remote server:
sudo nano /etc/rsyslog.d/50-default.conf
Add forwarding rules:
# Forward all logs to remote syslog server
*.* @@remote-syslog-server:514
# Or use TCP
*.* @@remote-syslog-server:514
# Or use UDP
*.* @remote-syslog-server:514
# Forward only specific logs
auth,authpriv.* @@remote-syslog-server:514
Restart rsyslog:
sudo systemctl restart rsyslog
Log Rotation for All Services
Configure global log rotation:
sudo nano /etc/logrotate.d/nodejs-apps
Add:
/var/www/*/logs/*.log {
daily
rotate 30
compress
delaycompress
notifempty
missingok
create 644 www-data www-data
sharedscripts
postrotate
# Reload PM2 to reopen log files
pm2 reloadLogs
endscript
}
Test log rotation:
sudo logrotate -d /etc/logrotate.d/nodejs-apps # Dry run
sudo logrotate -f /etc/logrotate.d/nodejs-apps # Force rotation
Real-time Log Analysis with multitail
# Install multitail
sudo apt install multitail -y
# View multiple log files
multitail /var/log/syslog /var/log/auth.log
# View PM2 logs
multitail ~/.pm2/logs/*-out.log
# With color scheme
multitail -c /var/log/syslog /var/log/auth.log
# Follow logs with filtering
multitail -e "error" /var/log/syslog
ELK Stack Basics (Optional)
For enterprise logging, consider ELK stack:
# Install Elasticsearch
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update && sudo apt install elasticsearch
# Install Logstash
sudo apt install logstash
# Install Kibana
sudo apt install kibana
# Start services
sudo systemctl start elasticsearch
sudo systemctl start logstash
sudo systemctl start kibana
This is beyond basic setup but provides centralized logging for large deployments.
13. Nginx Reverse Proxy (Advanced)
Install Nginx
# Install Nginx
sudo apt install nginx -y
# Start and enable Nginx
sudo systemctl start nginx
sudo systemctl enable nginx
# Check status
sudo systemctl status nginx
# Test configuration
sudo nginx -t
Basic Reverse Proxy Configuration
Create site configuration:
sudo nano /etc/nginx/sites-available/nodejs-app
Basic configuration:
server {
listen 80;
server_name yourdomain.com www.yourdomain.com;
location / {
proxy_pass http://localhost:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_cache_bypass $http_upgrade;
}
}
Enable the site:
sudo ln -s /etc/nginx/sites-available/nodejs-app /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
Advanced Reverse Proxy with Load Balancing
upstream nodejs_backend {
least_conn; # Load balancing method
server localhost:3000 weight=10 max_fails=3 fail_timeout=30s;
server localhost:3001 weight=10 max_fails=3 fail_timeout=30s;
server localhost:3002 weight=10 max_fails=3 fail_timeout=30s;
server localhost:3003 weight=10 max_fails=3 fail_timeout=30s;
# Health check
keepalive 32;
}
server {
listen 80;
listen [::]:80;
server_name yourdomain.com www.yourdomain.com;
# Security headers
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Referrer-Policy "no-referrer-when-downgrade" always;
add_header Content-Security-Policy "default-src 'self' http: https: data: blob: 'unsafe-inline'" always;
# Logs
access_log /var/log/nginx/nodejs-app-access.log;
error_log /var/log/nginx/nodejs-app-error.log;
# Client body size limit
client_max_body_size 100M;
# Gzip compression
gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_types text/plain text/css text/xml text/javascript application/json application/javascript application/xml+rss application/rss+xml font/truetype font/opentype application/vnd.ms-fontobject image/svg+xml;
location / {
proxy_pass http://nodejs_backend;
proxy_http_version 1.1;
# WebSocket support
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
# Headers
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $server_name;
# Timeouts
proxy_connect_timeout 60s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
# Buffering
proxy_buffering on;
proxy_buffer_size 4k;
proxy_buffers 8 4k;
proxy_busy_buffers_size 8k;
# Don't cache
proxy_cache_bypass $http_upgrade;
}
# Static files caching
location ~* \.(jpg|jpeg|png|gif|ico|css|js|svg|woff|woff2|ttf|eot)$ {
proxy_pass http://nodejs_backend;
expires 1y;
add_header Cache-Control "public, immutable";
}
# Health check endpoint
location /health {
proxy_pass http://nodejs_backend/health;
access_log off;
}
}
Rate Limiting
Add rate limiting to protect against abuse:
# Add to http block in /etc/nginx/nginx.conf
http {
# Rate limiting zones
limit_req_zone $binary_remote_addr zone=general:10m rate=10r/s;
limit_req_zone $binary_remote_addr zone=api:10m rate=5r/s;
limit_req_zone $binary_remote_addr zone=login:10m rate=1r/m;
# Connection limiting
limit_conn_zone $binary_remote_addr zone=addr:10m;
}
Use in server block:
server {
# ...
# General rate limit
limit_req zone=general burst=20 nodelay;
limit_conn addr 10;
location /api/ {
limit_req zone=api burst=10 nodelay;
proxy_pass http://nodejs_backend;
}
location /auth/login {
limit_req zone=login burst=2;
proxy_pass http://nodejs_backend;
}
}
Security Headers (Enhanced)
server {
# ...
# Strict Transport Security (HSTS)
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
# XSS Protection
add_header X-XSS-Protection "1; mode=block" always;
# Clickjacking Protection
add_header X-Frame-Options "SAMEORIGIN" always;
# MIME Type Sniffing Protection
add_header X-Content-Type-Options "nosniff" always;
# Referrer Policy
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
# Content Security Policy
add_header Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self' data:; connect-src 'self'; frame-ancestors 'self';" always;
# Permissions Policy
add_header Permissions-Policy "geolocation=(), microphone=(), camera=()" always;
}
GeoIP Blocking (Optional)
# Install GeoIP module
sudo apt install libnginx-mod-http-geoip2 -y
# Download GeoIP database
sudo mkdir -p /usr/share/GeoIP
cd /usr/share/GeoIP
sudo wget https://github.com/P3TERX/GeoLite.mmdb/raw/download/GeoLite2-Country.mmdb
Configure in nginx:
# In http block
http {
geoip2 /usr/share/GeoIP/GeoLite2-Country.mmdb {
$geoip2_data_country_code country iso_code;
$geoip2_data_country_name country names en;
}
map $geoip2_data_country_code $allowed_country {
default yes;
CN no; # Block China
RU no; # Block Russia
}
}
# In server block
server {
if ($allowed_country = no) {
return 403;
}
}
Nginx Caching
# In http block
http {
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=my_cache:10m max_size=1g inactive=60m use_temp_path=off;
}
# In server block
server {
location / {
proxy_cache my_cache;
proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
proxy_cache_background_update on;
proxy_cache_lock on;
# Cache based on these
proxy_cache_key "$scheme$request_method$host$request_uri";
# Cache valid responses
proxy_cache_valid 200 60m;
proxy_cache_valid 404 1m;
# Add cache status to response
add_header X-Cache-Status $upstream_cache_status;
proxy_pass http://nodejs_backend;
}
}
ModSecurity WAF Integration (Optional)
# Install ModSecurity
sudo apt install libmodsecurity3 libnginx-mod-http-modsecurity -y
# Download OWASP rules
sudo git clone https://github.com/coreruleset/coreruleset.git /etc/nginx/modsec/coreruleset
# Enable ModSecurity
sudo nano /etc/nginx/modsec/main.conf
Add:
SecRuleEngine On
SecRequestBodyAccess On
SecResponseBodyAccess On
SecResponseBodyMimeType text/plain text/html text/xml
Include /etc/nginx/modsec/coreruleset/crs-setup.conf
Include /etc/nginx/modsec/coreruleset/rules/*.conf
Enable in Nginx:
server {
modsecurity on;
modsecurity_rules_file /etc/nginx/modsec/main.conf;
}
Performance Tuning
Edit /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
worker_rlimit_nofile 65535;
pid /run/nginx.pid;
events {
worker_connections 4096;
use epoll;
multi_accept on;
}
http {
# Basic settings
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
server_tokens off;
# Buffer sizes
client_body_buffer_size 128k;
client_max_body_size 100m;
client_header_buffer_size 1k;
large_client_header_buffers 4 16k;
# Timeouts
client_body_timeout 12;
client_header_timeout 12;
send_timeout 10;
# File descriptors
open_file_cache max=200000 inactive=20s;
open_file_cache_valid 30s;
open_file_cache_min_uses 2;
open_file_cache_errors on;
}
14. SSL/TLS & Certificate Management
Install Certbot
# Install Certbot with Nginx plugin
sudo apt install certbot python3-certbot-nginx -y
# Verify installation
certbot --version
Obtain SSL Certificate
# For single domain
sudo certbot --nginx -d yourdomain.com
# For multiple domains
sudo certbot --nginx -d yourdomain.com -d www.yourdomain.com
# For wildcard certificate (requires DNS validation)
sudo certbot certonly --manual --preferred-challenges dns -d "*.yourdomain.com" -d yourdomain.com
Automated Certificate Renewal
Certbot automatically sets up renewal. Verify:
# Check renewal timer
sudo systemctl list-timers | grep certbot
# Test renewal (dry run)
sudo certbot renew --dry-run
# Force renewal
sudo certbot renew --force-renewal
Renewal Hooks
Create renewal hooks for custom actions:
# Pre-renewal hook
sudo nano /etc/letsencrypt/renewal-hooks/pre/01-pre-renewal.sh
Add:
#!/bin/bash
echo "Starting certificate renewal at $(date)" >> /var/log/certbot-renewal.log
# Post-renewal hook
sudo nano /etc/letsencrypt/renewal-hooks/post/01-post-renewal.sh
Add:
#!/bin/bash
echo "Certificate renewed at $(date)" >> /var/log/certbot-renewal.log
systemctl reload nginx
Make executable:
sudo chmod +x /etc/letsencrypt/renewal-hooks/pre/*
sudo chmod +x /etc/letsencrypt/renewal-hooks/post/*
SSL Labs A+ Rating Configuration
Update Nginx SSL configuration:
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name yourdomain.com www.yourdomain.com;
# SSL Certificate
ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
ssl_trusted_certificate /etc/letsencrypt/live/yourdomain.com/chain.pem;
# SSL Protocols and Ciphers
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
ssl_prefer_server_ciphers off;
# SSL Session
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:50m;
ssl_session_tickets off;
# OCSP Stapling
ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;
# Security Headers
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
# Disable SSL compression
ssl_compression off;
# Your application configuration
location / {
proxy_pass http://localhost:3000;
# ... rest of proxy configuration
}
}
# HTTP to HTTPS redirect
server {
listen 80;
listen [::]:80;
server_name yourdomain.com www.yourdomain.com;
# ACME challenge
location ^~ /.well-known/acme-challenge/ {
default_type "text/plain";
root /var/www/html;
}
# Redirect all other requests to HTTPS
location / {
return 301 https://$server_name$request_uri;
}
}
TLS 1.3 Only Configuration
For maximum security (may break older clients):
ssl_protocols TLSv1.3;
OCSP Stapling Configuration
# OCSP Stapling
ssl_stapling on;
ssl_stapling_verify on;
# OCSP responders
ssl_stapling_file /etc/nginx/ocsp/yourdomain.com.ocsp;
# DNS resolvers for OCSP
resolver 8.8.8.8 8.8.4.4 1.1.1.1 valid=300s;
resolver_timeout 5s;
Multi-Domain and SAN Certificates
# Generate certificate for multiple domains
sudo certbot --nginx -d example.com -d www.example.com -d api.example.com -d admin.example.com
# Or use certonly for manual installation
sudo certbot certonly --webroot -w /var/www/html -d example.com -d www.example.com
Certificate Monitoring
Create monitoring script:
sudo nano /usr/local/bin/check-ssl-expiry.sh
Add:
#!/bin/bash
DOMAIN="yourdomain.com"
DAYS_WARNING=30
ALERT_EMAIL="[email protected]"
# Get certificate expiry date
EXPIRY_DATE=$(echo | openssl s_client -servername $DOMAIN -connect $DOMAIN:443 2>/dev/null | openssl x509 -noout -enddate | cut -d= -f2)
# Convert to seconds
EXPIRY_EPOCH=$(date -d "$EXPIRY_DATE" +%s)
CURRENT_EPOCH=$(date +%s)
DAYS_UNTIL_EXPIRY=$(( ($EXPIRY_EPOCH - $CURRENT_EPOCH) / 86400 ))
if [ $DAYS_UNTIL_EXPIRY -lt $DAYS_WARNING ]; then
echo "SSL certificate for $DOMAIN expires in $DAYS_UNTIL_EXPIRY days" | mail -s "SSL Certificate Expiring Soon" $ALERT_EMAIL
fi
echo "SSL certificate for $DOMAIN expires in $DAYS_UNTIL_EXPIRY days"
Make executable and schedule:
sudo chmod +x /usr/local/bin/check-ssl-expiry.sh
# Add to crontab (check weekly)
sudo crontab -e
# Add: 0 0 * * 0 /usr/local/bin/check-ssl-expiry.sh
Sections 15-26: Advanced Topics
For the remaining sections covering advanced topics, please refer to UBUNTU_SERVER_PM2_SETUP_GUIDE_PART2.md:
- Section 15: Process Monitoring & Auto-Recovery (Monit)
- Section 16: Database Security Hardening (PostgreSQL, MongoDB, Redis)
- Section 17: Automated Backup Strategy
- Section 18: Application Performance Monitoring (APM)
- Section 19: Deployment Automation
- Section 20: High Availability Setup
- Section 21: Performance Optimization
- Section 22: Security Scanning & Hardening
- Section 23: Compliance & Logging
- Section 24: Maintenance & Operations
- Section 25: Advanced Troubleshooting & Debugging
- Section 26: Quick Reference & Cheatsheets
Summary
This guide has covered the essential foundation for running Node.js applications with PM2 in a production environment:
✅ Sections 1-8: System setup, SSH hardening, firewall, kernel hardening, AppArmor, auditing, intrusion detection, and monitoring tools
✅ Sections 9-11: Node.js/PM2 installation, advanced PM2 configuration, and custom monitoring
✅ Sections 12-14: Log management, Nginx reverse proxy, and SSL/TLS certificates
✅ Part 2: Advanced database security, backups, APM, deployment, HA, optimization, and troubleshooting
Quick Start Checklist
- [ ] Complete initial server setup and create non-root user
- [ ] Configure SSH keys and harden SSH configuration
- [ ] Setup UFW firewall and Fail2ban
- [ ] Apply kernel hardening with sysctl
- [ ] Install Node.js and PM2
- [ ] Configure PM2 with ecosystem file
- [ ] Setup Nginx reverse proxy
- [ ] Install SSL certificates with Certbot
- [ ] Configure monitoring and alerting
- [ ] Setup automated backups
- [ ] Test disaster recovery procedures
Support & Documentation
- Full documentation split across two files for comprehensive coverage
- Each section includes copy-paste ready commands
- Production-tested configurations
- Enterprise-grade security practices
Happy deploying! 🚀