Linux System Diagnostics
Understanding your system's hardware and software configuration
1.Displays kernel version and system architecture.
uname -a
2.Provides detailed information about the CPU.
lscpu
3.Lists all available block devices.
lsblk
4.Shows USB devices connected to the system.
lsusb
5.Shows how long the system has been running and load averages.Load average reflects the average number of processes waiting to run or use CPU resources over different time intervals, typically 1, 5, and 15 minutes.
uptime
Monitoring CPU and memory usage helps in identifying performance bottlenecks.
Commands:
htop is an interactive system monitor process viewer and process manager. It is designed as an alternative to the Unix program top. It shows a frequently updated list of the processes running on a computer, normally ordered by the amount of CPU usage.
Installation: on Ubuntu or Debian-based systems via command apt install htop or sudo yum install htop on CentOS
htop
vmstat -s # Provides a detailed breakdown of memory usage and other related metrics since the last system boot.
vmstat -s
980492 K total memory
480208 K used memory
132040 K active memory
582364 K inactive memory
67924 K free memory
21724 K buffer memory
410636 K swap cache
0 K total swap
free -h # Displays memory usage in "human-readable" format
free -h
mpstat
Linux 5.15.0-112-generic (testhost) 06/21/24 _x86_64_ (1 CPU)
10:37:12 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
10:37:12 all 1.68 0.11 1.38 0.14 0.00 0.04 1.16 0.00 0.00 95.49
Keeping track of disk usage and health is crucial for preventing data loss and ensuring efficient storage management.
df -h # Shows disk space usage in "human-readable" format
df -h
Filesystem Size Used Avail Use% Mounted on
udev 3.8G 0 3.8G 0% /dev
tmpfs 783M 1.7M 781M 1% /run
/dev/sda2 233G 25G 197G 12% /
tmpfs 3.9G 39M 3.8G 1% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 783M 80K 783M 1% /run/user/1000
du -sh /path/to/directory
fdisk -l # List all partitions
fdisk -l
Disk /dev/sda: 20 GiB, 21474836480 bytes, 41943040 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: XXXXXX-XXX-XXX-XXXX-XXXXXXXXXX
Device Start End Sectors Size Type
/dev/sda1 2048 4095 2048 1M BIOS boot
/dev/sda2 4096 41940607 41936512 20G Linux filesystem
iostat
Linux 5.15.0-112-generic (testhost) 06/21/24 _x86_64_ (1 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
1.68 0.11 1.41 0.14 1.16 95.50
Managing processes and services is essential for maintaining system stability.
ps aux # Lists all running processes
ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 168800 10944 ? Ss Jun12 0:02 /sbin/init
root 2 0.0 0.0 0 0 ? S Jun12 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? I< Jun12 0:00 [rcu_gp]
root 4 0.0 0.0 0 0 ? I< Jun12 0:00 [rcu_par_gp]
...
systemctl status service_name # Checks the status of a service
systemctl status apache2
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2024-05-29 16:06:04 UTC; 2 weeks 5 days ago
Docs: https://httpd.apache.org/docs/2.4/
Process: 1000205 ExecReload=/usr/sbin/apachectl graceful (code=exited, status=0/SUCCESS)
Main PID: 40385 (apache2)
Tasks: 55 (limit: 1013)
Memory: 8.8M
CPU: 1min 55.654s
CGroup: /system.slice/apache2.service
├─ 40385 /usr/sbin/apache2 -k start
├─1000209 /usr/sbin/apache2 -k start
systemctl restart service_name # Restarts a service.If a service is not behaving correctly, stopped responding, restarting it can often resolve the issue without needing a full system reboot. Also, when you modify a service config file, using systemctl restart applies the changes immediately.
systemctl enable service_name # Enables a service to start at boot.
kill -9 process_id(PID) # Forces termination of a process.
No output unless there is an error, the process is terminated forcefully
Analyzing logs and monitoring system activities can help in identifying issues and tracking system performance.
Linux log files are stored in plain-text and can be found in the /var/log directory
journalctl -f # will show you the latest system logs and keep updating the display with new log entries as they are created, which is useful for monitoring live system activity.
Essential Linux Log Files:
1.System logs:
/var/log/syslog or /var/log/messages - Stores all activity data across the global system. Activity for Redhat-based (CentOS or Rhel) stored in messages, while Debian-based (Ubuntu) systems are stored in syslog.
/var/log/dmesg dmesg allows reviewing messages stored in the Linux ring buffer, providing insights into hardware errors and startup issues. How To Use dmesg
dmesg -T
[Wed May 29 14:46:00 2024] Run /init as init process
[Wed May 29 14:46:00 2024] with arguments:
[Wed May 29 14:46:00 2024] /init
[Wed May 29 14:46:00 2024] with environment:
[Wed May 29 14:46:00 2024] HOME=/
[Wed May 29 14:46:00 2024] TERM=linux
[Wed May 29 14:46:00 2024] BOOT_IMAGE=/boot/vmlinuz-5.15.0-97-generic
[Wed May 29 14:46:00 2024] biosdevname=0
[Wed May 29 14:46:00 2024] netcfg/do_not_use_netplan=true
[Wed May 29 14:46:00 2024] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0
2.Service Logs:
/var/log/daemon.log - Tracks services running in the background that perform important tasks, but has no graphical output
journalctl # Views systemd logs and allows filtering Linux system logs to extract relevant information for monitoring and troubleshooting. How To Use journalctl
journalctl
-- Logs begin at Tue 2024-06-18 10:55:23 UTC, end at Tue 2024-06-18 12:06:19 UTC. --
Jun 18 10:55:23 hostname kernel: Linux version 5.4.0-74-generic (buildd@lcy01-amd64-013) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #83-Ubuntu SMP Thu May 6 10:34:06 UTC 2021 (Ubuntu 5.4.0-74.83-generic 5.4.101)
Jun 18 10:55:23 hostname kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-74-generic root=UUID=5d7f5dcd-1234-5678-9abc-def012345678 ro quiet splash vt.handoff=7
...
journalctl -u apache2
-- Logs begin at Tue 2024-06-18 10:55:23 UTC, end at Tue 2024-06-18 12:05:19 UTC. --
Jun 18 11:56:19 hostname systemd[1]: Starting The Apache HTTP Server...
Jun 18 11:56:19 hostname apache2[1342]: AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
Jun 18 11:56:19 hostname systemd[1]: Started The Apache HTTP Server.
/var/log/auth.log or /var/log/secure - Stores authentication logs, including both successful and failed logins and authentication methods. Debian/Ubuntu information is stored in /var/log/auth.log, while Redhat/CentrOS is stored in /var/log/secure.
4.Application Logs(examples):
/var/log/apache2/ - Apache HTTP server logs (access, error).
/var/log/mysql/ - MySQL database server logs.
5.Package Management Logs:
/var/log/apt directory - contains several log files that provide important information about apt-related activities on Debian-based systems.
/var/log/yum.log - Red Hat-based systems store the yum and dnf package manager logs, file provides information about the results of a specific command, such as any errors or warnings that were generated. For example, if a package update fails
6.Event Logs:
var/log/wtmp - Records of user logins and logouts.
var/log/lastlog - Information about the last logins for all users. This binary file can be read by command lastlog.