System Services and Systems

What Are System Services?

services are managed by Systemd
run continuously in the background
examples

SSH Service (sshd) - Waits quietly for an incoming network connection from a remote user
Web Server (apache2 or nginx) - waits for a browser to request a webpage
Print Spooler (cups) - waits for a document to be sent to the printer
Cron - waits for a specific time of day to run scheduled tasks

The PID 1 Controller

Init System is PID 1
when kernel finishes loading, it starts PID 1
in modern Linux PID1 is Systemd
Systemd creates a dependency tree

Understanding Init Systems: From SysVinit to Systemd

Systemd designed for speed and parallelism

Parallel Startup - starts as many services as possible at the same time
services used to be started sequentially
On-demabd Loading - Systemd can listen on a network port and only start the service when someone actually tries to connect
saves memory
Standardization - Systemd uses simple config files ending in .service

Systemd is default on almost every Linux distribution

Managing Services with systemctl

systemctl provides control panel

Checking the Status (status)
```
systemctl status <service name>
```
output returns
1. Loaded - shows where config is found nad whether it's enabled
  enabled means set to start on boot
2. Active - active (running) in green text
  inactive (dead) or failed means the service is down
3. Docs - links to man pages
4. Main PID - service's PID
5. CGroup - memory and CPU usage limits
6. logs - last few lines of log output
Starting and Stopping (start, stop) - to start a service
```
sudo systemctl start <service name>
```
to stop a service
```
sudo systemctl stop <service name>
```
commands do not output anything on success
Restarting and Reloading (restart, reload) - restart
```
sudo systemctl restart <service name>
```
to reload config without stopping the service
```
sudo systemctl reload <service name>
```
Enabling and Disabling (enable, disable)
- Start/Stop affects current state
- Enable/Disable affects future state (after boot)
to start a service automatically at boot
```
sudo systemctl enable <service name>
```
to stop a service from starting automatically at boot
```
sudo systemctl disable <service name>
```
when installing a new service
```
sudo systemctl enable --now  <service name>
```
--now both starts and enables the service
Masking (mask) -to ensure a service never runs
```
sudo systemctl mask <service name>
```
links the service file to /dev/null
makes it impossible to start
to undo use unmask
```
sudo systemctl unmask <service name>
```

Viewing Service Logs with journalctl

systemd's journalctl is a centralized logging system
collects output from every running service
stores data in binary format
to read logs use journalctl commands

The Firehose (journalctl)

shows every log message from every service

journalctl

Filtering by Service (-u)

to view logs from a specific service

journalctl -u <service name>

Following Logs in Real Time (-f)

works like tail
shows live feed of logs

journalctl -u <service name> -f

Filtering by Boot (-b)

see what happened during the current boot session

journalctl -b

see what happened during the previous boot session

journalctl -b -1

Filtering by Priority (-p)

log priority levels are

Info
Warning
Error

to only see errors

journalctl -p err

Filtering by Time (--since)

to see logs from the last hour

journalctl --since "1 hour ago"

to see logs from a specific time period

journalctl --since "2025-10-30 14:00:00" --until "2025-10-30 15:00:00"

Cleaning Logs

journal can grow large
to check space consumed by logs

journalctl --disk-usage

to clear old logs and keep only the last 1GB

sudo journalctl --vacuum-size=1G

Understanding Units: Service, Mount, Timer, and More

systemd also manages Units
a service is one type of unit
to see all units

systemctl list-units

Service Units (.service)

standard programs

Socket Units (.socket)

represents network socket of a file stream
Systemd can listen on a port (like port 80 for web) without actually running the web server
when a connection comes in

Systemd pauses the connection
starts the web server service
hands the connection over

Mount Units (.mount)

Systemd manages the mounting of hard drives and filesystems
/etc/fstab contains data entries which Systemd translates into Mount Units

Timer Units (.timer)

modern replacement for cron jobs
a timer unit is a schedule
are more powerful than Cron because they can handle dependencies

Target Units (.target)

a Target is a group of units
used to synchronize the boot process

Creating Custom Service Files

how to turn your own scripts into Systemd services
consider a Python script called backup_monitor.py which needs to run 24/7
run it a terminal, it will stop when the terminal closes
using nohup or backgrounding &, it won't restart if the computer reboots or if the script crashes
need to make it a first-class citizen managed by Systemd

Step 1 : Create the Script

make a dummy script for testing

nano /home/yourname/myscript.sh

the script content

#!/bin/bash 
while true do   
    echo "My service is running..."     
    sleep 10 done
done

make the script executable

chmod +x /home/yourname/myscript.sh

Step 2 : Creathe the Unit File

system configuration files live in

/etc/systemd/system

enter

sudo nano /etc/systemd/system/myservice.service

add the following content

[Unit] 
Description=My Custom Test Service
After=network.target 

[Service] 
Type=simple 
User=yourname 
ExecStart=/home/yourname/myscript.sh 
Restart=on-failure

[Install]
WantedBy=milti-user.target

explanation of content

[Unit]: Metadata
- Description: what appears in the log
- After=network.target: tells Systemd not to start script until networking is ready
[Service]: The Behavior
- Type=simple: The default. Means the process specified in ExecStart is the main service
- User=yourname: Crucial. By default, services run as root
  for security, always specify a regular user if root is not required
- ExecStart: The absolute path to the script
- Restart=on-failure: if the script crashes, Systemd will automatically restart it
[Install]: installation rules
- WantedBy=multi-user.target: "Start this service when the system enters standard non-graphical multi-user mode"
  effectively means "start on boot"

Step 3 : Reload and Start

need to have Systemd to scan for script

sudo systemctl daemon-reload

start the service

sudo systemctl start myservice

check the status

sudo systemctl status myservice

should se

Active (running)

check the logs

journalctl -u myservice

Step 4 : Test Resilence

verify the restart policy
kill the process manually
use PID from status output

kill -9 <PID>

wait a moment to give Systemd to restart the service
check status again
service should be running with a new PID

Step 5 : Clean Up

stop and disable service

sudo systemctl stop myservice 
sudo systemctl disable myservice 
sudo rm /etc/systemd/system/myservice.service 
sudo systemctl daemon-reload

System Startup and Boot Process Overview

sequence of events when a Linux box gets booted

Stage 1: BIOS /UEFI

when powered on the motherboard firmware (BIOS or UEFI) wakes up
performs a POST (Power On Self Test) to ensure RAM and CPU are working
looks for a bootable device (Hard Drive, USB, Network)

Stage 2: The Bootloader (GRUB)

firmware hands control to the Bootloader
on most Linux systems, this is GRUB (Grand Unified Bootloader)
GRUB's job is simple: Load the Linux Kernel into memory

Stage 3: The Kenel

initializes the hardware (drivers for graphics, disks, network)
the Kernel mounts the root filesystem (/)
the Kernel executes the very first program: /sbin/init (which is a link to Systemd)

Stage 4: Systemd (PID 1)

Systemd takes over
reads its "Target" configuration
the default target is graphical.target
Systemd calculates dependencies
works backward from the goal and starts everything required to reach it

Boot Target (Runlevels)

used to use Runlevels to define target types
Systemd uses targets mapping to similar concepts

poweroff.target (Runlevel 0) - shut down the system
rescur.target (Runlevel 1) - single-user mode
no network or graphical interface
used for repairing root password or fixing filesyestms
multi-user.target (RunLevel 3) - Standard system with networking and multiple users
no graphical interface
most servers run this way
graphical.target (Runlevel 5) - standard system with graphical interface
reboot.target (Runlevel 6) - reboot the system

Changing Targets

to change targets immediately

sudo systemctl isolate multi-user.target

to change default target for next boot

sudo systemctl set-default multi-user.target

Troubleshooting Service Failures

an administrator will spend a lot of time fixing services which refuse to start
a systematic workflow for trouble shooting

Scenario

Nginx web server is installed but not loading

Step 1 - Check Status
```
sudo systemctl status nginx
```
color dot will show the service's state
- green - service is running
  problem elsewhere (firewall or network issue)
- red (failed) - service tried to start and crashed
- gray (dead) - process never told to start

Step 2 - Read the Error Message

journalctl -u <service name>
journalctl -u nginx

Step 3 - Check the Full Logs
```
journalctl -u <service name> -e
journalctl -u ngix -e
```
-e flag jumps to end of the log
look for specific error
Step 4 - Check Configuration Syntax
most services have a config test command
service-dependent
commands read config files checking for typos
does not try to start service
Step 5 - Check Permissions
rwx------ (700) blocks everyone but the owner
Step 6 - The Reset
need to be aggressive
```
sudo systemctl stop nginx 
sudo killall nginx 
sudo systemctl start nginx
```
the second command makes certain no zombie processes remain

Summary

covered

The Controller - systemctl is the main tool
The Verbs - start, stop, restart, enable, disable
The Logs - journalctl can filter logs by service and time
The Customization - writing a .service to manage scripts
The Boot Process

key points

Service - background process managed by the init system
Systemd - modern init system, PID 1
systemctl status <service name> - check a service's health
systemctl enable <service name> - service start automatically on boot
journalctl -u <service name> -view logs specific to service
/etc/systemd/system - location for custom system unit files
daemon-reload - command used to reload after unit file edit
target - a synchronization point