`Prometheus` is an open-source monitoring and alerting toolkit designed for reliability and scalability
* Responsible for collecting and aggregating metrics
* Designed to monitor numeric time series data
* Collects metrics from configured targets at given intervals by scraping them through an HTTP endpoint (The target will host the collected metrics on a specific HTTP endpoint and all Prometheus has to do is sent a GET request to retrieve those metrics)
* Scraped metrics are stored in a time series database that can be queried using PromQL
* Can be configured to scrape time series data from a different path other than /metrics
* Follows a pull based model so it needs to have a list of all targets that it should scrape
# Install Prometheus
```Bash
#!/bin/bash
# Step 1: Create a Prometheus user
sudo useradd --no-create-home --shell /bin/false prometheus
# Step 2: Create necessary directories
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
# Step 3: Download Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.52.0/prometheus-2.52.0.linux-amd64.tar.gz
# Step 4: Extract and move binaries
tar -xvf prometheus-2.52.0.linux-amd64.tar.gz
cd prometheus-2.52.0.linux-amd64
sudo cp prometheus /usr/local/bin/
sudo cp promtool /usr/local/bin/
sudo cp -r consoles /etc/prometheus
sudo cp -r console_libraries /etc/prometheus
sudo cp prometheus.yml /etc/prometheus
# Step 5: Set ownership
sudo chown -R prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
sudo chown -R prometheus:prometheus /var/lib/prometheus
# Step 6: Create Prometheus systemd service
touch prometheus.service
echo """
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/ --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target
""" > prometheus.service
sudo mv prometheus.service /etc/systemd/system/
# Step 7: Start Prometheus
sudo systemctl daemon-reexec
sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus
# Step 8: Access Prometheur through the web on port 9090
```
# Prometheus Configuration
/etc/prometheus/prometheus.yml
```YAML
# Specify the default values for how Prometheus scrapes targets and evaluates alert rules
global:
scrap_interval: SECONDS # How often to scrape the targets
evaluation_interval: SECONDS # How often to evaluate rules
scrape_timeout: SECONDS # How long to wait for a target to respond during a scrape
# Specify the targets
scrape_configs:
- job_name: 'JOB_NAME'
static_configs:
- targets:
- 'IP_ADDRESS:PORT'
# Modify labels for targets
relabel_configs:
- source_labels: [__address__]
regex: '(.*):.*'
target_label: instance
replacement: '$1'
# Configure Alertmanager
alerting:
alertmanagers:
- static_configs:
- targets: ['IP_ADDRESS:PORT']
# Remote write
remote_write:
- url: "https://REMOTE_ENDPOINT"
```
| Job Name | Use Case |
| ------------------ | -------------------- |
| `prometheus` | Prometheus itself |
| `node_exporter` | Linux system metrics |
| `mysql` | MySQL database |
| `postgres` | PostgreSQL database |
| `nginx_exporter` | NGINX web server |
| `docker` | Docker containers |
| `kubernetes-nodes` | Kubernetes nodes |
| `blackbox` | Blackbox probes |
| `myapp-backend` | Custom app |