Was this page helpful?
Caution
You're viewing documentation for an unstable version of ScyllaDB Monitoring. Switch to the latest stable version.
Configure the Stack from One File¶
Instead of passing command-line flags to start-all.sh, you can describe the whole monitoring stack in one YAML file.
scylla_monitoring_config.py, the watcher, reads the file, applies it, and applies it again whenever it changes.
The file holds everything the stack needs: the targets, the alert rules, the Alertmanager configuration, the Grafana settings, the storage directories, TLS, and systemd drop-ins for host services such as node_exporter. The watcher generates the derived files from it, starts the stack, and on a later change runs only what that change needs: a change to the targets needs no restart, a change to the alert rules reloads Prometheus, and a new Grafana password is set in place.
Requirements¶
Python 3, with
PyYAML(python3-yaml) andjsonschema3.2 or later (python3-jsonschema).The watcher runs as a regular user who is a member of the
dockergroup.TLS certificates and systemd drop-ins need root. The watcher runs
scylla-monitoring-root.shfor them withsudo -n. Install the script root-owned and let the watcher’s user run it, and only it, for example:sudo install -o root -g root -m 755 scylla-monitoring-root.sh /usr/local/sbin/ echo 'scylla ALL=(root) NOPASSWD: /usr/local/sbin/scylla-monitoring-root.sh' | sudo tee /etc/sudoers.d/scylla-monitoring
Replace
scyllawith the watcher’s user. Without thetls,systemdanddockersections, the watcher does not need root.
The Configuration File¶
scylla-monitoring.schema.json, in the root of the monitoring directory, defines the file.
The description of each field names the start-all.sh flag it replaces and its default.
scylla-monitoring.example.yaml is a complete, commented example:
# An example of the configuration file applied by scylla_monitoring_config.py.
# scylla-monitoring.schema.json is the definition; this file is checked
# against it. Every write replaces the file whole; keep it 0600, it holds
# secrets.
version: 1
generation: "2026-10-21T10:00:00Z-1" # set by the writer; reported back in the status
cluster: cluster1 # the cluster label on the prometheus and grafana jobs, and the
# cluster external label; only for a monitor of one cluster,
# set to the cluster label of its targets
stack:
scylla_version: "2025.3" # -v
manager_version: "3.5" # -M; missing: the release default
alertmanager: true # false: --no-alertmanager
log_collector: alloy # --log-collector alloy|promtail|none
thanos_sidecar: true # --thanos-sc
local_thanos: false # --local-thanos
quick_startup: false # --quick-startup
# compose: true # docker compose; missing: --compose or WATCHER_COMPOSE=1, else start-all.sh
bind_address: 127.0.0.1 # -A; 127.0.0.1 when tls.enabled is true
docker_args: ["--restart=unless-stopped"] # -D
extra_flags: [] # start-all.sh syntax; takes precedence over the rest
storage:
prometheus_dir: /data/prometheus # -d
grafana_dir: /data/grafana # -G
alertmanager_dir: /data/alertmanager # -f
loki_dir: /data/loki # -k
retention_days: 30 # -b --storage.tsdb.retention.time=30d
prometheus:
external_url: https://example.com/prometheus # -b --web.external-url, with --web.route-prefix=/
grafana:
root_url: https://example.com/grafana # GF_SERVER_ROOT_URL
default_theme: light # GF_USERS_DEFAULT_THEME
anonymous_role: Viewer # -Q
thanos_datasource: true # false: --no-thanos-datasource
env: # -c, any other Grafana setting
GF_SECURITY_COOKIE_SECURE: "true"
admin_password: "change-me" # GRAFANA_ADMIN_PASSWORD
datasource:
scylla_user: "monitor" # SCYLLA_USER
scylla_password: "change-me" # SCYLLA_PSSWD
targets: # Prometheus file_sd entries
scylla: # hosts without a port; also scraped for node_exporter and the manager agent
- targets: ["10.0.0.1", "10.0.0.2"]
labels: {cluster: cluster1, dc: dc1}
- targets: ["10.0.0.3"]
labels: {cluster: cluster1, dc: dc2}
node_exporter: # optional; replaces the copy of scylla
- targets: ["10.0.1.1:9100"]
labels: {cluster: cluster1, dc: dc1, instance: "10.0.0.1"}
# manager_agent: # optional; replaces the copy of scylla; a target keeps its port
# - targets: ["10.0.0.1:26090"]
# labels: {cluster: cluster1, dc: dc1}
manager:
- targets: ["10.0.0.5:5090"]
vector_search: # present: --vector-search
- targets: ["10.0.0.6"]
labels: {cluster: cluster1}
alert_rules: # a Prometheus rule file
groups:
- name: scylla.rules
rules:
- alert: InstanceDown
expr: up{job="scylla"} == 0
for: 30s
labels:
severity: error
annotations:
description: "{{ $labels.instance }} has been down for more than 30 seconds."
summary: Instance {{ $labels.instance }} down
alertmanager_config: # an Alertmanager config file
route:
receiver: default
group_by: [alertname, cluster]
receivers:
- name: default
systemd: # drop-ins for host services, by unit
node-exporter.service: | # 127.0.0.1:9100 behind traefik
[Service]
Environment=NODE_EXPORTER_ARGS=--web.listen-address=127.0.0.1:9100
# docker: # the docker daemon's proxy; missing: docker is left as it is
# http_proxy: http://10.0.0.5:3128 # HTTP_PROXY
# https_proxy: http://10.0.0.5:3128 # HTTPS_PROXY
tls:
enabled: true # run traefik
listen_ip: 10.0.0.10 # PRIVATE_IP of start-traefik.sh
thanos_port: 10911 # behind :10911; 10911 the sidecar, 10903 the Thanos query
ca_crt: | # rootCA.crt
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
server_crt: | # server.crt
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
server_key: | # server.key
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
version, generation, stack and targets are required. A missing optional section keeps the
start-all.sh default: without tls no traefik runs, without systemd the host’s units are left as they are,
and without docker the host’s Docker daemon is left as it is.
stack.extra_flags takes any other start-all.sh flag, in start-all.sh syntax. A flag there takes
precedence over the same option derived from the rest of the file. The options the watcher sets from the file itself
(the targets, rule and Alertmanager files, --target-directory, --vector-search, --vector-store,
--cluster-label, --archive, --compose and -h) are rejected in extra_flags.
Writing the File¶
Always write the whole file, and replace it atomically: write a temporary file in the same directory and rename it into place.
The file holds secrets (the Grafana admin password, the CQL password and the TLS private key). Keep it
0600, owned by the watcher’s user.Set
generationto a new value on every write. The watcher reports it back in the status file, so you know when your change is applied.The watcher ignores fields it does not know. With
--strict, an unknown field rejects the file.
Running the Watcher¶
scylla_monitoring_config.py check|apply|run --config-file FILE --monitoring-dir DIR [options]
checkvalidates the file, renders the derived files into a temporary directory and prints thestart-all.shcommand line, with the secrets masked. It changes nothing, so it is safe to run next to a running watcher.applyapplies the file once.--forceapplies it as a first apply, restarting the whole stack.runapplies the file on start, then checks it every--poll-intervalseconds and applies it when it changes. On start, it checks that the stack is up; when it is not (for example, after a reboot), the file is applied whole. Until the file exists, it waits and changes nothing.
Options:
–config-file path The configuration file. It can be given more than once; the watcher watches every path and applies the one that changed last.
–monitoring-dir path The monitoring directory, where start-all.sh is. The watcher runs the start scripts from it.
–compose Run the stack with Docker Compose instead of start-all.sh, unless the file’s stack.compose says
otherwise. See Docker Compose.
–strict Reject fields the schema does not define.
–auto-revert When an apply fails, apply the previous configuration again.
–state-dir path Where the last applied configuration and the status are kept (default: <monitoring-dir>/config-state).
–staging-dir path Where the derived files are generated (default: <monitoring-dir>/config-build).
–cert-dir path Where the TLS certificates are installed (default: /etc/scylla-monitoring/certs).
–root-helper path The root helper (default: <monitoring-dir>/scylla-monitoring-root.sh).
–health-timeout seconds How long to wait for the components to become ready after an apply (default: 180).
–poll-interval seconds How often run checks the file (default: WATCHER_POLL_INTERVAL in env.sh, else 30).
A check only reads the file’s size and modification time, so a short interval, such as 5, costs nothing and
picks up a write within seconds.
Settings in env.sh¶
run reads these settings from env.sh in the monitoring directory. The machine images built from packer/ set
WATCHER_POLL_INTERVAL.
WATCHER_POLL_INTERVAL How often run checks the file, in seconds, when --poll-interval is not given (default: 30).
WATCHER_COMPOSE With WATCHER_COMPOSE=1, the stack runs with Docker Compose when the file does not set
stack.compose, as with --compose. The file’s stack.compose takes precedence; without either, the stack runs
one container at a time with start-all.sh. A change of stack.compose stops the stack (docker compose down,
or kill-all.sh) and starts it in the other mode.
Grafana settings can be set in env.sh too: the start scripts pass every GF_* variable that env.sh, or the
environment, sets to the Grafana container, in place of the stack’s default for that setting. A -c value, or an
-e/--env in -D or --limit grafana, that sets the same variable takes precedence. This works with
start-all.sh and with Docker Compose, with or without the watcher. The values go by name, so a secret such as
GF_DATABASE_PASSWORD is neither on the docker run command line nor in docker-compose.yml; with Compose it is
in .env, which only its owner can read. The machine images use it to keep Grafana’s first start short:
GF_PLUGINS_PREINSTALL_DISABLED=true
GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=scylladb-scylla-datasource
GF_PLUGINS_DISABLE_PLUGINS=cloudwatch,elasticsearch,mssql,grafana-postgresql-datasource,influxdb,mysql,grafana-pyroscope-datasource,stackdriver,zipkin,opentsdb,jaeger,tempo
Grafana then installs no plugin from grafana.com when it starts, so grafana/plugins holds only the plugins monitoring
ships (the ScyllaDB data source), and it loads none of its bundled data sources that the dashboards do not use.
Remove a line for Grafana’s default. The image’s env.sh writes each one as NAME=${NAME:-value}, so a value
exported in the environment of the start scripts overrides it.
Reading Grafana’s files on boot¶
A new machine loads its disk from the image on first read, which is slow, and Grafana reads a few hundred MB when it
starts. The machine images run scylla-monitoring-preread.service on every boot: it reads Grafana’s binary, its
bundled plugins that GF_PLUGINS_DISABLE_PLUGINS does not disable, and the stack’s grafana/plugins,
grafana/provisioning and grafana/build, while the machine waits to be configured. It starts no container, and
nothing waits for it: the stack starts whenever start-all.sh or the watcher starts it, and finds on the disk what was
read by then. On a later boot the files are already on the disk, and it ends in seconds.
It logs one line when it ends, with the MB of files it read:
$ journalctl -u scylla-monitoring-preread.service
Grafana pre-read: finished, 512 MB in 58s
Grafana pre-read: stopped by the 120s limit, 402 MB in 120s
Grafana pre-read: ended with read errors (status 123), 380 MB in 49s
Its settings are in env.sh:
GRAFANA_PREREAD GRAFANA_PREREAD=0 turns it off (default: on).
GRAFANA_PREREAD_TIMEOUT It stops after this many seconds (default: 300; the images set 120).
GRAFANA_PREREAD_PARALLEL How many files it reads at the same time (default: 4).
As a systemd Service¶
Run the watcher as a service, for example with /etc/systemd/system/scylla-monitoring-config.service:
[Unit]
Description=Configure the monitoring stack from its config file
After=docker.service network-online.target
Wants=network-online.target
PartOf=docker.service
[Service]
User=scylla
WorkingDirectory=/opt/scylla-monitoring
ExecStart=/usr/bin/python3 /opt/scylla-monitoring/scylla_monitoring_config.py run --monitoring-dir /opt/scylla-monitoring --config-file /etc/scylla-monitoring/scylla-monitoring.yaml --root-helper /usr/local/sbin/scylla-monitoring-root.sh --auto-revert
Restart=on-failure
[Install]
WantedBy=multi-user.target
To start a stopped watcher whenever the file is written, add a path unit, /etc/systemd/system/scylla-monitoring-config.path,
with one PathChanged= for each --config-file, and enable it:
[Unit]
Description=Start the monitoring config watcher when its config file is written
[Path]
PathChanged=/etc/scylla-monitoring/scylla-monitoring.yaml
Unit=scylla-monitoring-config.service
[Install]
WantedBy=multi-user.target
The watcher’s user needs to read the config file. It writes the status next to the file when it can write to that directory,
and always in the state directory. When the data directories are on a separate disk, add RequiresMountsFor=<data dir> in a drop-in.
systemctl stop scylla-monitoring-configstops the watcher;./kill-all.sh --traefik(ordocker compose down) then stops the containers, traefik included.systemctl start scylla-monitoring-configstarts the watcher, which brings the stack up.With the path unit, writing the file starts a stopped watcher too. A stop without a new file keeps the watcher and the stack down.
journalctl -u scylla-monitoring-configshows how long each apply took: the time of each step, the health check’s progress per component, and a summary line with the outcome and the total time.
How Changes Are Applied¶
Each apply validates the file, renders the derived files and checks them with promtool and amtool.
If the file or a derived file is invalid, the running stack is left as it is, and the status file says why.
Otherwise the watcher compares each section with the last applied configuration and runs what the changed sections need:
Change |
What runs |
|---|---|
|
nothing; Prometheus reads the new targets |
|
Prometheus reload |
|
Alertmanager reload |
|
the Prometheus configuration is generated again, then reloaded |
|
the password is set in Grafana |
|
the data source is written, then Grafana restarts |
|
Grafana deletes the Thanos data source, then restarts |
|
the certificates are installed, then traefik reloads them |
|
traefik reloads its configuration |
|
traefik starts, restarts or is removed |
|
the unit’s drop-in is written or removed, then the unit restarts |
|
Docker’s proxy is written or removed, then Docker and the stack restart |
|
nothing; |
|
the stack stops and starts in the other mode; nothing when the mode stays the same |
|
the stack restarts |
Adding or removing alert_rules, alertmanager_config or targets.vector_search also restarts the stack.
Before it applies the changes, the watcher checks that every component takes connections on its port, the log collector, the Grafana renderer, the Thanos sidecar and query and traefik included when they run. When one refuses them, as when its container crashed, the stack restarts along with the changes, so the next file the watcher accepts brings the stack back. A component that is starting or busy takes connections, so it is not restarted for it.
A systemd entry maps a unit name to the content of a drop-in, written as
/etc/systemd/system/<unit>.d/50-scylla-monitoring.conf. The watcher owns only that file.
Keep secrets out of drop-ins: systemd shows a unit’s settings, Environment= included, to every user on the host.
For example, the node_exporter unit installed by the ScyllaDB images ends its command line with $NODE_EXPORTER_ARGS,
so a drop-in can set its arguments:
systemd:
node-exporter.service: |
[Service]
Environment=NODE_EXPORTER_ARGS=--web.listen-address=127.0.0.1:9100
With tls.enabled, traefik serves the stack on tls.listen_ip with mutual TLS.
Set stack.bind_address to 127.0.0.1 as well, so traefik is the only way in.
When the stack restarts, traefik starts at the same time as the other containers.
With stack.quick_startup, start-traefik.sh --quick-startup does not wait until traefik routes a request;
the watcher’s health check waits for traefik’s ping endpoint with the other components instead.
Its port 10911 forwards to tls.thanos_port: 10911, the Thanos sidecar, by default, or 10903, the Thanos query that
stack.local_thanos runs, so a central Thanos querier connecting to port 10911 reaches the local query.
Its port 1514 forwards syslog to the log collector, and it is the one port that asks for no client certificate,
so limit who can reach it with a firewall. A node that sends its logs through traefik must:
Send syslog over TLS and verify traefik’s certificate with
tls.ca_crt. Plain TCP, as in the rsyslog example of the installation guide, is rejected on this port.Set the hostname of the syslog message to its own IP address. The collector sees every connection coming from traefik, so it takes the
instancelabel from that hostname when it is an IP address. With any other hostname the logs of all the nodes share oneinstance.
Without traefik nothing changes: the nodes send plain syslog to port 1514 of the collector, and instance is the
address they connect from, unless the hostname of the message is an IP address.
The docker section sets the proxy the Docker daemon uses, for example to pull images on a host without direct
Internet access:
docker:
http_proxy: http://10.0.0.5:3128
https_proxy: http://10.0.0.5:3128
The values are URLs; an HTTP proxy is given with http:// for https_proxy too. A URL can hold the proxy’s
credentials, such as http://user:password@10.0.0.5:3128. The watcher writes them as HTTP_PROXY and
HTTPS_PROXY in /etc/systemd/system/docker.service.d/50-scylla-monitoring.env, readable by root only, and adds the
drop-in 50-scylla-monitoring.conf next to it, which reads that file with EnvironmentFile=, so systemd does not
show the values to other users. Both files stay in place across reboots and upgrades, and the watcher restarts Docker
only when they change, or when an earlier restart did not happen. Restarting Docker also restarts a watcher that runs with PartOf=docker.service;
the watcher then applies the file again and completes the apply. Removing the section removes both files.
The Status File¶
After each apply, the watcher writes status.json in the state directory, and scylla-monitoring.status.json
next to each config file:
{"generation": "9f2c", "state": "applied", "time": "2026-10-20T10:00:00Z"}
{"generation": "a1b3", "state": "rejected", "error": "alert_rules: ...", "time": "2026-10-20T10:05:00Z"}
applied: the stack runs the configuration ofgeneration.rejected: the file is invalid; the stack still runsapplied_generation.failed: the apply failed; the stack is left as the failed apply left it, for inspection. The next apply runs every action, as a first apply does. An internal error of the watcher reports the generation of the file being applied when it is known, and a null generation otherwise.reverted: the apply failed and, with--auto-revert, the previous configuration was applied again. Docker’s proxy stays as the failed apply set it, so the revert runs without restarting Docker; the next apply sets the proxy again and restarts the stack, even when it writes the file that is already applied.
Error messages name the section and position, without the values, since the file holds secrets.
Note
start-all.sh and the start scripts still read env.sh from the monitoring directory, and what it sets can override the file.
The watcher logs a warning when env.sh exists.