# index.md

# ScyllaDB Monitoring Stack

ScyllaDB Monitoring Stack is a full stack for ScyllaDB monitoring and alerting. The stack contains open source tools including Prometheus and Grafana, as well as custom ScyllaDB dashboards and tooling.

![image](monitor.png)

The ScyllaDB Monitoring Stack consists of three components, wrapped in Docker containers:

* prometheus - collects and stores metrics
* alertmanager - handles alerts
* grafana - dashboard server

**Choose a topic to get started**:

* [User Guide](https://monitoring.docs.scylladb.com/stable/use-monitoring/index.md)
* [Download and Install](https://monitoring.docs.scylladb.com/stable/install/index.md)
* [Procedures](https://monitoring.docs.scylladb.com/stable/procedures/index.md)
* [Troubleshooting](https://monitoring.docs.scylladb.com/stable/troubleshooting/index.md)
* [Reference](https://monitoring.docs.scylladb.com/stable/reference/index.md)
* [Upgrade](https://monitoring.docs.scylladb.com/stable/upgrade/index.md)
* [ScyllaDB Monitoring Stack lesson](https://university.scylladb.com/courses/scylla-operations/lessons/scylla-monitoring/) on ScyllaDB University
* [ScyllaDB Monitoring Stack GitHub Project](https://github.com/scylladb/scylla-monitoring/)


# index.md

# ScyllaDB Monitoring Stack Procedures

There are several reference guides available which give additional information. Choose a topic to begin:

* [Cloud Users Datadog integration](https://monitoring.docs.scylladb.com/stable/procedures/datadog/cloud-integration.md)
* [Datadog Integration](https://monitoring.docs.scylladb.com/stable/procedures/datadog/index.md)
* [Alert Manager](https://monitoring.docs.scylladb.com/stable/procedures/alerts/index.md)
* [Adding and Modifying Dashboards](https://monitoring.docs.scylladb.com/stable/procedures/updating-dashboard.md)
* [Upgrade Guides](https://monitoring.docs.scylladb.com/stable/upgrade/index.md)


# index.md

# Upgrade Scylla Monitoring Stack

<div class="panel callout radius animated">
         <div class="row">
           <div class="medium-3 columns">
             <h5 id="getting-started">Upgrade Scylla Open Source</h5>
           </div>
           <div class="medium-9 columns">

Procedures for upgrading the Scylla Monitoring Stack.

* [Upgrade Guide - Monitoring 4.x to 4.y](https://monitoring.docs.scylladb.com/stable/upgrade/upgrade-guide-from-monitoring-4.x-to-monitoring-4.y.md)
* [Upgrade Guide - Monitoring 3.x to 4.y](https://monitoring.docs.scylladb.com/stable/upgrade/upgrade-guide-from-monitoring-3.x-to-monitoring-4.y.md)
* [Upgrade Guide - Monitoring 3.x to 3.y](https://monitoring.docs.scylladb.com/stable/upgrade/upgrade-guide-from-monitoring-3.x-to-monitoring-3.y.md)
* [Upgrade Guide - Monitoring 2.x to 3.y](https://monitoring.docs.scylladb.com/stable/upgrade/upgrade-guide-from-monitoring-2.x-to-monitoring-3.y.md)
* [Upgrade Guide - Monitoring 2.x to 2.y](https://monitoring.docs.scylladb.com/stable/upgrade/upgrade-guide-from-monitoring-2.x-to-monitoring-2.y.md)
* [Upgrade Guide - Monitoring 1.x to 2.x](https://monitoring.docs.scylladb.com/stable/upgrade/upgrade-guide-from-monitoring-1.x-to-monitoring-2.x.md)

</div>
</div>
</div>


# index.md

# Download and Install Scylla Monitoring Stack

Scylla Monitoring Stack is a full stack for Scylla monitoring and alerting. The stack contains open source tools including Prometheus and Grafana, as well as custom Scylla dashboards and tooling.

Choose a topic to begin:

* [Install Scylla Monitoring Stack](https://monitoring.docs.scylladb.com/stable/install/monitoring-stack.md)
* [The start-all.sh script](https://monitoring.docs.scylladb.com/stable/install/start-all.md)
* [Deploy Scylla Monitoring Stack Without Docker](https://monitoring.docs.scylladb.com/stable/install/monitor-without-docker.md)
* [Running using Docker Compose](https://monitoring.docs.scylladb.com/stable/install/docker-compose.md)
* [Using Thanos](https://monitoring.docs.scylladb.com/stable/install/thanos.md)


# index.md

# Troubleshooting Guide for Scylla Monitoring Stack

Choose a topic to begin:

* [Troubleshooting](https://monitoring.docs.scylladb.com/stable/troubleshooting/monitor-troubleshoot.md)
* [Troubleshooting Guide for Scylla Manager and Scylla Monitor Integration](https://docs.scylladb.com/troubleshooting/manager_monitoring_integration/)
* You can ask questions and discuss the ScyllaDB Monitoring Stack in the [ScyllaDB community forum](https://forum.scylladb.com/)  and on the [Slack channel](http://slack.scylladb.com/).


# index.md

# Scylla Monitoring Stack Reference Guide

There are several reference guides available which give additional information. Choose a topic to begin:

* [Support Matrix](https://monitoring.docs.scylladb.com/stable/reference/matrix.md)
* [Interfaces](https://monitoring.docs.scylladb.com/stable/reference/monitoring-apis.md)


# index.md

# Using Scylla Monitoring Stack

There are several dashboards you can use with Scylla Monitoring Stack. Choose a topic to begin:

* [CQL Optimization Dashboard](https://monitoring.docs.scylladb.com/stable/use-monitoring/cql-optimization.md)
* [Advisor](https://monitoring.docs.scylladb.com/stable/use-monitoring/advisor/index.md)


# index.md

# Scylla Monitoring Stack Alert Manager

Alerts are a Prometheus enhancement used to notify you that something is wrong in your system.
Choose a topic to begin:

* [Alerting](https://monitoring.docs.scylladb.com/stable/procedures/alerts/alerting.md)


# index.md

# Scylla Monitoring Datadog Integration

The safest way to use Datadog with Scylla is to load the metrics, using a  Datadog Agent, from a Prometheus server and not directly from  Scylla nodes.
The integration consists of:

1. Installing and configuring the Datadog Agent.
2. Add Datadog recording rules.
3. Loading Scylla dashboard to Datadog.
4. Optionally load Monitor (Alerts).

#### NOTE
Scylla Cloud users, Check the cloud users [specific guide](https://monitoring.docs.scylladb.com/stable/procedures/datadog/cloud-integration.md).

## Scylla Monitoring Datadog Integration Overview

A typical ScyllaDB cluster generates thousands of metrics, sometimes even tens of thousands.
The sheer number of metrics is too much for Datadog.

Instead of letting the Datadog agent scrap all metrics, the monitoring stack marks a small subset of metrics with a label and lets the Datadog agent scrap only those.
That labeling is done in two places: recording rules and the Prometheus relabel config.

Recording rules aggregate the metrics so that they will be reported per instance instead of per shard, and they mark the result with a label.

Prometheus relabel config marks metrics that are reported per instance, like disk and CPU.

## Install And configure the Datadog Agent

Start by following [Installation](https://docs.datadoghq.com/agent/) guide. The datadog agent should run on a machine that can reach the Prometheus server.

Once the Datadog agent is working, download the configuration file and place it under /etc/datadog-agent/conf.d/prometheus.d/conf.yaml
Download the configuration file [`conf.yaml`](https://monitoring.docs.scylladb.com/stable/_downloads/c0d96642845dab62e24773bd681249f8/conf.yaml) and replace the ip address of the Prometheus server.

### Post configuration

Restart the agent based on your installation. Scylla metrics should be visible in Datadog.

#### NOTE
By default, Datadog will not scrap per-shard metrics. To enable per-shard metrics, edit the conf.yaml file and replace dd=~”1” with dd=~”1|2”

## Add datadog recording rules

Download the rules configuration file [`datadog.rules.yml`](https://monitoring.docs.scylladb.com/stable/_downloads/0d91ecdf045d215637d56f0b81f5603a/datadog.rules.yml) if you need per-shard metrics, download [`datadog.rules-with-shards.yml`](https://monitoring.docs.scylladb.com/stable/_downloads/a7125545ebbccbf5ec65f35e45219c1b/datadog.rules-with-shards.yml) and place it under prometheus/prom_rules/.
Per-shards metrics adds load and cost to both the Prometheus server and Datadog agent and server, so only use it if needed.

## Upload the Dashboard

Download the dashboard file [`dashboard.json`](https://monitoring.docs.scylladb.com/stable/_downloads/8e23d84763f852c22d1a8bca59d8f420/dashboard.json).
Create a new dashboard in Datadog and import the json file you downloaded.

## Using the Dashboard

We created a Datadog dashboard that resembles the Grafana dashboards available to Scylla and Scylla-Cloud users.

![image](procedures/datadog/datadog.png)

The dashboard contains some specific filtering and perspectives:
First, you can choose between shard, instance, dc, or cluster view.
This will aggregate the metrics in the graphs accordingly.
Second, you can filter to see specific shards, nodes, or DCs.

#### NOTE
Pay attention that some of the combinations are conflicting. For example, you cannot filter by DC when looking at a cluster view. If no data displayed, remove the filters first.

## Adding Monitor

Alerts in Datadog called Monitor. Download the monitor file [`monitor.json`](https://monitoring.docs.scylladb.com/stable/_downloads/1d0fbd862a69c00f6c66cbfbb459380a/monitor.json). Go to the Monitor section in datadog and import the json.

## Adding more metrics to Datadog

To add a missing metric, do the following:

For ScyllaDB metrics, add a recording rule to report an aggregated per instance of that metric. We suggest placing it in a different YAML file to simplify an upgrade.

OS-related metrics originate from node_exporter, and Prometheus labels them during scraping.
To label a node exporter metrics, edit the prometheus/prometheus.yml.template search for the job_name: node_exporter and find the metric_relabel_configs.
You can add a metric name to the regex section.

If you think that metric is helpful for other users, open an issue, and we’ll add it.


# index.md

# Scylla Monitoring Stack Advisor

The Scylla Monitoring Stack Advisor is an element of the Scylla Monitoring Stack that recognizes bad practices, bad configurations, and potential problems and advises on how to solve them.

For example, the Advisor could warn about using large cells. Large cells in the data usually indicate a problem with the data model or a problem with the client code, and can impact system performance.

Each Advisor issue is explained in detail:

* [Some queries use ALLOW FILTERING](https://monitoring.docs.scylladb.com/stable/use-monitoring/advisor/cqlAllowFiltering.md)
* [Some queries use Consistency Level: ALL](https://monitoring.docs.scylladb.com/stable/use-monitoring/advisor/cqlCLAll.md)
* [Some queries use Consistency Level: ANY](https://monitoring.docs.scylladb.com/stable/use-monitoring/advisor/cqlCLAny.md)
* [Some queries are not token-aware](https://monitoring.docs.scylladb.com/stable/use-monitoring/advisor/cqlNoTokenAware.md)
* [Some SELECT queries are non-paged](https://monitoring.docs.scylladb.com/stable/use-monitoring/advisor/cqlNonPaged.md)
* [Some queries are non-prepared](https://monitoring.docs.scylladb.com/stable/use-monitoring/advisor/cqlNonPrepared.md)
* [Some operation failed due to unsatisfied consistency level](https://monitoring.docs.scylladb.com/stable/use-monitoring/advisor/nodeCLErrors.md)
* [I/O Errors can indicate a node with a faulty disk](https://monitoring.docs.scylladb.com/stable/use-monitoring/advisor/nodeIOErrors.md)
* [Some operations failed on the replica side](https://monitoring.docs.scylladb.com/stable/use-monitoring/advisor/nodeLocalErrors.md)
* [CQL queries are not balanced among shards](https://monitoring.docs.scylladb.com/stable/use-monitoring/advisor/nonBalancedcqlTraffic.md)
* [Prepared statements cache eviction](https://monitoring.docs.scylladb.com/stable/use-monitoring/advisor/preparedCacheEviction.md)


# alerting.md

# Alerting

Alerts are a Prometheus enhancement to notify that something is wrong in the system.

You can read more about [Prometheus alerting here](https://prometheus.io/docs/alerting/overview/)

By default, alerts are shown in the overview dashboard, but you can also configure the alerts to be sent to
other external systems like email or slack.

In general, alerts are generated by Prometheus when a specific condition is met within a certain period of time,
they are then sent to the Alertmanager, that can send them to external systems.

The Alertmanager also acts as a data source for Grafana so the active alerts are shown in the dashboard.

![image](procedures/alerts/monitoring_stack.png)

## Prometheus Alerts

The Prometheus alerts are defined in files placed under prometheus/prom_rules/. Prometheus will load any file that ends with yml or yaml.

Each Prometheus alert consists of:

- Name
- What happened
- For how long
- What to report

For example, let us look at the InstanceDown alert that comes by default as part of Scylla-Monitoring.

```yaml
- alert: InstanceDown
  expr: up == 0
  for: 60s
  labels:
    severity: "error"
  annotations:
    description: '{{ $labels.instance }} has been down for more than 30 seconds.'
    summary: Instance {{ $labels.instance }} down
```

The **expr** is a legal Prometheus expression, in this case, the **up** metric is equal to **0**.

The **for** set the minimal duration before the alert will be active. Prior to that, the alert will be in pending-mode and will not be sent.

The **labels** part holds additional labels that will be added to the alert, in the example, the **severity** label will be set to **error**.

**annotations** are special labels that add a description to the alert and can be used in emails and notification when the alert is forward to external systems.

### Alerts severity

Prometheus uses the following severities from low to high: **info**, **warn**, **error**, **critical**.

The Alertmanager inhibition rule (see the Alertmanager section) makes
sure that an alert with higher priority will mute the lower priority rule. For example, out-of-disk space alerts have different thresholds,
this way when the **error** alert for disk-full is firing, it silences the **warn** alert.

When adding your own alerts, you can use the severities to signal the action’s importance, for example **critical** would be a pager-duty, while an **error** will be an email.

### Adding Alerts

Add your own alerts in a separate file with a **yml** or **yaml** extension. Making it easier during upgrade.

### Modifying Alerts

It is a common practice to update the alerts that comes with the monitoring stack. Verify that the existing alerts suitable to your needs.
When modifying an alert, remember that it will be overridden on you next upgrade.

## Alertmanager

The [Alertmanager](https://prometheus.io/docs/alerting/alertmanager/) gets the alerts from the Prometheus server responsible for duplicating, grouping and routing.

The Alertmanager configuration is found in rule_config.yml file that is located in the prometheus directory.
The configuration consists of three parts:

- Routes: Represent a routing tree, the most specific rule, wins
- Inhibition: Mute an alert, based on another alert
- Receiver: Send a notification via email, sluck, etc’

Check the Alertmanager documentation for details on how to specify a specific receiver.

In the default configuration, the Grafana server will use the Alertmanager as a data source for the alert table found in the **overview** dashboard.

## Tips When Adding an Alert

**It’s worse having an alert that does not work, than not having an alert at all**

There are multiple ways you can use to make sure your alert is set correctly:

- Start with making sure that the full path (alert to mail or dashboard) is working a simple way is by negating the logic.
- Force the alert to be active by reducing the limits or the duration.
- When possible simulate the actual scenario and see that the alert works as expected.

For example, low available disk space.

Assume that you want an alert when you have lower than 10% disk space.
You can start by setting the alert that is fired when you have more than 10% disk space.
That would show that the alert is active and is propagated to the dashboard and to any external system you are using.
You can then set the alert rule to a value similar to the current situation.

Now you are sure that when setting the expression to alert when the free space below 10% the alert would work, but if possible,
simulate that situation, in this case by creating some big files.

Starting with a simple working expression helps you bisect problems.


# cloud-integration.md

# ScyllaDB Cloud Monitoring Datadog Integration

To allow external server scrapping, you will need to enable the [Prometheus proxy](https://cloud.docs.scylladb.com/stable/monitoring/cloud-prom-proxy.html).
The Datadog agent reads from the proxy, which reads from the Promethues server.

1. Installing and configuring the Datadog Agent.
2. Add Datadog recording rules.
3. Loading ScyllaDB dashboard to Datadog.
4. Optionally load Monitor (Alerts).

## ScyllaDB Monitoring Datadog Integration Overview

A typical ScyllaDB cluster generates thousands of metrics, sometimes even tens of thousands.
The sheer number of metrics is too much for Datadog.

Instead of letting the Datadog agent scrap all metrics, the monitoring stack marks a small subset of metrics with a label and lets the Datadog agent scrap only those.

## Install And configure the Datadog Agent

Start by installing DataDog Agent on a server with access to ScyllaDB Cloud Prometheus Proxy server, following [Installation](https://docs.datadoghq.com/agent/) guide.
The datadog agent should run on a machine that can reach the Prometheus Proxy server.

Once the Datadog agent is working, download the configuration file [`conf.yaml`](https://monitoring.docs.scylladb.com/stable/_downloads/55110650a2d5181badc62c6c37f68075/cloud-conf.yaml) move it to: /etc/datadog-agent/conf.d/prometheus.d/conf.yaml

Edit the file. You must replace the cluster id (CLUSTER_ID) and the token (TOKEN).

### Post configuration

Restart the agent based on your installation. ScyllaDB metrics should be visible in Datadog.

#### NOTE
By default, Datadog will not scrap per-shard metrics. To enable per-shard metrics, edit the conf.yaml file and replace dd=~”1” with dd=~”1|2”

## Upload the Dashboard

Download the dashboard file [`dashboard.json`](https://monitoring.docs.scylladb.com/stable/_downloads/8e23d84763f852c22d1a8bca59d8f420/dashboard.json).
Create a new dashboard in Datadog and import the json file you downloaded.

## Using the Dashboard

We created a Datadog dashboard that resembles the Grafana dashboards.

![image](procedures/datadog/datadog.png)

The dashboard contains some specific filtering and perspectives:
First, you can choose between shard, instance, dc, or cluster view.
This will aggregate the metrics in the graphs accordingly.
Second, you can filter to see specific shards, nodes, or DCs.

#### NOTE
Pay attention that some of the combinations are conflicting. For example, you cannot filter by DC when looking at a cluster view. If no data displayed, remove the filters first.

## Adding Monitor

Alerts in Datadog called Monitor. Download the monitor file [`monitor.json`](https://monitoring.docs.scylladb.com/stable/_downloads/1d0fbd862a69c00f6c66cbfbb459380a/monitor.json). Go to the Monitor section in datadog and import the json.


# cql-optimization.md

# The CQL Optimization

The CQL Optimization is part of the CQL Dashboard and is a tool to help identify potential issues with queries, data model and driver.

![image](use-monitoring/cql_optimization_master.png)

The upper part of the dashboard holds CQL related metrics.

The lower parts holds gauges and graphs. When inspecting the system, we like the gauge to be near zero and the graphs as low as possible.

#### NOTE
Besides your queries, there are queries generated by the cql driver and internal queries to the system tables which can be misleading when testing with low traffic.

The following sections describe each of the dashboard’s panel

## Prepared Statements

[Prepared statements](https://docs.scylladb.com/getting-started/definitions/#prepared-statements) are queries that are first defined as a template with place holders for the values and then that template is used
multiple times with different values.

Using prepared statements has the following benefits:

* The database only needs to parse the query once
* The driver can route the query to the right node
* Using place-holders and values is safer and prevents CQL-Injection

The **CQL Non-Prepared Queries** Gauge shows the percentage of queries that are not prepared.

The **CQL Non-Prepared Queries** Graph shows the rate of the queries. Make sure both are low.

## Token Aware

Scylla is a distributed database, where each node contains only part of the data, specifically a range of the token ring.
Ideally, a query would reach the node that holds the data (one of the replicas), failing to do so would mean the coordinator
needs to send the query internally to a replica. This results in higher latency and increased resource usage.

Typically, your driver would know how to route the queries to a replication node, but using non-prepared statements, non-token-aware driver
or load-balance can cause the query to reach a node that is not a replica.

The **Non-Token Aware** Gauge shows the percentage of queries that reached a node that does not hold that data (a node that is not a replica-node).

The **Non-Token Aware Queries** Graph shows the rate of the queries that did not reach a replica-node, make sure both are low.

## Paged Queries

By default, read queries are paged, this means that Scylla will break the results into multiple chunks limiting the reply size.
Non-Paged queries require all results be returned in one result increasing the overall load of the system and clients and should be avoided.

The **Non-Paged CQL Reads** Gauge shows the percentage of non-paged read queries that did not use paging.

The **Non-Paged CQL Reads** Graph shows the rate of the non-paged queries, make sure both are low.

## Reversed CQL Reads

ScyllaDB supports compound primary keys with a clustering column. This kind of primary keys allows an efficient way
to return sorted results that are sorted by the clustering column.

We strongly recommend using use the correct ordering.
Querying with an order different than the order the `CLUSTERING ORDER BY` was defined is supported, but not advisable.

For example, look at the following table:

```shell
CREATE TABLE ks1.table_demo (
   category text,
   type int,
   PRIMARY KEY (category, type))
WITH CLUSTERING ORDER BY (type DESC);
```

The following query uses reverse order:

```shell
select * from ks1.table_demo where category='cat1' order by type ASC;
```

The **Reversed CQL Reads** Gauge shows the percentage of read queries that use `ORDER BY` that is different than the `CLUSTERING ORDER BY`.

The **Reversed CQL Reads** Graph shows the rate of the read queries that use `ORDER BY` that is different than the `CLUSTERING ORDER BY`, make sure both are low.

## ALLOW FILTERING

Scylla supports server side data filtering that is not based on the primary key. This means Scylla would read data and then filter and
return part of it to the user. Data that is read and then filtered is an overhead to the system.

These kinds of queries can create a big load on the system, and should be used with care.

The CQL optimization dashboard, checks for two things related to queries that use `ALLOW FILTERING`: how many such queries exist; and how much of the data that was read was
dropped before returning to the client.

The **ALLOW FILTERING CQL Reads** Gauge shows the percentage of read queries that use `ALLOW FILTERING`.

The **ALLOW FILTERING CQL Reads** Graph shows the rate of the read queries that use `ALLOW FILTERING`, make sure both are low.

The **ALLOW FILTERING Filtered Rows** Gauge shows the percentage of rows that were read and then filtered, this is an indication of the additional overhead to the system.

The **ALLOW FILTERING Filtered Rows** Graph shows multiple graphs: the rows that were read, the rows that matched and the rows that were dropped. Rows that
were dropped are an additional overhead to the system.

## Consistency Level

Typically data in Scylla is duplicated into multiple replicas for availability reasons. A coordinator node would get the request and will send it
to the nodes holding the replicas.

The query Consistency Level determines how many replies from the replicas are required before the coordinator replies to the client.
The most common case is to use QUORUM, which means that when the coordinator gets a majority of the replies from the replicas, it will return success to the client.

Two consistency levels hold a potential problem and should be used with care `ANY` and `ALL`.

The **CQL ANY Queries** Gauge shows the percentage of queries that use Consistency Level `ANY`. Using consistency level ANY in a query may hurt persistency, if the node receiving the request fails, the data may be lost.

The **CQL ANY CL Queries** Graph shows the rate of the queries that use Consistency Level `ANY`, make sure both are low.

The **CQL ALL CL Queries** Gauge shows the percentage of queries that use Consistency Level `ALL`. Using consistency level ALL in a query may hurt availability, if a node is unavailable the operations will fail.

The **CQL ALL CL Queries** Graph shows the rate of the queries that use Consistency Level `ALL`, make sure both are low.

## Cross DC

Cross DC traffic is usually more expensive in terms of latencies and cost.
This metric reports on such traffic in situations were it could be avoided.

## Cross DC Consistency Level

Using consistency level QUORUM or consistency level ONE in a query when there is more than one DC may hurt performance,
as queries may end in the non-local DC. Use LOCAL_QUORUM and LOCAL_ONE instead.

## Cross DC read requests

#### NOTE
The CQL Optimization Dashboard relies on the definition of nodes per Data Center in the Scylla Monitoring Stack (prometheus/scylla_servers.yml) to match the Data Center names used in Scylla Cluster.
If this is not the case, you will see the wrong result.

In a typical situation, a client performs a read from the nearest data-center and that query is performed local to the data-center.
A read request that ends up causing traffic between data-centers adds additional overhead to the system.

The **Cross DC read requests** Gauge shows the percentage of read queries that caused a request to an external data-center, make sure it is low or zero.


# cqlAllowFiltering.md

# Some queries use ALLOW FILTERING

Scylla supports server-side data filtering that is not based on the primary key. This means Scylla would execute a *full scan* on the table: read **all** of the table’s data from disk, and then filter and return part of it to the user.  More information on [ALLOW FILTERING](https://docs.scylladb.com/getting-started/dml/#allowing-filtering).

These kinds of queries can create a bigger load on Scylla, and should be used with care.


# cqlCLAll.md

# Some queries use Consistency Level: ALL

Scylla stores your data in ReplicationFactor of nodes (replicas). Typically, for a consistency level, each piece of information is stored in multiple replicas. A query’s Consistency Level determines how many replicas will need to be queried before a reply is returned.

Using consistency level ALL in a query requires **all** replicas to be available and will fail if a node is unavailable, resulting in reduced availability. This means that the client will not get a result in case **one** of the replicas is down or not responding, reducing the HA of the system.

Consistency level ALL should be used with care and should be accompanied with deep understanding and fall back mechanisms for node unavailability.

## Link to Scylla university

[Lesson on Consistency](https://university.scylladb.com/courses/scylla-essentials-overview/lessons/high-availability/topic/consistency-level/#:~:text=ALL%20%E2%80%93%20A%20write%20must%20be%20written%20to%20all%20replicas%20in%20the%20cluster%2C%20a%20read%20waits%20for%20a%20response%20from%20all%20replicas.%20Provides%20the%20lowest%20availability%20with%20the%20highest%20consistency.)


# cqlCLAny.md

# Some queries use Consistency Level: ANY

Scylla stores your data in ReplicationFactor of nodes (replicas). A write query Consistency Level determines how many replicas need to acknowledge the write before a reply is returned.

Using consistency level ANY allows a write query to be acknowledged after storing the information on the coordinator, a non-replica (as hints), meaning the data will be acknowledged when in fact it is not yet persistent on disk. Use Consistency Level ANY with care.

## Link to Scylla university

[Lesson on Consistency](https://university.scylladb.com/courses/scylla-essentials-overview/lessons/high-availability/topic/consistency-level/#:~:text=Some%20of%20the%20most%20common,availability%20with%20the%20lowest%20consistency.)


# cqlNoTokenAware.md

# Some queries are not token-aware

Scylla is a distributed database, with each node containing only part of the data. Ideally, a query would reach the node that holds the data (one of the replicas). Failing to do so would mean the coordinator will need to send the query internally to a replica, resulting in higher latency and more resource usage.

Typically, your driver would know how to route the queries to a replication node, but if you’re using non-prepared statements, a non-token-aware load-balance policy can cause the query to reach a node that is not a replica.

## University link

[https://university.scylladb.com/courses/using-scylla-drivers/lessons/intro-and-recap-token-ring-architecture/](https://university.scylladb.com/courses/using-scylla-drivers/lessons/intro-and-recap-token-ring-architecture/)


# cqlNonPaged.md

# Some SELECT queries are non-paged

By default, read queries are paged, this means that Scylla breaks the results into multiple chunks (pages) limiting the reply size. Non-Paged queries require all results to be returned in one reply increasing the overall load on Scylla. Thus, drivers and clients should avoid sending these queries.

## Blog-post Links

[https://www.scylladb.com/2018/07/13/efficient-query-paging/](https://www.scylladb.com/2018/07/13/efficient-query-paging/)


# cqlNonPrepared.md

# Some queries are non-prepared

[Prepared Statements](https://docs.scylladb.com/getting-started/definitions/#prepared-statements) are an optimization that allows parsing a query only once and executing it multiple times with different concrete values.
As a rule of thumb, you should always favor prepared statements.


# docker-compose.md

# Using Docker Compose

Scylla-Monitoring Stack is container-based.
You can start and stop the Scylla Monitoring Stack with the start-all.sh and kill-all.sh scripts.

Docker Compose is an alternative method to start and stop the stack. It requires more manual steps, but once
configured, it simplifies starting and stopping the stack.

#### WARNING
*docker-compose up* **and** *start-all.sh* are two **alternative** ways to launch Scylla Monitoring Stack.
You should use **one** method, **not both**. In particular,  creating and updating *docker-compose.yml* is ignored
when using *start_all.sh*

#### NOTE
There is an experimental method that uses scripts to create the Docker Compose file. See the details below.

## Prerequisite

Make sure you have docker and docker-compose installed.

## Experimental - Configure docker-compose file using scripts

The `make-compose.sh` script accepts the same command-line arguments as `start-all.sh`. It generates a `docker-compose.yaml` file and a `.env` file.

Once the script is run, you can use `docker-compose up` to start the monitoring stack.
You can manually modify the generated Compose file, but be aware that running `make-compose.sh` script again, will overwrite it.

There is also an experimental `--compose` command-line option for `start-all.sh`. It will first, generates the Compose file
and then uses docker-compose to run the monitoring stack.

## Setting Prometheus

The Prometheus configuration file contains among others the IP address of the *alertmanager* and either the location
of the *scylla_server.yml* file or a Consul IP address of Scylla Manager, if Scylla Manager is used for server provisioning.

You can use ./prometheus-config.sh to generate the file, for example:

```shell
./prometheus-config.sh --compose
```

For production systems, It is advised to use an external directory for the Prometheus database. Make sure to create one and
update the docker-compose file accordingly (see the docker-compose example below).

## Setting Grafana Provisioning

Grafana reads its provisioning configuration from files, one for the data-source and one for the dashboards.
Note that the latter tells Grafana where the dashboards are located, it is not the dashboards themselves, which are
at a different location.

By default, Grafana iframe embedding is disabled. The generated `docker-compose.yml` and `.env` files
include `GF_SECURITY_ALLOW_EMBEDDING`, `GF_SECURITY_COOKIE_SECURE`, and `GF_SECURITY_COOKIE_SAMESITE`.
Use `--allow-embedding` or set `GF_SECURITY_ALLOW_EMBEDDING=true` in `env.sh` to enable embedding.

### Grafana Data-Source file

Run the following command to update the datasource:

```shell
./grafana-datasource.sh --compose
```

You can see the generated file under: grafana/provisioning/datasources/datasource.yaml

### Grafana Dashboard Load file

To set the dashboard load file, you can run the ./generate-sashboards.sh with the -t command line flag and the -v flag to specify the version.
For example, Scylla-enterprise version 2020.1:

```shell
./generate-dashboards.sh -t -v 2020.1
```

This command generates the files under: grafana/provisioning/dashboards/

## Docker Compose file

You can use the following example as a base for your docker compose.

Pass the following to a file called docker-compose.yml

```ruby
services:
  alertmanager:
    container_name: aalert
    image: prom/alertmanager:v0.34.1
    ports:
    - 9093:9093
    volumes:
    - ./prometheus/rule_config.yml:/etc/alertmanager/config.yml
  grafana:
    container_name: agraf
    environment:
    - GF_PANELS_DISABLE_SANITIZE_HTML=true
    - GF_PATHS_PROVISIONING=/var/lib/grafana/provisioning
    - GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS=scylladb-scylla-datasource
    # This is where you set Grafana security
    - GF_AUTH_BASIC_ENABLED=false
    - GF_AUTH_ANONYMOUS_ENABLED=true
    - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    - GF_SECURITY_ADMIN_PASSWORD=admin
    # To set your home dashboard uncomment the following line, set VERSION to be your current version
    #- GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH=/var/lib/grafana/dashboards/ver_VERSION/scylla-overview.VERSION.json
    image: grafana/grafana:13.2.2
    ports:
    - 3000:3000
    user: 1000:1000
    volumes:
    - ./grafana/build:/var/lib/grafana/dashboards
    - ./grafana/plugins:/var/lib/grafana/plugins
    - ./grafana/provisioning:/var/lib/grafana/provisioning
    # Uncomment the following line for grafana persistency
    # - path/to/grafana/dir:/var/lib/grafana
  loki:
    command:
    - --config.file=/mnt/config/loki-config.yaml
    container_name: loki
    image: grafana/loki:3.7.8
    ports:
    - 3100:3100
    volumes:
    - ./loki/rules:/etc/loki/rules
    - ./loki/conf:/mnt/config
  promotheus:
    command:
    - --config.file=/etc/prometheus/prometheus.yml
    container_name: aprom
    image: prom/prometheus:v3.14.0
    ports:
    - 9090:9090
    volumes:
    - ./prometheus/build/prometheus.yml:/etc/prometheus/prometheus.yml
    - ./prometheus/prom_rules/:/etc/prometheus/prom_rules/
    # instead of the following three targets, you can place three files under one directory and mount that directory
    # If you do, uncomment the following line and delete the three lines afterwards
    #- /path/to/targets:/etc/scylla.d/prometheus/targets/
    - ./prometheus/scylla_servers.yml:/etc/scylla.d/prometheus/targets/scylla_servers.yml
    - ./prometheus/scylla_manager_servers.yml:/etc/scylla.d/prometheus/targets/scylla_manager_servers.yml
    - ./prometheus/scylla_servers.yml:/etc/scylla.d/prometheus/targets/node_exporter_servers.yml
    
    # Uncomment the following line for prometheus persistency 
    # - path/to/data/dir:/prometheus/data
  promtail:
    command:
    - --config.file=/etc/promtail/config.yml
    container_name: promtail
    image: grafana/promtail:3.7.8
    ports:
    - 1514:1514
    - 9080:9080
    volumes:
    - ./loki/promtail/promtail_config.compose.yml:/etc/promtail/config.yml
version: '3'
```

### Start and Stop

To start the Scylla Monitoring Stack run `docker-compose up` and to stop run `docker-compose down`.


# heavyCompaction.md

# Compaction takes lots of memory and CPU

ScyllaDB runs compaction periodically as a background process. While running compaction is important, there are situations when
compaction takes too much CPU.
As a result, compaction impacts the overall system performance.

If this is the case, you can do one of the following:

* Statically limit the compaction shares with the `compaction_static_shares` option by setting a value between 50 and 1000:
  > * In the `scylla.yml` configuration file: `compaction_static_shares: 100`
  > * In the command line when starting ScyllaDB: `--compaction-static-shares 100`

  You may start by setting the value `100`. If read latency is impacted, which indicates that compaction is overly slowed down,
  you can increase the value to reach the balance between the system performance and read latency.
* Enforce `min_threshold` by setting `compaction_enforce_min_threshold: true` in the `scylla.yml` configuration file ([default is False](https://docs.scylladb.com/manual/stable/reference/configuration-parameters.html#confval-compaction_enforce_min_threshold)).
  As a result, ScyllaDB will compact only the buckets that contain the number of SSTables specified with `min_threshold`
  or more. See [STCS options](https://docs.scylladb.com/getting-started/compaction/#stcs-options) for details.


# intro.md

# ScyllaDB Monitoring Stack

ScyllaDB Monitoring Stack is a full stack for ScyllaDB monitoring and alerting. The stack contains open source tools including Prometheus and Grafana, as well as custom ScyllaDB dashboards and tooling.

![image](monitor.png)

The ScyllaDB Monitoring Stack consists of multiple components, wrapped in Docker containers:

* prometheus - Collects and stores metrics
* grafan-loki - Parses logs and generates metrics and alerts
* alertmanager - Handles alerts
* grafana - Dashboards server

A few optional components are used for additional services

* grafana-image-renderer - Allows you to download a dashboard as an image.
* Thanos sidecar - Allows a centralized Thanos server to read from the local Prometheus server.

## High Level Architecture

![image](monitoring_stack.png)

We use Prometheus for metrics collection and storage, and to generate alerts. Prometheus collects Scylla’s metrics from ScyllaDB and the
host metrics from the node_exporter agent that runs on the ScyllaDB server.

We use Loki for metrics and alerts generation based on logs, Loki gets the logs from rsyslog agents that run on each of the DB servers.

The alertmanager, receives alerts from Prometheus and Loki and distributes them to other systems like email and slack.

We use Grafana to display the dashboards. Grafana gets its data from Prometheus, the alertmanager and directly from ScyllaDB using CQL.

**Choose a topic to get started**:

* [User Guide](https://monitoring.docs.scylladb.com/stable/use-monitoring/index.md)
* [Download and Install](https://monitoring.docs.scylladb.com/stable/install/index.md)
* [Procedures](https://monitoring.docs.scylladb.com/stable/procedures/index.md)
* [Troubleshooting](https://monitoring.docs.scylladb.com/stable/troubleshooting/index.md)
* [Reference](https://monitoring.docs.scylladb.com/stable/reference/index.md)
* [ScyllaDB Monitoring Stack lesson](https://university.scylladb.com/courses/scylla-operations/lessons/scylla-monitoring/) on ScyllaDB University


# matrix.md

# Scylla Monitoring Stack Support Matrix

The following table shows which version of Scylla Monitoring Stack supports dashboards for which versions of Scylla and Scylla Manager.

| Scylla Monitoring Stack Version   | Scylla Open Source Version   | Scylla Enterprise Version              | Node_exporter[1] Version   | Scylla Manager Version   |
|-----------------------------------|------------------------------|----------------------------------------|----------------------------|--------------------------|
| 4.15                              |                              | 2024.x, 2025.x, 2026.1, 2026.2         | 1.7.0                      | 3.x                      |
| 4.15                              |                              | 2024.x, 2025.x, 2026.1, 2026.2         | 1.7.0                      | 3.x                      |
| 4.14                              |                              | 2024.x, 2025.x, 2026.1                 | 1.7.0                      | 3.x                      |
| 4.13                              |                              | 2024.x, 2025.x, 2026.1                 | 1.7.0                      | 3.x                      |
| 4.12                              |                              | 2024.x, 2025.1, 2025.2, 2025.3, 2025.4 | 1.7.0                      | 3.x                      |
| 4.11                              |                              | 2024.x, 2025.1, 2025.2, 2025.3         | 1.7.0                      | 3.x                      |
| 4.10                              |                              | 2024.1, 2024.2, 2025.1, 2025.2         | 1.7.0                      | 3.x                      |
| 4.9                               | 6.0, 6.1, 6.2                | 2022.2, 2023.1, 2024.1, 2024.2, 2025.1 | 1.7.0                      | 3.4                      |
| 4.8                               | 5.2, 5.4, 6.0, 6.1           | 2022.1, 2022.2, 2023.1, 2024.1, 2024.2 | 1.7.0                      | 3.3                      |
| 4.7                               | 5.2, 5.4, 6.0                | 2022.1, 2022.2, 2023.1, 2024.1         | 1.7.0                      | 3.2                      |
| 4.6                               | 5.2, 5.4                     | 2021.1, 2022.1, 2022.2, 2023.1, 2024.1 | 1.4.1                      | 3.0, 3.1, 3.2            |
| 4.5                               | 5.0,5.1,5.2,5.4              | 2020.1, 2021.1, 2022.1, 2022.2, 2023.1 | 1.4.1                      | 3.0, 3.1, 3.2            |
| 4.4.5                             | 5.0,5.1,5.2,5.4              | 2020.1, 2021.1, 2022.1, 2022.2, 2023.1 | 1.4.1                      | 3.0, 3.1, 3.2            |
| 4.4                               | 5.0,5.1,5.2                  | 2020.1, 2021.1, 2022.1, 2022.2, 2023.1 | 1.4.1                      | 3.0, 3.1                 |
| 4.3                               | 5.0,5.1,5.2                  | 2020.1, 2021.1, 2022.1, 2022.2         | 1.4.1                      | 2.5, 2.6, 3.0            |
| 4.2                               | 4.5,4.6,5.0,5.1              | 2020.1, 2021.1, 2022.1, 2022.2         | 1.4.1                      | 2.5, 2.6, 3.0            |
| 4.1                               | 4.5,4.6,5.0,5.1              | 2020.1, 2021.1, 2022.1, 2022.2         | 0.17                       | 2.5, 2.6, 3.0            |
| 4.0                               | 4.3, 4.4, 4.5, 4.6, 5.0      | 2020.1, 2021.1, 2022.1                 | 0.17                       | 2.3, 2.4, 2.5, 2.6, 3.0  |
| 3.11                              | 4.3, 4.4, 4.5, 4.6, 5.0      | 2020.1, 2021.1, 2022.2                 | 0.17                       | 2.3, 2.4, 2.5, 2.6       |
| 3.10                              | 4.3, 4.4, 4.5, 4.6           | 2020.1, 2021.1                         | 0.17                       | 2.3, 2.4, 2.5, 2.6       |
| 3.9                               | 4.3, 4.4, 4.5, 4.6           | 2020.1, 2021.1                         | 0.17                       | 2.3, 2.4, 2.5            |
| 3.8                               | 4.3, 4.4, 4.5                | 2020.1, 2021.1                         | 0.17                       | 2.3, 2.4, 2.5            |
| 3.7                               | 4.2, 4.3, 4.4                | 2019.1, 2020.1, 2021.1                 | 0.17                       | 2.2, 2.3                 |
| 3.6.3                             | 4.2, 4.3, 4.4                | 2019.1, 2020.1, 2021.1                 | 0.17                       | 2.2, 2.3, 2.1            |
| 3.6                               | 4.1, 4.2, 4.3                | 2019.1, 2020.1                         | 0.17                       | 2.1, 2.2, 2.3            |
| 3.6.1                             | 4.1, 4.2, 4.3                | 2019.1, 2020.1                         | 0.17                       | 2.1, 2.2                 |
| 3.6.1                             | 4.1, 4.2, 4.3, 4.4           | 2019.1, 2020.1                         | 0.17                       | 2.1, 2.2                 |
| 3.5                               | 3.3, 4.0, 4.1, 4.2           | 2019.1, 2020.1                         | 0.17                       | 2.0, 2.1, 2.2            |
| 3.4.3                             | 3.3, 4.0, 4.1, 4.2           | 2019.1, 2020.1                         | 0.17                       | 2.0, 2.1                 |
| 3.4                               | 3.3, 4.0, 4.1                | 2018.1, 2019.1                         | 0.17                       | 2.0, 2.1                 |
| 3.3                               | 3.1, 3.2, 3.3, 4.0           | 2018.1, 2019.1                         | 0.17                       | 1.4, 2.0, 2.1            |
| 3.2                               | 3.1, 3.2, 3.3                | 2018.1, 2019.1                         | 0.17                       | 1.4, 2.0                 |
| 3.1                               | 2.3, 3.0, 3.1, 3.2           | 2018.1, 2019.1                         | 0.17                       | 1.3, 1.4, 2.0            |
| 3.0                               | 2.3, 3.0, 3.1, 3.2           | 2018.1, 2019.1                         | 0.17                       | 1.3, 1.4                 |
| 2.4                               | 2.3, 3.0, 3.1                | 2018.1, 2019.1                         | 0.14, 0.17                 | 1.3, 1.4                 |
| 2.3                               | 2.3, 3.0                     | 2018.1, 2019.1                         | 0.14, 0.17                 | 1.3                      |
| 2.2                               | 2.3, 3.0                     | 2018.1, 2019.1                         | 0.14, 0.17                 | 1.3                      |
| 2.1                               | 2.3, 3.0                     | 2018.1                                 | 0.14, 0.17                 | 1.3                      |

[1] [Node_exporter](https://github.com/prometheus/node_exporter) is a monitoring agent running on each Scylla node and reports OS metrics to Prometheus


# min-prod-hw.md

# Minimal Production System Recommendations

* **CPU** - For clusters with up to 100 cores use 2vCPUS, for larger clusters 4vCPUs
* **Memory** - 15GB+ DRAM and proportional to the number of cores.
* **Disk** - persistent disk storage is proportional to the number of cores and Prometheus retention period (see the following section)
* **Network** - 1GbE/10GbE preferred

## Calculating Prometheus Minimal Disk Space requirement

Prometheus storage disk performance requirements: persistent block volume, for example an EC2 EBS volume

Prometheus storage disk volume requirement:  proportional to the number of metrics it holds and the default retention time. The default retention period is 15 days, and the disk requirement is around
3.5KB per series per day, assuming the default scraping interval of 20s.

For example, 100k series, with a retention time of 45 days, will need

```default
100k * 3.5KB * 45 ~ 15GB
```

To account for unexpected events, like replacing or adding nodes, we recommend allocating at least x2-3 the space, in this case, ~40GB.
Prometheus Storage disk does not have to be as fast as Scylla disk, and EC2 EBS, for example, is fast enough and provides HA out of the box.

## Calculating Prometheus Minimal Memory Space requirement

Prometheus uses more memory when querying over a longer duration (e.g. looking at a dashboard on a week view would take more memory than on an hourly duration).

For Prometheus alone, you should have 16KB of memory per series and it would use about 600MB of virtual memory per core.
Because Prometheus is so memory demanding, it is a good idea to add swap, so queries with a longer duration would not crash the server.

<script>
    //   series     = SERIES_CONST + SERIES_PER_NODE\*nodes + SERIES_PER_CORE\*cores + ...  (cores = total cores)
    //   rss_bytes  = RSS_CONST_BYTES + RSS_BYTES_PER_SERIES\*series
    //   disk_bytes = (DISK_CONST_BYTES_PER_DAY + DISK_BYTES_PER_SERIES_PER_DAY\*series) \* retention_days
    //                + WAL_CONST_BYTES + WAL_BYTES_PER_SERIES\*series
    const SERIES_CONST = 3786;
    const SERIES_PER_NODE = 1094;
    const SERIES_PER_CORE = 1185;
    const SERIES_PER_TABLE_PER_NODE = 17.86;
    const SERIES_PER_ACTIVE_TABLE_PER_NODE = 165.4;
    const SERIES_PER_SL_PER_CORE = 96.4;
    const SERIES_PER_ALTERNATOR_TABLE_PER_NODE = 20.9;
    const RSS_CONST_BYTES = 0;
    const RSS_BYTES_PER_SERIES = 16031;
    const DISK_CONST_BYTES_PER_DAY = 2096305;
    const DISK_BYTES_PER_SERIES_PER_DAY = 3574;
    const WAL_CONST_BYTES = 47523198;
    const WAL_BYTES_PER_SERIES = 2513;

    function formatBytes(bytes) {
        const units = ["B", "KB", "MB", "GB", "TB"];
        let i = 0;
        while (bytes >= 1024 && i < units.length - 1) {
            bytes /= 1024;
            i++;
        }
        return bytes.toFixed(i > 2 ? 2 : 0) + units[i];
    }

    function myFunction() {
        const value = (id) => parseInt(document.getElementById(id).value) || 0;
        const nodes = value('hosts');
        const cores = nodes \* value('shards');
        const tables = value('tables');
        const active = Math.min(value('active_tables'), tables);
        const sl = value('service_levels');
        const alternator = value('alternator_tables');
        const retention = value('retention');

        const series = SERIES_CONST + SERIES_PER_NODE \* nodes + SERIES_PER_CORE \* cores
            + SERIES_PER_TABLE_PER_NODE \* tables \* nodes + SERIES_PER_ACTIVE_TABLE_PER_NODE \* active \* nodes
            + SERIES_PER_SL_PER_CORE \* sl \* cores + SERIES_PER_ALTERNATOR_TABLE_PER_NODE \* alternator \* nodes;
        const memory = RSS_CONST_BYTES + RSS_BYTES_PER_SERIES \* series;
        const disk = (DISK_CONST_BYTES_PER_DAY + DISK_BYTES_PER_SERIES_PER_DAY \* series) \* retention
            + WAL_CONST_BYTES + WAL_BYTES_PER_SERIES \* series;

        document.getElementById('series').textContent = Math.round(series).toLocaleString();
        document.getElementById('memory').textContent = formatBytes(memory);
        document.getElementById('disk').textContent = formatBytes(disk);
    }
</script>
<div>
<table>
<tr><th># ScyllaDB Nodes</th><th># Cores Per ScyllaDB Node</th><th># Tables</th><th># Tables With Traffic</th><th># Service Levels</th><th># Alternator Tables</th><th>Prometheus Retention in Days</th></tr>
<tr>
<td><input type="number" id="hosts" oninput="myFunction()" value="3" placeholder="# Nodes" min="1"></td>
<td><input type="number" id="shards" oninput="myFunction()" value="16" placeholder="# Cores" min="1"></td>
<td><input type="number" id="tables" oninput="myFunction()" value="16" placeholder="# Tables" min="1"></td>
<td><input type="number" id="active_tables" oninput="myFunction()" value="16" placeholder="# Tables with traffic" min="0"></td>
<td><input type="number" id="service_levels" oninput="myFunction()" value="1" placeholder="# Service levels" min="1"></td>
<td><input type="number" id="alternator_tables" oninput="myFunction()" value="0" placeholder="# Alternator tables" min="0"></td>
<td><input type="number" id="retention" oninput="myFunction()" value="15" placeholder="# Days" min="1"></td>
</tr>
</table>
<table>
<tr><th># Series</th><th>Prometheus RAM</th><th>Prometheus Storage</th></tr>
<tr>
<td style="text-align:center"><span id="series">0</span></td>
<td style="text-align:center"><span id="memory">0</span></td>
<td style="text-align:center"><span id="disk">0</span></td>
</tr>
</table>
</div>
<script>
myFunction();
</script>
* **# Tables With Traffic** - tables that had reads or writes recently. Per-table latency histograms exist only for these tables, so they add many more series than idle tables.
* **# Service Levels** - user service levels, not counting the internal driver service level.

<!-- Rounded values used in the text above, written by prometheus_sizing.py --update-doc. -->


# monitor-description.md

ScyllaDB Monitoring Stack is a full stack for ScyllaDB monitoring and alerting. The stack contains open source tools including Prometheus and Grafana, as well as custom ScyllaDB dashboards and tooling.


# monitor-troubleshoot.md

# Troubleshoot Scylla Monitoring Stack

This document describes steps that need to be done to troubleshoot monitoring problems when using [Grafana/Prometheus](https://monitoring.docs.scylladb.com/stable/reference/monitoring-apis.md) monitoring tool.

## Problem

### Scylla-Manager 2.2 with Duplicate information

Scylla Manager 2.2 change the default metrics (Prometheus) reporting ports:

* For Manager server: from 56090 to 5090
* For Manager Agent: from 56090 to 5090

For backward compatibility, Scylla Monitoring Stack 3.5 default configuration reads from **both** Manager ports, old and new, so you do not have to update the Prometheus configuration when upgrading to Manager 2.2

However, if you configure `scylla_manager_server.yml` file with the new port, Scylla-Manager dashboard will report all metrics twice.

The easiest way around this is to edit `prometheus/prometheus.yml.template` and remove the `scylla_manager1` job.

Note that for this change to take effect you need to run `kill-all.sh` followed by `start-all.sh`.

### A Container Fails To Start

When running `./start-all.sh` a container can fail to start. For example you can see the following error message:

```shell
Wait for Prometheus container to start........Error: Prometheus container failed to start
```

Should this happen, check the Docker logs for more information.

```shell
docker logs aprom
```

Usually the reason for the failure is described in the logs.

### Files And Directory Permissions

#### NOTE
Avoid running Docker containers as root.

The preferred way of running the container is using a non root user.
See the [monitoring](https://monitoring.docs.scylladb.com/stable/install/monitoring-stack.md#docker-post-installation) Docker post-installation section.

If a container failed to start because of a permission problem, make sure
the external directory you are using is owned by the current user and that the current user has the proper permissions.

#### NOTE
If you started the container in the past as root, you may need to change the directory and files ownership and permissions.

For example if your Prometheus data directory is `/prom-data` and you are using `centos` user

```shell
ls -la /|grep prom-data

drwxr-xr-x    2 root root  4096 Jun 25 17:51 prom-data

sudo chown -R centos:centos /prom-data

ls -la /|grep prom-data

drwxr-xr-x    2 centos centos  4096 Jun 25 17:51 prom-data
```

### No Data Points

`No data points` on all data charts.

#### Solution

If there are no data points, or if a node appears to be unreachable when you know it is up, the immediate suspect is the Prometheus connectivity.

1. Login to the Prometheus console:
2. Point your browser to `http://{ip}:9090`, where {ip} is the Prometheus IP address.
3. Go to the target tabs: `http://{ip}:9090/targets` and see if any of the targets are down and if there are any error messages.

> * Make sure you are not using the local network for local IP range When using Docker containers, by default, the local IP range (127.0.0.X) is inside the Docker container and not the host local address. If you are trying to connect to a target via the local IP range from inside a Docker container, you need to use the `-l` flag to enable local network stack.
> * Confirm Prometheus is pointing to the wrong target. Check your `prometheus/scylla_servers.yml`. Make sure Prometheus is pulling data from the Scylla server.
> * Your dashboard and Scylla version may not be aligned. You can specify a specific Scylla version with `-v` flag to the start-all.sh script.

For example:

```shell
./start-all.sh -v 2024.1
```

More on start-all.sh [options](https://monitoring.docs.scylladb.com/stable/install/monitoring-stack.md).

### Grafana Chart Shows Error (!) Sign

Run this procedure on the Scylla Monitoring Stack server.

If the Grafana charts show an error (!) sign, there is a problem with the connection between Grafana and Prometheus.

#### Solution

On the Scylla Monitoring Stack server:

1. Check Prometheus is running using `docker ps`.

* If it is not running check the `prometheus.yml` for errors.

For example:

```shell
CONTAINER ID  IMAGE    COMMAND                  CREATED         STATUS         PORTS                                                    NAMES
41bd3db26240  monitor  "/docker-entrypoin..."   25 seconds ago  Up 23 seconds  7000-7001/tcp, 9042/tcp, 9160/tcp, 9180/tcp, 10000/tcp   monitor
```

* If it is running, go to “Data Source” in the Grafana GUI, choose Prometheus and click Test Connection.

### Grafana Shows Server Level Metrics, but not Scylla Metrics

Grafana shows server level metrics like disk usage, but not Scylla metrics.
Prometheus fails to fetch metrics from Scylla servers.

#### Solution

* Use `curl <scylla_node>:9180/metrics` to fetch metric data from Scylla.  If curl does not return data, the problem is the connectivity between the Scylla Monitoring Stack and Scylla server. In that case, check your IPs and firewalls.

For example

```shell
curl 172.17.0.2:9180/metrics
```

### Grafana Shows Scylla Metrics, but not Server Level Metrics

Grafana dashboards show Scylla metrics, such as load, but not server metrics such as disk usage.
Prometheus fails to fetch metrics from `node_exporter`.

#### Solution

1. Make sure that `node_exporter` is running on each Scylla server (by login to the machine and running `ps -ef | grep node_exporter`). `node_exporter` is installed with `scylla_setup`.
to check that `node_exporter` is installed, run `node_exporter --version`, If it is not, make sure to install and run it.

1. If it is running, use `curl http://<scylla_node>:9100/metrics` (where <scylla_node> is a Scylla server IP) to fetch metric data from the `node_exporter`.  If curl does not return data, the problem is the connectivity between Scylla Monitoring Stack and Scylla server. Please check your IPs and firewalls.

### Latencies Graphs Are empty

Starting from Scylla Monitoring version 3.8, Scylla Monitoring uses Prometheus’ recording rules for performance reasons. Recording rules perform some of the calculations when collecting the metrics, instead of when showing the dashboards.

During a transition period, Scylla Monitoring version 3.x has a fallback mechanism that shows data even if the recording rules are not present.

Scylla Monitoring versions 4.0 and newer rely only on recording rules.

If only the latency graphs are missing, it is because of missing recording rules.

This issue can be avoided in a clean installation, so if you are upgrading, it is recommended to perform a clean installation.

If you are using a standalone Prometheus server, make sure to copy the Prometheus configuration and recording rules as describe in [install without docker](https://monitoring.docs.scylladb.com/stable/install/monitor-without-docker.md#install-prometheus).

### Reducing the total number of metrics

In ScyllaDB, metrics are reported per shard (core) per node. A cluster with a high number of nodes and cores reports an increased number of metrics which might overload the Monitoring system like Prometheus or Datadog.
Below is one way to reduce the number of metrics reported per ScyllaDB Node.

#### Remove interrupts from node_exporter

By default, node_exporter reports interrupt metrics. You can disable interrupts reporting by editing
/etc/sysconfig/scylla-node-exporter and remove –collector.interrupts from it.

### Working with Wireshark

No metrics shown in the Scylla Monitoring Stack.

1. Install [wireshark](https://www.wireshark.org/#download)

2. Capture the traffic between the Scylla Monitoring Stack and Scylla node using the `tshark` command.
`tshark -i <network interface name> -f "dst port 9180"`

For example:

```shell
tshark -i eth0 -f "dst port 9180"
```

Capture from Scylla node towards Scylla Monitoring Stack server.

In this example, Scylla is running.

```shell
Monitor ip        Scylla node ip
199.203.229.89 -> 172.16.12.142 TCP 66 59212 > 9180 [ACK] Seq=317 Ack=78193 Win=158080 Len=0 TSval=79869679 TSecr=3347447210
```

In this example, Scylla is not running

```shell
Monitor ip        Scylla node ip
199.203.229.89 -> 172.16.12.142 TCP 74 60440 > 9180 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=79988291 TSecr=0 WS=128
```


# monitor-without-docker.md

# Deploying Scylla Monitoring Stack Without Docker

## Introduction

The following instructions will help to deploy [Scylla Monitoring Stack](https://monitoring.docs.scylladb.com/stable/install/monitoring-stack.md) in cases where you can not use the recommended Docker version.

Please note, Scylla recommends you use the Docker version as it will provide you with most updated, current Scylla Monitoring system.

Scylla Monitoring uses the following components:

* Alertmanager
* Grafana Loki
* Prometheus
* Grafana

The common scenario for users who use their own standalone installation, is that they already have such a server and would like to consolidate.
We assume that that you already have Prometheus and Grafana running but we will add minimal installation instruction for all components.

We suggest that you follow the installation instruction of each of those products from their official documentation. It is also recommended that all servers will run as a service.

## Minimal Production System Recommendations

* **CPU** - For clusters with up to 100 cores use 2vCPUS, for larger clusters 4vCPUs
* **Memory** - 15GB+ DRAM and proportional to the number of cores.
* **Disk** - persistent disk storage is proportional to the number of cores and Prometheus retention period (see the following section)
* **Network** - 1GbE/10GbE preferred

### Calculating Prometheus Minimal Disk Space requirement

Prometheus storage disk performance requirements: persistent block volume, for example an EC2 EBS volume

Prometheus storage disk volume requirement:  proportional to the number of metrics it holds and the default retention time. The default retention period is 15 days, and the disk requirement is around
3.5KB per series per day, assuming the default scraping interval of 20s.

For example, 100k series, with a retention time of 45 days, will need

```default
100k * 3.5KB * 45 ~ 15GB
```

To account for unexpected events, like replacing or adding nodes, we recommend allocating at least x2-3 the space, in this case, ~40GB.
Prometheus Storage disk does not have to be as fast as Scylla disk, and EC2 EBS, for example, is fast enough and provides HA out of the box.

### Calculating Prometheus Minimal Memory Space requirement

Prometheus uses more memory when querying over a longer duration (e.g. looking at a dashboard on a week view would take more memory than on an hourly duration).

For Prometheus alone, you should have 16KB of memory per series and it would use about 600MB of virtual memory per core.
Because Prometheus is so memory demanding, it is a good idea to add swap, so queries with a longer duration would not crash the server.

<script>
    //   series     = SERIES_CONST + SERIES_PER_NODE\*nodes + SERIES_PER_CORE\*cores + ...  (cores = total cores)
    //   rss_bytes  = RSS_CONST_BYTES + RSS_BYTES_PER_SERIES\*series
    //   disk_bytes = (DISK_CONST_BYTES_PER_DAY + DISK_BYTES_PER_SERIES_PER_DAY\*series) \* retention_days
    //                + WAL_CONST_BYTES + WAL_BYTES_PER_SERIES\*series
    const SERIES_CONST = 3786;
    const SERIES_PER_NODE = 1094;
    const SERIES_PER_CORE = 1185;
    const SERIES_PER_TABLE_PER_NODE = 17.86;
    const SERIES_PER_ACTIVE_TABLE_PER_NODE = 165.4;
    const SERIES_PER_SL_PER_CORE = 96.4;
    const SERIES_PER_ALTERNATOR_TABLE_PER_NODE = 20.9;
    const RSS_CONST_BYTES = 0;
    const RSS_BYTES_PER_SERIES = 16031;
    const DISK_CONST_BYTES_PER_DAY = 2096305;
    const DISK_BYTES_PER_SERIES_PER_DAY = 3574;
    const WAL_CONST_BYTES = 47523198;
    const WAL_BYTES_PER_SERIES = 2513;

    function formatBytes(bytes) {
        const units = ["B", "KB", "MB", "GB", "TB"];
        let i = 0;
        while (bytes >= 1024 && i < units.length - 1) {
            bytes /= 1024;
            i++;
        }
        return bytes.toFixed(i > 2 ? 2 : 0) + units[i];
    }

    function myFunction() {
        const value = (id) => parseInt(document.getElementById(id).value) || 0;
        const nodes = value('hosts');
        const cores = nodes \* value('shards');
        const tables = value('tables');
        const active = Math.min(value('active_tables'), tables);
        const sl = value('service_levels');
        const alternator = value('alternator_tables');
        const retention = value('retention');

        const series = SERIES_CONST + SERIES_PER_NODE \* nodes + SERIES_PER_CORE \* cores
            + SERIES_PER_TABLE_PER_NODE \* tables \* nodes + SERIES_PER_ACTIVE_TABLE_PER_NODE \* active \* nodes
            + SERIES_PER_SL_PER_CORE \* sl \* cores + SERIES_PER_ALTERNATOR_TABLE_PER_NODE \* alternator \* nodes;
        const memory = RSS_CONST_BYTES + RSS_BYTES_PER_SERIES \* series;
        const disk = (DISK_CONST_BYTES_PER_DAY + DISK_BYTES_PER_SERIES_PER_DAY \* series) \* retention
            + WAL_CONST_BYTES + WAL_BYTES_PER_SERIES \* series;

        document.getElementById('series').textContent = Math.round(series).toLocaleString();
        document.getElementById('memory').textContent = formatBytes(memory);
        document.getElementById('disk').textContent = formatBytes(disk);
    }
</script>
<div>
<table>
<tr><th># ScyllaDB Nodes</th><th># Cores Per ScyllaDB Node</th><th># Tables</th><th># Tables With Traffic</th><th># Service Levels</th><th># Alternator Tables</th><th>Prometheus Retention in Days</th></tr>
<tr>
<td><input type="number" id="hosts" oninput="myFunction()" value="3" placeholder="# Nodes" min="1"></td>
<td><input type="number" id="shards" oninput="myFunction()" value="16" placeholder="# Cores" min="1"></td>
<td><input type="number" id="tables" oninput="myFunction()" value="16" placeholder="# Tables" min="1"></td>
<td><input type="number" id="active_tables" oninput="myFunction()" value="16" placeholder="# Tables with traffic" min="0"></td>
<td><input type="number" id="service_levels" oninput="myFunction()" value="1" placeholder="# Service levels" min="1"></td>
<td><input type="number" id="alternator_tables" oninput="myFunction()" value="0" placeholder="# Alternator tables" min="0"></td>
<td><input type="number" id="retention" oninput="myFunction()" value="15" placeholder="# Days" min="1"></td>
</tr>
</table>
<table>
<tr><th># Series</th><th>Prometheus RAM</th><th>Prometheus Storage</th></tr>
<tr>
<td style="text-align:center"><span id="series">0</span></td>
<td style="text-align:center"><span id="memory">0</span></td>
<td style="text-align:center"><span id="disk">0</span></td>
</tr>
</table>
</div>
<script>
myFunction();
</script>
* **# Tables With Traffic** - tables that had reads or writes recently. Per-table latency histograms exist only for these tables, so they add many more series than idle tables.
* **# Service Levels** - user service levels, not counting the internal driver service level.

<!-- Rounded values used in the text above, written by prometheus_sizing.py --update-doc. -->

The main item to set an alert on is the available disk space in the monitoring system. Data is indefinitely accrued on the Prometheus data directory.
The current monitoring solution does not churn data.

#### NOTE
Confirm before installing, that your Grafana and Prometheus versions are supported by the Scylla Monitoring Stack version you want to install. Scylla-Monitoring follows the latest Prometheus and Grafana releases tightly. See the [Scylla Monitoring Stack Compatibility Matrix](https://monitoring.docs.scylladb.com/stable/install/monitoring-stack.md#monitoring-stack-prerequisites).

## Install Scylla Monitoring Stack

The following procedure uses a `CentOS 7` based instance

1. Download the latest Scylla Monitoring Stack release.

```sh
wget -O scylla-monitoring.tar.gz https://github.com/scylladb/scylla-monitoring/archive/refs/tags/4.16.1.tar.gz
```

1. Open the tar

`tar -xvf scylla-monitoring.tar.gz`

## Install Alertmanager

Tested with alertmanager 0.26.0 version

1. Install [alertmanager](https://prometheus.io/download/)

```shell
wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar -xvf alertmanager-*.linux-amd64.tar.gz
```

1. Copy the following file: `rule_config.yml` from `prometheus/` directory to `alertmanager.yml` in the alertmanager installation directory.

For example:

```shell
cp -p /home/centos/scylla-monitoring-4.16.1/prometheus/rule_config.yml alertmanager-0.26.0.linux-amd64/alertmanager.yml
```

1. Start the Alertmanager

For example:

```shell
cd alertmanager-0.26.0.linux-amd64/
./alertmanager
```

1. Verify that Alertmanager is up and running, point your browser to the Alertmanager IP:Port

For example:

```shell
http://192.168.135.166:9093/
```

![image](install/alertmanager.png)

## Install Grafana Loki

Loki is a log aggregation system inspired by Prometheus. Scylla Monitoring uses Loki for alerts and metrics generation. It does not replaces your centralized logging server, but it can, check
Loki-Grafana [documentation](https://grafana.com/docs/loki/latest/) if you want to use it for centralized log collection.

We recomand using Loki with containers, but you can install it locally as described in [Loki installation](https://grafana.com/docs/loki/latest/installation/local/)

You will need to run both Loki and Promtail. Loki responsible for log parsing and acts as a Grafana and Proemtheus data-source and Generate alerts that are sent to the Alertmanager.

Promtail load logs into Loki, there are multiple ways of doing that, we suggest to use of rsyslog, this way you can add Promtail (and Loki) as a second log collection server.

**Loki Related files**

Loki has a configuration file and a rule file. You need to copy and modify the configuration.

```shell
mkdir -p /etc/loki/rules
mkdir -p /etc/loki/config
cp loki/rules/scylla/* /etc/loki/rules
cp loki/conf/loki-config.template.yaml /etc/loki/config/loki-config.yaml
```

Edit `/etc/loki/config/loki-config.yaml` and replace `ALERTMANAGER` with the alertmanager ip:port (i.e. localhost:9093)

**Promtail Related files**

Promtail has a configuration file. You need to copy and modify the configuration.

```shell
mkdir -p /etc/promtail/
cp loki/promtail/promtail_config.template.yml /etc/promtail/config.yml
```

Edit `/etc/promtail/config.yml` and replace `LOKI_IP` with Loki’s ip:port (i.e. localhost:3100)

<a id="install-prometheus"></a>

## Install Prometheus

Tested with Prometheus version 2.49.1

#### NOTE
If you already have a prometheus server, beside the expected scrap jobs, make sure you take the Prometheus rules directory.
The files not only contains important alerts, they are containing recording rules, without it different asspects of the dashboards will not work.

1. Install [Prometheus](https://prometheus.io/download/)

```shell
wget https://github.com/prometheus/prometheus/releases/download/v2.49.1/prometheus-2.49.1.linux-amd64.tar.gz
tar -xvf prometheus-*.linux-amd64.tar.gz
```

1. Create Data and Config directories

```shell
sudo mkdir -p /prometheus/data
sudo mkdir -p /etc/prometheus/prom_rules/
sudo mkdir -p /etc/scylla.d/prometheus/
```

#### NOTE
To run Promethues as non-root user, you would need to give read write and execute permissions to those directories.

1. Copy the following files: `scylla_servers.yml`, `prometheus.rules.yml` from `prometheus/` directory to Prometheus installation directory.

Copy `prometheus/prometheus.yml.template` to `prometheus.yml`

For example:

```shell
cp scylla-monitoring-4.16.1/prometheus/prom_rules/*.yml /etc/prometheus/prom_rules/
cp scylla-monitoring-4.16.1/prometheus/prometheus.yml.template /etc/prometheus/prometheus.yml
```

1. Edit the `prometheus.yml` file to point to the correct static data sources.

#### NOTE
Make sure to include the `honor_labels: false` parameter in the prometheus.yml file.

```shell
vi /etc/prometheus/prometheus.yml
```

Set the alertmanger address and port by replacing `AM_ADDRESS` in the file.

For example if the alertmanager will run on the same host:

```shell
alerting:
   alertmanagers:
   - static_configs:
       - targets:
           - 127.0.0.1:9093
```

Replace the files in to point to the right local file, typically for scylla, node_exporter and manager_agent you can use the same file (scylla_servers.yml).

For example the scrape config for Scylla:

```shell
global:
  scrape_interval: 20s # By default, scrape targets every 20s second.
  scrape_timeout: 15s # Timeout before trying to scape a target again

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'scylla-monitor'

scrape_configs:
- job_name: scylla
  honor_labels: false
  file_sd_configs:
    - files:
      - /etc/scylla.d/prometheus/scylla_servers.yml
  relabel_configs:
    - source_labels: [__address__]
      regex:  '([^:]+)'
      target_label: __address__
      replacement: '${1}:9180'

    - source_labels: [__address__]
      regex:  '(.*):.+'
      target_label: instance
      replacement: '${1}'
  metric_relabel_configs:
    - source_labels: [__name__, scheduling_group_name]
      regex: '(scylla_storage_proxy_coordinator_.*_bucket;)(atexit|gossip|mem_compaction|memtable|streaming|background_reclaim|compaction|main|memtable_to_cache)'
      action: drop
    - source_labels: [version]
      regex:  '(.+)'
      target_label: CPU
      replacement: 'cpu'
    - source_labels: [version]
      regex:  '(.+)'
      target_label: CQL
      replacement: 'cql'
    - source_labels: [version]
      regex:  '(.+)'
      target_label: OS
      replacement: 'os'
    - source_labels: [version]
      regex:  '(.+)'
      target_label: IO
      replacement: 'io'
    - source_labels: [version]
      regex:  '(.+)'
      target_label: Errors
      replacement: 'errors'
    - regex: 'help|exported_instance'
      action: labeldrop
    - source_labels: [version]
      regex: '([0-9]+\.[0-9]+)(\.?[0-9]*).*'
      replacement: '$1$2'
      target_label: svr
    - source_labels: [__name__, quantile]
      regex: '(scylla_storage_proxy_coordinator_write_latency_summary;0.990*)'
      target_label: __name__
      replacement: 'wlatencyp99'
    - source_labels: [__name__, quantile]
      regex: '(scylla_storage_proxy_coordinator_write_latency_summary;0.950*)'
      target_label: __name__
      replacement: 'wlatencyp95'
    - source_labels: [__name__, quantile]
      regex: '(scylla_storage_proxy_coordinator_write_latency_summary;0.50*)'
      target_label: __name__
      replacement: 'wlatencya'
    - source_labels: [__name__, quantile]
      regex: '(scylla_storage_proxy_coordinator_read_latency_summary;0.990*)'
      target_label: __name__
      replacement: 'rlatencyp99'
    - source_labels: [__name__, quantile]
      regex: '(scylla_storage_proxy_coordinator_read_latency_summary;0.950*)'
      target_label: __name__
      replacement: 'rlatencyp95'
    - source_labels: [__name__, quantile]
      regex: '(scylla_storage_proxy_coordinator_read_latency_summary;0.50*)'
      target_label: __name__
      replacement: 'rlatencya'
    - source_labels: [__name__, quantile]
      regex: '(scylla_storage_proxy_coordinator_cas_write_latency_summary;0.950*)'
      target_label: __name__
      replacement: 'caswlatencyp95'
    - source_labels: [__name__, quantile]
      regex: '(scylla_storage_proxy_coordinator_cas_write_latency_summary;0.990*)'
      target_label: __name__
      replacement: 'caswlatencyp99'
    - source_labels: [__name__, quantile]
      regex: '(scylla_storage_proxy_coordinator_cas_write_latency_summary;0.50*)'
      target_label: __name__
      replacement: 'caswlatencya'
    - source_labels: [__name__, quantile]
      regex: '(scylla_storage_proxy_coordinator_cas_read_latency_summary;0.950*)'
      target_label: __name__
      replacement: 'casrlatencyp95'
    - source_labels: [__name__, quantile]
      regex: '(scylla_storage_proxy_coordinator_cas_read_latency_summary;0.990*)'
      target_label: __name__
      replacement: 'casrlatencyp99'
    - source_labels: [__name__, quantile]
      regex: '(scylla_storage_proxy_coordinator_cas_read_latency_summary;0.50*)'
      target_label: __name__
      replacement: 'casrlatencya'
    - source_labels: [quantile]
      regex: '(0\.[1-9]+)0*'
      target_label: quantile
      replacement: '${1}'
    - source_labels: [__name__]
      regex: '(.latency..?.?|cas.latency..?.?|scylla_.*_summary)'
      target_label: by
      replacement: 'instance,shard'
```

#### NOTE
You can change scrape_interval to a longer interval to save resources. If you do that, you must change the Grafan configuration to match it.

1. Create and set `scylla_servers.yml` file point to your Scylla nodes and `scylla_manager_server.yml` file to point to your Scylla Manager.

#### NOTE
There is no need to configure `node_exporter_server`. Instead, in the Prometheus scrape config of the node_exporter
you can use the same file you used for Scylla and Prometheus will assume you have a `node_exporter` running on each Scylla server.

An example for those files can be found under the Prometheus directory: `prometheus/scylla_servers.example.yml` and `prometheus/scylla_manager_servers.example.yml`

#### NOTE
You must have both files even if you are not using Scylla Manager

Add the labels for the cluster and data-center

`scylla_servers.yml`:

For example:

```shell
cat /etc/scylla.d/prometheus/scylla_servers.yml
# List Scylla end points

- targets:
  - 192.168.66.6
  - 192.168.66.244
  labels:
    cluster: cluster1
    dc: dc1
- targets:
    - 172.17.0.3
  labels:
    cluster: cluster1
    dc: dc2
```

#### NOTE
See the previous note about deprecating the `node_exporter_servers.yml` file.

`scylla_manager_server.yml`

For example:

```shell
cat /etc/scylla.d/prometheus/scylla_manager_server.yml
- targets:
  - 127.0.0.1:5090
```

1. Start Prometheus server:

For example:

```shell
cd prometheus-2.49.1.linux-amd64/
./prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path /prometheus/data
```

Data should start accumulate on: /prometheus/data

1. Verify that Prometheus is up and running, point your browser to the Prometheus IP:Port

For example:

```shell
http://192.168.135.166:9090/
```

![image](install/1.png)

Prometheus console should be visible

1. Verify that the node_exporter and scylla metrics accumulating in the server by executing a query through the console

For example:

`node_memory_MemFree`

![image](install/2.png)

And

`scylla_reactor_utilization`

![image](install/3.png)

At this point Scylla is emitting the metrics and Prometheus is able to store them.

## Install Grafana

Tested with Grafna 10.3.3

1. Install Grafana based on the instructions [here](http://docs.grafana.org/installation/)

Depends if you installed Grafana from a repository (yum install), or if you downloaded the zip version, the directory structure will be
different, we will assume that you used yum/dnf install.

1. Access Scylla-Grafana-monitoring directory

```shell
cd scylla-monitoring-4.16.1/
```

1. Copy the plugins to the grafana plugins directory (by default `/var/lib/grafana/`)

```shell
sudo cp -r grafana/plugins /var/lib/grafana/
```

1. Provision the Dashboards

For example Scylla Open-source version 5.4 and Scylla manager version 2.4

```shell
sudo cp grafana/load.yaml /etc/grafana/provisioning/dashboards/load.5.4.yaml
sudo cp grafana/load.yaml /etc/grafana/provisioning/dashboards/load.manager_3.3.yaml
sudo mkdir -p /var/lib/grafana/dashboards
sudo cp -r grafana/build/* /var/lib/grafana/dashboards
```

Edit the `load.*` you just copied. For example

```shell
$ cat /etc/grafana/provisioning/dashboards/load.manager_3.3.yaml
apiVersion: 1

providers:
- name: 'manager_3.3'
  orgId: 1
  folder: ''
  type: file
  disableDeletion: false
  allowUiUpdates: true
  updateIntervalSeconds: 10 #how often Grafana will scan for changed dashboards
  options:
    path: /var/lib/grafana/dashboards/manager_3.3

$ cat /etc/grafana/provisioning/dashboards/load.5.4.yaml
apiVersion: 1

providers:
- name: '5.4'
  orgId: 1
  folder: ''
  type: file
  disableDeletion: false
  allowUiUpdates: true
  updateIntervalSeconds: 10 #how often Grafana will scan for changed dashboards
  options:
    path: /var/lib/grafana/dashboards/ver_5.4
```

#### NOTE
A note about using folders, if you provision multiple Scylla versions, use the version as a folder name. Otherwise, no need to configure a FOLDER.

1. Set the data source by copy `datasource.yml` and edit it

```shell
sudo cp grafana/datasource.yml /etc/grafana/provisioning/datasources/
```

#### NOTE
Scylla uses a plugin to read from some system tables see the section below about using it.

You should set the Prometheus and the alertmanager IP and port.

For example

```shell
sudo cat /etc/grafana/provisioning/datasources/datasource.yml
apiVersion: 1
datasources:
 - name: prometheus
   type: prometheus
   url: http://192.168.135.167:9090
   access: proxy
   basicAuth: false
   jsonData:
      timeInterval: '20s' # Must match prometheus gloabal scrape_interval

 - name: alertmanager
   type: camptocamp-prometheus-alertmanager-datasource
   orgId: 1
   typeLogoUrl: public/img/icn-datasource.svg
   access: proxy
   url: http://192.168.135.166:9093
   password:
   user:
   database:
   basicAuth:
   isDefault:
   jsonData:
     implementation: 'prometheus'
```

#### NOTE
The timeInterval parameter must match Prometheus scrape_interval config.

1. Start the Grafana service

For Grafana installed with yum install

`sudo service grafana-server start`

1. Make sure Grafana is running

Point your browser to the Grafana server port 3000, the assumption is that Grafana and Prometheus are collocated on the same server.

![image](install/grafana.png)

## Using Scylla Plugin with Grafana

Scylla Monitoring uses a plugin to read from some of the System tables. For the plugin to work it needs to be installed, configured and there should be a CQL connectivity between the Scylla Monitoring and the Scylla servers.

Because the plugin gives access to the Scylla tables, we strongly encourage you to add a user with read-only access restricted to the system keyspace and configure the plugin to use that user.

### Setting a monitoring user

This part is optional, but is highly recommended. The instruction at [enable authorization](https://docs.scylladb.com/operating-scylla/security/enable-authorization/) covers all the following items in details.

* If you have not done so, [enable authorization](https://docs.scylladb.com/operating-scylla/security/enable-authorization/) first.
* Add a new ROLL for the scylla monitoring: `CREATE ROLE scylla_monitoring WITH PASSWORD = 'scylla_monitoring' AND LOGIN = true;` make sure to give it a proper password.
* Add SELECT permissions to the new user: `GRANT SELECT on KEYSPACE system TO scylla_monitoring`;

### Installing the Plugin

Grafana reads plugins from its plugin directory, copy Scylla Plugin from  ‘grafana/plugins/scylla-datasource’ as described in the Grafana installation section.

### Configure the Plugin

Add an entry to the datasource.yml file

```shell
- name: scylla-datasource
  type: scylladb-scylla-datasource
  orgId: 1
  isDefault:
  jsonData:
  host: ''
#  secureJsonData:
#    user: 'scylla_monitoring'
#    password: 'scylla_monitoring'
```

As mentioned previously it is safer to use a dedicated user/password for the monitoring with limited access privileges. Un-comment the relevant section if you do so, note that the username/password are only visible in the file.

### Enable the Plugin

Grafana will not load unsigned plugins, for that you will need to enable it with Grafana. Edit Grafana `grafana.ini` file and add
the line `allow_loading_unsigned_plugins: scylladb-scylla-datasource`.

See more about it the [Grafana configuration](https://grafana.com/docs/grafana/latest/administration/configuration/#allow_loading_unsigned_plugins).

### Embedding Dashboards

Iframe embedding is disabled by default. To enable it when installing Grafana manually, add the
following to your `grafana.ini` file:

```ini
[security]
allow_embedding = true
cookie_secure = true
cookie_samesite = none
```

`cookie_secure` requires HTTPS when embedding dashboards on a different site.

When using the monitoring stack scripts, set `GF_SECURITY_ALLOW_EMBEDDING=true` in `env.sh` or pass `--allow-embedding` to `start-all.sh` instead of editing `grafana.ini` directly.


# monitoring-apis.md

# Scylla Monitoring Stack Interfaces

Scylla exposes two interfaces for online monitoring, as described below

<a id="api-prometheus"></a>

## Prometheus API

By default, Scylla listens on port 9180 for [Prometheus](https://prometheus.io/) requests. To connect a Prometheus server to Scylla in your prometheus.yaml configuration file, add Scylla as a target with `your-ip:9180`

For more information on monitoring Scylla with Prometheus see [Scylla Monitoring Stack](https://monitoring.docs.scylladb.com/stable/install/monitoring-stack.md).

You can change the Prometheus listening address and port in scylla.yaml file

```yaml
# prometheus port
# By default, Scylla opens prometheus API port on port 9180
# setting the port to 0 will disable the prometheus API.
prometheus_port: 9180
#
# prometheus address
# By default, Scylla binds all interfaces to the prometheus API
# It is possible to restrict the listening address to a specific one
prometheus_address: 0.0.0.0
```


# monitoring-stack.md

# Install Scylla Monitoring Stack

This document describes the setup of Scylla Monitoring Stack, based on [Prometheus API](https://monitoring.docs.scylladb.com/stable/reference/monitoring-apis.md#api-prometheus).

The Scylla Monitoring Stack needs to be installed on a dedicated server, external to the Scylla cluster. Make sure the Scylla Monitoring Stack server has access to the Scylla nodes so that it can pull the metrics over the Prometheus API.

For evaluation, you can run Scylla Monitoring Stack on any server (or laptop) that can handle three Docker instances at the same time. For production, see recommendations below.

## Minimal Production System Recommendations

* **CPU** - For clusters with up to 100 cores use 2vCPUS, for larger clusters 4vCPUs
* **Memory** - 15GB+ DRAM and proportional to the number of cores.
* **Disk** - persistent disk storage is proportional to the number of cores and Prometheus retention period (see the following section)
* **Network** - 1GbE/10GbE preferred

### Calculating Prometheus Minimal Disk Space requirement

Prometheus storage disk performance requirements: persistent block volume, for example an EC2 EBS volume

Prometheus storage disk volume requirement:  proportional to the number of metrics it holds and the default retention time. The default retention period is 15 days, and the disk requirement is around
3.5KB per series per day, assuming the default scraping interval of 20s.

For example, 100k series, with a retention time of 45 days, will need

```default
100k * 3.5KB * 45 ~ 15GB
```

To account for unexpected events, like replacing or adding nodes, we recommend allocating at least x2-3 the space, in this case, ~40GB.
Prometheus Storage disk does not have to be as fast as Scylla disk, and EC2 EBS, for example, is fast enough and provides HA out of the box.

### Calculating Prometheus Minimal Memory Space requirement

Prometheus uses more memory when querying over a longer duration (e.g. looking at a dashboard on a week view would take more memory than on an hourly duration).

For Prometheus alone, you should have 16KB of memory per series and it would use about 600MB of virtual memory per core.
Because Prometheus is so memory demanding, it is a good idea to add swap, so queries with a longer duration would not crash the server.

<script>
    //   series     = SERIES_CONST + SERIES_PER_NODE\*nodes + SERIES_PER_CORE\*cores + ...  (cores = total cores)
    //   rss_bytes  = RSS_CONST_BYTES + RSS_BYTES_PER_SERIES\*series
    //   disk_bytes = (DISK_CONST_BYTES_PER_DAY + DISK_BYTES_PER_SERIES_PER_DAY\*series) \* retention_days
    //                + WAL_CONST_BYTES + WAL_BYTES_PER_SERIES\*series
    const SERIES_CONST = 3786;
    const SERIES_PER_NODE = 1094;
    const SERIES_PER_CORE = 1185;
    const SERIES_PER_TABLE_PER_NODE = 17.86;
    const SERIES_PER_ACTIVE_TABLE_PER_NODE = 165.4;
    const SERIES_PER_SL_PER_CORE = 96.4;
    const SERIES_PER_ALTERNATOR_TABLE_PER_NODE = 20.9;
    const RSS_CONST_BYTES = 0;
    const RSS_BYTES_PER_SERIES = 16031;
    const DISK_CONST_BYTES_PER_DAY = 2096305;
    const DISK_BYTES_PER_SERIES_PER_DAY = 3574;
    const WAL_CONST_BYTES = 47523198;
    const WAL_BYTES_PER_SERIES = 2513;

    function formatBytes(bytes) {
        const units = ["B", "KB", "MB", "GB", "TB"];
        let i = 0;
        while (bytes >= 1024 && i < units.length - 1) {
            bytes /= 1024;
            i++;
        }
        return bytes.toFixed(i > 2 ? 2 : 0) + units[i];
    }

    function myFunction() {
        const value = (id) => parseInt(document.getElementById(id).value) || 0;
        const nodes = value('hosts');
        const cores = nodes \* value('shards');
        const tables = value('tables');
        const active = Math.min(value('active_tables'), tables);
        const sl = value('service_levels');
        const alternator = value('alternator_tables');
        const retention = value('retention');

        const series = SERIES_CONST + SERIES_PER_NODE \* nodes + SERIES_PER_CORE \* cores
            + SERIES_PER_TABLE_PER_NODE \* tables \* nodes + SERIES_PER_ACTIVE_TABLE_PER_NODE \* active \* nodes
            + SERIES_PER_SL_PER_CORE \* sl \* cores + SERIES_PER_ALTERNATOR_TABLE_PER_NODE \* alternator \* nodes;
        const memory = RSS_CONST_BYTES + RSS_BYTES_PER_SERIES \* series;
        const disk = (DISK_CONST_BYTES_PER_DAY + DISK_BYTES_PER_SERIES_PER_DAY \* series) \* retention
            + WAL_CONST_BYTES + WAL_BYTES_PER_SERIES \* series;

        document.getElementById('series').textContent = Math.round(series).toLocaleString();
        document.getElementById('memory').textContent = formatBytes(memory);
        document.getElementById('disk').textContent = formatBytes(disk);
    }
</script>
<div>
<table>
<tr><th># ScyllaDB Nodes</th><th># Cores Per ScyllaDB Node</th><th># Tables</th><th># Tables With Traffic</th><th># Service Levels</th><th># Alternator Tables</th><th>Prometheus Retention in Days</th></tr>
<tr>
<td><input type="number" id="hosts" oninput="myFunction()" value="3" placeholder="# Nodes" min="1"></td>
<td><input type="number" id="shards" oninput="myFunction()" value="16" placeholder="# Cores" min="1"></td>
<td><input type="number" id="tables" oninput="myFunction()" value="16" placeholder="# Tables" min="1"></td>
<td><input type="number" id="active_tables" oninput="myFunction()" value="16" placeholder="# Tables with traffic" min="0"></td>
<td><input type="number" id="service_levels" oninput="myFunction()" value="1" placeholder="# Service levels" min="1"></td>
<td><input type="number" id="alternator_tables" oninput="myFunction()" value="0" placeholder="# Alternator tables" min="0"></td>
<td><input type="number" id="retention" oninput="myFunction()" value="15" placeholder="# Days" min="1"></td>
</tr>
</table>
<table>
<tr><th># Series</th><th>Prometheus RAM</th><th>Prometheus Storage</th></tr>
<tr>
<td style="text-align:center"><span id="series">0</span></td>
<td style="text-align:center"><span id="memory">0</span></td>
<td style="text-align:center"><span id="disk">0</span></td>
</tr>
</table>
</div>
<script>
myFunction();
</script>
* **# Tables With Traffic** - tables that had reads or writes recently. Per-table latency histograms exist only for these tables, so they add many more series than idle tables.
* **# Service Levels** - user service levels, not counting the internal driver service level.

<!-- Rounded values used in the text above, written by prometheus_sizing.py --update-doc. -->

<a id="monitoring-stack-prerequisites"></a>

## Prerequisites

* Follow the Installation Guide and install [docker](https://docs.docker.com/install/) on the Scylla Monitoring Stack Server. This server can be the same server that is running Scylla Manager. Alternatively, you can [Deploy Scylla Monitoring Stack Without Docker](https://monitoring.docs.scylladb.com/stable/install/monitor-without-docker.md).
* If you have Prometheus or Grafana installed, confirm that your version is supported by the Scylla Monitoring Stack version you want to install. Refer to the table below.

#### Scylla Monitoring Stack Compatibility Matrix

| Scylla Monitoring Stack Version   | Prometheus Version   | Grafana Version   |
|-----------------------------------|----------------------|-------------------|
| 4.16                              | 3.12.0               | 13.2.0            |
| 4.15                              | 3.11.3               | 12.4.3            |
| 4.14                              | 3.9.1                | 12.3.2            |
| 4.13                              | 3.8.1                | 12.3.1            |
| 4.12                              | 3.5.0                | 12.1.1            |
| 4.11                              | 3.4.1                | 12.0.2            |
| 4.10                              | 3.3.1                | 11.6.1            |
| 4.9                               | 3.1.0                | 11.4.0            |
| 4.8                               | 2.53.1               | 11.1.0            |
| 4.7                               | 2.50.1               | 11.0.0            |
| 4.6                               | 2.48.1               | 10.2.2            |
| 4.5                               | 2.47.1               | 10.1.5            |
| 4.4                               | 2.44.0               | 9.5.2             |
| 4.3                               | 2.42.0               | 9.3.8             |
| 4.2                               | 2.41.0               | 9.3.4             |
| 4.1                               | 2.38.0               | 9.1.0             |
| 4.0                               | 2.34.0               | 8.5.2             |
| 3.11                              | 2.32.0               | 8.3.4             |
| 3.10                              | 2.32.0               | 8.3.3             |
| 3.9.2                             | 2.29.1               | 8.2.7             |
| 3.9                               | 2.29.1               | 8.1.1             |
| 3.8                               | 2.27.1               | 7.5.7             |
| 3.7                               | 2.25.2               | 7.4.0             |
| 3.6                               | 2.18.1               | 7.3.5             |
| 3.5                               | 2.18.1               | 7.1.5             |
| 3.4                               | 2.18.1               | 6.7.3             |

<a id="docker-post-installation"></a>

## Docker Post Installation

Docker post installation guide can be found [here](https://docs.docker.com/install/linux/linux-postinstall/)

#### NOTE
Avoid running the container as root.

To avoid running docker as root, you should add the user you are going to use for Scylla Monitoring Stack to the Docker group.

1. Create the Docker group.

```sh
sudo groupadd docker
```

1. Add your user to the docker group. Log out and log in again. The new group will be active for this user on next login.

```sh
sudo usermod -aG docker $USER
```

1. Start Docker by calling:

```sh
sudo systemctl enable docker
```

## Install Scylla Monitoring Stack

**Procedure**

1. Download and extract the latest [Scylla Monitoring Stack binary](https://github.com/scylladb/scylla-monitoring/releases);.

```sh
wget https://github.com/scylladb/scylla-monitoring/archive/4.16.1.tar.gz
tar -xvf 4.16.1.tar.gz
cd scylla-monitoring-4.16.1
```

As an alternative, you can clone and use the Git repository directly.

```sh
git clone https://github.com/scylladb/scylla-monitoring.git
cd scylla-monitoring
git checkout branch-4.16
```

1. Start Docker service if needed

```sh
sudo systemctl restart docker
```

## Configure Scylla Monitoring Stack

To monitor the cluster, Scylla Monitoring Stack (Specifically the Prometheus Server) needs to know the IP of all the nodes and the IP of the Scylla Manager Server (if you are using Scylla Manager).

This configuration can be done from files, or using the [Consul](https://www.consul.io/) api.

Scylla Manager 2.0 and higher supports the Consul API.

### Configure Scylla nodes from files

1. Create `prometheus/scylla_servers.yml` with the targets’ IPs (the servers you wish to monitor).

#### NOTE
It is important that the name listed in `dc` in the `labels` matches the datacenter names used by Scylla.
Use the `nodetool status` command to validate the datacenter names used by Scylla.

For example:

```yaml
- targets:
      - 172.17.0.2
      - 172.17.0.3
  labels:
      cluster: cluster1
      dc: dc1
```

#### NOTE
If you want to add your managed cluster to Scylla Monitoring Stack, add the IPs of the nodes as well as the cluster name you used when you [added the cluster](https://manager.docs.scylladb.com/stable/add-a-cluster.html) to Scylla Manager. It is important that the label `cluster name` and the cluster name in Scylla Manager match.

*Using IPV6*

To use IPv6 inside scylla_server.yml, add the IPv6 addresses with their square brackets.

For example:

```yaml
- targets:
      - "[2600:1f18:26b1:3a00:fac8:118e:9199:67b9]"
      - "[2600:1f18:26b1:3a00:fac8:118e:9199:67ba]"
  labels:
      cluster: cluster1
      dc: dc1
```

#### NOTE
For IPv6 to work, both scylla Prometheus address and node_exporter’s –web.listen-address should be set to listen to an IPv6 address.

For general node information (disk, network, etc.) Scylla Monitoring Stack uses the `node_exporter` agent that runs on the same machine as Scylla does.
By default, Prometheus will assume you have a `node_exporter` running on each machine. If this is not the case, for example if Scylla runs in a container and the node_exporter runs on the host, you can override the `node_exporter`
targets configuration file by creating an additional file and passing it with the `-n` flag.

#### NOTE
By default, there is no need to create `node_exporter_server.yml`. Prometheus will use the same targets it uses for
Scylla and will assume you have a `node_exporter` running on each Scylla server.

If needed, you can set your own target file instead of the default `prometheus/scylla_servers.yml`, using the `-s` for Scylla target files.

For example:

```yaml
./start-all.sh -s my_scylla_server.yml -d prometheus_data
```

Mark the different Data Centers with Labels.

As can be seen in the examples, each target has its own set of labels to mark the cluster name and the data center (dc).
You can add multiple targets in the same file for multiple clusters or multiple data centers.

You can use the `genconfig.py` script to generate the server file. For example:

```yaml
./genconfig.py -d myconf -dc dc1:192.168.0.1,192.168.0.2 -dc dc2:192.168.0.3,192.168.0.4
```

This will generate a server file for four servers in two datacenters server `192.168.0.1` and `192.168.0.2` in dc1 and `192.168.0.3` and `192.168.0.4` in dc2.

OR

The `genconfig.py` script can also use `nodetool status` to generate the server file using the `-NS` flag.

```yaml
nodetool status | ./genconfig.py -NS
```

2. Connect to [Scylla Manager](https://scylladb.github.io/scylla-manager/) by creating `prometheus/scylla_manager_servers.yml`
If you are using Scylla Manager, you should set its IP and port in this file.

You must add a scylla_manager_servers.yml file even if you are not using the manager.
You can look at: `prometheus/scylla_manager_servers.example.yml` for an example.

For example if Scylla Manager host IP is 172.17.0.7 `prometheus/scylla_manager_servers.yml` would look like:

```yaml
# List Scylla Manager end points

- targets:
  - 172.17.0.7:5090
```

Note that you do not need to add labels to the Scylla Manager targets.

### Configure Scylla nodes using Scylla-Manager Consul API

Scylla Manager 2.0 has a [Consul](https://www.consul.io/) like API.

When using the manager as the configuration source, there is no need  to set any of the files.
Instead you should set the scylla-manager IP from the command line using the -L flag.

For example:

```yaml
./start-all.sh -L 10.10.0.1
```

#### NOTE
If you are running Scylla-Manager on the same host as Scylla-Monitoring you should use -l flag so that the localhost address
will be available from within the container.

### Connecting Scylla-Monitoring to ScyllaDB

Scylla-Monitoring version 3.5 and higher can read tables from a ScyllaDB node using CQL. If your ScyllaDB cluster is user/password protected (See [Scylla  Authorization](https://docs.scylladb.com/operating-scylla/security/enable-authorization/)) you should assign a user and password for the Scylla-Grafana connection.

You can limit the user to read only, currently it only read table from the system keyspace.

You can set a user and password from a file or environment variables.

If the environment variables **SCYLLA_USER** and  **SCYLLA_PSSWD** are set, they will be used.

To set the user and password from a file, edit grafana/datasource.scylla.yml. Uncomment the **secureJsonData** part and set the user and password.

#### NOTE
It is best to use a dedicated user and password with limited privileges.

### Use an external directory for the Prometheus data directory

The `-d` flag, places the Prometheus data directory outside of its container and by doing that makes it persistent.

#### NOTE
Specifying an external directory is important for systems in production. Without it,
every restart of the monitoring stack will result in metrics lost.

If the directory provided does not exist, the `start-all.sh` script will create it. Note that you should avoid running docker as root, the `start-all.sh` script
will use the user permissions that runs it. This is important if you want to place the prometheus directory not under the user path but somewhere else, for example `/prometheus-data`.

In that case, you need to create the directory before calling `start-all.sh` and make sure it has the right permissions for the user running the command.

### Add Additional Prometheus Targets

There are situations where you would like to monitor additional targets using the Prometheus server of the monitoring stack.
For example, an agent that runs on a firewall server.
The Prometheus server reads its targets from a file, this file is generated from a template when calling `start-all.sh`.
To add your targets you would need to edit the template file before calling `start-all.sh`.

The template file is either `prometheus/prometheus.yml.template` if Prometheus reads the Scylla target from file, or `prometheus/prometheus.consul.yml.template`
if Prometheus gets Scylla targets from the manager Consul API.

You can add a target at the end of the file, for example, the following example would read from a server with IP address 17.0.0.1 with a Prometheus port of 7000.

```yaml
- job_name: 'myservice'
  # Override the global default and scrape targets from this job every 5 seconds.
  scrape_interval: 5s
  static_configs:
    - targets:
      - 17.0.0.1:7000
```

## Start and Stop Scylla Monitoring Stack

### Start

```yaml
./start-all.sh -d prometheus_data
```

### Stop

```yaml
./kill-all.sh
```

### Start a Specific Scylla Monitoring Stack Version

By default, start-all.sh will start with dashboards for the latest Scylla Open source version and the latest Scylla Manager version.

You can specify specific scylla version with the `-v` flag and Scylla Manager version with `-M` flag.

Multiple versions are supported. For example:

```sh
./start-all.sh -v 2020.1,2019.1 -M 2.1 -d prometheus-data
```

will load the dashboards for Scylla Enterprise versions `2020.1` and `2019.1` and the dashboard for Scylla Manager `2.1`

### Accessing the localhost

The Prometheus server runs inside a Docker container if it needs to reach a target on the local- host: either Scylla or Scylla-Manager, it needs to use the host network and not the Docker network.
To do that run ./start-all.sh with the -l flag. For example:

```sh
./start-all.sh -l -d prometheus-data
```

### Configure rsyslog on each Scylla node

generates metrics and alerts from logs. To get full functionality, you should use [rsyslog](https://www.rsyslog.com/). Scylla Monitoring Stack will act as an additional rsyslog server.
Scylla Monitoring Stack collects Scylla logs using Loki and generates metrics and alerts based on these logs.
To use this feature, you need to direct logs from each Scylla node to Loki.
The recommended method to do this is by using [rsyslog](https://www.rsyslog.com/), where Scylla Monitoring Stack (Loki) acts as an additional rsyslog server.
.. note:: Scylla can send logs to more than one log collection service.

**Prerequisite**, make sure rsyslog is installed and running. If rsyslog is not installed, follow the installation [instruction](https://www.rsyslog.com/doc/v8-stable/installation/index.html).

Add scylla’s rsyslog configuration file. Add the file: `/etc/rsyslog.d/scylla.conf`.

If Scylla Monitoring Stack IP is 10.0.0.1, the file should look like

```sh
if $programname ==  'scylla' then @@10.0.0.1:1514;RSYSLOG_SyslogProtocol23Format
```

Restart rsyslog for the configuration to take effect.

```sh
systemctl restart rsyslog
```

## View Grafana Dashboards

Point your browser to `your-server-ip:3000`
By default, Grafana authentication is disabled. To enable it and set a password for user admin use the `-a` option.

Iframe embedding is disabled by default. Use `--allow-embedding`, set `GF_SECURITY_ALLOW_EMBEDDING=true` in the environment or in `env.sh`, to enable it. Cross-origin embedding requires HTTPS.
See [The start-all.sh Command](https://monitoring.docs.scylladb.com/stable/install/start-all.md) for details.


# nodeCLErrors.md

# Some operation failed due to unsatisfied consistency level

ScyllaDB uses data replication, which means that a query is sent to a coordinator node sends the query to the replica nodes (the nodes that hold and persist the replicated data). The coordinator collects the replies and returns an answer. A query [Consistency Level](https://docs.scylladb.com/glossary/#term-consistency-level-cl), determines the number of replicas that must reply before the coordinator returns an answer.

For example, if the data is replicated to 3 nodes (AKA replication factor 3) and the Consistency Level is quorum, the coordinator will wait for 2 replies before returning the answer.

When one or more nodes are down or unreachable, the Coordinator may fail with a Consistency Level Error because it cannot reach the required consistency level.


# nodeIOErrors.md

# I/O Errors can indicate a node with a faulty disk

I/O Errors are errors that originate from the hardware, most likely from a faulty disk. If this occurs, check the specific node’s hardware.


# nodeLocalErrors.md

# Some operations failed on the replica side

ScyllaDB uses data replication, which means that a query that is sent to a coordinator node sends the query to the replica nodes (the nodes that actually hold the replicated data). The coordinator then collects the replies and returns an answer.

An error on the replica side means that data failed to be persistent on that node, leaving data at risk. Check the node to identify the reasons for the errors.


# nonBalancedcqlTraffic.md

# CQL queries are not balanced among shards

For optimal performance, data and queries should be distributed evenly across nodes and shards. If some shards are getting more traffic they could become a bottleneck for Scylla.

There could be multiple explanations for these performance issues: data-model, non-prepared statement, or driver.

## Blog post link

[https://www.scylladb.com/2019/08/20/best-practices-for-data-modeling/](https://www.scylladb.com/2019/08/20/best-practices-for-data-modeling/)


# preparedCacheEviction.md

# Prepared statements cache eviction

Typically, prepared statements are prepared once and then used for a long period. Prepared statements are stored in the cache and only get evicted if they are not used.
If the prepared statements cache does get evicted, it’s an indication that something is wrong.
The two main sources are:

* A prepared statement that contains a field value, creating such a statement will result in a new prepared statement being stored each time, which defies the purpose of it.
* The prepared statements cache might be too small for the number of prepared statements.


# start-all.md

# The start-all.sh Command

Scylla Monitoring Stack is container-based. The simplest way to configure and start the monitoring is with the start-all.sh command.

The start-all.sh script is a small utility that sets the dashboards and starts the containers with the appropriate configuration.

## General Options

**-h, –help** Help, Print the help, and exit.

**–version** print the current Scylla-Monitoring stack version, and exit.

**-l** local. Use the host network. This is important when one of the containers needs access to an application that runs on the host.
For example, when Scylla Manager runs on the localhost next to the monitoring.
Because the monitoring applications run inside containers by default, their local IP address (127.0.0.1) is the container’s local IP address.
You cannot use port mapping when using the `-l` flag

**-A bind-to-ip-address** Bind the listening-address to an explicit IP address.

**-D encapsulate docker param** Allows passing additional parameters to all the docker containers. For example, to start the monitoring containers with a specific network driver use -D “–net=network_name”.

**–auto-restart** When set, Docker will automatically restart all the services inside the containers in case of a failure.

**–compose** This is an experimental option. When set, it initiates a two-stage process. Firstly, a `docker-compose.yml` file is created along with an `.env` file. Following this, the script executes `docker-compose up`. Please note that having docker-compose is a prerequisite.

**–scrap scrap-interval** Allows changing Prometheus scrap interval.

**–quick-startup** When set, the script will not validate that each of the processes start correctly. The benefit is a quicker startup time. The lack of validation makes handling errors harder, use at your own risk.

**–alternator** When set, the monitoring stack starts with Alternator as the main dashboard.

**–archive /path/to/prometheus/data** Run the monitoring stack using downloaded data.

**–vector-search /path/to/vector-search/vector_search_servers.yml** Read metrics from a vector-search service machine. This will also include the
vector-search service dashboard. When used with –target-directory, the file path
is ignored and the file is expected to be named vector_search_servers.yml.

## Grafana Related Commands

**-G path/to/grafana data-dir** Use an external directory for the Grafana database.
This flag places the Grafana data directory outside of its container and, by doing that, makes it persistent.
This is only important if you are creating your own dashboards using the Grafana GUI and wish to keep them.
If not used, each run of the containers will clear all of Grafana’s information.

**-v comma-separated versions** Each Scylla version comes with its own set of dashboards. By default, Grafana starts with the two latest versions. The `-v` flag allows specifying a specific version or versions.

**-M scylla-manager version** Each Scylla-Manager version has its own dashboard. By default, Grafana starts with the latest Scylla Manager version.  The `-M` flag allows specifying a specific version.

**-j dashboard** Allows adding dashboards to Grafana. Multiple parameters are supported.

**-c Grafana environment variable** Use this parameter to override Grafana’s configuration settings.  The `-c` flag allows adding an environment variable to Grafana and, by doing so, alters its configuration.
How Grafana [Configuration](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#override-configuration-with-environment-variables) work. For example to enable the new navigation, you add -c ‘GF_FEATURE_TOGGLES_ENABLE=newNavigation’ to start-all.sh.

**-g Grafana port** Override the default Grafana port. This is done using port mapping. Note that port mapping does not work when using the host network.

**-a admin password** Allows specifying the admin password.

**-Q Grafana anonymous role** By default, anonymous users have admin privileges. That means they can create and edit dashboards. The `-Q` flag changes this behavior by setting the role privileges to one of Admin, Editor, or Viewer.

**–allow-embedding** Enable embedding Grafana dashboards in iframes. Sets the required cookie security settings for cross-origin embedding.

**–disable-embedding** Disable iframe embedding and use Grafana’s default cookie security settings (`X-Frame-Options: deny`). This is the default.

### Embedding Grafana Dashboards

By default, iframe embedding is disabled. To enable it, use one of:

* The `--allow-embedding` command-line flag
* Environment variables (`GF_SECURITY_ALLOW_EMBEDDING`, `GF_SECURITY_COOKIE_SECURE`, `GF_SECURITY_COOKIE_SAMESITE`)
* An `env.sh` file in the monitoring directory (sourced by `start-all.sh`)

Example `env.sh`:

```shell
GF_SECURITY_ALLOW_EMBEDDING=true
```

When embedding is enabled, the monitoring stack configures Grafana with:

```ini
[security]
allow_embedding = true
cookie_secure = true
cookie_samesite = none
```

These map to the following environment variables in the Grafana container:

```shell
GF_SECURITY_ALLOW_EMBEDDING=true
GF_SECURITY_COOKIE_SECURE=true
GF_SECURITY_COOKIE_SAMESITE=none
```

When embedding dashboards on a different site, Grafana must be served over HTTPS. `cookie_secure` and `cookie_samesite = none` are required for cross-origin iframes in modern browsers.

To embed a dashboard:

```html
<iframe src="https://your-grafana-host/d/d/<uid>/<slug>?orgId=1&kiosk"
        width="100%" height="600"></iframe>
```

### Grafana LDAP support

Grafana supports [LDAP](https://grafana.com/docs/grafana/latest/auth/ldap/) for authentication and authorization.

Use the `-P` flag to supply an LDAP configuration file.

**-P ldap-config-file**

## Prometheus Related Commands

**-d path/to/data-dir** Use an external directory for the Prometheus data directory.
This flag places the Prometheus data directory outside of its container and by doing that makes it persistent.

#### NOTE
Specifying an external directory is important for systems in production. Without it,
every restart of the monitoring stack will result in metrics lost.

**-p prometheus-port** Override the default Prometheus port, this is done using port mapping, note that port mapping does not work when using the host network.

**-b command-line options** Allow adding command-line options that will be passed to the Prometheus server.

**-s scylla-target-file** Specify the location of the Scylla target files. This file contains the IP addresses of the Scylla nodes.

**-n node-target-file** Scylla Monitoring Stack collects OS metrics (Disk, network, etc.) using an agent called node_exporter. By default, Scylla Monitoring Stack assumes that there is a node_exporter running beside each Scylla node, for situations that this is not the case, for example, Scylla runs inside a container and the relevant metrics are of the host machine, it is possible to specify a target file for the node_exporter agents.

**-N manager target file** Specify the location of the Scylla Manager target file.

**-R prometheus-alert-file** By default Prometheus alert rules are found in `prometheus.rules.yml` in the `prometheus` directory. The `-R` flag allows specifying a different location.

**-L manager-address** Using Scylla Manager **Consul** API to resolve the servers’ IP address. When using this option, Prometheus will ignore the target files even if they are explicitly passed in the command line.

**–evaluation-interval duration** Override the default recording rules evaluation-interval.

**–no-cas** An optimization for users who do not use cas, Prometheus will drop all cas related metrics while scrapping

**–no-cdc** An optimization for users who do not use cdc, Prometheus will drop all cdc related metrics while scrapping

**–target-directory** Bind a target directory instead of specific files.
This is safer when replacing files while a container is running.
When using this option, the following files need to be added explicitly:
scylla_servers.yml, node_exporter_servers.yml, scylla_manager_agents.yml,
scylla_manager_servers.yml, and vector_search_servers.yml (if using the vector service).

### Prometheus Retention Period

Prometheus retention period is set for two weeks by default. A common request is how to set it to something else.
It is also an opportunity to demonstrate how to set a Prometheus specific command line option.
Prometheus storage configuration is covered [here](https://prometheus.io/docs/prometheus/latest/storage/#operational-aspects).

For example to set the retention time to 30 days add `-b "--storage.tsdb.retention.time=30d"` to the `start-all.sh` command

## Alert Manager

The Alertmanager handles the alerts and takes the following parameters:

**-m alertmanager-port** Override the default Alertmanager port, this is done using port mapping, note that port mapping does not work when using the host network.

**-r alert-manager-config** By default, the Alertmanager takes its configuration from `rule_config.yml` in the `prometheus` directory. The `-r` flag overrides it to another file.prometheus

**-C alertmanager-commands** Allows adding an arbitrary command line to the alertmanager container starting command.


# systemOverload.md

# System Overload

There could be multiple indications that a system is overloaded:

* Timeouts
* Requests shed - Requests are shed (dropped) when the system cannot process requests fast enough.
* CPU at 100% when no background process (like compaction or repair) runs.
* Queues are getting filled.

If you ruled out data-model problems and hardware failure, this could indicate you need to scale the system.


# thanos.md

# Using Thanos as Data Source With Scylla Monitoring Stack

Scylla-Monitoring uses [Prometheus](https://prometheus.io/) for metrics collection, which works out-of-the-box, but Prometheus does have limitations.
[Thanos](https://thanos.io/)  is an opensource solution which when used on top of Prometheus, provides additional functionalities such as:

* High-availability.
* Horizontal scaling.
* Backup.

The benefit is that with Thanos’ flexible design you can use some or all of these features depending on your requirements.

The rest of this document describes how to place Thanos in front of a multiple Prometheus servers and act as a Grafana datasource instead of Prometheus.

## Using Thanos As a Prometheus Aggregator

There are a few reasons why you would need multiple Prometheus servers: if the total number of your time series reaches millions you can reach the limit of a single Prometheus server capacity.
Sometimes it is also useful to limit the traffic between data centers, so you can have a Prometheus server per DC.

### Prometheus Configuration

We will assume you have two Prometheus servers running.

1. If you are running Prometheus using a container, you should use an **external** data directory, make sure it is reachable by other containers.
2. You will need to add the –web.enable-lifecycle flag to your Prometheus command-line option.

### Thanos sidecar

The Thanos sidecar is an agent that read from a local Prometheus. Thanos uses a single docker container for different uses, the container would act
differently based on the command line it gets.
You will need a sidecar for each of your Prometheus servers.
A docker command looks like:

```shell
docker run -d \
 -v /path/to/prom/dir:/data/prom:z \
 -i --name sidecar thanosio/thanos \
 sidecar \
 --grpc-address=0.0.0.0:10911 \
 --grpc-grace-period=1s \
 --http-address=0.0.0.0:10912 \
 --http-grace-period=1s \
 --prometheus.url=http://prometheus-ip:9090 \
 --tsdb.path=/data/prom \
 -p 10912:10912 \
 -p 10911:10911
```

After you run the sidecar you should be able to reach it from your browser at: `http://{ip}:10912`

### Thanos query

Thanos query is the aggregator, it expose a Prometheus like API and read from multiple thanos stores (in this case the Thanos stores are the sidecars).
You run Thanos query together with Scylla . Assuming that you have two sidecars running on IP addresses: ip1 and ip2,
Start the container  by running:

```shell
docker run -d \
 --name thanos -- thanosio/thanos \
   query \
   --debug.name=query0 \
   --log.level=debug \
   --grpc-address=0.0.0.0:10903 \
   --grpc-grace-period=1s \
   --http-address=0.0.0.0:10904 \
   --http-grace-period=1s \
   --query.replica-label=prometheus \
   --store={ip1}:10911 --store={ip2}:10911
```

After you run Thanos query, you can connect to its HTTP server, in the above example at `http://{ip}:10903`

### Update Scylla Data source

The last step is to update the Grafana data source to read from the local Thanos instead of from Prometheus. Edit grafana/datasource.yml
and replace DB_ADDRESS with {ip}:10903 (The IP address could be of the container as long as it is reachable).

The file you edit is a template file that replaces the file Grafana uses, next time you start.

Restart the Scylla Monitoring Stack it should now use Thanos.


# updating-dashboard.md

# Adding and Modifying Dashboards

This document explains how to update or create Grafana dashboards for the Scylla Monitoring Stack.

It covers dashboard templates and how to modify them.

> * [General Limitations](#general-limitations)
>   * [Consistency Between Restarts](#consistency-between-restarts)
>   * [Consistency Between Upgrades](#consistency-between-upgrades)
> * [Using Templated Dashboards](#using-templated-dashboards)
>   * [The Template Class System](#the-template-class-system)
>   * [Panel Example](#panel-example)
>   * [Grafana Formats and Layouts](#grafana-formats-and-layouts)
>   * [Generating the dashboards from templates (generate-dashboards.sh)](#generating-the-dashboards-from-templates-generate-dashboards-sh)
>   * [Validation](#validation)

## General Limitations

Scylla Monitoring Stack uses Grafana for its dashboards.
The dashboards are provisioned from files and are stored in the Grafana internal storage.
There are two potential consistency issues, covered below.

### Consistency Between Restarts

By default, the Grafana internal storage is within the container. That means that whenever you restart the Scylla Monitoring Stack (explicitly when restarting Grafana) any local changes will not be persisted.
If you are making changes and saving changes from the GUI make sure to configure an external directory for Grafana.

### Consistency Between Upgrades

As mentioned earlier, the dashboards are provisioned from files, this means that when the files are changed, any changes stored locally will be overridden. For this reason, do not make permanent changes to a dashboard, or your changes eventually will be lost.

#### NOTE
 You can save a dashboard change you made from the GUI, but it can be overridden. This should be avoided.

At large, we suggest maintaining your dashboards as files, as Scylla Monitoring does.

## Using Templated Dashboards

Scylla Monitoring uses dashboard templates as we found the Grafana dashboards in JSON format to be too verbose to be maintainable.

Each element in the dashboard file (Each JSON  object) contains all of its attributes and values.

For example a typical graph panel would look like this:

```json
{
    "aliasColors": {},
    "bars": false,
    "datasource": "prometheus",
    "editable": true,
    "error": false,
    "fill": 0,
    "grid": {
        "threshold1": null,
        "threshold1Color": "rgba(216, 200, 27, 0.27)",
        "threshold2": null,
        "threshold2Color": "rgba(234, 112, 112, 0.22)"
    },
    "gridPos": {
        "h": 6,
        "w": 10,
        "x": 0,
        "y": 4
    },
    "id": 2,
    "isNew": true,
    "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": false,
        "total": false,
        "values": false
    },
    "lines": true,
    "linewidth": 2,
    "links": [],
    "nullPointMode": "connected",
    "percentage": false,
    "pointradius": 5,
    "points": false,
    "renderer": "flot",
    "seriesOverrides": [
        {}
    ],
    "span": 5,
    "stack": false,
    "steppedLine": false,
    "targets": [
        {
            "expr": "sum(node_filesystem_avail) by (instance)",
            "intervalFactor": 1,
            "legendFormat": "",
            "refId": "A",
            "step": 1
        }
    ],
    "timeFrom": null,
    "timeShift": null,
    "title": "Available Disk Size",
    "tooltip": {
        "msResolution": false,
        "shared": true,
        "sort": 0,
        "value_type": "cumulative"
    },
    "transparent": false,
    "type": "graph",
    "xaxis": {
        "show": true
    },
    "yaxes": [
        {
            "format": "percent",
            "logBase": 1,
            "max": 101,
            "min": 0,
            "show": true
        },
        {
            "format": "short",
            "logBase": 1,
            "max": null,
            "min": null,
            "show": true
        }
    ]
}
```

As you can imagine, most panels would have similar values.

To reduce the redundancy of the Grafana JSON format, we added dashboard templates.

### The Template Class System

The Scylla Monitoring Stack dashboard templates use a `class` attribute that can be added to any JSON object in a template file.
The different classes are defined in a file.

The `class` system resembles CSS classes. It is hierarchical, so a `class` type definition can have a `class` attribute and
it would inherit that class attributes, the inherited class can add or override inherited attributes.

In the template file, you can also add or override attributes.

The Scylla Monitor generation script, uses the types.json file and a template file and creates a dashboard.

When generating dashboards, each class will be replaced by its definition.

For example, a row in the type.json is defined as:

```json
{
 "base_row": {
     "collapse": false,
     "editable": true
 },
 "row": {
     "class": "base_row",
     "height": "250px"
 }
 }
```

Will be used like in a template:

```json
{
     "class": "row",
     "height": "150px",
     "panels": [
     ]
}
```

And the output will be:

```json
{
     "class": "row",
     "collapse": false,
     "editable": true,
     "height": "150px",
     "panels": [

     ]
}
```

We can see that the template added the `panels` attribute and that it overrides the `height` attribute.

### Panel Example

Consider the following example that defines a row inside a dashboard with a graph
panel for the available disk size.

```json
{
     "class": "row",
     "panels": [
         {
             "class": "bytes_panel",
             "span": 3,
             "targets": [
                 {
                     "expr": "sum(node_filesystem_avail) by (instance)",
                     "intervalFactor": 1,
                     "legendFormat": "",
                     "metric": "",
                     "refId": "A",
                     "step": 1
                 }
             ],
             "title": "Available Disk Size"
         }
     ]
}
```

In the example, the bytes_panel class generates a graph with bytes as units (that would mean that your
Y axis units would adjust themselves to make the graph readable (i.e. GB, MB, bytes, etc’).

You can also see that the span attribute is overridden to set the panel size.

To get a grasp of the difference, take a look at the Grafana panel example and see how it looks originally.

### Grafana Formats and Layouts

The Grafana layout used to be based on rows, where each row contained multiple panels.
Each row would have a total of 12 panels and if the total span of the panels was larger than 12, it would
break them into multiple lines. This is no longer the case.

Starting from  Grafana version 5.0 and later, rows were no longer supported, they were replaced with a layout that uses
absolute positions (i.e. X,Y, height, width).

The server should be backward compatible, but we’ve found it had issues with parsing it correctly.
More so, absolute positions are impossible to get right when done by hand.

To overcome these issues, the dashboard generation script will generate the dashboards in the Grafana version 5.0 format.
In the transition, rows will be replaced with a calculated absolute position.

The panel’s height will be taken from their row. The span attribute is still supported as is row height.

You can use the gridPos attribute which is a Grafana 5.0 format, but unlike Grafana, you can use partial attributes.

gridPos has the following attributes:

```json
{
   "x": 0,
   "y": 0,
   "w": 24,
   "h": 4
 }
```

When using Scylla’s template you don’t need to supply all of the attributes, so for example to specify that a row is 2 units high you can use:

```json
{
   "gridPos": {
      "h": 2
    }
}
```

### Generating the dashboards from templates (generate-dashboards.sh)

#### Prerequisite

Python 3
pyyaml

make_dashboards.py is a utility that generates dashboards from templates or helps you update the templates when working in reverse mode (the -r flag).

Use the -h flag to get help information.

You can use the make_dashboards.py to generate a single dashboard, but it’s usually easier to use the
generate-dashboards.sh wrapper.

When you’re done changing an existing dashboard template, run the generate-dashboards.sh with the current version,
to replace your existing dashboards.

For example, if you are changing a dashboard in Scylla Enterprise version 2020.1 run:

`.\generate-dashboards.sh -v 2020.1`

#### NOTE
generate-dashboards.sh will update the dashboards in place, there is no need for a restart for the changes to take effect, just refresh the dashboard.

### Validation

After making changes to a template, run the `generate_generate-dashboards.sh` and make sure that it ran without any errors.

Refresh your browser for changes to take effect.
Make sure that your panels contain data, if not, maybe there is something wrong with your `expr` attribute.


# upgrade-guide-from-monitoring-1.x-to-monitoring-2.x.md

# Upgrade Guide - Scylla Monitoring 1.x to Scylla Monitoring 2.x

This document is a step by step procedure for upgrading ScyllaDB Monitoring Stack from version 1.x to 2.x

Scylla monitoring stack uses [Prometheus](https://prometheus.io) as its metrics database. The main differences between Scylla Monitoring 1.x and 2.x are moving from Prometheus version 1.x to Prometheus 2.x.
Since Prometheus is not backward compatible between these two versions, the upgrade procedure consists of running the two monitoring stack, old and new, in **parallel**, allowing the new stack to read metrics from the old one. This procedure will enable you to migrate your Scylla monitoring stack without losing historical metric data in the process.
Once the Prometheus retention time has passed, you can safely remove the old stack.

## Upgrade Procedure

### 1. Upgrade to the latest 1.x version

Before starting the upgrade procedure, make sure you are running the [latest 1.x version](https://github.com/scylladb/scylla-monitoring/releases/)

### 2. Install the new monitoring stack

1. Download the 2.x version from the [release](https://github.com/scylladb/scylla-monitoring/releases) page.
2. Unzip it into a **different** directory. This is important, as Prometheus is not backward compatible and would not be able to use the old data.
3. You can use the server definitions from the old monitoring stack by copying the target files from the old stack to the new one, located on the `prometheus/` directory:
   - scylla_servers.yml
   - scylla_manager_servers.yml
   - node_exporter_servers.yml
4. Start the new monitoring stack. If you are using Docker, make sure you are using `-g` `-p` and `-m` to specify different ports than the old monitoring stack. For example:

```bash
./start-all.sh -g 3001 -p 9091 -m 9094 -d /new-prometheus-data-path
```

#### NOTE
Make sure to use the `-d` option, letting Prometheus keep its data **outside** the Docker container. Fail to do so will result in loss of historical data in case you restart the monitoring stack.

While the **old** monitoring stack keeps working, you can take the **new** stack up and down to make sure everything works correctly. Note that if you are using non-default ports setting, you should use the same for stopping the stack:

```bash
./kill-all.sh -g 3001 -p 9091 -m 9094
```

#### Validation

Make sure the new monitoring stack is working before moving to 2.x as your main system. See that the graphs return data, and nodes are reachable.

### 3. Alerting Rules

Note that alerting rules moved to a yml file format. Make sure that all defined rules are taken.

### 4. Moving to Prometheus 2.x

Monitoring stack Version 2.0 upgrade the Prometheus version from 1.8 to 2.3. This upgrade is not backward compatible.
Prometheus Migration is cover [here](https://Prometheus.io/docs/Prometheus/latest/migration/).
Note that when using the docker containers, besides the data migration, the docker permissions were changed. This means that the permissions of the data directory will no longer work.

Do the following to allow the Prometheus in the new monitoring stack to read historical metrics from the old Prometheus:

#### a. Set the old system

The following steps will stop the **old** monitoring stack from reading new metrics while exposing an API for the **new** monitoring stack to read historical metrics from.

* In the **old** Prometheus prometheus.yml.template file, remove the `alerting`, `scrape_configs`, and `rule_files` sections, keeping only the `external_labels` section.
* Restart the **old** monitoring stack with, `kill-all.sh` followed by `start-all.sh` with command line flag `-b "-web.listen-address=:9111"`.

#### NOTE
After this phase, the **old** monitoring stack will not be updated with new metrics. It will only serve as a data source of historical data for the **new** stack

#### b. Set the new system

The following step will allow the **new** monitoring system to read historical metrics from the old system.

* In the Prometheus prometheus.yml.template file add the following at the end:

```bash
remote_read:
  - url: "http://{ip}:9111/api/v1/read"
```

Where {ip} is the ip of the old system.

* restart the **new** stack

### Validate the upgrade

You should be able to see the graphs on the new stack. Make sure you see the graphs’ history.
By default, the Prometheus retention period is 15 days, so after that period, it is safe to take down the old system and remove the remote_read from the new Prometheus configuration.

## Rollback

In the upgrade procedure, you set up a second monitoring stack. The old monitoring stack continues to work in parallel. To rollback, add back the Prometheus targets in the old system, and take down the new system.

## Related Links

* [Upgrade](https://monitoring.docs.scylladb.com/stable/upgrade/index.md)
* [Prometheus Migration](https://Prometheus.io/docs/Prometheus/latest/migration/)


# upgrade-guide-from-monitoring-2.x-to-monitoring-2.y.md

# Upgrade Guide - Scylla Monitoring 2.x to Scylla Monitoring 2.y

This document is a step by step procedure for upgrading  ScyllaDB Monitoring Stack from version 2.x to 2.y, for example, between 2.0 to 2.1.

## Upgrade Procedure

We recommend installing the new release next to the old one, running both in parallel, and making sure it is working as expected before uninstalling the old version.

Change to the directory you want to install the new Monitoring stack.
Download the latest release:
You can download the .zip or the .tar.

### Install 2.y (The new version)

```bash
wget -L https://github.com/scylladb/scylla-monitoring/archive/scylla-monitoring-2.y.zip
unzip scylla-monitoring-2.y.zip
cd scylla-monitoring-scylla-monitoring-2.y/
```

Replace “y” with the new minor release number, for example, 2.1.zip

### Setting the server’s files

Copy the `scylla_servers.yml` and `node_exporter_servers.yml` from the version that is already installed.

```bash
cp /path/to/monitoring/2.x/prometheus/scylla_servers.yml prometheus/
cp /path/to/monitoring/2.x/prometheus/node_exporter_servers.yml prometheus/
```

### Validate the new version is running the correct version

Starting from Scylla-Monitoring version 2.2, you can run:

```bash
./start-all.sh --version
```

To validate the Scylla-Monitoring version.

### Validate the version installed correctly

To validate that the Monitoring stack starts correctly, first in parallel to the current (2.x) stack.

```bash
./start-all.sh -p 9091 -g 3001 -m 9095
```

Browse to `http://{ip}:9091`
And check the Grafana dashboard

Note that we are using different ports numbers for Grafana, Prometheus, and the Alertmanager.

#### CAUTION
Important: do not use the local dir flag when testing!

When you are satisfied with the data in the dashboard, you can shut down the containers.

#### CAUTION
Important: Do not kill the 2.x version that is currently running.

Use the following command to kill the containers:

```bash
./kill-all.sh -p 9091 -g 3001 -m 9095
```

You can start and stop the new 2.y version while testing.

### Move to version 2.y (the new version)

Note: migrating will cause a few seconds of blackout in the system.

We assume that you are using external volume to store the metrics data.

#### Kill all containers

Follow the instruction on how to kill the 2.y version when in testing mode.

To kill the 2.x version containers, run:

```bash
./kill-all.sh
```

Start version 2.y in normal mode

From the new root of the scylla-monitoring-scylla-monitoring-2.y run

```bash
./start-all.sh -d /path/to/data/dir
```

Point your browser to `http://{ip}:3000` and see that the data is there.

### Rollback to version 2.x

To rollback during the testing mode, kill the 2.y containers as explained above, and the system will continue to operate normally.

To rollback to version 2.x after you completed the moving to version 2.y (as shown above).
Run:

```bash
./kill-all.sh
cd /path/to/scylla-grafana-2.x/
./start-all.sh -d /path/to/data/dir
```

## Related Links

* [Upgrade](https://monitoring.docs.scylladb.com/stable/upgrade/index.md)


# upgrade-guide-from-monitoring-2.x-to-monitoring-3.y.md

# Upgrade Guide - Scylla Monitoring 2.x to Scylla Monitoring 3.x

This document is a step by step procedure for upgrading ScyllaDB Monitoring Stack from version 2.x to 3.x

Switching from Scylla Monitoring 2.x to Scylla Monitoring 3.x is not fully backward compatible.
The changes affect dashboards’ names and metrics.

## Upgrade Procedure

### 1. Validate node_exporter version

Scylla Monitoring uses the node_exporter utility to collect OS-related metrics. By default, Scylla will install node_exporter version 0.17.
If you upgrade in the past from the older Scylla version (before Scylla 2.3), you should verify that you are running the correct node_exporter.
You can do that by running node_exporter –version on the machines running Scylla.

If you are running an older version of node_exporter you can use the helper script node_exporter_install –force that shipped with Scylla to force upgrade.

### 2. Install the new monitoring stack

1. Download the 3.x version from the [release](https://github.com/scylladb/scylla-monitoring/releases) page.
2. Unzip it into a **different** directory.
3. Copy the targets files from the old stack to the new one, located on the `prometheus/` sub-directory:
   - scylla_servers.yml (for example `cp /path/to/old/monitor/prometheus/scylla_servers.yml prometheus/`)
   - scylla_manager_servers.yml

   > #### NOTE
   > The targets files are no longer part of the release, make sure to copy them or the monitoring stack will not start.
4. Stop the old monitoring stack
   > ```bash
   > ./kill-all.sh
   > ```
5. Start the new monitoring stack
   > ```bash
   > ./start-all.sh -d /prometheus-data-path
   > ```

### Validate the upgrade

You should be able to see the graphs on the new stack. Make sure you see the graphs’ history.

Because of the metrics, name change graphs will display correctly but with a different color as of the time of the upgrade.
This is only true for the transition phase.

## Rollback

To rollback, simply kill the new stack, change to the old monitoring stack directory, and start it.

## Related Links

* [Upgrade](https://monitoring.docs.scylladb.com/stable/upgrade/index.md)


# upgrade-guide-from-monitoring-3.x-to-monitoring-3.y.md

# Upgrade Guide - Scylla Monitoring 3.x to Scylla Monitoring 3.y

This document is a step by step procedure for upgrading ScyllaDB Monitoring Stack from version 3.x to 3.y, for example, between 3.0 to 3.0.1.

## Upgrade Procedure

We recommend installing the new release next to the old one, running both in parallel, and making sure it is working as expected before uninstalling the old version.

Change to the directory you want to install the new Monitoring stack.
Download the latest release:
You can download the .zip or the .tar.

### Install 3.y (The new version)

```bash
wget -L https://github.com/scylladb/scylla-monitoring/archive/scylla-monitoring-3.y.zip
unzip scylla-monitoring-3.y.zip
cd scylla-monitoring-scylla-monitoring-3.y/
```

Replace “y” with the new minor release number, for example, 3.0.1.zip

### Setting the server’s files

Copy the `scylla_servers.yml` and `scylla_manager_servers.yml` from the version that is already installed.

```bash
cp /path/to/monitoring/3.x/prometheus/scylla_servers.yml prometheus/
cp /path/to/monitoring/3.x/prometheus/scylla_manager_servers.yml.yml prometheus/
```

### Validate the new version is running the correct version

Starting from Scylla-Monitoring version 2.2, you can run:

```bash
./start-all.sh --version
```

To validate the Scylla-Monitoring version.

### Validate the version installed correctly

To validate that the Monitoring stack starts correctly, first in parallel to the current (3.x) stack.

```bash
./start-all.sh -p 9091 -g 3001 -m 9095
```

Browse to `http://{ip}:9091`
And check the Grafana dashboard

Note that we are using different port numbers for Grafana, Prometheus, and the Alertmanager.

#### CAUTION
Important: do not use the local dir flag when testing!

When you are satisfied with the data in the dashboard, you can shut down the containers.

#### CAUTION
Important: Do not kill the 3.x version that is currently running.

### Killing the new 3.y Monitoring stack in testing mode

Use the following command to kill the containers:

```bash
./kill-all.sh -p 9091 -g 3001 -m 9095
```

You can start and stop the new 3.y version while testing.

### Move to version 3.y (the new version)

Note: migrating will cause a few seconds of blackout in the system.

We assume that you are using external volume to store the metrics data.

#### Kill all containers

At this point you have two monitoring stacks running side by side, you should kill both before
continuing.

Kill the newer version that runs in testing mode by following the instructions on how to [Killing the new 3.y Monitoring stack in testing mode]()
in the previous section

kill the older 3.x version containers by running:

```bash
./kill-all.sh
```

Start version 3.y in normal mode

From the new root of the scylla-monitoring-scylla-monitoring-3.y run

```bash
./start-all.sh -d /path/to/data/dir
```

Point your browser to `http://{ip}:3000` and see that the data is there.

### Rollback to version 3.x

To rollback during the testing mode, follow [Killing the new 3.y Monitoring stack in testing mode]() as explained previously
and the system will continue to operate normally.

To rollback to version 3.x after you completed the moving to version 3.y (as shown above).
Run:

```bash
./kill-all.sh
cd /path/to/scylla-grafana-3.x/
./start-all.sh -d /path/to/data/dir
```

## Related Links

* [Upgrade](https://monitoring.docs.scylladb.com/stable/upgrade/index.md)


# upgrade-guide-from-monitoring-3.x-to-monitoring-4.y.md

# Upgrade Guide - Scylla Monitoring 3.x to Scylla Monitoring 4.y

This document is a step by step procedure for upgrading ScyllaDB Monitoring Stack from version 3.x to 4.y, for example, between 3.9 to 4.0.0.

## Upgrade Procedure

We recommend installing the new release next to the old one. You can run both monitoring stacks in parallel, ensuring it is working as expected before uninstalling the old version.

Change to the directory you want to install the new Monitoring stack.
Download the latest release in the .zip or .tar format.

### Install 4.y (The new version)

```bash
wget -L https://github.com/scylladb/scylla-monitoring/archive/scylla-monitoring-4.y.zip
unzip scylla-monitoring-4.y.zip
cd scylla-monitoring-scylla-monitoring-4.y/
```

Replace “y” with the new minor and patch release number, for example, 4.0.0.zip

#### NOTE
For versions 4.7.\* and above, use the format 4.7.\*.zip, for example:

```bash
wget -L https://github.com/scylladb/scylla-monitoring/archive/4.7.1.zip
unzip 4.7.b.zip
cd scylla-monitoring-4.7.b/
```

### Setting the server’s files

Copy the `scylla_servers.yml` and `scylla_manager_servers.yml` from the version that is already installed.

```bash
cp /path/to/monitoring/3.x/prometheus/scylla_servers.yml prometheus/
cp /path/to/monitoring/3.x/prometheus/scylla_manager_servers.yml.yml prometheus/
```

### Validate the new version is running the correct version

run:

```bash
./start-all.sh --version
```

To validate the Scylla-Monitoring version.

## Running in test mode

This section is optional. It shows you how to run two monitoring stacks side by side. You can skip this section entirely and move to
switching to the new version section.

### Running second monitoring stack

We need to use different ports to run two monitoring stacks in parallel (i.e., the older 3.x version and the new 4.x stack).

```bash
./start-all.sh -p 9091 -g 3001 -m 9095
```

Browse to `http://{ip}:9091` and check the Grafana dashboard.

Note that we are using different port numbers for Grafana, Prometheus, and the Alertmanager.

#### CAUTION
Important: do not use the local dir flag when testing!

When you are satisfied with the data in the dashboard, you can shut down the containers.

#### CAUTION
Important: Do not kill the 3.x version that is currently running!

### Killing the new 4.y Monitoring stack in testing mode

Use the following command to kill the containers:

```bash
./kill-all.sh -p 9091 -g 3001 -m 9095
```

You can start and stop the new 4.y version while testing.

## Migrating

### Move to version 4.y (the new version)

Note: migrating will cause a few seconds of blackout in the system.

We assume that you are using external volume to store the metrics data.

#### Backup

We suggest to copy the Prometheus external directory first and use the copy as the data directory for the new monitoring stack.
Newer Monitoring stack uses newer Promethues versions, and keeping a backup of the prometheus dir would allow you to rollback.

#### Kill all containers

At this point you have two monitoring stacks installed with the older version running.

If you run the new version in testing mode kill it by following the instructions on how to [Killing the new 4.y Monitoring stack in testing mode]()
in the previous section.

kill the older 3.x version containers by running:

```bash
./kill-all.sh
```

Start version 4.y in normal mode

From the new root of the scylla-monitoring-scylla-monitoring-4.y run

```bash
./start-all.sh -d /path/to/copy/data/dir
```

Point your browser to `http://{ip}:3000` and see that the data is there.

### Rollback to version 3.x

To rollback during the testing mode, follow [Killing the new 4.y Monitoring stack in testing mode]() as explained previously
and the system will continue to operate normally.

To rollback to version 3.x after you completed moving to version 4.y (as shown above), run:

```bash
./kill-all.sh
cd /path/to/scylla-grafana-3.x/
./start-all.sh -d /path/to/original/data/dir
```

## Post-installation: Metrics back-filling

Starting from Scylla Monitoring version 3.8, Scylla Monitoring uses Prometheus’s recording rules for performance reasons.
Recording rules perform some of the calculations when collecting the metrics instead of when showing the dashboards.

For example, this is a recording rule that calculates the p99 write latency:

```yaml
- record: wlatencyp99
  expr: histogram_quantile(0.99, sum(rate(scylla_storage_proxy_coordinator_write_latency_bucket{}[60s])) by (cluster, dc, instance, shard, scheduling_group_name, le))
  labels:
    by: "instance,shard"
```

For a transition period, Scylla Monitoring version 3.x has a fall-back mechanism that if those recording rules
are not present data will still be shown.

Scylla Monitoring versions 4.0 and newer will rely only on recording rules.

> #### NOTE
> If you upgrade from a version older than 3.8 without back-filling, latency historical data will not be shown.
![image](upgrade/before-backfilling.png)

The following instructions are only relevant if you are upgrading from a version older than 3.8 to version 4.0 or higher,
or if you are looking at historical data collected before you upgraded to 3.8.

For example, you keep your data (retention period) for a year, and you upgraded to version 3.8 three months ago.

In this example you have recording rules data only for the last three months, to be able to look at older latency
information you will need to back-fill that missing period.

The following instructions are based on the recording rules backfilling section in the [Prometheus documentation](https://prometheus.io/docs/prometheus/latest/storage/#backfilling-for-recording-rules).

### Determine the backfilling period

When you run the backfilling process you need to determine the start time and end time.

#### Determine the start time

The start time is your Prometheus retention time, by default it is set to 15 days. if you are not sure what Prometheus retention time is, you can check by
logging in to your Prometheus server: http://{ip}:9090/status.

If you are running Scylla Monitoring version 3.8 or newer for longer than the retention period, you are done! You can skip the rest of this section.

For the rest of this example, we will assume that your retention time is 360 days.

#### Determine the end time

Typically, you need to back-fill the recording rules when you are using a long retention period, for example, you have a year of retention data,
and you upgraded to Scylla Monitoring 3.8 about three months ago.

If you open the Overview dashboard and look at your entire retention time (in our example 1 year) you will see that while most of the graphs do
show the data, the latency graphs have a missing period, in our example - from the entire year, the latency graph will only show the last three months.

That nine months gap (12 months minus 3) is what we want to fill with back-filling.

The point in time that the graphs start will be your back-filling end time. Check in the graph for the exact time.

### Backfilling Process

#### backup

If you have a long retention period you are using an external directory that holds the Prometheus data, back it up, in case
If you have a long retention period, you are using an external directory that holds the Prometheus data back it up; if something goes wrong in the process, you can revert the process.

To complete the process you will need to restart the monitoring stack at least once. If you are not using an external directory (The `-d`
command-line option) You cannot complete it.

#### Restart the monitoring stack

You need to stop the monitoring stack and run the `stat-all.sh` command with an additional flag:

`-b "--storage.tsdb.allow-overlapping-blocks"`

#### Create the data files

We will use the Promtool utility; it’s already installed for you if you are using the docker container.
You will need the start time and end time for the process, in our example the start time is 360 days ago and the end time is 90 days ago.

The start and end times are in epoc, so you will need to translate the times to epoc.  There are many ways to do this - for example, from the command line.
Run the following command to get the epoc time for 90 days ago: : `echo $((`date +%s` - 3600*24*90))`

Log in to your docker container and run the following (`start` and `end` should be the start and end in epoc time):

```bash
docker exec -it aprom sh
cd /prometheus/data/
promtool tsdb create-blocks-from rules \
--start $start \
--end $end \
--url http://localhost:9090 \
/etc/prometheus/prom_rules/back_fill/3.8/rules.1.yml
```

It will create a `data` directory in the directory where you run it.
The reason to run it under the `/prometheus/data/` is you can be sure Prometheus has write privileges there.

> #### NOTE
> Depending on the time range and the number of cores, the process can take a long time. During testing it took an hour for every week of data,
> for a cluster with a total of 100 cores. Make sure that the creation process is not inerupt. You can split the time range to smaller durations
> (e.g. instead of an entire year, do it a weeks at a time).

#### Copy the data files

Make sure that the process is completed successfully - don’t start this section before you complete the previous sections.

Copy the data files to the Prometheus directory:

```bash
cp data/* .
```

The rules will be evaluated next time Prometheus will perform compaction. You can force it by restarting the server using `docker restart aprom`

Follow the logs `docker logs aprom` to see that the process works as expected. If there are no errors, you should now be able to
see the latency graphs over your entire retention time.

![image](upgrade/after-backfilling.jpg)

## Related Links

* [Upgrade](https://monitoring.docs.scylladb.com/stable/upgrade/index.md)


# upgrade-guide-from-monitoring-4.x-to-monitoring-4.y.md

# Upgrade Guide - ScyllaDB Monitoring 4.x.a to ScyllaDB Monitoring 4.y.b

This document is a step by step procedure for upgrading ScyllaDB Monitoring Stack from version 4.x.a to 4.y.b, for example, between 4.0.0 to 4.0.1.

## Upgrade Procedure

We recommend installing the new release next to the old one, running both in parallel, and making sure it is working as expected before uninstalling the old version.

Change to the directory you want to install the new Monitoring stack.
Download the latest release:
You can download the .zip or the .tar.

### Install 4.y.b (The new version)

```bash
wget -L https://github.com/scylladb/scylla-monitoring/archive/scylla-monitoring-4.y.b.zip
unzip scylla-monitoring-4.y.b.zip
cd scylla-monitoring-scylla-monitoring-4.y.b/
```

Replace “y” with the new minor release number, for example, 4.0.1.zip

#### NOTE
For versions 4.7.\* and above, use the format 4.7.\*.zip, for example:

```bash
wget -L https://github.com/scylladb/scylla-monitoring/archive/4.7.1.zip
unzip 4.7.b.zip
cd scylla-monitoring-4.7.b/
```

### Setting the server’s files

Copy the target files `scylla_servers.yml` and `scylla_manager_servers.yml` from the version that is already installed.

```bash
cp /path/to/monitoring/4.x.a/prometheus/scylla_servers.yml prometheus/
cp /path/to/monitoring/4.x.a/prometheus/scylla_manager_servers.yml.yml prometheus/
```

#### Validate the port numbers

ScyllaDB-monitoring reads from ScyllaDB itself, from node_exporter for OS-related metrics, and from the ScyllaDB Manager agent.

Almost always, those targets use their default ports, and all share the same IP.
If you use the default port number, we recommend using the target file without ports and letting ScyllaDB monitoring add the default port number.
If the ScyllaDB Manager agent and node_exporter are running next to ScyllaDB on the same host (the default installation), use one target file for scylla_server, and the ScyllaDB monitoring will use that file with the correct ports for each target.

#### CAUTION
As of SyllaDB Monitoring 4.5, Specifying the node_exporter file (the -n flag) with the wrong ports will prevent ScyllaDB Monitoring from reading the metrics.

### Validate the new version is running the correct version

Run:

```bash
./start-all.sh --version
```

To validate the Scylla-Monitoring version.

### Validate the version installed correctly

To validate that the Monitoring stack starts correctly, first in parallel to the current (4.x.a) stack.

```bash
./start-all.sh -p 9091 -g 3001 -m 9095
```

Browse to `http://{ip}:9091`
And check the Grafana dashboard

Note that we are using different port numbers for Grafana, Prometheus, and the Alertmanager.

#### CAUTION
Important: do not use the local dir flag when testing!

When you are satisfied with the data in the dashboard, you can shut down the containers.

#### CAUTION
Important: Do not kill the 4.x.a version that is currently running.

### Killing the new 4.y.b Monitoring stack in testing mode

Use the following command to kill the containers:

```bash
./kill-all.sh -p 9091 -g 3001 -m 9095
```

You can start and stop the new 4.y.b version while testing.

### Move to version 4.y.b (the new version)

Note: migrating will cause a few seconds of blackout in the system.

We assume that you are using external volume to store the metrics data.

#### Kill all containers

At this point you have two monitoring stacks running side by side, you should kill both before
continuing.

Kill the newer version that runs in testing mode by following the instructions on how to [Killing the new 4.y.b Monitoring stack in testing mode]()
in the previous section

kill the older 4.x.a version containers by running:

```bash
./kill-all.sh
```

Start version 4.y.b in normal mode

From the new root of the scylla-monitoring-scylla-monitoring-4.y.b run

```bash
./start-all.sh -d /path/to/data/dir
```

Point your browser to `http://{ip}:3000` and see that the data is there.

### Rollback to version 4.x.a

To rollback during the testing mode, follow [Killing the new 4.y.b Monitoring stack in testing mode]() as explained previously
and the system will continue to operate normally.

To rollback to version 4.x.a after you completed the moving to version 4.y.b (as shown above).
Run:

```bash
./kill-all.sh
cd /path/to/scylla-grafana-4.x.a/
./start-all.sh -d /path/to/data/dir
```

## Related Links

* [ScyllaDB Monitoring](https://docs.scylladb.com/operating-scylla/monitoring/)
* [Upgrade](https://monitoring.docs.scylladb.com/stable/upgrade/index.md)