ScyllaDB University Live | Free Virtual Training Event
Learn more
ScyllaDB Documentation Logo Documentation
  • Server
  • Cloud
  • Tools
    • ScyllaDB Manager
    • ScyllaDB Monitoring Stack
    • ScyllaDB Operator
  • Drivers
    • CQL Drivers
    • DynamoDB Drivers
  • Resources
    • ScyllaDB University
    • Community Forum
    • Tutorials
Download
ScyllaDB Docs ScyllaDB Monitoring Download and Install Scylla Monitoring Stack Using Thanos as Data Source With Scylla Monitoring Stack

Caution

You're viewing documentation for a previous version of ScyllaDB Monitoring. Switch to the latest stable version.

Using Thanos as Data Source With Scylla Monitoring Stack¶

Scylla-Monitoring uses Prometheus for metrics collection, which works out-of-the-box, but Prometheus does have limitations. Thanos  is an opensource solution which when used on top of Prometheus, provides additional functionalities such as:

  • High-availability.

  • Horizontal scaling.

  • Backup.

The benefit is that with Thanos’ flexible design you can use some or all of these features depending on your requirements.

The rest of this document describes how to place Thanos in front of a multiple Prometheus servers and act as a Grafana datasource instead of Prometheus.

Using Thanos As a Prometheus Aggregator¶

There are a few reasons why you would need multiple Prometheus servers: if the total number of your time series reaches millions you can reach the limit of a single Prometheus server capacity. Sometimes it is also useful to limit the traffic between data centers, so you can have a Prometheus server per DC.

Prometheus Configuration¶

We will assume you have two Prometheus servers running.

  1. If you are running Prometheus using a container, you should use an external data directory, make sure it is reachable by other containers.

  2. You will need to add the –web.enable-lifecycle flag to your Prometheus command-line option.

Thanos sidecar¶

The Thanos sidecar is an agent that read from a local Prometheus. Thanos uses a single docker container for different uses, the container would act differently based on the command line it gets. You will need a sidecar for each of your Prometheus servers. A docker command looks like:

docker run -d \
 -v /path/to/prom/dir:/data/prom:z \
 -i --name sidecar thanosio/thanos \
 sidecar \
 --grpc-address=0.0.0.0:10911 \
 --grpc-grace-period=1s \
 --http-address=0.0.0.0:10912 \
 --http-grace-period=1s \
 --prometheus.url=http://prometheus-ip:9090 \
 --tsdb.path=/data/prom \
 -p 10912:10912 \
 -p 10911:10911

After you run the sidecar you should be able to reach it from your browser at: http://{ip}:10912

Thanos query¶

Thanos query is the aggregator, it expose a Prometheus like API and read from multiple thanos stores (in this case the Thanos stores are the sidecars). You run Thanos query together with Scylla . Assuming that you have two sidecars running on IP addresses: ip1 and ip2, Start the container by running:

docker run -d \
 --name thanos -- thanosio/thanos \
   query \
   --debug.name=query0 \
   --log.level=debug \
   --grpc-address=0.0.0.0:10903 \
   --grpc-grace-period=1s \
   --http-address=0.0.0.0:10904 \
   --http-grace-period=1s \
   --query.replica-label=prometheus \
   --store={ip1}:10911 --store={ip2}:10911

After you run Thanos query, you can connect to its HTTP server, in the above example at http://{ip}:10903

Update Scylla Data source¶

The last step is to update the Grafana data source to read from the local Thanos instead of from Prometheus. Edit grafana/datasource.yml and replace DB_ADDRESS with {ip}:10903 (The IP address could be of the container as long as it is reachable).

The file you edit is a template file that replaces the file Grafana uses, next time you start.

Restart the Scylla Monitoring Stack it should now use Thanos.

Was this page helpful?

PREVIOUS
Minimal Production System Recommendations
NEXT
Using Scylla Monitoring Stack
  • Create an issue
  • Edit this page

On this page

  • Using Thanos as Data Source With Scylla Monitoring Stack
    • Using Thanos As a Prometheus Aggregator
      • Prometheus Configuration
      • Thanos sidecar
      • Thanos query
      • Update Scylla Data source
ScyllaDB Monitoring
  • 4.7
    • 4.9
    • 4.8
    • 4.7
    • 4.6
    • 4.5
    • 4.4
    • 4.3
    • 4.2
    • 4.1
    • 4.0
    • 3.10
    • 3.9
    • 3.8
    • 3.7
    • 3.6
    • 3.5
  • Introduction
  • Download and Install
    • Install
    • The start-all.sh script
    • Deploy without Docker
    • Docker Compose
    • System Recommendations
    • Using Thanos
  • User Guide
    • CQL Optimization Dashboard
    • Advisor
      • Some queries use ALLOW FILTERING
      • Some queries use Consistency Level: ALL
      • Some queries use Consistency Level: ANY
      • Some queries are not token-aware
      • Some SELECT queries are non-paged
      • Some queries are non-prepared
      • Compaction takes lots of memory and CPU
      • Some operation failed due to unsatisfied consistency level
      • I/O Errors can indicate a node with a faulty disk
      • Some operations failed on the replica side
      • CQL queries are not balanced among shards
      • Prepared statements cache eviction
      • System Overload
  • Procedures
    • Cloud Users Datadog integration
    • Datadog Integration
    • Alert Manager
      • Alerting
    • Adding and Modifying Dashboards
    • Upgrade Guides
      • Monitoring 4.x to 4.y
      • Monitoring 3.x to 4.y
      • Monitoring 3.x to 3.y
      • Monitoring 2.x to 3.y
      • Monitoring 2.x to 2.y
      • Monitoring 1.x to 2.x
  • Upgrade
    • Monitoring 4.x to 4.y
    • Monitoring 3.x to 4.y
    • Monitoring 3.x to 3.y
    • Monitoring 2.x to 3.y
    • Monitoring 2.x to 2.y
    • Monitoring 1.x to 2.x
  • Troubleshooting
    • Troubleshooting
    • Troubleshooting Guide for Scylla Manager and Scylla Monitor Integration
  • Reference
    • Support Matrix
    • Interfaces
  • GitHub Project
Docs Tutorials University Contact Us About Us
© 2025, ScyllaDB. All rights reserved. | Terms of Service | Privacy Policy | ScyllaDB, and ScyllaDB Cloud, are registered trademarks of ScyllaDB, Inc.
Last updated on 04 May 2025.
Powered by Sphinx 7.4.7 & ScyllaDB Theme 1.8.6