Monitoring SingleStore pipelines with Prometheus

[elad.tamary]

Hi,
I saw you expose Prometheus compatible metrics using the memsql_exporter process.
Some related links:

Configure Monitoring · SingleStore Documentation

#SingleStore, #Prometheus, and #Grafana

Prometheus exporter for memsql?

We are heavy users of prometheus and am wondering if anyone has setup a prometheus exporter for memsql. It looks like the docs have some good starting points for what to monitor, but in my searching I couldn’t find a good prometheus exporter. I don’t mind writing one, but don’t want to duplicate work.

However, I couldn’t find pipeline related metrics on http://localhost:9104/metrics and not on http://localhost:9104/cluster-metrics

The only place I can find it is in http://localhost:9104/samples which is not Prometheus compatible structure.

What is the best way to have pipeline_batch information in Prometheus?

[roxannapourzand]

Hello,

Thank you for reaching out and welcome to our community!

For pipeline errors and batches (e.g., anything from information_schema.pipelines_* tables) we don’t yet expose that in the exporter, but it’s on our roadmap and we plan to release it with the next major version of SingleStore.

Our samples endpoint that you found exposes our mv_activities tables, which essentially is a sample of all activities running in the system as well as their respective resource usage (e.g., memory, disk, network… etc). It includes pipeline activities.

All in all, if you are looking for pipeline batch information and other errors, you will have to wait until the next release, but if you looking for resource data for pipeline activities, you can expose that to Prometheus if you are running version 7.3:

Starting in 7.3, we allow you to read samples in SingleStore’s monitoring solution as Prometheus-compatible metrics, but this setting is not enabled by default because SingleStore’s monitoring solution (that uses a SingleStore DB as a source for the monitoring data) expects this data in the non-metric format.

Our documentation is underway for this, but I can show you a sample prometheus.yml of how you would do this (see example below):

The high_cardinality_metrics setting is what dictates whether you want to collect data around query and resource usage (e.g., samples) as well. As the name alludes to, for clusters with large query workloads, the data volume when using high_cardinality_metrics =true may be large, so you should benchmark your disk usage and set a retention policy if necessary.

prometheus.yml

global:
 scrape_interval:     15s
scrape_configs:
 - job_name: 'memsql'
   metrics_path: '/cluster-metrics'
   params:
       high_cardinality_metrics: ['true']
   static_configs:
   - targets: ['<exporter_host>:9104']

[elad.tamary]

Thanks for the useful information.
It is exactly what I searched for .

I suggest to add possibility to decide which parts will be exported.
For example, on my environment memsql_info_schema_activities_extended_cumulative is responsible for 45K out of 50K metrics and I don’t need it at all.
Maybe in future releases you can consider giving the user possibility to decide which group of metrics he wants to export.

[roxannapourzand]

Hi Elad,

Glad to hear it was what you are looking for! As for your feature request - that’s a great idea. I’ve noted this in our tracking system.

It would be great if you could also file it on our public-facing page on the forum so that users can view & vote on the request:

https://www.singlestore.com/forum/c/feature-requests/

Cheers,
Roxanna