Promscale multi-tenancy for Prometheus

When Promscale executes a PromQL query it can return data for all tenants, or restrict its answers to a particular tenant or set of tenants. This allows a tenant to be given access to a particular Promscale PromQL endpoint for querying its own data, while ensuring that it cannot access other tenant’s data.

Multi-tenancy features:

  • Configure multi-tenancy using headers or external labels
  • Cross-tenant queries in PromQL and SQL
  • Allow data with no tenant information to be written or queried when multi-tenant mode is enabled
  • Restrict the set of valid tenants that a Promscale instance can ingest or query
  • Authorize tenants using

By default, Promscale ingests data without using multi-tenancy. To enable it, start Promscale with the -multi-tenancy flag. This allows Promscale to ingest data from all tenants. To restrict the list of tenants for Promscale ingestion and querying, list the tenant names with -multi-tenancy-valid-tenants=<tenant_names_seperated_by_commas>.

By default, when multi-tenancy is enabled, data is rejected if it does not include a tenant name. To accept data without tenant information, set the -multi-tenancy-allow-non-tenants flag.

note

When you set a CLI flag on a Promscale instance, remember to set it on all of your other Promscale instances as well, if you need them to have the same functionality.

To ingest all incoming tenant data regardless of tenant validation from Prometheus instances, along with non-tenant Prometheus instances:

note

The -multi-tenancy-valid-tenants flag defaults to allow-all.

Promscale accepts tenant information from headers or external_labels. It keeps the tenant name as a __tenant__: <tenant_name> label pair, which it attaches to the series ingested in the write request.

You can supply the tenant name in a header by using it as the value of the TENANT key. For example:

Alternatively, you can specify the header in the Prometheus remote-write configuration file, like this:

External labels

Data from tenants is kept with a __tenant__ label key, so you can use this key to query tenant data. Only tenants that are authorized under the -multi-tenancy-valid-tenants flag can be queried. Additionally, data without a tenant label can be queried only if you apply -multi-tenancy-allow-non-tenants.

If your query needs to be evaluated across multiple tenants, you can use metric_name{__tenant__=~"tenant-A|tenant-B"}.

In this example, Promscale contains data from tenant-A, and tenant-C. If Promscale is configured with -multi-tenancy-valid-tenants=tenant-A,tenant-B you can perform these PromQL queries:

  • metric_name{__tenant__=tenant-A} returns metric_name from tenant-A only.
  • metric_name{__tenant__=~tenant-A|tenant-B} returns metric_name from tenant-A and tenant-B.
  • metric_name{__tenant__=~tenant-A|tenant-C} returns metric_name from tenant-A but not from tenant-C. This is because the Promscale instance is configured to return data from only tenant-A and tenant-B.
  • metric_name returns from only tenant-A and tenant-B.

If the same database also contains samples without any tenant label, and -multi-tenancy-allow-non-tenants is applied, you can perform these PromQL queries:

  • metric_name{__tenant__=tenant-A} returns metric_name from tenant-A.
  • metric_name{__tenant__=tenant-A|} returns metric_name from non-tenants and tenant-A.
  • metric_name returns metric_name from non-tenants, tenant-A and tenant-B.
  • metric_name{__tenant__=""} returns metric_name from non-tenants only.
note

If you are querying multiple Promscale instances, you can configure individual instances to selectively authorize any valid tenant for queries, based on your requirements.