This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Documentation

Explore how to use Trickster to accelerate your projects. If you’re new to Trickster, check out Where to Place Trickster and the Quickstart.

1 - Quickstart

Try Trickster with Docker Compose and minimal setup.

This composition creates service containers for Prometheus, Grafana, Jaeger, Zipkin, Redis, Trickster and Mockster that together demonstrate several basic end-to-end configurations for running Trickster in your environment with different cache and tracing provider options.

Prerequisites

You should already have the following installed:

Starting the demo

  1. Clone the Trickster Github project.

  2. In the trickster directory, change your working directory to ./examples/docker-compose with the following command:

    cd examples/docker-compose

  3. To run the demo from the demo directory, enter the following command:

    docker-compose up -d

  4. You can interact with each of the services on their exposed ports (as defined in Compose file), or by running docker logs $container_name, docker attach $container_name, etc.

Exploring Trickster

Grafana

Once the composition is running, we recommend exploring with Grafana, at http://127.0.0.1:3000/. Grafana is pre-configured with datasources and a sample dashboard that are ready-to-use for the demo.

Jaeger UI

Jaeger UI is available at http://127.0.0.1:16686, which provides visualization of traces shipped by Trickster and Grafana. The more you use trickster-based data sources in Grafana, the more traces you will see in Jaeger. This composition runs the Jaeger All-in-One container. Trickster ships some traces to the Agent and others directly to the Collector, so as to demonstrate both capabilities. The Trickster config determines which upstream origin ships which traces where.

For a variety of configurations and other bootstrap data, review the various files in the docker-compose-data folder. This might be useful for configuring and using Trickster (or any of these other fantastic projects) in your own deployments. Try adding, removing, or changing some of the trickster configurations in ./docker-compose-data/trickster-config/trickster.yaml and then docker exec docker-compose_trickster_1 kill -1 1 into the Trickster container to apply the changes, or restart the environment altogether with docker-compose restart. Just be sure to make a backup of the original config first, so you don’t have to download it again later.

Example Datasources

The sim-* datasources generate on-the-fly simulation data for any possible time range, so you can immediately use them after starting up the environment. Note, however, that the simulated data is not representative of reality in any way.

The non-sim Prometheus container that backs the prom-* datasources polls the newly-running environment to generate metrics that will then populate the dashboard. Since the Prometheus container only collects and stores metrics while the environment is running, you’ll need to wait a minute or two for those datasources to show any data on the dashoard in real-time.

Getting Real Dashboard Data

Using datasources backed by the real Prometheus and Trickster (the prom-trickster-* datasources), rather than the simulator, to explore the dashboard is more desirable for the demo. It better conveys the shape and nature of the Trickster-specific metrics that might be unfamiliar. However, since there is no historical data in the demo composition, that creates an upfront barrier.

Keeping the dashboard open and auto-refreshing against any trickster-labeled datasource will help to generate real metrics in Trickster, such as request rates, cache hit rates, etc. Prometheus will collect and store those metrics, and the Grafana dashboard will query and render those metrics. So by keeping the demo dashboard open and refreshing, you are helping to generate the very metrics that the dashboard presents, making the demo much more visually useful while being very meta.

In addition to generating metrics, using the trickster-labeled datasources generates traces that are viewable in Jaeger UI, as described above.

Stopping the Demo and Cleaning Up

To stop and remove the demo, run docker-compose down in the ./examples/docker-compose directory.

2 - Getting Started

How to get up and running with Trickster.

2.1 - Where to Place Trickster

Depending upon the size of your existing or planned deployment, there are several placement configurations available. These designs are suggestions based on common usage, and you may find alternative or hybrid placement configurations that make the most sense for your situation, based on the activity of your Dashboard and TSDB instance(s).

Single “Everything”

Single “Everything” is the most common placement model. In this configuration, you have one optional dashboard endpoint, one Trickster endpoint and one HTTP or TSDB endpoint. Behind each endpoint, you may have a single instance or a cluster. Each component is only aware of the other component’s endpoint exposure and not the underlying configuration. This configuration represents a one-for-one-for-one deployment of your Dashboard, Origin, and Trickster endpoints.

Multiple Backends

In a Multiple Backend placement, you have one dashboard endpoint, one Trickster endpoint, and multiple TSDB and/or HTTP endpoints. Trickster is aware of each upstream endpoint and treats each as a unique backend to which it proxies and caches data independently from the others. Trickster routes a request to a specific backend based on Host Header or URL Path in the client request.

This setup may benefit situations where you have one or more static file server origins serving HTML, CSS and JavaScript assets and/or one or more API endpoints, all supporting a common platform.

For Time Series Dashboard acceleration, this is a good configuration to use when you have a single dashboard that displays data about multiple redundant clusters (each with its own TSDB), or when you have a single dashboard representing information about many different kinds of systems. For example, if you operate a “Dashboard as a Service” solution under which many teams use your Dashboard system by designing their own dashboard screens and bringing their own databases, a single Trickster endpoint can be used to accelerate dashboards for all of your customers.

You will need to configure each Trickster-to-TSDB mapping separately in your dashboard application as a separately named TSDB data source. Refer to the multi-origin documentation for configuring multi-origin support in Trickster and Grafana.

In this configuration, be aware that the default ‘memory’ cache may be underpowered depending on the number of customers, as well as the size and number of queries that need to be cached by each customer. Refer to the caches document to select and configure the caching layers as needed to meet your specific situation.

Multi-Trickster

In a Multi-Trickster configuration, you have one dashboard endpoint, multiple Trickster endpoints, and multiple TSDB or HTTP endpoints, with each Trickster Endpoint having a one-to-one mapping to a TSDB/HTTP Endpoint as a pair. This is a good design if Multiple Backends is not performant enough for the amount of activity associated with your solution (e.g., you need more Tricksters). If the Dashboard system owner is different from the TSDB system owner, either party could own and operate the Trickster instance.

2.2 - Configuring Trickster

There are 3 ways to configure Trickster, listed here in the order of evaluation.

  • Configuration File
  • Environment Variables
  • Command Line Arguments

Note that while the Configuration file provides a very robust number of knobs you can adjust, the ENV and CLI Args options support only basic use cases.

Internal Defaults

Internal Defaults are set for all configuration values, and are overridden by the configuration methods described below. All Internal Defaults are described in examples/conf/example.full.yaml comments.

Configuration Files

Trickster accepts a -config /path/to/trickster.yaml command line argument to specify a custom configuration file. The path can also name a directory containing configuration files. If a provided path cannot be accessed by Trickster, it will exit with a fatal error.

When a -config parameter is not provided, Trickster will check for the presence of a config file at /etc/trickster/trickster.yaml and load it if present, or proceed with the Internal Defaults if not present.

Refer to examples/conf/example.full.yaml for full documentation on format of a configuration file.

Multiple Configuration Files

When -config names a file, Trickster loads that file first and then loads supported files from a sibling conf.d directory, if the directory exists. The primary file can select a different include directory:

main:
  config_include_directory: config-parts

A relative config_include_directory is resolved from the directory containing the primary file. An explicitly configured include directory must exist. Only the primary file can set this option; included files cannot redirect configuration discovery.

When -config names a directory, Trickster loads supported files directly from that directory. The directory must contain at least one supported file, and files in this mode cannot set main.config_include_directory.

In both modes, Trickster:

  • Loads only direct regular files whose names do not start with ., with .conf, .yaml, or .yml extensions matched case-insensitively.
  • Loads directory entries in ascending lexical filename order. The primary file, when present, always comes first.
  • Recursively merges mappings. A later scalar or sequence replaces the earlier value, while a later mapping adds to or overrides individual keys in the earlier mapping.
  • Requires each file participating in a multi-source configuration to have a mapping root, one YAML document, and no duplicate keys.

For example, a fragment containing only backends.prometheus.origin_url can change that field without removing the other fields under the prometheus backend. Use null, rather than an empty mapping, when a later file must clear an earlier mapping value.

Configuring Secrets or Sensitive Information

Trickster supports Environment variable substitution in its configuration file where sensitive information is expected.

  • Supported via the following fields:
    • caches[*].redis.password, backends[*].healthcheck.headers, backends[*].cors.headers, backends[*].paths[*].cors.headers, backends[*].paths[*].request_headers, backends[*].paths[*].request_params, backends[*].paths[*].response_headers

Usage ${ENV_VAR_NAME}, example:

caches:
  default:
    redis:
      password: "${MY_REDIS_PW}"

Environment Variables

Trickster will then check for and evaluate the following Environment Variables:

  • TRK_ORIGIN_URL=http://prometheus.example.com:9090 - The default origin URL for proxying all http requests
  • TRK_ORIGIN_TYPE=prometheus - The type of supported backend server
  • TRK_LOG_LEVEL=INFO - Level of Logging that Trickster will output
  • TRK_PROXY_PORT=8480 -Listener port for the HTTP Proxy Endpoint
  • TRK_METRICS_PORT=8481 - Listener port for the Metrics and pprof debugging HTTP Endpoint

Command Line Arguments

Finally, Trickster will check for and evaluate the following Command Line Arguments:

  • -log-level INFO - Level of Logging that Trickster will output
  • -config /path/to/trickster.yaml - See Configuration Files section above
  • -origin-url http://prometheus.example.com:9090 - The default origin URL for proxying all http requests
  • -provider prometheus - The type of supported backend server
  • -proxy-port 8480 - Listener port for the HTTP Proxy Endpoint
  • -metrics-port 8481 - Listener port for the Metrics and pprof debugging HTTP Endpoint

Inbound Listeners

The top-level listeners map configures inbound listeners. Trickster always auto-defines three entries using the existing defaults: default, metrics, and mgmt.

Native MySQL listeners have additional protocol, authentication, TLS, and session-lifecycle requirements. See the MySQL Provider Guide before configuring protocol: mysql. ClickHouse Native listeners use protocol: clickhouse; see the ClickHouse Support Guide for ingress, origin, and TLS options.

listeners:
  default:
    address: ""
    port: 8480
    tls_address: ""
    tls_port: 8483
    connections_limit: 0
    read_header_timeout: 10s
  private_api:
    protocol: http
    address: 127.0.0.1
    port: 9080

backends:
  default:
    listener_names: [default, private_api]
    provider: prometheus
    origin_url: http://prometheus:9090
  private:
    listener_names: [private_api]
    provider: reverseproxy
    origin_url: http://private-origin

listener_names binds a backend to one or more compatible listeners. An ordinary unbound backend uses default; internal routing targets remain unexposed. A backend cannot select the reserved mgmt or metrics listeners, and validation fails for undefined or provider-incompatible listeners.

Each native listener maps to exactly one backend. Multiple HTTP listeners can share a backend, and ClickHouse can bind the same backend to HTTP and ClickHouse Native listeners.

A user-defined listener with no mapped backend is not started and produces a warning. A configured TLS port is enabled only when at least one backend mapped to that listener provides a valid frontend certificate and key in its tls section; otherwise Trickster disables that TLS port and logs a warning.

The top-level frontend section and listener address/port fields under metrics and mgmt remain supported during the compatibility period. Trickster logs deprecation warnings when those legacy listener settings are used. When the same built-in listener is present in listeners, its new configuration takes precedence.

Configuration Validation

Trickster can validate configuration files by running trickster -validate-config -config /path/to/config. Trickster will load the file or directory and exit with the validation result, without running the configuration.

Reloading the Configuration

Trickster can gracefully reload its configuration sources from disk without impacting the uptime and responsiveness of the application.

Trickster supports manual reloads by requesting an HTTP endpoint or sending a SIGHUP (e.g., kill -1 $TRICKSTER_PID) to the Trickster process. It can also poll the effective configuration sources automatically. In all cases, at least one effective configuration source must have changed since the configuration was loaded.

Automatic Config Reload

Trickster can poll its effective configuration sources and reload after a change. This is disabled by default. Set mgmt.auto_reload_interval to a positive duration to enable it:

mgmt:
  auto_reload_interval: 10s

Polling uses the same validation and graceful reload path as SIGHUP and the management endpoint. The interval itself is reloadable, so a successful configuration update can change or disable automatic reloads. Polling is suitable for Kubernetes ConfigMap projected volumes, whose atomic symlink updates are not reliably represented as writes to the mounted file by filesystem notification APIs.

Config Reload via SIGHUP

Once you have made the desired modifications to your config file, send a SIGHUP to the Trickster process by running kill -1 $TRICKSTER_PID. The Trickster log will indicate whether the reload attempt was successful or not.

Config Reload via HTTP Endpoint

Trickster provides an HTTP Endpoint for viewing the running Configuration, as well as requesting a configuration reload.

The reload endpoint is configured by default to listen on address 127.0.0.1 and port 8484, at /trickster/config/reload. These values can be customized, as demonstrated in the example.full.yaml The examples in this section will assume the defaults. Set the port to -1 to disable the reload HTTP interface altogether.

To reload the config, simply make a GET request to the reload endpoint. If an underlying configuration source has changed, or a supported file has been added to or removed from a configured directory, the configuration will be reloaded and the caller will receive a success response. If the configuration sources have not changed, the caller will receive an unsuccessful response, and reloading will be disabled for the duration of the Reload Rate Limiter. By default, this is 3 seconds, but can be customized as demonstrated in the example config file.

If a listener address or port changes, Trickster drains the old listener before starting its replacement. Listeners whose network settings do not change retain their open sockets and receive the refreshed router in place. Removed or newly unused listeners are drained and stopped, while newly mapped listeners are started. The drain period is configurable and defaults to 30 seconds. The Drain Timeout also applies to old log files when a new log filename is provided.

View the Running Configuration

Trickster also provides a http://127.0.0.1:8484/trickster/config endpoint, which returns the yaml output of the currently-running Trickster configuration. The YAML-formatted configuration will include all defaults populated, overlaid with any configuration file settings, command-line arguments and or applicable environment variables. By default, this interface is available only on the management listener. Set mgmt.config_handler_listener to metrics, both, or off to change where it is exposed. This path is configurable as demonstrated in the example config file.

Trickster also provides a sanitized view of the running configuration at http://127.0.0.1:8484/trickster/config/sanitized. If the config_handler_path is customized, append /sanitized to the configured path. The sanitized output deep-copies the running configuration, renames cache, backend, listener, and tracing resources by provider and sequence number (for example, prom-1, prom-2, alb-1, memory-1, listener-1, otlp-1), renames authenticators as auth1, auth2, etc., updates references to those resources in backend, path, ALB, rule, cache, tracing, listener, and authenticator mappings, replaces backend origin_url, Redis endpoint and endpoints, tracing endpoint, and Host-related request rewriter values with example.com, redacts per-path request and response header values, and replaces embedded authenticator users with user1: redacted, user2: redacted, etc. This endpoint is intended for sharing running configuration details in support requests without exposing private infrastructure names, origin endpoints, or user credentials.

2.3 - Trickster 2.0

An All-New Bag of Tricks

Trickster 2.0 is a near-complete rewrite of the project with performance, durability and extensibility in mind. We’ve made major architectural improvements, added new features and improved performance. Here’s a comprehensive overview of it all:

Features

  • Application Load Balancer (ALB): We have a brand new Application Load Balancer available as a backend provider type, with unique and powerful options, including:
    • Fanout and merge data from multiple time series backends into a single response
      • currently supports Prometheus backends
    • Fanout and return the first response received from any backend pool member
    • Fanout and return the response with the newest ‘Last-Modified’ header from all backend pool members
    • Routing a request to a given backend based on Basic Auth username
    • General Round Robin
  • Enhanced Health Checking: Health checking now supports automated health check polling for any backend, and provides a global health status endpoint. Automated health checks for an ALB pool member backend determines whether the ALB will route requests to it or not.
  • Cache Object Chunking: We now support Cache Object Chunking. This optional configuration allows a Time Series dataset to be chunked into multiple cache entries based on a configurable chunk size
    • Also supported by standard Reverse Proxy Cache for cache-chunking objects by a Byte Range size
    • Only the chunked cache entries needed to span the request range are inspected, rather than the entire time series
    • Significantly improves Redis and Filesystem performance of large timeseries records
  • Time Series Backend Request Sharding: We now support Time Series Backend Request Sharding. This optional feature allows a client request destined for a Time Series Backend to be cloned into multiple concurrent requests with different time ranges constituting the full uncached range
    • Backend shard responses + already-cached data are merged into a single downstream response
    • Cache Chunking and TS Backend Request Sharding work together seamlessly and can be used in any combination (on/on, on/off, off/on, off/off, including different cache chunk and request shard sizes).
  • Authenticator: We’ve added a new Authenticator feature so you can guard backends with Basic Auth or ClickHouse Auth
  • Enhanced Rules Engine: The Rules Engine now supports rmatch operations to permit regular expression-based routing against any part of the HTTP request
  • Request Rewriters: You can now chain a collection of request rewriters for more robust request transformation possibilities
  • Cache Purging: We now support purging specific cache items by Key (on the public ports) or Path (on the mgmt port). Read more in the cache documentation
  • Simulated Latency: You can use Trickster for Simulated Latency in lab environments
  • We’ve added support for InfluxDB 2.0 and Flux Query Language, plus InfluxDB 3.x via the native v3 HTTP API (SQL and InfluxQL) and an optional Apache Arrow Flight SQL (gRPC) proxy. See the InfluxDB support doc for details.

Configuration & Security

  • YAML Configuration: Trickster 2.0 uses YAML for configuration instead of TOML
  • Environment Variable Substitution: Environment Variable substitution is now possible in configuration files where sensitive information is expected
    • Supported via the following fields:
      • caches[*].redis.password, backends[*].healthcheck.headers, backends[*].paths[*].request_headers, backends[*].paths[*].request_params, backends[*].paths[*].response_headers, authenticators[*].users
    • Usage: password: ${MY_SECRET_VAR}
  • Request Body Size Limit: A configurable Request Body Size limit has been added for POST, PUT and PATCH requests, with a default of 10MB. Requests with a body size exceeding the limit will receive a 413 Content Too Large response. See Request Body Handling Customizations for more info
  • Zipkin Deprecation: Trickster uses packages provided by the OpenTelemetry project for Distributed Tracing capabilities. These external packages recently deprecated support for exporting spans via Zipkin, therefore Zipkin support has been removed for Trickster 2.0. Trickster retains support for OTLP.

Under the Hood & Performance

  • Common Time Series Format: We now use a common time series format internally for caching all supported TSDB’s, rather than implementing delta + merge algorithms per-provider.
  • New HTTP Request Router: We’ve switched to an all-new, made-for-proxies HTTP Request Router, which is up to 10X faster than the previous one
  • SQL Parser: We’ve switched from Regular Expression matches for SQL-based Time Series Backends to an extensible lexer/parser solution, providing better performance and accuracy
    • ClickHouse backend providers now use the new SQL Parser
  • Race Condition Fixes: We’ve eliminated nearly 100 race conditions and random panics
  • Expanded Compression Support: Compression support now includes options for Brotli and Zstd

Developer & Environment Improvements

  • Package Reorganization: We’ve re-organized many packages in the codebase to be more easily importable by other projects. A future maintenance release will add documentation to the examples folder for using Trickster packages in your own projects, including caching, acceleration and load balancing
  • For Trickster contributors, we have a new Docker Compose for developer environments.
  • CI/CD Enhancements: We have added new CI tools, including better linters and race condition checkers to enforce and ensure ongoing project quality
  • Docker Automation: We’ve updated our Docker automation:
    • The trickster-docker-images repo is now retired and image publishing is handled in the trickster repo
    • All merges to main will now push an image to Docker Hub at trickstercache/trickster:main as well as to trickstercache/trickster:<COMMIT_ID>
      • We no longer push images to the legacy tricksterio and tricksterproxy orgs on DockerHub. Everything is now and only trickstercache 🎉!
    • Images are also now pushed to the GitHub Container Repository as ghcr.io/trickstercache/trickster
  • Helm Charts: The Helm Charts repository is now updated for Trickster 2.0
  • Vendor Directory: We no longer include the vendor directory in the project repository and vendor is now in .gitignore. vendor will continue to be included in Release source tarballs

Documentation & Examples

  • Example Configurations: Example configurations are relocated to the examples directory
  • Docker Compose Demo: The Trickster docker-compose demo has been relocated to the examples directory and updated to use the latest version tags. This is the easiest way to try out Trickster 2.0!

Still to Come

A future Trickster 2.0.x will include the following additional features that didn’t quite make it to the finish line:

  • incorporate ALB examples into the docker-compose demo

Installing

You can build the 2.0 binary from the main branch, download binaries from the Releases page, or use the trickstercache/trickster Docker image tag in containerized environments

3 - Backends

Configuring the upstream origins that Trickster accelerates.

3.1 - Using Multiple Backends with a single Trickster instance

Trickster supports proxying to multiple backends in a single Trickster instance, by examining the inbound request and using a multiplexer to direct the proxied request to the correct upstream origin, in the same way that web servers support virtual hosting.

There are 2 ways to route to multiple backends.

  • HTTP Pathing
  • DNS Aliasing

Basic Usage

To utilize multiple backends, you must craft a Trickster configuration file to be read when Trickster starts up - operating with environment variables or command line arguments only supports accelerating a single backend. The example.full.yaml provides good documentation and commented sections demonstrating multiple backends. The config file should be placed in /etc/trickster/trickster.yaml unless you specify a different path when starting Trickster with the -config command line argument.

Each backend that your Trickster instance supports must be explicitly enumerated in the configuration file. Trickster does not support open proxying.

Example backend configuration:

backends:
  my-backend-01: # routed via http://trickster:8480/my-backend-01/
    provider: prometheus
    origin_url: http://my-origin-01.example.com

  my-backend-02: # routed via http://trickster:8480/my-backend-02/
    hosts: [ my-fqdn-02.example.com ] # or http://my-fqdn-02.example.com:8480/
    provider: prometheus
    origin_url: http://my-origin-02.example.com

  my-backend-03:
    path_routing_disabled: true # only routable via Host header, an ALB, or a Rule
    hosts: [ my-fqdn-03.example.com ] # or http://my-fqdn-03.example.com:8480/
    provider: prometheus
    origin_url: http://my-origin-02.example.com

Default Backend

Whether proxying to one or more upstreams, Trickster has the concept of a “default” backend, which means it does not require a specific DNS hostname in the request, or a specific URL path, in order to proxy the request to a known backend. When a default backend is configured, if the inbound request does not match any mapped backends by path or FQDN, the request will automatically be routed through the default backend. You are probably familiar with this behavior from when you first tried out Trickster with the using command line arguments.

Here’s an example: if you have Trickster configured with a backend named foo that proxies to http://foo/ and is configured as the default backend, then requesting http://trickster/image.jpg will initiate a proxy request to http://foo/image.jpg, without requiring the path be prefixed with /foo. But requesting to http://trickster/foo/image.jpg would also work.

The default backend can be configured by setting is_default: true for the backend you have elected to make the default. Having a default backend is optional. In a single-backend configuration, Trickster will automatically set the sole backend as is_default: true unless you explicitly set is_default: false in the configuration file. If you have multiple backends, and don’t wish to have a default backend, you can just omit the value for all backends. If you set is_default: true for more than one backend, Trickster will exit with a fatal error on startup.

Path-based Routing Configurations

In this mode, Trickster will use a single FQDN but still map to multiple upstream backends by path. This is the simplest setup and requires the least amount of work. The client will indicate which backend is desired in URL Path for the request.

Example Path-based Configuration with Multiple Backends:

backends:
  # backend1 backend
  backend1:
    origin_url: 'http://prometheus.example.com:9090'
    provider: prometheus
    cache_name: default
    is_default: true
  # "foo" backend
  foo:
    origin_url: 'http://influxdb-foo.example.com:9090'
    provider: influxdb
    cache_name: default
  # "bar" backend
  bar:
    origin_url: 'http://prometheus-bar.example.com:9090'
    provider: prometheus
    cache_name: default

Using HTTP Path as the Backend Routing Indicator

The client prefixes the Trickster request path with the Backend Name.

This is the recommended method for integrating with applications like Grafana.

Example Client Request URLs:

DNS Alias Configuration

In this mode, multiple DNS records point to a single Trickster instance. The FQDN used by the client to reach Trickster is mapped to specific backend configurations using the hosts list. In this mode, the URL Path is not considered during Backend Selection.

Example DNS-based Backend Configuration:

backends:
  # backend1 backend
  backend1:
    hosts: # users can route to this backend via these FQDNs, or via `/backend1`
      - 1.example.com
      - 2.example.com
    origin_url: 'http://prometheus.example.com:9090'
    provider: prometheus
    cache_name: default
    is_default: true
  # "foo" backend
  foo:
    hosts: # users can route to this backend via these FQDNs, or via `/foo`
      - trickster-foo.example.com
    origin_url: 'http://prometheus-foo.example.com:9090'
    provider: prometheus
    cache_name: default
  # "bar" backend
  bar:
    hosts: # users can route to this backend via these FQDNs, or via `/bar`
      - trickster-bar.example.com
    origin_url: 'http://prometheus-bar.example.com:9090'
    provider: prometheus
    cache_name: default

Example Client Request URLs:

Note: It is currently possible to specify the same FQDN in multiple backend configurations. You should not do this (obviously). A future enhancement will cause Trickster to exit fatally upon detection at startup.

Disabling Path-based Routing for a Backend

You may wish for a backend to be inaccessible via the /backend_name/ path, and only by Hostname or as the target of a rule or ALB. You can disable path routing by setting path_routing_disabled: true for the backend, as in this example, which requires the Request’s Host header match 1.example.com or 2.example.com in order to be routed to the backend:

backends:
  backend1:
    hosts:
      - 1.example.com
      - 2.example.com
    origin_url: 'http://prometheus.example.com:9090'
    provider: prometheus
    cache_name: default
    is_default: false
    path_routing_disabled: true # this will disable routing through /backend1

3.2 - Authenticator

Trickster 2.x provides an Authenticator capability that allows you to protect Backends with an Authentication layer.

Authenticator resources are defined globally by name, and then mapped into any Backend and/or Path configuration as needed. Authenticator users can be loaded from htpasswd or csv files, or directly in the Trickster config file. You can provide credentials in plaintext, bcrypt, apache md5-script, as well as legacy formats rsa-256 and rsa-512.

Authenticators work with all Backend provider types. Requests are handled by their respective Authenticators before all other Handlers (e.g., Caches, Rules, Request Rewrites, ALB Routes, etc.).

Native MySQL listeners terminate mysql_native_password authentication and have stricter credential-source requirements than HTTP authenticators. See the MySQL Provider Guide.

If a request is routed via a Trickster ALB or Rule Backend through to multiple other Backends - each having different Authenticator configurations - the authentication behavior is currently undefined. In an upcoming Beta release, we will define this use case to either use the Authenticator config (if set) of the very first Backend that handled the request; or to use the first defined Authenticator regardless of how deep into the Backend chain it is.

By default, when an Authenticator handles and successfully authenticates a request, the request’s Auth credentials (e.g., Authorization Header for Basic Auth) are stripped before the request is cloned for any necessary proxying. Setting proxy_preserve: true will preserve these headers instead of stripping them.

Path Protection

When you map an Authenticator to a Backend, all Paths defined for that Backend are protected by the Authenticator. However, you can override the Authenticator on a per-Path basis by including the authenticator_name in a Path config. To bypass a Backend-wide Authenticator in a Path config, use authenticator_name: none. This allows for a few possibilities:

  • A Backend config with a default / Backend-wide Authenticator that has:

    • specific Paths that do not require Authentication
    • specific Paths mapped to different Authenticators than the default

  • A Backend config with no Authenticator defined (so all Paths by default are unprotected) that has:

    • specific Paths mapped to different Authenticators than the default

See the example Backend configs below for more details.

Authenticator Providers

Trickster’s Authenticator feature currently supports Basic Auth and ClickHouse-compatible authentication. It was designed with extensibility in mind should there be value in adding additional Authentication providers.

Basic Auth Provider

Basic Auth is supported by using provider: basic in the Authenticator config.

By setting the showLoginForm: true config (see example_auth_1 in the blob below), Trickster will return a WWW-Authenticate: Basic realm="custom-realm-name" header on any request that requires but fails authentication, causing the login form to pop up. When showLoginForm is not present or non-true, Trickster responds with a 401 Unauthorized but does not ask the Basic Auth login form to show.

The realm attribute value defaults to the Authenticator name (e.g., example_auth_1) but can be overridden with the realm config as in the example.

If the user data changes (e.g. updated users_file contents or updated embedded users list), you must send a SIGHUP or other means to reload the Trickster config before the new user pool is processed.

ClickHouse Auth Provider

ClickHouse Auth is supported by using provider: clickhouse in the Authenticator config.

ClickHouse authentication is the same as Basic Auth, except you can also provide user and password URL params.

Example Authenticator Configs

# NOTE: Required options unrelated to Authenticators have been omitted from this
# example. It does not represent a fully-functioning Trickster Config.
# See the 'examples' directory for working copy/paste config examples.

backends:
  backend01:
    provider: reverseproxy # authenticators work with all backend providers
    authenticator_name: example_auth_1 # protects backend01 with example_auth_1 authenticator
    origin_url: https://example.com
    paths:
      - path: / # all requests are protected by example_auth_1 
        match_type: prefix
        handler: proxy

  backend02:
    provider: reverseproxy # no backend-wide authenticator
    origin_url: https://example.com
    paths:
      - path: / # requests will be allowed without auth except the 2 Paths below
        match_type: prefix
        handler: proxy
      - path: /private/
        authenticator_name: example_auth_2 # example_auth_2 protects this path only
        handler: proxy
      - path: /admin/
        authenticator_name: example_auth_3 # example_auth_3 protects this path only
        handler: proxy

  backend03:
    provider: reverseproxycache
    authenticator_name: example_auth_1 # protects backend03 with example_auth_1 authenticator
    origin_url: https://example.com
    paths:
      - path: / # requests will be challenged by example_auth_1 except the 2 Paths below
        match_type: prefix
        handler: proxy
      - path: /public/
        authenticator_name: none # requests to /public will be allowed without auth
        handler: proxy
      - path: /app/admin/
        authenticator_name: example_auth_2 # example_auth_2 protects this path, not auth_1
        handler: proxy

authenticators:
  # example_auth_1 loads users from a CSV and embeds a supplemental plaintext manifest
  # It also shows the login form to client browsers when login has failed
  example_auth_1:
    provider: basic # http basic auth (required)
    proxy_preserve: true # don't strip auth headers when proxying this request upstream
    users_file: /path/to/user-manifest.csv # optional users source file
    users_file_format: csv # required when users_file is set
    users: # optional embedded users manifest (username: credential)
      user1: red123
    config: # optional provider-specific configs
      showLoginForm: true # with basic auth, causes the browser to show the login form
      realm: custom-realm-name # realm would be example_auth_1 if not overridden here

  # example_auth_2 loads users from an htpasswd file (assumed bcrypted credentials)
  example_auth_2:
    provider: basic
    users_file: /path/to/user-manifest.htpasswd # optional users source file
    users_file_format: htpasswd # required when users_file is set

  # example_auth_3 loads users from the embedded users manifest, credentials already bcrypted
  example_auth_3:
    provider: basic
    users:
      user1: $2y$asf;j2ihj0h8vabjkwdqbv29hq

  # example_auth_4 loads users from the embedded manifest, credentials injected from env,
  # and supports ClickHouse query params (user/password) in addition to Basic Auth
  example_auth_4:
    provider: clickhouse
    users:
      user1: ${USER1_PASSWORD_ENV} # ${ENV_NAME} substitution is supported

3.3 - TLS Support

Trickster supports TLS on both the frontend server and backend clients.

Basics

To enable the TLS server, specify the tls_port, and optionally, the tls_address in one of the listeners of your config file. For example:

listeners:
  # default is built-in and is the default listener for backends
  # it uses these values by default:
  default:
    port: 8480
    tls_port: 8483
    # listen on all interfaces
    tls_address: ''

Note, Trickster will only start listening on a TLS port if at least one origin mapped to the named listener has a valid certificate and key configured.

Each origin section of a Trickster config file can be augmented with the optional tls section to modify TLS behavior for front-end and back-end requests. For example:

backends:
  example: # example backend
    tls:   # TLS settings for example backend
      # frontend configs
      full_chain_cert_path: '/path/to/my/cert.pem'
      private_key_path: '/path/to/my/key.pem'
      # backend configs
      insecure_skip_verify: true
      certificate_authority_paths: [ '/path/to/ca1.pem', '/path/to/ca2.pem' ]
      client_cert_path: '/path/to/client/cert.pem'
      client_key_path: '/path/to/client/key.pem'

Server Configs - used when responding to clients

Each backend contributes up to 1 certificate and key pair, as configured in the TLS section of the backend config (demonstrated above). A listener serves the certificates of all backends mapped to it, selecting per-handshake by SNI (see Certificate Selection (SNI) below), so a single TLS listener can serve many hostnames.

If the path to any configured Certificate or Key file is unreachable or unparsable, Trickster will exit upon startup with an error providing reasonable context.

You may use the same TLS certificate and key for multiple backends, depending upon how your Trickster configurations are laid out. Any certificates configured by Trickster must match the hostname header of the inbound http request (exactly, or by wildcard interpolation), or clients will likely reject the certificate for security issues.

Certificate Selection (SNI)

When a listener has multiple certificates, Trickster selects the certificate for each TLS handshake in this order:

  1. Exact match of the client’s SNI hostname against a certificate’s Subject Alternative Names
  2. Wildcard match (e.g. a *.example.com certificate for foo.example.com)
  3. A linear compatibility scan (covers clients that send no SNI, and IP SANs)
  4. The listener’s first certificate, if nothing else matches

The exact and wildcard lookups are index-based, so per-handshake selection cost is independent of the number of certificates a listener serves.

Automatic Certificate Rotation Detection

Trickster automatically detects when a serving certificate is renewed in place on disk — same file paths, config untouched, as performed by tools like certbot or cert-manager — and hot-swaps the renewed certificate into the live listener. This is entirely independent of configuration reloads: no manual reload, auto_reload_interval, or config change is required.

  • Detection is hybrid: filesystem events (via fsnotify) trigger a near-immediate check where the platform and filesystem support them, and a timer-based poll runs regardless — both as the fallback for deployments where filesystem events don’t work (e.g. some network and FUSE filesystems) and as a self-healing backstop for missed events. If event watching is unavailable, detection silently degrades to poll-only.
  • Every check compares file contents (not modification times or event payloads), so detection works across platforms and through the atomic symlink swaps used by Kubernetes Secret and projected volumes.
  • The certificate, key (and any associated CA bundle) files are watched and validated as one unit: if a poll observes a mid-rotation partial state (e.g. the cert file updated before the key file), the mismatched pair is never served; the last-good pair keeps serving and the change is retried on the next poll.
  • Read errors and invalid content never disable detection: the watcher keeps retrying, logs a warning after sustained failures, and the last-good certificate keeps serving. Deleting the files is treated as a persistent failure, not as certificate removal.
  • Startup behavior is unchanged: an unreadable or invalid configured certificate is still a fatal startup error. Only post-startup source failures are tolerated.

Rotation detection is on by default and is configured per listener. tls_watch_interval sets the fallback/backstop poll cadence (default 30s); filesystem events, where available, apply rotations within moments regardless of the interval. Setting the interval to 0 disables rotation detection entirely (events included):

listeners:
  default:
    tls_port: 8483
    # backstop poll interval for the cert/key files of mapped backends
    # (default 30s); filesystem events accelerate detection where supported.
    # set to 0 to disable automatic rotation detection.
    tls_watch_interval: 30s

Note: automatic rotation detection currently applies to HTTP(S) listeners. Native-protocol listeners (e.g. mysql) pick up rotated certificates on config reload.

Hot Swap and No-Close Semantics

Certificate swaps — whether from a config reload or automatic rotation detection — never restart, drain, or rebind the listener:

  • The certificate is consulted only at handshake time, so established connections (including keep-alive connections and in-flight requests) are untouched by a swap; they continue on the certificate they were handshaken with until they close naturally.
  • Only new handshakes see the new certificate.

Certificate Inventory (mgmt)

The mgmt listener exposes a read-only, per-listener certificate inventory at /trickster/certificates (configurable via mgmt.certificates_handler_path). Each entry reports the certificate’s id, source kind (file, memory or config), common name, subject alternative names, validity window and last-load time. The inventory never includes key material.

Observability

Certificate rotation and inventory are covered by the following metrics: trickster_tls_certificate_expiration_time_seconds, trickster_tls_certificate_last_load_time_seconds, trickster_tls_certificate_swaps_total, trickster_tls_certificate_validation_failures_total, trickster_tls_watcher_errors_total, and trickster_tls_certificate_store_size. See metrics.md and the example alerting rules in examples/alerting for cert-expiry, sustained-validation-failure and watcher-staleness alerts.

Client Configs - used when proxying to an origin

Each backend’s TLS configuration can also configure the https client used for making requests against the origin as demonstrated above.

insecure_skip_verify will instruct the http client to ignore hostname verification issues with the upstream origin’s certificate, and process the request anyway. This is analogous to -k | --insecure in curl.

certificate_authority_paths will provide the http client with a list of certificate authorities (used in addition to any OS-provided root CA’s) to use when determining the trust of an upstream origin’s TLS certificate. In all cases, the Root CA’s installed to the operating system on which Trickster is running are used for trust by the client.

To us Mutual Authentication with an upstream origin server, configure Trickster with Client Certificates using client_cert_path and client_key_path parameters, as shown above. You will likely need to also configure a custom CA in certificate_authority_paths to represent your certificate signer, unless it has been added to the underlying Operating System’s CA list.

3.4 - Health Checks

Trickster Service Health - Ping Endpoint

Trickster provides a /trickster/ping endpoint that returns a response of 200 OK and the word pong if Trickster is up and running. The /trickster/ping endpoint does not check any proxy configurations or upstream origins. The path to the Ping endpoint is configurable, see the configuration documentation for more information.

Upstream Connection Health - Backend Health Endpoints

Trickster offers health endpoints for monitoring the health of the Trickster service with respect to its upstream connection to origin servers.

General Endpoint

The main health check path is /trickster/health, which by default will return a text/plain summary of the backend health. You can request YAML or JSON format using the appropriate Accept header, or by providing a ?json or ?yaml query param.

Backend-Specific Endpoints

Each HTTP backend’s health check path is /trickster/health/BACKEND_NAME. For example, if your backend is named foo, you can perform a health check of the upstream server at http://<trickster_address:port>/trickster/health/foo. Native-protocol backends publish their scheduled status through the general endpoint without registering a synthetic HTTP origin route.

The backend health path prefix /trickster/health/ is customizable. See the example.full.yaml for more info about setting the health_handler_path configuration, or refer to this example:

mgmt:
  # this overrides the default '/trickster/health' to '/-/trickster/health'
  health_handler_path: /-/trickster/health

The behavior of a health request will vary based on the Backend provider, as each has their own health check protocol. For example, with Prometheus, Trickster makes a request to /query?query=up and (hopefully) receives a 200 OK, while for InfluxDB the request is to /ping which returns a 204 No Content.

Supported TSDB Providers are pre-configured in Trickster to perform a suitable health check operation, however these can be overridden in the configuration file.

For non-TSDB Backends, the default behavior is to make a GET request to http://origin_url:port/ and expect a 2xx response. However, all aspects of the Health Check request and expected response are configurable per-Backend.

Native MySQL Health Checks

The MySQL backend uses the same interval scheduler, timeout, transition thresholds, status registry, metrics, reload carryover, and shutdown lifecycle as HTTP health checks. Each probe opens a fresh connection with the backend’s configured origin credentials and TLS policy, completes authentication, executes COM_PING, and closes the connection.

backends:
  mysql1:
    provider: mysql
    origin_url: mysql://health-user:password@mysql.example:3306/analytics
    healthcheck:
      interval: 5s
      timeout: 3s
      failure_threshold: 3
      recovery_threshold: 3

Only interval, timeout, failure_threshold, and recovery_threshold apply to native probes. HTTP verbs, paths, headers, bodies, and expected HTTP response options are configuration errors for MySQL. Diagnostics exposed in health status and logs are limited to sanitized authentication, TLS, timeout, refused-connection, connection, and server-error categories.

Basic Health Check Configuration Example

backends:
  server1:
    provider: reverseproxycache
    origin_url: http://server1
    healthcheck: # all values below are optional
      verb: HEAD
      path: /health

Health Check With Exhaustive Request/Response Options

backends:
  server1:
    provider: reverseproxy
    origin_url: http://server1
    healthcheck: # all values below are optional
      # 
      ## customizing the health check request
      #
      verb: HEAD
      scheme: https
      host: alternate-hostname.example.com
      path: /health
      query: param1=value1&param2=value2
      headers:
        User-Agent: health-check-agent
      # if using a POST or PUT method, you can provide a string body
      # body: "my health check body"
      #
      ## customizing the expected response
      #
      # hc fails if a response takes longer than 1s
      timeout: 1000ms
      # hc fails if the response code is not in the list
      expected_codes: [ 200, 204, 206, 301, 302, 304 ]
      #
      # hc fails if these response headers are not present and have the expected value
      expected_headers:
        X-Health-Check-Status: success
      # hc fails if the stringified response body does not match the expected value
      expected_body: "pass"

See more examples in example.full.yaml.

Health Check Integrations with Application Load Balancers

By default, a Backend will only initiate a health check on-demand, upon receiving a request to its health endpoint.

To facilitate integrations with the Trickster Application Load Balancer provider, additional options provide for 1) timed interval health checks and 2) thresholding for consecutive successful or unsuccessful health checks that determine the backend’s overall health status.

Example Health Check Configuration for use in ALB

backends:
  server1:
    provider: reverseproxy
    origin_url: http://server1
    healthcheck:
      path: /health
      timeout: 1000ms      # timeout should be <= interval
      # for ALB integration:
      interval: 1000ms     # auto-poll health every 1s
      failure_threshold: 3  # backend is unhealthy after 3 consecutive failures
      recovery_threshold: 3 # backend is healthy after 3 consecutive successes

The Prometheus default probe is /api/v1/query?query=up. Some multi-tenant Prometheus gateways reject an unbounded up with 400 bad_data: "too many series found", which keeps the member out of any ALB pool it belongs to. Override healthcheck.query with a bounded expression the backend accepts (for example query=vector(1)) when probing such backends.

Other Ways to Monitor Health

In addition to the out-of-the-box health checks to determine up-or-down status, you may want to setup alarms and thresholds based on the metrics instrumented by Trickster. See metrics.md for collecting performance metrics about Trickster.

4 - Caching

Cache data stores and behaviors common to all of Trickster’s caching modes.

4.1 - Cache Overview

Supported Caches

There are several cache types supported by Trickster

  • In-Memory (default)
  • Filesystem
  • bbolt
  • BadgerDB
  • Redis (basic, cluster, and sentinel)

The sample configuration (examples/conf/example.full.yaml) demonstrates how to select and configure a particular cache type, as well as how to configure generic cache configurations such as Retention Policy.

In-Memory

In-Memory Cache is the default type that Trickster will implement if none of the other cache types are configured. The In-Memory cache utilizes a Golang sync.Map object for caching, which ensures atomic reads/writes against the cache with no possibility of data collisions. This option is good for both development environments and most smaller dashboard deployments.

When running Trickster in a Docker container, ensure your node hosting the container has enough memory available to accommodate the cache size of your footprint, or your container may be shut down by Docker with an Out of Memory error (#137). Similarly, when orchestrating with Kubernetes, set resource allocations accordingly.

Filesystem

The Filesystem Cache is a popular option when you have larger dashboard setup (e.g., many different dashboards with many varying queries, Dashboard as a Service for several teams running their own Prometheus instances, etc.) that requires more storage space than you wish to accommodate in RAM. A Filesystem Cache configuration keeps the Trickster RAM footprint small, and is generally comparable in performance to In-Memory. Trickster performance can be degraded when using the Filesystem Cache if disk i/o becomes a bottleneck (e.g., many concurrent dashboard users).

The default Filesystem Cache path is /tmp/trickster. The sample configuration demonstrates how to specify a custom cache path. Ensure that the user account running Trickster has read/write access to the custom directory or the application will exit on startup upon testing filesystem access. All users generally have access to /tmp so there is no concern about permissions in the default case.

bbolt

The BoltDB Cache is a popular key/value store, created by Ben Johnson. CoreOS’s bbolt fork is the version implemented in Trickster. A bbolt store is a filesystem-based solution that stores the entire database in a single file. Trickster, by default, creates the database at trickster.db and uses a bucket name of ’trickster’ for storing key/value data. See the example config file for details on customizing this aspect of your Trickster deployment. The same guidance about filesystem permissions described in the Filesystem Cache section above apply to a bbolt Cache.

BadgerDB

BadgerDB works similarly to bbolt, in that it is a filesystem-based key/value datastore. BadgerDB provides its own native object lifecycle management (TTL) and other additional features that distinguish it from bbolt. See the configuration for more info on using BadgerDB with Trickster.

Redis

Note: Trickster does not come with a Redis server. You must provide a pre-existing Redis endpoint for Trickster to use.

Redis is a good option for larger dashboard setups that also have heavy user traffic, where you might see degraded performance with a Filesystem Cache. This allows Trickster to scale better than a Filesystem Cache, but you will need to provide your own Redis instance at which to point your Trickster instance. The default Redis endpoint is redis:6379, and should work for most docker and kube deployments with containers or services named redis. The sample configuration demonstrates how to customize the Redis endpoint. In addition to supporting TCP endpoints, Trickster supports Unix sockets for Trickster and Redis running on the same VM or bare-metal host.

Ensure that your Redis instance is located close to your Trickster instance in order to minimize additional roundtrip latency.

In addition to basic Redis, Trickster also supports Redis Cluster and Redis Sentinel. Refer to the sample configuration for customizing the Redis client type.

Trickster supports Redis servers that use TLS encryption by setting use_tls: true in the config. Refer to the sample configuration for more info.

Purging an Item from the Cache

You can purge an item from the cache by making a call to the purge endpoint, as follows:

http://${trickster-address}:${mgmt-port}/trickster/purge/path/${backendName}/${path/to/purge}

For example, if you want to purge /api/v1/labels from backend prom1, a curl might look like:

curl http://localhost:8484/trickster/purge/path/prom1/api/v1/labels

Purging the Full Cache

Full Cache purges should not be necessary, but in the event that you wish to do so, the following steps should be followed based upon your selected Cache Type.

A future release will provide a mechanism to fully purge the cache (regardless of the underlying cache type) without stopping a running Trickster instance.

Purging In-Memory Cache

Since this cache type runs inside the virtual memory allocated to the Trickster process, bouncing the Trickster process or container will effectively purge the cache.

Purging Filesystem Cache

To completely purge a Filesystem-based Cache, you will need to:

  • Docker/Kube: delete the Trickster container (or mounted volume) and run a new one
  • Metal/VM: Stop the Trickster process and manually run rm -rf /tmp/trickster (or your custom-configured directory).

Purging Redis Cache

Connect to your Redis instance and issue a FLUSH command. Note that if your Redis instance supports more applications than Trickster, a FLUSH will clear the cache for all dependent applications.

Purging bbolt Cache

Stop the Trickster process and delete the configured bbolt file.

Purging BadgerDB Cache

Stop the Trickster process and delete the configured BadgerDB path.

Cache Status

Trickster reports several cache statuses in metrics, logs, tracing, and the X-Trickster-Result response header, which are listed and described in the table below.

StatusDescription
kmissThe requested object was not in cache and was fetched from the origin
rmissObject is in cache, but the specific data range requested (timestamps or byte ranges) was not
hitThe object was fully cached and served from cache to the client
phitThe object was cached for some of the data requested, but not all
nchitThe response was served from the Negative Cache
rhitThe object was served from cache to the client, after being revalidated for freshness against the origin
purgeThe cache key was purged as directed by a request or response header
proxy-onlyThe request was proxied 1:1 to the origin and not cached
proxy-errorThe upstream request needed to fulfill an associated client request returned an error
errorTrickster encountered a cache lookup or cache handling error
proxy-hitThe request joined an existing in-flight origin fetch for the same cache key

4.2 - Trickster Caching Retention Policies

Basic HTTP Backends

Trickster will respect HTTP 1.0, 1.1 and 2.0 caching directives from both the downstream client and the upstream origin when determining object cacheability and TTL. You can override the TTL by setting a custom Cache-Control header on a per-Path Config basis.

Cache Object Evictions

If you use a Trickster-managed cache (Memory, Filesystem, bbolt), then a maximum cache size is maintained by Trickster. You can configure the maximum size in number of bytes, number of objects, or both. See the example configuration for more information.

Once the cache has reached its configured maximum size of objects or bytes, Trickster will undergo an eviction routine that removes cache objects until the size has fallen below the configured maximums. Trickster-managed caches maintain a last access time for each cache object, and utilizes a Least Recently Used (LRU) methodology when selecting objects for eviction.

Caches whose object lifetimes are not managed internally by Trickster (Redis, BadgerDB) will use their own policies and methodologies for evicting cache records.

Time Series Backends

For non-time series responses from a TSDB, Trickster will adhere to HTTP caching rules as directed by the downstream client and upstream origin.

For time series data responses, Trickster will cache as follows:

TTL Settings

TTL settings for each Backend configured in Trickster can be customized independently of each other, and separate TTL configurations are available for timeseries objects, and fast forward data. See examples/conf/example.full.yaml for more info on configuring default TTLs.

Time Series Data Retention

Separately from the TTL of a time series cache object, Trickster allows you to control the size of each timeseries object, represented as a count of maximum timestamps in the cache object, on a per origin basis. This configuration is known as the timeseries_retention_factor (TRF), and has a default of 1024. Most dashboards for most users request and display approximately 300-to-400 timestamps, so the default TRF allows users to still recall recently-displayed data from the Trickster cache for a period of time after the data has aged off of real-time views.

If you have users with a high-resolution dashboard configuration (e.g., a 24-hour view with a 1-minute step, amounting to 1440 data points per graph), then you may benefit from increasing the timeseries_retention_factor accordingly. If you use a managed cache (see caches) and increase the timeseries_retention_factor, the overall size of your cache will not change; the result will be fewer objects in cache, with the timeseries objects having a larger share of the overall cache size with more aged data.

Time Series Data Evictions

Once the TRF is reached for a time series cache object, Trickster will undergo a timestamp eviction process for the record in question. Unlike the Cache Object Eviction, which removes an object from cache completely, TRF evictions examine the data set contained in a cache object and remove timestamped data in order to reduce the object size down to the TRF.

Time Series Data Evictions apply to all cached time series data sets, regardless of whether or not the cache object lifecycle is managed by Trickster.

Trickster provides two eviction methodologies (timeseries_eviction_method) for time series data eviction: oldest (default) and lru, and is configurable per-origin.

When timeseries_eviction_method is set to oldest, Trickster maintains time series data by calculating the “oldest cacheable timestamp” value upon each request, using time.Now().Add(step * timeseries_retention_factor * -1). Any queries for data older than the oldest cacheable timestamp are intelligently offloaded to the proxy since they will never be cached, and no data that is older than the oldest cacheable timestamp will be stored in the query’s cache record.

When timeseries_eviction_method is set to lru, Trickster will not calculate an oldest cacheable timestamp, but rather maintain a last-accessed time for each timestamp in the cache object, and evict the Least-Recently-Used items in order to maintain the cache size.

The advantage of the oldest methodology better cache performance, at the cost of not caching very old data. Thus, Trickster will be more performant computationally while providing a slightly lower cache hit rate. The lru methodology, since it requires accessing the cache on every request and maintaining access times for every timestamp, is computationally more expensive, but can achieve a higher cache hit rate since it permits caching data of any age, so long as it is accessed frequently enough to avoid eviction.

Most users will find the oldest methodology to meet their needs, so it is recommended to use lru only if you have a specific use case (e.g., dashboards with data from a diverse set of time ranges, where caching only relatively young data does not suffice).

4.3 - Chunked Caching

Overview

In some caching setups, users may want to increase timeseries_retention_factor or timeseries_ttl forms a given backend to a very large size (e.g., a duration of days or weeks). This can cause issues if the cache provider is filesystem or redis, because the entire time series is loaded to extract even just a few data points. Eventually, this could negate the effects of caching altogether.

To mitigate this, Trickster supports chunking cache data, by splitting large datasets into subdivisions of a configurable maximum size. The chunks are reconstituted upon retrieval. Only the chunks needed to service a client request are accessed, rather than the entire time series cache object.

Chunking can be configured per-cache and applies to both timeseries and byterange data.

Configuration

Chunked caching can be enabled and disabled using use_cache_chunking:

fs1:
    provider: filesystem
    use_cache_chunking: true
    timeseries_chunk_factor: 420
    byterange_chunk_size: 4096

timeseries_chunk_factor determines the maximum extent of timerange chunks, and byterange_chunk_size determines the maximum size of byterange chunks. See Detail for more information.

Detail

Timeseries

Timeseries chunking splits the timeseries to be cached into parts with the same duration, but not necessarily the same literal size.

  • Determine a chunk duration by multiplying the timerange step by timerange_chunk_factor (default 420)
  • Determine the smallest possible extent that is aligned to the epoch along the chunk duration, while containing the entire timeseries
  • To write: Write each chunk size subextent under a subkey
  • To read: Read each subkey and merge the timeseries results

Byterange

Byterange chunking splits the byterange into pieces with the same literal size. There are also some extra steps compared to the timeseries implementation to preserve the integrity of both full and partial responses being cached.

  • Determine a chunk size from byterange_chunk_size (default 4096)
  • Determine a range from the cache read/write request using the provided range or content length
  • Failure to determine a range on write results in an error
  • Failure to determine a range on read will read until the query fails
  • Determine a maximum range aligned along the chunk size that contains the entire byterange
  • To write: Write each chunk size range with RangeParts of all provided ranges cropped to that chunk range, under a subkey
  • To read: Read each subkey and reconstitute a body from RangeParts, if able

Full Example

This example has one Prometheus backend with a memory cache that has chunking enabled. The memory cache uses 380 as its timeseries chunk factor, and doesn’t define a byterange chunk size, so the default of 4096 will be used.

caches:
  mem1:
    provider: memory
    index:
      max_size_objects: 512
      max_size_backoff_objects: 128
    use_cache_chunking: true
    timeseries_chunk_factor: 380

backends:
  prom1:
    latency_max: 150ms
    latency_min: 50ms
    provider: prometheus
    origin_url: 'http://127.0.0.1:9090'
    cache_name: mem1

logging:
  log_level: warn

4.4 - Negative Caching

Negative Caching means to cache undesired HTTP responses for a very short period of time, in order to prevent overwhelming a system that would otherwise scale normally when desired, cacheable HTTP responses are being returned. For example, Trickster can be configured to cache 404 Not Found or 500 Internal Server Error responses for a short period of time, to ensure that a thundering herd of HTTP requests for a non-existent object, or unexpected downtime of a critical service, do not create an i/o bottleneck in your application pipeline.

Trickster supports negative caching of any status code >= 300 and < 600, on a per-Backend basis. In your Trickster configuration file, associate the desired Negative Cache Map to the desired Backend config. See the example.full.yaml, or refer to the snippet below for more information.

The Negative Cache Map must be an all-inclusive list of explicit status codes; there is currently no wildcard or status code range support for Negative Caching entries. By default, the Negative Cache Map is empty for all backend configs. The Negative Cache only applies to Cacheable Objects, and does not apply to Proxy-Only configurations.

For any response code handled by the Negative Cache, the response object’s effective cache TTL is explicitly overridden to the value of that code’s Negative Cache TTL, regardless of any response headers provided by the Backend concerning cacheability. All response headers are left in-tact and unmodified by Trickster’s Negative Cache, such that Negative Caching is transparent to the client. The X-Trickster-Result response header will indicate a response was served from the Negative Cache by providing a cache status of nchit.

Multiple negative cache configurations can be defined, and are referenced by name in the backend config. By default, a backend will use the ‘default’ Negative Cache config, which, by default is empty. The default can be easily populated in the config file, and additional configs can easily be added, as demonstrated below.

The format of a negative cache map entry is 'status_code': ttl.

Example Negative Caching Config

negative_caches:
  default:
    '404': 3s # cache 404 responses for 3 seconds
  foo:
    '404': 3s
    '500': 5s # caches 404 response for 3 seconds, and 500/502 for 5 seconds
    '502': 5s

backends:
  default:
    provider: rpc
    # by default will assume negative_cache_name = 'default'
  another:
    provider: rpc
    negative_cache_name: foo

5 - Object Caching

Accelerating generic HTTP objects with Trickster’s Reverse Proxy Cache.

5.1 - Byte Range Request Support

Trickster’s HTTP Reverse Proxy Cache offers best-in-class acceleration and caching of Byte Range Requests.

Much like its Time Series Delta Proxy Cache, Trickster’s Reverse Proxy Cache will determine what ranges are cached, and only request from the origin any uncached ranges needed to service the client request, reconstituting the ranges within the cache object. This ensures minimal response time for all Range requests.

In addition to supporting requests with a single Range (Range: bytes=0-5) Trickster also supports Multipart Range Requests (Range: bytes=0-5, 10-20).

Fronting Origins That Do Not Support Multipart Range Requests

In the event that an upstream origin supports serving a single Range, but does not support serving Multipart Range Requests, which is quite common, Trickster can transparently enable that support on behalf of the origin. To do so, Trickster offers a unique feature called Upstream Range Dearticulation, that will separate any ranges needed from the origin into individual, parallel HTTP requests, which are reconstituted by Trickster. This behavior can be enabled for any origin that only supports serving a single Range, by setting the origin configuration value dearticulate_upstream_ranges = true, as in this example:

backends:
  default:
    provider: reverseproxycache
    origin_url: 'http://example.com/'
    dearticulate_upstream_ranges: true

If you know that your clients will be making Range requests (even if they are not Multipart), check to ensure the configured origin supports Multipart Range requests. Use curl to request any static object from the origin, for which you know the size, and include a Multipart Range request; like curl -v -H 'Range: bytes=0-1, 3-4' 'http://example.com/object.js'. If the origin returns 200 OK and the entire object body, instead of 206 Partial Content and a multipart body, enable Upstream Range Dearticulation to ensure optimal performance.

This is important because a partial hit could result in multiple ranges being needed from the origin - even for a single-Range client request, depending upon what ranges are already in cache. If Upstream Range Dearticulation is disabled in this case, full objects could be unnecessarily returned from the Origin to Trickster, instead of small delta ranges, irrespective of the object’s overall size. This may or may not impact your use case.

Rule of thumb: If the origin does not support Multipart requests, enable Upstream Range Dearticulation in Trickster to compensate. Conversely, if the origin does support Multipart requests, do not enable Upstream Range Dearticulation.

Disabling Multipart Ranges to Clients

One of the great benefits of using Upstream Range Dearticulation is that it transparently enables Multipart Range support for clients, when fronting any origin that already supports serving just a single Range.

There may, however, be cases where you do not want to enable Multipart Range support for clients (since its paired Origin does not), but need Upstream Range Dearticulation to optimize Partial Hit fulfillments. For those cases, Trickster offers a setting to disable Multipart Range support for clients, while Upstream Range Dearticulation is enabled. Set multipart_ranges_disabled = true, as in the below example, and Trickster will strip Multipart Range Request headers, which will result in a 200 OK response with the full body. Client single Range requests are unaffected by this setting. This should only be set if you have a specific use case where clients should not be able to make multipart Range requests.

backends:
  default:
    provider: reverseproxycache
    origin_url: 'http://example.com/'
    dearticulate_upstream_ranges: true
    multipart_ranges_disabled: true

Partial Hit with Object Revalidation

As explained above, whenever the client makes a Range request, and only part of the Range is in the Trickster cache, Trickster will fetch the uncached Ranges from the Origin, then reconstitute and cache all of the accumulated Ranges, while also replying to the client with its requested Ranges.

In the event that a cache object returns 1) a partial hit, 2) that is no longer fresh, 3) but can be revalidated, based on a) the Origin’s provided caching directives or b) overridden by the Trickster operator’s explicit path-based Header configs; Trickster will revalidate the client’s requested-but-cached range from the origin with the appropriate revalidation headers.

In a Partial Hit with Revalidation, the revalidation request is made as a separate, parallel request to the origin alongside the uncached range request(s). If the revalidation succeeds, the cached range is merged with the newly-fetched range as if it had never expired. If the revalidation fails, the Origin will return the range needed by the client that was previously cached, or potentially the entire object - either of which are used to complete the ranges needed by the client and update the cache and caching policy for the object.

Range Miss with Object Revalidation

Trickster recognizes when an object exists in cache, but has none of the client’s requested Ranges. This is a state that lies between Cache Miss and Partial Hit, and is known as “Range Miss.” Range Misses can happen frequently on Range-requested objects.

When a Range Miss occurs against an object that also requires revalidation, Trickster will not initiate a parallel revalidation request, since none of the client’s requested Ranges are actually eligible for revalidation. Instead, Trickster will use the Response Headers returned by the Range Miss Request to perform a local revalidation of the cache object. If the object is revalidated, the new Ranges are merged with the cached Ranges before writing to cache based on the newly received Caching Policy. If the object is not revalidated, the cache object is created anew solely from the Range Miss Response.

Multiple Parts Require Revalidation

A situation can arise where there is a partial cache hit has multiple ranges that require revalidation before they can be used to satisfy the client. In these cases, Trickster will check if Upstream Range Dearticulation is enabled for the origin to determine how to resolve this condition. If Upstream Range Dearticulation is not enabled, Trickster trusts that the upstream origin will support Multipart Range Requests, and will include just the client’s needed-and-cached-but-expired ranges in the revalidation request. If Upstream Range Dearticulation is enabled, Trickster will forward, without modification, the client’s requested Ranges to the revalidation request to the origin. This behavior means Trickster currently does not support multiple parallel revalidation requests. Whenever the cache object requires revalidation, there will be only 1 revalidation request upstream, and 0 to N additional parallel upstream range requests as required to fulfill a partial hit.

If-Range Not Yet Supported

Trickster currently does not support revalidation based on If-Range request headers, for use with partial download resumptions by clients. If-Range headers are simply ignored by Trickster and passed through to the origin, which can result in unexpected behavior with the Trickster cache for that object.

We plan to provide full support for If-Range as part of Trickster 1.1 or 2.0

Mockster Byte Range

For verification of Trickster’s compatibility with Byte Range Requests (as well as Time Series data), we created a golang library and accompanying standalone application dubbed Mockster. Mockster’s Byte Range library simply prints out the Lorem ipsum ... sample text, pared down to the requested range or multipart ranges, with a few bells and whistles that allow you to customize its response for unit testing purposes. We make extensive use of Mockster in unit testing to verify the integrity of Trickster’s output after performing operations like merging disparate range parts, extracting ranges from other ranges, or from a full body, compressing adjacent ranges into a single range in the cache, etc.

It is fairly straightforward to run or import Mockster into your own applications. For examples of using it for Unit Testing, check out /internal/proxy/engines/objectproxycache_test.go.

5.2 - Collapsed Forwarding

Collapsed Forwarding is feature common among Reverse Proxy Cache solutions like Squid, Varnish and Apache Traffic Server. It works by ensuring only a single request to the upstream origin is performed for any object on a cache miss or revalidation attempt, no matter how many users are requesting the object at the same time.

Trickster has support for two types of Collapsed Forwarding: Basic (default) and Progressive

Basic Collapsed Forwarding

Basic Collapsed Forwarding is the default functionality for Trickster, and works by waitlisting all requests for a cacheable object while a cache miss is being serviced for the object, and then serving the waitlisted requests once the cache has been populated.

The feature is further detailed in the following diagram:

Progressive Collapsed Forwarding

Progressive Collapsed Forwarding (PCF) is an improvement upon the basic version, in that it eliminates the waitlist and serves all simultaneous requests concurrently while the object is still downloading from the server, similar to Apache Traffic Server’s “read-while-write” feature. This may be useful in low-latency applications such as DASH or HLS video delivery, since PCF minimizes Time to First Byte latency for extremely popular objects.

The feature is further detailed in the following diagram:

PCF for Proxy-Only Requests

Trickster provides a unique feature that implements PCF in Proxy-Only configurations, to bring the benefits of Collapsed Forwarding to HTTP Paths that are not configured to be routed through the Reverse Proxy Cache. (See Paths documentation for more info on routing).

The feature is further detailed in the following diagram:

How to enable Progressive Collapsed Forwarding

When configuring path configs as described in Paths Documentation add collapsed_forwarding: progressive in any path config using the proxy or proxycache handlers.

Example:

origins:
  test:
    paths:
      - path: /test_path1/
        match_type: prefix
        handler: proxycache
        collapsed_forwarding: progressive
      - path: /test_path2/
        match_type: prefix
        handler: proxy
        collapsed_forwarding: progressive

See the example.full.yaml for more configuration examples.

How to test Progressive Collapsed Forwarding

An easy way to test PCF is to set up your favorite file server to host a large file(Lighttpd, Nginx, Apache WS, etc.), In Trickster turn on PCF for that path config and try make simultaneous requests. If the networking between your machine and Trickster has enough bandwidth you should see both streaming at the equivalent rate as the origin request.

Example:

  • Run a Lighttpd instance or docker container on your local machine and make a large file available to be served
  • Run Trickster locally
  • Make multiple curl requests of the same object

You should see the speed limited on the origin request by your disk IO, and your speed between Trickster limited by Memory/CPU

6 - Time Series Caching

Accelerating time series databases with Trickster’s Delta Proxy Cache.

6.1 - Supported Providers

Trickster currently supports the following Providers:

Generic HTTP Reverse Proxy Cache

Trickster operates as a fully-featured and highly-customizable reverse proxy cache, designed to accelerate and scale upstream endpoints like API services and other simple http services. Specify 'reverseproxycache' or just 'rpc' as the Provider when configuring Trickster.

Time Series Databases

Prometheus

Trickster fully supports the Prometheus HTTP API (v1), including Prometheus 3.x features like native histograms and UTF-8 metric names. Specify 'prometheus' as the Provider when configuring Trickster. See the Prometheus Support Document for more information.

InfluxDB

Trickster supports InfluxDB 1.x, 2.x, and 3.x. Specify 'influxdb' as the Provider when configuring Trickster.

See the InfluxDB Support Document for more information.

ClickHouse

Trickster supports accelerating ClickHouse time series over both HTTP and the ClickHouse native binary protocol (port 9000), and is tested against the Vertamedia and official Grafana ClickHouse (v4+) datasource plugins. Specify 'clickhouse' as the Provider when configuring Trickster.

See the ClickHouse Support Document for more information.

Graphite

Trickster accelerates Graphite’s render API, including graphite-web, go-carbon and other Graphite-protocol backends. Specify 'graphite' as the Provider when configuring Trickster.

See the Graphite Support Document for more information.

MySQL

Trickster supports protocol-aware acceleration for supported MySQL servers and Grafana’s built-in MySQL data source. Specify mysql as the direct terminal provider and expose it through a listener with protocol: mysql.

See the MySQL Provider Guide for the supported server, client, SQL, authentication, TLS, caching, routing, and operations contract.

6.2 - Per-Query Time Series Instructions

Beginning with Trickster v1.1, certain features like Fast Forward can be toggled a per-query basis, to assist with compatibility in your environment. This allows the drafters of a query to have some say over toggling these features on queries they find to have issues running through Trickster. This is done by adding directives via query comments. For example, in Prometheus, you can end any query with # any comment following a hashtag, so you can place the per-query instructions there.

Supported Per-Query Instructions

Fast Forward Disable

Instruction trickster-fast-forward

Supported for: Prometheus (other time series do not currently implement Fast Forward)

Usage: go_goroutines{job="trickster"} # trickster-fast-forward:off

Notes: This can only be used to disable fast forward. A value of on will have no effect.

Backfill Tolerance

Instruction trickster-backfill-tolerance

Supported for: All time series backends

Usage: SELECT time, count(*) FROM table # trickster-backfill-tolerance:120

Notes: This overrides the backfill tolerance value for this query by the specified value (in seconds). Only integers are accepted.

6.3 - Query Range Limits

Trickster supports enforcing limits on the maximum time range (duration) of incoming queries. This feature protects both the Trickster cache and the downstream origin TSDBs from resource-intensive, oversized queries.

Overview

When a client sends a query spanning a duration larger than the configured limit, Trickster intercepts the request early in the pipeline. It rejects the query, returns an HTTP 400 Bad Request status, and increments a metric counter. This helps prevent large queries (e.g., querying 1 year of data at 15s resolution) from causing out-of-memory (OOM) issues or heavy processing spikes.

Configuration

You can configure the query range limit on any backend by setting the max_query_range option. The value is a duration string (e.g., 1h, 1d, 14d).

Here is a configuration example:

backends:
  prometheus_dev:
    provider: prometheus
    origin_url: http://prometheus-origin:9090
    max_query_range: 14d

Setting max_query_range: 0 or omitting the field disables the range limit enforcement.

Supported and Unsupported Backends

Query range limit enforcement is active on backends that implement the backends.TimeseriesBackend interface, which includes:

  • Supported Backends:
    • Prometheus
    • InfluxDB
    • ClickHouse
    • Application Load Balancers (ALBs) using Time Series Merge (TSM)
  • Unsupported Backends:
    • Standard HTTP Reverse Proxy backends (e.g., reverseproxycache) that do not parse the query timespan.

Rejection Behavior

When an incoming query’s duration exceeds the allowed max_query_range:

  1. HTTP Status: Trickster responds immediately with HTTP 400 Bad Request.
  2. Error Message: The response body contains an error message, e.g., query time range exceeds the allowed limit of 336h0m0s.
  3. Short-circuiting: Downstream origin TSDB instances are never contacted, conserving database resources.

Metrics and Logging

Trickster exposes metrics and logs to track rejected queries:

Metrics

  • Metric Name: trickster_proxy_query_range_rejected_total
  • Labels: backend (name of the backend rejecting the query)
  • Type: Counter
  • Description: Tracks the total number of queries rejected due to exceeding the max_query_range limit.

Logs

Upon query rejection, Trickster emits a structured warning log including information about the offending query:

[WARN] query rejected due to max_query_range limit (backendName=prometheus_dev, limit=336h0m0s, duration=360h0m0s, clientIP=127.0.0.1, path=/api/v1/query_range, statement=up)

Use with ALBs

Application Load Balancer (ALB) backends configured with the Time Series Merge (TSM) mechanism can enforce query range limits at the ALB entry point. This ensures that Trickster rejects oversized queries before scattering them across downstream pool member backends.

For details on using max_query_range with TSM ALBs, refer to the ALB Documentation.

6.4 - Timeseries Request Sharding

Overview

A shard means “a small part of a whole,” and Trickster supports the sharding of upstream HTTP requests when retrieving timeseries data. When configured for a given time series backend, Trickster will shard eligible requests by inspecting the time ranges needed from origin and subdividing them into smaller ranges that conform to the backend’s sharding configuration. Sharded requests are sent to the origin concurrently, and their responses are reconstituted back into a single dataset by Trickster after they’ve all been returned.

Mechanisms

Trickster support three main mechanisms for sharding:

  • Maximum Timestamps Per Shard - Trickster calculates number of expected unique timestamps in the response by dividing the requested time range size by the step cadence, and then subdivides the time ranges so that each sharded request’s time range will return no more timestamps than the configured maximum.
  • Maximum Time Range Width Per Shard - Trickster inspects each needed time range, and subdivides them such that each sharded request’s time range duration is no larger than the configured maximum.
  • Epoch-Aligned Maximum Time Range Width Per Shard - Trickster inspects each needed time range, and subdivides them such that each sharded request’s time range duration is no larger than the configured maximum, while also ensuring that each shard’s time boundaries are aligned to the Epoch based on the configured shard step size.

Configuring

Maximum Unique Timestamp Count Per Shard

In the Trickster configuration, use the shard_max_size_points configuration to shard requests by limiting the maximum number of unique timestamps in each sharded response.

backends:
  example:
    provider: prometheus
    origin_url: http://prometheus:9090
    shard_max_size_points: 10999

Maximum Time Range Width Per Shard

In the Trickster configuration, use the shard_max_size_time configuration to shard requests by limiting the maximum width of each sharded request’s time range.

backends:
  example:
    provider: 'prometheus'
    origin_url: http://prometheus:9090
    shard_max_size_time: 2h

Epoch-Aligned Maximum Time Range Width Per Shard

In the Trickster configuration, use the shard_step configuration to shard requests by limiting the maximum width of each sharded request’s time range, while ensuring shards align with the epoch on the configured cadence. This is useful for aligning shard boundaries with an upstream database’s partition boundaries, ensuring that sharded requests have as little partition overlap as possible.

backends:
  example:
    provider: 'prometheus'
    origin_url: http://prometheus:9090
    shard_step: 2h

shard_step can be used in conjunction with shard_max_size_time, so long as shard_max_size_time is perfectly divisible by shard_step. This combination configuration will align shards against the configured shard step, while sizing each shard’s time range to be multiple shard steps wide.

backends:
  example:
    provider: 'prometheus'
    origin_url: http://prometheus:9090
    shard_step: 2h
    shard_max_size_time: 4h

Neither shard_step or shard_max_size_time can be used in conjunction with shard_max_size_points.

6.5 - Apache Arrow Flight SQL Listeners

Trickster can serve Apache Arrow Flight SQL — the gRPC-based query protocol spoken by ADBC drivers, Grafana’s SQL-mode datasources, and the Arrow-native client SDKs — as a caching proxy in front of a Flight SQL-capable origin. The protocol support is vendor-neutral; each backend provider that adopts it supplies only its origin-specific wiring. InfluxDB 3.x is the first supported provider (see InfluxDB Support for its specifics and examples).

Enabling a Flight SQL listener

Define a listener with the flight-sql protocol and map exactly one compatible backend to it through the backend’s listener_names:

listeners:
  my-flight:
    protocol: flight-sql   # Apache Arrow Flight SQL over gRPC
    port: 8485
backends:
  influx3:
    provider: influxdb
    origin_url: 'http://influxdb3:8181/'
    listener_names: [ default, my-flight ]

Flight SQL listeners share Trickster’s standard listener lifecycle: connection limits (connections_limit), graceful drain on SIGTERM, and config reload (SIGHUP) — a reload with an unchanged backend configuration keeps serving on the existing socket, while a changed configuration drains the old server (closing its upstream connection) and rebinds.

TLS

The listener serves TLS when the mapped backend’s tls block presents a certificate and key, with in-place certificate rotation on config reload; without one it serves plaintext gRPC. The upstream dial is TLS-capable via provider options (for InfluxDB, influxdb.flight_upstream_tls).

Caching model

Every cache entry is scoped to the backend name plus a per-tenant namespace derived from the request metadata the provider declares (for InfluxDB: database, bucket-name, and a hash of authorization), mirroring the scope the upstream itself grants — tenants never share entries, and credentials never appear in cache keys.

Statement queries are served through three tiers:

  1. Delta proxy cache — statements the provider’s SQL analyzer classifies as delta-cacheable are cached by time extent: repeat and overlapping queries fetch only missing sub-ranges from the origin, and responses are rebuilt into Arrow record batches conforming to the response’s original schema (types, column order, nullability, and schema metadata are preserved). A statement’s ORDER BY is carried through reconstruction, so a delta hit returns rows in the order the statement asked for; ordering terms that do not resolve to a select-list output fall to the next tier. Responses whose schemas the delta model cannot represent fall automatically to the next tier as well.
  2. Object cache — everything else cacheable is stored as the verbatim Arrow IPC byte stream and returned byte-identically, with a short, provider-configured lifetime. Metadata RPCs use this tier.
  3. Proxy — nondeterministic and non-SELECT statements are never cached.

Serving-cost characteristics of the tiers are recorded in Flight SQL Cache Tier Benchmarks.

Supported RPC surface

  • Statement execution (GetFlightInfoStatement / DoGetStatement).
  • Result-set schema requests (GetSchemaStatement / GetSchemaPreparedStatement), which ADBC drivers probe on connect. When the origin itself does not implement its schema RPC (InfluxDB 3 Core among them), the schema is derived by executing the statement through the object cache, so the probe’s cost folds into the execution the client performs next.
  • Catalog metadata (GetTables, GetCatalogs, GetDbSchemas, GetTableTypes, GetSqlInfo), proxied and cached per tenant.
  • Type and key discovery (GetXdbcTypeInfo, GetPrimaryKeys, GetImportedKeys, GetExportedKeys, GetCrossReference), which JDBC- and ODBC-shaped clients probe during metadata discovery, likewise proxied and cached per tenant and per request shape.
  • The prepared-statement lifecycle (create, bind, execute, close). A parameterless prepared statement is served through the same three statement tiers — sharing cache entries with ad-hoc executions of the same text — while parameterized executions are cached whole, keyed by the bound parameter hash. Statements abandoned by disconnected clients are closed upstream after 15 minutes of inactivity.

A FlightInfo whose result the origin partitions across several endpoints is consumed in full: every endpoint’s ticket is resolved in order, and an endpoint that advertises its own location is retrieved from that location only when its exact host:port authority is explicitly allowlisted by the provider. This opt-in is required because request metadata, including credentials, is forwarded to the alternate host. A TLS primary is never downgraded to a plaintext alternate. Alternate connection caches are bounded (16 clients by default). Partitions that disagree on schema are refused rather than served partially.

Writes/updates/ingest, transactions, savepoints, Substrait plans, query cancellation, endpoint renewal, and session options return gRPC Unimplemented — they are outside a read-through cache’s contract. Clients requiring them should connect to the origin directly. Prepared-statement bindings are accepted as a single record batch; a client streaming several batches for one binding receives Unimplemented rather than a silently truncated binding.

Response size bound

Responses are buffered whole so they can be cached, so a single very large query would otherwise be able to exhaust the process heap. Each upstream response is bounded at 128MiB by default; a response exceeding the bound is refused with gRPC ResourceExhausted rather than being buffered. Providers expose the bound as a backend option (for InfluxDB, influxdb.flight_max_response_bytes; a negative value removes it). A separate aggregate budget bounds response bytes concurrently assembled or streamed (512MiB by default; influxdb.flight_max_buffered_bytes for InfluxDB).

Metrics

Statement executions record cache outcomes to the native SQL metrics under the flightsql dialect label: trickster_sql_query_cache_total (cache_mode x cache_status), the standard proxy request/points/duration metrics, trickster_sql_query_rewrite_failures_total for extent-rendering failures, and cache-error events on the shared cache metrics. See Metrics.

Architecture (for provider implementers)

The protocol lives in vendor-neutral packages:

  • pkg/proxy/flightsql — the gRPC protocol server, caching statement/ metadata/prepared handling, and the upstream client. A provider supplies: the upstream address and TLS posture; ForwardMetadataKeys (which inbound gRPC metadata flows to the origin); a KeyScoper covering every forwarded key that affects results; and optionally a DeltaConfig carrying its SQL dialect analyzer to enable the delta tier.
  • pkg/proxy/engines/nativedelta — the transport-agnostic delta-proxy-cache engine (shared with the MySQL native listener), driven by the frozen pkg/parsing/sqlanalyzer dialect contract.
  • pkg/timeseries/dataset/arrow — dataset ⇄ Arrow record batch conversion used by the delta tier.

pkg/backends/influxdb/native_listener.go is the reference wiring: a provider integrates by implementing the native.Adapter interface (declaring Protocol() == "flight-sql" and its own BackendProvider()) and constructing the flightsql server from its backend options.

6.6 - Providers

Guides for each supported time series provider.

6.6.1 - Prometheus Support

Trickster fully supports accelerating Prometheus, which we consider our First Class backend provider. They work great together, so you should give it a try!

Most configuration options that affect Prometheus reside in the main Backend config, since they generally apply to all TSDB providers alike.

Supported API Endpoints

Trickster supports the full Prometheus HTTP API (v1), including features introduced in Prometheus 3.x.

Cached Endpoints

EndpointCache StrategyScatter/Gather Merge
/api/v1/query_rangeDelta Proxy CacheYes
/api/v1/queryObject Proxy CacheYes
/api/v1/seriesObject Proxy CacheYes
/api/v1/labelsObject Proxy CacheYes
/api/v1/label/<name>/valuesObject Proxy CacheYes
/api/v1/alertsProxy + MergeYes
/api/v1/targetsObject Proxy CacheNo
/api/v1/targets/metadataObject Proxy CacheNo
/api/v1/rulesObject Proxy CacheNo
/api/v1/alertmanagersObject Proxy CacheNo
/api/v1/status/*Object Proxy CacheNo
/api/v1/query_exemplarsObject Proxy CacheNo
/api/v1/metadataObject Proxy CacheNo
/api/v1/format_queryObject Proxy CacheNo
/api/v1/parse_queryObject Proxy CacheNo
/api/v1/scrape_poolsObject Proxy CacheNo
/api/v1/featuresObject Proxy CacheNo

Proxied Endpoints (not cached)

EndpointNotes
/api/v1/notifications/liveSSE streaming
/api/v1/writeRemote write (v1 and v2)
/api/v1/otlp/v1/metricsOTLP ingestion
/api/v1/admin/*Explicitly unsupported (returns error)

All other /api/v1/* paths are reverse-proxied to the origin without caching.

Prometheus 3.x Features

  • Native histograms are fully supported in query and query_range responses, including mixed series with both float samples and histogram samples.
  • UTF-8 metric and label names (e.g., {"metric.name"}) are supported in queries and cache keys.
  • Query stats (stats=all parameter) are cache-key differentiated, so responses with and without stats are cached separately.

Injecting Labels

Trickster can inject labels on a per-backend basis into Prometheus responses before returning them to the caller.

Here is the basic configuration for adding labels:

backends:
  prom-1a:
    provider: prometheus
    origin_url: http://prometheus-us-east-1a:9090
    prometheus:
      labels:
        datacenter: us-east-1a

  prom-1b:
    provider: prometheus
    origin_url: http://prometheus-us-east-1b:9090
    prometheus:
      labels:
        datacenter: us-east-1b

Interaction with ALB Merge Strategy

When using label injection with an ALB configured for Time Series Merge, injected labels are automatically stripped from responses before merging. This ensures that series from different backends are aggregated correctly, and the injected labels do not appear in the final response to the caller. See the ALB Merge Strategy documentation for details.

Max Query Range Limitation

Trickster supports enforcing a max_query_range limit on Prometheus backends. For details on how to configure and use query range limits, see the Query Range Limits documentation.

6.6.2 - InfluxDB Support

Trickster provides support for accelerating InfluxDB queries that return time series data normally visualized on a dashboard. Acceleration works by using the Time Series Delta Proxy Cache to minimize the number and time range of queries to the upstream InfluxDB server.

Scope of Support

Trickster is tested with the built-in InfluxDB DataSource Plugin for Grafana v5.0.0.

Trickster uses InfluxDB-provided packages to parse and normalize queries for caching and acceleration. If you find query or response structures that are not yet supported, or providing inconsistent or unexpected results, we’d love for you to report those so we can further improve our InfluxDB support.

Trickster supports integrations with InfluxDB 1.x, 2.x, and 3.x.

InfluxDB 3.x Support

Trickster supports InfluxDB 3.x via both the native v3 API endpoints and the v1/v2 compatibility endpoints.

Supported v3 Endpoints

  • GET/POST /api/v3/query_sql — SQL queries with delta-proxy caching
  • GET/POST /api/v3/query_influxql — InfluxQL queries with delta-proxy caching
  • POST /api/v3/write_lp — line protocol writes (proxied, not cached)

All other v3 API paths (including the /api/v3/configure/* management endpoints, with any HTTP method) are proxied through untouched.

Query requests on both query endpoints may arrive as URL parameters (GET), an application/json document ({"q": ..., "db": ..., "format": ..., "params": ...}), a form-encoded body, or a raw statement body — all four shapes are parsed and cached equivalently, and the db and params values participate in the cache identity.

InfluxQL over v3

/api/v3/query_influxql requests use the same delta-proxy caching as the v1 /query endpoint (queries with a GROUP BY time(...) interval and a time-bounded WHERE clause), but speak the v3 request/response shapes: the v3 request document above and the v3 tabular response formats, including the iox::measurement column, which Trickster treats as a series tag alongside any GROUP BY tags.

SQL Query Caching

SQL queries using date_bin() or date_trunc() for time binning are parsed for time range extraction and step detection, enabling delta-proxy caching. Trickster extracts time ranges from WHERE clauses and intervals from date_bin(INTERVAL '...', time) or date_trunc('unit', time) in SELECT. Live, unaligned time ranges (the shape dashboard tools emit) are rounded inward to complete buckets. Grouped queries (GROUP BY 1, host) are cached per tag series.

Example query that Trickster will accelerate:

SELECT date_bin(INTERVAL '1 hour', time) AS time, avg(temperature)
FROM weather
WHERE time >= '2024-01-01 00:00:00' AND time < '2024-01-02 00:00:00'
GROUP BY 1

SELECT queries that cannot be delta-cached (no fixed-cadence time bucket, joins, subqueries, window functions, compound selects such as UNION, LIMIT, variable-length buckets like '1 month', or unsafe time predicates) fall back to the object proxy cache, which caches the whole response briefly and passes results through unchanged. Non-SELECT statements and parameterized queries (a params field in the request) are proxied to the origin without delta caching.

Queries without an upper time bound run to the present; for these, Trickster’s backfill tolerance is floored at one bucket so the still-filling final bucket is always refreshed from the origin rather than cached as complete.

Response Formats

Trickster supports the following v3 response formats, controlled by the format query parameter (in the URL or the request document) or, when no format is given, the Accept header (application/json, application/jsonl, text/csv, …):

  • json (default) — JSON array of objects
  • jsonl — JSON Lines (one JSON object per line)
  • csv — standard CSV with header row

The parquet and pretty formats are not supported for caching and will be proxied through.

v1/v2 Compatibility

InfluxDB 3.x ships with v1 and v2 compatibility endpoints. Trickster’s existing InfluxQL support works against these endpoints with no additional configuration — just point Trickster at the v3 instance and query via /query.

Flight SQL (gRPC)

InfluxDB 3.x exposes SQL via Apache Arrow Flight SQL on gRPC in addition to HTTP. Grafana’s InfluxDB datasource in SQL mode, the Python/Rust/Java SDKs, and ADBC all default to Flight SQL, so HTTP-only caching misses a significant fraction of real-world query traffic. The protocol-level (backend-agnostic) listener documentation lives in Apache Arrow Flight SQL Listeners; this section covers the InfluxDB specifics.

Trickster can expose a Flight SQL server that proxies to an upstream Flight SQL endpoint and caches the Arrow IPC byte stream. Enable it by defining a listener with the flight-sql protocol and mapping exactly one InfluxDB backend to it (alongside its usual HTTP listener):

listeners:
  influx3-flight:
    protocol: flight-sql   # Apache Arrow Flight SQL over gRPC
    port: 8485
backends:
  influx3:
    provider: influxdb
    origin_url: 'http://influxdb3:8181/'
    listener_names: [default, influx3-flight]
    influxdb:
      flight_upstream_address: 'influxdb3:8181'   # optional, defaults to origin_url host
      flight_max_response_bytes: 134217728        # optional, defaults to 128MiB
      flight_max_buffered_bytes: 536870912         # optional, aggregate in-flight budget; defaults to 512MiB
      flight_allowed_location_hosts:               # optional exact authorities for alternate endpoints
        - 'influxdb3-replica:8181'

The authorization and database headers are forwarded from the client through to the upstream, and every cache entry is keyed by the backend name, the database/bucket-name headers, and a hash of the authorization header — so tenants and databases never share cache entries. Requests that send no authorization header share one anonymous cache scope, mirroring the access the upstream would grant them.

Statement queries are served through a three-tier cache:

  1. Delta proxy cache — queries the SQL analyzer classifies as delta-cacheable (the same date_bin()/date_trunc() shapes as the HTTP path) are cached by time extent: repeat and overlapping queries fetch only the missing sub-ranges from the upstream, and responses are rebuilt into Arrow record batches conforming to the response’s original schema. A query’s ORDER BY is carried through that rebuild, so a cache hit returns rows in the requested order; ordering terms that do not resolve to a select-list output fall to the next tier. Entries use the backend’s timeseries_ttl and honor backfill_tolerance; still-filling buckets are always refetched. Responses whose Arrow schemas the delta model cannot represent (nested types, non-string dictionaries, …) automatically fall to the next tier.
  2. Object cache — everything else cacheable is stored as the verbatim Arrow IPC byte stream, returned byte-identically, with a lifetime of influxdb.flight_cache_ttl (default 60s). Metadata RPCs and prepared statements always use this tier.
  3. Proxy — statements referencing nondeterministic functions (now(), current_timestamp, random(), …) and non-SELECT statements are never cached.

Flight SQL TLS

The Flight listener serves TLS when its mapped backend’s tls block presents a certificate and key (the same mechanism HTTP listeners use); certificate rotation applies on config reload without dropping the listener. Without a certificate the listener serves plaintext gRPC. The upstream dial uses TLS when influxdb.flight_upstream_tls is set, honoring the backend tls block’s insecure_skip_verify.

Unsupported Flight SQL RPCs

Trickster proxies queries, result-set schema requests (GetSchemaStatement/GetSchemaPreparedStatement, which ADBC drivers probe on connect), the metadata RPCs below, and the prepared-statement lifecycle. Other Flight SQL RPCs — writes/updates/ingest (ExecuteUpdate, DoPut ingestion), transactions, savepoints, Substrait plans, query cancellation, endpoint renewal, and session options — return gRPC Unimplemented; clients requiring them should connect to InfluxDB directly. A prepared-statement binding must arrive as a single record batch; a client streaming several batches for one binding receives Unimplemented rather than a binding taken from its first batch alone.

Flight SQL listeners share Trickster’s standard listener lifecycle: connection limits (connections_limit), graceful drain on SIGTERM (active streams are drained until the configured drain timeout, then closed), and config reload (SIGHUP) — a reload with an unchanged backend configuration keeps serving on the existing socket, while a changed configuration drains the old server and rebinds.

Metadata RPCs

ADBC clients (Grafana’s SQL datasource, the Python adbc_driver_flightsql, etc.) probe the following RPCs on connect to populate schema browsers and query editors. Trickster proxies these to the upstream and caches the Arrow IPC response:

  • GetFlightInfoTables / DoGetTables
  • GetFlightInfoCatalogs / DoGetCatalogs
  • GetFlightInfoSchemas / DoGetDBSchemas
  • GetFlightInfoTableTypes / DoGetTableTypes
  • GetFlightInfoSqlInfo / DoGetSqlInfo
  • GetFlightInfoXdbcTypeInfo / DoGetXdbcTypeInfo
  • GetFlightInfoPrimaryKeys / DoGetPrimaryKeys
  • GetFlightInfoImportedKeys / DoGetImportedKeys
  • GetFlightInfoExportedKeys / DoGetExportedKeys
  • GetFlightInfoCrossReference / DoGetCrossReference

Flight SQL response size

Flight responses are buffered whole so they can be cached. influxdb.flight_max_response_bytes bounds one upstream response (128MiB by default), while influxdb.flight_max_buffered_bytes bounds response bytes concurrently assembled or streamed (512MiB by default). Exceeding either bound returns gRPC ResourceExhausted; a negative value removes that bound.

A FlightInfo partitioned across several endpoints is consumed in full. Alternate endpoint locations are followed only when their exact host:port authority appears in influxdb.flight_allowed_location_hosts, because the request’s authorization and database metadata are forwarded to that host. A TLS upstream never follows a plaintext alternate location. influxdb.flight_max_location_clients bounds cached alternate connections and defaults to 16.

Prepared Statements

Trickster proxies the full prepared statement lifecycle: CreatePreparedStatement, DoPutPreparedStatementQuery (parameter binding), DoGetPreparedStatement, ClosePreparedStatement. A parameterless prepared statement is equivalent to executing its statement text, so it is served through the same three cache tiers as plain statement queries — delta caching included — and shares their cache entries. Executions with bound parameters are cached whole, keyed by the bound parameter hash so two clients running the same statement with different values don’t alias. Prepared statements abandoned by disconnected clients are closed upstream after 15 minutes of inactivity.

Note: InfluxDB 3 Core 3.10 reports a parameter schema at prepare time but does not resolve bound values during query planning (upstream limitation). Parameterless prepared statements work end-to-end; parameterized ones require a newer Core or Enterprise build.

Flux Language Support

Trickster supports the Flux Query Language for general/basic usage with InfluxDB 1.x and 2.x. Flux is not supported in InfluxDB 3.x.

The delta-proxy cache accepts now() as a range() bound and handles queries with aggregateWindow(every: ...) – the common Grafana shape. Multi-table Flux CSV responses (one table per series in the result set) are also read correctly.

Trickster does not support advanced union-style queries (e.g., with multiple from clauses). In this rare use case, these responses will currently provide invalid data, however, a subsequent beta will proxy unsupported requests.

Trickster currently does not properly handle schema changes within a response CSV body (e.g., multiple CSVs in the same document with their own #annotation and header rows). We will fully support this use case in a future beta.

Max Query Range Limitation

Trickster supports enforcing a max_query_range limit on InfluxDB backends. For details on how to configure and use query range limits, see the Query Range Limits documentation.

6.6.3 - ClickHouse Support

Trickster will accelerate ClickHouse queries that return time series data normally visualized on a dashboard. Acceleration works by using the Time Series Delta Proxy Cache to minimize the number and time range of queries to the upstream ClickHouse server.

Scope of Support

Trickster is tested with the official ClickHouse DataSource Plugin for Grafana v4.21.1 and supports acceleration of queries constructed by this plugin using its built-in time macros like $__fromTime and $__toTime. Delta caching with this an other plugins (e.g., Altinity/Vertamedia) depends on the supported query shapes below. Trickster also supports several other query formats that return “time series like” data.

Trickster also supports the ClickHouse Go SDK (clickhouse-go/v2) over its HTTP and Native protocols, including clickhouse.OpenDB.

Native Binary Protocol Support

Inbound and upstream protocols are configured independently:

  • A named listener with protocol: clickhouse accepts Native client connections.
  • A ClickHouse backend with protocol: native uses the Native protocol for its origin. An empty protocol or protocol: http uses HTTP.
listeners:
  default:
    protocol: http
    port: 8480
  clickhouse-native:
    protocol: clickhouse
    port: 8487

backends:
  click1:
    provider: clickhouse
    origin_url: http://clickhouse:8123
    listener_names: [default, clickhouse-native]
    cache_name: default

This exposes one HTTP route at /click1/ and one Native listener on port 8487, backed by the same cache and HTTP ClickHouse origin. A Native listener must map to exactly one backend; ALB and user-router multiplexing are not supported.

To use a Native origin instead, set the backend protocol and point origin_url at ClickHouse’s Native port:

backends:
  click1:
    provider: clickhouse
    origin_url: http://clickhouse:9000
    protocol: native

All four HTTP/Native ingress and HTTP/Native origin combinations are supported. Native credentials, database, query settings, parameters, and query IDs are forwarded to the origin. Native identity pools are bounded to 64 credential/database combinations per backend.

TLS

Native listener TLS uses the backend’s server certificate. Set require_tls: true and provide both certificate paths:

listeners:
  clickhouse-native-tls:
    protocol: clickhouse
    port: 9441
    tls_watch_interval: 30s

backends:
  click1-tls:
    provider: clickhouse
    origin_url: http://clickhouse:8123
    listener_names: [clickhouse-native-tls]
    require_tls: true
    tls:
      full_chain_cert_path: /etc/trickster/tls/server.crt
      private_key_path: /etc/trickster/tls/server.key

Certificate changes are hot-swapped. Since require_tls applies to every binding of a backend, use a separate backend entry when plaintext and TLS listeners must coexist.

For a TLS Native origin, use an https origin URL with the Native protocol. The common backend TLS options configure origin verification and optional mutual TLS:

backends:
  click1:
    provider: clickhouse
    origin_url: https://clickhouse:9440
    protocol: native
    tls:
      certificate_authority_paths: [/etc/trickster/tls/origin-ca.crt]
      client_cert_path: /etc/trickster/tls/client.crt
      client_key_path: /etc/trickster/tls/client.key

Native Limitations

The Native listener supports SELECT queries, ping/pong, revision 54460 framing, structured errors, and LZ4 compression. INSERT, external tables, and session-changing USE or SET statements are rejected; select a database and settings per request. ClientCancel is accepted as a no-op and does not asynchronously cancel an active upstream request.

Supported native protocol data types: all integer types (8–256 bit), Float32/64, String, FixedString(N), DateTime, DateTime64, Date, Date32, UUID, IPv4, IPv6, Enum8/16, Bool, Nullable(T), Array(T), Map(K,V), Tuple(T1,T2,…), LowCardinality(T), and Decimal.

Native wire clients receive Native blocks. HTTP clients using a Native origin may request JSON, Native, CSV, or TSV-family output; compound columns in CSV/TSV and unsupported formats return an error. The modeler recognizes Native origin responses through the X-ClickHouse-Format response header, and HTTP Native framing honors client_protocol_version.

Trickster parses incoming ClickHouse statements into a full abstract syntax tree using the AfterShip ClickHouse SQL parser, then applies its own semantic analysis to determine whether a query is eligible for time series delta caching and, if so, its timestamp column, bucket cadence, time range, grouping tags, and cache identity. The cache key is derived from a canonical form of the query in which the requested time range is replaced with placeholders, so requests for different time ranges of the same logical series share one delta cache entry.

Trickster’s analysis fails closed: a valid query whose shape cannot be proven safe for delta caching is never rewritten approximately. It is instead served through the Object Proxy Cache (OPC) or proxied directly, and the classification reason is exported through the trickster_sql_query_analysis_total metric.

If you find query or response structures that are not yet supported, or providing inconsistent or unexpected results, we’d love for you to report those. We also always welcome any contributions around this functionality.

Delta-Cacheable Queries

To be eligible for the delta cache, a query must be a single SELECT statement containing a recognized time-bucketing expression in its select list, a supported time range in its WHERE or PREWHERE clause, and a GROUP BY clause that includes the time bucket. Each requirement is described below.

Time-Bucketing Expressions

Exactly one select-list expression must match a supported bucket form:

Grafana Plugin Format

SELECT intDiv(toUInt32(time_col), 60) * 60 [* 1000] [AS alias]

This is the approach used by the Grafana plugin. The argument to the ClickHouse intDiv function is the step value in seconds, since the toUInt32 function on a datetime column returns the Unix epoch seconds. An optional output multiplier of 1000, 1000000, or 1000000000 selects millisecond, microsecond, or nanosecond output timestamps.

ClickHouse Time Grouping Functions

SELECT toStartOfInterval(time_col, INTERVAL n unit) [AS alias]

with a positive constant n and a unit of millisecond, second, minute, hour, day, or week; or one of the fixed-period functions:

toStartOfNanosecond
toStartOfMicrosecond
toStartOfMillisecond
toStartOfSecond
toStartOfMinute
toStartOfFiveMinute
toStartOfTenMinutes
toStartOfFifteenMinutes
timeSlot
toStartOfHour
toStartOfDay
toStartOfWeek
toMonday

date_trunc('unit', time_col) and dateTrunc support the same fixed units. toStartOfWeek uses ClickHouse’s Sunday phase while toMonday and weekly date_trunc use Monday. Calendar-length month, quarter, and year buckets are served through the OPC.

The time column may be a plain, qualified (table.col), or quoted identifier, optionally wrapped in toDateTime, toInt32, or toUInt32. Integer constants defined in a scalar WITH clause may be used for the step value. Timezone-parameter variants of these functions (for example toStartOfHour(time_col, 'America/Denver')) are not eligible for delta caching and are served through the OPC. The current SQL parser does not accept INTERVAL MICROSECOND or INTERVAL NANOSECOND; use the corresponding fixed toStartOf... function.

Determining the Requested Time Range

Time range predicates must appear in a top-level AND conjunction of the WHERE or PREWHERE clause. Predicates joined by OR or negated with NOT make the query ineligible for delta caching.

Two predicate targets are supported, with different rules:

  • The raw time column (the column inside the bucket function): the lower bound must be inclusive (>=) and the upper bound exclusive (<), and both values must fall exactly on bucket boundaries. Other comparators — including BETWEEN — describe partial buckets whose aggregates cannot be safely cached, so those queries are served through the OPC.
  • The bucket alias (the output of the bucket expression): >, >=, <, <=, and BETWEEN are all supported, because bucket outputs are discrete; Trickster normalizes each comparator to the first and last included bucket.

Bound values may be expressed as epoch integers, ClickHouse string dates in the form 2006-01-02 15:04:05 (or date-only, or RFC3339), toDateTime(n), toDateTime64(n, precision), or toDate(n) wrappers, WITH-clause constants, or now()/now64() with optional addition or subtraction of seconds. DateTime64 precision is retained. Floating epoch bounds and timezone-qualified conversions such as toDateTime(n, 'America/Denver') are not eligible.

If no upper bound is present, Trickster caches results up to the current time and inserts a safe upper bound into origin requests automatically.

Examples of delta-cacheable time range clauses (for a one-minute bucket cadence):

WHERE t >= '2020-10-15 00:00:00' AND t <= '2020-10-16 12:00:00'  -- bucket alias
WHERE t BETWEEN 1574686320 AND 1574689920                        -- bucket alias
WHERE time_col >= toDateTime(1574686320) AND time_col < toDateTime(1574689920)
WHERE t >= now() - 3600 AND t < now()                            -- bucket alias

Secondary date-range predicates whose values match the primary range — such as the Date-typed partition filters emitted by the Grafana plugin — are recognized and rewritten in step with the primary range.

Grouping and Result Shape

The GROUP BY clause must include the time bucket (by alias or by its full expression), and every non-aggregate column in the select list must also be grouped. Grouped columns become the series tags in the cached time series. Queries using GROUP BY ... WITH CUBE/ROLLUP, grouping on expressions that are not selected, or leaving a selected dimension ungrouped are served through the OPC.

Output Formats

Delta-cacheable queries may specify FORMAT JSON, CSV, CSVWithNames, TabSeparated (TSV), TabSeparatedWithNames, or TabSeparatedWithNamesAndTypes, or omit the FORMAT clause. Trickster requests TSVWithNamesAndTypes from the origin and re-marshals cached data into the client’s requested format.

Non-Time-Series Queries

Queries that are not cacheable as time series — such as LIMIT-based queries, queries with set operations (UNION, EXCEPT, INTERSECT), SELECT 1 health checks, or SDK handshake requests — are transparently proxied to the upstream ClickHouse server. These requests are cached using the Object Proxy Cache (OPC) with per-query cache keys derived from the query and database URL parameters, ensuring that different SQL statements receive distinct cache entries.

Health and Ping Endpoint

Trickster exposes a /ping endpoint that returns a health check response, matching the endpoint provided by ClickHouse itself. This enables compatibility with clients and SDKs that probe /ping during connection initialization.

Normalization and “Fast Forwarding”

Trickster will always normalize the calculated time range to fit the step size, so small variations in the time range will still result in actual queries for the entire time “bucket”. In addition, Trickster will not cache the results for the portion of the query that is still active – i.e., within the current bucket or within the configured backfill tolerance setting (whichever is greater).

Per-query behavior can be adjusted with comment directives such as trickster-backfill-tolerance; see Per-Query Instructions.

Observability

Query classification outcomes are exported through two low-cardinality metrics that never include query text:

  • trickster_sql_query_analysis_total — labeled by backend, dialect, cache mode (delta, object, none), and a stable reason code such as delta_cacheable, unsafe_predicate, or unsupported_bucket.
  • trickster_sql_query_rewrite_failures_total — counts failures to render an origin request from a cached query plan.

With debug logging enabled, classification decisions are also logged with the same structured reason codes.

Max Query Range Limitation

Trickster supports enforcing a max_query_range limit on ClickHouse backends. For details on how to configure and use query range limits, see the Query Range Limits documentation.

6.6.4 - Graphite Provider

Trickster accelerates Graphite’s render API with the Delta Proxy Cache: it caches each metric at the origin’s native resolution and, on subsequent requests, fetches only the time ranges it does not already hold.

Graphite makes that harder than most time series databases, because a render request does not say what resolution it will be answered at. Whisper chooses an archive based on how old the query’s from is, and no API exposes a metric’s archive ladder. Trickster therefore learns each ladder by probing the origin, caches what it learns, and accelerates only the requests whose resolution it can predict. Everything else is served correctly through ordinary object caching.

Specify graphite as the provider:

backends:
  graphite1:
    provider: graphite
    origin_url: 'http://graphite.example.com:80'
    cache_name: default

That is the whole minimum configuration. Graphite’s cache sizing differs from every other provider’s, so the provider supplies its own defaults rather than asking operators to discover them; see Sizing for what they are and when to change them.

Compatibility

Tested against graphite-web 1.1.10 with Whisper storage, and written against its source. It should work with any Graphite-protocol origin that serves /render and /metrics/expand, including go-carbon’s carbonserver, graphite-clickhouse and carbonapi, but only graphite-web is verified.

ClientStatus
Grafana’s Graphite data sourceVerified, GET and POST form bodies
Direct /render API clientsVerified
Graphite Composer / dashboard UIProxied, not accelerated (image formats)
Response formatBehavior
jsonAccelerated. Also what Trickster requests upstream.
raw, csv, msgpackAccelerated; rendered from the cached series
png, svg, pdf, pickle, dygraph, rickshawProxied and object-cached

Trickster always requests format=json from the origin regardless of what the client asked for, and renders the client’s format from the cached series. JSON is the only render format that carries the tags object, which Grafana relies on, and it is what makes one cached series able to serve every output format.

Configuration

Every common backend option applies (cache_name, timeseries_ttl, timeseries_retention_factor, backfill_tolerance, max_object_size_bytes, timeout, healthcheck, paths, TLS, authenticators); two of them, max_object_size_bytes and timeseries_retention_factor, take Graphite-specific defaults, as Sizing explains.

Per-path request_headers and request_params apply to the synthetic requests resolution makes (probes, wildcard expansion) as well as to proxied traffic, so an origin behind a static API key or one that needs local=1 works for learning too. What cannot work is per-client upstream identity: resolution state (learned ladders, wildcard expansion tokens, the registry generation) is backend-wide, and synthetic requests carry only the backend’s configured identity. Trickster therefore declines acceleration — the request is served correctly through the object cache, which keys on the client’s Authorization and every declared result-affecting input — whenever a render request carries an Authorization header, a header named in the render path’s cache_key_headers, or a parameter or form field named in its cache_key_params/cache_key_form_fields (such as local, which selects the clustered origin view) that the path’s request_headers/request_params does not statically override, and whenever the render path or the backend itself has a request rewriter (req_rewriter_name) attached, since a rewriter can change the upstream host, path, headers, or parameters in ways synthetic requests do not see. The fallback is counted as trickster_graphite_fallbacks_total{reason="client_identity"}.

If every client shares one upstream identity — Grafana with a configured datasource credential is the common case — supply it in the graphite block to re-enable acceleration:

backends:
  graphite1:
    provider: graphite
    origin_url: 'http://graphite.example.com:80'
    graphite:
      origin_username: 'metrics'
      origin_password: '<the origin password>'

For origins using a non-Basic scheme, set origin_authorization to the verbatim header value (e.g., 'Bearer <token>') instead of the username/password pair.

The credential becomes the Authorization request header of every path — proxied client traffic, synthetic resolution requests (probes, wildcard expansion) and the default health check alike — so the origin sees one static identity, a client-supplied Authorization header is replaced rather than forwarded, and requests accelerate regardless of what the client sent. Cached objects are keyed on the configured credential, so rotating it invalidates previously cached responses and learned resolution state rather than serving the old tenant’s data. To authenticate Trickster’s own clients as well, attach an authenticator via authenticator_name: the validated client credential is stripped before proxying and the origin credential is applied in its place.

A path config that sets its own Authorization in request_headers (or removes it with -Authorization) overrides the origin credential for that path. Appending one with +Authorization alongside an origin credential is rejected at startup, since the effective header would be ambiguous. If per-path identities then diverge across the paths resolution uses — /render and /metrics/expand selecting different request_headers/request_params for synthetic GETs, or GET and POST /render pinned to different credentials — Trickster declines acceleration rather than mix upstream namespaces, counted as trickster_graphite_fallbacks_total{reason="resolution_identity"}.

The graphite block adds provider-specific options:

backends:
  graphite1:
    provider: graphite
    origin_url: 'http://graphite.example.com:80'
    graphite:
      time_zone: UTC
      passthrough_max_data_points: false
      find_cache_ttl: 1m
      resolution_registry:
        ttl: 24h
        negative_ttl: 30s
        max_entries: 100000
        persist: true
        probe_concurrency: 2
        probe_budget: 96
      static_retentions:
        - pattern: '^collectd\.'
          retentions: '10s:6h,1m:7d,10m:5y'
OptionDefaultPurpose
time_zoneUTCInterprets date-anchored from/until values (midnight, today, MM/DD/YY) when the request has no tz parameter. Set it to the origin’s graphite-web TIME_ZONE, or those queries resolve to different instants than the origin uses.
origin_username, origin_passwordunsetStatic HTTP Basic credential sent as the Authorization header of every upstream request — proxied, synthetic and health check alike. See Configuration above.
origin_authorizationunsetVerbatim Authorization header value for non-Basic schemes (e.g., Bearer <token>); mutually exclusive with origin_username/origin_password.
passthrough_max_data_pointsfalseSend the client’s maxDataPoints upstream instead of consolidating in Trickster. See maxDataPoints.
find_cache_ttl1mHow long a wildcard target’s expansion to concrete metric paths is reused. Lower it where metrics appear and disappear frequently.
max_targets_per_request128Renders carrying more targets than this are served through the object lane, untouched, before any parsing or per-target fan-out.
max_target_length16384A render with a longer single target expression (in bytes) is likewise served through the object lane untouched.
max_expanded_leaves4096A wildcard whose expansion matches more leaf paths than this is served through the object lane; the refusal is remembered for find_cache_ttl.
max_expansion_bytes2097152Bounds one expansion’s aggregate decoded leaf-name bytes the same way.
resolution_registry.ttl24hHow long a learned ladder is trusted. Whisper ladders change only when an operator runs whisper-resize.py, so this is deliberately long.
resolution_registry.negative_ttl30sInitial backoff after a failed resolution. Doubles per consecutive failure, capped at 10m.
resolution_registry.max_entries100000Bounds each registry layer; least-recently-used entries are evicted.
resolution_registry.persisttrueWrite learned ladders through to this backend’s cache so a restart does not relearn them.
resolution_registry.probe_concurrency2Simultaneous ladder-learning runs.
resolution_registry.probe_budget96Probes one learning run may issue before giving up.
static_retentionsemptySeed and override, from your storage-schemas.conf. See Static retentions.

Health checks

The default health check is GET /metrics/find?query=*, which every Graphite-protocol implementation serves. /version is not used because carbonapi and graphite-clickhouse answer it differently.

A configured origin credential rides on the probe even when the healthcheck block declares custom headers. To override it, set your own Authorization value there; to send the probe unauthenticated, set Authorization: ''.

Routed paths

PathHandling
/render (GET, POST)The render handler: delta cache, or object cache when the request is not accelerable
/metrics/find, /metrics/expand, /metrics/index.json, /tags, /tags/*Object-cached, 30s. These also back resolution’s own lookups, so caching them lowers probe cost.
/functions, /versionObject-cached, 1h
everything elseProxied

How resolution prediction works

Why it is necessary

Whisper picks an archive from the age of the query’s left edge, not from the width of the window:

diff = now - fromTime
for archive in header['archives']:
    if archive['retention'] >= diff:
        break
step = archive['secondsPerPoint']

So from=-7d&until=-6d and from=-7d&until=now are answered at the same step, and a query one second either side of a rung boundary is answered at different steps. The delta cache must know the step before it can compose a cache key or reason about which ranges it already holds — and the step is a property of the metric’s on-disk file, which nothing in the request reveals.

Static configuration is not sufficient on its own, because a Whisper file keeps whatever ladder it was created with. Editing storage-schemas.conf affects only newly created files, so in any long-lived installation the config and the files routinely disagree.

Probe and learn

On the first request for a metric Trickster does not know the ladder, so the request is served through the object cache — correct, just not delta-cached — and the response itself teaches the step at that age. In the background, Trickster then discovers the metric’s full ladder with a short series of synthetic /render probes: a geometric sweep outward in time to find where the step changes, then a binary search for each rung boundary, and a pair of probes to establish maxRetention. Each probe asks for a one-second window and pins the origin’s reference time with now, so it is cheap and exact.

Ladders are shared. They originate from storage-schemas.conf patterns, so a deployment with thousands of metrics typically has only a handful of distinct ladders; a new metric is first confirmed against the ladders already known (about 7 probes) and only discovered from scratch (about 40) if none matches. In the developer environment, 24 metrics across 4 ladders converge in about 380 probes total, after which probing stops entirely until the TTL expires.

Once a ladder is known, prediction is arithmetic and costs nothing.

Confidence levels

Every resolved target carries a confidence, exported as the confidence label on trickster_graphite_resolution_lookups_total:

ConfidenceMeaningBehavior
exactThe step was read from an origin response for this metric at this ageDelta cached
derivedComputed from known ladders — the LCM across a wildcard’s leaves, or a step-altering function Trickster understandsDelta cached
configuredFrom static_retentions only, not yet confirmed by probeDelta cached, and a confirming probe is scheduled
unknownNo usable stepObject cached

A request with several targets reports the weakest confidence among them.

Verification, and what happens when a prediction is wrong

The step is not merely predicted; it is checked. Every response Trickster models is compared against the predicted step. A mismatch means the cached ladder was wrong — a whisper-resize.py, a metric moved between schemas — and Trickster:

  1. discards the response rather than caching it under the predicted key,
  2. increments trickster_graphite_step_mispredictions_total and logs a warning,
  3. invalidates the registry generation, so every entry learned under the old assumption misses rather than colliding,
  4. relearns the affected ladders, and
  5. re-serves the request through the object cache, so the client still gets the correct answer.

One response shape cannot self-verify: JSON carries no explicit step, so a fetch that returns fewer than two points is consistent with any prediction. Trickster never trusts one. A delta fetch that would cover a single bucket — the tip fetch every steady-state refresh performs — is widened one bucket into the past, which cannot change the archive whisper selects (the pinned now keeps now - from identical) but makes the response two points, and therefore self-verifying, at no extra request cost. Any fetched response that still cannot prove its step — one point, or no series where the prediction promised data inside retention — is refused outright: nothing is cached, and the request is served through the object cache instead, while the background learner re-establishes the ladder.

trickster_graphite_step_mispredictions_total should be flat at zero. A non-zero value is not a client-visible error, but it does mean something the registry believed was wrong.

Static retentions

static_retentions mirrors storage-schemas.conf: an ordered list, first match wins, patterns matched anywhere in the metric path exactly as carbon applies them.

    graphite:
      static_retentions:
        - pattern: '^collectd\.'
          retentions: '10s:6h,1m:7d,10m:5y'
        - pattern: '\.count$'
          retentions: '1m:2d,5m:30d'
        - pattern: '.*'
          retentions: '1m:1d'

Retention syntax is Whisper’s: precision:retention pairs, where each part is a number with an optional unit (s, m for minutes, h, d, w, y), and a bare number in the second position is a point count. 1m:7d is a one-minute step kept for seven days.

This is a seed and an override, never the sole source of truth. A static match yields configured confidence and schedules a confirming probe; when the probe disagrees with the configuration, the probe wins and the ladder is relearned. That is deliberate: it means a stale static_retentions block degrades to a few extra probes rather than to incorrect data.

Use it to skip the warmup cost on a known-stable deployment, not as a substitute for learning.

Changing static_retentions invalidates the persisted registry on the next start, so a corrected block takes effect immediately rather than waiting out the TTL.

What is accelerated

A target is delta-cached only if every function in it satisfies two independent properties:

  1. Its step is predictable — the function does not change the resolution in a way Trickster cannot compute.
  2. It is range-decomposable — each output point is a function of the input points at that same timestamp, so the values for [t1,t2] are identical whether that window was fetched alone or as part of a wider one.

The second property is what delta caching actually requires, and it is stricter than it first appears. The v1 allowlist is deliberately small:

Cross-series aggregation — one output point per input timestamp: sumSeries (sum), averageSeries (avg), minSeries, maxSeries, diffSeries, multiplySeries, divideSeries, divideSeriesLists, stddevSeries, rangeOfSeries, countSeries, percentileOfSeries, aggregate, aggregateWithWildcards, aggregateSeriesLists, group, groupByNode, groupByNodes, asPercent (pct), weightedAverage, powSeries, unique, and the *SeriesLists and *WithWildcards variants.

Per-point transforms: scale, scaleToSeconds, offset, add, pow, exp, absolute, invert, squareRoot, sigmoid, logit, log, round, transformNull, isNonNull, removeAboveValue, removeBelowValue, consolidateBy, setXFilesFactor (xFilesFactor).

Naming, name-based filtering and cosmetics: alias, aliasSub, aliasByNode, aliasByMetric, aliasByTags, upper, lower, substr, exclude, grep, color, alpha, lineWidth, dashed, drawAsInfinite, secondYAxis, stacked, areaBetween.

Bare metric paths, wildcards (*, ?, [a-z], {a,b}) and pipe syntax are all accelerated.

What falls back, and why

Anything not on the list above is served through the object cache. The reason is recorded on trickster_graphite_fallbacks_total:

reasonCause
function_not_allowlistedA function in the target is not on the allowlist
unknown_stepThe step could not be resolved (metric not yet learned, probe failing, or the window is wholly beyond maxRetention)
missing_targetNo target parameter, or a wildcard that matches nothing
parse_errorThe target expression, from/until, or now did not parse
non_series_formatAn image or pickle format, or graphType=pie
multi_target_step_mismatchTargets resolve to different steps and could not be split
passthrough_max_data_pointspassthrough_max_data_points is on and the request carries maxDataPoints
mispredictionA response contradicted the predicted step
tz_unavailableThe request names a tz whose validity could not be verified within the timezone cold-load budget (a burst of unique hostile tz values can spend it); the request is served unaccelerated with the original tz forwarded, rather than being reinterpreted in the configured zone
client_identityThe request carries a result-affecting identity or view selector — an Authorization header, a header named in the render path’s cache_key_headers, or a parameter/form field it names in cache_key_params/cache_key_form_fields (local above all) — that the path’s request_headers/request_params does not statically override, or the render path or backend has a request rewriter — see below
resolution_identityThe configured path identities are mixed: /render and /metrics/expand select different request_headers/request_params for synthetic GETs, or the request’s path config carries a different identity than the synthetic one (e.g., GET and POST /render pinned to different credentials)

Notable functions that are not accelerated, and why:

  • Time-windowed: movingAverage, movingSum, movingMin, movingMax, movingMedian, exponentialMovingAverage, derivative, nonNegativeDerivative, perSecond, integral, stdev, holtWinters*, linearRegression. Each output point depends on input points before it, so a cached range cannot be reused at its edges.
  • Whole-range selecting or ranking: highest*, lowest*, sortBy*, limit, mostDeviant, *Percentile, currentAbove/Below, averageAbove/Below, maximumAbove/Below, filterSeries. Which series come back depends on the entire requested range.
  • Bucketing: summarize, hitcount, smartSummarize. These look decomposable — with alignToFrom=false, summarize’s buckets align to absolute interval boundaries — but the bucket covering either edge of the requested window is summarized from only the points inside that window. Measured on graphite-web 1.1.10, the same absolute one-hour bucket reported 134885.899, 302921.880 and 307388.038 over three different windows. Caching such a value and reusing it for a different window would return data the origin never produced, so these take the object lane.
  • Time-shifting: timeShift, timeStack, timeSlice, delay.
  • Generators and tag queries: constantLine, threshold, timeFunction, randomWalk, sinFunction, identity, seriesByTag, groupByTags, events, and template().

The object lane is still a cache: the whole response is cached by request URL with a TTL, keyed on every render parameter. It is always correct because it makes no claim about the response’s internal structure.

Multiple targets

Grafana sends every target of a panel in one /render call. Trickster splits a multi-target request into one delta-cached fetch per target and merges the results in target order, so targets on different ladders are still accelerated. If any single target cannot be accelerated, the whole request is served unaccelerated, because consolidation at the origin spans all series in the response and must be applied consistently.

maxDataPoints and consolidation

Grafana sends maxDataPoints on every panel, sized to the panel’s pixel width. Two panels of different widths showing the same query would otherwise be two different cache entries.

Trickster strips maxDataPoints from upstream requests, caches at the origin’s native resolution, and applies consolidation when rendering the response to each client. One cached series therefore serves every panel width, every output format, and noNullPoints, jsonp, pretty and tz variations.

The consolidation reproduces graphite-web’s renderViewJson exactly, including its start “nudge”, its treatment of xFilesFactor, the consolidateBy function, and the maxDataPoints=1 special case — verified byte-for-byte against a live origin.

Set passthrough_max_data_points: true if you need the origin’s own consolidation byte-for-byte instead. Requests carrying maxDataPoints are then served unaccelerated, which for Grafana means most requests, so this largely disables acceleration. It exists for operators who must guarantee identical bytes to a pre-existing consumer.

Sizing

Two of the common backend settings default differently for Graphite, because Graphite is fetched at its native resolution: maxDataPoints is stripped upstream (see maxDataPoints), so Trickster buffers and caches every point Whisper holds for the window, not the few hundred a dashboard draws. The generic defaults are sized for origins that consolidate before responding.

SettingGeneric defaultGraphite default
max_object_size_bytes512 KB64 MB
timeseries_retention_factor1024 points524288 points

The Graphite values come from pkg/backends/graphite/options/defaults.go and are applied only where the configuration is silent, so setting either one in the file always wins.

max_object_size_bytes

The delta cache must buffer an entire response to model it. At native resolution a wide window on a fine archive is large: in the developer environment, a 120-day panel over two series on a 5-minute archive is 34,560 points per series and about 1.6 MB of JSON — three times the generic 512 KB limit.

A response over the limit cannot be delta-cached. Trickster logs upstream response exceeded MaxObjectSizeBytes and serves the request through the object lane instead, which streams rather than buffers — so clients still get their data, but that panel is never accelerated. Raise the limit if you serve panels wider than the 64 MB default holds:

    max_object_size_bytes: 134217728   # 128MB

timeseries_retention_factor

The cache keeps this many points per series, cropping older ones. The generic default of 1024 points is only about 3.5 days at a 5-minute step, so a 90-day dashboard panel would have almost its entire range cropped after each fetch and refetched on the next — a partial hit that re-fetches nearly everything. The Graphite default of 524288 covers 5 years at a 5-minute step, or 60 days at a 10-second one.

Raise it if you serve a wider window at a finer step than that, and lower it to cap what one series may occupy:

    timeseries_retention_factor: 1048576

Cache storage

Trickster does not roll up between rungs: a 10-second cached chunk is never reused to answer a query that resolves to 60 seconds, because the rollup Trickster computed would not match what Whisper’s lower archive holds (which depends on the metric’s aggregationMethod and xFilesFactor). A different step is a different cache key.

The practical implication is that a metric is cached once per rung it is queried at. A dashboard with a 1-hour, a 24-hour and a 30-day view of the same metric holds three cached series for it, not one. Size the cache for the distinct (metric, step) pairs your dashboards produce, not for the number of distinct metrics.

Metrics, logs, and tracing

Graphite-specific metrics, in addition to the standard trickster_proxy_requests_total{provider="graphite"} and trickster_cache_* families:

MetricTypeNotes
trickster_graphite_resolution_lookups_total{backend_name,confidence,source}counterOne per resolved request
trickster_graphite_probes_total{backend_name,kind,result}counterShould spike at start and fall to zero
trickster_graphite_ladders{backend_name}gaugeDistinct ladders known; should flatten at a small number
trickster_graphite_registry_entries{backend_name,layer}gaugeleaf, ladder, target, negative
trickster_graphite_step_mispredictions_total{backend_name}counterShould be flat at zero
trickster_graphite_fallbacks_total{backend_name,reason}counterWhy requests were not accelerated

No metric path, target expression or query text ever appears in a label; every label value comes from a closed set. See metrics.md.

Logs: a Debug line per resolution decision carrying the target, a bucketed age, the lane, confidence, source and step (or reason when declined); Warn on every misprediction and on every negative-cache entry.

Traces: GraphiteExpand (a child of the request span), GraphiteProbe (kind, result, step) and GraphiteLearnLadder (ladder, probe count).

Operations and troubleshooting

Everything is falling back

Check trickster_graphite_fallbacks_total by reason.

  • unknown_step dominating right after a start is normal; it should fall as ladders are learned. If it does not, resolution is failing — look for graphite ladder learning failed; negative-cached warnings.
  • function_not_allowlisted means the dashboard uses functions outside the allowlist. That is expected and correct, not a defect.
  • non_series_format means the client is asking for images; those panels cannot be delta-cached.

Probing never quiets down

trickster_graphite_probes_total should approach zero after warmup and trickster_graphite_ladders should flatten. If probing continues:

  • Metric names may be high-cardinality or ephemeral, so new leaves appear constantly. Raise resolution_registry.max_entries, or accept the cost.
  • The registry may be evicting under max_entries pressure and relearning. Compare trickster_graphite_registry_entries{layer="leaf"} against the configured maximum.
  • Learning may be failing and retrying under backoff. Check the warnings.

Step mispredictions are non-zero

Something changed a metric’s on-disk ladder — usually whisper-resize.py, a schema edit followed by new file creation, or a metric that moved between schema patterns. Trickster recovers on its own (it relearns and re-serves the request unaccelerated), so this is not a client-visible error, but the counter should return to flat once the new ladders are learned. Persistent mispredictions on one namespace suggest an origin whose resolution is not a function of now - from at all — a clustered graphite-web fronting stores with different schemas, for example.

A panel is never accelerated but should be

  • Look for upstream response exceeded MaxObjectSizeBytes in the logs and raise max_object_size_bytes.
  • Check whether the panel’s window is wholly beyond the metric’s maxRetention; those requests have nothing to cache.
  • A single non-allowlisted target in a multi-target panel makes the whole panel unaccelerated.

Repeated partial hits on wide panels

Raise timeseries_retention_factor. A partial hit that re-fetches nearly the whole range on every request is the signature of cropping.

Verifying correctness against the origin

Because the object lane and the delta lane must both be byte-identical to the origin, the simplest check is to configure two data sources in Grafana — one direct to Graphite, one through Trickster — and compare panels. The developer environment (docs/developer/environment/) is set up this way, with a dashboard whose panels deliberately exercise archive boundaries, the retention edge, schema drift, non-allowlisted functions and mixed-ladder multi-target requests.

Known gaps

  • Fast Forward is not implemented. Graphite’s coarsest-rung behavior and its until > now clamp make the “one extra instant datapoint” approach ill-defined. fast_forward_disable is effectively always on.
  • No ALB / Time Series Merge support. Merging across replicas requires the replicas’ ladders to agree, an invariant not yet established. Graphite backends can still be members of non-TSM ALB mechanisms.
  • Non-decomposable functions are not accelerated. A future release may evaluate them inside Trickster over cached native-resolution leaves, which would move movingAverage, derivative, summarize, highest* and friends into the accelerated path.
  • No cross-resolution rollup, as described under Cache storage.
  • Tag-based queries (seriesByTag, groupByTags) are not accelerated. They are proxied and object-cached.
  • Non-ASCII metric names (graphite-web’s UTF8_METRICS) are not supported by the target parser; such targets fail to parse and take the object lane.

6.6.5 - MySQL Provider

Trickster can accept native MySQL client connections, proxy each authenticated session to a MySQL origin, and cache eligible text-protocol query results. This guide describes the initial supported contract. Behavior not listed here is rejected or proxied without caching; cache eligibility always fails closed.

Compatibility

The supported matrix is:

  • Oracle MySQL 8.4 LTS through 9.7 LTS;
  • Grafana 11.0 and above using Grafana’s built-in MySQL data source; and
  • native clients that use mysql_native_password and the supported commands below.

MariaDB, Percona Server, Aurora MySQL, other MySQL-compatible products, older Grafana releases, and third-party Grafana MySQL data sources are not part of the initial compatibility claim, but should work without problems. File an issue if you discover a compatibility issue.

The developer environment pins MySQL 8.4 and Grafana 13.

Direct backend configuration

The listener accepts the MySQL wire protocol on one TCP port. MySQL negotiates TLS in-band, so do not configure an HTTP-style tls_port.

listeners:
  mysql-native:
    protocol: mysql
    address: ""
    port: 8486
    connections_limit: 200
    mysql:
      handshake_timeout: 10s
      read_timeout: 30s
      write_timeout: 30s
      idle_timeout: 5m
      max_packet_size_bytes: 16777215
      max_query_size_bytes: 1048576

authenticators:
  mysql-clients:
    provider: basic
    users:
      grafana_reader: ${GRAFANA_MYSQL_PASSWORD}

caches:
  mysql-cache:
    provider: memory
    memory:
      max_size_bytes: 536870912

backends:
  mysql-primary:
    provider: mysql
    listener_names: [mysql-native]
    authenticator_name: mysql-clients
    origin_url: mysql://trickster_ro:REDACTED@mysql.example:3306/analytics
    cache_name: mysql-cache
    timeout: 30s
    max_concurrent_conns: 200
    max_object_size_bytes: 8388608
    mysql:
      max_result_rows: 100000
      max_result_size_bytes: 67108864
    healthcheck:
      interval: 5s
      timeout: 3s
      failure_threshold: 3
      recovery_threshold: 2

Exactly one direct MySQL backend or one supported MySQL User Router ALB maps to a MySQL listener. The origin URL must use the mysql scheme and include an origin username. Percent-encode reserved username, password, and database characters. Configuration stringification and the sanitized management configuration redact an embedded origin password, but the source configuration must still be protected as a secret.

The authenticator is the only downstream credential source. Trickster terminates downstream authentication; it does not pass client credentials to the origin or silently fall back to origin credentials. Native-password verification requires the original password, so MySQL authenticator entries must be plaintext values, including environment-expanded values or a protected CSV users file. Htpasswd hashes are rejected for this listener. Do not expose the raw configuration management endpoint outside a trusted administrative network.

TLS

Downstream TLS

Add a server certificate and key to the listener-facing backend:

backends:
  mysql-primary:
    # ...direct backend settings above...
    require_tls: true
    tls:
      full_chain_cert_path: /etc/trickster/tls/server.crt
      private_key_path: /etc/trickster/tls/server.key

Without a server pair, downstream TLS is disabled. With the pair, TLS is optional unless require_tls: true rejects plaintext clients. TLS 1.2 is the minimum. Downstream client certificates are not supported in the initial release.

Upstream TLS

The upstream modes are:

  • no upstream TLS fields: plaintext;
  • insecure_skip_verify: true: encryption without server verification;
  • one certificate_authority_paths entry: CA and hostname verification; and
  • a complete client_cert_path/client_key_path pair: mutual TLS, combined with verified or explicitly unverified server mode.
backends:
  mysql-primary:
    # ...direct backend settings above...
    tls:
      certificate_authority_paths:
        - /etc/trickster/tls/origin-ca.crt
      client_cert_path: /etc/trickster/tls/origin-client.crt
      client_key_path: /etc/trickster/tls/origin-client.key

Certificate/key settings must be complete pairs. More than one upstream CA path, require_tls without a downstream server pair, or an incomplete pair is a configuration error. There is no CA-verification-without-hostname mode.

Connections, limits, and lifecycle

Each authenticated downstream connection owns exactly one upstream connection. There is no pooling or multiplexing, which prevents transaction, database, and session state from leaking between clients. Size connections_limit and max_concurrent_conns together and leave capacity for health probes. A useful starting point is one admitted origin connection per downstream connection, plus normal operational headroom.

Listener limits protect the downstream boundary:

  • connections_limit limits concurrent client connections (0 is unlimited);
  • handshake, command-read, response-write, and idle timeouts bound each phase;
  • max_packet_size_bytes is fixed at 16 MiB minus one; and
  • max_query_size_bytes limits SQL text within that packet ceiling.

Backend limits protect the origin and cache:

  • timeout bounds connect and query work;
  • max_concurrent_conns bounds upstream admission;
  • mysql.max_result_rows and mysql.max_result_size_bytes bound results; and
  • max_object_size_bytes skips caching an otherwise valid oversized object.

Handshake, packet, query, read, write, and idle failures close the client connection. Origin timeout or concurrency rejection closes the affected upstream connection and returns a bounded MySQL error. A row/byte overflow closes both sides because a partial result may already have been emitted.

On shutdown, Trickster stops accepting new sessions and drains existing ones within the configured management drain window. A reload that changes MySQL authentication, listener transport, certificates, origin transport, routing, or relevant limits restarts and drains the listener; an established session is never silently moved to new credentials, TLS policy, or a different origin.

Protocol behavior

Supported commands are:

  • COM_QUERY containing one text-protocol statement;
  • COM_INIT_DB;
  • COM_PING;
  • COM_QUIT; and
  • COM_RESET_CONNECTION.

Reset discards the upstream connection and tracked session state. The next command opens a fresh origin connection.

The initial release rejects binary prepared statements; SQL PREPARE, EXECUTE, and DEALLOCATE; server cursors; binary results; compression; LOCAL INFILE and all LOAD statements; multi-statements; stored-procedure CALL; HELP, XA, HANDLER, and CACHE INDEX; executable version comments; unclassified text response shapes; COM_CHANGE_USER; replica registration; and binlog commands. Connection attributes may be syntactically accepted but are ignored for authentication, routing, cache identity, and query semantics. Malformed, oversized, timed-out, or partially streamed commands close the connection. A fully consumed unsupported command normally leaves it usable.

Successful DML and DDL are proxied and never cached. They do not invalidate existing OPC or DPC entries, so another session may see cached data until TTL expiry or eviction. Use short TTLs or purge affected cache data when immediate read-after-write visibility across sessions is required.

Cache classification

Trickster uses three outcomes:

  • Delta Proxy Cache (DPC): a supported deterministic aggregate time-series SELECT with one unambiguous time output, literal positive cadence, safe result shape and ordering, and an inclusive-lower/exclusive-upper raw-time predicate;
  • Object Proxy Cache (OPC): a deterministic single-result SELECT that is safe to cache as one object but is not proven reusable by time extent; and
  • proxy-only: mutations, transactions, unsupported or unsafe session state, non-deterministic or rejected shapes, and any query whose safety cannot be established.

DPC supports the tested zero-phase integer/cast and FLOOR(UNIX_TIMESTAMP(column) / n) * n bucket forms when n is the same positive integer literal on both sides. Dynamic, mismatched, non-positive, overflowing, ambiguous, or non-zero-phase forms fail closed. Qualified and backtick-quoted identifiers, multiple numeric values, and string dimensions are supported only in the shapes recorded by the compatibility corpus.

The authenticated downstream username, selected terminal backend, normalized database, supported time-zone literal, and query identity participate in cache keys. Cache entries are never shared across authenticated users merely because their SQL is identical.

Grafana macros and exact SQL shapes

Only patterns represented by the versioned compatibility corpus are published as supported. The corpus covers $__time, $__timeEpoch, $__timeFilter, $__timeFrom, $__timeTo, $__timeGroup, $__timeGroupAlias, $__unixEpochFilter, $__unixEpochFrom, $__unixEpochTo, $__unixEpochGroup, and $__unixEpochGroupAlias. Its documented minimum $__interval is one minute.

Grafana’s normal inclusive $__timeFilter expansion is OPC:

SELECT
  CAST(CAST(UNIX_TIMESTAMP(ts)/(300) AS SIGNED)*300 AS SIGNED) AS time,
  COUNT(*) AS samples
FROM telemetry
WHERE ts BETWEEN FROM_UNIXTIME(1785542400)
             AND FROM_UNIXTIME(1785628800)
GROUP BY time
ORDER BY time

Compose $__timeFrom() and $__timeTo() into a half-open predicate for DPC:

SELECT
  CAST(CAST(UNIX_TIMESTAMP(ts)/(60) AS SIGNED)*60 AS SIGNED) AS time,
  AVG(value) AS mean_value
FROM telemetry
WHERE site = 'denver'
  AND ts >= FROM_UNIXTIME(1785542400)
  AND ts < FROM_UNIXTIME(1785628800)
GROUP BY time
ORDER BY time

The epoch-second equivalent is:

SELECT
  CAST(CAST(epoch_seconds/(300) AS SIGNED)*300 AS SIGNED) AS time,
  SUM(requests) AS requests
FROM service_rollups
WHERE region = 'us-west'
  AND epoch_seconds >= 1785542400
  AND epoch_seconds < 1785628800
GROUP BY time
ORDER BY time

Trickster normalizes the lower bound up and the exclusive upper bound down to the cadence and caches only complete buckets. A range with no complete bucket normalizes to an empty range. Inclusive upper bounds and Grafana’s strict-lower $__unixEpochFilter expansion remain OPC because they do not prove the same complete-bucket semantics. Native DATETIME/TIMESTAMP, epoch-second integer, and the corpus’s epoch-nanosecond adaptation are supported in their recorded shapes.

Session state

The only cache-safe tracked state is the selected default database and one session-scoped literal SET time_zone = '<value>'. USE and COM_INIT_DB update the database portion of cache identity.

Transactions and savepoints bypass cache while active. Other SET forms, character set or collation changes, sql_mode, lc_time_names, user variables, temporary objects, locks, session-local functions, mutations, and unclassified state changes make that connection cache-unsafe. Reconnect or a successful COM_RESET_CONNECTION is required to restore the configured baseline.

Protocol-aware User Router

A native MySQL User Router selects one direct terminal MySQL backend from the verified downstream username:

listeners:
  mysql-routed:
    protocol: mysql
    port: 8486
    connections_limit: 200

authenticators:
  tenant-clients:
    provider: basic
    users:
      tenant_a_reader: ${TENANT_A_CLIENT_PASSWORD}
      tenant_b_reader: ${TENANT_B_CLIENT_PASSWORD}

caches:
  tenant-a-cache:
    provider: memory
  tenant-b-cache:
    provider: filesystem

backends:
  tenant-a-mysql:
    provider: mysql
    authenticator_name: tenant-clients
    origin_url: mysql://tenant_a_ro:REDACTED@mysql-a.example:3306/analytics
    cache_name: tenant-a-cache
    healthcheck:
      interval: 5s
      timeout: 3s

  tenant-b-mysql:
    provider: mysql
    authenticator_name: tenant-clients
    origin_url: mysql://tenant_b_ro:REDACTED@mysql-b.example:3306/analytics
    cache_name: tenant-b-cache
    healthcheck:
      interval: 5s
      timeout: 3s

  mysql-by-tenant:
    provider: alb
    listener_names: [mysql-routed]
    authenticator_name: tenant-clients
    alb:
      mechanism: ur
      user_router:
        default_backend: tenant-a-mysql
        users:
          tenant_a_reader:
            to_backend: tenant-a-mysql
          tenant_b_reader:
            to_backend: tenant-b-mysql

The listener-facing router owns the one downstream authentication exchange, admission, and TLS. Direct terminal MySQL backend entries must reference the same named authenticator to satisfy backend validation, but do not perform a second client exchange. The terminal owns its origin credentials, upstream TLS, cache, health, limits, and query policy. Routing occurs once before opening the upstream connection and remains sticky for the whole downstream session. Transactions, database changes, and session state never trigger rerouting.

Only direct MySQL terminal backends are supported. Nested ALBs, Rules, cycles, mixed terminal providers, empty routes, and MySQL to_user/to_credential remapping are configuration errors. A router requires an authenticator. An unmapped user uses default_backend when configured; without one, Trickster returns a bounded native MySQL no-route error. HTTP no_route_status_code is not sent on the native connection. A selected unhealthy or unavailable terminal fails as a MySQL availability error; the session is not silently routed elsewhere.

The verified username and selected terminal remain in cache identity. Route metrics use configured router/backend names and bounded outcomes, never the username.

Metrics, logs, and health

Important metrics include:

  • trickster_sql_query_analysis_total{backend_name,dialect,cache_mode,reason};
  • trickster_sql_query_rewrite_failures_total{backend_name,dialect,reason};
  • trickster_sql_query_cache_total{backend_name,dialect,cache_mode,cache_status};
  • trickster_proxy_request_duration_seconds and trickster_proxy_points_total with provider="mysql";
  • trickster_mysql_connections_total, trickster_mysql_active_connections, trickster_mysql_errors_total, trickster_mysql_route_selections_total, and trickster_mysql_command_duration_seconds; and
  • cache operation, usage, limit, and event metrics labeled by cache_name.

Example PromQL, replacing names to match the deployment:

sum by (cache_mode, reason) (rate(trickster_sql_query_analysis_total{backend_name="mysql1"}[5m]))
sum by (reason) (rate(trickster_sql_query_rewrite_failures_total{backend_name="mysql1"}[5m]))
sum by (cache_mode, cache_status) (rate(trickster_sql_query_cache_total{backend_name="mysql1"}[5m]))
histogram_quantile(0.95, sum by (le) (rate(trickster_proxy_request_duration_seconds_bucket{backend_name="mysql1",provider="mysql"}[5m])))
sum by (cache_status) (rate(trickster_proxy_points_total{backend_name="mysql1",provider="mysql"}[5m]))
sum by (operation, status) (rate(trickster_cache_operation_objects_total{cache_name="mysql-cache"}[5m]))
trickster_cache_usage_bytes{cache_name="mysql-cache"}
sum by (reason) (rate(trickster_cache_events_total{cache_name="mysql-cache",event="eviction"}[5m]))

Analysis logs contain bounded backend, cache mode, reason, and statement-type fields. Rewrite and protocol failures use bounded categories; SQL text, passwords, and usernames are not metric labels. Keep debug logging temporary in production because classification logs can be high volume.

Native health checks open a bounded connection with the backend’s configured origin credentials and TLS policy, then issue COM_PING. They use the common interval, timeout, failure, and recovery thresholds. HTTP health request fields do not apply. Health output reports sanitized authentication, TLS, timeout, refused-connection, and server-error categories. See Health Checks for management endpoint details.

Kubernetes deployment

The base examples expose the optional mysql Service/container port on 8486. The safe routed example keeps the complete credential-bearing configuration in deploy/kube/mysql-user-router-secret.yaml and replaces the base ConfigMap volume with deploy/kube/mysql-deployment-patch.yaml. Replace every placeholder and use an encrypted/external secret controller in production. Do not put an origin DSN, downstream passwords, private keys, or client keys in deploy/kube/configmap.yaml.

For direct TLS or mTLS, add the server, CA, and client certificate material to a Kubernetes TLS or opaque Secret, mount it read-only, and reference the mount paths from the backend tls block. Rotating that Secret requires the configuration/certificate reload path described above; established sessions are drained rather than changing transport identity in place.

Operations and troubleshooting

Kubernetes readiness

The native health scheduler does not delay listener startup or make the general health endpoint return a non-200 status. /trickster/ping proves only that the process is running, and a TCP readiness probe on port 8486 proves only that the native listener is accepting connections. The companion manifest uses that TCP probe. Treat the scheduled MySQL status published in /trickster/health?json as the origin-readiness signal in an external controller or monitoring rule; do not assume the endpoint’s HTTP status reflects origin health. Allow at least interval * failure_threshold or interval * recovery_threshold for a stable transition. User Router terminals report separately, and a failing selected terminal returns a native availability error rather than rerouting the session.

Capacity planning

Start from peak concurrent dashboard/client sessions. Reserve approximately one origin connection per admitted client, then add health and rollout headroom. Bound result rows/bytes below the pod or process memory budget and set max_object_size_bytes below both the result-byte limit and a practical fraction of cache capacity. For an in-memory cache, budget retained objects, index overhead, concurrent result encoding, and normal process memory. Use finite query, command, and idle timeouts so abandoned clients cannot retain connections indefinitely.

Repeated misses or proxy-only outcomes

  1. Group trickster_sql_query_analysis_total by cache_mode and reason.
  2. Confirm the query is deterministic, single-result, and outside a transaction or unsafe session.
  3. For DPC, inspect the expanded SQL: require a literal cadence and >= lower, < upper raw-time predicates.
  4. Confirm the requested interval contains at least one complete cadence bucket and that cache TTL, backfill tolerance, and retention are suitable.
  5. Check username, selected backend, database, and time zone; these intentionally isolate keys.
  6. Inspect cache operation status and eviction metrics for admission failures or churn.

Authentication and TLS

  • Authentication failure: verify the named authenticator, plaintext/CSV credential source, exact username, supported native-password plugin, and filesystem permissions on a mounted users file.
  • Downstream TLS failure: verify the server pair, require_tls, client TLS mode, hostname, and TLS 1.2+ support. Do not use tls_port.
  • Upstream TLS failure: verify the single CA path, hostname in the origin URL, complete client pair when using mTLS, file permissions, and whether insecure_skip_verify was intentionally selected.

Origin and cache failures

During an origin outage, health transitions and MySQL error metrics should rise; existing safe cache hits may continue, while misses fail. Restore origin health before increasing connection limits. A cache-provider failure should surface in cache operation/event metrics and logs; query handling fails closed to the provider’s normal proxy behavior rather than treating an unknown cache state as a hit.

Certificate rotation and relevant configuration reloads drain affected native listeners. Monitor active connections, authentication/TLS errors, route selection outcomes, health state, rewrite failures, cache status, and p95 latency until the old sessions have drained.

Rollout and rollback

Canary the listener/backend configuration with a limited client population. Compare against the pre-release baseline and roll back for a sustained increase of 5 percentage points in proxy/origin/protocol errors, 10 percentage points in proxy-only classifications, 1 percentage point in rewrite failures, or 20% in p95 proxy latency. Roll back by restoring the previous configuration and binary/image on the release branch, reloading or restarting Trickster, and allowing the old MySQL listener to drain. Do not move an established session between versions. Preserve metrics and sanitized logs for the incident record.

Environment variables and reload behavior

The global TRK_ORIGIN_URL and TRK_ORIGIN_TYPE variables map only to the default backend and are suitable only for a basic single-backend deployment. There are no one-to-one TRK_* variables for MySQL listener limits, backend limits, TLS, health, or User Router options; configure those in YAML. Authenticator users values support ${VARIABLE} expansion, and supported users files and secret volume mounts are preferred for production. The origin_url field does not expand ${VARIABLE} in structured YAML; store a configuration containing an embedded DSN as a protected secret, not a public ConfigMap.

Validate before rollout:

trickster -validate-config -config /etc/trickster/trickster.yaml

SIGHUP and the management reload endpoint reload a changed file. Authentication, TLS, transport, routing, and terminal runtime changes restart and drain the affected MySQL listener. Cache policy changes apply to new work; existing cache objects remain subject to their stored TTL and normal eviction.

Known compatibility gaps

The initial release does not claim support for:

  • MySQL-compatible server products outside Oracle MySQL 8.4-9.7;
  • downstream plugins other than mysql_native_password or downstream mTLS;
  • prepared/binary protocols, compression, local infile, multi-results, stored procedures, binlog/replication commands, or connection-attribute semantics;
  • general SQL parsing beyond the Vitess-supported and compatibility-corpus shapes;
  • window functions, grouping sets, rollups, ambiguous time axes or buckets, unsafe boolean predicates, non-deterministic outputs, dynamic/non-zero-phase buckets, or inclusive-range DPC;
  • arbitrary session-state caching or cross-session mutation invalidation; and
  • nested/mixed MySQL User Router topologies or origin credential remapping.

The executable SQL contract is pkg/backends/mysql/testdata/compatibility/v1.json. The maintainer-facing frozen release matrix is in docs/developer/mysql-release-contract.md.

7 - Routing & Load Balancing

Directing requests across multiple backends with the ALB, autodiscovery and Rule engine.

7.1 - Application Load Balancer

Trickster 2.x provides an Application Load Balancer that is easy to configure and provides unique features to aid with Scaling, High Availability and other applications. The ALB supports several balancing Mechanisms:

MechanismConfigProvidesDescription
Round RobinrrScalinga basic, stateless round robin between healthy pool members
Time Series MergetsmFederationuses scatter/gather to collect and merge data from multiple replica tsdb sources
First ResponsefrSpeedfans a request out to multiple backends, and returns the first response received
First Good ResponsefgrSpeedfans a request out to multiple backends, and returns the first response received with a status code < 400
Newest Last‑ModifiednlmFreshnessfans a request out to multiple backends, and returns the response with the newest Last-Modified header
User RouterurControlInspects the credentials in the Request and routes it based on the Username

Integration with Backends

The ALB works by applying a Mechanism to select one or more Backends from a list of Healthy Pool Members, through which to route a request. Pool member names represent Backend Configs (known in Trickster 0.x and 1.x as Origin Configs) that can be pre-existing or newly defined.

All settings and functions configured for a Backend are applicable to traffic routed via an ALB - caching, rewriters, rules, tracing, TLS, etc.

In Trickster configuration files, each ALB itself is a Backend, just like the pool members to which it routes. This makes it possible to configure infinite loops (e.g., where ALB1 has ALB2 in its pool, and ALB2 has ALB1 in its pool). However, at startup Trickster will validate ALB configurations by following all ALBs’ possible paths, and exit with a startup failure if any infinite loops are detected.

In addition to (or instead of) a static pool list, an ALB’s pool membership can be discovered and kept current automatically at runtime — from Kubernetes, DNS records, or a watched member-list file. See ALB Autodiscovery.

Mechanisms Deep Dive

Each mechanism has its own use cases and pitfalls. Be sure to read about each one to understand how they might apply to your situation.

Basic Round Robin

A basic Round Robin rotates through a pool of healthy backends used to service client requests. Each time a client request is made to Trickster, the round robiner will identify the next healthy backend in the rotation schedule and route the request to it.

The Trickster ALB is intended to support stateless workloads, and currently does not support Sticky Sessions or other advanced ALB capabilities.

Weighted Round Robin

Trickster supports Weighted Round Robin with a first-class integer weight on pool entries. A pool entry may be a plain backend name (weight 1) or a mapping with an explicit weight:

pool:
  - node01            # weight 1
  - name: node02
    weight: 3         # receives 3 of every 4 requests

Apportionment is exact: over any totalWeight consecutive requests against a stable healthy pool, each member is selected exactly weight times. Weights also carry through from autodiscovery sources that convey them (DNS SRV record weights, member-file weight fields); see ALB Autodiscovery.

The legacy workaround of repeating a member name multiple times in the pool list still functions, but explicit weights replace it and are preferred.

Weights apply to mechanisms that select a single member per request (round robin). Fan-out mechanisms (fr, fgr, nlm, tsm) dispatch to every healthy member regardless of weight.

More About Our Round Robin Mechanism

Trickster’s Round Robin Mechanism works by maintaining an atomic uint64 counter that increments each time a request is received by the ALB. With uniform weights, the ALB performs a modulo operation on the request’s counter value, with the denominator being the count of healthy backends in the pool; the resulting value, ranging from 0 to len(healthy_pool) - 1, indicates the assigned backend based on the counter and current pool size. With mixed weights, the modulo denominator becomes the pool’s total weight, and each member owns a contiguous weight-sized span of that rotation. Selection remains lock- and allocation-free in both forms.

Example Round Robin Configuration

backends:

  # traditional Trickster backend configurations

  node01:
    provider: reverseproxycache # will cache responses to the default memory cache
    path_routing_disabled: true # disables frontend request routing via /node01 path
    origin_url: https://node01.example.com # make requests with TLS
    tls: # this backend might use mutual TLS Auth
      client_cert_path: ./cert.pem
      client_key_path: ./cert.key

  node02:
    provider: reverseproxy      # requests will be proxy-only with no caching
    path_routing_disabled: true # disables frontend request routing via /node02 path
    origin_url: http://node-02.example.com # make unsecured requests
    request_headers: # this backend might use basic auth headers
      Authoriziation: "basic jdoe:${NODE_02_AUTH_TOKEN}"

  # Trickster 2.x ALB backend configuration, using above backends as pool members

  node-alb:
    provider: alb
    alb:
      mechanism: rr # round robin
      pool:
        - node01 # as named above; weight 1
        - node02
        # to weight the pool, use the mapping form on any entry:
        # - name: node02
        #   weight: 2 # node02 would receive 2 of every 3 requests

Here is the visual representation of this configuration:

Time Series Merge

The Time Series Merge mechanism supports both High Availability and federation. Each physical backend represents one logical data shard. Set the backend-level replica_group option to the same value on physical backends that are HA replicas of that shard. TSM first coalesces those replicas, using configured pool order to resolve overlapping points and later replicas to fill gaps, and then reduces the distinct logical shards.

When a TSM pool member is itself an ALB (for example, a round-robin ALB over Prometheus backends), set replica_group on that immediate nested ALB. Trickster uses the wrapper as the replica-group boundary while delegating TSM planning and finalization to its terminal Prometheus provider. Other ALBs and non-TSM providers cannot set an explicit replica group.

When replica_group is omitted, it defaults to the backend name, so existing configurations continue to treat every backend as a distinct shard. Explicitly set it for HA pools that use non-idempotent aggregations such as sum, count, or avg; otherwise replicas will be counted as separate data.

Replica grouping is backend-global, not ALB-specific. A backend cannot be a replica in one ALB and a disjoint shard in another; define a second backend entry if both views are required. Partially overlapping datasets are not representable: one backend belongs to one logical shard for all TSM queries. Injected labels remain useful output and routing metadata, but they do not establish replica provenance.

If replicas disagree at the same logical point, the first configured member wins deterministically and Trickster records a conflict metric and warning log. A failed replica does not make a response partial when another replica covers its group. If an entire logical group is unavailable, the response is marked partial and includes a warning.

For request paths that are not mergeable by the configured time series provider, TSM does not fan the request out. Those requests are dispatched directly to the first live pool target. The same first-live-target fallback is used when a request cannot be prepared for the merge path.

Merge Strategy

Within each configured replica group, TSM deduplicates values when merging series with identical labels — for each timestamp, only one replica value is kept. Across different groups it uses the query’s merge strategy.

For Federation use cases where backends hold different, non-overlapping data, Trickster automatically selects a merge strategy per query by inspecting the outermost PromQL aggregation operator. No configuration is required. This is particularly important for PromQL aggregation queries like sum() or avg(), which strip labels from results and cause series from different backends to appear identical.

Outer OperatorTrickster Merge Behavior
sumSum of values per unique label set + timestamp
countSum of values per unique label set + timestamp
count_valuesSum of values per unique label set + timestamp
minMinimum value per unique label set + timestamp
maxMaximum value per unique label set + timestamp
groupDeduplicate per unique label set + timestamp
avgDual queries (avg→sum and avg→count); weighted arithmetic mean per unique label set + timestamp
topk, bottomkQuery the inner expression across backends, merge it, then apply final top/bottom-k selection per timestamp and aggregation group
stddev, stdvarPool shard-local count, mean, and variance states, then finalize the global population variance or standard deviation
quantileQuery the inner expression, merge all float samples globally, then calculate the exact quantile per timestamp and aggregation group
limit_ratioApply Prometheus-compatible label-hash sampling, globally finalizing a supported inner aggregation when necessary
limitkQuery the inner expression, merge it globally, then retain the first k samples in stable TSM series order per timestamp and aggregation group
(none)Deduplicate (default)

For avg queries, Trickster issues two concurrent sub-queries per backend shard — one rewriting the outer avg to sum and another to count — then computes a true weighted arithmetic mean (sum_total / count_total) per series per timestamp. This avoids the skew introduced by a naïve avg-of-averages when backends have different data cardinalities.

For topk and bottomk, Trickster sends the inner expression to each backend, merges those inner results using the inner expression’s merge strategy, then applies the final rank-and-trim step per timestamp and aggregation group. This prevents each backend’s local topk/bottomk result from being weighted equally during the merge. If the inner expression is avg, Trickster still uses the weighted sum/count rewrite before applying the final rank. This also applies when the rank aggregation is wrapped in sort() or sort_desc().

For stddev and stdvar, Trickster requests the shard-local count, mean, and population variance and pools those states before finalizing the requested global value. Native histograms are excluded from this float-only calculation. Supported already-aggregated inner expressions such as count, min, max, and group are merged globally before the outer variance aggregation.

For quantile, Trickster sends the inner expression to every backend, globally merges supported inner aggregations, ignores native-histogram samples, and calculates Prometheus’s exact sort-and-interpolate value independently for each timestamp and group. Exact quantiles require every relevant float sample, so their fanout responses can be substantially larger than shard-local quantiles and remain subject to the configured response-capture limits. A capture-limit failure is returned rather than silently substituting an approximate result.

For limit_ratio, selection uses the same complete-label-set hash threshold as Prometheus. Supported inner aggregations are merged before the ratio is applied; shard-local expressions can be sampled by each backend because the hash decision for a given label set is independent of shard placement.

For limitk, Trickster reproduces the current Prometheus evaluator’s first-visited algorithm over TSM’s stable merged-series order: lexicographic JSON serialization of the complete label set, followed by the series name. Selection is independent for every timestamp and aggregation group, retains complete labels and both float and native-histogram samples, and does not rank candidates by value or label hash.

Prometheus does not define a canonical storage visitation order for limitk. A separate Prometheus deployment whose storage returns the same series in a different order may therefore select different labels. Trickster guarantees the requested cardinality, grouping, and repeatability for the same merged input, and fanout completion order does not affect the result. limitk remains an experimental PromQL operator, so this compatibility contract may change with Prometheus.

For an unsupported inner expression, Trickster retains the established per-shard fallback and injects a warnings entry in the Prometheus response body to alert the caller that results may be inaccurate.

When a non-dedup strategy is in effect and backends have injected labels configured, those labels are automatically stripped before merging. This ensures series from different backends hash identically for aggregation, and the injected labels do not appear in the response.

Native Histograms

Native histogram samples are preserved through the merge rather than being numerically aggregated. When a timestamp has a histogram on one backend and a float sample on another (or histograms on both), the histogram value is kept as-is — numeric aggregators like sum only apply across float samples. This prevents mixed-type series from being corrupted into garbage values when backends return a mix of float and histogram samples at the same timestamp.

Max Query Range Limitation

Trickster ALB supports enforcing a max_query_range duration on ALB backends. For details on how to configure and use query range limits, see the Query Range Limits documentation.

Providers Supporting Time Series Merge

Trickster currently supports Time Series Merging for the following TSDB Providers:

Provider Name
Prometheus

We hope to support more TSDB’s in the future and welcome any help!

Example TS Merge Configuration

backends:

  # prom01a and prom01b are redundant and poll the same targets
  prom01a:
    provider: prometheus
    replica_group: prom01
    origin_url: http://prom01a.example.com:9090
    prometheus:
      labels:
        region: us-east-1

  prom01b:
    provider: prometheus
    replica_group: prom01
    origin_url: http://prom01b.example.com:9090
      labels:
        region: us-east-1

  # prom-alb-01 scatter/gathers to prom01a and prom01b and merges responses for the caller.
  # Enforces a max 14-day time range limit on all incoming merge requests.
  prom-alb-01:
    provider: alb
    max_query_range: 14d
    alb:
      mechanism: tsm # time series merge
      pool: 
        - prom01a
        - prom01b

  # prom02 and prom03 poll unique targets but produce the same metric names as prom01a/b
  prom02:
    provider: prometheus
    origin_url: http://prom02.example.com:9090
      labels:
        region: us-east-2

  prom03:
    provider: prometheus
    origin_url: http://prom03.example.com:9090
      labels:
        region: us-west-1

  # prom-alb-all scatter/gathers prom01a/b, prom02 and prom03 and merges their responses
  # for the caller. The merge strategy is automatically selected per-query based on the
  # outer PromQL aggregation operator. Injected labels are automatically stripped before
  # merging so that series from different backends are combined correctly. Because prom01a
  # and prom01b are in the same replica_group, their values are de-duplicated before being
  # merged/reduced with prom02 and prom03.
  prom-alb-all:
    provider: alb
    alb:
      mechanism: tsm
      pool:
        - prom01a
        - prom01b
        - prom02
        - prom03

Here is the visual representation of a basic TS Merge configuration:

First Response

The First Response mechanism fans a request out to all healthy pool members, and returns the first response received back to the client. All other fanned out responses are cached (if applicable) but otherwise discarded. If one backend in the fanout has already cached the requested object, and the other backends do not, the cached response will return to the caller while the other backends in the fanout will cache their responses as well for subsequent requests through the ALB.

This mechanism works well when using Trickster as an HTTP object cache fronting multiple redundant origins, to ensure the fastest response possible is delivered to downstream clients - even if the HTTP Response Code indicates an error in the request or by the first backend to respond.

First Response Configuration Example

backends:
  node01:
    provider: reverseproxycache
    origin_url: http://node01.example.com

  node02:
    provider: reverseproxycache
    origin_url: http://node-02.example.com

  node-alb-fr:
    provider: alb
    alb:
      mechanism: fr # first response
      pool:
        - node01
        - node02

Here is the visual representation of this configuration:

First Good Response

The First Good Response (fgr) mechanism acts just as First Response does, except that it waits to return the first response with an HTTP Status Code < 400. If no fanned out response codes are in the acceptable range once all responses are returned (or the timeout has been reached), then the healthiest response, based on min(all_responses_status_codes), is used.

This mechanism is useful in applications such as live internet television. Consider an operational condition where an object may have been written to Origin 1, but not yet written to redundant Origin 2, while users have already received references to and begin requesting the object in a separate manifest. Trickster, when used as an ALB+Cache in this scenario, will poll both backends for the object and cache the positive responses from Origin 1 for serving subsequent requests locally, while a negative cache configuration will avoid potential 404 storms on Origin 2 until the object can be written by the replication process.

Custom Good Status Codes List

By default, fgr will return the first response with a status code < 400. However, you can optionally provide an explicit list of good status codes using the fgr.status_codes configuration setting, as shown in the example below. When set, Trickster will return the first response to be returned that has a status code found in the configured list.

First Good Response Configuration Example


negative-caches:
  default: # by default, backends use the 'default' negative cache
    "404": 500 # cache 404 responses for 500ms

backends:
  node01:
    provider: reverseproxycache
    origin_url: http://node-01.example.com

  node02:
    provider: reverseproxycache
    origin_url: http://node-02.example.com

  node-alb-fgr:
    provider: alb
    alb:
      mechanism: fgr # first good response
      pool:
        - node01
        - node02
      fgr:
        status_codes: [ 200, 201, 204 ] # only consider these codes when selecting a response

Here is the visual representation of this configuration:

Newest Last-Modified

The Newest Last-Modified mechanism is focused on providing the user with the newest representation of the response, rather than responding as quickly as possible. It will fan the client request out to all backends, and wait for all responses to come back (or the ALB timeout to be reached) before determining which response is returned to the user.

If at least one fanout response has a Last-Modified header, then any response not containing the header is discarded. The remaining responses are sorted based on their Last Modified header value, and the newest value determines which response is chosen.

This mechanism is useful in applications where an object residing at the same path on multiple origins is updated frequently, such as a DASH or HLS manifest for a live video broadcast. When using Trickster as an ALB+Cache in this scenario, it will poll both backends for the object, and ensure the newest version between them is used as the client response.

Note that with NLM, the response to the user is only as fast as the slowest backend to respond.

Newest Last-Modified Configuration Example

backends:
  node01:
    provider: reverseproxycache
    origin_url: http://node01.example.com

  node02:
    provider: reverseproxycache
    origin_url: http://node-02.example.com

  node-alb-nlm:
    provider: alb
    alb:
      mechanism: nlm # newest last modified
      pool:
        - node01
        - node02

Here is the visual representation of this configuration:

User Router

The User Router mechanism is used to control a Request’s destination Backend based on the username in the request. A default Backend (for no-user and users not in the manifest) can be configured, as well as a Backend per-user.

Native MySQL listeners use a deliberately narrower User Router topology than HTTP backends: one authenticated listener-facing User Router may select only direct terminal MySQL backends, selection is sticky for the session, and to_user/to_credential remapping is rejected. See the MySQL Provider Guide for the complete authentication, routing, health, cache-identity, and no-route contract.

When a User Router ALB is configured to use an Authenticator, the ALB can also modify a Request’s credentials before passing it off to the destination Backend. In the graphic below, user casey will be routed to the readersBackend, which proxies to a read-only database server with the dbreader credentials; while user taylor will be routed to the writersBackend, which proxies to a read-write database server with the dbwriter credentials. Here is the example configuration corresponding to the graphic:

Credential replacement is applied only when the user’s configured to_backend target is selected. When a request instead uses default_backend - because the username has no mapping, the mapping does not name a usable runtime target, or the mapped target is unavailable - the request retains its inbound credentials. If a user should receive replacement credentials when routed to the same Backend that also serves as the default, set that Backend explicitly as the user’s to_backend.

backends:
  readersBackend:
    provider: clickhouse
    origin_url: http://read.prod.db.com:8123/

  writersBackend:
    provider: clickhouse
    origin_url: http://write.prod.db.com:8123/

  click-lb-01:
    provider: alb
    authenticator_name: dbUsers
    alb:
      mechanism: ur # User Router Mechanism
      user_router: # User Router-specific configs
        default_backend: readersBackend # optional - users not in the list will route here, origin will 401
        users:
          casey:
            to_user: dbreader # replaces user casey with dbreader in the request's Authorization header
            to_credential: ${DB_READER_PW} # replaces credential in the Authorization header with this env
            to_backend: readersBackend # explicit selection applies casey's credential replacement
          taylor:
            to_user: dbwriter # replaces user taylor with dbwriter in the request's Authorization header
            to_credential: ${DB_WRITER_PW} # replaces credential in the Authorization header with this env
            to_backend: writersBackend # taylor is sent to the writers backend

authenticators:
  dbUsers:
    provider: clickhouse # use the clickhouse authenticator
    users_file: /path/to/user-manifest.csv # this file should include casey and taylor users
    users_file_format: csv # required when users_file is set

Supported Backend Provider Types

The User Router mechanism supports all Backend provider types for default_backend and to_backend values, including other User Router ALBs.

However, config validation will fail if:

  • there are any possible infinite loops between backends configured
  • users could ultimately be routed to different non-virtual (ALB/Rule) backend types by the same User Router ALB. The final ultimate route for all users must be of the same type (regardless of how many additional hops through ALBs and Rules the request would take).
    • In other words: user1 cannot be ultimately routed to a clickhouse backend and user2 be ultimately routed to a prometheus backend by the same User Router ALB.

User Router without an Authenticator

If a User Router ALB does not use an Authenticator, you can still configure user-specific Backend routes. In these cases Trickster will observe (but not authenticate) the username in the request and route based on the observed username. However, Trickster will exit with a validation failure on startup if a User Router ALB that does not utilize an Authenticator is configured to swap credentials. In short: users must be positively authenticated by a Trickster Authenticator for credential swapping to be permitted by the User Router ALB.

When a User Router ALB doesn’t use an Authenticator, Trickster uses the final destination Backend provider type to select a default Authenticator (operating in observe-only mode / no users manifest) for username observation. For clickhouse-destined User Routers, the observe only Authenticator provider is clickhouse. For all other backend provider types, the default the observe only Authenticator provider is basic (Basic Auth).

to_user / to_credential vs Backend Path Header Injection

It is still possible to insert credentials to a Backend proxy request using the request_headers Backend Path config. But any request_headers alterations configured for auth-related headers (e.g., Authorization) are performed by the Backend after being handled by a User Router; so they would overwrite any user-specific to_user and to_credential transformations performed by the User Router ALB.

Default Backend

As shown in the example config above, you can provide a default_backend config to a User Router, and users who are not in the user router list will be routed to this backend.

If you do not supply a default_backend, users who are not in the manifest will receive a default response of 502 Bad Gateway. You can customize the default response code by setting no_route_status_code to a value between 400 and 599 as in this example:

backends:
  prod-01:
    provider: reverseproxy
    origin_url: https://example.com/

  users-lb-01:
    provider: alb
    alb:
      mechanism: ur
      authenticator_name: all-users # not shown for brevity, see above examples
      user_router:
        no_route_status_code: 401 # unauthorized response for users not in allow list
        users: # allowed users
          casey:
            to_backend: prod-01
          taylor:
            to_backend: prod-01
          kris:
            to_backend: prod-01

User Router ALB Backend Pool and Health Checking

The User Router does not rotate through or fan out to a pool of Backends like the other ALB mechanisms. A healthy mapped target is selected directly. When a mapped target is unavailable, the request uses the healthy default_backend without applying the mapped target’s credential replacement. If neither target is available, the router uses its configured no-route response.

That fallback applies to HTTP requests. A native MySQL session whose username has an explicit mapping fails with a MySQL availability error when that mapped terminal is unavailable; it is never redirected to default_backend. Only an unmapped MySQL username may use the configured default terminal.

You can configure a User Router ALB’s backend destinations to be other ALBs with mechanisms that utilize healthchecked pools.

Bounding Per-Member Response Captures

ALB mechanisms that fan out (TSM, FR, FGR, NLM) buffer each pool member’s response in memory before merging or selecting a winner. Without a cap, one misbehaving upstream returning an oversized body can OOM the proxy – an N-way fanout multiplies that by N.

Trickster applies a default cap of 256 MiB per response. A member whose body exceeds the cap is treated as a partial failure: the merged response carries an X-Trickster-Result: phit marker and the trickster_alb_fanout_failures_total{mechanism, reason="truncated"} metric increments.

Override the cap at the backend or ALB level:

backends:
  default:
    max_capture_bytes: 67108864  # 64 MiB, applies to all backends (Prometheus, ClickHouse, ALB members, etc.)

  prom-alb-tsm:
    provider: alb
    alb:
      mechanism: tsm
      max_capture_bytes: 16777216  # 16 MiB, ALB-specific override
      pool:
        - prom01
        - prom02

The ALB-level value takes precedence over the backend-level value, which in turn takes precedence over the 256 MiB default.

Bounding Aggregate In-Flight Captures

max_capture_bytes caps each member’s response individually; a fanout to N members can still buffer up to N * max_capture_bytes in flight. For deployments with large pools or low memory ceilings, set max_fanout_capture_bytes to cap the aggregate buffer across all in-flight slots in a single fanout call. Slots dispatched after the aggregate budget would go negative are fail-fasted (marked Failed, no capture buffer allocated) before the upstream handler runs; the merge sees them as partial failures and the existing fallback path handles it.

backends:
  prom-alb-tsm:
    provider: alb
    alb:
      mechanism: tsm
      max_capture_bytes: 16777216         # 16 MiB per member
      max_fanout_capture_bytes: 67108864  # 64 MiB total across all in-flight slots
      pool:
        - prom01
        - prom02
        - prom03
        - prom04

max_fanout_capture_bytes defaults to 0 (no aggregate cap). Pick a value matching what your trickster instance can afford to buffer per request, independent of pool size.

Maintaining Healthy Pools With Automated Health Check Integrations

Health Checks are configured per-Backend as described in the Health documentation. Each Backend’s health checker will notify all ALB pools of which it is a member when its health status changes, so long as it has been configured with a health check interval for automated checking. When an ALB is notified that the state of a pool member has changed, the ALB will reconstruct its list of healthy pool members before serving the next request.

Health Check States

A backend will report one of three possible health states to its ALBs: unavailable (-1), unknown (0), or available (1).

Health-Based Backend Selection

Each ALB has a configurable healthy_floor value, which is the threshold for determining which pool members are included in the healthy pool, based on their instantaneous health state. The healthy_floor represents the minimum acceptable health state value for inclusion in the healthy pool. The default healthy_floor value is 0, meaning Backends in a state >= 0 (unknown and available) are included in the healthy pool. Setting healthy_floor: 1 would include only available Backends, while a value of -1 will include all backends in the configured pool, including those marked as unavailable.

Backends that do not have a health check interval configured will remain in a permanent state of unknown. Backends will also be in an unknown state from the time Trickster starts until the first of any configured automated health check is completed. A pool member in a permanent unknown state can never reach available, so a healthy_floor: 1 ALB whose members lack health checks would have an empty pool and return 502 for every request. To avoid that, Trickster resets such an ALB’s effective floor to 0 at startup, emits a warning naming the ALB and the un-probed members, and sets the trickster_alb_pool_floor_reset{backend_name} gauge to 1. Configure a health check interval on those members if you want healthy_floor: 1 to apply.

Setting healthy_floor below 0 admits members the probe has confirmed unavailable, not just members in the transient unknown state. If your goal is to keep traffic flowing during the cold-start window before the first probes complete, lower the pool members’ recovery_threshold so they transition out of unknown faster – don’t lower the floor. When healthy_floor < 0 Trickster emits a startup warning and sets the trickster_alb_pool_admits_failing{backend_name} gauge to 1.

Example ALB Configuration Routing Only To Known Healthy Backends

backends:
  prom01:
    provider: prometheus
    origin_url: http://prom01.example.com:9090
    healthcheck:
      interval: 1000ms # enables automatic health check polling for ALB pool reporting

  prom02:
    provider: prometheus
    origin_url: http://prom02.example.com:9090
    healthcheck:
      interval: 1000ms

  prom-alb-tsm:
    provider: alb
    alb:
      mechanism: tsm   # times series merge healthy pool members
      healthy_floor: 1 # only include Backends reporting as 'available' in the healthy pool
      pool:
        - prom01
        - prom02

All-Backends Health Status Page

Trickster 2.x provides a global health status page available at http://trickster:metrics-port/trickster/health or (the configured health_handler_path).

The global status page will display the health state about all backends configured for automated health checking. Here is an example configuration and a possible corresponding status page output:

backends:
  proxy-01:
    provider: reverseproxy
    origin_url: http://server01.example.com
    # not configured for automated health check polling

  prom-01:
    provider: prometheus
    origin_url: http://prom01.example.com:9090
    healthcheck:
      interval: 1000ms # enables automatic health check polling every 1s

  flux-01:
    provider: inflxudb
    origin_url: http://flux01.example.com:8086
    healthcheck:
      interval: 1000ms # enables automatic health check polling every 1s

$ curl "http://${trickster-fqdn}:8481/trickster/health"

Trickster Backend Health Status            last change: 2020-01-01 00:00:00 UTC
-------------------------------------------------------------------------------

prom-01      prometheus   available

flux-01      influxdb     unavailable since 2020-01-01 00:00:00 UTC
                                    
proxy-01     proxy        not configured for automated health checks

-------------------------------------------------------------------------------
You can also provide a 'Accept: application/json' Header or query param ?json

JSON Health Status

As the table footer from the plaintext version of the health status page indicates, you may also request a JSON version of the health status for machine consumption. The JSON version includes additional detail about any Backends marked as unavailable, and is structured as follows:

$ curl "http://${trickster-fqdn}:8481/trickster/health?json" | jq

{
  "title": "Trickster Backend Health Status",
  "updateTime": "2020-01-01 00:00:00 UTC",
  "available": [
    {
      "name": "flux-01",
      "provider": "influxdb"
    }
  ],
  "unavailable": [
    {
      "name": "prom-01",
      "provider": "prometheus",
      "downSince": "2020-01-01 00:00:00 UTC",
      "detail": "error probing target: dial tcp prometheus:9090: connect: connection refused"
    }
  ],
  "unchecked": [
    {
      "name": "proxy-01",
      "provider": "proxy"
    }
  ]
}

7.2 - ALB Autodiscovery

Autodiscovery keeps an ALB pool’s membership current at runtime: instead of hand-maintaining a pool list, Trickster watches a discovery source and adds, updates, and drains pool members as the source changes — with no config reload or listener restart. Discovered members are cloned from a designated template backend, so they inherit its caching, TLS, health check, path, and timeout configuration.

Supported Discovery Providers

There are several discovery providers supported by Trickster

  • Kubernetes (endpointslices, services, or pods, via the Kubernetes API)
  • DNS SRV records (dns_srv)
  • DNS A/AAAA records (dns_a)
  • Watched member-list file (file)

Support for additional providers (consul, ec2, gce, etcd, docker) is planned; until each lands, its users are generally served by the file provider (any external service-discovery tool can emit the member list) or by the DNS providers (e.g., Consul’s DNS interface, cloud private DNS zones, Docker’s embedded DNS). Developers can add providers against a simple interface; see the provider authoring guide.

Configuration Overview

Autodiscovery has three configuration parts:

  1. a named discoverer in the top-level discovery section — the provider and its connection-level settings (Kubernetes client, DNS resolver). Multiple ALBs can share one discoverer, and therefore one client/watch/resolver stack.
  2. a template backend — a normal backend definition marked is_template: true. Templates are never routed, never eligible for is_default, never static pool members, and don’t require an origin_url; they exist to be cloned per discovered member.
  3. the ALB’s alb.discovery block — which discoverer to use, a provider-specific query describing what to select, the template to clone, and guardrail options.
discovery:
  in-cluster:
    provider: kubernetes
    kubernetes:
      in_cluster: true

backends:
  prom-template:
    provider: prometheus
    is_template: true
    cache_name: default
    healthcheck:
      interval: 5s

  prom-alb:
    provider: alb
    alb:
      mechanism: tsmerge
      discovery:
        discoverer_name: in-cluster
        template_backend: prom-template
        query:
          kind: endpointslices
          namespace: monitoring
          service: prometheus
          port: web

A discovery-backed ALB may also list static pool members; discovered members are additive to them. Static pool entries (and discovered members from sources that convey weights) support integer weights for the round_robin mechanism:

pool:
  - static-member          # weight 1
  - name: bigger-member
    weight: 3

Discoverer Connection Options

Each entry in the top-level discovery section declares a provider and that provider’s connection-level block:

providerblockoptions
kuberneteskubernetesin_cluster (default true) or kubeconfig path — mutually exclusive
dns_srv, dns_adnsresolver (host:port; default: system resolver), interval (poll cadence, default 30s, min 1s; record TTLs act as a floor)
filefilepoll_interval (stat-poll fallback cadence, default 30s, min 1s)

A block is only valid on its own provider’s entries; anything else fails startup.

Template Backends

The template_backend is an ordinary backend definition marked is_template: true. A discovered member overrides exactly the template’s name, origin_url (and its derived scheme/host/path-prefix, with the member’s path prefix falling back to the template’s), and replica group (see below); the member’s weight applies to its pool entry. Every other option — cache settings, TLS client configuration, healthcheck, paths, request rewriters, authenticator, timeouts, concurrency limits, tracing — is inherited from the template.

Templates are excluded from everything that applies only to live backends: they are never routed, cannot be is_default, cannot appear in a static ALB pool, and do not require an origin_url. A template must use an origin-serving provider (not alb or rule), and a TSM-merged or replica_group_label-using ALB requires a time-series-merge-capable template provider.

The alb.discovery Block

optiondescriptiondefault
discoverer_namename of a discoverer in the top-level discovery section (required)
template_backendname of a backend with is_template: true (required)
queryprovider-specific selection (required; see each provider below)
min_membersreject snapshots that would shrink the discovered membership below this count, keeping the last-good pool (guards against source blips returning empty results). 0 disables.0
debounce_windowcoalesce membership changes arriving within this window into one pool update, damping flapping sources. 0 disables.0
startup_policyretry starts the ALB with only its static members and keeps retrying the discoverer; fail fails startup when the discoverer is unavailableretry
health_modeprobe: discovered members inherit the template’s active health check. provider: the discoverer’s readiness reporting (currently Kubernetes only) drives member health instead of active probesprobe

Health and Readiness Semantics

In probe mode, each discovered member gets its own health check from the template’s healthcheck config, and the ALB’s healthy_floor applies exactly as it does to static members. If the template has no probe interval and the floor requires passing probes, the floor is reset to 0 with a loud warning (as with static members).

In provider mode, readiness reported by the discoverer maps onto member health: ready members enter as passing (1), not-ready and terminating members as failing (-1), readiness-unknown members as unchecked (0) — so a healthy_floor of 1 excludes members until the provider reports them ready.

Members a provider reports as shutting down (terminating Kubernetes endpoints, deletion-stamped pods) are removed from the snapshot entirely, so they drain from the pool before the workload is killed — enabling zero-error rolling deploys. On removal, a member’s in-flight requests complete, its health check stops, its metrics series are deleted, and its idle upstream connections close after the configured drain timeout.

TSM Replica Groups

When the ALB uses the Time Series Merge mechanism, discovered members participate in replica_group semantics three ways, in priority order:

  1. Per-member groups from the discovery source (highest priority): the Kubernetes provider can read a configured label from each discovered workload via query.replica_group_label, and file provider entries may carry a replica_group field. Members sharing a group value are treated as HA replicas of one logical shard and coalesced; distinct values are distinct shards, merged. This handles sharded-HA topologies (e.g., Thanos-style Prometheus pairs) discovered with a single selector:

    backends:
  prom-alb:
    provider: alb
    alb:
      mechanism: tsmerge
      discovery:
        discoverer_name: in-cluster
        template_backend: prom-template
        query:
          kind: endpointslices
          namespace: monitoring
          service: prometheus
          port: web
          replica_group_label: prometheus/shard

  2. A replica_group set on the template backend: inherited by every member without a source-conveyed group — all such members are HA replicas of one shard (the common HA-pair case).

  3. Neither (default): each member is its own replica group, i.e. its own logical shard — a pure federation/merge of every member.

For the pods and service kinds the label is read from the discovered Pod or Service itself. For endpointslices, endpoints don’t carry pod labels, so setting replica_group_label joins a Pod watch in the query’s namespace to resolve each endpoint’s target pod (see the RBAC note below); an endpoint whose pod isn’t yet observed joins ungrouped and regroups automatically when the pod appears. A member whose group changes is rebuilt in place under the same name.

Because replica_group is only meaningful on time-series backends, configuration validation requires a TSM-capable template_backend whenever the ALB mechanism is tsmerge or replica_group_label is set. The DNS providers convey no grouping metadata; use the template-level group (mode 2) or the file provider for grouped non-Kubernetes pools.

Kubernetes

The kubernetes provider watches the cluster via the API server (shared informers; watch-driven, no polling) and supports three query kinds.

Connection options:

discovery:
  my-cluster:
    provider: kubernetes
    kubernetes:
      in_cluster: true              # use the pod's service account (default)
      # kubeconfig: /path/to/kubeconfig   # or run against a remote cluster

in_cluster and kubeconfig are mutually exclusive; with neither set, in-cluster is assumed.

Query Kinds

endpointslices (default) discovers the ready endpoint addresses of a named Service — the pod IPs behind it — and is the right choice for routing around the Service’s own load balancing:

query:
  kind: endpointslices
  namespace: monitoring
  service: prometheus
  port: web          # port name or number; optional when the service has one port

service discovers the ClusterIP of each Service matching a label selector (or one named Service), one member per Service:

query:
  kind: service
  namespace: monitoring
  selector:
    app: prometheus
  port: 9090

pods discovers pod IPs directly by label selector, without requiring a Service (the original ask of issue #609):

query:
  kind: pods
  namespace: monitoring
  selector:
    app: prometheus
  port: web

namespace defaults to the pod’s own namespace when running in-cluster.

Port and Scheme Resolution

port may be a named port, a number, or omitted when the target declares exactly one port; ambiguity is logged and the object skipped. The member scheme comes from query.scheme if set; otherwise a declared appProtocol: https on the selected port, or a trickster.io/scheme annotation on the watched Service/Pod, selects https; the default is http.

Zero-Error Rolling Deploys

Terminating endpoints are removed from the discovered membership as soon as the change is observed, so members drain out ahead of pod deletion. One piece belongs to the workload, though: Kubernetes marks an endpoint terminating and signals the container at the same moment, so — as with any EndpointSlice consumer, kube-proxy included — the pod must keep serving briefly while its removal propagates. Give discovered workloads a short preStop delay (a few seconds comfortably covers Trickster’s sub-second pickup):

lifecycle:
  preStop:
    sleep:         # native sleep action (Kubernetes >= 1.30); images with
      seconds: 5   # a shell can use exec with command: ["sleep", "5"]

Without it, an instantly-exiting container can produce a brief window of connection errors during rollouts — regardless of what is consuming the EndpointSlices.

RBAC

The provider needs only list and watch on the resources your query kinds use — it never reads Secrets or ConfigMaps and never writes:

query kindapiGroupresourceverbs
endpointslicesdiscovery.k8s.ioendpointsliceslist, watch
service"" (core)serviceslist, watch
pods"" (core)podslist, watch

Queries are namespace-scoped, so a namespaced Role/RoleBinding per watched namespace is sufficient and preferred:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: trickster-autodiscovery
  namespace: monitoring
rules:
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: trickster-autodiscovery
  namespace: monitoring
subjects:
  - kind: ServiceAccount
    name: trickster
    namespace: trickster
roleRef:
  kind: Role
  name: trickster-autodiscovery
  apiGroup: rbac.authorization.k8s.io

Add services and/or pods rules only for the kinds you configure. Note that using replica_group_label with the endpointslices kind joins a Pod watch, so it additionally requires pods list/watch in the query’s namespace. RBAC cannot scope below the resource level, so the grant covers all objects of that resource in the bound namespace. Out-of-cluster (kubeconfig-based) discoverers need the same permissions for their user or service account.

DNS SRV

The dns_srv provider polls SRV records. SRV target and port map to the member address; SRV weight maps to the member’s load-balancing weight; and only the highest-priority tier (the lowest priority value present in the answer) becomes members — lower tiers are treated as standby capacity managed on the DNS side.

discovery:
  corp-dns:
    provider: dns_srv
    dns:
      resolver: 10.0.0.53:53   # optional; default: the system resolver
      interval: 30s            # poll cadence (default 30s)

backends:
  my-alb:
    provider: alb
    alb:
      mechanism: rr
      discovery:
        discoverer_name: corp-dns
        template_backend: my-template
        query:
          srv_name: _prometheus._tcp.example.com
          # scheme: https      # optional; default http

Record TTLs act as a floor on the poll cadence: an answer is never re-resolved before its shortest TTL expires, so interval is the most frequent the provider will query. Resolution failures keep the last-good membership; an authoritative empty answer empties it.

DNS A/AAAA

The dns_a provider resolves a hostname’s A and AAAA records, with a fixed port and scheme from the query. This covers round-robin DNS, headless-service-style DNS outside Kubernetes, Docker’s embedded DNS, and Consul’s DNS interface without a bespoke provider:

query:
  hostname: prometheus.service.consul
  port: 9090           # required
  # scheme: https      # optional; default http

The dns connection options, poll cadence, TTL floor, and failure semantics are the same as dns_srv.

File

Note: like Prometheus’s file_sd, the file provider is the universal integration point — anything that can write a file (cron, sidecar, consul-template, your deploy tooling) can drive an ALB pool.

The provider watches a local YAML or JSON member-list file and applies it atomically on change:

discovery:
  external-sd:
    provider: file
    # file:
    #   poll_interval: 30s   # stat-poll fallback cadence (default 30s)

backends:
  my-alb:
    provider: alb
    alb:
      mechanism: rr
      discovery:
        discoverer_name: external-sd
        template_backend: my-template
        query:
          path: /etc/trickster/members.yaml

The file is a list of members:

- name: prom-1            # optional; defaults to the address
  address: 10.0.0.1:9090  # required host:port
- name: prom-2
  scheme: https           # optional; default http
  address: 10.0.0.2:9090
  path_prefix: /base      # optional
  weight: 3               # optional; default 1
  replica_group: shard-0  # optional; TSM replica group (see above)

Writers should replace the file atomically (write to a temp file in the same directory, then rename). A file that fails to read or parse keeps the last-good membership; an empty file is a valid, empty membership.

Change Detection

The provider uses two mechanisms together (via Trickster’s shared filesystem watcher), so updates are never missed:

  1. Filesystem notification (on the file’s parent directory, debounced): near-instant pickup of writes, atomic renames, and symlink swaps on local filesystems; a directory watch dropped by deletion or recreation is automatically re-armed.
  2. A content-comparing poll of the file (file.poll_interval, default 30s, minimum 1s): the guaranteed fallback wherever notification is unreliable or unavailable.

Guidance for Kubernetes-mounted member files:

  • ConfigMap / Secret volume mounts: kubelet applies updates with an atomic symlink swap inside the mount directory, which generates inotify events — notification works, and updates are picked up promptly. Note that kubelet itself syncs ConfigMap content periodically (typically up to a minute), which usually dominates end-to-end latency.
  • subPath mounts of a ConfigMap/Secret: Kubernetes never updates these after pod start — no mechanism (notification or polling) can see a change. Mount the directory, not a subPath.
  • Network- or FUSE-backed volumes (NFS PVs, some CSI drivers): inotify generally does not observe writes made by other clients, so the stat poll is the effective update mechanism — set file.poll_interval to the freshness your deployment needs.
  • emptyDir/hostPath written by a sidecar in the same pod: notification works.

The poll compares file content (not timestamps), so any change is detected within one poll_interval regardless of filesystem timestamp granularity.

Observing Discovered Members

Discovered members appear on the health status page alongside static pool members, under their generated backend names (<alb-name>-<member-name>), tagged with their provider, owning ALB, and discoverer (e.g., prometheus (prom-alb via in-cluster)).

Membership additions and removals are logged at info with the member name and origin (credentials embedded in origin URLs are masked), and the full membership is logged at debug on every change; all autodiscovery log events carry scope=discovery for easy filtering. Per-ALB member-count, member-change, snapshot-result, and refresh-staleness metrics — and per-discoverer refresh-error counters — are documented in metrics.md. When the ALB has a tracing_name configured, each membership reconcile cycle is traced as an alb.discovery.reconcile span via the standard tracing subsystem; the request hot path is never traced by discovery.

7.3 - Rule Backend

The Rule Backend is not really a true Backend; it only routes inbound requests to other configured Backends, based on how they match against the Rule’s cases.

A Rule is a single inspection operation performed against a single component of an inbound request, which determines the Next Backend to send the request to. The Next Backend can also be a rule Backend, so as to route requests through multiple Rules before arriving at a true Backend destination.

A rule can optionally rewrite multiple portions of the request before, during and after rule matching, by using request rewriters, which allows for powerful and limitless combinations of request rewriting and routing.

Rule Parts

A rule has several required parts, as follows:

Required Rule Parts

  • input_source - The part of the Request the Rule inspects
  • input_type - The source data type
  • operation - The operation taken on the input source
  • next_route - The Backend Name indicating the default next route for the Rule if no matching cases. Not required if redirect_url is provided.
  • redirect_url - The fully-qualified URL to issue as a 302 redirect to the client in the default case. Not required if next_route is provided.

Optional Rule Parts

  • input_key - case-sensitive lookup key; required when the source is header or URL param
  • input_encoding - the encoding of the input, which is decoded prior to performing the operation
  • input_index - when > -1, the source is split into parts and the input is extracted from parts[input_index]
  • input-delimiter - when input_index > -1, this delimiter is used to split the source into parts, and defaults to a standard space (’ ‘)
  • ingress_req_rewriter name - provides the name of a Request Rewriter to operate on the Request before rule execution.
  • egress_req_rewriter name - provides the name of a Request Rewriter to operate on the Request after rule execution.
  • nomatch_req_rewriter name - provides the name of a Request Rewriter to operate on the Request after rule execution if the request did not match any cases.
  • max_rule_executions - limits the number of rules a Request is passed through, and aborts with a 400 status code when exceeded. Default is 16.

input_source permitted values

source nameexample extracted value
urlhttps://example.com:8480/path1/path2?param1=value
url_no_paramshttps://example.com:8480/path1/path2
schemehttps
hostexample.com:8480
hostnameexample.com
port8480 (inferred from scheme when no port is provided)
path/path1/path2
params?param1=value
param(must be used with input_key as described below)
header(must be used with input_key as described below)

input_type permitted values and operations

type namepermitted operations
string (default)prefix, suffix, contains, eq, md5, sha1, modulo, rmatch
numeq, le, ge, gt, lt, modulo
booleq

Rule Cases

Rule cases define the possible values are able to alter the Request and change the next route.

Case Parts

Required Case Parts

  • matches - A string list of values applicable to this case.
  • next_route - The Backend Name indicating the next route for the Rule when a request matches this Case. Not required if redirect_url is provided.
  • redirect_url - The fully-qualified URL to issue as a 302 redirect to the client when the Request matches this Case. Not required if next_route is provided.

Optional Case Parts

  • req_rewriter name - provides the name of a Request Rewriter to operate on the Request when this case is matched.

Example Rule - Route Request by Basic Auth Username

In this example config, requests routed through the /example path will be compared against the rules and routed to either the Reader cluster or the Writer cluster. Curling http://trickster-host/example/path would route to the reader or writer cluster based on a provided Authorization header.

rules:
  example-user-router:
    # default route is reader cluster
    next_route: example-reader-cluster

    input_source: header
    input_key: Authorization
    input_type: string
    input_encoding: base64 # Authorization: Basic <base64string>
    input_index: 1         # Field 1 is the <base64string>
    input_delimiter: ' '   # Authorization Header field is space-delimited
    operation: prefix      # Basic Auth credentials are formatted as user:pass,
                           # so we can check if it is prefixed with $user:
    cases:
      - matches: # writers
          - 'johndoe:'
          - 'janedoe:'
        next_route: example-writer-cluster

backends:
  example:
    provider: rule
    rule_name: example-user-router

  example-reader-cluster:
    provider: rpc
    origin_url: 'http://reader-cluster.example.com'

  example-writer-cluster:
    provider: rpc
    origin_url: 'http://writer-cluster.example.com'
    path_routing_disabled: true  # restrict routing to this backend via rule only, so
                                 # users cannot directly access via /example-writer-cluster/

Example Rule - Route Request by Path Regex

In this example config, requests routed through the /example path will be compared against the rules and routed to either the Reader cluster or the Writer cluster. Curling http://trickster-host/example/reader and http://trickster-host/example/writer would route to the reader or writer cluster by matching the path.

rules:
  example-user-router:
    # default route is reader cluster
    next_route: example-reader-cluster

    input_source: path
    input_type: string
    operation: rmatch      # perform regex match against the path to see if it matches 'writer
    operation_arg: '^.*\/writer.*$'
    cases:
      - matches: 
          - 'true' # rmatch returns true when the input matches the regex; update next_route
        next_route: example-writer-cluster

backends:
  example:
    provider: rule
    rule_name: example-user-router

  example-reader-cluster:
    provider: rpc
    origin_url: 'http://reader-cluster.example.com'

  example-writer-cluster:
    provider: rpc
    origin_url: 'http://writer-cluster.example.com'
    path_routing_disabled: true  # restrict routing to this backend via rule only, so
                                 # users cannot directly access via /example-writer-cluster/

Example Rule - Rewrite a Hostname From a Regex Capture

An rmatch rule makes its numeric and named capture groups available to the matched case and egress request rewriters. This example routes a request containing a three-character tenant label and prefixes the destination hostname with that tenant.

request_rewriters:
  tenant-host:
    instructions:
      - [ 'hostname', 'set', '${tenant}.writer.example.com' ]

rules:
  tenant-router:
    next_route: example-reader-cluster
    input_source: path
    input_type: string
    operation: rmatch
    operation_arg: '\{mylabel="(?P<tenant>[a-z0-9]{3})"\}'
    cases:
      - matches: [ 'true' ]
        req_rewriter_name: tenant-host
        next_route: example-writer-cluster

backends:
  example:
    provider: rule
    rule_name: tenant-router

  example-reader-cluster:
    provider: rpc
    origin_url: 'http://reader-cluster.example.com'

  example-writer-cluster:
    provider: rpc
    origin_url: 'http://writer-cluster.example.com'
    path_routing_disabled: true

For input_source: path, matching uses Go’s decoded URL.Path. For example, %7Bmylabel%3D%22abc%22%7D is matched as {mylabel="abc"} and ${tenant} expands to abc. ${0} represents the complete regex match, while ${1} represents the first capture group. See Request Rewriters for token lifetime and safety details.

8 - Request Handling

Customizing how Trickster processes HTTP requests and responses.

8.1 - Customizing HTTP Path Behavior

Trickster supports, via configuration, customizing the upstream request and downstream response behavior on a per-Path, per-Backend basis, by providing a paths configuration section for each backend configuration. Here are the basic capabilities for customizing Path behavior:

  • Modify client request headers prior to contacting the origin while proxying
  • Modify origin response headers prior to processing the response object in Trickster and delivering to the client
  • Modify the response code and body
  • Limit the scope of a path by HTTP Method
  • Select the HTTP Handler for the path (proxy, proxycache or a published provider-specific handler)
  • Select which HTTP Headers, URL Parameters and other client request characteristics will be used to derive the Cache Key under which Trickster stores the object.
  • Disable Metrics Reporting for the path

Path Matching Scope

Paths are matchable as exact, prefix or regex

The default match is exact, meaning the client’s requested URL Path must be an exact match to the configured path in order to match and be handled by a given Path Config. For example a request to /foo/bar will not match an exact Path Config for /foo.

A prefix match will match any client-requested path to the Path Config with the longest prefix match. A prefix match Path Config to /foo will match /foo/bar as well as /foobar and /food. A basic string match is used to evaluate the incoming URL path, so it is recommended to consider finishing paths with a trailing /, like /foo/ in Path Configurations, if needed to avoid any unintentional matches.

Regex Paths

A path can also be a regular expression, evaluated against the client’s requested URL Path. A path is treated as a regex when either:

  • the path value starts with ^/ (or the escaped form ^\/) — this is auto-detected and applies regardless of any configured match_type; or
  • the Path Config explicitly sets match_type: regex, in which case the path need not start with ^/; Trickster will prepend a ^ anchor if absent, so match semantics are consistent.

Regex paths use Go’s RE2 syntax (linear-time matching; no backtracking). Patterns are always anchored at the start with ^. A $ end anchor is honored when present but never required — an unanchored end behaves like a prefix-style match, so ^/api/[0-9]+ matches /api/42 and /api/42/details alike, while ^/api/[0-9]+$ matches only /api/42.

Evaluation order: regex paths are evaluated only after both exact and prefix matching have missed. Within the regex tier, patterns are evaluated longest-pattern-string first; equal-length patterns are evaluated in the order they appear in the configuration; the first pattern that matches wins.

Catch-all warning: a classic catch-all path — match_type: prefix on /, or an effective catch-all like /api in front of regexes that all begin ^/api — always prefix-matches first and prevents the regex tier from ever being evaluated for those requests. When using regex paths, define the catch-all as a regex too (e.g. ^/.*), which sorts shortest and therefore evaluates last. Trickster logs a startup warning when a backend defines regex paths alongside a / prefix catch-all.

backends:
  default:
    provider: rpc
    origin_url: 'http://example.com'
    paths:
      # auto-detected as a regex path (starts with ^/)
      - path: '^/api/[0-9]+/results'
        methods: [ GET ]
        handler: proxycache
      # explicit opt-in; Trickster anchors this to ^/reports/(annual|monthly)/
      - path: '/reports/(annual|monthly)/'
        match_type: regex
        methods: [ GET ]
        handler: proxy
      # regex catch-all: evaluates last, does not shadow the regexes above
      - path: '^/.*'
        methods: [ '*' ]
        handler: proxy

Method Matching Scope

The methods section of a Path Config takes a string array of HTTP Methods that are routed through this Path Config. You can provide [ '*' ] to route all methods for this path.

Suggested Use Cases

  • Redirect a path by configuring Trickster to respond with a 302 response code and a Location header
  • Issue a blanket 401 Unauthorized code and custom response body to all requests for a given path.
  • Adjust Cache Control headers in either direction
  • Affix an Authorization header to requests proxied out by Trickster.
  • Control which paths are cached by Trickster, and which ones are simply proxied.

Request Rewriters

You can configure paths send inbound requests through a request rewriter that can modify any aspect of the inbound request (method, url, headers, etc.), before being processed by the path route. This means, when the path route inspects the request, it will have already been modified by the rewriter. Provide a rewriter with the req_rewriter_name config. It must map to a named/configured request rewriter (see request rewriters for more info). Note, you can also send requests through a rewriter at the backend level. If both are configured, backend-level rewriters are executed before path rewriters are.

request_rewriters:
  # this example request rewriter adds an additional header to the request
  # you can include as many instructions in rewriter as required
  example:
    instructions:
      - [ header, set, Example-Header-Name, Example Value ]

backends:
  default:
    provider: rpc
    origin_url: 'http://example.com'
    paths:
      - path: /
        req_rewriter_name: example

Header and Query Parameter Behavior

In addition to running the request through a named rewriter, it is currently possible to make similar changes to the request with legacy path features that are described in this section. Note that these are likely to be deprecated in a future Trickster release, in favor of the more versatile named rewriters described above, which accomplish the same thing. Currently, if both a named rewriter and legacy path-based rewriting configs are defined for a given path, the named rewriter will be executed first.

Basics

You can specify request query parameters, as well as request and response headers, to be Set, Appended or Removed.

Setting

To Set a header or parameter means to insert if non-existent, or fully replace if pre-existing. To set a header, provide the header name and value you wish to set in the Path Config request_params, request_headers or response_headers sections, in the format of 'Header-or-Parameter-Name' = 'Value'.

As an example, if the client request provides a Cache-Control: no-store header, a Path Config with a header ‘set’ directive for 'Cache-Control' = 'no-transform' will replace the no-store entirely with a no-transform; client requests that have no Cache-Control header that are routed through this Path will have the Trickster-configured header injected outright. The same logic applies to query parameters.

Environment Variable Substitution

The request_headers, request_params and response_headers sections support environment variable substitution. This means you can use environment variables in your header or query parameter values. Example:

backends:
  default:
    # ...
    paths:
      - path: /
        # ...
        request_params:
          'token': '${REQUEST_PARAM_TOKEN}'
        request_headers:
          'X-Auth-Token': '${REQUEST_HEADER_TOKEN}'
        response_headers:
          'X-Auth-Token': '${RESPONSE_HEADER_TOKEN}'

Appending

Appending a means inserting the header or parameter if it doesn’t exist, or appending the configured value(s) into a pre-existing header with the given name. To indicate an append behavior (as opposed to set), prefix the header or parameter name with a ‘+’ in the Path Config.

Example: if the client request provides a token=SomeHash parameter and the Path Config includes the parameter '+token' = 'ProxyHash', the effective parameter when forwarding the request to the origin will be token=SomeHash&token=ProxyHash.

Removing

Removing a header or parameter means to strip it from the HTTP Request or Response when present. To do so, prefix the header/parameter name with ‘-’, for example, -Cache-control: none. When removing headers, a value is required to be provided in order to conform to YAML specification; this value, however, is ineffectual. Note that there is currently no ability to remove a specific header value from a specific header - only the entire removal header. Consider setting the header value outright as described above, to strip any unwanted values.

Response Header Timing

Response Header injections occur as the object is received from the origin and before Trickster handles the object, meaning any caching response headers injected by Trickster will also be used by Trickster immediately to handle caching policies internally. This allows users to override cache controls from upstream systems if necessary to alter the actual caching behavior inside of Trickster. For example, InfluxDB sends down a Cache-Control: No-Cache header, which is fine for the user’s browser, but Trickster needs to ignore this header in order to accelerate InfluxDB; so the default Path Configs for InfluxDB actually removes this header.

Cache Key Components

By default, Trickster will use the HTTP Method, URL Path and any Authorization header to derive its Cache Key. In a Path Config, you may specify any additional HTTP headers and URL Parameters to be used for cache key derivation, as well as information in the Request Body.

Using Request Body Fields in Cache Key Hashing

Trickster supports the parsing of the HTTP Request body for the purpose of deriving the Cache Key for a cacheable object. Note that body parsing requires reading the entire request body into memory and parsing it before operating on the object. This will result in slightly higher resource utilization and latency, depending upon the size of the client request body.

Body parsing is supported when the request’s HTTP method is POST, PUT or PATCH, and the request Content-Type is either application/x-www-form-urlencoded, multipart/form-data, or application/json.

In a Path Config, provide the cache_key_form_fields setting with a list of form field names to include when hashing the cache key.

Trickster supports parsing of the Request body as a JSON document, including documents that are multiple levels deep, using a basic pathing convention of forward slashes, to indicate the path to a field that should be included in the cache key. Take the following JSON document:

{
    "requestType": "query",
    "query": {
        "table": "movies",
        "fields": "eidr,title",
        "filter": "year=1979"
    }
}

To include the requestType, table, fields, and filter fields from this document when hashing the cache key, you can provide the following setting in a Path Configuration:

cache_key_form_fields = [ 'requestType', 'query/table', 'query/fields', 'query/filter' ]

Example Reverse Proxy Cache Config with Path Customizations

backends:
  default:
    provider: rpc
    paths:
      # root path '/'
      - path: / # each path must be unique for the backend
        methods: [ '*' ] # All HTTP methods applicable to this config
        match_type: prefix # matches any path under '/'
        handler: proxy # proxy only, no caching (this is the default)
        # modify the query params en route to the origin; this adds authToken=${ROOT_REQUEST_AUTH_TOKEN}
        # (sourced from the environment variable ROOT_REQUEST_AUTH_TOKEN)
        request_params:
          authToken: ${ROOT_REQUEST_AUTH_TOKEN}
        # When a user requests a path matching this route, Trickster will
        # inject these headers into the request before contacting the Origin
        request_headers:
          Cache-Control: No-Transform
        # inject these headers into the response from the Origin
        # before replying to the client
        response_headers:
          Expires: '-1'
        # a path-level CORS policy overrides the backend policy; see docs/cors.md
        cors:
          mode: preserve
      - path: /images/
        methods:
          - GET
          - HEAD
        handler: proxycache # Trickster will cache the images directory
        match_type: prefix
        response_headers:
          Cache-Control: max-age=2592000 # cache for 30 days
      # but only cache this rotating image for 30 seconds
      - path: /images/rotating.jpg
        methods:
          - GET
        handler: proxycache
        match_type: exact
        response_headers:
          Cache-Control: max-age=30
          '-Expires': ''
      # redirect this sunsetted feature to a discontinued message
      - path: /blog
        methods:
          - '*'
        handler: localresponse
        match_type: prefix
        response_code: 302
        response_headers:
          Location: /discontinued
      # cache this API endpoint, keying on the query parameter
      - path: /api/
        methods:
          - GET
          - HEAD
        handler: proxycache
        match_type: prefix
        cache_key_params:
          - query
      # same API endpoint, different HTTP methods to route against, which are denied
      - path: /api/
        methods:
          - POST
          - PUT
          - PATCH
          - DELETE
          - OPTIONS
          - CONNECT
        handler: localresponse
        match_type: prefix
        response_code: 401
        response_body: this is a read-only api endpoint
      # cache the query endpoint, permitting GET, HEAD, POST
      - path: /api/query/
        methods:
          - GET
          - HEAD
          - POST
        handler: proxycache
        match_type: prefix
        cache_key_params:
          - query # for GET/HEAD
        cache_key_form_fields:
          - query # for POST

Modifying Behavior of Time Series Backend

Each of the Time Series Providers supported in Trickster comes with its own custom handlers and pre-defined Path Configs that are registered with the HTTP Router when Trickster starts up.

For example, when Trickster is configured to accelerate Prometheus, pre-defined Path Configs are registered to control how requests to /api/v1/query work differently from requests to /api/v1/query_range. For example, the /ap1/v1/query Path Config uses the query and time URL query parameters when creating the cache key, and is routed through the Object Proxy Cache; while the /api/v1/query_range Path Config uses the query, start, end and step parameters, and is routed through the Time Series Delta Proxy Cache.

In the Trickster config file, you can add your own Path Configs to your time series backend, as well override individual settings for any of the pre-defined Path Configs, and those custom settings will be applied at startup.

To know what configs you’d like to add or modify, take a look at the Trickster source code and examine the pre-definitions for the selected Backend Provider. Each supported Provider’s handlers and default Path Configs can be viewed under /pkg/backends/<provider>/routes.go. These files are in a standard format that are quite human-readable, even for a non-coder, so don’t be too intimidated. If you can understand Path Configs as YAML, you can understand them as Go code.

Examples of customizing Path Configs for Providers with Pre-Definitions:

backends:
  default:
    provider: prometheus
    paths:
      # route /api/v1/label* (including /labels/*)
      # through Proxy instead of ProxyCache as pre-defined
      - path: /api/v1/label
        methods:
          - GET
        match_type: prefix
        handler: proxy
      # route fictional new /api/v1/coffee to ProxyCache
      - path: /api/v1/coffee
        methods:
          - GET
        match_type: prefix
        handler: proxycache
        cache_key_params:
          - beans
      # block /api/v1/admin/ from being reachable via Trickster
      - path: /api/v1/admin/
        methods:
          - GET
          - POST
          - PUT
          - HEAD
          - DELETE
          - OPTIONS
        match_type: prefix
        handler: localresponse
        response_code: 401
        response_body: No soup for you!
        no_metrics: true

8.2 - Request Rewriters

A Request Rewriter is a named series of instructions that modifies any part of the incoming HTTP request. Request Rewriters are used in various parts of the Trickster configuration to make scoped changes. For example, a rewriter can modify the path, headers, parameters, etc. of a URL using mechanisms like search/replace, set and append.

In a configuration, request rewriters are represented as map of instructions, which themselves are represented as a list of string lists, in the following format:

request_rewriters:
  example_rewriter:
    instructions:
      - [ 'header', 'set', 'Cache-Control', 'max-age=60' ], # instruction 0
      - [ 'path', 'replace', '/cgi-bin/', '/' ],            # instruction 1
      - [ 'chain', 'exec', 'remove_accept_encoding' ]       # instruction 2

  remove_accept_encoding:
    instructions:
      - [ 'header', 'delete', 'Accept-Encoding' ] # instruction 0

In this case, any other configuration entity that supports mapping to a rewriter by name can do so with by referencing example_rewriter or remove_accept_encoding. Note that example_rewriter executes remove_accept_encoding using the chain instruction.

Where Rewriters Can Be Used

Rewriters are exposed as optional configurations for the following configuration constructs:

In a backend config, provide a req_rewriter_name to rewrite the Request using the named Request Rewriter, before it is handled by the Path route.

In a path config, provide a req_rewriter_name to rewrite the Request using the named Request Rewriter, before it is handled by the Path route.

In a rule config, provide ingress_req_rewriter_name, egress_req_rewriter_name and/or nomatch_req_rewriter_name configurations to rewrite the Request using the named Request Rewriter. The meaning of Ingress and Egress, in this case, are scoped to a Request’s traversal through the Rule in which these configuration values exist, and is unrelated to the wire traversal of the request. For ingress and egress, the rewriter is executed before or after, respectively, it is handled by the Rule (including any modifications made by a matching rule case). The No-Match Request Rewriter is only executed when the request does not match to any defined case.

In a Rule’s case configurations, provide req_rewriter_name. If there is a Rule Case match when executing the Rule against the incoming Request, the configured rewriter will execute on the Request before returning control back to the Rule to execute any configured egress request rewriter and hand the Request off to the next route.

In a Request Rewriter instruction using the chain instruction type. Provide the Rewriter Name as the third argument in the instruction as follows: [ 'chain', 'exec', '$rewriter_name']. See more information below.

Regex Capture Tokens

An rmatch rule can expose its regular expression matches to request rewriters. Numeric tokens use ${0} for the complete match and ${1}, ${2}, etc. for capture groups. A named capture such as (?P<tenant>[a-z0-9]{3}) is also available as ${tenant}.

Captures are available only to the matched case rewriter and the current rule’s egress rewriter. The ingress rewriter runs before matching and cannot use captures from its own rule, and captures are cleared before the request enters the next route. If the regular expression does not match, no capture tokens are available to the no-match or egress rewriter. Undefined tokens are left unchanged, while an optional capture group that did not participate in a successful match expands to an empty string.

Token expansion is supported in setter and appender values for headers and parameters, and in configured values for path, method, host, hostname, port and scheme instructions. Header and parameter replace/delete instructions also expand their key, search and replacement fields. Chained rewriters receive the same captures. For example:

request_rewriters:
  tenant-host:
    instructions:
      - [ 'hostname', 'set', '${tenant}.writer.example.com' ]
      - [ 'header', 'set', 'X-Tenant', '${1}' ]

Captured values are inserted without additional validation or escaping. Use restrictive regular expressions for values written to a hostname, path, header or query parameter. In particular, a client-controlled authority token can route a request, including configured upstream credentials, to an unintended host if the expression is too broad.

When Trickster constructs the final upstream URL, only host components explicitly changed by a request rewriter override the configured backend origin_url. A hostname rewrite preserves the origin_url port, while host replaces both hostname and port. The inbound request’s host never overrides origin_url by itself.

Instruction Construction Guide

header rewriters modify a header with a specific name and support the following operations.

header set

header set will set a specific header to a specific value.

['header', 'set', 'Header-Name', 'header value']

header replace

header replace performs a search/replace function on the Header value of the provided Name

['header', 'replace', 'Header-Name', 'search value', 'replacement value']

header delete

header delete removes, if present, the Header of the provided Name

['header', 'delete', 'Header-Name']

header append

header append appends an additional value to Header in the format of value1[, value2=subvalue, ...]

['header', 'append', 'Header-Name', 'additional header value']

path

path rewriters modify the full or partial path and support the following operations.

path set

['path', 'set', '/new/path'] sets the entire request path to /new/path

['path', 'set', 'awesome', 0 ] sets the first part of the path (zero-indexed, split on /) to ‘awesome’. For example, /new/path => /awesome/path

path replace

['path', 'replace', 'search', 'replacement'] search replaces against the entire path scalar

['path', 'replace', 'search', 'replacement', 1] search replaces against the second part of the path; For example /my/example-search/path => /my/example-replacement/path

param

param rewriters modify the URL Query Parameter of the specified name, and support the following operations

param set

param set sets the URL Query Parameter of the provided name to the provided value

['param', 'set', 'paramName', 'new param value']

param replace

param replace performs a search/replace function on the URL Query Parameter value of the provided name

['param', 'replace', 'paramName', 'search value', 'replacement value']

param delete

param delete removes, if present, the URL Query Parameter of the provided name

['param', 'delete', 'paramName']

param append

param append appends the provided name and value to the URL Query Parameters regardless of whether a parameter name already exists with the same or different value.

['param', 'append', 'paramName', 'additional param value']

params

params rewriters update the entire URL parameter collection as a URL-encoded scalar string.

params set

params set will replace the Request’s entire URL Query Parameter encoded string with the provided value. The provided value is assumed to already be URL-encoded.

['params', 'set', 'param1=value1&param2=value2']

To clear the URL parameters, use ['params', 'set', '']

params replace

params replace performs a search/replace operation on the url-encoded query string. The search and replacement values are assumed to already be URL-encoded.

['params', 'replace', 'search value', 'replacement value']

method

method rewriters update the HTTP request’s method

method set

method set sets the Request’s HTTP Method to the provided value. This value is not currently validated against known HTTP Method. The instruction should be configured to include a known and properly-formatted (all caps) HTTP Method.

['method', 'set', 'GET']

host

host rewriters update the HTTP Request’s host - defined as the hostname:port, as expressed in the Request’s Host header.

host set

host set sets the Request’s Host header to the provided value.

['host', 'set', 'my.new.hostname:9999']

['host', 'set', 'my.new.hostname'] Trickster will assume a standard source port 80/443 depending upon the URL scheme

host replace

host replace performs a search/replace operation on the Request’s Host Header.

['host', 'replace', ':8480', '']

['host', 'replace', ':443', ':8443']

['host', 'replace', 'example.com', 'trickstercache.org']

hostname

hostname rewriters update the HTTP Request’s hostname, without respect to the port.

hostname set

hostname set sets the Request’s hostname, without changing the port.

['hostname', 'set', 'my.new.hostname']

hostname replace

hostname replace performs a search/replace on the Request’s hostname, without respect to the port.

['hostname', 'replace', 'example.com', 'trickstercache.org']

port

port rewriters update the HTTP Request’s port, without respect to the hostname.

port set

port set sets the Request’s port.

['port', 'set', '8480']

port replace

port replace performs a search/replace on the port, as if it was a string. The search and replacement values must be integers and are not validated.

['port', 'replace', '8480', '']

port delete

port delete removes the port from the Request. This will cause the port to be assumed based on the URL scheme.

['port', 'delete']

scheme

scheme rewriters update the HTTP Request’s scheme (http or https).

scheme set

scheme set sets the scheme of the HTTP Request URL. This must be http or https in lowercase, and is not validated.

['scheme', 'set', 'https']

chain

chain rewriters do not directly rewrite the request, but execute other rewriters’ instructions before proceeding with the current rewriter’s remaining instructions (if any). You can create a rewriter with some reusable functionality and include that in other rewriters with a chain exec. Or you can define a rewriter that is just a list of other chained rewriters. Note that there is currently no validation of the configuration to prevent infinite cyclic chained rewriter calls. There is, however, a hard limit of 32 chained rules before a request will stop rewriting and proceed with being served by the backend.

chain exec executes the supplied rewriter name. Trickster will error at startup if the rewriter name is invalid. An example is provided in the sample yaml config at the top of this article.

['chain', 'exec', 'example_rewriter']

8.3 - Request Body Handling Customizations

Request Body Size Limiter

By default, the max allowed Request Body size is 10 MB. If the client request body in a POST, PUT or PATCH are over 10 MB, the request will receive a response of 413 Request Payload is too large

You can change (or bypass) this limit in the ’listeners’ Config Section per-listener:

listeners:
  default:
    max_request_body_size_bytes: 5120 # request bodies must be <= 5kb or returns 413

listeners:
  default:
    max_request_body_size_bytes: 5120 # request bodies > 5kb truncated to 5kb
    truncate_request_body_too_large: true # truncate request bodies that are too large

8.4 - Cross-Origin Resource Sharing

Trickster can control Cross-Origin Resource Sharing (CORS) response headers for each backend. A path can override its backend’s policy by providing its own cors block.

For compatibility, a backend or path without a cors block uses Trickster’s legacy behavior: Access-Control-Allow-Origin is set to *, while other CORS headers from the origin are left unchanged.

Modes

ModeBehavior
preserveReturn the origin’s Access-Control-* response headers unchanged.
mergePreserve origin CORS headers, then apply the configured headers as overrides.
replaceRemove all origin Access-Control-* response headers, then apply the configured headers.
disableRemove all origin Access-Control-* response headers. This mode cannot include a headers block.

When preserve or merge is configured, Trickster includes the request’s Origin header in the cache key. This prevents an origin-specific CORS response from being served to a request from another origin.

Backend Configuration

The following example replaces the origin’s CORS headers with a fixed policy:

backends:
  default:
    provider: reverseproxycache
    origin_url: http://api.example.com
    cors:
      mode: replace
      headers:
        Access-Control-Allow-Origin: https://dashboard.example.com
        Access-Control-Allow-Credentials: "true"
        Access-Control-Expose-Headers: X-Trickster-Result

To preserve the origin’s CORS headers while replacing selected values, use merge:

cors:
  mode: merge
  headers:
    Access-Control-Allow-Origin: https://trickster.example.com

To return all origin CORS headers unchanged, use preserve without a headers map:

cors:
  mode: preserve

To disable CORS headers, use disable without a headers block:

cors:
  mode: disable

Only Access-Control-* response headers can be configured in a cors.headers map. As with path response_headers, prefix a header name with - to remove that header in merge mode or + to append another value.

Path Overrides

A path-level policy replaces the backend policy for requests matching that path:

backends:
  default:
    provider: reverseproxycache
    origin_url: http://api.example.com
    cors:
      mode: preserve
    paths:
      - path: /private/
        match_type: prefix
        cors:
          mode: disable

For an ALB, user router, or rule backend that dispatches to another backend internally, the policy on the client-facing backend and path takes precedence. Internal routing therefore cannot change the CORS policy associated with the public Trickster route.

8.5 - Simulated Latency

Trickster supports simulating latency on a per-backend basis, and can simulate both consistent and random durations of latency. Simulated latency is introduced in the frontend part of the proxy, and thus works with any backend provider.

In rule, alb and other backend providers, where a request may transit multiple backend routes, only the Simulated Latency configs associated with the request entrypoint (first route) will be processed, and not any subsequent routes the request is sent through.

Consistent Latency Duration

In the Backend configuration, add a latency_min value > 0, and the configured amount of latency will be introduced for each incoming request.

Random Latency

To simulate random latency, set latency_max to a value > latency_min, which may be 0 for random latency. Trickster will introduce a random amount of latency between (inclusive) the provided min and max values.

Response Header

When simulated latency is applied to a request, an x-simulated-latency header will be included in the corresponding response indicating the duration of the latency applied in milliseconds. The format of the latency header is as follows:

GET /api/v1/query?query=up HTTP/1.1
Accept: */*
...

HTTP/1.1 200 OK
X-Simulated-Latency: 300ms
X-Trickster-Result: engine=DeltaProxyCache; status=hit; ffstatus=hit
Date: Mon, 06 Sep 2021 04:07:20 GMT
...

Example Config

backends:
  default:
    origin_url: https://www.example.com
    provider: reverseproxy
    #
    # introduce random latency between 50 and 150 milliseconds on each request
    latency_min: 50ms
    latency_max: 150ms

  backend2:
    origin_url: https://www.trickstercache.org
    provider: reverseproxy
    #
    # introduce latency of 300 milliseconds on each request
    latency_min: 300ms

9 - Observability

Metrics, logs, distributed tracing, and debugging Trickster’s behavior.

9.1 - Trickster Metrics

Trickster exposes a Prometheus /metrics endpoint with a customizable listener port number (default is 8481). For more information on customizing the metrics configuration, see configuring.md.

The following metrics are available for polling with any Trickster configuration:

  • trickster_build_info (Gauge) - This gauge is always 1 when Trickster is running

    • labels:
      • goversion - the version of go under which the running Trickster binary was built
      • revision - the commit ID on which the running Trickster binary was built
      • version - semantic version of the running Trickster binary

  • trickster_config_last_reload_successful (Gauge) - The value is 1 when true (the last config reload was successful) or 0 when false

  • trickster_config_last_reload_success_time_seconds (Gauge) - Epoch timestamp of the last successful configuration reload

  • trickster_frontend_requests_total (Counter) - Count of front end requests handled by Trickster

    • labels:
      • backend_name - the name of the configured backend handling the proxy request
      • provider - the type of the configured backend handling the proxy request
      • method - the HTTP Method of the proxied request
      • http_status - The HTTP response code provided by the backend
      • path - the Path portion of the requested URL

  • trickster_frontend_requests_duration_seconds (Histogram) - Histogram of front end request durations handled by Trickster

    • labels:
      • backend_name - the name of the configured backend handling the proxy request
      • provider - the type of the configured backend handling the proxy request
      • method - the HTTP Method of the proxied request
      • http_status - The HTTP response code provided by the backend
      • path - the Path portion of the requested URL

  • trickster_frontend_written_byte_total (Counter) - Count of bytes written in front end requests handled by Trickster

    • labels:
      • backend_name - the name of the configured backend handling the proxy request
      • provider - the type of the configured backend handling the proxy request
      • method - the HTTP Method of the proxied request
      • http_status - The HTTP response code provided by the backend
      • path - the Path portion of the requested URL

  • trickster_proxy_requests_total (Counter) - The total number of requests Trickster has handled.

    • labels:
      • backend_name - the name of the configured backend handling the proxy request
      • provider - the type of the configured backend handling the proxy request
      • method - the HTTP Method of the proxied request
      • cache_status - status codes are described here
      • http_status - The HTTP response code provided by the backend
      • path - the Path portion of the requested URL

  • trickster_proxy_points_total (Counter) - The total number of data points Trickster has handled.

    • labels:
      • backend_name - the name of the configured backend handling the proxy request
      • provider - the type of the configured backend handling the proxy request
      • cache_status - status codes are described here
      • path - the Path portion of the requested URL

  • trickster_proxy_request_duration_seconds (Histogram) - Time required to proxy a given Prometheus query.

    • labels:
      • backend_name - the name of the configured backend handling the proxy request
      • provider - the type of the configured backend handling the proxy request
      • method - the HTTP Method of the proxied request
      • cache_status - status codes are described here
      • http_status - The HTTP response code provided by the backend
      • path - the Path portion of the requested URL

  • trickster_proxy_max_connections (Gauge) - Trickster max number of allowed concurrent connections

  • trickster_proxy_active_connections (Gauge) - Trickster number of concurrent connections

  • trickster_proxy_requested_connections_total (Counter) - Trickster total number of connections requested by clients.

  • trickster_proxy_accepted_connections_total (Counter) - Trickster total number of accepted client connections.

  • trickster_proxy_closed_connections_total (Counter) - Trickster total number of administratively closed client connections.

  • trickster_proxy_failed_connections_total (Counter) - Trickster total number of failed client connections.

  • trickster_proxy_query_range_rejected_total (Counter) - Trickster total number of queries rejected due to exceeding the max_query_range limit.

    • labels:
      • backend - the name of the configured backend rejecting the query

  • trickster_graphite_resolution_lookups_total (Counter) - Count of Graphite step-resolution lookups. Labels never include a metric path or target expression.

    • labels:
      • backend_name - the name of the configured Graphite backend
      • confidence - how the step was established: exact (read from an origin response for this leaf set and age), derived (computed from known leaf ladders), configured (from static_retentions, not yet probe-confirmed), or unknown (no usable step; the request is served unaccelerated)
      • source - where it came from: registry, response, probe, static, function, or none

  • trickster_graphite_probes_total (Counter) - Count of synthetic requests issued to learn a metric’s archive ladder. Expect a spike at startup that collapses toward zero as ladders are learned.

    • labels:
      • backend_name - the name of the configured Graphite backend
      • kind - narrow (a one-second window that also discovers the retention edge), wide (what a real query at that age receives), or find (a /metrics/expand lookup)
      • result - step (a stepped series came back), empty (no series: beyond retention, or no such metric), or error

  • trickster_graphite_ladders (Gauge) - Number of distinct archive ladders known to the resolution registry. Ladders come from storage-schemas.conf patterns, so this should flatten at a small number.

    • labels:
      • backend_name - the name of the configured Graphite backend

  • trickster_graphite_registry_entries (Gauge) - Number of entries in each layer of the resolution registry.

    • labels:
      • backend_name - the name of the configured Graphite backend
      • layer - leaf (metric path to ladder), ladder (the ladders themselves), target (cached wildcard expansions), or negative (paths in resolution backoff)

  • trickster_graphite_step_mispredictions_total (Counter) - Count of origin responses whose step differed from the predicted step. This should always be zero. A non-zero value means a cached ladder was wrong; Trickster discards the prediction, relearns and re-serves the request unaccelerated, so clients still receive correct data.

    • labels:
      • backend_name - the name of the configured Graphite backend

  • trickster_graphite_fallbacks_total (Counter) - Count of render requests served without delta caching. Labels never include a target expression.

    • labels:
      • backend_name - the name of the configured Graphite backend
      • reason - parse_error, non_series_format, function_not_allowlisted, unknown_step, missing_target, multi_target_step_mismatch, passthrough_max_data_points, misprediction, client_identity, tz_unavailable, or resolution_identity

  • trickster_sql_query_analysis_total (Counter) - Count of SQL query cache-eligibility classifications. Labels never include query text.

    • labels:
      • backend_name - the name of the configured backend analyzing the query
      • dialect - the SQL dialect of the analyzing backend (e.g., clickhouse)
      • cache_mode - the strongest cache mode supported by the query (delta, object, or none)
      • reason - the stable classification reason code (e.g., delta_cacheable, unsafe_predicate, unsupported_bucket)

  • trickster_sql_query_rewrite_failures_total (Counter) - Count of SQL cache-miss extent rewrite failures. Labels never include query text.

    • labels:
      • backend_name - the name of the configured backend rendering the query
      • dialect - the SQL dialect of the rendering backend
      • reason - the fixed internal failure category

  • trickster_cache_operation_objects_total (Counter) - The total number of objects upon which the Trickster cache has operated.

    • labels:
      • cache_name - the name of the configured cache performing the operation$
      • provider - the type of the configured cache performing the operation
      • operation - the name of the operation being performed (read, write, etc.)
      • status - the result of the operation being performed

  • trickster_cache_operation_bytes_total (Counter) - The total number of bytes upon which the Trickster cache has operated.

    • labels:
      • cache_name - the name of the configured cache performing the operation$
      • provider - the type of the configured cache performing the operation
      • operation - the name of the operation being performed (read, write, etc.)
      • status - the result of the operation being performed

  • trickster_alb_pool_admits_failing (Gauge) - 1 when an ALB pool’s healthy_floor admits members in the unavailable state, 0 otherwise. See alb.md for the recommended floor.

    • labels:
      • backend_name - the name of the configured ALB backend

  • trickster_alb_pool_floor_reset (Gauge) - 1 when an ALB pool’s healthy_floor was reset to 0 at startup because pool members have no health check and could never reach the configured floor, 0 otherwise. See alb.md.

    • labels:
      • backend_name - the name of the configured ALB backend

The following metrics are available when ALB Autodiscovery is configured:

  • trickster_alb_discovery_members (Gauge) - Current number of discovered ALB pool members

    • labels:
      • alb_name - the name of the discovery-backed ALB backend
      • discoverer - the name of the discoverer serving the ALB

  • trickster_alb_discovery_member_changes_total (Counter) - Count of discovered pool member additions and removals

    • labels:
      • alb_name - the name of the discovery-backed ALB backend
      • discoverer - the name of the discoverer serving the ALB
      • event - add or remove

  • trickster_alb_discovery_snapshots_total (Counter) - Count of membership snapshots processed, by result

    • labels:
      • alb_name - the name of the discovery-backed ALB backend
      • discoverer - the name of the discoverer serving the ALB
      • result - applied (membership updated), unchanged (no-op), rejected (guardrail-refused, e.g. min_members), or partial (applied with member instantiation failures)

  • trickster_alb_discovery_last_refresh_success_time_seconds (Gauge) - Epoch timestamp of the last successfully processed snapshot, for staleness alerting

    • labels:
      • alb_name - the name of the discovery-backed ALB backend
      • discoverer - the name of the discoverer serving the ALB

  • trickster_discovery_refresh_errors_total (Counter) - Count of provider-side refresh/watch errors (DNS resolution failures, Kubernetes list/sync failures, member-file read/parse failures)

    • labels:
      • discoverer - the name of the discoverer experiencing the error
      • provider - the discoverer’s provider type

  • trickster_tls_certificate_expiration_time_seconds (Gauge) - NotAfter time of a serving TLS certificate, as unix seconds. See tls.md.

    • labels:
      • listener - the name of the listener serving the certificate
      • entry - the certificate’s source identity

  • trickster_tls_certificate_last_load_time_seconds (Gauge) - Epoch timestamp a serving TLS certificate was last loaded from its source

    • labels:
      • listener - the name of the listener serving the certificate
      • entry - the certificate’s source identity

  • trickster_tls_certificate_swaps_total (Counter) - Count of TLS certificates hot-swapped into a live listener by rotation detection

    • labels:
      • listener - the name of the listener serving the certificate
      • entry - the certificate’s source identity

  • trickster_tls_certificate_validation_failures_total (Counter) - Count of detected TLS certificate source changes that failed pair validation (e.g. a mid-rotation partial write) and were not swapped in

    • labels:
      • entry - the certificate’s source identity

  • trickster_tls_watcher_errors_total (Counter) - Count of errors reading watched TLS certificate source files

    • labels:
      • entry - the certificate’s source identity

  • trickster_tls_certificate_store_size (Gauge) - Number of certificates in a listener’s TLS certificate store

    • labels:
      • listener - the name of the listener

The following metrics are available only for Caches Types whose object lifecycle Trickster manages internally (Memory, Filesystem and bbolt):

  • trickster_cache_events_total (Counter) - The total number of events that change the Trickster cache, such as retention policy evictions.

    • labels:
      • cache_name - the name of the configured cache experiencing the event$
      • provider - the type of the configured cache experiencing the event
      • event - the name of the event being performed
      • reason - the reason the event occurred

  • trickster_cache_usage_objects (Gauge) - The current count of objects in the Trickster cache.

    • labels:
      • cache_name - the name of the configured cache$
      • provider - the type of the configured cache$

  • trickster_cache_usage_bytes (Gauge) - The current count of bytes in the Trickster cache.

    • labels:
      • cache_name - the name of the configured cache$
      • provider - the type of the configured cache$

  • trickster_cache_max_usage_objects (Gauge) - The maximum allowed size of the Trickster cache in objects.

    • labels:
      • cache_name - the name of the configured cache$
      • provider - the type of the configured cache

  • trickster_cache_max_usage_bytes (Gauge) - The maximum allowed size of the Trickster cache in bytes.

    • labels:
      • cache_name - the name of the configured cache$
      • provider - the type of the configured cache

In addition to these custom metrics, Trickster also exposes the standard Prometheus metrics that are part of the client_golang metrics instrumentation package, including memory and cpu utilization, etc.

9.2 - Distributed Tracing via OpenTelemetry

Trickster instruments Distributed Tracing with OpenTelemetry. We import the OpenTelemetry golang packages to instrument support for tracing.

As OpenTelemetry evolves to support additional exporter formats, we will work to extend Trickster to support those as quickly as possible. We also make a best effort to update our otel package imports to the latest releases, whenever we publish a new Trickster release. You can check the go.mod file to see which release of opentelemetry-go we are is using. In this view, to see which version of otel a specific Trickster release imports, use the branch selector dropdown to switch to the tag corresponding to that version of Trickster.

Supported Tracing Backends

  • Jaeger (via OTLP)
  • Console/Stdout (printed locally by the Trickster process)

Trickster’s OTLP exporter supports OTLP over HTTP and gRPC. HTTP is the default for existing OTLP configs. For Jaeger over OTLP/HTTP, configure Trickster with the collector’s HTTP endpoint, for example http://jaeger:4318/v1/traces. For Jaeger over OTLP/gRPC, set protocol: grpc and use the collector’s gRPC endpoint, for example http://jaeger:4317 from another Compose service or http://127.0.0.1:4317 from the host when using the developer Compose environment.

Configuration

Trickster allows the operator to configure multiple tracing configurations, which can be associated into each Backend configuration by name.

The example config has exhaustive examples of configuring Trickster for distributed tracing.

For OTLP tracing, protocol selects the exporter transport. Supported values are http and grpc; when omitted, Trickster uses http for backward compatibility with existing configs.

Context Propagation

When tracing is enabled for a Backend, Trickster uses the W3C Trace Context and Baggage propagators. It extracts incoming traceparent, tracestate, and baggage headers from client requests and injects the active outbound origin request span into the proxied request. This lets downstream Origins continue the same distributed trace instead of starting an unrelated trace.

Sampling

Trickster uses parent-based sampling for traced requests. The configured sample_rate controls root traces that Trickster starts when there is no sampled upstream trace context. When an incoming request already has a remote parent trace, Trickster follows the parent’s sampling decision so sampled upstream traces continue across the proxy boundary and unsampled upstream traces remain unsampled.

Span List

Trickster can insert several spans to the traces that it captures, depending upon the type and cacheability of the inbound client request, as described in the table below.

Span NameObserves when Trickster is:
requestinitially handling the client request by a Backend
QueryCachequerying the cache for an object
WriteCachewriting an object to the cache
DeltaProxyCacheRequesthandling a Time Series-based client request
FetchFastForwardmaking a Fast Forward request for time series data
FetchTimeSeriesretrieving time series data from an Origin
FetchRangeretrieving one sharded time range from an Origin
Fetchretrieving one object or range from an Origin
FetchRevalidationrevalidating a stale cache object against its Origin
ProxyRequestcommunicating with an Origin server to fulfill a client request
PrepareFetchReaderpreparing a client response from a cached or Origin response
CacheRevalidationrevalidating a stale cache object against its Origin
FetchObjectretrieving a non-time-series object from an Origin

Tags / Attributes

Trickster supports adding custom tags to every span via the configuration. Depending upon your preferred tracing backend, these may be referred to as attributes. See the example config for examples of adding custom attributes.

Trickster also supports omitting any tags that Trickster inserts by default. The list of default tags are below. For example on the “request” span, an http.url tag is attached with the current full URL. In deployments where that tag may introduce too much cardinality in your backend trace storage system, you may wish to omit that tag and rely on the more concise path tag. Each tracer config can be provided a string list of tags to omit from traces.

Attributes added to request and core proxy/cache/fetch spans

  • backend.name
  • backend.provider
  • cache.name
  • cache.provider
  • router.path - request path trimmed to the route match path for the request (e.g., /api/v1/query), good for aggregating when there are large variations in the full URL path
  • router.handler

These resource attributes are attached when the corresponding backend, cache, or route configuration is available to the request flow.

Attributes added to top level (request) span

  • http.url - the full HTTP request URL

Attributes added to cache/proxy spans

  • cache.status - the lookup or proxy cache status where available. See the cache status reference for a description of the attribute values.

Attributes added to proxy/fetch spans

  • http.status_code - the HTTP status returned by the Origin or generated proxy response, when available.

Attributes added to the FetchRevalidation span

  • isRange - is true if the client request includes an HTTP Range header

Attributes added to the FetchObject span

9.3 - Access and Error Logs

Trickster can write per-backend HTTP access logs and error logs, with customizable formats, rotation and retention. Both logs are off by default; each is enabled by configuring its filename.

Basic Configuration

backends:
  example1:
    provider: rp
    origin_url: http://example.com/
    access_log:
      filename: /var/log/trickster/example1.access.log
      error_filename: /var/log/trickster/example1.error.log
  • The access log receives one line per request handled by the backend and is written only when filename is set.
  • The error log receives one line per request whose response status is at or above error_threshold (default 400) and is written only when error_filename is set. An error-logged request also appears in the access log when both are configured.
  • Two backends may share a filename; they will safely share the underlying file and its rotation.
  • When instance_id is set in the main config, it is inserted into log filenames just as with the application log (e.g., example1.access.1.log).

Log Format

The format option accepts either a named preset or a custom format string using Apache-style % tokens, so the well-known conventions from Apache HTTP Server, Apache Traffic Server, Lighttpd, and similar servers apply directly.

Presets

NameDescription
commonNCSA Common Log Format: %h %l %u %t "%r" %>s %b
combinedApache/Nginx Combined format: common + Referer and User-Agent. This is the default.
extendedcombined + duration (ms), cache status and backend name
jsonOne JSON object per line with a fixed field set (see below)

Custom Formats

    access_log:
      filename: /var/log/trickster/example1.access.log
      format: '%h %u %t "%r" %>s %b %{ms}T %{cache-status}x'

Supported tokens:

TokenDescription
%h, %aclient IP address
%lremote logname (always -)
%uauthenticated username (from HTTP Basic Auth), else -
%trequest start time in CLF format: [26/Aug/2026:10:30:00 +0000]
%{sec}t, %{msec}t, %{usec}trequest start time as a Unix epoch value
%{LAYOUT}trequest start time in a custom Go time layout
%rfirst line of the request: GET /path?query HTTP/1.1
%mrequest method
%Urequest URL path
%qquery string, prefixed with ?, or empty when none
%Hrequest protocol (e.g., HTTP/1.1)
%s, %>sresponse status code
%bresponse body bytes, or - when zero (CLF style)
%Bresponse body bytes, numeric
%Drequest duration in microseconds
%Trequest duration in whole seconds
%{us}T, %{ms}T, %{s}Trequest duration in the given unit
%{Name}irequest header value
%{Name}oresponse header value
%{Name}crequest cookie value
%vrequested virtual host
%plistener port that served the request
%Alistener IP address that served the request
%%a literal %

Trickster-specific values use the %{key}x extension namespace:

TokenDescription
%{backend}xbackend name
%{provider}xbackend provider type
%{cache-status}xcache result (hit, phit, kmiss, …); see Cache Status
%{engine}xproxy engine that handled the request (e.g., DeltaProxyCache)
%{path-config}xthe matched path config path

Missing values render as -. Values derived from the request (like headers and usernames) are backslash-escaped so they cannot corrupt the log line structure. Unknown tokens fail validation at startup.

The json preset emits these fields per line: time, client_ip, user, method, path, query, proto, status, bytes, duration_ms, host, referer, user_agent, backend, provider, path_config, cache_status, engine.

Rotation and Retention

Access and error logs are rotated and pruned automatically, using nginx/logrotate-style numbered archives (example1.access.log.1.gz is the most recent archive, .2.gz the next, and so on).

    access_log:
      filename: /var/log/trickster/example1.access.log
      rotation:
        size: 256MB   # rotate when the live file would exceed this size (default 256MB)
        interval: 1d  # also rotate when the live file is older than this (default off)
      retention:
        count: 3      # keep at most 3 archives (default 80)
        age: 7d       # also prune archives older than this (default 7d)
      compress: true  # gzip archives (default true)
  • size and interval may be combined; the log rotates when either threshold is reached. Setting both to 0 disables rotation.
  • Sizes accept KB, MB, GB and TB suffixes (binary multiples), or a plain byte count.
  • retention.count: 0 disables count-based pruning and keeps all archives.
  • Writes are buffered for up to one second or 64 KiB. A process or machine crash can lose the buffered tail; an orderly shutdown flushes it.
  • Interval rotation keeps its epoch in a <filename>.rotation sidecar so a restart does not reset the interval clock.

Archives created by older Trickster releases use timestamped lumberjack names and are not included in numbered-archive retention. They form a bounded legacy set and may be removed manually after upgrading.

When upgrading from the original logging implementation, note these filename and retention changes:

  • retention.count: 0 now keeps all archives; configure a positive count to bound archive retention.
  • With main.instance_id enabled, filenames without a .log suffix now also include the instance ID (trickster.out becomes trickster.2.out). Update log shippers that still follow the unsuffixed filename.

The same rotation, retention and compress options are also available in the main logging: config section to control rotation of the Trickster application log, with the same defaults.

Error Log Settings

Each error_* option inherits its value from the corresponding access log option when unset:

    access_log:
      filename: /var/log/trickster/example1.access.log
      format: combined
      error_filename: /var/log/trickster/example1.error.log
      error_format: ''      # default: inherits format
      error_threshold: 400  # log responses with status >= this (default 400)
      error_rotation:       # default: inherits rotation
        size: 64MB
      error_retention:      # default: inherits retention
        count: 7
      error_compress: true  # default: inherits compress

9.4 - X-Trickster-Result Header

Trickster adds the X-Trickster-Result response header to describe how it handled a request. The header is intended for debugging cache behavior, proxy fallbacks, and partial origin fetches.

Example:

X-Trickster-Result: engine=DeltaProxyCache; status=phit; fetched=[1612804980000-1612808580000]; ffstatus=hit

The header value is a semicolon-separated list of fields. Optional fields are included only when Trickster has a value for them.

FieldDescription
engineThe proxy engine that handled the response, such as HTTPProxy, ObjectProxyCache, or DeltaProxyCache.
statusThe cache or proxy result. See Cache Status.
fetchedTime ranges fetched from the origin to satisfy the response. Ranges are formatted as start-end; multiple ranges are separated by semicolons inside the brackets.
ffstatusFast Forward cache result for time series requests. Possible values are hit, miss, off, or err.
failedTime ranges that Trickster attempted to fetch but could not fetch successfully. This usually appears with proxy-error or partial fanout failures.

Result Statuses

status uses the same values reported in metrics, logs, and tracing. Common examples are:

StatusMeaning
hitThe response was served fully from cache.
phitPart of the response was served from cache and part was fetched from the origin.
kmissTrickster had no object for the cache key and fetched the response from the origin.
rmissTrickster had an object for the cache key, but not for the requested range.
rhitTrickster revalidated a stale cached object against the origin and served it as a hit.
nchitThe response was served from the Negative Cache.
purgeThe cache key was purged as directed by a request or response header.
proxy-hitThe request joined an in-flight origin fetch for the same cache key.
proxy-onlyThe request was proxied to the origin without writing or reading a cache object.
proxy-errorAn upstream request needed for the response returned an error.
errorTrickster encountered a cache lookup or cache handling error.

Proxy-Only Results

status=proxy-only means the response came from the origin through Trickster’s proxy path, without a cache read or cache write for that request. It does not necessarily mean the origin response was wrong.

Common reasons include:

CauseExample
Backend or path is configured to bypass cachingprovider: reverseproxy or proxy_only: true.
Client request prevents cachingA request such as Cache-Control: no-cache can force Trickster to proxy and remove the existing object for that cache key.
Origin response is not cacheableFor example, response cache headers do not provide cacheability, or the response includes headers that Trickster treats as not cacheable.
Time series request cannot be parsed for delta cachingTrickster may fall back to object proxy cache for compatible requests, or proxy the request directly when it cannot safely cache the query shape.
Time series range is outside the retained cache windowOld data may be proxied without caching while newer ranges remain cacheable.

When investigating proxy-only, check the backend provider, any proxy_only setting, request and response Cache-Control headers, and whether the request shape is supported by the configured backend provider.

Fetched And Failed Ranges

fetched and failed describe the ranges Trickster fetched or failed to fetch while serving the response.

Example:

X-Trickster-Result: engine=DeltaProxyCache; status=phit; fetched=[1612804980000-1612808580000;1612812180000-1612815780000]

For time series responses, range values are Unix timestamps in milliseconds. A phit result with fetched ranges usually means Trickster had some of the requested data cached and fetched the missing ranges from the origin.

failed ranges indicate the origin request for those ranges failed. Depending on the proxy engine and fanout behavior, Trickster may return an error response or a partial response with failure metadata.

Fast Forward Status

ffstatus appears on time series responses when the Delta Proxy Cache checks Fast Forward data:

Fast Forward StatusMeaning
hitFast Forward data was served from cache.
missFast Forward data was fetched from the origin.
offFast Forward was not attempted for this request.
errFast Forward was attempted but failed or returned unusable data.

Fast Forward is only relevant for supported time series backends and only when the request is eligible for the latest datapoint optimization.