Explore how to use Trickster to accelerate your projects. If you’re new to Trickster, start with the Quick Start and Where to Place Trickster, then Configuring Trickster. Running on Kubernetes? The Kubernetes section covers deploying Trickster and using it as a Gateway API and Ingress controller.
This is the multi-page printable view of this section. Click here to print.
Documentation
- 1: Getting Started
- 1.1: Quick Start
- 1.2: Where to Place Trickster
- 1.3: Configuring Trickster
- 2: Backends
- 2.1: Using Multiple Backends with a single Trickster instance
- 2.2: Health Checks
- 2.3: Authenticator
- 2.4: AWS Integration
- 2.5: Origin Resolution via DNS SRV
- 2.6: Static File Server Backend
- 3: Listeners & TLS
- 3.1: TLS Support
- 3.2: HTTP/3
- 3.3: Apache Arrow Flight SQL Listeners
- 3.4: Automatic Certificates (ACME)
- 4: Caching
- 4.1: Cache Overview
- 4.2: Trickster Caching Retention Policies
- 4.3: Chunked Caching
- 4.4: Negative Caching
- 5: Object Caching
- 6: Time Series Caching
- 6.1: Supported Providers
- 6.2: Per-Query Time Series Instructions
- 6.3: Query Range Limits
- 6.4: Step Alignment
- 6.5: Timeseries Request Sharding
- 6.6: Providers
- 6.6.1: Prometheus Support
- 6.6.2: ClickHouse Support
- 6.6.3: InfluxDB Support
- 6.6.4: Apache Druid Support
- 6.6.5: Graphite Provider
- 6.6.6: PostgreSQL and TimescaleDB Provider
- 6.6.7: GreptimeDB Provider
- 6.6.8: QuestDB Provider
- 6.6.9: VictoriaMetrics Provider
- 6.6.10: MySQL Provider
- 7: Routing & Load Balancing
- 7.1: Application Load Balancer
- 7.2: ALB Autodiscovery
- 7.3: Rule Backend
- 8: Request Handling
- 8.1: Customizing HTTP Path Behavior
- 8.2: Request Rewriters
- 8.3: Request Body Handling Customizations
- 8.4: Cross-Origin Resource Sharing
- 8.5: Simulated Latency
- 9: Security & Policy
- 9.1: IP Access Lists
- 9.2: Geo ACLs
- 9.3: Rate Limiting
- 10: Kubernetes
- 10.1: Deploying on Kubernetes
- 10.2: Kubernetes Gateway API
- 10.3: Kubernetes Ingress v1
- 10.4: Kubernetes Cache Policy
- 10.5: Kubernetes Controller RBAC
- 11: Observability
- 11.1: Trickster Metrics
- 11.2: Distributed Tracing via OpenTelemetry
- 11.3: Access and Error Logs
- 11.4: X-Trickster-Result Header
- 12: Release Notes
- 12.1: Trickster 2.2
- 12.2: Trickster 2.1
- 12.3: Trickster 2.0
1 - Getting Started
1.1 - Quick Start
This composition creates service containers for Prometheus, Grafana, Jaeger, Zipkin, Redis, Trickster and Mockster that together demonstrate several basic end-to-end configurations for running Trickster in your environment with different cache and tracing provider options.
Prerequisites
You should already have the following installed:
Starting the demo
Clone the Trickster Github project.
In the
tricksterdirectory, change your working directory to./examples/docker-composewith the following command:cd examples/docker-composeTo run the demo from the demo directory, enter the following command:
docker-compose up -dYou can interact with each of the services on their exposed ports (as defined in Compose file), or by running
docker logs $container_name,docker attach $container_name, etc.
Exploring Trickster
Grafana
Once the composition is running, we recommend exploring with Grafana, at http://127.0.0.1:3000/. Grafana is pre-configured with datasources and a sample dashboard that are ready-to-use for the demo.
Jaeger UI
Jaeger UI is available at http://127.0.0.1:16686, which provides visualization of traces shipped by Trickster and Grafana. The more you use trickster-based data sources in Grafana, the more traces you will see in Jaeger. This composition runs the Jaeger All-in-One container. Trickster ships some traces to the Agent and others directly to the Collector, so as to demonstrate both capabilities. The Trickster config determines which upstream origin ships which traces where.
For a variety of configurations and other bootstrap data, review the various files in the docker-compose-data folder. This might be useful for configuring and using Trickster (or any of these other fantastic projects) in your own deployments. Try adding, removing, or changing some of the trickster configurations in ./docker-compose-data/trickster-config/trickster.yaml and then docker exec docker-compose_trickster_1 kill -1 1 into the Trickster container to apply the changes, or restart the environment altogether with docker-compose restart. Just be sure to make a backup of the original config first, so you don’t have to download it again later.
Example Datasources
The sim-* datasources generate on-the-fly simulation data for any possible time range, so you can immediately use them after starting up the environment. Note, however, that the simulated data is not representative of reality in any way.
The non-sim Prometheus container that backs the prom-* datasources polls the newly-running environment to generate metrics that will then populate the dashboard. Since the Prometheus container only collects and stores metrics while the environment is running, you’ll need to wait a minute or two for those datasources to show any data on the dashoard in real-time.
Getting Real Dashboard Data
Using datasources backed by the real Prometheus and Trickster (the prom-trickster-* datasources), rather than the simulator, to explore the dashboard is more desirable for the demo. It better conveys the shape and nature of the Trickster-specific metrics that might be unfamiliar. However, since there is no historical data in the demo composition, that creates an upfront barrier.
Keeping the dashboard open and auto-refreshing against any trickster-labeled datasource will help to generate real metrics in Trickster, such as request rates, cache hit rates, etc. Prometheus will collect and store those metrics, and the Grafana dashboard will query and render those metrics. So by keeping the demo dashboard open and refreshing, you are helping to generate the very metrics that the dashboard presents, making the demo much more visually useful while being very meta.
In addition to generating metrics, using the trickster-labeled datasources generates traces that are viewable in Jaeger UI, as described above.
Stopping the Demo and Cleaning Up
To stop and remove the demo, run docker-compose down in the ./examples/docker-compose directory.
1.2 - Where to Place Trickster
Depending upon the size of your existing or planned deployment, there are several placement configurations available. These designs are suggestions based on common usage, and you may find alternative or hybrid placement configurations that make the most sense for your situation, based on the activity of your Dashboard and TSDB instance(s).
Single “Everything”

Single “Everything” is the most common placement model. In this configuration, you have one optional dashboard endpoint, one Trickster endpoint and one HTTP or TSDB endpoint. Behind each endpoint, you may have a single instance or a cluster. Each component is only aware of the other component’s endpoint exposure and not the underlying configuration. This configuration represents a one-for-one-for-one deployment of your Dashboard, Origin, and Trickster endpoints.
Multiple Backends

In a Multiple Backend placement, you have one dashboard endpoint, one Trickster endpoint, and multiple TSDB and/or HTTP endpoints. Trickster is aware of each upstream endpoint and treats each as a unique backend to which it proxies and caches data independently from the others. Trickster routes a request to a specific backend based on Host Header or URL Path in the client request.
This setup may benefit situations where you have one or more static file server origins serving HTML, CSS and JavaScript assets and/or one or more API endpoints, all supporting a common platform.
For Time Series Dashboard acceleration, this is a good configuration to use when you have a single dashboard that displays data about multiple redundant clusters (each with its own TSDB), or when you have a single dashboard representing information about many different kinds of systems. For example, if you operate a “Dashboard as a Service” solution under which many teams use your Dashboard system by designing their own dashboard screens and bringing their own databases, a single Trickster endpoint can be used to accelerate dashboards for all of your customers.
You will need to configure each Trickster-to-TSDB mapping separately in your dashboard application as a separately named TSDB data source. Refer to the multi-origin documentation for configuring multi-origin support in Trickster and Grafana.
In this configuration, be aware that the default ‘memory’ cache may be underpowered depending on the number of customers, as well as the size and number of queries that need to be cached by each customer. Refer to the caches document to select and configure the caching layers as needed to meet your specific situation.
Multi-Trickster

In a Multi-Trickster configuration, you have one dashboard endpoint, multiple Trickster endpoints, and multiple TSDB or HTTP endpoints, with each Trickster Endpoint having a one-to-one mapping to a TSDB/HTTP Endpoint as a pair. This is a good design if Multiple Backends is not performant enough for the amount of activity associated with your solution (e.g., you need more Tricksters). If the Dashboard system owner is different from the TSDB system owner, either party could own and operate the Trickster instance.
1.3 - Configuring Trickster
There are 3 ways to configure Trickster, listed here in the order of evaluation.
- Configuration File
- Environment Variables
- Command Line Arguments
Note that while the Configuration file provides a very robust number of knobs you can adjust, the ENV and CLI Args options support only basic use cases.
Internal Defaults
Internal Defaults are set for all configuration values, and are overridden by the configuration methods described below. All Internal Defaults are described in examples/conf/example.full.yaml comments.
Configuration Files
Trickster accepts a -config /path/to/trickster.yaml command line argument to specify a custom configuration file. The path can also name a directory containing configuration files. If a provided path cannot be accessed by Trickster, it will exit with a fatal error.
When a -config parameter is not provided, Trickster will check for the presence of a config file at /etc/trickster/trickster.yaml and load it if present, or proceed with the Internal Defaults if not present.
Refer to examples/conf/example.full.yaml for full documentation on format of a configuration file.
Multiple Configuration Files
When -config names a file, Trickster loads that file first and then loads supported files from a sibling conf.d directory, if the directory exists. The primary file can select a different include directory:
main:
config_include_directory: config-parts
A relative config_include_directory is resolved from the directory containing the primary file. An explicitly configured include directory must exist. Only the primary file can set this option; included files cannot redirect configuration discovery.
When -config names a directory, Trickster loads supported files directly from that directory. The directory must contain at least one supported file, and files in this mode cannot set main.config_include_directory.
In both modes, Trickster:
- Loads only direct regular files whose names do not start with
., with.conf,.yaml, or.ymlextensions matched case-insensitively. - Loads directory entries in ascending lexical filename order. The primary file, when present, always comes first.
- Recursively merges mappings. A later scalar or sequence replaces the earlier value, while a later mapping adds to or overrides individual keys in the earlier mapping.
- Requires each file participating in a multi-source configuration to have a mapping root, one YAML document, and no duplicate keys.
For example, a fragment containing only backends.prometheus.origin_url can change that field without removing the other fields under the prometheus backend. Use null, rather than an empty mapping, when a later file must clear an earlier mapping value.
Reserved Names
Some object-name prefixes are reserved in every named section (backends, caches, listeners, discovery, rules, request_rewriters, negative_caches, tracing, authenticators, geo_locators, and geo_acls) for configuration that Trickster generates internally at runtime. A file or fragment that defines a name with a reserved prefix fails to load. The reserved prefixes are:
| Prefix | Producer |
|---|---|
kgw-- | Kubernetes Gateway/Ingress controller |
Generated configuration is merged after all files and fragments, with the same deep-merge behavior. It may only add objects under the named sections above, so it can never change main, frontend, logging, metrics, or mgmt settings or replace a file-defined object. A change to the generated configuration makes the running configuration stale for reload purposes in the same way as a change to a file, and every reload (SIGHUP, the reload handler, and auto_reload_interval) carries the current generated configuration forward.
Renamed Backend Options
This backend option was renamed. The former key is still accepted, and applies only when the current key is not set; configuration dumps show only the current key.
| Former key | Current key |
|---|---|
fastforward_ttl | partial_bucket_ttl |
Configuring Secrets or Sensitive Information
Trickster supports Environment variable substitution in its configuration file where sensitive information is expected.
- Supported via the following fields:
caches[*].redis.password,backends[*].healthcheck.headers,backends[*].cors.headers,backends[*].paths[*].cors.headers,backends[*].paths[*].request_headers,backends[*].paths[*].request_params,backends[*].paths[*].response_headers,backends[*].alb.user_router.users[*].to_credential,authenticators[*].users,discovery[*].headers
Usage ${ENV_VAR_NAME}, example:
caches:
default:
redis:
password: "${MY_REDIS_PW}"
Environment Variables
Trickster will then check for and evaluate the following Environment Variables:
TRK_ORIGIN_URL=http://prometheus.example.com:9090- The default origin URL for proxying all http requestsTRK_ORIGIN_TYPE=prometheus- The type of supported backend serverTRK_LOG_LEVEL=INFO- Level of Logging that Trickster will outputTRK_PROXY_PORT=8480-Listener port for the HTTP Proxy EndpointTRK_METRICS_PORT=8481- Listener port for the Metrics and pprof debugging HTTP Endpoint
Command Line Arguments
Finally, Trickster will check for and evaluate the following Command Line Arguments:
-log-level INFO- Level of Logging that Trickster will output-config /path/to/trickster.yaml- See Configuration Files section above-origin-url http://prometheus.example.com:9090- The default origin URL for proxying all http requests-provider prometheus- The type of supported backend server-proxy-port 8480- Listener port for the HTTP Proxy Endpoint-metrics-port 8481- Listener port for the Metrics and pprof debugging HTTP Endpoint
Inbound Listeners
The top-level listeners map configures inbound listeners. Trickster always auto-defines three entries using the existing defaults: default, metrics, and mgmt.
Native MySQL listeners have additional protocol, authentication, TLS, and
session-lifecycle requirements. See the MySQL Provider Guide before
configuring protocol: mysql.
ClickHouse Native listeners use protocol: clickhouse; see the
ClickHouse Support Guide for ingress, origin, and TLS options.
Media over QUIC listeners use protocol: moq and bind UDP; see the
MoQ Relay Guide for their options, certificates and session routing.
listeners:
default:
address: ""
port: 8480
tls_address: ""
tls_port: 8483
connections_limit: 0
read_header_timeout: 10s
private_api:
protocol: http
address: 127.0.0.1
port: 9080
backends:
default:
listener_names: [default, private_api]
provider: prometheus
origin_url: http://prometheus:9090
private:
listener_names: [private_api]
provider: reverseproxy
origin_url: http://private-origin
listener_names binds a backend to one or more compatible listeners. An ordinary unbound backend uses default; internal routing targets remain unexposed. A backend cannot select the reserved mgmt or metrics listeners, and validation fails for undefined or provider-incompatible listeners.
An ip_acl_name on a listener, backend, or path names an entry in ip_acls. A listener list and a backend list both apply. A path name replaces the backend list, and none clears it for that path.
Each native listener maps to exactly one backend, except a moq listener, which may serve several moq backends as long as no two claim the same hosts and paths: each session is routed by the authority and path it names. Multiple HTTP listeners can share a backend, ClickHouse can bind the same backend to HTTP and ClickHouse Native listeners, and a moq backend can serve a moq listener and, by WebTransport, HTTP listeners.
A rate_limiter_name on a listener, backend, or path names an entry in rate_limiters. The listener limiter is a separate outer layer. A path name replaces the backend limiter, and none clears it for that path.
Each native listener maps to exactly one backend. Multiple HTTP listeners can share a backend, and ClickHouse can bind the same backend to HTTP and ClickHouse Native listeners.
A user-defined listener with no mapped backend is not started and produces a warning. A configured TLS port is enabled only when at least one backend mapped to that listener provides a valid frontend certificate and key in its tls section; otherwise Trickster disables that TLS port and logs a warning.
Trusted Proxies
A listener behind a load balancer or another proxy sees that proxy’s address
as the connection peer. trusted_proxies lists the addresses and CIDRs of
the proxies in front of a listener; for a connection from one of them,
Trickster resolves the client’s address from the forwarding headers it
sent, taking the nearest Forwarded (for=) or X-Forwarded-For address
that is not itself a trusted proxy, or X-Real-IP when neither header is
present. A connection from any other peer is attributed to the peer itself,
whatever headers it carries, so a client cannot claim an address by sending
its own X-Forwarded-For. The resolved address is what the
access log records as %h and %a, and
what max_query_range rejections are logged against.
proxy_protocol: true accepts a PROXY protocol v1 or v2 header ahead of each
connection, which is how many load balancers pass the client’s address
through a TCP proxy. The header is honored from every peer when
trusted_proxies is empty, and only from a trusted proxy otherwise; a
connection from any other peer is read as plain traffic. The header
precedes the TLS handshake, so it works on a listener’s TLS port too, and
the address it carries is the connection peer for everything that follows,
including trusted_proxies matching. A connection that sends no header
within 10 seconds is closed. Enabling or disabling the PROXY protocol, or
changing trusted_proxies while it is enabled, restarts the listener on
reload; changing trusted_proxies alone is applied without a restart.
listeners:
default:
port: 8480
proxy_protocol: true
trusted_proxies: [10.0.0.0/8, 192.168.1.5]
Forwarding Headers to the Origin
A backend’s forwarded_headers (standard, x, both or none) chooses
which forwarding headers Trickster sends upstream. A client can put any
address it likes in the Forwarded, X-Forwarded-* and X-Real-IP headers
it sends, so Trickster believes them only from the listener’s
trusted_proxies, as Caddy and Traefik do:
- From a trusted proxy, Trickster appends its own hop to the hops the request
arrived with, and forwards its
X-Real-IP. - From any other peer, Trickster drops those hops and
X-Real-IP, and the origin receives Trickster’s hop alone, naming that peer.
The same applies on paths served by the passthrough handler. A listener
behind a load balancer must list the load balancer in trusted_proxies, or
the origin sees the load balancer, not the client, as the request’s source.
Connection Timeouts and Header Size
These options bound the time and memory a client can hold on an HTTP
listener. They apply to every endpoint of the listener and to the mgmt and
metrics listeners. Changing any of them restarts the listener on reload.
| Option | Default | Effect |
|---|---|---|
read_header_timeout | 10s | How long a client may take to send a request’s line and headers |
read_timeout | 0 (none) | How long a client may take to send a whole request, body included. Set it with care on listeners that receive large uploads |
idle_timeout | 2m | How long a keep-alive connection may wait for its next request before Trickster closes it. 0 keeps idle connections open indefinitely |
max_header_bytes | 0 (1 MB) | The most bytes Trickster reads for a request’s line and headers. HTTP/1.1 requests over it are answered with 431 Request Header Fields Too Large; HTTP/2 and HTTP/3 enforce it on the header block |
listeners:
default:
port: 8480
read_header_timeout: 10s
idle_timeout: 2m
max_header_bytes: 65536
- Without an idle timeout, a client can hold any number of idle connections
open, filling
connections_limitor exhausting file descriptors. - Go’s HTTP server reads up to 4 KB past
max_header_bytesbefore refusing a request, so the effective limit is slightly higher than the value set. - HTTP/3 has no whole-request deadline. On an HTTP/3 endpoint,
read_timeoutbounds reading a request’s body from the time the handler starts, in place ofread_header_timeout. QUIC’s own idle timeout also applies there. - These options have no effect on
tcp,tlsandudplisteners, whose timeouts are understream, or on native protocol listeners.
Path Normalization
Before routing a request, an HTTP listener cleans its path, then routes and
forwards the cleaned path, so the path a route matches is always the path its
origin receives. Without this agreement, a request for /public/../admin/x
would match a /public/ path, such as one with authenticator_name: none,
while an origin that resolves dot-segments served /admin/x. The
path_normalization block on each listener controls the cleaning:
| Option | Values | Default | Effect |
|---|---|---|---|
dot_segments | normalize, reject, off | normalize | normalize removes . and .. segments as RFC 3986 section 5.2.4 prescribes; reject answers any path holding one with 400 Bad Request; off routes and forwards them as received |
merge_slashes | true, false | false | collapses each run of slashes to one, so /a//b routes and forwards as /a/b |
escaped_slashes | keep, reject, unescape | keep | keep treats %2F as data within a segment and forwards it encoded; reject answers any path holding %2F with 400 Bad Request; unescape decodes %2F to a separator before the path is cleaned |
listeners:
default:
port: 8480
path_normalization:
dot_segments: normalize
merge_slashes: false
escaped_slashes: keep
- A percent-encoded dot (
%2E) counts as a dot, so/public/%2e%2e/admin/xis cleaned to/admin/xlike its literal form. - With
escaped_slashes: keep, a segment whose%2Fhides a dot-segment, such as..%2Fadmin, is refused with400 Bad Requestunlessdot_segmentsisoff. Origins disagree on whether%2Fseparates segments, so no single cleaned path is safe to route and forward. - The query string is never changed, and a path that needs no cleaning is routed without any allocation.
- The listener’s HTTP/1.1, HTTP/2 and HTTP/3 endpoints all apply the same options. Changing them applies on reload without restarting the listener.
- The access log records the request URI as the client sent it.
An origin that depends on receiving raw .. segments needs dot_segments: off
on the listeners that front it; any path-scoped control on such a listener can
then be bypassed by a client that sends dot-segments.
Stream Listeners
A listener whose protocol is tcp, tls or udp relays what it receives
without reading it. A tcp listener relays each accepted connection’s bytes
to its one mapped backend, and a udp listener relays each client’s
datagrams to its one mapped backend over a session of its own, so replies
find their way back. A tls listener relays TLS connections unterminated:
it reads the server name the client offers in its ClientHello, selects the
mapped backend whose hosts name it (a precise host first, then the longest
wildcard, *.example.com spanning one label and **.example.com any
number), or the one mapped backend with no hosts as the catch-all, and
then relays the whole connection, ClientHello included, so the backend
terminates TLS itself. A connection whose server name nothing routes, or
whose first bytes are not a ClientHello, is closed. A tls listener never
holds a certificate, so tls_port and tls_runtime_certs do not apply to
it; every stream listener uses port and address alone.
A stream listener’s backend is a reverseproxy (rp) backend, whose
origin_url supplies the host and port to dial and nothing more (the scheme
may be tcp:// or udp://), or an alb backend whose pool members are such
backends. The ALB may use any mechanism that commits to one member: rr,
p2c, lc, lt or hrw (see Load Balancing Stream
Listeners). Each connection or
session is committed to one pool member for its whole life, chosen as an HTTP
request’s is. By default a member that cannot be dialed refuses its share
rather than passing it to a sibling, so it is health checks, discovery
readiness or passive health that take a dead member out of rotation; set
alb.stream.connect_retries to try another member instead. A tcp:// member
with a healthcheck.interval is probed by opening a connection to it; a
udp:// member has no generic probe. A member whose origin host is under the
reserved .invalid domain, which can never resolve, refuses its share
without a lookup, which is how a share that must be refused is expressed. A
discovery-backed ALB works too, and a scheme of tcp or udp on its
query keeps the discovered members’ origins honest. No cache, path, handler
or HTTP setting applies to a stream backend.
listeners:
postgres:
protocol: tcp
port: 5432
stream:
connect_timeout: 5s
idle_timeout: 30m
sni:
protocol: tls
port: 8443
dns:
protocol: udp
port: 53
backends:
primary:
provider: rp
origin_url: tcp://db-primary:5432
listener_names: [postgres]
shop:
provider: rp
origin_url: tcp://shop-tls:8443
hosts: [shop.example.com]
listener_names: [sni]
other:
provider: rp
origin_url: tcp://catch-all-tls:8443
listener_names: [sni]
resolver:
provider: rp
origin_url: udp://10.0.0.53:53
listener_names: [dns]
The stream block tunes the relay: connect_timeout (default 10s) bounds
the name lookup and dial of the backend and, on a tls listener, the wait
for the client’s ClientHello; idle_timeout closes a connection over which
no byte has moved in either direction for that long (default none), and ends
a UDP session that has carried no datagram for that long (default 60s,
since a datagram flow has no close of its own). The idle timeout is the
connection’s: a client receiving a stream while sending nothing, or sending
one to a backend that answers only at the end, is not idle, while a write
blocked for the whole period is a stalled receiver and ends the connection.
Either side may half-close: a backend that finishes sending while the client
still has data to send is relayed as TCP allows, PROXY protocol and
connection limit included.
connections_limit bounds the connections a tcp or tls listener relays
at once, an accept beyond it waiting for one to end as on an HTTP listener,
and bounds the sessions a udp listener holds at once, a datagram from a
new client beyond it being dropped and counted as refused; a udp
listener with no limit holds at most 1024 sessions, since every open session
keeps an upstream socket, two workers and a reply buffer. The socket’s
receive loop never waits on the network: each session’s datagrams are queued
for a writer of its own, which relays them in order, so a backend that stops
accepting writes stalls that client alone. A session may hold sixteen
datagrams for its writer, and every session together eight megabytes, the
datagram each writer is in the middle of writing included; a datagram
beyond either is dropped and counted, as is one whose write blocked
for a full second, in trickster_proxy_stream_dropped_datagrams_total. A
session is opened on its worker too, so a slow or failing name lookup for
one client delays no other client’s datagrams and no shutdown; the datagrams
the client sent meanwhile are kept, up to four per session and a megabyte
across every session still opening, and relayed once the backend is reached,
ahead of anything sent later. At most 128 sessions open and 32 resolve or
dial at once, a resolved backend name being reused for thirty seconds, and a
datagram from a new client beyond those is dropped. A client
whose backend cannot be reached is remembered for five seconds, dropping
what it sends, rather than looked up again per datagram; such a client holds
no session slot and nothing it sends prolongs the memory, so a sender of
unreachable flows cannot refuse real ones. proxy_protocol and
trusted_proxies apply to tcp and tls listeners as to HTTP ones.
Changing a stream listener’s backends or stream block on reload swaps the
routing in place; connections already relayed keep the backend they reached.
A stream listener is drained on shutdown like any other: it stops accepting
connections, or datagrams from new clients, keeps relaying the established
connections and sessions until they end or the drain timeout passes, then
closes both sides of every relay and ends every pending dial.
A tcp, tls, http or https endpoint binds a TCP port and a udp
endpoint, like an HTTP/3 endpoint, a UDP port, so a tcp and a udp
listener may share a port number; validation refuses two endpoints of one
transport on one address and port.
The top-level frontend section and listener address/port fields under metrics and mgmt remain supported during the compatibility period. Trickster logs deprecation warnings when those legacy listener settings are used. When the same built-in listener is present in listeners, its new configuration takes precedence.
Configuration Validation
Trickster can validate configuration files by running trickster -validate-config -config /path/to/config. Trickster will load the file or directory and exit with the validation result, without running the configuration. The command runs every configuration check, including those that need the backend clients, such as route registration and sticky cookie conflicts. It opens no listener, cache or log file, builds no authenticator and starts no health check or discovery, so startup can still fail at one of those steps.
Reloading the Configuration
Trickster can gracefully reload its configuration sources from disk without impacting the uptime and responsiveness of the application.
Trickster supports manual reloads by requesting an HTTP endpoint or sending a SIGHUP (e.g., kill -1 $TRICKSTER_PID) to the Trickster process. It can also poll the effective configuration sources automatically. In all cases, at least one effective configuration source must have changed since the configuration was loaded.
Automatic Config Reload
Trickster can poll its effective configuration sources and reload after a change. This is disabled by default. Set mgmt.auto_reload_interval to a positive duration to enable it:
mgmt:
auto_reload_interval: 10s
Polling uses the same validation and graceful reload path as SIGHUP and the management endpoint. The interval itself is reloadable, so a successful configuration update can change or disable automatic reloads. Polling is suitable for Kubernetes ConfigMap projected volumes, whose atomic symlink updates are not reliably represented as writes to the mounted file by filesystem notification APIs.
Config Reload via SIGHUP
Once you have made the desired modifications to your config file, send a SIGHUP to the Trickster process by running kill -1 $TRICKSTER_PID. The Trickster log will indicate whether the reload attempt was successful or not.
Config Reload via HTTP Endpoint
Trickster provides an HTTP Endpoint for viewing the running Configuration, as well as requesting a configuration reload.
The reload endpoint is configured by default to listen on address 127.0.0.1 and port 8484, at /trickster/config/reload. These values can be customized, as demonstrated in the example.full.yaml The examples in this section will assume the defaults. Set the port to -1 to disable the reload HTTP interface altogether.
To reload the config, simply make a GET request to the reload endpoint. If an underlying configuration source has changed, or a supported file has been added to or removed from a configured directory, the configuration will be reloaded and the caller will receive a success response. If the configuration sources have not changed, the caller will receive an unsuccessful response, and reloading will be disabled for the duration of the Reload Rate Limiter. By default, this is 3 seconds, but can be customized as demonstrated in the example config file.
If a listener address or port changes, Trickster drains the old listener before starting its replacement. Listeners whose network settings do not change retain their open sockets and receive the refreshed router in place. Removed or newly unused listeners are drained and stopped, while newly mapped listeners are started. The drain period is configurable and defaults to 30 seconds. The Drain Timeout also applies to old log files when a new log filename is provided.
Graceful Shutdown and Readiness
On SIGTERM or SIGINT, Trickster shuts down in three steps:
- The readiness endpoint (default
/trickster/ready, configurable viamgmt.ready_handler_path) immediately begins returning503 draining. It is served on every proxy listener and on the management listener, and otherwise returns200 readyonly while every listener is serving. When thekubernetessection is configured it also returns503 not programmedfrom before the listeners open until the controller’s first translation is serving (a translation the data plane rejects programs nothing), so a new pod is not routed to before the routes it exists to serve are in place. - Listeners keep accepting connections for
mgmt.shutdown_delay(default0s) so load balancers that poll readiness can stop routing new traffic here first. - Listeners stop accepting and in-flight requests are given up to
mgmt.shutdown_drain_timeout(default: the value ofreload_drain_timeout, 30 seconds) to complete, after which any remaining connections are closed.
A second SIGTERM or SIGINT during the delay or drain closes all connections immediately.
/trickster/ping is a liveness check only; it returns 200 whenever the process can serve HTTP, including during a drain. In Kubernetes, use /trickster/ping for the liveness probe and /trickster/ready for the readiness probe. Pod deletion removes the pod from Service endpoints concurrently with sending SIGTERM, so either set shutdown_delay or add a preStop sleep of a few seconds to let endpoint updates propagate before the listeners close, and set terminationGracePeriodSeconds to at least the preStop sleep plus shutdown_delay plus shutdown_drain_timeout, with a few seconds of margin. deploy/kube/deployment.yaml shows this configuration.
View the Running Configuration
Trickster also provides a http://127.0.0.1:8484/trickster/config endpoint, which returns the yaml output of the currently-running Trickster configuration. The YAML-formatted configuration will include all defaults populated, overlaid with any configuration file settings, command-line arguments and or applicable environment variables. By default, this interface is available only on the management listener. Set mgmt.config_handler_listener to metrics, both, or off to change where it is exposed. This path is configurable as demonstrated in the example config file. Both views mask authenticator users, and under an authenticator’s config they mask the value of any key whose name contains secret, key, token or password, at any depth.
Trickster also provides a sanitized view of the running configuration at http://127.0.0.1:8484/trickster/config/sanitized. If the config_handler_path is customized, append /sanitized to the configured path. The sanitized output deep-copies the running configuration, renames cache, backend, listener, and tracing resources by provider and sequence number (for example, prom-1, prom-2, alb-1, memory-1, listener-1, otlp-1), renames authenticators as auth1, auth2, etc., renames geo locators, geo ACLs, IP access lists and rate limiters as geo-locator-1, geo-acl-1, ip-acl-1 and rate-limit-1, etc. (keeping default, which a geo ACL names by omission), redacts geofeed entries, geo ACL exempt addresses and IP access list entries other than all, updates references to those resources in backend, path, ALB, rule, cache, tracing, listener, and authenticator mappings and in the kubernetes section’s defaults and ingress.listener_names, replaces backend origin_url, Redis endpoint and endpoints, tracing endpoint, and Host-related request rewriter values with example.com, redacts per-path request and response header values, and replaces embedded authenticator users with user1: redacted, user2: redacted, etc. This endpoint is intended for sharing running configuration details in support requests without exposing private infrastructure names, origin endpoints, or user credentials.
Kubernetes Gateway and Ingress Controller
Trickster can act as a Kubernetes Gateway API and Ingress controller, programming its own data plane from the cluster’s routing objects. The controller does not exist unless the top-level kubernetes section is present; a Trickster that is not a gateway holds no watches, elects no leader, and generates no configuration. A complete deployment (RBAC, classes, ConfigMap, Deployment, Service) is in deploy/kube and described in kubernetes-deploy.md.
kubernetes:
enabled: true # default true when the section is present
connection:
in_cluster: true
gateway_class_controller_name: trickstercache.org/gateway-controller
ingress_class: trickster
watch_namespaces: [] # empty watches every namespace
resync_interval: 10m
debounce_window: 1s
read_only: false # true makes no writes to the cluster at all
leader_election:
enabled: true
name: trickster-gateway-controller
ingress:
listener_names: [web, websecure] # empty serves them on the default frontend
published_service:
namespace: trickster
name: trickster-gateway
defaults:
routing_mode: service # required: service or endpoint
cache_name: default
negative_cache_name: api-errors
tracing_name: otlp # operator-only; no annotation for these
req_rewriter_name: strip-internal-headers
authenticator_name: gateway-auth
geo_acl_name: north-america # operator-only; must place addresses, as stream routes take it too
ip_acl_name: office # operator-only; client_ip and reject
rate_limiter_name: per-client # operator-only; HTTP and gRPC routes, not stream routes
health_mode: provider
healthcheck: # the active probe used when health_mode is probe
path: /healthz
interval: 5s
sticky_secret_file: /etc/trickster/sticky/key # keys session tokens; the same on every replica
http3: true # generated HTTPS listeners also serve HTTP/3; default true
defaults.routing_mode is required and has no default. In service mode a generated backend sends traffic to the Service’s cluster IP and kube-proxy load balances it. In endpoint mode Trickster discovers the Service’s endpoints and load balances across them itself, which is what makes zero-error rolling deploys and per-endpoint health possible. The two have different failure modes, so Trickster refuses to guess: a configuration that omits the mode fails validation rather than silently picking one. In endpoint mode, defaults.health_mode decides whether discovered endpoints are trusted on their EndpointSlice readiness (provider, the default) or actively probed (probe), and defaults.healthcheck is the probe used in the latter case; unset, it probes the origin’s root every 5 seconds.
defaults.sticky_secret_file names a file of at least 32 bytes that keys the session tokens every generated ALB issues, for a route’s sessionPersistence or a sticky annotation, parameter or policy. Mount it from a Secret at the same path on every replica, so that each replica honors the others’ tokens. A GatewayClass’s sticky_secret parameter, which names a Secret, takes its place for that class’s routes. Unset, each process uses a random key of its own, which suits a single replica only. See kubernetes-gateway.md.
defaults.http3 decides whether every HTTPS listener the controller generates for a Gateway also serves HTTP/3, on the same port number over UDP, as the WebTransport sessions of a MoQ relay need; it is true unless set false, and a GatewayClass’s http3 parameter overrides it for the class. The Service in front of the controller must publish the UDP port for clients to reach it.
gateway_class_controller_name is the name this instance claims GatewayClasses with, and is also matched against an IngressClass’s spec.controller. Objects belonging to any other controller are ignored entirely and never receive status, because writing status onto another controller’s object is worse than ignoring it. An Ingress with no spec.ingressClassName is claimed only when one of this controller’s IngressClasses is annotated ingressclass.kubernetes.io/is-default-class: "true".
watch_namespaces and namespace_selector both narrow which namespaces the controller reads, and are mutually exclusive. watch_namespaces narrows the watches themselves, so a namespace-scoped RBAC grant is sufficient. namespace_selector watches all namespaces and filters by the namespace’s labels, which additionally requires cluster-wide read access to namespaces.
ingress.listener_names are the listeners claimed Ingresses are served on, named exactly as a backend names the listeners it is served on. A Gateway declares its own ports, but an Ingress has no way to, so its listeners are configured in the listeners section like any other and named here; naming none serves them on the default frontend, which is where a backend that names no listener is served. See kubernetes-ingress.md for how Ingress objects are translated and for the trickstercache.org/* annotations, and kubernetes-gateway.md for how Gateway API objects are translated, including how a GatewayClass’s parametersRef overrides defaults for its Gateways.
defaults names objects defined elsewhere in the configuration — a cache, a negative cache, a tracer, a request rewriter, an authenticator, a geo ACL, an IP access list, a rate limiter — and a name that is not defined fails startup, the same way a backend’s would. A route may override cache_name and negative_cache_name by annotation; the rest are operator settings only, because selecting a tracer is infrastructure and selecting an authenticator, a geo ACL, an access list, or a rate limiter is a capability. A rate limiter named in defaults must use HTTP keys, unit requests or unset, and an action other than close. See kubernetes-ingress.md. Caching behavior beyond what an annotation may carry — a time series provider, the cache key, the result header — is attached with the TricksterCachePolicy resource; see kubernetes-cache-policy.md.
read_only, leader_election and published_service govern what the controller writes back to the cluster. Every replica programs its own data plane; status on the claimed objects, and Events describing what could not be done with them, are written by one replica, elected over a Lease named by leader_election, or by every replica when leader_election.enabled is false. read_only: true writes nothing and needs no write permission. published_service names the Service whose addresses are published into Gateway and Ingress status; it is watched in its own namespace, which need not be a watched one. defaults.tracing_name also selects the tracer the controller’s own reconcile spans report to. See kubernetes-gateway.md for what is written and metrics.md for the controller’s metrics.
Generated objects are named with the reserved kgw-- prefix. Configuration files may not define an object with that prefix and the controller may not generate one without it, so generated and hand-written configuration can never collide in either direction. See kubernetes-rbac.md for the permissions the controller needs.
2 - Backends
2.1 - Using Multiple Backends with a single Trickster instance
Trickster supports proxying to multiple backends in a single Trickster instance, by examining the inbound request and using a multiplexer to direct the proxied request to the correct upstream origin, in the same way that web servers support virtual hosting.
There are 2 ways to route to multiple backends.
- HTTP Pathing
- DNS Aliasing
Basic Usage
To utilize multiple backends, you must craft a Trickster configuration file to be read when Trickster starts up - operating with environment variables or command line arguments only supports accelerating a single backend. The example.full.yaml provides good documentation and commented sections demonstrating multiple backends. The config file should be placed in /etc/trickster/trickster.yaml unless you specify a different path when starting Trickster with the -config command line argument.
Each backend that your Trickster instance supports must be explicitly enumerated in the configuration file. Trickster does not support open proxying.
Example backend configuration:
backends:
my-backend-01: # routed via http://trickster:8480/my-backend-01/
provider: prometheus
origin_url: http://my-origin-01.example.com
my-backend-02: # routed via http://trickster:8480/my-backend-02/
hosts: [ my-fqdn-02.example.com ] # or http://my-fqdn-02.example.com:8480/
provider: prometheus
origin_url: http://my-origin-02.example.com
my-backend-03:
path_routing_disabled: true # only routable via Host header, an ALB, or a Rule
hosts: [ my-fqdn-03.example.com ] # or http://my-fqdn-03.example.com:8480/
provider: prometheus
origin_url: http://my-origin-02.example.com
Default Backend
Whether proxying to one or more upstreams, Trickster has the concept of a “default” backend, which means it does not require a specific DNS hostname in the request, or a specific URL path, in order to proxy the request to a known backend. When a default backend is configured, if the inbound request does not match any mapped backends by path or FQDN, the request will automatically be routed through the default backend. You are probably familiar with this behavior from when you first tried out Trickster with the using command line arguments.
Here’s an example: if you have Trickster configured with a backend named foo that proxies to http://foo/ and is configured as the default backend, then requesting http://trickster/image.jpg will initiate a proxy request to http://foo/image.jpg, without requiring the path be prefixed with /foo. But requesting to http://trickster/foo/image.jpg would also work.
The default backend can be configured by setting is_default: true for the backend you have elected to make the default. Having a default backend is optional. In a single-backend configuration, Trickster will automatically set the sole backend as is_default: true unless you explicitly set is_default: false in the configuration file. If you have multiple backends, and don’t wish to have a default backend, you can just omit the value for all backends. If you set is_default: true for more than one backend, Trickster will exit with a fatal error on startup.
Path-based Routing Configurations
In this mode, Trickster will use a single FQDN but still map to multiple upstream backends by path. This is the simplest setup and requires the least amount of work. The client will indicate which backend is desired in URL Path for the request.
Example Path-based Configuration with Multiple Backends:
backends:
# backend1 backend
backend1:
origin_url: 'http://prometheus.example.com:9090'
provider: prometheus
cache_name: default
is_default: true
# "foo" backend
foo:
origin_url: 'http://influxdb-foo.example.com:9090'
provider: influxdb
cache_name: default
# "bar" backend
bar:
origin_url: 'http://prometheus-bar.example.com:9090'
provider: prometheus
cache_name: default
Using HTTP Path as the Backend Routing Indicator
The client prefixes the Trickster request path with the Backend Name.
This is the recommended method for integrating with applications like Grafana.
Example Client Request URLs:
To Request from Origin
foo: http://trickster.example.com:8480/foo/query?query=xxxTo Request from Origin
bar: http://trickster.example.com:8480/bar/query?query=xxxTo Request from Origin
backend1as default: http://trickster.example.com:8480/query?query=xxxTo Request from Origin
backend1(Method 2, with Origin Name): http://trickster.example.com:8480/backend1/query?query=xxxConfiguring Grafana to request from backend
foovia Trickster:

DNS Alias Configuration
In this mode, multiple DNS records point to a single Trickster instance. The FQDN used by the client to reach Trickster is mapped to specific backend configurations using the hosts list. In this mode, the URL Path is not considered during Backend Selection.
Host matching is case-insensitive and ignores the port. An entry may also be a single-label wildcard such as *.example.com, which matches foo.example.com but not example.com or a.b.example.com, or an any-depth wildcard such as **.example.com, which matches foo.example.com and a.b.example.com but not example.com; an exact entry on another backend always wins over a wildcard, and a *. entry wins over a **. entry for the same domain. A * anywhere else in an entry is rejected at load time. See Host Resolution for the full precedence order.
Example DNS-based Backend Configuration:
backends:
# backend1 backend
backend1:
hosts: # users can route to this backend via these FQDNs, or via `/backend1`
- 1.example.com
- 2.example.com
origin_url: 'http://prometheus.example.com:9090'
provider: prometheus
cache_name: default
is_default: true
# "foo" backend
foo:
hosts: # users can route to this backend via these FQDNs, or via `/foo`
- trickster-foo.example.com
origin_url: 'http://prometheus-foo.example.com:9090'
provider: prometheus
cache_name: default
# "bar" backend
bar:
hosts: # users can route to this backend via these FQDNs, or via `/bar`
- trickster-bar.example.com
origin_url: 'http://prometheus-bar.example.com:9090'
provider: prometheus
cache_name: default
Example Client Request URLs:
To Request from Origin
foo: http://trickster-foo.example.com:8480/query?query=xxxTo Request from Origin
bar: http://trickster-bar.example.com:8480/query?query=xxxTo Request from Origin
backend1as default: http://trickster.example.com:8480/query?query=xxxTo Request from Origin
backend1(Method 2, via FQDN): http://backend1.example.com:8480/query?query=xxx
Note: It is currently possible to specify the same FQDN in multiple backend configurations. You should not do this (obviously). A future enhancement will cause Trickster to exit fatally upon detection at startup.
Disabling Path-based Routing for a Backend
You may wish for a backend to be inaccessible via the /backend_name/ path, and only by Hostname or as the target of a rule or ALB. You can disable path routing by setting path_routing_disabled: true for the backend, as in this example, which requires the Request’s Host header match 1.example.com or 2.example.com in order to be routed to the backend:
backends:
backend1:
hosts:
- 1.example.com
- 2.example.com
origin_url: 'http://prometheus.example.com:9090'
provider: prometheus
cache_name: default
is_default: false
path_routing_disabled: true # this will disable routing through /backend1
2.2 - Health Checks
Trickster Service Health - Ping Endpoint
Trickster provides a /trickster/ping endpoint that returns a response of 200 OK and the word pong if Trickster is up and running. The /trickster/ping endpoint does not check any proxy configurations or upstream origins. The path to the Ping endpoint is configurable, see the configuration documentation for more information.
Upstream Connection Health - Backend Health Endpoints
Trickster offers health endpoints for monitoring the health of the Trickster service with respect to its upstream connection to origin servers.
General Endpoint
The main health check path is /trickster/health, which by default will return a text/plain summary of the backend health. You can request YAML or JSON format using the appropriate Accept header, or by providing a ?json or ?yaml query param.
ALBs and Their Pool Members
Each ALB is listed with its mechanism and its pool members, grouped by health. The text form appends them to the ALB’s line as u:[...] (unavailable), a:[...] (available), nc:[...] (not checked) and d:[...] (draining). The JSON and YAML forms name them unavailablePoolMembers, availablePoolMembers, uncheckedPoolMembers, initializingPoolMembers and drainingPoolMembers.
A draining member, one marked drain: true or a discovered one that is terminating but still serving, is listed under its health and again as draining: it still serves the sticky sessions it has, but takes no new work. An ALB with a pool is listed as available while it has a member it would send new work to, so an ALB whose members all drain is listed as unavailable, though it still serves their sessions. For example:
backends:
app1:
provider: reverseproxycache
origin_url: http://app1.example.com
healthcheck:
interval: 5s
app2:
provider: reverseproxycache
origin_url: http://app2.example.com
healthcheck:
interval: 5s
app:
provider: alb
alb:
mechanism: rr
pool:
- app1
- name: app2
drain: true
{
"title": "Trickster Backend Health Status",
"updateTime": "2026-09-26 00:00:00 UTC",
"available": [
{ "name": "app1", "provider": "rpc" },
{ "name": "app2", "provider": "rpc" },
{
"name": "app",
"provider": "alb",
"mechanism": "rr",
"availablePoolMembers": [ "app1", "app2" ],
"drainingPoolMembers": [ "app2" ]
}
]
}
See All-Backends Health Status Page for more about how ALBs are listed.
Backend-Specific Endpoints
Each HTTP backend’s health check path is /trickster/health/BACKEND_NAME. For example, if your backend is named foo, you can perform a health check of the upstream server at http://<trickster_address:port>/trickster/health/foo. Native-protocol backends publish their scheduled status through the general endpoint without registering a synthetic HTTP origin route.
The backend health path prefix /trickster/health/ is customizable. See the example.full.yaml for more info about setting the health_handler_path configuration, or refer to this example:
mgmt:
# this overrides the default '/trickster/health' to '/-/trickster/health'
health_handler_path: /-/trickster/health
The behavior of a health request will vary based on the Backend provider, as each has their own health check protocol. For example, with Prometheus, Trickster makes a request to /query?query=up and (hopefully) receives a 200 OK, while for InfluxDB the request is to /ping which returns a 204 No Content.
Supported TSDB Providers are pre-configured in Trickster to perform a suitable health check operation, however these can be overridden in the configuration file.
For non-TSDB Backends, the default behavior is to make a GET request to http://origin_url:port/ and expect a 2xx response. However, all aspects of the Health Check request and expected response are configurable per-Backend.
Native MySQL Health Checks
The MySQL backend uses the same interval scheduler, timeout, transition
thresholds, status registry, metrics, reload carryover, and shutdown lifecycle
as HTTP health checks. Each probe opens a fresh connection with the backend’s
configured origin credentials and TLS policy, completes authentication,
executes COM_PING, and closes the connection.
backends:
mysql1:
provider: mysql
origin_url: mysql://health-user:password@mysql.example:3306/analytics
healthcheck:
interval: 5s
timeout: 3s
failure_threshold: 3
recovery_threshold: 3
Only interval, timeout, failure_threshold, and recovery_threshold apply
to native probes. HTTP verbs, paths, headers, bodies, and expected HTTP response
options are configuration errors for MySQL. Diagnostics exposed in health
status and logs are limited to sanitized authentication, TLS, timeout,
refused-connection, connection, and server-error categories.
Native PostgreSQL Health Checks
A postgres or timescaledb backend is probed over the PostgreSQL wire
protocol with the same scheduler, timeout, thresholds, status registry and
metrics as every other health check. Each probe opens a fresh connection with
the backend’s origin_url credentials and upstream TLS mode, completes
authentication, waits for the origin’s ReadyForQuery, and ends the session
cleanly. A backend whose origin_url has no username (one that relays its
clients’ own authentication) can only be checked for accepting connections.
backends:
tsdb1:
provider: timescaledb
origin_url: postgres://health-user:password@tsdb.example:5432/analytics
healthcheck:
interval: 5s
timeout: 3s
failure_threshold: 3
recovery_threshold: 3
Only interval, timeout, failure_threshold, and recovery_threshold
apply. Failures are reported as a login rejection with the origin’s SQLSTATE,
a timeout, or a connection failure; user names, addresses and passwords never
appear in health status or logs. A native User Router refuses new sessions
for a user whose backend is failing its health check.
Basic Health Check Configuration Example
backends:
server1:
provider: reverseproxycache
origin_url: http://server1
healthcheck: # all values below are optional
verb: HEAD
path: /health
Health Check With Exhaustive Request/Response Options
backends:
server1:
provider: reverseproxy
origin_url: http://server1
healthcheck: # all values below are optional
#
## customizing the health check request
#
verb: HEAD
scheme: https
host: alternate-hostname.example.com
path: /health
query: param1=value1¶m2=value2
headers:
User-Agent: health-check-agent
# if using a POST or PUT method, you can provide a string body
# body: "my health check body"
#
## customizing the expected response
#
# hc fails if a response takes longer than 1s
timeout: 1000ms
# hc fails if the response code is not in the list
expected_codes: [ 200, 204, 206, 301, 302, 304 ]
#
# hc fails if these response headers are not present and have the expected value
expected_headers:
X-Health-Check-Status: success
# hc fails if the stringified response body does not match the expected value
expected_body: "pass"
See more examples in example.full.yaml.
Health Check Integrations with Application Load Balancers
By default, a Backend will only initiate a health check on-demand, upon receiving a request to its health endpoint.
To facilitate integrations with the Trickster Application Load Balancer provider, additional options provide for 1) timed interval health checks and 2) thresholding for consecutive successful or unsuccessful health checks that determine the backend’s overall health status.
Example Health Check Configuration for use in ALB
backends:
server1:
provider: reverseproxy
origin_url: http://server1
healthcheck:
path: /health
timeout: 1000ms # timeout should be <= interval
# for ALB integration:
interval: 1000ms # auto-poll health every 1s
failure_threshold: 3 # backend is unhealthy after 3 consecutive failures
recovery_threshold: 3 # backend is healthy after 3 consecutive successes
The Prometheus default probe is /api/v1/query?query=up. Some multi-tenant Prometheus gateways reject an unbounded up with 400 bad_data: "too many series found", which keeps the member out of any ALB pool it belongs to. Override healthcheck.query with a bounded expression the backend accepts (for example query=vector(1)) when probing such backends.
Other Ways to Monitor Health
In addition to the out-of-the-box health checks to determine up-or-down status, you may want to setup alarms and thresholds based on the metrics instrumented by Trickster. See metrics.md for collecting performance metrics about Trickster.
2.3 - Authenticator
Trickster 2.x provides an Authenticator capability that allows you to protect Backends with an Authentication layer.
Authenticator resources are defined globally by name, and then mapped into any Backend and/or Path configuration as needed. Authenticator users can be loaded from htpasswd or csv files, or directly in the Trickster config file. You can provide credentials in plaintext, bcrypt, Apache MD5 ($apr1$), MD5-crypt ($1$), as well as legacy formats sha-256-crypt and sha-512-crypt ($5$, $6$). PostgreSQL SCRAM-SHA-256 and md5 verifiers are accepted too.
A stored hash or verifier is only ever checked as that format, so a client that sends the stored hash itself as its password is rejected. Plaintext credentials are compared in constant time, and an empty stored credential never matches.
Authenticators work with all Backend provider types. Requests are handled by their respective Authenticators before all other Handlers (e.g., Caches, Rules, Request Rewrites, ALB Routes, etc.). Authentication sees the request as the client sent it: backend and path request rewriters run after it, so a rewriter cannot change the path or credentials that authentication judges.
Native MySQL listeners terminate mysql_native_password authentication and
have stricter credential-source requirements than HTTP authenticators. See the
MySQL Provider Guide.
If a request is routed via a Trickster ALB or Rule Backend through to multiple other Backends - each having different Authenticator configurations - the authentication behavior is currently undefined. In an upcoming Beta release, we will define this use case to either use the Authenticator config (if set) of the very first Backend that handled the request; or to use the first defined Authenticator regardless of how deep into the Backend chain it is.
By default, when an Authenticator handles and successfully authenticates a request, the request’s Auth credentials (e.g., Authorization Header for Basic Auth) are stripped before the request is cloned for any necessary proxying. Setting proxy_preserve: true will preserve these headers instead of stripping them.
Path Protection
When you map an Authenticator to a Backend, all Paths defined for that Backend are protected by the Authenticator. However, you can override the Authenticator on a per-Path basis by including the authenticator_name in a Path config. To bypass a Backend-wide Authenticator in a Path config, use authenticator_name: none. This allows for a few possibilities:
A Backend config with a default / Backend-wide Authenticator that has:
- specific Paths that do not require Authentication
- specific Paths mapped to different Authenticators than the default
A Backend config with no Authenticator defined (so all Paths by default are unprotected) that has:
- specific Paths mapped to different Authenticators than the default
See the example Backend configs below for more details.
Authenticator Providers
Trickster’s Authenticator feature currently supports Basic Auth and ClickHouse-compatible authentication. It was designed with extensibility in mind should there be value in adding additional Authentication providers.
Basic Auth Provider
Basic Auth is supported by using provider: basic in the Authenticator config.
By setting the showLoginForm: true config (see example_auth_1 in the blob below), Trickster will return a WWW-Authenticate: Basic realm="custom-realm-name" header on any request that requires but fails authentication, causing the login form to pop up. When showLoginForm is not present or non-true, Trickster responds with a 401 Unauthorized but does not ask the Basic Auth login form to show.
The realm attribute value defaults to the Authenticator name (e.g., example_auth_1) but can be overridden with the realm config as in the example.
If the user data changes (e.g. updated users_file contents or updated embedded users list), you must send a SIGHUP or other means to reload the Trickster config before the new user pool is processed.
ClickHouse Auth Provider
ClickHouse Auth is supported by using provider: clickhouse in the Authenticator config.
ClickHouse authentication is the same as Basic Auth, except you can also provide user and password URL params. Unless proxy_preserve: true is set, those URL params are removed along with the Authorization header before the request is proxied.
Example Authenticator Configs
# NOTE: Required options unrelated to Authenticators have been omitted from this
# example. It does not represent a fully-functioning Trickster Config.
# See the 'examples' directory for working copy/paste config examples.
backends:
backend01:
provider: reverseproxy # authenticators work with all backend providers
authenticator_name: example_auth_1 # protects backend01 with example_auth_1 authenticator
origin_url: https://example.com
paths:
- path: / # all requests are protected by example_auth_1
match_type: prefix
handler: proxy
backend02:
provider: reverseproxy # no backend-wide authenticator
origin_url: https://example.com
paths:
- path: / # requests will be allowed without auth except the 2 Paths below
match_type: prefix
handler: proxy
- path: /private/
authenticator_name: example_auth_2 # example_auth_2 protects this path only
handler: proxy
- path: /admin/
authenticator_name: example_auth_3 # example_auth_3 protects this path only
handler: proxy
backend03:
provider: reverseproxycache
authenticator_name: example_auth_1 # protects backend03 with example_auth_1 authenticator
origin_url: https://example.com
paths:
- path: / # requests will be challenged by example_auth_1 except the 2 Paths below
match_type: prefix
handler: proxy
- path: /public/
authenticator_name: none # requests to /public will be allowed without auth
handler: proxy
- path: /app/admin/
authenticator_name: example_auth_2 # example_auth_2 protects this path, not auth_1
handler: proxy
authenticators:
# example_auth_1 loads users from a CSV and embeds a supplemental plaintext manifest
# It also shows the login form to client browsers when login has failed
example_auth_1:
provider: basic # http basic auth (required)
proxy_preserve: true # don't strip auth headers when proxying this request upstream
users_file: /path/to/user-manifest.csv # optional users source file
users_file_format: csv # required when users_file is set: csv, csvNoHeader or htpasswd
users: # optional embedded users manifest (username: credential)
user1: red123
config: # optional provider-specific configs
showLoginForm: true # with basic auth, causes the browser to show the login form
realm: custom-realm-name # realm would be example_auth_1 if not overridden here
# example_auth_2 loads users from an htpasswd file (assumed bcrypted credentials)
example_auth_2:
provider: basic
users_file: /path/to/user-manifest.htpasswd # optional users source file
users_file_format: htpasswd # required when users_file is set
# example_auth_3 loads users from the embedded users manifest, credentials already bcrypted
example_auth_3:
provider: basic
users:
user1: $2y$asf;j2ihj0h8vabjkwdqbv29hq
# example_auth_4 loads users from the embedded manifest, credentials injected from env,
# and supports ClickHouse query params (user/password) in addition to Basic Auth
example_auth_4:
provider: clickhouse
users:
user1: ${USER1_PASSWORD_ENV} # ${ENV_NAME} substitution is supported
2.4 - AWS Integration
Trickster uses AWS credentials and SigV4 request signing in two places, and both are configured the same way:
- Signing outbound requests to an origin — the
sigv4block on a backend, below. - Autodiscovery of AWS resources — the
awsdiscovery provider, which reads an AWS API to keep an ALB pool current.
Credentials
Leaving the credential fields empty selects the standard AWS credential chain, in the order the AWS SDK resolves it:
- Environment variables (
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_SESSION_TOKEN) - The shared credentials and config files (
~/.aws/credentials,~/.aws/config), honoringprofile - Web-identity tokens — this is how EKS IAM Roles for Service Accounts (IRSA) and EKS Pod Identity work
- AWS SSO
- The EC2 instance metadata service (IMDSv2)
Prefer the chain over static keys wherever the platform provides one: on EKS use IRSA or Pod Identity, on EC2 use an instance profile. Static keys are supported for environments that have nothing else.
Credential Lifecycle
Credentials are resolved lazily, on the first signed request. Trickster therefore starts even when the instance metadata service is briefly unreachable. Concurrent first requests share one resolution.
- Failures are retried. A failed resolution is reused for 5 seconds and then tried again, so a momentary metadata or STS failure does not permanently disable signing, and an outage costs one blocked resolution per 5 seconds rather than one per request.
- Expiring credentials refresh early. Temporary credentials (IRSA, Pod
Identity, SSO, instance profiles,
role_arn) are refreshed between 2.5 and 5 minutes before they expire, rather than after, so requests are not signed with credentials that lapse in flight. - Rejected credentials are renewed once. When the origin answers 403
with
ExpiredToken,ExpiredTokenException,InvalidClientTokenId,UnrecognizedClientExceptionorInvalidSignatureException, Trickster renews the credentials and resends the request once:- Credentials that expire (IRSA, Pod Identity, SSO, instance profiles,
role_arn) are discarded and resolved again. Concurrent rejections of the same credentials cause a single refresh. - Credentials read from the shared credentials or config file, including temporary keys written there by a refresher, carry no expiry, so the files are read again instead, at most once every 5 seconds. A rewritten file therefore takes effect without a reload.
access_keyandsecret_keyset in thesigv4block, and keys from environment variables, cannot change while Trickster runs, so they are never retried.
- Credentials that expire (IRSA, Pod Identity, SSO, instance profiles,
- One cache per backend. A backend’s proxy and health check requests share one set of credentials.
- Clock skew. SigV4 signatures carry a timestamp, and AWS rejects
requests more than 5 minutes off its clock with
InvalidSignatureException. Keep the host clock synchronized.
Signing failures and credential retries are counted in
trickster_proxy_sigv4_events_total (see metrics) and
logged as warnings, at most once a minute per backend and kind.
Region
region may be set explicitly. When it is not, it is resolved from
AWS_REGION / AWS_DEFAULT_REGION, the shared config file, or instance
metadata. If none of those yields one, requests fail with an error naming
every source that was tried.
The sigv4 Backend Block
sigv4 signs Trickster’s outbound requests to a backend’s origin.
backends:
amp:
provider: prometheus
origin_url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-abc123
sigv4:
region: us-east-1
# credentials omitted: use the chain (IRSA, instance profile, ...)
# access_key: AKIA...
# secret_key: ... # redacted in config dumps and the health page
# profile: production
# role_arn: arn:aws:iam::123456789012:role/TricksterRead
# service: aps # default; see below
| option | meaning |
|---|---|
region | region to sign for; resolved from the environment when unset |
access_key, secret_key | a static credential pair — both or neither |
profile | a profile in the shared config file |
role_arn | a role to assume with whatever the chain resolves first |
service | the AWS service to sign for; defaults to aps |
The service default is aps — Amazon Managed Service for Prometheus.
That is deliberate: earlier releases could sign for nothing else, so every
existing config keeps working unchanged. Set service to sign for a
different AWS service (es for OpenSearch, and so on).
access_key and secret_key must be provided together. A config supplying
only one fails at startup rather than silently falling through to the
chain and authenticating as a different principal.
secret_key is redacted wherever configuration is emitted — the config
dump, the management API, logs, and error messages.
Amazon Managed Service for Prometheus
The common case. Point a prometheus backend at the workspace’s query URL
and add a sigv4 block; Trickster caches and accelerates AMP queries as it
does any other Prometheus origin.
backends:
amp:
provider: prometheus
origin_url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-abc123
sigv4:
region: us-east-1
The IAM principal needs aps:QueryMetrics, aps:GetSeries,
aps:GetLabels, and aps:GetMetricMetadata on the workspace.
Adding prometheus.flavor: amp restricts the backend to AMP’s read APIs
(query, query_range, series, labels, label/<name>/values,
metadata, rules and alerts). Every other path, including
/api/v1/remote_write, is answered by Trickster with an error instead of
being signed and forwarded. The flavor also adds the sigv4 block if it is
missing.
backends:
amp:
provider: prometheus
origin_url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-abc123
prometheus:
flavor: amp
sigv4:
region: us-east-1
Amazon CloudWatch PromQL
CloudWatch serves PromQL over a Prometheus-compatible API at
https://monitoring.<region>.amazonaws.com, covering metrics ingested
through OTLP and enriched AWS vended metrics. Queries are billed by the
samples they scan, so a dashboard that refreshes repeatedly pays again for
every sample it has already seen. Trickster’s delta cache fetches only the
new part of each refresh, and request collapsing merges concurrent viewers
into one upstream query.
backends:
cloudwatch:
provider: prometheus
prometheus:
flavor: cloudwatch
sigv4:
region: us-east-1
# credentials omitted: use the chain (IRSA, instance profile, ...)
Grafana’s Prometheus data source points at Trickster as it would at any Prometheus server; Trickster does the signing.
flavor: cloudwatch applies CloudWatch’s routes, limits and defaults:
- Signing: the
sigv4block is added if it is missing, and itsservicedefaults tomonitoring. - Origin:
origin_urldefaults tohttps://monitoring.<sigv4.region>.amazonaws.com. When the region comes from the environment instead ofsigv4.region, setorigin_urlexplicitly. Anorigin_urlandsigv4.regionnaming different regions fail at startup. - Routes: the PromQL API’s
query,query_range,series,labelsandlabel/<name>/valuesare served and cached. The read-only AWS API calls that Grafana’s CloudWatch data source makes are relayed uncached (see Grafana’s CloudWatch Data Source). Every other path and operation is answered by Trickster with an error instead of being signed and forwarded, so a client cannot use Trickster’s credentials to change anything. - Cache keys:
limit, which caps the series a query returns, is part of each cache key. - Truncated results: CloudWatch returns at most 500 series per query,
or
limitwhen smaller. A result cut short at that cap comes back with a warning and HTTP 200. Such a result is a sample of the series, and a later delta fetch can return a different sample, so Trickster never caches or merges one. It evicts any cached entry for the query, proxies the query to CloudWatch whole, and proxies that query directly for the backend’stimeseries_ttl. Each occurrence is counted intrickster_proxy_truncated_responses_total. Narrow the query, for example with an aggregation, to bring it back under the cap and into the cache. - Volatile window:
volatile_windowdefaults to2m, so the most recent two minutes are refetched while late data arrives. Any explicitvolatile_windoworvolatile_window_pointstakes precedence. - Health check: the default probe is
query=vector(1), which reads no metric and so scans no samples.
The IAM principal needs cloudwatch:GetMetricData and
cloudwatch:ListMetrics; series, labels and label values need only
cloudwatch:ListMetrics.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["cloudwatch:GetMetricData", "cloudwatch:ListMetrics"],
"Resource": "*"
}]
}
Grafana’s CloudWatch Data Source
Grafana’s Prometheus data source needs only the PromQL API. Grafana’s CloudWatch data source can also run PromQL, in its PromQL query mode, and works against Trickster too. Set the data source’s Endpoint to the Trickster backend’s URL, and its Default Region to the backend’s region. Any credentials will do, since Trickster signs with its own and ignores the data source’s.
That one Endpoint carries every AWS call the data source makes, not just PromQL. Trickster caches the PromQL calls and relays the others to their services uncached, re-signed with its own credentials for each service:
| service | relayed operations | used for |
|---|---|---|
| CloudWatch | ListMetrics, GetMetricData, DescribeAlarms, DescribeAlarmsForMetric, DescribeAlarmHistory | Save & test, metric queries, variables, annotations |
| CloudWatch Logs | DescribeLogGroups, GetLogGroupFields, StartQuery, GetQueryResults, StopQuery, GetLogEvents, ListAnomalies, ListAggregateLogGroupSummaries | Save & test, Logs queries |
| EC2 | DescribeRegions, DescribeInstances | the region picker, variables |
| Resource Groups Tagging | GetResources | variables |
| Observability Access Manager | ListSinks, ListAttachedLinks | the cross-account picker |
Any other operation is refused with AccessDeniedException. A request
signed for a region other than the backend’s is refused with
ValidationException rather than answered from the backend’s region. Run
one backend per region and select that region in Grafana. Grant
Trickster’s IAM identity the read permissions for the operations your
dashboards use; Save & test needs cloudwatch:ListMetrics and
logs:DescribeLogGroups.
CloudWatch Limits
These CloudWatch limits apply per account and are not changed by Trickster. Delta fetches and request collapsing reduce how often a dashboard reaches them.
| limit | value |
|---|---|
| query requests per second | 300 |
discovery requests (series, labels, label values) per second | 10 |
| concurrent queries, and concurrent discovery requests | 30 each |
| range per query, including range selectors and lookback | 7 days |
| execution timeout | 20 seconds |
Exceeding a rate or size limit returns HTTP 422, and exceeding concurrency
returns HTTP 429; Trickster does not cache either. Every selector must name
a metric: {__name__=~"..."} alone is rejected.
Making AWS Vended Metrics Queryable
Metrics ingested through OTLP are queryable as they arrive. Classic AWS vended metrics, such as DynamoDB’s, become queryable through PromQL only after both enrichment settings are turned on for the account and region:
aws cloudwatch start-otel-enrichment --region us-east-1
aws observabilityadmin start-telemetry-enrichment --region us-east-1
Enriched metrics carry their resource attributes as UTF-8 labels, such as
@aws.region and @resource.cloud.resource_id.
Cost Example
CloudWatch bills PromQL queries at $0.01 per million samples scanned. The figures below are an illustration, not a measurement; check the savings against your own bill.
Take a dashboard of 20 panels, each summing a rate over 50 series of one-minute data, showing the last 6 hours and refreshing every 60 seconds:
- Without Trickster: each refresh scans about 6 hours plus a 5-minute lookback per series: 50 × 365 ≈ 18,000 samples per panel, or 365,000 per refresh. That is about 526 million samples a day, or about $5.26 a day ($158 per 30 days) for one screen left open.
- With Trickster: after the first load, each refresh fetches only the new step and the 2-minute volatile window, plus the lookback: about 8 samples per series, 400 per panel, or 8,000 per refresh. That is about 11.5 million samples a day, or about $0.12 a day.
Every additional viewer of the same dashboard costs that much again without Trickster, and close to nothing with it, since concurrent requests for the same query are collapsed and later ones are served from the cache.
Live Tests
The CloudWatch tests in pkg/backends/prometheus/cloudwatch_live_test.go
run against a real account and are skipped by default. They need a metric
that is visible to PromQL in the account:
TRICKSTER_AWS_TEST=1 TRICKSTER_AWS_PROFILE=default TRICKSTER_CW_METRIC=ConsumedReadCapacityUnits go test ./pkg/backends/prometheus/ -run Live -v -count=1
Notes
- Trickster signs with its own credentials; a client’s signature cannot be
passed through, because Trickster rewrites requests (delta fetches change
startandend) and AWS ties each signature to the host it was made for. A backend with asigv4block therefore drops the SigV4 fields a client sends (Authorizationwhen it holds a SigV4 signature,X-Amz-Date,X-Amz-Security-TokenandX-Amz-Content-Sha256) before caching or forwarding the request. OtherAuthorizationvalues, such as basic auth for Trickster’s own authenticators, are kept. When pointing Grafana at Trickster, turn off the data source’s SigV4 auth; it is ignored either way. - Anyone who can reach a
sigv4backend reads with Trickster’s IAM identity, within the paths the backend serves. Give that identity only the read permissions listed above, and limit who can reach Trickster with network policy or one of Trickster’s authenticators. Serve different accounts or roles from separate backends, each with its ownrole_arn; cache entries are never shared between backends. - SigV4 signs a hash of the request body, so Trickster buffers a request
body in order to sign it. This applies only to backends with a
sigv4block. - SigV4 is not supported for the ClickHouse native protocol, which is not HTTP; configuring both fails at startup.
IAM for Autodiscovery
The aws discovery provider needs read-only permission for whichever
aws.service it is configured with.
aws.service | required IAM actions |
|---|---|
ec2 | ec2:DescribeInstances |
ecs | ecs:ListTasks, ecs:DescribeTasks |
A minimal policy for service: ec2:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "ec2:DescribeInstances",
"Resource": "*"
}]
}
ec2:DescribeInstances does not support resource-level permissions, so the
resource must be *; narrow the scope with a condition key if your
environment requires it. ecs:ListTasks and ecs:DescribeTasks can be
scoped to a cluster with the ecs:cluster condition key. Trickster only
ever reads.
2.5 - Origin Resolution via DNS SRV
By default, an HTTP backend dials its origin by looking up the A/AAAA records of the origin_url host and connecting to the port in the URL. With origin_resolution.mode: srv, the host is instead treated as a DNS SRV owner name. Each new connection goes to a target from the SRV answer, on the port the answer publishes.
This lets a backend reach services whose ports are known only to DNS, such as:
- ECS tasks in
bridgenetwork mode withhostPort: 0, registered in AWS Cloud Map with SRV records - Consul, Nomad or Kubernetes headless services exposed over DNS SRV
Combined with a rule and a hostname request rewriter, one backend can serve any number of SRV destinations, chosen per request.
Configuration
backends:
sites:
provider: rp
origin_url: 'http://_app._tcp.example.internal' # the port, if any, is ignored in srv mode
origin_resolution:
mode: srv # a (the default, A/AAAA lookup) | srv
resolver: '' # optional host:port of a DNS server to query directly
min_ttl: 5s # floor for cached answers
max_ttl: 60s # ceiling for cached answers
negative_ttl: 5s # how long NXDOMAIN or an empty answer is cached
tls_server_name: owner # owner (the default) | target
| setting | default | description |
|---|---|---|
mode | a | a dials the URL host’s A/AAAA addresses at the URL port, which is the behavior when origin_resolution is absent. srv resolves the URL host as an SRV owner name. |
resolver | The host:port of a DNS server to query directly. Direct queries make record TTLs visible. When empty, the Go standard library resolver is used. It reports no TTLs, so answers are cached for min_ttl. | |
min_ttl | 5s | The shortest time an answer is cached, whatever its record TTL |
max_ttl | 60s | The longest time an answer is cached, whatever its record TTL. It also bounds how long the last good answer keeps serving while lookups fail. |
negative_ttl | 5s | How long a name with nothing to dial is cached: NXDOMAIN, an empty answer, or a failed lookup with no last good answer to serve |
tls_server_name | owner | For https origins, the name the origin’s certificate is verified against: owner is the URL host, as with any other backend; target is the SRV target that was dialed |
resolver, the TTL settings and tls_server_name apply only to mode: srv; setting them with mode: a is a configuration error.
The owner name is used exactly as given. Trickster does not derive a service name such as _http._tcp from the URL scheme.
Routing Many Origins With One Backend
Without SRV resolution, a multi-tenant setup needs one backend per origin, each with its own fixed port, and every new origin is a config change and a reload. Origins whose ports are assigned dynamically can’t be expressed at all, because an A/AAAA record carries no port.
With SRV resolution, one rule, one rewriter, one router backend and one origin backend cover every origin that has SRV records. Adding an origin means publishing its DNS records; Trickster needs no config change or reload.
# DNS (example):
# alpha.origins.example.com. 30 IN SRV 1 1 31842 node-07.svc.example.com.
# _bravo._tcp.svc.example.com. 30 IN SRV 1 1 30117 node-02.svc.example.com.
# 30 IN SRV 1 1 30455 node-04.svc.example.com.
# ; aliases, so a public hostname doesn't have to equal the service name:
# bravo.origins.example.com. 60 IN CNAME _bravo._tcp.svc.example.com.
# bravo-legacy.origins.example.com. 60 IN CNAME _bravo._tcp.svc.example.com.
request_rewriters:
to-origin:
instructions:
# ${site} is the capture from the rule below. The rewritten name is
# SRV-resolved at dial time, and the port comes from the SRV answer.
- [ 'hostname', 'set', '${site}.origins.example.com' ]
rules:
site-router:
input_source: hostname
input_type: string
operation: rmatch
# keep the capture strict: it becomes the upstream hostname
operation_arg: '^(?P<site>[a-z0-9-]{1,63})\.example\.com$'
cases:
- matches: [ 'true' ]
req_rewriter_name: to-origin
next_route: origins
# anything that doesn't match the pattern
redirect_url: 'https://www.example.com/'
backends:
router:
provider: rule
rule_name: site-router
is_default: true
origins:
provider: rp
origin_url: 'http://origins.example.com' # placeholder; the rewriter sets the host per request
preserve_host: true # the origin still sees e.g. Host: alpha.example.com
path_routing_disabled: true
origin_resolution:
mode: srv
resolver: '10.0.0.2:53' # optional; a direct resolver gives real TTLs
min_ttl: 5s
max_ttl: 60s
negative_ttl: 5s
A request for alpha.example.com is rewritten to alpha.origins.example.com, which owns an SRV record, so it reaches node-07 on port 31842. A request for bravo-legacy.example.com is rewritten to bravo-legacy.origins.example.com, whose CNAME leads to the _bravo._tcp records. A host that matches the pattern but has no records gets a 502 Bad Gateway until its records are published.
Since the capture becomes a DNS name that Trickster resolves, keep the rule’s pattern strict. Every distinct name a client can produce costs a lookup, and its answer is cached.
How It Works
The transport’s dial is the only thing SRV resolution changes. The request URL keeps the SRV owner name, so URL handling, the Host header, logging, the cache key and the connection-pool key are the same as for any backend.
Target Selection
Targets are chosen as RFC 2782 describes:
- Targets in the lowest-priority tier (the smallest priority value) are tried first, ordered by weighted random selection. As RFC 2782 specifies, a target with weight
0has a small chance of being chosen first, and a tier whose weights are all0is ordered uniformly at random. - If a dial fails, the remaining targets in the tier are tried, then the next tier, all within the backend’s 10s connect timeout. Each target gets a share of the time left, which covers resolving its name and trying each of its addresses in turn, so neither an unreachable target nor an unreachable address can use it all up.
- A target of
.is ignored. An answer with no other targets means the service is not available.
Each target’s hostname is resolved to A/AAAA addresses with the same resolver. When the DNS server includes the target’s addresses in the additional section of the SRV answer, they are used without a further lookup. A target name that is an IP literal is dialed as given, and so is a URL host that a rewriter sets to an IP literal.
Caching
Answers are cached per owner name, and target addresses per target name. Each is kept for its record TTL, clamped to min_ttl and max_ttl. Concurrent dials for a name that needs a lookup share a single in-flight query.
When a cached answer expires, the dial waits up to 500ms for the refresh. If the refresh takes longer, the dial uses the last good answer while it is within max_ttl, and the refresh finishes in the background. When the lookup fails (a timeout, SERVFAIL or REFUSED), the last good answer keeps serving, retried no sooner than every min_ttl, until max_ttl has passed since it was looked up. NXDOMAIN and empty answers are answers rather than failures: they replace the last good answer at once and are cached for negative_ttl.
Each backend’s caches are bounded, so names derived from client-supplied hosts can’t grow them without limit. A config reload builds new caches, and in-flight requests finish on the old transport, as with any other reload.
Connection Pooling
max_idle_conns and max_concurrent_conns apply per SRV owner name, not per target, because the owner name is what the connection pool is keyed on. Idle connections to a target that has since left the answer are closed when they reach keep_alive_timeout.
TLS
For an https origin, the server name sent and verified is the SRV owner name (the URL host), as for any other backend. Set tls_server_name: target for services whose certificates name the target host instead. It can’t be combined with tls.server_name, which names a fixed server name for every target.
Health Checks
A backend health check probes through the same transport, so it is SRV-resolved too. Each probe reaches a single target, so it reports whether the service is reachable, not whether every target is.
Active per-target health checks, target ejection and load-balancing mechanisms are not part of SRV resolution. Use an ALB with dns_srv autodiscovery when you need them.
Validation
A backend with mode: srv fails validation when:
origin_urlis nothttp://orhttps://, or its host is an IP literalresolveris not ahost:portmin_ttlis greater thanmax_ttl, after defaults applytls_server_name: targetis combined withtls.server_name
origin_resolution is not supported on alb, rule and static backends, or with a non-HTTP protocol.
Metrics
trickster_proxy_origin_srv_lookups_totalcounts SRV lookups byresult. The results arehit,miss,stale(a failed refresh served the last good answer),negative(a cached NXDOMAIN, empty answer or failure) anderror.trickster_proxy_origin_srv_dial_attempts_totalcounts dial attempts to SRV targets by prioritytier(0is the preferred tier) andresult(successorfailure).
Both are labeled by backend_name. See metrics.
2.6 - Static File Server Backend
The Static Backend serves the contents of a local directory, rather than proxying requests to an upstream origin. Use it to host a website, a single-page application’s assets, a maintenance page, or any other set of files directly from Trickster, alongside (and with the same listeners, TLS, virtual hosting, authentication, logging and tracing as) your other Backends.
The file server is built into Trickster on the Go standard library, and provides:
GETandHEADfor files, withETagandLast-Modifiedvalidators, conditional requests (If-None-Match,If-Modified-Since,If-Match,If-Unmodified-Since), and single, multi-part andIf-Rangebyte range requests- a configurable default file (
index.html) for directory requests Content-Typedetection from a built-in table of common web types, with your own additions and overrides- a configurable
Cache-Controlheader, with overrides by file extension - a fileserver cache that holds small files in memory, kept current by filesystem change events (on Linux and Windows) and periodic revalidation, with the least recently used making room when it fills
- compression for the Backend’s
compressible_types, with each encoded rendition of a small file held in memory and reused - a not-found file, for a custom error page or a single-page application
- Prometheus metrics for the files served and the cache
- safe defaults: dotfiles are never served (other than
/.well-known/), directory listings are off, and no request can reach a file outside of the configured root
Configuring
A Static Backend sets its provider to static, and requires a static options block that names the root directory to serve.
backends:
website:
provider: static
hosts: [ www.example.com ]
static:
root: /var/www/html
Like any other Backend, this one is reachable at /website/ on its listeners, at / for requests whose Host is www.example.com, and at / for every request when it sets is_default: true.
Static Options
| Option | Default | Description |
|---|---|---|
root | (required) | Path to the directory holding the content to serve. It must exist and be a directory when the configuration is loaded. A relative path is resolved from Trickster’s working directory. |
default_file | index.html | The file served when a directory is requested (e.g., / or /docs/). It must be a plain file name that does not start with a . |
cache_control | The Cache-Control header value sent with every file. By default no Cache-Control header is sent, as with nginx and other common web servers, and a client judges how long a file stays fresh from its Last-Modified time. | |
cache_control_by_extension | A map of file extension to Cache-Control value, which takes precedence over cache_control for matching files. | |
response_headers | A map of additional headers to attach to every response. | |
mime_types | A map of file extension to Content-Type, which adds to and takes precedence over the built-in types. | |
not_found_file | A file, as a path within the root, that is served in place of a plain 404 response. See Not-Found File. | |
not_found_status | 404 | The status not_found_file is served with: 404 for an error page, or 200 for a single-page application. |
directory_listing | false | When true, a directory that has no default_file is answered with a listing of its contents. |
cache | Options for the Fileserver cache, which holds small files in memory, described below. |
File extensions are case-insensitive, and may be written with or without the leading dot.
A fuller example:
backends:
website:
provider: static
is_default: true
static:
root: /var/www/html
default_file: index.html
# html is revalidated on every use, so a deployment is visible immediately
cache_control: no-cache
cache_control_by_extension:
# fingerprinted assets never change, and can be cached for a year
.js: public, max-age=31536000, immutable
.css: public, max-age=31536000, immutable
.woff2: public, max-age=31536000, immutable
response_headers:
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
mime_types:
.md: text/markdown; charset=utf-8
not_found_file: errors/404.html
directory_listing: false
cache:
max_file_size_bytes: 1048576
max_size_bytes: 134217728
max_files: 10000
revalidation_interval: 10s
Backend Options
These general Backend options work with a Static Backend as they do with any other: hosts, any_host_routing, listener_names, is_default, path_routing_disabled, require_tls, tls, authenticator_name, cors, access_log, tracing_name, compressible_types, latency_min and latency_max.
Options that describe proxying, caching or routing to an origin have no meaning for a file server. Rather than being silently ignored, these fail configuration validation when set on a Static Backend: paths, req_rewriter_name, origin_url, rule_name, alb, prometheus, mysql, graphite, influxdb, sigv4, protocol, h2c_prior_knowledge, preserve_host, proxy_only and is_template. Likewise, a static block on a Backend of any other provider fails validation.
A Static Backend has no origin to probe, so it always reports as available on the health endpoint. It can be the target of a Rule or a member of an ALB pool.
Requiring Authentication
To require users to authenticate before any file is served, attach an Authenticator to the Backend. Requests that fail authentication are rejected before the file server is consulted, so they reveal nothing about which files exist.
authenticators:
staff:
provider: basic
users_file: /etc/trickster/htpasswd
users_file_format: htpasswd
backends:
intranet:
provider: static
authenticator_name: staff
static:
root: /var/www/intranet
Request Handling
| Request | Response |
|---|---|
| a file | 200 OK with the file |
| a directory, with a trailing slash | 200 OK with the directory’s default_file |
a directory with a trailing slash, but no default_file | 404 Not Found, or a listing when directory_listing is true |
| a directory, without a trailing slash | 301 Moved Permanently to the same path with a trailing slash |
| a file, with a trailing slash | 404 Not Found |
any path with a segment that starts with a ., other than a leading /.well-known/ | 404 Not Found |
| anything that does not exist, can’t be read, or resolves outside of the root | 404 Not Found |
OPTIONS | 204 No Content with an Allow header |
any method other than GET, HEAD or OPTIONS | 405 Method Not Allowed with an Allow header |
Every 404 Not Found in this table is answered with the not-found file when one is configured.
A missing default_file is deliberately a 404 rather than a 401 or 403: with listings off, a directory is not a resource that exists to be forbidden.
The trailing-slash redirect is relative (Location: docs/), so it is correct whether the Backend was reached by its hostname, as the default Backend, or under its /backend-name/ path.
Dotfiles
Any request path containing a segment that begins with a . is answered with a 404. This covers hidden files (/.env, /.htpasswd), everything beneath hidden directories (/.git/config), and attempts to reach them indirectly (/docs/../.env). Dotfiles are also left out of directory listings.
The one exception is /.well-known/, the location reserved by RFC 8615 for site-wide metadata such as security.txt, ACME HTTP-01 challenges and app-association files. It is exempt only as the first segment of the path (/docs/.well-known/ is refused), dotfiles within it are still refused (/.well-known/.secret), and it is still left out of directory listings. The list of exemptions is built in, and is deliberately not configurable.
Not-Found File
not_found_file names a file within the root to serve whenever the response would otherwise be a plain 404: a path that doesn’t exist, a directory with no default_file, a refused dotfile, and so on. not_found_status selects which of its two uses it is put to.
With the default status of 404, it is a custom error page. The page is sent as the body of the 404, with Cache-Control: no-cache and without the ETag, Last-Modified and Accept-Ranges headers, so that it can’t be revalidated, ranged over or reused as though it were the missing file. Any Range or conditional headers on the request are ignored.
static:
root: /var/www/html
not_found_file: errors/404.html
With a status of 200, it is the fallback for a single-page application, whose routes (/dashboard, /users/42/edit) exist only in the browser. Every unknown path is answered with the application itself, exactly as if the file had been requested by its own path: with its validators, its Cache-Control, conditional and byte range support, and its held renditions. Requests for files that do exist are unaffected.
static:
root: /var/www/app
not_found_file: index.html
not_found_status: 200
If the not-found file is itself missing, the response is a plain 404. A request using a method other than GET or HEAD is never answered with it. When directory_listing is on, a directory that can be listed is listed rather than answered with it.
Root Confinement and Symbolic Links
All file access is confined to the root by the operating system, not by path inspection alone. No request can read a file outside of the root, whether by .. traversal or through a symbolic link.
Symbolic links are followed only when they are relative links to a target inside the root. A link to anywhere outside of the root, and any link with an absolute target (even one that points inside the root), is answered with a 404.
Only regular files are served. Devices, sockets and named pipes inside the root are answered with a 404.
Content Types
A file’s Content-Type is resolved from, in order: the mime_types option, Trickster’s built-in table of common web types (HTML, CSS, JavaScript, JSON, WebAssembly, images, fonts, audio, video, archives, documents and more), and the host’s MIME database. The built-in table makes detection work the same in a minimal container image as on a full host. A file whose extension is unknown, or that has none, is identified from its first 512 bytes, and falls back to application/octet-stream.
Validators, Ranges and Compression
Every file is served with an ETag and a Last-Modified header, and matching conditional requests are answered with a 304 Not Modified.
Unless cache_control is set, no Cache-Control header is sent. A client then decides for itself how long to reuse a file before revalidating it, usually for a fraction of the time since the file was last modified, so files that have not changed in a long time are reused for longer. Where a change must be seen at once, such as the HTML entry point of an application with fingerprinted assets, set cache_control (or cache_control_by_extension) to no-cache, which has clients revalidate the file before each reuse.
The ETag is derived from the file’s modification time and size, the same two components used by nginx and other common web servers. Because it comes from file metadata, the ETag is the same whether a file is served from memory or from disk, and a HEAD or conditional request is answered from a stat of the file without reading it. Instances of Trickster serving copies of the same content issue the same ETag for a file, so long as the deployment preserves modification times (as rsync -a and tar do).
As with those servers, a file that is replaced by different content of exactly the same size and modification time keeps its ETag. If your deployment tooling pins or preserves modification times, give changed files a new name (such as a content fingerprint) or a new modification time.
Files whose Content-Type is in the Backend’s compressible_types are compressed for clients that accept it, in any of zstd, br, gzip and deflate. Those responses carry Vary: Accept-Encoding, and the weak form of the ETag (W/"..."), because the encoded bytes differ from the stored file the ETag describes. Byte ranges always address the stored file, so a 206 Partial Content response is never compressed, and carries the ETag unchanged.
How a file is compressed depends on whether the Fileserver cache can hold it. A file it can hold is compressed once, and the encoded rendition is held and reused, as described under Held Renditions. Any other file (a larger one, or any file when the cache is disabled) is compressed as it is sent, on every request.
Fileserver Cache
The Fileserver cache is configured in the cache block of the static options, and belongs to the Static Backend alone. It is unrelated to the caches Trickster uses to accelerate proxied Backends (including the memory cache provider): it is not configured in the caches section, a Static Backend has no cache_name, and nothing it holds is reachable through the cache purge endpoints.
Small files are read into memory the first time their content is needed, and served from memory afterward without touching the disk. Larger files are streamed from disk on every request.
Only a response that sends the whole file loads it. A HEAD request, a conditional request answered with a 304, and a byte range request are all served from disk without reading the file into memory, so they can’t be used to fill the cache. When many requests arrive at once for a file that is not yet held, it is read once and shared between them.
| Option | Default | Description |
|---|---|---|
disabled | false | When true, every request is served from disk. |
max_file_size_bytes | 1048576 (1 MiB) | The largest file that is held in memory. |
max_size_bytes | 134217728 (128 MiB) | The most memory the cache will use. Each held file counts as its size plus a fixed 1 KiB allowance for its bookkeeping, so empty and tiny files are bounded too. |
max_files | 10000 | The most objects that are held, which also bounds the number of directories that are watched to twice this number. A file as stored, and each held rendition of it, is an object. |
revalidation_interval | 10s | How often held files are compared to the disk, as a backstop to change events, and the only means of change detection on macOS and the BSDs. |
Capacity is reserved before a file is read, so the limits hold even while many different files are being loaded at once. Once either limit is reached, the least recently used objects make room for new ones.
Eviction
When a file needs room that the cache doesn’t have, by either limit, the least recently used objects are evicted to make it. Recency is tracked without slowing requests down: serving a held file only marks it as used, and eviction passes over each marked object once, clearing its mark, before it will remove it. An object is therefore evicted only if it has gone unread since eviction last reached it.
Room is only ever made for a file that is then held. Eviction first works out what would have to go, and goes ahead only if that is enough; a file that there isn’t room for is served from disk, and costs the cache nothing it holds. A request for a file that is never asked for again therefore can’t flush content that is in use.
The work a request will do to find room is bounded, however large the cache is: it passes over a limited number of recently used objects before giving up. Those it passed are no longer marked as recently used, so if requests for room keep coming, objects that aren’t being read in between give way to them, while objects in constant use never do. The objects a request does evict are in proportion to the size of the file they make way for. No single file is admitted at the cost of more than 128 objects. With the default limits that is rarely a constraint, as a full cache is usually full by its count of objects, and one eviction then makes room for a file of any size; it matters to a cache with a very high max_files that is full of very small files, in which a large file is served from disk instead.
A file is never evicted to make room for a rendition of itself, as a rendition is only held alongside the file it was made from. Where there is room for only one of them, it is the file that is held, and its renditions are made from memory for each request. Room held by files that are still being read into memory can’t be evicted either; in the unlikely event that those account for the whole cache, a new file is served from disk.
Held Renditions
A held file of a compressible type is also held in the encodings clients ask for, each as an object of its own beside the file as stored, so that a file is compressed once rather than on every request.
A request’s Accept-Encoding header is read for the encodings Trickster supports (zstd, br, gzip and deflate), most preferred first: by the weight (q) the client gave each, and among encodings of equal weight, or when no weights are given (as by browsers), by Trickster’s own preference in the order just listed. An encoding takes the weight of its own entry, or else that of the wildcard (*) if there is one. Encodings the client refused (q=0) are left out, and so are any it weighted below identity, since the file as stored is always there to be sent instead. Then:
- If the file is held in any of those encodings, the most preferred of the renditions that are held is served.
- Otherwise, if the file is held as stored, it is encoded from memory in the client’s most preferred encoding, and the new rendition is held.
- Otherwise, the file is read from disk and encoded in the client’s most preferred encoding, and both the file as stored and the new rendition are held. A rendition that is held in some other encoding is never decoded to stand in for the file.
A new rendition is streamed to the client as it is encoded, and what was sent is kept to be held, so the first response is not delayed by the rendition being made for later ones. That first response is sent without a Content-Length; once held, a rendition is sent with its own.
Renditions count toward max_size_bytes and max_files like any other object, are evicted independently of one another, and are all dropped together when their file changes on disk. A rendition is only made for a response that sends the whole file: not for a HEAD, a byte range, or a conditional request (which may turn out to need no body; if it does need one, it is compressed as it is sent). Files smaller than 512 bytes are always sent as stored, as an encoding’s own framing outweighs what it would save. When an encoding turns out not to make a file smaller, that encoding isn’t tried for the file again until it changes; the other encodings are unaffected, and a client that prefers the unhelpful one is given the next it accepts.
When nothing a client accepts is left (for example identity;q=0 from a client that accepts no supported encoding), the file is sent as stored rather than refused with a 406, as other web servers do.
Serving and the Cache
Holding a file never delays serving it. Held files are read without locking. Everything that changes the cache (holding a file or a rendition, or releasing room that turned out not to be needed) is done after, and apart from, the response that led to it. The one thing a request asks of the cache before it responds is to reserve room, which is what keeps memory within its limits however many files are being read at once; if the cache is busy at that moment, the request doesn’t wait, and is served without being held. The file is then held by a later request for it.
Change Detection
Trickster watches the directories of the files it holds. A directory that a change leaves with nothing held stays watched for one to two revalidation_intervals, since the changed file is usually requested again soon; one emptied by eviction is let go of at once. Letting go of a watch is done in the background, so evicting files never makes a request wait on the operating system. When a file is modified, replaced, renamed or removed, it is dropped from memory (typically within milliseconds) and the next request reads it from disk again. Renaming or removing a directory drops every held file beneath it. A changed file is never updated in place, and a read that overlaps a change is discarded rather than stored, so a request is never served a mix of old and new content from memory. A change touches only what it affects: the one file, or the files beneath the one directory. Every other held file stays in memory, and files that are being read into memory at that moment are unaffected, so a busy deployment does not push the rest of the site back to disk.
Every revalidation_interval, each held file’s size, modification time and identity are compared to the disk, which catches changes that raise no event: a filesystem without change notifications (such as some network and container bind mounts), a changed symbolic link target, or a replaced root. On such filesystems, revalidation_interval is the longest a stale file can be served; lower it, or disable the fileserver cache, if that matters for your content.
On macOS and the BSDs, the fileserver cache doesn’t use change events at all, and relies on revalidation alone, so a change can take up to revalidation_interval to be served. The event library’s kqueue support, which those platforms use, can close a file descriptor twice as a watch is removed, and the second close can land on a file that a request has open, so that another file’s content is served in its place (fsnotify#788). Change events will be enabled on these platforms once a fixed release is available. For local development on macOS, a short revalidation_interval (e.g., 1s) keeps edits visible quickly.
On every platform, a file that is found not to be the one at its path as it is opened is opened again, and answered with a 503 if it is still not; and a response sent from disk is broken off, rather than completed, if its file descriptor no longer refers to that file once the body is sent.
Atomic Deployments
Deploying by swapping a symbolic link is supported. When root is a link (e.g., /var/www/current → /var/www/releases/42) and the link is repointed, Trickster notices at the next revalidation_interval, switches to the new directory, and drops every held file. Requests in flight during the switch complete against the previous release.
Directory Listings
With directory_listing: true, a request for a directory that has no default_file is answered with an HTML page linking to the directory’s files and subdirectories, with their sizes and modification times. Dotfiles, links that can’t be followed, and anything that is not a regular file or directory are left out. A directory that does have a default_file is always answered with that file. Listings are generated per request, and are sent with Cache-Control: no-cache.
3 - Listeners & TLS
3.1 - TLS Support
Trickster supports TLS on both the frontend server and backend clients.
Basics
To enable the TLS server, specify the tls_port, and optionally, the tls_address in one of the listeners of your config file. For example:
listeners:
# default is built-in and is the default listener for backends
# it uses these values by default:
default:
port: 8480
tls_port: 8483
# listen on all interfaces
tls_address: ''
Note, Trickster will only start listening on a TLS port if at least one origin mapped to the named listener has a valid certificate and key configured, obtains its certificates through ACME, or the listener sets tls_runtime_certs.
Instead of certificate files, a backend can have Trickster obtain and renew its certificates from an ACME certificate authority such as Let’s Encrypt. See Automatic Certificates (ACME).
Each origin section of a Trickster config file can be augmented with the optional tls section to modify TLS behavior for front-end and back-end requests. For example:
backends:
example: # example backend
tls: # TLS settings for example backend
# frontend configs
full_chain_cert_path: '/path/to/my/cert.pem'
private_key_path: '/path/to/my/key.pem'
# backend configs
insecure_skip_verify: true
certificate_authority_paths: [ '/path/to/ca1.pem', '/path/to/ca2.pem' ]
certificate_authority_pem: |
-----BEGIN CERTIFICATE-----
...an inline CA bundle, for material that arrives as configuration...
-----END CERTIFICATE-----
server_name: origin.internal.example.com
exclude_system_roots: true
client_cert_path: '/path/to/client/cert.pem'
client_key_path: '/path/to/client/key.pem'
Server Configs - used when responding to clients
Each backend contributes up to 1 certificate and key pair, as configured in the TLS section of the backend config (demonstrated above). A listener serves the certificates of all backends mapped to it, selecting per-handshake by SNI (see Certificate Selection (SNI) below), so a single TLS listener can serve many hostnames.
If the path to any configured Certificate or Key file is unreachable or unparsable, Trickster will exit upon startup with an error providing reasonable context.
You may use the same TLS certificate and key for multiple backends, depending upon how your Trickster configurations are laid out. Any certificates configured by Trickster must match the hostname header of the inbound http request (exactly, or by wildcard interpolation), or clients will likely reject the certificate for security issues.
Certificate Selection (SNI)
When a listener has multiple certificates, Trickster selects the certificate for each TLS handshake in this order:
- Exact match of the client’s SNI hostname against a certificate’s Subject Alternative Names
- Wildcard match (e.g. a
*.example.comcertificate forfoo.example.com) - A linear compatibility scan (covers clients that send no SNI, and IP SANs)
- The listener’s first certificate, if nothing else matches
The exact and wildcard lookups are index-based, so per-handshake selection cost is independent of the number of certificates a listener serves.
Automatic Certificate Rotation Detection
Trickster automatically detects when a serving certificate is renewed in place on disk — same file paths, config untouched, as performed by tools like certbot or cert-manager — and hot-swaps the renewed certificate into the live listener. This is entirely independent of configuration reloads: no manual reload, auto_reload_interval, or config change is required.
- Detection is hybrid: filesystem events (via fsnotify) trigger a near-immediate check where the platform and filesystem support them, and a timer-based poll runs regardless — both as the fallback for deployments where filesystem events don’t work (e.g. some network and FUSE filesystems) and as a self-healing backstop for missed events. If event watching is unavailable, detection silently degrades to poll-only.
- Every check compares file contents (not modification times or event payloads), so detection works across platforms and through the atomic symlink swaps used by Kubernetes Secret and projected volumes.
- The certificate, key (and any associated CA bundle) files are watched and validated as one unit: if a poll observes a mid-rotation partial state (e.g. the cert file updated before the key file), the mismatched pair is never served; the last-good pair keeps serving and the change is retried on the next poll.
- Read errors and invalid content never disable detection: the watcher keeps retrying, logs a warning after sustained failures, and the last-good certificate keeps serving. Deleting the files is treated as a persistent failure, not as certificate removal.
- Startup behavior is unchanged: an unreadable or invalid configured certificate is still a fatal startup error. Only post-startup source failures are tolerated.
Rotation detection is on by default and is configured per listener. tls_watch_interval sets the fallback/backstop poll cadence (default 30s); filesystem events, where available, apply rotations within moments regardless of the interval. Setting the interval to 0 disables rotation detection entirely (events included):
listeners:
default:
tls_port: 8483
# backstop poll interval for the cert/key files of mapped backends
# (default 30s); filesystem events accelerate detection where supported.
# set to 0 to disable automatic rotation detection.
tls_watch_interval: 30s
Note: automatic rotation detection currently applies to HTTP(S) listeners. Native-protocol listeners (e.g. mysql) pick up rotated certificates on config reload.
Hot Swap and No-Close Semantics
Certificate swaps — whether from a config reload or automatic rotation detection — never restart, drain, or rebind the listener:
- The certificate is consulted only at handshake time, so established connections (including keep-alive connections and in-flight requests) are untouched by a swap; they continue on the certificate they were handshaken with until they close naturally.
- Only new handshakes see the new certificate.
Runtime Certificates
A listener normally serves TLS only when a mapped backend provides a certificate and key file. Setting tls_runtime_certs: true on a listener serves its tls_port even when no backend provides a file pair, so certificates can be supplied to the running process instead, for example by the Kubernetes Gateway/Ingress controller from TLS Secrets:
listeners:
default:
port: 8480
tls_port: 8483
tls_runtime_certs: true
Such a listener starts with an empty certificate store and fails handshakes until the first certificate is supplied. Certificates supplied at runtime are validated with the same pair-coherence checks as files, participate in SNI selection alongside any file-sourced certificates on the same listener, appear in the certificate inventory with source memory, and survive configuration reloads and file rotations: a reload replaces only the config-sourced certificates, and a rotation replaces only the file-sourced ones. tls_runtime_certs is valid only on http and moq listeners; on a moq listener it serves QUIC handshakes with the supplied certificates in place of a backend’s file pair.
Certificate Inventory (mgmt)
The mgmt listener exposes a read-only, per-listener certificate inventory at /trickster/certificates (configurable via mgmt.certificates_handler_path). Each entry reports the certificate’s id, source kind (file, memory, acme or config), common name, subject alternative names, validity window and last-load time. The inventory never includes key material.
Observability
Certificate rotation and inventory are covered by the following metrics: trickster_tls_certificate_expiration_time_seconds, trickster_tls_certificate_last_load_time_seconds, trickster_tls_certificate_swaps_total, trickster_tls_certificate_validation_failures_total, trickster_tls_watcher_errors_total, and trickster_tls_certificate_store_size. See metrics.md and the example alerting rules in examples/alerting for cert-expiry, sustained-validation-failure and watcher-staleness alerts.
Client Configs - used when proxying to an origin
Each backend’s TLS configuration can also configure the https client used for making requests against the origin as demonstrated above.
insecure_skip_verify will instruct the http client to ignore hostname verification issues with the upstream origin’s certificate, and process the request anyway. This is analogous to -k | --insecure in curl.
certificate_authority_paths will provide the http client with a list of certificate authorities (used in addition to any OS-provided root CA’s) to use when determining the trust of an upstream origin’s TLS certificate. In all cases, the Root CA’s installed to the operating system on which Trickster is running are used for trust by the client.
certificate_authority_pem adds certificate authorities the same way, from a PEM bundle written inline in the configuration rather than read from a file, for material that arrives as configuration (a Kubernetes BackendTLSPolicy’s CA bundle, for instance). A bundle holding no parsable certificate fails validation.
exclude_system_roots: true makes the configured certificate authorities (certificate_authority_paths and certificate_authority_pem) the only ones trusted for the origin, rather than additions to the operating system’s. Use it when the origin must present a certificate from one specific private authority and a certificate from any public authority would be an error. It requires at least one authority to be configured, since excluding the system roots with none would trust nothing.
server_name sets the hostname sent as SNI and verified against the origin’s certificate, when it differs from the host in origin_url — an origin reached by IP or by an internal alias whose certificate names something else. It changes nothing about where the connection goes or which Host header is sent.
To us Mutual Authentication with an upstream origin server, configure Trickster with Client Certificates using client_cert_path and client_key_path parameters, as shown above. You will likely need to also configure a custom CA in certificate_authority_paths to represent your certificate signer, unless it has been added to the underlying Operating System’s CA list.
3.2 - HTTP/3
Trickster can serve any HTTP listener’s routes over HTTP/3 (RFC 9114) in addition to HTTP/1.1 and HTTP/2, using QUIC over UDP.
HTTP/3 helps most where Trickster acts as an edge cache: loss recovery without head-of-line blocking benefits parallel byte-range fetches, and connection migration keeps a client’s session alive as it changes networks.
Enabling
HTTP/3 attaches to an existing http listener that already serves TLS. QUIC has
no cleartext mode, so a listener without a working TLS endpoint cannot serve it.
listeners:
default:
port: 8480
tls_port: 8483
http3:
enabled: true
That is the whole configuration for the common case. The UDP endpoint defaults to the same address and port as the TLS endpoint, so clients find HTTP/3 where they already found HTTPS.
Options
| Field | Default | Description |
|---|---|---|
enabled | false | Serves this listener’s routes over HTTP/3. |
address | the listener’s tls_address | IP for the UDP socket. |
port | the listener’s tls_port | UDP port to bind. |
advertised_port | port | Port published in Alt-Svc. Set this when a load balancer or NAT presents a different port to clients than the one bound here. |
listeners:
default:
tls_address: 0.0.0.0
tls_port: 8483
http3:
enabled: true
port: 8443 # bind UDP/8443
advertised_port: 443 # but tell clients to use 443
TLS certificates come from the backends mapped to the listener, exactly as they do for the TLS/TCP endpoint — see TLS. HTTP/3 uses the same certificates; no separate configuration is needed.
How clients discover HTTP/3
Browsers and most clients do not attempt HTTP/3 first. They connect over
TCP and look for an Alt-Svc response header naming an HTTP/3 endpoint:
Alt-Svc: h3=":8443"; ma=2592000
Trickster adds this header to every response from the listener’s TLS/TCP endpoint while HTTP/3 is enabled, so adoption is automatic and gradual. The TCP endpoint keeps serving HTTP/1.1 and HTTP/2 for clients that do not upgrade.
Protocol-upgrade requests (Connection: Upgrade) are not advertised, since such
a request hijacks its connection before response headers matter.
WebTransport
While a moq backend uses the listener, its HTTP/3 endpoint also
serves WebTransport (draft-ietf-webtrans-http3), which browsers use to reach a
MoQ relay. Each WebTransport session arrives as an Extended CONNECT request and
is routed like any other request, so the backend’s paths, hosts,
authenticator_name and ACLs apply to it. A reload that binds or unbinds a
moq backend changes how new connections are served, without restarting the
endpoint or touching the connections already open.
A WebTransport connection cannot carry an HTTP/3 GOAWAY, because WebTransport owns its control stream, so draining such an endpoint closes each connection once it carries no request. A WebTransport session counts as a request for its whole life, and whatever is still open when the drain timeout ends is closed.
Sharing the UDP port
The HTTP/3 endpoint may share its UDP address and port with a protocol: moq
listener (moq.md): each QUIC connection is handed to the listener
whose protocol it negotiated (h3 or a moqt-NN draft), and each listener
keeps its own certificates and routes. A protocol: udp listener cannot share
the port.
Limitations
- No protocol upgrades. HTTP/3 has no equivalent of the HTTP/1.1
101 Switching Protocolshandshake; RFC 9114 4.2 makes connection-specific headers malformed. WebSockets over HTTP/3 use Extended CONNECT (RFC 9220), which Trickster serves only for WebTransport sessions to amoqbackend. Send WebSocket traffic over the TCP endpoint, which supports it fully. - Inbound only. Trickster speaks HTTP/1.1 or HTTP/2 to origins regardless of
the protocol a client used to reach it, except that a
moqbackend reaches a WebTransport origin over HTTP/3. This matches Traefik; among major proxies only Envoy implements HTTP/3 to upstreams, and it needsAlt-Svccaching plus TCP fallback to do so safely. - UDP must reach the listener. Some networks block or throttle UDP/443. Clients that cannot establish QUIC silently keep using TCP, so this degrades rather than fails.
Operating notes
UDP receive buffer
QUIC moves far more data through a single socket than typical UDP workloads, and the kernel default receive buffer is often too small. On Linux, raise it:
sysctl -w net.core.rmem_max=7500000
sysctl -w net.core.wmem_max=7500000
Without this, a warning is logged at startup. In containers that cannot change
sysctls the warning is unactionable and can be silenced with
QUIC_GO_DISABLE_RECEIVE_BUFFER_WARNING=1.
Reloads
The HTTP/3 listener participates in the same reload and drain lifecycle as every other listener. Route changes are hot-swapped without rebinding the socket; changing the bound or advertised port restarts just that endpoint. A drained or removed endpoint releases its UDP port, so a later reload can bind it again; before 2.2, the port stayed bound until the process exited.
Trying it locally
Most systems ship a curl built without HTTP/3 (curl --version | grep HTTP3
to check). Trickster includes a small client so this is not a prerequisite:
make dev-certs # self-signed cert for localhost
go run ./hack/h3-client -url https://127.0.0.1:8483/ -ca docs/developer/environment/certs/trickster-dev.crt
The client prints the negotiated protocol, so HTTP/3.0 200 OK confirms the
request was served over QUIC.
3.3 - Apache Arrow Flight SQL Listeners
Trickster can serve Apache Arrow Flight SQL — the gRPC-based query protocol spoken by ADBC drivers, Grafana’s SQL-mode datasources, and the Arrow-native client SDKs — as a caching proxy in front of a Flight SQL-capable origin. The protocol support is vendor-neutral; each backend provider that adopts it supplies only its origin-specific wiring. InfluxDB 3.x is the first supported provider (see InfluxDB Support for its specifics and examples).
Enabling a Flight SQL listener
Define a listener with the flight-sql protocol and map exactly one
compatible backend to it through the backend’s listener_names:
listeners:
my-flight:
protocol: flight-sql # Apache Arrow Flight SQL over gRPC
port: 8485
backends:
influx3:
provider: influxdb
origin_url: 'http://influxdb3:8181/'
listener_names: [ default, my-flight ]
Flight SQL listeners share Trickster’s standard listener lifecycle:
connection limits (connections_limit), graceful drain on SIGTERM, and
config reload (SIGHUP) — a reload with an unchanged backend configuration
keeps serving on the existing socket, while a changed configuration drains
the old server (closing its upstream connection) and rebinds.
TLS
The listener serves TLS when the mapped backend’s tls block presents a
certificate and key, with in-place certificate rotation on config reload;
without one it serves plaintext gRPC. The upstream dial is TLS-capable via
provider options (for InfluxDB, influxdb.flight_upstream_tls).
Caching model
Every cache entry is scoped to the backend name plus a per-tenant namespace
derived from the request metadata the provider declares (for InfluxDB:
database, bucket-name, and a hash of authorization), mirroring the
scope the upstream itself grants — tenants never share entries, and
credentials never appear in cache keys.
Statement queries are served through three tiers:
- Delta proxy cache — statements the provider’s SQL analyzer classifies
as delta-cacheable are cached by time extent: repeat and overlapping
queries fetch only missing sub-ranges from the origin, and responses are
rebuilt into Arrow record batches conforming to the response’s original
schema (types, column order, nullability, and schema metadata are
preserved). A statement’s
ORDER BYis carried through reconstruction, so a delta hit returns rows in the order the statement asked for; ordering terms that do not resolve to a select-list output fall to the next tier. Responses whose schemas the delta model cannot represent fall automatically to the next tier as well. The partial buckets at a range’s edges are never delta-cached; the backend’s step alignment mode decides what the response shows for them:drop, which leaves them out, unless configured otherwise or chosen in a comment such as/* trickster-step-align:partial_end */. Thepartialmodes fetch each partial bucket as a statement of its own through the object cache, andoffanswers every statement from the object cache with its range as sent. - Object cache — everything else cacheable is stored as the verbatim Arrow IPC byte stream and returned byte-identically, with a short, provider-configured lifetime. Metadata RPCs use this tier.
- Proxy — nondeterministic and non-SELECT statements are never cached.
Serving-cost characteristics of the tiers are recorded in Flight SQL Cache Tier Benchmarks.
Supported RPC surface
- Statement execution (
GetFlightInfoStatement/DoGetStatement). - Result-set schema requests (
GetSchemaStatement/GetSchemaPreparedStatement), which ADBC drivers probe on connect. When the origin itself does not implement its schema RPC (InfluxDB 3 Core among them), the schema is derived by executing the statement through the object cache, so the probe’s cost folds into the execution the client performs next. - Catalog metadata (
GetTables,GetCatalogs,GetDbSchemas,GetTableTypes,GetSqlInfo), proxied and cached per tenant. - Type and key discovery (
GetXdbcTypeInfo,GetPrimaryKeys,GetImportedKeys,GetExportedKeys,GetCrossReference), which JDBC- and ODBC-shaped clients probe during metadata discovery, likewise proxied and cached per tenant and per request shape. - The prepared-statement lifecycle (create, bind, execute, close). A parameterless prepared statement is served through the same three statement tiers — sharing cache entries with ad-hoc executions of the same text — while parameterized executions are cached whole, keyed by the bound parameter hash. Statements abandoned by disconnected clients are closed upstream after 15 minutes of inactivity.
A FlightInfo whose result the origin partitions across several endpoints is
consumed in full: every endpoint’s ticket is resolved in order, and an
endpoint that advertises its own location is retrieved from that location
only when its exact host:port authority is explicitly allowlisted by the
provider. This opt-in is required because request metadata, including
credentials, is forwarded to the alternate host. A TLS primary is never
downgraded to a plaintext alternate. Alternate connection caches are bounded
(16 clients by default). Partitions that disagree on schema are refused rather
than served partially.
Writes/updates/ingest, transactions, savepoints, Substrait plans, query
cancellation, endpoint renewal, and session options return gRPC
Unimplemented — they are outside a read-through cache’s contract. Clients
requiring them should connect to the origin directly. Prepared-statement
bindings are accepted as a single record batch; a client streaming several
batches for one binding receives Unimplemented rather than a silently
truncated binding.
Response size bound
Responses are buffered whole so they can be cached, so a single very large
query would otherwise be able to exhaust the process heap. Each upstream
response is bounded at 128MiB by default; a response exceeding the bound is
refused with gRPC ResourceExhausted rather than being buffered. Providers
expose the bound as a backend option (for InfluxDB,
influxdb.flight_max_response_bytes; a negative value removes it). A separate
aggregate budget bounds response bytes concurrently assembled or streamed
(512MiB by default; influxdb.flight_max_buffered_bytes for InfluxDB).
Metrics
Statement executions record cache outcomes to the native SQL metrics under
the flightsql dialect label: trickster_sql_query_cache_total
(cache_mode x cache_status), the standard proxy request/points/duration
metrics, trickster_sql_query_rewrite_failures_total for extent-rendering
failures, and cache-error events on the shared cache metrics. See
Metrics.
Architecture (for provider implementers)
The protocol lives in vendor-neutral packages:
pkg/proxy/flightsql— the gRPC protocol server, caching statement/ metadata/prepared handling, and the upstream client. A provider supplies: the upstream address and TLS posture;ForwardMetadataKeys(which inbound gRPC metadata flows to the origin); aKeyScopercovering every forwarded key that affects results; and optionally aDeltaConfigcarrying its SQL dialect analyzer to enable the delta tier.pkg/proxy/engines/nativedelta— the transport-agnostic delta-proxy-cache engine (shared with the MySQL native listener), driven by the frozenpkg/parsing/sqlanalyzerdialect contract.pkg/timeseries/dataset/arrow— dataset ⇄ Arrow record batch conversion used by the delta tier.
pkg/backends/influxdb/native_listener.go is the reference wiring: a
provider integrates by implementing the native.Adapter interface
(declaring Protocol() == "flight-sql" and its own BackendProvider()) and
constructing the flightsql server from its backend options.
3.4 - Automatic Certificates (ACME)
Trickster can obtain and renew its own serving certificates from any certificate authority that speaks ACME (RFC 8555), such as Let’s Encrypt, ZeroSSL, Google Trust Services or a private step-ca. A backend opts in under its tls section, and Trickster issues a certificate for each of its hostnames, serves it from the listener’s certificate store, and renews it in the background before it expires.
ACME-managed certificates are served alongside certificates loaded from files and certificates supplied at runtime, and they use the same SNI selection, hot-swap and inventory machinery described in TLS Support. Issuance, renewal and storage all happen off the handshake path, so a certificate obtained through ACME costs nothing more per handshake than one loaded from a file.
Quick Start
listeners:
default:
port: 80
tls_port: 443
acme:
storage:
path: /var/lib/trickster/acme
issuers:
letsencrypt:
email: ops@example.com
agree_to_terms: true
backends:
site:
provider: reverseproxycache
origin_url: http://origin.internal:8080
hosts: [www.example.com, example.com]
tls:
acme:
issuer: letsencrypt
With this configuration, Trickster requests certificates for www.example.com and example.com from Let’s Encrypt, answers the CA’s validation on ports 80 and 443, and serves the certificates on the default listener as soon as they are issued.
The CA validates a name by connecting to it on port 80 (http-01) or port 443 (tls-alpn-01). When Trickster listens on other ports, such as the defaults of 8480 and 8483, forward 80 and 443 to them, or use the dns-01 challenge.
Try a new configuration against a staging directory first, such as https://acme-staging-v02.api.letsencrypt.org/directory, since production CAs rate-limit failed and duplicate orders.
Configuration
The acme Section
| Setting | Description | Default |
|---|---|---|
wait_on_startup | Holds readiness at startup until missing certificates are issued, or this long at most. See Startup Wait. | 0 (no wait) |
storage.provider | filesystem or redis. See Storage and Clusters. | filesystem |
storage.path | The storage directory, required for filesystem storage. | none |
storage.redis.cache_name | Borrows the connection settings of a named redis cache. | none |
storage.redis.connection | Redis connection settings, with the same fields as a cache’s redis section, when cache_name is not used. | none |
storage.redis.key_prefix | Namespaces every ACME key. | trickster:acme: |
storage.redis.lock_ttl | The lease of a distributed lock, refreshed while it is held. At least 5s. | 1m |
issuers | Named issuers, which backends reference. At least one is required. | none |
on_demand | Enables issuance during the first handshake for names no backend lists. See On-Demand TLS. | none |
Each issuer accepts:
| Setting | Description | Default |
|---|---|---|
directory_url | The CA’s ACME directory. | Let’s Encrypt production |
email | The account contact the CA sends expiry and policy notices to. | none |
agree_to_terms | Accepts the CA’s subscriber agreement. It must be true; Trickster never accepts terms implicitly. | false |
challenges | The challenge types the issuer may use: http-01, tls-alpn-01, dns-01. | [http-01, tls-alpn-01], or [dns-01] with a dns_provider |
key_type | ecdsa-p256, ecdsa-p384, rsa-2048, rsa-4096 or ed25519. | ecdsa-p256 |
profile | A certificate profile the CA advertises in its directory, such as Let’s Encrypt’s shortlived. | none |
external_account_binding | key_id, and hmac_key or hmac_key_file, for CAs that require an existing account, such as ZeroSSL and Google Trust Services. | none |
trusted_ca_paths | PEM files trusted, in addition to the system roots, for the CA’s own HTTPS endpoint, as a private CA needs. | none |
dns_provider | The DNS API that answers dns-01 challenges. See DNS-01. | none |
A Backend’s tls.acme
| Setting | Description | Default |
|---|---|---|
issuer | The issuer the backend’s certificates come from. | required |
domains | The names to request certificates for. | the backend’s hosts |
A backend that uses tls.acme cannot also set full_chain_cert_path or private_key_path. Its listeners must be http listeners with a tls_port. Other TLS settings, such as certificate_authority_paths for the origin, are unaffected.
Domains
Trickster requests one certificate per name. Names are lowercased, a trailing dot is dropped, and duplicates are removed. Each name must have at least two labels, and IP addresses are not supported.
- A wildcard such as
*.example.comrequires an issuer that usesdns-01. It covers exactly one label, so it matchesa.example.combut neitherexample.comnora.b.example.com. - A backend whose
hostsinclude an any-depth pattern such as**.example.commust list the names it wants certificates for intls.acme.domains, since no certificate can cover every depth. - A name belongs to exactly one issuer. Several backends may list it if they all use the same issuer.
Challenges
HTTP-01
The CA requests http://<name>/.well-known/acme-challenge/<token> on port 80. Trickster answers on the plaintext port of each listener that serves an http-01 domain, ahead of every route and middleware, so IP and geo access lists, rate limits and authenticators never block a CA’s validation. Only a pending token is answered. Any other request under /.well-known/acme-challenge/ gets an empty 404 and never reaches a backend. A listener that serves no http-01 domain does no extra work per request.
An issuer that allows http-01 but has no listener with a plaintext port cannot use it, and validation logs a warning.
TLS-ALPN-01
The CA connects to port 443 offering only the acme-tls/1 protocol (RFC 8737). Trickster answers those handshakes with the validation certificate, and every other handshake costs a single length comparison. TLS-ALPN-01 runs over TCP, so the HTTP/3 endpoint never answers it.
DNS-01
DNS-01 publishes a TXT record through a DNS provider’s API. It is the only challenge that can issue wildcards, and it works for names the CA cannot reach. An issuer using dns-01 uses it exclusively.
acme:
issuers:
letsencrypt-dns:
email: ops@example.com
agree_to_terms: true
dns_provider:
provider: cloudflare
cloudflare:
api_token_file: /etc/trickster/secrets/cloudflare-token
| Provider | Settings |
|---|---|
cloudflare | api_token or api_token_file (Zone.DNS:Edit), and optionally zone_token or zone_token_file |
route53 | Optionally region, profile, hosted_zone_id, and access_key_id with secret_access_key or secret_access_key_file; unset credentials use the AWS default credential chain |
rfc2136 | server (host:port), key_name, key_alg (such as hmac-sha256), and key or key_file; works with BIND, PowerDNS, Knot and other servers that accept dynamic updates |
Every provider also accepts propagation_delay, a wait before checking that the record is visible; propagation_timeout, which bounds that check (default 2 minutes, and a negative value skips it); and resolvers, the DNS servers (host:port) the check queries.
Secrets set directly in the configuration are redacted from the config handler and every config dump. The _file variants keep them out of the configuration altogether.
Storage and Clusters
ACME accounts, certificates, private keys and locks are kept in storage, so a restart serves the certificates it already has instead of ordering new ones.
Filesystem storage keeps everything under storage.path, in directories created with mode 0700 and files with mode 0600. Lock files make it safe for several processes on one host. It is not safe on network filesystems with weak locking semantics.
Redis storage lets every Trickster instance that shares the keyspace share certificates. Distributed locks ensure that only one instance orders or renews a given name. Any instance can answer a challenge another instance started, so a load balancer may send the CA’s validation to any of them. Every key carries one hash tag, so the keyspace stays within a single slot on Redis Cluster.
acme:
storage:
provider: redis
redis:
cache_name: shared-redis # borrow the connection settings of a redis cache
key_prefix: "trickster:acme:"
ACME storage never shares the cache’s keyspace, even when it borrows a cache’s connection settings. Eviction would destroy account keys and certificates, so the Redis server must not evict these keys: run it with maxmemory-policy noeviction, or without maxmemory.
Startup Wait
A brand-new deployment has no certificates until the CA issues them, and handshakes for those names fail in the meantime. wait_on_startup holds readiness until they are issued:
acme:
wait_on_startup: 90s
- Listeners bind immediately, because the CA’s challenges must reach them. What waits is the readiness endpoint (
/trickster/ready), which reports503with the bodycertificates pending, so load balancers and Kubernetes probes keep traffic away. - Only names without a usable certificate in storage are waited for, and an expired stored certificate counts as missing. A routine restart therefore becomes ready immediately.
- When the duration elapses, readiness is released anyway. Trickster logs an error for each name still missing, with the CA’s last error, and issuance keeps retrying in the background. Holding readiness indefinitely would let a CA outage take every instance out of rotation, including those serving backends that don’t use ACME.
- The wait applies only at startup. Names added by a configuration reload are issued in the background and never return readiness to
503.
On-Demand TLS
On-demand TLS issues a certificate during the first handshake for a name no backend lists. This suits a CDN or a SaaS platform whose customers bring their own domains. Because it lets clients cause certificate orders, it requires at least one gate:
acme:
on_demand:
issuer: letsencrypt
listeners: [default]
ask: http://127.0.0.1:9000/allowed # GET ?domain=<name>; a 200 permits issuance
allowed_domains: ["*.customers.example.com"] # names must also match one of these
decision_ttl: 5m # how long an answer or a refusal is remembered
rate_limit: 10 # most issuances started per minute
negative_cache_size: 10000
- When both
askandallowed_domainsare set, a name must pass both. - Refused names are remembered for
decision_ttl, so repeated handshakes for an unwanted name cost a table lookup, not a request toask. - The first handshake for a permitted name waits while the certificate is issued, which typically takes a few seconds. Later handshakes are answered from the listener’s certificate store.
- On-demand issuance cannot use a
dns-01issuer, and it applies only to TLS over TCP. Handshakes on the HTTP/3 endpoint are served from the store. - A handshake for a name the store already covers never reaches the on-demand gates.
Renewals and Reloads
Certificates are renewed during the last third of their lifetime, or earlier when the CA advises it through ACME Renewal Information (RFC 9773). A failed order or renewal is retried with backoff. Every renewal is hot-swapped into the listeners without dropping established connections.
A configuration reload reconciles what is managed:
- New names are ordered in the background.
- Removed names stop being served. Their certificates are never revoked, and they remain in storage, so adding a name back is immediate.
- When an issuer’s settings change, its names keep serving their current certificates until the reconfigured issuer replaces them.
- Changing
storagerestarts certificate management against the new storage.
Management API
The mgmt listener serves the ACME handler at /trickster/acme (configurable via mgmt.acme_handler_path). It is served only on the mgmt listener, never on proxy listeners.
GET /trickster/acmelists every managed name with its issuer.POST /trickster/acme?domain=<name>starts a forced renewal in the background and responds202 Accepted. An unmanaged name gets404.
ACME-managed certificates also appear in the certificate inventory with the source acme and the id acme:<issuer>:<name>.
Observability
The TLS certificate metrics, such as trickster_tls_certificate_expiration_time_seconds, cover ACME-managed certificates as well. These metrics are specific to ACME:
trickster_acme_orders_totalandtrickster_acme_renewals_total, byissuerandresult(successorfailure)trickster_acme_challenge_requests_total, bytype(http-01ortls-alpn-01) andresult(served,unknownorerror)trickster_acme_on_demand_decisions_total, byresult(allowed,refused,refused_cached,rate_limitedorask_error)trickster_acme_startup_wait_secondsandtrickster_acme_startup_wait_timeouts_total
See metrics.md for details.
Kubernetes
ACME is not supported together with the Kubernetes controller. With the controller, certificates come from TLS Secrets, and cert-manager is the established way to issue them. A Trickster Deployment that does not run the controller can use ACME, with a persistent volume for filesystem storage, or with Redis storage when it has several replicas.
Limitations
- ACME certificates are served on
httplisteners only. Native-protocol listeners (postgres,mysql,flight-sql,clickhouse) andtlsstream listeners use certificate files. - IP-address certificates and certificate revocation are not supported.
4 - Caching
4.1 - Cache Overview
Supported Caches
There are several cache types supported by Trickster
- In-Memory (default)
- Filesystem
- bbolt
- BadgerDB
- Redis (basic, cluster, and sentinel)
The sample configuration (examples/conf/example.full.yaml) demonstrates how to select and configure a particular cache type, as well as how to configure generic cache configurations such as Retention Policy.
In-Memory
In-Memory Cache is the default type that Trickster will implement if none of the other cache types are configured. The In-Memory cache utilizes a Golang sync.Map object for caching, which ensures atomic reads/writes against the cache with no possibility of data collisions. This option is good for both development environments and most smaller dashboard deployments.
When running Trickster in a Docker container, ensure your node hosting the container has enough memory available to accommodate the cache size of your footprint, or your container may be shut down by Docker with an Out of Memory error (#137). Similarly, when orchestrating with Kubernetes, set resource allocations accordingly.
Filesystem
The Filesystem Cache is a popular option when you need more storage space than you wish to accommodate in RAM: a large dashboard setup (e.g., many different dashboards with many varying queries, Dashboard as a Service for several teams running their own Prometheus instances, etc.), or large objects like media and downloads. A Filesystem Cache configuration keeps the Trickster RAM footprint small. Trickster performance can be degraded when using the Filesystem Cache if disk i/o becomes a bottleneck.
The default Filesystem Cache path is /tmp/trickster. The sample configuration demonstrates how to specify a custom cache path. Ensure that the user account running Trickster has read/write access to the custom directory or the application will exit on startup upon testing filesystem access. All users generally have access to /tmp so there is no concern about permissions in the default case.
A local filesystem (e.g., ext4 or XFS) is recommended. Network filesystems are discouraged, as they are slow to create, rename and list the many files a cache is made of.
See Disk Caches for how the Filesystem Cache lays out, recovers and bounds what it stores.
bbolt
The BoltDB Cache is a popular key/value store, created by Ben Johnson. CoreOS’s bbolt fork is the version implemented in Trickster. A bbolt store is a filesystem-based solution that stores the entire database in a single file. Trickster, by default, creates the database at trickster.db and uses a bucket name of ’trickster’ for storing key/value data. See the example config file for details on customizing this aspect of your Trickster deployment. The same guidance about filesystem permissions described in the Filesystem Cache section above apply to a bbolt Cache.
bbolt commits each write to disk before it returns, so it is slower to store an object than the Filesystem Cache, and is as fast or faster to read one. It is a good fit for caches of many small objects, since it does not spend a file on each.
See Disk Caches for how the bbolt Cache recovers and bounds what it stores.
Disk Caches
The Filesystem and bbolt Caches keep whatever they are given, so Trickster manages the lifecycle of their objects with a Cache Index, which it holds in memory and persists to the cache.
Stored Objects
Each object is stored with a header that holds its cache key, its expiration and a checksum of its content. An object that is found to be expired, incomplete or damaged when it is read is treated as a cache miss and removed. The checksum of an object of 10 MiB or more is verified after the object is returned rather than before, so that a large read is not held for it; such an object, if damaged, is served once and then removed. The format of what is stored is private to Trickster and versioned: objects stored by a version that used another format are treated as cache misses and removed, so a disk cache can be cold after an upgrade. The release notes say when that is so.
The Filesystem Cache writes each object to a new file and renames it into place, so a reader never finds part of an object, even after a crash. Files are spread across two levels of directories under cache_path, named for the first characters of the file’s name, to keep any one directory small:
/tmp/trickster/9f/86/9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
/tmp/trickster/_meta/
A file is named for its cache key when the key is a lowercase hex digest of 32 or 64 characters, as the keys of HTTP objects are, and otherwise for a hash of the key. The _meta directory holds the Cache Index.
Serving Large Objects
An object proxied through the Reverse Proxy Cache is stored in two sections: what describes it (status, headers, caching policy) and its body. When the whole of an object is cached uncompressed, a GET that is a fresh cache hit reads the body from the disk cache as it is written to the client, and a Range request reads only the ranges that were asked for. Neither holds the object in memory, whatever its size. Objects that Trickster compresses in the cache (see compressible_types) are read whole, as are objects that must first be revalidated.
Cache Index
| Setting | Default | Description |
|---|---|---|
reap_interval | 3s | how often expired objects are removed, and a cache that is over its size is brought back within it |
flush_interval | 5s | how often changes to the index are persisted |
index_expiry | 8760h | how old a persisted index can be, and still be used at startup |
max_size_bytes | 536870912 | the size in bytes at which least-recently-accessed objects are evicted |
max_size_backoff_bytes | 16777216 | how far under max_size_bytes an eviction takes the cache |
max_size_objects | 0 | the count of objects at which least-recently-accessed objects are evicted; 0 for no limit |
max_size_backoff_objects | 100 | how far under max_size_objects an eviction takes the cache |
scan_interval | 24h | how often the cache is swept; 0 to sweep only when the index cannot be trusted |
scan_batch_size | 512 | how many objects a sweep reads before it pauses |
scan_batch_pause | 50ms | how long a sweep pauses after each batch |
The Filesystem Cache also accepts min_free_bytes, the space to keep free on the filesystem that holds cache_path. When less is free, the cache evicts as it does when it is over max_size_bytes. It is supported on Unix-like systems.
Persistence. The index is persisted as a snapshot and a journal of the changes made since. Each flush_interval, the changes are appended to the journal; once the journal has grown to half the size of the snapshot, a new snapshot replaces both. Objects that expire within a minute of being stored are not persisted.
Recovery. When Trickster starts with an index that was persisted by a clean shutdown, it is used as it is. When the index is missing, expired, damaged, or was not closed by a clean shutdown, Trickster uses what it can of it and sweeps the cache in the background, while serving requests. A sweep reads the header of every object, scan_batch_size at a time, and lists the objects the index did not know of, so that nothing in the cache is orphaned. It also removes what can’t be served: objects that are expired or damaged, files left by an interrupted write, and objects stored by a version of Trickster that used another format.
Eviction. Objects are evicted in the order they were least recently accessed. In a cache of more than 1024 objects, the object to evict is the least recently accessed of 32 taken at random, which costs the same however large the cache is, and closely follows the true order. A write that takes the cache over its size has the reaper act within a second, without waiting for reap_interval.
BadgerDB
BadgerDB works similarly to bbolt, in that it is a filesystem-based key/value datastore. BadgerDB provides its own native object lifecycle management (TTL) and other additional features that distinguish it from bbolt. See the configuration for more info on using BadgerDB with Trickster.
Redis
Note: Trickster does not come with a Redis server. You must provide a pre-existing Redis endpoint for Trickster to use.
Redis is a good option for larger dashboard setups that also have heavy user traffic, where you might see degraded performance with a Filesystem Cache. This allows Trickster to scale better than a Filesystem Cache, but you will need to provide your own Redis instance at which to point your Trickster instance. The default Redis endpoint is redis:6379, and should work for most docker and kube deployments with containers or services named redis. The sample configuration demonstrates how to customize the Redis endpoint. In addition to supporting TCP endpoints, Trickster supports Unix sockets for Trickster and Redis running on the same VM or bare-metal host.
Ensure that your Redis instance is located close to your Trickster instance in order to minimize additional roundtrip latency.
In addition to basic Redis, Trickster also supports Redis Cluster and Redis Sentinel. Refer to the sample configuration for customizing the Redis client type.
Trickster supports Redis servers that use TLS encryption by setting use_tls: true in the config. Refer to the sample configuration for more info.
Purging an Item from the Cache
You can purge an item from the cache by making a call to the purge endpoint, as follows:
http://${trickster-address}:${mgmt-port}/trickster/purge/path/${backendName}/${path/to/purge}
For example, if you want to purge /api/v1/labels from backend prom1, a curl might look like:
curl http://localhost:8484/trickster/purge/path/prom1/api/v1/labels
Purging the Full Cache
Full Cache purges should not be necessary, but in the event that you wish to do so, the following steps should be followed based upon your selected Cache Type.
A future release will provide a mechanism to fully purge the cache (regardless of the underlying cache type) without stopping a running Trickster instance.
Purging In-Memory Cache
Since this cache type runs inside the virtual memory allocated to the Trickster process, bouncing the Trickster process or container will effectively purge the cache.
Purging Filesystem Cache
To completely purge a Filesystem-based Cache, you will need to:
- Docker/Kube: delete the Trickster container (or mounted volume) and run a new one
- Metal/VM: Stop the Trickster process and manually run
rm -rf /tmp/trickster(or your custom-configured directory).
Purging Redis Cache
Connect to your Redis instance and issue a FLUSH command. Note that if your Redis instance supports more applications than Trickster, a FLUSH will clear the cache for all dependent applications.
Purging bbolt Cache
Stop the Trickster process and delete the configured bbolt file.
Purging BadgerDB Cache
Stop the Trickster process and delete the configured BadgerDB path.
Cache Status
Trickster reports several cache statuses in metrics, logs, tracing, and the X-Trickster-Result response header, which are listed and described in the table below.
| Status | Description |
|---|---|
| kmiss | The requested object was not in cache and was fetched from the origin |
| rmiss | Object is in cache, but the specific data range requested (timestamps or byte ranges) was not |
| hit | The object was fully cached and served from cache to the client |
| phit | The object was cached for some of the data requested, but not all |
| nchit | The response was served from the Negative Cache |
| rhit | The object was served from cache to the client, after being revalidated for freshness against the origin |
| purge | The cache key was purged as directed by a request or response header |
| proxy-only | The request was proxied 1:1 to the origin and not cached |
| proxy-error | The upstream request needed to fulfill an associated client request returned an error |
| error | Trickster encountered a cache lookup or cache handling error |
| proxy-hit | The request joined an existing in-flight origin fetch for the same cache key |
4.2 - Trickster Caching Retention Policies
Basic HTTP Backends
Trickster will respect HTTP 1.0, 1.1 and 2.0 caching directives from both the downstream client and the upstream origin when determining object cacheability and TTL. You can override the TTL by setting a custom Cache-Control header on a per-Path Config basis.
Cache Object Evictions
If you use a Trickster-managed cache (Memory, Filesystem, bbolt), then a maximum cache size is maintained by Trickster. You can configure the maximum size in number of bytes, number of objects, or both. See the example configuration for more information.
Once the cache has reached its configured maximum size of objects or bytes, Trickster will undergo an eviction routine that removes cache objects until the size has fallen below the configured maximums. Trickster-managed caches maintain a last access time for each cache object, and utilizes a Least Recently Used (LRU) methodology when selecting objects for eviction.
Caches whose object lifetimes are not managed internally by Trickster (Redis, BadgerDB) will use their own policies and methodologies for evicting cache records.
Time Series Backends
For non-time series responses from a TSDB, Trickster will adhere to HTTP caching rules as directed by the downstream client and upstream origin.
For time series data responses, Trickster will cache as follows:
TTL Settings
TTL settings for each Backend configured in Trickster can be customized independently of each other, and separate TTL configurations are available for timeseries objects, and fast forward data. See examples/conf/example.full.yaml for more info on configuring default TTLs.
Time Series Data Retention
Separately from the TTL of a time series cache object, Trickster allows you to control the size of each timeseries object, represented as a count of maximum timestamps in the cache object, on a per origin basis. This configuration is known as the timeseries_retention_factor (TRF), and has a default of 1024. Most dashboards for most users request and display approximately 300-to-400 timestamps, so the default TRF allows users to still recall recently-displayed data from the Trickster cache for a period of time after the data has aged off of real-time views.
If you have users with a high-resolution dashboard configuration (e.g., a 24-hour view with a 1-minute step, amounting to 1440 data points per graph), then you may benefit from increasing the timeseries_retention_factor accordingly. If you use a managed cache (see caches) and increase the timeseries_retention_factor, the overall size of your cache will not change; the result will be fewer objects in cache, with the timeseries objects having a larger share of the overall cache size with more aged data.
Time Series Data Evictions
Once the TRF is reached for a time series cache object, Trickster will undergo a timestamp eviction process for the record in question. Unlike the Cache Object Eviction, which removes an object from cache completely, TRF evictions examine the data set contained in a cache object and remove timestamped data in order to reduce the object size down to the TRF.
Time Series Data Evictions apply to all cached time series data sets, regardless of whether or not the cache object lifecycle is managed by Trickster.
Trickster provides two eviction methodologies (timeseries_eviction_method) for time series data eviction: oldest (default) and lru, and is configurable per-origin.
When timeseries_eviction_method is set to oldest, Trickster maintains time series data by calculating the “oldest cacheable timestamp” value upon each request, using time.Now().Add(step * timeseries_retention_factor * -1). Any queries for data older than the oldest cacheable timestamp are intelligently offloaded to the proxy since they will never be cached, and no data that is older than the oldest cacheable timestamp will be stored in the query’s cache record.
When timeseries_eviction_method is set to lru, Trickster will not calculate an oldest cacheable timestamp, but rather maintain a last-accessed time for each timestamp in the cache object, and evict the Least-Recently-Used items in order to maintain the cache size.
The advantage of the oldest methodology better cache performance, at the cost of not caching very old data. Thus, Trickster will be more performant computationally while providing a slightly lower cache hit rate. The lru methodology, since it requires accessing the cache on every request and maintaining access times for every timestamp, is computationally more expensive, but can achieve a higher cache hit rate since it permits caching data of any age, so long as it is accessed frequently enough to avoid eviction.
Most users will find the oldest methodology to meet their needs, so it is recommended to use lru only if you have a specific use case (e.g., dashboards with data from a diverse set of time ranges, where caching only relatively young data does not suffice).
4.3 - Chunked Caching
Overview
In some caching setups, users may want to increase timeseries_retention_factor or timeseries_ttl forms a given backend to a very large size (e.g., a duration of days or weeks). This can cause issues if the cache provider is
filesystem or redis, because the entire time series is loaded to extract
even just a few data points. Eventually, this could negate the effects of
caching altogether.
To mitigate this, Trickster supports chunking cache data, by splitting large datasets into subdivisions of a configurable maximum size. The chunks are reconstituted upon retrieval. Only the chunks needed to service a client request are accessed, rather than the entire time series cache object.
Chunking can be configured per-cache and applies to both timeseries and byterange data.
Configuration
Chunked caching can be enabled and disabled using use_cache_chunking:
fs1:
provider: filesystem
use_cache_chunking: true
timeseries_chunk_factor: 420
byterange_chunk_size: 4096
timeseries_chunk_factor determines the maximum extent of timerange chunks, and byterange_chunk_size determines the maximum size of byterange chunks. See Detail for more information.
Changes to any of these settings take effect on a configuration reload, without a restart.
Detail
Timeseries
Timeseries chunking splits the timeseries to be cached into parts with the same duration, but not necessarily the same literal size.
- Determine a chunk duration by multiplying the timerange step by
timeseries_chunk_factor(default 420) - Determine the smallest possible extent that is aligned to the epoch along the chunk duration, while containing the entire timeseries
- To write: Write each chunk size subextent under a subkey
- To read: Read each subkey and merge the timeseries results
Byterange
Byterange chunking splits the byterange into pieces with the same literal size. There are also some extra steps compared to the timeseries implementation to preserve the integrity of both full and partial responses being cached.
- Determine a chunk size from
byterange_chunk_size(default 4096) - Determine a range from the cache read/write request using the provided range or content length
- Failure to determine a range on write results in an error
- Failure to determine a range on read will read until the query fails
- Determine a maximum range aligned along the chunk size that contains the entire byterange
- To write: Write each chunk size range with
RangePartsof all provided ranges cropped to that chunk range, under a subkey. A chunk that falls in a gap between cached ranges holds nothing and isn’t written. - To read: Read only the subkeys covering the requested range and reconstitute a body from
RangeParts. Cached ranges outside those chunks stay in the cache and remain listed in the meta document, so a later write leaves them intact. - A chunk that was evicted or expired is treated as a gap: its bytes are fetched from the origin again and rewritten, rather than served from the remaining chunks.
Full Example
This example has one Prometheus backend with a memory cache that has chunking enabled. The memory cache uses 380 as its timeseries chunk factor, and doesn’t define a byterange chunk size, so the default of 4096 will be used.
caches:
mem1:
provider: memory
index:
max_size_objects: 512
max_size_backoff_objects: 128
use_cache_chunking: true
timeseries_chunk_factor: 380
backends:
prom1:
latency_max: 150ms
latency_min: 50ms
provider: prometheus
origin_url: 'http://127.0.0.1:9090'
cache_name: mem1
logging:
log_level: warn
4.4 - Negative Caching
Negative Caching means to cache undesired HTTP responses for a very short period of time, in order to prevent overwhelming a system that would otherwise scale normally when desired, cacheable HTTP responses are being returned. For example, Trickster can be configured to cache 404 Not Found or 500 Internal Server Error responses for a short period of time, to ensure that a thundering herd of HTTP requests for a non-existent object, or unexpected downtime of a critical service, do not create an i/o bottleneck in your application pipeline.
Trickster supports negative caching of any status code >= 300 and < 600, on a per-Backend basis. In your Trickster configuration file, associate the desired Negative Cache Map to the desired Backend config. See the example.full.yaml, or refer to the snippet below for more information.
A moq backend uses its negative cache’s 404 TTL for an origin’s DOES_NOT_EXIST refusal of a track, MoQ’s equivalent of a 404, and no other entry.
The Negative Cache Map must be an all-inclusive list of explicit status codes; there is currently no wildcard or status code range support for Negative Caching entries. By default, the Negative Cache Map is empty for all backend configs. The Negative Cache only applies to Cacheable Objects, and does not apply to Proxy-Only configurations.
For any response code handled by the Negative Cache, the response object’s effective cache TTL is explicitly overridden to the value of that code’s Negative Cache TTL, regardless of any response headers provided by the Backend concerning cacheability. All response headers are left in-tact and unmodified by Trickster’s Negative Cache, such that Negative Caching is transparent to the client. The X-Trickster-Result response header will indicate a response was served from the Negative Cache by providing a cache status of nchit.
Multiple negative cache configurations can be defined, and are referenced by name in the backend config. By default, a backend will use the ‘default’ Negative Cache config, which, by default is empty. The default can be easily populated in the config file, and additional configs can easily be added, as demonstrated below.
The format of a negative cache map entry is 'status_code': ttl.
Example Negative Caching Config
negative_caches:
default:
'404': 3s # cache 404 responses for 3 seconds
foo:
'404': 3s
'500': 5s # caches 404 response for 3 seconds, and 500/502 for 5 seconds
'502': 5s
backends:
default:
provider: rpc
# by default will assume negative_cache_name = 'default'
another:
provider: rpc
negative_cache_name: foo
5 - Object Caching
5.1 - Byte Range Request Support
Trickster’s HTTP Reverse Proxy Cache offers best-in-class acceleration and caching of Byte Range Requests.
Much like its Time Series Delta Proxy Cache, Trickster’s Reverse Proxy Cache will determine what ranges are cached, and only request from the origin any uncached ranges needed to service the client request, reconstituting the ranges within the cache object. This ensures minimal response time for all Range requests.
In addition to supporting requests with a single Range (Range: bytes=0-5) Trickster also supports Multipart Range Requests (Range: bytes=0-5, 10-20).
Fronting Origins That Do Not Support Multipart Range Requests
In the event that an upstream origin supports serving a single Range, but does not support serving Multipart Range Requests, which is quite common, Trickster can transparently enable that support on behalf of the origin. To do so, Trickster offers a unique feature called Upstream Range Dearticulation, that will separate any ranges needed from the origin into individual, parallel HTTP requests, which are reconstituted by Trickster. This behavior can be enabled for any origin that only supports serving a single Range, by setting the origin configuration value dearticulate_upstream_ranges = true, as in this example:
backends:
default:
provider: reverseproxycache
origin_url: 'http://example.com/'
dearticulate_upstream_ranges: true
If you know that your clients will be making Range requests (even if they are not Multipart), check to ensure the configured origin supports Multipart Range requests. Use curl to request any static object from the origin, for which you know the size, and include a Multipart Range request; like curl -v -H 'Range: bytes=0-1, 3-4' 'http://example.com/object.js'. If the origin returns 200 OK and the entire object body, instead of 206 Partial Content and a multipart body, enable Upstream Range Dearticulation to ensure optimal performance.
This is important because a partial hit could result in multiple ranges being needed from the origin - even for a single-Range client request, depending upon what ranges are already in cache. If Upstream Range Dearticulation is disabled in this case, full objects could be unnecessarily returned from the Origin to Trickster, instead of small delta ranges, irrespective of the object’s overall size. This may or may not impact your use case.
Rule of thumb: If the origin does not support Multipart requests, enable Upstream Range Dearticulation in Trickster to compensate. Conversely, if the origin does support Multipart requests, do not enable Upstream Range Dearticulation.
Disabling Multipart Ranges to Clients
One of the great benefits of using Upstream Range Dearticulation is that it transparently enables Multipart Range support for clients, when fronting any origin that already supports serving just a single Range.
There may, however, be cases where you do not want to enable Multipart Range support for clients (since its paired Origin does not), but need Upstream Range Dearticulation to optimize Partial Hit fulfillments. For those cases, Trickster offers a setting to disable Multipart Range support for clients, while Upstream Range Dearticulation is enabled. Set multipart_ranges_disabled = true, as in the below example, and Trickster will strip Multipart Range Request headers, which will result in a 200 OK response with the full body. Client single Range requests are unaffected by this setting. This should only be set if you have a specific use case where clients should not be able to make multipart Range requests.
backends:
default:
provider: reverseproxycache
origin_url: 'http://example.com/'
dearticulate_upstream_ranges: true
multipart_ranges_disabled: true
Partial Hit with Object Revalidation
As explained above, whenever the client makes a Range request, and only part of the Range is in the Trickster cache, Trickster will fetch the uncached Ranges from the Origin, then reconstitute and cache all of the accumulated Ranges, while also replying to the client with its requested Ranges.
In the event that a cache object returns 1) a partial hit, 2) that is no longer fresh, 3) but can be revalidated, based on a) the Origin’s provided caching directives or b) overridden by the Trickster operator’s explicit path-based Header configs; Trickster will revalidate the client’s requested-but-cached range from the origin with the appropriate revalidation headers.
In a Partial Hit with Revalidation, the revalidation request is made as a separate, parallel request to the origin alongside the uncached range request(s). If the revalidation succeeds, the cached range is merged with the newly-fetched range as if it had never expired. If the revalidation fails, the Origin will return the range needed by the client that was previously cached, or potentially the entire object - either of which are used to complete the ranges needed by the client and update the cache and caching policy for the object.
Range Miss with Object Revalidation
Trickster recognizes when an object exists in cache, but has none of the client’s requested Ranges. This is a state that lies between Cache Miss and Partial Hit, and is known as “Range Miss.” Range Misses can happen frequently on Range-requested objects.
When a Range Miss occurs against an object that also requires revalidation, Trickster will not initiate a parallel revalidation request, since none of the client’s requested Ranges are actually eligible for revalidation. Instead, Trickster will use the Response Headers returned by the Range Miss Request to perform a local revalidation of the cache object. If the object is revalidated, the new Ranges are merged with the cached Ranges before writing to cache based on the newly received Caching Policy. If the object is not revalidated, the cache object is created anew solely from the Range Miss Response.
Multiple Parts Require Revalidation
A situation can arise where there is a partial cache hit has multiple ranges that require revalidation before they can be used to satisfy the client. In these cases, Trickster will check if Upstream Range Dearticulation is enabled for the origin to determine how to resolve this condition. If Upstream Range Dearticulation is not enabled, Trickster trusts that the upstream origin will support Multipart Range Requests, and will include just the client’s needed-and-cached-but-expired ranges in the revalidation request. If Upstream Range Dearticulation is enabled, Trickster will forward, without modification, the client’s requested Ranges to the revalidation request to the origin. This behavior means Trickster currently does not support multiple parallel revalidation requests. Whenever the cache object requires revalidation, there will be only 1 revalidation request upstream, and 0 to N additional parallel upstream range requests as required to fulfill a partial hit.
If-Range Not Yet Supported
Trickster currently does not support revalidation based on If-Range request headers, for use with partial download resumptions by clients. If-Range headers are simply ignored by Trickster and passed through to the origin, which can result in unexpected behavior with the Trickster cache for that object.
We plan to provide full support for If-Range as part of Trickster 1.1 or 2.0
Range Request Test Origin
To verify Trickster’s compatibility with Byte Range Requests, the repository includes a mock origin in pkg/testutil/mocks/rangesim. It serves the Lorem ipsum ... sample text at any path, pared down to the requested range or multipart ranges, with query parameters that customize its responses for testing. The unit tests use it extensively to verify the integrity of Trickster’s output after operations like merging disparate range parts, extracting ranges from other ranges or from a full body, and compressing adjacent ranges into a single range in the cache. For examples, see pkg/proxy/engines/objectproxycache_test.go.
The developer environment serves the same origin at http://127.0.0.1:8482/byterange/, behind its rpc1 and rpc2 backends.
5.2 - Collapsed Forwarding
Collapsed Forwarding is feature common among Reverse Proxy Cache solutions like Squid, Varnish and Apache Traffic Server. It works by ensuring only a single request to the upstream origin is performed for any object on a cache miss or revalidation attempt, no matter how many users are requesting the object at the same time.
Trickster has support for two types of Collapsed Forwarding: Basic (default) and Progressive
Basic Collapsed Forwarding
Basic Collapsed Forwarding is the default functionality for Trickster, and works by waitlisting all requests for a cacheable object while a cache miss is being serviced for the object, and then serving the waitlisted requests once the cache has been populated.
The feature is further detailed in the following diagram:

Progressive Collapsed Forwarding
Progressive Collapsed Forwarding (PCF) is an improvement upon the basic version, in that it eliminates the waitlist and serves all simultaneous requests concurrently while the object is still downloading from the server, similar to Apache Traffic Server’s “read-while-write” feature. This may be useful in low-latency applications such as DASH or HLS video delivery, since PCF minimizes Time to First Byte latency for extremely popular objects.
The feature is further detailed in the following diagram:

PCF for Proxy-Only Requests
Trickster provides a unique feature that implements PCF in Proxy-Only configurations, to bring the benefits of Collapsed Forwarding to HTTP Paths that are not configured to be routed through the Reverse Proxy Cache. (See Paths documentation for more info on routing).
The feature is further detailed in the following diagram:

When collapsing is refused
Collapsing delivers the same bytes to every client that joins, which is a stronger claim than caching makes. Trickster therefore applies the shared-cache rules from RFC 9111 before fanning a response out, and refuses when any of the following holds. A refused response is still served normally; each client simply gets its own fetch.
| Condition | Reason |
|---|---|
| Method is not GET or HEAD | Collapsing a non-idempotent request would mean one of them never executed (RFC 9110 9.2.1) |
| Response status is not 200 | Partial and error responses are not safely shareable |
Cache-Control: private or no-store | Explicitly single-user (RFC 9111 5.2.2.5, 5.2.2.7) |
Response carries Set-Cookie | Per-client state; sharing it would disclose it across users |
Request carried Authorization without public, s-maxage or must-revalidate on the response | RFC 9111 3.5 |
Vary: *, or a Vary field not listed in the path’s cache_key_headers | Two joiners could legitimately deserve different bytes |
Content-Type: text/event-stream | An event stream is per-subscriber, not a shared object |
Response is larger than the backend’s max_object_size_bytes | The shared buffer is bounded by that limit |
The default is to refuse: a missed collapse costs one extra origin fetch, while an incorrect one would serve one user’s response to another.
Object sizes
A response with a known Content-Length is buffered exactly. A response of
unknown length – a chunked transfer, which is common for live video manifests
and segments – is buffered as it arrives, growing up to the backend’s
max_object_size_bytes. A collapsed transfer that exceeds that limit is aborted
for every attached client rather than being silently truncated.
If the origin fails or ends a body early, every client attached to that collapse receives a failed response. An incomplete object is never presented as complete and is never written to cache.
How to enable Progressive Collapsed Forwarding
When configuring path configs as described in Paths Documentation add collapsed_forwarding: progressive in any path config using the proxy or proxycache handlers.
Example:
origins:
test:
paths:
- path: /test_path1/
match_type: prefix
handler: proxycache
collapsed_forwarding: progressive
- path: /test_path2/
match_type: prefix
handler: proxy
collapsed_forwarding: progressive
See the example.full.yaml for more configuration examples.
How to test Progressive Collapsed Forwarding
An easy way to test PCF is to set up your favorite file server to host a large file(Lighttpd, Nginx, Apache WS, etc.), In Trickster turn on PCF for that path config and try make simultaneous requests. If the networking between your machine and Trickster has enough bandwidth you should see both streaming at the equivalent rate as the origin request.
Example:
- Run a Lighttpd instance or docker container on your local machine and make a large file available to be served
- Run Trickster locally
- Make multiple curl requests of the same object
You should see the speed limited on the origin request by your disk IO, and your speed between Trickster limited by Memory/CPU
5.3 - Media over QUIC (MoQ) Relay
Trickster relays Media over QUIC Transport (MoQT) as a caching relay. Clients subscribe to tracks through Trickster, which holds one subscription per track toward its origin and fans each object out to every subscriber. It keeps the newest groups in memory and older ones in a cache, so FETCH and late joiners are served from Trickster with only the missing objects fetched upstream. Publishers can also send tracks straight to Trickster, making it the origin.
MoQT is a live-media transport: a track is a sequence of groups (for video, typically one group per GOP), each a sequence of objects, delivered over QUIC streams or datagrams. Trickster never parses the media inside objects; it caches and forwards them as opaque payloads.
Where Trickster fits

A publisher packages media into tracks and publishes them to an origin relay. A mid-tier
Trickster subscribes to the origin and serves everything below it: viewers directly, or an
optional edge tier of more Trickster nodes nearer the viewers. An edge node is a moq backend
whose origin_url names a mid-tier node’s moq listener. Each tier holds one upstream
subscription per track for all of its subscribers, with a live edge and a cache of its own, so a
tier adds capacity near the viewers while the tier above it sees one subscriber per node.
Two variations each change one part of this picture, and combine with it and with each other:
- Publishing: publishers send tracks straight to Trickster, which takes the origin relay’s place.
- Origin pools: a Trickster reaches several origins through an alb, which places each track on one of them and fails over between them.
Compatibility
MoQT is still an Internet-Draft, and implementations in the field speak different drafts. Trickster speaks drafts -17 through -22 and negotiates one per session; a session on one draft can relay a track whose origin speaks another.
| Draft | ALPN | Notes |
|---|---|---|
| -22 | moqt-22 | the newest and the preferred draft; LOCATION_FILTER carries its type |
| -21 | moqt-21 | editorial changes only |
| -20 | moqt-20 | fill fetch streams replace Joining FETCH |
| -19 | moqt-19 | range filters, MAX_REQUEST_UPDATES, multiple subscriptions per track |
| -18 | moqt-18 | moqt:// URIs, SUBSCRIBE_TRACKS, REDIRECT |
| -17 | moqt-17 | Required Request ID, PUBLISH_OK, the combined SUBSCRIBE_NAMESPACE |
Native QUIC clients negotiate the draft by ALPN, and WebTransport clients by
WT-Available-Protocols. Trickster prefers the newest draft, and offers origins every draft,
newest first. moq-dev’s relay before 0.17.3 and CLI before 0.14.3 encode -22 as -21; to work
with those releases, list -21 ahead of -22 in drafts and upstream.drafts. Drafts before -17
are not supported.
Interoperability is tested on every draft against moq-dev’s
relay (0.17.5) and moq CLI (0.14.5), including FETCH.
How clients reach the relay
- Native QUIC. A listener with
protocol: moqaccepts MoQT sessions on its UDP port. A session is routed to amoqbackend by the AUTHORITY and PATH of its SETUP, matched against the backend’shostsandpaths. - WebTransport. Browsers open WebTransport sessions on an HTTP listener that serves
HTTP/3. A
moqbackend’spathsbecome WebTransport routes there, matched like any other route by host and path. - Publishers use either path, with
PUBLISHorPUBLISH_NAMESPACE, when the backend accepts them.
All three meet in the backend’s relay, which is shared by every listener the backend is bound to.
Configuration
listeners:
# native QUIC; TLS is part of the QUIC handshake, so there is no tls_port
moq1:
protocol: moq
port: 443
# browsers arrive by WebTransport over the default listener's HTTP/3 endpoint
default:
port: 8480
tls_port: 8443
http3:
enabled: true
backends:
live:
provider: moq
# moqt:// for a native QUIC origin, https:// for a WebTransport one; omit it to serve
# only what publishers send
origin_url: moqt://origin.example.com:443/live
listener_names: [moq1, default]
hosts: [live.example.com]
cache_name: default
max_ttl: 1h
tls:
# served to clients of both listeners
full_chain_cert_path: /etc/trickster/tls/live.crt
private_key_path: /etc/trickster/tls/live.key
# trusted toward the origin
certificate_authority_paths: [/etc/trickster/tls/origin-ca.crt]
example.full.yaml documents every option.
The moq listener
| Option | Default | Description |
|---|---|---|
port, address | the UDP port and address to bind | |
moq.handshake_timeout | 5s | the time a client has to send SETUP |
moq.idle_timeout | 2m | the QUIC idle timeout |
moq.max_sessions | 10000 | sessions served at once; past it, a new connection is closed (“session limit reached”) |
moq.max_requests_per_session | 1024 | requests a client may have open at once; past it, a request is refused with EXCESSIVE_LOAD |
moq.max_filter_ranges, moq.max_request_updates | 16, 64 | advertised to clients in SETUP |
moq.drafts | [22, 21, 20, 19, 18, 17] | accepted drafts, most preferred first |
moq.implementation | trickster/<version> | the MOQT_IMPLEMENTATION setup option |
moq.goaway_uri | the moqt:// URI a GOAWAY names when the listener stops; see Reloads and shutdown |
A moq listener needs a certificate. It takes one from the tls block of a backend it
serves, or at runtime with tls_runtime_certs: true, as the
Kubernetes controller sets. Certificate files are watched and rotated as for any
listener (tls.md).
A moq listener may share its UDP address and port with the HTTP/3 endpoint of an HTTPS
listener, or with another moq listener, as long as they serve different ALPNs. Each QUIC
connection goes to the listener whose protocol it negotiated, and each keeps its own
certificates and routes. A moq listener never shares a port with a protocol: udp
listener.
The listener’s ip_acl_name is judged on the client’s first packet, before its TLS handshake, so a
refused client costs no handshake; drop and reject both refuse it there, since QUIC cannot
leave a client unanswered. Its decisions are counted in trickster_ip_acl_decisions_total with
scope="listener". The listener’s rate_limiter_name (see Limits) is judged once the
handshake has verified the client’s address. Once the session’s SETUP routes it to a backend, it
is judged by that backend’s ip_acl_name (counted with scope="backend") and geo_acl_name, so
backends sharing a listener keep their own lists. A session one refuses closes with
UNAUTHORIZED, and is logged at debug.
A reload judges the sessions already open by the new lists: the listener’s ip_acl_name, and
the ip_acl_name and geo_acl_name of each session’s backend, over native QUIC and WebTransport
alike. A session the new lists refuse is closed with UNAUTHORIZED. A WebTransport session is
not judged again by its HTTPS listener’s lists. The listener’s path_normalization applies to
the PATH of a SETUP before it is routed, and a path it refuses closes the session with
MALFORMED_PATH. Settings that shape TCP connections and HTTP requests (connections_limit,
the request timeouts and body limits, max_header_bytes, proxy_protocol, trusted_proxies)
have no effect on a moq listener, and are warned of at load.
Every path of a moq backend reaches the same tracks, whichever path a session named, so a
path cannot carry its own ip_acl_name, geo_acl_name, authenticator_name or
rate_limiter_name: one would guard nothing a session on another path could not reach. Serve
tracks that need other lists from a backend of their own, or grant them by namespace with
tokens. A path with a handler of its own answers HTTP requests, and keeps
its lists. A backend served by a moq listener cannot name an authenticator_name either,
since a native session carries no HTTP credentials; authorize its sessions with tokens.
cors, response_headers and mirrors have no effect on a session, and a request rewriter or
request_headers reaches only a WebTransport session’s CONNECT; each is warned of at load.
The moq backend
| Option | Default | Description |
|---|---|---|
origin_url | moqt://host[:port]/path (native QUIC) or https://host[:port]/path (WebTransport); optional | |
hosts, paths | any, / | the AUTHORITY and PATH, or the WebTransport request’s host and path, the backend serves |
cache_name | default | the cache older groups spill to |
max_ttl | 25h | the most a spilled group lives, and its lifetime when its track sets no MAX_CACHE_DURATION |
negative_cache_name | default | the negative cache whose 404 TTL keeps an origin’s DOES_NOT_EXIST refusal of a track; see Caching |
tls | the certificate served to clients, and the origin’s verification | |
moq.upstream.alb | an alb of moq backends to reach the origins through, in place of origin_url; see Origin pools | |
moq.upstream.drafts | [22, 21, 20, 19, 18, 17] | drafts offered to the origin, most preferred first |
moq.upstream.prewarm | false | keeps the upstream subscription forwarding while every subscriber is paused |
moq.upstream.reconnect_backoff | 1s | the first wait before re-subscribing a lost upstream subscription |
moq.upstream.linger | 5s | how long an upstream subscription stays after its last subscriber leaves |
moq.upstream.max_gap_fetches | 16 | upstream fetches one FETCH or fill may issue |
moq.live_edge.groups | 3 | groups of each track kept in memory |
moq.live_edge.max_size_bytes | 64MiB | payload bytes the live edge holds across the backend’s tracks |
moq.spill | true | writes groups leaving the live edge, and groups fetched upstream, to cache_name |
moq.fill_timeout | 2s | the wait for upstream fetches when a FETCH or fill names no FILL_TIMEOUT |
moq.max_object_size_bytes | 64MiB | the largest object; a larger one resets its stream |
moq.webtransport.* | see WebTransport and browsers | |
moq.publish.* | see Publishing | |
moq.authorization.* | see Authorization |
A moq backend without origin_url (or upstream.alb) has no origin. It serves what
publishers send, and validation warns when it accepts no publishers either. With an origin, the
health check completes a SETUP exchange with it on a fresh session; an origin-less backend is
left unprobed.
Routing sessions
A native session names its AUTHORITY and PATH in SETUP; when it names no authority, the TLS
server name stands in, and when it names no path, /. Among the backends on the listener, one
naming the host exactly wins over one naming it by wildcard, which wins over one naming no host.
Among those, the longest segment-aligned path prefix wins. A listener with one backend serves
every session with it, and a session no backend matches is closed with INVALID_PATH. No two
backends on a listener may claim the same host and path.
WebTransport and browsers
A moq backend bound to an HTTP listener serves WebTransport sessions on that listener’s HTTP/3
endpoint (http3.md); validation warns when the listener has none. The endpoint
switches to WebTransport only while a moq backend uses the listener, and a reload that binds
or unbinds one changes how new connections are served without restarting the endpoint.
Each of the backend’s paths accepts WebTransport sessions at and below it (a regex path
matches as a pattern), and a backend with no paths accepts them at /. The usual HTTP routing
applies: on a listener shared with other backends, give the moq backend hosts or make it
is_default, or reach it under its name (https://host/<backend>/). The CONNECT request runs
through the route’s middleware, so authenticator_name, IP and geo ACLs, metrics and access logs
all apply to it.
| Option | Default | Description |
|---|---|---|
moq.webtransport.drafts | [22, 21, 20, 19, 18, 17] | drafts accepted from WebTransport clients, most preferred first |
moq.webtransport.default_draft | 0 | the draft assumed for a client that offers no WT-Available-Protocols, and for a WebTransport origin that answers without WT-Protocol; 0 refuses both |
moq.webtransport.allowed_origins | browser origins (scheme://host[:port]) that may open sessions, or *; empty allows only an origin naming the requested host, and a client that sends no Origin is always allowed | |
moq.webtransport.max_sessions | 10000 | WebTransport sessions the backend serves at once |
moq.webtransport.goaway_uri | the https:// URI a GOAWAY names when the listener stops |
Trickster answers 403 for a refused origin, 405 for a method other than CONNECT, 400 for a
CONNECT that is not WebTransport or offers no usable draft, and 503 past max_sessions.
WebTransport sessions take the moq listener defaults for their limits. Chrome 143, Firefox 155
and Safari 26.4 send WT-Available-Protocols through the WebTransport protocols option; a
browser needs a certificate it trusts.
Toward an https:// origin, backends that reach the same host and port with the same tls
settings and drafts carry their WebTransport sessions on one QUIC connection, each backend with
its own session for its own path. The connection offers WebTransport flow control, which an
origin must also enable to take several sessions on one connection. An origin that refuses a
second session is dialed one connection per session from then on. The connection closes with
its last session. Native moqt:// origins get a connection per backend, since a native
connection is a single session.
Caching
Each track’s newest live_edge.groups groups stay in memory, within a backend-wide
live_edge.max_size_bytes budget that evicts the oldest groups first. Objects are stored once
and forwarded cut-through as their bytes arrive, so every subscriber and the cache share one
copy. A group leaving the live edge spills to cache_name as one entry, by reference in a
memory cache. Its key includes the backend’s name, so backends sharing a cache never read or
overwrite each other’s groups, even for tracks of the same name. It lives for the track’s
MAX_CACHE_DURATION less the oldest object’s age, capped by max_ttl. An object past its MAX_CACHE_DURATION is unknown again in both tiers and
is not forwarded. Spilled groups are indexed in memory, so after a restart a persistent cache’s
groups are fetched from the origin again rather than found. A track found malformed (a duplicate
object differing from the first copy, or an object past the end of its group or track) is
dropped uncached and its subscribers end with MALFORMED_TRACK.
An origin’s DOES_NOT_EXIST refusal of a track is MoQ’s 404. It is kept for the 404 TTL of
the backend’s negative_cache_name, and answers SUBSCRIBE, FETCH and TRACK_STATUS for the track
without asking the origin again. The default negative cache is empty, so nothing is kept unless
configured:
negative_caches:
default:
'404': 3s
Only the origin’s refusal is kept. A namespace publisher’s is not, and neither are codes that
depend on the requester or pass on their own (UNAUTHORIZED, the token codes, TIMEOUT,
INTERNAL_ERROR, EXCESSIVE_LOAD). A PUBLISH of the track, or a namespace arriving that covers
it, forgets the refusal at once, so a stream that starts after its viewers ask is served as soon
as it is published or announced. A backend keeps at most 4096 refusals. Clients that keep asking
for names that do not exist are a matter for rate limiting.
FETCH and fills (-20 and later) are served from the live edge and the cache where a group is
known in full. Each run of groups between them is fetched upstream with one request, in the
requested group order, and cached for the next request. Past max_gap_fetches requests, the
remaining gaps are reported as an End of Unknown Range. A group is
known in full only when every object in the requested range is present or known absent. A
publisher that never signals the end of its groups therefore has past groups fetched upstream
rather than answered incompletely. FILL_TIMEOUT (default fill_timeout, at most 30 seconds)
bounds the upstream wait. Past it, the rest of the range is reported as an End of Timed-Out
Range, and an unreachable origin as an End of Unknown Range. A FETCH that names no group order
is served ascending, as the drafts require, whatever the track’s publisher prefers. Joining
FETCH, on drafts through -19, is served the same way.
Subscriptions
One upstream subscription per track carries every downstream subscription to it, each filtered
at the relay by its own location window and range filters. A new subscription first gets the
newest group the relay holds, from the start of its window, and then what arrives. A player
joining at the current group, as moq-dev’s does with a Relative Start of 1, so gets a catalog
that was published once, and a subscription starting at the Next Object gets only what arrives.
Older groups come by fill or FETCH. Draft -22 §3.5 has a joiner ask for the current group with
a fill; the relay delivers it either way. SUBSCRIBE_OK waits until the upstream
subscription is established. Forwarding follows the subscribers: the upstream subscription
forwards while any subscriber does, and pauses otherwise unless upstream.prewarm keeps it
forwarding. A lost upstream subscription is re-subscribed with doubling backoff, up to five
attempts, while the downstream subscriptions stay open. That includes one the origin ends with
UPDATE_FAILED for refusing the relay’s own update, such as a resume or an EXPIRES renewal;
should the retries fail, its subscribers see INTERNAL_ERROR, since the update was never theirs. Only when that fails do they end with
PUBLISH_DONE. An upstream subscription lingers for upstream.linger after its last subscriber
leaves, so a returning viewer finds it warm.
When the upstream gives an EXPIRES, the relay asks to extend the subscription before it
elapses, with an empty REQUEST_UPDATE at three quarters of the time given. A new EXPIRES in
the answer schedules the next renewal; an answer without one, such as to a FORWARD change,
leaves the pending renewal as it was. The relay never ends a downstream subscription on a
timer, so it sends no EXPIRES downstream.
NEW_GROUP_REQUEST reaches the publisher as the spec requires of relays:
- with the upstream
SUBSCRIBEwhile the track has no established subscription; - afterwards, as a
REQUEST_UPDATEto every source, but only for a value of 0 or one past the Largest Group, on a track whoseDYNAMIC_GROUPSis 1, and only while no request at least as large is outstanding. A request is outstanding until the Largest Group increases.
Publishing
With moq.publish.enabled, a backend accepts PUBLISH and PUBLISH_NAMESPACE and becomes the
origin of what publishers send. Open ingest belongs behind a network boundary or
authorization.

Here Trickster takes the origin relay’s place in Where Trickster fits. Its subscribers, and any tiers below it, are served as they would be from a pulled track.
| Option | Default | Description |
|---|---|---|
moq.publish.enabled | false | accepts publishers |
moq.publish.namespaces | any | namespace prefixes publishers may claim, as fields joined by / (live/studio) |
moq.publish.max_publishers_per_track | 4 | publishers of one track at once; 0 allows any number |
A track can have several sources at once: publishers’ PUBLISHes, and the subscriptions the
relay opens toward every session that published a matching namespace, or toward the origin when
none did. The first copy of each object wins, an identical duplicate is dropped, and a differing
one makes the track malformed. A publisher can take over from another without its subscribers
noticing, and one whose session drops can return on a new session before the relay gives up on
the track. The relay never pauses a publisher, so an ingest backend’s cache keeps everything
published.
SUBSCRIBE_NAMESPACE is answered with the namespaces publishers announced, and kept current as
they come and go. A backend with an origin also learns the origin’s namespaces, in either of two
ways, and announces a namespace once however many sources report it:
- It forwards the request to the origin. What an origin reports is backend-wide, so every scope
shares the upstream requests, and their prefixes never overlap, as one session requires: a
prefix that an upstream request covers shares it, and a wider one replaces the narrower
requests it covers. A replaced request’s namespaces stay announced while the wider one
reports them again, so subscribers see no flap. A lost request withdraws what it reported and
is sent again. One the origin refuses is not retried, except a
PREFIX_OVERLAPrefusal, which the replacement can race. One whose last subscriber leaves lingers forupstream.linger. - It accepts
PUBLISH_NAMESPACEfrom the origin on its upstream session, as moq-dev’s relay sends, and reports those namespaces until the origin withdraws them or the session ends.
SUBSCRIBE_TRACKS offers each published track under the prefix with a relay PUBLISH. A
backend with an origin forwards it the same way, shared by prefix (as SUBSCRIBE_NAMESPACE with
the PUBLISH option on draft -17). Each PUBLISH the origin sends in answer becomes a source of
its track and is offered to every matching subscriber, in every scope. A subscriber in another
scope is served the track pulled from the origin with SUBSCRIBE. An origin PUBLISH that no
subscription covers is refused with UNINTERESTED.
A SUBSCRIBE naming a RENDEZVOUS_TIMEOUT waits up to that long (at most 30
seconds) for a publisher to arrive; without one, a track nobody publishes is refused with
DOES_NOT_EXIST. Publishers are refused with:
NOT_SUPPORTEDwhen the backend takes no publishers;UNAUTHORIZEDoutsidepublish.namespaces, or without a granted token;EXCESSIVE_LOADpastmax_publishers_per_track.
Authorization
moq.authorization.mode is none (the default) or token. In token mode, every request needs
an AUTHORIZATION_TOKEN that token_file grants, sent in the request or in the session’s SETUP.
# <token> <publish|subscribe|any> [namespace]
studio-encoder publish live/studio
viewer-7f3a subscribe live
ops-tools any
SUBSCRIBE, FETCH, TRACK_STATUS, SUBSCRIBE_NAMESPACE and SUBSCRIBE_TRACKS need a
subscribe grant, and PUBLISH and PUBLISH_NAMESPACE a publish grant. A grant’s namespace
must prefix the request’s, so a grant for live covers live/studio but not vod. A grant with
no namespace covers every namespace. The file is read again when it changes, checked at most
once a second; a file that no longer parses keeps its last grants, and one that cannot be read
at load fails the backend. A WebTransport CONNECT is also subject to the route’s
authenticator_name, before any MoQT request is made. A native session carries no HTTP
credentials, so a backend that a moq listener serves cannot name an authenticator, and tokens
are how its sessions are authorized.
Origin pools
moq.upstream.alb names an alb whose pool members are moq backends with origins.
The routed backend keeps its relay, cache, live edge, WebTransport, publishing and authorization,
and reaches its origins through the alb, which picks one member per track:
backends:
live:
provider: moq
listener_names: [moq1]
moq:
upstream:
alb: origins
origins:
provider: alb
alb:
mechanism: hrw
pool: [origin-a, origin-b]
origin-a:
provider: moq
origin_url: moqt://origin-a.example.com/
origin-b:
provider: moq
origin_url: moqt://origin-b.example.com/

The diagram shows this configuration, all in one Trickster process: viewers subscribe to live,
whose relay reaches Origin A and Origin B through the origins alb. In
Where Trickster fits, it replaces the mid-tier’s single origin.
The mechanism must pick a single member: rr, p2c, hrw (each track placed by its full name)
or lc. When a track’s origin fails, its upstream subscription moves to another member and its
subscribers see no PUBLISH_DONE. A member that fails its health check is passed over until it
recovers, and stream.passive_health on the alb ejects members whose origins cannot be dialed.
Pool members must be moq backends with origins; nested albs and discovery are refused.
The routed backend dials each member on a session of its own, apart from the member backend’s
own relay, so the two never share a Track Alias. A SUBSCRIBE_NAMESPACE or SUBSCRIBE_TRACKS
it forwards goes to the member the prefix places, and what that member’s origin publishes on the
session reaches the routed backend’s subscribers.
A member listed on an HTTP listener is reachable there at /<member name>/, as any backend is,
and judged by its own lists rather than the routed backend’s; give members
path_routing_disabled: true so only the routed backend serves sessions. The Kubernetes
controller does so for the members it generates.
Priorities
MoQT orders objects by subscriber priority, then publisher priority, then group order (each 0 to 255, lower first). QUIC offers eight stream urgencies (RFC 9218), so Trickster maps the two priorities onto urgencies 1 to 6:
| Subscriber priority | Publisher priority below 128 | Publisher priority 128 and above |
|---|---|---|
| 0–63 | 1 | 2 |
| 64–191 | 3 | 4 |
| 192–255 | 5 | 6 |
Control and request streams take urgency 0. Within an urgency, an ascending subscription’s streams are sent oldest first, while a descending subscription’s streams share bandwidth so its newest group keeps moving. Datagrams have no priority, and WebTransport streams take none either, since webtransport-go exposes no stream urgency.
Reloads and shutdown
A reload never ends a session. A session keeps the backend it was routed to for its life,
even when a reload removes that backend; only new sessions see the new configuration. Each
backend’s relay (its live edge, upstream subscriptions and upstream session) carries over to the
reloaded backend while its relay settings are unchanged. A reload that changes them starts a
new relay for new sessions, and the old one serves its sessions to their end. Publishing and
authorization policy takes effect when a reload commits. A reload restarts a moq listener only
when its address, port or moq block changes; its sessions are then drained as at shutdown.
When Trickster stops, each session gets a GOAWAY naming the listener’s goaway_uri (or the
backend’s webtransport.goaway_uri), with the drain’s remaining time as its timeout. New
requests on a departing session are answered with REDIRECT to that URI on drafts that have it
(-18 and later), or refused with GOING_AWAY. Sessions still open when the drain ends are closed
with GOAWAY_TIMEOUT.
Limits
| Limit | Value |
|---|---|
| sessions per listener | moq.max_sessions (10000) |
| open requests per session | moq.max_requests_per_session (1024), then EXCESSIVE_LOAD with a 1-second retry |
| WebTransport sessions per backend | moq.webtransport.max_sessions (10000) |
| object size | moq.max_object_size_bytes (64 MiB) |
| live edge | moq.live_edge.max_size_bytes (64 MiB) per backend |
| upstream fetches per FETCH or fill | moq.upstream.max_gap_fetches (16) |
| upstream wait for a FETCH or fill | FILL_TIMEOUT, at most 30 seconds |
| rendezvous wait | RENDEZVOUS_TIMEOUT, at most 30 seconds |
| data streams waiting for an unknown track alias | 64 per session, each for at most 2 seconds |
| namespaces | 32 fields and 4096 bytes |
remembered DOES_NOT_EXIST refusals | 4096 per backend |
Rate limiters keyed on client_ip, or on no key for one bucket, limit
clients by address:
- A
moqlistener’srate_limiter_namecounts its sessions (unitsessions), each judged when its connection arrives, after the listener’sip_acl_name. A session over the limit is closed before it begins. - A
moqbackend’srate_limiter_namecounts theSUBSCRIBE,FETCHandTRACK_STATUSrequests of its sessions (unitrequests), native and WebTransport alike, so a client’s sessions share one budget. A request over the limit is refused withEXCESSIVE_LOAD, and a Retry Interval of the wait until it could pass. - WebTransport sessions also pass the HTTPS listener’s limiter, and the backend’s counts each CONNECT as a request too.
rate_limiters:
moq-sessions:
keys: [client_ip]
limit: 20
window: 1m
moq-requests:
keys: [client_ip]
limit: 600
window: 1m
listeners:
moq1:
protocol: moq
port: 443
rate_limiter_name: moq-sessions
backends:
live:
provider: moq
rate_limiter_name: moq-requests
Only client_ip is a key a MoQ limiter can read, and close is not an action it takes. Their
decisions are counted in trickster_ratelimit_decisions_total with plane="moq".
Metrics and logs
| Metric | Labels |
|---|---|
trickster_moq_sessions_total | listener_name, backend_name, role, transport (quic, webtransport), draft, result (ok, refused, no_route, busy, failed) |
trickster_moq_active_sessions | listener_name, backend_name, role |
trickster_moq_requests_total | backend_name, type, result |
trickster_moq_active_subscriptions | backend_name, direction (upstream, downstream) |
trickster_moq_objects_total | backend_name, direction (in, out), delivery (stream, datagram, fetch) |
trickster_moq_bytes_total | backend_name, direction |
trickster_moq_dropped_objects_total | backend_name, reason (oversize, delivery_timeout, datagram_queue_full, spill_queue_full) |
trickster_moq_fetch_objects_total | backend_name, source (live_edge, cache, upstream) |
trickster_moq_live_edge_bytes | backend_name |
trickster_moq_publishers_active | backend_name |
trickster_moq_errors_total | backend_name, class |
Track and namespace names are never labels. Spilled groups count in the cache’s own
trickster_cache_* metrics. Each session’s close is logged at info with its listener, backend,
draft, transport, client, duration and reason.
A native session writes one line to its backend’s access_log when it ends, as a request does,
with the method SETUP, the session’s PATH as the path, its AUTHORITY as the host, the
negotiated ALPN (such as moqt-22) as the protocol, and the session’s duration; it has no
status, which is logged as 0. With the backend’s tracing_name, each native session is one
moq session span carrying moq.backend, moq.authority, moq.path and client.address, and
moq.end with why it ended. A WebTransport session is logged and traced as its CONNECT request,
and since a CONNECT lasts as long as its session, it is counted in
trickster_frontend_requests_total but not timed in trickster_frontend_requests_duration_seconds.
A moq backend’s health is served at /trickster/health/<backend name> on the metrics listener,
as the result of its origin probe. Purging by path does not apply to a moq backend, whose cache
entries are groups of tracks rather than responses to paths, and is refused; spilled groups
expire by max_ttl.
Kubernetes
The Kubernetes controller serves MoQT through a Gateway listener with the protocol
trickstercache.org/moq and UDPRoutes, and WebTransport through an HTTPRoute whose
TricksterCachePolicy names provider: moq. See
kubernetes-gateway.md.
Trying it locally
One command runs a harness: moq-dev’s relay as the origin, hack/diagstreamer publishing a live
test pattern into it, and Trickster in the foreground with
docs/developer/environment/trickster-config/moq.yaml. A second opens a player page, which
Trickster serves at https://localhost:8583/player/, in a Chrome profile that trusts the
development certificate. The page lists the broadcasts the relay announces and plays the selected
one with moq-dev’s <moq-watch> element, loaded from a CDN:
make developer-moq
make developer-moq-browser
ARGS passes flags to Trickster, as in make developer-moq ARGS="-log-level debug". The
harness’s Trickster listens on ports 8580 to 8590, clear of make serve-dev’s, so the two can
run side by side. The integration stack’s relay publishes the same UDP port as the harness’s, so
stop it first if it is running.
The developer environment’s moq profile runs moq-dev’s relay on UDP 10443, which serves as an
origin, and make dev-certs writes a self-signed certificate for localhost:
make dev-certs
make developer-start-moq
Point a relay at it, with a native listener on UDP 8490 and WebTransport on the default listener’s HTTP/3 endpoint:
listeners:
moq1:
protocol: moq
port: 8490
default:
tls_port: 8483
http3:
enabled: true
backends:
demo:
provider: moq
origin_url: moqt://127.0.0.1:10443/
listener_names: [moq1, default]
tls:
full_chain_cert_path: docs/developer/environment/certs/trickster-dev.crt
private_key_path: docs/developer/environment/certs/trickster-dev.key
# the dev relay generates its own certificate
insecure_skip_verify: true
hack/moq-client subscribes, publishes and fetches over native QUIC or WebTransport. It
publishes synthetic groups of random bytes behind a small header, which a subscriber checks
byte for byte:
# publish a group every second into the dev relay
go run ./hack/moq-client -url moqt://127.0.0.1:10443/ -insecure -pub demo/video
# subscribe through Trickster's moq listener, and by WebTransport under the backend's name
go run ./hack/moq-client -url moqt://localhost:8490/ -ca docs/developer/environment/certs/trickster-dev.crt -sub demo/video
go run ./hack/moq-client -url https://localhost:8483/demo/ -ca docs/developer/environment/certs/trickster-dev.crt -sub demo/video
# fetch groups behind the live edge, by the IDs the subscriber printed
go run ./hack/moq-client -url moqt://localhost:8490/ -ca docs/developer/environment/certs/trickster-dev.crt -fetch demo/video -from 1791148101 -to 1791148103
A published group’s ID is the Unix time it started, so a restarted publisher never reuses a
group a relay has cached. -draft offers a single draft, -token sends an
AUTHORIZATION_TOKEN, -groups stops after that many groups, and -h lists the rest. Each
received object prints verified, unverified (a payload without the synthetic header), or
CORRUPT, and the client exits nonzero when any object fails verification.
For real media, publish a fragmented MP4 with moq-dev’s CLI into the dev relay, or into a
backend that accepts publishers, and play it in moq-dev’s web player through Trickster’s
WebTransport endpoint with a certificate the browser trusts.
hack/diagstreamer generates a live feed with a clock and a
timing strip in the picture, as an ABR ladder of CMAF tracks with MSF and hang catalogs, and
publishes it itself:
go -C hack/diagstreamer run ./cmd/diagstreamer -out moq -moq-url moqt://localhost:8490/ -moq-insecure
moq --connect moqt://localhost:8490/ --broadcast diagstreamer.msf export --catalog-format hang fmp4 > out.mp4
Troubleshooting
- A session closes with
INVALID_PATH: no backend on the listener matches the session’s AUTHORITY and PATH. Checkhostsandpaths, or bind one backend to the listener. - WebTransport answers
404: the CONNECT matched no route. Give the backendhosts, make itis_default, or use the backend’s name as the path prefix. - WebTransport answers
400: the browser offered no draft the backend accepts. Checkmoq.webtransport.drafts, or setdefault_draftfor a client that offers none. - A subscription is refused with
DOES_NOT_EXIST: the origin does not have the track, or an origin-less backend has no publisher for it. - Objects stop and resume: compare
trickster_moq_dropped_objects_totalby reason, andtrickster_moq_active_subscriptions{direction="upstream"}for upstream losses. - FETCH reaches the origin every time: the groups never spilled (
moq.spill: false, or a cache that refused them, counted intrickster_moq_errors_total{class="cache"}), or the publisher never marks the end of a group.
Known gaps
- No TCP fallback: WebTransport over HTTP/2 is not served, so clients need UDP.
- No packaging: Trickster does not convert between MoQT and HLS, DASH or other formats.
- Upstream subscriptions are never filtered, so
MAX_FILTER_RANGESis not used toward origins. - On a draft -17 origin,
SUBSCRIBE_NAMESPACEasks for namespaces and for PUBLISHes alike, so a forwarded namespace request and a forwarded track request with overlapping prefixes conflict. The second is refused withPREFIX_OVERLAP, and given up after the retries. - moq-dev’s relay cannot be a WebTransport origin, because it does not support the
reset_stream_atextension that WebTransport over HTTP/3 needs. - Stream priorities and reliable reset boundaries are not available over WebTransport (see Priorities).
6 - Time Series Caching
6.1 - Supported Providers
Trickster currently supports the following Providers:
Generic HTTP Reverse Proxy Cache
Trickster operates as a fully-featured and highly-customizable reverse proxy cache, designed to accelerate and scale upstream endpoints like API services and other simple http services. Specify 'reverseproxycache' or just 'rpc' as the Provider when configuring Trickster.
Static File Server
Trickster can serve the contents of a local directory itself, with no upstream origin, for hosting websites and other static content. Specify 'static' as the Provider when configuring Trickster. See the Static File Server Document for more information.
Media over QUIC (MoQ) Relay
Trickster relays live media as a caching Media over QUIC Transport relay, drafts -17 through -22, to native QUIC clients on a protocol: moq listener and to browsers by WebTransport over HTTP/3. One upstream subscription per track fans out to every subscriber, the newest groups stay in memory and older ones in a cache for FETCH, and publishers can send tracks to Trickster directly. Specify 'moq' as the Provider when configuring Trickster. See the MoQ Relay Document for more information.
Time Series Databases
Prometheus
Trickster fully supports the Prometheus HTTP API (v1), including Prometheus 3.x features like native histograms and UTF-8 metric names. Specify 'prometheus' as the Provider when configuring Trickster. See the Prometheus Support Document for more information.
ClickHouse
Trickster supports accelerating ClickHouse time series over both HTTP and the ClickHouse native binary protocol (port 9000), and is tested against the Vertamedia and official Grafana ClickHouse (v4+) datasource plugins. Specify 'clickhouse' as the Provider when configuring Trickster.
See the ClickHouse Support Document for more information.
InfluxDB
Trickster supports InfluxDB 1.x, 2.x, and 3.x. Specify 'influxdb' as the Provider when configuring Trickster.
See the InfluxDB Support Document for more information.
Apache Druid
Trickster accelerates fixed-width native JSON timeseries, groupBy, and
topN queries plus eligible TIME_FLOOR SQL queries, with safe Object Proxy
Cache fallback for other read-query shapes. Specify 'druid' as the Provider
when configuring Trickster.
See the Apache Druid Support Document for more information.
Graphite
Trickster accelerates Graphite’s render API, including graphite-web, go-carbon
and other Graphite-protocol backends. Specify 'graphite' as the Provider when
configuring Trickster.
See the Graphite Support Document for more information.
PostgreSQL and TimescaleDB
Trickster accepts native PostgreSQL wire-protocol connections and accelerates
time-bucketed queries against PostgreSQL and TimescaleDB, including the
statements Grafana’s built-in PostgreSQL data source sends. Specify postgres
(or its alias timescaledb) as the provider and expose it through a listener
with protocol: postgres.
See the PostgreSQL and TimescaleDB Provider Guide for the supported clients, SQL, authentication, TLS, caching, routing, and operations contract.
GreptimeDB
Trickster accelerates eligible GreptimeDB SQL queries over HTTP, PostgreSQL
and MySQL, plus its Prometheus-compatible range API. Specify greptimedb as
the provider and map native listeners explicitly. See the
GreptimeDB Provider Guide for configuration, cache eligibility,
Grafana macros, authentication and upstream compatibility limits.
QuestDB
Trickster proxies QuestDB’s HTTP surface and accelerates eligible QuestDB SQL
over the PostgreSQL wire protocol. Specify questdb as the provider and map a
protocol: postgres listener for native clients. See the
QuestDB Provider Guide for configuration, cache eligibility,
authentication and upstream compatibility limits.
VictoriaMetrics
Trickster accelerates MetricsQL range queries through VictoriaMetrics’ Prometheus querying API,
caches settled instant and metadata queries, and serves its Graphite render, find and tags APIs,
for single-node VictoriaMetrics and a tenant’s vmselect paths. Specify victoriametrics as the
provider; it is not a flavor of prometheus, and time series merging (TSM) doesn’t support it yet.
See the VictoriaMetrics Provider Guide for configuration and cache eligibility.
MySQL
Trickster supports protocol-aware acceleration for supported MySQL
servers and Grafana’s built-in MySQL data source. Specify mysql as the direct
terminal provider and expose it through a listener with protocol: mysql.
See the MySQL Provider Guide for the supported server, client, SQL, authentication, TLS, caching, routing, and operations contract.
6.2 - Per-Query Time Series Instructions
A query can change how Trickster caches and serves it with directives written in its own comments, so a query’s author can adjust it without changing the backend’s configuration. For example, in PromQL:
sum(rate(http_requests_total[5m])) # trickster-step-align:drop
- Directives are read from comments only. The same text inside a string literal, or anywhere else in the query, is not a directive.
- Several directives can share one comment, separated by spaces. When a directive appears more than once, the last one wins, and an invalid value is ignored.
- Directives change how a query is cached and served, never what a complete bucket holds, so they are not part of the cache key: the same query with and without them shares one cache entry.
- The query is sent to the origin as written; the origin ignores the comment.
- Directives apply to queries Trickster caches as time series. A query served from the object cache, or proxied, is unaffected.
Directives
| Directive | Values | Effect |
|---|---|---|
trickster-step-align | truncate, drop, partial, partial_start, partial_end, off | Selects the query’s step alignment mode, overriding the backend’s. A mode the query doesn’t support falls back to its default, counted in trickster_step_alignment_fallbacks_total. |
trickster-volatile-window | whole seconds, or a duration such as 90s or 5m | Replaces the backend’s volatile_window for this query. |
trickster-fast-forward | off | Turns Fast Forward off for this Prometheus query. A trickster-step-align directive in the same query takes precedence. |
Under an ALB that applies a step alignment mode to its pool, a
trickster-step-align directive wins when every pool member supports the mode it
names; otherwise the ALB’s mode applies.
Comment Syntax by Backend
| Backend and query language | Comments |
|---|---|
| Prometheus, GreptimeDB PromQL, VictoriaMetrics MetricsQL | # ... to the end of the line |
| InfluxQL | -- ... to the end of the line, /* ... */ |
| Flux | // ... to the end of the line |
| InfluxDB SQL, Flight SQL, PostgreSQL, TimescaleDB, Druid SQL, GreptimeDB SQL and PostgreSQL | -- ... to the end of the line, /* ... */ |
| MySQL, GreptimeDB MySQL | -- ... (with a space or tab after the dashes), # ..., /* ... */ |
| ClickHouse | -- ... to the end of the line, /* ... */ |
| Druid native queries | the query’s context map, keyed by the directive’s full name |
| Graphite, InfluxDB Prometheus remote read | none |
Notes:
ClickHouse accepts
#comments too, but Trickster’s ClickHouse SQL parser doesn’t, so a statement using one is served from the object cache, or proxied, and its directives don’t apply.A PostgreSQL escape string (
E'...') is a string literal, so directive text inside one is not a directive.A Druid native query has no comments, so it carries directives as context keys, which Trickster leaves out of the cache key and Druid ignores:
"context": { "trickster-step-align": "partial_end", "trickster-volatile-window": "2m" }A Druid SQL query can use both. For the same directive, its SQL comment wins over its
context.
6.3 - Query Range Limits
Trickster supports enforcing limits on the maximum time range (duration) of incoming queries. This feature protects both the Trickster cache and the downstream origin TSDBs from resource-intensive, oversized queries.
Overview
When a client sends a query spanning a duration larger than the configured limit, Trickster intercepts the request early in the pipeline. It rejects the query, returns an HTTP 400 Bad Request status, and increments a metric counter. This helps prevent large queries (e.g., querying 1 year of data at 15s resolution) from causing out-of-memory (OOM) issues or heavy processing spikes.
Configuration
You can configure the query range limit on any backend by setting the max_query_range option. The value is a duration string (e.g., 1h, 1d, 14d).
Here is a configuration example:
backends:
prometheus_dev:
provider: prometheus
origin_url: http://prometheus-origin:9090
max_query_range: 14d
Setting max_query_range: 0 or omitting the field disables the range limit enforcement.
How the Range Is Measured
The limit is compared with the time range as the client sent it, before step alignment or any other adjustment Trickster makes to the range it sends to the origin. A query without an end time is measured to the time of the request.
Supported and Unsupported Backends
Query range limit enforcement is active on backends that implement the backends.TimeseriesBackend interface, which includes:
- Supported Backends:
- Prometheus
- InfluxDB
- ClickHouse
- Application Load Balancers (ALBs) using Time Series Merge (TSM)
- Unsupported Backends:
- Standard HTTP Reverse Proxy backends (e.g.,
reverseproxycache) that do not parse the query timespan.
- Standard HTTP Reverse Proxy backends (e.g.,
Rejection Behavior
When an incoming query’s duration exceeds the allowed max_query_range:
- HTTP Status: Trickster responds immediately with HTTP
400 Bad Request. - Error Message: The response body contains an error message, e.g.,
query time range exceeds the allowed limit of 336h0m0s. - Short-circuiting: Downstream origin TSDB instances are never contacted, conserving database resources.
Metrics and Logging
Trickster exposes metrics and logs to track rejected queries:
Metrics
- Metric Name:
trickster_proxy_query_range_rejected_total - Labels:
backend(name of the backend rejecting the query) - Type: Counter
- Description: Tracks the total number of queries rejected due to exceeding the
max_query_rangelimit.
Logs
Upon query rejection, Trickster emits a structured warning log including information about the offending query:
[WARN] query rejected due to max_query_range limit (backendName=prometheus_dev, limit=336h0m0s, duration=360h0m0s, clientIP=127.0.0.1, path=/api/v1/query_range, statement=up)
Use with ALBs
Application Load Balancer (ALB) backends configured with the Time Series Merge (TSM) mechanism can enforce query range limits at the ALB entry point. This ensures that Trickster rejects oversized queries before scattering them across downstream pool member backends.
For details on using max_query_range with TSM ALBs, refer to the ALB Documentation.
6.4 - Step Alignment
Dashboards ask for ranges like “the last 6 hours”, so the start and end of every request are a few seconds later than the last one. Time series databases group data into steps, or buckets, on a fixed grid, and a range that starts or ends between two grid points leaves a partial bucket at that edge: a bucket that holds only some of the rows in its span.
Step alignment is what Trickster does with those partial buckets. Other tools call the same idea “aligning queries with their step” (Thanos, Mimir, Cortex and Loki) or “dropping partial buckets” (Kibana). In those tools it is an on/off switch; Trickster offers several modes, chosen per backend and, when needed, per query.

Why It Matters
Trickster’s time series cache holds complete buckets only. A partial bucket’s value depends on exactly where the client’s range started or ended, so caching it as if it were complete would serve wrong values to the next dashboard that asks for that bucket. However a mode treats partial buckets, it never writes one to the time series cache. Complete buckets are what make repeat and overlapping requests fast.
Modes
| Mode | Start edge | End edge | What the client sees |
|---|---|---|---|
truncate | the whole bucket that contains the start | left out | points on the grid from the bucket containing the start; the first includes rows before the start |
drop | left out | left out | complete buckets only, all of them inside the requested range |
partial | the rows from the start to the next grid point | the rows from the last grid point to the end | what the origin would answer for the range: complete buckets plus both partial ones |
partial_start | as partial | as truncate | the origin’s first bucket, and no partial bucket at the end |
partial_end | as truncate | as partial | a live right edge, with the start snapped to the grid |
off | not applied | not applied | the origin’s own answer to the range as sent, cached as an object for one minute |
- A range whose start and end are on the grid gets the same answer in every mode.
truncateanddropdiffer only at the start: at the end both stop before the bucket the end falls in.- The live bucket, the one that contains the current time, is always
partial. It appears only in
partialandpartial_end, whose end edge is partial. Undertruncate,dropandpartial_starta range that runs to now, or has no end at all, stops before it. - A range with no complete bucket, one shorter than a step or one that
crosses a single grid point, is sent to the origin unchanged in every mode.
The origin’s answer is cached as an object for
partial_bucket_ttl, keyed on the range as sent. offsends each request to the origin with the range the client sent, and caches the response as an object for one minute under a key that includes that range. Identical requests share it, but dashboards with relative ranges rarely send the same range twice. Use it when a client needs the origin’s exact output, such as a conformance test, and the time series cache can’t provide it.
Example
With a one-minute step and a client range of 10:00:30 to 10:05:20:
| Mode | Buckets in the response |
|---|---|
truncate | 10:00 (whole) through 10:04 |
drop | 10:01 through 10:04 |
partial | 10:00 (rows from 10:00:30), 10:01 through 10:04, 10:05 (rows up to 10:05:20) |
partial_start | 10:00 (rows from 10:00:30), 10:01 through 10:04 |
partial_end | 10:00 (whole), 10:01 through 10:04, 10:05 (rows up to 10:05:20) |
In every mode, only the buckets 10:01 through 10:04 come from, and go into, the
time series cache. The whole 10:00 bucket that truncate returns is complete,
and is cached too.
How Partial Buckets Are Fetched
In the partial modes, each partial bucket is its own small query to the
origin, bounded by the client’s range and sent through the object cache:
- It is cached for
partial_bucket_ttl(15 seconds unless configured, and never more thanmax_ttl), keyed on its exact range. A partial bucket is short-lived: the next refresh usually asks for a different one. - Its rows are merged into the response after the cached complete buckets. They are never written to the time series cache.
- A partial end bucket never runs past the client’s end. When the end is written
as
now()or left open, the query is sent that way, so every request within the same bucket shares one object; an end written as a timestamp, as Grafana sends it, makes a new object on each refresh. - A partial bucket that fails is left out of the response, and reported as
err. - A fetch doesn’t change the request’s cache status (
hit,phitand so on). Each is reported in thepartial_bucketsfield of theX-Trickster-Resultheader.
Cost
The partial modes trade origin requests for fidelity at the edges. With a
Grafana dashboard refreshing a relative range, where both edges move on each
refresh:
| Refresh | truncate or drop | partial_end | partial |
|---|---|---|---|
| every complete bucket cached | 0 | 1 | 2 |
| a new bucket has closed since the last refresh | 1 | 2 | 3 |
| nothing cached for the query | 1 | 2 | 3 |
Counts are origin requests; a partial bucket that is still in the object cache
costs none. Each partial bucket query scans at most one bucket of raw rows,
though with a high-cardinality GROUP BY it still returns one row per series.
The defaults keep today’s behavior, so the extra requests are opt-in.
Prometheus And Fast Forward
A Prometheus range query evaluates at points rather than aggregating buckets, so it has no partial start bucket. Its modes are:
truncate: points from the grid point at or before the start.drop: points from the grid point at or after the start, so every point lies inside the range.partial_end, the default:truncate, plus the live point at the current time when the range reaches it. This is Trickster’s Fast Forward, and is reported asffstatus. It is skipped when the step is no longer thanpartial_bucket_ttl.off.
partial and partial_start aren’t supported. fast_forward_disable: true
selects truncate as the default, and a backend can’t set both it and
step_alignment. The trickster-fast-forward:off directive still turns Fast
Forward off for one query.
VictoriaMetrics
A VictoriaMetrics range query is served on the grid VictoriaMetrics itself evaluates it on: a
range of 50 or more points starts on a step boundary, and a shorter one keeps its own start.
Every point is on that grid, so truncate, the default, and drop answer alike. partial_end
isn’t offered, since VictoriaMetrics answers the live edge with its latency offset, not as an
instant query at the end. Expressions that depend on the whole requested range, such as
range_avg or topk_avg, run as off; a mode or directive naming another mode is counted as a
fallback. See the VictoriaMetrics Provider Guide.
Configuration
step_alignment is a backend option; when it is unset, the provider’s default
applies:
backends:
ch1:
provider: clickhouse
origin_url: http://clickhouse:8123
# truncate | drop | partial | partial_start | partial_end | off
step_alignment: partial_end
# how long a partial bucket is kept in the object cache
partial_bucket_ttl: 15s
- A mode the provider doesn’t support fails the configuration when it loads.
- Setting
offon aproxy_onlybackend logs a warning, because the backend caches nothing anyway. volatile_windowandvolatile_window_pointswork the same in every mode: they apply to complete buckets, the only ones the time series cache holds.- The object cache entries of
off, of ranges with no complete bucket, and of partial buckets are stored for requests that carry credentials, keyed on those credentials, as the time series cache stores them.
Per Query
A query can choose its own mode with the trickster-step-align directive, written
in a comment in the query’s own language:
sum(rate(http_requests_total[5m])) # trickster-step-align:drop
SELECT ... /* trickster-step-align:partial_end */
The directive always overrides the backend’s mode. When the query’s language or
shape doesn’t support the mode it names, the query runs in its default mode and
the fallback is counted in trickster_step_alignment_fallbacks_total. See
Per-Query Instructions for each language’s
comment syntax.
Load Balancers
On an ALB, step_alignment applies one mode to every pool member,
whatever the mechanism, so that members answer on one grid; for requests
through the ALB, it replaces each member’s own mode. Without it, only a tsm
ALB applies a mode.
- A
tsmALB without its ownstep_alignmentapplies its first configured member’s mode, so merged series line up. A leader outage doesn’t change it, because the first configured member is used, not the first healthy one. - For a static pool, a member that can’t apply the ALB’s mode fails the
configuration and is named in the error. When autodiscovered members can’t,
the pool falls back to
truncateif they all support it, and otherwise each member uses its own mode; either way a warning is logged and added to merged responses. - A query’s
trickster-step-aligndirective wins over the ALB’s mode when every member supports the mode it names. - At startup, Trickster warns about each ALB whose members would apply different modes.
Support by Backend
| Backend and query path | Supported modes | Default |
|---|---|---|
| Prometheus range queries | truncate, drop, partial_end, off | partial_end (truncate with fast_forward_disable) |
| Graphite render | truncate, off | truncate |
| InfluxDB InfluxQL (1.x and 3.x) | all | partial_end |
| InfluxDB Flux | truncate, off | truncate |
| InfluxDB SQL, over HTTP and Flight SQL | all | drop |
| InfluxDB Prometheus remote read | truncate | truncate |
| ClickHouse, over HTTP and the native protocol | all | drop |
| Druid native queries | all | partial |
| Druid SQL | all | drop |
| PostgreSQL and TimescaleDB | all | drop |
| MySQL | all | drop |
| GreptimeDB PromQL | truncate, drop, partial_end, off | partial_end |
| GreptimeDB SQL, over HTTP, MySQL and PostgreSQL | all | drop |
| QuestDB SQL over PostgreSQL | all | drop |
| VictoriaMetrics MetricsQL range queries | truncate, drop, off | truncate |
A backend accepts any mode one of its paths supports; a query on a path that
doesn’t support the configured mode runs in that path’s default. Graphite has no
partial buckets: whisper stores whole buckets and returns only those. Flux
labels windows by their stop time, so its partial buckets would land on
different labels; it supports truncate and off for now. Remote read returns
raw samples, which have no grid.
Observability
| Signal | What it reports |
|---|---|
partial_buckets in X-Trickster-Result | each partial bucket’s fetched range, edge and object cache status |
trickster_proxy_partial_bucket_fetches_total{backend_name, provider, edge, status} | partial bucket fetches, by edge (start or end) and status |
trickster_step_alignment_fallbacks_total{backend_name, requested, applied} | requests for a mode the query doesn’t support, served in its default mode |
span attribute step_alignment.mode | the mode a request was served in |
Troubleshooting
- The first and last points are missing. The backend’s mode is
drop, the SQL default.partialreturns what the origin would;truncatereturns a whole first bucket. - The first point includes data from before the range. The mode is
truncateorpartial_end, which return the whole bucket at the start.partialorpartial_startreturn only the rows inside the range. - Partial bucket fetches miss on every refresh. The query’s end is a
timestamp, as Grafana writes it, so each refresh asks for a different partial
bucket. That is expected: each is one small query. Ends written as
now()share one object per bucket. trickster_step_alignment_fallbacks_totalis rising. Queries, or a directive, are asking for a mode their path doesn’t support. Therequestedandappliedlabels name both.- A merged ALB response carries a warning about step alignment. Some autodiscovered pool members can’t apply the ALB’s mode; see Load Balancers.
6.5 - Timeseries Request Sharding
Overview
A shard means “a small part of a whole,” and Trickster supports the sharding of upstream HTTP requests when retrieving timeseries data. When configured for a given time series backend, Trickster will shard eligible requests by inspecting the time ranges needed from origin and subdividing them into smaller ranges that conform to the backend’s sharding configuration. Sharded requests are sent to the origin concurrently, and their responses are reconstituted back into a single dataset by Trickster after they’ve all been returned.
Mechanisms
Trickster support three main mechanisms for sharding:
- Maximum Timestamps Per Shard - Trickster calculates number of expected unique timestamps in the response by dividing the requested time range size by the step cadence, and then subdivides the time ranges so that each sharded request’s time range will return no more timestamps than the configured maximum.

- Maximum Time Range Width Per Shard - Trickster inspects each needed time range, and subdivides them such that each sharded request’s time range duration is no larger than the configured maximum.

- Epoch-Aligned Maximum Time Range Width Per Shard - Trickster inspects each needed time range, and subdivides them such that each sharded request’s time range duration is no larger than the configured maximum, while also ensuring that each shard’s time boundaries are aligned to the Epoch based on the configured shard step size.

Configuring
Maximum Unique Timestamp Count Per Shard
In the Trickster configuration, use the shard_max_size_points configuration to shard requests by limiting the maximum number of unique timestamps in each sharded response.
backends:
example:
provider: prometheus
origin_url: http://prometheus:9090
shard_max_size_points: 10999
Maximum Time Range Width Per Shard
In the Trickster configuration, use the shard_max_size_time configuration to shard requests by limiting the maximum width of each sharded request’s time range.
backends:
example:
provider: 'prometheus'
origin_url: http://prometheus:9090
shard_max_size_time: 2h
Epoch-Aligned Maximum Time Range Width Per Shard
In the Trickster configuration, use the shard_step configuration to shard requests by limiting the maximum width of each sharded request’s time range, while ensuring shards align with the epoch on the configured cadence. This is useful for aligning shard boundaries with an upstream database’s partition boundaries, ensuring that sharded requests have as little partition overlap as possible.
backends:
example:
provider: 'prometheus'
origin_url: http://prometheus:9090
shard_step: 2h
shard_step can be used in conjunction with shard_max_size_time, so long as shard_max_size_time is perfectly divisible by shard_step. This combination configuration will align shards against the configured shard step, while sizing each shard’s time range to be multiple shard steps wide.
backends:
example:
provider: 'prometheus'
origin_url: http://prometheus:9090
shard_step: 2h
shard_max_size_time: 4h
Neither shard_step or shard_max_size_time can be used in conjunction with shard_max_size_points.
6.6 - Providers
6.6.1 - Prometheus Support
Trickster fully supports accelerating Prometheus, which we consider our First Class backend provider. They work great together, so you should give it a try!
Most configuration options that affect Prometheus reside in the main Backend config, since they generally apply to all TSDB providers alike.
Supported API Endpoints
Trickster supports the full Prometheus HTTP API (v1), including features introduced in Prometheus 3.x.
Cached Endpoints
| Endpoint | Cache Strategy | Scatter/Gather Merge |
|---|---|---|
/api/v1/query_range | Delta Proxy Cache | Yes |
/api/v1/query | Object Proxy Cache | Yes |
/api/v1/series | Object Proxy Cache | Yes |
/api/v1/labels | Object Proxy Cache | Yes |
/api/v1/label/<name>/values | Object Proxy Cache | Yes |
/api/v1/alerts | Proxy + Merge | Yes |
/api/v1/targets | Object Proxy Cache | No |
/api/v1/targets/metadata | Object Proxy Cache | No |
/api/v1/rules | Object Proxy Cache | No |
/api/v1/alertmanagers | Object Proxy Cache | No |
/api/v1/status/* | Object Proxy Cache | No |
/api/v1/query_exemplars | Object Proxy Cache | No |
/api/v1/metadata | Object Proxy Cache | No |
/api/v1/format_query | Object Proxy Cache | No |
/api/v1/parse_query | Object Proxy Cache | No |
/api/v1/scrape_pools | Object Proxy Cache | No |
/api/v1/features | Object Proxy Cache | No |
Range Query Notes
- A
stepmay be fractional seconds (for example1.5or0.5), as Prometheus accepts; Trickster keeps the fraction and sends sub-secondstartandendvalues with millisecond precision. - A
query_rangewhose expression uses the@ start()or@ end()modifier is proxied without caching. Those modifiers resolve against each request’s own range, so results fetched for part of a range could not be combined. A fixed@ <timestamp>is cached normally.
Step Alignment
Range queries default to the partial_end step alignment mode: points on the step grid from the grid point at or before start, plus Trickster’s Fast Forward point at the current time when the range reaches it. Fast Forward’s point is fetched through the object cache for partial_bucket_ttl, and skipped when the step is no longer than that.
truncateis the same without Fast Forward. It is the default withfast_forward_disable: true, which can’t be set alongsidestep_alignment.dropstarts at the grid point at or afterstart, so no point falls before the requested range.offsends each range query as the client sent it, and caches the response as an object for one minute.partialandpartial_startaren’t supported: PromQL evaluates at instants, so there is no partial bucket at the start.
A query can choose its own mode, or turn Fast Forward off, with a comment: up # trickster-step-align:drop or up # trickster-fast-forward:off. See Per-Query Instructions.
Proxied Endpoints (not cached)
| Endpoint | Notes |
|---|---|
/api/v1/notifications/live | SSE streaming |
/api/v1/write | Remote write (v1 and v2) |
/api/v1/otlp/v1/metrics | OTLP ingestion |
/api/v1/admin/* | Explicitly unsupported (returns error) |
All other /api/v1/* paths are reverse-proxied to the origin without caching.
Prometheus 3.x Features
- Native histograms are fully supported in query and query_range responses, including mixed series with both float samples and histogram samples.
- UTF-8 metric and label names (e.g.,
{"metric.name"}) are supported in queries and cache keys. - Query stats (
stats=allparameter) are cache-key differentiated, so responses with and without stats are cached separately.
Flavors
prometheus.flavor adapts a backend to a Prometheus-compatible managed
service, applying its supported routes, limits and defaults.
| flavor | service |
|---|---|
cloudwatch | Amazon CloudWatch PromQL |
amp | Amazon Managed Service for Prometheus |
mimir | Grafana Mimir |
thanos | Thanos |
backends:
cloudwatch:
provider: prometheus
prometheus:
flavor: cloudwatch
sigv4:
region: us-east-1
The cloudwatch and amp flavors limit the backend to the routes the service supports and
answer every other path themselves. See AWS Integration for what
each one sets. The mimir and thanos flavors change no routes; they select
the series source that TSM global evaluation reads, and that
evaluation reads their newest 5 minutes of samples again each time, since
their replicas and ingesters may still be receiving them.
Series Sources
An ALB using TSM global evaluation reads raw
series from its members rather than query results. series_api selects how
a backend serves them, and series_endpoint where:
series_api | protocol | default for flavor | series_endpoint |
|---|---|---|---|
remote_read | Prometheus remote read over HTTP, as streamed chunks or sampled responses | mimir | optional; an absolute http or https URL. The default is the origin_url path plus /api/v1/read |
storeapi | Thanos StoreAPI Series over gRPC | thanos | required; grpc://host:port for plaintext, or grpcs://host:port for TLS with the backend’s tls client settings |
Any Prometheus backend can set series_api, since Prometheus itself serves
remote read. A StoreAPI endpoint can be a Thanos Sidecar, Store Gateway,
Receive or Querier. The cloudwatch and amp flavors serve no raw series, and
reject both options.
backends:
prom-a:
provider: prometheus
origin_url: http://prom-a:9090
prometheus:
series_api: remote_read
mimir:
provider: prometheus
origin_url: http://mimir:8080/prometheus # remote read at /prometheus/api/v1/read
prometheus:
flavor: mimir
thanos-a:
provider: prometheus
origin_url: http://prom-a:9090
prometheus:
flavor: thanos
series_endpoint: grpc://prom-a-sidecar:10901
Remote read uses the backend’s HTTP client settings, and sends the client’s
end-to-end request headers plus the request_headers the backend’s matching
path sets, as a proxied request would; a tenant header such as
X-Scope-OrgID, or an Authorization header, reaches the source this way.
StoreAPI calls carry the same headers as gRPC metadata.
Both transports ask for compressed answers, which mostly shrinks the label sets the chunks travel with. Remote read asks for gzip, which Mimir answers with; Prometheus answers remote read uncompressed. StoreAPI calls ask for snappy, as Thanos components support; a store that rejects it is read again at once without compression. After that read succeeds, compression stays disabled until Trickster restarts or reloads.
A backend opens its StoreAPI connection on its first read, and closes it once a reload, shutdown, or autodiscovery replaces the backend and its requests have drained, as described for global evaluation.
Injecting Labels
Trickster can inject labels on a per-backend basis into Prometheus responses before returning them to the caller.
Here is the basic configuration for adding labels:
backends:
prom-1a:
provider: prometheus
origin_url: http://prometheus-us-east-1a:9090
prometheus:
labels:
datacenter: us-east-1a
prom-1b:
provider: prometheus
origin_url: http://prometheus-us-east-1b:9090
prometheus:
labels:
datacenter: us-east-1b
Interaction with ALB Merge Strategy
When using label injection with an ALB configured for Time Series Merge, injected labels are automatically stripped from responses before merging. This ensures that series from different backends are aggregated correctly, and the injected labels do not appear in the final response to the caller. See the ALB Merge Strategy documentation for details.
Max Query Range Limitation
Trickster supports enforcing a max_query_range limit on Prometheus backends. For details on how to configure and use query range limits, see the Query Range Limits documentation.
6.6.2 - ClickHouse Support
Trickster will accelerate ClickHouse queries that return time series data normally visualized on a dashboard. Acceleration works by using the Time Series Delta Proxy Cache to minimize the number and time range of queries to the upstream ClickHouse server.
Scope of Support
Trickster is tested with the official ClickHouse DataSource Plugin for Grafana v4.21.1 and supports acceleration of queries constructed by this plugin using its built-in time macros like $__fromTime and $__toTime. Delta caching with this an other plugins (e.g., Altinity/Vertamedia) depends on the supported query shapes below. Trickster also supports several other query formats that return “time series like” data.
Trickster also supports the ClickHouse Go SDK (clickhouse-go/v2) over its HTTP and Native protocols, including clickhouse.OpenDB.
Native Binary Protocol Support
Inbound and upstream protocols are configured independently:
- A named listener with
protocol: clickhouseaccepts Native client connections. - A ClickHouse backend with
protocol: nativeuses the Native protocol for its origin. An empty protocol orprotocol: httpuses HTTP.
listeners:
default:
protocol: http
port: 8480
clickhouse-native:
protocol: clickhouse
port: 8487
backends:
click1:
provider: clickhouse
origin_url: http://clickhouse:8123
listener_names: [default, clickhouse-native]
cache_name: default
This exposes one HTTP route at /click1/ and one Native listener on port 8487, backed by the same cache and HTTP ClickHouse origin. A Native listener must map to exactly one backend; ALB and user-router multiplexing are not supported.
To use a Native origin instead, set the backend protocol and point origin_url at ClickHouse’s Native port:
backends:
click1:
provider: clickhouse
origin_url: http://clickhouse:9000
protocol: native
All four HTTP/Native ingress and HTTP/Native origin combinations are supported. Native credentials, database, query settings, parameters, and query IDs are forwarded to the origin. Native identity pools are bounded to 64 credential/database combinations per backend.
TLS
Native listener TLS uses the backend’s server certificate. Set require_tls: true and provide both certificate paths:
listeners:
clickhouse-native-tls:
protocol: clickhouse
port: 9441
tls_watch_interval: 30s
backends:
click1-tls:
provider: clickhouse
origin_url: http://clickhouse:8123
listener_names: [clickhouse-native-tls]
require_tls: true
tls:
full_chain_cert_path: /etc/trickster/tls/server.crt
private_key_path: /etc/trickster/tls/server.key
Certificate changes are hot-swapped. Since require_tls applies to every binding of a backend, use a separate backend entry when plaintext and TLS listeners must coexist.
For a TLS Native origin, use an https origin URL with the Native protocol. The common backend TLS options configure origin verification and optional mutual TLS:
backends:
click1:
provider: clickhouse
origin_url: https://clickhouse:9440
protocol: native
tls:
certificate_authority_paths: [/etc/trickster/tls/origin-ca.crt]
client_cert_path: /etc/trickster/tls/client.crt
client_key_path: /etc/trickster/tls/client.key
Native Limitations
The Native listener supports SELECT queries, ping/pong, revision 54460 framing, structured errors, and LZ4 compression. INSERT, external tables, and session-changing USE or SET statements are rejected; select a database and settings per request. ClientCancel is accepted as a no-op and does not asynchronously cancel an active upstream request.
Supported native protocol data types: all integer types (8–256 bit), Float32/64, String, FixedString(N), DateTime, DateTime64, Date, Date32, UUID, IPv4, IPv6, Enum8/16, Bool, Nullable(T), Array(T), Map(K,V), Tuple(T1,T2,…), LowCardinality(T), and Decimal.
Delta-cacheable query results are decoded into Trickster’s dataset model and re-encoded in the client’s requested format. Both the TSV and the FORMAT Native origin readers support every scalar type above, Nullable(T), LowCardinality(T), Array(T), Map(K, V) and Tuple(...) (including named elements and nesting); compound values are carried in ClickHouse’s text-literal form ([1,'a'], {'k':1}, ('a',1)) and parsed back when re-encoding to Native. Nested, Variant, Dynamic and JSON columns are rejected with an explicit error on the delta path rather than decoded incorrectly; such queries should use a non-delta-cacheable shape.
Native wire clients receive Native blocks. HTTP clients using a Native origin may request JSON, Native, CSV, or TSV-family output; compound columns in CSV/TSV and unsupported formats return an error. The modeler recognizes Native origin responses through the X-ClickHouse-Format response header, and HTTP Native framing honors client_protocol_version.
Trickster parses incoming ClickHouse statements into a full abstract syntax tree using the AfterShip ClickHouse SQL parser, then applies its own semantic analysis to determine whether a query is eligible for time series delta caching and, if so, its timestamp column, bucket cadence, time range, grouping tags, and cache identity. The cache key is derived from a canonical form of the query in which the requested time range is replaced with placeholders, so requests for different time ranges of the same logical series share one delta cache entry.
Trickster’s analysis fails closed: a valid query whose shape cannot be proven safe for delta caching is never rewritten approximately. It is instead served through the Object Proxy Cache (OPC) or proxied directly, and the classification reason is exported through the trickster_sql_query_analysis_total metric.
If you find query or response structures that are not yet supported, or providing inconsistent or unexpected results, we’d love for you to report those. We also always welcome any contributions around this functionality.
Delta-Cacheable Queries
To be eligible for the delta cache, a query must be a single SELECT statement containing a recognized time-bucketing expression in its select list, a supported time range in its WHERE or PREWHERE clause, and a GROUP BY clause that includes the time bucket. Each requirement is described below.
Time-Bucketing Expressions
Exactly one select-list expression must match a supported bucket form:
Grafana Plugin Format
SELECT intDiv(toUInt32(time_col), 60) * 60 [* 1000] [AS alias]
This is the approach used by the Grafana plugin. The argument to the ClickHouse intDiv function is the step value in seconds, since the toUInt32 function on a datetime column returns the Unix epoch seconds. An optional output multiplier of 1000, 1000000, or 1000000000 selects millisecond, microsecond, or nanosecond output timestamps.
ClickHouse Time Grouping Functions
SELECT toStartOfInterval(time_col, INTERVAL n unit) [AS alias]
with a positive constant n and a unit of millisecond, second, minute, hour, day, or week; or one of the fixed-period functions:
toStartOfNanosecond
toStartOfMicrosecond
toStartOfMillisecond
toStartOfSecond
toStartOfMinute
toStartOfFiveMinute
toStartOfTenMinutes
toStartOfFifteenMinutes
timeSlot
toStartOfHour
toStartOfDay
toStartOfWeek
toMonday
date_trunc('unit', time_col) and dateTrunc support the same fixed units. toStartOfWeek uses ClickHouse’s Sunday phase while toMonday and weekly date_trunc use Monday. Calendar-length month, quarter, and year buckets are served through the OPC.
The time column may be a plain, qualified (table.col), or quoted identifier, optionally wrapped in toDateTime, toInt32, or toUInt32. Integer constants defined in a scalar WITH clause may be used for the step value. Timezone-parameter variants of these functions (for example toStartOfHour(time_col, 'America/Denver')) are not eligible for delta caching and are served through the OPC. The current SQL parser does not accept INTERVAL MICROSECOND or INTERVAL NANOSECOND; use the corresponding fixed toStartOf... function.
Determining the Requested Time Range
Time range predicates must appear in a top-level AND conjunction of the WHERE or PREWHERE clause. Predicates joined by OR or negated with NOT make the query ineligible for delta caching.
Two predicate targets are supported, with different rules:
- The raw time column (the column inside the bucket function): the lower bound must be inclusive (
>=, or the lower end ofBETWEEN); a strict>is served through the OPC. The upper bound may be exclusive (<) or inclusive (<=, or the upper end ofBETWEEN). Values that do not fall on bucket boundaries — such as the live ranges produced by Grafana’s$__fromTimeand$__toTimemacros — leave partial buckets at the edges, which are never cached as complete aggregates; the backend’s step alignment mode decides what the response shows for them. Under the default,drop, they are left out, and an inclusive upper bound leaves out the bucket that contains it, because that bucket is only partly covered. A range with no complete bucket is sent to ClickHouse as written, and its response is cached as an object forpartial_bucket_ttl. - The bucket alias (the output of the bucket expression):
>,>=,<,<=, andBETWEENare all supported, because bucket outputs are discrete; Trickster aligns each comparator to the first and last included bucket.
Bound values may be expressed as epoch integers, ClickHouse string dates in the form 2006-01-02 15:04:05 (or date-only, or RFC3339), toDateTime(n), toDateTime64(n, precision), or toDate(n) wrappers, WITH-clause constants, or now()/now64() with optional addition or subtraction of seconds. DateTime64 precision is retained. Floating epoch bounds and timezone-qualified conversions such as toDateTime(n, 'America/Denver') are not eligible.
If no upper bound is present, Trickster inserts a safe upper bound into origin requests automatically and caches every complete bucket up to the current time. The still-filling bucket is never cached: under the default step_alignment, drop, the response ends before it, and the partial and partial_end modes fetch it from the origin on each request.
Examples of delta-cacheable time range clauses (for a one-minute bucket cadence):
WHERE t >= '2020-10-15 00:00:00' AND t <= '2020-10-16 12:00:00' -- bucket alias
WHERE t BETWEEN 1574686320 AND 1574689920 -- bucket alias
WHERE time_col >= toDateTime(1574686320) AND time_col < toDateTime(1574689920)
WHERE t >= now() - 3600 AND t < now() -- bucket alias
Secondary date-range predicates whose values match the primary range — such as the Date-typed partition filters emitted by the Grafana plugin — are recognized and rewritten in step with the primary range.
Grouping and Result Shape
The GROUP BY clause must include the time bucket (by alias or by its full expression), and every non-aggregate column in the select list must also be grouped. Grouped columns become the series tags in the cached time series. Queries using GROUP BY ... WITH CUBE/ROLLUP, grouping on expressions that are not selected, or leaving a selected dimension ungrouped are served through the OPC.
Queries whose values in one time bucket can depend on other buckets, or on the whole result, are also served through the OPC:
LIMIT,LIMIT BYandTOP;- window functions (
OVER (...)or aWINDOWclause) and cross-row functions such asneighbor,lagInFrame,leadInFrameand therunning*family; WITH TOTALSandORDER BY ... WITH FILL(includingINTERPOLATE);- a query-level
SETTINGSclause; - subqueries, common table expressions and joins, which can carry time filters of their own (
ARRAY JOINis still delta-cacheable).
Ordering
Trickster rebuilds a delta-cached response from its cached buckets in ascending time order. A query is therefore delta-cacheable only with no ORDER BY, or with a single ascending term on the time bucket (by alias, by the bucket expression, or by its position, as in Grafana’s ORDER BY time). Any other ordering — descending time, additional terms, or other columns — is served through the OPC, where the origin’s row order is kept.
Output Formats
Delta-cacheable queries may specify FORMAT JSON, CSV, CSVWithNames, TabSeparated (TSV), TabSeparatedWithNames, or TabSeparatedWithNamesAndTypes, or omit the FORMAT clause. Trickster requests TSVWithNamesAndTypes from the origin and re-marshals cached data into the client’s requested format.
Trickster writes each format as ClickHouse writes it:
- NULL is
\Nin TSV and CSV, andnullin JSON. An empty string stays an empty string. - Numbers are bare JSON numbers, and floats use ClickHouse’s text (
1e21,1e-7,nan,inf). JSON writes NaN and infinities asnull. - CSV quotes every text value. A
FixedStringis padded to its width. - Compound values:
Array,MapandTuplevalues are their ClickHouse literals in TSV and CSV, and JSON arrays and objects in JSON.
Responses honor output_format_json_quote_64bit_integers, output_format_json_quote_decimals, output_format_json_quote_denormals and date_time_output_format.
Time Zones
A DateTime without a zone in its type is written in the session’s time zone: the request’s session_timezone URL parameter, or else the server’s. Trickster caches UTC and writes each response in its client’s zone, so clients in different zones share cache entries.
The settings that change only how a response is written aren’t part of the cache key: date_time_output_format, the JSON quote settings, default_format, and client_protocol_version.
The zone is part of the key only when it changes which rows a query returns:
- Buckets that start on the zone’s clock: a daily bucket starts at the zone’s midnight, and an hourly bucket in a zone offset by a half hour starts on the half hour. The zone is keyed unless every offset it has over the queried range is a whole number of buckets. So hourly and finer buckets in New York or UTC share entries; Kolkata hourly buckets, and daily buckets outside UTC, don’t.
- Zone-sensitive expressions: another date or time function that reads the zone, such as
toHour()ortoday(), or text compared as a time.
A query whose time bounds are text (ts >= '2026-09-29 00:00:00') or dates (toDate(…)) is read by ClickHouse in the session’s zone. Outside UTC it’s served through the OPC rather than delta-cached.
Trickster asks the origin for date_time_output_format=iso, which writes every DateTime as UTC. Its cache holds UTC, without the ambiguity of the hour a clock repeats when daylight saving time ends. It writes its DateTime64 range bounds with an explicit 'UTC' zone, so the session’s zone doesn’t move them.
Trickster learns the server’s zone from the X-ClickHouse-Timezone header of the origin’s responses. A backend that hasn’t seen one yet asks the server with SELECT timezone() before it caches a query. If that probe fails, for example because the origin requires credentials, Trickster proxies the request uncached to learn the zone from the response. If the zone is still unknown after that, Trickster assumes UTC and logs a warning.
Non-Time-Series Queries
Queries that are not cacheable as time series — such as LIMIT-based queries, queries with set operations (UNION, EXCEPT, INTERSECT), SELECT 1 health checks, or SDK handshake requests — are transparently proxied to the upstream ClickHouse server. These requests are cached using the Object Proxy Cache (OPC).
Cache Keys
Both the OPC and the delta proxy cache key a request on its SQL statement and on every other URL parameter, because query parameters (param_<name> values for {name:Type} placeholders) and settings can change the result. The native listener forwards its query parameters and settings the same way. Only transport parameters that cannot change the result are left out of the key: query_id, session_timeout, session_check, send_progress_in_http_headers, http_headers_progress_interval_ms, wait_end_of_query, buffer_size, log_comment, log_queries, quota_key, and add_http_cors_header.
Requests that carry a session_id are proxied without caching. A session’s SET statements change the results of later queries in that session, and that state is not part of any cache key.
Health and Ping Endpoint
Trickster exposes a /ping endpoint that returns a health check response, matching the endpoint provided by ClickHouse itself. This enables compatibility with clients and SDKs that probe /ping during connection initialization.
Step Alignment
ClickHouse supports every step alignment mode, and defaults to drop: responses hold complete buckets only, so the partial buckets at the edges of a live range, and the still-filling bucket at the present, are left out. truncate answers the whole first bucket instead.
partial, partial_start and partial_end add the partial buckets as ClickHouse computes them over the client’s own range. Each is a small query of its own, sent through the object cache and kept for partial_bucket_ttl, never in the time series cache, so these modes cost up to two extra origin queries per request. Fast Forward doesn’t apply to ClickHouse; partial_end shows the still-filling bucket instead.
Complete buckets inside the configured volatile_window are refetched until they settle. A query can choose its own mode or window with a comment directive, such as /* trickster-step-align:partial_end */; see Per-Query Instructions. Directives in # comments aren’t read, because Trickster’s ClickHouse SQL parser rejects them: a statement with one is served through the OPC, or proxied.
Observability
Query classification outcomes are exported through two low-cardinality metrics that never include query text:
trickster_sql_query_analysis_total— labeled by backend, dialect, cache mode (delta,object,none), and a stable reason code such asdelta_cacheable,unsafe_predicate, orunsupported_bucket.trickster_sql_query_rewrite_failures_total— counts failures to render an origin request from a cached query plan.
With debug logging enabled, classification decisions are also logged with the same structured reason codes.
Max Query Range Limitation
Trickster supports enforcing a max_query_range limit on ClickHouse backends. For details on how to configure and use query range limits, see the Query Range Limits documentation.
6.6.3 - InfluxDB Support
Trickster provides support for accelerating InfluxDB queries that return time series data normally visualized on a dashboard. Acceleration works by using the Time Series Delta Proxy Cache to minimize the number and time range of queries to the upstream InfluxDB server.
Scope of Support
Trickster is tested with the built-in InfluxDB DataSource Plugin for Grafana v5.0.0.
Trickster uses InfluxDB-provided packages to parse and normalize queries for caching and acceleration. If you find query or response structures that are not yet supported, or providing inconsistent or unexpected results, we’d love for you to report those so we can further improve our InfluxDB support.
Trickster supports integrations with InfluxDB 1.x, 2.x, and 3.x.
Prometheus Remote Read
For InfluxDB 1.x, Trickster accelerates Prometheus remote-read requests sent to
POST /api/v1/prom/read. Point Prometheus at the same endpoint on Trickster and
retain the InfluxDB query parameters, for example:
remote_read:
- url: http://trickster:8480/api/v1/prom/read?db=metrics&rp=autogen
The InfluxDB endpoint supports one query per request and the Prometheus
SAMPLES response type. Trickster delta-caches requests with that shape. Other
request shapes continue through the normal proxy path so that InfluxDB remains
responsible for its response and error behavior.
Raw remote-read samples do not advertise a guaranteed interval, so point-count
sharding cannot split their extents without risking gaps. Requests use the
normal proxy path when shard_max_size_points is enabled; time-based sharding
remains supported. Cache retention (oldest and lru) and volatile_window_points
use a positive hints.step_ms; a request without that hint uses the
normal proxy path, including Prometheus instant queries with a zero step hint.
The hint is retained in serialized cache entries and does not change the 1 ms
precision used to locate missing raw samples.
Cacheable remote reads request Snappy directly from InfluxDB to avoid an extra
HTTP compression layer. Reconstructed responses carry Content-Encoding: snappy
on cache misses, partial hits, and full hits.
The db, rp, u, and p query parameters and the Authorization header are
part of the cache identity. They are forwarded unchanged unless a path-level
request rewrite or header configuration overrides them.
InfluxDB 3.x Support
Trickster supports InfluxDB 3.x via both the native v3 API endpoints and the v1/v2 compatibility endpoints.
Supported v3 Endpoints
GET/POST /api/v3/query_sql— SQL queries with delta-proxy cachingGET/POST /api/v3/query_influxql— InfluxQL queries with delta-proxy cachingPOST /api/v3/write_lp— line protocol writes (proxied, not cached)
All other v3 API paths (including the /api/v3/configure/* management endpoints, with any HTTP method) are proxied through untouched.
Query requests on both query endpoints may arrive as URL parameters (GET), an application/json document ({"q": ..., "db": ..., "format": ..., "params": ...}), a form-encoded body, or a raw statement body — all four shapes are parsed and cached equivalently, and the db and params values participate in the cache identity.
InfluxQL over v3
/api/v3/query_influxql requests use the same delta-proxy caching as the v1 /query endpoint (queries with a GROUP BY time(...) interval, optionally with an offset such as time(1h, 15m), and a time-bounded WHERE clause; a non-UTC tz() clause shifts bucket boundaries by the zone offset, so those queries use the object proxy cache instead), but speak the v3 request/response shapes: the v3 request document above and the v3 tabular response formats, including the iox::measurement column, which Trickster treats as a series tag alongside any GROUP BY tags.
On both /query and /api/v3/query_influxql, statements whose values in one time bucket depend on other buckets or on the whole result use the object proxy cache instead of delta caching: fill(previous) and fill(linear); transformations such as derivative(), non_negative_derivative(), difference(), moving_average(), cumulative_sum(), elapsed(), integral() and the other technical analysis functions; LIMIT, OFFSET, SLIMIT and SOFFSET; and subqueries, whose own time ranges are not rewritten per fetch.
SQL Query Caching
SQL queries using date_bin() or date_trunc() for time binning are parsed for time range extraction and step detection, enabling delta-proxy caching. Trickster extracts time ranges from WHERE clauses and intervals from date_bin(INTERVAL '...', time) or date_trunc('unit', time) in SELECT. Live, unaligned time ranges (the shape dashboard tools emit) are rounded inward to complete buckets. Grouped queries (GROUP BY 1, host) are cached per tag series.
Example query that Trickster will accelerate:
SELECT date_bin(INTERVAL '1 hour', time) AS time, avg(temperature)
FROM weather
WHERE time >= '2024-01-01 00:00:00' AND time < '2024-01-02 00:00:00'
GROUP BY 1
SELECT queries that cannot be delta-cached (no fixed-cadence time bucket, joins, subqueries, window functions, compound selects such as UNION, LIMIT, variable-length buckets like '1 month', or unsafe time predicates) fall back to the object proxy cache, which caches the whole response briefly and passes results through unchanged. Non-SELECT statements and parameterized queries (a params field in the request) are proxied to the origin without delta caching.
The still-filling final bucket of a range that reaches the present is never cached. Under the SQL default step_alignment, drop, the response ends before it; the partial and partial_end modes, and InfluxQL’s default partial_end, fetch it from the origin on each request. volatile_window applies only to complete buckets.
Step Alignment
Each query language has its own step alignment support and default:
| Query language | Supported modes | Default |
|---|---|---|
| InfluxQL, 1.x and over 3.x | all | partial_end |
| SQL, over HTTP and Flight SQL | all | drop |
| Flux | truncate, off | truncate |
| Prometheus remote read | truncate | truncate |
A backend’s step_alignment may name any of these modes; a query in a language that doesn’t support it runs in that language’s default, and the fallback is counted in trickster_step_alignment_fallbacks_total. The partial modes fetch each partial bucket as a small query of its own through the object cache, so they cost up to two extra origin queries per request. A query can choose its own mode with a comment: -- trickster-step-align:drop in InfluxQL and SQL, // trickster-step-align:off in Flux. See Per-Query Instructions.
Response Formats
Trickster supports the following v3 response formats, controlled by the format query parameter (in the URL or the request document) or, when no format is given, the Accept header (application/json, application/jsonl, text/csv, …):
json(default) — JSON array of objectsjsonl— JSON Lines (one JSON object per line)csv— standard CSV with header row
The parquet and pretty formats are not supported for caching and will be proxied through.
InfluxDB 3 leaves a row’s null values out of JSON and JSON Lines, so a column can first appear on any row. Trickster reads every column a response names, on whichever row first names it.
v1/v2 Compatibility
InfluxDB 3.x ships with v1 and v2 compatibility endpoints. Trickster’s existing InfluxQL support works against these endpoints with no additional configuration — just point Trickster at the v3 instance and query via /query.
Flight SQL (gRPC)
InfluxDB 3.x exposes SQL via Apache Arrow Flight SQL on gRPC in addition to HTTP. Grafana’s InfluxDB datasource in SQL mode, the Python/Rust/Java SDKs, and ADBC all default to Flight SQL, so HTTP-only caching misses a significant fraction of real-world query traffic. The protocol-level (backend-agnostic) listener documentation lives in Apache Arrow Flight SQL Listeners; this section covers the InfluxDB specifics.
Trickster can expose a Flight SQL server that proxies to an upstream Flight SQL endpoint and caches the Arrow IPC byte stream. Enable it by defining a listener with the flight-sql protocol and mapping exactly one InfluxDB backend to it (alongside its usual HTTP listener):
listeners:
influx3-flight:
protocol: flight-sql # Apache Arrow Flight SQL over gRPC
port: 8485
backends:
influx3:
provider: influxdb
origin_url: 'http://influxdb3:8181/'
listener_names: [default, influx3-flight]
influxdb:
flight_upstream_address: 'influxdb3:8181' # optional, defaults to origin_url host
flight_max_response_bytes: 134217728 # optional, defaults to 128MiB
flight_max_buffered_bytes: 536870912 # optional, aggregate in-flight budget; defaults to 512MiB
flight_allowed_location_hosts: # optional exact authorities for alternate endpoints
- 'influxdb3-replica:8181'
The authorization and database headers are forwarded from the client through to the upstream, and every cache entry is keyed by the backend name, the database/bucket-name headers, and a hash of the authorization header — so tenants and databases never share cache entries. Requests that send no authorization header share one anonymous cache scope, mirroring the access the upstream would grant them.
Statement queries are served through a three-tier cache:
- Delta proxy cache — queries the SQL analyzer classifies as delta-cacheable (the same
date_bin()/date_trunc()shapes as the HTTP path) are cached by time extent: repeat and overlapping queries fetch only the missing sub-ranges from the upstream, and responses are rebuilt into Arrow record batches conforming to the response’s original schema. A query’sORDER BYis carried through that rebuild, so a cache hit returns rows in the requested order; ordering terms that do not resolve to a select-list output fall to the next tier. Entries use the backend’stimeseries_ttland honorvolatile_window. Under the defaultstep_alignment,drop, a range holding at least one complete bucket returns only complete buckets: partial buckets at either end of the range, including the still-filling bucket of a range that reaches the present, are left out. Thepartial,partial_startandpartial_endmodes fetch those buckets from the upstream over the client’s own range, through the object cache forpartial_bucket_ttl, and never delta-cache them. A range with no complete bucket gets the upstream’s own result for the statement, partial and still-filling buckets included, cached as an object forpartial_bucket_ttl. Responses whose Arrow schemas the delta model cannot represent (nested types, non-string dictionaries, …) automatically fall to the next tier. - Object cache — everything else cacheable is stored as the verbatim Arrow IPC byte stream, returned byte-identically, with a lifetime of
influxdb.flight_cache_ttl(default 60s). Metadata RPCs and prepared statements always use this tier. - Proxy — statements referencing nondeterministic functions (
now(),current_timestamp,random(), …) and non-SELECT statements are never cached.
Flight SQL TLS
The Flight listener serves TLS when its mapped backend’s tls block presents a certificate and key (the same mechanism HTTP listeners use); certificate rotation applies on config reload without dropping the listener. Without a certificate the listener serves plaintext gRPC. The upstream dial uses TLS when influxdb.flight_upstream_tls is set, honoring the backend tls block’s insecure_skip_verify.
Unsupported Flight SQL RPCs
Trickster proxies queries, result-set schema requests (GetSchemaStatement/GetSchemaPreparedStatement, which ADBC drivers probe on connect), the metadata RPCs below, and the prepared-statement lifecycle. Other Flight SQL RPCs — writes/updates/ingest (ExecuteUpdate, DoPut ingestion), transactions, savepoints, Substrait plans, query cancellation, endpoint renewal, and session options — return gRPC Unimplemented; clients requiring them should connect to InfluxDB directly. A prepared-statement binding must arrive as a single record batch; a client streaming several batches for one binding receives Unimplemented rather than a binding taken from its first batch alone.
Flight SQL listeners share Trickster’s standard listener lifecycle: connection limits (connections_limit), graceful drain on SIGTERM (active streams are drained until the configured drain timeout, then closed), and config reload (SIGHUP) — a reload with an unchanged backend configuration keeps serving on the existing socket, while a changed configuration drains the old server and rebinds.
Metadata RPCs
ADBC clients (Grafana’s SQL datasource, the Python adbc_driver_flightsql, etc.) probe the following RPCs on connect to populate schema browsers and query editors. Trickster proxies these to the upstream and caches the Arrow IPC response:
GetFlightInfoTables/DoGetTablesGetFlightInfoCatalogs/DoGetCatalogsGetFlightInfoSchemas/DoGetDBSchemasGetFlightInfoTableTypes/DoGetTableTypesGetFlightInfoSqlInfo/DoGetSqlInfoGetFlightInfoXdbcTypeInfo/DoGetXdbcTypeInfoGetFlightInfoPrimaryKeys/DoGetPrimaryKeysGetFlightInfoImportedKeys/DoGetImportedKeysGetFlightInfoExportedKeys/DoGetExportedKeysGetFlightInfoCrossReference/DoGetCrossReference
Flight SQL response size
Flight responses are buffered whole so they can be cached. influxdb.flight_max_response_bytes bounds one upstream response (128MiB by default), while influxdb.flight_max_buffered_bytes bounds response bytes concurrently assembled or streamed (512MiB by default). Exceeding either bound returns gRPC ResourceExhausted; a negative value removes that bound.
A FlightInfo partitioned across several endpoints is consumed in full. Alternate endpoint locations are followed only when their exact host:port authority appears in influxdb.flight_allowed_location_hosts, because the request’s authorization and database metadata are forwarded to that host. A TLS upstream never follows a plaintext alternate location. influxdb.flight_max_location_clients bounds cached alternate connections and defaults to 16.
Prepared Statements
Trickster proxies the full prepared statement lifecycle: CreatePreparedStatement, DoPutPreparedStatementQuery (parameter binding), DoGetPreparedStatement, ClosePreparedStatement. A parameterless prepared statement is equivalent to executing its statement text, so it is served through the same three cache tiers as plain statement queries — delta caching included — and shares their cache entries. Executions with bound parameters are cached whole, keyed by the bound parameter hash so two clients running the same statement with different values don’t alias. Prepared statements abandoned by disconnected clients are closed upstream after 15 minutes of inactivity.
Note: InfluxDB 3 Core 3.10 reports a parameter schema at prepare time but does not resolve bound values during query planning (upstream limitation). Parameterless prepared statements work end-to-end; parameterized ones require a newer Core or Enterprise build.
Flux Language Support
Trickster supports the Flux Query Language for general/basic usage with InfluxDB 1.x and 2.x. Flux is not supported in InfluxDB 3.x.
The delta-proxy cache accepts now() as a range() bound and handles queries with aggregateWindow(every: ...) – the common Grafana shape. Multi-table Flux CSV responses (one table per series in the result set) are also read correctly.
InfluxDB answers Flux queries only in annotated CSV. Trickster can also answer them in JSON when the request’s Accept header asks for application/json. Each table’s records are written as objects, and a NaN or infinite value is written as null, as InfluxDB 3’s JSON writes it.
aggregateWindow may also set offset and timeSrc ("_stop", the default, or "_start"); Trickster aligns its fetches to the resulting window boundaries and labels. Queries whose windows cannot be mapped onto a fixed step grid are proxied without delta caching: a bare window() (its records keep their own _time), a period that differs from every, or a location.
Queries whose values in one window depend on other windows, or on the whole range, are also proxied without delta caching:
limit(),tail(),derivative(),difference(),cumulativeSum(),movingAverage()and the other moving-average and technical analysis functions,elapsed(),integral(),stateCount()andstateDuration();fill(usePrevious: true);timeShift(), which moves labels off the window grid;- reducers, selectors and
sort()that followaggregateWindow(), such asmax(),last()ortop(), which collapse or reorder every window; - more than one
from(), as in union and join queries.
Trickster currently does not properly handle schema changes within a response CSV body (e.g., multiple CSVs in the same document with their own #annotation and header rows). We will fully support this use case in a future beta.
Max Query Range Limitation
Trickster supports enforcing a max_query_range limit on InfluxDB backends. For details on how to configure and use query range limits, see the Query Range Limits documentation.
6.6.4 - Apache Druid Support
Trickster accelerates eligible Apache Druid native JSON queries with the Time
Series Delta Proxy Cache (DPC). Configure druid as the backend provider and
point origin_url at a Druid Broker or Router.
backends:
druid1:
provider: druid
origin_url: http://druid-router:8888
cache_name: default
volatile_window: 60s
timeseries_retention_factor: 2048
Native query acceleration
POST /druid/v2 uses delta caching when all of these conditions hold:
queryTypeistimeseries,groupBy, ortopN.intervalscontains exactly one ISO-8601 half-open interval.granularityhas a fixed width:- a simple granularity from
secondthroughday; - a positive
durationgranularity in milliseconds; or - a fixed ISO-8601
periodgranularity in UTC, such asPT15MorP1D.
- a simple granularity from
- The selected context does not request an alternate native response shape.
Trickster removes the interval from the logical cache identity and rewrites
only missing extents into Druid’s [start,end) form. Druid’s end is exclusive,
so the final cached bucket is rendered as extent.End + granularity.
An interval boundary inside a bucket is handled by the backend’s
step_alignment, which defaults to partial for native queries: complete
buckets come from the delta cache, and each edge bucket is Druid’s answer over
the part of it the interval covers, fetched through the Object Proxy Cache for
partial_bucket_ttl and never delta cached. A native query can choose its own
mode, or its own volatile window, with keys in its context map, such as
"trickster-step-align": "drop", which Trickster leaves out of the cache key
and Druid ignores. See Step Alignment and
Per-Query Instructions.
The response model preserves native timeseries, groupBy, and topN JSON
shapes. Grouping dimensions become DataSet tags internally, and per-bucket
positions keep groupBy and topN rows in Druid’s order. A descending: true
query is cached in time order like any other and written newest first, as Druid
writes it. See Response fidelity.
Object-cache fallback
A valid read query that is unsafe for delta merging automatically uses the Object Proxy Cache (OPC) with a one-minute fallback TTL when Druid does not provide explicit freshness headers. This includes:
- other native query types such as
scan,search,segmentMetadata,datasourceMetadata, andtimeBoundary; - multiple intervals;
all,none,week,month,quarter, andyearsimple granularities;- calendar-width periods or period granularities in a non-UTC time zone;
- timeseries
limit, which keeps only the first rows of the whole result; - groupBy limits or dimension-first result ordering; and
- response-changing contexts such as
bySegment,serializeDateTimeAsLong, timeseriesgrandTotal, or groupByresultAsArray.
The following native context keys are transport controls and are omitted from
the cache identity: queryId, sqlQueryId, priority, timeout, and
queryDeadline. They remain unchanged in the request sent to Druid. Semantic
context keys, including skipEmptyBuckets, remain part of the cache identity.
Druid SQL acceleration
POST /druid/v2/sql uses the same delta cache for a deliberately conservative
subset of Druid SQL. The request must be a JSON object using either the default
or explicit resultFormat: "object", or resultFormat: "array" with
header: true. The statement must be a single-table SELECT that has:
- one
TIME_FLOOR(__time, <fixed UTC period>)bucket expression with an explicit alias; - a
GROUP BYcontaining that bucket and every selected dimension; and - a complete lower/upper time range on
__time(unaligned edges follow the backend’sstep_alignment,dropby default for SQL; partial edge buckets are never delta cached). A SQL comment such as-- trickster-step-align:partialchooses the query’s own mode, and wins over the same key in the request’scontext.
The shared CockroachDB SQL analyzer canonicalizes the statement and renders
each missing extent while preserving the original JSON context on the wire.
Literal MILLIS_TO_TIMESTAMP(...) bounds generated by the Grafana Druid plugin
are supported. Object rows and header-plus-array rows are converted to the
standard Trickster DataSet internally and emitted in their requested shape
after cache merging. Other valid SELECT statements and response formats
remain safe OPC fallbacks; non-read statements, SQL task requests, and malformed
requests are proxied.
Response fidelity
A response built from the cache is written the way Druid writes it:
- Numbers, objects and arrays are written exactly as Druid wrote them, so a
double keeps its form, such as
5319.0or9.999999999999999E22. - Strings and keys are escaped the way Druid’s JSON writer escapes them.
- Each row’s members are in Druid’s order. For
timeseriesandgroupByresults, that is the order of Druid’s Java hash maps. - Native responses end without a newline, and SQL responses end with one.
- SQL rows without an
ORDER BYfollow Druid’s order: by theGROUP BYcolumns, with the time bucket first unless theGROUP BYends with it. - Rows that tie on an
ORDER BYalso follow Druid’s order. - Nulls sort lowest, as in Druid: first when ascending and last when descending.
Two things can still differ from the response Druid builds for the same request, and both also vary between Druid’s own responses:
- topN key order. Druid orders a topN row’s keys differently depending on whether its segment cache served the row, so one query can come back with either order. Trickster writes the dimension, then the aggregations, then the post-aggregations.
- Floating-point sums. Druid adds a bucket’s values in an order that
depends on the query’s interval. So the last digits of a
doubleSumorfloatSumcan differ between two Druid queries that cover the same bucket. A cached bucket keeps the value from the query that fetched it.
Route policy
| Route | Method | Policy |
|---|---|---|
/druid/v2 | POST | DPC when eligible, otherwise OPC or proxy |
/druid/v2/sql | POST | DPC for eligible SQL, otherwise OPC or proxy |
/druid/v2/sql/task | POST | Proxy only |
/druid/v2/datasources... | GET | OPC |
/status/health | GET | Health probe; expects true |
| all other routes | any | Proxy only |
SQL ingestion and management endpoints, ALB time-series merging,
scan/search delta caching, and Fast Forward are not supported. Fast Forward
is disabled for every Druid backend. A 60-second volatile_window is used when
the backend does not configure one, so recently ingested buckets can be
refreshed before segments settle.
Observability
trickster_druid_query_analysis_totalcounts classifications by backend, cache mode (delta,object, orproxy), and stable reason code.trickster_druid_query_rewrite_failures_totalcounts failed missing-extent rewrites by backend and fixed failure category.
Neither metric includes query text, datasource names, or request IDs.
6.6.5 - Graphite Provider
Trickster accelerates Graphite’s render API with the Delta Proxy Cache: it caches each metric at the origin’s native resolution and, on subsequent requests, fetches only the time ranges it does not already hold.
Graphite makes that harder than most time series databases, because a render
request does not say what resolution it will be answered at. Whisper chooses an
archive based on how old the query’s from is, and no API exposes a metric’s
archive ladder. Trickster therefore learns each ladder by probing the origin,
caches what it learns, and accelerates only the requests whose resolution it can
predict. Everything else is served correctly through ordinary object caching.
Specify graphite as the provider:
backends:
graphite1:
provider: graphite
origin_url: 'http://graphite.example.com:80'
cache_name: default
That is the whole minimum configuration. Graphite’s cache sizing differs from every other provider’s, so the provider supplies its own defaults rather than asking operators to discover them; see Sizing for what they are and when to change them.
Compatibility
Tested against graphite-web 1.1.10 with Whisper storage, and written against
its source. It should work with any Graphite-protocol origin that serves
/render and /metrics/expand, including go-carbon’s carbonserver,
graphite-clickhouse and carbonapi, but only graphite-web is verified.
| Client | Status |
|---|---|
| Grafana’s Graphite data source | Verified, GET and POST form bodies |
Direct /render API clients | Verified |
| Graphite Composer / dashboard UI | Proxied, not accelerated (image formats) |
| Response format | Behavior |
|---|---|
json | Accelerated. Also what Trickster requests upstream. |
raw, csv, msgpack | Accelerated; rendered from the cached series |
png, svg, pdf, pickle, dygraph, rickshaw | Proxied and object-cached |
Trickster always requests format=json from the origin regardless of what the
client asked for, and renders the client’s format from the cached series. JSON
is the only render format that carries the tags object, which Grafana relies
on, and it is what makes one cached series able to serve every output format.
Configuration
Every common backend option applies (cache_name, timeseries_ttl,
timeseries_retention_factor, volatile_window, max_object_size_bytes,
timeout, healthcheck, paths, TLS, authenticators); two of them,
max_object_size_bytes and timeseries_retention_factor, take
Graphite-specific defaults, as Sizing explains.
Per-path request_headers and request_params apply to the synthetic
requests resolution makes (probes, wildcard expansion) as well as to proxied
traffic, so an origin behind a static API key or one that needs local=1
works for learning too. What cannot work is per-client upstream identity:
resolution state (learned ladders, wildcard expansion tokens, the registry
generation) is backend-wide, and synthetic requests carry only the backend’s
configured identity. Trickster therefore declines acceleration — the
request is served correctly through the object cache, which keys on the
client’s Authorization and every declared result-affecting input —
whenever a render request carries an Authorization header, a header named
in the render path’s cache_key_headers, or a parameter or form field
named in its cache_key_params/cache_key_form_fields (such as local,
which selects the clustered origin view) that the path’s
request_headers/request_params does not statically override, and
whenever the render path or the backend itself has a request rewriter
(req_rewriter_name) attached, since a rewriter can change the upstream
host, path, headers, or parameters in ways synthetic requests do not see. The fallback is counted as
trickster_graphite_fallbacks_total{reason="client_identity"}.
If every client shares one upstream identity — Grafana with a configured
datasource credential is the common case — supply it in the graphite
block to re-enable acceleration:
backends:
graphite1:
provider: graphite
origin_url: 'http://graphite.example.com:80'
graphite:
origin_username: 'metrics'
origin_password: '<the origin password>'
For origins using a non-Basic scheme, set origin_authorization to the
verbatim header value (e.g., 'Bearer <token>') instead of the
username/password pair.
The credential becomes the Authorization request header of every path —
proxied client traffic, synthetic resolution requests (probes, wildcard
expansion) and the default health check alike — so the origin sees one
static identity, a client-supplied Authorization header is replaced
rather than forwarded, and requests accelerate regardless of what the
client sent. Cached objects are keyed on the configured credential, so
rotating it invalidates previously cached responses and learned resolution
state rather than serving the old tenant’s data. To authenticate
Trickster’s own clients as well, attach an
authenticator via authenticator_name: the
validated client credential is stripped before proxying and the origin
credential is applied in its place.
A path config that sets its own Authorization in request_headers (or
removes it with -Authorization) overrides the origin credential for that
path. Appending one with +Authorization alongside an origin credential is
rejected at startup, since the effective header would be ambiguous. If
per-path identities then diverge across the paths resolution
uses — /render and /metrics/expand selecting different
request_headers/request_params for synthetic GETs, or GET and POST
/render pinned to different credentials — Trickster declines
acceleration rather than mix upstream namespaces, counted as
trickster_graphite_fallbacks_total{reason="resolution_identity"}.
The graphite block adds provider-specific options:
backends:
graphite1:
provider: graphite
origin_url: 'http://graphite.example.com:80'
graphite:
time_zone: UTC
passthrough_max_data_points: false
find_cache_ttl: 1m
resolution_registry:
ttl: 24h
negative_ttl: 30s
max_entries: 100000
persist: true
probe_concurrency: 2
probe_budget: 96
static_retentions:
- pattern: '^collectd\.'
retentions: '10s:6h,1m:7d,10m:5y'
| Option | Default | Purpose |
|---|---|---|
time_zone | UTC | Interprets date-anchored from/until values (midnight, today, MM/DD/YY) when the request has no tz parameter. Set it to the origin’s graphite-web TIME_ZONE, or those queries resolve to different instants than the origin uses. |
origin_username, origin_password | unset | Static HTTP Basic credential sent as the Authorization header of every upstream request — proxied, synthetic and health check alike. See Configuration above. |
origin_authorization | unset | Verbatim Authorization header value for non-Basic schemes (e.g., Bearer <token>); mutually exclusive with origin_username/origin_password. |
passthrough_max_data_points | false | Send the client’s maxDataPoints upstream instead of consolidating in Trickster. See maxDataPoints. |
find_cache_ttl | 1m | How long a wildcard target’s expansion to concrete metric paths is reused. Lower it where metrics appear and disappear frequently. |
max_targets_per_request | 128 | Renders carrying more targets than this are served through the object lane, untouched, before any parsing or per-target fan-out. |
max_target_length | 16384 | A render with a longer single target expression (in bytes) is likewise served through the object lane untouched. |
max_expanded_leaves | 4096 | A wildcard whose expansion matches more leaf paths than this is served through the object lane; the refusal is remembered for find_cache_ttl. |
max_expansion_bytes | 2097152 | Bounds one expansion’s aggregate decoded leaf-name bytes the same way. |
resolution_registry.ttl | 24h | How long a learned ladder is trusted. Whisper ladders change only when an operator runs whisper-resize.py, so this is deliberately long. |
resolution_registry.negative_ttl | 30s | Initial backoff after a failed resolution. Doubles per consecutive failure, capped at 10m. |
resolution_registry.max_entries | 100000 | Bounds each registry layer; least-recently-used entries are evicted. |
resolution_registry.persist | true | Write learned ladders through to this backend’s cache so a restart does not relearn them. |
resolution_registry.probe_concurrency | 2 | Simultaneous ladder-learning runs. |
resolution_registry.probe_budget | 96 | Probes one learning run may issue before giving up. |
static_retentions | empty | Seed and override, from your storage-schemas.conf. See Static retentions. |
Health checks
The default health check is GET /metrics/find?query=*, which every
Graphite-protocol implementation serves. /version is not used because
carbonapi and graphite-clickhouse answer it differently.
A configured origin credential rides on the probe even when the
healthcheck block declares custom headers. To override it, set your own
Authorization value there; to send the probe unauthenticated, set
Authorization: ''.
Routed paths
| Path | Handling |
|---|---|
/render (GET, POST) | The render handler: delta cache, or object cache when the request is not accelerable |
/metrics/find, /metrics/expand, /metrics/index.json, /tags, /tags/* | Object-cached, 30s. These also back resolution’s own lookups, so caching them lowers probe cost. |
/functions, /version | Object-cached, 1h |
| everything else | Proxied |
How resolution prediction works
Why it is necessary
Whisper picks an archive from the age of the query’s left edge, not from the width of the window:
diff = now - fromTime
for archive in header['archives']:
if archive['retention'] >= diff:
break
step = archive['secondsPerPoint']
So from=-7d&until=-6d and from=-7d&until=now are answered at the same
step, and a query one second either side of a rung boundary is answered at
different steps. The delta cache must know the step before it can compose a
cache key or reason about which ranges it already holds — and the step is a
property of the metric’s on-disk file, which nothing in the request reveals.
Static configuration is not sufficient on its own, because a Whisper file keeps
whatever ladder it was created with. Editing storage-schemas.conf affects only
newly created files, so in any long-lived installation the config and the files
routinely disagree.
Probe and learn
On the first request for a metric Trickster does not know the ladder, so the
request is served through the object cache — correct, just not delta-cached —
and the response itself teaches the step at that age. In the background,
Trickster then discovers the metric’s full ladder with a short series of
synthetic /render probes: a geometric sweep outward in time to find where the
step changes, then a binary search for each rung boundary, and a pair of probes
to establish maxRetention. Each probe asks for a one-second window and pins
the origin’s reference time with now, so it is cheap and exact.
Ladders are shared. They originate from storage-schemas.conf patterns, so a
deployment with thousands of metrics typically has only a handful of distinct
ladders; a new metric is first confirmed against the ladders already known
(about 7 probes) and only discovered from scratch (about 40) if none matches.
In the developer environment, 24 metrics across 4 ladders converge in about 380
probes total, after which probing stops entirely until the TTL expires.
Once a ladder is known, prediction is arithmetic and costs nothing.
Confidence levels
Every resolved target carries a confidence, exported as the confidence label
on trickster_graphite_resolution_lookups_total:
| Confidence | Meaning | Behavior |
|---|---|---|
exact | The step was read from an origin response for this metric at this age | Delta cached |
derived | Computed from known ladders — the shared step of a wildcard’s leaves, or a step-altering function Trickster understands | Delta cached |
configured | From static_retentions only, not yet confirmed by probe | Delta cached, and a confirming probe is scheduled |
unknown | No usable step | Object cached |
A request with several targets reports the weakest confidence among them.
Verification, and what happens when a prediction is wrong
The step is not merely predicted; it is checked. Every response Trickster
models is compared against the predicted step. A mismatch means the cached
ladder was wrong — a whisper-resize.py, a metric moved between schemas — and
Trickster:
- discards the response rather than caching it under the predicted key,
- increments
trickster_graphite_step_mispredictions_totaland logs a warning, - invalidates the registry generation, so every entry learned under the old assumption misses rather than colliding,
- relearns the affected ladders, and
- re-serves the request through the object cache, so the client still gets the correct answer.
One response shape cannot self-verify: JSON carries no explicit step, so a
fetch that returns fewer than two points is consistent with any prediction.
Trickster never trusts one. A delta fetch that would cover a single bucket —
the tip fetch every steady-state refresh performs — is widened one bucket
into the past, which cannot change the archive whisper selects (the pinned
now keeps now - from identical) but makes the response two points, and
therefore self-verifying, at no extra request cost. Any fetched response
that still cannot prove its step — one point, or no series where the
prediction promised data inside retention — is refused outright: nothing is
cached, and the request is served through the object cache instead, while
the background learner re-establishes the ladder.
trickster_graphite_step_mispredictions_total should be flat at zero. A
non-zero value is not a client-visible error, but it does mean something the
registry believed was wrong.
Static retentions
static_retentions mirrors storage-schemas.conf: an ordered list, first
match wins, patterns matched anywhere in the metric path exactly as carbon
applies them.
graphite:
static_retentions:
- pattern: '^collectd\.'
retentions: '10s:6h,1m:7d,10m:5y'
- pattern: '\.count$'
retentions: '1m:2d,5m:30d'
- pattern: '.*'
retentions: '1m:1d'
Retention syntax is Whisper’s: precision:retention pairs, where each part is
a number with an optional unit (s, m for minutes, h, d, w, y), and a
bare number in the second position is a point count. 1m:7d is a one-minute
step kept for seven days.
This is a seed and an override, never the sole source of truth. A static
match yields configured confidence and schedules a confirming probe; when the
probe disagrees with the configuration, the probe wins and the ladder is
relearned. That is deliberate: it means a stale static_retentions block
degrades to a few extra probes rather than to incorrect data.
Use it to skip the warmup cost on a known-stable deployment, not as a substitute for learning.
Changing static_retentions invalidates the persisted registry on the next
start, so a corrected block takes effect immediately rather than waiting out
the TTL.
What is accelerated
A target is delta-cached only if every function in it satisfies two independent properties:
- Its step is predictable — the function does not change the resolution in a way Trickster cannot compute.
- It is range-decomposable — each output point is a function of the input
points at that same timestamp, so the values for
[t1,t2]are identical whether that window was fetched alone or as part of a wider one.
The second property is what delta caching actually requires, and it is stricter than it first appears. The v1 allowlist is deliberately small:
Cross-series aggregation — one output point per input timestamp:
sumSeries (sum), averageSeries (avg), minSeries, maxSeries,
diffSeries, multiplySeries, divideSeries, divideSeriesLists,
stddevSeries, rangeOfSeries, countSeries, percentileOfSeries,
aggregate, aggregateWithWildcards, aggregateSeriesLists, group,
groupByNode, groupByNodes, asPercent (pct), weightedAverage,
powSeries, unique, and the *SeriesLists and *WithWildcards variants.
Per-point transforms: scale, scaleToSeconds, offset, add, pow,
exp, absolute, invert, squareRoot, sigmoid, logit, log, round,
transformNull, isNonNull, removeAboveValue, removeBelowValue,
consolidateBy, setXFilesFactor (xFilesFactor).
Naming, name-based filtering and cosmetics: alias, aliasSub,
aliasByNode, aliasByMetric, aliasByTags, upper, lower, substr,
exclude, grep, color, alpha, lineWidth, dashed, drawAsInfinite,
secondYAxis, stacked, areaBetween.
Bare metric paths, wildcards (*, ?, [a-z], {a,b}) and pipe syntax are
all accelerated.
What falls back, and why
Anything not on the list above is served through the object cache. The reason
is recorded on trickster_graphite_fallbacks_total:
reason | Cause |
|---|---|
function_not_allowlisted | A function in the target is not on the allowlist |
unknown_step | The step could not be resolved (metric not yet learned, probe failing, or the window is wholly beyond maxRetention) |
missing_target | No target parameter, or a wildcard that matches nothing |
mixed_steps | A function combines series stored at different steps. graphite-web consolidates them to their least common multiple starting from each series’ first point, so the buckets move with from and cannot be stitched |
parse_error | The target expression, from/until, or now did not parse |
non_series_format | An image or pickle format, or graphType=pie |
multi_target_step_mismatch | Targets resolve to different steps and could not be split |
passthrough_max_data_points | passthrough_max_data_points is on and the request carries maxDataPoints |
misprediction | A response contradicted the predicted step |
tz_unavailable | The request names a tz whose validity could not be verified within the timezone cold-load budget (a burst of unique hostile tz values can spend it); the request is served unaccelerated with the original tz forwarded, rather than being reinterpreted in the configured zone |
client_identity | The request carries a result-affecting identity or view selector — an Authorization header, a header named in the render path’s cache_key_headers, or a parameter/form field it names in cache_key_params/cache_key_form_fields (local above all) — that the path’s request_headers/request_params does not statically override, or the render path or backend has a request rewriter — see below |
resolution_identity | The configured path identities are mixed: /render and /metrics/expand select different request_headers/request_params for synthetic GETs, or the request’s path config carries a different identity than the synthetic one (e.g., GET and POST /render pinned to different credentials) |
Notable functions that are not accelerated, and why:
- Time-windowed:
movingAverage,movingSum,movingMin,movingMax,movingMedian,exponentialMovingAverage,derivative,nonNegativeDerivative,perSecond,integral,stdev,holtWinters*,linearRegression. Each output point depends on input points before it, so a cached range cannot be reused at its edges. - Whole-range selecting or ranking:
highest*,lowest*,sortBy*,limit,mostDeviant,*Percentile,currentAbove/Below,averageAbove/Below,maximumAbove/Below,filterSeries. Which series come back depends on the entire requested range. - Bucketing:
summarize,hitcount,smartSummarize. These look decomposable — withalignToFrom=false,summarize’s buckets align to absolute interval boundaries — but the bucket covering either edge of the requested window is summarized from only the points inside that window. Measured on graphite-web 1.1.10, the same absolute one-hour bucket reported134885.899,302921.880and307388.038over three different windows. Caching such a value and reusing it for a different window would return data the origin never produced, so these take the object lane. - Time-shifting:
timeShift,timeStack,timeSlice,delay. - Generators and tag queries:
constantLine,threshold,timeFunction,randomWalk,sinFunction,identity,seriesByTag,groupByTags,events, andtemplate().
The object lane is still a cache: the whole response is cached by request URL with a TTL, keyed on every render parameter. It is always correct because it makes no claim about the response’s internal structure.
Multiple targets
Grafana sends every target of a panel in one /render call. Trickster splits a
multi-target request into one delta-cached fetch per target and merges the
results in target order, so targets on different ladders are still
accelerated. If any single target cannot be accelerated, the whole request is
served unaccelerated, because consolidation at the origin spans all series in
the response and must be applied consistently.
maxDataPoints and consolidation
Grafana sends maxDataPoints on every panel, sized to the panel’s pixel width.
Two panels of different widths showing the same query would otherwise be two
different cache entries.
Trickster strips maxDataPoints from upstream requests, caches at the
origin’s native resolution, and applies consolidation when rendering the
response to each client. One cached series therefore serves every panel width,
every output format, and noNullPoints, jsonp, pretty and tz variations.
The consolidation reproduces graphite-web’s renderViewJson exactly, including
its start “nudge”, its treatment of xFilesFactor, the consolidateBy
function, and the maxDataPoints=1 special case — verified byte-for-byte
against a live origin.
Set passthrough_max_data_points: true if you need the origin’s own
consolidation byte-for-byte instead. Requests carrying maxDataPoints are then
served unaccelerated, which for Grafana means most requests, so this largely
disables acceleration. It exists for operators who must guarantee identical
bytes to a pre-existing consumer.
Sizing
Two of the common backend settings default differently for Graphite, because
Graphite is fetched at its native resolution: maxDataPoints is stripped
upstream (see maxDataPoints), so Trickster
buffers and caches every point Whisper holds for the window, not the few
hundred a dashboard draws. The generic defaults are sized for origins that
consolidate before responding.
| Setting | Generic default | Graphite default |
|---|---|---|
max_object_size_bytes | 512 KB | 64 MB |
timeseries_retention_factor | 1024 points | 524288 points |
The Graphite values come from pkg/backends/graphite/options/defaults.go and are applied only where the configuration is silent, so setting either one in the file always wins.
max_object_size_bytes
The delta cache must buffer an entire response to model it. At native resolution a wide window on a fine archive is large: in the developer environment, a 120-day panel over two series on a 5-minute archive is 34,560 points per series and about 1.6 MB of JSON — three times the generic 512 KB limit.
A response over the limit cannot be delta-cached. Trickster logs
upstream response exceeded MaxObjectSizeBytes and serves the request through
the object lane instead, which streams rather than buffers — so clients still
get their data, but that panel is never accelerated. Raise the limit if you
serve panels wider than the 64 MB default holds:
max_object_size_bytes: 134217728 # 128MB
timeseries_retention_factor
The cache keeps this many points per series, cropping older ones. The generic default of 1024 points is only about 3.5 days at a 5-minute step, so a 90-day dashboard panel would have almost its entire range cropped after each fetch and refetched on the next — a partial hit that re-fetches nearly everything. The Graphite default of 524288 covers 5 years at a 5-minute step, or 60 days at a 10-second one.
Raise it if you serve a wider window at a finer step than that, and lower it to cap what one series may occupy:
timeseries_retention_factor: 1048576
Cache storage
Trickster does not roll up between rungs: a 10-second cached chunk is never
reused to answer a query that resolves to 60 seconds, because the rollup
Trickster computed would not match what Whisper’s lower archive holds (which
depends on the metric’s aggregationMethod and xFilesFactor). A different
step is a different cache key.
The practical implication is that a metric is cached once per rung it is
queried at. A dashboard with a 1-hour, a 24-hour and a 30-day view of the
same metric holds three cached series for it, not one. Size the cache for the
distinct (metric, step) pairs your dashboards produce, not for the number of
distinct metrics.
Metrics, logs, and tracing
Graphite-specific metrics, in addition to the standard
trickster_proxy_requests_total{provider="graphite"} and trickster_cache_*
families:
| Metric | Type | Notes |
|---|---|---|
trickster_graphite_resolution_lookups_total{backend_name,confidence,source} | counter | One per resolved request |
trickster_graphite_probes_total{backend_name,kind,result} | counter | Should spike at start and fall to zero |
trickster_graphite_ladders{backend_name} | gauge | Distinct ladders known; should flatten at a small number |
trickster_graphite_registry_entries{backend_name,layer} | gauge | leaf, ladder, target, negative |
trickster_graphite_step_mispredictions_total{backend_name} | counter | Should be flat at zero |
trickster_graphite_fallbacks_total{backend_name,reason} | counter | Why requests were not accelerated |
No metric path, target expression or query text ever appears in a label; every label value comes from a closed set. See metrics.md.
Logs: a Debug line per resolution decision carrying the target, a bucketed
age, the lane, confidence, source and step (or reason when declined); Warn on
every misprediction and on every negative-cache entry.
Traces: GraphiteExpand (a child of the request span), GraphiteProbe
(kind, result, step) and GraphiteLearnLadder (ladder, probe count).
Operations and troubleshooting
Everything is falling back
Check trickster_graphite_fallbacks_total by reason.
unknown_stepdominating right after a start is normal; it should fall as ladders are learned. If it does not, resolution is failing — look forgraphite ladder learning failed; negative-cachedwarnings.function_not_allowlistedmeans the dashboard uses functions outside the allowlist. That is expected and correct, not a defect.non_series_formatmeans the client is asking for images; those panels cannot be delta-cached.
Probing never quiets down
trickster_graphite_probes_total should approach zero after warmup and
trickster_graphite_ladders should flatten. If probing continues:
- Metric names may be high-cardinality or ephemeral, so new leaves appear
constantly. Raise
resolution_registry.max_entries, or accept the cost. - The registry may be evicting under
max_entriespressure and relearning. Comparetrickster_graphite_registry_entries{layer="leaf"}against the configured maximum. - Learning may be failing and retrying under backoff. Check the warnings.
Step mispredictions are non-zero
Something changed a metric’s on-disk ladder — usually whisper-resize.py, a
schema edit followed by new file creation, or a metric that moved between
schema patterns. Trickster recovers on its own (it relearns and re-serves the
request unaccelerated), so this is not a client-visible error, but the counter
should return to flat once the new ladders are learned. Persistent
mispredictions on one namespace suggest an origin whose resolution is not a
function of now - from at all — a clustered graphite-web fronting stores with
different schemas, for example.
A panel is never accelerated but should be
- Look for
upstream response exceeded MaxObjectSizeBytesin the logs and raisemax_object_size_bytes. - Check whether the panel’s window is wholly beyond the metric’s
maxRetention; those requests have nothing to cache. - A single non-allowlisted target in a multi-target panel makes the whole panel unaccelerated.
Repeated partial hits on wide panels
Raise timeseries_retention_factor. A partial hit that re-fetches nearly the
whole range on every request is the signature of cropping.
Verifying correctness against the origin
Because the object lane and the delta lane must both be byte-identical to the
origin, the simplest check is to configure two data sources in Grafana — one
direct to Graphite, one through Trickster — and compare panels. The developer
environment (docs/developer/environment/) is set up this way, with a
dashboard whose panels deliberately exercise archive boundaries, the retention
edge, schema drift, non-allowlisted functions and mixed-ladder multi-target
requests.
Step alignment
Graphite supports the truncate step alignment mode, its default, and off. Whisper stores and returns whole buckets only, so there are no partial buckets for the other modes to fetch, and configuring one fails at load. off sends each render request to Graphite as the client sent it, through the fallback lane, cached as an object for one minute. A render target has no comment syntax, so Graphite queries can’t carry directives.
Known gaps
- Fast Forward is not implemented. Graphite’s coarsest-rung behavior and
its
until > nowclamp make the “one extra instant datapoint” approach ill-defined.fast_forward_disableis effectively always on. - No ALB / Time Series Merge support. Merging across replicas requires the replicas’ ladders to agree, an invariant not yet established. Graphite backends can still be members of non-TSM ALB mechanisms.
- Non-decomposable functions are not accelerated. A future release may
evaluate them inside Trickster over cached native-resolution leaves, which
would move
movingAverage,derivative,summarize,highest*and friends into the accelerated path. - No cross-resolution rollup, as described under Cache storage.
- Tag-based queries (
seriesByTag,groupByTags) are not accelerated. They are proxied and object-cached. - Non-ASCII metric names (graphite-web’s
UTF8_METRICS) are not supported by the target parser; such targets fail to parse and take the object lane.
6.6.6 - PostgreSQL and TimescaleDB Provider
Trickster can accept native PostgreSQL client connections, relay each session to a PostgreSQL or TimescaleDB origin, and cache the results of eligible queries. Dashboards that bucket rows by time are served from Trickster’s delta cache: a refresh fetches only the buckets the cache does not hold yet.
This guide describes the supported contract. Anything not listed here is relayed to the origin without caching; cache eligibility always fails closed.
Compatibility
- PostgreSQL 14 and later, with or without the TimescaleDB extension.
- Grafana’s built-in PostgreSQL data source (Grafana 11 and later), with its TimescaleDB option on or off.
- Any client that speaks protocol version 3.0 or 3.2 (
psql, libpq, pgx, JDBC, …). Every client is relayed; which statements are cached is described under Cache classification.
Other engines that speak the PostgreSQL wire protocol are not part of this provider’s compatibility claim. The developer environment pins PostgreSQL 18 with TimescaleDB 2.30 and Grafana 13.
Provider names: postgres, and its alias timescaledb. Both select the same
engine, and metrics report provider="postgres".
Direct backend configuration
A postgres listener accepts the connections and maps to exactly one backend
(or one User Router) through the backend’s
listener_names.
listeners:
tsdb:
protocol: postgres
port: 5432
authenticators:
tsdb-clients:
provider: basic
users:
grafana_ro: ${GRAFANA_RO_PASSWORD}
backends:
tsdb1:
provider: timescaledb
origin_url: postgres://trickster_ro:REPLACE_ME@tsdb.example:5432/metrics
listener_names: [tsdb]
authenticator_name: tsdb-clients
cache_name: default
healthcheck:
interval: 5s
timeout: 3s
A complete annotated example is in
examples/conf/example.full.yaml.
Authentication modes
The backend’s authenticator decides who authenticates the client.
With authenticator_name (recommended). Trickster authenticates the
client itself against the authenticator’s users and then logs in to the origin
with the origin_url credentials. Clients are offered SCRAM-SHA-256, plus
SCRAM-SHA-256-PLUS (channel binding) when the listener has a certificate.
Authenticator entries may be plaintext passwords or PostgreSQL
SCRAM-SHA-256$... verifiers copied from pg_authid. An entry stored as a
PostgreSQL md5 hash or a crypt hash needs the client’s cleartext password,
which is refused on a non-TLS connection unless the listener sets
postgres.allow_cleartext_without_tls (or postgres.allow_md5 for the legacy
md5 exchange). Every client shares the origin role named in origin_url, so
give that role the least privilege the dashboards need. Replication
connections are refused in this mode.
Without one (or with observe_only: true). The client’s own
authentication exchange is relayed to the origin untouched and Trickster holds
no credentials. One caveat: SCRAM-SHA-256-PLUS cannot succeed when both the
listener and the origin use TLS, because the client binds to Trickster’s
certificate and the origin expects its own. Clients that require channel
binding must use the authenticator mode.
In both modes the cache is partitioned by the client’s user and database, so one user’s cached answers are never served to another.
TLS
TLS is negotiated in-band on the listener port, as PostgreSQL does it; there
is no separate TLS port. Configure the backend’s tls.full_chain_cert_path
and tls.private_key_path to let clients upgrade, and set require_tls: true
to refuse plaintext clients.
TLS toward the origin is independent of the listener. Set the backend’s
postgres.upstream_tls_mode:
| Mode | Meaning |
|---|---|
disable (default) | plaintext |
require | TLS without certificate verification |
verify-ca | the origin’s chain must verify against tls.certificate_authority_paths |
verify-full | as verify-ca, and the certificate must match the origin host name |
tls.client_cert_path and tls.client_key_path add a client certificate.
Connections, limits, and lifecycle
A result that outgrows the result limits is streamed to the client through a fixed buffer, however large its rows are; the limits bound what Trickster holds in memory, not what a client may read.
Each client connection gets its own origin connection for its whole life, so session state never crosses clients, and Trickster adds no pooling. Put a pooler such as PgBouncer between Trickster and the origin if you need one.
| Setting | Where | Default | Effect |
|---|---|---|---|
max_concurrent_conns | backend | unlimited | origin connections; further clients get 53300 too many clients |
connections_limit | listener | unlimited | accepted client connections |
postgres.handshake_timeout | listener | 10s | startup and authentication |
postgres.read_timeout / write_timeout | listener | 30s | one message in flight |
postgres.idle_timeout | listener | 5m | wait for a client’s next message |
postgres.max_query_size_bytes | listener | 1 MiB | larger statements are relayed without being analyzed |
postgres.max_result_rows | backend | 100000 | larger results are relayed without being cached |
postgres.max_result_size_bytes | backend | 64 MiB | as above |
timeout | backend | 1m | origin connect, and the wait for a cached statement’s result |
Cancel requests work through the listener: clients receive a Trickster-issued cancellation key that maps to the origin’s, and a client that disconnects mid-query has its statement canceled at the origin. A configuration reload that changes a listener’s identity (origin, credentials, TLS, routes, limits) restarts that listener and drains its sessions; other reloads leave sessions alone.
Protocol behavior
Everything a client sends is relayed: the simple and extended query protocols,
COPY, LISTEN/NOTIFY, transactions, multi-statement queries, notices and
errors (with their SQLSTATE) all reach the client as the origin sent them.
Only one kind of message is ever answered from the cache: a simple-protocol
Query holding a single read statement, sent while the session is idle (no
open transaction, nothing in flight). That is what Grafana, psql and most
reporting tools send. Statements sent with the extended protocol
(Parse/Bind/Execute, i.e. prepared or parameterized statements) are relayed
and never cached. For a driver that defaults to it, select its simple
protocol mode if you want caching (pgx: default_query_exec_mode=simple_protocol;
JDBC: preferQueryMode=simple).
Cached answers are the origin’s own bytes: the row description and rows are stored and replayed exactly as received, in text format. Trickster reads only the bucket column, to place rows on the time axis.
Cache classification
Each analyzed statement is counted in
trickster_sql_query_analysis_total{backend_name,dialect,cache_mode,reason}.
cache_mode | Meaning |
|---|---|
delta | cached per time bucket; only missing buckets are fetched |
object | the whole result is cached under the statement’s exact text |
none | relayed, never cached |
Delta-cacheable statements
A statement uses the delta cache when it is a single-table SELECT that
groups by one recognized time bucket and bounds the bucketed column with a
literal (or now()-relative) lower limit and, optionally, an upper limit
joined by AND. A range with no upper limit runs to the present.
| Bucket | Notes |
|---|---|
time_bucket(width, col) | TimescaleDB; what Grafana’s $__timeGroup writes with the TimescaleDB option on |
time_bucket(width, col, TIMESTAMPTZ '...') | a typed origin |
time_bucket(width, col, '...'::interval) | a typed offset |
time_bucket_gapfill(width, col) | a single series, both bounds in WHERE, no locf or interpolate |
floor(extract(epoch from col)/N)*N | $__timeGroup with the TimescaleDB option off; also with date_part('epoch', col) |
floor((col)/N)*N | $__unixEpochGroup, over a column of epoch seconds with epoch-second bounds |
date_bin(width, col, origin) | PostgreSQL 14+ |
date_trunc('unit', col) | second to week, and only while the session TimeZone is UTC |
A width is a fixed-length interval: '300.000s', '5 minutes', '1h30m',
INTERVAL '1 hour', '7 days'. Month and year widths never match.
FILTER, ordered-set aggregates, TimescaleDB hyperfunctions such as first
and last, extra GROUP BY columns, and ORDER BY on the bucket (either
direction) are all fine.
Four behaviors are worth knowing:
- Live ranges lose their partial edge buckets by default. Under the
default
step_alignment,drop, bounds that are not on the bucket grid, as Grafana’snow-relative ranges never are, are rounded inward, so the partial first and last buckets are left out of the answer rather than cached as if they were complete. The still-filling bucket that holds the present is partial too, so a range that reaches the present ends before it.col <= Xkeeps X’s bucket only when X is the last instant of it (...:59.999999). Thepartial,partial_startandpartial_endmodes instead fetch those buckets from the origin over the client’s own range, through the object cache forpartial_bucket_ttl, and never cache them with the complete ones, at a cost of up to two small origin queries per request;truncateanswers the whole first bucket. A query can choose its own mode with a comment, such as-- trickster-step-align:partial. See Step Alignment. - A range with no complete bucket is answered by the origin. When nothing
complete remains, as for a range inside one bucket, across a single
boundary, or starting inside the still-filling bucket, the client gets the
origin’s own result for its statement, partial and still-filling buckets
included, cached as an object for
partial_bucket_ttl. - Zone-less literals need a UTC session. PostgreSQL reads
'2026-09-17 00:00:00'andTIMESTAMP '...'in the sessionTimeZone, so such bounds (and origins) qualify only while that zone is UTC. Grafana’sZ-suffixed bounds always qualify. - Ordering within a bucket is the origin’s. Buckets are merged in time
order and each keeps the row order the origin returned.
ORDER BYmust lead with the bucket or be absent; any other ordering makes the statement an object.
Statements cached as objects
Everything else that is a deterministic read: statements with no bucket, a
LIMIT, a join, a subquery or CTE, DISTINCT, HAVING, window functions,
set-returning functions, a gapfill outside the form above, month or year
buckets, the time zone form of time_bucket, unsafe time predicates (OR,
NOT, <>, a strict > lower bound on the raw column), and statements the
SQL parser cannot write back faithfully (FROM ONLY, json casts,
U&'...' strings) or cannot parse at all (named arguments such as
origin => ..., GROUP BY ROLLUP). The reason label says which.
An object is keyed on the statement’s exact text. A dashboard panel whose
range ends at now is therefore a new object on every refresh, and is served
from cache only while its range is unchanged. Objects live for
timeseries_ttl, so a repeated SELECT count(*) FROM t is answered from
cache for that long; lower it, or set proxy_only: true, if that is too
stale for your clients.
Statements never cached
- Anything that is not a
SELECT, and aSELECTthat writes or locks (WITH x AS (DELETE ... RETURNING *) SELECT ...,FOR UPDATE,SELECT ... INTO). - A statement that calls a volatile or side-effecting function, or reads the
clock anywhere but in a time bound:
random(),now(),current_date,nextval(),pg_sleep(),pg_notify(), advisory locks and similar (reason="nondeterministic"). - Multi-statement queries, statements inside a transaction, pipelined
statements, and statements larger than
postgres.max_query_size_bytes. - Everything in a session whose state Trickster can no longer vouch for; see Session state.
Trickster recognizes volatile functions by name. A user-defined volatile
function, or one that changes a session setting as a side effect, is not
detectable. Do not route such statements through a caching backend; use a
proxy_only backend for them.
Continuous aggregates
Querying a continuous aggregate works like querying a table. To stay on the delta cache, re-bucket the aggregate’s bucket column and group by it:
SELECT time_bucket('15 minutes', bucket) AS "time", cab_type AS metric, sum(trips) AS trips
FROM trips_15m
WHERE bucket >= $__timeFrom() AND bucket < $__timeTo()
GROUP BY 1, 2 ORDER BY 1
A bare SELECT bucket, trips FROM trips_15m WHERE ... has no bucket function
and no GROUP BY, so it is cached as an object. Real-time aggregates change
their newest buckets as data arrives; volatile_window controls how much
of the recent past Trickster re-fetches on each request.
Grafana macros and exact SQL shapes
These are the statements Grafana 13’s PostgreSQL data source sends, and how they are classified.
| Macros | Sent as | Mode |
|---|---|---|
$__timeGroup(col,'5m'), TimescaleDB on | time_bucket('300.000s',col) AS "time" | delta |
$__timeGroup(col,'5m'), TimescaleDB off | floor(extract(epoch from col)/300)*300 AS "time" | delta |
$__timeGroupAlias(col,$__interval) | as above, with the panel’s interval | delta |
$__timeFilter(col) | col BETWEEN '...Z' AND '...Z' | delta with a bucket |
$__timeFrom(), $__timeTo() | '...Z' | delta with a bucket |
$__unixEpochGroup(col,'10m') | floor((col)/600)*600 | delta |
$__unixEpochFilter(col) | col >= N AND col <= M | delta with a bucket |
$__time(col), $__timeEpoch(col) | a projection, no bucket | object |
$__unixEpochNanoFilter(col) | nanosecond integers | object |
The minimum supported $__interval is 1 minute. Prefer
col >= $__timeFrom() AND col < $__timeTo() over $__timeFilter(col): both
are delta-cacheable, but the half-open form states exactly the range that is
cached. Grafana’s pgx driver sends an empty -- ping query before each panel
query; it is relayed and left out of the request metrics.
The executable form of this contract is
pkg/backends/postgres/testdata/compatibility/v1.json.
Session state
Cached answers depend on session settings: TimeZone changes how every
timestamptz is rendered, search_path changes which table a name means.
Trickster therefore partitions the cache by a session identity: the user, the
database, every setting the origin announces (TimeZone, DateStyle,
IntervalStyle, client_encoding, search_path on PostgreSQL 18, …), and
role, extra_float_digits and bytea_output from the client’s own SET
statements. Two sessions share cached answers only when all of it matches.
A session that does something Trickster cannot follow stops being cached
until it reconnects (reason="session_state"): any other SET (for example
a custom app.tenant used by row-level security), set_config(),
SELECT ... INTO, DDL, PREPARE/EXECUTE, DO, CALL, and fast-path
function calls. A session with standard_conforming_strings off is relayed
uncached as well. Transactions, DML, SHOW, EXPLAIN, and harmless settings
such as statement_timeout and application_name do not end caching.
PostgreSQL never announces extra_float_digits or bytea_output, and a role
or database default can change them. With an authenticator, Trickster reads
the session’s effective values once at origin login, makes them part of the
session identity, and returns to them on RESET. When the client’s own login
is relayed, Trickster cannot ask, so it only knows a value the client set
itself.
Results whose bucket column cannot be read safely are cached as objects
instead: under a non-ISO DateStyle, and for a float8 epoch bucket
($__unixEpochGroup, or the date_part('epoch', ...) form) whenever
extra_float_digits is negative or unknown, since PostgreSQL may then render
1788998400 as 2e+09. In relay-authentication mode such a panel needs the
client to SET extra_float_digits (0 or higher) to use the delta cache.
Failure behavior
The cache always fails open to a plain relay of the client’s own statement:
when the origin rejects a statement Trickster rewrote for a sub-range
(trickster_sql_query_rewrite_failures_total{reason="origin_rejected"}), or
when a result outgrows the result limits (reason="result_size"). Such a
statement skips the cache until a marker expires, so it is not retried on
every refresh. Origin errors reach the client verbatim and are never cached.
Protocol-aware User Router
A postgres listener can front a User Router ALB whose targets are
postgres/timescaledb backends. Trickster authenticates the client against
the router’s authenticator, picks the backend mapped to the startup user (or
default_backend), and logs in to that backend’s origin with its origin_url
credentials. The session stays on that backend until it ends.
- The router requires an authenticator. Targets need origin credentials and no authenticator or listener of their own.
- A user with no route, or whose backend is failing its health check, is
refused with SQLSTATE
28000. Sessions never fail over to another backend. to_userandto_credentialare not supported.- Each target keeps its own cache, limits and metrics.
A complete example is in
examples/conf/postgres-user-router.yaml.
Metrics, logs, and health
| Metric | Labels |
|---|---|
trickster_sql_query_analysis_total | backend_name, dialect, cache_mode, reason |
trickster_sql_query_cache_total | backend_name, dialect, cache_mode, cache_status |
trickster_sql_query_rewrite_failures_total | backend_name, dialect, reason |
trickster_proxy_requests_total, ..._duration_seconds, trickster_proxy_points_total | the common request labels, method="QUERY" |
trickster_pgwire_connections_total, trickster_pgwire_active_connections | backend_name (and event) |
trickster_pgwire_errors_total | backend_name, class |
trickster_pgwire_route_selections_total | router_name, backend_name, outcome |
Query text, user names in labels, passwords and origin addresses are never
logged or exported. The health check logs in to the origin on a fresh
connection with the origin_url credentials and waits for ReadyForQuery;
see Health Checks.
Operations and troubleshooting
A panel is never a hit. Look up its statement’s reason in the analysis
metric. unsupported_limit, unsupported_bucket and unsupported_format
mean it is an object, and an object whose range ends at now changes every
refresh. session_state means the session did something unmodeled; find the
SET the client sends at connect time. pipelined and in_transaction
point at a driver that wraps reads in transactions or uses the extended
protocol.
The first and last points are missing. Expected on unaligned ranges; see Delta-cacheable statements.
origin_rejected rewrite failures. Trickster re-spells the statement
when it fetches a sub-range, and the origin refused the result. The client is
unaffected, since its own statement is relayed, but the statement is not
cached. Please file an issue with the statement.
Stale data after a bulk rewrite. Cached buckets older than the volatile
window are not re-fetched until they expire (timeseries_ttl). After
rewriting history, restart Trickster when using the memory cache, or clear
the backend’s keys from a persistent one.
Rollout and rollback
Canary the listener with a limited client population and compare with the
pre-release baseline. Roll back on a sustained increase of 5 percentage points
in trickster_pgwire_errors_total or proxy-error outcomes, 1 percentage point
in rewrite failures, or 20% in p95 request duration. Roll back by restoring
the previous configuration and binary, reloading or restarting Trickster, and
letting the listener drain; sessions are never moved between versions.
Setting proxy_only: true on the backend is the quickest mitigation: it
keeps the listener up and turns off all statement inspection and caching.
Environment variables and reload behavior
There are no TRK_* variables for postgres listener or backend options;
configure them in YAML. Authenticator users values expand ${VARIABLE};
origin_url does not, so keep a configuration with an embedded password in a
secret store. Validate before rollout:
trickster -validate-config -config /etc/trickster/trickster.yaml
Known compatibility gaps
- Extended-protocol (prepared, parameterized, binary-format) statements are relayed but never cached.
- Month, quarter and year buckets, the time zone overload of
time_bucket, integer-timetime_bucket(bigint, col), and explicitstart/finisharguments totime_bucket_gapfillare objects, not deltas. date_truncand zone-less time literals are delta-cacheable only in UTC sessions.- User-defined volatile functions and functions that change session settings cannot be detected.
- Downstream client-certificate authentication, GSSAPI encryption, and channel binding in relay-authentication mode are not supported.
- No cross-session invalidation: a write through one session does not evict what another session cached.
6.6.7 - GreptimeDB Provider
Use provider: greptimedb for GreptimeDB’s HTTP, PostgreSQL and MySQL query
surfaces. One backend may be mapped to all three listener protocols. The
provider reuses the SQL and Prometheus cache engines, with GreptimeDB-specific
parsing, session settings, result types and API paths.
Surfaces
| Surface | Origin port | Behavior |
|---|---|---|
HTTP /v1/sql | 4000 | Eligible SELECT results use delta or object caching |
| PostgreSQL wire | 4003 | Eligible simple queries use delta or object caching; extended queries are relayed |
| MySQL wire | 4002 | Eligible text queries use delta or object caching |
/v1/prometheus/api/v1/query_range | 4000 | Prometheus range caching and supported time-series merges |
| Prometheus instant and metadata APIs | 4000 | Object caching subject to HTTP cache policy |
Ingest, TQL, /v1/promql, other HTTP APIs | 4000 | Proxied without SQL/PromQL delta rewriting |
| gRPC | 4001 | Not implemented by this provider |
Wire compatibility does not make GreptimeDB a PostgreSQL or MySQL server.
Use the greptimedb provider even on native listeners. For example, MySQL’s
SHOW COUNT(*) WARNINGS is not a supported GreptimeDB statement, and its
PostgreSQL session settings have different semantics.
The developer environment pins an official nightly that includes GreptimeDB #9295, the fix for Grafana’s comment-only PostgreSQL health query. See the developer environment for the exact image and reproducible checks. The tested Grafana plugin is the bundled PostgreSQL datasource, not a GreptimeDB-specific plugin.
Step Alignment
GreptimeDB’s surfaces follow the step alignment of the query language they carry:
| Surface | Supported modes | Default |
|---|---|---|
| Prometheus HTTP API | truncate, drop, partial_end, off | partial_end, which is Fast Forward (truncate with fast_forward_disable: true) |
| SQL, over HTTP, MySQL and PostgreSQL | all | drop |
A backend’s step_alignment may name any mode either supports; a PromQL query asked for partial or partial_start runs in its default instead, and the fallback is counted in trickster_step_alignment_fallbacks_total. The partial modes fetch each partial bucket as a small query of its own through the object cache, so they cost up to two extra origin queries per request. A query can choose its own mode with a comment, # ... in PromQL and -- ... or /* ... */ in SQL; see Per-Query Instructions.
Configuration
The complete example exposes HTTP on 8480, PostgreSQL on 8489 and MySQL on 8491. A minimal mixed backend is:
listeners:
greptime-pg:
protocol: postgres
port: 8489
greptime-mysql:
protocol: mysql
port: 8491
authenticators:
greptime-readers:
provider: basic
proxy_preserve: true
users:
grafana_ro: ${GRAFANA_RO_PASSWORD}
backends:
greptime:
provider: greptimedb
origin_url: http://greptime.example:4000
listener_names: [default, greptime-pg, greptime-mysql]
authenticator_name: greptime-readers
postgres:
upstream_url: postgres://trickster_ro:REPLACE_ME@greptime.example:4003/public
mysql:
upstream_url: mysql://trickster_ro:REPLACE_ME@greptime.example:4002/public
HTTP uses origin_url; postgres.upstream_url and mysql.upstream_url
separately identify native endpoints and credentials. Never put the HTTP
port in either native URL. Without a native override, an HTTP origin’s host
is reused with the engine’s native default port, not its HTTP port or userinfo.
Explicit URLs are recommended. URL credentials do not expand environment
variables; authenticator user values do.
List only default for an HTTP-only backend, or just the chosen native
listener for a native-only backend. A mixed backend probes HTTP /health;
a native-only backend uses its native authenticated probe. Listener, origin,
credential or protocol changes restart and drain affected native listeners.
Authentication And TLS
For HTTP, proxy_preserve: true forwards client credentials to GreptimeDB.
Authenticated object responses are not shared unless the origin explicitly
permits HTTP caching. SQL and PromQL cache identities retain their database,
request parameters, effective timezone and authentication partitioning.
For PostgreSQL, an authenticator terminates client authentication at Trickster; the separate upstream URL supplies the origin role. Without an authenticator, authentication is passed through to GreptimeDB. MySQL requires a configured authenticator. Give every origin role only the permissions its clients need; terminating authentication does not preserve separate upstream roles for each client.
Native TLS and limits use the existing postgres and mysql option blocks;
see PostgreSQL and MySQL. Support for an option in
Trickster is not a promise that a particular GreptimeDB deployment supports
the corresponding server feature. The loopback developer environment uses
plaintext and published test credentials and must not be exposed publicly.
SQL Cache Classification
Delta caching requires one provably deterministic time-bucketed SELECT with supported grouping, ordering and time bounds. Other supported reads can use whole-result object caching. Writes, volatile queries and session-unsafe statements bypass caches; a parser rejection never makes a write cacheable.
PostgreSQL supports fixed-width date_bin intervals or compact widths such
as 5m, UTC fixed-width date_trunc, and epoch buckets of the form
floor(extract(epoch FROM ts)/300)*300 or date_part. Calendar widths,
ambiguous time predicates, unknown timezone-dependent semantics, and
submicrosecond PostgreSQL text buckets do not get delta plans. Extended
protocol messages are always relayed.
HTTP SQL supports GET and form-encoded POST with the default greptimedb_v1
response format. A delta response retains typed schema, row ordering, NULLs
and exact integer values. Like PostgreSQL, under the default drop step
alignment, unaligned SQL bounds round inward to complete buckets before delta
caching and partial edge buckets are omitted.
Alternate formats, limit, unknown options and ranges with no complete bucket
retain the original query. Rebuilt responses do not claim the origin’s
execution duration or execution metrics. The default JSON response is decoded
row by row into a DataSet, and its validated client serialization is reused.
MySQL supports DATE_BIN('1m', ts, FROM_UNIXTIME(0)) and fixed-width
DATE_TRUNC buckets with verified UTC sessions, whole-second cadence and
half-open bounds, which the default drop step alignment
rounds inward to complete buckets. Other bucket origins,
subsecond cadence, inclusive upper bounds and ranges with no complete bucket
use the original query instead. Timestamp results keep up to nine
fractional digits, text groups are compared case-sensitively, and NULL ordering
follows GreptimeDB. Unsupported or failed session changes conservatively
disable caching for that connection.
PromQL range endpoints round down to epoch-aligned steps, including 500ms steps, so equivalent aligned ranges share cached samples. Instant and metadata timestamps are unchanged. Route defaults do not grant permission to share authenticated object responses: only an origin’s explicit cache policy can authorize that sharing.
Grafana Macros
Use the built-in PostgreSQL datasource with TimescaleDB disabled and a minimum interval of one minute. Native epoch-second columns are integers, not timestamps cast to integers (GreptimeDB timestamp casts can yield nanoseconds).
| Macro family | GreptimeDB behavior |
|---|---|
$__time, $__timeEpoch | Supported projection; not a bucket by itself |
$__timeFilter, $__timeFrom, $__timeTo | Supported timestamp bounds |
$__timeGroup, $__timeGroupAlias | Fixed epoch-floor buckets with TimescaleDB mode off |
$__unixEpochFilter, $__unixEpochFrom, $__unixEpochTo | Numeric epoch-second bounds |
$__unixEpochGroup, $__unixEpochGroupAlias | Fixed buckets over an epoch-second column |
$__unixEpochNanoFilter | Supported SQL predicate; not an automatic delta plan |
$__interval | Use a positive supported fixed duration |
The compatibility corpus
records real Grafana expansions, including unaligned windows and conservative
fallbacks. Grafana’s MySQL macros are not interchangeable: UNIX_TIMESTAMP
is unsupported in the tested GreptimeDB image. Use explicit GreptimeDB SQL
when connecting a MySQL client.
Known Limits
RANGE ... ALIGNremains whole-result caching; TQL is passthrough. There is no advanced RANGE/FILL dependency rewriting.- PostgreSQL text timestamps carry microseconds even for a TIMESTAMP(9) column. MySQL and HTTP have separate precision contracts.
- Transaction commands accepted by GreptimeDB are compatibility stubs, not proof of transactional isolation. PostgreSQL cancellation is not a supported upstream capability in the tested image.
SET LOCALpersists at session scope in GreptimeDB. Effective settings are probed and tracked; requested startup parameters alone are not proof of the resulting timezone. The development read-only account cannot change it.- PostgreSQL session load balancing is unsupported. MySQL listeners use the existing native MySQL routing rules.
- PromQL
count_valuescurrently loses its grouping label upstream; merges reject that response rather than claim a correct aggregate. - Ingestion, mutations, enterprise-only features and gRPC caching are outside this provider’s accelerated contract.
Run the acceptance suites against an
isolated seeded environment. Inspect cache counters as well as results:
trickster_sql_query_analysis_total, trickster_sql_query_cache_total, and
trickster_sql_query_rewrite_failures_total distinguish actual cache hits,
fallbacks and failed rewrites. SQL dialect labels are greptimedb; transport
metrics retain the native protocol name.
6.6.8 - QuestDB Provider
Use provider: questdb for a QuestDB backend that exposes both the HTTP SQL
surface and the PostgreSQL wire protocol. Trickster keeps the HTTP surface as
an authenticated passthrough and uses the native pgwire path for the verified
SQL relay and cache contract.
Surfaces
| Surface | Origin | Behavior |
|---|---|---|
| HTTP SQL, Web Console and other HTTP paths | QuestDB HTTP, normally 9000 | Passthrough; no HTTP SQL delta cache is applied |
| PostgreSQL wire protocol | QuestDB pgwire, normally 8812 | Simple-query relay, object cache and verified fixed-width delta cache |
| Extended pgwire statements | QuestDB pgwire | Relayed; extended statements are not analyzed for delta caching |
The native listener uses the shared PostgreSQL wire adapter. A QuestDB backend
can therefore be mapped to an HTTP listener and a protocol: postgres
listener at the same time. The provider name, rather than postgres, selects
QuestDB’s SQL analyzer and result semantics.
Configuration
The HTTP origin and native origin are separate. Put the HTTP endpoint in
origin_url and the pgwire endpoint, including its credentials, in
postgres.upstream_url:
listeners:
questdb-pg:
protocol: postgres
port: 8490
authenticators:
questdb-readers:
provider: basic
users:
grafana_ro: ${QUESTDB_CLIENT_PASSWORD}
backends:
questdb:
provider: questdb
origin_url: https://questdb.example:9000
listener_names: [default, questdb-pg]
authenticator_name: questdb-readers
postgres:
upstream_url: postgres://grafana_ro:REPLACE_ME@questdb.example:8812/qdb
upstream_tls_mode: disable
healthcheck:
interval: 5s
timeout: 3s
Use TLS and deployment-specific credentials in a real installation. The
developer environment uses cleartext QuestDB pgwire because QuestDB OSS does
not provide pgwire TLS in the tested image. proxy_preserve is only suitable
when the HTTP origin is configured to accept the same client credential; the
native upstream URL is the source of the pgwire origin credential.
If the backend is native-only, map only questdb-pg and use a PostgreSQL
origin_url with an explicit user. Without an HTTP listener, the provider
uses its authenticated pgwire health probe. With an HTTP listener, the health
check uses authenticated QuestDB /execute and the native listener still uses
the postgres.upstream_url.
SQL Cache Contract
The QuestDB analyzer has a deliberately narrow delta-cache surface:
| Query shape | Result |
|---|---|
SAMPLE BY 5m, 15m, and other positive fixed s/m/h/d widths | Delta cache |
SAMPLE BY ... FILL(NULL) | Object cache; fill rows depend on the selected range |
timestamp_floor('5m', timestamp_column) with grouped output | Delta cache |
floor(extract(epoch FROM ts)/N)*N | Delta cache when the bounds and result are renderable |
| Other deterministic reads | Object cache or relay, according to the normal pgwire policy |
FILL(PREV), SAMPLE BY FROM ... TO ..., calendar M widths, ambiguous grouping | Object/relay; no speculative rewrite |
| Writes, volatile statements, malformed or unknown time ranges | Relay; never a delta plan |
For SAMPLE BY, QuestDB applies the designated timestamp to the first plain
selected column and implicitly groups the remaining plain selected columns.
The adapter only accepts that unambiguous form. An explicit GROUP BY, a
computed first item or another ambiguous shape fails closed rather than
guessing the series identity.
The pgwire result path maps QuestDB’s verified PostgreSQL type OIDs and
microsecond timestamp text to the shared time-series representation. Session
settings that can affect results or rendering, including timezone, date style,
interval style, float/bytea output and search path, remain part of the cache
identity. Unknown startup settings also partition the identity until observed;
unknown SET statements bypass caching conservatively.
Developer Environment
The reproducible developer environment adds a pinned QuestDB OSS image, a
read-only pgwire role, an HTTP admin role for seeding, and the official
questdb-questdb-datasource Grafana plugin. It reuses the shared generated
trips fixture rather than downloading another dataset.
See QuestDB Details for
ports, seed commands, dashboard datasource names and direct-versus-proxy
checks. The versioned analyzer contract is in
pkg/backends/questdb/testdata/compatibility.
The integration conformance target compares direct and proxied QuestDB results, protocol errors and session state. It also checks object and delta cache counters, so a successful passthrough response cannot be mistaken for a cache hit.
Known Limits
- HTTP SQL is passthrough in this provider; native pgwire is the only path with QuestDB-specific delta caching in this change.
- Extended pgwire messages are relayed and are not delta-cacheable.
- QuestDB
FILL(PREV), clause-ownedFROM/TOranges and calendar-month widths remain object/relay paths. - QuestDB OSS cleartext pgwire and the credentials in the developer compose file are for isolated local testing only.
6.6.9 - VictoriaMetrics Provider
Use provider: victoriametrics for VictoriaMetrics: single-node, or one tenant of a cluster’s
vmselect. Trickster accelerates MetricsQL range queries through VictoriaMetrics’ Prometheus
querying API, caches settled instant and metadata queries, and serves its Graphite render, find
and tags APIs. VictoriaMetrics is its own provider rather than a flavor of prometheus:
MetricsQL is a different language, and native VictoriaMetrics, not PromQL, decides every answer.
Surfaces
| Surface | Routes | Behavior |
|---|---|---|
| MetricsQL range queries | /api/v1/query_range | Delta proxy cache, or exact-request object cache for expressions that depend on the whole range |
| MetricsQL instant queries | /api/v1/query | Object cache once the evaluation time has settled; relayed near the live edge |
| Metadata | /api/v1/series, /api/v1/labels, /api/v1/label/<name>/values | Object cache, 30 seconds |
| Other read APIs | /api/v1/status/..., /api/v1/metadata and the rest of the Prometheus catalog | Object cache, 30 seconds, keyed on every parameter |
| Graphite render | /render | Exact-request object cache for settled absolute ranges; relayed otherwise |
| Graphite discovery | /metrics/find, /metrics/expand, /tags..., /functions | Object cache, 30 seconds |
| Graphite tag registration | /tags/tagSeries, /tags/tagMultiSeries, /tags/delSeries | Relayed uncached |
| Everything else | writes, imports, exports, /api/v1/admin/..., /internal/..., /federate, /vmui | Relayed uncached |
Every Graphite route is also served under /graphite, as single-node VictoriaMetrics serves it.
Configuration
origin_url is the base of the Prometheus querying API: the server for single-node
VictoriaMetrics, or a tenant’s /select/<tenant>/prometheus path on vmselect. The Graphite
APIs are served from the same server, with a trailing /prometheus replaced by /graphite, as
vmselect’s tenant paths require.
backends:
vm:
provider: victoriametrics
origin_url: http://victoriametrics:8428
vm-tenant:
provider: victoriametrics
# Graphite requests go to http://vmselect:8481/select/0/graphite
origin_url: http://vmselect:8481/select/0/prometheus
victoriametrics:
# replaces the derived Graphite path, e.g. for a gateway with its own routes
graphite_path: /select/0/graphite
# set when VictoriaMetrics runs with -search.disableCache (see Range Queries)
search_disable_cache: false
| Option | Default | Description |
|---|---|---|
victoriametrics.graphite_path | derived | The upstream path of the Graphite APIs: an absolute path, without a query |
victoriametrics.search_disable_cache | false | Mirrors the origin’s -search.disableCache flag |
volatile_window | 60s | How long recent points stay provisional; keep it longer than the origin’s -search.latencyOffset (30s) plus its ingestion delay |
Authentication headers, TLS and request rewriting work as for any HTTP backend; requests that
carry Authorization are cached under keys that include it. The default health check runs
query=1 against the query API, so it also checks a tenant’s path.
Range Queries
Trickster answers a range query on the grid VictoriaMetrics evaluates it on. With its response
cache on, VictoriaMetrics aligns a range of 50 or more points to start on a step boundary and
keeps the number of points; a shorter range keeps the requested start. Trickster rewrites the
range the same way before looking in its cache, so cached and native answers have the same
timestamps. Grafana aligns its ranges itself, so its requests are unchanged. With
search_disable_cache: true, every range keeps the requested start, as VictoriaMetrics then does.
Cache Eligibility
Each MetricsQL statement is parsed once with VictoriaMetrics’ own parser
(metricsql, pinned to the version the provider
was validated with); WITH templates are expanded before classification.
| Statement | Path |
|---|---|
Selectors, rollups (rate, increase, *_over_time, rollup, with or without a window), aggregations, topk/bottomk, binary operators, label and math functions, offset, subqueries, @ with a timestamp | Delta proxy cache |
range_*, running_*, keep_last_value, keep_next_value, interpolate, remove_resets, smooth_exponential, sort*, limit_offset, drop_common_labels, drop_empty_series, union, buckets_limit, start(), end(); topk_*/bottomk_* variants, limitk, any, outliers*; the limit modifier; @ with anything but a timestamp | Exact-request object cache (step alignment off) |
now(), rand(), rand_normal(), rand_exponential(), and statements that don’t parse | Relayed uncached |
A statement that doesn’t parse is relayed, so VictoriaMetrics returns its own error. A request is also relayed uncached when it:
- sets
nocacheortrace; - has a parameter the route doesn’t read, or a parameter repeated, or the same parameter in both the URL and a POST form;
- uses a time VictoriaMetrics would read differently than Trickster: anything but Unix seconds
(with at most millisecond precision) or RFC 3339 with a zone, such as
-1h,now, a zoneless time, or milliseconds; - is an instant query within the volatile window of now, or without a
time; - is a POST that isn’t
application/x-www-form-urlencoded.
round_digits, latency_offset, max_lookback, extra_label, extra_filters[], limit and
optimize_repeated_binary_op_subexprs are part of a response’s cache identity; timeout and
denyPartialResponse can only fail a request and are not. The classification of each request
is counted in trickster_victoriametrics_query_analysis_total.
Fidelity
VictoriaMetrics evaluates a point from the samples it fetched for the whole request: an implicit
window, and whether rate and similar functions use the sample before their window, depend on a
scrape interval estimated from the last 20 sample intervals it read. Where a series’ scrape
interval changes, a range assembled from separate fetches can differ slightly from one
nocache=1 evaluation of it. VictoriaMetrics’ own response cache, which is on by default,
assembles ranges from separate evaluations the same way, so the delta proxy cache’s answers match
what VictoriaMetrics serves by default. For a single exact evaluation, set
step_alignment: off on the backend, or write # trickster-step-align:off in the query.
Points within the volatile window of now, which VictoriaMetrics replaces within its
-search.latencyOffset, are fetched on every request and never stored. A request’s own
latency_offset widens that window.
Partial Responses
When some vmstorage nodes don’t answer, vmselect marks its response "isPartial":true.
Trickster never caches such a response: a range query is relayed to the origin, and other API
responses are marked Cache-Control: no-store. Responses served from cache for an origin that
reports isPartial include "isPartial":false, as vmselect writes it. The stats object
VictoriaMetrics adds to query responses describes one evaluation and is not kept in cached
responses.
Graphite
The Graphite APIs use the Storage-Step request header, which sets the grid the render API
returns; it is part of a render response’s cache identity, with every query parameter. A render
is consolidated to maxDataPoints over its whole range, so renders are cached only as exact
requests, never assembled from separate ranges. A render is cached only when from and until
are Unix timestamps and until is outside the volatile window; Grafana sends both that way.
Time Series Merging
The tsm ALB mechanism doesn’t support VictoriaMetrics backends yet: the provider offers no merge
planning, so MetricsQL is never merged with PromQL’s semantics, and requests through a tsm pool
of VictoriaMetrics backends fail. Other ALB mechanisms work as for any backend.
Known Limits
- Delta caching is not offered for the Graphite render API.
partial_end(Fast Forward) is not offered.- An unfiltered
/tagsor/tags/<tag>returns nothing for series VictoriaMetrics stored as labels;/tags/autoComplete/tags?expr=...,/tags/findSeriesandseriesByTagfind them. This is VictoriaMetrics’ behavior and is relayed as is. - Requests through the frontend’s
/graphitealias are mapped to the Graphite path for vmselect origins too, where vmselect has no such alias. - Raw-series export (
/api/v1/export) is relayed uncached.
Developer Environment
The developer environment runs single-node VictoriaMetrics v1.153.0 with seeded trips history
and fixtures, the victoriametrics1 backend, and the Trips (VictoriaMetrics) and
VictoriaMetrics Graphite dashboards. See
VictoriaMetrics Details.
6.6.10 - MySQL Provider
Trickster can accept native MySQL client connections, proxy each authenticated session to a MySQL origin, and cache eligible text-protocol query results. This guide describes the initial supported contract. Behavior not listed here is rejected or proxied without caching; cache eligibility always fails closed.
Compatibility
The supported matrix is:
- Oracle MySQL 8.4 LTS through 9.7 LTS;
- Grafana 11.0 and above using Grafana’s built-in MySQL data source; and
- native clients that use
mysql_native_passwordand the supported commands below.
MariaDB, Percona Server, Aurora MySQL, other MySQL-compatible products, older Grafana releases, and third-party Grafana MySQL data sources are not part of the initial compatibility claim, but should work without problems. File an issue if you discover a compatibility issue.
The developer environment pins MySQL 8.4 and Grafana 13.
Direct backend configuration
The listener accepts the MySQL wire protocol on one TCP port. MySQL negotiates
TLS in-band, so do not configure an HTTP-style tls_port.
listeners:
mysql-native:
protocol: mysql
address: ""
port: 8486
connections_limit: 200
mysql:
handshake_timeout: 10s
read_timeout: 30s
write_timeout: 30s
idle_timeout: 5m
max_packet_size_bytes: 16777215
max_query_size_bytes: 1048576
authenticators:
mysql-clients:
provider: basic
users:
grafana_reader: ${GRAFANA_MYSQL_PASSWORD}
caches:
mysql-cache:
provider: memory
memory:
max_size_bytes: 536870912
backends:
mysql-primary:
provider: mysql
listener_names: [mysql-native]
authenticator_name: mysql-clients
origin_url: mysql://trickster_ro:REDACTED@mysql.example:3306/analytics
cache_name: mysql-cache
timeout: 30s
max_concurrent_conns: 200
max_object_size_bytes: 8388608
mysql:
max_result_rows: 100000
max_result_size_bytes: 67108864
healthcheck:
interval: 5s
timeout: 3s
failure_threshold: 3
recovery_threshold: 2
Exactly one direct MySQL backend, one supported MySQL User Router ALB, or one
session-balancing ALB maps to a MySQL
listener. The origin URL must use the mysql scheme and include an
origin username. Percent-encode reserved username, password, and database
characters. Configuration stringification and the sanitized management
configuration redact an embedded origin password, but the source configuration
must still be protected as a secret.
The authenticator is the only downstream credential source. Trickster terminates downstream authentication; it does not pass client credentials to the origin or silently fall back to origin credentials. Native-password verification requires the original password, so MySQL authenticator entries must be plaintext values, including environment-expanded values or a protected CSV users file. Htpasswd hashes are rejected for this listener. Do not expose the raw configuration management endpoint outside a trusted administrative network.
TLS
Downstream TLS
Add a server certificate and key to the listener-facing backend:
backends:
mysql-primary:
# ...direct backend settings above...
require_tls: true
tls:
full_chain_cert_path: /etc/trickster/tls/server.crt
private_key_path: /etc/trickster/tls/server.key
Without a server pair, downstream TLS is disabled. With the pair, TLS is
optional unless require_tls: true rejects plaintext clients. TLS 1.2 is the
minimum. Downstream client certificates are not supported in the initial
release.
Upstream TLS
The upstream modes are:
- no upstream TLS fields: plaintext;
insecure_skip_verify: true: encryption without server verification;- one
certificate_authority_pathsentry: CA and hostname verification; and - a complete
client_cert_path/client_key_pathpair: mutual TLS, combined with verified or explicitly unverified server mode.
backends:
mysql-primary:
# ...direct backend settings above...
tls:
certificate_authority_paths:
- /etc/trickster/tls/origin-ca.crt
client_cert_path: /etc/trickster/tls/origin-client.crt
client_key_path: /etc/trickster/tls/origin-client.key
Certificate/key settings must be complete pairs. More than one upstream CA
path, require_tls without a downstream server pair, or an incomplete pair is
a configuration error. There is no CA-verification-without-hostname mode.
Connections, limits, and lifecycle
Each authenticated downstream connection owns exactly one upstream connection.
There is no pooling or multiplexing, which prevents transaction, database, and
session state from leaking between clients. Size connections_limit and
max_concurrent_conns together and leave capacity for health probes. A useful
starting point is one admitted origin connection per downstream connection,
plus normal operational headroom.
Listener limits protect the downstream boundary:
connections_limitlimits concurrent client connections (0is unlimited);- handshake, command-read, response-write, and idle timeouts bound each phase;
max_packet_size_bytesis fixed at 16 MiB minus one; andmax_query_size_byteslimits SQL text within that packet ceiling.
Backend limits protect the origin and cache:
timeoutbounds connect and query work;max_concurrent_connsbounds upstream admission;mysql.max_result_rowsandmysql.max_result_size_bytesbound results; andmax_object_size_bytesskips caching an otherwise valid oversized object.
Handshake, packet, query, read, write, and idle failures close the client connection. Origin timeout or concurrency rejection closes the affected upstream connection and returns a bounded MySQL error. A row/byte overflow closes both sides because a partial result may already have been emitted.
On shutdown, Trickster stops accepting new sessions and drains existing ones within the configured management drain window. A reload that changes MySQL authentication, listener transport, certificates, origin transport, routing, or relevant limits restarts and drains the listener; an established session is never silently moved to new credentials, TLS policy, or a different origin.
Protocol behavior
Supported commands are:
COM_QUERYcontaining one text-protocol statement;COM_INIT_DB;COM_PING;COM_QUIT; andCOM_RESET_CONNECTION.
Reset discards the upstream connection and tracked session state. The next command opens a fresh origin connection.
The initial release rejects binary prepared statements; SQL PREPARE,
EXECUTE, and DEALLOCATE; server cursors; binary results; compression;
LOCAL INFILE and all LOAD statements; multi-statements; stored-procedure
CALL; HELP, XA, HANDLER, and CACHE INDEX; executable version comments;
unclassified text response shapes; COM_CHANGE_USER; replica registration; and
binlog commands. Connection attributes may be syntactically accepted but are
ignored for authentication, routing, cache identity, and query semantics.
Malformed, oversized, timed-out, or partially streamed commands close the
connection. A fully consumed unsupported command normally leaves it usable.
Successful DML and DDL are proxied and never cached. They do not invalidate existing OPC or DPC entries, so another session may see cached data until TTL expiry or eviction. Use short TTLs or purge affected cache data when immediate read-after-write visibility across sessions is required.
Cache classification
Trickster uses three outcomes:
- Delta Proxy Cache (DPC): a supported deterministic aggregate time-series
SELECTwith one unambiguous time output, literal positive cadence, safe result shape and ordering, and an inclusive-lower/exclusive-upper raw-time predicate; - Object Proxy Cache (OPC): a deterministic single-result
SELECTthat is safe to cache as one object but is not proven reusable by time extent; and - proxy-only: mutations, transactions, unsupported or unsafe session state, non-deterministic or rejected shapes, and any query whose safety cannot be established.
DPC supports the tested zero-phase integer/cast and
FLOOR(UNIX_TIMESTAMP(column) / n) * n bucket forms when n is the same
positive integer literal on both sides. Dynamic, mismatched, non-positive,
overflowing, ambiguous, or non-zero-phase forms fail closed. Qualified and
backtick-quoted identifiers, multiple numeric values, and string dimensions
are supported only in the shapes recorded by the compatibility corpus.
The authenticated downstream username, selected terminal backend, normalized database, supported time-zone literal, and query identity participate in cache keys. Cache entries are never shared across authenticated users merely because their SQL is identical.
Grafana macros and exact SQL shapes
Only patterns represented by the versioned compatibility corpus are published
as supported. The corpus covers $__time, $__timeEpoch, $__timeFilter,
$__timeFrom, $__timeTo, $__timeGroup, $__timeGroupAlias,
$__unixEpochFilter, $__unixEpochFrom, $__unixEpochTo,
$__unixEpochGroup, and $__unixEpochGroupAlias. Its documented minimum
$__interval is one minute.
Grafana’s normal inclusive $__timeFilter expansion is delta-cached (DPC):
SELECT
CAST(CAST(UNIX_TIMESTAMP(ts)/(300) AS SIGNED)*300 AS SIGNED) AS time,
COUNT(*) AS samples
FROM telemetry
WHERE ts BETWEEN FROM_UNIXTIME(1785542400)
AND FROM_UNIXTIME(1785628800)
GROUP BY time
ORDER BY time
A half-open predicate composed from $__timeFrom() and $__timeTo() is also DPC:
SELECT
CAST(CAST(UNIX_TIMESTAMP(ts)/(60) AS SIGNED)*60 AS SIGNED) AS time,
AVG(value) AS mean_value
FROM telemetry
WHERE site = 'denver'
AND ts >= FROM_UNIXTIME(1785542400)
AND ts < FROM_UNIXTIME(1785628800)
GROUP BY time
ORDER BY time
The epoch-second equivalent is:
SELECT
CAST(CAST(epoch_seconds/(300) AS SIGNED)*300 AS SIGNED) AS time,
SUM(requests) AS requests
FROM service_rollups
WHERE region = 'us-west'
AND epoch_seconds >= 1785542400
AND epoch_seconds < 1785628800
GROUP BY time
ORDER BY time
Trickster caches only complete buckets. Under the default step_alignment,
drop, it rounds the lower bound up and the upper bound down to the cadence,
and an inclusive upper bound, such as the end of BETWEEN, also drops the
bucket that contains it, because that bucket is only partly covered. The
partial, partial_start and partial_end modes fetch those edge buckets
from the origin over the client’s own bounds, through the object cache for
partial_bucket_ttl, at a cost of up to two small origin queries per request,
and truncate answers the whole first bucket. A range with no complete bucket
is answered with the origin’s own result for the statement, cached as an object
for partial_bucket_ttl. A query can choose its own mode with a comment, such
as /* trickster-step-align:partial */. See Step Alignment.
Grafana’s strict-lower $__unixEpochFilter expansion remains OPC, because a
strict lower bound does not cover the first bucket completely. Native DATETIME/TIMESTAMP, epoch-second integer,
and the corpus’s epoch-nanosecond adaptation are supported in their recorded
shapes.
Session state
The only cache-safe tracked state is the selected default database and one
session-scoped literal SET time_zone = '<value>'. USE and COM_INIT_DB
update the database portion of cache identity.
Transactions and savepoints bypass cache while active. Other SET forms,
character set or collation changes, sql_mode, lc_time_names, user
variables, temporary objects, locks, session-local functions, mutations, and
unclassified state changes make that connection cache-unsafe. Reconnect or a
successful COM_RESET_CONNECTION is required to restore the configured
baseline.
Protocol-aware User Router
A native MySQL User Router selects one direct terminal MySQL backend from the verified downstream username:
listeners:
mysql-routed:
protocol: mysql
port: 8486
connections_limit: 200
authenticators:
tenant-clients:
provider: basic
users:
tenant_a_reader: ${TENANT_A_CLIENT_PASSWORD}
tenant_b_reader: ${TENANT_B_CLIENT_PASSWORD}
caches:
tenant-a-cache:
provider: memory
tenant-b-cache:
provider: filesystem
backends:
tenant-a-mysql:
provider: mysql
authenticator_name: tenant-clients
origin_url: mysql://tenant_a_ro:REDACTED@mysql-a.example:3306/analytics
cache_name: tenant-a-cache
healthcheck:
interval: 5s
timeout: 3s
tenant-b-mysql:
provider: mysql
authenticator_name: tenant-clients
origin_url: mysql://tenant_b_ro:REDACTED@mysql-b.example:3306/analytics
cache_name: tenant-b-cache
healthcheck:
interval: 5s
timeout: 3s
mysql-by-tenant:
provider: alb
listener_names: [mysql-routed]
authenticator_name: tenant-clients
alb:
mechanism: ur
user_router:
default_backend: tenant-a-mysql
users:
tenant_a_reader:
to_backend: tenant-a-mysql
tenant_b_reader:
to_backend: tenant-b-mysql
The listener-facing router owns the one downstream authentication exchange, admission, and TLS. Direct terminal MySQL backend entries must reference the same named authenticator to satisfy backend validation, but do not perform a second client exchange. The terminal owns its origin credentials, upstream TLS, cache, health, limits, and query policy. Routing occurs once before opening the upstream connection and remains sticky for the whole downstream session. Transactions, database changes, and session state never trigger rerouting.
Only direct MySQL terminal backends are supported. Nested ALBs, Rules, cycles,
mixed terminal providers, empty routes, and MySQL to_user/to_credential
remapping are configuration errors. A router requires an authenticator. An
unmapped user uses default_backend when configured; without one, Trickster
returns a bounded native MySQL no-route error. HTTP no_route_status_code is
not sent on the native connection. A selected unhealthy or unavailable
terminal fails as a MySQL availability error; the session is not silently
routed elsewhere.
The verified username and selected terminal remain in cache identity. Route metrics use configured router/backend names and bounded outcomes, never the username.
Balancing sessions across replicas
A MySQL listener can also map to an ALB that balances its sessions across a pool of direct MySQL backends, such as read replicas:
backends:
replica-1:
provider: mysql
authenticator_name: app-clients
origin_url: mysql://app_ro:REDACTED@replica-1.example:3306/analytics
healthcheck:
interval: 5s
replica-2:
provider: mysql
authenticator_name: app-clients
origin_url: mysql://app_ro:REDACTED@replica-2.example:3306/analytics
healthcheck:
interval: 5s
replicas:
provider: alb
listener_names: [mysql-replicas]
authenticator_name: app-clients
alb:
mechanism: lc # rr, p2c, lc or hrw
pool:
- replica-1
- name: replica-2
weight: 2
The ALB owns the downstream authentication exchange, admission, and TLS, as a
User Router does, and each pool member owns its origin credentials, cache,
health, and query policy. A session is committed to one member after it
authenticates and stays there until it ends. lc and p2c compare members by
their open sessions; hrw keeps a client address (hrw.key: client_ip, the
default) or a user name (hrw.key: user) on one member. weight, backup
members, and healthy_floor apply as they do for any ALB. lt, the fanout
mechanisms, nested ALBs, mixed providers, and autodiscovery are configuration
errors. See
Load Balancing Native Protocol Sessions.
A change of mechanism or weight applies to new sessions on reload; a change to the set of pool members restarts the listener.
Metrics, logs, and health
Important metrics include:
trickster_sql_query_analysis_total{backend_name,dialect,cache_mode,reason};trickster_sql_query_rewrite_failures_total{backend_name,dialect,reason};trickster_sql_query_cache_total{backend_name,dialect,cache_mode,cache_status};trickster_proxy_request_duration_secondsandtrickster_proxy_points_totalwithprovider="mysql";trickster_mysql_connections_total,trickster_mysql_active_connections,trickster_mysql_errors_total,trickster_mysql_route_selections_total, andtrickster_mysql_command_duration_seconds; and- cache operation, usage, limit, and event metrics labeled by
cache_name.
Example PromQL, replacing names to match the deployment:
sum by (cache_mode, reason) (rate(trickster_sql_query_analysis_total{backend_name="mysql1"}[5m]))
sum by (reason) (rate(trickster_sql_query_rewrite_failures_total{backend_name="mysql1"}[5m]))
sum by (cache_mode, cache_status) (rate(trickster_sql_query_cache_total{backend_name="mysql1"}[5m]))
histogram_quantile(0.95, sum by (le) (rate(trickster_proxy_request_duration_seconds_bucket{backend_name="mysql1",provider="mysql"}[5m])))
sum by (cache_status) (rate(trickster_proxy_points_total{backend_name="mysql1",provider="mysql"}[5m]))
sum by (operation, status) (rate(trickster_cache_operation_objects_total{cache_name="mysql-cache"}[5m]))
trickster_cache_usage_bytes{cache_name="mysql-cache"}
sum by (reason) (rate(trickster_cache_events_total{cache_name="mysql-cache",event="eviction"}[5m]))
Analysis logs contain bounded backend, cache mode, reason, and statement-type fields. Rewrite and protocol failures use bounded categories; SQL text, passwords, and usernames are not metric labels. Keep debug logging temporary in production because classification logs can be high volume.
Native health checks open a bounded connection with the backend’s configured
origin credentials and TLS policy, then issue COM_PING. They use the common
interval, timeout, failure, and recovery thresholds. HTTP health request fields
do not apply. Health output reports sanitized authentication, TLS, timeout,
refused-connection, and server-error categories. See Health Checks
for management endpoint details.
Kubernetes deployment
The base examples expose the optional mysql Service/container port on 8486.
The safe routed example keeps the complete credential-bearing configuration in
deploy/kube/mysql-user-router-secret.yaml
and replaces the base ConfigMap volume with
deploy/kube/mysql-deployment-patch.yaml.
Replace every placeholder and use an encrypted/external secret controller in
production. Do not put an origin DSN, downstream passwords, private keys, or
client keys in deploy/kube/configmap.yaml.
For direct TLS or mTLS, add the server, CA, and client certificate material to
a Kubernetes TLS or opaque Secret, mount it read-only, and reference the mount
paths from the backend tls block. Rotating that Secret requires the
configuration/certificate reload path described above; established sessions
are drained rather than changing transport identity in place.
Operations and troubleshooting
Kubernetes readiness
The native health scheduler does not delay listener startup or make the general
health endpoint return a non-200 status. /trickster/ping proves only that the
process is running, and a TCP readiness probe on port 8486 proves only that the
native listener is accepting connections. The companion manifest uses that TCP
probe. Treat the scheduled MySQL status published in /trickster/health?json
as the origin-readiness signal in an external controller or monitoring rule;
do not assume the endpoint’s HTTP status reflects origin health. Allow at least
interval * failure_threshold or interval * recovery_threshold for a stable
transition. User Router terminals report separately, and a failing selected
terminal returns a native availability error rather than rerouting the session.
Capacity planning
Start from peak concurrent dashboard/client sessions. Reserve approximately
one origin connection per admitted client, then add health and rollout
headroom. Bound result rows/bytes below the pod or process memory budget and
set max_object_size_bytes below both the result-byte limit and a practical
fraction of cache capacity. For an in-memory cache, budget retained objects,
index overhead, concurrent result encoding, and normal process memory. Use
finite query, command, and idle timeouts so abandoned clients cannot retain
connections indefinitely.
Repeated misses or proxy-only outcomes
- Group
trickster_sql_query_analysis_totalbycache_modeandreason. - Confirm the query is deterministic, single-result, and outside a transaction or unsafe session.
- For DPC, inspect the expanded SQL: require a literal cadence and
>=lower,<upper raw-time predicates. - Confirm the requested interval contains at least one complete cadence bucket and that cache TTL, volatile window, and retention are suitable.
- Check username, selected backend, database, and time zone; these intentionally isolate keys.
- Inspect cache operation status and eviction metrics for admission failures or churn.
Authentication and TLS
- Authentication failure: verify the named authenticator, plaintext/CSV credential source, exact username, supported native-password plugin, and filesystem permissions on a mounted users file.
- Downstream TLS failure: verify the server pair,
require_tls, client TLS mode, hostname, and TLS 1.2+ support. Do not usetls_port. - Upstream TLS failure: verify the single CA path, hostname in the origin URL,
complete client pair when using mTLS, file permissions, and whether
insecure_skip_verifywas intentionally selected.
Origin and cache failures
During an origin outage, health transitions and MySQL error metrics should rise; existing safe cache hits may continue, while misses fail. Restore origin health before increasing connection limits. A cache-provider failure should surface in cache operation/event metrics and logs; query handling fails closed to the provider’s normal proxy behavior rather than treating an unknown cache state as a hit.
Certificate rotation and relevant configuration reloads drain affected native listeners. Monitor active connections, authentication/TLS errors, route selection outcomes, health state, rewrite failures, cache status, and p95 latency until the old sessions have drained.
Rollout and rollback
Canary the listener/backend configuration with a limited client population. Compare against the pre-release baseline and roll back for a sustained increase of 5 percentage points in proxy/origin/protocol errors, 10 percentage points in proxy-only classifications, 1 percentage point in rewrite failures, or 20% in p95 proxy latency. Roll back by restoring the previous configuration and binary/image on the release branch, reloading or restarting Trickster, and allowing the old MySQL listener to drain. Do not move an established session between versions. Preserve metrics and sanitized logs for the incident record.
Environment variables and reload behavior
The global TRK_ORIGIN_URL and TRK_ORIGIN_TYPE variables map only to the
default backend and are suitable only for a basic single-backend deployment.
There are no one-to-one TRK_* variables for MySQL listener limits, backend
limits, TLS, health, or User Router options; configure those in YAML.
Authenticator users values support ${VARIABLE} expansion, and supported
users files and secret volume mounts are preferred for production. The
origin_url field does not expand ${VARIABLE} in structured YAML; store a
configuration containing an embedded DSN as a protected secret, not a public
ConfigMap.
Validate before rollout:
trickster -validate-config -config /etc/trickster/trickster.yaml
SIGHUP and the management reload endpoint reload a changed file. Authentication, TLS, transport, routing, and terminal runtime changes restart and drain the affected MySQL listener. Cache policy changes apply to new work; existing cache objects remain subject to their stored TTL and normal eviction.
Known compatibility gaps
The initial release does not claim support for:
- MySQL-compatible server products outside Oracle MySQL 8.4-9.7;
- downstream plugins other than
mysql_native_passwordor downstream mTLS; - prepared/binary protocols, compression, local infile, multi-results, stored procedures, binlog/replication commands, or connection-attribute semantics;
- general SQL parsing beyond the Vitess-supported and compatibility-corpus shapes;
- window functions, grouping sets, rollups, ambiguous time axes or buckets, unsafe boolean predicates, non-deterministic outputs, dynamic/non-zero-phase buckets, or inclusive-range DPC;
- arbitrary session-state caching or cross-session mutation invalidation; and
- nested/mixed MySQL User Router topologies or origin credential remapping.
Streamed result metadata and FOUND_ROWS()
Vitess does not expose the terminal status flags and warning count of a
streamed result set. To preserve those fields for the downstream client,
Trickster issues SHOW COUNT(*) WARNINGS on the same upstream connection
after the stream ends. That diagnostic statement can update connection-local
MySQL diagnostics, so FOUND_ROWS() immediately after a streamed result may
reflect the hidden warning-count query rather than the client’s previous
statement.
This is a known compatibility gap in the Vitess streaming API, not a Trickster-specific SQL parsing defect. A Vitess API that exposes terminal result metadata is required to remove the workaround.
The executable SQL contract is
pkg/backends/mysql/testdata/compatibility/v1.json. The maintainer-facing frozen release matrix is in
docs/developer/mysql-release-contract.md.
7 - Routing & Load Balancing
7.1 - Application Load Balancer
Trickster 2.x provides an Application Load Balancer that is easy to configure and provides unique features to aid with Scaling, High Availability and other applications. The ALB supports several balancing Mechanisms:
| Mechanism | Config | Provides | Description |
|---|---|---|---|
| Round Robin | rr | Scaling | a basic, stateless round robin between healthy pool members |
| Power of Two Choices | p2c | Scaling | draws two healthy pool members at random and routes to the one with fewer requests in flight |
| Least Connections | lc | Scaling | routes to the healthy pool member with the fewest requests in flight |
| Least Time | lt | Speed | routes to the healthy pool member with the lowest response latency, scaled by its requests in flight |
| Highest Random Weight | hrw | Affinity | consistently routes each client, tenant or other key to the same healthy pool member |
| Time Series Merge | tsm | Federation | uses scatter/gather to collect and merge data from multiple replica tsdb sources |
| First Response | fr | Speed | fans a request out to multiple backends, and returns the first response received |
| First Good Response | fgr | Speed | fans a request out to multiple backends, and returns the first response received with a status code < 400 |
| Newest Last‑Modified | nlm | Freshness | fans a request out to multiple backends, and returns the response with the newest Last-Modified header |
| User Router | ur | Control | Inspects the credentials in the Request and routes it based on the Username |
| Connect Race | race | Speed | connects a tcp or tls stream connection to several pool members at once and relays over the first to connect |
| UDP Mirror | mirror | Replication | copies every datagram of a udp stream session to every healthy pool member, and answers from the first |
Integration with Backends
The ALB works by applying a Mechanism to select one or more Backends from a list of Healthy Pool Members, through which to route a request. Pool member names represent Backend Configs (known in Trickster 0.x and 1.x as Origin Configs) that can be pre-existing or newly defined.
All settings and functions configured for a Backend are applicable to traffic routed via an ALB - caching, rewriters, rules, tracing, TLS, etc.
In Trickster configuration files, each ALB itself is a Backend, just like the pool members to which it routes. This makes it possible to configure infinite loops (e.g., where ALB1 has ALB2 in its pool, and ALB2 has ALB1 in its pool). However, at startup Trickster will validate ALB configurations by following all ALBs’ possible paths, and exit with a startup failure if any infinite loops are detected.
In addition to (or instead of) a static pool list, an ALB’s pool membership can be discovered and kept current automatically at runtime — from Kubernetes, the AWS, Google Cloud and Azure APIs, the Docker Engine, the Consul or Nomad service registries, DNS records, an HTTP endpoint, or a watched member-list file. See ALB Autodiscovery.
Mechanisms Deep Dive
Each mechanism has its own use cases and pitfalls. Be sure to read about each one to understand how they might apply to your situation.
Basic Round Robin
A basic Round Robin rotates through a pool of healthy backends used to service client requests. Each time a client request is made to Trickster, the round robiner will identify the next healthy backend in the rotation schedule and route the request to it.
To keep each client on the member it was first sent to, add Sticky Sessions to the ALB. For affinity without session state, see Highest Random Weight.
Weighted Round Robin
Trickster supports Weighted Round Robin with a first-class integer weight on pool entries. A pool entry may be a plain backend name (weight 1) or a mapping with an explicit weight:
pool:
- node01 # weight 1
- name: node02
weight: 3 # receives 3 of every 4 requests
Apportionment is exact: over any totalWeight consecutive requests against a stable healthy pool, each member is selected exactly weight times. A heavier member’s turns are spread through the rotation rather than taken back to back: two members weighted 3 and 2 are served A B A A B, not A A A B B. Weights also carry through from autodiscovery sources that convey them (DNS SRV record weights, member-file weight fields); see ALB Autodiscovery.
Weights apply to every mechanism that selects a single member per request, though only Round Robin makes them an exact guarantee; see Weights and the Selection Mechanisms. Fan-out mechanisms (fr, fgr, nlm, tsm) dispatch to every healthy member regardless of weight.
More About Our Round Robin Mechanism
Trickster’s Round Robin Mechanism works by maintaining an atomic uint64 counter that increments each time a request is received by the ALB. With uniform weights, the ALB performs a modulo operation on the request’s counter value, with the denominator being the count of healthy backends in the pool; the resulting value, ranging from 0 to len(healthy_pool) - 1, indicates the assigned backend based on the counter and current pool size. With mixed weights, the modulo denominator becomes the pool’s total weight, and the result indexes a rotation schedule, computed once each time the set of healthy members changes, in which every member’s turns are evenly spaced. Selection remains lock- and allocation-free in both forms. The counter starts at a random value, so a fleet of Trickster replicas started together does not send its first requests to the same pool member.
Example Round Robin Configuration
backends:
# traditional Trickster backend configurations
node01:
provider: reverseproxycache # will cache responses to the default memory cache
path_routing_disabled: true # disables frontend request routing via /node01 path
origin_url: https://node01.example.com # make requests with TLS
tls: # this backend might use mutual TLS Auth
client_cert_path: ./cert.pem
client_key_path: ./cert.key
node02:
provider: reverseproxy # requests will be proxy-only with no caching
path_routing_disabled: true # disables frontend request routing via /node02 path
origin_url: http://node-02.example.com # make unsecured requests
request_headers: # this backend might use basic auth headers
Authoriziation: "basic jdoe:${NODE_02_AUTH_TOKEN}"
# Trickster 2.x ALB backend configuration, using above backends as pool members
node-alb:
provider: alb
alb:
mechanism: rr # round robin
pool:
- node01 # as named above; weight 1
- node02
# to weight the pool, use the mapping form on any entry:
# - name: node02
# weight: 2 # node02 would receive 2 of every 3 requests
Here is the visual representation of this configuration:

Power of Two Choices
The Power of Two Choices (p2c) mechanism draws two healthy pool members at random and routes the request to whichever has fewer requests in flight. It keeps load nearly as even as inspecting every member would, at a cost that does not grow with the size of the pool, and unlike Round Robin it reacts to a member that has become slow: requests pile up there, so it loses more of its draws. It is a good default for large pools and for requests whose cost varies widely.
backends:
api:
provider: alb
alb:
mechanism: p2c # or power_of_two_choices
pool: [ node01, node02, node03 ]
Least Connections
The Least Connections (lc) mechanism routes each request to the healthy pool member with the fewest requests in flight. It reads every member on every request, so it suits small pools; Power of Two Choices approximates it for large ones. While several members are tied, as all are when the pool is idle, they take turns.
backends:
api:
provider: alb
alb:
mechanism: lc # or least_connections
pool: [ node01, node02 ]
Least Time
The Least Time (lt) mechanism routes each request to the healthy pool member that has been answering fastest. Each member’s score is its latency average multiplied by one more than its requests in flight, and the lowest score wins, so a fast member is preferred until it is busy enough that a slower one would answer sooner.
Latency is the time from routing a request to the first byte of its response. The average rises at once when a member slows down and falls gradually, over lt.decay, as it recovers. A few details keep the ranking honest:
- A member with no requests yet, such as one just discovered, is scored as its fastest peer. It shares that peer’s requests until its own first response ranks it, so it is neither flooded as the apparent fastest nor left waiting for a turn that an idle pool would never give it.
- A failed request is recorded as a long latency rather than a short one, so a member that returns errors in a millisecond never looks fast. By default a response of
502,503or504is a failure, as is one that was never completed;lt.status_codessets the codes that count as a good answer instead, as bare codes, inclusive ranges, or both. A request the client abandoned counts neither way. - A member’s average fades while it is passed over, so one ranked last on an old measurement or a past failure is tried again rather than ignored for good.
- Averages survive a configuration reload and autodiscovery membership changes.
When the pool is idle and its members are equally loaded, every request goes to the fastest member. That is the mechanism working as intended; choose p2c or lc to spread idle traffic instead.
backends:
video:
provider: alb
alb:
mechanism: lt # or least_time
pool: [ edge01, edge02 ]
lt:
decay: 10s # default
status_codes: [ { start: 200, end: 499 } ] # optional; default is every code but 502, 503 and 504
Highest Random Weight
The Highest Random Weight (hrw) mechanism, also known as rendezvous hashing, routes every request that shares a key to the same healthy pool member. Use it to keep a client or tenant on one member’s warm cache. When a member leaves the pool, only the keys it owned move, each to a different remaining member; when it returns, exactly those keys move back. The mapping depends only on the key and the members’ names, so every Trickster replica agrees on it and a restart does not change it.
hrw.key selects what is hashed:
| Key | Follows |
|---|---|
client_ip (default) | the client’s IP address, after trusted proxy resolution. The port is never part of the key. IPv6 addresses are keyed on their leading hrw.ipv6_prefix bits (default 64), because privacy addressing changes the rest of a client’s address over time. |
host | the request’s host name, without its port and without regard to case |
header:<name> | the first value of the named request header |
cookie:<name> | the value of the named cookie |
query:<name> | the value of the named query string parameter, as written in the URL |
sni | the TLS server name the client offered; only for an ALB that serves a tls stream listener |
proxy_tlv:<type> | the value of a PROXY protocol version 2 TLV, such as proxy_tlv:0xEA for an AWS VPC endpoint ID; only for an ALB that serves a tcp or tls stream listener with proxy_protocol enabled. The type is one byte, written in decimal or 0x hex. |
user | the user name a session authenticated as; only for an ALB that serves a native protocol listener |
A request that lacks the configured key, such as one without the header, has no affinity to preserve and is routed to a member at random.
hrw balances keys, not requests. With many keys of similar volume the members’ loads even out, but a single very busy key is always served by one member.
hrw keeps no state, so a key moves when a member joins the pool and wins it, and a client with no stable key has no affinity at all. To keep a client on its member for as long as its session lasts, whatever the pool does, add Sticky Sessions, which work with hrw as with every mechanism that selects one member.
backends:
tenants:
provider: alb
alb:
mechanism: hrw # or highest_random_weight
pool: [ cache01, cache02, cache03 ]
hrw:
key: header:X-Tenant
Sticky Sessions
An ALB whose mechanism selects one member per request, rr, p2c, lc, lt or hrw, can keep each client on the member it was first sent to. The mechanism chooses a new client’s member; after that, the client’s requests go to the same member for as long as its session lasts and that member is available, even as members join or leave the pool. The same holds for the connections of a stream listener and the sessions of a native protocol listener; see Stream and native listeners.
backends:
app:
provider: alb
alb:
mechanism: p2c
pool: [ app1, app2, app3 ]
sticky:
ttl: 8h
secret_file: /etc/trickster/sticky.key
A session is kept in one of three ways, set by sticky.mode:
| Mode | How the member is remembered | Suits |
|---|---|---|
cookie (the default on http listeners) | The ALB issues a signed token naming the member in a cookie, which the browser sends back. The ALB keeps no state. | browsers |
header | The same token in a response header, X-Trickster-Session by default, which the client sends back in a request header of the same name. | API clients and SDKs that do not keep cookies |
table (the default on stream and native listeners) | The ALB remembers each client’s member by a key read from its requests, connections or sessions, such as its address or a header. | clients with a stable identifier of their own, upstreams that hand out their own session ids, and every listener that is not http |
The settings, with their defaults:
sticky:
mode: "" # cookie, header or table; unset is cookie on http listeners, table on the others
ttl: 1h # a session's lifetime from its first request; 0 is no limit
idle: 0 # ends a session left unused, as Session lifetimes describes; 0 never does
on_unavailable: repick # repick or reject; see below
secret: "" # the key tokens are signed with; see below
secret_file: "" # a file holding it, in place of secret
cookie: # cookie mode only
name: trickster_sticky
path: /
domain: "" # unset keeps the cookie to the host that set it
secure: auto # auto marks it Secure when the request arrived over TLS; or true, false
http_only: true
same_site: lax # lax, strict or none, which requires secure: true
lifetime: permanent # or session, which never sets Max-Age; see below
mark_private: false
header: # header mode only
name: X-Trickster-Session
table: # table mode only
key: client_ip
learn: request # or response; see below
ipv6_prefix: 64
max_entries: 100000
ttl and idle are 0 or at least 1s. With mode set, a cookie, header or table block that the mode never reads is a configuration error. With mode unset, a cookie block (for http listeners) and a table block (for the others) may both be set, and a header block is refused.
Other settings that are refused: sticky on a mechanism other than rr, p2c, lc, lt and hrw; cookie or header mode on an ALB that serves no http listener; both secret and secret_file; a cookie.path that does not begin with /; a cookie name beginning __Secure- or __Host- without secure: true, or a __Host- one with a domain or a path other than /; same_site: none without secure: true; table.learn: response with a key other than header:<name> or cookie:<name>; and an ipv6_prefix above 128.
Tokens
In cookie and header mode the token carries the member, when the session began and when the token was issued, signed with HMAC-SHA256 under a key derived from sticky.secret. A token is bound to the ALB that issued it, so another ALB does not honor it. A client cannot choose its own member, and a token that is altered, signed with another key, issued by another ALB or past its lifetime is treated as no token: the request starts a new session. It is never an error.
A token is issued with the response to a client’s first request, including the 101 Switching Protocols that opens a WebSocket, when its session moves to another member, and, with idle set, once more than half of idle has passed since the token was issued, so that an active session does not lapse. A request sent to the member its token names is otherwise answered with no token, so a steady client sees none after its first.
A cookie carries a Max-Age that ends with its token when ttl is set. With ttl: 0, or with lifetime: session, it is a browser-session cookie, and the token itself still enforces ttl and idle. secure: auto marks the cookie Secure when the request reached Trickster over TLS; behind a load balancer that terminates TLS and forwards plain HTTP to Trickster, set secure: true. mark_private: true adds Cache-Control: private to a response that sets the cookie, unless that response is already private or no-store; see Sticky Sessions and Caching.
Two ALBs that set the same cookie (by name, domain and path) for a host they both serve would each replace the other’s token in a browser, so Trickster refuses to start with such a configuration. A browser sends a host’s cookies to every port and scheme, so this holds across listeners. To resolve it, give one of them its own cookie.name.
ALBs whose served hosts do not overlap may share a cookie name, as long as the cookie sets no domain. A cookie with no domain goes back only to the host that set it. The hosts an ALB serves are:
- every host, when it is the default backend, or when one of its paths registers on a listener and it has
any_host_routingor path routing on (the default). Path routing answers/<alb name>/...on every host, whateverhostslists. A path registers on a listener unless it is markeddispatch_onlyor names a handler the ALB lacks. The ALB’s default paths count, unlesspath_defaults_disabledis set; - otherwise, the
hostsits listener-registered paths answer for, joined by the hosts of its dispatchers, since a dispatched request keeps the host it arrived on. Its dispatchers are the ALBs that name it in their pool oruser_router, and the rules that route to it. An ALB withpath_routing_disabled: trueand nohosts, or whose paths are alldispatch_only, serves only its dispatchers’ hosts; - no host, when nothing but a mirror reaches it. A mirror discards its target’s response, so no cookie from it reaches a browser, and such an ALB never conflicts.
The secret
Every Trickster replica behind one address, and every restart, must sign with the same key for a session to survive moving between them. Set secret to at least 32 bytes, or secret_file to a file that holds them, such as a mounted Kubernetes Secret; trailing line breaks in the file are ignored. Tokens made with one key are never honored with another. A secret that still contains ${, an environment variable reference that was not expanded, is refused rather than used as a publicly known key.
With neither set, the ALB signs with a random key that lasts as long as the process, and Trickster logs a warning each time it loads such a configuration, for every ALB that issues tokens on an http listener. That suits a single instance that can afford to start every session over when it restarts. A configuration reload keeps the process’s key, so a reload alone never ends a session. See Sticky Sessions Across Replicas.
When the pinned member is unavailable
With on_unavailable: repick, the default, a request whose member is unavailable goes to the member the mechanism chooses, and its session moves there. With reject, the ALB answers 503 Service Unavailable instead, for applications that cannot survive a move, and the client keeps its token so that it returns to its member once that member is available again. A member that has left the pool has no session to return to, so its clients start new sessions in either case.
A member that is draining keeps its sessions while it is available.
On an http listener a session is pinned by its first response, whatever its status, so a session that began on a member answering with errors stays there until that member is marked unavailable, by its health check or by the pool’s floor.
Table mode
In table mode the ALB keeps a table of each client’s member, by table.key, which takes the same values as hrw.key. A request without the key is balanced by the mechanism and not remembered. An entry lasts ttl from when it was stored and ends early when unused for idle. A full table drops the least recently used of a sample of its entries to make room.
With table.learn: response and a header:<name> or cookie:<name> key, the ALB also remembers the value a member hands the client in its response, in the header of that name or in the Set-Cookie that sets that cookie. That keeps a session on the member that created it from the client’s very next request, which is what an upstream that issues its own session ids needs. For example, for an MCP server whose replicas each know only the sessions they created:
sticky:
mode: table
table:
key: header:Mcp-Session-Id
learn: response
A response that sets a new value pins that value to the member too; the entry for the old value lasts until it expires.
A table is kept in each Trickster instance’s memory. It survives a configuration reload that leaves the ALB’s mode, ttl, idle and table settings unchanged, but not a restart, and it is not shared between replicas: use cookie or header mode where sessions must outlive either, or see Sticky Sessions Across Replicas. With key: client_ip, every client behind one NAT address shares a session.
Session lifetimes
ttl and idle bound a session in every mode, but they count from different moments for a token and for a table entry:
cookie and header mode | table mode | |
|---|---|---|
ttl counts from | the session’s first request. A refreshed token keeps that start; a session that moves to another member starts over. | when the entry was stored. An entry stored again, because its session moved, starts over. |
idle counts from | when the token was last issued. A request more than half of idle after that is answered with a fresh token, so an unused session ends between half of idle and idle after its last request. | the entry’s last use, to within a second. |
| after a reload | the new ttl and idle apply at once to every token already issued, since a token records only when its session began and when it was issued | a change to mode, ttl, idle or table starts the ALB’s table empty |
Renaming the ALB or a member ends the sessions they had, since tokens and table entries refer to both by name. Moving from one mode to another starts every session over, as does a change of the signing key.
Stream and native listeners
A tcp, tls or udp listener, or a native protocol listener such as mysql, carries no token, so an ALB keeps the sessions of its connections there in a table. mode is best left unset: an ALB that serves both http and stream listeners then issues cookies to its requests and keeps its connections in a table. cookie and header mode are refused on a listener that is not http. With mode: table, requests and connections share one table, so a client whose requests and connections share a key keeps one member for both; table.key must then be readable on every listener the ALB serves, which in practice means client_ip.
table.key is read from each connection or session, as hrw.key is:
| Listener | table.key |
|---|---|
tcp | client_ip; proxy_tlv:<type> with proxy_protocol |
tls | client_ip, sni; proxy_tlv:<type> with proxy_protocol |
udp | client_ip |
native (mysql) | client_ip, user |
client_ip is the address a PROXY protocol header names when the listener trusts one. user is the name a native session authenticated as. A connection or session without its key is balanced by the mechanism and not remembered.
A connection is pinned to its member once it reaches it: a tcp or tls connection when it connects, a udp session when its member first answers, or when a one-way member’s session ends with no port-unreachable, and a native session when it is handed to its member. A member that cannot be reached is never pinned. A udp session already stays with its member for its whole life; what a pin adds is that the client’s next session, from any port, lands on the same member. A table entry is read only when a connection or session begins, so idle counts from the client’s last new connection or session, however long that one lasts.
With on_unavailable: reject, a connection whose pinned member is unavailable is refused, and so is one whose pinned member cannot be connected, however many connect_retries the ALB has: the retries would move the session. With repick, a failed connect is offered to other members as connect_retries allows, and its session moves to the one that connects. A native session that is refused fails to route, as it would with no member available.
Sticky Sessions and Nested ALBs
When a sticky ALB’s pool has ALBs in it, such as one per region or per service, its token or table entry keeps the whole path: the inner ALB the client was sent to and that ALB’s member. The inner ALBs need no sticky block of their own. An inner ALB whose mechanism fans requests out, or a User Router, is itself the session’s member, and chooses among its own members as it would with no session. When only the inner member becomes unavailable, the session stays with its inner ALB and moves within it. On a stream listener, a connection retried after its inner member failed to connect is likewise offered the inner ALB’s other members first.
When the inner ALB has no member left that can take the session (a draining member the session is pinned to still can), the session moves with on_unavailable: repick to the first other member the mechanism offers that can take it, and is refused with reject, keeping its token or table entry for when its inner ALB recovers. A new session that the mechanism sends to an inner ALB with no member is answered 502 Bad Gateway, or refused on a stream listener, as it would be with no sessions. Set propagate_health: true on the inner ALBs (see ALBs as Pool Members) to keep new sessions away from one while it has no healthy member.
The session covers one level of nesting: an ALB further in selects its member, or keeps sessions of its own if it has a sticky block, as it would with no outer session. An inner ALB directly in the sticky ALB’s pool follows the outer session, and uses a sticky block of its own only for requests that reach it without going through the outer ALB.
On a stream listener only the ALB that the listener maps to keeps sessions: an inner ALB’s own sticky block applies to its http listeners, and to stream listeners mapped to it directly, never to connections that reach it through another ALB.
Sticky Sessions and Caching
In front of cacheable content, prefer table mode. It adds nothing to responses and sends members nothing the client did not, so every cache between the client and the origin sees exactly what it would see without sessions. Weigh its two limits first:
- A session is only as stable as its key. With
key: client_ip, every client behind one NAT address or forward proxy shares one session, and so one member, and a client whose address changes, such as a phone moving between networks, starts a new session. Aheader:<name>orcookie:<name>key needs a value that the client sends, or that the application hands it andlearn: responsepicks up. A request without the key is balanced afresh every time. - A table belongs to one Trickster instance. A restart empties it, and another replica does not know its sessions. With several replicas, keep each client on one replica, or pair the table with
hrwon the same key; see Sticky Sessions Across Replicas.
Where those limits rule table mode out, use cookie mode with mark_private: true. header mode suits API clients with no shared cache in front of Trickster, since its token rides in responses and it has no mark_private.
A token interacts with caches in three ways:
- A member’s cache never stores it. The ALB adds a token to a response after its member has answered, so a response served from a member’s cache is given a token when its session needs one, like any other.
- The token travels upstream. A member receives the request with its token still in the
Cookieheader, or in theheader.nameheader. An origin that answersVary: Cookiemakes a member cache one copy per distinctCookieheader, and a cookie token gives every session a distinct one, so such content is cached once per session. - A cache in front of Trickster sees the token. HTTP lets a shared cache, such as a CDN, store a response that sets a cookie and replay it, cookie included, to other clients, which joins them to that session’s member. A token in a response header is replayed the same way to clients that echo it back. That costs balance, not security, since a pin grants no access. In cookie mode,
mark_private: truestops it by making each response that sets the cookieprivate, so a shared cache does not store that response. Only a session’s first response, a move and an idle refresh carry a token, so most responses stay shareable.
Sticky Sessions Across Replicas
In cookie and header mode the client carries its session, so every Trickster replica behind one address honors it, and so does a restarted one, as long as each replica:
- signs with the same key, from
secretorsecret_file(see The secret); - configures the ALB under the same name, since a token is bound to it;
- names the pool members alike, since a token refers to its member by name. A discovered member is named after its ALB and the member’s discovered name, or its address when it has none, so replicas that discover the same members agree.
Changing the key starts every session over once, since a token signed with the old key is no longer honored. The Kubernetes controller keys the ALBs it generates from a Secret; see The session key.
In table mode each replica keeps its own table, and a client that reaches a replica that has not seen it before is balanced there as a new session. To keep such a client on its member anyway, either:
- keep each client on one replica, with source-address affinity in whatever balances the replicas; or
- use
mechanism: hrwwithhrw.keyset to the same key astable.key. Every replica then sends a new session to the same member for as long as their pools agree, and a replica’s table keeps the sessions it has seen in place when the pool changes. A replica that has not seen a session chooses its member afresh after such a change.
table.learn: response learns a value only on the replica whose member set it, and the request that began that session had no value for hrw to hash, so it needs each client kept on one replica.
Sticky Session Metrics
trickster_alb_sticky_total{alb_name, result} counts requests, connections and native sessions by how their session fared: hit (sent to its member), miss (no token or table entry), expired, invalid, repick (its member was unavailable and it moved) and rejected. In table mode an expired entry counts as a miss. A request, connection or native session is counted once it reaches its member, or once it is refused; one that reaches no member, which the ALB answers 502 Bad Gateway or closes, is not counted. trickster_alb_sticky_entries{alb_name} is the number of entries in an ALB’s table.
WebSockets and Other Protocol Upgrades
A request that asks to switch protocols, such as a WebSocket handshake, is tunneled by the pool member it is sent to when the ALB’s mechanism sends each request to one member (rr, p2c, lc, lt, hrw) or is the User Router. The mechanisms that fan a request out (fr, fgr, nlm, tsm) have no one member to tunnel to, so they ignore the upgrade and serve the request as an ordinary one, as HTTP permits.
Load Balancing Stream Listeners
The mechanisms that select one member, rr, p2c, lc, lt and hrw, also balance the connections of a tcp or tls stream listener and the sessions of a udp one. They are the same mechanisms with the same weights; only what they measure differs. The mechanisms that fan a request out (fr, fgr, nlm, tsm) and the User Router need an HTTP request, and are refused on a stream listener. Two more mechanisms, race and mirror, serve only stream listeners.
| http | tcp and tls | udp | |
|---|---|---|---|
| unit of work | a request | a connection | a session: one client address and port |
in flight (p2c, lc, lt) | requests being served | connections open | sessions open |
latency (lt.signal) | first_write: the first byte sent to the client | connect (default): the time to connect to the member; or first_byte: the member’s first byte | first_reply: the member’s first datagram |
hrw.key | client_ip, host, header:, cookie:, query: | client_ip; sni on a tls listener; proxy_tlv:<type> with proxy_protocol | client_ip |
A connection or session stays on the member it was given until it ends, whatever the mechanism. client_ip is the address a PROXY protocol header names when the listener trusts one. To keep a client’s later connections on one member too, give the ALB a sticky block.
Two settings apply only to an ALB that selects one member on a stream listener, under alb.stream:
backends:
pg:
provider: alb
listener_names: [ postgres ]
alb:
mechanism: p2c
pool: [ pg1, pg2, pg3 ]
stream:
connect_retries: 1 # default 0
passive_health: # off unless present
failures: 3 # default
eject: 30s # default
max_ejected_percent: 50 # default
connect_retriesis how many other members atcportlsconnection is offered when it cannot connect to the one it was given. All attempts share the listener’sstream.connect_timeout. The default, 0, refuses the connection, which is what keeps a weighted split exact: a member under the reserved.invaliddomain exists to refuse its share, and is never retried past.udphas no connect to fail, so it is not retried.passive_healthtakes a member out of the pool afterfailuresconsecutive failed connects, foreject, without waiting for a health check. Only failures to reach the member count, never anything it sent. At mostmax_ejected_percentof the pool is out at once, and the last live member is never ejected. Whenejectends the member returns, unless its health check has it down. Onudp, where nothing connects, a member that answers a datagram with a port-unreachable is what counts as a failure.
Health checks work as they do for HTTP pools: a tcp:// member with a healthcheck.interval is probed by opening a connection to it. A udp:// member has no generic probe; rely on autodiscovery readiness or on passive_health.
Connect Race and UDP Mirror
Two mechanisms exist only for stream listeners, because they commit one flow to several members at once. Both use every healthy member that has an address to dial, ignore weight, and take neither connect_retries nor passive_health. An ALB that uses one must be mapped to the stream listener directly; it cannot be a member of another ALB’s pool.
The Connect Race (race, or connect_race) mechanism serves tcp and tls listeners. Each client connection is connected to several members at once, within the listener’s stream.connect_timeout; the first member to connect carries the connection, and the other attempts are closed. A member that is down or slow to accept costs the client nothing. stream.race_width is how many members are raced, from 2 to 8; the default is every member, up to 4. In a pool wider than the race, each connection starts one member further along, so the connects are shared across the pool. A race opens and discards connections on the members that lose, so use it where a connect is cheap for the member.
The UDP Mirror (mirror, or udp_mirror) mechanism serves udp listeners. Every datagram a client sends is copied to every healthy member, which suits one-way protocols such as statsd, syslog and NetFlow. The first healthy member in pool order answers the session: only its replies are relayed to the client, and the other members’ replies are discarded. A mirror pool has at most 8 members.
backends:
statsd:
provider: alb
listener_names: [ statsd ]
alb:
mechanism: mirror
pool: [ statsd1, statsd2 ]
db:
provider: alb
listener_names: [ postgres ]
alb:
mechanism: race
pool: [ pg1, pg2, pg3 ]
stream:
race_width: 2
Load Balancing Native Protocol Sessions
A listener that speaks a backend’s own wire protocol, such as a mysql listener, can map to an ALB that uses rr, p2c, lc or hrw. Each client session is committed to one pool member once it authenticates, and stays there until it ends; a session is the unit of work, so p2c and lc compare members by their open sessions. hrw.key is client_ip or user. lt is not available, since a session reports no latency to rank members by. Every pool member must be a backend of the listener’s own provider, listed directly, and autodiscovery is not supported. The ALB authenticates the listener’s clients, so it carries the authenticator_name that a User Router on the same listener would.
backends:
replicas:
provider: alb
listener_names: [ mysql ]
authenticator_name: mysql-clients
alb:
mechanism: lc
pool: [ replica1, replica2 ]
With a sticky block, a client’s later sessions keep to the member its first one was given, by client_ip or, with table.key: user, by the name it authenticated as.
Weights and the Selection Mechanisms
Every mechanism that selects one member per request honors the pool weight, but what a weight promises differs:
| Mechanism | A weight is | Guarantee |
|---|---|---|
| rr | a share of requests | exact: weight of every totalWeight consecutive requests |
| p2c | a capacity | proportional on average: heavier members are drawn more often and compared by requests in flight per unit of weight |
| hrw | a share of the keys | proportional on average over many keys; changing a weight moves as few keys as possible |
| lc | a capacity | members are kept at equal requests in flight per unit of weight |
| lt | a bias | the score is divided by the weight, which shifts load under contention; an idle pool still sends every request to its best-scoring member |
If you need a guaranteed split, use rr.
Time Series Merge
The Time Series Merge mechanism supports both High Availability and federation. Each physical backend represents one logical data shard. Set the backend-level replica_group option to the same value on physical backends that are HA replicas of that shard. TSM first coalesces those replicas, using configured pool order to resolve overlapping points and later replicas to fill gaps, and then reduces the distinct logical shards.
When a TSM pool member is itself an ALB (for example, a round-robin ALB over
Prometheus backends), set replica_group on that immediate nested ALB. Trickster
uses the wrapper as the replica-group boundary while delegating TSM planning and
finalization to its terminal Prometheus provider. Other ALBs and non-TSM
providers cannot set an explicit replica group.
When replica_group is omitted, it defaults to the backend name, so existing configurations continue to treat every backend as a distinct shard. Explicitly set it for HA pools that use non-idempotent aggregations such as sum, count, or avg; otherwise replicas will be counted as separate data.
Replica grouping is backend-global, not ALB-specific. A backend cannot be a replica in one ALB and a disjoint shard in another; define a second backend entry if both views are required. Partially overlapping datasets are not representable: one backend belongs to one logical shard for all TSM queries. Injected labels remain useful output and routing metadata, but they do not establish replica provenance.
If replicas disagree at the same logical point, the first configured member wins deterministically and Trickster records a conflict metric and warning log. A failed replica does not make a response partial when another replica covers its group. If an entire logical group is unavailable, the response is marked partial and includes a warning.
For request paths that are not mergeable by the configured time series provider, TSM does not fan the request out. Those requests are dispatched directly to the first live pool target. The same first-live-target fallback is used when a request cannot be prepared for the merge path.
By default, TSM merges the answers its members give to a query. A TSM ALB can instead read its members’ raw series and evaluate each query once, the way Thanos does; see Global Evaluation.
Merge Strategy
Within each configured replica group, TSM deduplicates values when merging series with identical labels — for each timestamp, only one replica value is kept. Across different groups it uses the query’s merge strategy.
For Federation use cases where backends hold different, non-overlapping data, Trickster automatically selects a merge strategy per query by parsing it with the upstream Prometheus PromQL parser and inspecting the outermost aggregation operator. No configuration is required. Because selection uses the parsed expression, redundant parentheses, comments, whitespace, keyword case, and by/without placement do not change the selected strategy. This is particularly important for PromQL aggregation queries like sum() or avg(), which strip labels from results and cause series from different backends to appear identical.
| Outer Operator | Trickster Merge Behavior |
|---|---|
sum | Sum of values per unique label set + timestamp |
count | Sum of values per unique label set + timestamp |
count_values | Sum of values per unique label set + timestamp |
min | Minimum value per unique label set + timestamp |
max | Maximum value per unique label set + timestamp |
group | Deduplicate per unique label set + timestamp |
avg | Dual queries (avg→sum and avg→count); weighted arithmetic mean per unique label set + timestamp |
topk, bottomk | Query the inner expression across backends, merge it, then apply final top/bottom-k selection per timestamp and aggregation group |
stddev, stdvar | Pool shard-local count, mean, and variance states, then finalize the global population variance or standard deviation |
quantile | Query the inner expression, merge all float samples globally, then calculate the exact quantile per timestamp and aggregation group |
limit_ratio | Apply Prometheus-compatible label-hash sampling, globally finalizing a supported inner aggregation when necessary |
limitk | Query the inner expression, merge it globally, then retain the first k samples in stable TSM series order per timestamp and aggregation group |
count or group over an inner count or group of a selector | Query the inner aggregation, union its label sets across backends, then count or group each label set once per timestamp and aggregation group |
sum, count, or count_values followed by or vector(0) | Sum of values per unique label set + timestamp |
| (none) | Deduplicate (default) |
For avg queries, Trickster issues two concurrent sub-queries per backend shard — one rewriting the outer avg to sum and another to count — then computes a true weighted arithmetic mean (sum_total / count_total) per series per timestamp. This avoids the skew introduced by a naïve avg-of-averages when backends have different data cardinalities.
When an outer aggregation’s input contains a nested aggregation (other than the count and group forms below), binary expression, or function that needs globally complete input, Trickster retains the aggregation’s established merge strategy and adds a warning. This fail-open behavior preserves correct results when the input series are colocated while noting that results can be inaccurate when matching series are split across shards.
For an outer count or group whose input is a count or group of a vector selector, such as count(count by (pod) (up)), the result depends only on which label sets the inner aggregation produces. Trickster sends the inner aggregation to each backend, unions the label sets the backends return, and applies the outer operator once per timestamp and aggregation group. A label set returned by several backends counts once, so replicated or partially overlapping backends don’t inflate the count. The selector may use label matchers, offset, and @, and the aggregation may be wrapped in sort(), sort_desc(), or a literal scalar operation such as > 1. A label set counts at a timestamp when any backend returned it there. Inner inputs that apply functions, filters, or binary operations can change which label sets a backend returns, so they keep the warning described above. When a backend fails, the result covers the backends that answered and is marked as a partial hit.
For sum, count, or count_values followed by or vector(0), Trickster sends the complete expression to each backend and sums the results, because a backend without matching series contributes only the explicit zero. This requires an aggregation input that each backend can evaluate independently: no nested aggregation, binary expression, or function that needs globally complete input. When the or uses on(...) or ignoring(...) label matching, this applies only to sum or count without by or without grouping; other forms use the warning fallback described below.
For topk and bottomk, Trickster sends the inner expression to each backend, merges those inner results using the inner expression’s merge strategy, then applies the final rank-and-trim step per timestamp and aggregation group. This prevents each backend’s local topk/bottomk result from being weighted equally during the merge. If the inner expression is avg, Trickster still uses the weighted sum/count rewrite before applying the final rank. This also applies when the rank aggregation is wrapped in sort() or sort_desc().
For stddev and stdvar, Trickster requests the shard-local count, mean, and population variance and pools those states before finalizing the requested global value. Native histograms are excluded from this float-only calculation. Supported already-aggregated inner expressions such as count, min, max, and group are merged globally before the outer variance aggregation.
For quantile, Trickster sends the inner expression to every backend, globally merges supported inner aggregations, ignores native-histogram samples, and calculates Prometheus’s exact sort-and-interpolate value independently for each timestamp and group. Exact quantiles require every relevant float sample, so their fanout responses can be substantially larger than shard-local quantiles and remain subject to the configured response-capture limits. A capture-limit failure is returned rather than silently substituting an approximate result.
For limit_ratio, selection uses the same complete-label-set hash threshold as Prometheus. Supported inner aggregations are merged before the ratio is applied; shard-local expressions can be sampled by each backend because the hash decision for a given label set is independent of shard placement.
For limitk, Trickster reproduces the current Prometheus evaluator’s first-visited algorithm over TSM’s stable merged-series order: lexicographic JSON serialization of the complete label set, followed by the series name. Selection is independent for every timestamp and aggregation group, retains complete labels and both float and native-histogram samples, and does not rank candidates by value or label hash.
Prometheus does not define a canonical storage visitation order for limitk. A separate Prometheus deployment whose storage returns the same series in a different order may therefore select different labels. Trickster guarantees the requested cardinality, grouping, and repeatability for the same merged input, and fanout completion order does not affect the result. limitk remains an experimental PromQL operator, so this compatibility contract may change with Prometheus.
For an unsupported inner expression, Trickster retains the established per-shard fallback and injects a warnings entry in the Prometheus response body to alert the caller that results may be inaccurate.
The same deduplicating fallback and warning apply when an aggregation is not the query’s outermost operation (for example, histogram_quantile(0.9, sum by (le) (rate(x_bucket[5m])))), when an aggregation is combined with a binary expression other than the supported or vector(0) form, and when the query cannot be parsed as PromQL, such as a query that uses another dialect’s extensions.
When a non-dedup strategy is in effect, or Trickster sends backends an inner expression to finalize after merging, and backends have injected labels configured, those labels are automatically stripped before merging. This ensures series from different backends hash identically for aggregation, and the injected labels do not appear in the response.
Native Histograms
Native histogram samples are preserved through the merge rather than being numerically aggregated. When a timestamp has a histogram on one backend and a float sample on another (or histograms on both), the histogram value is kept as-is — numeric aggregators like sum only apply across float samples. This prevents mixed-type series from being corrupted into garbage values when backends return a mix of float and histogram samples at the same timestamp.
Max Query Range Limitation
Trickster ALB supports enforcing a max_query_range duration on ALB backends. For details on how to configure and use query range limits, see the Query Range Limits documentation.
Step Alignment
Members’ responses combine at identical timestamps, so members must answer on one grid: a member that drops the partial bucket at the start while another returns it whole would merge into a series whose first points are summed over a subset of members. A tsm ALB therefore applies one step alignment mode to every member. It uses its own step_alignment when set; otherwise it uses its first configured member’s mode, whether or not that member is healthy, so a leader’s outage doesn’t move every open dashboard’s grid. A query’s trickster-step-align directive wins when every member supports the mode it names.
A static pool member that can’t apply the chosen mode fails the configuration. When autodiscovered members can’t, the pool uses truncate if every member supports it, or otherwise leaves each member to its own mode, and adds a warning to the merged response.
step_alignment on an ALB using any other mechanism applies that mode to every member too, so the chart’s shape doesn’t depend on which member answered. Without it, those mechanisms leave members to their own modes, and Trickster logs a warning at startup for each ALB whose members differ.
Members With a Series Limit
Some managed services, such as Amazon CloudWatch PromQL, cut a query result short at a series limit and flag it with a warning. Each member applies its own limit, so a member’s truncated result is proxied and never cached, while the other members’ results are cached as usual. The merged response then covers only the series each member returned, and it carries the truncation warning so the client can tell.
Global Evaluation
Merging members’ answers is exact only for queries whose aggregation can be split across shards, as described in Merge Strategy. A query such as sum(rate(x[5m])) over replicas that differ by a label, or histogram_quantile(...) over buckets split across members, needs every raw sample in one place. With tsm.evaluation: global, a TSM ALB answers each instant and range query the way Thanos Querier does: it reads the raw series the query selects from every member, deduplicates replicas, and evaluates the whole expression once in an embedded Prometheus engine.
backends:
mimir-east:
provider: prometheus
origin_url: http://mimir-east:8080/prometheus
prometheus:
flavor: mimir
mimir-west:
provider: prometheus
origin_url: http://mimir-west:8080/prometheus
prometheus:
flavor: mimir
mimir-global:
provider: alb
cache_name: default
alb:
mechanism: tsm
tsm:
evaluation: global # merge (the default) or global
replica_labels: [replica] # labels that tell copies of one series apart; no default
deduplication: penalty # penalty (the default) or chain
partial_response: true # the default
max_samples: 50000000 # the default
query_timeout: 2m # the default
max_concurrent_queries: 20 # the default
max_fetched_series: 0 # the default, unlimited
max_fetched_samples: 0 # the default, unlimited
raw_cache_window: 31d # the default
pool:
- mimir-east
- mimir-west
The evaluation is set per ALB, and covers every query the ALB receives; other TSM ALBs keep merging. Requests for other paths, such as labels and series, are merged as usual. The settings below evaluation are valid only with evaluation: global.
Members. Every pool member must be a prometheus backend with a series source: flavor: mimir reads Prometheus remote read, flavor: thanos reads Thanos StoreAPI at the member’s series_endpoint, and any other Prometheus backend can set series_api. Configuration fails for any other member, including a nested ALB. With autodiscovery, the template_backend must be such a backend. A template’s series_endpoint, including the one StoreAPI requires, is used unchanged by every discovered member, so every member reads that one store and Trickster logs a warning at startup; remote read without a series_endpoint instead reads each member’s own origin. A discovery query.scheme of tcp or udp gives no such origin and fails configuration, and a discovered member whose series endpoint still can’t be formed is logged and left out of the pool. Every live member is read, whatever its replica_group. Each read reuses the member’s TLS settings, and carries the client’s request headers and the member path’s request_headers.
Deduplication. Series whose labels match once replica_labels are removed are replicas of one series, and the result keeps one series per such label set, without the replica labels. deduplication: penalty follows Thanos: it reads one replica until that replica falls silent for twice its sampling interval, then switches to another. When the query reads the series through rate, increase, irate or resets, a switch to a replica whose counter is behind is adjusted so that it doesn’t read as a counter reset. When replicas sample at the same instant, the earliest-configured member’s replica is read first. deduplication: chain instead keeps every distinct timestamp across replicas, and the earliest-configured member’s value where they share one. Series with identical labels from different members are always combined like chain. Leaving replica_labels empty still combines identical series. A native histogram’s counter reset hint describes its change from the previous sample of its own source, so where deduplication moves to another replica or copy, or passes over samples, the hint is cleared and the engine finds any reset from the counts, as Prometheus does where it merges series.
Partial responses. When a member is unavailable or fails to answer, by default the result is evaluated over the members that answered, and carries a warning naming the member, as Thanos does with partial responses enabled. Why the member failed is logged rather than returned, since a transport error can name internal hosts. Set partial_response: false to fail such a query with a 503 instead, as soon as a member fails, stopping the reads of the others. A request’s partial_response parameter overrides the ALB’s setting. A failed evaluation is returned as a Prometheus API error, never replaced by a merged answer.
Limits. max_samples bounds the samples the engine holds at once, like Prometheus’s --query.max-samples. query_timeout bounds an evaluation from its arrival, including any wait for one of the ALB’s max_concurrent_queries slots; a request’s timeout parameter can shorten it. max_fetched_series and max_fetched_samples bound the raw series and samples one evaluation reads from all its members before deduplication: the samples each selector reads within its range, whether from a member or the raw series cache, and the series with any such sample. What the raw series cache serves for a selector is counted before any member is read for it, and what members send as their answers arrive, so a read stops as soon as its samples and the cache’s pass the allowance. As a member’s series are mostly those the cache serves too, a read counts series only past as many as the cache served, and each selector’s exact count is settled once its answer is assembled. A series counts once for each selector and member, however many frames, messages, or uncached time ranges carry it. With max_fetched_series set, series are told apart by a hash of their labels as they arrive, so a copy of a series that a member sends apart from the rest of it counts once too; without it, such a copy may count again. A read stops as soon as the evaluation’s allowance runs out, and a read several evaluations share counts against each of them, stopping only when none can afford more. Each member’s raw response, and each StoreAPI message or remote-read frame in it, is bounded by the ALB’s max_capture_bytes. Only the chunks and samples within a selector’s range are kept from a response, and what is kept from one response, with the buckets its native histogram chunks lay out as they are read, may take at most four times max_capture_bytes in memory once decoded, so a response can’t expand far past its size even when the fetch limits are unlimited. Deduplication reads every copy and replica of a series at once, so the buckets their chunks lay out count together. Responses of float chunks stay within that bound; native histograms expand more, integer ones sent as samples to around eight times their size, so a response of them can reach it before max_capture_bytes. A histogram chunk whose bucket layout declares more buckets than its data holds is refused as malformed, as is one whose samples couldn’t be read: spans offset back after the first, in a schema finer than the engine reads, or custom-bucket spans offset back, on the negative side, or reaching past their bounds. A histogram sent as a sample is refused for the same layouts, and for spans laying out more or fewer buckets than it holds. max_capture_bytes also bounds the result an evaluation encodes as its response, whether or not the result can be cached: encoding stops as soon as the result passes it, and the query fails whole with execution, never returning part of the result. A failed evaluation’s error stays within it too, or within 4 KiB when that is more, leaving out warnings and shortening its message as needed. A result takes some 15 to 35 bytes of JSON for each sample it returns, where members’ chunk-encoded reads often take one or two, so an ALB that returns raw series over wide ranges needs a max_capture_bytes sized for its results as well as its members’ reads. A query that reaches one of these limits or times out fails, even when partial responses are allowed; a limit a member enforces itself, such as Mimir’s query limits, is that member’s failure. Errors use Prometheus’s types and status codes: bad_data (400), execution (422, which includes the limits), canceled (499), internal (500), and timeout or unavailable (503). Once a member’s read of one of a query’s selectors reaches a limit, the other members’ reads of that selector stop, and the query fails with execution, even when any other of its reads also failed, of that selector or another, or it also timed out or its client canceled it as the limit was reached.
The engine. The engine is the Prometheus version Trickster is built with, with a 5-minute lookback, the @ modifier, negative offsets and the experimental functions enabled, a 1-minute step for subqueries that name none, and at most 11,000 points per series. The query’s limit parameter is honored. A request is validated as Prometheus validates it before any cache lookup or raw read, except that a range query’s step must be a whole number of milliseconds, the engine’s resolution, rather than be evaluated at another step; a timeout must be positive, a limit a non-negative integer, and partial_response a boolean.
Caching. A global ALB keeps each complete evaluation in its own cache_name (default when unset), whole. Its range queries are not delta-cached, as a Prometheus backend’s are: penalty deduplication follows replicas from a range’s first sample, so samples evaluated as part of a range can differ from the same samples evaluated with the whole of it. A cached result answers only the same query with the same time, or the same start, end and step, so a dashboard’s moving range is evaluated each time, while repeats and concurrent viewers are answered from the cache. A result that reaches into the recent past, within the 5-minute lookback or the ALB’s volatile_window, is kept for up to 30 seconds, while samples may still arrive; an older one is kept for the ALB’s timeseries_ttl. A partial result or a failed evaluation is never cached, and is answered with Cache-Control: no-store.
The cache key holds the query’s limit, and its timeout and partial_response resolved against the ALB’s settings, so requests that share a result or an evaluation agree on how it ends. It also holds the Authorization header, the X-Scope-OrgID and THANOS-TENANT tenant headers, the cache_key_headers, cache_key_params and cache_key_form_fields configured on the ALB’s query paths and on the ALB path a request arrives on, such as an alias like /prometheus/api/v1/query_range, and the cache_key_headers configured on its members’ query paths, so no tenant or scope reads another’s results. Results are kept under a namespace that digests the ALB’s members, in order, with each one’s series endpoint, the request headers its query paths send, and its TLS client and SigV4 settings. The namespace changes when membership does, as when a discovery adds or removes a member, and when a member’s configuration does, but not with health, so results kept before a member’s outage stay usable. A request’s evaluation reads the members whose namespace keys its result, even when the pool changes while it runs.
Raw series caching. Beneath its results, a global ALB with a cache also keeps the raw series it reads, so each member is read only for samples the cache doesn’t hold. A member’s series are kept per selector, in two-hour blocks aligned to the epoch, keyed by the member’s series endpoint and configuration (as for the namespace above), the selector’s matchers, and the identity headers that key results, as the member receives them. Samples within a member’s volatile window are read from it by every evaluation and never kept, since replicas’ samples may still arrive; older samples are read once and kept for the ALB’s timeseries_ttl. A dashboard’s moving range therefore reads only its newest minutes from each member, and a wider range reads only the part the cache lacks. The volatile window is 5 minutes for members with flavor: mimir or flavor: thanos, and none for other members; an ALB that sets volatile_window uses it for every member instead, so set it to cover any out-of-order window your members accept. Samples older than raw_cache_window, 31 days by default, aren’t kept either: a range reaching further back reads that older part from the members every time, and the rest from the cache. The window counts whole blocks, and each block a range spans is one cache lookup per selector and member. A selector with an exact metric name can also be answered from the cached blocks of a broader one on the same metric, member and identity, whose matchers are all among its own: up{job="api",code=~"5.."} from up{job="api"}’s blocks, filtered to the series it matches. Each Trickster instance remembers in memory which selectors have cached blocks; nothing more is stored. Concurrent evaluations needing the same samples share one read, and a selector’s missing ranges are read from a member a few at a time. Every chunk of a member’s answer is read through, and the bucket layout of every histogram in it checked, whether the answer arrives as chunks or as samples, before it is kept, so a chunk or histogram that can’t be read fails that member’s read rather than being cached. An answer a member gave incompletely, or failed to give, is never kept. If keeping an answer fails anyway, the evaluations that read it already have their answer, the failure is logged, and its samples are read from the member next time; a cached block that fails as they are merged into it is dropped, and its samples are read again too. The ALB’s max_object_size_bytes, 64 MB by default for a TSM ALB, bounds each block as it bounds each cached result: samples that would take a block past it aren’t kept, so they are read from the members every time while the block keeps what it held. A block refused some samples isn’t tried with them again until timeseries_ttl passes, unless it no longer holds what it held then, as once evicted, when they may fit; each refusal counts in trickster_cache_events_total as event error with reason tsm_global_raw_max_object_size. Blocks are keyed by the limit too, so ALBs of different limits sharing a cache keep their own, and changing the limit starts new blocks while the old ones expire. A memory cache measures a block by the memory it holds, and other caches by its encoded size. The small chunks that frequent refreshes add to a block are encoded again into full ones as they accumulate, so a block kept current every few seconds ends as compact as one read whole. Each selector reads exactly the samples from its start to its end, cached or not, so cached and uncached evaluations agree, and penalty deduplication starts at a selector’s first sample.
Concurrent requests for the same result share one evaluation. The request that starts it owns it: when its client goes away, the evaluation and its raw reads are canceled, and a request that joined it runs the query again.
Reloads and shutdown. A configuration reload leaves the replaced ALB’s evaluations running for the reload_drain_timeout. Then that ALB admits no more evaluations and cancels those still running, which are answered unavailable (503) unless one already failed with execution, as on reaching a limit, which decides its answer as it would a timeout’s, and waits up to the same timeout again for raw series blocks still being stored; nothing a raw read returns from then on is stored. Only after that are its members’ StoreAPI connections closed, discovered members’ included. A memory or redis cache that the new configuration changes or removes, and that a replaced global ALB keeps its results in, is closed after that too, so the blocks still being stored reach it; until then, a memory cache’s objects stay in memory alongside those of the cache replacing it. A cache that keeps its objects in local files (filesystem, bbolt, badger) is closed as the reload applies, since its replacement can open the same files, so it refuses what the replaced ALB stores in it during the drain. A changed or removed cache that no global ALB uses is closed as the reload applies too, once the operations under way in it end, whatever its provider. A reload that fails restores the previous ALB, which keeps reading its discovered members over the connections it already has. Shutdown does the same once the listeners have drained: retiring the evaluators and closing the caches share one shutdown_drain_timeout. A discovered member that autodiscovery removes or rebuilds has its StoreAPI connection closed once the reload_drain_timeout passes, and one it refuses as it builds it, at once.
Reading raw series moves much more data than reading answers, as it does in Thanos: plan max_capture_bytes and the fetch limits for the widest dashboard ranges. To compare global evaluation with Thanos or with merging, see the mimir and thanos profiles of the developer environment.
Providers Supporting Time Series Merge
Trickster currently supports Time Series Merging for the following TSDB Providers:
| Provider Name |
|---|
| Prometheus |
We hope to support more TSDB’s in the future and welcome any help!
Example TS Merge Configuration
backends:
# prom01a and prom01b are redundant and poll the same targets
prom01a:
provider: prometheus
replica_group: prom01
origin_url: http://prom01a.example.com:9090
prometheus:
labels:
region: us-east-1
prom01b:
provider: prometheus
replica_group: prom01
origin_url: http://prom01b.example.com:9090
labels:
region: us-east-1
# prom-alb-01 scatter/gathers to prom01a and prom01b and merges responses for the caller.
# Enforces a max 14-day time range limit on all incoming merge requests.
prom-alb-01:
provider: alb
max_query_range: 14d
alb:
mechanism: tsm # or time_series_merge
pool:
- prom01a
- prom01b
# prom02 and prom03 poll unique targets but produce the same metric names as prom01a/b
prom02:
provider: prometheus
origin_url: http://prom02.example.com:9090
labels:
region: us-east-2
prom03:
provider: prometheus
origin_url: http://prom03.example.com:9090
labels:
region: us-west-1
# prom-alb-all scatter/gathers prom01a/b, prom02 and prom03 and merges their responses
# for the caller. The merge strategy is automatically selected per-query based on the
# outer PromQL aggregation operator. Injected labels are automatically stripped before
# merging so that series from different backends are combined correctly. Because prom01a
# and prom01b are in the same replica_group, their values are de-duplicated before being
# merged/reduced with prom02 and prom03.
prom-alb-all:
provider: alb
alb:
mechanism: tsm
pool:
- prom01a
- prom01b
- prom02
- prom03
Here is the visual representation of a basic TS Merge configuration:

First Response
The First Response mechanism fans a request out to all healthy pool members, and returns the first response received back to the client. All other fanned out responses are cached (if applicable) but otherwise discarded. If one backend in the fanout has already cached the requested object, and the other backends do not, the cached response will return to the caller while the other backends in the fanout will cache their responses as well for subsequent requests through the ALB.
This mechanism works well when using Trickster as an HTTP object cache fronting multiple redundant origins, to ensure the fastest response possible is delivered to downstream clients - even if the HTTP Response Code indicates an error in the request or by the first backend to respond.
First Response Configuration Example
backends:
node01:
provider: reverseproxycache
origin_url: http://node01.example.com
node02:
provider: reverseproxycache
origin_url: http://node-02.example.com
node-alb-fr:
provider: alb
alb:
mechanism: fr # or first_response
pool:
- node01
- node02
Here is the visual representation of this configuration:

First Good Response
The First Good Response (fgr) mechanism acts just as First Response does, except that it waits to return the first response with an HTTP Status Code < 400. If no fanned out response codes are in the acceptable range once all responses are returned (or the timeout has been reached), then the healthiest response, based on min(all_responses_status_codes), is used.
This mechanism is useful in applications such as live internet television. Consider an operational condition where an object may have been written to Origin 1, but not yet written to redundant Origin 2, while users have already received references to and begin requesting the object in a separate manifest. Trickster, when used as an ALB+Cache in this scenario, will poll both backends for the object and cache the positive responses from Origin 1 for serving subsequent requests locally, while a negative cache configuration will avoid potential 404 storms on Origin 2 until the object can be written by the replication process.
Custom Good Status Codes List
By default, fgr will return the first response with a status code < 400. However, you can optionally provide an explicit list of good status codes using the fgr.status_codes configuration setting, as shown in the example below. When set, Trickster will return the first response to be returned that has a status code found in the configured list. An entry may be a single code or an inclusive range, and the two forms mix: status_codes: [ { start: 200, end: 299 }, 304 ].
First Good Response Configuration Example
negative-caches:
default: # by default, backends use the 'default' negative cache
"404": 500 # cache 404 responses for 500ms
backends:
node01:
provider: reverseproxycache
origin_url: http://node-01.example.com
node02:
provider: reverseproxycache
origin_url: http://node-02.example.com
node-alb-fgr:
provider: alb
alb:
mechanism: fgr # or first_good_response
pool:
- node01
- node02
fgr:
status_codes: [ 200, 201, 204 ] # only consider these codes when selecting a response
Here is the visual representation of this configuration:

Newest Last-Modified
The Newest Last-Modified mechanism is focused on providing the user with the newest representation of the response, rather than responding as quickly as possible. It will fan the client request out to all backends, and wait for all responses to come back (or the ALB timeout to be reached) before determining which response is returned to the user.
If at least one fanout response has a Last-Modified header, then any response not containing the header is discarded. The remaining responses are sorted based on their Last Modified header value, and the newest value determines which response is chosen.
This mechanism is useful in applications where an object residing at the same path on multiple origins is updated frequently, such as a DASH or HLS manifest for a live video broadcast. When using Trickster as an ALB+Cache in this scenario, it will poll both backends for the object, and ensure the newest version between them is used as the client response.
Note that with NLM, the response to the user is only as fast as the slowest backend to respond.
Newest Last-Modified Configuration Example
backends:
node01:
provider: reverseproxycache
origin_url: http://node01.example.com
node02:
provider: reverseproxycache
origin_url: http://node-02.example.com
node-alb-nlm:
provider: alb
alb:
mechanism: nlm # or newest_last_modified
pool:
- node01
- node02
Here is the visual representation of this configuration:

User Router
The User Router mechanism is used to control a Request’s destination Backend based on the username in the request. A default Backend (for no-user and users not in the manifest) can be configured, as well as a Backend per-user.
Native MySQL listeners use a deliberately narrower User Router topology than
HTTP backends: one authenticated listener-facing User Router may select only
direct terminal MySQL backends, selection is sticky for the session, and
to_user/to_credential remapping is rejected. See the
MySQL Provider Guide for the complete
authentication, routing, health, cache-identity, and no-route contract.
When a User Router ALB is configured to use an Authenticator, the ALB can also modify a Request’s credentials before passing it off to the destination Backend. In the graphic below, user casey will be routed to the readersBackend, which proxies to a read-only database server with the dbreader credentials; while user taylor will be routed to the writersBackend, which proxies to a read-write database server with the dbwriter credentials. Here is the example configuration corresponding to the graphic:
Credential replacement is applied only when the user’s configured to_backend target is selected. When a request instead uses default_backend - because the username has no mapping, the mapping does not name a usable runtime target, or the mapped target is unavailable - the request retains its inbound credentials. If a user should receive replacement credentials when routed to the same Backend that also serves as the default, set that Backend explicitly as the user’s to_backend.
backends:
readersBackend:
provider: clickhouse
origin_url: http://read.prod.db.com:8123/
writersBackend:
provider: clickhouse
origin_url: http://write.prod.db.com:8123/
click-lb-01:
provider: alb
authenticator_name: dbUsers
alb:
mechanism: ur # or user_router
user_router: # User Router-specific configs
default_backend: readersBackend # optional - users not in the list will route here, origin will 401
users:
casey:
to_user: dbreader # replaces user casey with dbreader in the request's Authorization header
to_credential: ${DB_READER_PW} # replaces credential in the Authorization header with this env
to_backend: readersBackend # explicit selection applies casey's credential replacement
taylor:
to_user: dbwriter # replaces user taylor with dbwriter in the request's Authorization header
to_credential: ${DB_WRITER_PW} # replaces credential in the Authorization header with this env
to_backend: writersBackend # taylor is sent to the writers backend
authenticators:
dbUsers:
provider: clickhouse # use the clickhouse authenticator
users_file: /path/to/user-manifest.csv # this file should include casey and taylor users
users_file_format: csv # required when users_file is set

Supported Backend Provider Types
The User Router mechanism supports all Backend provider types for default_backend and to_backend values, including other User Router ALBs.
However, config validation will fail if:
- there are any possible infinite loops between backends configured
- users could ultimately be routed to different non-virtual (ALB/Rule) backend types by the same User Router ALB. The final ultimate route for all users must be of the same type (regardless of how many additional hops through ALBs and Rules the request would take).
- In other words: user1 cannot be ultimately routed to a
clickhousebackend and user2 be ultimately routed to aprometheusbackend by the same User Router ALB.
- In other words: user1 cannot be ultimately routed to a
User Router without an Authenticator
If a User Router ALB does not use an Authenticator, you can still configure user-specific Backend routes. In these cases Trickster will observe (but not authenticate) the username in the request and route based on the observed username. However, Trickster will exit with a validation failure on startup if a User Router ALB that does not utilize an Authenticator is configured to swap credentials. In short: users must be positively authenticated by a Trickster Authenticator for credential swapping to be permitted by the User Router ALB.
When a User Router ALB doesn’t use an Authenticator, Trickster uses the final destination Backend provider type to select a default Authenticator (operating in observe-only mode / no users manifest) for username observation. For clickhouse-destined User Routers, the observe only Authenticator provider is clickhouse. For all other backend provider types, the default the observe only Authenticator provider is basic (Basic Auth).
to_user / to_credential vs Backend Path Header Injection
It is still possible to insert credentials to a Backend proxy request using the request_headers Backend Path config. But any request_headers alterations configured for auth-related headers (e.g., Authorization) are performed by the Backend after being handled by a User Router; so they would overwrite any user-specific to_user and to_credential transformations performed by the User Router ALB.
Default Backend
As shown in the example config above, you can provide a default_backend config to a User Router, and users who are not in the user router list will be routed to this backend.
If you do not supply a default_backend, users who are not in the manifest will receive a default response of 502 Bad Gateway. You can customize the default response code by setting no_route_status_code to a value between 400 and 599 as in this example:
backends:
prod-01:
provider: reverseproxy
origin_url: https://example.com/
users-lb-01:
provider: alb
alb:
mechanism: ur
authenticator_name: all-users # not shown for brevity, see above examples
user_router:
no_route_status_code: 401 # unauthorized response for users not in allow list
users: # allowed users
casey:
to_backend: prod-01
taylor:
to_backend: prod-01
kris:
to_backend: prod-01
User Router ALB Backend Pool and Health Checking
The User Router does not rotate through or fan out to a pool of Backends like
the other ALB mechanisms. A healthy mapped target is selected directly. When a
mapped target is unavailable, the request uses the healthy default_backend
without applying the mapped target’s credential replacement. If neither target
is available, the router uses its configured no-route response.
That fallback applies to HTTP requests. A native MySQL session whose username
has an explicit mapping fails with a MySQL availability error when that mapped
terminal is unavailable; it is never redirected to default_backend. Only an
unmapped MySQL username may use the configured default terminal.
You can configure a User Router ALB’s backend destinations to be other ALBs with mechanisms that utilize healthchecked pools.
Bounding Per-Member Response Captures
ALB mechanisms that fan out (TSM, FR, FGR, NLM) buffer each pool member’s response in memory before merging or selecting a winner. Without a cap, one misbehaving upstream returning an oversized body can OOM the proxy – an N-way fanout multiplies that by N.
Trickster applies a default cap of 256 MiB per response. A member whose body exceeds the cap is treated as a partial failure: the merged response carries an X-Trickster-Result: phit marker and the trickster_alb_fanout_failures_total{mechanism, reason="truncated"} metric increments.
Override the cap at the backend or ALB level:
backends:
default:
max_capture_bytes: 67108864 # 64 MiB, applies to all backends (Prometheus, ClickHouse, ALB members, etc.)
prom-alb-tsm:
provider: alb
alb:
mechanism: tsm
max_capture_bytes: 16777216 # 16 MiB, ALB-specific override
pool:
- prom01
- prom02
The ALB-level value takes precedence over the backend-level value, which in turn takes precedence over the 256 MiB default.
A TSM ALB with evaluation: global applies the cap to each member’s raw series response and to the result each evaluation encodes as its response. Unlike the fanout mechanisms above, it doesn’t treat a member’s response over the cap as a partial failure: the whole query fails with a Prometheus execution error (422), even when partial responses are allowed, with no X-Trickster-Result: phit marker and no truncated count in trickster_alb_fanout_failures_total. A result that would exceed the cap fails with the same error rather than being returned in part.
Bounding Aggregate In-Flight Captures
max_capture_bytes caps each member’s response individually; a fanout to N members can still buffer up to N * max_capture_bytes in flight. For deployments with large pools or low memory ceilings, set max_fanout_capture_bytes to cap the aggregate buffer across all in-flight slots in a single fanout call. Slots dispatched after the aggregate budget would go negative are fail-fasted (marked Failed, no capture buffer allocated) before the upstream handler runs; the merge sees them as partial failures and the existing fallback path handles it.
backends:
prom-alb-tsm:
provider: alb
alb:
mechanism: tsm
max_capture_bytes: 16777216 # 16 MiB per member
max_fanout_capture_bytes: 67108864 # 64 MiB total across all in-flight slots
pool:
- prom01
- prom02
- prom03
- prom04
max_fanout_capture_bytes defaults to 0 (no aggregate cap). Pick a value matching what your trickster instance can afford to buffer per request, independent of pool size.
Maintaining Healthy Pools With Automated Health Check Integrations
Health Checks are configured per-Backend as described in the Health documentation. Each Backend’s health checker will notify all ALB pools of which it is a member when its health status changes, so long as it has been configured with a health check interval for automated checking. When an ALB is notified that the state of a pool member has changed, the ALB will reconstruct its list of healthy pool members before serving the next request.
Health Check States
A backend will report one of three possible health states to its ALBs: unavailable (-1), unknown (0), or available (1).
Health-Based Backend Selection
Each ALB has a configurable healthy_floor value, which is the threshold for determining which pool members are included in the healthy pool, based on their instantaneous health state. The healthy_floor represents the minimum acceptable health state value for inclusion in the healthy pool. The default healthy_floor value is 0, meaning Backends in a state >= 0 (unknown and available) are included in the healthy pool. Setting healthy_floor: 1 would include only available Backends, while a value of -1 will include all backends in the configured pool, including those marked as unavailable.
Backends that do not have a health check interval configured will remain in a permanent state of unknown. Backends will also be in an unknown state from the time Trickster starts until the first of any configured automated health check is completed. A pool member in a permanent unknown state can never reach available, so a healthy_floor: 1 ALB whose members lack health checks would have an empty pool and return 502 for every request. To avoid that, Trickster resets such an ALB’s effective floor to 0 at startup, emits a warning naming the ALB and the un-probed members, and sets the trickster_alb_pool_floor_reset{backend_name} gauge to 1. Configure a health check interval on those members if you want healthy_floor: 1 to apply.
Setting healthy_floor below 0 admits members the probe has confirmed unavailable, not just members in the transient unknown state. If your goal is to keep traffic flowing during the cold-start window before the first probes complete, lower the pool members’ recovery_threshold so they transition out of unknown faster – don’t lower the floor. When healthy_floor < 0 Trickster emits a startup warning and sets the trickster_alb_pool_admits_failing{backend_name} gauge to 1.
Backup Pool Members
A pool member marked backup: true stands by: it receives traffic only while no other member of the pool is in the healthy pool. As soon as one of the other members returns, the backup members stand down again, apart from the sticky sessions they took meanwhile, which stay with them until they end. This applies to every mechanism that has a pool, and to stream and native protocol listeners as well as HTTP. A pool must have at least one member that is not a backup, unless its other members come from autodiscovery; discovered members are never backups.
backends:
db:
provider: alb
listener_names: [ postgres ]
alb:
mechanism: rr
healthy_floor: 1
pool:
- primary
- name: standby
backup: true
Failover depends on the ALB learning that its other members are down, so give them a health check interval or, on a stream listener, stream.passive_health. While an ALB with backup members is dispatching to them, the trickster_alb_pool_on_backup{backend_name} gauge is 1, and a warning is logged when it fails over.
Draining Pool Members
A pool member marked drain: true takes no new work but keeps the sticky sessions it already has, for as long as it is available, so that it can be retired without ending them. Every mechanism leaves a draining member out when it selects a member or fans a request out; only a sticky session can reach it. A pool must have at least one member that is not draining, unless its other members come from autodiscovery. When every member of the primary tier drains, new work goes to the backup members.
Draining ends no session by itself: a session keeps its draining member until its ttl or idle ends it, the member leaves the pool, or the member becomes unavailable. To retire a member without cutting sessions short, keep it draining until they end: a session lasts at most ttl, and ends sooner once unused for idle.
Discovered members drain too: a Kubernetes endpoint that is terminating but still serving, or a deleted pod that is still ready, stays in the pool as a draining member until it stops serving, so a rolling restart moves new sessions to the new pods while the old ones finish theirs. See autodiscovery. This applies to every discovered pool, sticky or not.
Draining members are listed on the health status page and exported by the trickster_alb_member_draining{alb_name, member} gauge, which is 1 for each draining member.
A Time Series Merge ALB, which keeps no sessions, merges nothing from a draining member. Draining the only live member of a replica_group therefore leaves that shard out of its merged results, and the ALB warns of a degraded pool, as it would had the member failed.
backends:
app:
provider: alb
alb:
mechanism: rr
sticky: {}
pool:
- app1
- app2
- name: app3
drain: true
ALBs as Pool Members
An ALB that is a member of another ALB’s pool is always treated as available, even when its own healthy pool is empty: it keeps its share of the outer pool’s traffic and fails it. That is deliberate for weighted splits, where an empty member must not shift its share onto its siblings. Set propagate_health: true on the inner ALB to change that. It then reports unavailable to the pools it belongs to while it has no healthy member, and available otherwise, so an outer pool whose healthy_floor excludes unavailable members sends that share to its other members instead.
backends:
region-east:
provider: alb
alb:
mechanism: rr
propagate_health: true
pool: [ east1, east2 ]
global:
provider: alb
alb:
mechanism: rr
pool:
- region-east
- name: region-west
backup: true
Example ALB Configuration Routing Only To Known Healthy Backends
backends:
prom01:
provider: prometheus
origin_url: http://prom01.example.com:9090
healthcheck:
interval: 1000ms # enables automatic health check polling for ALB pool reporting
prom02:
provider: prometheus
origin_url: http://prom02.example.com:9090
healthcheck:
interval: 1000ms
prom-alb-tsm:
provider: alb
alb:
mechanism: tsm # times series merge healthy pool members
healthy_floor: 1 # only include Backends reporting as 'available' in the healthy pool
pool:
- prom01
- prom02
All-Backends Health Status Page
Trickster 2.x provides a global health status page available at http://trickster:metrics-port/trickster/health or (the configured health_handler_path).
The global status page will display the health state about all backends configured for automated health checking, and every ALB with its pool members. Here is an example configuration and a possible corresponding status page output:
backends:
proxy-01:
provider: reverseproxy
origin_url: http://server01.example.com
# not configured for automated health check polling
prom-01:
provider: prometheus
origin_url: http://prom01.example.com:9090
healthcheck:
interval: 1000ms # enables automatic health check polling every 1s
prom-02:
provider: prometheus
origin_url: http://prom02.example.com:9090
healthcheck:
interval: 1000ms
prom-03:
provider: prometheus
origin_url: http://prom03.example.com:9090
healthcheck:
interval: 1000ms
prom-alb:
provider: alb
alb:
mechanism: rr
pool:
- prom-01
- prom-02
- name: prom-03
drain: true
$ curl "http://${trickster-fqdn}:8481/trickster/health"
Trickster Backend Health Status last change: 2026-09-26 00:00:00 UTC
-------------------------------------------------------------------------------
prom-02 prometheus unavailable since 2026-09-26 00:00:00 UTC
prom-01 prometheus available
prom-03 prometheus available
prom-alb alb (rr) available u:[prom-02] a:[prom-01,prom-03] d:[prom-03]
proxy-01 reverseproxy not configured for automated health checks
-------------------------------------------------------------------------------
For JSON, provide a 'Accept: application/json' Header or query param ?json
For YAML, provide a 'Accept: application/yaml' Header or query param ?yaml
ALB Pool Members on the Status Page
Each ALB is listed with its pool members grouped by health: u:[...] (unavailable), a:[...] (available) and nc:[...] (not checked) in the text form, and unavailablePoolMembers, availablePoolMembers, uncheckedPoolMembers and initializingPoolMembers in the JSON and YAML forms. Draining members are also listed under their health, and again in d:[...] (text) or drainingPoolMembers (JSON and YAML). An ALB is listed as available while its pool has a member it would send new work to: one that meets its healthy_floor, is not draining, and is in the tier in use. An ALB whose members all drain is therefore unavailable, whatever their health, although it still serves their sticky sessions. In the example, prom-alb is available because prom-01 takes new work; were prom-01 down too, it would be listed as unavailable while prom-03 went on serving the sessions pinned to it.
JSON Health Status
As the footer of the plaintext version of the health status page indicates, you may also request a JSON or YAML version of the health status for machine consumption. The JSON version includes additional detail about any Backends marked as unavailable, and is structured as follows:
$ curl "http://${trickster-fqdn}:8481/trickster/health?json" | jq
{
"title": "Trickster Backend Health Status",
"updateTime": "2026-09-26 00:00:00 UTC",
"unavailable": [
{
"name": "prom-02",
"provider": "prometheus",
"downSince": "2026-09-26 00:00:00 UTC",
"detail": "error probing target: dial tcp prom02.example.com:9090: connect: connection refused"
}
],
"available": [
{
"name": "prom-01",
"provider": "prometheus"
},
{
"name": "prom-03",
"provider": "prometheus"
},
{
"name": "prom-alb",
"provider": "alb",
"mechanism": "rr",
"availablePoolMembers": [
"prom-01",
"prom-03"
],
"unavailablePoolMembers": [
"prom-02"
],
"drainingPoolMembers": [
"prom-03"
]
}
],
"unchecked": [
{
"name": "proxy-01",
"provider": "reverseproxy"
}
]
}
7.2 - ALB Autodiscovery
Autodiscovery keeps an ALB pool’s membership current at runtime:
instead of hand-maintaining a pool list, Trickster watches a discovery
source and adds, updates, and drains pool members as the source changes —
with no config reload or listener restart. Discovered members are cloned
from a designated template backend, so they inherit its caching, TLS,
health check, path, and timeout configuration.
Supported Discovery Providers
| provider | discovers | change detection |
|---|---|---|
kubernetes | endpointslices, services or pods via the Kubernetes API | watch (event-driven) |
dns_srv | SRV records | poll |
dns_a | A/AAAA records | poll |
file | a member-list file | filesystem notify + poll |
http_sd | a member-list HTTP endpoint | poll (ETag-aware) |
consul | Consul service instances | blocking query (event-driven) |
nomad | Nomad’s native service registry | blocking query (event-driven) |
aws | EC2 instances or ECS tasks, by aws.service | poll |
gcp | Compute Engine instances, by gcp.service | poll |
azure | Virtual Machines, by azure.service | poll |
docker | Docker Engine containers | poll |
For other services, users are generally served by the file provider (any
external service-discovery tool can emit the member list), by
http_sd (the universal adapter — a few lines of
glue serving a member list, with no Trickster change), or by the DNS
providers (e.g. Consul’s DNS interface, cloud private DNS zones, Docker’s
embedded DNS). Developers can add providers against a simple interface; see
the provider authoring guide.
Configuration Overview
Autodiscovery has three configuration parts:
- a named discoverer in the top-level
discoverysection — the provider and its connection-level settings (Kubernetes client, DNS resolver). Multiple ALBs can share one discoverer, and therefore one client/watch/resolver stack. - a template backend — a normal backend definition marked
is_template: true. Templates are never routed, never eligible foris_default, never static pool members, and don’t require anorigin_url; they exist to be cloned per discovered member. - the ALB’s
alb.discoveryblock — which discoverer to use, a provider-specificquerydescribing what to select, the template to clone, and guardrail options.
discovery:
in-cluster:
provider: kubernetes
kubernetes:
in_cluster: true
backends:
prom-template:
provider: prometheus
is_template: true
cache_name: default
healthcheck:
interval: 5s
prom-alb:
provider: alb
alb:
mechanism: tsmerge
discovery:
discoverer_name: in-cluster
template_backend: prom-template
query:
kind: endpointslices
namespace: monitoring
service: prometheus
port: web
A discovery-backed ALB may also list static pool members; discovered
members are additive to them. Static pool entries (and discovered members
from sources that convey weights) support integer weights for the
round_robin mechanism:
pool:
- static-member # weight 1
- name: bigger-member
weight: 3
Discoverer Connection Options
Each entry in the top-level discovery section declares a provider and
that provider’s connection-level block:
| provider | block | options |
|---|---|---|
kubernetes | kubernetes | in_cluster (default true) or kubeconfig path — mutually exclusive; plus qps, burst, user_agent, timeout |
dns_srv, dns_a | dns | resolver (host:port; default: system resolver), interval (poll cadence, default 30s, min 1s; record TTLs act as a floor) |
file | file | poll_interval (stat-poll fallback cadence, default 30s, min 1s) |
http_sd | http + http_sd | connection settings in the shared http block (below); http_sd.format selects the member-list document: trickster (default) or prometheus |
consul | http + consul | connection settings in the shared http block; consul.datacenter, namespace, partition, wait, allow_stale, only_passing, warning_is_ready |
nomad | http + nomad | connection settings in the shared http block; nomad.namespace, region, wait, allow_stale |
aws | aws (+ optional http) | aws.service (required: ec2 or ecs), region, and the credential fields; the endpoint is derived, so http.endpoint is an optional override |
gcp | gcp (+ optional http) | gcp.service (required; gce), gcp.project (from the metadata server when unset) and credentials_file; the endpoint is the Compute API, so http.endpoint is an optional override |
docker | docker (+ optional http) | docker.api_version; the endpoint defaults to the well-known socket, so http.endpoint is an optional override (unix:// or tcp://) |
azure | azure (+ optional http) | azure.service (required; vm), azure.subscription_id (required), credentials and cloud; the endpoint comes from the cloud, so http.endpoint is an optional override |
A block is only valid on its own provider’s entries; anything else fails startup.
The shared http block
Providers that discover members by polling an HTTP endpoint share one
connection block rather than each defining its own, so that configuring a
second such provider does not mean learning a second vocabulary. It is
required by http_sd, and rejected on providers that do not poll HTTP.
| option | meaning |
|---|---|
endpoint | base URL of the service to poll (http/https, host required) |
interval | poll cadence (default 30s, min 1s) |
timeout | bound on a single poll (default 10s, min 100ms) |
tls | outbound client TLS: client_cert_path/client_key_path, certificate_authority_paths, insecure_skip_verify |
headers | headers set on every request; where a registry’s credential is a bespoke header (X-Consul-Token, X-Nomad-Token), it goes here |
username, password | static HTTP Basic credential |
bearer_token | sent as Authorization: Bearer <token> |
bearer_token_file | path re-read before each poll, so a rotated credential is picked up without a restart — prefer this over bearer_token for anything that expires |
follow_redirects | default false; a redirect away from the configured endpoint is surfaced rather than chased |
username/password and the bearer-token fields are mutually exclusive,
as are the two bearer-token forms: a config that sets both fails startup
instead of silently preferring one, which is how an operator ends up
debugging a 401 against a config that looks correct.
Providers whose normal poll includes a server-side wait (blocking queries)
need timeout comfortably above that wait. There is no second,
client-level timeout underneath it that could cut a long poll short.
Template Backends
The template_backend is an ordinary backend definition marked
is_template: true. A discovered member overrides exactly the template’s
name, origin_url (and its derived scheme/host/path-prefix, with
the member’s path prefix falling back to the template’s), and replica
group (see below); the member’s weight applies to its pool entry. Every
other option — cache settings, TLS client configuration, healthcheck,
paths, request rewriters, authenticator, timeouts, concurrency limits,
tracing — is inherited from the template.
Templates are excluded from everything that applies only to live
backends: they are never routed, cannot be is_default, cannot appear in
a static ALB pool, and do not require an origin_url. A template must
use an origin-serving provider (not alb or rule), and a TSM-merged or
replica_group_label-using ALB requires a time-series-merge-capable
template provider. A TSM ALB with evaluation: global requires a
prometheus template with a
series source. Discovery replaces only
the origin, so a template’s series_endpoint, including the one Thanos
StoreAPI requires, is used unchanged by every discovered member: each
member reads that same store, and each read counts separately against
max_fetched_series and max_fetched_samples. A pool of sharded Thanos
sidecars discovered this way therefore reads only the one shard the
endpoint names, and Trickster logs a warning at startup for any such
template. Global evaluation over discovered members suits remote read
without a series_endpoint, as with flavor: mimir, which reads
/api/v1/read beneath each member’s own origin. That origin must be
http or https, so a query.scheme of tcp or udp fails
configuration, and a discovered member whose endpoint still can’t be
formed is logged and not added to the pool. See
Global Evaluation.
The alb.discovery Block
| option | description | default |
|---|---|---|
discoverer_name | name of a discoverer in the top-level discovery section (required) | |
template_backend | name of a backend with is_template: true (required) | |
query | provider-specific selection (required; see each provider below) | |
min_members | reject snapshots that would shrink the discovered membership below this count, keeping the last-good pool (guards against source blips returning empty results). 0 disables. | 0 |
debounce_window | coalesce membership changes arriving within this window into one pool update, damping flapping sources. 0 disables. | 0 |
startup_policy | retry starts the ALB with only its static members and keeps retrying the discoverer; fail fails startup when the discoverer is unavailable | retry |
health_mode | probe: discovered members inherit the template’s active health check. provider: the discoverer’s readiness reporting (currently Kubernetes only) drives member health instead of active probes | probe |
Health and Readiness Semantics
In probe mode, each discovered member gets its own health check from the
template’s healthcheck config, and the ALB’s healthy_floor applies
exactly as it does to static members. If the template has no probe
interval and the floor requires passing probes, the floor is reset to 0
with a loud warning (as with static members).
A new member’s first probe runs the moment it is registered, and a member
the provider reports ready is admitted to the pool immediately, ahead
of that first result. This matters for rolling deploys: an orchestrator
retires the old workload on the very readiness signal that announced the
new one, so a pool that held the newcomer in unknown until its own probe
completed would have no admissible member in between, and every request in
that window would fail with a 502 even though a healthy workload was
serving. From its first result on, the probe governs the member exactly as
it does an established one: a member the provider reports not-ready, or
one whose readiness the provider does not convey, waits for its first
passing probe as before.
In provider mode, readiness reported by the discoverer maps onto member
health: ready and terminating members enter as passing (1), not-ready
members as failing (-1), readiness-unknown members as unchecked (0) — so a
healthy_floor of 1 excludes members until the provider reports them
ready.
Members a provider reports as shutting down but still serving (terminating
Kubernetes endpoints whose serving condition holds, deletion-stamped pods
that are still ready) stay in the pool as
draining members: from the moment the
change is observed they take no new requests, connections or sessions, while
sticky sessions already pinned to them keep
reaching them for as long as they are available. Once the provider reports
that a member no longer serves, it is removed from the snapshot, before
the workload is killed — enabling zero-error rolling deploys. On removal, a
member’s in-flight requests complete, its health check stops, its metrics
series are deleted, and its idle upstream connections close after the
configured drain timeout. Draining members count toward min_members and
trickster_alb_discovery_members, and are shown on the
health status page and by the
trickster_alb_member_draining gauge.
TSM Replica Groups
When the ALB uses the Time Series Merge
mechanism, discovered members participate in replica_group semantics
three ways, in priority order:
Per-member groups from the discovery source (highest priority): the Kubernetes provider can read a configured label from each discovered workload via
query.replica_group_label, andfileprovider entries may carry areplica_groupfield. Members sharing a group value are treated as HA replicas of one logical shard and coalesced; distinct values are distinct shards, merged. This handles sharded-HA topologies (e.g., Thanos-style Prometheus pairs) discovered with a single selector:backends: prom-alb: provider: alb alb: mechanism: tsmerge discovery: discoverer_name: in-cluster template_backend: prom-template query: kind: endpointslices namespace: monitoring service: prometheus port: web replica_group_label: prometheus/shardA
replica_groupset on the template backend: inherited by every member without a source-conveyed group — all such members are HA replicas of one shard (the common HA-pair case).Neither (default): each member is its own replica group, i.e. its own logical shard — a pure federation/merge of every member.
For the pods and service kinds the label is read from the discovered
Pod or Service itself. For endpointslices, endpoints don’t carry pod
labels, so setting replica_group_label joins a Pod watch in the query’s
namespace to resolve each endpoint’s target pod (see the RBAC note below);
an endpoint whose pod isn’t yet observed joins ungrouped and regroups
automatically when the pod appears. A member whose group changes is
rebuilt in place under the same name.
Because replica_group is only meaningful on time-series backends,
configuration validation requires a TSM-capable template_backend
whenever the ALB mechanism is tsmerge or replica_group_label is set.
The DNS providers convey no grouping metadata; use the template-level
group (mode 2) or the file provider for grouped non-Kubernetes pools.
Query Vocabulary
Every ALB’s alb.discovery.query block is drawn from one shared vocabulary.
A field set for a provider that does not accept it fails startup rather
than being silently ignored, so a query aimed at the wrong provider is a
config error rather than an empty pool.
| field | kubernetes | dns_srv | dns_a | file | http_sd | consul | nomad | aws | gcp | azure | docker |
|---|---|---|---|---|---|---|---|---|---|---|---|
kind | ● | ||||||||||
namespace | ● | ||||||||||
service | ● | ● | ● | ● | |||||||
selector | ● | ||||||||||
srv_name | ● | ||||||||||
hostname | ● | ||||||||||
path | ● | ● | |||||||||
filter | ● | ● | ● | ||||||||
filters | ● | ● | |||||||||
tags | ● | ● | ● | ● | ● | ||||||
cluster | ● | ||||||||||
network | ● | ||||||||||
address_type | ● | ● | ● | ● | |||||||
port | ● | ● | ● | ● | ● | ● | |||||
port_label | ● | ● | ● | ● | |||||||
scheme | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | |
replica_group_label | ● | ● | ● | ● | ● | ● |
Most fields mean the same thing everywhere:
scheme—http(default) orhttpsfor the member’s origin URL.port— a static port applied to every member.port_label— reads the port from the provider’s own metadata namespace: an EC2 or Azure tag, a GCE label (then instance metadata), a Docker container label. It wins overportper member, so a fleet can share one static default with individual machines overriding it.address_type—private(default),public, oripv6. Which address to take where a resource has several.replica_group_label— reads a TSM replica group from the same namespace asport_label.tags— narrows by tag/label presence, and is a conjunction: every listed tag must be present.filter/filters— provider-native selection evaluated server-side.filteris an expression string (Consul, Nomad, GCE);filtersis a name→values map (AWSFilter.N, Docker’s filter document). These are passed through, so an expression the upstream does not understand fails the refresh loudly rather than matching nothing.
Hosts versus endpoints is the distinction that decides whether a port is
required. A cloud inventory API returns hosts — aws service: ec2,
gcp, and azure return addresses with no port, so one of port or
port_label is required. Kubernetes and docker return endpoints
that carry their own ports, so a port is optional there and is resolved
from the object when unambiguous.
The kubernetes Provider
The kubernetes provider watches the cluster via the API server (shared
informers; watch-driven, no polling) and supports three query kinds.
Connection options:
discovery:
my-cluster:
provider: kubernetes
kubernetes:
in_cluster: true # use the pod's service account (default)
# kubeconfig: /path/to/kubeconfig # or run against a remote cluster
qps: 20 # sustained API request rate (default 20)
burst: 40 # API request burst allowance (default 40)
timeout: 10s # bounds one-shot API calls (default 10s)
# user_agent: my-trickster/1 # default: trickster/<version> (<os>/<arch>)
in_cluster and kubeconfig are mutually exclusive; with neither set,
in-cluster is assumed.
qps and burst are the client-side rate limit on API requests. The
defaults are raised over the client-go defaults (5 and 10), which are sized
for a one-shot CLI rather than a process that watches many objects.
timeout bounds a single one-shot API call, such as the connectivity
preflight run under startup_policy: fail. It is deliberately not applied
to the REST client as a whole, whose timeout would also truncate the
long-running watch streams the informers depend on.
Shared Watches
Discoverers that agree on the connection (API server, credentials, identity) and on the query’s server-side filtering (namespace, label selector, field selector) share one set of informers, so several ALBs selecting the same Service cost the API server one watch rather than one each. A watch is released when the last subscription using it stops, so unsubscribing one ALB never disturbs another. Nothing needs to be configured for this; it follows from the discoverer and query values being identical.
Startup Preflight
When any ALB bound to a kubernetes discoverer sets startup_policy: fail,
the discoverer contacts the API server’s /version endpoint before any
watch is established, bounded by timeout. Informers are lazy, so without
this check an unreachable API server is indistinguishable from a Service
with no endpoints yet: the process would start clean and serve an empty
pool. Under the default startup_policy: retry the preflight is skipped and
the discoverer keeps watching for the API server to come back.
Query Kinds
endpointslices (default) discovers the endpoint addresses of a named
Service — the pod IPs behind it — and is the right choice for routing
around the Service’s own load balancing. Terminating endpoints that still
serve join as draining members:
query:
kind: endpointslices
namespace: monitoring
service: prometheus
port: web # port name or number; optional when the service has one port
service discovers the ClusterIP of each Service matching a label
selector (or one named Service), one member per Service:
query:
kind: service
namespace: monitoring
selector:
app: prometheus
port: 9090
pods discovers pod IPs directly by label selector, without requiring
a Service (the original ask of issue #609):
query:
kind: pods
namespace: monitoring
selector:
app: prometheus
port: web
namespace defaults to the pod’s own namespace when running in-cluster.
Port and Scheme Resolution
port may be a named port, a number, or omitted when the target declares
exactly one port; ambiguity is logged and the object skipped. The member
scheme comes from query.scheme if set; otherwise a declared
appProtocol: https on the selected port, or a trickstercache.org/scheme
annotation on the watched Service/Pod, selects https; the default is
http.
Zero-Error Rolling Deploys
Terminating endpoints stop taking new work as soon as the change is
observed: they drain, keeping only their sticky sessions, and leave the
discovered membership once they stop serving, ahead of pod deletion.
One piece belongs to the workload, though: Kubernetes marks an endpoint
terminating and signals the container at the same moment, so — as with
any EndpointSlice consumer, kube-proxy included — the pod must keep
serving briefly while its removal propagates. Give discovered workloads a
short preStop delay (a few seconds comfortably covers Trickster’s
sub-second pickup):
lifecycle:
preStop:
sleep: # native sleep action (Kubernetes >= 1.30); images with
seconds: 5 # a shell can use exec with command: ["sleep", "5"]
Without it, an instantly-exiting container can produce a brief window of connection errors during rollouts — regardless of what is consuming the EndpointSlices.
RBAC
The provider needs only list and watch on the resources your query kinds use — it never reads Secrets or ConfigMaps and never writes:
query kind | apiGroup | resource | verbs |
|---|---|---|---|
endpointslices | discovery.k8s.io | endpointslices | list, watch |
service | "" (core) | services | list, watch |
pods | "" (core) | pods | list, watch |
Queries are namespace-scoped, so a namespaced Role/RoleBinding per
watched namespace is sufficient and preferred:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: trickster-autodiscovery
namespace: monitoring
rules:
- apiGroups: ["discovery.k8s.io"]
resources: ["endpointslices"]
verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: trickster-autodiscovery
namespace: monitoring
subjects:
- kind: ServiceAccount
name: trickster
namespace: trickster
roleRef:
kind: Role
name: trickster-autodiscovery
apiGroup: rbac.authorization.k8s.io
Add services and/or pods rules only for the kinds you configure. Note
that using replica_group_label with the endpointslices kind joins a
Pod watch, so it additionally requires pods list/watch in the query’s
namespace. RBAC cannot scope below the resource level, so the grant covers
all objects of that resource in the bound namespace. Out-of-cluster
(kubeconfig-based) discoverers need the same permissions for their user or
service account.
The dns_srv Provider
The dns_srv provider polls SRV records. SRV target and port map to the
member address; SRV weight maps to the member’s load-balancing weight; and
only the highest-priority tier (the lowest priority value present in the
answer) becomes members — lower tiers are treated as standby capacity
managed on the DNS side.
discovery:
corp-dns:
provider: dns_srv
dns:
resolver: 10.0.0.53:53 # optional; default: the system resolver
interval: 30s # poll cadence (default 30s)
backends:
my-alb:
provider: alb
alb:
mechanism: rr
discovery:
discoverer_name: corp-dns
template_backend: my-template
query:
srv_name: _prometheus._tcp.example.com
# scheme: https # optional; default http
Record TTLs act as a floor on the poll cadence: an answer is never
re-resolved before its shortest TTL expires, so interval is the most
frequent the provider will query. Resolution failures keep the last-good
membership; an authoritative empty answer empties it.
The dns_a Provider
The dns_a provider resolves a hostname’s A and AAAA records, with a
fixed port and scheme from the query. This covers round-robin DNS,
headless-service-style DNS outside Kubernetes, Docker’s embedded DNS, and
Consul’s DNS interface without a bespoke provider:
query:
hostname: prometheus.service.consul
port: 9090 # required
# scheme: https # optional; default http
The dns connection options, poll cadence, TTL floor, and failure
semantics are the same as dns_srv.
The file Provider
Note: like Prometheus’s file_sd, the file provider is the universal
integration point — anything that can write a file (cron, sidecar,
consul-template, your deploy tooling) can drive an ALB pool.
The provider watches a local YAML or JSON member-list file and applies it atomically on change:
discovery:
external-sd:
provider: file
# file:
# poll_interval: 30s # stat-poll fallback cadence (default 30s)
backends:
my-alb:
provider: alb
alb:
mechanism: rr
discovery:
discoverer_name: external-sd
template_backend: my-template
query:
path: /etc/trickster/members.yaml
The file is a list of members:
- name: prom-1 # optional; defaults to the address
address: 10.0.0.1:9090 # required host:port
- name: prom-2
scheme: https # optional; default http
address: 10.0.0.2:9090
path_prefix: /base # optional
weight: 3 # optional; default 1
replica_group: shard-0 # optional; TSM replica group (see above)
Writers should replace the file atomically (write to a temp file in the same directory, then rename). A file that fails to read or parse keeps the last-good membership; an empty file is a valid, empty membership.
Change Detection
The provider uses two mechanisms together (via Trickster’s shared filesystem watcher), so updates are never missed:
- Filesystem notification (on the file’s parent directory, debounced): near-instant pickup of writes, atomic renames, and symlink swaps on local filesystems; a directory watch dropped by deletion or recreation is automatically re-armed.
- A content-comparing poll of the file (
file.poll_interval, default30s, minimum1s): the guaranteed fallback wherever notification is unreliable or unavailable.
Guidance for Kubernetes-mounted member files:
- ConfigMap / Secret volume mounts: kubelet applies updates with an atomic symlink swap inside the mount directory, which generates inotify events — notification works, and updates are picked up promptly. Note that kubelet itself syncs ConfigMap content periodically (typically up to a minute), which usually dominates end-to-end latency.
subPathmounts of a ConfigMap/Secret: Kubernetes never updates these after pod start — no mechanism (notification or polling) can see a change. Mount the directory, not asubPath.- Network- or FUSE-backed volumes (NFS PVs, some CSI drivers):
inotify generally does not observe writes made by other clients, so the
stat poll is the effective update mechanism — set
file.poll_intervalto the freshness your deployment needs. emptyDir/hostPathwritten by a sidecar in the same pod: notification works.
The poll compares file content (not timestamps), so any change is
detected within one poll_interval regardless of filesystem timestamp
granularity.
The http_sd Provider
http_sd fetches a member list from an HTTP endpoint. It is the universal
adapter: any service-discovery system Trickster has no in-tree provider for
can feed it through a few lines of glue that serve a member list, with no
Trickster change and no restart.
discovery:
fleet:
provider: http_sd
http:
endpoint: https://sd.example.com
interval: 15s
bearer_token_file: /var/run/secrets/sd-token
http_sd:
format: trickster
backends:
prom-template:
provider: prometheus
is_template: true
prom-alb:
provider: alb
alb:
mechanism: tsm
discovery:
discoverer_name: fleet
template_backend: prom-template
query:
path: /pools/prometheus
The query’s path is optional and is appended to the endpoint, so one
server can serve a different member list per ALB while they share the
discoverer’s connection settings. The query’s scheme supplies the scheme
for prometheus-format targets, which are bare host:port; native-format
entries carry their own and override it.
Formats. trickster is the same document the file provider reads and
is the default:
- name: prom-1
scheme: https
address: 10.0.0.1:9090
path_prefix: /base
weight: 2
replica_group: shard-0
prometheus is the document Prometheus’s own file_sd and http_sd
consume, so an existing endpoint can be pointed at Trickster unchanged:
[{"targets": ["10.0.0.1:9090"], "labels": {"env": "prod"}}]
A group’s __scheme__ label overrides the query scheme, letting one
endpoint serve mixed-scheme members. Note that the Prometheus format cannot
express weight or replica_group — deployments that need weighted
pools or TSM replica groups want the native format.
The format is named explicitly rather than sniffed. The two documents are structurally distinguishable, but guessing means a typo in one format can parse as a valid, wrong membership in the other, and the cost of guessing wrong is a silently drained pool.
Efficiency. Requests carry X-Prometheus-Refresh-Interval-Seconds, as
Prometheus’s does, so servers that generate member lists on demand can pace
their own work. ETag is honored: an unchanged membership costs one
conditional request and a 304, with no re-parse and no snapshot. The
validator is only stored after a document parses, so a rejected document
can never be confirmed as current by a later 304.
Failure. A transport error, an unexpected status, an oversized body
(the limit is 16 MiB), or a document that will not parse all keep the
last-good membership and are logged once per failure streak and counted on
trickster_discovery_refresh_errors_total. An endpoint that
authoritatively returns no members ([]) is a valid membership and is
applied, so a scaled-to-zero pool can be reported as such.
The consul Provider
consul reads service instances from Consul’s health endpoint. It is
event-driven rather than polled: each request is a Consul blocking
query, so the server parks it until the service changes or wait elapses.
A membership change is observed within a round trip instead of within a
poll interval, and a stable service costs one parked connection rather
than a request per interval.
discovery:
consul-dc1:
provider: consul
http:
endpoint: http://127.0.0.1:8500
# a rotated ACL token; Consul accepts the Authorization Bearer scheme
# as an equivalent to its own X-Consul-Token header
bearer_token_file: /var/run/secrets/consul-token
consul:
datacenter: dc1
wait: 5m # how long a blocking query parks; 1s–10m
allow_stale: true # answer from any server, not only the leader
backends:
prom-template:
provider: prometheus
is_template: true
prom-alb:
provider: alb
alb:
mechanism: tsm
health_mode: provider # Consul's own checks decide readiness
discovery:
discoverer_name: consul-dc1
template_backend: prom-template
query:
service: prometheus
tags: [production]
# filter: 'Service.Meta.version == "2"'
# replica_group_label: shard # read from the service's Meta
Readiness. Consul reports per-instance check status, so this is the
first provider outside kubernetes that can honestly answer “is this member
ready”, which makes health_mode: provider meaningful for VM and container
fleets. An instance’s readiness is its worst check: all passing is
ready, critical and maintenance are not ready, and warning is ready by
default (matching how Consul treats warning for DNS) — set
warning_is_ready: false to drain warning instances instead. A status a
future Consul release introduces is treated as not-ready rather than
ignored.
Failing instances are reported as NotReady rather than omitted, so that an
ALB using the default health_mode: probe can decide for itself and a
wholly-unhealthy service does not look like an empty one. Set
only_passing: true to have Consul filter them out server-side instead.
Weights. Consul’s own Weights.Passing / Weights.Warning map onto
member weights, including the passing/warning distinction — an operator who
has already told Consul the relative capacity of each instance does not have
to tell Trickster again.
Addresses. A service that registers its own address overrides its node’s, which is how sidecars and containers with their own routable address are represented. An instance with no usable address or port fails the whole refresh rather than being silently dropped, so a pool never quietly shrinks because of a catalog change nobody noticed.
Labels. Members carry service, service_id, node, datacenter,
status, and tags (comma-bracketed, as ,a,b,). Service metadata is
carried as meta_<key> so that an operator-defined key cannot shadow a
Trickster-assigned label.
Timeouts. http.timeout must outlast consul.wait, because a blocking
query legitimately takes that long. Its default is derived from the wait
rather than shared with the other HTTP providers (Consul adds up to
wait/16 of its own jitter, which the margin covers), and a config that
sets it too low is rejected at startup rather than producing a stream of
timeouts. http.interval is not the poll cadence here — with blocking
queries there is no cadence — it is the retry delay after a failure.
The nomad Provider
nomad reads service instances from Nomad’s native service registry
(Nomad 1.3+). Like consul it is event-driven, using the same HashiCorp
blocking-query protocol, so a membership change is observed within a round
trip rather than within a poll interval.
discovery:
nomad-eu:
provider: nomad
http:
endpoint: http://127.0.0.1:4646
# a rotated ACL token; Nomad accepts the Authorization Bearer scheme
# as an equivalent to its own X-Nomad-Token header
bearer_token_file: /var/run/secrets/nomad-token
nomad:
namespace: default
region: eu-1
wait: 5m
allow_stale: true
backends:
prom-alb:
provider: alb
alb:
discovery:
discoverer_name: nomad-eu
template_backend: prom-template
query:
service: prometheus
tags: [production]
# filter: 'JobID == "monitoring"'
Native registry, not Consul. This reads the registry a job selects with
provider = "nomad" in its service block. Jobs that register into Consul
instead are discovered with the consul provider, and that is the more
capable choice where it applies: Nomad’s service endpoint carries no
per-instance check state, so members are reported ReadyUnknown and
health_mode: provider falls back to Trickster’s own probes. A deployment
that wants discovery-conveyed readiness should register its services into
Consul.
Tags filter client-side. Unlike Consul’s catalog endpoint, Nomad’s
service endpoint has no tag parameter, so query.tags is applied by
Trickster after the response arrives. It is a conjunction — every listed tag
must be present. query.filter is passed through to Nomad and evaluated
server-side.
Labels. Members carry service, service_id, job_id, alloc_id,
node_id, namespace, datacenter, and tags (comma-bracketed). The
allocation and job identifiers are what an operator needs to trace a member
back to the workload that registered it.
Timeouts work exactly as for consul: http.timeout must outlast
nomad.wait, its default is derived from the wait, and http.interval is
the retry delay after a failure rather than a poll cadence.
The aws Provider
aws discovers members from an AWS API, selected by aws.service: ec2
for instances, ecs for tasks. It is required — with more than one AWS
API supported, defaulting would be an arbitrary guess at which one you
meant, so a config that omits it fails at startup. Further AWS sources
arrive as new service values rather than new providers, inheriting this
provider’s credentials, signing, pagination and options.
discovery:
fleet:
provider: aws
aws:
service: ec2 # required: ec2 or ecs
region: us-east-1
# credentials omitted: use the standard chain (IRSA, instance
# profile, environment, shared config). See docs/aws.md.
http:
interval: 60s # instance inventories change slowly; poll gently
backends:
prom-alb:
provider: alb
alb:
discovery:
discoverer_name: fleet
template_backend: prom-template
query:
filters:
tag:service: [prometheus]
instance-state-name: [running]
port_label: trickster-port # read the port from an EC2 tag
address_type: private # private (default), public, or ipv6
# port: 9090 # or a static port for every member
# replica_group_label: shard
Credentials, region resolution and IAM are documented once in
AWS Integration. The IAM principal needs
ec2:DescribeInstances.
Hosts, not endpoints. An instance inventory returns hosts with several addresses and no port, so two query fields exist that the registry providers do not need:
address_type—private(default),public, oripv6— selects which of the instance’s addresses becomes the member address.port_label— names an EC2 tag whose value is the member’s port.portsupplies a static one. At least one is required, and they compose: where both are set, the tag wins per instance andportis the fallback, which is what makesport_labelsafe to adopt incrementally across a fleet.
Selection. filters is passed to EC2 as Filter.N, evaluated
server-side — use tag:<key> to filter on a tag value, and any other
DescribeInstances filter
name. tags additionally requires the presence of tag keys, applied after
the response arrives.
Instance state. running is Ready and pending is NotReady.
Instances that are shutting-down, stopping, stopped or terminated
are omitted entirely rather than reported unready, so they drain from
pools before they stop answering — the same rule the kubernetes provider
applies to terminating endpoints. A state a future EC2 release introduces is
treated as not-ready rather than assumed healthy.
Instances that cannot become members are excluded, not fatal. An
instance with no port_label tag and no static port, or with no address
of the requested type, is skipped and logged (once, until the set of
excluded instances changes) rather than failing the refresh. This differs
deliberately from the consul and nomad providers, where a malformed
entry fails the whole refresh: a service registry contains only instances of
the service, so a bad entry means the API is broken, while an EC2 inventory
routinely contains hosts that simply are not tagged yet. Failing there would
drain a working pool because of one unrelated instance.
Labels. Members carry instance_id, instance_type, instance_state,
image_id, availability_zone, vpc_id, subnet_id, architecture,
private_ip, public_ip, private_dns and public_dns. Instance tags are
carried as tag_<Key>, prefixed so an operator-defined tag cannot shadow a
Trickster-assigned label. The Name tag becomes the member name, falling
back to the instance id.
Endpoint. Derived from the region and service
(https://ec2.<region>.amazonaws.com). Set http.endpoint to override it
for a VPC endpoint, a FIPS endpoint, or a test double; a region is required
when it is not overridden, because the endpoint is built from it.
service: ecs
Discovers ECS tasks, including Fargate, which service: ec2
structurally cannot see — a Fargate task has no EC2 instance behind it.
discovery:
tasks:
provider: aws
aws:
service: ecs
region: us-east-1
http:
interval: 30s
backends:
prom-alb:
provider: alb
alb:
discovery:
discoverer_name: tasks
template_backend: prom-template
query:
cluster: prod # ECS cluster; the account default when unset
service: prometheus # ECS service name; optional
port_label: trickster-port
# port: 9090 # or a static port for every task
The IAM principal needs ecs:ListTasks and ecs:DescribeTasks.
awsvpc only. Each task in awsvpc mode has its own elastic network
interface, so DescribeTasks alone yields a routable address. That is the
only mode Fargate offers and the default for new EC2-launch-type services.
Under bridge or host networking the address belongs to the container
instance rather than the task, and resolving it would take two further API
calls, a second signing service and broader IAM — so such tasks are
excluded with a reason saying so rather than silently missing. If you
need bridge or host mode, say so and it can be added.
Tags must be propagated. port_label reads an ECS task tag, and ECS
does not copy service tags onto tasks unless the service is created with
--propagate-tags SERVICE. Trickster asks DescribeTasks for tags
explicitly (include: [TAGS]); if port_label finds nothing, check the
propagation setting first.
Task state. RUNNING is Ready; PROVISIONING, PENDING and
ACTIVATING are NotReady; DEACTIVATING, STOPPING, DEPROVISIONING
and STOPPED are omitted entirely, so tasks drain from pools before they
stop answering. Container health is only reported when the task definition
declares a health check — UNHEALTHY is NotReady, and UNKNOWN or absent
is treated as ready, because the alternative would make every
un-instrumented task permanently unusable.
Selection is by cluster and service; filters and address_type are
rejected for ecs, since ECS selects by cluster rather than by instance
attribute and an awsvpc task has exactly one address. tags still filters on
tag presence after the response arrives.
Labels. Members carry task_arn, task_id, cluster,
task_definition, group, launch_type, availability_zone,
task_status and health_status, plus task tags as tag_<Key>. The Name
tag becomes the member name, falling back to the task id.
Churn. A task that disappears between ListTasks and DescribeTasks is
ordinary churn; it is reported as an exclusion rather than silently
shrinking the pool.
The gcp Provider
gcp reads a Google Cloud API named by gcp.service. The provider is
named for the cloud rather than for Compute Engine, matching aws — Google
Cloud APIs outside Compute Engine belong here too, and would sit oddly under
a provider called gce.
service is required even though gce is the only value today. A
default added now could never be removed, and every service added later
would then be reached by opting out of a value the operator never chose.
A value names the product an operator would recognize, not the API that
serves it — the same convention as aws.service. That matters here: Cloud
Load Balancing is served by the Compute API alongside instances, so naming
these for the API would collide where naming them for the product does not.
service: gce
Discovers Google Compute Engine instances through the Compute API’s
instances.aggregatedList, which covers every zone in the project in
one paged call — so no zone list has to be configured or kept current.
discovery:
fleet:
provider: gcp
gcp:
service: gce # required
project: my-project # from the metadata server when unset
# credentials_file: /etc/trickster/sa.json # ADC when unset
http:
interval: 60s # instance inventories change slowly
backends:
prom-alb:
provider: alb
alb:
discovery:
discoverer_name: fleet
template_backend: prom-template
query:
filter: 'labels.role = "prometheus" AND status = "RUNNING"'
tags: [http-server] # network tags, matched on presence
port_label: port # instance label, or metadata key
address_type: private # private (default), public, or ipv6
# port: 9090 # or a static port for every member
# replica_group_label: shard
Credentials. Leaving credentials_file empty selects Application
Default Credentials: GOOGLE_APPLICATION_CREDENTIALS, gcloud user
credentials, Workload Identity on GKE, or the instance metadata server
on GCE. Prefer those over a key file wherever the platform offers one.
Credentials resolve lazily and only successes are cached, so Trickster
starts even when the metadata server is briefly unreachable and a momentary
failure does not permanently disable discovery.
credentials_file must be a service account key. The credential type is
required rather than taken from the file: an external_account or
impersonated_service_account configuration can name an arbitrary token URL
or local executable, so accepting whichever type a file happens to declare
would hand credential resolution somewhere unintended. For user credentials,
use ADC instead of this field.
The IAM principal needs compute.instances.list on the project — the
roles/compute.viewer role includes it. The OAuth scope requested is
compute.readonly; Trickster never mutates a project.
Project. Taken from gcp.project, then from the credentials, then from
the metadata server when Trickster runs on GCE. It is deliberately not
required in config, because reading it from the metadata server is the
idiomatic deployment.
Hosts, not endpoints, exactly as for aws service: ec2:
address_type chooses the address and port/port_label supplies the
port, with the label winning per instance and the static port as fallback.
port_label reads an instance label first, then instance metadata —
both are key/value namespaces on a GCE instance, and a deployment already
carrying the value in metadata does not have to move it.
Instance status. RUNNING is Ready; PROVISIONING, STAGING and
REPAIRING are NotReady; STOPPING, SUSPENDING, SUSPENDED and
TERMINATED are omitted entirely, so instances drain from pools before
they stop answering. A status a future Compute Engine release introduces is
treated as not-ready rather than assumed healthy.
Selection. filter is a GCE filter expression, evaluated server-side.
tags matches GCE network tags, which are names without values, so it
filters on presence. Requests set returnPartialSuccess, so one unreachable
zone contributes no instances rather than failing the whole refresh.
Labels. Members carry instance_id, instance_name, status, zone,
machine_type, network, subnetwork, private_ip, public_ip, and
tags (comma-bracketed). Instance labels are carried as label_<key>,
prefixed so a user-defined label cannot shadow a Trickster-assigned one.
Resource URLs are shortened to their last segment, since a member label full
of https://www.googleapis.com/compute/v1/... is unreadable.
The azure Provider
azure reads an Azure Resource Manager API named by azure.service.
service is required even though vm is the only value today, for the
same reason as aws.service and gcp.service: a default added now could
never be removed.
service: vm
Discovers Virtual Machines, joining them to their network interfaces to resolve addresses.
discovery:
fleet:
provider: azure
azure:
service: vm # required
subscription_id: 00000000-0000-0000-0000-000000000000 # required
# resource_group: prod-rg # narrows every list; recommended
# credentials omitted: use the managed identity of the VM or AKS pod
# tenant_id: ... # with a service principal:
# client_id: ...
# client_secret: ...
# federated_token_file: /var/run/secrets/azure/tokens/azure-identity-token
# cloud: public # public (default), usgovernment, china
# power_state: false # one extra list call; see Readiness
http:
interval: 60s # vm inventories change slowly
backends:
prom-alb:
provider: alb
alb:
discovery:
discoverer_name: fleet
template_backend: prom-template
query:
tags: [prometheus] # vm tag names, matched on presence
port_label: port # a vm tag holding the port
# port: 9090 # or a static port for every member
# address_type: private # private (default), public, or ipv6
# replica_group_label: shard
Credentials. Leaving the credential fields empty selects the managed
identity of the Azure VM or AKS pod Trickster runs on, via the instance
metadata service; set client_id alone to select a user-assigned
identity. On AKS with workload identity, set federated_token_file to
the projected token path — the platform writes and rotates that file, so
Trickster holds no secret, and the file is re-read on every token
acquisition. A client_secret service principal is the fallback where
neither is available; it is redacted from config dumps and the management
API.
There is deliberately no credential chaining. A configured credential that fails is an error, not a reason to quietly fall back to the instance metadata service and authenticate as a different principal.
Permissions. The principal needs only Reader on the subscription, or
on the resource group named by resource_group. The specific actions are
Microsoft.Compute/virtualMachines/read and
Microsoft.Network/networkInterfaces/read, plus
Microsoft.Network/publicIPAddresses/read when address_type: public.
power_state: true additionally requires read at subscription scope,
even with resource_group set — see Readiness.
If discovery finds nothing, check resource provider registration first.
A subscription where Microsoft.Compute is not registered answers the VM
list with HTTP 200 and an empty result, not an error, so the pool is
silently empty. New subscriptions start unregistered. Trickster detects this
case and logs it, but the fix is outside Trickster:
az provider register -n Microsoft.Compute && az provider register -n Microsoft.Network
Clouds. cloud selects both the ARM endpoint and the Entra ID login
endpoint together, since keeping two URLs consistent by hand is exactly the
kind of thing that silently half-works. http.endpoint overrides the ARM
endpoint alone, for a private ARM proxy or a test server.
API versions. ARM requires an explicit api-version on every request,
so both are pinned rather than omitted: compute_api_version (default
2024-07-01) for the VM list and network_api_version (default
2024-05-01) for interfaces and public addresses. Pinning means the
response shape cannot change under Trickster when Microsoft ships a new
version. Override either to move forward deliberately.
The join. An Azure VM carries no address. Addresses live on network
interfaces, which the VM references by resource id, and a public address is
a further reference from the interface to a publicIPAddresses resource. A
refresh is therefore two list calls — three with address_type: public —
joined in memory. Resource ids are matched case-insensitively, because
Azure treats them that way and the casing genuinely differs between APIs; a
VM’s interface reference commonly spells the resource group differently from
the interface list. A VM whose references resolve to nothing is excluded
with a message saying so specifically, since that is the symptom a broken
join produces.
resource_group is worth setting on a large subscription: it narrows every
list rather than enumerating thousands of machines to find a handful.
Hosts, not endpoints, exactly as for aws service: ec2 and gcp
service: gce: address_type chooses the address and port/port_label
supplies the port, with the tag winning per VM and the static port as
fallback. VM tags are matched case-insensitively, as Azure treats them,
so a machine tagged Role is found by a query for role.
Readiness. Off by default, ReadyUnknown for every member. The VM list
carries provisioning state, not power state, and a provisioned VM may be
stopped — reporting Ready on that basis would assert something Azure never
said. Setting power_state: true requests the instance view, which makes
running VMs Ready and removes stopped ones from membership entirely so
they drain from pools. The cost is one extra list call per refresh, not one
call per VM: the whole-subscription list accepts statusOnly=true and
returns every machine’s instance view at once. Trickster’s own active health
checks are the alternative, and cover the case either way.
statusOnly is a parameter of the subscription-wide list only. The
resource-group-scoped list accepts it, returns 200, and silently ignores
it, so Trickster always issues this one call at subscription scope — which
is why power_state: true needs subscription-scope read even when
resource_group is set. If the status list ever comes back with no instance
view for any machine, the refresh fails and keeps the last-good
membership rather than reading every VM as stopped and emptying the pool.
Labels. Members carry vm_name, vm_id, location, vm_size,
resource_group, private_ip, public_ip, and power_state when
requested. VM tags are carried as tag_<key>, prefixed so a user-defined
tag cannot shadow a Trickster-assigned label.
The docker Provider
docker discovers containers through the Docker Engine API’s
GET /containers/json, polled on http.interval.
discovery:
containers:
provider: docker
docker: {} # api_version: v1.41 by default
http:
# endpoint: unix:///var/run/docker.sock # the default
# endpoint: tcp://dockerhost:2376 # remote daemon; add a tls block
interval: 30s
backends:
prom-alb:
provider: alb
alb:
discovery:
discoverer_name: containers
template_backend: prom-template
query:
filters:
label: [com.example.discover=yes]
# network: backend # required only when a container is on several
# port: 9090 # or port_label, when the container exposes several
# address_type: private # private (default), public, or ipv6
# replica_group_label: com.example.shard
Endpoint. Taken from http.endpoint, defaulting to
unix:///var/run/docker.sock. Both the unix:// and tcp:// forms
DOCKER_HOST uses are accepted, so one can be pasted into the other. A
tcp:// endpoint takes its client TLS from the shared http.tls block —
that is how a remote daemon’s mutual TLS is configured — and becomes
https on the wire when one is present. TLS on a unix:// endpoint is
rejected rather than ignored.
Access. Trickster needs read access to the Docker socket, which on most
hosts means membership of the docker group. That access is equivalent to
root on the host, so prefer a read-only socket proxy in production over
mounting the socket directly. Trickster only ever issues GET /containers/json.
API version. Pinned to v1.41 (Docker 20.10 and later) rather than
left off. An unversioned request binds to whatever the daemon’s newest
version happens to be, so the response shape would change under Trickster
when the host upgrades Docker. Override with docker.api_version to use a
newer one.
Endpoints, not hosts. Unlike the cloud providers, the Engine API
reports ports, so port is not required. A container exposing exactly
one TCP port resolves automatically; one exposing several is excluded with
a reason naming the candidates, rather than having a guess made for it. UDP
ports are never candidates, which is what makes the single-port case common
in practice — a container exposing one TCP port beside two UDP ports is
unambiguous. port_label reads the port from a container label and wins
over a static port per container.
Addresses. address_type: private (the default) uses the container’s IP
on its network, with the container-internal port. public uses the host
binding instead — the address and port are taken from the same binding, so
a container publishing one port on 127.0.0.1 and another on 0.0.0.0 does
not produce a mismatched pair. A wildcard binding (0.0.0.0 or ::) is not
dialable, so it resolves to 127.0.0.1. ipv6 uses the container’s global
IPv6 address.
Networks. A container on exactly one network needs no network. One on
several must name which, rather than having a map iteration pick — that
would differ between polls and churn the pool. A container attached to no
network is excluded.
Readiness. State: running is required; every other state leaves
membership entirely, so containers drain from pools as they stop. Health
comes from the container’s HEALTHCHECK: healthy is Ready, unhealthy
and starting are NotReady, and a container with no healthcheck
declared is ReadyUnknown, not Ready — the daemon knows the process
started, not that it is serving, and Trickster’s own health checks cover
that. Note that GET /containers/json reports health only inside its
human-readable Status string; the per-container inspect that carries a
structured Health object would cost one request per container per poll.
Selection. filters is passed to the Engine API as its own filter
document and evaluated server-side, so the daemon does the narrowing:
label, name, status, health, network, ancestor and the rest. A
filter name the daemon does not recognize is a 400 and fails the refresh
loudly, rather than being ignored into a silently empty pool. When the query
sets no status filter, status: [running] is added, so a host with a long
history of exited containers is not listed in full every poll.
Labels. Members carry container_id (short form), container_name,
image, state, network and private_ip. Container labels are carried
as label_<key>, prefixed so a user-defined label cannot shadow a
Trickster-assigned one — Compose’s own labels arrive as
label_com.docker.compose.service and the like.
Upstream Permissions
Every provider is read-only; Trickster never mutates a discovery source. The least privilege each needs:
| provider | required access |
|---|---|
kubernetes | list + watch on the resources your query kinds touch — see RBAC |
consul | an ACL token with service:read (and node:read) for the queried services |
nomad | an ACL token with read-job on the namespace |
aws (ec2) | ec2:DescribeInstances |
aws (ecs) | ecs:ListTasks, ecs:DescribeTasks (add ec2:DescribeInstances for EC2-launch-type tasks) |
gcp | compute.instances.list — roles/compute.viewer includes it |
azure | Reader on the subscription or resource group; power_state: true needs it at subscription scope |
docker | read access to the Engine socket (the docker group, or a read-only socket proxy) |
http_sd | whatever the endpoint itself requires; see connection options |
Credentials are never required in config where the platform can supply
them: aws uses the standard credential chain (IRSA, instance profile,
environment), gcp uses Application Default Credentials (Workload Identity
on GKE, the instance service account on GCE), and azure uses the managed
identity of the VM or AKS pod. Prefer those over stored secrets wherever
they are available — see each provider’s section. Secrets that are
configured (aws.secret_key, azure.client_secret) redact themselves from
config dumps and the management API.
Observing Discovered Members
Discovered members appear on the health status page
alongside static pool members, under their generated backend names
(<alb-name>-<member-name>), tagged with their provider, owning ALB, and
discoverer (e.g., prometheus (prom-alb via in-cluster)).
Membership additions and removals are logged at info with the member name
and origin (credentials embedded in origin URLs are masked), and the full
membership is logged at debug on every change; all autodiscovery log
events carry scope=discovery for easy filtering. Per-ALB member-count,
member-change, snapshot-result, and refresh-staleness metrics — and
per-discoverer refresh-error counters — are documented in
metrics.md. When the ALB has a tracing_name configured,
each membership reconcile cycle is traced as an alb.discovery.reconcile
span via the standard tracing subsystem; the request hot
path is never traced by discovery.
7.3 - Rule Backend
The Rule Backend is not really a true Backend; it only routes inbound requests to other configured Backends, based on how they match against the Rule’s cases.
A Rule is a single inspection operation performed against a single component of an inbound request, which determines the Next Backend to send the request to. The Next Backend can also be a rule Backend, so as to route requests through multiple Rules before arriving at a true Backend destination.
A request that asks to switch protocols, such as a WebSocket handshake, is tunneled by the backend the rule sends it to.
A rule can optionally rewrite multiple portions of the request before, during and after rule matching, by using request rewriters, which allows for powerful and limitless combinations of request rewriting and routing.
Rule Parts
A rule has several required parts, as follows:
Required Rule Parts
input_source- The part of the Request the Rule inspectsinput_type- The source data typeoperation- The operation taken on the input sourcenext_route- The Backend Name indicating the default next route for the Rule if no matching cases. Not required ifredirect_urlis provided.redirect_url- The fully-qualified URL to issue as a 302 redirect to the client in the default case. Not required ifnext_routeis provided.
Optional Rule Parts
input_key- case-sensitive lookup key; required when the source isheaderor URLparaminput_encoding- the encoding of the input, which is decoded prior to performing the operationinput_index- when > -1, the source is split into parts and the input is extracted from parts[input_index]input-delimiter- when input_index > -1, this delimiter is used to split the source into parts, and defaults to a standard space (’ ‘)ingress_req_rewriter name- provides the name of a Request Rewriter to operate on the Request before rule execution.egress_req_rewriter name- provides the name of a Request Rewriter to operate on the Request after rule execution.nomatch_req_rewriter name- provides the name of a Request Rewriter to operate on the Request after rule execution if the request did not match any cases.max_rule_executions- limits the number of rules a Request is passed through, and aborts with a 400 status code when exceeded. Default is 16. The first rule a request reaches sets its budget from this value; each later rule it passes through may lower the budget but not raise it.
input_source permitted values
| source name | example extracted value |
|---|---|
| url | https://example.com:8480/path1/path2?param1=value |
| url_no_params | https://example.com:8480/path1/path2 |
| scheme | https |
| host | example.com:8480 |
| hostname | example.com |
| port | 8480 (inferred from scheme when no port is provided) |
| path | /path1/path2 |
| params | param1=value |
| param | (must be used with input_key as described below) |
| header | (must be used with input_key as described below) |
| has_param | true or false: whether the query parameter named by input_key is present, even if empty |
| has_header | true or false: whether the header named by input_key is present, even if empty |
For a client request, url, scheme, host, hostname and port describe the URL the client requested: the Host header supplies the host and port, and the scheme is https when the request arrived over TLS.
input_type permitted values and operations
| type name | permitted operations |
|---|---|
| string (default) | prefix, suffix, contains, eq, md5, sha1, modulo, rmatch |
| num | eq, le, ge, gt, lt, modulo |
| bool | eq |
Rule Cases
Rule cases define the possible values are able to alter the Request and change the next route.
Case Parts
Required Case Parts
matches- A string list of values applicable to this case.next_route- The Backend Name indicating the next route for the Rule when a request matches this Case. Not required ifredirect_urlis provided.redirect_url- The fully-qualified URL to issue as a 302 redirect to the client when the Request matches this Case. Not required ifnext_routeis provided.
Optional Case Parts
req_rewriter name- provides the name of a Request Rewriter to operate on the Request when this case is matched.
Example Rule - Route Request by Basic Auth Username
In this example config, requests routed through the /example path will be compared against the rules and routed to either the Reader cluster or the Writer cluster. Curling http://trickster-host/example/path would route to the reader or writer cluster based on a provided Authorization header.
rules:
example-user-router:
# default route is reader cluster
next_route: example-reader-cluster
input_source: header
input_key: Authorization
input_type: string
input_encoding: base64 # Authorization: Basic <base64string>
input_index: 1 # Field 1 is the <base64string>
input_delimiter: ' ' # Authorization Header field is space-delimited
operation: prefix # Basic Auth credentials are formatted as user:pass,
# so we can check if it is prefixed with $user:
cases:
- matches: # writers
- 'johndoe:'
- 'janedoe:'
next_route: example-writer-cluster
backends:
example:
provider: rule
rule_name: example-user-router
example-reader-cluster:
provider: rpc
origin_url: 'http://reader-cluster.example.com'
example-writer-cluster:
provider: rpc
origin_url: 'http://writer-cluster.example.com'
path_routing_disabled: true # restrict routing to this backend via rule only, so
# users cannot directly access via /example-writer-cluster/
Example Rule - Route Request by Path Regex
In this example config, requests routed through the /example path will be compared against the rules and routed to either the Reader cluster or the Writer cluster. Curling http://trickster-host/example/reader and http://trickster-host/example/writer would route to the reader or writer cluster by matching the path.
rules:
example-user-router:
# default route is reader cluster
next_route: example-reader-cluster
input_source: path
input_type: string
operation: rmatch # perform regex match against the path to see if it matches 'writer
operation_arg: '^.*\/writer.*$'
cases:
- matches:
- 'true' # rmatch returns true when the input matches the regex; update next_route
next_route: example-writer-cluster
backends:
example:
provider: rule
rule_name: example-user-router
example-reader-cluster:
provider: rpc
origin_url: 'http://reader-cluster.example.com'
example-writer-cluster:
provider: rpc
origin_url: 'http://writer-cluster.example.com'
path_routing_disabled: true # restrict routing to this backend via rule only, so
# users cannot directly access via /example-writer-cluster/
Example Rule - Rewrite a Hostname From a Regex Capture
An rmatch rule makes its numeric and named capture groups available to the matched case and egress request rewriters. This example routes a request containing a three-character tenant label and prefixes the destination hostname with that tenant.
request_rewriters:
tenant-host:
instructions:
- [ 'hostname', 'set', '${tenant}.writer.example.com' ]
rules:
tenant-router:
next_route: example-reader-cluster
input_source: path
input_type: string
operation: rmatch
operation_arg: '\{mylabel="(?P<tenant>[a-z0-9]{3})"\}'
cases:
- matches: [ 'true' ]
req_rewriter_name: tenant-host
next_route: example-writer-cluster
backends:
example:
provider: rule
rule_name: tenant-router
example-reader-cluster:
provider: rpc
origin_url: 'http://reader-cluster.example.com'
example-writer-cluster:
provider: rpc
origin_url: 'http://writer-cluster.example.com'
path_routing_disabled: true
For input_source: path, matching uses Go’s decoded URL.Path. For example, %7Bmylabel%3D%22abc%22%7D is matched as {mylabel="abc"} and ${tenant} expands to abc. ${0} represents the complete regex match, while ${1} represents the first capture group. See Request Rewriters for token lifetime and safety details.
When the destination hostname is a DNS SRV owner name, the destination backend can resolve it with origin_resolution.mode: srv, which takes the port from the SRV answer, so one backend can reach many origins.
8 - Request Handling
8.1 - Customizing HTTP Path Behavior
Trickster supports, via configuration, customizing the upstream request and downstream response behavior on a per-Path, per-Backend basis, by providing a paths configuration section for each backend configuration. Here are the basic capabilities for customizing Path behavior:
- Modify client request headers prior to contacting the origin while proxying
- Modify origin response headers prior to processing the response object in Trickster and delivering to the client
- Modify the response code and body
- Limit the scope of a path by HTTP Method
- Select the HTTP Handler for the path (
proxy,proxycacheor a published provider-specific handler) - Select which HTTP Headers, URL Parameters and other client request characteristics will be used to derive the Cache Key under which Trickster stores the object.
- Disable Metrics Reporting for the path
- Hide the
X-Trickster-Resultresponse header from the client
Path Matching Scope
Paths are matchable as exact, prefix, segment or regex
The default match is exact, meaning the client’s requested URL Path must be an exact match to the configured path in order to match and be handled by a given Path Config. For example a request to /foo/bar will not match an exact Path Config for /foo.
A prefix match will match any client-requested path to the Path Config with the longest prefix match. A prefix match Path Config to /foo will match /foo/bar as well as /foobar and /food. A basic string match is used to evaluate the incoming URL path, so it is recommended to consider finishing paths with a trailing /, like /foo/ in Path Configurations, if needed to avoid any unintentional matches.
A segment match is a prefix match that only matches on a path segment boundary: a segment Path Config for /foo matches /foo, /foo/ and /foo/bar, but not /foobar. A segment path ending in /, like /foo/, matches /foo/bar but not /foo. Segment and prefix paths share one tier and are evaluated longest path first, so the two kinds can be mixed freely. This is the matching the Kubernetes Ingress Prefix and Gateway API PathPrefix types define.
A segment and a prefix path may name the same path; both are kept, and they are tried in the order described under Header and Query Parameter Conditions below, since neither is more specific than the other in the router’s eyes. A plain prefix listed first therefore answers everything the segment would, leaving it unreached, so list the segment first when you want it to serve the paths on a boundary.
Header and Query Parameter Conditions
A Path Config may also be conditioned on the request’s headers and query parameters with match_headers and match_query_params. Each is a list of conditions with a name, a value and an optional regex flag. A condition is satisfied when the named header or parameter is present and its first value equals value, or, with regex: true, matches value as an RE2 regular expression. The expression is not anchored, so anchor it (^...$) to require a whole-value match. A header or parameter that is absent never satisfies a condition, even one an empty value would satisfy; a header that is present and empty does. Header names are matched case-insensitively; parameter names are case-sensitive, as query parameters are.
These are match conditions and are distinct from request_headers and request_params, which modify the request after it has matched.
paths:
- path: /api/
match_type: prefix
match_headers:
- name: X-Tenant
value: gold
handler: proxycache
...
- path: /api/
match_type: prefix
match_query_params:
- name: version
value: '^v[0-9]+$'
regex: true
handler: proxy
...
- path: /api/
match_type: prefix
handler: proxy
...
Conditioned paths on the same path and methods are tried in ascending match_order, then in the order they appear in the configuration, and the first one whose conditions the request satisfies handles it. A path with no conditions always matches, so it must rank last among the paths sharing its path and method: list it after them, or give it a higher match_order. match_order is what orders candidates that come from different backends, since backends are a map with no declaration order; equal orders across backends are tried in backend name order. A request satisfying none of the conditioned paths continues to the less specific tiers as if the path did not exist: a shorter prefix, a regex path, a less specific host, and finally a 404. A request for a path whose methods do not include the request method is answered 405 as always; conditions do not change that.
Compiled once at load, conditions cost nothing for paths that declare none: a router with no conditioned paths matches exactly as it did before they existed. The query string is parsed at most once per request, and only when a reachable conditioned path names a query parameter.
Regex Paths
A path can also be a regular expression, evaluated against the client’s requested URL Path. A path is treated as a regex when either:
- the path value starts with
^/(or the escaped form^\/) — this is auto-detected and applies regardless of any configuredmatch_type; or - the Path Config explicitly sets
match_type: regex, in which case the path need not start with^/; Trickster will prepend a^anchor if absent, so match semantics are consistent.
Regex paths use Go’s RE2 syntax (linear-time matching; no backtracking). Patterns are always anchored at the start with ^. A $ end anchor is honored when present but never required — an unanchored end behaves like a prefix-style match, so ^/api/[0-9]+ matches /api/42 and /api/42/details alike, while ^/api/[0-9]+$ matches only /api/42.
Evaluation order: regex paths are evaluated only after both exact and prefix matching have missed. Within the regex tier, patterns are evaluated longest-pattern-string first; equal-length patterns are evaluated in the order they appear in the configuration; the first pattern that matches wins.
Provider Default Paths
Each backend provider defines its own default paths, which are merged with the paths you configure: a configured path replaces a default with the same path and methods, and any remaining defaults are kept. This is why a backend that configures only /api still answers every other path through its provider’s catch-all / route.
Set path_defaults_disabled: true on a backend to register only its configured paths. The backend then answers exactly the paths it lists and returns 404 for everything else. This is how the Kubernetes controller keeps a generated backend to the paths its route declares, and it is also useful for narrowly scoping a backend by hand. A backend with defaults disabled and no configured paths serves nothing.
Host Resolution
A backend’s paths are matched either for the hostnames it lists in hosts, or, when it sets any_host_routing: true, for every hostname reaching its listeners. The two are mutually exclusive and configuring both fails validation. A backend that sets neither is reachable only through its /backend_name/ path, an ALB, or a rule.
When a backend lists hosts, its paths are matched only for requests whose Host header (port excluded, compared case-insensitively) matches an entry. An entry may be an exact hostname, a single-label wildcard such as *.example.com, which matches api.example.com but not example.com or a.b.example.com, or an any-depth wildcard such as **.example.com, which matches api.example.com and a.b.example.com but still not example.com. A wildcard must be the leading label and nothing else in the entry may contain a *; a backend may not list both spellings for one domain.
The router resolves each request by host tier and stops at the first match: the exact hostname; then, at each label boundary of the hostname working outward, a *. wildcard covering the hostname (its first boundary only) before a **. wildcard at the same boundary; then the global routes, which are those of backends using any_host_routing or is_default. So for a.b.example.com the order is a.b.example.com, *.b.example.com, **.b.example.com, **.example.com, **.com, then global. Within each tier the path tiers run in the order above: exact, then longest prefix, then regex. So an exact host beats a wildcard host, a wildcard beats a global route, and a host-specific regex path beats a global exact path. Any-depth wildcards cost one map lookup per label boundary, and only when a backend registers one.
Hiding the Result Header
Every response carries X-Trickster-Result, which says how the request was handled (trickster-result.md). Set hide_result_header: true on a path to withhold it from the client, for a path facing the public where the cache behavior is nobody’s business but the operator’s. Metrics and the access log still record the result: the value is kept for the access log’s %{cache-status}x and %{engine}x fields, so hiding it from clients does not blind the operator. The decision is the serving path’s: when an ALB or rule path that hides the header dispatches to a backend whose path does not, the header is served, and only a response the dispatching path answered itself is withheld. The Kubernetes controller sets it from a TricksterCachePolicy with resultHeader: Hide.
paths:
- path: /
match_type: prefix
handler: proxycache
hide_result_header: true
Geo ACLs on Paths
A path’s geo_acl_name replaces its backend’s geo ACL for requests on that path, and geo_acl_name: none clears the backend’s, as authenticator_name does. A path judges HTTP requests only; a native protocol or stream listener judges sessions by the backend’s geo ACL.
backends:
video:
provider: rpc
origin_url: 'http://video.example.com'
geo_acl_name: north-america
paths:
- path: /trailers/
match_type: prefix
handler: proxycache
geo_acl_name: none # trailers are served everywhere
- path: /eu/
match_type: prefix
handler: proxycache
geo_acl_name: europe # this path is judged by another geo ACL
Dispatch-Only Paths
A path with dispatch_only: true is registered on the backend’s own router only, never on a listener. It is reachable when another backend hands the request over — an ALB selecting this backend from its pool, or a rule whose next_route names it — and is invisible to clients otherwise, even when the backend lists hosts or sets any_host_routing. Use it to keep the entry point for a path on one backend while the backend that finally serves it stays reachable only through that entry point. The Kubernetes controller relies on it: a route that another route’s header match fronts keeps its path dispatch-only, so the rule that tests the header is the only thing registered on the listener.
Catch-all warning: a classic catch-all path — match_type: prefix on /, or an effective catch-all like /api in front of regexes that all begin ^/api — always prefix-matches first and prevents the regex tier from ever being evaluated for those requests. When using regex paths, define the catch-all as a regex too (e.g. ^/.*), which sorts shortest and therefore evaluates last. Trickster logs a startup warning when a backend defines regex paths alongside a / prefix catch-all.
backends:
default:
provider: rpc
origin_url: 'http://example.com'
paths:
# auto-detected as a regex path (starts with ^/)
- path: '^/api/[0-9]+/results'
methods: [ GET ]
handler: proxycache
# explicit opt-in; Trickster anchors this to ^/reports/(annual|monthly)/
- path: '/reports/(annual|monthly)/'
match_type: regex
methods: [ GET ]
handler: proxy
# regex catch-all: evaluates last, does not shadow the regexes above
- path: '^/.*'
methods: [ '*' ]
handler: proxy
Timeouts and Retries
A backend’s timeout bounds how long the origin may take to begin a
response and how long its body may stall. A path may bound more: timeout
is a deadline on the whole upstream exchange for requests on the path,
retries included, and attempt_timeout a deadline on each attempt; both
cancel the upstream request when reached, and attempt_timeout may not
exceed timeout. A retry block repeats a failed idempotent request
(GET, HEAD, OPTIONS, TRACE, PUT, DELETE, with a body only when
it can be replayed) up to attempts more times: always after a connection
failure, and after a response whose status is in codes. backoff waits
between attempts; a request the client abandoned is never retried. Retries
are drawn from a budget per path, budget_percent of the requests the path
saw in the last ten seconds with three always allowed, so an origin that
is down is not swamped by retries of everything that reaches it. Retried
requests are counted in trickster_proxy_upstream_retries_total. A budget_percent of 100 removes the budget, so every eligible request is retried; a request whose timeout or attempt_timeout runs out with no attempt answered is answered 504, while an origin that cannot be reached at all is answered 502.
backends:
default:
provider: rpc
origin_url: 'http://example.com'
paths:
- path: /api/
match_type: prefix
methods: [ GET, POST ]
handler: proxycache
timeout: 5s
attempt_timeout: 2s
retry:
attempts: 2
codes: [ 502, 503 ]
backoff: 100ms
budget_percent: 20
Mirroring Requests
A path’s mirrors list sends a copy of each request it serves to other
backends and discards the copies’ responses; the client’s response comes
from the path’s own handler alone. A copy is served through the mirror
backend’s router, so it follows that backend’s paths and handlers, on its
own goroutine after the original has been handed to the path’s handler; a
bodied request is buffered so every reader gets it. Each mirror’s percent
selects the share of requests copied (all of them when unset) and its
max_in_flight bounds the copies in progress at once (64 when unset); a
copy the bound refuses is dropped. A copy is never mirrored again, however
the mirror backend’s paths are configured. Copies are counted in
trickster_proxy_mirror_requests_total as sent or dropped. Every mirror
backend must exist and may not be a template.
backends:
default:
provider: rpc
origin_url: 'http://example.com'
paths:
- path: /api/
match_type: prefix
methods: [ '*' ]
handler: proxy
mirrors:
- backend_name: shadow
percent: 10
shadow:
provider: rp
origin_url: 'http://shadow.example.com'
Forwarding Trailers
Proxy and caching handlers relay origin trailers automatically: the client’s
TE: trailers reaches the origin, and trailers follow the response body.
A response served from cache has no trailers, so gRPC paths should use the
proxy handler, which the Kubernetes controller selects for every GRPCRoute.
Method Matching Scope
The methods section of a Path Config takes a string array of HTTP Methods that are routed through this Path Config. You can provide [ '*' ] to route all methods for this path.
The QUERY Method
Trickster supports the HTTP QUERY method (RFC 10008). A QUERY is a safe, idempotent, cacheable request that carries its query in the request body.
QUERYis cacheable, likeGETandHEAD. Its cache key always includes the request body, theContent-Type(case-insensitive) and anyContent-Encoding, soQUERYrequests with different content never share an entry. AQUERYand aGETfor the same URL are cached separately.- A successful
QUERYdoes not invalidate cached responses for its URL, because it is safe. A successful unsafe request, such as aPOST, invalidates the cachedGETandHEADresponses for its URL but not cachedQUERYresponses. Those expire at the end of their freshness lifetime. - A conditional
QUERYis evaluated as it would be forGET. ARangeheader on aQUERYis ignored, and the full response is returned. - The
reverseproxycacheprovider’s default paths accept and cacheQUERY. Paths using theproxyhandler relay aQUERYto the origin as sent.[ '*' ]includesQUERY.
Time series origins do not accept QUERY, so on the provider query endpoints below Trickster forwards a QUERY to the origin as a POST. It first checks the request’s Content-Type: a missing one is answered with 400 and an unsupported one with 415. These endpoints advertise their media types in an Accept-Query response header, unless the origin sends its own. A QUERY and the equivalent POST share cache entries.
| Provider | Endpoints | QUERY media types |
|---|---|---|
| Prometheus, and the Prometheus APIs that GreptimeDB and VictoriaMetrics serve | /api/v1/query_range, /api/v1/query, /api/v1/series, /api/v1/labels, /api/v1/query_exemplars, /api/v1/format_query, /api/v1/parse_query | application/x-www-form-urlencoded |
| InfluxDB | /query | application/x-www-form-urlencoded |
| InfluxDB | /api/v2/query | application/json, application/vnd.flux |
| InfluxDB | /api/v3/query_sql, /api/v3/query_influxql | application/json, application/x-www-form-urlencoded |
| ClickHouse | / | text/plain, application/sql |
| Graphite | /render, /metrics/find, /metrics/expand, /metrics/index.json, /tags, /tags/ | application/x-www-form-urlencoded |
| Druid | /druid/v2, /druid/v2/sql | application/json |
| VictoriaMetrics | /render, /metrics/find, /metrics/expand, and each under /graphite | application/x-www-form-urlencoded |
| GreptimeDB | /v1/sql | application/x-www-form-urlencoded |
A configured path is handled the same way when it uses one of these endpoints’ handlers (for example, handler: query_range) and its methods include QUERY. If a configured path replaces a provider default for only some of its methods, such as [ GET, POST ], the default still serves QUERY on that path. Add QUERY to the configured path’s methods to change that too.
Suggested Use Cases
- Redirect a path by configuring Trickster to respond with a
302response code and aLocationheader - Issue a blanket
401 Unauthorizedcode and custom response body to all requests for a given path. - Adjust Cache Control headers in either direction
- Affix an Authorization header to requests proxied out by Trickster.
- Control which paths are cached by Trickster, and which ones are simply proxied.
Request Rewriters
You can configure paths send inbound requests through a request rewriter that can modify any aspect of the inbound request (method, url, headers, etc.), before being processed by the path route. This means, when the path route inspects the request, it will have already been modified by the rewriter. Provide a rewriter with the req_rewriter_name config. It must map to a named/configured request rewriter (see request rewriters for more info). Note, you can also send requests through a rewriter at the backend level. If both are configured, backend-level rewriters are executed before path rewriters are.
request_rewriters:
# this example request rewriter adds an additional header to the request
# you can include as many instructions in rewriter as required
example:
instructions:
- [ header, set, Example-Header-Name, Example Value ]
backends:
default:
provider: rpc
origin_url: 'http://example.com'
paths:
- path: /
req_rewriter_name: example
Header and Query Parameter Behavior
In addition to running the request through a named rewriter, it is currently possible to make similar changes to the request with legacy path features that are described in this section. Note that these are likely to be deprecated in a future Trickster release, in favor of the more versatile named rewriters described above, which accomplish the same thing. Currently, if both a named rewriter and legacy path-based rewriting configs are defined for a given path, the named rewriter will be executed first.
Basics
You can specify request query parameters, as well as request and response headers, to be Set, Appended or Removed.
Setting
To Set a header or parameter means to insert if non-existent, or fully replace if pre-existing. To set a header, provide the header name and value you wish to set in the Path Config request_params, request_headers or response_headers sections, in the format of 'Header-or-Parameter-Name' = 'Value'.
The Host request header is special: it is what the upstream connection sends as its Host, so a request_headers entry naming it, under any spelling of the name, replaces the Host the origin sees rather than adding a header line. +Host behaves as a set, since Host carries one value, and -Host clears the override so the origin’s own host is sent.
As an example, if the client request provides a Cache-Control: no-store header, a Path Config with a header ‘set’ directive for 'Cache-Control' = 'no-transform' will replace the no-store entirely with a no-transform; client requests that have no Cache-Control header that are routed through this Path will have the Trickster-configured header injected outright. The same logic applies to query parameters.
Environment Variable Substitution
The request_headers, request_params and response_headers sections support environment variable substitution. This means you can use environment variables in your header or query parameter values. Example:
backends:
default:
# ...
paths:
- path: /
# ...
request_params:
'token': '${REQUEST_PARAM_TOKEN}'
request_headers:
'X-Auth-Token': '${REQUEST_HEADER_TOKEN}'
response_headers:
'X-Auth-Token': '${RESPONSE_HEADER_TOKEN}'
Appending
Appending a means inserting the header or parameter if it doesn’t exist, or appending the configured value(s) into a pre-existing header with the given name. To indicate an append behavior (as opposed to set), prefix the header or parameter name with a ‘+’ in the Path Config.
Example: if the client request provides a token=SomeHash parameter and the Path Config includes the parameter '+token' = 'ProxyHash', the effective parameter when forwarding the request to the origin will be token=SomeHash&token=ProxyHash.
Removing
Removing a header or parameter means to strip it from the HTTP Request or Response when present. To do so, prefix the header/parameter name with ‘-’, for example, -Cache-control: none. When removing headers, a value is required to be provided in order to conform to YAML specification; this value, however, is ineffectual. Note that there is currently no ability to remove a specific header value from a specific header - only the entire removal header. Consider setting the header value outright as described above, to strip any unwanted values.
Response Header Timing
Response Header injections occur as the object is received from the origin and before Trickster handles the object, meaning any caching response headers injected by Trickster will also be used by Trickster immediately to handle caching policies internally. This allows users to override cache controls from upstream systems if necessary to alter the actual caching behavior inside of Trickster. For example, InfluxDB sends down a Cache-Control: No-Cache header, which is fine for the user’s browser, but Trickster needs to ignore this header in order to accelerate InfluxDB; so the default Path Configs for InfluxDB actually removes this header.
Cache Key Components
By default, Trickster will use the HTTP Method, URL Path and any Authorization header to derive its Cache Key. On a backend with preserve_host, the client’s Host is part of the key as well, since the origin sees it and may answer by it; a path whose request_headers fixes Host (a Host or +Host entry with a value, or -Host) sends one value, so its objects are keyed without it, while an empty Host entry changes nothing and keys as if absent. In a Path Config, you may specify any additional HTTP headers and URL Parameters to be used for cache key derivation, as well as information in the Request Body.
Using Request Body Fields in Cache Key Hashing
Trickster supports the parsing of the HTTP Request body for the purpose of deriving the Cache Key for a cacheable object. Note that body parsing requires reading the entire request body into memory and parsing it before operating on the object. This will result in slightly higher resource utilization and latency, depending upon the size of the client request body.
Body parsing is supported when the request’s HTTP method is POST, PUT, PATCH or QUERY, and the request Content-Type is either application/x-www-form-urlencoded, multipart/form-data, or application/json.
In a Path Config, provide the cache_key_form_fields setting with a list of form field names to include when hashing the cache key.
Trickster supports parsing of the Request body as a JSON document, including documents that are multiple levels deep, using a basic pathing convention of forward slashes, to indicate the path to a field that should be included in the cache key. Take the following JSON document:
{
"requestType": "query",
"query": {
"table": "movies",
"fields": "eidr,title",
"filter": "year=1979"
}
}
To include the requestType, table, fields, and filter fields from this document when hashing the cache key, you can provide the following setting in a Path Configuration:
cache_key_form_fields = [ 'requestType', 'query/table', 'query/fields', 'query/filter' ]
Example Reverse Proxy Cache Config with Path Customizations
backends:
default:
provider: rpc
paths:
# root path '/'
- path: / # each path must be unique for the backend
methods: [ '*' ] # All HTTP methods applicable to this config
match_type: prefix # matches any path under '/'
handler: proxy # proxy only, no caching (this is the default)
# modify the query params en route to the origin; this adds authToken=${ROOT_REQUEST_AUTH_TOKEN}
# (sourced from the environment variable ROOT_REQUEST_AUTH_TOKEN)
request_params:
authToken: ${ROOT_REQUEST_AUTH_TOKEN}
# When a user requests a path matching this route, Trickster will
# inject these headers into the request before contacting the Origin
request_headers:
Cache-Control: No-Transform
# inject these headers into the response from the Origin
# before replying to the client
response_headers:
Expires: '-1'
# a path-level CORS policy overrides the backend policy; see docs/cors.md
cors:
mode: preserve
- path: /images/
methods:
- GET
- HEAD
handler: proxycache # Trickster will cache the images directory
match_type: prefix
response_headers:
Cache-Control: max-age=2592000 # cache for 30 days
# but only cache this rotating image for 30 seconds
- path: /images/rotating.jpg
methods:
- GET
handler: proxycache
match_type: exact
response_headers:
Cache-Control: max-age=30
'-Expires': ''
# redirect this sunsetted feature to a discontinued message
- path: /blog
methods:
- '*'
handler: localresponse
match_type: prefix
response_code: 302
response_headers:
Location: /discontinued
# redirect plaintext requests to the same URL over TLS: the redirect
# handler builds the Location from the request as the path's rewriter
# leaves it, so only the parts the rewriter set change
- path: /account
methods:
- '*'
handler: redirect
match_type: prefix
response_code: 301
req_rewriter_name: to-https
# cache this API endpoint, keying on the query parameter
- path: /api/
methods:
- GET
- HEAD
handler: proxycache
match_type: prefix
cache_key_params:
- query
# same API endpoint, different HTTP methods to route against, which are denied
- path: /api/
methods:
- POST
- PUT
- PATCH
- DELETE
- OPTIONS
- CONNECT
handler: localresponse
match_type: prefix
response_code: 401
response_body: this is a read-only api endpoint
# cache the query endpoint, permitting GET, HEAD, POST
- path: /api/query/
methods:
- GET
- HEAD
- POST
handler: proxycache
match_type: prefix
cache_key_params:
- query # for GET/HEAD
cache_key_form_fields:
- query # for POST
Redirecting Requests
The redirect handler answers a path with a redirection instead of an upstream request. Its Location is the request’s own URL as the path’s request rewriter and request_headers left it: a rewriter instruction that sets the scheme, hostname, port or path decides that part of the Location, a Host entry in request_headers decides the hostname when the rewriter did not, and whatever neither set is taken from the incoming request. The path’s other request_headers are applied too, though nothing upstream receives them. An explicit scheme with no explicit port drops the request’s port, since the redirect target is that scheme’s well-known port, and a port that is the well-known one for the scheme is omitted. The path’s response_code selects the redirection status (302 unless it names another 3xx), and its response_headers are applied, except that the composed Location always wins: a response_headers entry naming Location is overwritten by it. The redirect handler is registered by the rp and rpc providers. Like localresponse, it answers from configuration alone, so a request carrying Connection: Upgrade receives the redirection rather than being tunneled to the origin as it would be on a proxying path.
request_rewriters:
to-https:
instructions:
- [ scheme, set, https ]
backends:
default:
provider: rp
origin_url: 'http://example.com'
paths:
- path: /account
match_type: prefix
handler: redirect
response_code: 301
req_rewriter_name: to-https
A request for http://shop.example.com:8080/account/settings?tab=1 is answered with 301 and Location: https://shop.example.com/account/settings?tab=1. A fixed Location that ignores the request is better expressed with the localresponse handler, as in the example above.
Modifying Behavior of Time Series Backend
Each of the Time Series Providers supported in Trickster comes with its own custom handlers and pre-defined Path Configs that are registered with the HTTP Router when Trickster starts up.
For example, when Trickster is configured to accelerate Prometheus, pre-defined Path Configs are registered to control how requests to /api/v1/query work differently from requests to /api/v1/query_range. For example, the /ap1/v1/query Path Config uses the query and time URL query parameters when creating the cache key, and is routed through the Object Proxy Cache; while the /api/v1/query_range Path Config uses the query, start, end and step parameters, and is routed through the Time Series Delta Proxy Cache.
In the Trickster config file, you can add your own Path Configs to your time series backend, as well override individual settings for any of the pre-defined Path Configs, and those custom settings will be applied at startup.
To know what configs you’d like to add or modify, take a look at the Trickster source code and examine the pre-definitions for the selected Backend Provider. Each supported Provider’s handlers and default Path Configs can be viewed under /pkg/backends/<provider>/routes.go. These files are in a standard format that are quite human-readable, even for a non-coder, so don’t be too intimidated. If you can understand Path Configs as YAML, you can understand them as Go code.
Examples of customizing Path Configs for Providers with Pre-Definitions:
backends:
default:
provider: prometheus
paths:
# route /api/v1/label* (including /labels/*)
# through Proxy instead of ProxyCache as pre-defined
- path: /api/v1/label
methods:
- GET
match_type: prefix
handler: proxy
# route fictional new /api/v1/coffee to ProxyCache
- path: /api/v1/coffee
methods:
- GET
match_type: prefix
handler: proxycache
cache_key_params:
- beans
# block /api/v1/admin/ from being reachable via Trickster
- path: /api/v1/admin/
methods:
- GET
- POST
- PUT
- HEAD
- DELETE
- OPTIONS
match_type: prefix
handler: localresponse
response_code: 401
response_body: No soup for you!
no_metrics: true
8.2 - Request Rewriters
A Request Rewriter is a named series of instructions that modifies any part of the incoming HTTP request. Request Rewriters are used in various parts of the Trickster configuration to make scoped changes. For example, a rewriter can modify the path, headers, parameters, etc. of a URL using mechanisms like search/replace, set and append.
In a configuration, request rewriters are represented as map of instructions, which themselves are represented as a list of string lists, in the following format:
request_rewriters:
example_rewriter:
instructions:
- [ 'header', 'set', 'Cache-Control', 'max-age=60' ], # instruction 0
- [ 'path', 'replace', '/cgi-bin/', '/' ], # instruction 1
- [ 'chain', 'exec', 'remove_accept_encoding' ] # instruction 2
remove_accept_encoding:
instructions:
- [ 'header', 'delete', 'Accept-Encoding' ] # instruction 0
In this case, any other configuration entity that supports mapping to a rewriter by name can do so with by referencing example_rewriter or remove_accept_encoding. Note that example_rewriter executes remove_accept_encoding using the chain instruction.
Where Rewriters Can Be Used
Rewriters are exposed as optional configurations for the following configuration constructs:
In a backend config, provide a req_rewriter_name to rewrite the Request using the named Request Rewriter, before it is handled by the Path route.
In a path config, provide a req_rewriter_name to rewrite the Request using the named Request Rewriter, before it is handled by the Path route.
In a rule config, provide ingress_req_rewriter_name, egress_req_rewriter_name and/or nomatch_req_rewriter_name configurations to rewrite the Request using the named Request Rewriter. The meaning of Ingress and Egress, in this case, are scoped to a Request’s traversal through the Rule in which these configuration values exist, and is unrelated to the wire traversal of the request. For ingress and egress, the rewriter is executed before or after, respectively, it is handled by the Rule (including any modifications made by a matching rule case). The No-Match Request Rewriter is only executed when the request does not match to any defined case.
In a Rule’s case configurations, provide req_rewriter_name. If there is a Rule Case match when executing the Rule against the incoming Request, the configured rewriter will execute on the Request before returning control back to the Rule to execute any configured egress request rewriter and hand the Request off to the next route.
In a Request Rewriter instruction using the chain instruction type. Provide the Rewriter Name as the third argument in the instruction as follows: [ 'chain', 'exec', '$rewriter_name']. See more information below.
Regex Capture Tokens
An rmatch rule, or a backend path with match_type: regex, can expose its regular expression matches to request rewriters. Numeric tokens use ${0} for the complete match and ${1}, ${2}, etc. for capture groups. A named capture such as (?P<tenant>[a-z0-9]{3}) is also available as ${tenant}.
For a regex path, captures are taken from the request path as the client sent it and are available to that path’s req_rewriter_name rewriter and to the backend-level rewriter. Only paths whose rewriters actually use tokens pay the capture cost; every other route is matched without allocation. This is how an Ingress trickstercache.org/rewrite-target: /${2} is expressed:
request_rewriters:
strip-app:
instructions:
- [ 'path', 'set', '/${2}' ]
backends:
app:
provider: rp
origin_url: 'http://app:8080'
paths:
- path: '^/app(/|$)(.*)'
req_rewriter_name: strip-app
Captures are available only to the matched case rewriter and the current rule’s egress rewriter. The ingress rewriter runs before matching and cannot use captures from its own rule, and captures are cleared before the request enters the next route. If the regular expression does not match, no capture tokens are available to the no-match or egress rewriter. Undefined tokens are left unchanged, while an optional capture group that did not participate in a successful match expands to an empty string.
Token expansion is supported in setter and appender values for headers and parameters, and in configured values for path, method, host, hostname, port and scheme instructions. Header and parameter replace/delete instructions also expand their key, search and replacement fields. Chained rewriters receive the same captures. For example:
request_rewriters:
tenant-host:
instructions:
- [ 'hostname', 'set', '${tenant}.writer.example.com' ]
- [ 'header', 'set', 'X-Tenant', '${1}' ]
Captured values are inserted without additional validation or escaping. Use restrictive regular expressions for values written to a hostname, path, header or query parameter. In particular, a client-controlled authority token can route a request, including configured upstream credentials, to an unintended host if the expression is too broad.
When Trickster constructs the final upstream URL, only host components explicitly changed by a request rewriter override the configured backend origin_url. A hostname rewrite preserves the origin_url port, while host replaces both hostname and port. The inbound request’s host never overrides origin_url by itself.
Instruction Construction Guide
header
header rewriters modify a header with a specific name and support the following operations.
header set
header set will set a specific header to a specific value.
['header', 'set', 'Header-Name', 'header value']
header replace
header replace performs a search/replace function on the Header value of the provided Name
['header', 'replace', 'Header-Name', 'search value', 'replacement value']
header delete
header delete removes, if present, the Header of the provided Name
['header', 'delete', 'Header-Name']
header append
header append appends an additional value to Header in the format of value1[, value2=subvalue, ...]
['header', 'append', 'Header-Name', 'additional header value']
path
path rewriters modify the full or partial path and support the following operations.
path set
['path', 'set', '/new/path'] sets the entire request path to /new/path
['path', 'set', 'awesome', 0 ] sets the first part of the path (zero-indexed, split on /) to ‘awesome’. For example, /new/path => /awesome/path
path replace
['path', 'replace', 'search', 'replacement'] search replaces against the entire path scalar
['path', 'replace', 'search', 'replacement', 1] search replaces against the second part of the path; For example /my/example-search/path => /my/example-replacement/path
path prefix-replace
['path', 'prefix-replace', '/old', '/new'] replaces a leading path prefix and keeps the remainder. The prefix must match on a segment boundary, so /old matches /old and /old/items but not /older. For example /old/items?x=1 => /new/items?x=1, and /old => /new. A replacement of / strips the prefix: /old/items => /items. Paths that do not start with the prefix are left unchanged. This is the equivalent of the Gateway API URLRewrite ReplacePrefixMatch filter.
param
param rewriters modify the URL Query Parameter of the specified name, and support the following operations
param set
param set sets the URL Query Parameter of the provided name to the provided value
['param', 'set', 'paramName', 'new param value']
param replace
param replace performs a search/replace function on the URL Query Parameter value of the provided name
['param', 'replace', 'paramName', 'search value', 'replacement value']
param delete
param delete removes, if present, the URL Query Parameter of the provided name
['param', 'delete', 'paramName']
param append
param append appends the provided name and value to the URL Query Parameters regardless of whether a parameter name already exists with the same or different value.
['param', 'append', 'paramName', 'additional param value']
params
params rewriters update the entire URL parameter collection as a URL-encoded scalar string.
params set
params set will replace the Request’s entire URL Query Parameter encoded string with the provided value. The provided value is assumed to already be URL-encoded.
['params', 'set', 'param1=value1¶m2=value2']
To clear the URL parameters, use ['params', 'set', '']
params replace
params replace performs a search/replace operation on the url-encoded query string. The search and replacement values are assumed to already be URL-encoded.
['params', 'replace', 'search value', 'replacement value']
method
method rewriters update the HTTP request’s method
method set
method set sets the Request’s HTTP Method to the provided value. This value is not currently validated against known HTTP Method. The instruction should be configured to include a known and properly-formatted (all caps) HTTP Method.
['method', 'set', 'GET']
host
host rewriters update the HTTP Request’s host - defined as the hostname:port, as expressed in the Request’s Host header.
host set
host set sets the Request’s Host header to the provided value.
['host', 'set', 'my.new.hostname:9999']
['host', 'set', 'my.new.hostname'] Trickster will assume a standard source port 80/443 depending upon the URL scheme
host replace
host replace performs a search/replace operation on the Request’s Host Header.
['host', 'replace', ':8480', '']
['host', 'replace', ':443', ':8443']
['host', 'replace', 'example.com', 'trickstercache.org']
hostname
hostname rewriters update the HTTP Request’s hostname, without respect to the port.
hostname set
hostname set sets the Request’s hostname, without changing the port.
['hostname', 'set', 'my.new.hostname']
hostname replace
hostname replace performs a search/replace on the Request’s hostname, without respect to the port.
['hostname', 'replace', 'example.com', 'trickstercache.org']
port
port rewriters update the HTTP Request’s port, without respect to the hostname.
port set
port set sets the Request’s port.
['port', 'set', '8480']
port replace
port replace performs a search/replace on the port, as if it was a string. The search and replacement values must be integers and are not validated.
['port', 'replace', '8480', '']
port delete
port delete removes the port from the Request. This will cause the port to be assumed based on the URL scheme.
['port', 'delete']
scheme
scheme rewriters update the HTTP Request’s scheme (http or https).
scheme set
scheme set sets the scheme of the HTTP Request URL. This must be http or https in lowercase, and is not validated.
['scheme', 'set', 'https']
chain
chain rewriters do not directly rewrite the request, but execute other rewriters’ instructions before proceeding with the current rewriter’s remaining instructions (if any). You can create a rewriter with some reusable functionality and include that in other rewriters with a chain exec. Or you can define a rewriter that is just a list of other chained rewriters. Note that there is currently no validation of the configuration to prevent infinite cyclic chained rewriter calls. There is, however, a hard limit of 32 chained rewriter runs each time a rewriter executes; once it is reached, the remaining chained rewriters are skipped and the request proceeds to be served by the backend.
chain exec executes the supplied rewriter name. Trickster will error at startup if the rewriter name is invalid. An example is provided in the sample yaml config at the top of this article.
['chain', 'exec', 'example_rewriter']
8.3 - Request Body Handling Customizations
Request Body Size Limiter
By default, the max allowed Request Body size is 10 MB. If the client request
body in a POST, PUT, PATCH or QUERY is over 10 MB, the request will receive a
response of 413 Request Payload is too large
You can change (or bypass) this limit in the ’listeners’ Config Section per-listener:
listeners:
default:
max_request_body_size_bytes: 5120 # request bodies must be <= 5kb or returns 413
listeners:
default:
max_request_body_size_bytes: 5120 # request bodies > 5kb truncated to 5kb
truncate_request_body_too_large: true # truncate request bodies that are too large
8.4 - Cross-Origin Resource Sharing
Trickster can control Cross-Origin Resource Sharing (CORS) response headers for each backend. A path can override its backend’s policy by providing its own cors block.
For compatibility, a backend or path without a cors block uses Trickster’s legacy behavior: Access-Control-Allow-Origin is set to *, while other CORS headers from the origin are left unchanged.
Modes
| Mode | Behavior |
|---|---|
preserve | Return the origin’s Access-Control-* response headers unchanged. |
merge | Preserve origin CORS headers, then apply the configured headers as overrides. |
replace | Remove all origin Access-Control-* response headers, then apply the configured headers. |
disable | Remove all origin Access-Control-* response headers. This mode cannot include a headers block. |
When preserve or merge is configured, Trickster includes the request’s Origin header in the cache key. This prevents an origin-specific CORS response from being served to a request from another origin.
Backend Configuration
The following example replaces the origin’s CORS headers with a fixed policy:
backends:
default:
provider: reverseproxycache
origin_url: http://api.example.com
cors:
mode: replace
headers:
Access-Control-Allow-Origin: https://dashboard.example.com
Access-Control-Allow-Credentials: "true"
Access-Control-Expose-Headers: X-Trickster-Result
To preserve the origin’s CORS headers while replacing selected values, use merge:
cors:
mode: merge
headers:
Access-Control-Allow-Origin: https://trickster.example.com
To return all origin CORS headers unchanged, use preserve without a headers map:
cors:
mode: preserve
To disable CORS headers, use disable without a headers block:
cors:
mode: disable
Only Access-Control-* response headers can be configured in a cors.headers map. As with path response_headers, prefix a header name with - to remove that header in merge mode or + to append another value.
Path Overrides
A path-level policy replaces the backend policy for requests matching that path:
backends:
default:
provider: reverseproxycache
origin_url: http://api.example.com
cors:
mode: preserve
paths:
- path: /private/
match_type: prefix
cors:
mode: disable
For an ALB, user router, or rule backend that dispatches to another backend internally, the policy on the client-facing backend and path takes precedence. Internal routing therefore cannot change the CORS policy associated with the public Trickster route.
8.5 - Simulated Latency
Trickster supports simulating latency on a per-backend basis, and can simulate both consistent and random durations of latency. Simulated latency is introduced in the frontend part of the proxy, and thus works with any backend provider.
In rule, alb and other backend providers, where a request may transit multiple backend routes, only the Simulated Latency configs associated with the request entrypoint (first route) will be processed, and not any subsequent routes the request is sent through.
Consistent Latency Duration
In the Backend configuration, add a latency_min value > 0, and the configured amount of latency will be introduced for each incoming request.
Random Latency
To simulate random latency, set latency_max to a value > latency_min, which may be 0 for random latency. Trickster will introduce a random amount of latency between (inclusive) the provided min and max values.
Response Header
When simulated latency is applied to a request, an x-simulated-latency header will be included in the corresponding response indicating the duration of the latency applied in milliseconds. The format of the latency header is as follows:
GET /api/v1/query?query=up HTTP/1.1
Accept: */*
...
HTTP/1.1 200 OK
X-Simulated-Latency: 300ms
X-Trickster-Result: engine=DeltaProxyCache; status=hit; ffstatus=hit
Date: Mon, 06 Sep 2021 04:07:20 GMT
...
Example Config
backends:
default:
origin_url: https://www.example.com
provider: reverseproxy
#
# introduce random latency between 50 and 150 milliseconds on each request
latency_min: 50ms
latency_max: 150ms
backend2:
origin_url: https://www.trickstercache.org
provider: reverseproxy
#
# introduce latency of 300 milliseconds on each request
latency_min: 300ms
9 - Security & Policy
9.1 - IP Access Lists
ip_acls defines named IPv4 and IPv6 access lists. A listener, backend, or path selects one with ip_acl_name. The exhaustive example is examples/conf/example.full.yaml.
ip_acls:
office:
match: longest # longest (default) | ordered
default: deny # deny (default) | allow
allow: [10.0.0.0/8, 2001:db8::/32]
deny: [10.0.99.0/24]
allow_file: /etc/trickster/office-allow.lst
deny_file: /etc/trickster/blocklist.lst
source: client_ip # client_ip (default) | peer
action: reject # reject (default) | drop
status: 403 # 400-599; default 403
listeners:
default:
ip_acl_name: office
backends:
api:
ip_acl_name: partners
paths:
- path: /public/
ip_acl_name: none
- path: /admin/
ip_acl_name: office
Configuration and composition
Each entry under ip_acls is one named list. ip_acl_name attaches that name to a listener, a backend, or a path. none is reserved and is not a list name. On a listener or a backend it is an undefined name and the configuration fails to load. On a path it is the clear operation below.
Three attachments can apply to one HTTP request, in two decisions:
- The listener list is its own decision. It does not inherit, replace, or clear a backend or path list.
- The route decision is the path list when the path names one. That name replaces the backend list for that path, and the decision is counted with scope
path. An empty pathip_acl_nameinherits the backend list, and that decision is counted with scopebackend.ip_acl_name: noneclears the backend list for that path, so the route makes no access-list decision.
When both the listener and the route have a list, the request must be allowed by each. A denial from either one stops the request.
Match modes
match is longest or ordered. Empty means longest. default is the verdict when nothing matches. Empty means deny. allow is the other value.
longest uses the longest matching prefix. An equal-length allow and deny is a deny, and the load reports a warning. allow, deny, allow_file, and deny_file belong to this mode. rules in this mode fails the load.
ordered walks rules from top to bottom and uses the first match. allow, deny, allow_file, and deny_file next to rules fail the load; each rule sets exactly one of those. An empty rules list is valid. With default: deny it is the same warning as any other list that has no entries: every address is denied. A rule that an earlier rule already covers completely is unreachable and is a load warning. The commented legacy list in example.full.yaml is this mode:
ip_acls:
legacy:
match: ordered
default: deny
rules:
- deny: 10.1.2.0/24
- allow: 10.0.0.0/8
- deny: all
A list with no entries and default: deny is valid and warns, because every address is denied. That warning covers an empty longest list and an empty ordered rule list alike.
Addresses and files
An entry is an address or a CIDR, in IPv4 or IPv6. Host bits in a CIDR are masked. all means 0.0.0.0/0 and ::/0. ::/0 alone is IPv6. An IPv4-mapped IPv6 address is matched as its IPv4 form. At request time an address that cannot be parsed is denied, whatever default is. A zoned IPv6 address matches no prefix, so default applies.
allow_file and deny_file are one entry per line. A blank line, or a line whose first non-space character is #, is skipped. The file is read when the configuration is loaded and on every configuration reload. A missing or unreadable file fails that load. A reload happens only when a configuration file has changed, and an access-list file is not one: after editing only the list, update a configuration file’s modification time (touch it, for example) before sending the reload, or the reload is skipped and the old list stays.
client_ip and peer
source is client_ip or peer. Empty means client_ip. peer is valid only on a listener. A backend or path that names a peer list fails validation.
client_ip is the address already resolved through the listener’s trusted_proxies and, where the listener reads one, its PROXY protocol header. See Trusted Proxies. peer is the socket peer. The two are not interchangeable across protocols.
HTTP and HTTP/3
On HTTP and HTTP/3, client_ip is judged in middleware, after trusted-proxy and PROXY resolution. A peer list on HTTP/1 or HTTP/2 is judged when the TCP connection is accepted and is not judged again in middleware. HTTP/3 has no TCP accept, so a peer list is judged in middleware from the QUIC peer address.
TCP, TLS, and UDP
On a tcp or tls listener, peer is judged at accept, before the PROXY header and before TLS. client_ip is judged later, during stream admission, from the client address on the flow. That address is the socket peer, or the address from a trusted PROXY header when the listener has read one.
UDP has no TCP accept. A UDP listener list, whichever its source, and the UDP backend list are judged during stream admission when the flow opens. Later datagrams of a flow that already opened are not judged again. A denied UDP flow stays denied for the datagrams that follow.
A backend peer list is not a valid attachment, so stream admission does not enforce one.
Stream admission enforces the list on the backend selected by the listener table, including a front ALB. Lists on stream ALB pool members, nested ALBs, or discovery templates are not supported and fail configuration validation. This includes named path lists on those members or templates, unless HTTP also serves the member: a path list then judges its HTTP requests. Attach the policy to the listener or front ALB instead. Geo ACLs follow the same rule.
Native protocols
A native listener (mysql, postgres, clickhouse, flight-sql) judges its list at accept. client_ip on that listener is the socket peer, and only when proxy_protocol is off. client_ip together with proxy_protocol is refused, because the socket peer and the address in the header are different addresses. client_ip with proxy_protocol and no trusted_proxies still loads on a listener that reads a PROXY header, including udp, and warns: an empty trusted-proxy list believes every peer’s header.
Native sessions are judged by the listener list only, so a backend that has an access list and is mapped to a native listener loads with a warning. The configuration remains valid. A ClickHouse native listener also sends each query through the backend’s routes with the session’s address, so the backend or path list judges every query: reject fails the query with exception 62 and keeps the session open, and drop closes the session. A ClickHouse backend served only over HTTP does not get the warning.
Actions
action is reject or drop. Empty means reject. status is the HTTP status used by reject, from 400 to 599. Empty means 403. Both actions count as a deny in the decision metric below.
HTTP
reject writes that status, an empty body, and Cache-Control: no-store. The response then follows the normal access-log path.
drop aborts the request with http.ErrAbortHandler. No response bytes are written, and the access log does not write its completion line. See access-logs.md.
drop is refused when the list is attached to a backend or path that an HTTP listener serves, including a backend mapped to both HTTP and a stream listener. HTTP reachability includes dispatch through ALB pools, discovery templates, rules, user routers, and mirrors. A backend mapped only to tcp, tls, or udp may use drop.
Streams
On TCP and TLS, reject resets the connection and drop closes it. Neither writes an HTTP response. On UDP, both actions are the admission verdict for that flow: the flow is not opened, and later datagrams from that client are not admitted again. An HTTP drop does not describe this path.
Readiness
The listener list wraps every handler on that listener. On a proxy listener that includes the readiness handler. On the management listener it includes that listener’s routes. Readiness is not exempt from the list. A client the list denies does not receive a ready response from a listener that list protects.
Reload
Access-list files are read again only when the configuration is loaded or reloaded, and a reload is skipped while no configuration file has changed; see Addresses and files. An access-list change does not by itself rebind the listener: the running listener swaps in the new list. Connections accepted after the reload, and stream flows admitted after the reload, are judged by the new list.
A request already in progress is not judged a second time. An established TCP or TLS flow keeps the upstream it was relayed to and is not judged again. An established UDP flow is not admitted again. A flow that was denied stays denied; a flow that was allowed is not closed because the list changed.
Metrics
trickster_ip_acl_decisions_total counts each decision an attached list actually makes. The labels are:
| Label | Values |
|---|---|
ip_acl | the list name |
scope | listener, backend, or path |
verdict | allow or deny |
reject and drop are both deny. The scope is the attachment that enforced the list: listener for the listener, path when the path named its own list, and backend when the path inherited the backend list. Each enforcement point increments the counter once. A TCP or TLS peer decision counted at accept is not counted again in HTTP middleware or in stream admission. See metrics.md.
Debug logging
A denial writes one application log line at debug: ip acl denied, with ip_acl, scope, address, and action. Allows are not logged. The default log level does not show the line. This is separate from the access log and from the decision metric.
Kubernetes
Generated backends can take a list from kubernetes.defaults.ip_acl_name, or from the GatewayClass parameter ip_acl_name when that parameter is set. Only a list with source: client_ip and action: reject is eligible. There is no Ingress annotation and no cache-policy field for the name. See kubernetes-ingress.md and kubernetes-gateway.md.
Compile time
Compiling a file of 100,000 IPv4 allow entries was measured with go test -bench=BenchmarkCompileFile100k -benchmem -count=1 ./pkg/proxy/ipacl/ on darwin/arm64 (Apple M1):
BenchmarkCompileFile100k-8 24 55277351 ns/op 46640989 B/op 600334 allocs/op
That is the measured cost of compiling that file, not of matching one address against an already compiled list.
9.2 - Geo ACLs
Geo ACLs restrict backends and paths by where their clients are. Other products call this geo-restriction, geo-blocking, geo-filtering or geofencing. A geo ACL allows or denies countries, first-level subdivisions (states, provinces, regions) and continents, and refuses a client in the form its protocol expects: an HTTP status and body, a native database error, or a reset stream.
Two top-level sections configure it:
geo_locatorsare named lookups that place a client address: a MaxMind DB format database, a list of RFC 8805 geofeed entries, or a location header a CDN in front of Trickster sets.geo_aclsare named lists of the locations that may, or may not, pass. Each names the locator it uses, so many geo ACLs share one database.
Backends and paths name a geo ACL with geo_acl_name.
Trickster ships no location database and downloads none. You provide the file and keep it current; see Where location data comes from. To add a source of location data, see the locator provider guide.
A geo ACL is a compliance and licensing control, not a security boundary. A VPN, a proxy or Tor places a client wherever it chooses.
Example
geo_locators:
default: # a geo ACL uses the locator named default unless it names another
provider: mmdb
mmdb:
path: /var/lib/GeoIP/GeoLite2-Country.mmdb
city:
provider: mmdb
mmdb:
path: /var/lib/GeoIP/GeoLite2-City.mmdb
corrections:
provider: geofeed
geofeed:
entries:
- 203.0.113.0/24,DE,DE-BE
files: [ /etc/trickster/private-relay.csv ]
geo_acls:
north-america:
allow: [ US, CA, MX ]
exempt: [ private ]
no-restricted-states:
geo_locator_name: city # subdivisions need a City-class database
deny: [ US-TX, US-UT ]
europe:
allow: [ continent:EU ]
backends:
video:
provider: reverseproxycache
origin_url: http://video.example.com
geo_acl_name: north-america
paths:
- path: /trailers/
geo_acl_name: none # clears the backend's geo ACL for this path
- path: /eu/
geo_acl_name: europe # replaces it for this path
orders:
provider: mysql
geo_acl_name: no-restricted-states # a refused session gets MySQL error 1130
example.full.yaml describes every option.
Locators
Every locator has a provider and one options block named for it. A block for another provider, or
an unknown provider, fails validation.
mmdb
Reads a MaxMind DB format file. That covers MaxMind GeoIP2 and GeoLite2, DB-IP, IPinfo, and IP2Location’s MMDB editions.
| Option | Default | Meaning |
|---|---|---|
path | required | the database file |
schema | auto | where records keep the location: auto, geoip2, ipinfo or custom |
fields | the record paths of country (required), continent and subdivision, for schema: custom | |
reload_interval | 5m | how often the file is checked for a replacement, besides change events; 10s to 24h |
max_age | 0 (never) | log a warning when the loaded file was built longer ago than this |
- Schemas.
geoip2readscountry.iso_code,continent.codeandsubdivisions[0].iso_code; MaxMind, DB-IP and IP2Location’s MMDB editions use it.ipinforeadscountry_codeandcontinent_code, falling back to thecountryandcontinentcodes of IPinfo’s older files.autochooses by the file’s database type and confirms against the file’s own records. A file thatautocannot read fails to load, and the error namesschema: custom. - Custom paths are lists of map keys and array indexes, such as
[subdivisions, 0, iso_code]. - Fields a file serves. A Country file serves countries and continents. A file serves subdivisions only when one of its records carries one, whatever its database type says; a load searches the whole file before deciding it serves none. A geo ACL with subdivision entries refuses a locator that serves none, when the configuration is applied.
- In memory. The file is read into memory, not mapped, so an updater that rewrites it in place can never fault the process. It costs the file’s size, and twice that while a replacement loads. GeoLite2 Country is about 8 MB and GeoLite2 City about 65 MB; use a Country file in tight pods.
- An IPv4-only file logs a warning, since it places no IPv6 client.
geofeed
Places addresses by RFC 8805 lines:
prefix,country,region,city,postal code. City and postal code are ignored.
| Option | Default | Meaning |
|---|---|---|
entries | inline lines | |
files | geofeed files, watched for changes | |
reload_interval | 5m | as for mmdb |
- At least one of
entriesandfilesis required. An inline entry that does not parse fails validation; a line in a file that does not parse is skipped, and a warning counts the skipped lines and shows the first. - The longest prefix holding an address wins. An inline entry beats a file’s for the same prefix, which makes inline entries a place for corrections.
- A line with every location field empty places no one: an address under it has no location, even when a shorter prefix would place it.
regionis a full ISO 3166-2 code, such asUS-AL, whose country must agree with the line’s. A country’s continent is derived from the country.- Feeds such as Apple’s iCloud Private Relay egress ranges are published in this format.
header
Reads a location header that a trusted upstream set, such as Cloudflare’s CF-IPCountry or
CloudFront’s CloudFront-Viewer-Country and CloudFront-Viewer-Country-Region.
| Option | Default | Meaning |
|---|---|---|
country | required | the header holding a country code |
subdivision | a header holding a subdivision code, with or without its country | |
continent | a header holding a continent code; otherwise the country’s continent | |
unknown_values | [XX, T1, ZZ, '-'] | values that mean no location |
- Headers are believed only from the listener’s
trusted_proxies. From any other peer they are not read, and the request has no location. A listener with notrusted_proxiesbelieves no header, and a load warning says so. - It judges HTTP requests only. A geo ACL whose locator is
headercannot gate a backend that a native protocol or stream listener serves; validation refuses it.
Replacing files
A replaced mmdb or geofeed file is loaded while serving, with no configuration reload. Write the
new file and rename it over the old one: a file written in place can be read half-written. A
replacement that fails to load leaves the last good file serving, counts a failed reload and logs
one warning. Replacements are refused when:
- an
mmdbfile does not open, or serves fewer location fields than the file it replaces; - a
geofeedfile changed while it was read, or lost every entry where the old file had some.
A locator whose options are unchanged is kept across configuration reloads, with its loaded file.
Geo ACLs
| Option | Default | Meaning |
|---|---|---|
geo_locator_name | default | the locator that places clients |
allow | the only locations allowed | |
deny | the locations denied | |
unknown | as an unlisted location | allow or deny: the verdict for a client with no location |
exempt | addresses and prefixes allowed with no lookup; private is every non-routable range | |
action | reject | reject, or count, which counts a denial and allows the client |
message | see below | what a refused client is told, in every protocol |
response | the HTTP refusal’s status, headers and body |
Entries
A geo ACL has an allow list or a deny list, never both. An entry is one of:
| Spelling | Matches | Example |
|---|---|---|
| two letters | a country (ISO 3166-1 alpha-2) | US |
| country, hyphen, 1 to 3 letters or digits | a first-level subdivision (ISO 3166-2) | US-TX, FR-75 |
continent: and two letters | a continent: AF AN AS EU NA OC SA | continent:EU |
- Entries ignore case.
- A list is a union:
allow: [US, CA-QC]is all of the United States and Quebec only. - Country codes are checked against the assigned ISO 3166-1 codes, plus
XKfor Kosovo, which the databases use. An unassigned code fails with a suggestion:UKis not assigned; useGB. - Continents take the
continent:prefix becauseAF,AS,NAandSAare also countries.
How a client is judged
- An
exemptaddress is allowed, with no lookup. - The locator places the address.
- A client with no location (the locator had no answer, the lookup failed, or the address did not
parse) takes the
unknownverdict. Unset, that is the verdict of an unlisted location: denied by anallowlist, allowed by adenylist. - Otherwise an
allowlist allows only what it lists, and adenylist denies only what it lists. - A denial under
action: countis counted and allowed. On HTTP the request goes on carrying anX-Trickster-Geo-Deniedheader naming the geo ACL, for the origin’s logs.
private stands for loopback, RFC 1918, link-local, carrier-grade NAT (100.64.0.0/10) and IPv6
unique local addresses. No database places them, so an allow list refuses them unless they are
exempt; a load warning says so for an allow list with neither exempt nor unknown.
Where geo ACLs apply
- Backends and paths. A path’s
geo_acl_namereplaces its backend’s, andgeo_acl_name: noneon a path clears it.noneis valid on paths only. Provider paths a configuration does not name take the backend’s geo ACL. - HTTP. The geo ACL runs in the route chain just outside the authenticator: a refused request never reaches a credential check, a mirror, the cache or the origin, and is logged and counted like any response. A backend’s geo ACL is on its own route too, which ALB pools and rules dispatch into, so a pool member’s geo ACL judges what an ALB sends it. Put the geo ACL on the backend a client reaches first: a fanout ALB treats a member’s refusal as that member’s response.
- Native protocols. A
mysql,postgres,clickhouseorflight-sqllistener judges each session by the geo ACL of the backend it maps to, before any credential is checked.flight-sqljudges every call.clickhousealso judges every query, since it sends each one through its backend’s HTTP route for/: the backend’s geo ACL, or that of a/path that names its own, can refuse a query on an open session after a reload or a replaced file. See Refusals for how that differs from a refused session. Otherwise a session admitted before a reload is not judged again. A backend that a native listener’s ALB routes sessions to may not name a geo ACL, since the session is judged by the ALB’s before it is routed; a path-level geo ACL on such a member judges its own HTTP requests. - Stream listeners. A
tcporudplistener judges each connection or flow by its backend’s geo ACL; atlslistener judges by the backend the server name selects. A refusal resets atcportlsconnection and drops audpflow’s datagrams; no message can be sent on an opaque stream. An ALB on a stream listener dials its members without entering their routes, so a backend or discovery template in its pool may not name a geo ACL: set it on the ALB, which judges the connection before a member is dialed. A path-level geo ACL on such a member is refused too, unless HTTP also serves the member, whose requests it then judges. IP access lists follow the same rule. - Not gated: the readiness path and the
mgmtandmetricslisteners. - Other gates on the same route judge independently, and any refusal ends the request.
The client address
- On HTTP it is the address the listener’s
trusted_proxiesresolved from the forwarding headers, or else the peer’s. A mirrored copy carries it, as does each query of a ClickHouse native session. - On native and stream listeners it is the peer, or the source a trusted PROXY protocol header named.
- A wrong
trusted_proxiesmakes every client the proxy. Behind a private address that is no location for everyone: anallowlist refuses every client, which is noticed at once, and adenylist allows every client, which is not. Watch the rate oftrickster_geo_locator_lookups_total{result="not_found"}; a listener withproxy_protocoland notrusted_proxieslogs a load warning, since any client can name its own source.
Refusals
Every protocol carries the geo ACL’s message, which defaults to
This resource is not available in your geographical area. It is one line of at most 255 bytes.
| Client | Shown |
|---|---|
| HTTP | 403 Forbidden and the message, as text/plain; charset=utf-8 |
mysql | ERROR 1130 (HY000): <message> |
psql | FATAL: <message>, SQLSTATE 28000 |
clickhouse-client | Code: 195. DB::Exception: <message> (IP_ADDRESS_NOT_ALLOWED) |
| Flight SQL | gRPC status PermissionDenied with the message |
tcp, tls | a reset |
udp | nothing; the datagrams are dropped |
A clickhouse query refused on a session that was already admitted fails like any other query:
Code: 62. DB::Exception: <message>, where the message is the HTTP refusal’s body, so a
response.body replaces it. The session stays open, and the refusal is counted with
plane="http". Only a new session gets exception 195.
HTTP refusals:
response.statusis from 400 to 599. No status code is defined for a geographic refusal, so 403 is the default, as CDNs answer.- 451 is for a legal demand (RFC 7725), not a
licensing or business restriction. A 451 should explain the demand in its body and carry a
Link: <uri>; rel="blocked-by"header; a load warning names a 451 without one. response.headersare merged over the defaultCache-Control: no-storeandContent-Type.response.bodyreplaces the message and its newline, for a JSON or HTML refusal.
Caches in front of Trickster
Trickster marks its own refusals no-store. A shared cache in front of Trickster may still store
an allowed client’s response and serve it to anyone. Serve a restricted path behind such a cache
with Cache-Control: private, or apply the same restriction at that cache.
Where location data comes from
| Source | Format | Terms | Updates |
|---|---|---|---|
| MaxMind GeoIP2 (paid), GeoLite2 (free) | MMDB | an account and license key to download; attribution; a GeoLite file is replaced within 30 days of a release | GeoLite twice a week |
| DB-IP Lite | MMDB | CC BY 4.0, with a link back | monthly |
| IPinfo Lite | MMDB | CC BY-SA 4.0, with attribution; a token to download | frequent |
| IP2Location LITE | MMDB editions for some packages | free with attribution; sign-up | monthly |
| CDN headers | a request header | the CDN’s | live |
| Geofeeds | RFC 8805 CSV | the publisher’s | the publisher’s |
- Licenses bind whoever distributes the file. Trickster ships none; a container image that bakes a GeoLite file in redistributes it.
- On Kubernetes, MaxMind’s
geoipupdateimage can run as an init container and a sidecar writing to a volume that Trickster reads; see Deploying on Kubernetes. - Accuracy falls with precision. MaxMind cites country accuracy above 99% with VPNs left out, state
or province accuracy of 55% to 80%, and city accuracy of 20% to 75%. A subdivision list refuses
some clients wrongly and admits others;
exemptand a geofeed of corrections are the remedies. - An address can move between networks daily.
max_ageand the build-time gauge show a stale file. - Whether a restriction is lawful where it applies is the operator’s question.
Metrics
| Metric | Labels |
|---|---|
trickster_geo_acl_decisions_total | geo_acl, plane (http, native, stream), verdict (allow, deny, count, exempt) |
trickster_geo_locator_lookups_total | geo_locator, result (found, not_found, error) |
trickster_geo_locator_reloads_total | geo_locator, result (success, error) |
trickster_geo_locator_build_timestamp_seconds | geo_locator; mmdb only |
No metric carries a country. Refusals log at debug level only, so a flood of refused clients is not
a flood of log lines; a MySQL listener’s failed logins log at debug too, and are counted by
trickster_mysql_connection_errors_total.
Performance
Measured on an Apple M3 Max, none allocating:
| What | Time |
|---|---|
a geofeed lookup, 10 or 300,000 entries | 11 to 15 ns |
a geo ACL check over a geofeed | about 25 ns |
| the HTTP gate, an allowed request | about 50 ns over the route without it |
an mmdb lookup, test database | about 105 ns |
An mmdb file is decoded once per record, then answered from a bounded cache of decoded records.
A route with no geo ACL is unchanged, and a native listener without one pays one atomic load per
session.
Kubernetes
A geo ACL is an operator control, like an authenticator, so the controller takes it only from operator-tier settings:
kubernetes.defaults.geo_acl_namegates every route the controller generates. Since stream routes take it too, it must name a geo ACL whose locator places addresses.- A GatewayClass’s parameters can name
geo_acl_namefor the class’s routes. A geo ACL whose locator reads headers serves the class’s HTTP routes, and refuses its stream routes rather than serving them ungated. - No Ingress annotation and no
TricksterCachePolicyfield names one, since whoever writes those in their own namespace could name a looser geo ACL, or none. - The geo ACL goes on the backend a route attaches, never on a pool member, an endpoint template or a mirror target, so each request is judged once.
See Kubernetes Ingress and Kubernetes Gateway API.
9.3 - Rate Limiting
Trickster counts requests, connections, sessions, or datagrams against named limiters and refuses what exceeds a limit. Limiters are defined once under rate_limiters and attached by rate_limiter_name. The exhaustive example is examples/conf/example.full.yaml.
rate_limiters:
per-client:
keys: [client_ip] # empty means one bucket for every event
ipv6_prefix: 64 # leading bits of an IPv6 client address; default 64
limit: 600 # events per window
window: 1m # 1s to 1h; default 1m
algorithm: sliding_window # the only algorithm
missing_key: exempt # exempt (default) | shared
action: reject # reject (default) | count | close
policy_headers: none # none (default) | ietf | legacy
max_keys: 100000 # buckets tracked; default 100000
max_keys_action: allow # allow (default) | reject
# unit: requests # leave unset when the limiter is used on more than one plane
response:
status: 429 # 400-599; default 429
body: "Too Many Requests\n"
headers:
Cache-Control: no-store
listeners:
default:
rate_limiter_name: per-client
backends:
api:
rate_limiter_name: per-client
paths:
- path: /healthz
rate_limiter_name: none # clears the backend limiter; the listener limiter remains
Where a limiter applies
rate_limiter_name attaches a limiter to a listener, a backend, or a path. none is reserved and is not a limiter name. On a listener or a backend it is an undefined name and the configuration fails to load. On a path it clears the backend limiter for that path.
Two decisions can apply to one HTTP request:
- The listener limiter is its own decision. A path name does not replace it, and
nonedoes not clear it. - The route decision is the path limiter when the path names one. That name replaces the backend limiter for that path. An empty path
rate_limiter_nameinherits the backend limiter.noneclears it, so the route makes no rate-limit decision.
When both are set, each judges the request. Naming the same limiter on the listener and on the route spends two events from that limiter’s buckets. A listener allowance can be spent by a request that a later route access list denies. A route access-list denial happens before the route limiter, so it does not spend the route allowance.
The same limiter name is one set of buckets everywhere it is attached, for as long as its keys, IPv6 prefix, window, limit, and max_keys stay the same. missing_key, max_keys_action, action, the HTTP response, and policy_headers can change on reload while those buckets stay.
unit says what an event is: requests, connections, sessions, or datagrams. Leave it unset when the limiter is attached on more than one plane. An unset unit uses requests on HTTP, connections on tcp and tls, and sessions on udp, and those events share one quota. A limiter attached on more than one plane with unit set fails to load. close is valid only on stream listeners. A native listener (mysql, postgres, clickhouse, flight-sql) cannot name a limiter. A moq listener can: it counts sessions, by client_ip or in one bucket, and a moq backend’s limiter counts its sessions’ MoQT requests; see Media over QUIC.
A cache hit is still an event: the limiter judges the request before the cache.
The estimate
algorithm is sliding_window. Empty means that. Each bucket keeps the count for the previous window and the count for the current one. At a moment frac of the way through the window, the estimate is the previous count times (1 - frac), plus the current count. An event is allowed when the estimate plus one is at most limit. The event that would pass the limit is not added, except in count mode, which records it and still lets it through.
This is an estimate, not a log of every event. A burst at the end of one window and the start of the next can weigh as less than the number of events, so a client can be allowed slightly past what an exact sliding log would allow.
A bucket that is not read for a little more than two windows is dropped. The table may drop it up to a second early, or up to a sixteenth of that idle time when that is shorter.
Keys
keys is up to five attributes, in order. Together they are one bucket. Empty keys is one bucket for every event.
| Key | Read on |
|---|---|
client_ip | HTTP, HTTP/3, tcp, tls, and udp. The client address after trusted_proxies, never the port. IPv6 keeps the leading ipv6_prefix bits (default 64), so a client that rotates its privacy address stays on one key. |
host | HTTP. The request host, without its port, without regard to case. |
header:<name> | HTTP. The first value of that request header. |
cookie:<name> | HTTP. The value of that request cookie. |
query:<name> | HTTP. The raw value of that query parameter. |
query | HTTP. The query string as the client sent it. |
method | HTTP. The request method. |
path | HTTP. The request path, without the query string, after path normalization. |
sni | tls only. The server name the client offered. It is known after the handshake, so the connection is counted then. |
proxy_tlv:<type> | tcp and tls listeners that accept a PROXY protocol header. <type> is the TLV type, as a decimal or hex integer. |
user | The authenticated name on a native session. No listener that can carry a limiter reads it, so a limiter keyed on user cannot be attached. |
A key the attachment cannot read fails to load. A listener limiter can use only keys that need nothing but the connection or the request itself.
missing_key is exempt or shared. Empty means exempt. exempt lets an event through without counting it when a key has no value, such as a header the request did not send. shared puts every such event in one extra bucket.
client_ip is the address Trickster already resolved. Clients behind one NAT share a key. A listener behind a proxy that is missing from trusted_proxies sees that proxy as the client, so every request shares the proxy’s key. On a stream listener that reads a PROXY header and lists no trusted_proxies, a client_ip limiter still loads and warns: an empty trusted-proxy list believes every peer’s header.
Two different identities that hash to the same 64-bit key share a bucket, and therefore one quota. A collision spends the allowance faster. It does not open a second one.
Capacity
max_keys is how many buckets the limiter tracks. Empty means 100000. The table is split into as many as 64 shards, and each shard rounds its share up, so the table can hold up to 63 buckets past max_keys. A new key can also be refused while the table is still under max_keys, once the shard it lands in is full.
max_keys_action is what a new key gets when it does not fit. Empty means allow: the event proceeds and is not counted. reject treats it as over the limit. Both record the full result.
Each tracked key is a 40-byte counter plus the table entry that holds it. On a 64-bit process that comes to about 110 bytes of heap per key, including the map slot.
HTTP
On an HTTP or HTTP/3 listener the limiter sits after the listener IP access list and path normalization, and before the router. The route limiter sits after the route’s IP and geo access lists and before the authenticator. The readiness path, after normalization, is exempt on every listener except the metrics listener. A health check is not exempt. An ACME http-01 challenge is answered before any limiter.
action is reject or count on HTTP. Empty means reject. close fails to load when the limiter is attached to anything HTTP serves.
reject writes the configured response. The status is 400–599 and defaults to 429. The body defaults to Too Many Requests\n and is omitted for HEAD. Cache-Control: no-store is always set. Retry-After is the wait until a new event could pass, in whole seconds, rounded up, and at most about two windows. A full table that rejects uses the window length. A response header cannot replace Retry-After, the quota headers below, Content-Length, Transfer-Encoding, or a hop-by-hop field.
policy_headers adds quota fields to responses the limiter allowed as well as to refusals:
noneadds none.ietfaddsRateLimit-Policy("<name>";q=<limit>;w=<window seconds>) andRateLimit("<name>";r=<remaining>;t=<seconds until the window ages out>). Each limiter adds its own pair.legacysetsX-RateLimit-Limit,X-RateLimit-Remaining, andX-RateLimit-Resetwhen those fields are still absent, so the route limiter’s values win over the listener’s.
count forwards the event after recording it, including one that is over the limit and a new key a rejecting full table would have stopped. The limiter appends its name to the request header X-Trickster-Rate-Limited. A listener that has a limiter, or that serves a backend or path which names one, removes any value the client sent before judging.
A refusal, a counted event, and a full table write one debug log line, rate limit decision, with the limiter name and the result. Allows are not logged.
Streams
A tcp or tls listener counts a connection unless unit says otherwise. A udp listener counts a session unless unit is datagrams.
connectionsare counted when the connection is accepted, fromclient_ipandproxy_tlv. A key ofsni, or an IP or geo access list that still judges after the flow is known, waits until then, so that list can refuse the connection before it is counted.sniis always counted after the handshake.sessionsare counted once, when the UDP flow opens. A refused session is held until a new event could pass, and that wait does not count again. A full table that rejects holds for the window.datagramsare counted on every datagram, including the first.
reject resets a TCP or TLS connection and drops a UDP datagram. close finishes a TCP or TLS connection and drops a UDP datagram. count allows the event after the same single charge. A connection that ends before the stage that counts it is not counted. That includes a TLS name that routes nowhere.
A ClickHouse query that arrives over HTTP uses the HTTP route limiter. That is separate from a limiter on a native session, which a native listener does not enforce.
Reload and restart
A reload that keeps the limiter’s name, keys, IPv6 prefix, window, limit, and max_keys keeps its buckets. Changing any of those starts that limiter empty. Removing the limiter drops its buckets after the new configuration is in place. A reload that fails to apply leaves the previous buckets alone. Restarting the process starts every limiter empty. The clock is monotonic for the life of the process, so a step of the wall clock does not roll a window backward or forward.
Metrics
trickster_ratelimit_decisions_total counts each decision. The labels are limiter, plane (http, stream or moq), and result:
result | Meaning |
|---|---|
allowed | counted, and at or under the limit |
limited | over the limit, refused, and not counted |
counted | over the limit in count mode, recorded, and still forwarded |
exempt | not counted: a missing key under exempt, or the readiness path |
full | a new key that did not fit. allow forwards it without counting; reject refuses it |
trickster_ratelimit_keys is the number of buckets that limiter holds at scrape time, labeled by limiter. See metrics.md.
Kubernetes
kubernetes.defaults.rate_limiter_name puts one limiter on every HTTP and gRPC route the controller generates. A stream route does not take it. A GatewayClass parameter of the same name overrides it for that class. The limiter must use HTTP keys, a unit of requests or unset, and an action other than close. There is no Ingress annotation and no TricksterCachePolicy field, and a generated route cannot set the name to none. The name is not copied onto pool members, endpoint templates, mirror targets, or generated listeners. See kubernetes-ingress.md and kubernetes-gateway.md.
10 - Kubernetes
10.1 - Deploying on Kubernetes
deploy/kube carries two raw-YAML deployments. configmap.yaml,
deployment.yaml and service.yaml run Trickster as a caching proxy with
the exhaustive example configuration; the gateway-*.yaml manifests run it
as the Kubernetes Gateway API and Ingress controller described in
kubernetes-gateway.md and
kubernetes-ingress.md. The Helm chart at
https://github.com/trickstercache/helm-charts is built from these
manifests, so what they do is what the chart does with its values applied.
What is in deploy/kube
| File | What it is |
|---|---|
gateway-namespace.yaml | the trickster namespace every controller manifest names |
gateway-rbac.yaml | ServiceAccount, ClusterRole and binding, and the leader election Role, per kubernetes-rbac.md |
gateway-classes.yaml | the trickster GatewayClass and IngressClass |
gateway-configmap.yaml | the controller’s configuration: Ingress listeners, probe listener, the kubernetes section |
gateway-deployment.yaml | two replicas, readiness and liveness probes, drain settings, a restricted security context |
gateway-service.yaml | the LoadBalancer whose address is published into status, and a ClusterIP for metrics |
gateway-pdb.yaml | keeps one replica through voluntary disruptions |
gateway-hpa.yaml | optional CPU autoscaling |
crds/trickstercachepolicies.yaml | the TricksterCachePolicy resource (kubernetes-cache-policy.md) |
configmap.yaml, deployment.yaml, service.yaml | the plain caching proxy |
graphite-*.yaml, mysql-*.yaml | demonstrations layered on the plain proxy |
Installing the controller
The Gateway API is an add-on, installed from its own release. Install the
version the controller is built against (sigs.k8s.io/gateway-api in
go.mod); the standard channel carries the v1 kinds, and the
experimental channel additionally carries HTTPRoute retries and the
TCPRoute, TLSRoute and UDPRoute kinds. The TricksterCachePolicy definition
ships in deploy/kube/crds. Both go in before the controller starts, since
it probes for them once at startup:
make -C deploy/kube install-gateway-crds # standard channel
make -C deploy/kube install-gateway-crds GATEWAY_API_CHANNEL=experimental
A cluster that will never serve the Gateway API skips the first apply and
the GatewayClass in gateway-classes.yaml; the controller then serves
Ingress objects alone. Then the controller itself, in this order:
make -C deploy/kube bootstrap-trickster-gateway
which applies the namespace, RBAC, classes, ConfigMap, Deployment, Service and PodDisruptionBudget and waits for the rollout. A pod is ready only once its controller has translated the cluster’s routing objects and the data plane serves them, so the rollout completing means the controller is serving. Verify what it claimed:
kubectl get gatewayclass trickster # ACCEPTED True
kubectl get ingressclass trickster
kubectl -n trickster get service trickster-gateway # the published address
kubectl get gateway,httproute,ingress -A # PROGRAMMED, the address, and Accepted parents
A Gateway of the class opens the ports it declares; an Ingress naming the class is served on the listeners the ConfigMap configures:
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: edge
namespace: shop
spec:
gatewayClassName: trickster
listeners:
- name: http
port: 80
protocol: HTTP
Configuration
gateway-configmap.yaml is the primary configuration file, mounted at
/etc/trickster/trickster.yaml. It carries what the cluster’s routing
objects cannot: the listeners claimed Ingresses are served on, the listener
the probes use, logging, the kubernetes section, and the operator tier
(kubernetes.defaults) no route may change. Everything the controller
generates from Gateways, routes and Ingresses is merged after it, through
the same validation, and may only add objects: it can never change main,
logging, mgmt or a listener the file defines. Every option is described
in configmap.yaml, which is the example configuration wrapped in a
ConfigMap.
Fragments in conf.d
A second ConfigMap, trickster-gateway-conf-d, is mounted at
/etc/trickster/conf.d and is optional. Each of its keys becomes a file
there, and every .yaml, .yml or .conf file in the directory is merged
into the primary file in name order, as described under
Multiple Configuration Files. It is
where the operator tier is provisioned without touching the primary file:
the caches, negative caches, tracers, request rewriters, authenticators and
geo ACLs that kubernetes.defaults, a GatewayClass’s parameters, a
TricksterCachePolicy or an Ingress annotation may then select by name.
apiVersion: v1
kind: ConfigMap
metadata:
name: trickster-gateway-conf-d
namespace: trickster
data:
caches.yaml: |
caches:
objects:
provider: memory
index:
max_size_bytes: 536870912
authenticators.yaml: |
authenticators:
gateway-auth:
provider: basic
users:
api: ${API_PASSWORD_HASH}
A chart renders its values into exactly these fragments; a hand-run deployment edits them in place.
How a change reaches the pod
Two patterns, which may be combined:
In-place reload. The manifests set mgmt.auto_reload_interval, so the
running process polls its files and reloads when one changes, through the
same validation and graceful path a SIGHUP uses. The kubelet refreshes a
mounted ConfigMap within about a minute of an edit (its sync period), by an
atomic symlink swap the poll sees and filesystem notifications do not, which
is why polling is the mechanism. A reload that fails validation leaves the
previous configuration serving, logged as a failed reload; caches and
listeners whose settings did not change are kept, and a listener whose
settings did change is restarted with a drain. The volumes are whole-volume
mounts and never subPath mounts: a subPath mount is a copy the kubelet
does not refresh, so the poll would see nothing.
Rollout on a checksum. A chart that wants every configuration change to
pass through a rollout, with the Deployment’s history as its rollback, puts
a hash of the rendered ConfigMap in a pod template annotation
(checksum/config in gateway-deployment.yaml); a changed hash rolls the
pods. Readiness holds each new pod out of the Service until its controller
has programmed the routes, and the old pod drains only after it has been
removed, so the rollout is invisible to clients. This is the pattern for a
change the in-place reload cannot make, and for a Secret consumed through
the environment.
Secrets
Credentials never go in a ConfigMap. The configuration expands ${VARIABLE}
in its credential-bearing fields, listed under
Configuring Secrets:
authenticator users, a Redis password, a user router’s to_credential, and
header values in path, CORS, health check and discovery blocks. Mount a
Secret into the environment
(envFrom.secretRef in gateway-deployment.yaml) and reference its keys.
A container’s environment is fixed at start, so a rotated Secret reaches
the pod through a rollout, not a reload. TLS certificates for HTTPS Gateway
listeners and Ingress TLS sections are read from the kubernetes.io/tls
Secrets the objects reference and pushed into the listeners at runtime,
rotated without a reload; no certificate is mounted.
Location databases
A geo ACL reads a location database that Trickster neither
ships nor downloads; the operator mounts it. MaxMind’s geoipupdate image
fetches GeoLite2 or GeoIP2 files with an account ID and license key. Run it
as an init container, so the pod starts with a file, and as a sidecar that
keeps it current, both writing to a volume that Trickster reads:
spec:
volumes:
- name: geoip
emptyDir: {}
initContainers:
- name: geoip-init
image: ghcr.io/maxmind/geoipupdate # pin a release
env:
- { name: GEOIPUPDATE_EDITION_IDS, value: GeoLite2-Country }
- { name: GEOIPUPDATE_ACCOUNT_ID, valueFrom: { secretKeyRef: { name: geoip, key: account-id } } }
- { name: GEOIPUPDATE_LICENSE_KEY, valueFrom: { secretKeyRef: { name: geoip, key: license-key } } }
volumeMounts: [ { name: geoip, mountPath: /usr/share/GeoIP } ]
containers:
- name: geoip-update
image: ghcr.io/maxmind/geoipupdate
env:
- { name: GEOIPUPDATE_EDITION_IDS, value: GeoLite2-Country }
- { name: GEOIPUPDATE_FREQUENCY, value: "72" } # hours between updates; keeps it running
- { name: GEOIPUPDATE_ACCOUNT_ID, valueFrom: { secretKeyRef: { name: geoip, key: account-id } } }
- { name: GEOIPUPDATE_LICENSE_KEY, valueFrom: { secretKeyRef: { name: geoip, key: license-key } } }
volumeMounts: [ { name: geoip, mountPath: /usr/share/GeoIP } ]
- name: trickster
volumeMounts: [ { name: geoip, mountPath: /var/lib/GeoIP, readOnly: true } ]
geoipupdate writes a new file and renames it over the old, which a geo
locator swaps in while serving. A City file takes about 65 MB of memory per
locator, and twice that while a replacement loads; size the pod’s limit for
it, or use a Country file. A Helm chart needs values for the volume and the two containers.
Ports and addresses
A Gateway listener binds the port it declares, in the pod, and
gateway-service.yaml forwards by port number, so a Gateway declaring 80 is
reached at the load balancer’s port 80. The pod runs as an unprivileged user
and binds 80 and 443 through the net.ipv4.ip_unprivileged_port_start
sysctl the Deployment sets, which every current Kubernetes release treats as
safe. The alternative, adding NET_BIND_SERVICE to the container, does not
reach a non-root process: the capability is dropped at execve unless the
binary carries file capabilities, which the image does not set. A Gateway
declaring some other port needs a matching Service entry.
The Service also forwards UDP 443, where an HTTPS listener on 443 serves
HTTP/3 and WebTransport and a trickstercache.org/moq listener on 443 serves
MoQ (MoQ relays). A LoadBalancer mixing
TCP and UDP ports needs Kubernetes 1.26 or later and a load balancer that
supports it; where yours does not, remove the https-udp entry, or move it
to a Service of its own. A Gateway listener on another UDP port needs an
entry of its own with protocol: UDP.
A port is bound once, so a port belongs either to a Gateway or to a listener
configured in the ConfigMap. The manifests give 80 and 443 to Gateways and
serve Ingresses on 8080 and 8443, which the Service also forwards. To serve
Ingresses on 80 and 443 instead, move the web and websecure listeners to
those ports in the ConfigMap and have Gateways declare others; the Service
forwards by number and needs no change.
The addresses published into Gateway and Ingress status are those of the
Service named by kubernetes.published_service, a LoadBalancer here. On a
cluster with no load balancer implementation, give the Service externalIPs
or make it a NodePort, or install one such as MetalLB; the status address
follows whatever the Service is assigned.
Every Gateway of the class is served by every replica at the one published
address, and two Gateways declaring one port merge on it as described under
Listeners. A deployment that needs
Gateways at distinct addresses runs one controller Deployment per
GatewayClass: a copy of the manifests with its own namespace or names, its
own Service, and a distinct gateway_class_controller_name (and
ingress_class) that only that class names.
Scaling and availability
The Deployment starts two replicas spread across nodes, with a
PodDisruptionBudget keeping one through drains. Every replica watches the
cluster, translates and serves every route; one, elected over the Lease
kubernetes.leader_election names, writes status and Events, so adding
replicas adds data-plane capacity and nothing else, which is what the
optional HorizontalPodAutoscaler scales on. The resource requests are
starting points.
A rolling update surges one pod in before one goes (maxSurge: 1,
maxUnavailable: 0), and terminationGracePeriodSeconds covers the
preStop sleep (which lets the endpoint removal propagate before SIGTERM)
plus mgmt.shutdown_drain_timeout, with margin; see
Graceful shutdown.
RBAC
gateway-rbac.yaml grants exactly the verbs
kubernetes-rbac.md lists, cluster-wide, because the
shipped configuration watches every namespace. A controller narrowed with
kubernetes.watch_namespaces binds the namespaced kinds through a Role in
each watched namespace and keeps only gatewayclasses, ingressclasses and
namespaces cluster-wide; read_only: true drops every write. The
endpoint routing mode adds endpointslices, left commented in the file.
The image
trickstercache/trickster (also ghcr.io/trickstercache/trickster) is a
statically linked binary built with CGO_ENABLED=0 on a distroless static
base: no shell, no package manager, no libc. It runs as nobody (65534) and
the manifests pin that with runAsNonRoot and a read-only root filesystem.
The image carries the CA bundle, the license notices of every linked
dependency under /licenses, and the example configuration at
/etc/trickster/trickster.yaml, which the mounted ConfigMap replaces.
Images are signed; the README shows the cosign verify invocation. The
Kubernetes client and Gateway API libraries the controller is built on are a
fixed part of the binary whether or not the kubernetes section is
configured, and a Trickster without that section holds no watches and opens
no API connection.
The plain caching proxy
configmap.yaml is examples/conf/example.full.yaml wrapped in a
ConfigMap, generated by make kube-configmap and checked in CI, so the two
never drift; edit the example, not the ConfigMap. deployment.yaml mounts it
and service.yaml exposes the proxy and metrics ports.
graphite-deployment-patch.yaml and mysql-deployment-patch.yaml layer an
in-cluster Graphite and a credential-bearing MySQL configuration on top of
it; each file describes how it is applied.
10.2 - Kubernetes Gateway API
Trickster serves the gateway.networking.k8s.io/v1 objects it claims —
GatewayClass, Gateway, HTTPRoute and ReferenceGrant — translating them into
its own configuration and reloading onto it. Enable the controller with the
top-level kubernetes section (configuring.md) and grant
it the permissions in kubernetes-rbac.md. Ingress v1
is served alongside; see kubernetes-ingress.md.
Caching behavior — a cache, TTLs, a time series provider, the cache key — is
attached to Gateways, routes and Services with the TricksterCachePolicy
resource; see kubernetes-cache-policy.md.
The Gateway API is optional at runtime. Its CRDs are an add-on that most clusters do not have, so the controller probes for the group at startup and, where it is absent, watches none of its kinds and serves Ingress alone.
What is claimed
A GatewayClass belongs to this controller when its spec.controllerName
equals kubernetes.gateway_class_controller_name. A Gateway belongs to it
when its spec.gatewayClassName names a claimed class, and a route of any
served kind (HTTPRoute, GRPCRoute, TCPRoute, TLSRoute, UDPRoute) attaches
through a parentRef naming a claimed Gateway. Anything else is ignored
entirely: not translated, not counted, and never statused. A parentRef
naming a Gateway this controller does not claim is another controller’s
business and draws no complaint.
GatewayClass parameters
A GatewayClass may carry a parametersRef to a ConfigMap. Its data keys
are the kubernetes.defaults fields, spelled the same way, and override
those defaults for every route served through a Gateway of the class:
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: trickster-cached
spec:
controllerName: trickstercache.org/gateway-controller
parametersRef:
group: ""
kind: ConfigMap
name: cached-gateway-params
namespace: trickster
---
apiVersion: v1
kind: ConfigMap
metadata:
name: cached-gateway-params
namespace: trickster
data:
cache_name: objects
negative_cache_name: api-errors
timeout: 45s
authenticator_name: gateway-auth
ip_acl_name: office
rate_limiter_name: per-client
| Key | Value |
|---|---|
routing_mode | service or endpoint |
cache_name, negative_cache_name | a configured cache or negative cache |
tracing_name, req_rewriter_name, authenticator_name | a configured tracer, request rewriter or authenticator |
geo_acl_name | a configured geo ACL, which gates the backend each of the class’s routes attaches, and none of its pool members, templates or mirror targets. A stream route needs one whose locator places addresses; under a geo ACL that reads headers, the class’s stream routes are refused rather than served ungated |
ip_acl_name | a configured access list with source: client_ip and action: reject. When set, it overrides kubernetes.defaults.ip_acl_name for this class’s route backends. See ip-acl.md |
rate_limiter_name | a configured rate limiter with HTTP keys, unit requests or unset, and an action other than close. When set, it overrides kubernetes.defaults.rate_limiter_name for this class’s HTTP and gRPC route backends. A stream route does not take it. See rate-limiting.md |
timeout | a duration with a unit, such as 30s |
health_mode | probe or provider, for generated discovery-backed ALBs |
load_balancing | rr, p2c, lc, lt or hrw: how traffic is spread across a Service’s endpoints in the endpoint routing mode |
load_balancing_key | what hrw keeps on one endpoint: client_ip, or sni on a TLSRoute; an HTTP route may also use host, header:<name>, cookie:<name> or query:<name> |
sticky | cookie, header, table or none: keeps a client on the endpoint it first reached, on an HTTP route in the endpoint routing mode, and in a table on a stream route; see Session persistence |
sticky_key | what table mode keeps a client’s endpoint by, from the load_balancing_key vocabulary; client_ip unless set, or when the route’s listener cannot read it |
sticky_ttl, sticky_idle | a duration of at least 1s: a session ends that long after it began (1h unless set), or once unused that long. A route’s sessionPersistence does not use them |
sticky_secret | the name of a Secret, in the ConfigMap’s namespace and labeled trickstercache.org/sticky-key, whose key entry keys every session token the class’s ALBs issue; see The session key |
http3 | true or false: whether the class’s HTTPS listeners also serve HTTP/3, on the same port number over UDP, which WebTransport sessions need; kubernetes.defaults.http3 when unset, which is true unless set. Gateways of several classes sharing a port take the oldest Gateway’s setting, and the others are told by Event |
A non-empty ip_acl_name is copied onto the generated HTTP, gRPC, stream, and MoQ route backends, in both the service and endpoint routing modes. It is not copied onto pool members, endpoint templates, or mirrors. An empty value leaves kubernetes.defaults.ip_acl_name in place. No route or policy field sets the name to none. A name that is not configured, or a list that is not client_ip and reject, cannot be honored: the class is not served, as for any other parameter that names something the configuration does not define.
A non-empty rate_limiter_name is copied onto the generated HTTP and gRPC route backends, in both routing modes. It is not copied onto stream route backends, pool members, endpoint templates, mirrors, or generated listeners. An empty value leaves kubernetes.defaults.rate_limiter_name in place. No route or policy field sets the name to none. A name that is not configured, or a limiter a generated HTTP route cannot use, cannot be honored: that class is not served. Another class keeps the parameters it can honor.
Unlike an Ingress annotation, a GatewayClass may set the operator-tier names,
because a GatewayClass is cluster-scoped infrastructure and whoever can write
its ConfigMap decides the defaults for the class. A TricksterCachePolicy on
a Gateway, an HTTPRoute, one of its rules, or a backend’s Service is written
over the class’s parameters, most specific last; see
kubernetes-cache-policy.md.
Because those parameters carry operator controls, a class whose parameters
cannot be honored is not served: none of its Gateways open a port and
none of the routes attached to them are emitted, until the parameters are
fixed. That covers a parametersRef that cannot be read (wrong kind, no
namespace, ConfigMap absent or outside the watched namespaces) and any key
that is unknown, does not parse, or names something the configuration does
not define. Every affected Gateway is reported with the reason. A ConfigMap
with no keys is a class with no overrides, and is served.
Listeners
Each Gateway listener becomes a Trickster listener bound to the declared
port. HTTP opens a plaintext port; HTTPS opens a TLS port with
tls_runtime_certs, whose certificates arrive from the referenced Secrets at
runtime, and serves HTTP/3 on the same port number over UDP
unless the class’s http3 parameter turns it off; TCP, TLS and UDP open a
stream listener of the matching protocol,
which relays what it receives without reading it; and the implementation-specific
trickstercache.org/moq opens a MoQ listener (see MoQ relays).
Listeners on one port — within a Gateway or across Gateways — merge onto one
Trickster listener, because a port can be bound once; a UDP listener binds
in its own port space, so it may share a port number with a TCP, TLS,
HTTP or HTTPS listener. A trickstercache.org/moq listener binds UDP too,
so it may not share a port with a UDP listener, while it shares one socket
with the HTTP/3 side of an HTTPS listener on the same number, each QUIC
connection dispatched by the protocol it negotiates. Two listeners on one port must serve one
protocol; where they do not, the listener of the older Gateway (then the
earlier listener) keeps the port and the other is reported and not served.
Listeners of different Gateways on one port may share a hostname, or have
none: each admits its own routes and the port serves the union, with
overlapping routes resolved by the ordinary precedence (the older route
wins). On HTTPS the merged listeners’ certificates share one store, which
answers for a name with the first certificate carrying it, so two listeners
on one port, of one Gateway or two, may not bring different certificates
carrying one name (a wildcard being a name of its own); the older Gateway’s
listener, then the earlier one, keeps the name and the other is reported
with HostnameConflict and not served until a rotation ends the overlap. A
Secret rotated to unusable material withdraws nothing, so on a port whose
store holds a certificate under it that certificate keeps its names until a
usable rotation replaces it, while on a port holding nothing under it the
Secret claims nothing.
Two Gateways naming one hostname on one port must also name the same
certificate. Hostless HTTPS listeners merge whatever their certificates,
each selected by the names it carries, so long as those names are disjoint.
A Gateway may not repeat a hostname on a port within itself, which the API
refuses before it reaches the controller.
hostname may be precise or a single leading wildcard label
(*.example.com). It restricts the routes the listener admits, as described
below. A TCP or UDP listener cannot carry one, since nothing in the
stream names a host; a listener that declares one is refused with
UnsupportedValue. A TLS listener’s hostname is matched against the server
name a client offers, and a trickstercache.org/moq listener’s against the
authority a session names.
On an HTTPS or trickstercache.org/moq listener tls.mode must be Terminate (the default). Every
certificateRefs entry must name a kubernetes.io/tls Secret, in the
Gateway’s namespace or in one whose ReferenceGrant permits it. A listener
that cannot terminate TLS at all — no tls block, no certificateRefs, or
Passthrough — is not served. A reference that merely does not resolve yet
is reported and the listener still opens, so that creating the Secret later
heals it without any further change. Certificates never travel in
configuration: they are supplied to the running listener and re-supplied when
the Secret changes, so a rotation costs no reload. See
tls.md for how a listener holds them and for
the certificate inventory that lists them.
On a TLS listener tls.mode must be Passthrough: the listener relays the
client’s handshake to the backend the server name selects and terminates
nothing, so it holds no certificate and certificateRefs are ignored. The
mode defaults to Terminate, which is refused with UnsupportedValue
rather than assumed; a TLS listener that would terminate and forward plain
TCP is not served in this release.
allowedRoutes.namespaces.from is honored: Same (the default), All, and
Selector against the labels of the route’s Namespace. allowedRoutes.kinds
may list the kinds the listener’s protocol carries: HTTPRoute and
GRPCRoute on HTTP and HTTPS, TCPRoute on TCP, TLSRoute on TLS
and UDPRoute on UDP and trickstercache.org/moq. Any other kind is reported as unsupported on the
listener, and a listener admitting no supported kind admits no route.
Deviation: allowedRoutes decides which routes attach to a listener; it
does not fence requests. Listeners on one port share one router, which
resolves a request by hostname across every route served on the port, so a
request for shop.example.com whose path no route on the
shop.example.com listener claims may be answered by a route attached to a
*.example.com listener on the same port — including a route from a
namespace the precise listener would not have admitted. The Gateway API’s
GatewayHTTPListenerIsolation feature, under which the most specific
listener alone answers its hostname, is not supported.
A listener no route has attached to is bound all the same: a generated
backend of its own answers 404 for whatever arrives, so the port is open
and an HTTPS listener already holds its certificates before its first
route, and the listener is Programmed as soon as it is declared.
Not supported in this release, reported and ignored: spec.addresses
(addresses come from kubernetes.published_service), spec.tls, and
spec.allowedListeners (ListenerSets). A Gateway naming
spec.infrastructure.parametersRef is refused with InvalidParameters and
not served, since nothing reads Gateway parameters and serving it without
them would misrepresent what was asked.
Binding a port below 1024 inside a container requires running as root,
granting NET_BIND_SERVICE, or the net.ipv4.ip_unprivileged_port_start
sysctl the deployment in deploy/kube sets
(kubernetes-deploy.md); a
Gateway declaring port 80 behind a Service that maps it from a high
container port is not yet expressible, so declare the port the pod may bind.
Route attachment and hostnames
A parentRef selects a claimed Gateway, optionally narrowed by
sectionName or port. The route attaches to each selected listener whose
allowedRoutes admit it and whose hostname intersects the route’s:
- a listener with no hostname admits whatever the route names, or every hostname when the route names none;
- a route naming no hostname takes the listener’s;
- otherwise each route hostname is kept when it equals the listener’s or
falls within its wildcard (
*.example.comadmitsshop.example.comanddeep.shop.example.com), and a wildcard route hostname is narrowed to a precise listener it covers.
When both sides name hostnames and none agree, the route is not accepted on that listener, and a parentRef none of whose listeners accept the route is reported with the reason.
The route is then served once per port and hostname it was accepted on. Listeners that merged onto one port serve it for the union of the hostnames they admitted.
A wildcard hostname spans any number of labels, as the Gateway API defines:
a route served on *.example.com answers shop.example.com and
deep.shop.example.com, but not example.com. See
paths.md for the order hosts are tried in.
Matches and precedence
| HTTPRoute | Trickster |
|---|---|
path.type: Exact | an exact path match |
path.type: PathPrefix | a whole-segment prefix match |
path.type: RegularExpression | a regular expression match, anchored at the start |
no path | PathPrefix / |
method, headers, queryParams | conditions on the matched path |
backendRefs with weight | a weighted round-robin pool over the references |
A Kubernetes PathPrefix matches whole path elements: /foo matches /foo
and /foo/bar but never /foobar, and a trailing slash means nothing.
Requests are resolved by host first — the exact hostname, then wildcards covering it from the nearest label boundary outward, then hostless routes — then by path, exact before longest prefix before regular expression, then by method, and finally by the match’s header and query parameter conditions, most specific first. The Gateway API’s own precedence decides among matches that meet on one path and method: a match naming a method first, then more header matches, then more query parameter matches, then the older route.
- A header or query value is compared whole, and a
RegularExpressionone must match the whole value. The field must be present to match at all: a pattern the empty string satisfies, such as.*, matches a field that is present and empty but not one that is absent. A match naming one header or query parameter twice uses the first condition, whatever the case. - A method that no match claims on a path some match did claim reaches the
covering match on the same host where one exists, so a
POSTto a path that only namesGETstill reaches the prefix behind it, and answers 404 otherwise. That fill does not cross into a less specific host tier. - Two matches testing the same path, method and predicates are a duplicate:
the older route keeps it (ties on creation time break on namespace then
name) and the other is reported. An
Exactmatch outranks aPathPrefixon the same path and a match naming a method outranks one that does not, whatever their ages, which is the order the Gateway API defines.
Backend references
Each backendRefs entry must be a Service (the default kind) with a port
that exists on it. A reference into another namespace needs a ReferenceGrant
in that namespace permitting HTTPRoute from the route’s namespace to
Service, optionally by name. weight defaults to 1; a weight of 0 sends no
traffic and drops the reference.
The request reaches the Service with the Host header the client sent, and
cached objects are keyed by
it, so two hosts one backend serves never share an object; a URLRewrite
hostname replaces it, and then keys nothing. A Service port whose
appProtocol is kubernetes.io/h2c is spoken to in cleartext HTTP/2 by
prior knowledge. A headless Service (clusterIP: None) resolves to its
pods: under service routing it is dialed on the port’s numeric
targetPort, and one naming its targetPort by name, which only its pods
resolve, is refused there with UnsupportedValue; under endpoint routing
the pods are discovered through the EndpointSlices, which name the port, so
both shapes are served. The routing mode that decides is the one the route’s
policies leave in force (class parameters, route, rule, then the Service’s
own policy).
A reference that cannot be resolved — unsupported kind, no permitting grant,
Service or port absent — keeps its slot and its weight, and its share of the
traffic answers with a fixed 500, which is what the Gateway API requires.
A rule with no usable reference answers 500 outright. Because Services are
watched, creating the Service later heals the route. Several references
become a weighted round-robin pool, apportioned exactly by integer weight.
BackendTLSPolicy
A BackendTLSPolicy whose targetRefs name a Service, optionally one named
port of it through sectionName, makes every connection to that Service
TLS, verified as the policy says:
apiVersion: gateway.networking.k8s.io/v1
kind: BackendTLSPolicy
metadata:
name: web-tls
namespace: shop
spec:
targetRefs:
- group: ""
kind: Service
name: web-svc
sectionName: https
validation:
caCertificateRefs:
- group: ""
kind: ConfigMap
name: web-ca
hostname: web.shop.internal
validation.hostname is sent as the SNI and is the name the certificate is
verified against. caCertificateRefs name ConfigMaps, or kubernetes.io/tls
Secrets, in the policy’s namespace whose ca.crt key holds a PEM bundle;
wellKnownCACertificates: System trusts the system store instead. The two
are alternatives, as the API defines them: a bundle from caCertificateRefs
is the whole of the trust for that backend, so a certificate from a public
authority is refused, and only a policy selecting System trusts the system
roots. The bundle travels in configuration, so a rotated bundle costs a
reload; a CA certificate is public material. A port-specific policy
outranks one for the whole Service, and of two policies selecting one target
the older wins and the other is reported.
A policy that cannot be honored — a reference that does not resolve or holds
no certificate, a wildcard hostname, subjectAltNames, an unknown well-known
set — makes every backendRef it governs invalid: the reference answers
500 as an unresolvable one does, because connecting without the
verification the policy asked for would be worse. options are
implementation-specific and are reported and ignored. Only Service
targets are supported.
Filters
Filters on a rule apply to every request it matches; filters on a
backendRef apply after them, to the requests that reference receives.
RequestHeaderModifier, ResponseHeaderModifier, URLRewrite,
RequestRedirect and RequestMirror are supported.
Within one header modifier a header may be named once, across set, add
and remove and whatever its case; a filter naming Authorization under
set and authorization under remove has no defined order and makes the
route not served. Header names are case-insensitive throughout. Across
filters, modifications fold into one operation per header in the order they
apply, the rule’s filters before a backendRef’s: a set followed by an add
of the same header sets the joined value; an add after a remove is a
set; a remove after a set removes.
replacePrefixMatch replaces the declared prefix on a segment boundary, so
with a PathPrefix of /api and a replacement of /v2, /api becomes
/v2, /api/ becomes /v2/ and /api/orders becomes /v2/orders; an
empty replacement is /. It requires the rule to have exactly one match,
of type PathPrefix, and on a backendRef it also requires the rule to have
exactly one backendRef: behind a weighted dispatch the member no longer
knows which prefix was matched. A rule and one of its backendRefs may not
both rewrite the path.
RequestRedirect answers with the status it names (301, 303, 307 or
308, or 302 by default) and a Location composed from the request as the filter leaves
it: scheme, hostname, port and path replace their parts of the
request URL and the rest is kept. An explicit scheme with no port sends
the client to the scheme’s well-known port; neither scheme nor port
sends it back to the Gateway listener’s port, whatever port the request’s
Host header carried; and a port that is the well-known one for the scheme
is omitted. A redirecting rule forwards nothing, not even a request carrying
Connection: Upgrade, which is answered with the redirection like any
other. It is still a backend the operator’s controls apply to: an
authenticator from kubernetes.defaults or the GatewayClass’s parameters
guards it exactly as it guards a forwarding rule, and a
ResponseHeaderModifier on the same rule applies to the redirection.
backendRefs written on a redirecting rule are ignored and reported, and it
may not also carry a URLRewrite. Filters apply in the order declared, and a
redirect ends the request: a RequestHeaderModifier ahead of it modifies the
request the redirect answers, so a Host it sets is the Location’s
hostname when the redirect names none, while one after it has no request
left to modify and is reported and ignored. The redirect filter alone
defines the Location: a ResponseHeaderModifier on the same rule that sets, adds or
removes Location, in any case, makes the route not served rather than
being applied in one order or the other and silently losing to the
redirect. Express the target through the redirect’s own scheme,
hostname, port and path.
RequestMirror copies the share of requests it names — percent, or
fraction rounded to a whole percent — to the Service its backendRef
resolves to, discarding their responses; see
Mirroring Requests for what the data plane
does with a copy. The reference resolves exactly as a forwarding
backendRef does, ReferenceGrants and BackendTLSPolicy included. One that
does not resolve is the one filter failure the API defines as partial: the
mirror is dropped, the route’s ResolvedRefs is False, and the rest of
the route is served. A mirror on a rule fires once per request, ahead of a
weighted dispatch; one on a backendRef fires for the requests that
reference receives, and a rule served by one backendRef carrying both
fires both. A filter copying nothing (percent: 0) makes the route
not served.
A filter this build cannot honor — ExtensionRef, CORS, ExternalAuth,
a repeated filter type, a malformed hostname, header, scheme, port or
status — makes the route not served until it is fixed, because ignoring
a filter would serve the route wrongly rather than partially.
Timeouts and retry
A rule’s timeouts and retry apply to every request it serves, behind a
weighted dispatch included; see
Timeouts and Retries for their exact
semantics. timeouts.request bounds the whole upstream exchange, retries
included, and timeouts.backendRequest each attempt; a zero duration
bounds nothing, and backendRequest may not exceed request. A request
that runs out of either with no attempt answered is answered 504. retry
repeats an idempotent request after a connection failure and after a
response whose status is in codes, attempts times (once when unset, at
most ten), waiting backoff between attempts. The Gateway API defines no
retry budget, so every eligible request is retried. A value the data plane
cannot express makes the route not served. A rule’s sessionPersistence is
described next.
Session persistence
A rule’s sessionPersistence keeps a client on the backend it first reached,
on the ALB that serves the rule, as Sticky Sessions
describes. The client carries a signed token naming its backend, in a cookie
or a request header, so any replica holding the same key honors it.
| Field | Lowered to |
|---|---|
type: Cookie (the default) | sticky.mode: cookie |
type: Header | sticky.mode: header: the token is issued in a response header, and the client sends it back in a request header of the same name |
sessionName | the cookie’s or header’s name |
absoluteTimeout | sticky.ttl; unset, a session has no absolute limit. A limit under one second is kept as one second, since a token keeps whole seconds |
cookieConfig.lifetimeType: Session (the default) | sticky.cookie.lifetime: session: the cookie has no Max-Age and ends with the browser session, while absoluteTimeout still ends the token |
cookieConfig.lifetimeType: Permanent | the cookie’s Max-Age ends with the token |
The Gateway API version this build reads has no idle timeout for a route’s
session, and the class’s sticky_ttl and sticky_idle parameters do not
apply to one.
Where the session is kept depends on the rule’s backendRefs and the routing mode:
- Several backendRefs. The rule’s round-robin ALB keeps the session, and
its token names the Service. In the endpoint routing mode it also names the
endpoint within that Service, so a session stays on one pod. In the
servicerouting mode kube-proxy chooses the endpoint, so a session stays on its Service only; the controller reports this for the rule. - One backendRef. Only the endpoint routing mode has an ALB between the
rule and the Service’s pods, and that ALB keeps the session. In the
servicerouting mode the rule is served without a session, and the controller reports it.
An unnamed cookie is named for the ALB that sets it,
trickster_sticky_<hash>, where the hash is 12 hex digits of the ALB’s
name, since ALBs that serve one host must not share a cookie. A generated
ALB is named for its route and its place among the route’s rules, for each
of the route’s hostnames and listener ports, so renaming the route, or
adding a rule, hostname or port ahead of it, renames the cookie and starts
its sessions over. An unnamed header is X-Trickster-Session. A
sessionName beginning __Host- or __Secure- makes the cookie always
Secure, as a browser requires, so it works only over HTTPS.
A named cookie is set by one generated ALB per host. A browser sends a
host’s cookies to every port and scheme it serves, and each rule and each
listener port has an ALB of its own, whose tokens the others do not honor,
so this holds across every rule and every listener, HTTP and HTTPS alike.
When two rules set the same sessionName for a host they both serve, the
older route keeps it (within one route, the earlier rule), and the other is
served without a session and told why. A rule served on two ports for one
host, such as a route attached to both an HTTP and an HTTPS listener, keeps
a named session on only one of them; leave sessionName unset there, so
that each port’s ALB names a cookie of its own. A route served under
several hostnames uses its name once per host, which is allowed. A
sessionPersistence value that cannot be expressed is reported, and the
rule is served without a session.
Each of these problems is reported as a Warning Event with the reason
Rejected on the route, or on the XBackendTrafficPolicy; the route’s
Accepted condition does not change, since the rule is still served.
A GRPCRoute rule’s sessionPersistence is lowered the same way. gRPC
clients rarely keep cookies, so give a GRPCRoute type: Header, and have
the client send back the header’s value as request metadata.
The token travels upstream with the request, and is added to the response
outside any cache, so it is never part of a cached object. Generated ALBs do
not mark a response that sets the cookie private, so a shared cache in
front of Trickster may store it and hand the cookie to other clients; see
Sticky Sessions and Caching. In front
of cacheable content prefer table mode, which sessionPersistence cannot
ask for: leave the rule without one, and set sticky: table in a
TricksterCachePolicy or the class parameters. That keeps each client on its
endpoint within a Service in the endpoint routing mode, with the key and
replica limits that section describes.
The session key
Every replica must key tokens alike, or each refuses the others’ tokens. A
GatewayClass names the key’s Secret with its sticky_secret parameter:
apiVersion: v1
kind: ConfigMap
metadata:
name: cached-gateway-params
namespace: trickster
data:
sticky_secret: trickster-sticky-key
---
apiVersion: v1
kind: Secret
metadata:
name: trickster-sticky-key
namespace: trickster
labels:
trickstercache.org/sticky-key: ""
type: Opaque
stringData:
key: <at least 32 random bytes>
The Secret must be in the parameters ConfigMap’s namespace. A
namespace/name value is an invalid parameter, so the class is not served
(InvalidParameters) until it is fixed. It must
carry the trickstercache.org/sticky-key label, since the controller watches
only labeled Secrets beside TLS ones. Its key entry must hold at least 32
bytes. Every replica reads the same Secret, so every replica keys tokens
alike. A change to it reaches the ALBs without a restart. Replacing the key
starts every session over once, since tokens made with the old key are no
longer honored. A Secret that is missing, unlabeled or too short is reported
on the GatewayClass (InvalidParameters), which is still served, and until it
is fixed its ALBs key their tokens with kubernetes.defaults.sticky_secret_file
when that is set, and per process otherwise. The label is the same text as
the trickstercache.org/sticky-key Ingress annotation, but it is a label on
the Secret, not an annotation on a route. The key is kept in memory. The configuration view
redacts it, and it is never logged.
Without a class Secret, tokens are keyed with the file that
kubernetes.defaults.sticky_secret_file names, which is how an Ingress-only
deployment configures it. Mount the file from a Secret at the same path on
every replica:
kubernetes:
defaults:
routing_mode: endpoint
sticky_secret_file: /etc/trickster/sticky/key
The file must exist and hold at least 32 bytes, after trailing line breaks, or the configuration is refused. It is read when the configuration loads, so a rotated file takes effect at the next configuration reload.
With neither set, each process keys its tokens with a random key of its own. A replica then refuses the others’ tokens and moves the session, and a restart ends every session. That suits a single replica only, and Trickster logs a warning for each ALB that issues tokens with such a key.
XBackendTrafficPolicy
Where the cluster serves the experimental
gateway.networking.x-k8s.io/v1alpha1 XBackendTrafficPolicy, its
sessionPersistence applies to every HTTPRoute and GRPCRoute backendRef
naming a targeted Service, lowered as above onto the ALB that balances that
Service’s endpoints. Ingress backends and stream routes do not read it. That
ALB exists only in the endpoint routing mode, so elsewhere the policy is
ignored, without a report. A named cookie is claimed per backendRef and host
as a rule’s is, so where two routes reach the Service on one host, only the
older keeps the session. A rule’s own sessionPersistence takes precedence,
since its token already carries the Service and its endpoint. A targetRef
must be a Service in the policy’s namespace. When two policies target one
Service, the older one applies and the younger one is reported; a policy
whose sessionPersistence cannot be expressed claims nothing, and a younger
valid one applies instead. retryConstraint is reported and ignored. The
policy’s status is not written back. The controller reads the kind only
where it may list and watch it; see kubernetes-rbac.md.
A TricksterCachePolicy’s sticky fields,
and the sticky class parameters above, apply to a Service’s endpoint ALB
when neither the rule nor the Service’s traffic policy asks for a session.
GRPCRoute
A GRPCRoute attaches, ranks and resolves exactly as an HTTPRoute does, and
lowers onto what gRPC is on the wire: every call is a POST to
/{service}/{method}. A method match with service and method is an
exact path; service alone is the path prefix beneath the service; method
alone matches the method under any service; and a RegularExpression type
matches each segment as a pattern, an absent one matching any segment. A
headers match is a header match. RequestHeaderModifier,
ResponseHeaderModifier and RequestMirror are the HTTPRoute’s filters;
ExtensionRef is not honored. The generated backends select the proxy
handler, which relays trailers, and speak cleartext HTTP/2 by prior
knowledge to a plaintext Service, or HTTP/2 over TLS to one a
BackendTLSPolicy governs, since gRPC needs HTTP/2 end to end. A
TricksterCachePolicy on a Gateway or a Service still governs a GRPCRoute’s
backends for the operator-tier controls, but a GRPCRoute caches nothing and
a policy cannot target one.
An HTTP or HTTPS listener admits both kinds by default and
allowedRoutes.kinds may restrict it to one; supportedKinds says which.
An HTTPRoute and a GRPCRoute attached to one listener with intersecting
hostnames cannot both be served: the older is, and the newer is refused
with Accepted: False and reason RouteConflict, which is the resolution
the Gateway API defines. The controller watches GRPCRoutes only in a cluster
whose Gateway API serves the kind; see
kubernetes-rbac.md.
TCPRoute, TLSRoute and UDPRoute
The stream route kinds attach as an HTTPRoute does, through a parentRef
naming a claimed Gateway, narrowed by sectionName or port and subject to
the listener’s allowedRoutes; a TCPRoute attaches only to TCP listeners,
a TLSRoute only to TLS listeners and a UDPRoute only to UDP listeners,
and one naming a listener of another protocol is refused with
NotAllowedByListeners. Each is served with exactly one rule, since nothing
in a stream selects among rules; a route declaring more is refused with
UnsupportedValue. The rule’s backendRefs resolve as an HTTPRoute’s do
(a Service port, ReferenceGrants for another namespace), with their weights,
and an unresolvable reference keeps its share of the connections and refuses
them, as the Gateway API requires; a BackendTLSPolicy does not apply, since
the stream is relayed unread.
A TCP or UDP listener carries one route: the oldest route naming it is
served and every later one is refused with RouteConflict. A TLSRoute’s
hostnames are the server names it serves, intersected with the listener’s
hostname as an HTTPRoute’s are; a route naming none serves every server name
the listener admits. Listeners on one port bind one socket, so each server
name on a port belongs to one route, the oldest, however many listeners on
that port admitted it: a route admitted by a wildcard listener and a precise
one on the same port is served once there, a newer route that names only
names already served on the port is refused with RouteConflict, and one
that names some is served for the rest and told which it lost. A route
admitted by several Gateways on one port takes the first Gateway’s policy.
A connection is relayed to the route whose hostname matches the server name
it offers, a precise name before a wildcard, and a connection whose server
name no route serves, or that offers no ClientHello at all, is closed.
A backendRef’s port is resolved by the route’s transport: a TCPRoute or TLSRoute selects the Service port of that number carrying TCP and a UDPRoute the one carrying UDP, so a Service exposing one port number over both, with different names and targets, is reached at the right target in either mode; a Service exposing the number over the other transport alone is an unresolved reference, keeping its weight and refusing its share.
Every connection, and every UDP client’s session, is committed to one
backendRef by weighted round robin, and in the endpoint routing mode to one
of that Service’s ready endpoints, in turn unless the GatewayClass’s
load_balancing parameter names another mechanism. The split between
backendRefs is always round robin, because Gateway API makes those weights an
exact apportionment; the mechanism applies within each Service. A key that a
route’s listener cannot read, such as sni on a TCPRoute, is ignored in
favor of the client address. A member that cannot be dialed
refuses the connection rather than passing it to a sibling, an unresolved
reference refuses its share without a lookup, and it is the Service’s
readiness that takes an endpoint out of rotation. The
health_mode parameter does not apply to stream members, which are always
judged by readiness, since no probe speaks the protocol they carry. A stream
route caches nothing, and a TricksterCachePolicy cannot target a TCPRoute or
TLSRoute; one on a UDPRoute applies only where a trickstercache.org/moq
listener serves it (MoQ relays), and is reported elsewhere. It
reads only its routing mode, load balancing and sticky settings, from the
class’s parameters or a TricksterCachePolicy on its Gateway, and its routing
mode also from kubernetes.defaults.routing_mode. A stream route’s
session is kept in a table whichever of cookie, header or table
sticky names, and not at all with none or when unset: by client_ip,
or by sni on a TLSRoute when sticky_key says so. The ALB the listener
maps to keeps it for the whole path, so in the endpoint routing mode a
client returns to the same Service and pod, and in the service routing
mode a route with several backendRefs keeps a client on its Service.
The controller watches the three kinds only in a cluster whose experimental
Gateway API channel serves them (gateway.networking.k8s.io/v1alpha2); see
kubernetes-rbac.md. Their status is written as an
HTTPRoute’s is.
MoQ relays
Trickster relays Media over QUIC Transport (MoQ) through a Gateway
in two ways, each generating a moq backend:
- Native QUIC. A
trickstercache.org/moqlistener opens a MoQ listener on the declared UDP port. It terminates TLS as anHTTPSlistener does, with certificates supplied from the referenced Secrets at runtime, and carriesUDPRoutes. Each route is served with one rule, and the oldest route naming a hostname on a port serves it, as a TLSRoute serves its server names. A backendRef names a Service port carrying UDP where a native MoQ origin listens, dialed asmoqt://<service>.<namespace>.svc:<port>/. - WebTransport. An
HTTPRouterule whoseTricksterCachePolicynamesprovider: moqrelays the WebTransport sessions browsers open. The policy may target the route, the rule, or every Service the rule reaches (or one port of each); a rule whose backendRefs mix MoQ origins with other Services is refused. The rule’s listener must be anHTTPSlistener serving HTTP/3; on any other the rule is refused, withResolvedRefsFalseand reasonUnsupportedProtocol. Each match accepts sessions at and below its path (aRegularExpressionpath as a pattern), and a backendRef names a Service port carrying UDP where a WebTransport origin listens, dialed ashttps://<service>.<namespace>.svc:<port>/. Filters, timeouts, retries and session persistence shape HTTP requests, which a session is not, so they are reported and ignored.
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: media
namespace: infra
spec:
gatewayClassName: trickster
listeners:
- name: moq
port: 443
protocol: trickstercache.org/moq
hostname: live.example.com
tls:
certificateRefs:
- name: live-tls
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: UDPRoute
metadata:
name: live
namespace: media
spec:
parentRefs:
- name: media
namespace: infra
rules:
- backendRefs:
- name: origin
port: 4443
A rule with several backendRefs relays each track from one of them: the
generated backend reaches its origins through an ALB named for it with an
-origins suffix, by hrw unless the policy’s loadBalancing names rr,
p2c or lc, apportioned by the backendRefs’ weights. When an origin fails,
its tracks move to the others, and an unresolved reference keeps its weight
with an origin that never answers, so its tracks move too.
The relay takes cacheName (where groups leaving the live edge are kept),
maxTTL and loadBalancing from the class’s parameters and the policies
governing the route and its Services; the policy fields only HTTP responses
have are reported as ignored when set (see
kubernetes-cache-policy.md).
QUIC is always encrypted: without a BackendTLSPolicy on the Service, the
origin’s certificate is not verified, trusting the cluster network as a
plaintext HTTP backend does, and with one it is verified against the
policy’s CA certificates and hostname. A relay always addresses the
Service’s cluster IP, whatever the routing mode, since it keeps each track
on one origin for as long as it relays it. On a native listener, each
route’s generated backend carries the class’s ip_acl_name (else
kubernetes.defaults.ip_acl_name) and its geo ACL, as on any other route,
and a session is judged by the lists of the backend it is routed to, so
routes sharing the listener keep their own. A native session carries no
HTTP credentials, so a class with an authenticator_name (or
kubernetes.defaults.authenticator_name) does not serve its MoQ UDPRoutes:
each is refused with UnsupportedValue rather than served
unauthenticated. Serve them from a class without one.
A client reaches a UDP port only where the Service in front of the
controller publishes it. gateway-service.yaml
publishes UDP 443 beside TCP 443, which carries HTTP/3, WebTransport and a
trickstercache.org/moq listener on 443 alike; a listener on another port
needs a UDP entry of its own.
Endpoint routing mode
With routing_mode: endpoint, from kubernetes.defaults or a GatewayClass’s
parameters, a backendRef is served by an ALB whose pool is the Service’s
ready endpoints, discovered from its EndpointSlices, rather than a backend
addressing the Service’s cluster IP. Each pool carries everything the
backendRef would have carried — cache, timeout, TLS, filters. Endpoint churn
reaches the pools without a configuration reload, and a rolling restart
drains terminating endpoints before their pods stop: they take no new
clients, keep the sessions pinned to them, and leave the pool once they stop
serving. The controller’s
service account needs endpointslices list and watch for it; see
kubernetes-rbac.md.
health_mode decides how a discovered member is judged healthy:
provider (the default) trusts the EndpointSlice’s readiness, which is
what the pod’s own readiness probe already established; probe runs an
active health check, configured by kubernetes.defaults.healthcheck or,
when that is unset, a probe of the origin’s root every 5 seconds. See alb-autodiscovery.md
for the semantics of both.
Status
Every claimed object is told what became of it, by one replica at a time (see Leader election):
- GatewayClass:
AcceptedisTruefor a claimed class, orFalsewith reasonInvalidParameterswhen itsparametersRefcannot be honored, in which case none of its Gateways is served. - Gateway:
AcceptedisTruewhen every listener is valid,Truewith reasonListenersNotValidwhen only some are, andFalsewhen none is.ProgrammedfollowsAccepted, and is lowered toFalsewith reasonPendingwhen the data plane rejected the generated configuration, orAddressNotAssignedwhenkubernetes.published_serviceis configured but that Service has no address yet.status.addressescarries the published Service’s load balancer addresses, or failing those its external IPs. - Gateway listeners: one entry per declared listener, with
Accepted,Programmed,ResolvedRefsandConflicted,supportedKinds(the kinds the listener’s protocol carries, those of themallowedRoutes.kindsadmits, or nothing when it admits no kind this controller serves) andattachedRoutes, which counts only routes that are served. A refused listener keeps its entry and says why (UnsupportedProtocol,UnsupportedValue,ProtocolConflict,HostnameConflict,InvalidCertificateRef). A certificateRef that does not resolve lowersResolvedRefs(InvalidCertificateRef,RefNotPermitted) without refusing the listener, and an unsupported route kind lowers it withInvalidRouteKinds. - HTTPRoute (and every other route kind): one
parentsentry, under this controller’s name, per parentRef naming a claimed Gateway.AcceptedisTruewhen at least one listener accepted the route, orFalsewithNoMatchingParent,NotAllowedByListeners,NoMatchingListenerHostname,RouteConflictfor a stream route whose listener or server names an older route already serves, orUnsupportedValuefor an unroutable hostname, a filter that cannot be honored, or a stream route with more than one rule, each of which unserves the whole route.ResolvedRefsisFalsewithBackendNotFound,InvalidKind,RefNotPermittedorUnsupportedProtocol(a BackendTLSPolicy that cannot be honored) when any backendRef did not resolve; the route stays attached and the unresolved reference answers with an error. Entries other controllers wrote are left untouched, and a parentRef naming a Gateway this controller does not claim gets no entry.
Every condition’s observedGeneration is the generation that was
translated, and a verdict is written only onto that generation of that
object, so an object edited after translation, or recreated under the same
name, is never labeled with a verdict about its predecessor.
A Gateway whose class stops being accepted keeps its Gateway and listener
entries — the listeners Accepted on their own terms, Programmed: False
because the Gateway is not served — and every HTTPRoute naming it is refused
with NoMatchingParent and the class’s reason, so a class rejection reaches
the routes rather than leaving them with the acceptance they had. An HTTPS
listener is Programmed only while the data plane holds a usable certificate
for it: one whose only material cannot be parsed is Programmed: False with
reason Invalid, while a rotation to unusable material leaves the certificate
already serving in place, so the listener stays Programmed with
ResolvedRefs: False saying why the new material was not taken.
Conditions of other types and other controllers’ entries are kept, and an
object whose status already says what the pass concluded is not written at
all, so a resync that changes nothing makes no API call. A write the API
server refuses is logged, counted in
trickster_kgw_status_write_failures_total, and retried on the next pass.
Status writing never delays the routes and certificates a pass carries.
Status is not withdrawn from an object once it stops being claimed; the new
owner overwrites it.
Events
Everything a pass could not do with an object is a Warning Event on that
object, visible in kubectl describe: Rejected for a translation problem,
InvalidAnnotation for a rejected annotation, InvalidCertificate for a TLS
Secret that is missing or unusable, CertificateRejected for one a
listener’s certificate store refused, InvalidParameters for a
GatewayClass’s parameters, and Invalid, Conflicted or TargetNotFound
for a TricksterCachePolicy. A claimed GatewayClass gets a Normal Accepted
Event. An Event is published when the problem first appears, and again when
the replica publishing it takes over leadership, not on every resync;
client-go’s aggregation folds repeats and rate-limits per object.
Leader election
Every replica watches, translates and programs its own data plane. Status and
Events are written by one replica, elected over a Lease
(kubernetes.leader_election), so two replicas never fight over one
object’s status. Every write is bound to the leadership term it began under,
so a replica that loses the Lease stops writing at once and cannot overwrite
what its successor wrote. A replica that loses the election keeps serving.
leader_election.enabled: false makes every replica a writer, for a
single-replica deployment that wants no Lease; read_only: true makes none
of them one, and needs no write permission at all. The Lease lives in
leader_election.namespace, the pod’s own namespace by default, and
lease_duration must be at least one second because the Lease API stores it
in whole seconds.
The leader is visible as trickster_kgw_leader; the controller’s other
metrics are in metrics.md. When kubernetes.defaults.tracing_name
names a tracer, every reconcile pass is traced on it: kgw.reconcile, with
kgw.translate, kgw.compile, kgw.apply and kgw.certificates beneath
it, and each status write as a kgw.status trace of its own, since it runs
on its own worker.
Pod readiness
The readiness endpoint (/trickster/ready; see
Graceful shutdown) reports
503 not programmed from before the listeners open until the controller’s
first translation is serving, and 200 ready from then on. A translation the
data plane rejects programs nothing, so the pod stays not ready until a later
reconcile succeeds. A pod is therefore not added to its Service’s endpoints
until the routes it exists to serve are in place, which is what makes a
rolling update of the controller Deployment invisible to clients. A
controller that cannot start, because the API server is unreachable or the
service account lacks a permission, keeps the pod not ready, so the rollout
waits rather than replacing a working pod with one that serves nothing. A
reload that changes the kubernetes section restarts the controller without
touching readiness, since the routes already programmed keep serving; a
reload that enables the section after it was off holds readiness again until
the new controller publishes.
Tuning for generic web ingress
The defaults suit a caching proxy in front of an API. A Gateway serving
arbitrary web traffic through Trickster should consider these settings on
the listeners it generates and in kubernetes.defaults:
| Setting | Where | Suggested | Why |
|---|---|---|---|
max_request_body_size_bytes | listener | 10 MiB (the default), higher for upload paths | A request body above it is refused with 413 before it reaches an origin; truncate_request_body_too_large is for logging, not proxying |
read_header_timeout | listener | 10s | Bounds a client that sends headers slowly; has no effect on the body or the response |
idle_timeout | listener | 2m (the default) | Closes keep-alive connections that wait too long for their next request; 0 never closes them |
max_header_bytes | listener | 64 KiB | Bounds the memory one request’s headers can take; the default is 1 MB |
connections_limit | listener | 0 (unlimited), or the pod’s file descriptor budget | A limit blocks accepts rather than refusing them |
proxy_protocol, trusted_proxies | listener | the load balancer’s addresses | The real client address in logs and max_query_range decisions; see Trusted Proxies |
timeout | kubernetes.defaults | 60s (the default) | Bounds how long an origin may take to start a response and how long its body may stall; a route’s timeouts bound more |
max_object_size_bytes | backend | 512 KiB (the default), higher for large cached objects | A response above it is served but never cached; streaming responses are unaffected |
access_log | kubernetes.defaults | the json preset to stdout | One line per request with the route, upstream and request identifiers; see access-logs.md |
A response the origin streams is relayed as it arrives on every handler; the caching handlers buffer only what they store. Range requests are served through the cache as described in range_request.md.
Conformance
The controller is tested against the upstream
Gateway API conformance suite
for the GATEWAY-HTTP profile on every change, against the experimental CRD
channel. make kind-conformance (or kind-conformance-docker on macOS) runs
it against a local kind cluster, and CI publishes the report as a workflow
artifact.
The core result is partial rather than a core conformance claim: every core
test passes but HTTPRouteMultipleGateways, which is skipped. It expects two
Gateways declaring one port to answer at distinct addresses, which one
process serving every Gateway of a class at one address cannot do; the
Gateways merge on the port as described under Listeners. A deployment that
needs Gateways at distinct addresses runs one controller Deployment per
GatewayClass (kubernetes-deploy.md).
The suite at the version this build pins has no session persistence tests,
and names no feature for it, so sessionPersistence is outside the report.
The kind scenario TestGatewaySessionPersistenceKind covers it instead,
with a cookie and a header session over two Services.
The report lists the extended features claimed. They cover query parameter, method and port matching, request and response header modification on rules and backendRefs, path, host, scheme and port rewrites and redirects, request mirroring, request and backend timeouts, retries, named route rules, and WebSocket and h2c backend protocols.
Not supported
| Feature | Status |
|---|---|
GatewayHTTPListenerIsolation | Not supported; see the deviation under Listeners |
spec.addresses, GatewayStaticAddresses | Addresses come only from kubernetes.published_service |
spec.infrastructure.parametersRef | The Gateway is refused with InvalidParameters |
ListenerSets (spec.allowedListeners) | Reported and ignored |
| Frontend and backend client certificates | Not supported |
CORS and ExternalAuth filters | Refused; the route is not served |
Multiple RequestMirror filters on one rule | A rule carries one mirror |
BackendTLSPolicy status, subjectAltNames | The policy is honored; its status is not written back and subjectAltNames is refused |
A TLS listener in Terminate mode | Not served; Passthrough only |
XBackendTrafficPolicy status and retryConstraint | Its sessionPersistence is honored; its status is not written back and retryConstraint is ignored |
sessionPersistence in the service routing mode | Kept only on a rule with several backendRefs, and then on the Service alone; see Session persistence |
| Rate limiting | Rate-limit at the load balancer in front, or at the origin |
| MoQ publishing through a Gateway | A generated moq backend relays from its origins and accepts no publishers |
| MoQ listener and relay tuning | Drafts, session limits, GOAWAY URIs and the live edge take their defaults |
MoQ relays in the endpoint routing mode | A relay addresses the Service’s cluster IP; see MoQ relays |
Generated configuration
Every listener, backend, ALB and request rewriter the controller generates
carries the reserved kgw-- prefix and a name derived from the Kubernetes
object’s identity, so the names you see in logs, metrics and the management
API are stable across restarts. A route served on two listeners is served by
one set of objects per port and hostname, each with its own cache keys, so an
object cached through one port is not served through the other.
Generated configuration is merged onto the file configuration and then loaded, validated and applied by the same code that loads a configuration file, so it is exempt from no check. A translation the daemon rejects leaves the last good configuration serving and is not carried into later reloads.
10.3 - Kubernetes Ingress v1
Trickster serves Kubernetes networking.k8s.io/v1 Ingress objects it claims,
translating them into its own configuration and reloading onto it. Enable the
controller with the top-level kubernetes section
(configuring.md) and grant it the permissions in
kubernetes-rbac.md.
What is claimed
An Ingress belongs to this controller when:
- its
spec.ingressClassNamenames an IngressClass whosespec.controllermatcheskubernetes.gateway_class_controller_name; or - it names no class, and one of this controller’s IngressClasses is annotated
ingressclass.kubernetes.io/is-default-class: "true"; or - it carries the deprecated
kubernetes.io/ingress.classannotation naming the configuredkubernetes.ingress_class.
Setting kubernetes.ingress_class narrows ownership to that one class even
when others name this controller, which is how two Trickster instances divide
a cluster. Anything not claimed is ignored entirely: not translated, not
counted, and never statused.
Listeners
An Ingress cannot describe the port it is served on, so it names the
listeners instead — exactly as a backend names the listeners it is served
on. They are ordinary listeners, configured in the listeners section, so
their ports, bind addresses, body limits, timeouts and TLS settings are
tuned the same way every other Trickster listener’s are:
listeners:
web:
port: 8080
max_request_body_size_bytes: 10485760
websecure:
tls_port: 8443
# the certificates arrive at runtime from the Secrets the claimed
# Ingresses reference, so the port is kept open with none behind it
tls_runtime_certs: true
kubernetes:
ingress:
listener_names:
- web
- websecure
Every claimed route is served on all of them. Naming none serves claimed Ingresses on the default frontend, which is where a backend that names no listener is served. A listener that does not exist fails validation, the same way an undefined listener named by a backend does.
A single listener may serve both ports, so listeners: {web: {port: 8080, tls_port: 8443, tls_runtime_certs: true}} named on its own is equally
valid. Binding 80 and 443 inside a container requires running as root,
granting NET_BIND_SERVICE, or the net.ipv4.ip_unprivileged_port_start
sysctl the deployment in deploy/kube sets
(kubernetes-deploy.md); an
install behind a load balancer that terminates TLS simply configures no TLS
port.
Rules and paths
| Ingress | Trickster |
|---|---|
rules[].host | the backend’s hosts; a single leading *. wildcard is honored |
| a rule with no host | any_host_routing |
pathType: Exact | match_type: exact |
pathType: Prefix | match_type: segment |
pathType: ImplementationSpecific | as Prefix, or regex with trickstercache.org/use-regex |
backend.service | a generated backend at http://<service>.<namespace>.svc:<port> |
spec.defaultBackend | a hostless catch-all ordered after every other route |
A Kubernetes Prefix matches whole path elements: /foo matches /foo and
/foo/bar but never /foobar, and a trailing slash means nothing, so
/foo/ and /foo are the same rule. That is the router’s segment match
type, so a Prefix path is one generated path; see
paths.md.
The request reaches the Service with the Host header the client sent, so an
application that reads it sees the Ingress host rather than its own Service
name; cached objects are keyed by it, so a rule with no host serving several
never hands one host’s object to another.
The referenced Service and port must exist in the cluster. A backend that
cannot be resolved still answers, with a fixed 500, rather than silently
vanishing from the routing table; the reason is logged. Because Services are
watched, creating the Service later heals the route without any further
action. backend.resource references are not supported.
Conflicts
Two Ingresses may claim the same host. The router resolves overlapping paths on its own — exact before prefix before regex, longest first within each tier — so the only genuine conflict is a duplicate of host and path once both have been lowered. That is broken in favor of the older object; ties on creation time are broken by namespace and name, so every replica reaches the same answer. A declaration is reported only when it loses everything it asked for, and keeps whatever else it claimed.
An Exact rule outranks a Prefix rule for the same path, whichever object
declared it and whichever is older, which is the precedence Kubernetes
defines. The prefix keeps everything below that path, so the two coexist and
neither is reported as a conflict.
spec.defaultBackend answers whatever no rule matched, so it is emitted as a
catch-all ordered after every other route, including a hostless regular
expression rule. Because it is the controller’s own invention rather than
something the operator wrote, any declared rule that lowers onto the same
route takes it — a hostless / rule with use-regex, for instance — and the
unreachable default backend is reported. Two Ingresses declaring a default
backend do conflict, and the older one wins.
Regular expression paths are anchored before anything else looks at them, so
/api/(.*) and ^/api/(.*) are one route rather than two that would race to
register the same pattern.
TLS
Each spec.tls[].secretName must name a kubernetes.io/tls Secret in the
Ingress’s own namespace. The certificate never travels in configuration: it
is supplied to the TLS listener at runtime and re-supplied whenever the
Secret’s contents change, so rotating a Secret costs no reload at all. See
tls.md.
Certificates reach whichever of the named listeners actually serves TLS; that is a property of the listener’s own configuration, so a plaintext one is simply skipped. A listener’s certificates live only as long as the listener does, so a configuration change that removes one takes its certificates with it; they are installed again when it comes back, without the Secret having to change.
Because only kubernetes.io/tls Secrets are watched, a Secret of any other
type reads as absent, and is reported that way.
Annotations
Annotations outside the trickstercache.org/ namespace are another controller’s
business and are ignored. One inside it that is unknown, or whose value does
not parse, is rejected: the annotation is not applied, the rest of the
object still translates, and the rejection is logged. Failing the whole
object would let one typo delete a live route; applying it silently would
leave an operator believing a setting is in force when it is not.
| Annotation | Value | Effect |
|---|---|---|
trickstercache.org/handler | proxy, proxycache | selects the path handler; proxycache makes the generated backend cache-capable |
trickstercache.org/cache-name | a configured cache name | which cache a caching route uses |
trickstercache.org/max-ttl | a duration, e.g. 10m | caps how long a cached object is served before revalidation |
trickstercache.org/negative-cache-name | a configured negative cache name | how long error responses are cached |
trickstercache.org/timeout | a duration | the upstream request timeout |
trickstercache.org/collapsed-forwarding | basic, progressive | collapses concurrent requests for one object |
trickstercache.org/request-headers | Name: value per line | header updates on the way upstream |
trickstercache.org/response-headers | Name: value per line | header updates on the way back |
trickstercache.org/cors-mode | preserve, merge, replace, disable | how origin CORS headers are combined with the configured ones |
trickstercache.org/cors-headers | Name: value per line | the CORS headers merge and replace apply |
trickstercache.org/use-regex | true, false | compiles this object’s ImplementationSpecific paths as anchored regular expressions |
trickstercache.org/rewrite-target | a path | rewrites the matched path on the way upstream |
trickstercache.org/health-mode | probe, provider | how discovered members are judged healthy in the endpoint routing mode |
trickstercache.org/load-balancing | rr, p2c, lc, lt, hrw | how traffic is spread across a Service’s endpoints in the endpoint routing mode; rr unless set |
trickstercache.org/load-balancing-key | client_ip, host, header:<name>, cookie:<name>, query:<name> | what hrw keeps on one endpoint |
trickstercache.org/sticky | cookie, header, table, none | keeps a client on the endpoint it first reached, in the endpoint routing mode; see below |
trickstercache.org/sticky-key | client_ip, host, header:<name>, cookie:<name>, query:<name> | what table mode keeps a client’s endpoint by; client_ip unless set. sni, user and proxy_tlv:<type> are accepted but cannot be read from a request, so they keep client_ip |
trickstercache.org/sticky-ttl, trickstercache.org/sticky-idle | a duration of at least 1s | a session ends that long after it began (1h unless set), or once unused that long |
trickstercache.org/step-alignment | truncate, drop, partial, partial_start, partial_end, off | the step alignment mode of the time series backend a cache policy’s provider makes; it applies only where that provider supports the mode, and the provider’s default applies otherwise |
Durations require a unit: 600 is rejected, 600s is not.
In a header list, a name prefixed with - deletes the header and one
prefixed with + appends to it rather than replacing:
metadata:
annotations:
trickstercache.org/request-headers: |
X-Forwarded-Host: shop.example.com
-X-Internal-Token:
trickstercache.org/response-headers: |
+Vary: Accept-Encoding
cache-name and negative-cache-name select among things the operator has
already configured; a name the configuration does not define is rejected
like any other bad value, because emitting it would fail validation for the
whole generated configuration and stop every other route in the cluster from
reconciling. The route keeps serving, on the defaults.
That is the line the annotation set is drawn on: an annotation may select
among what the operator provisioned, but never grant a capability or remove
a control. What an annotation may not carry — a time series provider, the
cache key components, hiding the X-Trickster-Result header — belongs on a
TricksterCachePolicy targeting the Ingress or its Service, a resource with
RBAC of its own; a policy on the Ingress is written over its annotations. See
kubernetes-cache-policy.md. Anything on the
far side of that line is an operator setting under kubernetes.defaults,
where it applies to every generated backend and no Ingress author can change
it:
kubernetes.defaults | Effect |
|---|---|
cache_name, negative_cache_name | the defaults a route may override by annotation |
tracing_name | the configured tracer generated backends report to |
req_rewriter_name | a configured rewriter every generated backend runs, ahead of any route’s own |
authenticator_name | the configured authenticator every generated backend is behind |
geo_acl_name | the configured geo ACL that gates the backend each route attaches |
ip_acl_name | the access list every generated backend names; see ip-acl.md |
rate_limiter_name | the rate limiter every generated HTTP and gRPC route is behind; see rate-limiting.md |
authenticator_name and geo_acl_name in particular have no annotation and
will not get one: an annotation that can name an authenticator or a geo ACL
is one that can also omit it or name a looser one, and whoever can create an
Ingress in their own namespace would then be able to take their route out
from behind it. A name in defaults that
the configuration does not define fails startup, rather than the first
reconcile, because it is the operator’s own mistake to see immediately.
ip_acl_name likewise has no Ingress annotation and no TricksterCachePolicy
field. An Ingress author cannot select another list or clear the configured
default with none. The list must use source: client_ip and action: reject.
A name the configuration does not define, and a list that is not eligible,
both fail startup. The list syntax is in ip-acl.md.
rate_limiter_name is the same kind of operator setting. It has no Ingress
annotation and no TricksterCachePolicy field, and an Ingress author cannot
clear it with none. It is copied onto generated HTTP and gRPC route backends
only. A stream route does not take it, and neither do pool members, endpoint
templates, mirror targets, or generated listeners. The limiter must use HTTP
keys, a unit of requests or unset, and an action other than close. A name
the configuration does not define, and a limiter that is not eligible, both
fail startup. See rate-limiting.md.
max-ttl, cache-name and negative-cache-name describe caching, so they take
effect only on a route that caches — one whose effective handler is
proxycache, either from trickstercache.org/handler or from a configured
kubernetes.defaults.cache_name. On a non-caching route they are emitted
nowhere, because a non-caching backend would ignore them.
Rewriting
trickstercache.org/rewrite-target replaces the part of the path the rule matched:
- an
Exactor regular expression match knows the whole path it matched, so the path is set outright; - a
Prefixmatch knows only its leading segments, so only those are replaced and the rest of the path is carried through.
With trickstercache.org/use-regex: "true", capture groups in the path are
available to the target as ${1}, ${2}, and so on (${0} is the whole
match, and named groups are available under their names). The pattern is
anchored at the start for you, since the API server requires every Ingress
path to begin with /:
metadata:
annotations:
trickstercache.org/use-regex: "true"
trickstercache.org/rewrite-target: /v2/${1}
spec:
rules:
- host: shop.example.com
http:
paths:
- path: /api/(.*)
pathType: ImplementationSpecific
backend:
service:
name: web-svc
port:
number: 8080
A request for /api/orders reaches the Service as /v2/orders.
Endpoint routing mode
With kubernetes.defaults.routing_mode: endpoint, a rule’s backend is an ALB
whose pool is the Service’s ready endpoints, discovered from its
EndpointSlices, rather than a backend addressing the Service’s cluster IP:
the controller generates a discovery entry over its own connection, a
template backend carrying the rule’s settings, and a discovery-backed ALB
whose query selects the Service’s port. Endpoint churn then reaches the pool
without a configuration reload, and a rolling restart of the Deployment
behind the Service drains terminating endpoints before their pods stop: they
take no new clients, keep the sessions pinned to them, and leave the pool
once they stop serving.
The controller’s service account needs endpointslices list and watch for
it; see kubernetes-rbac.md.
kubernetes.defaults.health_mode decides how a discovered member is judged
healthy, and trickstercache.org/health-mode overrides it per Ingress:
provider (the default) trusts the EndpointSlice’s readiness, which the
pod’s own readiness probe established; probe runs an active health check
from the generated template, configured by kubernetes.defaults.healthcheck
or, when that is unset, a probe of the origin’s root every 5 seconds. See
alb-autodiscovery.md for the semantics of both.
trickstercache.org/sticky keeps each client on the endpoint it first
reached, on the Ingress backend’s endpoint ALB, as
Sticky Sessions describes:
cookieissues a signed token in a cookie named for that ALB,trickster_sticky_<hash>. The cookie’sMax-Ageends with its token, at moststicky-ttl(1hunless set) after the session began.headerissues it in theX-Trickster-Sessionresponse header, for a client that sends it back in the request header of that name.tablekeeps the pin itself, bysticky-key. It adds nothing to responses, so prefer it in front of cacheable content, within the key and replica limits that Sticky Sessions and Caching describes.
An Ingress’s annotations are its least specific settings, so none
changes nothing in an annotation. It is for a
TricksterCachePolicy on the Ingress or a
Service, which may set or override sticky for its backends, and whose
none turns off the annotation’s.
The annotation has no effect in the service routing mode, where kube-proxy
chooses the endpoint. kubernetes.defaults.sticky_secret_file keys the
tokens, as configuring.md describes. Without it they are
honored only by the replica that issued them, and only until it restarts,
and Trickster logs a warning for each ALB that issues such tokens.
Status and Events
When kubernetes.published_service names the Service in front of Trickster,
its load balancer addresses (or, failing those, its external IPs) are written
into every claimed Ingress’s status.loadBalancer by the replica holding the
leader election Lease; see
kubernetes-gateway.md. Without it
nothing is written there. Whatever could not be done with an Ingress — a
rejected annotation, a missing TLS Secret, a path that could not be lowered,
a host and path lost to an older Ingress — is a Warning Event on the
Ingress, visible in kubectl describe ingress.
Beyond what an Ingress expresses
Ingress is served indefinitely. Settings an Ingress cannot express — a weighted canary, a redirect, TLS to a backend, TLS passthrough, per-rule timeouts and retries — are Gateway API features; see kubernetes-gateway.md. Both kinds are served at once, so a host may be moved to an HTTPRoute without disturbing the rest.
A TricksterCachePolicy may target an Ingress, one of its rules’ Services,
or an HTTPRoute, so caching behavior can be moved off annotations
independently. These are the equivalents:
trickstercache.org/ annotation | TricksterCachePolicy field |
|---|---|
handler | handler |
cache-name | cacheName |
negative-cache-name | negativeCacheName |
max-ttl | maxTTL |
timeout | timeout, or an HTTPRoute rule’s timeouts |
collapsed-forwarding | collapsedForwarding |
request-headers | requestHeaders, a map, or a RequestHeaderModifier filter |
response-headers | responseHeaders, a map, or a ResponseHeaderModifier filter |
cors-mode, cors-headers | cors.mode, cors.headers |
health-mode | healthMode |
load-balancing, load-balancing-key | loadBalancing, loadBalancingKey |
sticky, sticky-key, sticky-ttl, sticky-idle | sticky, stickyKey, stickyTTL, stickyIdle, or an HTTPRoute rule’s sessionPersistence for a cookie or header session |
step-alignment | stepAlignment |
use-regex | an HTTPRoute path match of type RegularExpression |
rewrite-target | a URLRewrite filter, whose ReplacePrefixMatch replaces the matched prefix and ReplaceFullPath the whole path |
A header list becomes a map with the same - and + prefixes. A
rewrite-target using regular expression captures has no Gateway API
equivalent: URLRewrite cannot reference captures, so a RegularExpression
match paired with ReplaceFullPath rewrites to a fixed path only. A policy
on an Ingress is written over its annotations; see
kubernetes-cache-policy.md.
Generated configuration
Every backend, ALB and request rewriter the controller generates carries the
reserved kgw-- prefix and a name derived from the Kubernetes object’s
identity, so the names you see in logs, metrics and the management API are
stable across restarts. Listeners are not generated for Ingress: the ones an
Ingress is served on are the operator’s.
Generated configuration is merged onto the file configuration and then loaded, validated and applied by the same code that loads a configuration file, so it is exempt from no check. A translation the daemon rejects leaves the last good configuration serving and is not carried into later reloads.
10.4 - Kubernetes Cache Policy
TricksterCachePolicy is the custom resource that attaches Trickster’s caching behavior to
the objects the Kubernetes controller serves. It carries everything the
trickstercache.org/* Ingress annotations carry
(kubernetes-ingress.md), and three things they
never will: a time series provider, the cache key components, and whether the
X-Trickster-Result header reaches the client. A custom resource has RBAC of its own, so
whoever may create one is decided by the cluster rather than by whoever may edit a route.
Installing the resource
The definition lives at deploy/kube/crds/trickstercachepolicies.yaml. Install it before the controller starts:
kubectl apply -f deploy/kube/crds/trickstercachepolicies.yaml
kubectl wait --for=condition=Established crd/trickstercachepolicies.trickstercache.org
The controller probes for the resource once at startup, as it does for the Gateway API,
and where the cluster does not serve it, serves every route without one. Installing the
definition later takes effect at the next restart or kubernetes configuration change.
The controller’s service account needs list and watch on trickstercachepolicies and
update on trickstercachepolicies/status; see
kubernetes-rbac.md.
Targets and precedence
A policy governs the objects its targetRefs name, in the policy’s own namespace:
kind | group | governs | sectionName |
|---|---|---|---|
Gateway | gateway.networking.k8s.io | every route attached to the Gateway | not supported |
HTTPRoute | gateway.networking.k8s.io | every rule of the route | one rule, by the rule’s name |
UDPRoute | gateway.networking.k8s.io | the route, where a trickstercache.org/moq listener serves it; see MoQ relays | not supported |
Service | "" | every backendRef and Ingress backend naming the Service | one named port |
Ingress | networking.k8s.io | every rule of the Ingress | not supported |
group may be omitted; when given it must be the kind’s own.
Several policies may govern one rule. They are applied least specific first, each field an inner one sets written over the outer one’s, and a field none of them sets takes the configured default:
- the GatewayClass’s
parametersRef(kubernetes-gateway.md), or an Ingress’s annotations; - a policy on the Gateway;
- a policy on the HTTPRoute, UDPRoute or Ingress;
- a policy on the route’s rule, by
sectionName; - a policy on the backend’s Service, then one on the Service’s port.
Header updates fold in that order, one operation per header whatever its spelling: an
inherited Authorization set and a more specific -Authorization leave the header removed,
and a more specific +Vary appends to an inherited Vary. A map may name a header once,
across spellings and operators, because a map is applied in no particular order.
cacheKeyParams and cacheKeyHeaders replace as a whole: a policy that omits a list
inherits it, and one that sets it empty (cacheKeyHeaders: []) clears what was inherited.
A Service policy applies to one backendRef of a rule and not its siblings, so two members of
a weighted rule may be served differently, resultHeader included.
Of two policies naming one target, the older one governs it (ties on creation time break on
namespace then name) and the newer one is reported Conflicted on that target. A policy any
field of which cannot be honored — an unknown value, a duration without a unit, a
cacheName the configuration does not define — governs nothing at all and is reported
Invalid, since a route governed by half a policy would look configured and be something
else.
apiVersion: trickstercache.org/v1alpha1
kind: TricksterCachePolicy
metadata:
name: metrics
namespace: monitoring
spec:
targetRefs:
- kind: Service
name: prometheus
sectionName: http
provider: prometheus
cacheName: timeseries
timeout: 60s
resultHeader: Hide
Fields
| Field | Value | Effect |
|---|---|---|
handler | proxy, proxycache | the path handler; proxycache makes the backend cache-capable |
provider | prometheus, influxdb, clickhouse, graphite, moq | serves the target through a time series provider, or as a MoQ relay; see below |
cacheName | a configured cache | which cache a caching route uses |
negativeCacheName | a configured negative cache | how long error responses are cached |
maxTTL | a duration, e.g. 10m | caps how long a cached object is served before revalidation |
timeout | a duration | the upstream request timeout |
collapsedForwarding | basic, progressive | collapses concurrent requests for one object |
cacheKeyParams | query parameter names | hashed into the cache key of every caching path |
cacheKeyHeaders | header names | hashed into the cache key of every caching path |
requestHeaders | a map of header updates | header updates on the way upstream |
responseHeaders | a map of header updates | header updates on the way back |
cors.mode | preserve, merge, replace, disable | how origin CORS headers combine with the configured ones |
cors.headers | a map of headers | the CORS headers merge and replace apply |
healthMode | probe, provider | how discovered members are judged healthy in the endpoint routing mode |
loadBalancing | rr, p2c, lc, lt, hrw | how traffic is spread across a Service’s endpoints in the endpoint routing mode; rr unless set. See the ALB mechanisms |
loadBalancingKey | client_ip, host, header:<name>, cookie:<name>, query:<name> | what hrw keeps on one endpoint; client_ip unless set |
sticky | cookie, header, table, none | keeps a client on the endpoint it first reached, on an HTTP route or Ingress in the endpoint routing mode; none turns off a less specific setting’s. A cookie’s Max-Age ends with its token, at most stickyTTL after the session began. A TCP, TLS or UDP route keeps it in a table. See kubernetes-gateway.md |
stickyKey | client_ip, host, header:<name>, cookie:<name>, query:<name>, sni | what table mode keeps a client’s endpoint by; client_ip unless set, and a key the route’s listener cannot read (sni on an HTTP route, for one) is left at that |
stickyTTL, stickyIdle | a duration of at least 1s | a session ends that long after it began (1h unless set), or once unused that long |
resultHeader | Expose, Hide | whether X-Trickster-Result reaches the client; see below |
stepAlignment | truncate, drop, partial, partial_start, partial_end, off | the step alignment mode of the time series backend provider makes; a mode the policy’s provider doesn’t support makes the policy Invalid |
In a header map a name prefixed with - deletes the header and one prefixed with +
appends to it rather than replacing, exactly as in the annotations and in Trickster’s own
request_headers. Durations require a unit. cacheName and negativeCacheName select
among what the operator configured; the operator-tier names — a tracer, a request
rewriter, an authenticator, a geo ACL, an IP access list — have no field here either, for the
reason given in the Ingress document: a policy that could name an authenticator or an access
control list could also omit one. A route’s geo ACL stays as kubernetes.defaults or its class
set it, whatever policy the route carries.
maxTTL, cacheName, negativeCacheName, cacheKeyParams and cacheKeyHeaders take
effect only on a route that caches: one whose effective handler is proxycache, from
handler, a configured cacheName, or a provider.
Provider-aware acceleration
provider makes the generated backend that provider — prometheus, influxdb,
clickhouse or graphite — instead of a reverse proxy cache, so a request for
/api/v1/query_range behind a prometheus policy reaches the Delta Proxy Cache and is
served by time range rather than as an opaque object, exactly as it would from a
hand-configured provider: prometheus backend. See the provider documents
(prometheus.md, influxdb.md,
clickhouse.md, graphite.md) for what each accelerates.
A provider never widens a route. Its accelerated paths are reachable only beneath the
route’s own PathPrefix match, and carry that match’s headers, query parameters and
methods: a request that does not satisfy the match, that names a method the match did not
allow, or that asks for a provider path outside the prefix is served exactly as it would
be without the provider — by the next route the router would have chosen, or not at all. A
provider path that another route on the same host owns, by declaring it exactly or by a
longer prefix, is left to that route.
Each accelerated path keeps the provider’s own cache key components (query, start,
end, step for a range query), which are what make acceleration possible, and takes the
policy’s cacheKeyParams and cacheKeyHeaders beside them, its header updates over the
provider’s own, its CORS policy, its collapsed forwarding and its result header
disposition. Behind a weighted or endpoint-mode dispatch, each member is accelerated under
its own effective policy.
Three things follow:
- The route must expose the provider’s API at its native paths. The router selects the
handler by the request path before any rewrite, so a
PathPrefixof/or/apiputs the provider’s paths in reach, while a route that serves Prometheus under/promand rewrites the prefix away never lets a request reach/api/v1/query_rangeon the provider. Serve a provider at its own paths, on a hostname of its own where it has to share a listener. - Only a
PathPrefixmatch accelerates. AnExactmatch names one path, and aRegularExpressionmatch cannot be intersected with the provider’s paths ahead of time; requests under either are served by the provider backend’s plain object cache. - A route may not declare a path the provider predefines. An
Exactmatch for/api/v1/query_rangewould take the path from the provider’s handler. The controller refuses that: the provider is withheld from that rule alone, the rest of the policy still applies, and both the route and the policy are told by Event. The root path is not a conflict, since every provider predefines it as a plain proxy catch-all.
The generated backend carries only what the policy and the configured defaults describe:
an origin, a cache, timeouts, headers. An IP access list name, when the backend has one,
comes from kubernetes.defaults.ip_acl_name or the GatewayClass ip_acl_name parameter,
not from this policy. A rate limiter name, when the backend has one, comes from
kubernetes.defaults.rate_limiter_name or the GatewayClass rate_limiter_name parameter,
the same way. This policy cannot set either name or clear it. Provider settings with no policy field — a
Prometheus instant_round, an InfluxDB flux block, a Graphite render section — take
their defaults. MySQL is served over its own wire protocol rather than HTTP and cannot be
selected.
MoQ relays
provider: moq makes the generated backend a MoQ Transport relay rather
than an HTTP proxy. On an HTTPRoute it relays the WebTransport sessions that arrive over
HTTP/3; on a UDPRoute served by a trickstercache.org/moq listener, which makes a relay
of the route with or without a policy, it relays native QUIC sessions. A policy on a
Service makes a relay of every HTTPRoute rule that reaches only MoQ origins. See
kubernetes-gateway.md for both kinds of listener.
A relay applies four fields: cacheName, the cache that groups leaving its in-memory
live edge are kept in; maxTTL, how long a kept group lives at most (the backend default of
25 hours unless set); negativeCacheName, whose 404 entry keeps an origin’s
DOES_NOT_EXIST refusal of a track (see MoQ caching); and
loadBalancing, how a rule with several backendRefs places each track on one of them
(rr, p2c, lc or hrw, hrw unless set). The others shape HTTP requests and
responses, which a session has none of: handler, timeout,
collapsedForwarding, cacheKeyParams, cacheKeyHeaders, requestHeaders,
responseHeaders, cors, healthMode, loadBalancingKey, sticky, resultHeader and
stepAlignment. Each one set is reported by Event, and the rest of the policy applies.
Cache key components
A cached object is stored under a key derived from the backend, the path and, for a
caching path, whatever cacheKeyParams and cacheKeyHeaders name; see
paths.md. The policy’s components reach every caching
path the route serves, the provider’s predefined paths included, where they join the
provider’s own. Name a header whose value partitions the cache — a tenant, an
authorization scope — and every value gets an object of its own.
Exposing the result header
Every Trickster response carries X-Trickster-Result, which says how the request was
handled (trickster-result.md). It is a debugging aid, and on a
gateway in front of the public it discloses which paths are cached and how. resultHeader: Hide withholds it from the client on every path the policy governs. Metrics and the
access log still record the result: the value is kept for the access log’s
%{cache-status}x and %{engine}x fields, so an operator sees what the client does not;
hand-written configuration does the same with the hide_result_header path option
(paths.md). The disposition is
the serving path’s: behind a weighted rule that hides the header, a member whose Service
policy says Expose exposes it, and a response the dispatch answered itself, with no
member selected, follows the rule’s.
Shared caches
The controller’s generated backends use whatever kubernetes.defaults.cache_name, a
GatewayClass’s parameters, or a policy names, and the default in-memory cache when nothing
does. Memory is right for one replica and for objects that are cheap to fetch again; it is
per replica, so two replicas behind one Service each fetch and store their own copy, and a
restart starts cold.
For several replicas, or for time series that are expensive to backfill, configure a Redis cache (caches.md) and name it:
caches:
shared:
provider: redis
redis:
endpoint: redis.trickster.svc:6379
kubernetes:
defaults:
routing_mode: service
cache_name: shared
Every replica then reads and writes one store, a request served by any replica warms the
cache for all of them, and a rolling restart keeps what was cached. A policy may still
select a different configured cache for the routes it governs, so a shared store for time
series and memory for everything else is one cacheName on one policy.
Cache keys are stable across restarts and replicas. They are derived from the Kubernetes object’s identity rather than the rule’s position, so inserting a rule does not invalidate what its neighbors cached. Renaming a route or moving it between namespaces changes the key; editing its rules does not.
Status and Events
The controller writes one status.ancestors entry per targetRefs entry, under its own
controllerName, with an Accepted condition:
reason | meaning |
|---|---|
Accepted | the policy governs the target |
Conflicted | an older policy governs the target; the message names it |
TargetNotFound | the target is not in a watched namespace |
Invalid | the target’s kind, group or sectionName is not supported, the target is named twice, or the policy as a whole cannot be honored; the message says which |
Status is written by the replica holding the leader election Lease, as every other status
is (kubernetes-gateway.md), and by none under
read_only. Entries other controllers wrote are kept. The same verdicts, and a provider
withheld from a rule, are Warning Events on the policy, visible in kubectl describe trickstercachepolicy; the withheld provider is an Event on the route as well.
A policy’s target is judged to exist when the object is in a watched namespace, whether or
not this controller claims it: a policy on an HTTPRoute attached to another controller’s
Gateway is Accepted and has no effect.
10.5 - Kubernetes Controller RBAC
Every permission the Kubernetes Gateway/Ingress controller uses, and why. This is the source the Helm chart’s Role and ClusterRole are generated from; a verb that does not appear here is one the controller does not use.
The controller is entirely absent unless the top-level kubernetes
configuration section is present, so a Trickster deployment that is not a
gateway needs none of this.
Watched resources
The controller reads these to build its routing model. It never writes them
themselves; the get verbs on the four statused kinds exist only to re-read
an object whose status write was refused as a conflict, and a read-only
instance does not need them.
| API group | Resource | Verbs | Scope | Why |
|---|---|---|---|---|
gateway.networking.k8s.io | gatewayclasses | get, list, watch | Cluster | Deciding which classes name this controller; get re-reads one whose status write conflicted |
gateway.networking.k8s.io | gateways | get, list, watch | Namespaced | Listeners, hostnames, TLS references; get as above |
gateway.networking.k8s.io | httproutes | get, list, watch | Namespaced | Route matches and backend references; get as above |
gateway.networking.k8s.io | grpcroutes | get, list, watch | Namespaced | gRPC method matches and backend references; only in a cluster whose Gateway API serves the kind, and get as above |
gateway.networking.k8s.io | tcproutes, tlsroutes, udproutes | get, list, watch | Namespaced | Stream route backend references and, for TLSRoute, server names; only in a cluster whose experimental Gateway API channel (v1alpha2) serves the kind, and get as above |
gateway.networking.k8s.io | referencegrants | list, watch | Namespaced | Permitting cross-namespace Service and Secret references |
gateway.networking.k8s.io | backendtlspolicies | list, watch | Namespaced | TLS to backends; only in a cluster whose Gateway API serves the kind |
gateway.networking.x-k8s.io | xbackendtrafficpolicies | list, watch | Namespaced | Session persistence a Service asks for; only in a cluster that serves the experimental kind |
networking.k8s.io | ingressclasses | list, watch | Cluster | Deciding which classes name this controller, and which is the cluster default |
networking.k8s.io | ingresses | get, list, watch | Namespaced | Hosts, paths, backends, TLS references; get as above |
| (core) | services | list, watch | Namespaced | Resolving backend references to an address and port; also in published_service’s namespace, for its addresses |
| (core) | secrets | list, watch | Namespaced | TLS certificates for HTTPS listeners, and the session token key a GatewayClass’s sticky_secret names |
| (core) | configmaps | list, watch | Namespaced | A GatewayClass’s parametersRef and a BackendTLSPolicy’s CA bundle; only in a cluster serving the Gateway API |
| (core) | namespaces | list, watch | Cluster | When namespace_selector is configured, or the cluster serves the Gateway API |
discovery.k8s.io | endpointslices | list, watch | Namespaced | Only with routing_mode: endpoint: the generated discoverer reads the endpoints of every referenced Service over the controller’s connection |
trickstercache.org | trickstercachepolicies | get, list, watch | Namespaced | Caching policy on Gateways, routes and Services; only in a cluster that serves the resource, and get as above |
The gateway.networking.k8s.io grants are needed only in a cluster that
serves the Gateway API; its CRDs are an add-on. The controller probes for the
group at startup and, where it is absent, watches none of its kinds and serves
Ingress objects alone. Granting the verbs for resources the cluster does not
define is harmless, so one Role covers both cases; a cluster that will never
install the CRDs can leave them out.
secrets are read through two watches, each narrowed server-side: one to
type=kubernetes.io/tls, and, in a cluster that serves the Gateway API, one
to Secrets labeled trickstercache.org/sticky-key. The controller therefore never holds an
application’s credentials in memory. The grant
itself cannot express that narrowing, which is the strongest reason to scope
the controller to named namespaces where that is possible.
namespaces is requested when namespace_selector is configured, and in any
cluster that serves the Gateway API, because a Gateway listener may admit
routes by the labels of their namespace (allowedRoutes.namespaces.from: Selector) and only the Namespace object carries those labels. An Ingress-only
controller using watch_namespaces avoids both the cluster-wide namespace
read and the cluster-scoped Role binding for it.
configmaps is requested only in a cluster that serves the Gateway API, for
the ConfigMap a GatewayClass’s parametersRef may name and the CA bundle a
BackendTLSPolicy’s caCertificateRefs may name. It is watched in the same
namespaces as Services, so a reference into a namespace outside
watch_namespaces is reported as not found.
backendtlspolicies graduated to the Gateway API’s v1 later than the core
kinds, so a cluster may serve the group without it. The controller reads the
served resources at startup and builds the informer only where the kind is
served; a policy on a cluster without it is simply never seen.
xbackendtrafficpolicies is in the Gateway API’s experimental
gateway.networking.x-k8s.io group, installed by its experimental channel.
Before building its informer, the controller asks the API server whether it
may list and watch the kind in every watched namespace. It asks with a
SelfSubjectAccessReview, which Kubernetes’ default system:basic-user role
lets every authenticated identity create, so that needs no grant here. Where
the answer is no, it
logs a warning and never reads the kind, rather than waiting on an informer
that would never sync. A review that fails is treated as a refusal. An
upgrade that keeps an older Role therefore starts, and the kind’s
sessionPersistence is ignored until the grant is added and the controller
restarts, since it asks only at startup.
trickstercachepolicies is this project’s own custom resource
(kubernetes-cache-policy.md), installed from
deploy/kube/crds. The controller probes for it at startup exactly as it
probes for the Gateway API and watches it only where the cluster serves it,
so the grant is harmless on a cluster without the definition.
EndpointSlices are not watched by the controller itself: endpoint churn is
the autodiscovery provider’s job and reaches the data plane through pool
mutation rather than a configuration reload. In the endpoint routing mode
the controller generates a discoverer over its own connection, so the
service account then also needs the endpointslices grant above in every
namespace whose Services routes reference, exactly as a hand-configured
discoverer would; see the RBAC section of
alb-autodiscovery.md. In the service routing
mode it is not needed.
Cluster writes
Status and Events are the only writes the controller makes; everything above
is read. Setting read_only: true gives up all of them, and the grant below
along with them, for an instance whose service account cannot or should not
write to the cluster. It still watches, translates and serves traffic —
read-only describes the relationship with the cluster, not the data plane.
| API group | Resource | Verbs | Scope | Why |
|---|---|---|---|---|
gateway.networking.k8s.io | gatewayclasses/status | update | Cluster | Accepted condition on claimed classes |
gateway.networking.k8s.io | gateways/status | update | Namespaced | Accepted and Programmed conditions, listener conditions, and published addresses |
gateway.networking.k8s.io | httproutes/status | update | Namespaced | Per-parent Accepted and ResolvedRefs conditions |
gateway.networking.k8s.io | grpcroutes/status | update | Namespaced | The same, for GRPCRoutes |
gateway.networking.k8s.io | tcproutes/status, tlsroutes/status, udproutes/status | update | Namespaced | The same, for the stream route kinds where the cluster serves them |
networking.k8s.io | ingresses/status | update | Namespaced | status.loadBalancer addresses |
trickstercache.org | trickstercachepolicies/status | update | Namespaced | Per-target Accepted conditions |
| (core) | events | create, patch | Namespaced | Translation errors, rejected annotations, certificate failures, class acceptance; patch is how client-go counts a repeated Event |
Leader election
Required only when leader_election.enabled is true (the default) and the
instance is not read-only: the election decides which replica writes status
and Events, so an instance that writes neither contends for nothing. Every
replica programs its own data plane regardless, so a replica that loses an
election still serves traffic.
| API group | Resource | Verbs | Scope | Why |
|---|---|---|---|---|
coordination.k8s.io | leases | get, create, update | Namespaced | The election itself, in leader_election.namespace |
Address publishing
Required only when published_service is configured: the controller watches
that one Service (list, watch on services, narrowed by a field selector
to its name) in the Service’s own namespace, which need not be one of the
watched namespaces, and publishes its assigned addresses into Gateway and
Ingress status.
11 - Observability
11.1 - Trickster Metrics
Trickster exposes a Prometheus /metrics endpoint with a customizable listener port number (default is 8481). For more information on customizing the metrics configuration, see configuring.md.
The following metrics are available for polling with any Trickster configuration:
trickster_build_info(Gauge) - This gauge is always 1 when Trickster is running- labels:
goversion- the version of go under which the running Trickster binary was builtrevision- the commit ID on which the running Trickster binary was builtversion- semantic version of the running Trickster binary
- labels:
trickster_config_last_reload_successful(Gauge) - The value is 1 when true (the last config reload was successful) or 0 when falsetrickster_config_last_reload_success_time_seconds(Gauge) - Epoch timestamp of the last successful configuration reloadtrickster_frontend_requests_total(Counter) - Count of front end requests handled by Trickster- labels:
backend_name- the name of the configured backend handling the proxy requestprovider- the type of the configured backend handling the proxy requestmethod- the HTTP Method of the proxied requesthttp_status- The HTTP response code provided by the backendpath- the Path portion of the requested URL
- labels:
trickster_frontend_requests_duration_seconds(Histogram) - Histogram of front end request durations handled by Trickster. ACONNECT, whose duration is that of the tunnel or WebTransport session it opens, is not observed- labels:
backend_name- the name of the configured backend handling the proxy requestprovider- the type of the configured backend handling the proxy requestmethod- the HTTP Method of the proxied requesthttp_status- The HTTP response code provided by the backendpath- the Path portion of the requested URL
- labels:
trickster_frontend_written_byte_total(Counter) - Count of bytes written in front end requests handled by Trickster- labels:
backend_name- the name of the configured backend handling the proxy requestprovider- the type of the configured backend handling the proxy requestmethod- the HTTP Method of the proxied requesthttp_status- The HTTP response code provided by the backendpath- the Path portion of the requested URL
- labels:
trickster_proxy_requests_total(Counter) - The total number of requests Trickster has handled.- labels:
backend_name- the name of the configured backend handling the proxy requestprovider- the type of the configured backend handling the proxy requestmethod- the HTTP Method of the proxied requestcache_status- status codes are described herehttp_status- The HTTP response code provided by the backendpath- the Path portion of the requested URL
- labels:
trickster_proxy_upstream_retries_total(Counter) - The number of upstream requests retried under a path’sretrypolicy.- labels:
backend_name- the name of the configured backend handling the proxy requestpath- the configured path whose policy retried the request
- labels:
trickster_proxy_mirror_requests_total(Counter) - The number of requests copied to a path’smirrorbackend.- labels:
backend_name- the name of the configured backend handling the proxy requestmirror_backend- the backend receiving the copiesresult-sent, ordroppedwhen the mirror’s in-flight bound was reached
- labels:
trickster_proxy_sigv4_events_total(Counter) - The number of SigV4 signing events on backends with asigv4block.- labels:
backend_name- the name of the configured backend whose request was signedevent-sign_failurewhen a request could not be signed and was not sent, orcredentials_retrywhen the origin rejected expiring credentials and the request was resent with refreshed ones
- labels:
trickster_proxy_origin_srv_lookups_total(Counter) - The number of SRV lookups made by backends using SRV origin resolution.- labels:
backend_name- the name of the configured backend that dialed its originresult-hit,miss,stale(a failed refresh served the last good answer),negative(a cached NXDOMAIN, empty answer or failure) orerror
- labels:
trickster_proxy_origin_srv_dial_attempts_total(Counter) - The number of dial attempts to SRV targets by backends using SRV origin resolution.- labels:
backend_name- the name of the configured backend that dialed its origintier- the target’s priority tier, where0is the preferred tierresult-successorfailure
- labels:
trickster_proxy_truncated_responses_total(Counter) - The number of time series fetches the origin truncated at its series limit, which Trickster proxies rather than caches. Only backends whose flavor declares a series limit report it.- labels:
backend_name- the name of the configured backend whose fetch was truncated
- labels:
trickster_accesslog_dropped_lines_total(Counter) - The number of access and error log lines dropped because the log could not accept them.- labels:
backend_name- the name of the configured backend whose logger dropped the linelog-accessorerror
- labels:
trickster_ip_acl_decisions_total(Counter) - Decisions made by an attached IP access list. See ip-acl.md.- labels:
ip_acl- the name of the access listscope-listener,backend, orpathverdict-allowordeny.rejectanddropare bothdeny
- labels:
trickster_ratelimit_decisions_total(Counter) - Decisions made by an attached rate limiter. See rate-limiting.md.- labels:
limiter- the name of the rate limiterplane-http,streamormoqresult-allowed,limited,counted,exempt, orfull
- labels:
trickster_ratelimit_keys(Gauge) - Buckets a rate limiter holds at scrape time.- labels:
limiter- the name of the rate limiter
- labels:
trickster_proxy_points_total(Counter) - The total number of data points Trickster has handled.- labels:
backend_name- the name of the configured backend handling the proxy requestprovider- the type of the configured backend handling the proxy requestcache_status- status codes are described herepath- the Path portion of the requested URL
- labels:
trickster_proxy_timeseries_offgrid_extents_total(Counter) - The number of time series fetch ranges whose bounds fell between buckets and were narrowed to whole buckets before fetching. A non-zero value points to a step or phase mismatch between Trickster and the origin; please report it.- labels:
backend_name- the name of the configured backend fetching the rangeprovider- the type of the configured backend, or the native protocol (e.g.,mysql)
- labels:
trickster_proxy_partial_bucket_fetches_total(Counter) - The number of partial bucket fetches through the object cache, under apartialstep alignment mode.- labels:
backend_name- the name of the configured backendprovider- the backend provideredge-startorend: the edge of the requested range the bucket sits onstatus- the object cache result, such ashitorkmiss, orerrwhen the fetch failed and the bucket was left out of the response
- labels:
trickster_step_alignment_fallbacks_total(Counter) - The number of requests for a step alignment mode the query doesn’t support, served in the query’s default mode instead.- labels:
backend_name- the name of the configured backendrequested- the mode asked for, by the backend’s configuration, an ALB, or atrickster-step-aligndirectiveapplied- the mode the request was served in
- labels:
trickster_geo_acl_decisions_total(Counter) - The number of clients a geo ACL judged.- labels:
geo_acl- the name of the geo ACLplane-http,nativeorstream: the kind of listener the client was judged onverdict-allow,deny,count(denied, but the geo ACL only counts) orexempt(allowed with no lookup), astrickster_ip_acl_decisions_totalnames its verdicts
- labels:
trickster_geo_locator_lookups_total(Counter) - The number of lookups a geo locator answered. A rising rate ofnot_foundcan mean the listener’strusted_proxiesis wrong, so every client resolves to a proxy’s private address.- labels:
geo_locator- the name of the geo locatorresult-found,not_foundorerror
- labels:
trickster_geo_locator_reloads_total(Counter) - The number of replaced data files a geo locator loaded or refused, keeping the last good one.- labels:
geo_locator- the name of the geo locatorresult-successorerror
- labels:
trickster_geo_locator_build_timestamp_seconds(Gauge) - When the database anmmdbgeo locator has loaded was built, in seconds since the epoch.- labels:
geo_locator- the name of the geo locator
- labels:
trickster_proxy_request_duration_seconds(Histogram) - Time required to proxy a given Prometheus query.- labels:
backend_name- the name of the configured backend handling the proxy requestprovider- the type of the configured backend handling the proxy requestmethod- the HTTP Method of the proxied requestcache_status- status codes are described herehttp_status- The HTTP response code provided by the backendpath- the Path portion of the requested URL
- labels:
trickster_proxy_max_connections(Gauge) - Trickster max number of allowed concurrent connectionstrickster_proxy_active_connections(Gauge) - Trickster number of concurrent connectionstrickster_proxy_requested_connections_total(Counter) - Trickster total number of connections requested by clients.trickster_proxy_accepted_connections_total(Counter) - Trickster total number of accepted client connections.trickster_proxy_closed_connections_total(Counter) - Trickster total number of administratively closed client connections.trickster_proxy_failed_connections_total(Counter) - Trickster total number of failed client connections.trickster_proxy_stream_connections_total(Counter) - The number of connections and UDP sessions accepted bytcp,tlsandudplisteners.- labels:
listener_name- the name of the configured listenerprotocol-tcp,tlsorudpresult-proxied, or why the connection was closed instead:not_tls(atlslistener received no ClientHello),no_route(no backend routes the server name),no_upstream(the backend’s pool has no dialable member, or the member chosen refuses its share),dial_failed,refused(audplistener at its session limit, or a connection arriving as the listener closes), ordenied(turned away by the listener’s admission control before anything was relayed)
- labels:
trickster_proxy_stream_active_connections(Gauge) - The number of connections and UDP sessions stream listeners are relaying.- labels:
listener_name- the name of the configured listenerprotocol-tcp,tlsorudp
- labels:
trickster_proxy_stream_dropped_datagrams_total(Counter) - The number of datagramsudplisteners dropped rather than relayed.- labels:
listener_name- the name of the configured listenerreason-queue_full(the client’s flow, or every flow together, already held its allowance of datagrams waiting to be written),write_timeout(the write to the backend blocked for the whole write bound) ordenied(turned away by the listener’s admission control)
- labels:
trickster_proxy_stream_bytes_total(Counter) - The bytes relayed by stream listeners.- labels:
listener_name- the name of the configured listenerprotocol-tcp,tlsorudpdirection-infrom the client to the backend,outfrom the backend to the client
- labels:
trickster_proxy_stream_member_connections_total(Counter) - The number of connections and UDP sessions a stream listener committed to an ALB pool member.- labels:
listener_name- the name of the configured listenerprotocol-tcp,tlsorudpbackend_name- the name of the pool member backendresult-proxied,dial_failed(the member could not be connected to) orunreachable(audpmember answered a datagram with a port-unreachable)
- labels:
trickster_proxy_stream_member_active_connections(Gauge) - The number of connections and UDP sessions open to an ALB pool member.- labels:
listener_name- the name of the configured listenerprotocol-tcp,tlsorudpbackend_name- the name of the pool member backend
- labels:
trickster_proxy_stream_member_connect_duration_seconds(Histogram) - The time taken to connect to an ALB pool member.- labels:
listener_name- the name of the configured listenerprotocol-tcp,tlsorudpbackend_name- the name of the pool member backend
- labels:
trickster_proxy_query_range_rejected_total(Counter) - Trickster total number of queries rejected due to exceeding themax_query_rangelimit.- labels:
backend- the name of the configured backend rejecting the query
- labels:
trickster_graphite_resolution_lookups_total(Counter) - Count of Graphite step-resolution lookups. Labels never include a metric path or target expression.- labels:
backend_name- the name of the configured Graphite backendconfidence- how the step was established:exact(read from an origin response for this leaf set and age),derived(computed from known leaf ladders),configured(fromstatic_retentions, not yet probe-confirmed), orunknown(no usable step; the request is served unaccelerated)source- where it came from:registry,response,probe,static,function, ornone
- labels:
trickster_graphite_probes_total(Counter) - Count of synthetic requests issued to learn a metric’s archive ladder. Expect a spike at startup that collapses toward zero as ladders are learned.- labels:
backend_name- the name of the configured Graphite backendkind-narrow(a one-second window that also discovers the retention edge),wide(what a real query at that age receives), orfind(a/metrics/expandlookup)result-step(a stepped series came back),empty(no series: beyond retention, or no such metric), orerror
- labels:
trickster_graphite_ladders(Gauge) - Number of distinct archive ladders known to the resolution registry. Ladders come fromstorage-schemas.confpatterns, so this should flatten at a small number.- labels:
backend_name- the name of the configured Graphite backend
- labels:
trickster_graphite_registry_entries(Gauge) - Number of entries in each layer of the resolution registry.- labels:
backend_name- the name of the configured Graphite backendlayer-leaf(metric path to ladder),ladder(the ladders themselves),target(cached wildcard expansions), ornegative(paths in resolution backoff)
- labels:
trickster_graphite_step_mispredictions_total(Counter) - Count of origin responses whose step differed from the predicted step. This should always be zero. A non-zero value means a cached ladder was wrong; Trickster discards the prediction, relearns and re-serves the request unaccelerated, so clients still receive correct data.- labels:
backend_name- the name of the configured Graphite backend
- labels:
trickster_graphite_fallbacks_total(Counter) - Count of render requests served without delta caching. Labels never include a target expression.- labels:
backend_name- the name of the configured Graphite backendreason-parse_error,non_series_format,function_not_allowlisted,unknown_step,missing_target,mixed_steps,multi_target_step_mismatch,passthrough_max_data_points,misprediction,client_identity,tz_unavailable, orresolution_identity
- labels:
trickster_sql_query_analysis_total(Counter) - Count of SQL query cache-eligibility classifications. Labels never include query text.- labels:
backend_name- the name of the configured backend analyzing the querydialect- the SQL dialect of the analyzing backend (e.g.,clickhouse)cache_mode- the strongest cache mode supported by the query (delta,object, ornone)reason- the stable classification reason code (e.g.,delta_cacheable,unsafe_predicate,unsupported_bucket)
- labels:
trickster_sql_query_rewrite_failures_total(Counter) - Count of SQL cache-miss extent rewrite failures. Labels never include query text.- labels:
backend_name- the name of the configured backend rendering the querydialect- the SQL dialect of the rendering backendreason- the fixed internal failure category
- labels:
trickster_druid_query_analysis_total(Counter) - Count of native Druid query cache-eligibility classifications. Labels never include query text or datasource names.- labels:
backend_name- the configured Druid backendcache_mode-delta,object, orproxyreason- the stable classification reason code
- labels:
trickster_victoriametrics_query_analysis_total(Counter) - Count of MetricsQL API request cache-eligibility classifications. Labels never include query text.- labels:
backend_name- the configured VictoriaMetrics backendcache_mode-delta,object, orproxyreason- the stable classification reason code
- labels:
trickster_druid_query_rewrite_failures_total(Counter) - Count of Druid cache-miss extent rewrite failures.- labels:
backend_name- the configured Druid backendreason- the fixed internal failure category
- labels:
trickster_cache_operation_objects_total(Counter) - The total number of objects upon which the Trickster cache has operated.- labels:
cache_name- the name of the configured cache performing the operationprovider- the type of the configured cache performing the operationoperation- the name of the operation being performed (read, write, etc.)status- the result of the operation being performed
- labels:
trickster_cache_operation_duration_seconds(Histogram) - The time, in seconds, required to perform an operation on the Trickster cache. Deletions include both requested removals and index reaper evictions.- labels:
cache_name- the name of the configured cache performing the operationprovider- the type of the configured cache performing the operationoperation- the name of the operation being performed (get,set,setDirect,del)status- the result of the operation being performed (e.g.,hit,kmissfor a full key miss,none)
- labels:
trickster_cache_operation_bytes_total(Counter) - The total number of bytes upon which the Trickster cache has operated. Deletions (del) record bytes only for cache providers that use an index, since other providers don’t track object sizes.- labels:
cache_name- the name of the configured cache performing the operationprovider- the type of the configured cache performing the operationoperation- the name of the operation being performed (read, write, etc.)status- the result of the operation being performed
- labels:
trickster_alb_pool_admits_failing(Gauge) - 1 when an ALB pool’shealthy_flooradmits members in theunavailablestate, 0 otherwise. See alb.md for the recommended floor.- labels:
backend_name- the name of the configured ALB backend
- labels:
trickster_alb_pool_floor_reset(Gauge) - 1 when an ALB pool’shealthy_floorwas reset to 0 at startup because pool members have no health check and could never reach the configured floor, 0 otherwise. See alb.md.trickster_alb_pool_on_backup(Gauge) - 1 while an ALB pool that hasbackupmembers is dispatching to them because no other member is available, 0 otherwise. Present only for pools with backup members. See alb.md.- labels:
backend_name- the name of the configured ALB backend
- labels:
trickster_alb_member_inflight(Gauge) - Current number of requests in flight to an ALB pool member. Exported for the mechanisms that track it (p2c,lc,lt), for requests and for stream connections and sessions alike; read when the metrics endpoint is scraped, at no cost to request routing.- labels:
alb_name- the name of the configured ALB backendmember- the name of the pool member backend
- labels:
trickster_alb_member_draining(Gauge) - 1 for each ALB pool member that is draining: markeddrain: true, or discovered while terminating but still serving. It keeps its sticky sessions and takes no new work. Members that are not draining have no series; read when the metrics endpoint is scraped.- labels:
alb_name- the name of the configured ALB backendmember- the name of the pool member backend
- labels:
trickster_alb_member_ejections_total(Counter) - The number of timesalb.stream.passive_healthtook a pool member out of selection after repeated connect failures.- labels:
alb_name- the name of the configured ALB backendmember- the name of the pool member backend
- labels:
trickster_alb_sticky_total(Counter) - The number of requests, stream connections and native sessions through an ALB with sticky sessions, by how their session fared. Each is counted once it reaches its member or is refused, and not at all when it reaches no member; see Sticky Session Metrics.- labels:
alb_name- the name of the configured ALB backendresult-hit(sent to the member its session is pinned to),miss(no token or table entry),expired(a token past itsttloridle; an expired table entry is amiss),invalid(a token that is altered, signed with another key or issued by another ALB),repick(its member was unavailable and the session moved) orrejected(its member was unavailable andon_unavailable: rejectrefused it)
- labels:
trickster_alb_sticky_entries(Gauge) - The number of entries in the table of an ALB that keeps sticky sessions in a table, on any listener, expired entries not yet removed included; read when the metrics endpoint is scraped.- labels:
alb_name- the name of the configured ALB backend
- labels:
The following metrics are available for ALBs that use TSM global evaluation. Queries answered from the ALB’s cache aren’t evaluated, so they aren’t counted.
trickster_alb_global_evaluations_total(Counter) - The number of TSM global evaluations- labels:
alb_name- the name of the configured ALB backendstatus-success,partial(evaluated without an unavailable member), or the Prometheus API error type that answered the query:bad_data,execution,timeout,canceled,unavailableorinternal
- labels:
trickster_alb_global_evaluation_duration_seconds(Histogram) - The time TSM global evaluations take, from arrival to response, including raw series retrieval and any wait for an evaluation slot- labels:
alb_name- the name of the configured ALB backend
- labels:
trickster_alb_global_fetched_series_total(Counter) - The number of raw series TSM global evaluations read from pool members or the raw series cache, before deduplication- labels:
alb_name- the name of the configured ALB backend
- labels:
trickster_alb_global_fetched_samples_total(Counter) - The number of raw samples TSM global evaluations read from pool members or the raw series cache, before deduplication- labels:
alb_name- the name of the configured ALB backend
- labels:
trickster_alb_global_member_failures_total(Counter) - The number of raw series reads from a pool member that failed during TSM global evaluation- labels:
alb_name- the name of the configured ALB backendmember- the name of the pool member backend
- labels:
trickster_alb_global_raw_reads_total(Counter) - The number of raw series reads TSM global evaluations made of pool members, by how the raw series cache answered them- labels:
alb_name- the name of the configured ALB backendstatus-hitwhen only samples within the volatile window were read from the member,phitwhen older samples the cache lacked were read too,kmisswhen the cache held none, andproxy-onlywhen every sample was within the volatile window, or the read couldn’t be cached
- labels:
trickster_alb_global_raw_subsumed_total(Counter) - The number of raw series reads TSM global evaluations answered in part from the cached raw series of a broader selector on the same metric- labels:
alb_name- the name of the configured ALB backend
- labels:
The following metrics are available when ALB Autodiscovery is configured:
trickster_alb_discovery_members(Gauge) - Current number of discovered ALB pool members- labels:
alb_name- the name of the discovery-backed ALB backenddiscoverer- the name of the discoverer serving the ALB
- labels:
trickster_alb_discovery_member_changes_total(Counter) - Count of discovered pool member additions and removals- labels:
alb_name- the name of the discovery-backed ALB backenddiscoverer- the name of the discoverer serving the ALBevent-addorremove
- labels:
trickster_alb_discovery_snapshots_total(Counter) - Count of membership snapshots processed, by result- labels:
alb_name- the name of the discovery-backed ALB backenddiscoverer- the name of the discoverer serving the ALBresult-applied(membership updated),unchanged(no-op),rejected(guardrail-refused, e.g.min_members), orpartial(applied with member instantiation failures)
- labels:
trickster_alb_discovery_last_refresh_success_time_seconds(Gauge) - Epoch timestamp of the last successfully processed snapshot, for staleness alerting- labels:
alb_name- the name of the discovery-backed ALB backenddiscoverer- the name of the discoverer serving the ALB
- labels:
trickster_discovery_refresh_errors_total(Counter) - Count of provider-side refresh/watch errors (DNS resolution failures, Kubernetes list/sync failures, member-file read/parse failures)- labels:
discoverer- the name of the discoverer experiencing the errorprovider- the discoverer’s provider type
- labels:
trickster_tls_certificate_expiration_time_seconds(Gauge) - NotAfter time of a serving TLS certificate, as unix seconds. See tls.md.- labels:
listener- the name of the listener serving the certificateentry- the certificate’s source identity
- labels:
trickster_tls_certificate_last_load_time_seconds(Gauge) - Epoch timestamp a serving TLS certificate was last loaded from its source- labels:
listener- the name of the listener serving the certificateentry- the certificate’s source identity
- labels:
trickster_tls_certificate_swaps_total(Counter) - Count of TLS certificates hot-swapped into a live listener by rotation detection- labels:
listener- the name of the listener serving the certificateentry- the certificate’s source identity
- labels:
trickster_tls_certificate_validation_failures_total(Counter) - Count of detected TLS certificate source changes that failed pair validation (e.g. a mid-rotation partial write) and were not swapped in- labels:
entry- the certificate’s source identity
- labels:
trickster_tls_watcher_errors_total(Counter) - Count of errors reading watched TLS certificate source files- labels:
entry- the certificate’s source identity
- labels:
trickster_tls_certificate_store_size(Gauge) - Number of certificates in a listener’s TLS certificate store- labels:
listener- the name of the listener
- labels:
trickster_acme_orders_total(Counter) - Count of first-time ACME certificate orders. See acme.md.- labels:
issuer- the name of the ACME issuerresult-successorfailure
- labels:
trickster_acme_renewals_total(Counter) - Count of ACME certificate renewals- labels:
issuer- the name of the ACME issuerresult-successorfailure
- labels:
trickster_acme_challenge_requests_total(Counter) - Count of ACME challenge requests received by listeners- labels:
type-http-01ortls-alpn-01result-served,unknown(no matching pending challenge) orerror
- labels:
trickster_acme_on_demand_decisions_total(Counter) - Count of on-demand issuance decisions- labels:
result-allowed,refused,refused_cached,rate_limitedorask_error
- labels:
trickster_acme_startup_wait_seconds(Gauge) - Seconds startup readiness was held waiting for missing ACME certificatestrickster_acme_startup_wait_timeouts_total(Counter) - Count of startup waits that timed out before every ACME certificate was issued
The following metrics are available only for Caches Types whose object lifecycle Trickster manages internally (Memory, Filesystem and bbolt):
trickster_cache_events_total(Counter) - The total number of events that change the Trickster cache, such as retention policy evictions.- labels:
cache_name- the name of the configured cache experiencing the event$provider- the type of the configured cache experiencing the eventevent- the name of the event being performedreason- the reason the event occurred
- events and reasons:
eventreasoncounts evictionttlremovals of expired objects, a batch at a time evictionsize_bytes,size_objectsremovals of least-recently-accessed objects from a cache over its size evictionfree_spaceremovals of least-recently-accessed objects for min_free_bytes(Filesystem)invalid_objectexpired,corrupt,unknown_formatobjects found to be unservable when read or swept, and removed (Filesystem and bbolt) indexcompactionreplacements of the Cache Index journal by a snapshot (Filesystem and bbolt) indexsweepcompleted sweeps of the cache (Filesystem and bbolt) sweepadoptedobjects a sweep found in the cache that the Cache Index did not list sweepdroppedobjects a sweep found the Cache Index to list that the cache no longer held
- labels:
trickster_cache_usage_objects(Gauge) - The current count of objects in the Trickster cache.- labels:
cache_name- the name of the configured cache$provider- the type of the configured cache$
- labels:
trickster_cache_usage_bytes(Gauge) - The current count of bytes in the Trickster cache.- labels:
cache_name- the name of the configured cache$provider- the type of the configured cache$
- labels:
trickster_cache_max_usage_objects(Gauge) - The maximum allowed size of the Trickster cache in objects.- labels:
cache_name- the name of the configured cache$provider- the type of the configured cache
- labels:
trickster_cache_max_usage_bytes(Gauge) - The maximum allowed size of the Trickster cache in bytes.- labels:
cache_name- the name of the configured cache$provider- the type of the configured cache
- labels:
The following metrics are available for Static File Server Backends. Requests they serve are also counted, like those of any other Backend, by the trickster_frontend_requests_* metrics with a provider of static. Their Fileserver cache is separate from the caches above, and is not reported by the trickster_cache_* metrics.
trickster_fileserver_responses_total(Counter) - The total number of files served, by how the Fileserver cache figured in the response. Responses that send no file (such as a404with no not-found file, a redirect or a directory listing) are not counted.- labels:
backend_name- the name of the configured backendcache_status-hit(served as it was held),phit(the file was held, and was encoded for the response and the rendition then held),kmiss(read from disk for the response, and then held) ordisk(sent from disk without being held, as for a large file, a byte range, aHEADor a304)encoding- the encoding of the rendition the file server sent:identity,zstd,br,gzipordeflate. A response counted asidentitymay still be encoded on its way out, as a large compressible file is.
- labels:
trickster_fileserver_cache_events_total(Counter) - The total number of objects removed from the Fileserver cache.- labels:
backend_name- the name of the configured backendevent-eviction(the least recently used, removed to make room) orinvalidation(removed because the file changed on disk, or the cache was stopped)
- labels:
A backend’s series are published only once it is in service, so a configuration that is rejected publishes nothing. They are deleted when a reload removes or renames the backend; across a reload that keeps its name, the counters carry on rather than start over. The four gauges that follow are published only while the backend has a Fileserver cache, and are removed when it is disabled.
trickster_fileserver_cache_usage_objects(Gauge) - The current count of objects in the Fileserver cache, including files being read into it. Each held rendition of a file is an object.- labels:
backend_name- the name of the configured backend
- labels:
trickster_fileserver_cache_usage_bytes(Gauge) - The current accounted size of the Fileserver cache in bytes, which includes each object’s bookkeeping allowance.- labels:
backend_name- the name of the configured backend
- labels:
trickster_fileserver_cache_max_usage_objects(Gauge) - The configuredmax_filesof the Fileserver cache.- labels:
backend_name- the name of the configured backend
- labels:
trickster_fileserver_cache_max_usage_bytes(Gauge) - The configuredmax_size_bytesof the Fileserver cache.- labels:
backend_name- the name of the configured backend
- labels:
The following metrics are available when the Kubernetes Gateway/Ingress controller is enabled (the top-level kubernetes section; see kubernetes-gateway.md):
trickster_moq_sessions_total(Counter) - Count of MoQ Transport sessions by outcome- labels:
listener_name- the listener the session arrived onbackend_name- the backend it was routed to, empty when it was refused before routingrole-downstreamtransport-quicorwebtransportdraft- the negotiated draft, empty when none was negotiatedresult-ok,refused,no_route,busyorfailed
- labels:
trickster_moq_active_sessions(Gauge) - Current MoQ Transport sessions- labels:
listener_name- the listener serving the sessionsbackend_name- the backend they were routed torole-downstream
- labels:
trickster_moq_requests_total(Counter) - Count of MoQ Transport requests by type and result- labels:
backend_name- the moq backendtype-subscribe,fetch,track_status,publish,publish_namespace,subscribe_namespace,subscribe_tracksorotherresult-okorrefused
- labels:
trickster_moq_active_subscriptions(Gauge) - Current MoQ Transport subscriptions- labels:
backend_name- the moq backenddirection-upstream(the relay’s subscriptions toward origins and publishers) ordownstream
- labels:
trickster_moq_objects_total(Counter) - Count of MoQ Transport objects by direction and delivery- labels:
backend_name- the moq backenddirection-inoroutdelivery-stream,datagramorfetch
- labels:
trickster_moq_bytes_total(Counter) - Count of MoQ Transport object payload and property bytes- labels:
backend_name- the moq backenddirection-inorout
- labels:
trickster_moq_dropped_objects_total(Counter) - Count of MoQ Transport objects not forwarded- labels:
backend_name- the moq backendreason-oversize,delivery_timeout,datagram_queue_fullorspill_queue_full
- labels:
trickster_moq_fetch_objects_total(Counter) - Count of objects served to FETCH and fills, by where they came from- labels:
backend_name- the moq backendsource-live_edge,cacheorupstream
- labels:
trickster_moq_live_edge_bytes(Gauge) - Current payload bytes held in a moq backend’s in-memory live edge- labels:
backend_name- the moq backend
- labels:
trickster_moq_publishers_active(Gauge) - CurrentPUBLISHsubscriptions publishers hold open to a moq backend- labels:
backend_name- the moq backend
- labels:
trickster_moq_errors_total(Counter) - Count of MoQ Transport failures- labels:
backend_name- the moq backendclass-malformed_trackorcache
- labels:
trickster_kgw_reconciles_total(Counter) - Count of controller reconcile passes, by result- labels:
result-applied(the data plane was reloaded),unchanged(the generated configuration was already running), orerror(the pass failed to compile or apply)
- labels:
trickster_kgw_reconcile_duration_seconds(Histogram) - Duration of a whole reconcile pass, from reading the caches to writing statustrickster_kgw_reconcile_errors_total(Counter) - Count of reconcile passes that failed in a stage- labels:
stage-compile,apply,certificates(a certificate a listener’s store refused) orstatus(a status write the API server refused)
- labels:
trickster_kgw_watch_events_total(Counter) - Count of Kubernetes watch events received by the controller- labels:
kind- the object kind (Gateway,HTTPRoute,Ingress,Secret,Service, …)event-add,updateordelete
- labels:
trickster_kgw_translate_duration_seconds(Histogram) - Duration of translating the watched objects into the routing modeltrickster_kgw_apply_duration_seconds(Histogram) - Duration of applying generated configuration to the data plane, observed only on passes that reload ittrickster_kgw_generated_objects(Gauge) - Size of the routing model the last pass produced- labels:
kind-listeners,routes,backends,certificatesorpolicies
- labels:
trickster_kgw_status_write_failures_total(Counter) - Count of status writes the API server refused, after conflict retries- labels:
kind- the kind of the object whose status could not be written
- labels:
trickster_kgw_leader(Gauge) - 1 when this replica holds the leader election Lease and so writes status and Events, 0 otherwisetrickster_kgw_last_successful_sync_time_seconds(Gauge) - Epoch timestamp of the last reconcile pass whose generated configuration is running, for staleness alertingtrickster_kgw_route_info(Gauge) - A constant 1 per generated backend, joining it to the Kubernetes object it serves- labels:
kind-HTTPRouteorIngressroute- the object’s namenamespace- the object’s namespacebackend_name- the generated backend’s name, as thebackend_namelabel of the request metrics carries it
For example, request rates per Kubernetes route:
sum by (kind, namespace, route) ( rate(trickster_proxy_requests_total[5m]) * on (backend_name) group_left (kind, namespace, route) trickster_kgw_route_info )- labels:
In addition to these custom metrics, Trickster also exposes the standard Prometheus metrics that are part of the client_golang metrics instrumentation package, including memory and cpu utilization, etc.
11.2 - Distributed Tracing via OpenTelemetry
Trickster instruments Distributed Tracing with OpenTelemetry. We import the OpenTelemetry golang packages to instrument support for tracing.
As OpenTelemetry evolves to support additional exporter formats, we will work to extend Trickster to support those as quickly as possible. We also make a best effort to update our otel package imports to the latest releases, whenever we publish a new Trickster release. You can check the go.mod file to see which release of opentelemetry-go we are is using. In this view, to see which version of otel a specific Trickster release imports, use the branch selector dropdown to switch to the tag corresponding to that version of Trickster.
Supported Tracing Backends
- Jaeger (via OTLP)
- Console/Stdout (printed locally by the Trickster process)
Trickster’s OTLP exporter supports OTLP over HTTP and gRPC. HTTP is the default
for existing OTLP configs. For Jaeger over OTLP/HTTP, configure Trickster with
the collector’s HTTP endpoint, for example http://jaeger:4318/v1/traces. For
Jaeger over OTLP/gRPC, set protocol: grpc and use the collector’s gRPC
endpoint, for example http://jaeger:4317 from another Compose service or
http://127.0.0.1:4317 from the host when using the developer Compose
environment.
Configuration
Trickster allows the operator to configure multiple tracing configurations, which can be associated into each Backend configuration by name.
The example config has exhaustive examples of configuring Trickster for distributed tracing.
For OTLP tracing, protocol selects the exporter transport. Supported values
are http and grpc; when omitted, Trickster uses http for backward
compatibility with existing configs.
Context Propagation
When tracing is enabled for a Backend, Trickster uses the W3C Trace Context and Baggage propagators. It extracts incoming traceparent, tracestate, and baggage headers from client requests and injects the active outbound origin request span into the proxied request. This lets downstream Origins continue the same distributed trace instead of starting an unrelated trace.
Sampling
Trickster uses parent-based sampling for traced requests. The configured sample_rate controls root traces that Trickster starts when there is no sampled upstream trace context. When an incoming request already has a remote parent trace, Trickster follows the parent’s sampling decision so sampled upstream traces continue across the proxy boundary and unsampled upstream traces remain unsampled.
Span List
Trickster can insert several spans to the traces that it captures, depending upon the type and cacheability of the inbound client request, as described in the table below.
| Span Name | Observes when Trickster is: |
|---|---|
| request | initially handling the client request by a Backend |
| QueryCache | querying the cache for an object |
| WriteCache | writing an object to the cache |
| DeltaProxyCacheRequest | handling a Time Series-based client request |
| FetchFastForward | making a Fast Forward request for time series data |
| FetchTimeSeries | retrieving time series data from an Origin |
| FetchRange | retrieving one sharded time range from an Origin |
| Fetch | retrieving one object or range from an Origin |
| FetchRevalidation | revalidating a stale cache object against its Origin |
| ProxyRequest | communicating with an Origin server to fulfill a client request |
| PrepareFetchReader | preparing a client response from a cached or Origin response |
| CacheRevalidation | revalidating a stale cache object against its Origin |
| FetchObject | retrieving a non-time-series object from an Origin |
Tags / Attributes
Trickster supports adding custom tags to every span via the configuration. Depending upon your preferred tracing backend, these may be referred to as attributes. See the example config for examples of adding custom attributes.
Trickster also supports omitting any tags that Trickster inserts by default. The list of default tags are below. For example on the “request” span, an http.url tag is attached with the current full URL. In deployments where that tag may introduce too much cardinality in your backend trace storage system, you may wish to omit that tag and rely on the more concise path tag. Each tracer config can be provided a string list of tags to omit from traces.
Attributes added to request and core proxy/cache/fetch spans
backend.namebackend.providercache.namecache.providerrouter.path- request path trimmed to the route match path for the request (e.g.,/api/v1/query), good for aggregating when there are large variations in the full URL pathrouter.handler
These resource attributes are attached when the corresponding backend, cache, or route configuration is available to the request flow.
Attributes added to top level (request) span
http.url- the full HTTP request URL
Attributes added to cache/proxy spans
cache.status- the lookup or proxy cache status where available. See the cache status reference for a description of the attribute values.
Attributes added to proxy/fetch spans
http.status_code- the HTTP status returned by the Origin or generated proxy response, when available.
Attributes added to the FetchRevalidation span
isRange- is true if the client request includes an HTTPRangeheader
Attributes added to the FetchObject span
isPCF- is true if the origin is configured for Progressive Collapsed Forwarding
11.3 - Access and Error Logs
Trickster can write HTTP access logs and error logs, per backend or for the
whole process, with customizable formats, rotation and retention. Both logs
are off by default; each is enabled by configuring its filename, which may be
a file path or the stdout or stderr stream.
Basic Configuration
backends:
example1:
provider: rp
origin_url: http://example.com/
access_log:
filename: /var/log/trickster/example1.access.log
error_filename: /var/log/trickster/example1.error.log
- The access log receives one line per request handled by the backend and is
written only when
filenameis set. An HTTP listener access list whose action isdropaborts the request withhttp.ErrAbortHandlerinside the access-log middleware’snext.ServeHTTP, before that completion line is written, so the dropped request has no access-log line.rejectwrites the list’s HTTP status and is logged on this path. See ip-acl.md. - The error log receives one line per request whose response status is at or
above
error_threshold(default400) and is written only whenerror_filenameis set. An error-logged request also appears in the access log when both are configured. A listener access-listrejectwhose status is at or above that threshold is included. Adropwrites no status, so it is not an error-log line either. - Two backends may share a filename; they will safely share the underlying file and its rotation.
- When
instance_idis set in the main config, it is inserted into log filenames just as with the application log (e.g.,example1.access.1.log).
Default Access Log
A top-level access_log section applies to every backend that does not define its own access_log, and also captures requests that no backend route handled: router 404s and the built-in ping, readiness, health and management endpoints. Those unmatched lines report - for the backend and provider. Requests on the metrics listener are never access-logged.
access_log:
filename: stdout
format: json
backends:
api:
provider: rp
origin_url: http://api:8080/
quiet:
provider: rp
origin_url: http://quiet:8080/
access_log: {}
A backend’s own access_log replaces the default entirely rather than merging with it, so the empty block on quiet disables access logging for that backend. Lines for a backend carry its name and provider whichever configuration produced them, and a request is logged exactly once even when both the backend and the default write to the same target.
Logging to Standard Output
filename and error_filename accept the special values stdout and stderr in place of a path, for container platforms that collect logs from the process streams. A stream is never rotated or pruned, ignores rotation, retention, compress and instance_id, and may be shared by any number of backends and the default access log regardless of their other settings. Lines are still buffered for up to one second before being written.
access_log:
filename: stdout
error_filename: stderr
Log Format
The format option accepts either a named preset or a custom format string
using Apache-style % tokens, so the well-known conventions from Apache
HTTP Server, Apache Traffic Server, Lighttpd, and similar servers apply
directly.
Presets
| Name | Description |
|---|---|
common | NCSA Common Log Format: %h %l %u %t "%r" %>s %b |
combined | Apache/Nginx Combined format: common + Referer and User-Agent. This is the default. |
extended | combined + duration (ms), cache status and backend name |
json | One JSON object per line with a fixed field set (see below) |
Custom Formats
access_log:
filename: /var/log/trickster/example1.access.log
format: '%h %u %t "%r" %>s %b %{ms}T %{cache-status}x'
Supported tokens:
| Token | Description |
|---|---|
%h, %a | client IP address, resolved through the listener’s trusted_proxies when configured |
%{c}a | IP address of the connection peer, which is the proxy when one is trusted |
%l | remote logname (always -) |
%u | authenticated username (from HTTP Basic Auth), else - |
%t | request start time in CLF format: [26/Aug/2026:10:30:00 +0000] |
%{sec}t, %{msec}t, %{usec}t | request start time as a Unix epoch value |
%{LAYOUT}t | request start time in a custom Go time layout |
%r | first line of the request: GET /path?query HTTP/1.1 |
%m | request method |
%U | request URL path |
%q | query string, prefixed with ?, or empty when none |
%H | request protocol (e.g., HTTP/1.1) |
%s, %>s | response status code |
%b | response body bytes, or - when zero (CLF style) |
%B | response body bytes, numeric |
%D | request duration in microseconds |
%T | request duration in whole seconds |
%{us}T, %{ms}T, %{s}T | request duration in the given unit |
%{Name}i | request header value |
%{Name}o | response header value |
%{Name}c | request cookie value |
%v | requested virtual host |
%p | listener port that served the request |
%A | listener IP address that served the request |
%% | a literal % |
Trickster-specific values use the %{key}x extension namespace:
| Token | Description |
|---|---|
%{backend}x | backend name |
%{provider}x | backend provider type |
%{cache-status}x | cache result (hit, phit, kmiss, …); see Cache Status |
%{engine}x | proxy engine that handled the request (e.g., DeltaProxyCache) |
%{path-config}x | the matched path config path |
%{upstream-addr}x | host:port of the origin the request was proxied to, or - when none was contacted |
%{upstream-status}x | status code the origin answered with |
%{upstream-duration}x | duration of the origin exchange in milliseconds |
%{trace-id}x, %{span-id}x | the request’s trace and span identifiers when tracing is enabled |
%{request-id}x | the request’s X-Request-ID; see below |
%{key}e | a static value declared under extra; see below |
Missing values render as -. Values derived from the request (like headers
and usernames) are backslash-escaped so they cannot corrupt the log line
structure. Unknown tokens fail validation at startup.
The json preset emits these fields per line: time, client_ip,
remote_ip, user, method, path, query, proto, status, bytes,
duration_ms, host, referer, user_agent, backend, provider,
path_config, cache_status, engine, upstream_addr, upstream_status,
upstream_duration_ms, trace_id, span_id, request_id, and an extra
object holding the declared extra values when there are any.
Client Address
%h and %a are the client’s address as Trickster resolved it. Without
trusted_proxies on the listener that is the address of the connection
peer. With it, a connection from a trusted proxy is attributed to the
nearest address in Forwarded or X-Forwarded-For that is not itself a
trusted proxy, or to X-Real-IP when neither is present, so a log line
names the client behind a load balancer rather than the balancer. %{c}a
is always the connection peer. See Trusted Proxies
for the listener settings.
Request ID
A format that logs %{request-id}x gives every request an identifier: the
X-Request-ID header the client sent, or a random 128-bit hexadecimal
value assigned on arrival. The identifier is set on the request, so the
origin receives it in X-Request-ID, and echoed on the response, so a
client can quote it. A format that does not log it assigns none.
Extra Values
extra declares static values for a backend’s log lines, rendered by
%{key}e and emitted as the json preset’s extra object:
backends:
example:
access_log:
filename: /var/log/trickster/example.access.log
format: '%h %t "%r" %>s %b %{team}e'
extra:
team: payments
A key may not contain %, { or }. A token naming an undeclared key
renders -. The Kubernetes controller declares route_kind,
route_namespace and route_name on every backend it generates, so a line
names the Ingress, HTTPRoute or GRPCRoute it served.
Credential Redaction
Trickster masks credentials before they reach a log line. The masked value
is REDACTED, and names stay visible.
- Query parameters: values of
password,passwd,pass,key,api_key,apikey,access_token,token,code,state,client_secret,signatureandsig, matched without regard to case. This covers%r,%q, thejsonpreset’squeryfield, and the query of aRefererorLocationheader. - Headers: values of
Authorization,Proxy-Authorization,Cookie,Set-Cookie,X-Api-Key,Api-Key,X-Goog-Api-KeyandX-Amz-Security-Token, logged through%{Name}ior%{Name}o. - Cookies:
%{name}cmasks only the cookies named in configuration; none are masked by default.
The same query and URL redaction applies to the URLs and upstream errors in Trickster’s application log, including its debug lines.
logging.redact adds names to each list, or turns redaction off entirely:
logging:
redact:
query_params: [session_id]
headers: [X-Internal-Token]
cookies: [grafana_session]
# enabled: false # redaction is on unless explicitly disabled
Dropped Lines
A log line the writer cannot accept, because its bounded buffer is full or
the file cannot be opened, is dropped rather than blocking the request, and
counted in the trickster_accesslog_dropped_lines_total metric by backend
and log (access or error).
Rotation and Retention
Access and error logs are rotated and pruned automatically, using
nginx/logrotate-style numbered archives (example1.access.log.1.gz is the
most recent archive, .2.gz the next, and so on).
access_log:
filename: /var/log/trickster/example1.access.log
rotation:
size: 256MB # rotate when the live file would exceed this size (default 256MB)
interval: 1d # also rotate when the live file is older than this (default off)
retention:
count: 3 # keep at most 3 archives (default 80)
age: 7d # also prune archives older than this (default 7d)
compress: true # gzip archives (default true)
sizeandintervalmay be combined; the log rotates when either threshold is reached. Setting both to0disables rotation.- Sizes accept
KB,MB,GBandTBsuffixes (binary multiples), or a plain byte count. retention.count: 0disables count-based pruning and keeps all archives.- Writes are buffered for up to one second or 64 KiB. A process or machine crash can lose the buffered tail; an orderly shutdown flushes it.
- Interval rotation keeps its epoch in a
<filename>.rotationsidecar so a restart does not reset the interval clock.
Archives created by older Trickster releases use timestamped lumberjack names and are not included in numbered-archive retention. They form a bounded legacy set and may be removed manually after upgrading.
When upgrading from the original logging implementation, note these filename and retention changes:
retention.count: 0now keeps all archives; configure a positive count to bound archive retention.- With
main.instance_idenabled, filenames without a.logsuffix now also include the instance ID (trickster.outbecomestrickster.2.out). Update log shippers that still follow the unsuffixed filename.
The same rotation, retention and compress options are also available
in the main logging: config section to control rotation of the Trickster
application log, with the same defaults.
Error Log Settings
Each error_* option inherits its value from the corresponding access log
option when unset:
access_log:
filename: /var/log/trickster/example1.access.log
format: combined
error_filename: /var/log/trickster/example1.error.log
error_format: '' # default: inherits format
error_threshold: 400 # log responses with status >= this (default 400)
error_rotation: # default: inherits rotation
size: 64MB
error_retention: # default: inherits retention
count: 7
error_compress: true # default: inherits compress
11.4 - X-Trickster-Result Header
Trickster adds the X-Trickster-Result response header to describe how it handled a request. The header is intended for debugging cache behavior, proxy fallbacks, and partial origin fetches. A path may withhold it from the client with hide_result_header: true (paths.md); the access log and metrics still record the result.
Example:
X-Trickster-Result: engine=DeltaProxyCache; status=phit; fetched=[1612804980000-1612808580000]; ffstatus=hit
The header value is a semicolon-separated list of fields. Optional fields are included only when Trickster has a value for them.
| Field | Description |
|---|---|
engine | The proxy engine that handled the response, such as HTTPProxy, ObjectProxyCache, or DeltaProxyCache. |
status | The cache or proxy result. See Cache Status. |
fetched | Time ranges fetched from the origin to satisfy the response. Ranges are formatted as start-end; multiple ranges are separated by semicolons inside the brackets. |
ffstatus | Fast Forward cache result for time series requests. Possible values are hit, kmiss, off, or err. |
partial_buckets | The partial buckets fetched for a time series response under a partial step alignment mode, each as start-end:edge:status, separated by semicolons inside the brackets. |
failed | Time ranges that Trickster attempted to fetch but could not fetch successfully. This usually appears with proxy-error or partial fanout failures. |
Result Statuses
status uses the same values reported in metrics, logs, and tracing. Common examples are:
| Status | Meaning |
|---|---|
hit | The response was served fully from cache. |
phit | Part of the response was served from cache and part was fetched from the origin. |
kmiss | Trickster had no object for the cache key and fetched the response from the origin. |
rmiss | Trickster had an object for the cache key, but not for the requested range. |
rhit | Trickster revalidated a stale cached object against the origin and served it as a hit. |
nchit | The response was served from the Negative Cache. |
purge | The cache key was purged as directed by a request or response header. |
proxy-hit | The request joined an in-flight origin fetch for the same cache key. |
proxy-only | The request was proxied to the origin without writing or reading a cache object. |
proxy-error | An upstream request needed for the response returned an error. |
error | Trickster encountered a cache lookup or cache handling error. |
Proxy-Only Results
status=proxy-only means the response came from the origin through Trickster’s proxy path, without a cache read or cache write for that request. It does not necessarily mean the origin response was wrong.
Common reasons include:
| Cause | Example |
|---|---|
| Backend or path is configured to bypass caching | provider: reverseproxy or proxy_only: true. |
| Client request prevents caching | A request such as Cache-Control: no-cache can force Trickster to proxy and remove the existing object for that cache key. |
| Origin response is not cacheable | For example, response cache headers do not provide cacheability, or the response includes headers that Trickster treats as not cacheable. |
| Time series request cannot be parsed for delta caching | Trickster may fall back to object proxy cache for compatible requests, or proxy the request directly when it cannot safely cache the query shape. |
| Time series range is outside the retained cache window | Old data may be proxied without caching while newer ranges remain cacheable. |
When investigating proxy-only, check the backend provider, any proxy_only setting, request and response Cache-Control headers, and whether the request shape is supported by the configured backend provider.
Fetched And Failed Ranges
fetched and failed describe the ranges Trickster fetched or failed to fetch while serving the response.
Example:
X-Trickster-Result: engine=DeltaProxyCache; status=phit; fetched=[1612804980000-1612808580000;1612812180000-1612815780000]
For time series responses, range values are Unix timestamps in milliseconds. A phit result with fetched ranges usually means Trickster had some of the requested data cached and fetched the missing ranges from the origin.
failed ranges indicate the origin request for those ranges failed. Depending on the proxy engine and fanout behavior, Trickster may return an error response or a partial response with failure metadata.
Fast Forward Status
ffstatus appears on time series responses when the Delta Proxy Cache checks Fast Forward data:
| Fast Forward Status | Meaning |
|---|---|
hit | Fast Forward data was served from cache. |
kmiss | Fast Forward data was fetched from the origin. |
off | Fast Forward was not attempted for this request. |
err | Fast Forward was attempted but failed or returned unusable data. |
Fast Forward is only relevant for supported time series backends and only when the request is eligible for the latest datapoint optimization.
Partial Buckets
partial_buckets appears on time series responses whose step alignment mode fetched partial buckets at the edges of the requested range:
X-Trickster-Result: engine=DeltaProxyCache; status=hit; partial_buckets=[1612804950000-1612804980000:start:hit;1612808580000-1612808595000:end:kmiss]
Each entry is one partial bucket:
| Part | Meaning |
|---|---|
start-end | The bucket’s fetched range, in Unix milliseconds, half-open. For a range with no end, end is the time of the request. |
| edge | start or end: the edge of the requested range the bucket sits on. |
| status | The object cache result for the bucket, such as hit or kmiss, or err when the fetch failed and the bucket was left out of the response. |
Partial bucket fetches don’t change the request’s status: a response whose complete buckets all came from cache is a hit, whatever its partial buckets needed. Under an ALB that merges its members’ responses, entries for the same range and edge are merged, and differing statuses become phit.
12 - Release Notes
12.1 - Trickster 2.2
Trickster 2.2 extends many of the new features introduced in 2.1, while adding support for delta-caching several new TSDB’s that use the Postgres Query Dialect, as well as a new Global Query Evaluator that works with mixed Prom, Thanos and Mimir deployments. 2.2 adds multiple security and enforcement features, including ACME automatic cert renewal, IP ACLs, Rate Limiting and GeoACLs; and new CDN-grade enhancements including MoQ support.
Load Balancing and Scaling
L4 Load Balancing - We’ve extended the HTTP L7 ALB to support Layer 4 as well (TCP and UDP), with a number of new balancing mechanisms.
Sticky Sessions - We now support configurable sticky sessions for the Load Balancer feature.
New Acceleration-supported TSDBs
TimescaleDB - You can now accelerate TimescaleDB with the delta proxy cache! If you are tired of playing whack-a-mole with new continuous aggregates to manage performance, Trickster can stop the madness. Even better - any Postgres-compatible database can be fronted by Trickster for a SELECT result cache.
GreptimeDB - We’ve added GreptimeDB as an acceleration-supported backend time series provider.
QuestDB - And we also now support accelerating QuestDB.
VictoriaMetrics - And we also now support accelerating VictoriaMetrics on the MetricsQL and Graphite HTTP endpoints.
CloudWatch Metrics - We now support accelerating AWS CloudWatch Metrics on the new PromQL endpoints. Classic GetMetricData endpoints are not yet supported.
TSM Global Evaluation - A Time Series Merge ALB can now evaluate each PromQL query once over the raw series of every member, deduplicating replicas the way Thanos does, rather than merging the members’ answers. Members serve raw series through Prometheus remote read (the new mimir flavor) or the Thanos StoreAPI (the new thanos flavor), and complete results are cached, while the members’ raw series are delta-cached, so each member is read only for the samples the cache lacks. Merging also now counts label sets returned by several members once for count or group over an inner count or group. See Global Evaluation.
HTTP Reverse Proxy Cache, CDN and Streaming Media
Media over QUIC (MoQ) - Building on the HTTP/3 and QUIC support introduced in 2.1, Trickster is now a caching Media over QUIC Transport relay, for drafts -17 through -22. Native QUIC clients connect to a protocol: moq listener and browsers connect by WebTransport over HTTP/3. One upstream subscription per track fans out to every subscriber, the newest groups stay in memory and older ones in the cache for FETCH, and publishers can send tracks straight to Trickster. Origin pools, token authorization and the Kubernetes Gateway API are supported too. See Media over QUIC for details.
HTTP/3 on Kubernetes Gateways - HTTPS listeners the Kubernetes controller generates now also serve HTTP/3, and WebTransport, on the same port number over UDP. Turn it off with kubernetes.defaults.http3: false or a GatewayClass’s http3 parameter. See GatewayClass parameters.
HTTP/3 Port Release - A reload that removes or drains an HTTP/3 endpoint now releases its UDP port. In 2.1, the port stayed bound until the process exited, so a later reload could not bind it again.
HTTP QUERY Method - Trickster accepts and caches the QUERY method (RFC 10008), keyed on the request body. Time series query endpoints forward a QUERY to their origins as a POST and advertise Accept-Query. See The QUERY Method.
Automatic Certificates (ACME) - Trickster can now obtain and renew its own serving certificates from Let’s Encrypt or any other ACME certificate authority. See Automatic Certificates for details.
Disk Caches - The Filesystem and bbolt caches are rebuilt for large caches and large objects. See Disk Caches for details.
Chunked Caching Fixes - use_cache_chunking: true is honored from configuration again; since 2.0 it was silently reset to false at load. Chunked byterange caching no longer zeroes stored chunks on a partial hit, stores range misses, retries chunks that were evicted rather than serving their bytes as zeros, and no longer truncates bodies when chunks are read concurrently. A configuration reload that changes a memory cache’s chunking settings now rebuilds the cache. See Chunked Caching.
SRV Origin Resolution - A backend can resolve its origin host as a DNS SRV owner name, dialing a target and port from the SRV answer, with failover across targets and tiers. Combined with a rule and a hostname rewriter, one backend can route to any number of origins published in DNS, including ECS, Consul and Nomad services with dynamically assigned ports. See Origin Resolution via DNS SRV.
Idle Connection Timeouts - HTTP listeners now close a keep-alive connection that waits more than 2 minutes for its next request; until now idle connections were never closed. The new idle_timeout, read_timeout and max_header_bytes listener options tune this. See Connection Timeouts and Header Size.
Security & Policy
Path Normalization - HTTP listeners now remove . and .. path segments before routing and forward the cleaned path, closing a bypass of path-scoped controls such as authenticator_name: none; see Path Normalization for the new path_normalization options and opt-out.
Forwarded Hops From Trusted Proxies Only - Trickster now forwards the Forwarded, X-Forwarded-* and X-Real-IP values a request arrived with only when one of the listener’s trusted_proxies delivered it, appending its own hop; from any other peer the origin receives Trickster’s hop alone, so a client cannot hand the origin a forged address. Passthrough paths, which forwarded no prior hops before, now forward a trusted proxy’s. A listener behind a load balancer must list it in trusted_proxies for the origin to keep seeing client addresses. See Forwarding Headers to the Origin.
Rate Limiting - A named limiter counts requests, connections, sessions, or datagrams and refuses what exceeds its limit. It attaches at the listener, backend, and path. The listener limiter is its own layer. See Rate Limiting.
IP ACLs - We’ve also added IP Access Control Lists to restrict access to certain backend resources by IP. it can be attached at the listener, backend, and path levels, with most specific winning.
Geo ACLs - We now support Geolocation Access Control lists to restrict content to geographical areas through IP -> Location translation via industry-standard locator services (MaxMind, RFC 8805 geofeeds, Header from trusted downstreams). Bring your own licensed database.
Fixes
Log Level Names Ignore Case - log_level, TRK_LOG_LEVEL and -log-level now accept a level name in any case. Until now, an uppercase name such as DEBUG passed config validation but logged at info, and a config with no logging block warned that its own default level was unknown. See Environment Variables and Command Line Arguments.
Rule Inputs Read the Requested Host - A rule’s url, url_no_params, scheme, host, hostname and port inputs now describe the URL the client requested, taking the host from the Host header and the scheme from whether the request arrived over TLS. Until now they read only the request URL, which carries no host or scheme on a client request, so rules matching on these inputs never matched. port now also infers 80 or 443 from the scheme when the host names no port, as documented. A request rewriter’s host, hostname and port replace instructions likewise now act on the requested host. See input_source permitted values.
12.2 - Trickster 2.1
Trickster 2.1 adds tons of new features to put acceleration in even more places: Kubernetes routing, more supported backend providers, and modern protocols. 👌
The next release of Trickster will be v2.2, which has been added to the Roadmap.
A Brand New Trick: Mecone v1.0
Alongside Trickster v2.1, we’ve launched a new companion project, Mecone (pronounced like McConey) - a Reverse Proxy conformance tester that is extensible via YAML configs. We use Mecone to measure Trickster’s conformance to the HTTP protocol specifications, and also compare its conformance against other industry solutions via published quarterly reports. Check out the Mecone repo to see how Trickster stacks up today in our maiden State of the Industry report.
Trickster v2.1 Features
Request Routing
Kubernetes Gateway API and Ingress Controller - Trickster now runs as a Kubernetes controller, serving
gateway.networking.k8s.ioGatewayClass, Gateway, HTTPRoute, GRPCRoute, TCPRoute, TLSRoute, UDPRoute, ReferenceGrant and BackendTLSPolicy objects, andnetworking.k8s.io/v1Ingress objects, translating them into its own configuration and reloading onto it in-process. Routes may be served through the Service’s cluster IP or load balanced across discovered endpoints, TLS certificates arrive from Kubernetes Secrets without a reload, and status, conditions and Events are written back to every claimed object. Enable it with the new top-levelkubernetessection. See kubernetes-gateway.md, kubernetes-ingress.md, kubernetes-rbac.md and kubernetes-deploy.md.- TricksterCachePolicy Custom Resource - A custom resource that attaches caching behavior — a cache, TTLs, cache key components, CORS, header updates and time series acceleration — to Gateways, HTTPRoutes, Ingresses and Services, so a Prometheus or ClickHouse Service behind a route is accelerated rather than merely proxied. See kubernetes-cache-policy.md.
Auto-discovery - The ALB can now manage pool members through common auto-discovery mechanisms such as Kubernetes APIs, DNS A and SRV records, etc.
Regex Path Matching - You can now define path routes with regexes to match incoming requests, and expose their capture groups to request rewriters.
Wildcard Host Routing - A backend’s
hostsmay name a single-label wildcard (*.example.com) or an any-depth wildcard (**.example.com), resolved by specificity ahead of global routes. See Path Configuration Documentation.
Supported Backend Providers
We now support accelerating InfluxDB 3.x, including over Flight SQL (gRPC). Support for InfluxDB 1.x and 2.x remains and is unchanged.
We’ve expanded ClickHouse delta proxy cache support to include the native (TCP/binary) protocol. You can configure a ClickHouse Backend that accepts HTTP and proxies to an origin via Native, or conversely a Backend that accepts Native and proxies via HTTP. A single Backend configuration can also listen on both Native and HTTP.
We now support accelerating Graphite.
We now support accelerating Apache Druid.
We now support accelerating MySQL with Time Series-like queries (e.g., Grafana dashboards).
HTTP Protocol Conformance
HTTP/2 and HTTP/3 - Between the client and Trickster, cleartext listeners now accept prior-knowledge cleartext HTTP/2 (h2c) alongside HTTP/1.1, as gRPC and other h2c clients require; no configuration is required. Between Trickster and the origin, HTTP/2 is negotiated automatically via ALPN with
https://origins that offer it, also with no configuration. Forhttp://origins that serve only h2c, set the newh2c_prior_knowledgebackend option; it is opt-in because there is no HTTP/1.1 fallback for such an origin. Listeners can also now serve their routes over HTTP/3 (QUIC) alongside HTTP/1.1 and HTTP/2. Enable the newhttp3block on any TLS-enabled listener; responses on the TLS endpoint advertise the HTTP/3 endpoint viaAlt-Svcso clients upgrade on their own. See http3.md.Protocol Upgrades - WebSocket and other
Connection: Upgraderequests are now tunneled end-to-end rather than rejected, including through paths that are otherwise cached.Streaming - Responses with an unknown length and Server-Sent Events streams are now flushed to the client as bytes arrive rather than buffered, and HTTP trailers are passed through.
Stream Listeners - Listeners can now relay TCP connections, TLS connections by server name without terminating them, and UDP datagrams to a backend or a load balancer pool, without reading what passes. Set
protocol: tcp,tlsorudpon a listener; see configuring.md. The Kubernetes controller serves TCPRoute, TLSRoute and UDPRoute through them.
Configuration, Observability & Security
Logging - We’ve added support for customizable access logging and error logging per backend in NCSA format, a top-level
access_logthat captures every request no backend handled, and logging to stdout or stderr for containerized deployments. See access-logs.md.Config Files - We now support loading multiple config files in the same subdirectory below the main config. We’ve also added automatic config reloading when the file contents change - including automatic detection and reloading when a TLS certificate is swapped out. See the Configuring Documentation for more info.
TLS Certificate Rotation and Runtime Certificates - Serving certificates renewed in place on disk by tools such as certbot or cert-manager are detected and hot-swapped into the live listener, with no reload and without dropping established connections;
tls_watch_intervaltunes the backstop poll. The newtls_runtime_certslistener option keeps a TLS port open with no certificate files behind it, so certificates can be supplied to the running process instead — which is how the Kubernetes controller serves Gateway and Ingress TLS from Secrets. The mgmt listener exposes a read-only certificate inventory at/trickster/certificates, and certificate expiry, load, swap and validation metrics are available with example alerting rules. See tls.md.Graceful Shutdown and Readiness - A new readiness endpoint (
/trickster/ready) reports whether the process is serving, andmgmt.shutdown_delayandmgmt.shutdown_drain_timeouthold listeners open and then drain in-flight requests on SIGTERM, so a rolling deployment is invisible to clients./trickster/pingremains a liveness check. See configuring.md.Real Client Addresses - Listeners now accept the PROXY protocol, and the new
trusted_proxieslistener option resolves the real client address fromForwarded,X-Forwarded-FororX-Real-IPonly when the connection comes from a trusted address. The resolved address is what the access log records and whatmax_query_rangerejections are logged against. See Trusted Proxies.
Developer Environment
All of the new supported backend time series providers are included in the Developer Environment Docker Compose.
We’ve refactored the Developer Environment Data Seeder job to use 100% locally generated data. No more big S3 file downloads. This also speeds up the GitHub CI/CD integration tests that depend on the dev env. Seed data is generated once as a TSV, and all seedable databases (MySQL, Druid, ClickHouse) mount the same file to load the same data to enable cross-provider testing and verification.
12.3 - Trickster 2.0
An All-New Bag of Tricks
Trickster 2.0 is a near-complete rewrite of the project with performance, durability and extensibility in mind. We’ve made major architectural improvements, added new features and improved performance. Here’s a comprehensive overview of it all:
Features
- Application Load Balancer (ALB): We have a brand new Application Load Balancer available as a backend provider type, with unique and powerful options, including:
- Fanout and merge data from multiple time series backends into a single response
- currently supports Prometheus backends
- Fanout and return the first response received from any backend pool member
- Fanout and return the response with the newest ‘Last-Modified’ header from all backend pool members
- Routing a request to a given backend based on Basic Auth username
- General Round Robin
- Fanout and merge data from multiple time series backends into a single response
- Enhanced Health Checking: Health checking now supports automated health check polling for any backend, and provides a global health status endpoint. Automated health checks for an ALB pool member backend determines whether the ALB will route requests to it or not.
- Cache Object Chunking: We now support Cache Object Chunking. This optional configuration allows a Time Series dataset to be chunked into multiple cache entries based on a configurable chunk size
- Also supported by standard Reverse Proxy Cache for cache-chunking objects by a Byte Range size
- Only the chunked cache entries needed to span the request range are inspected, rather than the entire time series
- Significantly improves Redis and Filesystem performance of large timeseries records
- Time Series Backend Request Sharding: We now support Time Series Backend Request Sharding. This optional feature allows a client request destined for a Time Series Backend to be cloned into multiple concurrent requests with different time ranges constituting the full uncached range
- Backend shard responses + already-cached data are merged into a single downstream response
- Cache Chunking and TS Backend Request Sharding work together seamlessly and can be used in any combination (on/on, on/off, off/on, off/off, including different cache chunk and request shard sizes).
- Authenticator: We’ve added a new Authenticator feature so you can guard backends with Basic Auth or ClickHouse Auth
- Enhanced Rules Engine: The Rules Engine now supports
rmatchoperations to permit regular expression-based routing against any part of the HTTP request - Request Rewriters: You can now chain a collection of request rewriters for more robust request transformation possibilities
- Cache Purging: We now support purging specific cache items by Key (on the public ports) or Path (on the mgmt port). Read more in the cache documentation
- Simulated Latency: You can use Trickster for Simulated Latency in lab environments
- We’ve added support for InfluxDB 2.0 and Flux Query Language, plus InfluxDB 3.x via the native v3 HTTP API (SQL and InfluxQL) and an optional Apache Arrow Flight SQL (gRPC) proxy. See the InfluxDB support doc for details.
Configuration & Security
- YAML Configuration: Trickster 2.0 uses YAML for configuration instead of TOML
- Environment Variable Substitution: Environment Variable substitution is now possible in configuration files where sensitive information is expected
- Supported via the following fields:
caches[*].redis.password,backends[*].healthcheck.headers,backends[*].paths[*].request_headers,backends[*].paths[*].request_params,backends[*].paths[*].response_headers,authenticators[*].users
- Usage:
password: ${MY_SECRET_VAR}
- Supported via the following fields:
- Request Body Size Limit: A configurable Request Body Size limit has been added for
POST,PUTandPATCHrequests, with a default of 10MB. Requests with a body size exceeding the limit will receive a413 Content Too Largeresponse. See Request Body Handling Customizations for more info - Zipkin Deprecation: Trickster uses packages provided by the OpenTelemetry project for Distributed Tracing capabilities. These external packages recently deprecated support for exporting spans via Zipkin, therefore Zipkin support has been removed for Trickster 2.0. Trickster retains support for OTLP.
Under the Hood & Performance
- Common Time Series Format: We now use a common time series format internally for caching all supported TSDB’s, rather than implementing delta + merge algorithms per-provider.
- New HTTP Request Router: We’ve switched to an all-new, made-for-proxies HTTP Request Router, which is up to 10X faster than the previous one
- SQL Parser: We’ve switched from Regular Expression matches for SQL-based Time Series Backends to an extensible lexer/parser solution, providing better performance and accuracy
- ClickHouse backend providers now use the new SQL Parser
- Race Condition Fixes: We’ve eliminated nearly 100 race conditions and random panics
- Expanded Compression Support: Compression support now includes options for Brotli and Zstd
Developer & Environment Improvements
- Package Reorganization: We’ve re-organized many packages in the codebase to be more easily importable by other projects. A future maintenance release will add documentation to the examples folder for using Trickster packages in your own projects, including caching, acceleration and load balancing
- For Trickster contributors, we have a new Docker Compose for developer environments.
- CI/CD Enhancements: We have added new CI tools, including better linters and race condition checkers to enforce and ensure ongoing project quality
- Docker Automation: We’ve updated our Docker automation:
- The trickster-docker-images repo is now retired and image publishing is handled in the trickster repo
- All merges to main will now push an image to Docker Hub at
trickstercache/trickster:mainas well as totrickstercache/trickster:<COMMIT_ID>- We no longer push images to the legacy
tricksterioandtricksterproxyorgs on DockerHub. Everything is now and onlytrickstercache🎉!
- We no longer push images to the legacy
- Images are also now pushed to the GitHub Container Repository as
ghcr.io/trickstercache/trickster
- Helm Charts: The Helm Charts repository is now updated for Trickster 2.0
- Vendor Directory: We no longer include the
vendordirectory in the project repository andvendoris now in.gitignore.vendorwill continue to be included in Release source tarballs
Documentation & Examples
- Example Configurations: Example configurations are relocated to the examples directory
- Docker Compose Demo: The Trickster docker-compose demo has been relocated to the examples directory and updated to use the latest version tags. This is the easiest way to try out Trickster 2.0!
Still to Come
A future Trickster 2.0.x will include the following additional features that didn’t quite make it to the finish line:
- incorporate ALB examples into the docker-compose demo
Installing
You can build the 2.0 binary from the main branch, download binaries from the Releases page, or use the trickstercache/trickster Docker image tag in containerized environments