This is the multi-page printable view of this section. Click here to print.
Backends
1 - Using Multiple Backends with a single Trickster instance
Trickster supports proxying to multiple backends in a single Trickster instance, by examining the inbound request and using a multiplexer to direct the proxied request to the correct upstream origin, in the same way that web servers support virtual hosting.
There are 2 ways to route to multiple backends.
- HTTP Pathing
- DNS Aliasing
Basic Usage
To utilize multiple backends, you must craft a Trickster configuration file to be read when Trickster starts up - operating with environment variables or command line arguments only supports accelerating a single backend. The example.full.yaml provides good documentation and commented sections demonstrating multiple backends. The config file should be placed in /etc/trickster/trickster.yaml unless you specify a different path when starting Trickster with the -config command line argument.
Each backend that your Trickster instance supports must be explicitly enumerated in the configuration file. Trickster does not support open proxying.
Example backend configuration:
backends:
my-backend-01: # routed via http://trickster:8480/my-backend-01/
provider: prometheus
origin_url: http://my-origin-01.example.com
my-backend-02: # routed via http://trickster:8480/my-backend-02/
hosts: [ my-fqdn-02.example.com ] # or http://my-fqdn-02.example.com:8480/
provider: prometheus
origin_url: http://my-origin-02.example.com
my-backend-03:
path_routing_disabled: true # only routable via Host header, an ALB, or a Rule
hosts: [ my-fqdn-03.example.com ] # or http://my-fqdn-03.example.com:8480/
provider: prometheus
origin_url: http://my-origin-02.example.com
Default Backend
Whether proxying to one or more upstreams, Trickster has the concept of a “default” backend, which means it does not require a specific DNS hostname in the request, or a specific URL path, in order to proxy the request to a known backend. When a default backend is configured, if the inbound request does not match any mapped backends by path or FQDN, the request will automatically be routed through the default backend. You are probably familiar with this behavior from when you first tried out Trickster with the using command line arguments.
Here’s an example: if you have Trickster configured with a backend named foo that proxies to http://foo/ and is configured as the default backend, then requesting http://trickster/image.jpg will initiate a proxy request to http://foo/image.jpg, without requiring the path be prefixed with /foo. But requesting to http://trickster/foo/image.jpg would also work.
The default backend can be configured by setting is_default: true for the backend you have elected to make the default. Having a default backend is optional. In a single-backend configuration, Trickster will automatically set the sole backend as is_default: true unless you explicitly set is_default: false in the configuration file. If you have multiple backends, and don’t wish to have a default backend, you can just omit the value for all backends. If you set is_default: true for more than one backend, Trickster will exit with a fatal error on startup.
Path-based Routing Configurations
In this mode, Trickster will use a single FQDN but still map to multiple upstream backends by path. This is the simplest setup and requires the least amount of work. The client will indicate which backend is desired in URL Path for the request.
Example Path-based Configuration with Multiple Backends:
backends:
# backend1 backend
backend1:
origin_url: 'http://prometheus.example.com:9090'
provider: prometheus
cache_name: default
is_default: true
# "foo" backend
foo:
origin_url: 'http://influxdb-foo.example.com:9090'
provider: influxdb
cache_name: default
# "bar" backend
bar:
origin_url: 'http://prometheus-bar.example.com:9090'
provider: prometheus
cache_name: default
Using HTTP Path as the Backend Routing Indicator
The client prefixes the Trickster request path with the Backend Name.
This is the recommended method for integrating with applications like Grafana.
Example Client Request URLs:
To Request from Origin
foo: http://trickster.example.com:8480/foo/query?query=xxxTo Request from Origin
bar: http://trickster.example.com:8480/bar/query?query=xxxTo Request from Origin
backend1as default: http://trickster.example.com:8480/query?query=xxxTo Request from Origin
backend1(Method 2, with Origin Name): http://trickster.example.com:8480/backend1/query?query=xxxConfiguring Grafana to request from backend
foovia Trickster:

DNS Alias Configuration
In this mode, multiple DNS records point to a single Trickster instance. The FQDN used by the client to reach Trickster is mapped to specific backend configurations using the hosts list. In this mode, the URL Path is not considered during Backend Selection.
Host matching is case-insensitive and ignores the port. An entry may also be a single-label wildcard such as *.example.com, which matches foo.example.com but not example.com or a.b.example.com, or an any-depth wildcard such as **.example.com, which matches foo.example.com and a.b.example.com but not example.com; an exact entry on another backend always wins over a wildcard, and a *. entry wins over a **. entry for the same domain. A * anywhere else in an entry is rejected at load time. See Host Resolution for the full precedence order.
Example DNS-based Backend Configuration:
backends:
# backend1 backend
backend1:
hosts: # users can route to this backend via these FQDNs, or via `/backend1`
- 1.example.com
- 2.example.com
origin_url: 'http://prometheus.example.com:9090'
provider: prometheus
cache_name: default
is_default: true
# "foo" backend
foo:
hosts: # users can route to this backend via these FQDNs, or via `/foo`
- trickster-foo.example.com
origin_url: 'http://prometheus-foo.example.com:9090'
provider: prometheus
cache_name: default
# "bar" backend
bar:
hosts: # users can route to this backend via these FQDNs, or via `/bar`
- trickster-bar.example.com
origin_url: 'http://prometheus-bar.example.com:9090'
provider: prometheus
cache_name: default
Example Client Request URLs:
To Request from Origin
foo: http://trickster-foo.example.com:8480/query?query=xxxTo Request from Origin
bar: http://trickster-bar.example.com:8480/query?query=xxxTo Request from Origin
backend1as default: http://trickster.example.com:8480/query?query=xxxTo Request from Origin
backend1(Method 2, via FQDN): http://backend1.example.com:8480/query?query=xxx
Note: It is currently possible to specify the same FQDN in multiple backend configurations. You should not do this (obviously). A future enhancement will cause Trickster to exit fatally upon detection at startup.
Disabling Path-based Routing for a Backend
You may wish for a backend to be inaccessible via the /backend_name/ path, and only by Hostname or as the target of a rule or ALB. You can disable path routing by setting path_routing_disabled: true for the backend, as in this example, which requires the Request’s Host header match 1.example.com or 2.example.com in order to be routed to the backend:
backends:
backend1:
hosts:
- 1.example.com
- 2.example.com
origin_url: 'http://prometheus.example.com:9090'
provider: prometheus
cache_name: default
is_default: false
path_routing_disabled: true # this will disable routing through /backend1
2 - Health Checks
Trickster Service Health - Ping Endpoint
Trickster provides a /trickster/ping endpoint that returns a response of 200 OK and the word pong if Trickster is up and running. The /trickster/ping endpoint does not check any proxy configurations or upstream origins. The path to the Ping endpoint is configurable, see the configuration documentation for more information.
Upstream Connection Health - Backend Health Endpoints
Trickster offers health endpoints for monitoring the health of the Trickster service with respect to its upstream connection to origin servers.
General Endpoint
The main health check path is /trickster/health, which by default will return a text/plain summary of the backend health. You can request YAML or JSON format using the appropriate Accept header, or by providing a ?json or ?yaml query param.
ALBs and Their Pool Members
Each ALB is listed with its mechanism and its pool members, grouped by health. The text form appends them to the ALB’s line as u:[...] (unavailable), a:[...] (available), nc:[...] (not checked) and d:[...] (draining). The JSON and YAML forms name them unavailablePoolMembers, availablePoolMembers, uncheckedPoolMembers, initializingPoolMembers and drainingPoolMembers.
A draining member, one marked drain: true or a discovered one that is terminating but still serving, is listed under its health and again as draining: it still serves the sticky sessions it has, but takes no new work. An ALB with a pool is listed as available while it has a member it would send new work to, so an ALB whose members all drain is listed as unavailable, though it still serves their sessions. For example:
backends:
app1:
provider: reverseproxycache
origin_url: http://app1.example.com
healthcheck:
interval: 5s
app2:
provider: reverseproxycache
origin_url: http://app2.example.com
healthcheck:
interval: 5s
app:
provider: alb
alb:
mechanism: rr
pool:
- app1
- name: app2
drain: true
{
"title": "Trickster Backend Health Status",
"updateTime": "2026-09-26 00:00:00 UTC",
"available": [
{ "name": "app1", "provider": "rpc" },
{ "name": "app2", "provider": "rpc" },
{
"name": "app",
"provider": "alb",
"mechanism": "rr",
"availablePoolMembers": [ "app1", "app2" ],
"drainingPoolMembers": [ "app2" ]
}
]
}
See All-Backends Health Status Page for more about how ALBs are listed.
Backend-Specific Endpoints
Each HTTP backend’s health check path is /trickster/health/BACKEND_NAME. For example, if your backend is named foo, you can perform a health check of the upstream server at http://<trickster_address:port>/trickster/health/foo. Native-protocol backends publish their scheduled status through the general endpoint without registering a synthetic HTTP origin route.
The backend health path prefix /trickster/health/ is customizable. See the example.full.yaml for more info about setting the health_handler_path configuration, or refer to this example:
mgmt:
# this overrides the default '/trickster/health' to '/-/trickster/health'
health_handler_path: /-/trickster/health
The behavior of a health request will vary based on the Backend provider, as each has their own health check protocol. For example, with Prometheus, Trickster makes a request to /query?query=up and (hopefully) receives a 200 OK, while for InfluxDB the request is to /ping which returns a 204 No Content.
Supported TSDB Providers are pre-configured in Trickster to perform a suitable health check operation, however these can be overridden in the configuration file.
For non-TSDB Backends, the default behavior is to make a GET request to http://origin_url:port/ and expect a 2xx response. However, all aspects of the Health Check request and expected response are configurable per-Backend.
Native MySQL Health Checks
The MySQL backend uses the same interval scheduler, timeout, transition
thresholds, status registry, metrics, reload carryover, and shutdown lifecycle
as HTTP health checks. Each probe opens a fresh connection with the backend’s
configured origin credentials and TLS policy, completes authentication,
executes COM_PING, and closes the connection.
backends:
mysql1:
provider: mysql
origin_url: mysql://health-user:password@mysql.example:3306/analytics
healthcheck:
interval: 5s
timeout: 3s
failure_threshold: 3
recovery_threshold: 3
Only interval, timeout, failure_threshold, and recovery_threshold apply
to native probes. HTTP verbs, paths, headers, bodies, and expected HTTP response
options are configuration errors for MySQL. Diagnostics exposed in health
status and logs are limited to sanitized authentication, TLS, timeout,
refused-connection, connection, and server-error categories.
Native PostgreSQL Health Checks
A postgres or timescaledb backend is probed over the PostgreSQL wire
protocol with the same scheduler, timeout, thresholds, status registry and
metrics as every other health check. Each probe opens a fresh connection with
the backend’s origin_url credentials and upstream TLS mode, completes
authentication, waits for the origin’s ReadyForQuery, and ends the session
cleanly. A backend whose origin_url has no username (one that relays its
clients’ own authentication) can only be checked for accepting connections.
backends:
tsdb1:
provider: timescaledb
origin_url: postgres://health-user:password@tsdb.example:5432/analytics
healthcheck:
interval: 5s
timeout: 3s
failure_threshold: 3
recovery_threshold: 3
Only interval, timeout, failure_threshold, and recovery_threshold
apply. Failures are reported as a login rejection with the origin’s SQLSTATE,
a timeout, or a connection failure; user names, addresses and passwords never
appear in health status or logs. A native User Router refuses new sessions
for a user whose backend is failing its health check.
Basic Health Check Configuration Example
backends:
server1:
provider: reverseproxycache
origin_url: http://server1
healthcheck: # all values below are optional
verb: HEAD
path: /health
Health Check With Exhaustive Request/Response Options
backends:
server1:
provider: reverseproxy
origin_url: http://server1
healthcheck: # all values below are optional
#
## customizing the health check request
#
verb: HEAD
scheme: https
host: alternate-hostname.example.com
path: /health
query: param1=value1¶m2=value2
headers:
User-Agent: health-check-agent
# if using a POST or PUT method, you can provide a string body
# body: "my health check body"
#
## customizing the expected response
#
# hc fails if a response takes longer than 1s
timeout: 1000ms
# hc fails if the response code is not in the list
expected_codes: [ 200, 204, 206, 301, 302, 304 ]
#
# hc fails if these response headers are not present and have the expected value
expected_headers:
X-Health-Check-Status: success
# hc fails if the stringified response body does not match the expected value
expected_body: "pass"
See more examples in example.full.yaml.
Health Check Integrations with Application Load Balancers
By default, a Backend will only initiate a health check on-demand, upon receiving a request to its health endpoint.
To facilitate integrations with the Trickster Application Load Balancer provider, additional options provide for 1) timed interval health checks and 2) thresholding for consecutive successful or unsuccessful health checks that determine the backend’s overall health status.
Example Health Check Configuration for use in ALB
backends:
server1:
provider: reverseproxy
origin_url: http://server1
healthcheck:
path: /health
timeout: 1000ms # timeout should be <= interval
# for ALB integration:
interval: 1000ms # auto-poll health every 1s
failure_threshold: 3 # backend is unhealthy after 3 consecutive failures
recovery_threshold: 3 # backend is healthy after 3 consecutive successes
The Prometheus default probe is /api/v1/query?query=up. Some multi-tenant Prometheus gateways reject an unbounded up with 400 bad_data: "too many series found", which keeps the member out of any ALB pool it belongs to. Override healthcheck.query with a bounded expression the backend accepts (for example query=vector(1)) when probing such backends.
Other Ways to Monitor Health
In addition to the out-of-the-box health checks to determine up-or-down status, you may want to setup alarms and thresholds based on the metrics instrumented by Trickster. See metrics.md for collecting performance metrics about Trickster.
3 - Authenticator
Trickster 2.x provides an Authenticator capability that allows you to protect Backends with an Authentication layer.
Authenticator resources are defined globally by name, and then mapped into any Backend and/or Path configuration as needed. Authenticator users can be loaded from htpasswd or csv files, or directly in the Trickster config file. You can provide credentials in plaintext, bcrypt, Apache MD5 ($apr1$), MD5-crypt ($1$), as well as legacy formats sha-256-crypt and sha-512-crypt ($5$, $6$). PostgreSQL SCRAM-SHA-256 and md5 verifiers are accepted too.
A stored hash or verifier is only ever checked as that format, so a client that sends the stored hash itself as its password is rejected. Plaintext credentials are compared in constant time, and an empty stored credential never matches.
Authenticators work with all Backend provider types. Requests are handled by their respective Authenticators before all other Handlers (e.g., Caches, Rules, Request Rewrites, ALB Routes, etc.). Authentication sees the request as the client sent it: backend and path request rewriters run after it, so a rewriter cannot change the path or credentials that authentication judges.
Native MySQL listeners terminate mysql_native_password authentication and
have stricter credential-source requirements than HTTP authenticators. See the
MySQL Provider Guide.
If a request is routed via a Trickster ALB or Rule Backend through to multiple other Backends - each having different Authenticator configurations - the authentication behavior is currently undefined. In an upcoming Beta release, we will define this use case to either use the Authenticator config (if set) of the very first Backend that handled the request; or to use the first defined Authenticator regardless of how deep into the Backend chain it is.
By default, when an Authenticator handles and successfully authenticates a request, the request’s Auth credentials (e.g., Authorization Header for Basic Auth) are stripped before the request is cloned for any necessary proxying. Setting proxy_preserve: true will preserve these headers instead of stripping them.
Path Protection
When you map an Authenticator to a Backend, all Paths defined for that Backend are protected by the Authenticator. However, you can override the Authenticator on a per-Path basis by including the authenticator_name in a Path config. To bypass a Backend-wide Authenticator in a Path config, use authenticator_name: none. This allows for a few possibilities:
A Backend config with a default / Backend-wide Authenticator that has:
- specific Paths that do not require Authentication
- specific Paths mapped to different Authenticators than the default
A Backend config with no Authenticator defined (so all Paths by default are unprotected) that has:
- specific Paths mapped to different Authenticators than the default
See the example Backend configs below for more details.
Authenticator Providers
Trickster’s Authenticator feature currently supports Basic Auth and ClickHouse-compatible authentication. It was designed with extensibility in mind should there be value in adding additional Authentication providers.
Basic Auth Provider
Basic Auth is supported by using provider: basic in the Authenticator config.
By setting the showLoginForm: true config (see example_auth_1 in the blob below), Trickster will return a WWW-Authenticate: Basic realm="custom-realm-name" header on any request that requires but fails authentication, causing the login form to pop up. When showLoginForm is not present or non-true, Trickster responds with a 401 Unauthorized but does not ask the Basic Auth login form to show.
The realm attribute value defaults to the Authenticator name (e.g., example_auth_1) but can be overridden with the realm config as in the example.
If the user data changes (e.g. updated users_file contents or updated embedded users list), you must send a SIGHUP or other means to reload the Trickster config before the new user pool is processed.
ClickHouse Auth Provider
ClickHouse Auth is supported by using provider: clickhouse in the Authenticator config.
ClickHouse authentication is the same as Basic Auth, except you can also provide user and password URL params. Unless proxy_preserve: true is set, those URL params are removed along with the Authorization header before the request is proxied.
Example Authenticator Configs
# NOTE: Required options unrelated to Authenticators have been omitted from this
# example. It does not represent a fully-functioning Trickster Config.
# See the 'examples' directory for working copy/paste config examples.
backends:
backend01:
provider: reverseproxy # authenticators work with all backend providers
authenticator_name: example_auth_1 # protects backend01 with example_auth_1 authenticator
origin_url: https://example.com
paths:
- path: / # all requests are protected by example_auth_1
match_type: prefix
handler: proxy
backend02:
provider: reverseproxy # no backend-wide authenticator
origin_url: https://example.com
paths:
- path: / # requests will be allowed without auth except the 2 Paths below
match_type: prefix
handler: proxy
- path: /private/
authenticator_name: example_auth_2 # example_auth_2 protects this path only
handler: proxy
- path: /admin/
authenticator_name: example_auth_3 # example_auth_3 protects this path only
handler: proxy
backend03:
provider: reverseproxycache
authenticator_name: example_auth_1 # protects backend03 with example_auth_1 authenticator
origin_url: https://example.com
paths:
- path: / # requests will be challenged by example_auth_1 except the 2 Paths below
match_type: prefix
handler: proxy
- path: /public/
authenticator_name: none # requests to /public will be allowed without auth
handler: proxy
- path: /app/admin/
authenticator_name: example_auth_2 # example_auth_2 protects this path, not auth_1
handler: proxy
authenticators:
# example_auth_1 loads users from a CSV and embeds a supplemental plaintext manifest
# It also shows the login form to client browsers when login has failed
example_auth_1:
provider: basic # http basic auth (required)
proxy_preserve: true # don't strip auth headers when proxying this request upstream
users_file: /path/to/user-manifest.csv # optional users source file
users_file_format: csv # required when users_file is set: csv, csvNoHeader or htpasswd
users: # optional embedded users manifest (username: credential)
user1: red123
config: # optional provider-specific configs
showLoginForm: true # with basic auth, causes the browser to show the login form
realm: custom-realm-name # realm would be example_auth_1 if not overridden here
# example_auth_2 loads users from an htpasswd file (assumed bcrypted credentials)
example_auth_2:
provider: basic
users_file: /path/to/user-manifest.htpasswd # optional users source file
users_file_format: htpasswd # required when users_file is set
# example_auth_3 loads users from the embedded users manifest, credentials already bcrypted
example_auth_3:
provider: basic
users:
user1: $2y$asf;j2ihj0h8vabjkwdqbv29hq
# example_auth_4 loads users from the embedded manifest, credentials injected from env,
# and supports ClickHouse query params (user/password) in addition to Basic Auth
example_auth_4:
provider: clickhouse
users:
user1: ${USER1_PASSWORD_ENV} # ${ENV_NAME} substitution is supported
4 - AWS Integration
Trickster uses AWS credentials and SigV4 request signing in two places, and both are configured the same way:
- Signing outbound requests to an origin — the
sigv4block on a backend, below. - Autodiscovery of AWS resources — the
awsdiscovery provider, which reads an AWS API to keep an ALB pool current.
Credentials
Leaving the credential fields empty selects the standard AWS credential chain, in the order the AWS SDK resolves it:
- Environment variables (
AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEY,AWS_SESSION_TOKEN) - The shared credentials and config files (
~/.aws/credentials,~/.aws/config), honoringprofile - Web-identity tokens — this is how EKS IAM Roles for Service Accounts (IRSA) and EKS Pod Identity work
- AWS SSO
- The EC2 instance metadata service (IMDSv2)
Prefer the chain over static keys wherever the platform provides one: on EKS use IRSA or Pod Identity, on EC2 use an instance profile. Static keys are supported for environments that have nothing else.
Credential Lifecycle
Credentials are resolved lazily, on the first signed request. Trickster therefore starts even when the instance metadata service is briefly unreachable. Concurrent first requests share one resolution.
- Failures are retried. A failed resolution is reused for 5 seconds and then tried again, so a momentary metadata or STS failure does not permanently disable signing, and an outage costs one blocked resolution per 5 seconds rather than one per request.
- Expiring credentials refresh early. Temporary credentials (IRSA, Pod
Identity, SSO, instance profiles,
role_arn) are refreshed between 2.5 and 5 minutes before they expire, rather than after, so requests are not signed with credentials that lapse in flight. - Rejected credentials are renewed once. When the origin answers 403
with
ExpiredToken,ExpiredTokenException,InvalidClientTokenId,UnrecognizedClientExceptionorInvalidSignatureException, Trickster renews the credentials and resends the request once:- Credentials that expire (IRSA, Pod Identity, SSO, instance profiles,
role_arn) are discarded and resolved again. Concurrent rejections of the same credentials cause a single refresh. - Credentials read from the shared credentials or config file, including temporary keys written there by a refresher, carry no expiry, so the files are read again instead, at most once every 5 seconds. A rewritten file therefore takes effect without a reload.
access_keyandsecret_keyset in thesigv4block, and keys from environment variables, cannot change while Trickster runs, so they are never retried.
- Credentials that expire (IRSA, Pod Identity, SSO, instance profiles,
- One cache per backend. A backend’s proxy and health check requests share one set of credentials.
- Clock skew. SigV4 signatures carry a timestamp, and AWS rejects
requests more than 5 minutes off its clock with
InvalidSignatureException. Keep the host clock synchronized.
Signing failures and credential retries are counted in
trickster_proxy_sigv4_events_total (see metrics) and
logged as warnings, at most once a minute per backend and kind.
Region
region may be set explicitly. When it is not, it is resolved from
AWS_REGION / AWS_DEFAULT_REGION, the shared config file, or instance
metadata. If none of those yields one, requests fail with an error naming
every source that was tried.
The sigv4 Backend Block
sigv4 signs Trickster’s outbound requests to a backend’s origin.
backends:
amp:
provider: prometheus
origin_url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-abc123
sigv4:
region: us-east-1
# credentials omitted: use the chain (IRSA, instance profile, ...)
# access_key: AKIA...
# secret_key: ... # redacted in config dumps and the health page
# profile: production
# role_arn: arn:aws:iam::123456789012:role/TricksterRead
# service: aps # default; see below
| option | meaning |
|---|---|
region | region to sign for; resolved from the environment when unset |
access_key, secret_key | a static credential pair — both or neither |
profile | a profile in the shared config file |
role_arn | a role to assume with whatever the chain resolves first |
service | the AWS service to sign for; defaults to aps |
The service default is aps — Amazon Managed Service for Prometheus.
That is deliberate: earlier releases could sign for nothing else, so every
existing config keeps working unchanged. Set service to sign for a
different AWS service (es for OpenSearch, and so on).
access_key and secret_key must be provided together. A config supplying
only one fails at startup rather than silently falling through to the
chain and authenticating as a different principal.
secret_key is redacted wherever configuration is emitted — the config
dump, the management API, logs, and error messages.
Amazon Managed Service for Prometheus
The common case. Point a prometheus backend at the workspace’s query URL
and add a sigv4 block; Trickster caches and accelerates AMP queries as it
does any other Prometheus origin.
backends:
amp:
provider: prometheus
origin_url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-abc123
sigv4:
region: us-east-1
The IAM principal needs aps:QueryMetrics, aps:GetSeries,
aps:GetLabels, and aps:GetMetricMetadata on the workspace.
Adding prometheus.flavor: amp restricts the backend to AMP’s read APIs
(query, query_range, series, labels, label/<name>/values,
metadata, rules and alerts). Every other path, including
/api/v1/remote_write, is answered by Trickster with an error instead of
being signed and forwarded. The flavor also adds the sigv4 block if it is
missing.
backends:
amp:
provider: prometheus
origin_url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-abc123
prometheus:
flavor: amp
sigv4:
region: us-east-1
Amazon CloudWatch PromQL
CloudWatch serves PromQL over a Prometheus-compatible API at
https://monitoring.<region>.amazonaws.com, covering metrics ingested
through OTLP and enriched AWS vended metrics. Queries are billed by the
samples they scan, so a dashboard that refreshes repeatedly pays again for
every sample it has already seen. Trickster’s delta cache fetches only the
new part of each refresh, and request collapsing merges concurrent viewers
into one upstream query.
backends:
cloudwatch:
provider: prometheus
prometheus:
flavor: cloudwatch
sigv4:
region: us-east-1
# credentials omitted: use the chain (IRSA, instance profile, ...)
Grafana’s Prometheus data source points at Trickster as it would at any Prometheus server; Trickster does the signing.
flavor: cloudwatch applies CloudWatch’s routes, limits and defaults:
- Signing: the
sigv4block is added if it is missing, and itsservicedefaults tomonitoring. - Origin:
origin_urldefaults tohttps://monitoring.<sigv4.region>.amazonaws.com. When the region comes from the environment instead ofsigv4.region, setorigin_urlexplicitly. Anorigin_urlandsigv4.regionnaming different regions fail at startup. - Routes: the PromQL API’s
query,query_range,series,labelsandlabel/<name>/valuesare served and cached. The read-only AWS API calls that Grafana’s CloudWatch data source makes are relayed uncached (see Grafana’s CloudWatch Data Source). Every other path and operation is answered by Trickster with an error instead of being signed and forwarded, so a client cannot use Trickster’s credentials to change anything. - Cache keys:
limit, which caps the series a query returns, is part of each cache key. - Truncated results: CloudWatch returns at most 500 series per query,
or
limitwhen smaller. A result cut short at that cap comes back with a warning and HTTP 200. Such a result is a sample of the series, and a later delta fetch can return a different sample, so Trickster never caches or merges one. It evicts any cached entry for the query, proxies the query to CloudWatch whole, and proxies that query directly for the backend’stimeseries_ttl. Each occurrence is counted intrickster_proxy_truncated_responses_total. Narrow the query, for example with an aggregation, to bring it back under the cap and into the cache. - Volatile window:
volatile_windowdefaults to2m, so the most recent two minutes are refetched while late data arrives. Any explicitvolatile_windoworvolatile_window_pointstakes precedence. - Health check: the default probe is
query=vector(1), which reads no metric and so scans no samples.
The IAM principal needs cloudwatch:GetMetricData and
cloudwatch:ListMetrics; series, labels and label values need only
cloudwatch:ListMetrics.
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["cloudwatch:GetMetricData", "cloudwatch:ListMetrics"],
"Resource": "*"
}]
}
Grafana’s CloudWatch Data Source
Grafana’s Prometheus data source needs only the PromQL API. Grafana’s CloudWatch data source can also run PromQL, in its PromQL query mode, and works against Trickster too. Set the data source’s Endpoint to the Trickster backend’s URL, and its Default Region to the backend’s region. Any credentials will do, since Trickster signs with its own and ignores the data source’s.
That one Endpoint carries every AWS call the data source makes, not just PromQL. Trickster caches the PromQL calls and relays the others to their services uncached, re-signed with its own credentials for each service:
| service | relayed operations | used for |
|---|---|---|
| CloudWatch | ListMetrics, GetMetricData, DescribeAlarms, DescribeAlarmsForMetric, DescribeAlarmHistory | Save & test, metric queries, variables, annotations |
| CloudWatch Logs | DescribeLogGroups, GetLogGroupFields, StartQuery, GetQueryResults, StopQuery, GetLogEvents, ListAnomalies, ListAggregateLogGroupSummaries | Save & test, Logs queries |
| EC2 | DescribeRegions, DescribeInstances | the region picker, variables |
| Resource Groups Tagging | GetResources | variables |
| Observability Access Manager | ListSinks, ListAttachedLinks | the cross-account picker |
Any other operation is refused with AccessDeniedException. A request
signed for a region other than the backend’s is refused with
ValidationException rather than answered from the backend’s region. Run
one backend per region and select that region in Grafana. Grant
Trickster’s IAM identity the read permissions for the operations your
dashboards use; Save & test needs cloudwatch:ListMetrics and
logs:DescribeLogGroups.
CloudWatch Limits
These CloudWatch limits apply per account and are not changed by Trickster. Delta fetches and request collapsing reduce how often a dashboard reaches them.
| limit | value |
|---|---|
| query requests per second | 300 |
discovery requests (series, labels, label values) per second | 10 |
| concurrent queries, and concurrent discovery requests | 30 each |
| range per query, including range selectors and lookback | 7 days |
| execution timeout | 20 seconds |
Exceeding a rate or size limit returns HTTP 422, and exceeding concurrency
returns HTTP 429; Trickster does not cache either. Every selector must name
a metric: {__name__=~"..."} alone is rejected.
Making AWS Vended Metrics Queryable
Metrics ingested through OTLP are queryable as they arrive. Classic AWS vended metrics, such as DynamoDB’s, become queryable through PromQL only after both enrichment settings are turned on for the account and region:
aws cloudwatch start-otel-enrichment --region us-east-1
aws observabilityadmin start-telemetry-enrichment --region us-east-1
Enriched metrics carry their resource attributes as UTF-8 labels, such as
@aws.region and @resource.cloud.resource_id.
Cost Example
CloudWatch bills PromQL queries at $0.01 per million samples scanned. The figures below are an illustration, not a measurement; check the savings against your own bill.
Take a dashboard of 20 panels, each summing a rate over 50 series of one-minute data, showing the last 6 hours and refreshing every 60 seconds:
- Without Trickster: each refresh scans about 6 hours plus a 5-minute lookback per series: 50 × 365 ≈ 18,000 samples per panel, or 365,000 per refresh. That is about 526 million samples a day, or about $5.26 a day ($158 per 30 days) for one screen left open.
- With Trickster: after the first load, each refresh fetches only the new step and the 2-minute volatile window, plus the lookback: about 8 samples per series, 400 per panel, or 8,000 per refresh. That is about 11.5 million samples a day, or about $0.12 a day.
Every additional viewer of the same dashboard costs that much again without Trickster, and close to nothing with it, since concurrent requests for the same query are collapsed and later ones are served from the cache.
Live Tests
The CloudWatch tests in pkg/backends/prometheus/cloudwatch_live_test.go
run against a real account and are skipped by default. They need a metric
that is visible to PromQL in the account:
TRICKSTER_AWS_TEST=1 TRICKSTER_AWS_PROFILE=default TRICKSTER_CW_METRIC=ConsumedReadCapacityUnits go test ./pkg/backends/prometheus/ -run Live -v -count=1
Notes
- Trickster signs with its own credentials; a client’s signature cannot be
passed through, because Trickster rewrites requests (delta fetches change
startandend) and AWS ties each signature to the host it was made for. A backend with asigv4block therefore drops the SigV4 fields a client sends (Authorizationwhen it holds a SigV4 signature,X-Amz-Date,X-Amz-Security-TokenandX-Amz-Content-Sha256) before caching or forwarding the request. OtherAuthorizationvalues, such as basic auth for Trickster’s own authenticators, are kept. When pointing Grafana at Trickster, turn off the data source’s SigV4 auth; it is ignored either way. - Anyone who can reach a
sigv4backend reads with Trickster’s IAM identity, within the paths the backend serves. Give that identity only the read permissions listed above, and limit who can reach Trickster with network policy or one of Trickster’s authenticators. Serve different accounts or roles from separate backends, each with its ownrole_arn; cache entries are never shared between backends. - SigV4 signs a hash of the request body, so Trickster buffers a request
body in order to sign it. This applies only to backends with a
sigv4block. - SigV4 is not supported for the ClickHouse native protocol, which is not HTTP; configuring both fails at startup.
IAM for Autodiscovery
The aws discovery provider needs read-only permission for whichever
aws.service it is configured with.
aws.service | required IAM actions |
|---|---|
ec2 | ec2:DescribeInstances |
ecs | ecs:ListTasks, ecs:DescribeTasks |
A minimal policy for service: ec2:
{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": "ec2:DescribeInstances",
"Resource": "*"
}]
}
ec2:DescribeInstances does not support resource-level permissions, so the
resource must be *; narrow the scope with a condition key if your
environment requires it. ecs:ListTasks and ecs:DescribeTasks can be
scoped to a cluster with the ecs:cluster condition key. Trickster only
ever reads.
5 - Origin Resolution via DNS SRV
By default, an HTTP backend dials its origin by looking up the A/AAAA records of the origin_url host and connecting to the port in the URL. With origin_resolution.mode: srv, the host is instead treated as a DNS SRV owner name. Each new connection goes to a target from the SRV answer, on the port the answer publishes.
This lets a backend reach services whose ports are known only to DNS, such as:
- ECS tasks in
bridgenetwork mode withhostPort: 0, registered in AWS Cloud Map with SRV records - Consul, Nomad or Kubernetes headless services exposed over DNS SRV
Combined with a rule and a hostname request rewriter, one backend can serve any number of SRV destinations, chosen per request.
Configuration
backends:
sites:
provider: rp
origin_url: 'http://_app._tcp.example.internal' # the port, if any, is ignored in srv mode
origin_resolution:
mode: srv # a (the default, A/AAAA lookup) | srv
resolver: '' # optional host:port of a DNS server to query directly
min_ttl: 5s # floor for cached answers
max_ttl: 60s # ceiling for cached answers
negative_ttl: 5s # how long NXDOMAIN or an empty answer is cached
tls_server_name: owner # owner (the default) | target
| setting | default | description |
|---|---|---|
mode | a | a dials the URL host’s A/AAAA addresses at the URL port, which is the behavior when origin_resolution is absent. srv resolves the URL host as an SRV owner name. |
resolver | The host:port of a DNS server to query directly. Direct queries make record TTLs visible. When empty, the Go standard library resolver is used. It reports no TTLs, so answers are cached for min_ttl. | |
min_ttl | 5s | The shortest time an answer is cached, whatever its record TTL |
max_ttl | 60s | The longest time an answer is cached, whatever its record TTL. It also bounds how long the last good answer keeps serving while lookups fail. |
negative_ttl | 5s | How long a name with nothing to dial is cached: NXDOMAIN, an empty answer, or a failed lookup with no last good answer to serve |
tls_server_name | owner | For https origins, the name the origin’s certificate is verified against: owner is the URL host, as with any other backend; target is the SRV target that was dialed |
resolver, the TTL settings and tls_server_name apply only to mode: srv; setting them with mode: a is a configuration error.
The owner name is used exactly as given. Trickster does not derive a service name such as _http._tcp from the URL scheme.
Routing Many Origins With One Backend
Without SRV resolution, a multi-tenant setup needs one backend per origin, each with its own fixed port, and every new origin is a config change and a reload. Origins whose ports are assigned dynamically can’t be expressed at all, because an A/AAAA record carries no port.
With SRV resolution, one rule, one rewriter, one router backend and one origin backend cover every origin that has SRV records. Adding an origin means publishing its DNS records; Trickster needs no config change or reload.
# DNS (example):
# alpha.origins.example.com. 30 IN SRV 1 1 31842 node-07.svc.example.com.
# _bravo._tcp.svc.example.com. 30 IN SRV 1 1 30117 node-02.svc.example.com.
# 30 IN SRV 1 1 30455 node-04.svc.example.com.
# ; aliases, so a public hostname doesn't have to equal the service name:
# bravo.origins.example.com. 60 IN CNAME _bravo._tcp.svc.example.com.
# bravo-legacy.origins.example.com. 60 IN CNAME _bravo._tcp.svc.example.com.
request_rewriters:
to-origin:
instructions:
# ${site} is the capture from the rule below. The rewritten name is
# SRV-resolved at dial time, and the port comes from the SRV answer.
- [ 'hostname', 'set', '${site}.origins.example.com' ]
rules:
site-router:
input_source: hostname
input_type: string
operation: rmatch
# keep the capture strict: it becomes the upstream hostname
operation_arg: '^(?P<site>[a-z0-9-]{1,63})\.example\.com$'
cases:
- matches: [ 'true' ]
req_rewriter_name: to-origin
next_route: origins
# anything that doesn't match the pattern
redirect_url: 'https://www.example.com/'
backends:
router:
provider: rule
rule_name: site-router
is_default: true
origins:
provider: rp
origin_url: 'http://origins.example.com' # placeholder; the rewriter sets the host per request
preserve_host: true # the origin still sees e.g. Host: alpha.example.com
path_routing_disabled: true
origin_resolution:
mode: srv
resolver: '10.0.0.2:53' # optional; a direct resolver gives real TTLs
min_ttl: 5s
max_ttl: 60s
negative_ttl: 5s
A request for alpha.example.com is rewritten to alpha.origins.example.com, which owns an SRV record, so it reaches node-07 on port 31842. A request for bravo-legacy.example.com is rewritten to bravo-legacy.origins.example.com, whose CNAME leads to the _bravo._tcp records. A host that matches the pattern but has no records gets a 502 Bad Gateway until its records are published.
Since the capture becomes a DNS name that Trickster resolves, keep the rule’s pattern strict. Every distinct name a client can produce costs a lookup, and its answer is cached.
How It Works
The transport’s dial is the only thing SRV resolution changes. The request URL keeps the SRV owner name, so URL handling, the Host header, logging, the cache key and the connection-pool key are the same as for any backend.
Target Selection
Targets are chosen as RFC 2782 describes:
- Targets in the lowest-priority tier (the smallest priority value) are tried first, ordered by weighted random selection. As RFC 2782 specifies, a target with weight
0has a small chance of being chosen first, and a tier whose weights are all0is ordered uniformly at random. - If a dial fails, the remaining targets in the tier are tried, then the next tier, all within the backend’s 10s connect timeout. Each target gets a share of the time left, which covers resolving its name and trying each of its addresses in turn, so neither an unreachable target nor an unreachable address can use it all up.
- A target of
.is ignored. An answer with no other targets means the service is not available.
Each target’s hostname is resolved to A/AAAA addresses with the same resolver. When the DNS server includes the target’s addresses in the additional section of the SRV answer, they are used without a further lookup. A target name that is an IP literal is dialed as given, and so is a URL host that a rewriter sets to an IP literal.
Caching
Answers are cached per owner name, and target addresses per target name. Each is kept for its record TTL, clamped to min_ttl and max_ttl. Concurrent dials for a name that needs a lookup share a single in-flight query.
When a cached answer expires, the dial waits up to 500ms for the refresh. If the refresh takes longer, the dial uses the last good answer while it is within max_ttl, and the refresh finishes in the background. When the lookup fails (a timeout, SERVFAIL or REFUSED), the last good answer keeps serving, retried no sooner than every min_ttl, until max_ttl has passed since it was looked up. NXDOMAIN and empty answers are answers rather than failures: they replace the last good answer at once and are cached for negative_ttl.
Each backend’s caches are bounded, so names derived from client-supplied hosts can’t grow them without limit. A config reload builds new caches, and in-flight requests finish on the old transport, as with any other reload.
Connection Pooling
max_idle_conns and max_concurrent_conns apply per SRV owner name, not per target, because the owner name is what the connection pool is keyed on. Idle connections to a target that has since left the answer are closed when they reach keep_alive_timeout.
TLS
For an https origin, the server name sent and verified is the SRV owner name (the URL host), as for any other backend. Set tls_server_name: target for services whose certificates name the target host instead. It can’t be combined with tls.server_name, which names a fixed server name for every target.
Health Checks
A backend health check probes through the same transport, so it is SRV-resolved too. Each probe reaches a single target, so it reports whether the service is reachable, not whether every target is.
Active per-target health checks, target ejection and load-balancing mechanisms are not part of SRV resolution. Use an ALB with dns_srv autodiscovery when you need them.
Validation
A backend with mode: srv fails validation when:
origin_urlis nothttp://orhttps://, or its host is an IP literalresolveris not ahost:portmin_ttlis greater thanmax_ttl, after defaults applytls_server_name: targetis combined withtls.server_name
origin_resolution is not supported on alb, rule and static backends, or with a non-HTTP protocol.
Metrics
trickster_proxy_origin_srv_lookups_totalcounts SRV lookups byresult. The results arehit,miss,stale(a failed refresh served the last good answer),negative(a cached NXDOMAIN, empty answer or failure) anderror.trickster_proxy_origin_srv_dial_attempts_totalcounts dial attempts to SRV targets by prioritytier(0is the preferred tier) andresult(successorfailure).
Both are labeled by backend_name. See metrics.
6 - Static File Server Backend
The Static Backend serves the contents of a local directory, rather than proxying requests to an upstream origin. Use it to host a website, a single-page application’s assets, a maintenance page, or any other set of files directly from Trickster, alongside (and with the same listeners, TLS, virtual hosting, authentication, logging and tracing as) your other Backends.
The file server is built into Trickster on the Go standard library, and provides:
GETandHEADfor files, withETagandLast-Modifiedvalidators, conditional requests (If-None-Match,If-Modified-Since,If-Match,If-Unmodified-Since), and single, multi-part andIf-Rangebyte range requests- a configurable default file (
index.html) for directory requests Content-Typedetection from a built-in table of common web types, with your own additions and overrides- a configurable
Cache-Controlheader, with overrides by file extension - a fileserver cache that holds small files in memory, kept current by filesystem change events (on Linux and Windows) and periodic revalidation, with the least recently used making room when it fills
- compression for the Backend’s
compressible_types, with each encoded rendition of a small file held in memory and reused - a not-found file, for a custom error page or a single-page application
- Prometheus metrics for the files served and the cache
- safe defaults: dotfiles are never served (other than
/.well-known/), directory listings are off, and no request can reach a file outside of the configured root
Configuring
A Static Backend sets its provider to static, and requires a static options block that names the root directory to serve.
backends:
website:
provider: static
hosts: [ www.example.com ]
static:
root: /var/www/html
Like any other Backend, this one is reachable at /website/ on its listeners, at / for requests whose Host is www.example.com, and at / for every request when it sets is_default: true.
Static Options
| Option | Default | Description |
|---|---|---|
root | (required) | Path to the directory holding the content to serve. It must exist and be a directory when the configuration is loaded. A relative path is resolved from Trickster’s working directory. |
default_file | index.html | The file served when a directory is requested (e.g., / or /docs/). It must be a plain file name that does not start with a . |
cache_control | The Cache-Control header value sent with every file. By default no Cache-Control header is sent, as with nginx and other common web servers, and a client judges how long a file stays fresh from its Last-Modified time. | |
cache_control_by_extension | A map of file extension to Cache-Control value, which takes precedence over cache_control for matching files. | |
response_headers | A map of additional headers to attach to every response. | |
mime_types | A map of file extension to Content-Type, which adds to and takes precedence over the built-in types. | |
not_found_file | A file, as a path within the root, that is served in place of a plain 404 response. See Not-Found File. | |
not_found_status | 404 | The status not_found_file is served with: 404 for an error page, or 200 for a single-page application. |
directory_listing | false | When true, a directory that has no default_file is answered with a listing of its contents. |
cache | Options for the Fileserver cache, which holds small files in memory, described below. |
File extensions are case-insensitive, and may be written with or without the leading dot.
A fuller example:
backends:
website:
provider: static
is_default: true
static:
root: /var/www/html
default_file: index.html
# html is revalidated on every use, so a deployment is visible immediately
cache_control: no-cache
cache_control_by_extension:
# fingerprinted assets never change, and can be cached for a year
.js: public, max-age=31536000, immutable
.css: public, max-age=31536000, immutable
.woff2: public, max-age=31536000, immutable
response_headers:
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
mime_types:
.md: text/markdown; charset=utf-8
not_found_file: errors/404.html
directory_listing: false
cache:
max_file_size_bytes: 1048576
max_size_bytes: 134217728
max_files: 10000
revalidation_interval: 10s
Backend Options
These general Backend options work with a Static Backend as they do with any other: hosts, any_host_routing, listener_names, is_default, path_routing_disabled, require_tls, tls, authenticator_name, cors, access_log, tracing_name, compressible_types, latency_min and latency_max.
Options that describe proxying, caching or routing to an origin have no meaning for a file server. Rather than being silently ignored, these fail configuration validation when set on a Static Backend: paths, req_rewriter_name, origin_url, rule_name, alb, prometheus, mysql, graphite, influxdb, sigv4, protocol, h2c_prior_knowledge, preserve_host, proxy_only and is_template. Likewise, a static block on a Backend of any other provider fails validation.
A Static Backend has no origin to probe, so it always reports as available on the health endpoint. It can be the target of a Rule or a member of an ALB pool.
Requiring Authentication
To require users to authenticate before any file is served, attach an Authenticator to the Backend. Requests that fail authentication are rejected before the file server is consulted, so they reveal nothing about which files exist.
authenticators:
staff:
provider: basic
users_file: /etc/trickster/htpasswd
users_file_format: htpasswd
backends:
intranet:
provider: static
authenticator_name: staff
static:
root: /var/www/intranet
Request Handling
| Request | Response |
|---|---|
| a file | 200 OK with the file |
| a directory, with a trailing slash | 200 OK with the directory’s default_file |
a directory with a trailing slash, but no default_file | 404 Not Found, or a listing when directory_listing is true |
| a directory, without a trailing slash | 301 Moved Permanently to the same path with a trailing slash |
| a file, with a trailing slash | 404 Not Found |
any path with a segment that starts with a ., other than a leading /.well-known/ | 404 Not Found |
| anything that does not exist, can’t be read, or resolves outside of the root | 404 Not Found |
OPTIONS | 204 No Content with an Allow header |
any method other than GET, HEAD or OPTIONS | 405 Method Not Allowed with an Allow header |
Every 404 Not Found in this table is answered with the not-found file when one is configured.
A missing default_file is deliberately a 404 rather than a 401 or 403: with listings off, a directory is not a resource that exists to be forbidden.
The trailing-slash redirect is relative (Location: docs/), so it is correct whether the Backend was reached by its hostname, as the default Backend, or under its /backend-name/ path.
Dotfiles
Any request path containing a segment that begins with a . is answered with a 404. This covers hidden files (/.env, /.htpasswd), everything beneath hidden directories (/.git/config), and attempts to reach them indirectly (/docs/../.env). Dotfiles are also left out of directory listings.
The one exception is /.well-known/, the location reserved by RFC 8615 for site-wide metadata such as security.txt, ACME HTTP-01 challenges and app-association files. It is exempt only as the first segment of the path (/docs/.well-known/ is refused), dotfiles within it are still refused (/.well-known/.secret), and it is still left out of directory listings. The list of exemptions is built in, and is deliberately not configurable.
Not-Found File
not_found_file names a file within the root to serve whenever the response would otherwise be a plain 404: a path that doesn’t exist, a directory with no default_file, a refused dotfile, and so on. not_found_status selects which of its two uses it is put to.
With the default status of 404, it is a custom error page. The page is sent as the body of the 404, with Cache-Control: no-cache and without the ETag, Last-Modified and Accept-Ranges headers, so that it can’t be revalidated, ranged over or reused as though it were the missing file. Any Range or conditional headers on the request are ignored.
static:
root: /var/www/html
not_found_file: errors/404.html
With a status of 200, it is the fallback for a single-page application, whose routes (/dashboard, /users/42/edit) exist only in the browser. Every unknown path is answered with the application itself, exactly as if the file had been requested by its own path: with its validators, its Cache-Control, conditional and byte range support, and its held renditions. Requests for files that do exist are unaffected.
static:
root: /var/www/app
not_found_file: index.html
not_found_status: 200
If the not-found file is itself missing, the response is a plain 404. A request using a method other than GET or HEAD is never answered with it. When directory_listing is on, a directory that can be listed is listed rather than answered with it.
Root Confinement and Symbolic Links
All file access is confined to the root by the operating system, not by path inspection alone. No request can read a file outside of the root, whether by .. traversal or through a symbolic link.
Symbolic links are followed only when they are relative links to a target inside the root. A link to anywhere outside of the root, and any link with an absolute target (even one that points inside the root), is answered with a 404.
Only regular files are served. Devices, sockets and named pipes inside the root are answered with a 404.
Content Types
A file’s Content-Type is resolved from, in order: the mime_types option, Trickster’s built-in table of common web types (HTML, CSS, JavaScript, JSON, WebAssembly, images, fonts, audio, video, archives, documents and more), and the host’s MIME database. The built-in table makes detection work the same in a minimal container image as on a full host. A file whose extension is unknown, or that has none, is identified from its first 512 bytes, and falls back to application/octet-stream.
Validators, Ranges and Compression
Every file is served with an ETag and a Last-Modified header, and matching conditional requests are answered with a 304 Not Modified.
Unless cache_control is set, no Cache-Control header is sent. A client then decides for itself how long to reuse a file before revalidating it, usually for a fraction of the time since the file was last modified, so files that have not changed in a long time are reused for longer. Where a change must be seen at once, such as the HTML entry point of an application with fingerprinted assets, set cache_control (or cache_control_by_extension) to no-cache, which has clients revalidate the file before each reuse.
The ETag is derived from the file’s modification time and size, the same two components used by nginx and other common web servers. Because it comes from file metadata, the ETag is the same whether a file is served from memory or from disk, and a HEAD or conditional request is answered from a stat of the file without reading it. Instances of Trickster serving copies of the same content issue the same ETag for a file, so long as the deployment preserves modification times (as rsync -a and tar do).
As with those servers, a file that is replaced by different content of exactly the same size and modification time keeps its ETag. If your deployment tooling pins or preserves modification times, give changed files a new name (such as a content fingerprint) or a new modification time.
Files whose Content-Type is in the Backend’s compressible_types are compressed for clients that accept it, in any of zstd, br, gzip and deflate. Those responses carry Vary: Accept-Encoding, and the weak form of the ETag (W/"..."), because the encoded bytes differ from the stored file the ETag describes. Byte ranges always address the stored file, so a 206 Partial Content response is never compressed, and carries the ETag unchanged.
How a file is compressed depends on whether the Fileserver cache can hold it. A file it can hold is compressed once, and the encoded rendition is held and reused, as described under Held Renditions. Any other file (a larger one, or any file when the cache is disabled) is compressed as it is sent, on every request.
Fileserver Cache
The Fileserver cache is configured in the cache block of the static options, and belongs to the Static Backend alone. It is unrelated to the caches Trickster uses to accelerate proxied Backends (including the memory cache provider): it is not configured in the caches section, a Static Backend has no cache_name, and nothing it holds is reachable through the cache purge endpoints.
Small files are read into memory the first time their content is needed, and served from memory afterward without touching the disk. Larger files are streamed from disk on every request.
Only a response that sends the whole file loads it. A HEAD request, a conditional request answered with a 304, and a byte range request are all served from disk without reading the file into memory, so they can’t be used to fill the cache. When many requests arrive at once for a file that is not yet held, it is read once and shared between them.
| Option | Default | Description |
|---|---|---|
disabled | false | When true, every request is served from disk. |
max_file_size_bytes | 1048576 (1 MiB) | The largest file that is held in memory. |
max_size_bytes | 134217728 (128 MiB) | The most memory the cache will use. Each held file counts as its size plus a fixed 1 KiB allowance for its bookkeeping, so empty and tiny files are bounded too. |
max_files | 10000 | The most objects that are held, which also bounds the number of directories that are watched to twice this number. A file as stored, and each held rendition of it, is an object. |
revalidation_interval | 10s | How often held files are compared to the disk, as a backstop to change events, and the only means of change detection on macOS and the BSDs. |
Capacity is reserved before a file is read, so the limits hold even while many different files are being loaded at once. Once either limit is reached, the least recently used objects make room for new ones.
Eviction
When a file needs room that the cache doesn’t have, by either limit, the least recently used objects are evicted to make it. Recency is tracked without slowing requests down: serving a held file only marks it as used, and eviction passes over each marked object once, clearing its mark, before it will remove it. An object is therefore evicted only if it has gone unread since eviction last reached it.
Room is only ever made for a file that is then held. Eviction first works out what would have to go, and goes ahead only if that is enough; a file that there isn’t room for is served from disk, and costs the cache nothing it holds. A request for a file that is never asked for again therefore can’t flush content that is in use.
The work a request will do to find room is bounded, however large the cache is: it passes over a limited number of recently used objects before giving up. Those it passed are no longer marked as recently used, so if requests for room keep coming, objects that aren’t being read in between give way to them, while objects in constant use never do. The objects a request does evict are in proportion to the size of the file they make way for. No single file is admitted at the cost of more than 128 objects. With the default limits that is rarely a constraint, as a full cache is usually full by its count of objects, and one eviction then makes room for a file of any size; it matters to a cache with a very high max_files that is full of very small files, in which a large file is served from disk instead.
A file is never evicted to make room for a rendition of itself, as a rendition is only held alongside the file it was made from. Where there is room for only one of them, it is the file that is held, and its renditions are made from memory for each request. Room held by files that are still being read into memory can’t be evicted either; in the unlikely event that those account for the whole cache, a new file is served from disk.
Held Renditions
A held file of a compressible type is also held in the encodings clients ask for, each as an object of its own beside the file as stored, so that a file is compressed once rather than on every request.
A request’s Accept-Encoding header is read for the encodings Trickster supports (zstd, br, gzip and deflate), most preferred first: by the weight (q) the client gave each, and among encodings of equal weight, or when no weights are given (as by browsers), by Trickster’s own preference in the order just listed. An encoding takes the weight of its own entry, or else that of the wildcard (*) if there is one. Encodings the client refused (q=0) are left out, and so are any it weighted below identity, since the file as stored is always there to be sent instead. Then:
- If the file is held in any of those encodings, the most preferred of the renditions that are held is served.
- Otherwise, if the file is held as stored, it is encoded from memory in the client’s most preferred encoding, and the new rendition is held.
- Otherwise, the file is read from disk and encoded in the client’s most preferred encoding, and both the file as stored and the new rendition are held. A rendition that is held in some other encoding is never decoded to stand in for the file.
A new rendition is streamed to the client as it is encoded, and what was sent is kept to be held, so the first response is not delayed by the rendition being made for later ones. That first response is sent without a Content-Length; once held, a rendition is sent with its own.
Renditions count toward max_size_bytes and max_files like any other object, are evicted independently of one another, and are all dropped together when their file changes on disk. A rendition is only made for a response that sends the whole file: not for a HEAD, a byte range, or a conditional request (which may turn out to need no body; if it does need one, it is compressed as it is sent). Files smaller than 512 bytes are always sent as stored, as an encoding’s own framing outweighs what it would save. When an encoding turns out not to make a file smaller, that encoding isn’t tried for the file again until it changes; the other encodings are unaffected, and a client that prefers the unhelpful one is given the next it accepts.
When nothing a client accepts is left (for example identity;q=0 from a client that accepts no supported encoding), the file is sent as stored rather than refused with a 406, as other web servers do.
Serving and the Cache
Holding a file never delays serving it. Held files are read without locking. Everything that changes the cache (holding a file or a rendition, or releasing room that turned out not to be needed) is done after, and apart from, the response that led to it. The one thing a request asks of the cache before it responds is to reserve room, which is what keeps memory within its limits however many files are being read at once; if the cache is busy at that moment, the request doesn’t wait, and is served without being held. The file is then held by a later request for it.
Change Detection
Trickster watches the directories of the files it holds. A directory that a change leaves with nothing held stays watched for one to two revalidation_intervals, since the changed file is usually requested again soon; one emptied by eviction is let go of at once. Letting go of a watch is done in the background, so evicting files never makes a request wait on the operating system. When a file is modified, replaced, renamed or removed, it is dropped from memory (typically within milliseconds) and the next request reads it from disk again. Renaming or removing a directory drops every held file beneath it. A changed file is never updated in place, and a read that overlaps a change is discarded rather than stored, so a request is never served a mix of old and new content from memory. A change touches only what it affects: the one file, or the files beneath the one directory. Every other held file stays in memory, and files that are being read into memory at that moment are unaffected, so a busy deployment does not push the rest of the site back to disk.
Every revalidation_interval, each held file’s size, modification time and identity are compared to the disk, which catches changes that raise no event: a filesystem without change notifications (such as some network and container bind mounts), a changed symbolic link target, or a replaced root. On such filesystems, revalidation_interval is the longest a stale file can be served; lower it, or disable the fileserver cache, if that matters for your content.
On macOS and the BSDs, the fileserver cache doesn’t use change events at all, and relies on revalidation alone, so a change can take up to revalidation_interval to be served. The event library’s kqueue support, which those platforms use, can close a file descriptor twice as a watch is removed, and the second close can land on a file that a request has open, so that another file’s content is served in its place (fsnotify#788). Change events will be enabled on these platforms once a fixed release is available. For local development on macOS, a short revalidation_interval (e.g., 1s) keeps edits visible quickly.
On every platform, a file that is found not to be the one at its path as it is opened is opened again, and answered with a 503 if it is still not; and a response sent from disk is broken off, rather than completed, if its file descriptor no longer refers to that file once the body is sent.
Atomic Deployments
Deploying by swapping a symbolic link is supported. When root is a link (e.g., /var/www/current → /var/www/releases/42) and the link is repointed, Trickster notices at the next revalidation_interval, switches to the new directory, and drops every held file. Requests in flight during the switch complete against the previous release.
Directory Listings
With directory_listing: true, a request for a directory that has no default_file is answered with an HTML page linking to the directory’s files and subdirectories, with their sizes and modification times. Dotfiles, links that can’t be followed, and anything that is not a regular file or directory are left out. A directory that does have a default_file is always answered with that file. Listings are generated per request, and are sent with Cache-Control: no-cache.