This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Backends

Configuring the upstream origins that Trickster accelerates: multiple backends, health checks, authentication and AWS request signing.

1 - Using Multiple Backends with a single Trickster instance

Trickster supports proxying to multiple backends in a single Trickster instance, by examining the inbound request and using a multiplexer to direct the proxied request to the correct upstream origin, in the same way that web servers support virtual hosting.

There are 2 ways to route to multiple backends.

  • HTTP Pathing
  • DNS Aliasing

Basic Usage

To utilize multiple backends, you must craft a Trickster configuration file to be read when Trickster starts up - operating with environment variables or command line arguments only supports accelerating a single backend. The example.full.yaml provides good documentation and commented sections demonstrating multiple backends. The config file should be placed in /etc/trickster/trickster.yaml unless you specify a different path when starting Trickster with the -config command line argument.

Each backend that your Trickster instance supports must be explicitly enumerated in the configuration file. Trickster does not support open proxying.

Example backend configuration:

backends:
  my-backend-01: # routed via http://trickster:8480/my-backend-01/
    provider: prometheus
    origin_url: http://my-origin-01.example.com

  my-backend-02: # routed via http://trickster:8480/my-backend-02/
    hosts: [ my-fqdn-02.example.com ] # or http://my-fqdn-02.example.com:8480/
    provider: prometheus
    origin_url: http://my-origin-02.example.com

  my-backend-03:
    path_routing_disabled: true # only routable via Host header, an ALB, or a Rule
    hosts: [ my-fqdn-03.example.com ] # or http://my-fqdn-03.example.com:8480/
    provider: prometheus
    origin_url: http://my-origin-02.example.com

Default Backend

Whether proxying to one or more upstreams, Trickster has the concept of a “default” backend, which means it does not require a specific DNS hostname in the request, or a specific URL path, in order to proxy the request to a known backend. When a default backend is configured, if the inbound request does not match any mapped backends by path or FQDN, the request will automatically be routed through the default backend. You are probably familiar with this behavior from when you first tried out Trickster with the using command line arguments.

Here’s an example: if you have Trickster configured with a backend named foo that proxies to http://foo/ and is configured as the default backend, then requesting http://trickster/image.jpg will initiate a proxy request to http://foo/image.jpg, without requiring the path be prefixed with /foo. But requesting to http://trickster/foo/image.jpg would also work.

The default backend can be configured by setting is_default: true for the backend you have elected to make the default. Having a default backend is optional. In a single-backend configuration, Trickster will automatically set the sole backend as is_default: true unless you explicitly set is_default: false in the configuration file. If you have multiple backends, and don’t wish to have a default backend, you can just omit the value for all backends. If you set is_default: true for more than one backend, Trickster will exit with a fatal error on startup.

Path-based Routing Configurations

In this mode, Trickster will use a single FQDN but still map to multiple upstream backends by path. This is the simplest setup and requires the least amount of work. The client will indicate which backend is desired in URL Path for the request.

Example Path-based Configuration with Multiple Backends:

backends:
  # backend1 backend
  backend1:
    origin_url: 'http://prometheus.example.com:9090'
    provider: prometheus
    cache_name: default
    is_default: true
  # "foo" backend
  foo:
    origin_url: 'http://influxdb-foo.example.com:9090'
    provider: influxdb
    cache_name: default
  # "bar" backend
  bar:
    origin_url: 'http://prometheus-bar.example.com:9090'
    provider: prometheus
    cache_name: default

Using HTTP Path as the Backend Routing Indicator

The client prefixes the Trickster request path with the Backend Name.

This is the recommended method for integrating with applications like Grafana.

Example Client Request URLs:

DNS Alias Configuration

In this mode, multiple DNS records point to a single Trickster instance. The FQDN used by the client to reach Trickster is mapped to specific backend configurations using the hosts list. In this mode, the URL Path is not considered during Backend Selection.

Host matching is case-insensitive and ignores the port. An entry may also be a single-label wildcard such as *.example.com, which matches foo.example.com but not example.com or a.b.example.com, or an any-depth wildcard such as **.example.com, which matches foo.example.com and a.b.example.com but not example.com; an exact entry on another backend always wins over a wildcard, and a *. entry wins over a **. entry for the same domain. A * anywhere else in an entry is rejected at load time. See Host Resolution for the full precedence order.

Example DNS-based Backend Configuration:

backends:
  # backend1 backend
  backend1:
    hosts: # users can route to this backend via these FQDNs, or via `/backend1`
      - 1.example.com
      - 2.example.com
    origin_url: 'http://prometheus.example.com:9090'
    provider: prometheus
    cache_name: default
    is_default: true
  # "foo" backend
  foo:
    hosts: # users can route to this backend via these FQDNs, or via `/foo`
      - trickster-foo.example.com
    origin_url: 'http://prometheus-foo.example.com:9090'
    provider: prometheus
    cache_name: default
  # "bar" backend
  bar:
    hosts: # users can route to this backend via these FQDNs, or via `/bar`
      - trickster-bar.example.com
    origin_url: 'http://prometheus-bar.example.com:9090'
    provider: prometheus
    cache_name: default

Example Client Request URLs:

Note: It is currently possible to specify the same FQDN in multiple backend configurations. You should not do this (obviously). A future enhancement will cause Trickster to exit fatally upon detection at startup.

Disabling Path-based Routing for a Backend

You may wish for a backend to be inaccessible via the /backend_name/ path, and only by Hostname or as the target of a rule or ALB. You can disable path routing by setting path_routing_disabled: true for the backend, as in this example, which requires the Request’s Host header match 1.example.com or 2.example.com in order to be routed to the backend:

backends:
  backend1:
    hosts:
      - 1.example.com
      - 2.example.com
    origin_url: 'http://prometheus.example.com:9090'
    provider: prometheus
    cache_name: default
    is_default: false
    path_routing_disabled: true # this will disable routing through /backend1

2 - Health Checks

Trickster Service Health - Ping Endpoint

Trickster provides a /trickster/ping endpoint that returns a response of 200 OK and the word pong if Trickster is up and running. The /trickster/ping endpoint does not check any proxy configurations or upstream origins. The path to the Ping endpoint is configurable, see the configuration documentation for more information.

Upstream Connection Health - Backend Health Endpoints

Trickster offers health endpoints for monitoring the health of the Trickster service with respect to its upstream connection to origin servers.

General Endpoint

The main health check path is /trickster/health, which by default will return a text/plain summary of the backend health. You can request YAML or JSON format using the appropriate Accept header, or by providing a ?json or ?yaml query param.

ALBs and Their Pool Members

Each ALB is listed with its mechanism and its pool members, grouped by health. The text form appends them to the ALB’s line as u:[...] (unavailable), a:[...] (available), nc:[...] (not checked) and d:[...] (draining). The JSON and YAML forms name them unavailablePoolMembers, availablePoolMembers, uncheckedPoolMembers, initializingPoolMembers and drainingPoolMembers.

A draining member, one marked drain: true or a discovered one that is terminating but still serving, is listed under its health and again as draining: it still serves the sticky sessions it has, but takes no new work. An ALB with a pool is listed as available while it has a member it would send new work to, so an ALB whose members all drain is listed as unavailable, though it still serves their sessions. For example:

backends:
  app1:
    provider: reverseproxycache
    origin_url: http://app1.example.com
    healthcheck:
      interval: 5s
  app2:
    provider: reverseproxycache
    origin_url: http://app2.example.com
    healthcheck:
      interval: 5s
  app:
    provider: alb
    alb:
      mechanism: rr
      pool:
        - app1
        - name: app2
          drain: true
{
  "title": "Trickster Backend Health Status",
  "updateTime": "2026-09-26 00:00:00 UTC",
  "available": [
    { "name": "app1", "provider": "rpc" },
    { "name": "app2", "provider": "rpc" },
    {
      "name": "app",
      "provider": "alb",
      "mechanism": "rr",
      "availablePoolMembers": [ "app1", "app2" ],
      "drainingPoolMembers": [ "app2" ]
    }
  ]
}

See All-Backends Health Status Page for more about how ALBs are listed.

Backend-Specific Endpoints

Each HTTP backend’s health check path is /trickster/health/BACKEND_NAME. For example, if your backend is named foo, you can perform a health check of the upstream server at http://<trickster_address:port>/trickster/health/foo. Native-protocol backends publish their scheduled status through the general endpoint without registering a synthetic HTTP origin route.

The backend health path prefix /trickster/health/ is customizable. See the example.full.yaml for more info about setting the health_handler_path configuration, or refer to this example:

mgmt:
  # this overrides the default '/trickster/health' to '/-/trickster/health'
  health_handler_path: /-/trickster/health

The behavior of a health request will vary based on the Backend provider, as each has their own health check protocol. For example, with Prometheus, Trickster makes a request to /query?query=up and (hopefully) receives a 200 OK, while for InfluxDB the request is to /ping which returns a 204 No Content.

Supported TSDB Providers are pre-configured in Trickster to perform a suitable health check operation, however these can be overridden in the configuration file.

For non-TSDB Backends, the default behavior is to make a GET request to http://origin_url:port/ and expect a 2xx response. However, all aspects of the Health Check request and expected response are configurable per-Backend.

Native MySQL Health Checks

The MySQL backend uses the same interval scheduler, timeout, transition thresholds, status registry, metrics, reload carryover, and shutdown lifecycle as HTTP health checks. Each probe opens a fresh connection with the backend’s configured origin credentials and TLS policy, completes authentication, executes COM_PING, and closes the connection.

backends:
  mysql1:
    provider: mysql
    origin_url: mysql://health-user:password@mysql.example:3306/analytics
    healthcheck:
      interval: 5s
      timeout: 3s
      failure_threshold: 3
      recovery_threshold: 3

Only interval, timeout, failure_threshold, and recovery_threshold apply to native probes. HTTP verbs, paths, headers, bodies, and expected HTTP response options are configuration errors for MySQL. Diagnostics exposed in health status and logs are limited to sanitized authentication, TLS, timeout, refused-connection, connection, and server-error categories.

Native PostgreSQL Health Checks

A postgres or timescaledb backend is probed over the PostgreSQL wire protocol with the same scheduler, timeout, thresholds, status registry and metrics as every other health check. Each probe opens a fresh connection with the backend’s origin_url credentials and upstream TLS mode, completes authentication, waits for the origin’s ReadyForQuery, and ends the session cleanly. A backend whose origin_url has no username (one that relays its clients’ own authentication) can only be checked for accepting connections.

backends:
  tsdb1:
    provider: timescaledb
    origin_url: postgres://health-user:password@tsdb.example:5432/analytics
    healthcheck:
      interval: 5s
      timeout: 3s
      failure_threshold: 3
      recovery_threshold: 3

Only interval, timeout, failure_threshold, and recovery_threshold apply. Failures are reported as a login rejection with the origin’s SQLSTATE, a timeout, or a connection failure; user names, addresses and passwords never appear in health status or logs. A native User Router refuses new sessions for a user whose backend is failing its health check.

Basic Health Check Configuration Example

backends:
  server1:
    provider: reverseproxycache
    origin_url: http://server1
    healthcheck: # all values below are optional
      verb: HEAD
      path: /health

Health Check With Exhaustive Request/Response Options

backends:
  server1:
    provider: reverseproxy
    origin_url: http://server1
    healthcheck: # all values below are optional
      # 
      ## customizing the health check request
      #
      verb: HEAD
      scheme: https
      host: alternate-hostname.example.com
      path: /health
      query: param1=value1&param2=value2
      headers:
        User-Agent: health-check-agent
      # if using a POST or PUT method, you can provide a string body
      # body: "my health check body"
      #
      ## customizing the expected response
      #
      # hc fails if a response takes longer than 1s
      timeout: 1000ms
      # hc fails if the response code is not in the list
      expected_codes: [ 200, 204, 206, 301, 302, 304 ]
      #
      # hc fails if these response headers are not present and have the expected value
      expected_headers:
        X-Health-Check-Status: success
      # hc fails if the stringified response body does not match the expected value
      expected_body: "pass"

See more examples in example.full.yaml.

Health Check Integrations with Application Load Balancers

By default, a Backend will only initiate a health check on-demand, upon receiving a request to its health endpoint.

To facilitate integrations with the Trickster Application Load Balancer provider, additional options provide for 1) timed interval health checks and 2) thresholding for consecutive successful or unsuccessful health checks that determine the backend’s overall health status.

Example Health Check Configuration for use in ALB

backends:
  server1:
    provider: reverseproxy
    origin_url: http://server1
    healthcheck:
      path: /health
      timeout: 1000ms      # timeout should be <= interval
      # for ALB integration:
      interval: 1000ms     # auto-poll health every 1s
      failure_threshold: 3  # backend is unhealthy after 3 consecutive failures
      recovery_threshold: 3 # backend is healthy after 3 consecutive successes

The Prometheus default probe is /api/v1/query?query=up. Some multi-tenant Prometheus gateways reject an unbounded up with 400 bad_data: "too many series found", which keeps the member out of any ALB pool it belongs to. Override healthcheck.query with a bounded expression the backend accepts (for example query=vector(1)) when probing such backends.

Other Ways to Monitor Health

In addition to the out-of-the-box health checks to determine up-or-down status, you may want to setup alarms and thresholds based on the metrics instrumented by Trickster. See metrics.md for collecting performance metrics about Trickster.

3 - Authenticator

Trickster 2.x provides an Authenticator capability that allows you to protect Backends with an Authentication layer.

Authenticator resources are defined globally by name, and then mapped into any Backend and/or Path configuration as needed. Authenticator users can be loaded from htpasswd or csv files, or directly in the Trickster config file. You can provide credentials in plaintext, bcrypt, Apache MD5 ($apr1$), MD5-crypt ($1$), as well as legacy formats sha-256-crypt and sha-512-crypt ($5$, $6$). PostgreSQL SCRAM-SHA-256 and md5 verifiers are accepted too.

A stored hash or verifier is only ever checked as that format, so a client that sends the stored hash itself as its password is rejected. Plaintext credentials are compared in constant time, and an empty stored credential never matches.

Authenticators work with all Backend provider types. Requests are handled by their respective Authenticators before all other Handlers (e.g., Caches, Rules, Request Rewrites, ALB Routes, etc.). Authentication sees the request as the client sent it: backend and path request rewriters run after it, so a rewriter cannot change the path or credentials that authentication judges.

Native MySQL listeners terminate mysql_native_password authentication and have stricter credential-source requirements than HTTP authenticators. See the MySQL Provider Guide.

If a request is routed via a Trickster ALB or Rule Backend through to multiple other Backends - each having different Authenticator configurations - the authentication behavior is currently undefined. In an upcoming Beta release, we will define this use case to either use the Authenticator config (if set) of the very first Backend that handled the request; or to use the first defined Authenticator regardless of how deep into the Backend chain it is.

By default, when an Authenticator handles and successfully authenticates a request, the request’s Auth credentials (e.g., Authorization Header for Basic Auth) are stripped before the request is cloned for any necessary proxying. Setting proxy_preserve: true will preserve these headers instead of stripping them.

Path Protection

When you map an Authenticator to a Backend, all Paths defined for that Backend are protected by the Authenticator. However, you can override the Authenticator on a per-Path basis by including the authenticator_name in a Path config. To bypass a Backend-wide Authenticator in a Path config, use authenticator_name: none. This allows for a few possibilities:

  • A Backend config with a default / Backend-wide Authenticator that has:

    • specific Paths that do not require Authentication
    • specific Paths mapped to different Authenticators than the default
  • A Backend config with no Authenticator defined (so all Paths by default are unprotected) that has:

    • specific Paths mapped to different Authenticators than the default

See the example Backend configs below for more details.

Authenticator Providers

Trickster’s Authenticator feature currently supports Basic Auth and ClickHouse-compatible authentication. It was designed with extensibility in mind should there be value in adding additional Authentication providers.

Basic Auth Provider

Basic Auth is supported by using provider: basic in the Authenticator config.

By setting the showLoginForm: true config (see example_auth_1 in the blob below), Trickster will return a WWW-Authenticate: Basic realm="custom-realm-name" header on any request that requires but fails authentication, causing the login form to pop up. When showLoginForm is not present or non-true, Trickster responds with a 401 Unauthorized but does not ask the Basic Auth login form to show.

The realm attribute value defaults to the Authenticator name (e.g., example_auth_1) but can be overridden with the realm config as in the example.

If the user data changes (e.g. updated users_file contents or updated embedded users list), you must send a SIGHUP or other means to reload the Trickster config before the new user pool is processed.

ClickHouse Auth Provider

ClickHouse Auth is supported by using provider: clickhouse in the Authenticator config.

ClickHouse authentication is the same as Basic Auth, except you can also provide user and password URL params. Unless proxy_preserve: true is set, those URL params are removed along with the Authorization header before the request is proxied.

Example Authenticator Configs

# NOTE: Required options unrelated to Authenticators have been omitted from this
# example. It does not represent a fully-functioning Trickster Config.
# See the 'examples' directory for working copy/paste config examples.

backends:
  backend01:
    provider: reverseproxy # authenticators work with all backend providers
    authenticator_name: example_auth_1 # protects backend01 with example_auth_1 authenticator
    origin_url: https://example.com
    paths:
      - path: / # all requests are protected by example_auth_1 
        match_type: prefix
        handler: proxy

  backend02:
    provider: reverseproxy # no backend-wide authenticator
    origin_url: https://example.com
    paths:
      - path: / # requests will be allowed without auth except the 2 Paths below
        match_type: prefix
        handler: proxy
      - path: /private/
        authenticator_name: example_auth_2 # example_auth_2 protects this path only
        handler: proxy
      - path: /admin/
        authenticator_name: example_auth_3 # example_auth_3 protects this path only
        handler: proxy

  backend03:
    provider: reverseproxycache
    authenticator_name: example_auth_1 # protects backend03 with example_auth_1 authenticator
    origin_url: https://example.com
    paths:
      - path: / # requests will be challenged by example_auth_1 except the 2 Paths below
        match_type: prefix
        handler: proxy
      - path: /public/
        authenticator_name: none # requests to /public will be allowed without auth
        handler: proxy
      - path: /app/admin/
        authenticator_name: example_auth_2 # example_auth_2 protects this path, not auth_1
        handler: proxy

authenticators:
  # example_auth_1 loads users from a CSV and embeds a supplemental plaintext manifest
  # It also shows the login form to client browsers when login has failed
  example_auth_1:
    provider: basic # http basic auth (required)
    proxy_preserve: true # don't strip auth headers when proxying this request upstream
    users_file: /path/to/user-manifest.csv # optional users source file
    users_file_format: csv # required when users_file is set: csv, csvNoHeader or htpasswd
    users: # optional embedded users manifest (username: credential)
      user1: red123
    config: # optional provider-specific configs
      showLoginForm: true # with basic auth, causes the browser to show the login form
      realm: custom-realm-name # realm would be example_auth_1 if not overridden here

  # example_auth_2 loads users from an htpasswd file (assumed bcrypted credentials)
  example_auth_2:
    provider: basic
    users_file: /path/to/user-manifest.htpasswd # optional users source file
    users_file_format: htpasswd # required when users_file is set

  # example_auth_3 loads users from the embedded users manifest, credentials already bcrypted
  example_auth_3:
    provider: basic
    users:
      user1: $2y$asf;j2ihj0h8vabjkwdqbv29hq

  # example_auth_4 loads users from the embedded manifest, credentials injected from env,
  # and supports ClickHouse query params (user/password) in addition to Basic Auth
  example_auth_4:
    provider: clickhouse
    users:
      user1: ${USER1_PASSWORD_ENV} # ${ENV_NAME} substitution is supported

4 - AWS Integration

Trickster uses AWS credentials and SigV4 request signing in two places, and both are configured the same way:

  • Signing outbound requests to an origin — the sigv4 block on a backend, below.
  • Autodiscovery of AWS resources — the aws discovery provider, which reads an AWS API to keep an ALB pool current.

Credentials

Leaving the credential fields empty selects the standard AWS credential chain, in the order the AWS SDK resolves it:

  1. Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN)
  2. The shared credentials and config files (~/.aws/credentials, ~/.aws/config), honoring profile
  3. Web-identity tokens — this is how EKS IAM Roles for Service Accounts (IRSA) and EKS Pod Identity work
  4. AWS SSO
  5. The EC2 instance metadata service (IMDSv2)

Prefer the chain over static keys wherever the platform provides one: on EKS use IRSA or Pod Identity, on EC2 use an instance profile. Static keys are supported for environments that have nothing else.

Credential Lifecycle

Credentials are resolved lazily, on the first signed request. Trickster therefore starts even when the instance metadata service is briefly unreachable. Concurrent first requests share one resolution.

  • Failures are retried. A failed resolution is reused for 5 seconds and then tried again, so a momentary metadata or STS failure does not permanently disable signing, and an outage costs one blocked resolution per 5 seconds rather than one per request.
  • Expiring credentials refresh early. Temporary credentials (IRSA, Pod Identity, SSO, instance profiles, role_arn) are refreshed between 2.5 and 5 minutes before they expire, rather than after, so requests are not signed with credentials that lapse in flight.
  • Rejected credentials are renewed once. When the origin answers 403 with ExpiredToken, ExpiredTokenException, InvalidClientTokenId, UnrecognizedClientException or InvalidSignatureException, Trickster renews the credentials and resends the request once:
    • Credentials that expire (IRSA, Pod Identity, SSO, instance profiles, role_arn) are discarded and resolved again. Concurrent rejections of the same credentials cause a single refresh.
    • Credentials read from the shared credentials or config file, including temporary keys written there by a refresher, carry no expiry, so the files are read again instead, at most once every 5 seconds. A rewritten file therefore takes effect without a reload.
    • access_key and secret_key set in the sigv4 block, and keys from environment variables, cannot change while Trickster runs, so they are never retried.
  • One cache per backend. A backend’s proxy and health check requests share one set of credentials.
  • Clock skew. SigV4 signatures carry a timestamp, and AWS rejects requests more than 5 minutes off its clock with InvalidSignatureException. Keep the host clock synchronized.

Signing failures and credential retries are counted in trickster_proxy_sigv4_events_total (see metrics) and logged as warnings, at most once a minute per backend and kind.

Region

region may be set explicitly. When it is not, it is resolved from AWS_REGION / AWS_DEFAULT_REGION, the shared config file, or instance metadata. If none of those yields one, requests fail with an error naming every source that was tried.

The sigv4 Backend Block

sigv4 signs Trickster’s outbound requests to a backend’s origin.

backends:
  amp:
    provider: prometheus
    origin_url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-abc123
    sigv4:
      region: us-east-1
      # credentials omitted: use the chain (IRSA, instance profile, ...)
      # access_key: AKIA...
      # secret_key: ...        # redacted in config dumps and the health page
      # profile: production
      # role_arn: arn:aws:iam::123456789012:role/TricksterRead
      # service: aps           # default; see below
optionmeaning
regionregion to sign for; resolved from the environment when unset
access_key, secret_keya static credential pair — both or neither
profilea profile in the shared config file
role_arna role to assume with whatever the chain resolves first
servicethe AWS service to sign for; defaults to aps

The service default is aps — Amazon Managed Service for Prometheus. That is deliberate: earlier releases could sign for nothing else, so every existing config keeps working unchanged. Set service to sign for a different AWS service (es for OpenSearch, and so on).

access_key and secret_key must be provided together. A config supplying only one fails at startup rather than silently falling through to the chain and authenticating as a different principal.

secret_key is redacted wherever configuration is emitted — the config dump, the management API, logs, and error messages.

Amazon Managed Service for Prometheus

The common case. Point a prometheus backend at the workspace’s query URL and add a sigv4 block; Trickster caches and accelerates AMP queries as it does any other Prometheus origin.

backends:
  amp:
    provider: prometheus
    origin_url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-abc123
    sigv4:
      region: us-east-1

The IAM principal needs aps:QueryMetrics, aps:GetSeries, aps:GetLabels, and aps:GetMetricMetadata on the workspace.

Adding prometheus.flavor: amp restricts the backend to AMP’s read APIs (query, query_range, series, labels, label/<name>/values, metadata, rules and alerts). Every other path, including /api/v1/remote_write, is answered by Trickster with an error instead of being signed and forwarded. The flavor also adds the sigv4 block if it is missing.

backends:
  amp:
    provider: prometheus
    origin_url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-abc123
    prometheus:
      flavor: amp
    sigv4:
      region: us-east-1

Amazon CloudWatch PromQL

CloudWatch serves PromQL over a Prometheus-compatible API at https://monitoring.<region>.amazonaws.com, covering metrics ingested through OTLP and enriched AWS vended metrics. Queries are billed by the samples they scan, so a dashboard that refreshes repeatedly pays again for every sample it has already seen. Trickster’s delta cache fetches only the new part of each refresh, and request collapsing merges concurrent viewers into one upstream query.

backends:
  cloudwatch:
    provider: prometheus
    prometheus:
      flavor: cloudwatch
    sigv4:
      region: us-east-1
      # credentials omitted: use the chain (IRSA, instance profile, ...)

Grafana’s Prometheus data source points at Trickster as it would at any Prometheus server; Trickster does the signing.

flavor: cloudwatch applies CloudWatch’s routes, limits and defaults:

  • Signing: the sigv4 block is added if it is missing, and its service defaults to monitoring.
  • Origin: origin_url defaults to https://monitoring.<sigv4.region>.amazonaws.com. When the region comes from the environment instead of sigv4.region, set origin_url explicitly. An origin_url and sigv4.region naming different regions fail at startup.
  • Routes: the PromQL API’s query, query_range, series, labels and label/<name>/values are served and cached. The read-only AWS API calls that Grafana’s CloudWatch data source makes are relayed uncached (see Grafana’s CloudWatch Data Source). Every other path and operation is answered by Trickster with an error instead of being signed and forwarded, so a client cannot use Trickster’s credentials to change anything.
  • Cache keys: limit, which caps the series a query returns, is part of each cache key.
  • Truncated results: CloudWatch returns at most 500 series per query, or limit when smaller. A result cut short at that cap comes back with a warning and HTTP 200. Such a result is a sample of the series, and a later delta fetch can return a different sample, so Trickster never caches or merges one. It evicts any cached entry for the query, proxies the query to CloudWatch whole, and proxies that query directly for the backend’s timeseries_ttl. Each occurrence is counted in trickster_proxy_truncated_responses_total. Narrow the query, for example with an aggregation, to bring it back under the cap and into the cache.
  • Volatile window: volatile_window defaults to 2m, so the most recent two minutes are refetched while late data arrives. Any explicit volatile_window or volatile_window_points takes precedence.
  • Health check: the default probe is query=vector(1), which reads no metric and so scans no samples.

The IAM principal needs cloudwatch:GetMetricData and cloudwatch:ListMetrics; series, labels and label values need only cloudwatch:ListMetrics.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["cloudwatch:GetMetricData", "cloudwatch:ListMetrics"],
    "Resource": "*"
  }]
}

Grafana’s CloudWatch Data Source

Grafana’s Prometheus data source needs only the PromQL API. Grafana’s CloudWatch data source can also run PromQL, in its PromQL query mode, and works against Trickster too. Set the data source’s Endpoint to the Trickster backend’s URL, and its Default Region to the backend’s region. Any credentials will do, since Trickster signs with its own and ignores the data source’s.

That one Endpoint carries every AWS call the data source makes, not just PromQL. Trickster caches the PromQL calls and relays the others to their services uncached, re-signed with its own credentials for each service:

servicerelayed operationsused for
CloudWatchListMetrics, GetMetricData, DescribeAlarms, DescribeAlarmsForMetric, DescribeAlarmHistorySave & test, metric queries, variables, annotations
CloudWatch LogsDescribeLogGroups, GetLogGroupFields, StartQuery, GetQueryResults, StopQuery, GetLogEvents, ListAnomalies, ListAggregateLogGroupSummariesSave & test, Logs queries
EC2DescribeRegions, DescribeInstancesthe region picker, variables
Resource Groups TaggingGetResourcesvariables
Observability Access ManagerListSinks, ListAttachedLinksthe cross-account picker

Any other operation is refused with AccessDeniedException. A request signed for a region other than the backend’s is refused with ValidationException rather than answered from the backend’s region. Run one backend per region and select that region in Grafana. Grant Trickster’s IAM identity the read permissions for the operations your dashboards use; Save & test needs cloudwatch:ListMetrics and logs:DescribeLogGroups.

CloudWatch Limits

These CloudWatch limits apply per account and are not changed by Trickster. Delta fetches and request collapsing reduce how often a dashboard reaches them.

limitvalue
query requests per second300
discovery requests (series, labels, label values) per second10
concurrent queries, and concurrent discovery requests30 each
range per query, including range selectors and lookback7 days
execution timeout20 seconds

Exceeding a rate or size limit returns HTTP 422, and exceeding concurrency returns HTTP 429; Trickster does not cache either. Every selector must name a metric: {__name__=~"..."} alone is rejected.

Making AWS Vended Metrics Queryable

Metrics ingested through OTLP are queryable as they arrive. Classic AWS vended metrics, such as DynamoDB’s, become queryable through PromQL only after both enrichment settings are turned on for the account and region:

aws cloudwatch start-otel-enrichment --region us-east-1
aws observabilityadmin start-telemetry-enrichment --region us-east-1

Enriched metrics carry their resource attributes as UTF-8 labels, such as @aws.region and @resource.cloud.resource_id.

Cost Example

CloudWatch bills PromQL queries at $0.01 per million samples scanned. The figures below are an illustration, not a measurement; check the savings against your own bill.

Take a dashboard of 20 panels, each summing a rate over 50 series of one-minute data, showing the last 6 hours and refreshing every 60 seconds:

  • Without Trickster: each refresh scans about 6 hours plus a 5-minute lookback per series: 50 × 365 ≈ 18,000 samples per panel, or 365,000 per refresh. That is about 526 million samples a day, or about $5.26 a day ($158 per 30 days) for one screen left open.
  • With Trickster: after the first load, each refresh fetches only the new step and the 2-minute volatile window, plus the lookback: about 8 samples per series, 400 per panel, or 8,000 per refresh. That is about 11.5 million samples a day, or about $0.12 a day.

Every additional viewer of the same dashboard costs that much again without Trickster, and close to nothing with it, since concurrent requests for the same query are collapsed and later ones are served from the cache.

Live Tests

The CloudWatch tests in pkg/backends/prometheus/cloudwatch_live_test.go run against a real account and are skipped by default. They need a metric that is visible to PromQL in the account:

TRICKSTER_AWS_TEST=1 TRICKSTER_AWS_PROFILE=default TRICKSTER_CW_METRIC=ConsumedReadCapacityUnits go test ./pkg/backends/prometheus/ -run Live -v -count=1

Notes

  • Trickster signs with its own credentials; a client’s signature cannot be passed through, because Trickster rewrites requests (delta fetches change start and end) and AWS ties each signature to the host it was made for. A backend with a sigv4 block therefore drops the SigV4 fields a client sends (Authorization when it holds a SigV4 signature, X-Amz-Date, X-Amz-Security-Token and X-Amz-Content-Sha256) before caching or forwarding the request. Other Authorization values, such as basic auth for Trickster’s own authenticators, are kept. When pointing Grafana at Trickster, turn off the data source’s SigV4 auth; it is ignored either way.
  • Anyone who can reach a sigv4 backend reads with Trickster’s IAM identity, within the paths the backend serves. Give that identity only the read permissions listed above, and limit who can reach Trickster with network policy or one of Trickster’s authenticators. Serve different accounts or roles from separate backends, each with its own role_arn; cache entries are never shared between backends.
  • SigV4 signs a hash of the request body, so Trickster buffers a request body in order to sign it. This applies only to backends with a sigv4 block.
  • SigV4 is not supported for the ClickHouse native protocol, which is not HTTP; configuring both fails at startup.

IAM for Autodiscovery

The aws discovery provider needs read-only permission for whichever aws.service it is configured with.

aws.servicerequired IAM actions
ec2ec2:DescribeInstances
ecsecs:ListTasks, ecs:DescribeTasks

A minimal policy for service: ec2:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "ec2:DescribeInstances",
    "Resource": "*"
  }]
}

ec2:DescribeInstances does not support resource-level permissions, so the resource must be *; narrow the scope with a condition key if your environment requires it. ecs:ListTasks and ecs:DescribeTasks can be scoped to a cluster with the ecs:cluster condition key. Trickster only ever reads.

5 - Origin Resolution via DNS SRV

By default, an HTTP backend dials its origin by looking up the A/AAAA records of the origin_url host and connecting to the port in the URL. With origin_resolution.mode: srv, the host is instead treated as a DNS SRV owner name. Each new connection goes to a target from the SRV answer, on the port the answer publishes.

This lets a backend reach services whose ports are known only to DNS, such as:

  • ECS tasks in bridge network mode with hostPort: 0, registered in AWS Cloud Map with SRV records
  • Consul, Nomad or Kubernetes headless services exposed over DNS SRV

Combined with a rule and a hostname request rewriter, one backend can serve any number of SRV destinations, chosen per request.

Configuration

backends:
  sites:
    provider: rp
    origin_url: 'http://_app._tcp.example.internal'  # the port, if any, is ignored in srv mode
    origin_resolution:
      mode: srv              # a (the default, A/AAAA lookup) | srv
      resolver: ''           # optional host:port of a DNS server to query directly
      min_ttl: 5s            # floor for cached answers
      max_ttl: 60s           # ceiling for cached answers
      negative_ttl: 5s       # how long NXDOMAIN or an empty answer is cached
      tls_server_name: owner # owner (the default) | target
settingdefaultdescription
modeaa dials the URL host’s A/AAAA addresses at the URL port, which is the behavior when origin_resolution is absent. srv resolves the URL host as an SRV owner name.
resolverThe host:port of a DNS server to query directly. Direct queries make record TTLs visible. When empty, the Go standard library resolver is used. It reports no TTLs, so answers are cached for min_ttl.
min_ttl5sThe shortest time an answer is cached, whatever its record TTL
max_ttl60sThe longest time an answer is cached, whatever its record TTL. It also bounds how long the last good answer keeps serving while lookups fail.
negative_ttl5sHow long a name with nothing to dial is cached: NXDOMAIN, an empty answer, or a failed lookup with no last good answer to serve
tls_server_nameownerFor https origins, the name the origin’s certificate is verified against: owner is the URL host, as with any other backend; target is the SRV target that was dialed

resolver, the TTL settings and tls_server_name apply only to mode: srv; setting them with mode: a is a configuration error.

The owner name is used exactly as given. Trickster does not derive a service name such as _http._tcp from the URL scheme.

Routing Many Origins With One Backend

Without SRV resolution, a multi-tenant setup needs one backend per origin, each with its own fixed port, and every new origin is a config change and a reload. Origins whose ports are assigned dynamically can’t be expressed at all, because an A/AAAA record carries no port.

With SRV resolution, one rule, one rewriter, one router backend and one origin backend cover every origin that has SRV records. Adding an origin means publishing its DNS records; Trickster needs no config change or reload.

# DNS (example):
#   alpha.origins.example.com.    30 IN SRV 1 1 31842 node-07.svc.example.com.
#   _bravo._tcp.svc.example.com.  30 IN SRV 1 1 30117 node-02.svc.example.com.
#                                 30 IN SRV 1 1 30455 node-04.svc.example.com.
#   ; aliases, so a public hostname doesn't have to equal the service name:
#   bravo.origins.example.com.        60 IN CNAME _bravo._tcp.svc.example.com.
#   bravo-legacy.origins.example.com. 60 IN CNAME _bravo._tcp.svc.example.com.

request_rewriters:
  to-origin:
    instructions:
      # ${site} is the capture from the rule below. The rewritten name is
      # SRV-resolved at dial time, and the port comes from the SRV answer.
      - [ 'hostname', 'set', '${site}.origins.example.com' ]

rules:
  site-router:
    input_source: hostname
    input_type: string
    operation: rmatch
    # keep the capture strict: it becomes the upstream hostname
    operation_arg: '^(?P<site>[a-z0-9-]{1,63})\.example\.com$'
    cases:
      - matches: [ 'true' ]
        req_rewriter_name: to-origin
        next_route: origins
    # anything that doesn't match the pattern
    redirect_url: 'https://www.example.com/'

backends:
  router:
    provider: rule
    rule_name: site-router
    is_default: true

  origins:
    provider: rp
    origin_url: 'http://origins.example.com'  # placeholder; the rewriter sets the host per request
    preserve_host: true                       # the origin still sees e.g. Host: alpha.example.com
    path_routing_disabled: true
    origin_resolution:
      mode: srv
      resolver: '10.0.0.2:53'                 # optional; a direct resolver gives real TTLs
      min_ttl: 5s
      max_ttl: 60s
      negative_ttl: 5s

A request for alpha.example.com is rewritten to alpha.origins.example.com, which owns an SRV record, so it reaches node-07 on port 31842. A request for bravo-legacy.example.com is rewritten to bravo-legacy.origins.example.com, whose CNAME leads to the _bravo._tcp records. A host that matches the pattern but has no records gets a 502 Bad Gateway until its records are published.

Since the capture becomes a DNS name that Trickster resolves, keep the rule’s pattern strict. Every distinct name a client can produce costs a lookup, and its answer is cached.

How It Works

The transport’s dial is the only thing SRV resolution changes. The request URL keeps the SRV owner name, so URL handling, the Host header, logging, the cache key and the connection-pool key are the same as for any backend.

Target Selection

Targets are chosen as RFC 2782 describes:

  • Targets in the lowest-priority tier (the smallest priority value) are tried first, ordered by weighted random selection. As RFC 2782 specifies, a target with weight 0 has a small chance of being chosen first, and a tier whose weights are all 0 is ordered uniformly at random.
  • If a dial fails, the remaining targets in the tier are tried, then the next tier, all within the backend’s 10s connect timeout. Each target gets a share of the time left, which covers resolving its name and trying each of its addresses in turn, so neither an unreachable target nor an unreachable address can use it all up.
  • A target of . is ignored. An answer with no other targets means the service is not available.

Each target’s hostname is resolved to A/AAAA addresses with the same resolver. When the DNS server includes the target’s addresses in the additional section of the SRV answer, they are used without a further lookup. A target name that is an IP literal is dialed as given, and so is a URL host that a rewriter sets to an IP literal.

Caching

Answers are cached per owner name, and target addresses per target name. Each is kept for its record TTL, clamped to min_ttl and max_ttl. Concurrent dials for a name that needs a lookup share a single in-flight query.

When a cached answer expires, the dial waits up to 500ms for the refresh. If the refresh takes longer, the dial uses the last good answer while it is within max_ttl, and the refresh finishes in the background. When the lookup fails (a timeout, SERVFAIL or REFUSED), the last good answer keeps serving, retried no sooner than every min_ttl, until max_ttl has passed since it was looked up. NXDOMAIN and empty answers are answers rather than failures: they replace the last good answer at once and are cached for negative_ttl.

Each backend’s caches are bounded, so names derived from client-supplied hosts can’t grow them without limit. A config reload builds new caches, and in-flight requests finish on the old transport, as with any other reload.

Connection Pooling

max_idle_conns and max_concurrent_conns apply per SRV owner name, not per target, because the owner name is what the connection pool is keyed on. Idle connections to a target that has since left the answer are closed when they reach keep_alive_timeout.

TLS

For an https origin, the server name sent and verified is the SRV owner name (the URL host), as for any other backend. Set tls_server_name: target for services whose certificates name the target host instead. It can’t be combined with tls.server_name, which names a fixed server name for every target.

Health Checks

A backend health check probes through the same transport, so it is SRV-resolved too. Each probe reaches a single target, so it reports whether the service is reachable, not whether every target is.

Active per-target health checks, target ejection and load-balancing mechanisms are not part of SRV resolution. Use an ALB with dns_srv autodiscovery when you need them.

Validation

A backend with mode: srv fails validation when:

  • origin_url is not http:// or https://, or its host is an IP literal
  • resolver is not a host:port
  • min_ttl is greater than max_ttl, after defaults apply
  • tls_server_name: target is combined with tls.server_name

origin_resolution is not supported on alb, rule and static backends, or with a non-HTTP protocol.

Metrics

  • trickster_proxy_origin_srv_lookups_total counts SRV lookups by result. The results are hit, miss, stale (a failed refresh served the last good answer), negative (a cached NXDOMAIN, empty answer or failure) and error.
  • trickster_proxy_origin_srv_dial_attempts_total counts dial attempts to SRV targets by priority tier (0 is the preferred tier) and result (success or failure).

Both are labeled by backend_name. See metrics.

6 - Static File Server Backend

The Static Backend serves the contents of a local directory, rather than proxying requests to an upstream origin. Use it to host a website, a single-page application’s assets, a maintenance page, or any other set of files directly from Trickster, alongside (and with the same listeners, TLS, virtual hosting, authentication, logging and tracing as) your other Backends.

The file server is built into Trickster on the Go standard library, and provides:

  • GET and HEAD for files, with ETag and Last-Modified validators, conditional requests (If-None-Match, If-Modified-Since, If-Match, If-Unmodified-Since), and single, multi-part and If-Range byte range requests
  • a configurable default file (index.html) for directory requests
  • Content-Type detection from a built-in table of common web types, with your own additions and overrides
  • a configurable Cache-Control header, with overrides by file extension
  • a fileserver cache that holds small files in memory, kept current by filesystem change events (on Linux and Windows) and periodic revalidation, with the least recently used making room when it fills
  • compression for the Backend’s compressible_types, with each encoded rendition of a small file held in memory and reused
  • a not-found file, for a custom error page or a single-page application
  • Prometheus metrics for the files served and the cache
  • safe defaults: dotfiles are never served (other than /.well-known/), directory listings are off, and no request can reach a file outside of the configured root

Configuring

A Static Backend sets its provider to static, and requires a static options block that names the root directory to serve.

backends:
  website:
    provider: static
    hosts: [ www.example.com ]
    static:
      root: /var/www/html

Like any other Backend, this one is reachable at /website/ on its listeners, at / for requests whose Host is www.example.com, and at / for every request when it sets is_default: true.

Static Options

OptionDefaultDescription
root(required)Path to the directory holding the content to serve. It must exist and be a directory when the configuration is loaded. A relative path is resolved from Trickster’s working directory.
default_fileindex.htmlThe file served when a directory is requested (e.g., / or /docs/). It must be a plain file name that does not start with a .
cache_controlThe Cache-Control header value sent with every file. By default no Cache-Control header is sent, as with nginx and other common web servers, and a client judges how long a file stays fresh from its Last-Modified time.
cache_control_by_extensionA map of file extension to Cache-Control value, which takes precedence over cache_control for matching files.
response_headersA map of additional headers to attach to every response.
mime_typesA map of file extension to Content-Type, which adds to and takes precedence over the built-in types.
not_found_fileA file, as a path within the root, that is served in place of a plain 404 response. See Not-Found File.
not_found_status404The status not_found_file is served with: 404 for an error page, or 200 for a single-page application.
directory_listingfalseWhen true, a directory that has no default_file is answered with a listing of its contents.
cacheOptions for the Fileserver cache, which holds small files in memory, described below.

File extensions are case-insensitive, and may be written with or without the leading dot.

A fuller example:

backends:
  website:
    provider: static
    is_default: true
    static:
      root: /var/www/html
      default_file: index.html
      # html is revalidated on every use, so a deployment is visible immediately
      cache_control: no-cache
      cache_control_by_extension:
        # fingerprinted assets never change, and can be cached for a year
        .js: public, max-age=31536000, immutable
        .css: public, max-age=31536000, immutable
        .woff2: public, max-age=31536000, immutable
      response_headers:
        X-Content-Type-Options: nosniff
        X-Frame-Options: DENY
      mime_types:
        .md: text/markdown; charset=utf-8
      not_found_file: errors/404.html
      directory_listing: false
      cache:
        max_file_size_bytes: 1048576
        max_size_bytes: 134217728
        max_files: 10000
        revalidation_interval: 10s

Backend Options

These general Backend options work with a Static Backend as they do with any other: hosts, any_host_routing, listener_names, is_default, path_routing_disabled, require_tls, tls, authenticator_name, cors, access_log, tracing_name, compressible_types, latency_min and latency_max.

Options that describe proxying, caching or routing to an origin have no meaning for a file server. Rather than being silently ignored, these fail configuration validation when set on a Static Backend: paths, req_rewriter_name, origin_url, rule_name, alb, prometheus, mysql, graphite, influxdb, sigv4, protocol, h2c_prior_knowledge, preserve_host, proxy_only and is_template. Likewise, a static block on a Backend of any other provider fails validation.

A Static Backend has no origin to probe, so it always reports as available on the health endpoint. It can be the target of a Rule or a member of an ALB pool.

Requiring Authentication

To require users to authenticate before any file is served, attach an Authenticator to the Backend. Requests that fail authentication are rejected before the file server is consulted, so they reveal nothing about which files exist.

authenticators:
  staff:
    provider: basic
    users_file: /etc/trickster/htpasswd
    users_file_format: htpasswd

backends:
  intranet:
    provider: static
    authenticator_name: staff
    static:
      root: /var/www/intranet

Request Handling

RequestResponse
a file200 OK with the file
a directory, with a trailing slash200 OK with the directory’s default_file
a directory with a trailing slash, but no default_file404 Not Found, or a listing when directory_listing is true
a directory, without a trailing slash301 Moved Permanently to the same path with a trailing slash
a file, with a trailing slash404 Not Found
any path with a segment that starts with a ., other than a leading /.well-known/404 Not Found
anything that does not exist, can’t be read, or resolves outside of the root404 Not Found
OPTIONS204 No Content with an Allow header
any method other than GET, HEAD or OPTIONS405 Method Not Allowed with an Allow header

Every 404 Not Found in this table is answered with the not-found file when one is configured.

A missing default_file is deliberately a 404 rather than a 401 or 403: with listings off, a directory is not a resource that exists to be forbidden.

The trailing-slash redirect is relative (Location: docs/), so it is correct whether the Backend was reached by its hostname, as the default Backend, or under its /backend-name/ path.

Dotfiles

Any request path containing a segment that begins with a . is answered with a 404. This covers hidden files (/.env, /.htpasswd), everything beneath hidden directories (/.git/config), and attempts to reach them indirectly (/docs/../.env). Dotfiles are also left out of directory listings.

The one exception is /.well-known/, the location reserved by RFC 8615 for site-wide metadata such as security.txt, ACME HTTP-01 challenges and app-association files. It is exempt only as the first segment of the path (/docs/.well-known/ is refused), dotfiles within it are still refused (/.well-known/.secret), and it is still left out of directory listings. The list of exemptions is built in, and is deliberately not configurable.

Not-Found File

not_found_file names a file within the root to serve whenever the response would otherwise be a plain 404: a path that doesn’t exist, a directory with no default_file, a refused dotfile, and so on. not_found_status selects which of its two uses it is put to.

With the default status of 404, it is a custom error page. The page is sent as the body of the 404, with Cache-Control: no-cache and without the ETag, Last-Modified and Accept-Ranges headers, so that it can’t be revalidated, ranged over or reused as though it were the missing file. Any Range or conditional headers on the request are ignored.

    static:
      root: /var/www/html
      not_found_file: errors/404.html

With a status of 200, it is the fallback for a single-page application, whose routes (/dashboard, /users/42/edit) exist only in the browser. Every unknown path is answered with the application itself, exactly as if the file had been requested by its own path: with its validators, its Cache-Control, conditional and byte range support, and its held renditions. Requests for files that do exist are unaffected.

    static:
      root: /var/www/app
      not_found_file: index.html
      not_found_status: 200

If the not-found file is itself missing, the response is a plain 404. A request using a method other than GET or HEAD is never answered with it. When directory_listing is on, a directory that can be listed is listed rather than answered with it.

All file access is confined to the root by the operating system, not by path inspection alone. No request can read a file outside of the root, whether by .. traversal or through a symbolic link.

Symbolic links are followed only when they are relative links to a target inside the root. A link to anywhere outside of the root, and any link with an absolute target (even one that points inside the root), is answered with a 404.

Only regular files are served. Devices, sockets and named pipes inside the root are answered with a 404.

Content Types

A file’s Content-Type is resolved from, in order: the mime_types option, Trickster’s built-in table of common web types (HTML, CSS, JavaScript, JSON, WebAssembly, images, fonts, audio, video, archives, documents and more), and the host’s MIME database. The built-in table makes detection work the same in a minimal container image as on a full host. A file whose extension is unknown, or that has none, is identified from its first 512 bytes, and falls back to application/octet-stream.

Validators, Ranges and Compression

Every file is served with an ETag and a Last-Modified header, and matching conditional requests are answered with a 304 Not Modified.

Unless cache_control is set, no Cache-Control header is sent. A client then decides for itself how long to reuse a file before revalidating it, usually for a fraction of the time since the file was last modified, so files that have not changed in a long time are reused for longer. Where a change must be seen at once, such as the HTML entry point of an application with fingerprinted assets, set cache_control (or cache_control_by_extension) to no-cache, which has clients revalidate the file before each reuse.

The ETag is derived from the file’s modification time and size, the same two components used by nginx and other common web servers. Because it comes from file metadata, the ETag is the same whether a file is served from memory or from disk, and a HEAD or conditional request is answered from a stat of the file without reading it. Instances of Trickster serving copies of the same content issue the same ETag for a file, so long as the deployment preserves modification times (as rsync -a and tar do).

As with those servers, a file that is replaced by different content of exactly the same size and modification time keeps its ETag. If your deployment tooling pins or preserves modification times, give changed files a new name (such as a content fingerprint) or a new modification time.

Files whose Content-Type is in the Backend’s compressible_types are compressed for clients that accept it, in any of zstd, br, gzip and deflate. Those responses carry Vary: Accept-Encoding, and the weak form of the ETag (W/"..."), because the encoded bytes differ from the stored file the ETag describes. Byte ranges always address the stored file, so a 206 Partial Content response is never compressed, and carries the ETag unchanged.

How a file is compressed depends on whether the Fileserver cache can hold it. A file it can hold is compressed once, and the encoded rendition is held and reused, as described under Held Renditions. Any other file (a larger one, or any file when the cache is disabled) is compressed as it is sent, on every request.

Fileserver Cache

The Fileserver cache is configured in the cache block of the static options, and belongs to the Static Backend alone. It is unrelated to the caches Trickster uses to accelerate proxied Backends (including the memory cache provider): it is not configured in the caches section, a Static Backend has no cache_name, and nothing it holds is reachable through the cache purge endpoints.

Small files are read into memory the first time their content is needed, and served from memory afterward without touching the disk. Larger files are streamed from disk on every request.

Only a response that sends the whole file loads it. A HEAD request, a conditional request answered with a 304, and a byte range request are all served from disk without reading the file into memory, so they can’t be used to fill the cache. When many requests arrive at once for a file that is not yet held, it is read once and shared between them.

OptionDefaultDescription
disabledfalseWhen true, every request is served from disk.
max_file_size_bytes1048576 (1 MiB)The largest file that is held in memory.
max_size_bytes134217728 (128 MiB)The most memory the cache will use. Each held file counts as its size plus a fixed 1 KiB allowance for its bookkeeping, so empty and tiny files are bounded too.
max_files10000The most objects that are held, which also bounds the number of directories that are watched to twice this number. A file as stored, and each held rendition of it, is an object.
revalidation_interval10sHow often held files are compared to the disk, as a backstop to change events, and the only means of change detection on macOS and the BSDs.

Capacity is reserved before a file is read, so the limits hold even while many different files are being loaded at once. Once either limit is reached, the least recently used objects make room for new ones.

Eviction

When a file needs room that the cache doesn’t have, by either limit, the least recently used objects are evicted to make it. Recency is tracked without slowing requests down: serving a held file only marks it as used, and eviction passes over each marked object once, clearing its mark, before it will remove it. An object is therefore evicted only if it has gone unread since eviction last reached it.

Room is only ever made for a file that is then held. Eviction first works out what would have to go, and goes ahead only if that is enough; a file that there isn’t room for is served from disk, and costs the cache nothing it holds. A request for a file that is never asked for again therefore can’t flush content that is in use.

The work a request will do to find room is bounded, however large the cache is: it passes over a limited number of recently used objects before giving up. Those it passed are no longer marked as recently used, so if requests for room keep coming, objects that aren’t being read in between give way to them, while objects in constant use never do. The objects a request does evict are in proportion to the size of the file they make way for. No single file is admitted at the cost of more than 128 objects. With the default limits that is rarely a constraint, as a full cache is usually full by its count of objects, and one eviction then makes room for a file of any size; it matters to a cache with a very high max_files that is full of very small files, in which a large file is served from disk instead.

A file is never evicted to make room for a rendition of itself, as a rendition is only held alongside the file it was made from. Where there is room for only one of them, it is the file that is held, and its renditions are made from memory for each request. Room held by files that are still being read into memory can’t be evicted either; in the unlikely event that those account for the whole cache, a new file is served from disk.

Held Renditions

A held file of a compressible type is also held in the encodings clients ask for, each as an object of its own beside the file as stored, so that a file is compressed once rather than on every request.

A request’s Accept-Encoding header is read for the encodings Trickster supports (zstd, br, gzip and deflate), most preferred first: by the weight (q) the client gave each, and among encodings of equal weight, or when no weights are given (as by browsers), by Trickster’s own preference in the order just listed. An encoding takes the weight of its own entry, or else that of the wildcard (*) if there is one. Encodings the client refused (q=0) are left out, and so are any it weighted below identity, since the file as stored is always there to be sent instead. Then:

  1. If the file is held in any of those encodings, the most preferred of the renditions that are held is served.
  2. Otherwise, if the file is held as stored, it is encoded from memory in the client’s most preferred encoding, and the new rendition is held.
  3. Otherwise, the file is read from disk and encoded in the client’s most preferred encoding, and both the file as stored and the new rendition are held. A rendition that is held in some other encoding is never decoded to stand in for the file.

A new rendition is streamed to the client as it is encoded, and what was sent is kept to be held, so the first response is not delayed by the rendition being made for later ones. That first response is sent without a Content-Length; once held, a rendition is sent with its own.

Renditions count toward max_size_bytes and max_files like any other object, are evicted independently of one another, and are all dropped together when their file changes on disk. A rendition is only made for a response that sends the whole file: not for a HEAD, a byte range, or a conditional request (which may turn out to need no body; if it does need one, it is compressed as it is sent). Files smaller than 512 bytes are always sent as stored, as an encoding’s own framing outweighs what it would save. When an encoding turns out not to make a file smaller, that encoding isn’t tried for the file again until it changes; the other encodings are unaffected, and a client that prefers the unhelpful one is given the next it accepts.

When nothing a client accepts is left (for example identity;q=0 from a client that accepts no supported encoding), the file is sent as stored rather than refused with a 406, as other web servers do.

Serving and the Cache

Holding a file never delays serving it. Held files are read without locking. Everything that changes the cache (holding a file or a rendition, or releasing room that turned out not to be needed) is done after, and apart from, the response that led to it. The one thing a request asks of the cache before it responds is to reserve room, which is what keeps memory within its limits however many files are being read at once; if the cache is busy at that moment, the request doesn’t wait, and is served without being held. The file is then held by a later request for it.

Change Detection

Trickster watches the directories of the files it holds. A directory that a change leaves with nothing held stays watched for one to two revalidation_intervals, since the changed file is usually requested again soon; one emptied by eviction is let go of at once. Letting go of a watch is done in the background, so evicting files never makes a request wait on the operating system. When a file is modified, replaced, renamed or removed, it is dropped from memory (typically within milliseconds) and the next request reads it from disk again. Renaming or removing a directory drops every held file beneath it. A changed file is never updated in place, and a read that overlaps a change is discarded rather than stored, so a request is never served a mix of old and new content from memory. A change touches only what it affects: the one file, or the files beneath the one directory. Every other held file stays in memory, and files that are being read into memory at that moment are unaffected, so a busy deployment does not push the rest of the site back to disk.

Every revalidation_interval, each held file’s size, modification time and identity are compared to the disk, which catches changes that raise no event: a filesystem without change notifications (such as some network and container bind mounts), a changed symbolic link target, or a replaced root. On such filesystems, revalidation_interval is the longest a stale file can be served; lower it, or disable the fileserver cache, if that matters for your content.

On macOS and the BSDs, the fileserver cache doesn’t use change events at all, and relies on revalidation alone, so a change can take up to revalidation_interval to be served. The event library’s kqueue support, which those platforms use, can close a file descriptor twice as a watch is removed, and the second close can land on a file that a request has open, so that another file’s content is served in its place (fsnotify#788). Change events will be enabled on these platforms once a fixed release is available. For local development on macOS, a short revalidation_interval (e.g., 1s) keeps edits visible quickly.

On every platform, a file that is found not to be the one at its path as it is opened is opened again, and answered with a 503 if it is still not; and a response sent from disk is broken off, rather than completed, if its file descriptor no longer refers to that file once the body is sent.

Atomic Deployments

Deploying by swapping a symbolic link is supported. When root is a link (e.g., /var/www/current → /var/www/releases/42) and the link is repointed, Trickster notices at the next revalidation_interval, switches to the new directory, and drops every held file. Requests in flight during the switch complete against the previous release.

Directory Listings

With directory_listing: true, a request for a directory that has no default_file is answered with an HTML page linking to the directory’s files and subdirectories, with their sizes and modification times. Dotfiles, links that can’t be followed, and anything that is not a regular file or directory are left out. A directory that does have a default_file is always answered with that file. Listings are generated per request, and are sent with Cache-Control: no-cache.