This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Backends

Configuring the upstream origins that Trickster accelerates.

1 - Using Multiple Backends with a single Trickster instance

Trickster supports proxying to multiple backends in a single Trickster instance, by examining the inbound request and using a multiplexer to direct the proxied request to the correct upstream origin, in the same way that web servers support virtual hosting.

There are 2 ways to route to multiple backends.

  • HTTP Pathing
  • DNS Aliasing

Basic Usage

To utilize multiple backends, you must craft a Trickster configuration file to be read when Trickster starts up - operating with environment variables or command line arguments only supports accelerating a single backend. The example.full.yaml provides good documentation and commented sections demonstrating multiple backends. The config file should be placed in /etc/trickster/trickster.yaml unless you specify a different path when starting Trickster with the -config command line argument.

Each backend that your Trickster instance supports must be explicitly enumerated in the configuration file. Trickster does not support open proxying.

Example backend configuration:

backends:
  my-backend-01: # routed via http://trickster:8480/my-backend-01/
    provider: prometheus
    origin_url: http://my-origin-01.example.com

  my-backend-02: # routed via http://trickster:8480/my-backend-02/
    hosts: [ my-fqdn-02.example.com ] # or http://my-fqdn-02.example.com:8480/
    provider: prometheus
    origin_url: http://my-origin-02.example.com

  my-backend-03:
    path_routing_disabled: true # only routable via Host header, an ALB, or a Rule
    hosts: [ my-fqdn-03.example.com ] # or http://my-fqdn-03.example.com:8480/
    provider: prometheus
    origin_url: http://my-origin-02.example.com

Default Backend

Whether proxying to one or more upstreams, Trickster has the concept of a “default” backend, which means it does not require a specific DNS hostname in the request, or a specific URL path, in order to proxy the request to a known backend. When a default backend is configured, if the inbound request does not match any mapped backends by path or FQDN, the request will automatically be routed through the default backend. You are probably familiar with this behavior from when you first tried out Trickster with the using command line arguments.

Here’s an example: if you have Trickster configured with a backend named foo that proxies to http://foo/ and is configured as the default backend, then requesting http://trickster/image.jpg will initiate a proxy request to http://foo/image.jpg, without requiring the path be prefixed with /foo. But requesting to http://trickster/foo/image.jpg would also work.

The default backend can be configured by setting is_default: true for the backend you have elected to make the default. Having a default backend is optional. In a single-backend configuration, Trickster will automatically set the sole backend as is_default: true unless you explicitly set is_default: false in the configuration file. If you have multiple backends, and don’t wish to have a default backend, you can just omit the value for all backends. If you set is_default: true for more than one backend, Trickster will exit with a fatal error on startup.

Path-based Routing Configurations

In this mode, Trickster will use a single FQDN but still map to multiple upstream backends by path. This is the simplest setup and requires the least amount of work. The client will indicate which backend is desired in URL Path for the request.

Example Path-based Configuration with Multiple Backends:

backends:
  # backend1 backend
  backend1:
    origin_url: 'http://prometheus.example.com:9090'
    provider: prometheus
    cache_name: default
    is_default: true
  # "foo" backend
  foo:
    origin_url: 'http://influxdb-foo.example.com:9090'
    provider: influxdb
    cache_name: default
  # "bar" backend
  bar:
    origin_url: 'http://prometheus-bar.example.com:9090'
    provider: prometheus
    cache_name: default

Using HTTP Path as the Backend Routing Indicator

The client prefixes the Trickster request path with the Backend Name.

This is the recommended method for integrating with applications like Grafana.

Example Client Request URLs:

DNS Alias Configuration

In this mode, multiple DNS records point to a single Trickster instance. The FQDN used by the client to reach Trickster is mapped to specific backend configurations using the hosts list. In this mode, the URL Path is not considered during Backend Selection.

Example DNS-based Backend Configuration:

backends:
  # backend1 backend
  backend1:
    hosts: # users can route to this backend via these FQDNs, or via `/backend1`
      - 1.example.com
      - 2.example.com
    origin_url: 'http://prometheus.example.com:9090'
    provider: prometheus
    cache_name: default
    is_default: true
  # "foo" backend
  foo:
    hosts: # users can route to this backend via these FQDNs, or via `/foo`
      - trickster-foo.example.com
    origin_url: 'http://prometheus-foo.example.com:9090'
    provider: prometheus
    cache_name: default
  # "bar" backend
  bar:
    hosts: # users can route to this backend via these FQDNs, or via `/bar`
      - trickster-bar.example.com
    origin_url: 'http://prometheus-bar.example.com:9090'
    provider: prometheus
    cache_name: default

Example Client Request URLs:

Note: It is currently possible to specify the same FQDN in multiple backend configurations. You should not do this (obviously). A future enhancement will cause Trickster to exit fatally upon detection at startup.

Disabling Path-based Routing for a Backend

You may wish for a backend to be inaccessible via the /backend_name/ path, and only by Hostname or as the target of a rule or ALB. You can disable path routing by setting path_routing_disabled: true for the backend, as in this example, which requires the Request’s Host header match 1.example.com or 2.example.com in order to be routed to the backend:

backends:
  backend1:
    hosts:
      - 1.example.com
      - 2.example.com
    origin_url: 'http://prometheus.example.com:9090'
    provider: prometheus
    cache_name: default
    is_default: false
    path_routing_disabled: true # this will disable routing through /backend1

2 - Authenticator

Trickster 2.x provides an Authenticator capability that allows you to protect Backends with an Authentication layer.

Authenticator resources are defined globally by name, and then mapped into any Backend and/or Path configuration as needed. Authenticator users can be loaded from htpasswd or csv files, or directly in the Trickster config file. You can provide credentials in plaintext, bcrypt, apache md5-script, as well as legacy formats rsa-256 and rsa-512.

Authenticators work with all Backend provider types. Requests are handled by their respective Authenticators before all other Handlers (e.g., Caches, Rules, Request Rewrites, ALB Routes, etc.).

Native MySQL listeners terminate mysql_native_password authentication and have stricter credential-source requirements than HTTP authenticators. See the MySQL Provider Guide.

If a request is routed via a Trickster ALB or Rule Backend through to multiple other Backends - each having different Authenticator configurations - the authentication behavior is currently undefined. In an upcoming Beta release, we will define this use case to either use the Authenticator config (if set) of the very first Backend that handled the request; or to use the first defined Authenticator regardless of how deep into the Backend chain it is.

By default, when an Authenticator handles and successfully authenticates a request, the request’s Auth credentials (e.g., Authorization Header for Basic Auth) are stripped before the request is cloned for any necessary proxying. Setting proxy_preserve: true will preserve these headers instead of stripping them.

Path Protection

When you map an Authenticator to a Backend, all Paths defined for that Backend are protected by the Authenticator. However, you can override the Authenticator on a per-Path basis by including the authenticator_name in a Path config. To bypass a Backend-wide Authenticator in a Path config, use authenticator_name: none. This allows for a few possibilities:

  • A Backend config with a default / Backend-wide Authenticator that has:

    • specific Paths that do not require Authentication
    • specific Paths mapped to different Authenticators than the default

  • A Backend config with no Authenticator defined (so all Paths by default are unprotected) that has:

    • specific Paths mapped to different Authenticators than the default

See the example Backend configs below for more details.

Authenticator Providers

Trickster’s Authenticator feature currently supports Basic Auth and ClickHouse-compatible authentication. It was designed with extensibility in mind should there be value in adding additional Authentication providers.

Basic Auth Provider

Basic Auth is supported by using provider: basic in the Authenticator config.

By setting the showLoginForm: true config (see example_auth_1 in the blob below), Trickster will return a WWW-Authenticate: Basic realm="custom-realm-name" header on any request that requires but fails authentication, causing the login form to pop up. When showLoginForm is not present or non-true, Trickster responds with a 401 Unauthorized but does not ask the Basic Auth login form to show.

The realm attribute value defaults to the Authenticator name (e.g., example_auth_1) but can be overridden with the realm config as in the example.

If the user data changes (e.g. updated users_file contents or updated embedded users list), you must send a SIGHUP or other means to reload the Trickster config before the new user pool is processed.

ClickHouse Auth Provider

ClickHouse Auth is supported by using provider: clickhouse in the Authenticator config.

ClickHouse authentication is the same as Basic Auth, except you can also provide user and password URL params.

Example Authenticator Configs

# NOTE: Required options unrelated to Authenticators have been omitted from this
# example. It does not represent a fully-functioning Trickster Config.
# See the 'examples' directory for working copy/paste config examples.

backends:
  backend01:
    provider: reverseproxy # authenticators work with all backend providers
    authenticator_name: example_auth_1 # protects backend01 with example_auth_1 authenticator
    origin_url: https://example.com
    paths:
      - path: / # all requests are protected by example_auth_1 
        match_type: prefix
        handler: proxy

  backend02:
    provider: reverseproxy # no backend-wide authenticator
    origin_url: https://example.com
    paths:
      - path: / # requests will be allowed without auth except the 2 Paths below
        match_type: prefix
        handler: proxy
      - path: /private/
        authenticator_name: example_auth_2 # example_auth_2 protects this path only
        handler: proxy
      - path: /admin/
        authenticator_name: example_auth_3 # example_auth_3 protects this path only
        handler: proxy

  backend03:
    provider: reverseproxycache
    authenticator_name: example_auth_1 # protects backend03 with example_auth_1 authenticator
    origin_url: https://example.com
    paths:
      - path: / # requests will be challenged by example_auth_1 except the 2 Paths below
        match_type: prefix
        handler: proxy
      - path: /public/
        authenticator_name: none # requests to /public will be allowed without auth
        handler: proxy
      - path: /app/admin/
        authenticator_name: example_auth_2 # example_auth_2 protects this path, not auth_1
        handler: proxy

authenticators:
  # example_auth_1 loads users from a CSV and embeds a supplemental plaintext manifest
  # It also shows the login form to client browsers when login has failed
  example_auth_1:
    provider: basic # http basic auth (required)
    proxy_preserve: true # don't strip auth headers when proxying this request upstream
    users_file: /path/to/user-manifest.csv # optional users source file
    users_file_format: csv # required when users_file is set
    users: # optional embedded users manifest (username: credential)
      user1: red123
    config: # optional provider-specific configs
      showLoginForm: true # with basic auth, causes the browser to show the login form
      realm: custom-realm-name # realm would be example_auth_1 if not overridden here

  # example_auth_2 loads users from an htpasswd file (assumed bcrypted credentials)
  example_auth_2:
    provider: basic
    users_file: /path/to/user-manifest.htpasswd # optional users source file
    users_file_format: htpasswd # required when users_file is set

  # example_auth_3 loads users from the embedded users manifest, credentials already bcrypted
  example_auth_3:
    provider: basic
    users:
      user1: $2y$asf;j2ihj0h8vabjkwdqbv29hq

  # example_auth_4 loads users from the embedded manifest, credentials injected from env,
  # and supports ClickHouse query params (user/password) in addition to Basic Auth
  example_auth_4:
    provider: clickhouse
    users:
      user1: ${USER1_PASSWORD_ENV} # ${ENV_NAME} substitution is supported

3 - TLS Support

Trickster supports TLS on both the frontend server and backend clients.

Basics

To enable the TLS server, specify the tls_port, and optionally, the tls_address in one of the listeners of your config file. For example:

listeners:
  # default is built-in and is the default listener for backends
  # it uses these values by default:
  default:
    port: 8480
    tls_port: 8483
    # listen on all interfaces
    tls_address: ''

Note, Trickster will only start listening on a TLS port if at least one origin mapped to the named listener has a valid certificate and key configured.

Each origin section of a Trickster config file can be augmented with the optional tls section to modify TLS behavior for front-end and back-end requests. For example:

backends:
  example: # example backend
    tls:   # TLS settings for example backend
      # frontend configs
      full_chain_cert_path: '/path/to/my/cert.pem'
      private_key_path: '/path/to/my/key.pem'
      # backend configs
      insecure_skip_verify: true
      certificate_authority_paths: [ '/path/to/ca1.pem', '/path/to/ca2.pem' ]
      client_cert_path: '/path/to/client/cert.pem'
      client_key_path: '/path/to/client/key.pem'

Server Configs - used when responding to clients

Each backend contributes up to 1 certificate and key pair, as configured in the TLS section of the backend config (demonstrated above). A listener serves the certificates of all backends mapped to it, selecting per-handshake by SNI (see Certificate Selection (SNI) below), so a single TLS listener can serve many hostnames.

If the path to any configured Certificate or Key file is unreachable or unparsable, Trickster will exit upon startup with an error providing reasonable context.

You may use the same TLS certificate and key for multiple backends, depending upon how your Trickster configurations are laid out. Any certificates configured by Trickster must match the hostname header of the inbound http request (exactly, or by wildcard interpolation), or clients will likely reject the certificate for security issues.

Certificate Selection (SNI)

When a listener has multiple certificates, Trickster selects the certificate for each TLS handshake in this order:

  1. Exact match of the client’s SNI hostname against a certificate’s Subject Alternative Names
  2. Wildcard match (e.g. a *.example.com certificate for foo.example.com)
  3. A linear compatibility scan (covers clients that send no SNI, and IP SANs)
  4. The listener’s first certificate, if nothing else matches

The exact and wildcard lookups are index-based, so per-handshake selection cost is independent of the number of certificates a listener serves.

Automatic Certificate Rotation Detection

Trickster automatically detects when a serving certificate is renewed in place on disk — same file paths, config untouched, as performed by tools like certbot or cert-manager — and hot-swaps the renewed certificate into the live listener. This is entirely independent of configuration reloads: no manual reload, auto_reload_interval, or config change is required.

  • Detection is hybrid: filesystem events (via fsnotify) trigger a near-immediate check where the platform and filesystem support them, and a timer-based poll runs regardless — both as the fallback for deployments where filesystem events don’t work (e.g. some network and FUSE filesystems) and as a self-healing backstop for missed events. If event watching is unavailable, detection silently degrades to poll-only.
  • Every check compares file contents (not modification times or event payloads), so detection works across platforms and through the atomic symlink swaps used by Kubernetes Secret and projected volumes.
  • The certificate, key (and any associated CA bundle) files are watched and validated as one unit: if a poll observes a mid-rotation partial state (e.g. the cert file updated before the key file), the mismatched pair is never served; the last-good pair keeps serving and the change is retried on the next poll.
  • Read errors and invalid content never disable detection: the watcher keeps retrying, logs a warning after sustained failures, and the last-good certificate keeps serving. Deleting the files is treated as a persistent failure, not as certificate removal.
  • Startup behavior is unchanged: an unreadable or invalid configured certificate is still a fatal startup error. Only post-startup source failures are tolerated.

Rotation detection is on by default and is configured per listener. tls_watch_interval sets the fallback/backstop poll cadence (default 30s); filesystem events, where available, apply rotations within moments regardless of the interval. Setting the interval to 0 disables rotation detection entirely (events included):

listeners:
  default:
    tls_port: 8483
    # backstop poll interval for the cert/key files of mapped backends
    # (default 30s); filesystem events accelerate detection where supported.
    # set to 0 to disable automatic rotation detection.
    tls_watch_interval: 30s

Note: automatic rotation detection currently applies to HTTP(S) listeners. Native-protocol listeners (e.g. mysql) pick up rotated certificates on config reload.

Hot Swap and No-Close Semantics

Certificate swaps — whether from a config reload or automatic rotation detection — never restart, drain, or rebind the listener:

  • The certificate is consulted only at handshake time, so established connections (including keep-alive connections and in-flight requests) are untouched by a swap; they continue on the certificate they were handshaken with until they close naturally.
  • Only new handshakes see the new certificate.

Certificate Inventory (mgmt)

The mgmt listener exposes a read-only, per-listener certificate inventory at /trickster/certificates (configurable via mgmt.certificates_handler_path). Each entry reports the certificate’s id, source kind (file, memory or config), common name, subject alternative names, validity window and last-load time. The inventory never includes key material.

Observability

Certificate rotation and inventory are covered by the following metrics: trickster_tls_certificate_expiration_time_seconds, trickster_tls_certificate_last_load_time_seconds, trickster_tls_certificate_swaps_total, trickster_tls_certificate_validation_failures_total, trickster_tls_watcher_errors_total, and trickster_tls_certificate_store_size. See metrics.md and the example alerting rules in examples/alerting for cert-expiry, sustained-validation-failure and watcher-staleness alerts.

Client Configs - used when proxying to an origin

Each backend’s TLS configuration can also configure the https client used for making requests against the origin as demonstrated above.

insecure_skip_verify will instruct the http client to ignore hostname verification issues with the upstream origin’s certificate, and process the request anyway. This is analogous to -k | --insecure in curl.

certificate_authority_paths will provide the http client with a list of certificate authorities (used in addition to any OS-provided root CA’s) to use when determining the trust of an upstream origin’s TLS certificate. In all cases, the Root CA’s installed to the operating system on which Trickster is running are used for trust by the client.

To us Mutual Authentication with an upstream origin server, configure Trickster with Client Certificates using client_cert_path and client_key_path parameters, as shown above. You will likely need to also configure a custom CA in certificate_authority_paths to represent your certificate signer, unless it has been added to the underlying Operating System’s CA list.

4 - Health Checks

Trickster Service Health - Ping Endpoint

Trickster provides a /trickster/ping endpoint that returns a response of 200 OK and the word pong if Trickster is up and running. The /trickster/ping endpoint does not check any proxy configurations or upstream origins. The path to the Ping endpoint is configurable, see the configuration documentation for more information.

Upstream Connection Health - Backend Health Endpoints

Trickster offers health endpoints for monitoring the health of the Trickster service with respect to its upstream connection to origin servers.

General Endpoint

The main health check path is /trickster/health, which by default will return a text/plain summary of the backend health. You can request YAML or JSON format using the appropriate Accept header, or by providing a ?json or ?yaml query param.

Backend-Specific Endpoints

Each HTTP backend’s health check path is /trickster/health/BACKEND_NAME. For example, if your backend is named foo, you can perform a health check of the upstream server at http://<trickster_address:port>/trickster/health/foo. Native-protocol backends publish their scheduled status through the general endpoint without registering a synthetic HTTP origin route.

The backend health path prefix /trickster/health/ is customizable. See the example.full.yaml for more info about setting the health_handler_path configuration, or refer to this example:

mgmt:
  # this overrides the default '/trickster/health' to '/-/trickster/health'
  health_handler_path: /-/trickster/health

The behavior of a health request will vary based on the Backend provider, as each has their own health check protocol. For example, with Prometheus, Trickster makes a request to /query?query=up and (hopefully) receives a 200 OK, while for InfluxDB the request is to /ping which returns a 204 No Content.

Supported TSDB Providers are pre-configured in Trickster to perform a suitable health check operation, however these can be overridden in the configuration file.

For non-TSDB Backends, the default behavior is to make a GET request to http://origin_url:port/ and expect a 2xx response. However, all aspects of the Health Check request and expected response are configurable per-Backend.

Native MySQL Health Checks

The MySQL backend uses the same interval scheduler, timeout, transition thresholds, status registry, metrics, reload carryover, and shutdown lifecycle as HTTP health checks. Each probe opens a fresh connection with the backend’s configured origin credentials and TLS policy, completes authentication, executes COM_PING, and closes the connection.

backends:
  mysql1:
    provider: mysql
    origin_url: mysql://health-user:password@mysql.example:3306/analytics
    healthcheck:
      interval: 5s
      timeout: 3s
      failure_threshold: 3
      recovery_threshold: 3

Only interval, timeout, failure_threshold, and recovery_threshold apply to native probes. HTTP verbs, paths, headers, bodies, and expected HTTP response options are configuration errors for MySQL. Diagnostics exposed in health status and logs are limited to sanitized authentication, TLS, timeout, refused-connection, connection, and server-error categories.

Basic Health Check Configuration Example

backends:
  server1:
    provider: reverseproxycache
    origin_url: http://server1
    healthcheck: # all values below are optional
      verb: HEAD
      path: /health

Health Check With Exhaustive Request/Response Options

backends:
  server1:
    provider: reverseproxy
    origin_url: http://server1
    healthcheck: # all values below are optional
      # 
      ## customizing the health check request
      #
      verb: HEAD
      scheme: https
      host: alternate-hostname.example.com
      path: /health
      query: param1=value1&param2=value2
      headers:
        User-Agent: health-check-agent
      # if using a POST or PUT method, you can provide a string body
      # body: "my health check body"
      #
      ## customizing the expected response
      #
      # hc fails if a response takes longer than 1s
      timeout: 1000ms
      # hc fails if the response code is not in the list
      expected_codes: [ 200, 204, 206, 301, 302, 304 ]
      #
      # hc fails if these response headers are not present and have the expected value
      expected_headers:
        X-Health-Check-Status: success
      # hc fails if the stringified response body does not match the expected value
      expected_body: "pass"

See more examples in example.full.yaml.

Health Check Integrations with Application Load Balancers

By default, a Backend will only initiate a health check on-demand, upon receiving a request to its health endpoint.

To facilitate integrations with the Trickster Application Load Balancer provider, additional options provide for 1) timed interval health checks and 2) thresholding for consecutive successful or unsuccessful health checks that determine the backend’s overall health status.

Example Health Check Configuration for use in ALB

backends:
  server1:
    provider: reverseproxy
    origin_url: http://server1
    healthcheck:
      path: /health
      timeout: 1000ms      # timeout should be <= interval
      # for ALB integration:
      interval: 1000ms     # auto-poll health every 1s
      failure_threshold: 3  # backend is unhealthy after 3 consecutive failures
      recovery_threshold: 3 # backend is healthy after 3 consecutive successes

The Prometheus default probe is /api/v1/query?query=up. Some multi-tenant Prometheus gateways reject an unbounded up with 400 bad_data: "too many series found", which keeps the member out of any ALB pool it belongs to. Override healthcheck.query with a bounded expression the backend accepts (for example query=vector(1)) when probing such backends.

Other Ways to Monitor Health

In addition to the out-of-the-box health checks to determine up-or-down status, you may want to setup alarms and thresholds based on the metrics instrumented by Trickster. See metrics.md for collecting performance metrics about Trickster.