This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Routing & Load Balancing

Directing requests across multiple backends with the ALB, autodiscovery and Rule engine.

1 - Application Load Balancer

Trickster 2.x provides an Application Load Balancer that is easy to configure and provides unique features to aid with Scaling, High Availability and other applications. The ALB supports several balancing Mechanisms:

MechanismConfigProvidesDescription
Round RobinrrScalinga basic, stateless round robin between healthy pool members
Power of Two Choicesp2cScalingdraws two healthy pool members at random and routes to the one with fewer requests in flight
Least ConnectionslcScalingroutes to the healthy pool member with the fewest requests in flight
Least TimeltSpeedroutes to the healthy pool member with the lowest response latency, scaled by its requests in flight
Highest Random WeighthrwAffinityconsistently routes each client, tenant or other key to the same healthy pool member
Time Series MergetsmFederationuses scatter/gather to collect and merge data from multiple replica tsdb sources
First ResponsefrSpeedfans a request out to multiple backends, and returns the first response received
First Good ResponsefgrSpeedfans a request out to multiple backends, and returns the first response received with a status code < 400
Newest Last‑ModifiednlmFreshnessfans a request out to multiple backends, and returns the response with the newest Last-Modified header
User RouterurControlInspects the credentials in the Request and routes it based on the Username
Connect RaceraceSpeedconnects a tcp or tls stream connection to several pool members at once and relays over the first to connect
UDP MirrormirrorReplicationcopies every datagram of a udp stream session to every healthy pool member, and answers from the first

Integration with Backends

The ALB works by applying a Mechanism to select one or more Backends from a list of Healthy Pool Members, through which to route a request. Pool member names represent Backend Configs (known in Trickster 0.x and 1.x as Origin Configs) that can be pre-existing or newly defined.

All settings and functions configured for a Backend are applicable to traffic routed via an ALB - caching, rewriters, rules, tracing, TLS, etc.

In Trickster configuration files, each ALB itself is a Backend, just like the pool members to which it routes. This makes it possible to configure infinite loops (e.g., where ALB1 has ALB2 in its pool, and ALB2 has ALB1 in its pool). However, at startup Trickster will validate ALB configurations by following all ALBs’ possible paths, and exit with a startup failure if any infinite loops are detected.

In addition to (or instead of) a static pool list, an ALB’s pool membership can be discovered and kept current automatically at runtime — from Kubernetes, the AWS, Google Cloud and Azure APIs, the Docker Engine, the Consul or Nomad service registries, DNS records, an HTTP endpoint, or a watched member-list file. See ALB Autodiscovery.

Mechanisms Deep Dive

Each mechanism has its own use cases and pitfalls. Be sure to read about each one to understand how they might apply to your situation.

Basic Round Robin

A basic Round Robin rotates through a pool of healthy backends used to service client requests. Each time a client request is made to Trickster, the round robiner will identify the next healthy backend in the rotation schedule and route the request to it.

To keep each client on the member it was first sent to, add Sticky Sessions to the ALB. For affinity without session state, see Highest Random Weight.

Weighted Round Robin

Trickster supports Weighted Round Robin with a first-class integer weight on pool entries. A pool entry may be a plain backend name (weight 1) or a mapping with an explicit weight:

pool:
  - node01            # weight 1
  - name: node02
    weight: 3         # receives 3 of every 4 requests

Apportionment is exact: over any totalWeight consecutive requests against a stable healthy pool, each member is selected exactly weight times. A heavier member’s turns are spread through the rotation rather than taken back to back: two members weighted 3 and 2 are served A B A A B, not A A A B B. Weights also carry through from autodiscovery sources that convey them (DNS SRV record weights, member-file weight fields); see ALB Autodiscovery.

Weights apply to every mechanism that selects a single member per request, though only Round Robin makes them an exact guarantee; see Weights and the Selection Mechanisms. Fan-out mechanisms (fr, fgr, nlm, tsm) dispatch to every healthy member regardless of weight.

More About Our Round Robin Mechanism

Trickster’s Round Robin Mechanism works by maintaining an atomic uint64 counter that increments each time a request is received by the ALB. With uniform weights, the ALB performs a modulo operation on the request’s counter value, with the denominator being the count of healthy backends in the pool; the resulting value, ranging from 0 to len(healthy_pool) - 1, indicates the assigned backend based on the counter and current pool size. With mixed weights, the modulo denominator becomes the pool’s total weight, and the result indexes a rotation schedule, computed once each time the set of healthy members changes, in which every member’s turns are evenly spaced. Selection remains lock- and allocation-free in both forms. The counter starts at a random value, so a fleet of Trickster replicas started together does not send its first requests to the same pool member.

Example Round Robin Configuration

backends:

  # traditional Trickster backend configurations

  node01:
    provider: reverseproxycache # will cache responses to the default memory cache
    path_routing_disabled: true # disables frontend request routing via /node01 path
    origin_url: https://node01.example.com # make requests with TLS
    tls: # this backend might use mutual TLS Auth
      client_cert_path: ./cert.pem
      client_key_path: ./cert.key

  node02:
    provider: reverseproxy      # requests will be proxy-only with no caching
    path_routing_disabled: true # disables frontend request routing via /node02 path
    origin_url: http://node-02.example.com # make unsecured requests
    request_headers: # this backend might use basic auth headers
      Authoriziation: "basic jdoe:${NODE_02_AUTH_TOKEN}"

  # Trickster 2.x ALB backend configuration, using above backends as pool members

  node-alb:
    provider: alb
    alb:
      mechanism: rr # round robin
      pool:
        - node01 # as named above; weight 1
        - node02
        # to weight the pool, use the mapping form on any entry:
        # - name: node02
        #   weight: 2 # node02 would receive 2 of every 3 requests

Here is the visual representation of this configuration:

Power of Two Choices

The Power of Two Choices (p2c) mechanism draws two healthy pool members at random and routes the request to whichever has fewer requests in flight. It keeps load nearly as even as inspecting every member would, at a cost that does not grow with the size of the pool, and unlike Round Robin it reacts to a member that has become slow: requests pile up there, so it loses more of its draws. It is a good default for large pools and for requests whose cost varies widely.

backends:
  api:
    provider: alb
    alb:
      mechanism: p2c # or power_of_two_choices
      pool: [ node01, node02, node03 ]

Least Connections

The Least Connections (lc) mechanism routes each request to the healthy pool member with the fewest requests in flight. It reads every member on every request, so it suits small pools; Power of Two Choices approximates it for large ones. While several members are tied, as all are when the pool is idle, they take turns.

backends:
  api:
    provider: alb
    alb:
      mechanism: lc # or least_connections
      pool: [ node01, node02 ]

Least Time

The Least Time (lt) mechanism routes each request to the healthy pool member that has been answering fastest. Each member’s score is its latency average multiplied by one more than its requests in flight, and the lowest score wins, so a fast member is preferred until it is busy enough that a slower one would answer sooner.

Latency is the time from routing a request to the first byte of its response. The average rises at once when a member slows down and falls gradually, over lt.decay, as it recovers. A few details keep the ranking honest:

  • A member with no requests yet, such as one just discovered, is scored as its fastest peer. It shares that peer’s requests until its own first response ranks it, so it is neither flooded as the apparent fastest nor left waiting for a turn that an idle pool would never give it.
  • A failed request is recorded as a long latency rather than a short one, so a member that returns errors in a millisecond never looks fast. By default a response of 502, 503 or 504 is a failure, as is one that was never completed; lt.status_codes sets the codes that count as a good answer instead, as bare codes, inclusive ranges, or both. A request the client abandoned counts neither way.
  • A member’s average fades while it is passed over, so one ranked last on an old measurement or a past failure is tried again rather than ignored for good.
  • Averages survive a configuration reload and autodiscovery membership changes.

When the pool is idle and its members are equally loaded, every request goes to the fastest member. That is the mechanism working as intended; choose p2c or lc to spread idle traffic instead.

backends:
  video:
    provider: alb
    alb:
      mechanism: lt # or least_time
      pool: [ edge01, edge02 ]
      lt:
        decay: 10s # default
        status_codes: [ { start: 200, end: 499 } ] # optional; default is every code but 502, 503 and 504

Highest Random Weight

The Highest Random Weight (hrw) mechanism, also known as rendezvous hashing, routes every request that shares a key to the same healthy pool member. Use it to keep a client or tenant on one member’s warm cache. When a member leaves the pool, only the keys it owned move, each to a different remaining member; when it returns, exactly those keys move back. The mapping depends only on the key and the members’ names, so every Trickster replica agrees on it and a restart does not change it.

hrw.key selects what is hashed:

KeyFollows
client_ip (default)the client’s IP address, after trusted proxy resolution. The port is never part of the key. IPv6 addresses are keyed on their leading hrw.ipv6_prefix bits (default 64), because privacy addressing changes the rest of a client’s address over time.
hostthe request’s host name, without its port and without regard to case
header:<name>the first value of the named request header
cookie:<name>the value of the named cookie
query:<name>the value of the named query string parameter, as written in the URL
snithe TLS server name the client offered; only for an ALB that serves a tls stream listener
proxy_tlv:<type>the value of a PROXY protocol version 2 TLV, such as proxy_tlv:0xEA for an AWS VPC endpoint ID; only for an ALB that serves a tcp or tls stream listener with proxy_protocol enabled. The type is one byte, written in decimal or 0x hex.
userthe user name a session authenticated as; only for an ALB that serves a native protocol listener

A request that lacks the configured key, such as one without the header, has no affinity to preserve and is routed to a member at random.

hrw balances keys, not requests. With many keys of similar volume the members’ loads even out, but a single very busy key is always served by one member.

hrw keeps no state, so a key moves when a member joins the pool and wins it, and a client with no stable key has no affinity at all. To keep a client on its member for as long as its session lasts, whatever the pool does, add Sticky Sessions, which work with hrw as with every mechanism that selects one member.

backends:
  tenants:
    provider: alb
    alb:
      mechanism: hrw # or highest_random_weight
      pool: [ cache01, cache02, cache03 ]
      hrw:
        key: header:X-Tenant

Sticky Sessions

An ALB whose mechanism selects one member per request, rr, p2c, lc, lt or hrw, can keep each client on the member it was first sent to. The mechanism chooses a new client’s member; after that, the client’s requests go to the same member for as long as its session lasts and that member is available, even as members join or leave the pool. The same holds for the connections of a stream listener and the sessions of a native protocol listener; see Stream and native listeners.

backends:
  app:
    provider: alb
    alb:
      mechanism: p2c
      pool: [ app1, app2, app3 ]
      sticky:
        ttl: 8h
        secret_file: /etc/trickster/sticky.key

A session is kept in one of three ways, set by sticky.mode:

ModeHow the member is rememberedSuits
cookie (the default on http listeners)The ALB issues a signed token naming the member in a cookie, which the browser sends back. The ALB keeps no state.browsers
headerThe same token in a response header, X-Trickster-Session by default, which the client sends back in a request header of the same name.API clients and SDKs that do not keep cookies
table (the default on stream and native listeners)The ALB remembers each client’s member by a key read from its requests, connections or sessions, such as its address or a header.clients with a stable identifier of their own, upstreams that hand out their own session ids, and every listener that is not http

The settings, with their defaults:

      sticky:
        mode: ""                 # cookie, header or table; unset is cookie on http listeners, table on the others
        ttl: 1h                  # a session's lifetime from its first request; 0 is no limit
        idle: 0                  # ends a session left unused, as Session lifetimes describes; 0 never does
        on_unavailable: repick   # repick or reject; see below
        secret: ""               # the key tokens are signed with; see below
        secret_file: ""          # a file holding it, in place of secret
        cookie:                  # cookie mode only
          name: trickster_sticky
          path: /
          domain: ""             # unset keeps the cookie to the host that set it
          secure: auto           # auto marks it Secure when the request arrived over TLS; or true, false
          http_only: true
          same_site: lax         # lax, strict or none, which requires secure: true
          lifetime: permanent    # or session, which never sets Max-Age; see below
          mark_private: false
        header:                  # header mode only
          name: X-Trickster-Session
        table:                   # table mode only
          key: client_ip
          learn: request         # or response; see below
          ipv6_prefix: 64
          max_entries: 100000

ttl and idle are 0 or at least 1s. With mode set, a cookie, header or table block that the mode never reads is a configuration error. With mode unset, a cookie block (for http listeners) and a table block (for the others) may both be set, and a header block is refused.

Other settings that are refused: sticky on a mechanism other than rr, p2c, lc, lt and hrw; cookie or header mode on an ALB that serves no http listener; both secret and secret_file; a cookie.path that does not begin with /; a cookie name beginning __Secure- or __Host- without secure: true, or a __Host- one with a domain or a path other than /; same_site: none without secure: true; table.learn: response with a key other than header:<name> or cookie:<name>; and an ipv6_prefix above 128.

Tokens

In cookie and header mode the token carries the member, when the session began and when the token was issued, signed with HMAC-SHA256 under a key derived from sticky.secret. A token is bound to the ALB that issued it, so another ALB does not honor it. A client cannot choose its own member, and a token that is altered, signed with another key, issued by another ALB or past its lifetime is treated as no token: the request starts a new session. It is never an error.

A token is issued with the response to a client’s first request, including the 101 Switching Protocols that opens a WebSocket, when its session moves to another member, and, with idle set, once more than half of idle has passed since the token was issued, so that an active session does not lapse. A request sent to the member its token names is otherwise answered with no token, so a steady client sees none after its first.

A cookie carries a Max-Age that ends with its token when ttl is set. With ttl: 0, or with lifetime: session, it is a browser-session cookie, and the token itself still enforces ttl and idle. secure: auto marks the cookie Secure when the request reached Trickster over TLS; behind a load balancer that terminates TLS and forwards plain HTTP to Trickster, set secure: true. mark_private: true adds Cache-Control: private to a response that sets the cookie, unless that response is already private or no-store; see Sticky Sessions and Caching.

Two ALBs that set the same cookie (by name, domain and path) for a host they both serve would each replace the other’s token in a browser, so Trickster refuses to start with such a configuration. A browser sends a host’s cookies to every port and scheme, so this holds across listeners. To resolve it, give one of them its own cookie.name.

ALBs whose served hosts do not overlap may share a cookie name, as long as the cookie sets no domain. A cookie with no domain goes back only to the host that set it. The hosts an ALB serves are:

  • every host, when it is the default backend, or when one of its paths registers on a listener and it has any_host_routing or path routing on (the default). Path routing answers /<alb name>/... on every host, whatever hosts lists. A path registers on a listener unless it is marked dispatch_only or names a handler the ALB lacks. The ALB’s default paths count, unless path_defaults_disabled is set;
  • otherwise, the hosts its listener-registered paths answer for, joined by the hosts of its dispatchers, since a dispatched request keeps the host it arrived on. Its dispatchers are the ALBs that name it in their pool or user_router, and the rules that route to it. An ALB with path_routing_disabled: true and no hosts, or whose paths are all dispatch_only, serves only its dispatchers’ hosts;
  • no host, when nothing but a mirror reaches it. A mirror discards its target’s response, so no cookie from it reaches a browser, and such an ALB never conflicts.

The secret

Every Trickster replica behind one address, and every restart, must sign with the same key for a session to survive moving between them. Set secret to at least 32 bytes, or secret_file to a file that holds them, such as a mounted Kubernetes Secret; trailing line breaks in the file are ignored. Tokens made with one key are never honored with another. A secret that still contains ${, an environment variable reference that was not expanded, is refused rather than used as a publicly known key.

With neither set, the ALB signs with a random key that lasts as long as the process, and Trickster logs a warning each time it loads such a configuration, for every ALB that issues tokens on an http listener. That suits a single instance that can afford to start every session over when it restarts. A configuration reload keeps the process’s key, so a reload alone never ends a session. See Sticky Sessions Across Replicas.

When the pinned member is unavailable

With on_unavailable: repick, the default, a request whose member is unavailable goes to the member the mechanism chooses, and its session moves there. With reject, the ALB answers 503 Service Unavailable instead, for applications that cannot survive a move, and the client keeps its token so that it returns to its member once that member is available again. A member that has left the pool has no session to return to, so its clients start new sessions in either case.

A member that is draining keeps its sessions while it is available.

On an http listener a session is pinned by its first response, whatever its status, so a session that began on a member answering with errors stays there until that member is marked unavailable, by its health check or by the pool’s floor.

Table mode

In table mode the ALB keeps a table of each client’s member, by table.key, which takes the same values as hrw.key. A request without the key is balanced by the mechanism and not remembered. An entry lasts ttl from when it was stored and ends early when unused for idle. A full table drops the least recently used of a sample of its entries to make room.

With table.learn: response and a header:<name> or cookie:<name> key, the ALB also remembers the value a member hands the client in its response, in the header of that name or in the Set-Cookie that sets that cookie. That keeps a session on the member that created it from the client’s very next request, which is what an upstream that issues its own session ids needs. For example, for an MCP server whose replicas each know only the sessions they created:

      sticky:
        mode: table
        table:
          key: header:Mcp-Session-Id
          learn: response

A response that sets a new value pins that value to the member too; the entry for the old value lasts until it expires.

A table is kept in each Trickster instance’s memory. It survives a configuration reload that leaves the ALB’s mode, ttl, idle and table settings unchanged, but not a restart, and it is not shared between replicas: use cookie or header mode where sessions must outlive either, or see Sticky Sessions Across Replicas. With key: client_ip, every client behind one NAT address shares a session.

Session lifetimes

ttl and idle bound a session in every mode, but they count from different moments for a token and for a table entry:

cookie and header modetable mode
ttl counts fromthe session’s first request. A refreshed token keeps that start; a session that moves to another member starts over.when the entry was stored. An entry stored again, because its session moved, starts over.
idle counts fromwhen the token was last issued. A request more than half of idle after that is answered with a fresh token, so an unused session ends between half of idle and idle after its last request.the entry’s last use, to within a second.
after a reloadthe new ttl and idle apply at once to every token already issued, since a token records only when its session began and when it was issueda change to mode, ttl, idle or table starts the ALB’s table empty

Renaming the ALB or a member ends the sessions they had, since tokens and table entries refer to both by name. Moving from one mode to another starts every session over, as does a change of the signing key.

Stream and native listeners

A tcp, tls or udp listener, or a native protocol listener such as mysql, carries no token, so an ALB keeps the sessions of its connections there in a table. mode is best left unset: an ALB that serves both http and stream listeners then issues cookies to its requests and keeps its connections in a table. cookie and header mode are refused on a listener that is not http. With mode: table, requests and connections share one table, so a client whose requests and connections share a key keeps one member for both; table.key must then be readable on every listener the ALB serves, which in practice means client_ip.

table.key is read from each connection or session, as hrw.key is:

Listenertable.key
tcpclient_ip; proxy_tlv:<type> with proxy_protocol
tlsclient_ip, sni; proxy_tlv:<type> with proxy_protocol
udpclient_ip
native (mysql)client_ip, user

client_ip is the address a PROXY protocol header names when the listener trusts one. user is the name a native session authenticated as. A connection or session without its key is balanced by the mechanism and not remembered.

A connection is pinned to its member once it reaches it: a tcp or tls connection when it connects, a udp session when its member first answers, or when a one-way member’s session ends with no port-unreachable, and a native session when it is handed to its member. A member that cannot be reached is never pinned. A udp session already stays with its member for its whole life; what a pin adds is that the client’s next session, from any port, lands on the same member. A table entry is read only when a connection or session begins, so idle counts from the client’s last new connection or session, however long that one lasts.

With on_unavailable: reject, a connection whose pinned member is unavailable is refused, and so is one whose pinned member cannot be connected, however many connect_retries the ALB has: the retries would move the session. With repick, a failed connect is offered to other members as connect_retries allows, and its session moves to the one that connects. A native session that is refused fails to route, as it would with no member available.

Sticky Sessions and Nested ALBs

When a sticky ALB’s pool has ALBs in it, such as one per region or per service, its token or table entry keeps the whole path: the inner ALB the client was sent to and that ALB’s member. The inner ALBs need no sticky block of their own. An inner ALB whose mechanism fans requests out, or a User Router, is itself the session’s member, and chooses among its own members as it would with no session. When only the inner member becomes unavailable, the session stays with its inner ALB and moves within it. On a stream listener, a connection retried after its inner member failed to connect is likewise offered the inner ALB’s other members first.

When the inner ALB has no member left that can take the session (a draining member the session is pinned to still can), the session moves with on_unavailable: repick to the first other member the mechanism offers that can take it, and is refused with reject, keeping its token or table entry for when its inner ALB recovers. A new session that the mechanism sends to an inner ALB with no member is answered 502 Bad Gateway, or refused on a stream listener, as it would be with no sessions. Set propagate_health: true on the inner ALBs (see ALBs as Pool Members) to keep new sessions away from one while it has no healthy member.

The session covers one level of nesting: an ALB further in selects its member, or keeps sessions of its own if it has a sticky block, as it would with no outer session. An inner ALB directly in the sticky ALB’s pool follows the outer session, and uses a sticky block of its own only for requests that reach it without going through the outer ALB.

On a stream listener only the ALB that the listener maps to keeps sessions: an inner ALB’s own sticky block applies to its http listeners, and to stream listeners mapped to it directly, never to connections that reach it through another ALB.

Sticky Sessions and Caching

In front of cacheable content, prefer table mode. It adds nothing to responses and sends members nothing the client did not, so every cache between the client and the origin sees exactly what it would see without sessions. Weigh its two limits first:

  • A session is only as stable as its key. With key: client_ip, every client behind one NAT address or forward proxy shares one session, and so one member, and a client whose address changes, such as a phone moving between networks, starts a new session. A header:<name> or cookie:<name> key needs a value that the client sends, or that the application hands it and learn: response picks up. A request without the key is balanced afresh every time.
  • A table belongs to one Trickster instance. A restart empties it, and another replica does not know its sessions. With several replicas, keep each client on one replica, or pair the table with hrw on the same key; see Sticky Sessions Across Replicas.

Where those limits rule table mode out, use cookie mode with mark_private: true. header mode suits API clients with no shared cache in front of Trickster, since its token rides in responses and it has no mark_private.

A token interacts with caches in three ways:

  • A member’s cache never stores it. The ALB adds a token to a response after its member has answered, so a response served from a member’s cache is given a token when its session needs one, like any other.
  • The token travels upstream. A member receives the request with its token still in the Cookie header, or in the header.name header. An origin that answers Vary: Cookie makes a member cache one copy per distinct Cookie header, and a cookie token gives every session a distinct one, so such content is cached once per session.
  • A cache in front of Trickster sees the token. HTTP lets a shared cache, such as a CDN, store a response that sets a cookie and replay it, cookie included, to other clients, which joins them to that session’s member. A token in a response header is replayed the same way to clients that echo it back. That costs balance, not security, since a pin grants no access. In cookie mode, mark_private: true stops it by making each response that sets the cookie private, so a shared cache does not store that response. Only a session’s first response, a move and an idle refresh carry a token, so most responses stay shareable.

Sticky Sessions Across Replicas

In cookie and header mode the client carries its session, so every Trickster replica behind one address honors it, and so does a restarted one, as long as each replica:

  • signs with the same key, from secret or secret_file (see The secret);
  • configures the ALB under the same name, since a token is bound to it;
  • names the pool members alike, since a token refers to its member by name. A discovered member is named after its ALB and the member’s discovered name, or its address when it has none, so replicas that discover the same members agree.

Changing the key starts every session over once, since a token signed with the old key is no longer honored. The Kubernetes controller keys the ALBs it generates from a Secret; see The session key.

In table mode each replica keeps its own table, and a client that reaches a replica that has not seen it before is balanced there as a new session. To keep such a client on its member anyway, either:

  • keep each client on one replica, with source-address affinity in whatever balances the replicas; or
  • use mechanism: hrw with hrw.key set to the same key as table.key. Every replica then sends a new session to the same member for as long as their pools agree, and a replica’s table keeps the sessions it has seen in place when the pool changes. A replica that has not seen a session chooses its member afresh after such a change.

table.learn: response learns a value only on the replica whose member set it, and the request that began that session had no value for hrw to hash, so it needs each client kept on one replica.

Sticky Session Metrics

trickster_alb_sticky_total{alb_name, result} counts requests, connections and native sessions by how their session fared: hit (sent to its member), miss (no token or table entry), expired, invalid, repick (its member was unavailable and it moved) and rejected. In table mode an expired entry counts as a miss. A request, connection or native session is counted once it reaches its member, or once it is refused; one that reaches no member, which the ALB answers 502 Bad Gateway or closes, is not counted. trickster_alb_sticky_entries{alb_name} is the number of entries in an ALB’s table.

WebSockets and Other Protocol Upgrades

A request that asks to switch protocols, such as a WebSocket handshake, is tunneled by the pool member it is sent to when the ALB’s mechanism sends each request to one member (rr, p2c, lc, lt, hrw) or is the User Router. The mechanisms that fan a request out (fr, fgr, nlm, tsm) have no one member to tunnel to, so they ignore the upgrade and serve the request as an ordinary one, as HTTP permits.

Load Balancing Stream Listeners

The mechanisms that select one member, rr, p2c, lc, lt and hrw, also balance the connections of a tcp or tls stream listener and the sessions of a udp one. They are the same mechanisms with the same weights; only what they measure differs. The mechanisms that fan a request out (fr, fgr, nlm, tsm) and the User Router need an HTTP request, and are refused on a stream listener. Two more mechanisms, race and mirror, serve only stream listeners.

httptcp and tlsudp
unit of worka requesta connectiona session: one client address and port
in flight (p2c, lc, lt)requests being servedconnections opensessions open
latency (lt.signal)first_write: the first byte sent to the clientconnect (default): the time to connect to the member; or first_byte: the member’s first bytefirst_reply: the member’s first datagram
hrw.keyclient_ip, host, header:, cookie:, query:client_ip; sni on a tls listener; proxy_tlv:<type> with proxy_protocolclient_ip

A connection or session stays on the member it was given until it ends, whatever the mechanism. client_ip is the address a PROXY protocol header names when the listener trusts one. To keep a client’s later connections on one member too, give the ALB a sticky block.

Two settings apply only to an ALB that selects one member on a stream listener, under alb.stream:

backends:
  pg:
    provider: alb
    listener_names: [ postgres ]
    alb:
      mechanism: p2c
      pool: [ pg1, pg2, pg3 ]
      stream:
        connect_retries: 1       # default 0
        passive_health:          # off unless present
          failures: 3            # default
          eject: 30s             # default
          max_ejected_percent: 50 # default
  • connect_retries is how many other members a tcp or tls connection is offered when it cannot connect to the one it was given. All attempts share the listener’s stream.connect_timeout. The default, 0, refuses the connection, which is what keeps a weighted split exact: a member under the reserved .invalid domain exists to refuse its share, and is never retried past. udp has no connect to fail, so it is not retried.
  • passive_health takes a member out of the pool after failures consecutive failed connects, for eject, without waiting for a health check. Only failures to reach the member count, never anything it sent. At most max_ejected_percent of the pool is out at once, and the last live member is never ejected. When eject ends the member returns, unless its health check has it down. On udp, where nothing connects, a member that answers a datagram with a port-unreachable is what counts as a failure.

Health checks work as they do for HTTP pools: a tcp:// member with a healthcheck.interval is probed by opening a connection to it. A udp:// member has no generic probe; rely on autodiscovery readiness or on passive_health.

Connect Race and UDP Mirror

Two mechanisms exist only for stream listeners, because they commit one flow to several members at once. Both use every healthy member that has an address to dial, ignore weight, and take neither connect_retries nor passive_health. An ALB that uses one must be mapped to the stream listener directly; it cannot be a member of another ALB’s pool.

The Connect Race (race, or connect_race) mechanism serves tcp and tls listeners. Each client connection is connected to several members at once, within the listener’s stream.connect_timeout; the first member to connect carries the connection, and the other attempts are closed. A member that is down or slow to accept costs the client nothing. stream.race_width is how many members are raced, from 2 to 8; the default is every member, up to 4. In a pool wider than the race, each connection starts one member further along, so the connects are shared across the pool. A race opens and discards connections on the members that lose, so use it where a connect is cheap for the member.

The UDP Mirror (mirror, or udp_mirror) mechanism serves udp listeners. Every datagram a client sends is copied to every healthy member, which suits one-way protocols such as statsd, syslog and NetFlow. The first healthy member in pool order answers the session: only its replies are relayed to the client, and the other members’ replies are discarded. A mirror pool has at most 8 members.

backends:
  statsd:
    provider: alb
    listener_names: [ statsd ]
    alb:
      mechanism: mirror
      pool: [ statsd1, statsd2 ]
  db:
    provider: alb
    listener_names: [ postgres ]
    alb:
      mechanism: race
      pool: [ pg1, pg2, pg3 ]
      stream:
        race_width: 2

Load Balancing Native Protocol Sessions

A listener that speaks a backend’s own wire protocol, such as a mysql listener, can map to an ALB that uses rr, p2c, lc or hrw. Each client session is committed to one pool member once it authenticates, and stays there until it ends; a session is the unit of work, so p2c and lc compare members by their open sessions. hrw.key is client_ip or user. lt is not available, since a session reports no latency to rank members by. Every pool member must be a backend of the listener’s own provider, listed directly, and autodiscovery is not supported. The ALB authenticates the listener’s clients, so it carries the authenticator_name that a User Router on the same listener would.

backends:
  replicas:
    provider: alb
    listener_names: [ mysql ]
    authenticator_name: mysql-clients
    alb:
      mechanism: lc
      pool: [ replica1, replica2 ]

With a sticky block, a client’s later sessions keep to the member its first one was given, by client_ip or, with table.key: user, by the name it authenticated as.

Weights and the Selection Mechanisms

Every mechanism that selects one member per request honors the pool weight, but what a weight promises differs:

MechanismA weight isGuarantee
rra share of requestsexact: weight of every totalWeight consecutive requests
p2ca capacityproportional on average: heavier members are drawn more often and compared by requests in flight per unit of weight
hrwa share of the keysproportional on average over many keys; changing a weight moves as few keys as possible
lca capacitymembers are kept at equal requests in flight per unit of weight
lta biasthe score is divided by the weight, which shifts load under contention; an idle pool still sends every request to its best-scoring member

If you need a guaranteed split, use rr.

Time Series Merge

The Time Series Merge mechanism supports both High Availability and federation. Each physical backend represents one logical data shard. Set the backend-level replica_group option to the same value on physical backends that are HA replicas of that shard. TSM first coalesces those replicas, using configured pool order to resolve overlapping points and later replicas to fill gaps, and then reduces the distinct logical shards.

When a TSM pool member is itself an ALB (for example, a round-robin ALB over Prometheus backends), set replica_group on that immediate nested ALB. Trickster uses the wrapper as the replica-group boundary while delegating TSM planning and finalization to its terminal Prometheus provider. Other ALBs and non-TSM providers cannot set an explicit replica group.

When replica_group is omitted, it defaults to the backend name, so existing configurations continue to treat every backend as a distinct shard. Explicitly set it for HA pools that use non-idempotent aggregations such as sum, count, or avg; otherwise replicas will be counted as separate data.

Replica grouping is backend-global, not ALB-specific. A backend cannot be a replica in one ALB and a disjoint shard in another; define a second backend entry if both views are required. Partially overlapping datasets are not representable: one backend belongs to one logical shard for all TSM queries. Injected labels remain useful output and routing metadata, but they do not establish replica provenance.

If replicas disagree at the same logical point, the first configured member wins deterministically and Trickster records a conflict metric and warning log. A failed replica does not make a response partial when another replica covers its group. If an entire logical group is unavailable, the response is marked partial and includes a warning.

For request paths that are not mergeable by the configured time series provider, TSM does not fan the request out. Those requests are dispatched directly to the first live pool target. The same first-live-target fallback is used when a request cannot be prepared for the merge path.

By default, TSM merges the answers its members give to a query. A TSM ALB can instead read its members’ raw series and evaluate each query once, the way Thanos does; see Global Evaluation.

Merge Strategy

Within each configured replica group, TSM deduplicates values when merging series with identical labels — for each timestamp, only one replica value is kept. Across different groups it uses the query’s merge strategy.

For Federation use cases where backends hold different, non-overlapping data, Trickster automatically selects a merge strategy per query by parsing it with the upstream Prometheus PromQL parser and inspecting the outermost aggregation operator. No configuration is required. Because selection uses the parsed expression, redundant parentheses, comments, whitespace, keyword case, and by/without placement do not change the selected strategy. This is particularly important for PromQL aggregation queries like sum() or avg(), which strip labels from results and cause series from different backends to appear identical.

Outer OperatorTrickster Merge Behavior
sumSum of values per unique label set + timestamp
countSum of values per unique label set + timestamp
count_valuesSum of values per unique label set + timestamp
minMinimum value per unique label set + timestamp
maxMaximum value per unique label set + timestamp
groupDeduplicate per unique label set + timestamp
avgDual queries (avg→sum and avg→count); weighted arithmetic mean per unique label set + timestamp
topk, bottomkQuery the inner expression across backends, merge it, then apply final top/bottom-k selection per timestamp and aggregation group
stddev, stdvarPool shard-local count, mean, and variance states, then finalize the global population variance or standard deviation
quantileQuery the inner expression, merge all float samples globally, then calculate the exact quantile per timestamp and aggregation group
limit_ratioApply Prometheus-compatible label-hash sampling, globally finalizing a supported inner aggregation when necessary
limitkQuery the inner expression, merge it globally, then retain the first k samples in stable TSM series order per timestamp and aggregation group
count or group over an inner count or group of a selectorQuery the inner aggregation, union its label sets across backends, then count or group each label set once per timestamp and aggregation group
sum, count, or count_values followed by or vector(0)Sum of values per unique label set + timestamp
(none)Deduplicate (default)

For avg queries, Trickster issues two concurrent sub-queries per backend shard — one rewriting the outer avg to sum and another to count — then computes a true weighted arithmetic mean (sum_total / count_total) per series per timestamp. This avoids the skew introduced by a naïve avg-of-averages when backends have different data cardinalities.

When an outer aggregation’s input contains a nested aggregation (other than the count and group forms below), binary expression, or function that needs globally complete input, Trickster retains the aggregation’s established merge strategy and adds a warning. This fail-open behavior preserves correct results when the input series are colocated while noting that results can be inaccurate when matching series are split across shards.

For an outer count or group whose input is a count or group of a vector selector, such as count(count by (pod) (up)), the result depends only on which label sets the inner aggregation produces. Trickster sends the inner aggregation to each backend, unions the label sets the backends return, and applies the outer operator once per timestamp and aggregation group. A label set returned by several backends counts once, so replicated or partially overlapping backends don’t inflate the count. The selector may use label matchers, offset, and @, and the aggregation may be wrapped in sort(), sort_desc(), or a literal scalar operation such as > 1. A label set counts at a timestamp when any backend returned it there. Inner inputs that apply functions, filters, or binary operations can change which label sets a backend returns, so they keep the warning described above. When a backend fails, the result covers the backends that answered and is marked as a partial hit.

For sum, count, or count_values followed by or vector(0), Trickster sends the complete expression to each backend and sums the results, because a backend without matching series contributes only the explicit zero. This requires an aggregation input that each backend can evaluate independently: no nested aggregation, binary expression, or function that needs globally complete input. When the or uses on(...) or ignoring(...) label matching, this applies only to sum or count without by or without grouping; other forms use the warning fallback described below.

For topk and bottomk, Trickster sends the inner expression to each backend, merges those inner results using the inner expression’s merge strategy, then applies the final rank-and-trim step per timestamp and aggregation group. This prevents each backend’s local topk/bottomk result from being weighted equally during the merge. If the inner expression is avg, Trickster still uses the weighted sum/count rewrite before applying the final rank. This also applies when the rank aggregation is wrapped in sort() or sort_desc().

For stddev and stdvar, Trickster requests the shard-local count, mean, and population variance and pools those states before finalizing the requested global value. Native histograms are excluded from this float-only calculation. Supported already-aggregated inner expressions such as count, min, max, and group are merged globally before the outer variance aggregation.

For quantile, Trickster sends the inner expression to every backend, globally merges supported inner aggregations, ignores native-histogram samples, and calculates Prometheus’s exact sort-and-interpolate value independently for each timestamp and group. Exact quantiles require every relevant float sample, so their fanout responses can be substantially larger than shard-local quantiles and remain subject to the configured response-capture limits. A capture-limit failure is returned rather than silently substituting an approximate result.

For limit_ratio, selection uses the same complete-label-set hash threshold as Prometheus. Supported inner aggregations are merged before the ratio is applied; shard-local expressions can be sampled by each backend because the hash decision for a given label set is independent of shard placement.

For limitk, Trickster reproduces the current Prometheus evaluator’s first-visited algorithm over TSM’s stable merged-series order: lexicographic JSON serialization of the complete label set, followed by the series name. Selection is independent for every timestamp and aggregation group, retains complete labels and both float and native-histogram samples, and does not rank candidates by value or label hash.

Prometheus does not define a canonical storage visitation order for limitk. A separate Prometheus deployment whose storage returns the same series in a different order may therefore select different labels. Trickster guarantees the requested cardinality, grouping, and repeatability for the same merged input, and fanout completion order does not affect the result. limitk remains an experimental PromQL operator, so this compatibility contract may change with Prometheus.

For an unsupported inner expression, Trickster retains the established per-shard fallback and injects a warnings entry in the Prometheus response body to alert the caller that results may be inaccurate.

The same deduplicating fallback and warning apply when an aggregation is not the query’s outermost operation (for example, histogram_quantile(0.9, sum by (le) (rate(x_bucket[5m])))), when an aggregation is combined with a binary expression other than the supported or vector(0) form, and when the query cannot be parsed as PromQL, such as a query that uses another dialect’s extensions.

When a non-dedup strategy is in effect, or Trickster sends backends an inner expression to finalize after merging, and backends have injected labels configured, those labels are automatically stripped before merging. This ensures series from different backends hash identically for aggregation, and the injected labels do not appear in the response.

Native Histograms

Native histogram samples are preserved through the merge rather than being numerically aggregated. When a timestamp has a histogram on one backend and a float sample on another (or histograms on both), the histogram value is kept as-is — numeric aggregators like sum only apply across float samples. This prevents mixed-type series from being corrupted into garbage values when backends return a mix of float and histogram samples at the same timestamp.

Max Query Range Limitation

Trickster ALB supports enforcing a max_query_range duration on ALB backends. For details on how to configure and use query range limits, see the Query Range Limits documentation.

Step Alignment

Members’ responses combine at identical timestamps, so members must answer on one grid: a member that drops the partial bucket at the start while another returns it whole would merge into a series whose first points are summed over a subset of members. A tsm ALB therefore applies one step alignment mode to every member. It uses its own step_alignment when set; otherwise it uses its first configured member’s mode, whether or not that member is healthy, so a leader’s outage doesn’t move every open dashboard’s grid. A query’s trickster-step-align directive wins when every member supports the mode it names.

A static pool member that can’t apply the chosen mode fails the configuration. When autodiscovered members can’t, the pool uses truncate if every member supports it, or otherwise leaves each member to its own mode, and adds a warning to the merged response.

step_alignment on an ALB using any other mechanism applies that mode to every member too, so the chart’s shape doesn’t depend on which member answered. Without it, those mechanisms leave members to their own modes, and Trickster logs a warning at startup for each ALB whose members differ.

Members With a Series Limit

Some managed services, such as Amazon CloudWatch PromQL, cut a query result short at a series limit and flag it with a warning. Each member applies its own limit, so a member’s truncated result is proxied and never cached, while the other members’ results are cached as usual. The merged response then covers only the series each member returned, and it carries the truncation warning so the client can tell.

Global Evaluation

Merging members’ answers is exact only for queries whose aggregation can be split across shards, as described in Merge Strategy. A query such as sum(rate(x[5m])) over replicas that differ by a label, or histogram_quantile(...) over buckets split across members, needs every raw sample in one place. With tsm.evaluation: global, a TSM ALB answers each instant and range query the way Thanos Querier does: it reads the raw series the query selects from every member, deduplicates replicas, and evaluates the whole expression once in an embedded Prometheus engine.

backends:
  mimir-east:
    provider: prometheus
    origin_url: http://mimir-east:8080/prometheus
    prometheus:
      flavor: mimir
  mimir-west:
    provider: prometheus
    origin_url: http://mimir-west:8080/prometheus
    prometheus:
      flavor: mimir
  mimir-global:
    provider: alb
    cache_name: default
    alb:
      mechanism: tsm
      tsm:
        evaluation: global        # merge (the default) or global
        replica_labels: [replica] # labels that tell copies of one series apart; no default
        deduplication: penalty    # penalty (the default) or chain
        partial_response: true    # the default
        max_samples: 50000000     # the default
        query_timeout: 2m         # the default
        max_concurrent_queries: 20 # the default
        max_fetched_series: 0     # the default, unlimited
        max_fetched_samples: 0    # the default, unlimited
        raw_cache_window: 31d     # the default
      pool:
        - mimir-east
        - mimir-west

The evaluation is set per ALB, and covers every query the ALB receives; other TSM ALBs keep merging. Requests for other paths, such as labels and series, are merged as usual. The settings below evaluation are valid only with evaluation: global.

Members. Every pool member must be a prometheus backend with a series source: flavor: mimir reads Prometheus remote read, flavor: thanos reads Thanos StoreAPI at the member’s series_endpoint, and any other Prometheus backend can set series_api. Configuration fails for any other member, including a nested ALB. With autodiscovery, the template_backend must be such a backend. A template’s series_endpoint, including the one StoreAPI requires, is used unchanged by every discovered member, so every member reads that one store and Trickster logs a warning at startup; remote read without a series_endpoint instead reads each member’s own origin. A discovery query.scheme of tcp or udp gives no such origin and fails configuration, and a discovered member whose series endpoint still can’t be formed is logged and left out of the pool. Every live member is read, whatever its replica_group. Each read reuses the member’s TLS settings, and carries the client’s request headers and the member path’s request_headers.

Deduplication. Series whose labels match once replica_labels are removed are replicas of one series, and the result keeps one series per such label set, without the replica labels. deduplication: penalty follows Thanos: it reads one replica until that replica falls silent for twice its sampling interval, then switches to another. When the query reads the series through rate, increase, irate or resets, a switch to a replica whose counter is behind is adjusted so that it doesn’t read as a counter reset. When replicas sample at the same instant, the earliest-configured member’s replica is read first. deduplication: chain instead keeps every distinct timestamp across replicas, and the earliest-configured member’s value where they share one. Series with identical labels from different members are always combined like chain. Leaving replica_labels empty still combines identical series. A native histogram’s counter reset hint describes its change from the previous sample of its own source, so where deduplication moves to another replica or copy, or passes over samples, the hint is cleared and the engine finds any reset from the counts, as Prometheus does where it merges series.

Partial responses. When a member is unavailable or fails to answer, by default the result is evaluated over the members that answered, and carries a warning naming the member, as Thanos does with partial responses enabled. Why the member failed is logged rather than returned, since a transport error can name internal hosts. Set partial_response: false to fail such a query with a 503 instead, as soon as a member fails, stopping the reads of the others. A request’s partial_response parameter overrides the ALB’s setting. A failed evaluation is returned as a Prometheus API error, never replaced by a merged answer.

Limits. max_samples bounds the samples the engine holds at once, like Prometheus’s --query.max-samples. query_timeout bounds an evaluation from its arrival, including any wait for one of the ALB’s max_concurrent_queries slots; a request’s timeout parameter can shorten it. max_fetched_series and max_fetched_samples bound the raw series and samples one evaluation reads from all its members before deduplication: the samples each selector reads within its range, whether from a member or the raw series cache, and the series with any such sample. What the raw series cache serves for a selector is counted before any member is read for it, and what members send as their answers arrive, so a read stops as soon as its samples and the cache’s pass the allowance. As a member’s series are mostly those the cache serves too, a read counts series only past as many as the cache served, and each selector’s exact count is settled once its answer is assembled. A series counts once for each selector and member, however many frames, messages, or uncached time ranges carry it. With max_fetched_series set, series are told apart by a hash of their labels as they arrive, so a copy of a series that a member sends apart from the rest of it counts once too; without it, such a copy may count again. A read stops as soon as the evaluation’s allowance runs out, and a read several evaluations share counts against each of them, stopping only when none can afford more. Each member’s raw response, and each StoreAPI message or remote-read frame in it, is bounded by the ALB’s max_capture_bytes. Only the chunks and samples within a selector’s range are kept from a response, and what is kept from one response, with the buckets its native histogram chunks lay out as they are read, may take at most four times max_capture_bytes in memory once decoded, so a response can’t expand far past its size even when the fetch limits are unlimited. Deduplication reads every copy and replica of a series at once, so the buckets their chunks lay out count together. Responses of float chunks stay within that bound; native histograms expand more, integer ones sent as samples to around eight times their size, so a response of them can reach it before max_capture_bytes. A histogram chunk whose bucket layout declares more buckets than its data holds is refused as malformed, as is one whose samples couldn’t be read: spans offset back after the first, in a schema finer than the engine reads, or custom-bucket spans offset back, on the negative side, or reaching past their bounds. A histogram sent as a sample is refused for the same layouts, and for spans laying out more or fewer buckets than it holds. max_capture_bytes also bounds the result an evaluation encodes as its response, whether or not the result can be cached: encoding stops as soon as the result passes it, and the query fails whole with execution, never returning part of the result. A failed evaluation’s error stays within it too, or within 4 KiB when that is more, leaving out warnings and shortening its message as needed. A result takes some 15 to 35 bytes of JSON for each sample it returns, where members’ chunk-encoded reads often take one or two, so an ALB that returns raw series over wide ranges needs a max_capture_bytes sized for its results as well as its members’ reads. A query that reaches one of these limits or times out fails, even when partial responses are allowed; a limit a member enforces itself, such as Mimir’s query limits, is that member’s failure. Errors use Prometheus’s types and status codes: bad_data (400), execution (422, which includes the limits), canceled (499), internal (500), and timeout or unavailable (503). Once a member’s read of one of a query’s selectors reaches a limit, the other members’ reads of that selector stop, and the query fails with execution, even when any other of its reads also failed, of that selector or another, or it also timed out or its client canceled it as the limit was reached.

The engine. The engine is the Prometheus version Trickster is built with, with a 5-minute lookback, the @ modifier, negative offsets and the experimental functions enabled, a 1-minute step for subqueries that name none, and at most 11,000 points per series. The query’s limit parameter is honored. A request is validated as Prometheus validates it before any cache lookup or raw read, except that a range query’s step must be a whole number of milliseconds, the engine’s resolution, rather than be evaluated at another step; a timeout must be positive, a limit a non-negative integer, and partial_response a boolean.

Caching. A global ALB keeps each complete evaluation in its own cache_name (default when unset), whole. Its range queries are not delta-cached, as a Prometheus backend’s are: penalty deduplication follows replicas from a range’s first sample, so samples evaluated as part of a range can differ from the same samples evaluated with the whole of it. A cached result answers only the same query with the same time, or the same start, end and step, so a dashboard’s moving range is evaluated each time, while repeats and concurrent viewers are answered from the cache. A result that reaches into the recent past, within the 5-minute lookback or the ALB’s volatile_window, is kept for up to 30 seconds, while samples may still arrive; an older one is kept for the ALB’s timeseries_ttl. A partial result or a failed evaluation is never cached, and is answered with Cache-Control: no-store.

The cache key holds the query’s limit, and its timeout and partial_response resolved against the ALB’s settings, so requests that share a result or an evaluation agree on how it ends. It also holds the Authorization header, the X-Scope-OrgID and THANOS-TENANT tenant headers, the cache_key_headers, cache_key_params and cache_key_form_fields configured on the ALB’s query paths and on the ALB path a request arrives on, such as an alias like /prometheus/api/v1/query_range, and the cache_key_headers configured on its members’ query paths, so no tenant or scope reads another’s results. Results are kept under a namespace that digests the ALB’s members, in order, with each one’s series endpoint, the request headers its query paths send, and its TLS client and SigV4 settings. The namespace changes when membership does, as when a discovery adds or removes a member, and when a member’s configuration does, but not with health, so results kept before a member’s outage stay usable. A request’s evaluation reads the members whose namespace keys its result, even when the pool changes while it runs.

Raw series caching. Beneath its results, a global ALB with a cache also keeps the raw series it reads, so each member is read only for samples the cache doesn’t hold. A member’s series are kept per selector, in two-hour blocks aligned to the epoch, keyed by the member’s series endpoint and configuration (as for the namespace above), the selector’s matchers, and the identity headers that key results, as the member receives them. Samples within a member’s volatile window are read from it by every evaluation and never kept, since replicas’ samples may still arrive; older samples are read once and kept for the ALB’s timeseries_ttl. A dashboard’s moving range therefore reads only its newest minutes from each member, and a wider range reads only the part the cache lacks. The volatile window is 5 minutes for members with flavor: mimir or flavor: thanos, and none for other members; an ALB that sets volatile_window uses it for every member instead, so set it to cover any out-of-order window your members accept. Samples older than raw_cache_window, 31 days by default, aren’t kept either: a range reaching further back reads that older part from the members every time, and the rest from the cache. The window counts whole blocks, and each block a range spans is one cache lookup per selector and member. A selector with an exact metric name can also be answered from the cached blocks of a broader one on the same metric, member and identity, whose matchers are all among its own: up{job="api",code=~"5.."} from up{job="api"}’s blocks, filtered to the series it matches. Each Trickster instance remembers in memory which selectors have cached blocks; nothing more is stored. Concurrent evaluations needing the same samples share one read, and a selector’s missing ranges are read from a member a few at a time. Every chunk of a member’s answer is read through, and the bucket layout of every histogram in it checked, whether the answer arrives as chunks or as samples, before it is kept, so a chunk or histogram that can’t be read fails that member’s read rather than being cached. An answer a member gave incompletely, or failed to give, is never kept. If keeping an answer fails anyway, the evaluations that read it already have their answer, the failure is logged, and its samples are read from the member next time; a cached block that fails as they are merged into it is dropped, and its samples are read again too. The ALB’s max_object_size_bytes, 64 MB by default for a TSM ALB, bounds each block as it bounds each cached result: samples that would take a block past it aren’t kept, so they are read from the members every time while the block keeps what it held. A block refused some samples isn’t tried with them again until timeseries_ttl passes, unless it no longer holds what it held then, as once evicted, when they may fit; each refusal counts in trickster_cache_events_total as event error with reason tsm_global_raw_max_object_size. Blocks are keyed by the limit too, so ALBs of different limits sharing a cache keep their own, and changing the limit starts new blocks while the old ones expire. A memory cache measures a block by the memory it holds, and other caches by its encoded size. The small chunks that frequent refreshes add to a block are encoded again into full ones as they accumulate, so a block kept current every few seconds ends as compact as one read whole. Each selector reads exactly the samples from its start to its end, cached or not, so cached and uncached evaluations agree, and penalty deduplication starts at a selector’s first sample.

Concurrent requests for the same result share one evaluation. The request that starts it owns it: when its client goes away, the evaluation and its raw reads are canceled, and a request that joined it runs the query again.

Reloads and shutdown. A configuration reload leaves the replaced ALB’s evaluations running for the reload_drain_timeout. Then that ALB admits no more evaluations and cancels those still running, which are answered unavailable (503) unless one already failed with execution, as on reaching a limit, which decides its answer as it would a timeout’s, and waits up to the same timeout again for raw series blocks still being stored; nothing a raw read returns from then on is stored. Only after that are its members’ StoreAPI connections closed, discovered members’ included. A memory or redis cache that the new configuration changes or removes, and that a replaced global ALB keeps its results in, is closed after that too, so the blocks still being stored reach it; until then, a memory cache’s objects stay in memory alongside those of the cache replacing it. A cache that keeps its objects in local files (filesystem, bbolt, badger) is closed as the reload applies, since its replacement can open the same files, so it refuses what the replaced ALB stores in it during the drain. A changed or removed cache that no global ALB uses is closed as the reload applies too, once the operations under way in it end, whatever its provider. A reload that fails restores the previous ALB, which keeps reading its discovered members over the connections it already has. Shutdown does the same once the listeners have drained: retiring the evaluators and closing the caches share one shutdown_drain_timeout. A discovered member that autodiscovery removes or rebuilds has its StoreAPI connection closed once the reload_drain_timeout passes, and one it refuses as it builds it, at once.

Reading raw series moves much more data than reading answers, as it does in Thanos: plan max_capture_bytes and the fetch limits for the widest dashboard ranges. To compare global evaluation with Thanos or with merging, see the mimir and thanos profiles of the developer environment.

Providers Supporting Time Series Merge

Trickster currently supports Time Series Merging for the following TSDB Providers:

Provider Name
Prometheus

We hope to support more TSDB’s in the future and welcome any help!

Example TS Merge Configuration

backends:

  # prom01a and prom01b are redundant and poll the same targets
  prom01a:
    provider: prometheus
    replica_group: prom01
    origin_url: http://prom01a.example.com:9090
    prometheus:
      labels:
        region: us-east-1

  prom01b:
    provider: prometheus
    replica_group: prom01
    origin_url: http://prom01b.example.com:9090
      labels:
        region: us-east-1

  # prom-alb-01 scatter/gathers to prom01a and prom01b and merges responses for the caller.
  # Enforces a max 14-day time range limit on all incoming merge requests.
  prom-alb-01:
    provider: alb
    max_query_range: 14d
    alb:
      mechanism: tsm # or time_series_merge
      pool: 
        - prom01a
        - prom01b

  # prom02 and prom03 poll unique targets but produce the same metric names as prom01a/b
  prom02:
    provider: prometheus
    origin_url: http://prom02.example.com:9090
      labels:
        region: us-east-2

  prom03:
    provider: prometheus
    origin_url: http://prom03.example.com:9090
      labels:
        region: us-west-1

  # prom-alb-all scatter/gathers prom01a/b, prom02 and prom03 and merges their responses
  # for the caller. The merge strategy is automatically selected per-query based on the
  # outer PromQL aggregation operator. Injected labels are automatically stripped before
  # merging so that series from different backends are combined correctly. Because prom01a
  # and prom01b are in the same replica_group, their values are de-duplicated before being
  # merged/reduced with prom02 and prom03.
  prom-alb-all:
    provider: alb
    alb:
      mechanism: tsm
      pool:
        - prom01a
        - prom01b
        - prom02
        - prom03

Here is the visual representation of a basic TS Merge configuration:

First Response

The First Response mechanism fans a request out to all healthy pool members, and returns the first response received back to the client. All other fanned out responses are cached (if applicable) but otherwise discarded. If one backend in the fanout has already cached the requested object, and the other backends do not, the cached response will return to the caller while the other backends in the fanout will cache their responses as well for subsequent requests through the ALB.

This mechanism works well when using Trickster as an HTTP object cache fronting multiple redundant origins, to ensure the fastest response possible is delivered to downstream clients - even if the HTTP Response Code indicates an error in the request or by the first backend to respond.

First Response Configuration Example

backends:
  node01:
    provider: reverseproxycache
    origin_url: http://node01.example.com

  node02:
    provider: reverseproxycache
    origin_url: http://node-02.example.com

  node-alb-fr:
    provider: alb
    alb:
      mechanism: fr # or first_response
      pool:
        - node01
        - node02

Here is the visual representation of this configuration:

First Good Response

The First Good Response (fgr) mechanism acts just as First Response does, except that it waits to return the first response with an HTTP Status Code < 400. If no fanned out response codes are in the acceptable range once all responses are returned (or the timeout has been reached), then the healthiest response, based on min(all_responses_status_codes), is used.

This mechanism is useful in applications such as live internet television. Consider an operational condition where an object may have been written to Origin 1, but not yet written to redundant Origin 2, while users have already received references to and begin requesting the object in a separate manifest. Trickster, when used as an ALB+Cache in this scenario, will poll both backends for the object and cache the positive responses from Origin 1 for serving subsequent requests locally, while a negative cache configuration will avoid potential 404 storms on Origin 2 until the object can be written by the replication process.

Custom Good Status Codes List

By default, fgr will return the first response with a status code < 400. However, you can optionally provide an explicit list of good status codes using the fgr.status_codes configuration setting, as shown in the example below. When set, Trickster will return the first response to be returned that has a status code found in the configured list. An entry may be a single code or an inclusive range, and the two forms mix: status_codes: [ { start: 200, end: 299 }, 304 ].

First Good Response Configuration Example


negative-caches:
  default: # by default, backends use the 'default' negative cache
    "404": 500 # cache 404 responses for 500ms

backends:
  node01:
    provider: reverseproxycache
    origin_url: http://node-01.example.com

  node02:
    provider: reverseproxycache
    origin_url: http://node-02.example.com

  node-alb-fgr:
    provider: alb
    alb:
      mechanism: fgr # or first_good_response
      pool:
        - node01
        - node02
      fgr:
        status_codes: [ 200, 201, 204 ] # only consider these codes when selecting a response

Here is the visual representation of this configuration:

Newest Last-Modified

The Newest Last-Modified mechanism is focused on providing the user with the newest representation of the response, rather than responding as quickly as possible. It will fan the client request out to all backends, and wait for all responses to come back (or the ALB timeout to be reached) before determining which response is returned to the user.

If at least one fanout response has a Last-Modified header, then any response not containing the header is discarded. The remaining responses are sorted based on their Last Modified header value, and the newest value determines which response is chosen.

This mechanism is useful in applications where an object residing at the same path on multiple origins is updated frequently, such as a DASH or HLS manifest for a live video broadcast. When using Trickster as an ALB+Cache in this scenario, it will poll both backends for the object, and ensure the newest version between them is used as the client response.

Note that with NLM, the response to the user is only as fast as the slowest backend to respond.

Newest Last-Modified Configuration Example

backends:
  node01:
    provider: reverseproxycache
    origin_url: http://node01.example.com

  node02:
    provider: reverseproxycache
    origin_url: http://node-02.example.com

  node-alb-nlm:
    provider: alb
    alb:
      mechanism: nlm # or newest_last_modified
      pool:
        - node01
        - node02

Here is the visual representation of this configuration:

User Router

The User Router mechanism is used to control a Request’s destination Backend based on the username in the request. A default Backend (for no-user and users not in the manifest) can be configured, as well as a Backend per-user.

Native MySQL listeners use a deliberately narrower User Router topology than HTTP backends: one authenticated listener-facing User Router may select only direct terminal MySQL backends, selection is sticky for the session, and to_user/to_credential remapping is rejected. See the MySQL Provider Guide for the complete authentication, routing, health, cache-identity, and no-route contract.

When a User Router ALB is configured to use an Authenticator, the ALB can also modify a Request’s credentials before passing it off to the destination Backend. In the graphic below, user casey will be routed to the readersBackend, which proxies to a read-only database server with the dbreader credentials; while user taylor will be routed to the writersBackend, which proxies to a read-write database server with the dbwriter credentials. Here is the example configuration corresponding to the graphic:

Credential replacement is applied only when the user’s configured to_backend target is selected. When a request instead uses default_backend - because the username has no mapping, the mapping does not name a usable runtime target, or the mapped target is unavailable - the request retains its inbound credentials. If a user should receive replacement credentials when routed to the same Backend that also serves as the default, set that Backend explicitly as the user’s to_backend.

backends:
  readersBackend:
    provider: clickhouse
    origin_url: http://read.prod.db.com:8123/

  writersBackend:
    provider: clickhouse
    origin_url: http://write.prod.db.com:8123/

  click-lb-01:
    provider: alb
    authenticator_name: dbUsers
    alb:
      mechanism: ur # or user_router
      user_router: # User Router-specific configs
        default_backend: readersBackend # optional - users not in the list will route here, origin will 401
        users:
          casey:
            to_user: dbreader # replaces user casey with dbreader in the request's Authorization header
            to_credential: ${DB_READER_PW} # replaces credential in the Authorization header with this env
            to_backend: readersBackend # explicit selection applies casey's credential replacement
          taylor:
            to_user: dbwriter # replaces user taylor with dbwriter in the request's Authorization header
            to_credential: ${DB_WRITER_PW} # replaces credential in the Authorization header with this env
            to_backend: writersBackend # taylor is sent to the writers backend

authenticators:
  dbUsers:
    provider: clickhouse # use the clickhouse authenticator
    users_file: /path/to/user-manifest.csv # this file should include casey and taylor users
    users_file_format: csv # required when users_file is set

Supported Backend Provider Types

The User Router mechanism supports all Backend provider types for default_backend and to_backend values, including other User Router ALBs.

However, config validation will fail if:

  • there are any possible infinite loops between backends configured
  • users could ultimately be routed to different non-virtual (ALB/Rule) backend types by the same User Router ALB. The final ultimate route for all users must be of the same type (regardless of how many additional hops through ALBs and Rules the request would take).
    • In other words: user1 cannot be ultimately routed to a clickhouse backend and user2 be ultimately routed to a prometheus backend by the same User Router ALB.

User Router without an Authenticator

If a User Router ALB does not use an Authenticator, you can still configure user-specific Backend routes. In these cases Trickster will observe (but not authenticate) the username in the request and route based on the observed username. However, Trickster will exit with a validation failure on startup if a User Router ALB that does not utilize an Authenticator is configured to swap credentials. In short: users must be positively authenticated by a Trickster Authenticator for credential swapping to be permitted by the User Router ALB.

When a User Router ALB doesn’t use an Authenticator, Trickster uses the final destination Backend provider type to select a default Authenticator (operating in observe-only mode / no users manifest) for username observation. For clickhouse-destined User Routers, the observe only Authenticator provider is clickhouse. For all other backend provider types, the default the observe only Authenticator provider is basic (Basic Auth).

to_user / to_credential vs Backend Path Header Injection

It is still possible to insert credentials to a Backend proxy request using the request_headers Backend Path config. But any request_headers alterations configured for auth-related headers (e.g., Authorization) are performed by the Backend after being handled by a User Router; so they would overwrite any user-specific to_user and to_credential transformations performed by the User Router ALB.

Default Backend

As shown in the example config above, you can provide a default_backend config to a User Router, and users who are not in the user router list will be routed to this backend.

If you do not supply a default_backend, users who are not in the manifest will receive a default response of 502 Bad Gateway. You can customize the default response code by setting no_route_status_code to a value between 400 and 599 as in this example:

backends:
  prod-01:
    provider: reverseproxy
    origin_url: https://example.com/

  users-lb-01:
    provider: alb
    alb:
      mechanism: ur
      authenticator_name: all-users # not shown for brevity, see above examples
      user_router:
        no_route_status_code: 401 # unauthorized response for users not in allow list
        users: # allowed users
          casey:
            to_backend: prod-01
          taylor:
            to_backend: prod-01
          kris:
            to_backend: prod-01

User Router ALB Backend Pool and Health Checking

The User Router does not rotate through or fan out to a pool of Backends like the other ALB mechanisms. A healthy mapped target is selected directly. When a mapped target is unavailable, the request uses the healthy default_backend without applying the mapped target’s credential replacement. If neither target is available, the router uses its configured no-route response.

That fallback applies to HTTP requests. A native MySQL session whose username has an explicit mapping fails with a MySQL availability error when that mapped terminal is unavailable; it is never redirected to default_backend. Only an unmapped MySQL username may use the configured default terminal.

You can configure a User Router ALB’s backend destinations to be other ALBs with mechanisms that utilize healthchecked pools.

Bounding Per-Member Response Captures

ALB mechanisms that fan out (TSM, FR, FGR, NLM) buffer each pool member’s response in memory before merging or selecting a winner. Without a cap, one misbehaving upstream returning an oversized body can OOM the proxy – an N-way fanout multiplies that by N.

Trickster applies a default cap of 256 MiB per response. A member whose body exceeds the cap is treated as a partial failure: the merged response carries an X-Trickster-Result: phit marker and the trickster_alb_fanout_failures_total{mechanism, reason="truncated"} metric increments.

Override the cap at the backend or ALB level:

backends:
  default:
    max_capture_bytes: 67108864  # 64 MiB, applies to all backends (Prometheus, ClickHouse, ALB members, etc.)

  prom-alb-tsm:
    provider: alb
    alb:
      mechanism: tsm
      max_capture_bytes: 16777216  # 16 MiB, ALB-specific override
      pool:
        - prom01
        - prom02

The ALB-level value takes precedence over the backend-level value, which in turn takes precedence over the 256 MiB default.

A TSM ALB with evaluation: global applies the cap to each member’s raw series response and to the result each evaluation encodes as its response. Unlike the fanout mechanisms above, it doesn’t treat a member’s response over the cap as a partial failure: the whole query fails with a Prometheus execution error (422), even when partial responses are allowed, with no X-Trickster-Result: phit marker and no truncated count in trickster_alb_fanout_failures_total. A result that would exceed the cap fails with the same error rather than being returned in part.

Bounding Aggregate In-Flight Captures

max_capture_bytes caps each member’s response individually; a fanout to N members can still buffer up to N * max_capture_bytes in flight. For deployments with large pools or low memory ceilings, set max_fanout_capture_bytes to cap the aggregate buffer across all in-flight slots in a single fanout call. Slots dispatched after the aggregate budget would go negative are fail-fasted (marked Failed, no capture buffer allocated) before the upstream handler runs; the merge sees them as partial failures and the existing fallback path handles it.

backends:
  prom-alb-tsm:
    provider: alb
    alb:
      mechanism: tsm
      max_capture_bytes: 16777216         # 16 MiB per member
      max_fanout_capture_bytes: 67108864  # 64 MiB total across all in-flight slots
      pool:
        - prom01
        - prom02
        - prom03
        - prom04

max_fanout_capture_bytes defaults to 0 (no aggregate cap). Pick a value matching what your trickster instance can afford to buffer per request, independent of pool size.

Maintaining Healthy Pools With Automated Health Check Integrations

Health Checks are configured per-Backend as described in the Health documentation. Each Backend’s health checker will notify all ALB pools of which it is a member when its health status changes, so long as it has been configured with a health check interval for automated checking. When an ALB is notified that the state of a pool member has changed, the ALB will reconstruct its list of healthy pool members before serving the next request.

Health Check States

A backend will report one of three possible health states to its ALBs: unavailable (-1), unknown (0), or available (1).

Health-Based Backend Selection

Each ALB has a configurable healthy_floor value, which is the threshold for determining which pool members are included in the healthy pool, based on their instantaneous health state. The healthy_floor represents the minimum acceptable health state value for inclusion in the healthy pool. The default healthy_floor value is 0, meaning Backends in a state >= 0 (unknown and available) are included in the healthy pool. Setting healthy_floor: 1 would include only available Backends, while a value of -1 will include all backends in the configured pool, including those marked as unavailable.

Backends that do not have a health check interval configured will remain in a permanent state of unknown. Backends will also be in an unknown state from the time Trickster starts until the first of any configured automated health check is completed. A pool member in a permanent unknown state can never reach available, so a healthy_floor: 1 ALB whose members lack health checks would have an empty pool and return 502 for every request. To avoid that, Trickster resets such an ALB’s effective floor to 0 at startup, emits a warning naming the ALB and the un-probed members, and sets the trickster_alb_pool_floor_reset{backend_name} gauge to 1. Configure a health check interval on those members if you want healthy_floor: 1 to apply.

Setting healthy_floor below 0 admits members the probe has confirmed unavailable, not just members in the transient unknown state. If your goal is to keep traffic flowing during the cold-start window before the first probes complete, lower the pool members’ recovery_threshold so they transition out of unknown faster – don’t lower the floor. When healthy_floor < 0 Trickster emits a startup warning and sets the trickster_alb_pool_admits_failing{backend_name} gauge to 1.

Backup Pool Members

A pool member marked backup: true stands by: it receives traffic only while no other member of the pool is in the healthy pool. As soon as one of the other members returns, the backup members stand down again, apart from the sticky sessions they took meanwhile, which stay with them until they end. This applies to every mechanism that has a pool, and to stream and native protocol listeners as well as HTTP. A pool must have at least one member that is not a backup, unless its other members come from autodiscovery; discovered members are never backups.

backends:
  db:
    provider: alb
    listener_names: [ postgres ]
    alb:
      mechanism: rr
      healthy_floor: 1
      pool:
        - primary
        - name: standby
          backup: true

Failover depends on the ALB learning that its other members are down, so give them a health check interval or, on a stream listener, stream.passive_health. While an ALB with backup members is dispatching to them, the trickster_alb_pool_on_backup{backend_name} gauge is 1, and a warning is logged when it fails over.

Draining Pool Members

A pool member marked drain: true takes no new work but keeps the sticky sessions it already has, for as long as it is available, so that it can be retired without ending them. Every mechanism leaves a draining member out when it selects a member or fans a request out; only a sticky session can reach it. A pool must have at least one member that is not draining, unless its other members come from autodiscovery. When every member of the primary tier drains, new work goes to the backup members.

Draining ends no session by itself: a session keeps its draining member until its ttl or idle ends it, the member leaves the pool, or the member becomes unavailable. To retire a member without cutting sessions short, keep it draining until they end: a session lasts at most ttl, and ends sooner once unused for idle.

Discovered members drain too: a Kubernetes endpoint that is terminating but still serving, or a deleted pod that is still ready, stays in the pool as a draining member until it stops serving, so a rolling restart moves new sessions to the new pods while the old ones finish theirs. See autodiscovery. This applies to every discovered pool, sticky or not.

Draining members are listed on the health status page and exported by the trickster_alb_member_draining{alb_name, member} gauge, which is 1 for each draining member.

A Time Series Merge ALB, which keeps no sessions, merges nothing from a draining member. Draining the only live member of a replica_group therefore leaves that shard out of its merged results, and the ALB warns of a degraded pool, as it would had the member failed.

backends:
  app:
    provider: alb
    alb:
      mechanism: rr
      sticky: {}
      pool:
        - app1
        - app2
        - name: app3
          drain: true

ALBs as Pool Members

An ALB that is a member of another ALB’s pool is always treated as available, even when its own healthy pool is empty: it keeps its share of the outer pool’s traffic and fails it. That is deliberate for weighted splits, where an empty member must not shift its share onto its siblings. Set propagate_health: true on the inner ALB to change that. It then reports unavailable to the pools it belongs to while it has no healthy member, and available otherwise, so an outer pool whose healthy_floor excludes unavailable members sends that share to its other members instead.

backends:
  region-east:
    provider: alb
    alb:
      mechanism: rr
      propagate_health: true
      pool: [ east1, east2 ]
  global:
    provider: alb
    alb:
      mechanism: rr
      pool:
        - region-east
        - name: region-west
          backup: true

Example ALB Configuration Routing Only To Known Healthy Backends

backends:
  prom01:
    provider: prometheus
    origin_url: http://prom01.example.com:9090
    healthcheck:
      interval: 1000ms # enables automatic health check polling for ALB pool reporting

  prom02:
    provider: prometheus
    origin_url: http://prom02.example.com:9090
    healthcheck:
      interval: 1000ms

  prom-alb-tsm:
    provider: alb
    alb:
      mechanism: tsm   # times series merge healthy pool members
      healthy_floor: 1 # only include Backends reporting as 'available' in the healthy pool
      pool:
        - prom01
        - prom02

All-Backends Health Status Page

Trickster 2.x provides a global health status page available at http://trickster:metrics-port/trickster/health or (the configured health_handler_path).

The global status page will display the health state about all backends configured for automated health checking, and every ALB with its pool members. Here is an example configuration and a possible corresponding status page output:

backends:
  proxy-01:
    provider: reverseproxy
    origin_url: http://server01.example.com
    # not configured for automated health check polling

  prom-01:
    provider: prometheus
    origin_url: http://prom01.example.com:9090
    healthcheck:
      interval: 1000ms # enables automatic health check polling every 1s

  prom-02:
    provider: prometheus
    origin_url: http://prom02.example.com:9090
    healthcheck:
      interval: 1000ms

  prom-03:
    provider: prometheus
    origin_url: http://prom03.example.com:9090
    healthcheck:
      interval: 1000ms

  prom-alb:
    provider: alb
    alb:
      mechanism: rr
      pool:
        - prom-01
        - prom-02
        - name: prom-03
          drain: true
$ curl "http://${trickster-fqdn}:8481/trickster/health"

Trickster Backend Health Status            last change: 2026-09-26 00:00:00 UTC
-------------------------------------------------------------------------------

prom-02    prometheus     unavailable since 2026-09-26 00:00:00 UTC

prom-01    prometheus     available
prom-03    prometheus     available
prom-alb   alb (rr)       available u:[prom-02] a:[prom-01,prom-03] d:[prom-03]

proxy-01   reverseproxy   not configured for automated health checks

-------------------------------------------------------------------------------
For JSON, provide a 'Accept: application/json' Header or query param ?json
For YAML, provide a 'Accept: application/yaml' Header or query param ?yaml

ALB Pool Members on the Status Page

Each ALB is listed with its pool members grouped by health: u:[...] (unavailable), a:[...] (available) and nc:[...] (not checked) in the text form, and unavailablePoolMembers, availablePoolMembers, uncheckedPoolMembers and initializingPoolMembers in the JSON and YAML forms. Draining members are also listed under their health, and again in d:[...] (text) or drainingPoolMembers (JSON and YAML). An ALB is listed as available while its pool has a member it would send new work to: one that meets its healthy_floor, is not draining, and is in the tier in use. An ALB whose members all drain is therefore unavailable, whatever their health, although it still serves their sticky sessions. In the example, prom-alb is available because prom-01 takes new work; were prom-01 down too, it would be listed as unavailable while prom-03 went on serving the sessions pinned to it.

JSON Health Status

As the footer of the plaintext version of the health status page indicates, you may also request a JSON or YAML version of the health status for machine consumption. The JSON version includes additional detail about any Backends marked as unavailable, and is structured as follows:

$ curl "http://${trickster-fqdn}:8481/trickster/health?json" | jq

{
  "title": "Trickster Backend Health Status",
  "updateTime": "2026-09-26 00:00:00 UTC",
  "unavailable": [
    {
      "name": "prom-02",
      "provider": "prometheus",
      "downSince": "2026-09-26 00:00:00 UTC",
      "detail": "error probing target: dial tcp prom02.example.com:9090: connect: connection refused"
    }
  ],
  "available": [
    {
      "name": "prom-01",
      "provider": "prometheus"
    },
    {
      "name": "prom-03",
      "provider": "prometheus"
    },
    {
      "name": "prom-alb",
      "provider": "alb",
      "mechanism": "rr",
      "availablePoolMembers": [
        "prom-01",
        "prom-03"
      ],
      "unavailablePoolMembers": [
        "prom-02"
      ],
      "drainingPoolMembers": [
        "prom-03"
      ]
    }
  ],
  "unchecked": [
    {
      "name": "proxy-01",
      "provider": "reverseproxy"
    }
  ]
}

2 - ALB Autodiscovery

Autodiscovery keeps an ALB pool’s membership current at runtime: instead of hand-maintaining a pool list, Trickster watches a discovery source and adds, updates, and drains pool members as the source changes — with no config reload or listener restart. Discovered members are cloned from a designated template backend, so they inherit its caching, TLS, health check, path, and timeout configuration.

Supported Discovery Providers

providerdiscoverschange detection
kubernetesendpointslices, services or pods via the Kubernetes APIwatch (event-driven)
dns_srvSRV recordspoll
dns_aA/AAAA recordspoll
filea member-list filefilesystem notify + poll
http_sda member-list HTTP endpointpoll (ETag-aware)
consulConsul service instancesblocking query (event-driven)
nomadNomad’s native service registryblocking query (event-driven)
awsEC2 instances or ECS tasks, by aws.servicepoll
gcpCompute Engine instances, by gcp.servicepoll
azureVirtual Machines, by azure.servicepoll
dockerDocker Engine containerspoll

For other services, users are generally served by the file provider (any external service-discovery tool can emit the member list), by http_sd (the universal adapter — a few lines of glue serving a member list, with no Trickster change), or by the DNS providers (e.g. Consul’s DNS interface, cloud private DNS zones, Docker’s embedded DNS). Developers can add providers against a simple interface; see the provider authoring guide.

Configuration Overview

Autodiscovery has three configuration parts:

  1. a named discoverer in the top-level discovery section — the provider and its connection-level settings (Kubernetes client, DNS resolver). Multiple ALBs can share one discoverer, and therefore one client/watch/resolver stack.
  2. a template backend — a normal backend definition marked is_template: true. Templates are never routed, never eligible for is_default, never static pool members, and don’t require an origin_url; they exist to be cloned per discovered member.
  3. the ALB’s alb.discovery block — which discoverer to use, a provider-specific query describing what to select, the template to clone, and guardrail options.
discovery:
  in-cluster:
    provider: kubernetes
    kubernetes:
      in_cluster: true

backends:
  prom-template:
    provider: prometheus
    is_template: true
    cache_name: default
    healthcheck:
      interval: 5s

  prom-alb:
    provider: alb
    alb:
      mechanism: tsmerge
      discovery:
        discoverer_name: in-cluster
        template_backend: prom-template
        query:
          kind: endpointslices
          namespace: monitoring
          service: prometheus
          port: web

A discovery-backed ALB may also list static pool members; discovered members are additive to them. Static pool entries (and discovered members from sources that convey weights) support integer weights for the round_robin mechanism:

pool:
  - static-member          # weight 1
  - name: bigger-member
    weight: 3

Discoverer Connection Options

Each entry in the top-level discovery section declares a provider and that provider’s connection-level block:

providerblockoptions
kuberneteskubernetesin_cluster (default true) or kubeconfig path — mutually exclusive; plus qps, burst, user_agent, timeout
dns_srv, dns_adnsresolver (host:port; default: system resolver), interval (poll cadence, default 30s, min 1s; record TTLs act as a floor)
filefilepoll_interval (stat-poll fallback cadence, default 30s, min 1s)
http_sdhttp + http_sdconnection settings in the shared http block (below); http_sd.format selects the member-list document: trickster (default) or prometheus
consulhttp + consulconnection settings in the shared http block; consul.datacenter, namespace, partition, wait, allow_stale, only_passing, warning_is_ready
nomadhttp + nomadconnection settings in the shared http block; nomad.namespace, region, wait, allow_stale
awsaws (+ optional http)aws.service (required: ec2 or ecs), region, and the credential fields; the endpoint is derived, so http.endpoint is an optional override
gcpgcp (+ optional http)gcp.service (required; gce), gcp.project (from the metadata server when unset) and credentials_file; the endpoint is the Compute API, so http.endpoint is an optional override
dockerdocker (+ optional http)docker.api_version; the endpoint defaults to the well-known socket, so http.endpoint is an optional override (unix:// or tcp://)
azureazure (+ optional http)azure.service (required; vm), azure.subscription_id (required), credentials and cloud; the endpoint comes from the cloud, so http.endpoint is an optional override

A block is only valid on its own provider’s entries; anything else fails startup.

The shared http block

Providers that discover members by polling an HTTP endpoint share one connection block rather than each defining its own, so that configuring a second such provider does not mean learning a second vocabulary. It is required by http_sd, and rejected on providers that do not poll HTTP.

optionmeaning
endpointbase URL of the service to poll (http/https, host required)
intervalpoll cadence (default 30s, min 1s)
timeoutbound on a single poll (default 10s, min 100ms)
tlsoutbound client TLS: client_cert_path/client_key_path, certificate_authority_paths, insecure_skip_verify
headersheaders set on every request; where a registry’s credential is a bespoke header (X-Consul-Token, X-Nomad-Token), it goes here
username, passwordstatic HTTP Basic credential
bearer_tokensent as Authorization: Bearer <token>
bearer_token_filepath re-read before each poll, so a rotated credential is picked up without a restart — prefer this over bearer_token for anything that expires
follow_redirectsdefault false; a redirect away from the configured endpoint is surfaced rather than chased

username/password and the bearer-token fields are mutually exclusive, as are the two bearer-token forms: a config that sets both fails startup instead of silently preferring one, which is how an operator ends up debugging a 401 against a config that looks correct.

Providers whose normal poll includes a server-side wait (blocking queries) need timeout comfortably above that wait. There is no second, client-level timeout underneath it that could cut a long poll short.

Template Backends

The template_backend is an ordinary backend definition marked is_template: true. A discovered member overrides exactly the template’s name, origin_url (and its derived scheme/host/path-prefix, with the member’s path prefix falling back to the template’s), and replica group (see below); the member’s weight applies to its pool entry. Every other option — cache settings, TLS client configuration, healthcheck, paths, request rewriters, authenticator, timeouts, concurrency limits, tracing — is inherited from the template.

Templates are excluded from everything that applies only to live backends: they are never routed, cannot be is_default, cannot appear in a static ALB pool, and do not require an origin_url. A template must use an origin-serving provider (not alb or rule), and a TSM-merged or replica_group_label-using ALB requires a time-series-merge-capable template provider. A TSM ALB with evaluation: global requires a prometheus template with a series source. Discovery replaces only the origin, so a template’s series_endpoint, including the one Thanos StoreAPI requires, is used unchanged by every discovered member: each member reads that same store, and each read counts separately against max_fetched_series and max_fetched_samples. A pool of sharded Thanos sidecars discovered this way therefore reads only the one shard the endpoint names, and Trickster logs a warning at startup for any such template. Global evaluation over discovered members suits remote read without a series_endpoint, as with flavor: mimir, which reads /api/v1/read beneath each member’s own origin. That origin must be http or https, so a query.scheme of tcp or udp fails configuration, and a discovered member whose endpoint still can’t be formed is logged and not added to the pool. See Global Evaluation.

The alb.discovery Block

optiondescriptiondefault
discoverer_namename of a discoverer in the top-level discovery section (required)
template_backendname of a backend with is_template: true (required)
queryprovider-specific selection (required; see each provider below)
min_membersreject snapshots that would shrink the discovered membership below this count, keeping the last-good pool (guards against source blips returning empty results). 0 disables.0
debounce_windowcoalesce membership changes arriving within this window into one pool update, damping flapping sources. 0 disables.0
startup_policyretry starts the ALB with only its static members and keeps retrying the discoverer; fail fails startup when the discoverer is unavailableretry
health_modeprobe: discovered members inherit the template’s active health check. provider: the discoverer’s readiness reporting (currently Kubernetes only) drives member health instead of active probesprobe

Health and Readiness Semantics

In probe mode, each discovered member gets its own health check from the template’s healthcheck config, and the ALB’s healthy_floor applies exactly as it does to static members. If the template has no probe interval and the floor requires passing probes, the floor is reset to 0 with a loud warning (as with static members).

A new member’s first probe runs the moment it is registered, and a member the provider reports ready is admitted to the pool immediately, ahead of that first result. This matters for rolling deploys: an orchestrator retires the old workload on the very readiness signal that announced the new one, so a pool that held the newcomer in unknown until its own probe completed would have no admissible member in between, and every request in that window would fail with a 502 even though a healthy workload was serving. From its first result on, the probe governs the member exactly as it does an established one: a member the provider reports not-ready, or one whose readiness the provider does not convey, waits for its first passing probe as before.

In provider mode, readiness reported by the discoverer maps onto member health: ready and terminating members enter as passing (1), not-ready members as failing (-1), readiness-unknown members as unchecked (0) — so a healthy_floor of 1 excludes members until the provider reports them ready.

Members a provider reports as shutting down but still serving (terminating Kubernetes endpoints whose serving condition holds, deletion-stamped pods that are still ready) stay in the pool as draining members: from the moment the change is observed they take no new requests, connections or sessions, while sticky sessions already pinned to them keep reaching them for as long as they are available. Once the provider reports that a member no longer serves, it is removed from the snapshot, before the workload is killed — enabling zero-error rolling deploys. On removal, a member’s in-flight requests complete, its health check stops, its metrics series are deleted, and its idle upstream connections close after the configured drain timeout. Draining members count toward min_members and trickster_alb_discovery_members, and are shown on the health status page and by the trickster_alb_member_draining gauge.

TSM Replica Groups

When the ALB uses the Time Series Merge mechanism, discovered members participate in replica_group semantics three ways, in priority order:

  1. Per-member groups from the discovery source (highest priority): the Kubernetes provider can read a configured label from each discovered workload via query.replica_group_label, and file provider entries may carry a replica_group field. Members sharing a group value are treated as HA replicas of one logical shard and coalesced; distinct values are distinct shards, merged. This handles sharded-HA topologies (e.g., Thanos-style Prometheus pairs) discovered with a single selector:

    backends:
      prom-alb:
        provider: alb
        alb:
          mechanism: tsmerge
          discovery:
            discoverer_name: in-cluster
            template_backend: prom-template
            query:
              kind: endpointslices
              namespace: monitoring
              service: prometheus
              port: web
              replica_group_label: prometheus/shard
    
  2. A replica_group set on the template backend: inherited by every member without a source-conveyed group — all such members are HA replicas of one shard (the common HA-pair case).

  3. Neither (default): each member is its own replica group, i.e. its own logical shard — a pure federation/merge of every member.

For the pods and service kinds the label is read from the discovered Pod or Service itself. For endpointslices, endpoints don’t carry pod labels, so setting replica_group_label joins a Pod watch in the query’s namespace to resolve each endpoint’s target pod (see the RBAC note below); an endpoint whose pod isn’t yet observed joins ungrouped and regroups automatically when the pod appears. A member whose group changes is rebuilt in place under the same name.

Because replica_group is only meaningful on time-series backends, configuration validation requires a TSM-capable template_backend whenever the ALB mechanism is tsmerge or replica_group_label is set. The DNS providers convey no grouping metadata; use the template-level group (mode 2) or the file provider for grouped non-Kubernetes pools.

Query Vocabulary

Every ALB’s alb.discovery.query block is drawn from one shared vocabulary. A field set for a provider that does not accept it fails startup rather than being silently ignored, so a query aimed at the wrong provider is a config error rather than an empty pool.

fieldkubernetesdns_srvdns_afilehttp_sdconsulnomadawsgcpazuredocker
kind●
namespace●
service●●●●
selector●
srv_name●
hostname●
path●●
filter●●●
filters●●
tags●●●●●
cluster●
network●
address_type●●●●
port●●●●●●
port_label●●●●
scheme●●●●●●●●●●
replica_group_label●●●●●●

Most fields mean the same thing everywhere:

  • scheme — http (default) or https for the member’s origin URL.
  • port — a static port applied to every member.
  • port_label — reads the port from the provider’s own metadata namespace: an EC2 or Azure tag, a GCE label (then instance metadata), a Docker container label. It wins over port per member, so a fleet can share one static default with individual machines overriding it.
  • address_type — private (default), public, or ipv6. Which address to take where a resource has several.
  • replica_group_label — reads a TSM replica group from the same namespace as port_label.
  • tags — narrows by tag/label presence, and is a conjunction: every listed tag must be present.
  • filter / filters — provider-native selection evaluated server-side. filter is an expression string (Consul, Nomad, GCE); filters is a name→values map (AWS Filter.N, Docker’s filter document). These are passed through, so an expression the upstream does not understand fails the refresh loudly rather than matching nothing.

Hosts versus endpoints is the distinction that decides whether a port is required. A cloud inventory API returns hosts — aws service: ec2, gcp, and azure return addresses with no port, so one of port or port_label is required. Kubernetes and docker return endpoints that carry their own ports, so a port is optional there and is resolved from the object when unambiguous.

The kubernetes Provider

The kubernetes provider watches the cluster via the API server (shared informers; watch-driven, no polling) and supports three query kinds.

Connection options:

discovery:
  my-cluster:
    provider: kubernetes
    kubernetes:
      in_cluster: true              # use the pod's service account (default)
      # kubeconfig: /path/to/kubeconfig   # or run against a remote cluster
      qps: 20                       # sustained API request rate (default 20)
      burst: 40                     # API request burst allowance (default 40)
      timeout: 10s                  # bounds one-shot API calls (default 10s)
      # user_agent: my-trickster/1  # default: trickster/<version> (<os>/<arch>)

in_cluster and kubeconfig are mutually exclusive; with neither set, in-cluster is assumed.

qps and burst are the client-side rate limit on API requests. The defaults are raised over the client-go defaults (5 and 10), which are sized for a one-shot CLI rather than a process that watches many objects.

timeout bounds a single one-shot API call, such as the connectivity preflight run under startup_policy: fail. It is deliberately not applied to the REST client as a whole, whose timeout would also truncate the long-running watch streams the informers depend on.

Shared Watches

Discoverers that agree on the connection (API server, credentials, identity) and on the query’s server-side filtering (namespace, label selector, field selector) share one set of informers, so several ALBs selecting the same Service cost the API server one watch rather than one each. A watch is released when the last subscription using it stops, so unsubscribing one ALB never disturbs another. Nothing needs to be configured for this; it follows from the discoverer and query values being identical.

Startup Preflight

When any ALB bound to a kubernetes discoverer sets startup_policy: fail, the discoverer contacts the API server’s /version endpoint before any watch is established, bounded by timeout. Informers are lazy, so without this check an unreachable API server is indistinguishable from a Service with no endpoints yet: the process would start clean and serve an empty pool. Under the default startup_policy: retry the preflight is skipped and the discoverer keeps watching for the API server to come back.

Query Kinds

endpointslices (default) discovers the endpoint addresses of a named Service — the pod IPs behind it — and is the right choice for routing around the Service’s own load balancing. Terminating endpoints that still serve join as draining members:

query:
  kind: endpointslices
  namespace: monitoring
  service: prometheus
  port: web          # port name or number; optional when the service has one port

service discovers the ClusterIP of each Service matching a label selector (or one named Service), one member per Service:

query:
  kind: service
  namespace: monitoring
  selector:
    app: prometheus
  port: 9090

pods discovers pod IPs directly by label selector, without requiring a Service (the original ask of issue #609):

query:
  kind: pods
  namespace: monitoring
  selector:
    app: prometheus
  port: web

namespace defaults to the pod’s own namespace when running in-cluster.

Port and Scheme Resolution

port may be a named port, a number, or omitted when the target declares exactly one port; ambiguity is logged and the object skipped. The member scheme comes from query.scheme if set; otherwise a declared appProtocol: https on the selected port, or a trickstercache.org/scheme annotation on the watched Service/Pod, selects https; the default is http.

Zero-Error Rolling Deploys

Terminating endpoints stop taking new work as soon as the change is observed: they drain, keeping only their sticky sessions, and leave the discovered membership once they stop serving, ahead of pod deletion. One piece belongs to the workload, though: Kubernetes marks an endpoint terminating and signals the container at the same moment, so — as with any EndpointSlice consumer, kube-proxy included — the pod must keep serving briefly while its removal propagates. Give discovered workloads a short preStop delay (a few seconds comfortably covers Trickster’s sub-second pickup):

lifecycle:
  preStop:
    sleep:         # native sleep action (Kubernetes >= 1.30); images with
      seconds: 5   # a shell can use exec with command: ["sleep", "5"]

Without it, an instantly-exiting container can produce a brief window of connection errors during rollouts — regardless of what is consuming the EndpointSlices.

RBAC

The provider needs only list and watch on the resources your query kinds use — it never reads Secrets or ConfigMaps and never writes:

query kindapiGroupresourceverbs
endpointslicesdiscovery.k8s.ioendpointsliceslist, watch
service"" (core)serviceslist, watch
pods"" (core)podslist, watch

Queries are namespace-scoped, so a namespaced Role/RoleBinding per watched namespace is sufficient and preferred:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: trickster-autodiscovery
  namespace: monitoring
rules:
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: trickster-autodiscovery
  namespace: monitoring
subjects:
  - kind: ServiceAccount
    name: trickster
    namespace: trickster
roleRef:
  kind: Role
  name: trickster-autodiscovery
  apiGroup: rbac.authorization.k8s.io

Add services and/or pods rules only for the kinds you configure. Note that using replica_group_label with the endpointslices kind joins a Pod watch, so it additionally requires pods list/watch in the query’s namespace. RBAC cannot scope below the resource level, so the grant covers all objects of that resource in the bound namespace. Out-of-cluster (kubeconfig-based) discoverers need the same permissions for their user or service account.

The dns_srv Provider

The dns_srv provider polls SRV records. SRV target and port map to the member address; SRV weight maps to the member’s load-balancing weight; and only the highest-priority tier (the lowest priority value present in the answer) becomes members — lower tiers are treated as standby capacity managed on the DNS side.

discovery:
  corp-dns:
    provider: dns_srv
    dns:
      resolver: 10.0.0.53:53   # optional; default: the system resolver
      interval: 30s            # poll cadence (default 30s)

backends:
  my-alb:
    provider: alb
    alb:
      mechanism: rr
      discovery:
        discoverer_name: corp-dns
        template_backend: my-template
        query:
          srv_name: _prometheus._tcp.example.com
          # scheme: https      # optional; default http

Record TTLs act as a floor on the poll cadence: an answer is never re-resolved before its shortest TTL expires, so interval is the most frequent the provider will query. Resolution failures keep the last-good membership; an authoritative empty answer empties it.

The dns_a Provider

The dns_a provider resolves a hostname’s A and AAAA records, with a fixed port and scheme from the query. This covers round-robin DNS, headless-service-style DNS outside Kubernetes, Docker’s embedded DNS, and Consul’s DNS interface without a bespoke provider:

query:
  hostname: prometheus.service.consul
  port: 9090           # required
  # scheme: https      # optional; default http

The dns connection options, poll cadence, TTL floor, and failure semantics are the same as dns_srv.

The file Provider

Note: like Prometheus’s file_sd, the file provider is the universal integration point — anything that can write a file (cron, sidecar, consul-template, your deploy tooling) can drive an ALB pool.

The provider watches a local YAML or JSON member-list file and applies it atomically on change:

discovery:
  external-sd:
    provider: file
    # file:
    #   poll_interval: 30s   # stat-poll fallback cadence (default 30s)

backends:
  my-alb:
    provider: alb
    alb:
      mechanism: rr
      discovery:
        discoverer_name: external-sd
        template_backend: my-template
        query:
          path: /etc/trickster/members.yaml

The file is a list of members:

- name: prom-1            # optional; defaults to the address
  address: 10.0.0.1:9090  # required host:port
- name: prom-2
  scheme: https           # optional; default http
  address: 10.0.0.2:9090
  path_prefix: /base      # optional
  weight: 3               # optional; default 1
  replica_group: shard-0  # optional; TSM replica group (see above)

Writers should replace the file atomically (write to a temp file in the same directory, then rename). A file that fails to read or parse keeps the last-good membership; an empty file is a valid, empty membership.

Change Detection

The provider uses two mechanisms together (via Trickster’s shared filesystem watcher), so updates are never missed:

  1. Filesystem notification (on the file’s parent directory, debounced): near-instant pickup of writes, atomic renames, and symlink swaps on local filesystems; a directory watch dropped by deletion or recreation is automatically re-armed.
  2. A content-comparing poll of the file (file.poll_interval, default 30s, minimum 1s): the guaranteed fallback wherever notification is unreliable or unavailable.

Guidance for Kubernetes-mounted member files:

  • ConfigMap / Secret volume mounts: kubelet applies updates with an atomic symlink swap inside the mount directory, which generates inotify events — notification works, and updates are picked up promptly. Note that kubelet itself syncs ConfigMap content periodically (typically up to a minute), which usually dominates end-to-end latency.
  • subPath mounts of a ConfigMap/Secret: Kubernetes never updates these after pod start — no mechanism (notification or polling) can see a change. Mount the directory, not a subPath.
  • Network- or FUSE-backed volumes (NFS PVs, some CSI drivers): inotify generally does not observe writes made by other clients, so the stat poll is the effective update mechanism — set file.poll_interval to the freshness your deployment needs.
  • emptyDir/hostPath written by a sidecar in the same pod: notification works.

The poll compares file content (not timestamps), so any change is detected within one poll_interval regardless of filesystem timestamp granularity.

The http_sd Provider

http_sd fetches a member list from an HTTP endpoint. It is the universal adapter: any service-discovery system Trickster has no in-tree provider for can feed it through a few lines of glue that serve a member list, with no Trickster change and no restart.

discovery:
  fleet:
    provider: http_sd
    http:
      endpoint: https://sd.example.com
      interval: 15s
      bearer_token_file: /var/run/secrets/sd-token
    http_sd:
      format: trickster

backends:
  prom-template:
    provider: prometheus
    is_template: true
  prom-alb:
    provider: alb
    alb:
      mechanism: tsm
      discovery:
        discoverer_name: fleet
        template_backend: prom-template
        query:
          path: /pools/prometheus

The query’s path is optional and is appended to the endpoint, so one server can serve a different member list per ALB while they share the discoverer’s connection settings. The query’s scheme supplies the scheme for prometheus-format targets, which are bare host:port; native-format entries carry their own and override it.

Formats. trickster is the same document the file provider reads and is the default:

- name: prom-1
  scheme: https
  address: 10.0.0.1:9090
  path_prefix: /base
  weight: 2
  replica_group: shard-0

prometheus is the document Prometheus’s own file_sd and http_sd consume, so an existing endpoint can be pointed at Trickster unchanged:

[{"targets": ["10.0.0.1:9090"], "labels": {"env": "prod"}}]

A group’s __scheme__ label overrides the query scheme, letting one endpoint serve mixed-scheme members. Note that the Prometheus format cannot express weight or replica_group — deployments that need weighted pools or TSM replica groups want the native format.

The format is named explicitly rather than sniffed. The two documents are structurally distinguishable, but guessing means a typo in one format can parse as a valid, wrong membership in the other, and the cost of guessing wrong is a silently drained pool.

Efficiency. Requests carry X-Prometheus-Refresh-Interval-Seconds, as Prometheus’s does, so servers that generate member lists on demand can pace their own work. ETag is honored: an unchanged membership costs one conditional request and a 304, with no re-parse and no snapshot. The validator is only stored after a document parses, so a rejected document can never be confirmed as current by a later 304.

Failure. A transport error, an unexpected status, an oversized body (the limit is 16 MiB), or a document that will not parse all keep the last-good membership and are logged once per failure streak and counted on trickster_discovery_refresh_errors_total. An endpoint that authoritatively returns no members ([]) is a valid membership and is applied, so a scaled-to-zero pool can be reported as such.

The consul Provider

consul reads service instances from Consul’s health endpoint. It is event-driven rather than polled: each request is a Consul blocking query, so the server parks it until the service changes or wait elapses. A membership change is observed within a round trip instead of within a poll interval, and a stable service costs one parked connection rather than a request per interval.

discovery:
  consul-dc1:
    provider: consul
    http:
      endpoint: http://127.0.0.1:8500
      # a rotated ACL token; Consul accepts the Authorization Bearer scheme
      # as an equivalent to its own X-Consul-Token header
      bearer_token_file: /var/run/secrets/consul-token
    consul:
      datacenter: dc1
      wait: 5m          # how long a blocking query parks; 1s–10m
      allow_stale: true # answer from any server, not only the leader

backends:
  prom-template:
    provider: prometheus
    is_template: true
  prom-alb:
    provider: alb
    alb:
      mechanism: tsm
      health_mode: provider   # Consul's own checks decide readiness
      discovery:
        discoverer_name: consul-dc1
        template_backend: prom-template
        query:
          service: prometheus
          tags: [production]
          # filter: 'Service.Meta.version == "2"'
          # replica_group_label: shard   # read from the service's Meta

Readiness. Consul reports per-instance check status, so this is the first provider outside kubernetes that can honestly answer “is this member ready”, which makes health_mode: provider meaningful for VM and container fleets. An instance’s readiness is its worst check: all passing is ready, critical and maintenance are not ready, and warning is ready by default (matching how Consul treats warning for DNS) — set warning_is_ready: false to drain warning instances instead. A status a future Consul release introduces is treated as not-ready rather than ignored.

Failing instances are reported as NotReady rather than omitted, so that an ALB using the default health_mode: probe can decide for itself and a wholly-unhealthy service does not look like an empty one. Set only_passing: true to have Consul filter them out server-side instead.

Weights. Consul’s own Weights.Passing / Weights.Warning map onto member weights, including the passing/warning distinction — an operator who has already told Consul the relative capacity of each instance does not have to tell Trickster again.

Addresses. A service that registers its own address overrides its node’s, which is how sidecars and containers with their own routable address are represented. An instance with no usable address or port fails the whole refresh rather than being silently dropped, so a pool never quietly shrinks because of a catalog change nobody noticed.

Labels. Members carry service, service_id, node, datacenter, status, and tags (comma-bracketed, as ,a,b,). Service metadata is carried as meta_<key> so that an operator-defined key cannot shadow a Trickster-assigned label.

Timeouts. http.timeout must outlast consul.wait, because a blocking query legitimately takes that long. Its default is derived from the wait rather than shared with the other HTTP providers (Consul adds up to wait/16 of its own jitter, which the margin covers), and a config that sets it too low is rejected at startup rather than producing a stream of timeouts. http.interval is not the poll cadence here — with blocking queries there is no cadence — it is the retry delay after a failure.

The nomad Provider

nomad reads service instances from Nomad’s native service registry (Nomad 1.3+). Like consul it is event-driven, using the same HashiCorp blocking-query protocol, so a membership change is observed within a round trip rather than within a poll interval.

discovery:
  nomad-eu:
    provider: nomad
    http:
      endpoint: http://127.0.0.1:4646
      # a rotated ACL token; Nomad accepts the Authorization Bearer scheme
      # as an equivalent to its own X-Nomad-Token header
      bearer_token_file: /var/run/secrets/nomad-token
    nomad:
      namespace: default
      region: eu-1
      wait: 5m
      allow_stale: true

backends:
  prom-alb:
    provider: alb
    alb:
      discovery:
        discoverer_name: nomad-eu
        template_backend: prom-template
        query:
          service: prometheus
          tags: [production]
          # filter: 'JobID == "monitoring"'

Native registry, not Consul. This reads the registry a job selects with provider = "nomad" in its service block. Jobs that register into Consul instead are discovered with the consul provider, and that is the more capable choice where it applies: Nomad’s service endpoint carries no per-instance check state, so members are reported ReadyUnknown and health_mode: provider falls back to Trickster’s own probes. A deployment that wants discovery-conveyed readiness should register its services into Consul.

Tags filter client-side. Unlike Consul’s catalog endpoint, Nomad’s service endpoint has no tag parameter, so query.tags is applied by Trickster after the response arrives. It is a conjunction — every listed tag must be present. query.filter is passed through to Nomad and evaluated server-side.

Labels. Members carry service, service_id, job_id, alloc_id, node_id, namespace, datacenter, and tags (comma-bracketed). The allocation and job identifiers are what an operator needs to trace a member back to the workload that registered it.

Timeouts work exactly as for consul: http.timeout must outlast nomad.wait, its default is derived from the wait, and http.interval is the retry delay after a failure rather than a poll cadence.

The aws Provider

aws discovers members from an AWS API, selected by aws.service: ec2 for instances, ecs for tasks. It is required — with more than one AWS API supported, defaulting would be an arbitrary guess at which one you meant, so a config that omits it fails at startup. Further AWS sources arrive as new service values rather than new providers, inheriting this provider’s credentials, signing, pagination and options.

discovery:
  fleet:
    provider: aws
    aws:
      service: ec2      # required: ec2 or ecs
      region: us-east-1
      # credentials omitted: use the standard chain (IRSA, instance
      # profile, environment, shared config). See docs/aws.md.
    http:
      interval: 60s   # instance inventories change slowly; poll gently

backends:
  prom-alb:
    provider: alb
    alb:
      discovery:
        discoverer_name: fleet
        template_backend: prom-template
        query:
          filters:
            tag:service: [prometheus]
            instance-state-name: [running]
          port_label: trickster-port   # read the port from an EC2 tag
          address_type: private        # private (default), public, or ipv6
          # port: 9090                 # or a static port for every member
          # replica_group_label: shard

Credentials, region resolution and IAM are documented once in AWS Integration. The IAM principal needs ec2:DescribeInstances.

Hosts, not endpoints. An instance inventory returns hosts with several addresses and no port, so two query fields exist that the registry providers do not need:

  • address_type — private (default), public, or ipv6 — selects which of the instance’s addresses becomes the member address.
  • port_label — names an EC2 tag whose value is the member’s port. port supplies a static one. At least one is required, and they compose: where both are set, the tag wins per instance and port is the fallback, which is what makes port_label safe to adopt incrementally across a fleet.

Selection. filters is passed to EC2 as Filter.N, evaluated server-side — use tag:<key> to filter on a tag value, and any other DescribeInstances filter name. tags additionally requires the presence of tag keys, applied after the response arrives.

Instance state. running is Ready and pending is NotReady. Instances that are shutting-down, stopping, stopped or terminated are omitted entirely rather than reported unready, so they drain from pools before they stop answering — the same rule the kubernetes provider applies to terminating endpoints. A state a future EC2 release introduces is treated as not-ready rather than assumed healthy.

Instances that cannot become members are excluded, not fatal. An instance with no port_label tag and no static port, or with no address of the requested type, is skipped and logged (once, until the set of excluded instances changes) rather than failing the refresh. This differs deliberately from the consul and nomad providers, where a malformed entry fails the whole refresh: a service registry contains only instances of the service, so a bad entry means the API is broken, while an EC2 inventory routinely contains hosts that simply are not tagged yet. Failing there would drain a working pool because of one unrelated instance.

Labels. Members carry instance_id, instance_type, instance_state, image_id, availability_zone, vpc_id, subnet_id, architecture, private_ip, public_ip, private_dns and public_dns. Instance tags are carried as tag_<Key>, prefixed so an operator-defined tag cannot shadow a Trickster-assigned label. The Name tag becomes the member name, falling back to the instance id.

Endpoint. Derived from the region and service (https://ec2.<region>.amazonaws.com). Set http.endpoint to override it for a VPC endpoint, a FIPS endpoint, or a test double; a region is required when it is not overridden, because the endpoint is built from it.

service: ecs

Discovers ECS tasks, including Fargate, which service: ec2 structurally cannot see — a Fargate task has no EC2 instance behind it.

discovery:
  tasks:
    provider: aws
    aws:
      service: ecs
      region: us-east-1
    http:
      interval: 30s

backends:
  prom-alb:
    provider: alb
    alb:
      discovery:
        discoverer_name: tasks
        template_backend: prom-template
        query:
          cluster: prod           # ECS cluster; the account default when unset
          service: prometheus     # ECS service name; optional
          port_label: trickster-port
          # port: 9090            # or a static port for every task

The IAM principal needs ecs:ListTasks and ecs:DescribeTasks.

awsvpc only. Each task in awsvpc mode has its own elastic network interface, so DescribeTasks alone yields a routable address. That is the only mode Fargate offers and the default for new EC2-launch-type services. Under bridge or host networking the address belongs to the container instance rather than the task, and resolving it would take two further API calls, a second signing service and broader IAM — so such tasks are excluded with a reason saying so rather than silently missing. If you need bridge or host mode, say so and it can be added.

Tags must be propagated. port_label reads an ECS task tag, and ECS does not copy service tags onto tasks unless the service is created with --propagate-tags SERVICE. Trickster asks DescribeTasks for tags explicitly (include: [TAGS]); if port_label finds nothing, check the propagation setting first.

Task state. RUNNING is Ready; PROVISIONING, PENDING and ACTIVATING are NotReady; DEACTIVATING, STOPPING, DEPROVISIONING and STOPPED are omitted entirely, so tasks drain from pools before they stop answering. Container health is only reported when the task definition declares a health check — UNHEALTHY is NotReady, and UNKNOWN or absent is treated as ready, because the alternative would make every un-instrumented task permanently unusable.

Selection is by cluster and service; filters and address_type are rejected for ecs, since ECS selects by cluster rather than by instance attribute and an awsvpc task has exactly one address. tags still filters on tag presence after the response arrives.

Labels. Members carry task_arn, task_id, cluster, task_definition, group, launch_type, availability_zone, task_status and health_status, plus task tags as tag_<Key>. The Name tag becomes the member name, falling back to the task id.

Churn. A task that disappears between ListTasks and DescribeTasks is ordinary churn; it is reported as an exclusion rather than silently shrinking the pool.

The gcp Provider

gcp reads a Google Cloud API named by gcp.service. The provider is named for the cloud rather than for Compute Engine, matching aws — Google Cloud APIs outside Compute Engine belong here too, and would sit oddly under a provider called gce.

service is required even though gce is the only value today. A default added now could never be removed, and every service added later would then be reached by opting out of a value the operator never chose.

A value names the product an operator would recognize, not the API that serves it — the same convention as aws.service. That matters here: Cloud Load Balancing is served by the Compute API alongside instances, so naming these for the API would collide where naming them for the product does not.

service: gce

Discovers Google Compute Engine instances through the Compute API’s instances.aggregatedList, which covers every zone in the project in one paged call — so no zone list has to be configured or kept current.

discovery:
  fleet:
    provider: gcp
    gcp:
      service: gce               # required
      project: my-project        # from the metadata server when unset
      # credentials_file: /etc/trickster/sa.json   # ADC when unset
    http:
      interval: 60s              # instance inventories change slowly

backends:
  prom-alb:
    provider: alb
    alb:
      discovery:
        discoverer_name: fleet
        template_backend: prom-template
        query:
          filter: 'labels.role = "prometheus" AND status = "RUNNING"'
          tags: [http-server]     # network tags, matched on presence
          port_label: port        # instance label, or metadata key
          address_type: private   # private (default), public, or ipv6
          # port: 9090            # or a static port for every member
          # replica_group_label: shard

Credentials. Leaving credentials_file empty selects Application Default Credentials: GOOGLE_APPLICATION_CREDENTIALS, gcloud user credentials, Workload Identity on GKE, or the instance metadata server on GCE. Prefer those over a key file wherever the platform offers one. Credentials resolve lazily and only successes are cached, so Trickster starts even when the metadata server is briefly unreachable and a momentary failure does not permanently disable discovery.

credentials_file must be a service account key. The credential type is required rather than taken from the file: an external_account or impersonated_service_account configuration can name an arbitrary token URL or local executable, so accepting whichever type a file happens to declare would hand credential resolution somewhere unintended. For user credentials, use ADC instead of this field.

The IAM principal needs compute.instances.list on the project — the roles/compute.viewer role includes it. The OAuth scope requested is compute.readonly; Trickster never mutates a project.

Project. Taken from gcp.project, then from the credentials, then from the metadata server when Trickster runs on GCE. It is deliberately not required in config, because reading it from the metadata server is the idiomatic deployment.

Hosts, not endpoints, exactly as for aws service: ec2: address_type chooses the address and port/port_label supplies the port, with the label winning per instance and the static port as fallback. port_label reads an instance label first, then instance metadata — both are key/value namespaces on a GCE instance, and a deployment already carrying the value in metadata does not have to move it.

Instance status. RUNNING is Ready; PROVISIONING, STAGING and REPAIRING are NotReady; STOPPING, SUSPENDING, SUSPENDED and TERMINATED are omitted entirely, so instances drain from pools before they stop answering. A status a future Compute Engine release introduces is treated as not-ready rather than assumed healthy.

Selection. filter is a GCE filter expression, evaluated server-side. tags matches GCE network tags, which are names without values, so it filters on presence. Requests set returnPartialSuccess, so one unreachable zone contributes no instances rather than failing the whole refresh.

Labels. Members carry instance_id, instance_name, status, zone, machine_type, network, subnetwork, private_ip, public_ip, and tags (comma-bracketed). Instance labels are carried as label_<key>, prefixed so a user-defined label cannot shadow a Trickster-assigned one. Resource URLs are shortened to their last segment, since a member label full of https://www.googleapis.com/compute/v1/... is unreadable.

The azure Provider

azure reads an Azure Resource Manager API named by azure.service. service is required even though vm is the only value today, for the same reason as aws.service and gcp.service: a default added now could never be removed.

service: vm

Discovers Virtual Machines, joining them to their network interfaces to resolve addresses.

discovery:
  fleet:
    provider: azure
    azure:
      service: vm                # required
      subscription_id: 00000000-0000-0000-0000-000000000000  # required
      # resource_group: prod-rg  # narrows every list; recommended
      # credentials omitted: use the managed identity of the VM or AKS pod
      # tenant_id: ...           # with a service principal:
      # client_id: ...
      # client_secret: ...
      # federated_token_file: /var/run/secrets/azure/tokens/azure-identity-token
      # cloud: public            # public (default), usgovernment, china
      # power_state: false       # one extra list call; see Readiness
    http:
      interval: 60s              # vm inventories change slowly

backends:
  prom-alb:
    provider: alb
    alb:
      discovery:
        discoverer_name: fleet
        template_backend: prom-template
        query:
          tags: [prometheus]     # vm tag names, matched on presence
          port_label: port       # a vm tag holding the port
          # port: 9090           # or a static port for every member
          # address_type: private # private (default), public, or ipv6
          # replica_group_label: shard

Credentials. Leaving the credential fields empty selects the managed identity of the Azure VM or AKS pod Trickster runs on, via the instance metadata service; set client_id alone to select a user-assigned identity. On AKS with workload identity, set federated_token_file to the projected token path — the platform writes and rotates that file, so Trickster holds no secret, and the file is re-read on every token acquisition. A client_secret service principal is the fallback where neither is available; it is redacted from config dumps and the management API.

There is deliberately no credential chaining. A configured credential that fails is an error, not a reason to quietly fall back to the instance metadata service and authenticate as a different principal.

Permissions. The principal needs only Reader on the subscription, or on the resource group named by resource_group. The specific actions are Microsoft.Compute/virtualMachines/read and Microsoft.Network/networkInterfaces/read, plus Microsoft.Network/publicIPAddresses/read when address_type: public. power_state: true additionally requires read at subscription scope, even with resource_group set — see Readiness.

If discovery finds nothing, check resource provider registration first. A subscription where Microsoft.Compute is not registered answers the VM list with HTTP 200 and an empty result, not an error, so the pool is silently empty. New subscriptions start unregistered. Trickster detects this case and logs it, but the fix is outside Trickster:

az provider register -n Microsoft.Compute && az provider register -n Microsoft.Network

Clouds. cloud selects both the ARM endpoint and the Entra ID login endpoint together, since keeping two URLs consistent by hand is exactly the kind of thing that silently half-works. http.endpoint overrides the ARM endpoint alone, for a private ARM proxy or a test server.

API versions. ARM requires an explicit api-version on every request, so both are pinned rather than omitted: compute_api_version (default 2024-07-01) for the VM list and network_api_version (default 2024-05-01) for interfaces and public addresses. Pinning means the response shape cannot change under Trickster when Microsoft ships a new version. Override either to move forward deliberately.

The join. An Azure VM carries no address. Addresses live on network interfaces, which the VM references by resource id, and a public address is a further reference from the interface to a publicIPAddresses resource. A refresh is therefore two list calls — three with address_type: public — joined in memory. Resource ids are matched case-insensitively, because Azure treats them that way and the casing genuinely differs between APIs; a VM’s interface reference commonly spells the resource group differently from the interface list. A VM whose references resolve to nothing is excluded with a message saying so specifically, since that is the symptom a broken join produces.

resource_group is worth setting on a large subscription: it narrows every list rather than enumerating thousands of machines to find a handful.

Hosts, not endpoints, exactly as for aws service: ec2 and gcp service: gce: address_type chooses the address and port/port_label supplies the port, with the tag winning per VM and the static port as fallback. VM tags are matched case-insensitively, as Azure treats them, so a machine tagged Role is found by a query for role.

Readiness. Off by default, ReadyUnknown for every member. The VM list carries provisioning state, not power state, and a provisioned VM may be stopped — reporting Ready on that basis would assert something Azure never said. Setting power_state: true requests the instance view, which makes running VMs Ready and removes stopped ones from membership entirely so they drain from pools. The cost is one extra list call per refresh, not one call per VM: the whole-subscription list accepts statusOnly=true and returns every machine’s instance view at once. Trickster’s own active health checks are the alternative, and cover the case either way.

statusOnly is a parameter of the subscription-wide list only. The resource-group-scoped list accepts it, returns 200, and silently ignores it, so Trickster always issues this one call at subscription scope — which is why power_state: true needs subscription-scope read even when resource_group is set. If the status list ever comes back with no instance view for any machine, the refresh fails and keeps the last-good membership rather than reading every VM as stopped and emptying the pool.

Labels. Members carry vm_name, vm_id, location, vm_size, resource_group, private_ip, public_ip, and power_state when requested. VM tags are carried as tag_<key>, prefixed so a user-defined tag cannot shadow a Trickster-assigned label.

The docker Provider

docker discovers containers through the Docker Engine API’s GET /containers/json, polled on http.interval.

discovery:
  containers:
    provider: docker
    docker: {}                 # api_version: v1.41 by default
    http:
      # endpoint: unix:///var/run/docker.sock   # the default
      # endpoint: tcp://dockerhost:2376         # remote daemon; add a tls block
      interval: 30s

backends:
  prom-alb:
    provider: alb
    alb:
      discovery:
        discoverer_name: containers
        template_backend: prom-template
        query:
          filters:
            label: [com.example.discover=yes]
          # network: backend      # required only when a container is on several
          # port: 9090            # or port_label, when the container exposes several
          # address_type: private # private (default), public, or ipv6
          # replica_group_label: com.example.shard

Endpoint. Taken from http.endpoint, defaulting to unix:///var/run/docker.sock. Both the unix:// and tcp:// forms DOCKER_HOST uses are accepted, so one can be pasted into the other. A tcp:// endpoint takes its client TLS from the shared http.tls block — that is how a remote daemon’s mutual TLS is configured — and becomes https on the wire when one is present. TLS on a unix:// endpoint is rejected rather than ignored.

Access. Trickster needs read access to the Docker socket, which on most hosts means membership of the docker group. That access is equivalent to root on the host, so prefer a read-only socket proxy in production over mounting the socket directly. Trickster only ever issues GET /containers/json.

API version. Pinned to v1.41 (Docker 20.10 and later) rather than left off. An unversioned request binds to whatever the daemon’s newest version happens to be, so the response shape would change under Trickster when the host upgrades Docker. Override with docker.api_version to use a newer one.

Endpoints, not hosts. Unlike the cloud providers, the Engine API reports ports, so port is not required. A container exposing exactly one TCP port resolves automatically; one exposing several is excluded with a reason naming the candidates, rather than having a guess made for it. UDP ports are never candidates, which is what makes the single-port case common in practice — a container exposing one TCP port beside two UDP ports is unambiguous. port_label reads the port from a container label and wins over a static port per container.

Addresses. address_type: private (the default) uses the container’s IP on its network, with the container-internal port. public uses the host binding instead — the address and port are taken from the same binding, so a container publishing one port on 127.0.0.1 and another on 0.0.0.0 does not produce a mismatched pair. A wildcard binding (0.0.0.0 or ::) is not dialable, so it resolves to 127.0.0.1. ipv6 uses the container’s global IPv6 address.

Networks. A container on exactly one network needs no network. One on several must name which, rather than having a map iteration pick — that would differ between polls and churn the pool. A container attached to no network is excluded.

Readiness. State: running is required; every other state leaves membership entirely, so containers drain from pools as they stop. Health comes from the container’s HEALTHCHECK: healthy is Ready, unhealthy and starting are NotReady, and a container with no healthcheck declared is ReadyUnknown, not Ready — the daemon knows the process started, not that it is serving, and Trickster’s own health checks cover that. Note that GET /containers/json reports health only inside its human-readable Status string; the per-container inspect that carries a structured Health object would cost one request per container per poll.

Selection. filters is passed to the Engine API as its own filter document and evaluated server-side, so the daemon does the narrowing: label, name, status, health, network, ancestor and the rest. A filter name the daemon does not recognize is a 400 and fails the refresh loudly, rather than being ignored into a silently empty pool. When the query sets no status filter, status: [running] is added, so a host with a long history of exited containers is not listed in full every poll.

Labels. Members carry container_id (short form), container_name, image, state, network and private_ip. Container labels are carried as label_<key>, prefixed so a user-defined label cannot shadow a Trickster-assigned one — Compose’s own labels arrive as label_com.docker.compose.service and the like.

Upstream Permissions

Every provider is read-only; Trickster never mutates a discovery source. The least privilege each needs:

providerrequired access
kuberneteslist + watch on the resources your query kinds touch — see RBAC
consulan ACL token with service:read (and node:read) for the queried services
nomadan ACL token with read-job on the namespace
aws (ec2)ec2:DescribeInstances
aws (ecs)ecs:ListTasks, ecs:DescribeTasks (add ec2:DescribeInstances for EC2-launch-type tasks)
gcpcompute.instances.list — roles/compute.viewer includes it
azureReader on the subscription or resource group; power_state: true needs it at subscription scope
dockerread access to the Engine socket (the docker group, or a read-only socket proxy)
http_sdwhatever the endpoint itself requires; see connection options

Credentials are never required in config where the platform can supply them: aws uses the standard credential chain (IRSA, instance profile, environment), gcp uses Application Default Credentials (Workload Identity on GKE, the instance service account on GCE), and azure uses the managed identity of the VM or AKS pod. Prefer those over stored secrets wherever they are available — see each provider’s section. Secrets that are configured (aws.secret_key, azure.client_secret) redact themselves from config dumps and the management API.

Observing Discovered Members

Discovered members appear on the health status page alongside static pool members, under their generated backend names (<alb-name>-<member-name>), tagged with their provider, owning ALB, and discoverer (e.g., prometheus (prom-alb via in-cluster)).

Membership additions and removals are logged at info with the member name and origin (credentials embedded in origin URLs are masked), and the full membership is logged at debug on every change; all autodiscovery log events carry scope=discovery for easy filtering. Per-ALB member-count, member-change, snapshot-result, and refresh-staleness metrics — and per-discoverer refresh-error counters — are documented in metrics.md. When the ALB has a tracing_name configured, each membership reconcile cycle is traced as an alb.discovery.reconcile span via the standard tracing subsystem; the request hot path is never traced by discovery.

3 - Rule Backend

The Rule Backend is not really a true Backend; it only routes inbound requests to other configured Backends, based on how they match against the Rule’s cases.

A Rule is a single inspection operation performed against a single component of an inbound request, which determines the Next Backend to send the request to. The Next Backend can also be a rule Backend, so as to route requests through multiple Rules before arriving at a true Backend destination.

A request that asks to switch protocols, such as a WebSocket handshake, is tunneled by the backend the rule sends it to.

A rule can optionally rewrite multiple portions of the request before, during and after rule matching, by using request rewriters, which allows for powerful and limitless combinations of request rewriting and routing.

Rule Parts

A rule has several required parts, as follows:

Required Rule Parts

  • input_source - The part of the Request the Rule inspects
  • input_type - The source data type
  • operation - The operation taken on the input source
  • next_route - The Backend Name indicating the default next route for the Rule if no matching cases. Not required if redirect_url is provided.
  • redirect_url - The fully-qualified URL to issue as a 302 redirect to the client in the default case. Not required if next_route is provided.

Optional Rule Parts

  • input_key - case-sensitive lookup key; required when the source is header or URL param
  • input_encoding - the encoding of the input, which is decoded prior to performing the operation
  • input_index - when > -1, the source is split into parts and the input is extracted from parts[input_index]
  • input-delimiter - when input_index > -1, this delimiter is used to split the source into parts, and defaults to a standard space (’ ‘)
  • ingress_req_rewriter name - provides the name of a Request Rewriter to operate on the Request before rule execution.
  • egress_req_rewriter name - provides the name of a Request Rewriter to operate on the Request after rule execution.
  • nomatch_req_rewriter name - provides the name of a Request Rewriter to operate on the Request after rule execution if the request did not match any cases.
  • max_rule_executions - limits the number of rules a Request is passed through, and aborts with a 400 status code when exceeded. Default is 16. The first rule a request reaches sets its budget from this value; each later rule it passes through may lower the budget but not raise it.

input_source permitted values

source nameexample extracted value
urlhttps://example.com:8480/path1/path2?param1=value
url_no_paramshttps://example.com:8480/path1/path2
schemehttps
hostexample.com:8480
hostnameexample.com
port8480 (inferred from scheme when no port is provided)
path/path1/path2
paramsparam1=value
param(must be used with input_key as described below)
header(must be used with input_key as described below)
has_paramtrue or false: whether the query parameter named by input_key is present, even if empty
has_headertrue or false: whether the header named by input_key is present, even if empty

For a client request, url, scheme, host, hostname and port describe the URL the client requested: the Host header supplies the host and port, and the scheme is https when the request arrived over TLS.

input_type permitted values and operations

type namepermitted operations
string (default)prefix, suffix, contains, eq, md5, sha1, modulo, rmatch
numeq, le, ge, gt, lt, modulo
booleq

Rule Cases

Rule cases define the possible values are able to alter the Request and change the next route.

Case Parts

Required Case Parts

  • matches - A string list of values applicable to this case.
  • next_route - The Backend Name indicating the next route for the Rule when a request matches this Case. Not required if redirect_url is provided.
  • redirect_url - The fully-qualified URL to issue as a 302 redirect to the client when the Request matches this Case. Not required if next_route is provided.

Optional Case Parts

  • req_rewriter name - provides the name of a Request Rewriter to operate on the Request when this case is matched.

Example Rule - Route Request by Basic Auth Username

In this example config, requests routed through the /example path will be compared against the rules and routed to either the Reader cluster or the Writer cluster. Curling http://trickster-host/example/path would route to the reader or writer cluster based on a provided Authorization header.

rules:
  example-user-router:
    # default route is reader cluster
    next_route: example-reader-cluster

    input_source: header
    input_key: Authorization
    input_type: string
    input_encoding: base64 # Authorization: Basic <base64string>
    input_index: 1         # Field 1 is the <base64string>
    input_delimiter: ' '   # Authorization Header field is space-delimited
    operation: prefix      # Basic Auth credentials are formatted as user:pass,
                           # so we can check if it is prefixed with $user:
    cases:
      - matches: # writers
          - 'johndoe:'
          - 'janedoe:'
        next_route: example-writer-cluster

backends:
  example:
    provider: rule
    rule_name: example-user-router

  example-reader-cluster:
    provider: rpc
    origin_url: 'http://reader-cluster.example.com'

  example-writer-cluster:
    provider: rpc
    origin_url: 'http://writer-cluster.example.com'
    path_routing_disabled: true  # restrict routing to this backend via rule only, so
                                 # users cannot directly access via /example-writer-cluster/

Example Rule - Route Request by Path Regex

In this example config, requests routed through the /example path will be compared against the rules and routed to either the Reader cluster or the Writer cluster. Curling http://trickster-host/example/reader and http://trickster-host/example/writer would route to the reader or writer cluster by matching the path.

rules:
  example-user-router:
    # default route is reader cluster
    next_route: example-reader-cluster

    input_source: path
    input_type: string
    operation: rmatch      # perform regex match against the path to see if it matches 'writer
    operation_arg: '^.*\/writer.*$'
    cases:
      - matches: 
          - 'true' # rmatch returns true when the input matches the regex; update next_route
        next_route: example-writer-cluster

backends:
  example:
    provider: rule
    rule_name: example-user-router

  example-reader-cluster:
    provider: rpc
    origin_url: 'http://reader-cluster.example.com'

  example-writer-cluster:
    provider: rpc
    origin_url: 'http://writer-cluster.example.com'
    path_routing_disabled: true  # restrict routing to this backend via rule only, so
                                 # users cannot directly access via /example-writer-cluster/

Example Rule - Rewrite a Hostname From a Regex Capture

An rmatch rule makes its numeric and named capture groups available to the matched case and egress request rewriters. This example routes a request containing a three-character tenant label and prefixes the destination hostname with that tenant.

request_rewriters:
  tenant-host:
    instructions:
      - [ 'hostname', 'set', '${tenant}.writer.example.com' ]

rules:
  tenant-router:
    next_route: example-reader-cluster
    input_source: path
    input_type: string
    operation: rmatch
    operation_arg: '\{mylabel="(?P<tenant>[a-z0-9]{3})"\}'
    cases:
      - matches: [ 'true' ]
        req_rewriter_name: tenant-host
        next_route: example-writer-cluster

backends:
  example:
    provider: rule
    rule_name: tenant-router

  example-reader-cluster:
    provider: rpc
    origin_url: 'http://reader-cluster.example.com'

  example-writer-cluster:
    provider: rpc
    origin_url: 'http://writer-cluster.example.com'
    path_routing_disabled: true

For input_source: path, matching uses Go’s decoded URL.Path. For example, %7Bmylabel%3D%22abc%22%7D is matched as {mylabel="abc"} and ${tenant} expands to abc. ${0} represents the complete regex match, while ${1} represents the first capture group. See Request Rewriters for token lifetime and safety details.

When the destination hostname is a DNS SRV owner name, the destination backend can resolve it with origin_resolution.mode: srv, which takes the port from the SRV answer, so one backend can reach many origins.