This is the multi-page printable view of this section. Click here to print.
Routing & Load Balancing
1 - Application Load Balancer
Trickster 2.x provides an Application Load Balancer that is easy to configure and provides unique features to aid with Scaling, High Availability and other applications. The ALB supports several balancing Mechanisms:
| Mechanism | Config | Provides | Description |
|---|---|---|---|
| Round Robin | rr | Scaling | a basic, stateless round robin between healthy pool members |
| Power of Two Choices | p2c | Scaling | draws two healthy pool members at random and routes to the one with fewer requests in flight |
| Least Connections | lc | Scaling | routes to the healthy pool member with the fewest requests in flight |
| Least Time | lt | Speed | routes to the healthy pool member with the lowest response latency, scaled by its requests in flight |
| Highest Random Weight | hrw | Affinity | consistently routes each client, tenant or other key to the same healthy pool member |
| Time Series Merge | tsm | Federation | uses scatter/gather to collect and merge data from multiple replica tsdb sources |
| First Response | fr | Speed | fans a request out to multiple backends, and returns the first response received |
| First Good Response | fgr | Speed | fans a request out to multiple backends, and returns the first response received with a status code < 400 |
| Newest Last‑Modified | nlm | Freshness | fans a request out to multiple backends, and returns the response with the newest Last-Modified header |
| User Router | ur | Control | Inspects the credentials in the Request and routes it based on the Username |
| Connect Race | race | Speed | connects a tcp or tls stream connection to several pool members at once and relays over the first to connect |
| UDP Mirror | mirror | Replication | copies every datagram of a udp stream session to every healthy pool member, and answers from the first |
Integration with Backends
The ALB works by applying a Mechanism to select one or more Backends from a list of Healthy Pool Members, through which to route a request. Pool member names represent Backend Configs (known in Trickster 0.x and 1.x as Origin Configs) that can be pre-existing or newly defined.
All settings and functions configured for a Backend are applicable to traffic routed via an ALB - caching, rewriters, rules, tracing, TLS, etc.
In Trickster configuration files, each ALB itself is a Backend, just like the pool members to which it routes. This makes it possible to configure infinite loops (e.g., where ALB1 has ALB2 in its pool, and ALB2 has ALB1 in its pool). However, at startup Trickster will validate ALB configurations by following all ALBs’ possible paths, and exit with a startup failure if any infinite loops are detected.
In addition to (or instead of) a static pool list, an ALB’s pool membership can be discovered and kept current automatically at runtime — from Kubernetes, the AWS, Google Cloud and Azure APIs, the Docker Engine, the Consul or Nomad service registries, DNS records, an HTTP endpoint, or a watched member-list file. See ALB Autodiscovery.
Mechanisms Deep Dive
Each mechanism has its own use cases and pitfalls. Be sure to read about each one to understand how they might apply to your situation.
Basic Round Robin
A basic Round Robin rotates through a pool of healthy backends used to service client requests. Each time a client request is made to Trickster, the round robiner will identify the next healthy backend in the rotation schedule and route the request to it.
To keep each client on the member it was first sent to, add Sticky Sessions to the ALB. For affinity without session state, see Highest Random Weight.
Weighted Round Robin
Trickster supports Weighted Round Robin with a first-class integer weight on pool entries. A pool entry may be a plain backend name (weight 1) or a mapping with an explicit weight:
pool:
- node01 # weight 1
- name: node02
weight: 3 # receives 3 of every 4 requests
Apportionment is exact: over any totalWeight consecutive requests against a stable healthy pool, each member is selected exactly weight times. A heavier member’s turns are spread through the rotation rather than taken back to back: two members weighted 3 and 2 are served A B A A B, not A A A B B. Weights also carry through from autodiscovery sources that convey them (DNS SRV record weights, member-file weight fields); see ALB Autodiscovery.
Weights apply to every mechanism that selects a single member per request, though only Round Robin makes them an exact guarantee; see Weights and the Selection Mechanisms. Fan-out mechanisms (fr, fgr, nlm, tsm) dispatch to every healthy member regardless of weight.
More About Our Round Robin Mechanism
Trickster’s Round Robin Mechanism works by maintaining an atomic uint64 counter that increments each time a request is received by the ALB. With uniform weights, the ALB performs a modulo operation on the request’s counter value, with the denominator being the count of healthy backends in the pool; the resulting value, ranging from 0 to len(healthy_pool) - 1, indicates the assigned backend based on the counter and current pool size. With mixed weights, the modulo denominator becomes the pool’s total weight, and the result indexes a rotation schedule, computed once each time the set of healthy members changes, in which every member’s turns are evenly spaced. Selection remains lock- and allocation-free in both forms. The counter starts at a random value, so a fleet of Trickster replicas started together does not send its first requests to the same pool member.
Example Round Robin Configuration
backends:
# traditional Trickster backend configurations
node01:
provider: reverseproxycache # will cache responses to the default memory cache
path_routing_disabled: true # disables frontend request routing via /node01 path
origin_url: https://node01.example.com # make requests with TLS
tls: # this backend might use mutual TLS Auth
client_cert_path: ./cert.pem
client_key_path: ./cert.key
node02:
provider: reverseproxy # requests will be proxy-only with no caching
path_routing_disabled: true # disables frontend request routing via /node02 path
origin_url: http://node-02.example.com # make unsecured requests
request_headers: # this backend might use basic auth headers
Authoriziation: "basic jdoe:${NODE_02_AUTH_TOKEN}"
# Trickster 2.x ALB backend configuration, using above backends as pool members
node-alb:
provider: alb
alb:
mechanism: rr # round robin
pool:
- node01 # as named above; weight 1
- node02
# to weight the pool, use the mapping form on any entry:
# - name: node02
# weight: 2 # node02 would receive 2 of every 3 requests
Here is the visual representation of this configuration:

Power of Two Choices
The Power of Two Choices (p2c) mechanism draws two healthy pool members at random and routes the request to whichever has fewer requests in flight. It keeps load nearly as even as inspecting every member would, at a cost that does not grow with the size of the pool, and unlike Round Robin it reacts to a member that has become slow: requests pile up there, so it loses more of its draws. It is a good default for large pools and for requests whose cost varies widely.
backends:
api:
provider: alb
alb:
mechanism: p2c # or power_of_two_choices
pool: [ node01, node02, node03 ]
Least Connections
The Least Connections (lc) mechanism routes each request to the healthy pool member with the fewest requests in flight. It reads every member on every request, so it suits small pools; Power of Two Choices approximates it for large ones. While several members are tied, as all are when the pool is idle, they take turns.
backends:
api:
provider: alb
alb:
mechanism: lc # or least_connections
pool: [ node01, node02 ]
Least Time
The Least Time (lt) mechanism routes each request to the healthy pool member that has been answering fastest. Each member’s score is its latency average multiplied by one more than its requests in flight, and the lowest score wins, so a fast member is preferred until it is busy enough that a slower one would answer sooner.
Latency is the time from routing a request to the first byte of its response. The average rises at once when a member slows down and falls gradually, over lt.decay, as it recovers. A few details keep the ranking honest:
- A member with no requests yet, such as one just discovered, is scored as its fastest peer. It shares that peer’s requests until its own first response ranks it, so it is neither flooded as the apparent fastest nor left waiting for a turn that an idle pool would never give it.
- A failed request is recorded as a long latency rather than a short one, so a member that returns errors in a millisecond never looks fast. By default a response of
502,503or504is a failure, as is one that was never completed;lt.status_codessets the codes that count as a good answer instead, as bare codes, inclusive ranges, or both. A request the client abandoned counts neither way. - A member’s average fades while it is passed over, so one ranked last on an old measurement or a past failure is tried again rather than ignored for good.
- Averages survive a configuration reload and autodiscovery membership changes.
When the pool is idle and its members are equally loaded, every request goes to the fastest member. That is the mechanism working as intended; choose p2c or lc to spread idle traffic instead.
backends:
video:
provider: alb
alb:
mechanism: lt # or least_time
pool: [ edge01, edge02 ]
lt:
decay: 10s # default
status_codes: [ { start: 200, end: 499 } ] # optional; default is every code but 502, 503 and 504
Highest Random Weight
The Highest Random Weight (hrw) mechanism, also known as rendezvous hashing, routes every request that shares a key to the same healthy pool member. Use it to keep a client or tenant on one member’s warm cache. When a member leaves the pool, only the keys it owned move, each to a different remaining member; when it returns, exactly those keys move back. The mapping depends only on the key and the members’ names, so every Trickster replica agrees on it and a restart does not change it.
hrw.key selects what is hashed:
| Key | Follows |
|---|---|
client_ip (default) | the client’s IP address, after trusted proxy resolution. The port is never part of the key. IPv6 addresses are keyed on their leading hrw.ipv6_prefix bits (default 64), because privacy addressing changes the rest of a client’s address over time. |
host | the request’s host name, without its port and without regard to case |
header:<name> | the first value of the named request header |
cookie:<name> | the value of the named cookie |
query:<name> | the value of the named query string parameter, as written in the URL |
sni | the TLS server name the client offered; only for an ALB that serves a tls stream listener |
proxy_tlv:<type> | the value of a PROXY protocol version 2 TLV, such as proxy_tlv:0xEA for an AWS VPC endpoint ID; only for an ALB that serves a tcp or tls stream listener with proxy_protocol enabled. The type is one byte, written in decimal or 0x hex. |
user | the user name a session authenticated as; only for an ALB that serves a native protocol listener |
A request that lacks the configured key, such as one without the header, has no affinity to preserve and is routed to a member at random.
hrw balances keys, not requests. With many keys of similar volume the members’ loads even out, but a single very busy key is always served by one member.
hrw keeps no state, so a key moves when a member joins the pool and wins it, and a client with no stable key has no affinity at all. To keep a client on its member for as long as its session lasts, whatever the pool does, add Sticky Sessions, which work with hrw as with every mechanism that selects one member.
backends:
tenants:
provider: alb
alb:
mechanism: hrw # or highest_random_weight
pool: [ cache01, cache02, cache03 ]
hrw:
key: header:X-Tenant
Sticky Sessions
An ALB whose mechanism selects one member per request, rr, p2c, lc, lt or hrw, can keep each client on the member it was first sent to. The mechanism chooses a new client’s member; after that, the client’s requests go to the same member for as long as its session lasts and that member is available, even as members join or leave the pool. The same holds for the connections of a stream listener and the sessions of a native protocol listener; see Stream and native listeners.
backends:
app:
provider: alb
alb:
mechanism: p2c
pool: [ app1, app2, app3 ]
sticky:
ttl: 8h
secret_file: /etc/trickster/sticky.key
A session is kept in one of three ways, set by sticky.mode:
| Mode | How the member is remembered | Suits |
|---|---|---|
cookie (the default on http listeners) | The ALB issues a signed token naming the member in a cookie, which the browser sends back. The ALB keeps no state. | browsers |
header | The same token in a response header, X-Trickster-Session by default, which the client sends back in a request header of the same name. | API clients and SDKs that do not keep cookies |
table (the default on stream and native listeners) | The ALB remembers each client’s member by a key read from its requests, connections or sessions, such as its address or a header. | clients with a stable identifier of their own, upstreams that hand out their own session ids, and every listener that is not http |
The settings, with their defaults:
sticky:
mode: "" # cookie, header or table; unset is cookie on http listeners, table on the others
ttl: 1h # a session's lifetime from its first request; 0 is no limit
idle: 0 # ends a session left unused, as Session lifetimes describes; 0 never does
on_unavailable: repick # repick or reject; see below
secret: "" # the key tokens are signed with; see below
secret_file: "" # a file holding it, in place of secret
cookie: # cookie mode only
name: trickster_sticky
path: /
domain: "" # unset keeps the cookie to the host that set it
secure: auto # auto marks it Secure when the request arrived over TLS; or true, false
http_only: true
same_site: lax # lax, strict or none, which requires secure: true
lifetime: permanent # or session, which never sets Max-Age; see below
mark_private: false
header: # header mode only
name: X-Trickster-Session
table: # table mode only
key: client_ip
learn: request # or response; see below
ipv6_prefix: 64
max_entries: 100000
ttl and idle are 0 or at least 1s. With mode set, a cookie, header or table block that the mode never reads is a configuration error. With mode unset, a cookie block (for http listeners) and a table block (for the others) may both be set, and a header block is refused.
Other settings that are refused: sticky on a mechanism other than rr, p2c, lc, lt and hrw; cookie or header mode on an ALB that serves no http listener; both secret and secret_file; a cookie.path that does not begin with /; a cookie name beginning __Secure- or __Host- without secure: true, or a __Host- one with a domain or a path other than /; same_site: none without secure: true; table.learn: response with a key other than header:<name> or cookie:<name>; and an ipv6_prefix above 128.
Tokens
In cookie and header mode the token carries the member, when the session began and when the token was issued, signed with HMAC-SHA256 under a key derived from sticky.secret. A token is bound to the ALB that issued it, so another ALB does not honor it. A client cannot choose its own member, and a token that is altered, signed with another key, issued by another ALB or past its lifetime is treated as no token: the request starts a new session. It is never an error.
A token is issued with the response to a client’s first request, including the 101 Switching Protocols that opens a WebSocket, when its session moves to another member, and, with idle set, once more than half of idle has passed since the token was issued, so that an active session does not lapse. A request sent to the member its token names is otherwise answered with no token, so a steady client sees none after its first.
A cookie carries a Max-Age that ends with its token when ttl is set. With ttl: 0, or with lifetime: session, it is a browser-session cookie, and the token itself still enforces ttl and idle. secure: auto marks the cookie Secure when the request reached Trickster over TLS; behind a load balancer that terminates TLS and forwards plain HTTP to Trickster, set secure: true. mark_private: true adds Cache-Control: private to a response that sets the cookie, unless that response is already private or no-store; see Sticky Sessions and Caching.
Two ALBs that set the same cookie (by name, domain and path) for a host they both serve would each replace the other’s token in a browser, so Trickster refuses to start with such a configuration. A browser sends a host’s cookies to every port and scheme, so this holds across listeners. To resolve it, give one of them its own cookie.name.
ALBs whose served hosts do not overlap may share a cookie name, as long as the cookie sets no domain. A cookie with no domain goes back only to the host that set it. The hosts an ALB serves are:
- every host, when it is the default backend, or when one of its paths registers on a listener and it has
any_host_routingor path routing on (the default). Path routing answers/<alb name>/...on every host, whateverhostslists. A path registers on a listener unless it is markeddispatch_onlyor names a handler the ALB lacks. The ALB’s default paths count, unlesspath_defaults_disabledis set; - otherwise, the
hostsits listener-registered paths answer for, joined by the hosts of its dispatchers, since a dispatched request keeps the host it arrived on. Its dispatchers are the ALBs that name it in their pool oruser_router, and the rules that route to it. An ALB withpath_routing_disabled: trueand nohosts, or whose paths are alldispatch_only, serves only its dispatchers’ hosts; - no host, when nothing but a mirror reaches it. A mirror discards its target’s response, so no cookie from it reaches a browser, and such an ALB never conflicts.
The secret
Every Trickster replica behind one address, and every restart, must sign with the same key for a session to survive moving between them. Set secret to at least 32 bytes, or secret_file to a file that holds them, such as a mounted Kubernetes Secret; trailing line breaks in the file are ignored. Tokens made with one key are never honored with another. A secret that still contains ${, an environment variable reference that was not expanded, is refused rather than used as a publicly known key.
With neither set, the ALB signs with a random key that lasts as long as the process, and Trickster logs a warning each time it loads such a configuration, for every ALB that issues tokens on an http listener. That suits a single instance that can afford to start every session over when it restarts. A configuration reload keeps the process’s key, so a reload alone never ends a session. See Sticky Sessions Across Replicas.
When the pinned member is unavailable
With on_unavailable: repick, the default, a request whose member is unavailable goes to the member the mechanism chooses, and its session moves there. With reject, the ALB answers 503 Service Unavailable instead, for applications that cannot survive a move, and the client keeps its token so that it returns to its member once that member is available again. A member that has left the pool has no session to return to, so its clients start new sessions in either case.
A member that is draining keeps its sessions while it is available.
On an http listener a session is pinned by its first response, whatever its status, so a session that began on a member answering with errors stays there until that member is marked unavailable, by its health check or by the pool’s floor.
Table mode
In table mode the ALB keeps a table of each client’s member, by table.key, which takes the same values as hrw.key. A request without the key is balanced by the mechanism and not remembered. An entry lasts ttl from when it was stored and ends early when unused for idle. A full table drops the least recently used of a sample of its entries to make room.
With table.learn: response and a header:<name> or cookie:<name> key, the ALB also remembers the value a member hands the client in its response, in the header of that name or in the Set-Cookie that sets that cookie. That keeps a session on the member that created it from the client’s very next request, which is what an upstream that issues its own session ids needs. For example, for an MCP server whose replicas each know only the sessions they created:
sticky:
mode: table
table:
key: header:Mcp-Session-Id
learn: response
A response that sets a new value pins that value to the member too; the entry for the old value lasts until it expires.
A table is kept in each Trickster instance’s memory. It survives a configuration reload that leaves the ALB’s mode, ttl, idle and table settings unchanged, but not a restart, and it is not shared between replicas: use cookie or header mode where sessions must outlive either, or see Sticky Sessions Across Replicas. With key: client_ip, every client behind one NAT address shares a session.
Session lifetimes
ttl and idle bound a session in every mode, but they count from different moments for a token and for a table entry:
cookie and header mode | table mode | |
|---|---|---|
ttl counts from | the session’s first request. A refreshed token keeps that start; a session that moves to another member starts over. | when the entry was stored. An entry stored again, because its session moved, starts over. |
idle counts from | when the token was last issued. A request more than half of idle after that is answered with a fresh token, so an unused session ends between half of idle and idle after its last request. | the entry’s last use, to within a second. |
| after a reload | the new ttl and idle apply at once to every token already issued, since a token records only when its session began and when it was issued | a change to mode, ttl, idle or table starts the ALB’s table empty |
Renaming the ALB or a member ends the sessions they had, since tokens and table entries refer to both by name. Moving from one mode to another starts every session over, as does a change of the signing key.
Stream and native listeners
A tcp, tls or udp listener, or a native protocol listener such as mysql, carries no token, so an ALB keeps the sessions of its connections there in a table. mode is best left unset: an ALB that serves both http and stream listeners then issues cookies to its requests and keeps its connections in a table. cookie and header mode are refused on a listener that is not http. With mode: table, requests and connections share one table, so a client whose requests and connections share a key keeps one member for both; table.key must then be readable on every listener the ALB serves, which in practice means client_ip.
table.key is read from each connection or session, as hrw.key is:
| Listener | table.key |
|---|---|
tcp | client_ip; proxy_tlv:<type> with proxy_protocol |
tls | client_ip, sni; proxy_tlv:<type> with proxy_protocol |
udp | client_ip |
native (mysql) | client_ip, user |
client_ip is the address a PROXY protocol header names when the listener trusts one. user is the name a native session authenticated as. A connection or session without its key is balanced by the mechanism and not remembered.
A connection is pinned to its member once it reaches it: a tcp or tls connection when it connects, a udp session when its member first answers, or when a one-way member’s session ends with no port-unreachable, and a native session when it is handed to its member. A member that cannot be reached is never pinned. A udp session already stays with its member for its whole life; what a pin adds is that the client’s next session, from any port, lands on the same member. A table entry is read only when a connection or session begins, so idle counts from the client’s last new connection or session, however long that one lasts.
With on_unavailable: reject, a connection whose pinned member is unavailable is refused, and so is one whose pinned member cannot be connected, however many connect_retries the ALB has: the retries would move the session. With repick, a failed connect is offered to other members as connect_retries allows, and its session moves to the one that connects. A native session that is refused fails to route, as it would with no member available.
Sticky Sessions and Nested ALBs
When a sticky ALB’s pool has ALBs in it, such as one per region or per service, its token or table entry keeps the whole path: the inner ALB the client was sent to and that ALB’s member. The inner ALBs need no sticky block of their own. An inner ALB whose mechanism fans requests out, or a User Router, is itself the session’s member, and chooses among its own members as it would with no session. When only the inner member becomes unavailable, the session stays with its inner ALB and moves within it. On a stream listener, a connection retried after its inner member failed to connect is likewise offered the inner ALB’s other members first.
When the inner ALB has no member left that can take the session (a draining member the session is pinned to still can), the session moves with on_unavailable: repick to the first other member the mechanism offers that can take it, and is refused with reject, keeping its token or table entry for when its inner ALB recovers. A new session that the mechanism sends to an inner ALB with no member is answered 502 Bad Gateway, or refused on a stream listener, as it would be with no sessions. Set propagate_health: true on the inner ALBs (see ALBs as Pool Members) to keep new sessions away from one while it has no healthy member.
The session covers one level of nesting: an ALB further in selects its member, or keeps sessions of its own if it has a sticky block, as it would with no outer session. An inner ALB directly in the sticky ALB’s pool follows the outer session, and uses a sticky block of its own only for requests that reach it without going through the outer ALB.
On a stream listener only the ALB that the listener maps to keeps sessions: an inner ALB’s own sticky block applies to its http listeners, and to stream listeners mapped to it directly, never to connections that reach it through another ALB.
Sticky Sessions and Caching
In front of cacheable content, prefer table mode. It adds nothing to responses and sends members nothing the client did not, so every cache between the client and the origin sees exactly what it would see without sessions. Weigh its two limits first:
- A session is only as stable as its key. With
key: client_ip, every client behind one NAT address or forward proxy shares one session, and so one member, and a client whose address changes, such as a phone moving between networks, starts a new session. Aheader:<name>orcookie:<name>key needs a value that the client sends, or that the application hands it andlearn: responsepicks up. A request without the key is balanced afresh every time. - A table belongs to one Trickster instance. A restart empties it, and another replica does not know its sessions. With several replicas, keep each client on one replica, or pair the table with
hrwon the same key; see Sticky Sessions Across Replicas.
Where those limits rule table mode out, use cookie mode with mark_private: true. header mode suits API clients with no shared cache in front of Trickster, since its token rides in responses and it has no mark_private.
A token interacts with caches in three ways:
- A member’s cache never stores it. The ALB adds a token to a response after its member has answered, so a response served from a member’s cache is given a token when its session needs one, like any other.
- The token travels upstream. A member receives the request with its token still in the
Cookieheader, or in theheader.nameheader. An origin that answersVary: Cookiemakes a member cache one copy per distinctCookieheader, and a cookie token gives every session a distinct one, so such content is cached once per session. - A cache in front of Trickster sees the token. HTTP lets a shared cache, such as a CDN, store a response that sets a cookie and replay it, cookie included, to other clients, which joins them to that session’s member. A token in a response header is replayed the same way to clients that echo it back. That costs balance, not security, since a pin grants no access. In cookie mode,
mark_private: truestops it by making each response that sets the cookieprivate, so a shared cache does not store that response. Only a session’s first response, a move and an idle refresh carry a token, so most responses stay shareable.
Sticky Sessions Across Replicas
In cookie and header mode the client carries its session, so every Trickster replica behind one address honors it, and so does a restarted one, as long as each replica:
- signs with the same key, from
secretorsecret_file(see The secret); - configures the ALB under the same name, since a token is bound to it;
- names the pool members alike, since a token refers to its member by name. A discovered member is named after its ALB and the member’s discovered name, or its address when it has none, so replicas that discover the same members agree.
Changing the key starts every session over once, since a token signed with the old key is no longer honored. The Kubernetes controller keys the ALBs it generates from a Secret; see The session key.
In table mode each replica keeps its own table, and a client that reaches a replica that has not seen it before is balanced there as a new session. To keep such a client on its member anyway, either:
- keep each client on one replica, with source-address affinity in whatever balances the replicas; or
- use
mechanism: hrwwithhrw.keyset to the same key astable.key. Every replica then sends a new session to the same member for as long as their pools agree, and a replica’s table keeps the sessions it has seen in place when the pool changes. A replica that has not seen a session chooses its member afresh after such a change.
table.learn: response learns a value only on the replica whose member set it, and the request that began that session had no value for hrw to hash, so it needs each client kept on one replica.
Sticky Session Metrics
trickster_alb_sticky_total{alb_name, result} counts requests, connections and native sessions by how their session fared: hit (sent to its member), miss (no token or table entry), expired, invalid, repick (its member was unavailable and it moved) and rejected. In table mode an expired entry counts as a miss. A request, connection or native session is counted once it reaches its member, or once it is refused; one that reaches no member, which the ALB answers 502 Bad Gateway or closes, is not counted. trickster_alb_sticky_entries{alb_name} is the number of entries in an ALB’s table.
WebSockets and Other Protocol Upgrades
A request that asks to switch protocols, such as a WebSocket handshake, is tunneled by the pool member it is sent to when the ALB’s mechanism sends each request to one member (rr, p2c, lc, lt, hrw) or is the User Router. The mechanisms that fan a request out (fr, fgr, nlm, tsm) have no one member to tunnel to, so they ignore the upgrade and serve the request as an ordinary one, as HTTP permits.
Load Balancing Stream Listeners
The mechanisms that select one member, rr, p2c, lc, lt and hrw, also balance the connections of a tcp or tls stream listener and the sessions of a udp one. They are the same mechanisms with the same weights; only what they measure differs. The mechanisms that fan a request out (fr, fgr, nlm, tsm) and the User Router need an HTTP request, and are refused on a stream listener. Two more mechanisms, race and mirror, serve only stream listeners.
| http | tcp and tls | udp | |
|---|---|---|---|
| unit of work | a request | a connection | a session: one client address and port |
in flight (p2c, lc, lt) | requests being served | connections open | sessions open |
latency (lt.signal) | first_write: the first byte sent to the client | connect (default): the time to connect to the member; or first_byte: the member’s first byte | first_reply: the member’s first datagram |
hrw.key | client_ip, host, header:, cookie:, query: | client_ip; sni on a tls listener; proxy_tlv:<type> with proxy_protocol | client_ip |
A connection or session stays on the member it was given until it ends, whatever the mechanism. client_ip is the address a PROXY protocol header names when the listener trusts one. To keep a client’s later connections on one member too, give the ALB a sticky block.
Two settings apply only to an ALB that selects one member on a stream listener, under alb.stream:
backends:
pg:
provider: alb
listener_names: [ postgres ]
alb:
mechanism: p2c
pool: [ pg1, pg2, pg3 ]
stream:
connect_retries: 1 # default 0
passive_health: # off unless present
failures: 3 # default
eject: 30s # default
max_ejected_percent: 50 # default
connect_retriesis how many other members atcportlsconnection is offered when it cannot connect to the one it was given. All attempts share the listener’sstream.connect_timeout. The default, 0, refuses the connection, which is what keeps a weighted split exact: a member under the reserved.invaliddomain exists to refuse its share, and is never retried past.udphas no connect to fail, so it is not retried.passive_healthtakes a member out of the pool afterfailuresconsecutive failed connects, foreject, without waiting for a health check. Only failures to reach the member count, never anything it sent. At mostmax_ejected_percentof the pool is out at once, and the last live member is never ejected. Whenejectends the member returns, unless its health check has it down. Onudp, where nothing connects, a member that answers a datagram with a port-unreachable is what counts as a failure.
Health checks work as they do for HTTP pools: a tcp:// member with a healthcheck.interval is probed by opening a connection to it. A udp:// member has no generic probe; rely on autodiscovery readiness or on passive_health.
Connect Race and UDP Mirror
Two mechanisms exist only for stream listeners, because they commit one flow to several members at once. Both use every healthy member that has an address to dial, ignore weight, and take neither connect_retries nor passive_health. An ALB that uses one must be mapped to the stream listener directly; it cannot be a member of another ALB’s pool.
The Connect Race (race, or connect_race) mechanism serves tcp and tls listeners. Each client connection is connected to several members at once, within the listener’s stream.connect_timeout; the first member to connect carries the connection, and the other attempts are closed. A member that is down or slow to accept costs the client nothing. stream.race_width is how many members are raced, from 2 to 8; the default is every member, up to 4. In a pool wider than the race, each connection starts one member further along, so the connects are shared across the pool. A race opens and discards connections on the members that lose, so use it where a connect is cheap for the member.
The UDP Mirror (mirror, or udp_mirror) mechanism serves udp listeners. Every datagram a client sends is copied to every healthy member, which suits one-way protocols such as statsd, syslog and NetFlow. The first healthy member in pool order answers the session: only its replies are relayed to the client, and the other members’ replies are discarded. A mirror pool has at most 8 members.
backends:
statsd:
provider: alb
listener_names: [ statsd ]
alb:
mechanism: mirror
pool: [ statsd1, statsd2 ]
db:
provider: alb
listener_names: [ postgres ]
alb:
mechanism: race
pool: [ pg1, pg2, pg3 ]
stream:
race_width: 2
Load Balancing Native Protocol Sessions
A listener that speaks a backend’s own wire protocol, such as a mysql listener, can map to an ALB that uses rr, p2c, lc or hrw. Each client session is committed to one pool member once it authenticates, and stays there until it ends; a session is the unit of work, so p2c and lc compare members by their open sessions. hrw.key is client_ip or user. lt is not available, since a session reports no latency to rank members by. Every pool member must be a backend of the listener’s own provider, listed directly, and autodiscovery is not supported. The ALB authenticates the listener’s clients, so it carries the authenticator_name that a User Router on the same listener would.
backends:
replicas:
provider: alb
listener_names: [ mysql ]
authenticator_name: mysql-clients
alb:
mechanism: lc
pool: [ replica1, replica2 ]
With a sticky block, a client’s later sessions keep to the member its first one was given, by client_ip or, with table.key: user, by the name it authenticated as.
Weights and the Selection Mechanisms
Every mechanism that selects one member per request honors the pool weight, but what a weight promises differs:
| Mechanism | A weight is | Guarantee |
|---|---|---|
| rr | a share of requests | exact: weight of every totalWeight consecutive requests |
| p2c | a capacity | proportional on average: heavier members are drawn more often and compared by requests in flight per unit of weight |
| hrw | a share of the keys | proportional on average over many keys; changing a weight moves as few keys as possible |
| lc | a capacity | members are kept at equal requests in flight per unit of weight |
| lt | a bias | the score is divided by the weight, which shifts load under contention; an idle pool still sends every request to its best-scoring member |
If you need a guaranteed split, use rr.
Time Series Merge
The Time Series Merge mechanism supports both High Availability and federation. Each physical backend represents one logical data shard. Set the backend-level replica_group option to the same value on physical backends that are HA replicas of that shard. TSM first coalesces those replicas, using configured pool order to resolve overlapping points and later replicas to fill gaps, and then reduces the distinct logical shards.
When a TSM pool member is itself an ALB (for example, a round-robin ALB over
Prometheus backends), set replica_group on that immediate nested ALB. Trickster
uses the wrapper as the replica-group boundary while delegating TSM planning and
finalization to its terminal Prometheus provider. Other ALBs and non-TSM
providers cannot set an explicit replica group.
When replica_group is omitted, it defaults to the backend name, so existing configurations continue to treat every backend as a distinct shard. Explicitly set it for HA pools that use non-idempotent aggregations such as sum, count, or avg; otherwise replicas will be counted as separate data.
Replica grouping is backend-global, not ALB-specific. A backend cannot be a replica in one ALB and a disjoint shard in another; define a second backend entry if both views are required. Partially overlapping datasets are not representable: one backend belongs to one logical shard for all TSM queries. Injected labels remain useful output and routing metadata, but they do not establish replica provenance.
If replicas disagree at the same logical point, the first configured member wins deterministically and Trickster records a conflict metric and warning log. A failed replica does not make a response partial when another replica covers its group. If an entire logical group is unavailable, the response is marked partial and includes a warning.
For request paths that are not mergeable by the configured time series provider, TSM does not fan the request out. Those requests are dispatched directly to the first live pool target. The same first-live-target fallback is used when a request cannot be prepared for the merge path.
By default, TSM merges the answers its members give to a query. A TSM ALB can instead read its members’ raw series and evaluate each query once, the way Thanos does; see Global Evaluation.
Merge Strategy
Within each configured replica group, TSM deduplicates values when merging series with identical labels — for each timestamp, only one replica value is kept. Across different groups it uses the query’s merge strategy.
For Federation use cases where backends hold different, non-overlapping data, Trickster automatically selects a merge strategy per query by parsing it with the upstream Prometheus PromQL parser and inspecting the outermost aggregation operator. No configuration is required. Because selection uses the parsed expression, redundant parentheses, comments, whitespace, keyword case, and by/without placement do not change the selected strategy. This is particularly important for PromQL aggregation queries like sum() or avg(), which strip labels from results and cause series from different backends to appear identical.
| Outer Operator | Trickster Merge Behavior |
|---|---|
sum | Sum of values per unique label set + timestamp |
count | Sum of values per unique label set + timestamp |
count_values | Sum of values per unique label set + timestamp |
min | Minimum value per unique label set + timestamp |
max | Maximum value per unique label set + timestamp |
group | Deduplicate per unique label set + timestamp |
avg | Dual queries (avg→sum and avg→count); weighted arithmetic mean per unique label set + timestamp |
topk, bottomk | Query the inner expression across backends, merge it, then apply final top/bottom-k selection per timestamp and aggregation group |
stddev, stdvar | Pool shard-local count, mean, and variance states, then finalize the global population variance or standard deviation |
quantile | Query the inner expression, merge all float samples globally, then calculate the exact quantile per timestamp and aggregation group |
limit_ratio | Apply Prometheus-compatible label-hash sampling, globally finalizing a supported inner aggregation when necessary |
limitk | Query the inner expression, merge it globally, then retain the first k samples in stable TSM series order per timestamp and aggregation group |
count or group over an inner count or group of a selector | Query the inner aggregation, union its label sets across backends, then count or group each label set once per timestamp and aggregation group |
sum, count, or count_values followed by or vector(0) | Sum of values per unique label set + timestamp |
| (none) | Deduplicate (default) |
For avg queries, Trickster issues two concurrent sub-queries per backend shard — one rewriting the outer avg to sum and another to count — then computes a true weighted arithmetic mean (sum_total / count_total) per series per timestamp. This avoids the skew introduced by a naïve avg-of-averages when backends have different data cardinalities.
When an outer aggregation’s input contains a nested aggregation (other than the count and group forms below), binary expression, or function that needs globally complete input, Trickster retains the aggregation’s established merge strategy and adds a warning. This fail-open behavior preserves correct results when the input series are colocated while noting that results can be inaccurate when matching series are split across shards.
For an outer count or group whose input is a count or group of a vector selector, such as count(count by (pod) (up)), the result depends only on which label sets the inner aggregation produces. Trickster sends the inner aggregation to each backend, unions the label sets the backends return, and applies the outer operator once per timestamp and aggregation group. A label set returned by several backends counts once, so replicated or partially overlapping backends don’t inflate the count. The selector may use label matchers, offset, and @, and the aggregation may be wrapped in sort(), sort_desc(), or a literal scalar operation such as > 1. A label set counts at a timestamp when any backend returned it there. Inner inputs that apply functions, filters, or binary operations can change which label sets a backend returns, so they keep the warning described above. When a backend fails, the result covers the backends that answered and is marked as a partial hit.
For sum, count, or count_values followed by or vector(0), Trickster sends the complete expression to each backend and sums the results, because a backend without matching series contributes only the explicit zero. This requires an aggregation input that each backend can evaluate independently: no nested aggregation, binary expression, or function that needs globally complete input. When the or uses on(...) or ignoring(...) label matching, this applies only to sum or count without by or without grouping; other forms use the warning fallback described below.
For topk and bottomk, Trickster sends the inner expression to each backend, merges those inner results using the inner expression’s merge strategy, then applies the final rank-and-trim step per timestamp and aggregation group. This prevents each backend’s local topk/bottomk result from being weighted equally during the merge. If the inner expression is avg, Trickster still uses the weighted sum/count rewrite before applying the final rank. This also applies when the rank aggregation is wrapped in sort() or sort_desc().
For stddev and stdvar, Trickster requests the shard-local count, mean, and population variance and pools those states before finalizing the requested global value. Native histograms are excluded from this float-only calculation. Supported already-aggregated inner expressions such as count, min, max, and group are merged globally before the outer variance aggregation.
For quantile, Trickster sends the inner expression to every backend, globally merges supported inner aggregations, ignores native-histogram samples, and calculates Prometheus’s exact sort-and-interpolate value independently for each timestamp and group. Exact quantiles require every relevant float sample, so their fanout responses can be substantially larger than shard-local quantiles and remain subject to the configured response-capture limits. A capture-limit failure is returned rather than silently substituting an approximate result.
For limit_ratio, selection uses the same complete-label-set hash threshold as Prometheus. Supported inner aggregations are merged before the ratio is applied; shard-local expressions can be sampled by each backend because the hash decision for a given label set is independent of shard placement.
For limitk, Trickster reproduces the current Prometheus evaluator’s first-visited algorithm over TSM’s stable merged-series order: lexicographic JSON serialization of the complete label set, followed by the series name. Selection is independent for every timestamp and aggregation group, retains complete labels and both float and native-histogram samples, and does not rank candidates by value or label hash.
Prometheus does not define a canonical storage visitation order for limitk. A separate Prometheus deployment whose storage returns the same series in a different order may therefore select different labels. Trickster guarantees the requested cardinality, grouping, and repeatability for the same merged input, and fanout completion order does not affect the result. limitk remains an experimental PromQL operator, so this compatibility contract may change with Prometheus.
For an unsupported inner expression, Trickster retains the established per-shard fallback and injects a warnings entry in the Prometheus response body to alert the caller that results may be inaccurate.
The same deduplicating fallback and warning apply when an aggregation is not the query’s outermost operation (for example, histogram_quantile(0.9, sum by (le) (rate(x_bucket[5m])))), when an aggregation is combined with a binary expression other than the supported or vector(0) form, and when the query cannot be parsed as PromQL, such as a query that uses another dialect’s extensions.
When a non-dedup strategy is in effect, or Trickster sends backends an inner expression to finalize after merging, and backends have injected labels configured, those labels are automatically stripped before merging. This ensures series from different backends hash identically for aggregation, and the injected labels do not appear in the response.
Native Histograms
Native histogram samples are preserved through the merge rather than being numerically aggregated. When a timestamp has a histogram on one backend and a float sample on another (or histograms on both), the histogram value is kept as-is — numeric aggregators like sum only apply across float samples. This prevents mixed-type series from being corrupted into garbage values when backends return a mix of float and histogram samples at the same timestamp.
Max Query Range Limitation
Trickster ALB supports enforcing a max_query_range duration on ALB backends. For details on how to configure and use query range limits, see the Query Range Limits documentation.
Step Alignment
Members’ responses combine at identical timestamps, so members must answer on one grid: a member that drops the partial bucket at the start while another returns it whole would merge into a series whose first points are summed over a subset of members. A tsm ALB therefore applies one step alignment mode to every member. It uses its own step_alignment when set; otherwise it uses its first configured member’s mode, whether or not that member is healthy, so a leader’s outage doesn’t move every open dashboard’s grid. A query’s trickster-step-align directive wins when every member supports the mode it names.
A static pool member that can’t apply the chosen mode fails the configuration. When autodiscovered members can’t, the pool uses truncate if every member supports it, or otherwise leaves each member to its own mode, and adds a warning to the merged response.
step_alignment on an ALB using any other mechanism applies that mode to every member too, so the chart’s shape doesn’t depend on which member answered. Without it, those mechanisms leave members to their own modes, and Trickster logs a warning at startup for each ALB whose members differ.
Members With a Series Limit
Some managed services, such as Amazon CloudWatch PromQL, cut a query result short at a series limit and flag it with a warning. Each member applies its own limit, so a member’s truncated result is proxied and never cached, while the other members’ results are cached as usual. The merged response then covers only the series each member returned, and it carries the truncation warning so the client can tell.
Global Evaluation
Merging members’ answers is exact only for queries whose aggregation can be split across shards, as described in Merge Strategy. A query such as sum(rate(x[5m])) over replicas that differ by a label, or histogram_quantile(...) over buckets split across members, needs every raw sample in one place. With tsm.evaluation: global, a TSM ALB answers each instant and range query the way Thanos Querier does: it reads the raw series the query selects from every member, deduplicates replicas, and evaluates the whole expression once in an embedded Prometheus engine.
backends:
mimir-east:
provider: prometheus
origin_url: http://mimir-east:8080/prometheus
prometheus:
flavor: mimir
mimir-west:
provider: prometheus
origin_url: http://mimir-west:8080/prometheus
prometheus:
flavor: mimir
mimir-global:
provider: alb
cache_name: default
alb:
mechanism: tsm
tsm:
evaluation: global # merge (the default) or global
replica_labels: [replica] # labels that tell copies of one series apart; no default
deduplication: penalty # penalty (the default) or chain
partial_response: true # the default
max_samples: 50000000 # the default
query_timeout: 2m # the default
max_concurrent_queries: 20 # the default
max_fetched_series: 0 # the default, unlimited
max_fetched_samples: 0 # the default, unlimited
raw_cache_window: 31d # the default
pool:
- mimir-east
- mimir-west
The evaluation is set per ALB, and covers every query the ALB receives; other TSM ALBs keep merging. Requests for other paths, such as labels and series, are merged as usual. The settings below evaluation are valid only with evaluation: global.
Members. Every pool member must be a prometheus backend with a series source: flavor: mimir reads Prometheus remote read, flavor: thanos reads Thanos StoreAPI at the member’s series_endpoint, and any other Prometheus backend can set series_api. Configuration fails for any other member, including a nested ALB. With autodiscovery, the template_backend must be such a backend. A template’s series_endpoint, including the one StoreAPI requires, is used unchanged by every discovered member, so every member reads that one store and Trickster logs a warning at startup; remote read without a series_endpoint instead reads each member’s own origin. A discovery query.scheme of tcp or udp gives no such origin and fails configuration, and a discovered member whose series endpoint still can’t be formed is logged and left out of the pool. Every live member is read, whatever its replica_group. Each read reuses the member’s TLS settings, and carries the client’s request headers and the member path’s request_headers.
Deduplication. Series whose labels match once replica_labels are removed are replicas of one series, and the result keeps one series per such label set, without the replica labels. deduplication: penalty follows Thanos: it reads one replica until that replica falls silent for twice its sampling interval, then switches to another. When the query reads the series through rate, increase, irate or resets, a switch to a replica whose counter is behind is adjusted so that it doesn’t read as a counter reset. When replicas sample at the same instant, the earliest-configured member’s replica is read first. deduplication: chain instead keeps every distinct timestamp across replicas, and the earliest-configured member’s value where they share one. Series with identical labels from different members are always combined like chain. Leaving replica_labels empty still combines identical series. A native histogram’s counter reset hint describes its change from the previous sample of its own source, so where deduplication moves to another replica or copy, or passes over samples, the hint is cleared and the engine finds any reset from the counts, as Prometheus does where it merges series.
Partial responses. When a member is unavailable or fails to answer, by default the result is evaluated over the members that answered, and carries a warning naming the member, as Thanos does with partial responses enabled. Why the member failed is logged rather than returned, since a transport error can name internal hosts. Set partial_response: false to fail such a query with a 503 instead, as soon as a member fails, stopping the reads of the others. A request’s partial_response parameter overrides the ALB’s setting. A failed evaluation is returned as a Prometheus API error, never replaced by a merged answer.
Limits. max_samples bounds the samples the engine holds at once, like Prometheus’s --query.max-samples. query_timeout bounds an evaluation from its arrival, including any wait for one of the ALB’s max_concurrent_queries slots; a request’s timeout parameter can shorten it. max_fetched_series and max_fetched_samples bound the raw series and samples one evaluation reads from all its members before deduplication: the samples each selector reads within its range, whether from a member or the raw series cache, and the series with any such sample. What the raw series cache serves for a selector is counted before any member is read for it, and what members send as their answers arrive, so a read stops as soon as its samples and the cache’s pass the allowance. As a member’s series are mostly those the cache serves too, a read counts series only past as many as the cache served, and each selector’s exact count is settled once its answer is assembled. A series counts once for each selector and member, however many frames, messages, or uncached time ranges carry it. With max_fetched_series set, series are told apart by a hash of their labels as they arrive, so a copy of a series that a member sends apart from the rest of it counts once too; without it, such a copy may count again. A read stops as soon as the evaluation’s allowance runs out, and a read several evaluations share counts against each of them, stopping only when none can afford more. Each member’s raw response, and each StoreAPI message or remote-read frame in it, is bounded by the ALB’s max_capture_bytes. Only the chunks and samples within a selector’s range are kept from a response, and what is kept from one response, with the buckets its native histogram chunks lay out as they are read, may take at most four times max_capture_bytes in memory once decoded, so a response can’t expand far past its size even when the fetch limits are unlimited. Deduplication reads every copy and replica of a series at once, so the buckets their chunks lay out count together. Responses of float chunks stay within that bound; native histograms expand more, integer ones sent as samples to around eight times their size, so a response of them can reach it before max_capture_bytes. A histogram chunk whose bucket layout declares more buckets than its data holds is refused as malformed, as is one whose samples couldn’t be read: spans offset back after the first, in a schema finer than the engine reads, or custom-bucket spans offset back, on the negative side, or reaching past their bounds. A histogram sent as a sample is refused for the same layouts, and for spans laying out more or fewer buckets than it holds. max_capture_bytes also bounds the result an evaluation encodes as its response, whether or not the result can be cached: encoding stops as soon as the result passes it, and the query fails whole with execution, never returning part of the result. A failed evaluation’s error stays within it too, or within 4 KiB when that is more, leaving out warnings and shortening its message as needed. A result takes some 15 to 35 bytes of JSON for each sample it returns, where members’ chunk-encoded reads often take one or two, so an ALB that returns raw series over wide ranges needs a max_capture_bytes sized for its results as well as its members’ reads. A query that reaches one of these limits or times out fails, even when partial responses are allowed; a limit a member enforces itself, such as Mimir’s query limits, is that member’s failure. Errors use Prometheus’s types and status codes: bad_data (400), execution (422, which includes the limits), canceled (499), internal (500), and timeout or unavailable (503). Once a member’s read of one of a query’s selectors reaches a limit, the other members’ reads of that selector stop, and the query fails with execution, even when any other of its reads also failed, of that selector or another, or it also timed out or its client canceled it as the limit was reached.
The engine. The engine is the Prometheus version Trickster is built with, with a 5-minute lookback, the @ modifier, negative offsets and the experimental functions enabled, a 1-minute step for subqueries that name none, and at most 11,000 points per series. The query’s limit parameter is honored. A request is validated as Prometheus validates it before any cache lookup or raw read, except that a range query’s step must be a whole number of milliseconds, the engine’s resolution, rather than be evaluated at another step; a timeout must be positive, a limit a non-negative integer, and partial_response a boolean.
Caching. A global ALB keeps each complete evaluation in its own cache_name (default when unset), whole. Its range queries are not delta-cached, as a Prometheus backend’s are: penalty deduplication follows replicas from a range’s first sample, so samples evaluated as part of a range can differ from the same samples evaluated with the whole of it. A cached result answers only the same query with the same time, or the same start, end and step, so a dashboard’s moving range is evaluated each time, while repeats and concurrent viewers are answered from the cache. A result that reaches into the recent past, within the 5-minute lookback or the ALB’s volatile_window, is kept for up to 30 seconds, while samples may still arrive; an older one is kept for the ALB’s timeseries_ttl. A partial result or a failed evaluation is never cached, and is answered with Cache-Control: no-store.
The cache key holds the query’s limit, and its timeout and partial_response resolved against the ALB’s settings, so requests that share a result or an evaluation agree on how it ends. It also holds the Authorization header, the X-Scope-OrgID and THANOS-TENANT tenant headers, the cache_key_headers, cache_key_params and cache_key_form_fields configured on the ALB’s query paths and on the ALB path a request arrives on, such as an alias like /prometheus/api/v1/query_range, and the cache_key_headers configured on its members’ query paths, so no tenant or scope reads another’s results. Results are kept under a namespace that digests the ALB’s members, in order, with each one’s series endpoint, the request headers its query paths send, and its TLS client and SigV4 settings. The namespace changes when membership does, as when a discovery adds or removes a member, and when a member’s configuration does, but not with health, so results kept before a member’s outage stay usable. A request’s evaluation reads the members whose namespace keys its result, even when the pool changes while it runs.
Raw series caching. Beneath its results, a global ALB with a cache also keeps the raw series it reads, so each member is read only for samples the cache doesn’t hold. A member’s series are kept per selector, in two-hour blocks aligned to the epoch, keyed by the member’s series endpoint and configuration (as for the namespace above), the selector’s matchers, and the identity headers that key results, as the member receives them. Samples within a member’s volatile window are read from it by every evaluation and never kept, since replicas’ samples may still arrive; older samples are read once and kept for the ALB’s timeseries_ttl. A dashboard’s moving range therefore reads only its newest minutes from each member, and a wider range reads only the part the cache lacks. The volatile window is 5 minutes for members with flavor: mimir or flavor: thanos, and none for other members; an ALB that sets volatile_window uses it for every member instead, so set it to cover any out-of-order window your members accept. Samples older than raw_cache_window, 31 days by default, aren’t kept either: a range reaching further back reads that older part from the members every time, and the rest from the cache. The window counts whole blocks, and each block a range spans is one cache lookup per selector and member. A selector with an exact metric name can also be answered from the cached blocks of a broader one on the same metric, member and identity, whose matchers are all among its own: up{job="api",code=~"5.."} from up{job="api"}’s blocks, filtered to the series it matches. Each Trickster instance remembers in memory which selectors have cached blocks; nothing more is stored. Concurrent evaluations needing the same samples share one read, and a selector’s missing ranges are read from a member a few at a time. Every chunk of a member’s answer is read through, and the bucket layout of every histogram in it checked, whether the answer arrives as chunks or as samples, before it is kept, so a chunk or histogram that can’t be read fails that member’s read rather than being cached. An answer a member gave incompletely, or failed to give, is never kept. If keeping an answer fails anyway, the evaluations that read it already have their answer, the failure is logged, and its samples are read from the member next time; a cached block that fails as they are merged into it is dropped, and its samples are read again too. The ALB’s max_object_size_bytes, 64 MB by default for a TSM ALB, bounds each block as it bounds each cached result: samples that would take a block past it aren’t kept, so they are read from the members every time while the block keeps what it held. A block refused some samples isn’t tried with them again until timeseries_ttl passes, unless it no longer holds what it held then, as once evicted, when they may fit; each refusal counts in trickster_cache_events_total as event error with reason tsm_global_raw_max_object_size. Blocks are keyed by the limit too, so ALBs of different limits sharing a cache keep their own, and changing the limit starts new blocks while the old ones expire. A memory cache measures a block by the memory it holds, and other caches by its encoded size. The small chunks that frequent refreshes add to a block are encoded again into full ones as they accumulate, so a block kept current every few seconds ends as compact as one read whole. Each selector reads exactly the samples from its start to its end, cached or not, so cached and uncached evaluations agree, and penalty deduplication starts at a selector’s first sample.
Concurrent requests for the same result share one evaluation. The request that starts it owns it: when its client goes away, the evaluation and its raw reads are canceled, and a request that joined it runs the query again.
Reloads and shutdown. A configuration reload leaves the replaced ALB’s evaluations running for the reload_drain_timeout. Then that ALB admits no more evaluations and cancels those still running, which are answered unavailable (503) unless one already failed with execution, as on reaching a limit, which decides its answer as it would a timeout’s, and waits up to the same timeout again for raw series blocks still being stored; nothing a raw read returns from then on is stored. Only after that are its members’ StoreAPI connections closed, discovered members’ included. A memory or redis cache that the new configuration changes or removes, and that a replaced global ALB keeps its results in, is closed after that too, so the blocks still being stored reach it; until then, a memory cache’s objects stay in memory alongside those of the cache replacing it. A cache that keeps its objects in local files (filesystem, bbolt, badger) is closed as the reload applies, since its replacement can open the same files, so it refuses what the replaced ALB stores in it during the drain. A changed or removed cache that no global ALB uses is closed as the reload applies too, once the operations under way in it end, whatever its provider. A reload that fails restores the previous ALB, which keeps reading its discovered members over the connections it already has. Shutdown does the same once the listeners have drained: retiring the evaluators and closing the caches share one shutdown_drain_timeout. A discovered member that autodiscovery removes or rebuilds has its StoreAPI connection closed once the reload_drain_timeout passes, and one it refuses as it builds it, at once.
Reading raw series moves much more data than reading answers, as it does in Thanos: plan max_capture_bytes and the fetch limits for the widest dashboard ranges. To compare global evaluation with Thanos or with merging, see the mimir and thanos profiles of the developer environment.
Providers Supporting Time Series Merge
Trickster currently supports Time Series Merging for the following TSDB Providers:
| Provider Name |
|---|
| Prometheus |
We hope to support more TSDB’s in the future and welcome any help!
Example TS Merge Configuration
backends:
# prom01a and prom01b are redundant and poll the same targets
prom01a:
provider: prometheus
replica_group: prom01
origin_url: http://prom01a.example.com:9090
prometheus:
labels:
region: us-east-1
prom01b:
provider: prometheus
replica_group: prom01
origin_url: http://prom01b.example.com:9090
labels:
region: us-east-1
# prom-alb-01 scatter/gathers to prom01a and prom01b and merges responses for the caller.
# Enforces a max 14-day time range limit on all incoming merge requests.
prom-alb-01:
provider: alb
max_query_range: 14d
alb:
mechanism: tsm # or time_series_merge
pool:
- prom01a
- prom01b
# prom02 and prom03 poll unique targets but produce the same metric names as prom01a/b
prom02:
provider: prometheus
origin_url: http://prom02.example.com:9090
labels:
region: us-east-2
prom03:
provider: prometheus
origin_url: http://prom03.example.com:9090
labels:
region: us-west-1
# prom-alb-all scatter/gathers prom01a/b, prom02 and prom03 and merges their responses
# for the caller. The merge strategy is automatically selected per-query based on the
# outer PromQL aggregation operator. Injected labels are automatically stripped before
# merging so that series from different backends are combined correctly. Because prom01a
# and prom01b are in the same replica_group, their values are de-duplicated before being
# merged/reduced with prom02 and prom03.
prom-alb-all:
provider: alb
alb:
mechanism: tsm
pool:
- prom01a
- prom01b
- prom02
- prom03
Here is the visual representation of a basic TS Merge configuration:

First Response
The First Response mechanism fans a request out to all healthy pool members, and returns the first response received back to the client. All other fanned out responses are cached (if applicable) but otherwise discarded. If one backend in the fanout has already cached the requested object, and the other backends do not, the cached response will return to the caller while the other backends in the fanout will cache their responses as well for subsequent requests through the ALB.
This mechanism works well when using Trickster as an HTTP object cache fronting multiple redundant origins, to ensure the fastest response possible is delivered to downstream clients - even if the HTTP Response Code indicates an error in the request or by the first backend to respond.
First Response Configuration Example
backends:
node01:
provider: reverseproxycache
origin_url: http://node01.example.com
node02:
provider: reverseproxycache
origin_url: http://node-02.example.com
node-alb-fr:
provider: alb
alb:
mechanism: fr # or first_response
pool:
- node01
- node02
Here is the visual representation of this configuration:

First Good Response
The First Good Response (fgr) mechanism acts just as First Response does, except that it waits to return the first response with an HTTP Status Code < 400. If no fanned out response codes are in the acceptable range once all responses are returned (or the timeout has been reached), then the healthiest response, based on min(all_responses_status_codes), is used.
This mechanism is useful in applications such as live internet television. Consider an operational condition where an object may have been written to Origin 1, but not yet written to redundant Origin 2, while users have already received references to and begin requesting the object in a separate manifest. Trickster, when used as an ALB+Cache in this scenario, will poll both backends for the object and cache the positive responses from Origin 1 for serving subsequent requests locally, while a negative cache configuration will avoid potential 404 storms on Origin 2 until the object can be written by the replication process.
Custom Good Status Codes List
By default, fgr will return the first response with a status code < 400. However, you can optionally provide an explicit list of good status codes using the fgr.status_codes configuration setting, as shown in the example below. When set, Trickster will return the first response to be returned that has a status code found in the configured list. An entry may be a single code or an inclusive range, and the two forms mix: status_codes: [ { start: 200, end: 299 }, 304 ].
First Good Response Configuration Example
negative-caches:
default: # by default, backends use the 'default' negative cache
"404": 500 # cache 404 responses for 500ms
backends:
node01:
provider: reverseproxycache
origin_url: http://node-01.example.com
node02:
provider: reverseproxycache
origin_url: http://node-02.example.com
node-alb-fgr:
provider: alb
alb:
mechanism: fgr # or first_good_response
pool:
- node01
- node02
fgr:
status_codes: [ 200, 201, 204 ] # only consider these codes when selecting a response
Here is the visual representation of this configuration:

Newest Last-Modified
The Newest Last-Modified mechanism is focused on providing the user with the newest representation of the response, rather than responding as quickly as possible. It will fan the client request out to all backends, and wait for all responses to come back (or the ALB timeout to be reached) before determining which response is returned to the user.
If at least one fanout response has a Last-Modified header, then any response not containing the header is discarded. The remaining responses are sorted based on their Last Modified header value, and the newest value determines which response is chosen.
This mechanism is useful in applications where an object residing at the same path on multiple origins is updated frequently, such as a DASH or HLS manifest for a live video broadcast. When using Trickster as an ALB+Cache in this scenario, it will poll both backends for the object, and ensure the newest version between them is used as the client response.
Note that with NLM, the response to the user is only as fast as the slowest backend to respond.
Newest Last-Modified Configuration Example
backends:
node01:
provider: reverseproxycache
origin_url: http://node01.example.com
node02:
provider: reverseproxycache
origin_url: http://node-02.example.com
node-alb-nlm:
provider: alb
alb:
mechanism: nlm # or newest_last_modified
pool:
- node01
- node02
Here is the visual representation of this configuration:

User Router
The User Router mechanism is used to control a Request’s destination Backend based on the username in the request. A default Backend (for no-user and users not in the manifest) can be configured, as well as a Backend per-user.
Native MySQL listeners use a deliberately narrower User Router topology than
HTTP backends: one authenticated listener-facing User Router may select only
direct terminal MySQL backends, selection is sticky for the session, and
to_user/to_credential remapping is rejected. See the
MySQL Provider Guide for the complete
authentication, routing, health, cache-identity, and no-route contract.
When a User Router ALB is configured to use an Authenticator, the ALB can also modify a Request’s credentials before passing it off to the destination Backend. In the graphic below, user casey will be routed to the readersBackend, which proxies to a read-only database server with the dbreader credentials; while user taylor will be routed to the writersBackend, which proxies to a read-write database server with the dbwriter credentials. Here is the example configuration corresponding to the graphic:
Credential replacement is applied only when the user’s configured to_backend target is selected. When a request instead uses default_backend - because the username has no mapping, the mapping does not name a usable runtime target, or the mapped target is unavailable - the request retains its inbound credentials. If a user should receive replacement credentials when routed to the same Backend that also serves as the default, set that Backend explicitly as the user’s to_backend.
backends:
readersBackend:
provider: clickhouse
origin_url: http://read.prod.db.com:8123/
writersBackend:
provider: clickhouse
origin_url: http://write.prod.db.com:8123/
click-lb-01:
provider: alb
authenticator_name: dbUsers
alb:
mechanism: ur # or user_router
user_router: # User Router-specific configs
default_backend: readersBackend # optional - users not in the list will route here, origin will 401
users:
casey:
to_user: dbreader # replaces user casey with dbreader in the request's Authorization header
to_credential: ${DB_READER_PW} # replaces credential in the Authorization header with this env
to_backend: readersBackend # explicit selection applies casey's credential replacement
taylor:
to_user: dbwriter # replaces user taylor with dbwriter in the request's Authorization header
to_credential: ${DB_WRITER_PW} # replaces credential in the Authorization header with this env
to_backend: writersBackend # taylor is sent to the writers backend
authenticators:
dbUsers:
provider: clickhouse # use the clickhouse authenticator
users_file: /path/to/user-manifest.csv # this file should include casey and taylor users
users_file_format: csv # required when users_file is set

Supported Backend Provider Types
The User Router mechanism supports all Backend provider types for default_backend and to_backend values, including other User Router ALBs.
However, config validation will fail if:
- there are any possible infinite loops between backends configured
- users could ultimately be routed to different non-virtual (ALB/Rule) backend types by the same User Router ALB. The final ultimate route for all users must be of the same type (regardless of how many additional hops through ALBs and Rules the request would take).
- In other words: user1 cannot be ultimately routed to a
clickhousebackend and user2 be ultimately routed to aprometheusbackend by the same User Router ALB.
- In other words: user1 cannot be ultimately routed to a
User Router without an Authenticator
If a User Router ALB does not use an Authenticator, you can still configure user-specific Backend routes. In these cases Trickster will observe (but not authenticate) the username in the request and route based on the observed username. However, Trickster will exit with a validation failure on startup if a User Router ALB that does not utilize an Authenticator is configured to swap credentials. In short: users must be positively authenticated by a Trickster Authenticator for credential swapping to be permitted by the User Router ALB.
When a User Router ALB doesn’t use an Authenticator, Trickster uses the final destination Backend provider type to select a default Authenticator (operating in observe-only mode / no users manifest) for username observation. For clickhouse-destined User Routers, the observe only Authenticator provider is clickhouse. For all other backend provider types, the default the observe only Authenticator provider is basic (Basic Auth).
to_user / to_credential vs Backend Path Header Injection
It is still possible to insert credentials to a Backend proxy request using the request_headers Backend Path config. But any request_headers alterations configured for auth-related headers (e.g., Authorization) are performed by the Backend after being handled by a User Router; so they would overwrite any user-specific to_user and to_credential transformations performed by the User Router ALB.
Default Backend
As shown in the example config above, you can provide a default_backend config to a User Router, and users who are not in the user router list will be routed to this backend.
If you do not supply a default_backend, users who are not in the manifest will receive a default response of 502 Bad Gateway. You can customize the default response code by setting no_route_status_code to a value between 400 and 599 as in this example:
backends:
prod-01:
provider: reverseproxy
origin_url: https://example.com/
users-lb-01:
provider: alb
alb:
mechanism: ur
authenticator_name: all-users # not shown for brevity, see above examples
user_router:
no_route_status_code: 401 # unauthorized response for users not in allow list
users: # allowed users
casey:
to_backend: prod-01
taylor:
to_backend: prod-01
kris:
to_backend: prod-01
User Router ALB Backend Pool and Health Checking
The User Router does not rotate through or fan out to a pool of Backends like
the other ALB mechanisms. A healthy mapped target is selected directly. When a
mapped target is unavailable, the request uses the healthy default_backend
without applying the mapped target’s credential replacement. If neither target
is available, the router uses its configured no-route response.
That fallback applies to HTTP requests. A native MySQL session whose username
has an explicit mapping fails with a MySQL availability error when that mapped
terminal is unavailable; it is never redirected to default_backend. Only an
unmapped MySQL username may use the configured default terminal.
You can configure a User Router ALB’s backend destinations to be other ALBs with mechanisms that utilize healthchecked pools.
Bounding Per-Member Response Captures
ALB mechanisms that fan out (TSM, FR, FGR, NLM) buffer each pool member’s response in memory before merging or selecting a winner. Without a cap, one misbehaving upstream returning an oversized body can OOM the proxy – an N-way fanout multiplies that by N.
Trickster applies a default cap of 256 MiB per response. A member whose body exceeds the cap is treated as a partial failure: the merged response carries an X-Trickster-Result: phit marker and the trickster_alb_fanout_failures_total{mechanism, reason="truncated"} metric increments.
Override the cap at the backend or ALB level:
backends:
default:
max_capture_bytes: 67108864 # 64 MiB, applies to all backends (Prometheus, ClickHouse, ALB members, etc.)
prom-alb-tsm:
provider: alb
alb:
mechanism: tsm
max_capture_bytes: 16777216 # 16 MiB, ALB-specific override
pool:
- prom01
- prom02
The ALB-level value takes precedence over the backend-level value, which in turn takes precedence over the 256 MiB default.
A TSM ALB with evaluation: global applies the cap to each member’s raw series response and to the result each evaluation encodes as its response. Unlike the fanout mechanisms above, it doesn’t treat a member’s response over the cap as a partial failure: the whole query fails with a Prometheus execution error (422), even when partial responses are allowed, with no X-Trickster-Result: phit marker and no truncated count in trickster_alb_fanout_failures_total. A result that would exceed the cap fails with the same error rather than being returned in part.
Bounding Aggregate In-Flight Captures
max_capture_bytes caps each member’s response individually; a fanout to N members can still buffer up to N * max_capture_bytes in flight. For deployments with large pools or low memory ceilings, set max_fanout_capture_bytes to cap the aggregate buffer across all in-flight slots in a single fanout call. Slots dispatched after the aggregate budget would go negative are fail-fasted (marked Failed, no capture buffer allocated) before the upstream handler runs; the merge sees them as partial failures and the existing fallback path handles it.
backends:
prom-alb-tsm:
provider: alb
alb:
mechanism: tsm
max_capture_bytes: 16777216 # 16 MiB per member
max_fanout_capture_bytes: 67108864 # 64 MiB total across all in-flight slots
pool:
- prom01
- prom02
- prom03
- prom04
max_fanout_capture_bytes defaults to 0 (no aggregate cap). Pick a value matching what your trickster instance can afford to buffer per request, independent of pool size.
Maintaining Healthy Pools With Automated Health Check Integrations
Health Checks are configured per-Backend as described in the Health documentation. Each Backend’s health checker will notify all ALB pools of which it is a member when its health status changes, so long as it has been configured with a health check interval for automated checking. When an ALB is notified that the state of a pool member has changed, the ALB will reconstruct its list of healthy pool members before serving the next request.
Health Check States
A backend will report one of three possible health states to its ALBs: unavailable (-1), unknown (0), or available (1).
Health-Based Backend Selection
Each ALB has a configurable healthy_floor value, which is the threshold for determining which pool members are included in the healthy pool, based on their instantaneous health state. The healthy_floor represents the minimum acceptable health state value for inclusion in the healthy pool. The default healthy_floor value is 0, meaning Backends in a state >= 0 (unknown and available) are included in the healthy pool. Setting healthy_floor: 1 would include only available Backends, while a value of -1 will include all backends in the configured pool, including those marked as unavailable.
Backends that do not have a health check interval configured will remain in a permanent state of unknown. Backends will also be in an unknown state from the time Trickster starts until the first of any configured automated health check is completed. A pool member in a permanent unknown state can never reach available, so a healthy_floor: 1 ALB whose members lack health checks would have an empty pool and return 502 for every request. To avoid that, Trickster resets such an ALB’s effective floor to 0 at startup, emits a warning naming the ALB and the un-probed members, and sets the trickster_alb_pool_floor_reset{backend_name} gauge to 1. Configure a health check interval on those members if you want healthy_floor: 1 to apply.
Setting healthy_floor below 0 admits members the probe has confirmed unavailable, not just members in the transient unknown state. If your goal is to keep traffic flowing during the cold-start window before the first probes complete, lower the pool members’ recovery_threshold so they transition out of unknown faster – don’t lower the floor. When healthy_floor < 0 Trickster emits a startup warning and sets the trickster_alb_pool_admits_failing{backend_name} gauge to 1.
Backup Pool Members
A pool member marked backup: true stands by: it receives traffic only while no other member of the pool is in the healthy pool. As soon as one of the other members returns, the backup members stand down again, apart from the sticky sessions they took meanwhile, which stay with them until they end. This applies to every mechanism that has a pool, and to stream and native protocol listeners as well as HTTP. A pool must have at least one member that is not a backup, unless its other members come from autodiscovery; discovered members are never backups.
backends:
db:
provider: alb
listener_names: [ postgres ]
alb:
mechanism: rr
healthy_floor: 1
pool:
- primary
- name: standby
backup: true
Failover depends on the ALB learning that its other members are down, so give them a health check interval or, on a stream listener, stream.passive_health. While an ALB with backup members is dispatching to them, the trickster_alb_pool_on_backup{backend_name} gauge is 1, and a warning is logged when it fails over.
Draining Pool Members
A pool member marked drain: true takes no new work but keeps the sticky sessions it already has, for as long as it is available, so that it can be retired without ending them. Every mechanism leaves a draining member out when it selects a member or fans a request out; only a sticky session can reach it. A pool must have at least one member that is not draining, unless its other members come from autodiscovery. When every member of the primary tier drains, new work goes to the backup members.
Draining ends no session by itself: a session keeps its draining member until its ttl or idle ends it, the member leaves the pool, or the member becomes unavailable. To retire a member without cutting sessions short, keep it draining until they end: a session lasts at most ttl, and ends sooner once unused for idle.
Discovered members drain too: a Kubernetes endpoint that is terminating but still serving, or a deleted pod that is still ready, stays in the pool as a draining member until it stops serving, so a rolling restart moves new sessions to the new pods while the old ones finish theirs. See autodiscovery. This applies to every discovered pool, sticky or not.
Draining members are listed on the health status page and exported by the trickster_alb_member_draining{alb_name, member} gauge, which is 1 for each draining member.
A Time Series Merge ALB, which keeps no sessions, merges nothing from a draining member. Draining the only live member of a replica_group therefore leaves that shard out of its merged results, and the ALB warns of a degraded pool, as it would had the member failed.
backends:
app:
provider: alb
alb:
mechanism: rr
sticky: {}
pool:
- app1
- app2
- name: app3
drain: true
ALBs as Pool Members
An ALB that is a member of another ALB’s pool is always treated as available, even when its own healthy pool is empty: it keeps its share of the outer pool’s traffic and fails it. That is deliberate for weighted splits, where an empty member must not shift its share onto its siblings. Set propagate_health: true on the inner ALB to change that. It then reports unavailable to the pools it belongs to while it has no healthy member, and available otherwise, so an outer pool whose healthy_floor excludes unavailable members sends that share to its other members instead.
backends:
region-east:
provider: alb
alb:
mechanism: rr
propagate_health: true
pool: [ east1, east2 ]
global:
provider: alb
alb:
mechanism: rr
pool:
- region-east
- name: region-west
backup: true
Example ALB Configuration Routing Only To Known Healthy Backends
backends:
prom01:
provider: prometheus
origin_url: http://prom01.example.com:9090
healthcheck:
interval: 1000ms # enables automatic health check polling for ALB pool reporting
prom02:
provider: prometheus
origin_url: http://prom02.example.com:9090
healthcheck:
interval: 1000ms
prom-alb-tsm:
provider: alb
alb:
mechanism: tsm # times series merge healthy pool members
healthy_floor: 1 # only include Backends reporting as 'available' in the healthy pool
pool:
- prom01
- prom02
All-Backends Health Status Page
Trickster 2.x provides a global health status page available at http://trickster:metrics-port/trickster/health or (the configured health_handler_path).
The global status page will display the health state about all backends configured for automated health checking, and every ALB with its pool members. Here is an example configuration and a possible corresponding status page output:
backends:
proxy-01:
provider: reverseproxy
origin_url: http://server01.example.com
# not configured for automated health check polling
prom-01:
provider: prometheus
origin_url: http://prom01.example.com:9090
healthcheck:
interval: 1000ms # enables automatic health check polling every 1s
prom-02:
provider: prometheus
origin_url: http://prom02.example.com:9090
healthcheck:
interval: 1000ms
prom-03:
provider: prometheus
origin_url: http://prom03.example.com:9090
healthcheck:
interval: 1000ms
prom-alb:
provider: alb
alb:
mechanism: rr
pool:
- prom-01
- prom-02
- name: prom-03
drain: true
$ curl "http://${trickster-fqdn}:8481/trickster/health"
Trickster Backend Health Status last change: 2026-09-26 00:00:00 UTC
-------------------------------------------------------------------------------
prom-02 prometheus unavailable since 2026-09-26 00:00:00 UTC
prom-01 prometheus available
prom-03 prometheus available
prom-alb alb (rr) available u:[prom-02] a:[prom-01,prom-03] d:[prom-03]
proxy-01 reverseproxy not configured for automated health checks
-------------------------------------------------------------------------------
For JSON, provide a 'Accept: application/json' Header or query param ?json
For YAML, provide a 'Accept: application/yaml' Header or query param ?yaml
ALB Pool Members on the Status Page
Each ALB is listed with its pool members grouped by health: u:[...] (unavailable), a:[...] (available) and nc:[...] (not checked) in the text form, and unavailablePoolMembers, availablePoolMembers, uncheckedPoolMembers and initializingPoolMembers in the JSON and YAML forms. Draining members are also listed under their health, and again in d:[...] (text) or drainingPoolMembers (JSON and YAML). An ALB is listed as available while its pool has a member it would send new work to: one that meets its healthy_floor, is not draining, and is in the tier in use. An ALB whose members all drain is therefore unavailable, whatever their health, although it still serves their sticky sessions. In the example, prom-alb is available because prom-01 takes new work; were prom-01 down too, it would be listed as unavailable while prom-03 went on serving the sessions pinned to it.
JSON Health Status
As the footer of the plaintext version of the health status page indicates, you may also request a JSON or YAML version of the health status for machine consumption. The JSON version includes additional detail about any Backends marked as unavailable, and is structured as follows:
$ curl "http://${trickster-fqdn}:8481/trickster/health?json" | jq
{
"title": "Trickster Backend Health Status",
"updateTime": "2026-09-26 00:00:00 UTC",
"unavailable": [
{
"name": "prom-02",
"provider": "prometheus",
"downSince": "2026-09-26 00:00:00 UTC",
"detail": "error probing target: dial tcp prom02.example.com:9090: connect: connection refused"
}
],
"available": [
{
"name": "prom-01",
"provider": "prometheus"
},
{
"name": "prom-03",
"provider": "prometheus"
},
{
"name": "prom-alb",
"provider": "alb",
"mechanism": "rr",
"availablePoolMembers": [
"prom-01",
"prom-03"
],
"unavailablePoolMembers": [
"prom-02"
],
"drainingPoolMembers": [
"prom-03"
]
}
],
"unchecked": [
{
"name": "proxy-01",
"provider": "reverseproxy"
}
]
}
2 - ALB Autodiscovery
Autodiscovery keeps an ALB pool’s membership current at runtime:
instead of hand-maintaining a pool list, Trickster watches a discovery
source and adds, updates, and drains pool members as the source changes —
with no config reload or listener restart. Discovered members are cloned
from a designated template backend, so they inherit its caching, TLS,
health check, path, and timeout configuration.
Supported Discovery Providers
| provider | discovers | change detection |
|---|---|---|
kubernetes | endpointslices, services or pods via the Kubernetes API | watch (event-driven) |
dns_srv | SRV records | poll |
dns_a | A/AAAA records | poll |
file | a member-list file | filesystem notify + poll |
http_sd | a member-list HTTP endpoint | poll (ETag-aware) |
consul | Consul service instances | blocking query (event-driven) |
nomad | Nomad’s native service registry | blocking query (event-driven) |
aws | EC2 instances or ECS tasks, by aws.service | poll |
gcp | Compute Engine instances, by gcp.service | poll |
azure | Virtual Machines, by azure.service | poll |
docker | Docker Engine containers | poll |
For other services, users are generally served by the file provider (any
external service-discovery tool can emit the member list), by
http_sd (the universal adapter — a few lines of
glue serving a member list, with no Trickster change), or by the DNS
providers (e.g. Consul’s DNS interface, cloud private DNS zones, Docker’s
embedded DNS). Developers can add providers against a simple interface; see
the provider authoring guide.
Configuration Overview
Autodiscovery has three configuration parts:
- a named discoverer in the top-level
discoverysection — the provider and its connection-level settings (Kubernetes client, DNS resolver). Multiple ALBs can share one discoverer, and therefore one client/watch/resolver stack. - a template backend — a normal backend definition marked
is_template: true. Templates are never routed, never eligible foris_default, never static pool members, and don’t require anorigin_url; they exist to be cloned per discovered member. - the ALB’s
alb.discoveryblock — which discoverer to use, a provider-specificquerydescribing what to select, the template to clone, and guardrail options.
discovery:
in-cluster:
provider: kubernetes
kubernetes:
in_cluster: true
backends:
prom-template:
provider: prometheus
is_template: true
cache_name: default
healthcheck:
interval: 5s
prom-alb:
provider: alb
alb:
mechanism: tsmerge
discovery:
discoverer_name: in-cluster
template_backend: prom-template
query:
kind: endpointslices
namespace: monitoring
service: prometheus
port: web
A discovery-backed ALB may also list static pool members; discovered
members are additive to them. Static pool entries (and discovered members
from sources that convey weights) support integer weights for the
round_robin mechanism:
pool:
- static-member # weight 1
- name: bigger-member
weight: 3
Discoverer Connection Options
Each entry in the top-level discovery section declares a provider and
that provider’s connection-level block:
| provider | block | options |
|---|---|---|
kubernetes | kubernetes | in_cluster (default true) or kubeconfig path — mutually exclusive; plus qps, burst, user_agent, timeout |
dns_srv, dns_a | dns | resolver (host:port; default: system resolver), interval (poll cadence, default 30s, min 1s; record TTLs act as a floor) |
file | file | poll_interval (stat-poll fallback cadence, default 30s, min 1s) |
http_sd | http + http_sd | connection settings in the shared http block (below); http_sd.format selects the member-list document: trickster (default) or prometheus |
consul | http + consul | connection settings in the shared http block; consul.datacenter, namespace, partition, wait, allow_stale, only_passing, warning_is_ready |
nomad | http + nomad | connection settings in the shared http block; nomad.namespace, region, wait, allow_stale |
aws | aws (+ optional http) | aws.service (required: ec2 or ecs), region, and the credential fields; the endpoint is derived, so http.endpoint is an optional override |
gcp | gcp (+ optional http) | gcp.service (required; gce), gcp.project (from the metadata server when unset) and credentials_file; the endpoint is the Compute API, so http.endpoint is an optional override |
docker | docker (+ optional http) | docker.api_version; the endpoint defaults to the well-known socket, so http.endpoint is an optional override (unix:// or tcp://) |
azure | azure (+ optional http) | azure.service (required; vm), azure.subscription_id (required), credentials and cloud; the endpoint comes from the cloud, so http.endpoint is an optional override |
A block is only valid on its own provider’s entries; anything else fails startup.
The shared http block
Providers that discover members by polling an HTTP endpoint share one
connection block rather than each defining its own, so that configuring a
second such provider does not mean learning a second vocabulary. It is
required by http_sd, and rejected on providers that do not poll HTTP.
| option | meaning |
|---|---|
endpoint | base URL of the service to poll (http/https, host required) |
interval | poll cadence (default 30s, min 1s) |
timeout | bound on a single poll (default 10s, min 100ms) |
tls | outbound client TLS: client_cert_path/client_key_path, certificate_authority_paths, insecure_skip_verify |
headers | headers set on every request; where a registry’s credential is a bespoke header (X-Consul-Token, X-Nomad-Token), it goes here |
username, password | static HTTP Basic credential |
bearer_token | sent as Authorization: Bearer <token> |
bearer_token_file | path re-read before each poll, so a rotated credential is picked up without a restart — prefer this over bearer_token for anything that expires |
follow_redirects | default false; a redirect away from the configured endpoint is surfaced rather than chased |
username/password and the bearer-token fields are mutually exclusive,
as are the two bearer-token forms: a config that sets both fails startup
instead of silently preferring one, which is how an operator ends up
debugging a 401 against a config that looks correct.
Providers whose normal poll includes a server-side wait (blocking queries)
need timeout comfortably above that wait. There is no second,
client-level timeout underneath it that could cut a long poll short.
Template Backends
The template_backend is an ordinary backend definition marked
is_template: true. A discovered member overrides exactly the template’s
name, origin_url (and its derived scheme/host/path-prefix, with
the member’s path prefix falling back to the template’s), and replica
group (see below); the member’s weight applies to its pool entry. Every
other option — cache settings, TLS client configuration, healthcheck,
paths, request rewriters, authenticator, timeouts, concurrency limits,
tracing — is inherited from the template.
Templates are excluded from everything that applies only to live
backends: they are never routed, cannot be is_default, cannot appear in
a static ALB pool, and do not require an origin_url. A template must
use an origin-serving provider (not alb or rule), and a TSM-merged or
replica_group_label-using ALB requires a time-series-merge-capable
template provider. A TSM ALB with evaluation: global requires a
prometheus template with a
series source. Discovery replaces only
the origin, so a template’s series_endpoint, including the one Thanos
StoreAPI requires, is used unchanged by every discovered member: each
member reads that same store, and each read counts separately against
max_fetched_series and max_fetched_samples. A pool of sharded Thanos
sidecars discovered this way therefore reads only the one shard the
endpoint names, and Trickster logs a warning at startup for any such
template. Global evaluation over discovered members suits remote read
without a series_endpoint, as with flavor: mimir, which reads
/api/v1/read beneath each member’s own origin. That origin must be
http or https, so a query.scheme of tcp or udp fails
configuration, and a discovered member whose endpoint still can’t be
formed is logged and not added to the pool. See
Global Evaluation.
The alb.discovery Block
| option | description | default |
|---|---|---|
discoverer_name | name of a discoverer in the top-level discovery section (required) | |
template_backend | name of a backend with is_template: true (required) | |
query | provider-specific selection (required; see each provider below) | |
min_members | reject snapshots that would shrink the discovered membership below this count, keeping the last-good pool (guards against source blips returning empty results). 0 disables. | 0 |
debounce_window | coalesce membership changes arriving within this window into one pool update, damping flapping sources. 0 disables. | 0 |
startup_policy | retry starts the ALB with only its static members and keeps retrying the discoverer; fail fails startup when the discoverer is unavailable | retry |
health_mode | probe: discovered members inherit the template’s active health check. provider: the discoverer’s readiness reporting (currently Kubernetes only) drives member health instead of active probes | probe |
Health and Readiness Semantics
In probe mode, each discovered member gets its own health check from the
template’s healthcheck config, and the ALB’s healthy_floor applies
exactly as it does to static members. If the template has no probe
interval and the floor requires passing probes, the floor is reset to 0
with a loud warning (as with static members).
A new member’s first probe runs the moment it is registered, and a member
the provider reports ready is admitted to the pool immediately, ahead
of that first result. This matters for rolling deploys: an orchestrator
retires the old workload on the very readiness signal that announced the
new one, so a pool that held the newcomer in unknown until its own probe
completed would have no admissible member in between, and every request in
that window would fail with a 502 even though a healthy workload was
serving. From its first result on, the probe governs the member exactly as
it does an established one: a member the provider reports not-ready, or
one whose readiness the provider does not convey, waits for its first
passing probe as before.
In provider mode, readiness reported by the discoverer maps onto member
health: ready and terminating members enter as passing (1), not-ready
members as failing (-1), readiness-unknown members as unchecked (0) — so a
healthy_floor of 1 excludes members until the provider reports them
ready.
Members a provider reports as shutting down but still serving (terminating
Kubernetes endpoints whose serving condition holds, deletion-stamped pods
that are still ready) stay in the pool as
draining members: from the moment the
change is observed they take no new requests, connections or sessions, while
sticky sessions already pinned to them keep
reaching them for as long as they are available. Once the provider reports
that a member no longer serves, it is removed from the snapshot, before
the workload is killed — enabling zero-error rolling deploys. On removal, a
member’s in-flight requests complete, its health check stops, its metrics
series are deleted, and its idle upstream connections close after the
configured drain timeout. Draining members count toward min_members and
trickster_alb_discovery_members, and are shown on the
health status page and by the
trickster_alb_member_draining gauge.
TSM Replica Groups
When the ALB uses the Time Series Merge
mechanism, discovered members participate in replica_group semantics
three ways, in priority order:
Per-member groups from the discovery source (highest priority): the Kubernetes provider can read a configured label from each discovered workload via
query.replica_group_label, andfileprovider entries may carry areplica_groupfield. Members sharing a group value are treated as HA replicas of one logical shard and coalesced; distinct values are distinct shards, merged. This handles sharded-HA topologies (e.g., Thanos-style Prometheus pairs) discovered with a single selector:backends: prom-alb: provider: alb alb: mechanism: tsmerge discovery: discoverer_name: in-cluster template_backend: prom-template query: kind: endpointslices namespace: monitoring service: prometheus port: web replica_group_label: prometheus/shardA
replica_groupset on the template backend: inherited by every member without a source-conveyed group — all such members are HA replicas of one shard (the common HA-pair case).Neither (default): each member is its own replica group, i.e. its own logical shard — a pure federation/merge of every member.
For the pods and service kinds the label is read from the discovered
Pod or Service itself. For endpointslices, endpoints don’t carry pod
labels, so setting replica_group_label joins a Pod watch in the query’s
namespace to resolve each endpoint’s target pod (see the RBAC note below);
an endpoint whose pod isn’t yet observed joins ungrouped and regroups
automatically when the pod appears. A member whose group changes is
rebuilt in place under the same name.
Because replica_group is only meaningful on time-series backends,
configuration validation requires a TSM-capable template_backend
whenever the ALB mechanism is tsmerge or replica_group_label is set.
The DNS providers convey no grouping metadata; use the template-level
group (mode 2) or the file provider for grouped non-Kubernetes pools.
Query Vocabulary
Every ALB’s alb.discovery.query block is drawn from one shared vocabulary.
A field set for a provider that does not accept it fails startup rather
than being silently ignored, so a query aimed at the wrong provider is a
config error rather than an empty pool.
| field | kubernetes | dns_srv | dns_a | file | http_sd | consul | nomad | aws | gcp | azure | docker |
|---|---|---|---|---|---|---|---|---|---|---|---|
kind | ● | ||||||||||
namespace | ● | ||||||||||
service | ● | ● | ● | ● | |||||||
selector | ● | ||||||||||
srv_name | ● | ||||||||||
hostname | ● | ||||||||||
path | ● | ● | |||||||||
filter | ● | ● | ● | ||||||||
filters | ● | ● | |||||||||
tags | ● | ● | ● | ● | ● | ||||||
cluster | ● | ||||||||||
network | ● | ||||||||||
address_type | ● | ● | ● | ● | |||||||
port | ● | ● | ● | ● | ● | ● | |||||
port_label | ● | ● | ● | ● | |||||||
scheme | ● | ● | ● | ● | ● | ● | ● | ● | ● | ● | |
replica_group_label | ● | ● | ● | ● | ● | ● |
Most fields mean the same thing everywhere:
scheme—http(default) orhttpsfor the member’s origin URL.port— a static port applied to every member.port_label— reads the port from the provider’s own metadata namespace: an EC2 or Azure tag, a GCE label (then instance metadata), a Docker container label. It wins overportper member, so a fleet can share one static default with individual machines overriding it.address_type—private(default),public, oripv6. Which address to take where a resource has several.replica_group_label— reads a TSM replica group from the same namespace asport_label.tags— narrows by tag/label presence, and is a conjunction: every listed tag must be present.filter/filters— provider-native selection evaluated server-side.filteris an expression string (Consul, Nomad, GCE);filtersis a name→values map (AWSFilter.N, Docker’s filter document). These are passed through, so an expression the upstream does not understand fails the refresh loudly rather than matching nothing.
Hosts versus endpoints is the distinction that decides whether a port is
required. A cloud inventory API returns hosts — aws service: ec2,
gcp, and azure return addresses with no port, so one of port or
port_label is required. Kubernetes and docker return endpoints
that carry their own ports, so a port is optional there and is resolved
from the object when unambiguous.
The kubernetes Provider
The kubernetes provider watches the cluster via the API server (shared
informers; watch-driven, no polling) and supports three query kinds.
Connection options:
discovery:
my-cluster:
provider: kubernetes
kubernetes:
in_cluster: true # use the pod's service account (default)
# kubeconfig: /path/to/kubeconfig # or run against a remote cluster
qps: 20 # sustained API request rate (default 20)
burst: 40 # API request burst allowance (default 40)
timeout: 10s # bounds one-shot API calls (default 10s)
# user_agent: my-trickster/1 # default: trickster/<version> (<os>/<arch>)
in_cluster and kubeconfig are mutually exclusive; with neither set,
in-cluster is assumed.
qps and burst are the client-side rate limit on API requests. The
defaults are raised over the client-go defaults (5 and 10), which are sized
for a one-shot CLI rather than a process that watches many objects.
timeout bounds a single one-shot API call, such as the connectivity
preflight run under startup_policy: fail. It is deliberately not applied
to the REST client as a whole, whose timeout would also truncate the
long-running watch streams the informers depend on.
Shared Watches
Discoverers that agree on the connection (API server, credentials, identity) and on the query’s server-side filtering (namespace, label selector, field selector) share one set of informers, so several ALBs selecting the same Service cost the API server one watch rather than one each. A watch is released when the last subscription using it stops, so unsubscribing one ALB never disturbs another. Nothing needs to be configured for this; it follows from the discoverer and query values being identical.
Startup Preflight
When any ALB bound to a kubernetes discoverer sets startup_policy: fail,
the discoverer contacts the API server’s /version endpoint before any
watch is established, bounded by timeout. Informers are lazy, so without
this check an unreachable API server is indistinguishable from a Service
with no endpoints yet: the process would start clean and serve an empty
pool. Under the default startup_policy: retry the preflight is skipped and
the discoverer keeps watching for the API server to come back.
Query Kinds
endpointslices (default) discovers the endpoint addresses of a named
Service — the pod IPs behind it — and is the right choice for routing
around the Service’s own load balancing. Terminating endpoints that still
serve join as draining members:
query:
kind: endpointslices
namespace: monitoring
service: prometheus
port: web # port name or number; optional when the service has one port
service discovers the ClusterIP of each Service matching a label
selector (or one named Service), one member per Service:
query:
kind: service
namespace: monitoring
selector:
app: prometheus
port: 9090
pods discovers pod IPs directly by label selector, without requiring
a Service (the original ask of issue #609):
query:
kind: pods
namespace: monitoring
selector:
app: prometheus
port: web
namespace defaults to the pod’s own namespace when running in-cluster.
Port and Scheme Resolution
port may be a named port, a number, or omitted when the target declares
exactly one port; ambiguity is logged and the object skipped. The member
scheme comes from query.scheme if set; otherwise a declared
appProtocol: https on the selected port, or a trickstercache.org/scheme
annotation on the watched Service/Pod, selects https; the default is
http.
Zero-Error Rolling Deploys
Terminating endpoints stop taking new work as soon as the change is
observed: they drain, keeping only their sticky sessions, and leave the
discovered membership once they stop serving, ahead of pod deletion.
One piece belongs to the workload, though: Kubernetes marks an endpoint
terminating and signals the container at the same moment, so — as with
any EndpointSlice consumer, kube-proxy included — the pod must keep
serving briefly while its removal propagates. Give discovered workloads a
short preStop delay (a few seconds comfortably covers Trickster’s
sub-second pickup):
lifecycle:
preStop:
sleep: # native sleep action (Kubernetes >= 1.30); images with
seconds: 5 # a shell can use exec with command: ["sleep", "5"]
Without it, an instantly-exiting container can produce a brief window of connection errors during rollouts — regardless of what is consuming the EndpointSlices.
RBAC
The provider needs only list and watch on the resources your query kinds use — it never reads Secrets or ConfigMaps and never writes:
query kind | apiGroup | resource | verbs |
|---|---|---|---|
endpointslices | discovery.k8s.io | endpointslices | list, watch |
service | "" (core) | services | list, watch |
pods | "" (core) | pods | list, watch |
Queries are namespace-scoped, so a namespaced Role/RoleBinding per
watched namespace is sufficient and preferred:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: trickster-autodiscovery
namespace: monitoring
rules:
- apiGroups: ["discovery.k8s.io"]
resources: ["endpointslices"]
verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: trickster-autodiscovery
namespace: monitoring
subjects:
- kind: ServiceAccount
name: trickster
namespace: trickster
roleRef:
kind: Role
name: trickster-autodiscovery
apiGroup: rbac.authorization.k8s.io
Add services and/or pods rules only for the kinds you configure. Note
that using replica_group_label with the endpointslices kind joins a
Pod watch, so it additionally requires pods list/watch in the query’s
namespace. RBAC cannot scope below the resource level, so the grant covers
all objects of that resource in the bound namespace. Out-of-cluster
(kubeconfig-based) discoverers need the same permissions for their user or
service account.
The dns_srv Provider
The dns_srv provider polls SRV records. SRV target and port map to the
member address; SRV weight maps to the member’s load-balancing weight; and
only the highest-priority tier (the lowest priority value present in the
answer) becomes members — lower tiers are treated as standby capacity
managed on the DNS side.
discovery:
corp-dns:
provider: dns_srv
dns:
resolver: 10.0.0.53:53 # optional; default: the system resolver
interval: 30s # poll cadence (default 30s)
backends:
my-alb:
provider: alb
alb:
mechanism: rr
discovery:
discoverer_name: corp-dns
template_backend: my-template
query:
srv_name: _prometheus._tcp.example.com
# scheme: https # optional; default http
Record TTLs act as a floor on the poll cadence: an answer is never
re-resolved before its shortest TTL expires, so interval is the most
frequent the provider will query. Resolution failures keep the last-good
membership; an authoritative empty answer empties it.
The dns_a Provider
The dns_a provider resolves a hostname’s A and AAAA records, with a
fixed port and scheme from the query. This covers round-robin DNS,
headless-service-style DNS outside Kubernetes, Docker’s embedded DNS, and
Consul’s DNS interface without a bespoke provider:
query:
hostname: prometheus.service.consul
port: 9090 # required
# scheme: https # optional; default http
The dns connection options, poll cadence, TTL floor, and failure
semantics are the same as dns_srv.
The file Provider
Note: like Prometheus’s file_sd, the file provider is the universal
integration point — anything that can write a file (cron, sidecar,
consul-template, your deploy tooling) can drive an ALB pool.
The provider watches a local YAML or JSON member-list file and applies it atomically on change:
discovery:
external-sd:
provider: file
# file:
# poll_interval: 30s # stat-poll fallback cadence (default 30s)
backends:
my-alb:
provider: alb
alb:
mechanism: rr
discovery:
discoverer_name: external-sd
template_backend: my-template
query:
path: /etc/trickster/members.yaml
The file is a list of members:
- name: prom-1 # optional; defaults to the address
address: 10.0.0.1:9090 # required host:port
- name: prom-2
scheme: https # optional; default http
address: 10.0.0.2:9090
path_prefix: /base # optional
weight: 3 # optional; default 1
replica_group: shard-0 # optional; TSM replica group (see above)
Writers should replace the file atomically (write to a temp file in the same directory, then rename). A file that fails to read or parse keeps the last-good membership; an empty file is a valid, empty membership.
Change Detection
The provider uses two mechanisms together (via Trickster’s shared filesystem watcher), so updates are never missed:
- Filesystem notification (on the file’s parent directory, debounced): near-instant pickup of writes, atomic renames, and symlink swaps on local filesystems; a directory watch dropped by deletion or recreation is automatically re-armed.
- A content-comparing poll of the file (
file.poll_interval, default30s, minimum1s): the guaranteed fallback wherever notification is unreliable or unavailable.
Guidance for Kubernetes-mounted member files:
- ConfigMap / Secret volume mounts: kubelet applies updates with an atomic symlink swap inside the mount directory, which generates inotify events — notification works, and updates are picked up promptly. Note that kubelet itself syncs ConfigMap content periodically (typically up to a minute), which usually dominates end-to-end latency.
subPathmounts of a ConfigMap/Secret: Kubernetes never updates these after pod start — no mechanism (notification or polling) can see a change. Mount the directory, not asubPath.- Network- or FUSE-backed volumes (NFS PVs, some CSI drivers):
inotify generally does not observe writes made by other clients, so the
stat poll is the effective update mechanism — set
file.poll_intervalto the freshness your deployment needs. emptyDir/hostPathwritten by a sidecar in the same pod: notification works.
The poll compares file content (not timestamps), so any change is
detected within one poll_interval regardless of filesystem timestamp
granularity.
The http_sd Provider
http_sd fetches a member list from an HTTP endpoint. It is the universal
adapter: any service-discovery system Trickster has no in-tree provider for
can feed it through a few lines of glue that serve a member list, with no
Trickster change and no restart.
discovery:
fleet:
provider: http_sd
http:
endpoint: https://sd.example.com
interval: 15s
bearer_token_file: /var/run/secrets/sd-token
http_sd:
format: trickster
backends:
prom-template:
provider: prometheus
is_template: true
prom-alb:
provider: alb
alb:
mechanism: tsm
discovery:
discoverer_name: fleet
template_backend: prom-template
query:
path: /pools/prometheus
The query’s path is optional and is appended to the endpoint, so one
server can serve a different member list per ALB while they share the
discoverer’s connection settings. The query’s scheme supplies the scheme
for prometheus-format targets, which are bare host:port; native-format
entries carry their own and override it.
Formats. trickster is the same document the file provider reads and
is the default:
- name: prom-1
scheme: https
address: 10.0.0.1:9090
path_prefix: /base
weight: 2
replica_group: shard-0
prometheus is the document Prometheus’s own file_sd and http_sd
consume, so an existing endpoint can be pointed at Trickster unchanged:
[{"targets": ["10.0.0.1:9090"], "labels": {"env": "prod"}}]
A group’s __scheme__ label overrides the query scheme, letting one
endpoint serve mixed-scheme members. Note that the Prometheus format cannot
express weight or replica_group — deployments that need weighted
pools or TSM replica groups want the native format.
The format is named explicitly rather than sniffed. The two documents are structurally distinguishable, but guessing means a typo in one format can parse as a valid, wrong membership in the other, and the cost of guessing wrong is a silently drained pool.
Efficiency. Requests carry X-Prometheus-Refresh-Interval-Seconds, as
Prometheus’s does, so servers that generate member lists on demand can pace
their own work. ETag is honored: an unchanged membership costs one
conditional request and a 304, with no re-parse and no snapshot. The
validator is only stored after a document parses, so a rejected document
can never be confirmed as current by a later 304.
Failure. A transport error, an unexpected status, an oversized body
(the limit is 16 MiB), or a document that will not parse all keep the
last-good membership and are logged once per failure streak and counted on
trickster_discovery_refresh_errors_total. An endpoint that
authoritatively returns no members ([]) is a valid membership and is
applied, so a scaled-to-zero pool can be reported as such.
The consul Provider
consul reads service instances from Consul’s health endpoint. It is
event-driven rather than polled: each request is a Consul blocking
query, so the server parks it until the service changes or wait elapses.
A membership change is observed within a round trip instead of within a
poll interval, and a stable service costs one parked connection rather
than a request per interval.
discovery:
consul-dc1:
provider: consul
http:
endpoint: http://127.0.0.1:8500
# a rotated ACL token; Consul accepts the Authorization Bearer scheme
# as an equivalent to its own X-Consul-Token header
bearer_token_file: /var/run/secrets/consul-token
consul:
datacenter: dc1
wait: 5m # how long a blocking query parks; 1s–10m
allow_stale: true # answer from any server, not only the leader
backends:
prom-template:
provider: prometheus
is_template: true
prom-alb:
provider: alb
alb:
mechanism: tsm
health_mode: provider # Consul's own checks decide readiness
discovery:
discoverer_name: consul-dc1
template_backend: prom-template
query:
service: prometheus
tags: [production]
# filter: 'Service.Meta.version == "2"'
# replica_group_label: shard # read from the service's Meta
Readiness. Consul reports per-instance check status, so this is the
first provider outside kubernetes that can honestly answer “is this member
ready”, which makes health_mode: provider meaningful for VM and container
fleets. An instance’s readiness is its worst check: all passing is
ready, critical and maintenance are not ready, and warning is ready by
default (matching how Consul treats warning for DNS) — set
warning_is_ready: false to drain warning instances instead. A status a
future Consul release introduces is treated as not-ready rather than
ignored.
Failing instances are reported as NotReady rather than omitted, so that an
ALB using the default health_mode: probe can decide for itself and a
wholly-unhealthy service does not look like an empty one. Set
only_passing: true to have Consul filter them out server-side instead.
Weights. Consul’s own Weights.Passing / Weights.Warning map onto
member weights, including the passing/warning distinction — an operator who
has already told Consul the relative capacity of each instance does not have
to tell Trickster again.
Addresses. A service that registers its own address overrides its node’s, which is how sidecars and containers with their own routable address are represented. An instance with no usable address or port fails the whole refresh rather than being silently dropped, so a pool never quietly shrinks because of a catalog change nobody noticed.
Labels. Members carry service, service_id, node, datacenter,
status, and tags (comma-bracketed, as ,a,b,). Service metadata is
carried as meta_<key> so that an operator-defined key cannot shadow a
Trickster-assigned label.
Timeouts. http.timeout must outlast consul.wait, because a blocking
query legitimately takes that long. Its default is derived from the wait
rather than shared with the other HTTP providers (Consul adds up to
wait/16 of its own jitter, which the margin covers), and a config that
sets it too low is rejected at startup rather than producing a stream of
timeouts. http.interval is not the poll cadence here — with blocking
queries there is no cadence — it is the retry delay after a failure.
The nomad Provider
nomad reads service instances from Nomad’s native service registry
(Nomad 1.3+). Like consul it is event-driven, using the same HashiCorp
blocking-query protocol, so a membership change is observed within a round
trip rather than within a poll interval.
discovery:
nomad-eu:
provider: nomad
http:
endpoint: http://127.0.0.1:4646
# a rotated ACL token; Nomad accepts the Authorization Bearer scheme
# as an equivalent to its own X-Nomad-Token header
bearer_token_file: /var/run/secrets/nomad-token
nomad:
namespace: default
region: eu-1
wait: 5m
allow_stale: true
backends:
prom-alb:
provider: alb
alb:
discovery:
discoverer_name: nomad-eu
template_backend: prom-template
query:
service: prometheus
tags: [production]
# filter: 'JobID == "monitoring"'
Native registry, not Consul. This reads the registry a job selects with
provider = "nomad" in its service block. Jobs that register into Consul
instead are discovered with the consul provider, and that is the more
capable choice where it applies: Nomad’s service endpoint carries no
per-instance check state, so members are reported ReadyUnknown and
health_mode: provider falls back to Trickster’s own probes. A deployment
that wants discovery-conveyed readiness should register its services into
Consul.
Tags filter client-side. Unlike Consul’s catalog endpoint, Nomad’s
service endpoint has no tag parameter, so query.tags is applied by
Trickster after the response arrives. It is a conjunction — every listed tag
must be present. query.filter is passed through to Nomad and evaluated
server-side.
Labels. Members carry service, service_id, job_id, alloc_id,
node_id, namespace, datacenter, and tags (comma-bracketed). The
allocation and job identifiers are what an operator needs to trace a member
back to the workload that registered it.
Timeouts work exactly as for consul: http.timeout must outlast
nomad.wait, its default is derived from the wait, and http.interval is
the retry delay after a failure rather than a poll cadence.
The aws Provider
aws discovers members from an AWS API, selected by aws.service: ec2
for instances, ecs for tasks. It is required — with more than one AWS
API supported, defaulting would be an arbitrary guess at which one you
meant, so a config that omits it fails at startup. Further AWS sources
arrive as new service values rather than new providers, inheriting this
provider’s credentials, signing, pagination and options.
discovery:
fleet:
provider: aws
aws:
service: ec2 # required: ec2 or ecs
region: us-east-1
# credentials omitted: use the standard chain (IRSA, instance
# profile, environment, shared config). See docs/aws.md.
http:
interval: 60s # instance inventories change slowly; poll gently
backends:
prom-alb:
provider: alb
alb:
discovery:
discoverer_name: fleet
template_backend: prom-template
query:
filters:
tag:service: [prometheus]
instance-state-name: [running]
port_label: trickster-port # read the port from an EC2 tag
address_type: private # private (default), public, or ipv6
# port: 9090 # or a static port for every member
# replica_group_label: shard
Credentials, region resolution and IAM are documented once in
AWS Integration. The IAM principal needs
ec2:DescribeInstances.
Hosts, not endpoints. An instance inventory returns hosts with several addresses and no port, so two query fields exist that the registry providers do not need:
address_type—private(default),public, oripv6— selects which of the instance’s addresses becomes the member address.port_label— names an EC2 tag whose value is the member’s port.portsupplies a static one. At least one is required, and they compose: where both are set, the tag wins per instance andportis the fallback, which is what makesport_labelsafe to adopt incrementally across a fleet.
Selection. filters is passed to EC2 as Filter.N, evaluated
server-side — use tag:<key> to filter on a tag value, and any other
DescribeInstances filter
name. tags additionally requires the presence of tag keys, applied after
the response arrives.
Instance state. running is Ready and pending is NotReady.
Instances that are shutting-down, stopping, stopped or terminated
are omitted entirely rather than reported unready, so they drain from
pools before they stop answering — the same rule the kubernetes provider
applies to terminating endpoints. A state a future EC2 release introduces is
treated as not-ready rather than assumed healthy.
Instances that cannot become members are excluded, not fatal. An
instance with no port_label tag and no static port, or with no address
of the requested type, is skipped and logged (once, until the set of
excluded instances changes) rather than failing the refresh. This differs
deliberately from the consul and nomad providers, where a malformed
entry fails the whole refresh: a service registry contains only instances of
the service, so a bad entry means the API is broken, while an EC2 inventory
routinely contains hosts that simply are not tagged yet. Failing there would
drain a working pool because of one unrelated instance.
Labels. Members carry instance_id, instance_type, instance_state,
image_id, availability_zone, vpc_id, subnet_id, architecture,
private_ip, public_ip, private_dns and public_dns. Instance tags are
carried as tag_<Key>, prefixed so an operator-defined tag cannot shadow a
Trickster-assigned label. The Name tag becomes the member name, falling
back to the instance id.
Endpoint. Derived from the region and service
(https://ec2.<region>.amazonaws.com). Set http.endpoint to override it
for a VPC endpoint, a FIPS endpoint, or a test double; a region is required
when it is not overridden, because the endpoint is built from it.
service: ecs
Discovers ECS tasks, including Fargate, which service: ec2
structurally cannot see — a Fargate task has no EC2 instance behind it.
discovery:
tasks:
provider: aws
aws:
service: ecs
region: us-east-1
http:
interval: 30s
backends:
prom-alb:
provider: alb
alb:
discovery:
discoverer_name: tasks
template_backend: prom-template
query:
cluster: prod # ECS cluster; the account default when unset
service: prometheus # ECS service name; optional
port_label: trickster-port
# port: 9090 # or a static port for every task
The IAM principal needs ecs:ListTasks and ecs:DescribeTasks.
awsvpc only. Each task in awsvpc mode has its own elastic network
interface, so DescribeTasks alone yields a routable address. That is the
only mode Fargate offers and the default for new EC2-launch-type services.
Under bridge or host networking the address belongs to the container
instance rather than the task, and resolving it would take two further API
calls, a second signing service and broader IAM — so such tasks are
excluded with a reason saying so rather than silently missing. If you
need bridge or host mode, say so and it can be added.
Tags must be propagated. port_label reads an ECS task tag, and ECS
does not copy service tags onto tasks unless the service is created with
--propagate-tags SERVICE. Trickster asks DescribeTasks for tags
explicitly (include: [TAGS]); if port_label finds nothing, check the
propagation setting first.
Task state. RUNNING is Ready; PROVISIONING, PENDING and
ACTIVATING are NotReady; DEACTIVATING, STOPPING, DEPROVISIONING
and STOPPED are omitted entirely, so tasks drain from pools before they
stop answering. Container health is only reported when the task definition
declares a health check — UNHEALTHY is NotReady, and UNKNOWN or absent
is treated as ready, because the alternative would make every
un-instrumented task permanently unusable.
Selection is by cluster and service; filters and address_type are
rejected for ecs, since ECS selects by cluster rather than by instance
attribute and an awsvpc task has exactly one address. tags still filters on
tag presence after the response arrives.
Labels. Members carry task_arn, task_id, cluster,
task_definition, group, launch_type, availability_zone,
task_status and health_status, plus task tags as tag_<Key>. The Name
tag becomes the member name, falling back to the task id.
Churn. A task that disappears between ListTasks and DescribeTasks is
ordinary churn; it is reported as an exclusion rather than silently
shrinking the pool.
The gcp Provider
gcp reads a Google Cloud API named by gcp.service. The provider is
named for the cloud rather than for Compute Engine, matching aws — Google
Cloud APIs outside Compute Engine belong here too, and would sit oddly under
a provider called gce.
service is required even though gce is the only value today. A
default added now could never be removed, and every service added later
would then be reached by opting out of a value the operator never chose.
A value names the product an operator would recognize, not the API that
serves it — the same convention as aws.service. That matters here: Cloud
Load Balancing is served by the Compute API alongside instances, so naming
these for the API would collide where naming them for the product does not.
service: gce
Discovers Google Compute Engine instances through the Compute API’s
instances.aggregatedList, which covers every zone in the project in
one paged call — so no zone list has to be configured or kept current.
discovery:
fleet:
provider: gcp
gcp:
service: gce # required
project: my-project # from the metadata server when unset
# credentials_file: /etc/trickster/sa.json # ADC when unset
http:
interval: 60s # instance inventories change slowly
backends:
prom-alb:
provider: alb
alb:
discovery:
discoverer_name: fleet
template_backend: prom-template
query:
filter: 'labels.role = "prometheus" AND status = "RUNNING"'
tags: [http-server] # network tags, matched on presence
port_label: port # instance label, or metadata key
address_type: private # private (default), public, or ipv6
# port: 9090 # or a static port for every member
# replica_group_label: shard
Credentials. Leaving credentials_file empty selects Application
Default Credentials: GOOGLE_APPLICATION_CREDENTIALS, gcloud user
credentials, Workload Identity on GKE, or the instance metadata server
on GCE. Prefer those over a key file wherever the platform offers one.
Credentials resolve lazily and only successes are cached, so Trickster
starts even when the metadata server is briefly unreachable and a momentary
failure does not permanently disable discovery.
credentials_file must be a service account key. The credential type is
required rather than taken from the file: an external_account or
impersonated_service_account configuration can name an arbitrary token URL
or local executable, so accepting whichever type a file happens to declare
would hand credential resolution somewhere unintended. For user credentials,
use ADC instead of this field.
The IAM principal needs compute.instances.list on the project — the
roles/compute.viewer role includes it. The OAuth scope requested is
compute.readonly; Trickster never mutates a project.
Project. Taken from gcp.project, then from the credentials, then from
the metadata server when Trickster runs on GCE. It is deliberately not
required in config, because reading it from the metadata server is the
idiomatic deployment.
Hosts, not endpoints, exactly as for aws service: ec2:
address_type chooses the address and port/port_label supplies the
port, with the label winning per instance and the static port as fallback.
port_label reads an instance label first, then instance metadata —
both are key/value namespaces on a GCE instance, and a deployment already
carrying the value in metadata does not have to move it.
Instance status. RUNNING is Ready; PROVISIONING, STAGING and
REPAIRING are NotReady; STOPPING, SUSPENDING, SUSPENDED and
TERMINATED are omitted entirely, so instances drain from pools before
they stop answering. A status a future Compute Engine release introduces is
treated as not-ready rather than assumed healthy.
Selection. filter is a GCE filter expression, evaluated server-side.
tags matches GCE network tags, which are names without values, so it
filters on presence. Requests set returnPartialSuccess, so one unreachable
zone contributes no instances rather than failing the whole refresh.
Labels. Members carry instance_id, instance_name, status, zone,
machine_type, network, subnetwork, private_ip, public_ip, and
tags (comma-bracketed). Instance labels are carried as label_<key>,
prefixed so a user-defined label cannot shadow a Trickster-assigned one.
Resource URLs are shortened to their last segment, since a member label full
of https://www.googleapis.com/compute/v1/... is unreadable.
The azure Provider
azure reads an Azure Resource Manager API named by azure.service.
service is required even though vm is the only value today, for the
same reason as aws.service and gcp.service: a default added now could
never be removed.
service: vm
Discovers Virtual Machines, joining them to their network interfaces to resolve addresses.
discovery:
fleet:
provider: azure
azure:
service: vm # required
subscription_id: 00000000-0000-0000-0000-000000000000 # required
# resource_group: prod-rg # narrows every list; recommended
# credentials omitted: use the managed identity of the VM or AKS pod
# tenant_id: ... # with a service principal:
# client_id: ...
# client_secret: ...
# federated_token_file: /var/run/secrets/azure/tokens/azure-identity-token
# cloud: public # public (default), usgovernment, china
# power_state: false # one extra list call; see Readiness
http:
interval: 60s # vm inventories change slowly
backends:
prom-alb:
provider: alb
alb:
discovery:
discoverer_name: fleet
template_backend: prom-template
query:
tags: [prometheus] # vm tag names, matched on presence
port_label: port # a vm tag holding the port
# port: 9090 # or a static port for every member
# address_type: private # private (default), public, or ipv6
# replica_group_label: shard
Credentials. Leaving the credential fields empty selects the managed
identity of the Azure VM or AKS pod Trickster runs on, via the instance
metadata service; set client_id alone to select a user-assigned
identity. On AKS with workload identity, set federated_token_file to
the projected token path — the platform writes and rotates that file, so
Trickster holds no secret, and the file is re-read on every token
acquisition. A client_secret service principal is the fallback where
neither is available; it is redacted from config dumps and the management
API.
There is deliberately no credential chaining. A configured credential that fails is an error, not a reason to quietly fall back to the instance metadata service and authenticate as a different principal.
Permissions. The principal needs only Reader on the subscription, or
on the resource group named by resource_group. The specific actions are
Microsoft.Compute/virtualMachines/read and
Microsoft.Network/networkInterfaces/read, plus
Microsoft.Network/publicIPAddresses/read when address_type: public.
power_state: true additionally requires read at subscription scope,
even with resource_group set — see Readiness.
If discovery finds nothing, check resource provider registration first.
A subscription where Microsoft.Compute is not registered answers the VM
list with HTTP 200 and an empty result, not an error, so the pool is
silently empty. New subscriptions start unregistered. Trickster detects this
case and logs it, but the fix is outside Trickster:
az provider register -n Microsoft.Compute && az provider register -n Microsoft.Network
Clouds. cloud selects both the ARM endpoint and the Entra ID login
endpoint together, since keeping two URLs consistent by hand is exactly the
kind of thing that silently half-works. http.endpoint overrides the ARM
endpoint alone, for a private ARM proxy or a test server.
API versions. ARM requires an explicit api-version on every request,
so both are pinned rather than omitted: compute_api_version (default
2024-07-01) for the VM list and network_api_version (default
2024-05-01) for interfaces and public addresses. Pinning means the
response shape cannot change under Trickster when Microsoft ships a new
version. Override either to move forward deliberately.
The join. An Azure VM carries no address. Addresses live on network
interfaces, which the VM references by resource id, and a public address is
a further reference from the interface to a publicIPAddresses resource. A
refresh is therefore two list calls — three with address_type: public —
joined in memory. Resource ids are matched case-insensitively, because
Azure treats them that way and the casing genuinely differs between APIs; a
VM’s interface reference commonly spells the resource group differently from
the interface list. A VM whose references resolve to nothing is excluded
with a message saying so specifically, since that is the symptom a broken
join produces.
resource_group is worth setting on a large subscription: it narrows every
list rather than enumerating thousands of machines to find a handful.
Hosts, not endpoints, exactly as for aws service: ec2 and gcp
service: gce: address_type chooses the address and port/port_label
supplies the port, with the tag winning per VM and the static port as
fallback. VM tags are matched case-insensitively, as Azure treats them,
so a machine tagged Role is found by a query for role.
Readiness. Off by default, ReadyUnknown for every member. The VM list
carries provisioning state, not power state, and a provisioned VM may be
stopped — reporting Ready on that basis would assert something Azure never
said. Setting power_state: true requests the instance view, which makes
running VMs Ready and removes stopped ones from membership entirely so
they drain from pools. The cost is one extra list call per refresh, not one
call per VM: the whole-subscription list accepts statusOnly=true and
returns every machine’s instance view at once. Trickster’s own active health
checks are the alternative, and cover the case either way.
statusOnly is a parameter of the subscription-wide list only. The
resource-group-scoped list accepts it, returns 200, and silently ignores
it, so Trickster always issues this one call at subscription scope — which
is why power_state: true needs subscription-scope read even when
resource_group is set. If the status list ever comes back with no instance
view for any machine, the refresh fails and keeps the last-good
membership rather than reading every VM as stopped and emptying the pool.
Labels. Members carry vm_name, vm_id, location, vm_size,
resource_group, private_ip, public_ip, and power_state when
requested. VM tags are carried as tag_<key>, prefixed so a user-defined
tag cannot shadow a Trickster-assigned label.
The docker Provider
docker discovers containers through the Docker Engine API’s
GET /containers/json, polled on http.interval.
discovery:
containers:
provider: docker
docker: {} # api_version: v1.41 by default
http:
# endpoint: unix:///var/run/docker.sock # the default
# endpoint: tcp://dockerhost:2376 # remote daemon; add a tls block
interval: 30s
backends:
prom-alb:
provider: alb
alb:
discovery:
discoverer_name: containers
template_backend: prom-template
query:
filters:
label: [com.example.discover=yes]
# network: backend # required only when a container is on several
# port: 9090 # or port_label, when the container exposes several
# address_type: private # private (default), public, or ipv6
# replica_group_label: com.example.shard
Endpoint. Taken from http.endpoint, defaulting to
unix:///var/run/docker.sock. Both the unix:// and tcp:// forms
DOCKER_HOST uses are accepted, so one can be pasted into the other. A
tcp:// endpoint takes its client TLS from the shared http.tls block —
that is how a remote daemon’s mutual TLS is configured — and becomes
https on the wire when one is present. TLS on a unix:// endpoint is
rejected rather than ignored.
Access. Trickster needs read access to the Docker socket, which on most
hosts means membership of the docker group. That access is equivalent to
root on the host, so prefer a read-only socket proxy in production over
mounting the socket directly. Trickster only ever issues GET /containers/json.
API version. Pinned to v1.41 (Docker 20.10 and later) rather than
left off. An unversioned request binds to whatever the daemon’s newest
version happens to be, so the response shape would change under Trickster
when the host upgrades Docker. Override with docker.api_version to use a
newer one.
Endpoints, not hosts. Unlike the cloud providers, the Engine API
reports ports, so port is not required. A container exposing exactly
one TCP port resolves automatically; one exposing several is excluded with
a reason naming the candidates, rather than having a guess made for it. UDP
ports are never candidates, which is what makes the single-port case common
in practice — a container exposing one TCP port beside two UDP ports is
unambiguous. port_label reads the port from a container label and wins
over a static port per container.
Addresses. address_type: private (the default) uses the container’s IP
on its network, with the container-internal port. public uses the host
binding instead — the address and port are taken from the same binding, so
a container publishing one port on 127.0.0.1 and another on 0.0.0.0 does
not produce a mismatched pair. A wildcard binding (0.0.0.0 or ::) is not
dialable, so it resolves to 127.0.0.1. ipv6 uses the container’s global
IPv6 address.
Networks. A container on exactly one network needs no network. One on
several must name which, rather than having a map iteration pick — that
would differ between polls and churn the pool. A container attached to no
network is excluded.
Readiness. State: running is required; every other state leaves
membership entirely, so containers drain from pools as they stop. Health
comes from the container’s HEALTHCHECK: healthy is Ready, unhealthy
and starting are NotReady, and a container with no healthcheck
declared is ReadyUnknown, not Ready — the daemon knows the process
started, not that it is serving, and Trickster’s own health checks cover
that. Note that GET /containers/json reports health only inside its
human-readable Status string; the per-container inspect that carries a
structured Health object would cost one request per container per poll.
Selection. filters is passed to the Engine API as its own filter
document and evaluated server-side, so the daemon does the narrowing:
label, name, status, health, network, ancestor and the rest. A
filter name the daemon does not recognize is a 400 and fails the refresh
loudly, rather than being ignored into a silently empty pool. When the query
sets no status filter, status: [running] is added, so a host with a long
history of exited containers is not listed in full every poll.
Labels. Members carry container_id (short form), container_name,
image, state, network and private_ip. Container labels are carried
as label_<key>, prefixed so a user-defined label cannot shadow a
Trickster-assigned one — Compose’s own labels arrive as
label_com.docker.compose.service and the like.
Upstream Permissions
Every provider is read-only; Trickster never mutates a discovery source. The least privilege each needs:
| provider | required access |
|---|---|
kubernetes | list + watch on the resources your query kinds touch — see RBAC |
consul | an ACL token with service:read (and node:read) for the queried services |
nomad | an ACL token with read-job on the namespace |
aws (ec2) | ec2:DescribeInstances |
aws (ecs) | ecs:ListTasks, ecs:DescribeTasks (add ec2:DescribeInstances for EC2-launch-type tasks) |
gcp | compute.instances.list — roles/compute.viewer includes it |
azure | Reader on the subscription or resource group; power_state: true needs it at subscription scope |
docker | read access to the Engine socket (the docker group, or a read-only socket proxy) |
http_sd | whatever the endpoint itself requires; see connection options |
Credentials are never required in config where the platform can supply
them: aws uses the standard credential chain (IRSA, instance profile,
environment), gcp uses Application Default Credentials (Workload Identity
on GKE, the instance service account on GCE), and azure uses the managed
identity of the VM or AKS pod. Prefer those over stored secrets wherever
they are available — see each provider’s section. Secrets that are
configured (aws.secret_key, azure.client_secret) redact themselves from
config dumps and the management API.
Observing Discovered Members
Discovered members appear on the health status page
alongside static pool members, under their generated backend names
(<alb-name>-<member-name>), tagged with their provider, owning ALB, and
discoverer (e.g., prometheus (prom-alb via in-cluster)).
Membership additions and removals are logged at info with the member name
and origin (credentials embedded in origin URLs are masked), and the full
membership is logged at debug on every change; all autodiscovery log
events carry scope=discovery for easy filtering. Per-ALB member-count,
member-change, snapshot-result, and refresh-staleness metrics — and
per-discoverer refresh-error counters — are documented in
metrics.md. When the ALB has a tracing_name configured,
each membership reconcile cycle is traced as an alb.discovery.reconcile
span via the standard tracing subsystem; the request hot
path is never traced by discovery.
3 - Rule Backend
The Rule Backend is not really a true Backend; it only routes inbound requests to other configured Backends, based on how they match against the Rule’s cases.
A Rule is a single inspection operation performed against a single component of an inbound request, which determines the Next Backend to send the request to. The Next Backend can also be a rule Backend, so as to route requests through multiple Rules before arriving at a true Backend destination.
A request that asks to switch protocols, such as a WebSocket handshake, is tunneled by the backend the rule sends it to.
A rule can optionally rewrite multiple portions of the request before, during and after rule matching, by using request rewriters, which allows for powerful and limitless combinations of request rewriting and routing.
Rule Parts
A rule has several required parts, as follows:
Required Rule Parts
input_source- The part of the Request the Rule inspectsinput_type- The source data typeoperation- The operation taken on the input sourcenext_route- The Backend Name indicating the default next route for the Rule if no matching cases. Not required ifredirect_urlis provided.redirect_url- The fully-qualified URL to issue as a 302 redirect to the client in the default case. Not required ifnext_routeis provided.
Optional Rule Parts
input_key- case-sensitive lookup key; required when the source isheaderor URLparaminput_encoding- the encoding of the input, which is decoded prior to performing the operationinput_index- when > -1, the source is split into parts and the input is extracted from parts[input_index]input-delimiter- when input_index > -1, this delimiter is used to split the source into parts, and defaults to a standard space (’ ‘)ingress_req_rewriter name- provides the name of a Request Rewriter to operate on the Request before rule execution.egress_req_rewriter name- provides the name of a Request Rewriter to operate on the Request after rule execution.nomatch_req_rewriter name- provides the name of a Request Rewriter to operate on the Request after rule execution if the request did not match any cases.max_rule_executions- limits the number of rules a Request is passed through, and aborts with a 400 status code when exceeded. Default is 16. The first rule a request reaches sets its budget from this value; each later rule it passes through may lower the budget but not raise it.
input_source permitted values
| source name | example extracted value |
|---|---|
| url | https://example.com:8480/path1/path2?param1=value |
| url_no_params | https://example.com:8480/path1/path2 |
| scheme | https |
| host | example.com:8480 |
| hostname | example.com |
| port | 8480 (inferred from scheme when no port is provided) |
| path | /path1/path2 |
| params | param1=value |
| param | (must be used with input_key as described below) |
| header | (must be used with input_key as described below) |
| has_param | true or false: whether the query parameter named by input_key is present, even if empty |
| has_header | true or false: whether the header named by input_key is present, even if empty |
For a client request, url, scheme, host, hostname and port describe the URL the client requested: the Host header supplies the host and port, and the scheme is https when the request arrived over TLS.
input_type permitted values and operations
| type name | permitted operations |
|---|---|
| string (default) | prefix, suffix, contains, eq, md5, sha1, modulo, rmatch |
| num | eq, le, ge, gt, lt, modulo |
| bool | eq |
Rule Cases
Rule cases define the possible values are able to alter the Request and change the next route.
Case Parts
Required Case Parts
matches- A string list of values applicable to this case.next_route- The Backend Name indicating the next route for the Rule when a request matches this Case. Not required ifredirect_urlis provided.redirect_url- The fully-qualified URL to issue as a 302 redirect to the client when the Request matches this Case. Not required ifnext_routeis provided.
Optional Case Parts
req_rewriter name- provides the name of a Request Rewriter to operate on the Request when this case is matched.
Example Rule - Route Request by Basic Auth Username
In this example config, requests routed through the /example path will be compared against the rules and routed to either the Reader cluster or the Writer cluster. Curling http://trickster-host/example/path would route to the reader or writer cluster based on a provided Authorization header.
rules:
example-user-router:
# default route is reader cluster
next_route: example-reader-cluster
input_source: header
input_key: Authorization
input_type: string
input_encoding: base64 # Authorization: Basic <base64string>
input_index: 1 # Field 1 is the <base64string>
input_delimiter: ' ' # Authorization Header field is space-delimited
operation: prefix # Basic Auth credentials are formatted as user:pass,
# so we can check if it is prefixed with $user:
cases:
- matches: # writers
- 'johndoe:'
- 'janedoe:'
next_route: example-writer-cluster
backends:
example:
provider: rule
rule_name: example-user-router
example-reader-cluster:
provider: rpc
origin_url: 'http://reader-cluster.example.com'
example-writer-cluster:
provider: rpc
origin_url: 'http://writer-cluster.example.com'
path_routing_disabled: true # restrict routing to this backend via rule only, so
# users cannot directly access via /example-writer-cluster/
Example Rule - Route Request by Path Regex
In this example config, requests routed through the /example path will be compared against the rules and routed to either the Reader cluster or the Writer cluster. Curling http://trickster-host/example/reader and http://trickster-host/example/writer would route to the reader or writer cluster by matching the path.
rules:
example-user-router:
# default route is reader cluster
next_route: example-reader-cluster
input_source: path
input_type: string
operation: rmatch # perform regex match against the path to see if it matches 'writer
operation_arg: '^.*\/writer.*$'
cases:
- matches:
- 'true' # rmatch returns true when the input matches the regex; update next_route
next_route: example-writer-cluster
backends:
example:
provider: rule
rule_name: example-user-router
example-reader-cluster:
provider: rpc
origin_url: 'http://reader-cluster.example.com'
example-writer-cluster:
provider: rpc
origin_url: 'http://writer-cluster.example.com'
path_routing_disabled: true # restrict routing to this backend via rule only, so
# users cannot directly access via /example-writer-cluster/
Example Rule - Rewrite a Hostname From a Regex Capture
An rmatch rule makes its numeric and named capture groups available to the matched case and egress request rewriters. This example routes a request containing a three-character tenant label and prefixes the destination hostname with that tenant.
request_rewriters:
tenant-host:
instructions:
- [ 'hostname', 'set', '${tenant}.writer.example.com' ]
rules:
tenant-router:
next_route: example-reader-cluster
input_source: path
input_type: string
operation: rmatch
operation_arg: '\{mylabel="(?P<tenant>[a-z0-9]{3})"\}'
cases:
- matches: [ 'true' ]
req_rewriter_name: tenant-host
next_route: example-writer-cluster
backends:
example:
provider: rule
rule_name: tenant-router
example-reader-cluster:
provider: rpc
origin_url: 'http://reader-cluster.example.com'
example-writer-cluster:
provider: rpc
origin_url: 'http://writer-cluster.example.com'
path_routing_disabled: true
For input_source: path, matching uses Go’s decoded URL.Path. For example, %7Bmylabel%3D%22abc%22%7D is matched as {mylabel="abc"} and ${tenant} expands to abc. ${0} represents the complete regex match, while ${1} represents the first capture group. See Request Rewriters for token lifetime and safety details.
When the destination hostname is a DNS SRV owner name, the destination backend can resolve it with origin_resolution.mode: srv, which takes the port from the SRV answer, so one backend can reach many origins.