ALB Autodiscovery

Autodiscovery keeps an ALB pool’s membership current at runtime: instead of hand-maintaining a pool list, Trickster watches a discovery source and adds, updates, and drains pool members as the source changes — with no config reload or listener restart. Discovered members are cloned from a designated template backend, so they inherit its caching, TLS, health check, path, and timeout configuration.

Supported Discovery Providers

For other services, users are generally served by the file provider (any external service-discovery tool can emit the member list), by http_sd (the universal adapter — a few lines of glue serving a member list, with no Trickster change), or by the DNS providers (e.g. Consul’s DNS interface, cloud private DNS zones, Docker’s embedded DNS). Developers can add providers against a simple interface; see the provider authoring guide.

Configuration Overview

Autodiscovery has three configuration parts:

1. a named discoverer in the top-level discovery section — the provider and its connection-level settings (Kubernetes client, DNS resolver). Multiple ALBs can share one discoverer, and therefore one client/watch/resolver stack.
2. a template backend — a normal backend definition marked is_template: true. Templates are never routed, never eligible for is_default, never static pool members, and don’t require an origin_url; they exist to be cloned per discovered member.
3. the ALB’s alb.discovery block — which discoverer to use, a provider-specific query describing what to select, the template to clone, and guardrail options.

discovery:
  in-cluster:
    provider: kubernetes
    kubernetes:
      in_cluster: true

backends:
  prom-template:
    provider: prometheus
    is_template: true
    cache_name: default
    healthcheck:
      interval: 5s

  prom-alb:
    provider: alb
    alb:
      mechanism: tsmerge
      discovery:
        discoverer_name: in-cluster
        template_backend: prom-template
        query:
          kind: endpointslices
          namespace: monitoring
          service: prometheus
          port: web

A discovery-backed ALB may also list static pool members; discovered members are additive to them. Static pool entries (and discovered members from sources that convey weights) support integer weights for the round_robin mechanism:

pool:
  - static-member          # weight 1
  - name: bigger-member
    weight: 3

Discoverer Connection Options

Each entry in the top-level discovery section declares a provider and that provider’s connection-level block:

A block is only valid on its own provider’s entries; anything else fails startup.

The shared http block

Providers that discover members by polling an HTTP endpoint share one connection block rather than each defining its own, so that configuring a second such provider does not mean learning a second vocabulary. It is required by http_sd, and rejected on providers that do not poll HTTP.

username/password and the bearer-token fields are mutually exclusive, as are the two bearer-token forms: a config that sets both fails startup instead of silently preferring one, which is how an operator ends up debugging a 401 against a config that looks correct.

Providers whose normal poll includes a server-side wait (blocking queries) need timeout comfortably above that wait. There is no second, client-level timeout underneath it that could cut a long poll short.

Template Backends

The template_backend is an ordinary backend definition marked is_template: true. A discovered member overrides exactly the template’s name, origin_url (and its derived scheme/host/path-prefix, with the member’s path prefix falling back to the template’s), and replica group (see below); the member’s weight applies to its pool entry. Every other option — cache settings, TLS client configuration, healthcheck, paths, request rewriters, authenticator, timeouts, concurrency limits, tracing — is inherited from the template.

Templates are excluded from everything that applies only to live backends: they are never routed, cannot be is_default, cannot appear in a static ALB pool, and do not require an origin_url. A template must use an origin-serving provider (not alb or rule), and a TSM-merged or replica_group_label-using ALB requires a time-series-merge-capable template provider.

The alb.discovery Block

Health and Readiness Semantics

In probe mode, each discovered member gets its own health check from the template’s healthcheck config, and the ALB’s healthy_floor applies exactly as it does to static members. If the template has no probe interval and the floor requires passing probes, the floor is reset to 0 with a loud warning (as with static members).

A new member’s first probe runs the moment it is registered, and a member the provider reports ready is admitted to the pool immediately, ahead of that first result. This matters for rolling deploys: an orchestrator retires the old workload on the very readiness signal that announced the new one, so a pool that held the newcomer in unknown until its own probe completed would have no admissible member in between, and every request in that window would fail with a 502 even though a healthy workload was serving. From its first result on, the probe governs the member exactly as it does an established one: a member the provider reports not-ready, or one whose readiness the provider does not convey, waits for its first passing probe as before.

In provider mode, readiness reported by the discoverer maps onto member health: ready members enter as passing (1), not-ready and terminating members as failing (-1), readiness-unknown members as unchecked (0) — so a healthy_floor of 1 excludes members until the provider reports them ready.

Members a provider reports as shutting down (terminating Kubernetes endpoints, deletion-stamped pods) are removed from the snapshot entirely, so they drain from the pool before the workload is killed — enabling zero-error rolling deploys. On removal, a member’s in-flight requests complete, its health check stops, its metrics series are deleted, and its idle upstream connections close after the configured drain timeout.

TSM Replica Groups

When the ALB uses the Time Series Merge mechanism, discovered members participate in replica_group semantics three ways, in priority order:

1. Per-member groups from the discovery source (highest priority): the Kubernetes provider can read a configured label from each discovered workload via query.replica_group_label, and file provider entries may carry a replica_group field. Members sharing a group value are treated as HA replicas of one logical shard and coalesced; distinct values are distinct shards, merged. This handles sharded-HA topologies (e.g., Thanos-style Prometheus pairs) discovered with a single selector:

   backends:
     prom-alb:
       provider: alb
       alb:
         mechanism: tsmerge
         discovery:
           discoverer_name: in-cluster
           template_backend: prom-template
           query:
             kind: endpointslices
             namespace: monitoring
             service: prometheus
             port: web
             replica_group_label: prometheus/shard
   

2. A replica_group set on the template backend: inherited by every member without a source-conveyed group — all such members are HA replicas of one shard (the common HA-pair case).

3. Neither (default): each member is its own replica group, i.e. its own logical shard — a pure federation/merge of every member.

For the pods and service kinds the label is read from the discovered Pod or Service itself. For endpointslices, endpoints don’t carry pod labels, so setting replica_group_label joins a Pod watch in the query’s namespace to resolve each endpoint’s target pod (see the RBAC note below); an endpoint whose pod isn’t yet observed joins ungrouped and regroups automatically when the pod appears. A member whose group changes is rebuilt in place under the same name.

Because replica_group is only meaningful on time-series backends, configuration validation requires a TSM-capable template_backend whenever the ALB mechanism is tsmerge or replica_group_label is set. The DNS providers convey no grouping metadata; use the template-level group (mode 2) or the file provider for grouped non-Kubernetes pools.

Query Vocabulary

Every ALB’s alb.discovery.query block is drawn from one shared vocabulary. A field set for a provider that does not accept it fails startup rather than being silently ignored, so a query aimed at the wrong provider is a config error rather than an empty pool.

Most fields mean the same thing everywhere:

• scheme — http (default) or https for the member’s origin URL.
• port — a static port applied to every member.
• port_label — reads the port from the provider’s own metadata namespace: an EC2 or Azure tag, a GCE label (then instance metadata), a Docker container label. It wins over port per member, so a fleet can share one static default with individual machines overriding it.
• address_type — private (default), public, or ipv6. Which address to take where a resource has several.
• replica_group_label — reads a TSM replica group from the same namespace as port_label.
• tags — narrows by tag/label presence, and is a conjunction: every listed tag must be present.
• filter / filters — provider-native selection evaluated server-side. filter is an expression string (Consul, Nomad, GCE); filters is a name→values map (AWS Filter.N, Docker’s filter document). These are passed through, so an expression the upstream does not understand fails the refresh loudly rather than matching nothing.

Hosts versus endpoints is the distinction that decides whether a port is required. A cloud inventory API returns hosts — aws service: ec2, gcp, and azure return addresses with no port, so one of port or port_label is required. Kubernetes and docker return endpoints that carry their own ports, so a port is optional there and is resolved from the object when unambiguous.

The kubernetes Provider

The kubernetes provider watches the cluster via the API server (shared informers; watch-driven, no polling) and supports three query kinds.

Connection options:

discovery:
  my-cluster:
    provider: kubernetes
    kubernetes:
      in_cluster: true              # use the pod's service account (default)
      # kubeconfig: /path/to/kubeconfig   # or run against a remote cluster
      qps: 20                       # sustained API request rate (default 20)
      burst: 40                     # API request burst allowance (default 40)
      timeout: 10s                  # bounds one-shot API calls (default 10s)
      # user_agent: my-trickster/1  # default: trickster/<version> (<os>/<arch>)

in_cluster and kubeconfig are mutually exclusive; with neither set, in-cluster is assumed.

qps and burst are the client-side rate limit on API requests. The defaults are raised over the client-go defaults (5 and 10), which are sized for a one-shot CLI rather than a process that watches many objects.

timeout bounds a single one-shot API call, such as the connectivity preflight run under startup_policy: fail. It is deliberately not applied to the REST client as a whole, whose timeout would also truncate the long-running watch streams the informers depend on.

Shared Watches

Discoverers that agree on the connection (API server, credentials, identity) and on the query’s server-side filtering (namespace, label selector, field selector) share one set of informers, so several ALBs selecting the same Service cost the API server one watch rather than one each. A watch is released when the last subscription using it stops, so unsubscribing one ALB never disturbs another. Nothing needs to be configured for this; it follows from the discoverer and query values being identical.

Startup Preflight

When any ALB bound to a kubernetes discoverer sets startup_policy: fail, the discoverer contacts the API server’s /version endpoint before any watch is established, bounded by timeout. Informers are lazy, so without this check an unreachable API server is indistinguishable from a Service with no endpoints yet: the process would start clean and serve an empty pool. Under the default startup_policy: retry the preflight is skipped and the discoverer keeps watching for the API server to come back.

Query Kinds

endpointslices (default) discovers the ready endpoint addresses of a named Service — the pod IPs behind it — and is the right choice for routing around the Service’s own load balancing:

query:
  kind: endpointslices
  namespace: monitoring
  service: prometheus
  port: web          # port name or number; optional when the service has one port

service discovers the ClusterIP of each Service matching a label selector (or one named Service), one member per Service:

query:
  kind: service
  namespace: monitoring
  selector:
    app: prometheus
  port: 9090

pods discovers pod IPs directly by label selector, without requiring a Service (the original ask of issue #609):

query:
  kind: pods
  namespace: monitoring
  selector:
    app: prometheus
  port: web

namespace defaults to the pod’s own namespace when running in-cluster.

Port and Scheme Resolution

port may be a named port, a number, or omitted when the target declares exactly one port; ambiguity is logged and the object skipped. The member scheme comes from query.scheme if set; otherwise a declared appProtocol: https on the selected port, or a trickstercache.org/scheme annotation on the watched Service/Pod, selects https; the default is http.

Zero-Error Rolling Deploys

Terminating endpoints are removed from the discovered membership as soon as the change is observed, so members drain out ahead of pod deletion. One piece belongs to the workload, though: Kubernetes marks an endpoint terminating and signals the container at the same moment, so — as with any EndpointSlice consumer, kube-proxy included — the pod must keep serving briefly while its removal propagates. Give discovered workloads a short preStop delay (a few seconds comfortably covers Trickster’s sub-second pickup):

lifecycle:
  preStop:
    sleep:         # native sleep action (Kubernetes >= 1.30); images with
      seconds: 5   # a shell can use exec with command: ["sleep", "5"]

Without it, an instantly-exiting container can produce a brief window of connection errors during rollouts — regardless of what is consuming the EndpointSlices.

RBAC

The provider needs only list and watch on the resources your query kinds use — it never reads Secrets or ConfigMaps and never writes:

Queries are namespace-scoped, so a namespaced Role/RoleBinding per watched namespace is sufficient and preferred:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: trickster-autodiscovery
  namespace: monitoring
rules:
  - apiGroups: ["discovery.k8s.io"]
    resources: ["endpointslices"]
    verbs: ["list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: trickster-autodiscovery
  namespace: monitoring
subjects:
  - kind: ServiceAccount
    name: trickster
    namespace: trickster
roleRef:
  kind: Role
  name: trickster-autodiscovery
  apiGroup: rbac.authorization.k8s.io

Add services and/or pods rules only for the kinds you configure. Note that using replica_group_label with the endpointslices kind joins a Pod watch, so it additionally requires pods list/watch in the query’s namespace. RBAC cannot scope below the resource level, so the grant covers all objects of that resource in the bound namespace. Out-of-cluster (kubeconfig-based) discoverers need the same permissions for their user or service account.

The dns_srv Provider

The dns_srv provider polls SRV records. SRV target and port map to the member address; SRV weight maps to the member’s load-balancing weight; and only the highest-priority tier (the lowest priority value present in the answer) becomes members — lower tiers are treated as standby capacity managed on the DNS side.

discovery:
  corp-dns:
    provider: dns_srv
    dns:
      resolver: 10.0.0.53:53   # optional; default: the system resolver
      interval: 30s            # poll cadence (default 30s)

backends:
  my-alb:
    provider: alb
    alb:
      mechanism: rr
      discovery:
        discoverer_name: corp-dns
        template_backend: my-template
        query:
          srv_name: _prometheus._tcp.example.com
          # scheme: https      # optional; default http

Record TTLs act as a floor on the poll cadence: an answer is never re-resolved before its shortest TTL expires, so interval is the most frequent the provider will query. Resolution failures keep the last-good membership; an authoritative empty answer empties it.

The dns_a Provider

The dns_a provider resolves a hostname’s A and AAAA records, with a fixed port and scheme from the query. This covers round-robin DNS, headless-service-style DNS outside Kubernetes, Docker’s embedded DNS, and Consul’s DNS interface without a bespoke provider:

query:
  hostname: prometheus.service.consul
  port: 9090           # required
  # scheme: https      # optional; default http

The dns connection options, poll cadence, TTL floor, and failure semantics are the same as dns_srv.

The file Provider

Note: like Prometheus’s file_sd, the file provider is the universal integration point — anything that can write a file (cron, sidecar, consul-template, your deploy tooling) can drive an ALB pool.

The provider watches a local YAML or JSON member-list file and applies it atomically on change:

discovery:
  external-sd:
    provider: file
    # file:
    #   poll_interval: 30s   # stat-poll fallback cadence (default 30s)

backends:
  my-alb:
    provider: alb
    alb:
      mechanism: rr
      discovery:
        discoverer_name: external-sd
        template_backend: my-template
        query:
          path: /etc/trickster/members.yaml

The file is a list of members:

- name: prom-1            # optional; defaults to the address
  address: 10.0.0.1:9090  # required host:port
- name: prom-2
  scheme: https           # optional; default http
  address: 10.0.0.2:9090
  path_prefix: /base      # optional
  weight: 3               # optional; default 1
  replica_group: shard-0  # optional; TSM replica group (see above)

Writers should replace the file atomically (write to a temp file in the same directory, then rename). A file that fails to read or parse keeps the last-good membership; an empty file is a valid, empty membership.

Change Detection

The provider uses two mechanisms together (via Trickster’s shared filesystem watcher), so updates are never missed:

1. Filesystem notification (on the file’s parent directory, debounced): near-instant pickup of writes, atomic renames, and symlink swaps on local filesystems; a directory watch dropped by deletion or recreation is automatically re-armed.
2. A content-comparing poll of the file (file.poll_interval, default 30s, minimum 1s): the guaranteed fallback wherever notification is unreliable or unavailable.

Guidance for Kubernetes-mounted member files:

• ConfigMap / Secret volume mounts: kubelet applies updates with an atomic symlink swap inside the mount directory, which generates inotify events — notification works, and updates are picked up promptly. Note that kubelet itself syncs ConfigMap content periodically (typically up to a minute), which usually dominates end-to-end latency.
• subPath mounts of a ConfigMap/Secret: Kubernetes never updates these after pod start — no mechanism (notification or polling) can see a change. Mount the directory, not a subPath.
• Network- or FUSE-backed volumes (NFS PVs, some CSI drivers): inotify generally does not observe writes made by other clients, so the stat poll is the effective update mechanism — set file.poll_interval to the freshness your deployment needs.
• emptyDir/hostPath written by a sidecar in the same pod: notification works.

The poll compares file content (not timestamps), so any change is detected within one poll_interval regardless of filesystem timestamp granularity.

The http_sd Provider

http_sd fetches a member list from an HTTP endpoint. It is the universal adapter: any service-discovery system Trickster has no in-tree provider for can feed it through a few lines of glue that serve a member list, with no Trickster change and no restart.

discovery:
  fleet:
    provider: http_sd
    http:
      endpoint: https://sd.example.com
      interval: 15s
      bearer_token_file: /var/run/secrets/sd-token
    http_sd:
      format: trickster

backends:
  prom-template:
    provider: prometheus
    is_template: true
  prom-alb:
    provider: alb
    alb:
      mechanism: tsm
      discovery:
        discoverer_name: fleet
        template_backend: prom-template
        query:
          path: /pools/prometheus

The query’s path is optional and is appended to the endpoint, so one server can serve a different member list per ALB while they share the discoverer’s connection settings. The query’s scheme supplies the scheme for prometheus-format targets, which are bare host:port; native-format entries carry their own and override it.

Formats. trickster is the same document the file provider reads and is the default:

- name: prom-1
  scheme: https
  address: 10.0.0.1:9090
  path_prefix: /base
  weight: 2
  replica_group: shard-0

prometheus is the document Prometheus’s own file_sd and http_sd consume, so an existing endpoint can be pointed at Trickster unchanged:

[{"targets": ["10.0.0.1:9090"], "labels": {"env": "prod"}}]

A group’s __scheme__ label overrides the query scheme, letting one endpoint serve mixed-scheme members. Note that the Prometheus format cannot express weight or replica_group — deployments that need weighted pools or TSM replica groups want the native format.

The format is named explicitly rather than sniffed. The two documents are structurally distinguishable, but guessing means a typo in one format can parse as a valid, wrong membership in the other, and the cost of guessing wrong is a silently drained pool.

Efficiency. Requests carry X-Prometheus-Refresh-Interval-Seconds, as Prometheus’s does, so servers that generate member lists on demand can pace their own work. ETag is honored: an unchanged membership costs one conditional request and a 304, with no re-parse and no snapshot. The validator is only stored after a document parses, so a rejected document can never be confirmed as current by a later 304.

Failure. A transport error, an unexpected status, an oversized body (the limit is 16 MiB), or a document that will not parse all keep the last-good membership and are logged once per failure streak and counted on trickster_discovery_refresh_errors_total. An endpoint that authoritatively returns no members ([]) is a valid membership and is applied, so a scaled-to-zero pool can be reported as such.

The consul Provider

consul reads service instances from Consul’s health endpoint. It is event-driven rather than polled: each request is a Consul blocking query, so the server parks it until the service changes or wait elapses. A membership change is observed within a round trip instead of within a poll interval, and a stable service costs one parked connection rather than a request per interval.

discovery:
  consul-dc1:
    provider: consul
    http:
      endpoint: http://127.0.0.1:8500
      # a rotated ACL token; Consul accepts the Authorization Bearer scheme
      # as an equivalent to its own X-Consul-Token header
      bearer_token_file: /var/run/secrets/consul-token
    consul:
      datacenter: dc1
      wait: 5m          # how long a blocking query parks; 1s–10m
      allow_stale: true # answer from any server, not only the leader

backends:
  prom-template:
    provider: prometheus
    is_template: true
  prom-alb:
    provider: alb
    alb:
      mechanism: tsm
      health_mode: provider   # Consul's own checks decide readiness
      discovery:
        discoverer_name: consul-dc1
        template_backend: prom-template
        query:
          service: prometheus
          tags: [production]
          # filter: 'Service.Meta.version == "2"'
          # replica_group_label: shard   # read from the service's Meta

Readiness. Consul reports per-instance check status, so this is the first provider outside kubernetes that can honestly answer “is this member ready”, which makes health_mode: provider meaningful for VM and container fleets. An instance’s readiness is its worst check: all passing is ready, critical and maintenance are not ready, and warning is ready by default (matching how Consul treats warning for DNS) — set warning_is_ready: false to drain warning instances instead. A status a future Consul release introduces is treated as not-ready rather than ignored.

Failing instances are reported as NotReady rather than omitted, so that an ALB using the default health_mode: probe can decide for itself and a wholly-unhealthy service does not look like an empty one. Set only_passing: true to have Consul filter them out server-side instead.

Weights. Consul’s own Weights.Passing / Weights.Warning map onto member weights, including the passing/warning distinction — an operator who has already told Consul the relative capacity of each instance does not have to tell Trickster again.

Addresses. A service that registers its own address overrides its node’s, which is how sidecars and containers with their own routable address are represented. An instance with no usable address or port fails the whole refresh rather than being silently dropped, so a pool never quietly shrinks because of a catalog change nobody noticed.

Labels. Members carry service, service_id, node, datacenter, status, and tags (comma-bracketed, as ,a,b,). Service metadata is carried as meta_<key> so that an operator-defined key cannot shadow a Trickster-assigned label.

Timeouts. http.timeout must outlast consul.wait, because a blocking query legitimately takes that long. Its default is derived from the wait rather than shared with the other HTTP providers (Consul adds up to wait/16 of its own jitter, which the margin covers), and a config that sets it too low is rejected at startup rather than producing a stream of timeouts. http.interval is not the poll cadence here — with blocking queries there is no cadence — it is the retry delay after a failure.

The nomad Provider

nomad reads service instances from Nomad’s native service registry (Nomad 1.3+). Like consul it is event-driven, using the same HashiCorp blocking-query protocol, so a membership change is observed within a round trip rather than within a poll interval.

discovery:
  nomad-eu:
    provider: nomad
    http:
      endpoint: http://127.0.0.1:4646
      # a rotated ACL token; Nomad accepts the Authorization Bearer scheme
      # as an equivalent to its own X-Nomad-Token header
      bearer_token_file: /var/run/secrets/nomad-token
    nomad:
      namespace: default
      region: eu-1
      wait: 5m
      allow_stale: true

backends:
  prom-alb:
    provider: alb
    alb:
      discovery:
        discoverer_name: nomad-eu
        template_backend: prom-template
        query:
          service: prometheus
          tags: [production]
          # filter: 'JobID == "monitoring"'

Native registry, not Consul. This reads the registry a job selects with provider = "nomad" in its service block. Jobs that register into Consul instead are discovered with the consul provider, and that is the more capable choice where it applies: Nomad’s service endpoint carries no per-instance check state, so members are reported ReadyUnknown and health_mode: provider falls back to Trickster’s own probes. A deployment that wants discovery-conveyed readiness should register its services into Consul.

Tags filter client-side. Unlike Consul’s catalog endpoint, Nomad’s service endpoint has no tag parameter, so query.tags is applied by Trickster after the response arrives. It is a conjunction — every listed tag must be present. query.filter is passed through to Nomad and evaluated server-side.

Labels. Members carry service, service_id, job_id, alloc_id, node_id, namespace, datacenter, and tags (comma-bracketed). The allocation and job identifiers are what an operator needs to trace a member back to the workload that registered it.

Timeouts work exactly as for consul: http.timeout must outlast nomad.wait, its default is derived from the wait, and http.interval is the retry delay after a failure rather than a poll cadence.

The aws Provider

aws discovers members from an AWS API, selected by aws.service: ec2 for instances, ecs for tasks. It is required — with more than one AWS API supported, defaulting would be an arbitrary guess at which one you meant, so a config that omits it fails at startup. Further AWS sources arrive as new service values rather than new providers, inheriting this provider’s credentials, signing, pagination and options.

discovery:
  fleet:
    provider: aws
    aws:
      service: ec2      # required: ec2 or ecs
      region: us-east-1
      # credentials omitted: use the standard chain (IRSA, instance
      # profile, environment, shared config). See docs/aws.md.
    http:
      interval: 60s   # instance inventories change slowly; poll gently

backends:
  prom-alb:
    provider: alb
    alb:
      discovery:
        discoverer_name: fleet
        template_backend: prom-template
        query:
          filters:
            tag:service: [prometheus]
            instance-state-name: [running]
          port_label: trickster-port   # read the port from an EC2 tag
          address_type: private        # private (default), public, or ipv6
          # port: 9090                 # or a static port for every member
          # replica_group_label: shard

Credentials, region resolution and IAM are documented once in AWS Integration. The IAM principal needs ec2:DescribeInstances.

Hosts, not endpoints. An instance inventory returns hosts with several addresses and no port, so two query fields exist that the registry providers do not need:

• address_type — private (default), public, or ipv6 — selects which of the instance’s addresses becomes the member address.
• port_label — names an EC2 tag whose value is the member’s port. port supplies a static one. At least one is required, and they compose: where both are set, the tag wins per instance and port is the fallback, which is what makes port_label safe to adopt incrementally across a fleet.

Selection. filters is passed to EC2 as Filter.N, evaluated server-side — use tag:<key> to filter on a tag value, and any other DescribeInstances filter name. tags additionally requires the presence of tag keys, applied after the response arrives.

Instance state. running is Ready and pending is NotReady. Instances that are shutting-down, stopping, stopped or terminated are omitted entirely rather than reported unready, so they drain from pools before they stop answering — the same rule the kubernetes provider applies to terminating endpoints. A state a future EC2 release introduces is treated as not-ready rather than assumed healthy.

Instances that cannot become members are excluded, not fatal. An instance with no port_label tag and no static port, or with no address of the requested type, is skipped and logged (once, until the set of excluded instances changes) rather than failing the refresh. This differs deliberately from the consul and nomad providers, where a malformed entry fails the whole refresh: a service registry contains only instances of the service, so a bad entry means the API is broken, while an EC2 inventory routinely contains hosts that simply are not tagged yet. Failing there would drain a working pool because of one unrelated instance.

Labels. Members carry instance_id, instance_type, instance_state, image_id, availability_zone, vpc_id, subnet_id, architecture, private_ip, public_ip, private_dns and public_dns. Instance tags are carried as tag_<Key>, prefixed so an operator-defined tag cannot shadow a Trickster-assigned label. The Name tag becomes the member name, falling back to the instance id.

Endpoint. Derived from the region and service (https://ec2.<region>.amazonaws.com). Set http.endpoint to override it for a VPC endpoint, a FIPS endpoint, or a test double; a region is required when it is not overridden, because the endpoint is built from it.

service: ecs

Discovers ECS tasks, including Fargate, which service: ec2 structurally cannot see — a Fargate task has no EC2 instance behind it.

discovery:
  tasks:
    provider: aws
    aws:
      service: ecs
      region: us-east-1
    http:
      interval: 30s

backends:
  prom-alb:
    provider: alb
    alb:
      discovery:
        discoverer_name: tasks
        template_backend: prom-template
        query:
          cluster: prod           # ECS cluster; the account default when unset
          service: prometheus     # ECS service name; optional
          port_label: trickster-port
          # port: 9090            # or a static port for every task

The IAM principal needs ecs:ListTasks and ecs:DescribeTasks.

awsvpc only. Each task in awsvpc mode has its own elastic network interface, so DescribeTasks alone yields a routable address. That is the only mode Fargate offers and the default for new EC2-launch-type services. Under bridge or host networking the address belongs to the container instance rather than the task, and resolving it would take two further API calls, a second signing service and broader IAM — so such tasks are excluded with a reason saying so rather than silently missing. If you need bridge or host mode, say so and it can be added.

Tags must be propagated. port_label reads an ECS task tag, and ECS does not copy service tags onto tasks unless the service is created with --propagate-tags SERVICE. Trickster asks DescribeTasks for tags explicitly (include: [TAGS]); if port_label finds nothing, check the propagation setting first.

Task state. RUNNING is Ready; PROVISIONING, PENDING and ACTIVATING are NotReady; DEACTIVATING, STOPPING, DEPROVISIONING and STOPPED are omitted entirely, so tasks drain from pools before they stop answering. Container health is only reported when the task definition declares a health check — UNHEALTHY is NotReady, and UNKNOWN or absent is treated as ready, because the alternative would make every un-instrumented task permanently unusable.

Selection is by cluster and service; filters and address_type are rejected for ecs, since ECS selects by cluster rather than by instance attribute and an awsvpc task has exactly one address. tags still filters on tag presence after the response arrives.

Labels. Members carry task_arn, task_id, cluster, task_definition, group, launch_type, availability_zone, task_status and health_status, plus task tags as tag_<Key>. The Name tag becomes the member name, falling back to the task id.

Churn. A task that disappears between ListTasks and DescribeTasks is ordinary churn; it is reported as an exclusion rather than silently shrinking the pool.

The gcp Provider

gcp reads a Google Cloud API named by gcp.service. The provider is named for the cloud rather than for Compute Engine, matching aws — Google Cloud APIs outside Compute Engine belong here too, and would sit oddly under a provider called gce.

service is required even though gce is the only value today. A default added now could never be removed, and every service added later would then be reached by opting out of a value the operator never chose.

A value names the product an operator would recognize, not the API that serves it — the same convention as aws.service. That matters here: Cloud Load Balancing is served by the Compute API alongside instances, so naming these for the API would collide where naming them for the product does not.

service: gce

Discovers Google Compute Engine instances through the Compute API’s instances.aggregatedList, which covers every zone in the project in one paged call — so no zone list has to be configured or kept current.

discovery:
  fleet:
    provider: gcp
    gcp:
      service: gce               # required
      project: my-project        # from the metadata server when unset
      # credentials_file: /etc/trickster/sa.json   # ADC when unset
    http:
      interval: 60s              # instance inventories change slowly

backends:
  prom-alb:
    provider: alb
    alb:
      discovery:
        discoverer_name: fleet
        template_backend: prom-template
        query:
          filter: 'labels.role = "prometheus" AND status = "RUNNING"'
          tags: [http-server]     # network tags, matched on presence
          port_label: port        # instance label, or metadata key
          address_type: private   # private (default), public, or ipv6
          # port: 9090            # or a static port for every member
          # replica_group_label: shard

Credentials. Leaving credentials_file empty selects Application Default Credentials: GOOGLE_APPLICATION_CREDENTIALS, gcloud user credentials, Workload Identity on GKE, or the instance metadata server on GCE. Prefer those over a key file wherever the platform offers one. Credentials resolve lazily and only successes are cached, so Trickster starts even when the metadata server is briefly unreachable and a momentary failure does not permanently disable discovery.

credentials_file must be a service account key. The credential type is required rather than taken from the file: an external_account or impersonated_service_account configuration can name an arbitrary token URL or local executable, so accepting whichever type a file happens to declare would hand credential resolution somewhere unintended. For user credentials, use ADC instead of this field.

The IAM principal needs compute.instances.list on the project — the roles/compute.viewer role includes it. The OAuth scope requested is compute.readonly; Trickster never mutates a project.

Project. Taken from gcp.project, then from the credentials, then from the metadata server when Trickster runs on GCE. It is deliberately not required in config, because reading it from the metadata server is the idiomatic deployment.

Hosts, not endpoints, exactly as for aws service: ec2: address_type chooses the address and port/port_label supplies the port, with the label winning per instance and the static port as fallback. port_label reads an instance label first, then instance metadata — both are key/value namespaces on a GCE instance, and a deployment already carrying the value in metadata does not have to move it.

Instance status. RUNNING is Ready; PROVISIONING, STAGING and REPAIRING are NotReady; STOPPING, SUSPENDING, SUSPENDED and TERMINATED are omitted entirely, so instances drain from pools before they stop answering. A status a future Compute Engine release introduces is treated as not-ready rather than assumed healthy.

Selection. filter is a GCE filter expression, evaluated server-side. tags matches GCE network tags, which are names without values, so it filters on presence. Requests set returnPartialSuccess, so one unreachable zone contributes no instances rather than failing the whole refresh.

Labels. Members carry instance_id, instance_name, status, zone, machine_type, network, subnetwork, private_ip, public_ip, and tags (comma-bracketed). Instance labels are carried as label_<key>, prefixed so a user-defined label cannot shadow a Trickster-assigned one. Resource URLs are shortened to their last segment, since a member label full of https://www.googleapis.com/compute/v1/... is unreadable.

The azure Provider

azure reads an Azure Resource Manager API named by azure.service. service is required even though vm is the only value today, for the same reason as aws.service and gcp.service: a default added now could never be removed.

service: vm

Discovers Virtual Machines, joining them to their network interfaces to resolve addresses.

discovery:
  fleet:
    provider: azure
    azure:
      service: vm                # required
      subscription_id: 00000000-0000-0000-0000-000000000000  # required
      # resource_group: prod-rg  # narrows every list; recommended
      # credentials omitted: use the managed identity of the VM or AKS pod
      # tenant_id: ...           # with a service principal:
      # client_id: ...
      # client_secret: ...
      # federated_token_file: /var/run/secrets/azure/tokens/azure-identity-token
      # cloud: public            # public (default), usgovernment, china
      # power_state: false       # one extra list call; see Readiness
    http:
      interval: 60s              # vm inventories change slowly

backends:
  prom-alb:
    provider: alb
    alb:
      discovery:
        discoverer_name: fleet
        template_backend: prom-template
        query:
          tags: [prometheus]     # vm tag names, matched on presence
          port_label: port       # a vm tag holding the port
          # port: 9090           # or a static port for every member
          # address_type: private # private (default), public, or ipv6
          # replica_group_label: shard

Credentials. Leaving the credential fields empty selects the managed identity of the Azure VM or AKS pod Trickster runs on, via the instance metadata service; set client_id alone to select a user-assigned identity. On AKS with workload identity, set federated_token_file to the projected token path — the platform writes and rotates that file, so Trickster holds no secret, and the file is re-read on every token acquisition. A client_secret service principal is the fallback where neither is available; it is redacted from config dumps and the management API.

There is deliberately no credential chaining. A configured credential that fails is an error, not a reason to quietly fall back to the instance metadata service and authenticate as a different principal.

Permissions. The principal needs only Reader on the subscription, or on the resource group named by resource_group. The specific actions are Microsoft.Compute/virtualMachines/read and Microsoft.Network/networkInterfaces/read, plus Microsoft.Network/publicIPAddresses/read when address_type: public. power_state: true additionally requires read at subscription scope, even with resource_group set — see Readiness.

If discovery finds nothing, check resource provider registration first. A subscription where Microsoft.Compute is not registered answers the VM list with HTTP 200 and an empty result, not an error, so the pool is silently empty. New subscriptions start unregistered. Trickster detects this case and logs it, but the fix is outside Trickster:

az provider register -n Microsoft.Compute && az provider register -n Microsoft.Network

Clouds. cloud selects both the ARM endpoint and the Entra ID login endpoint together, since keeping two URLs consistent by hand is exactly the kind of thing that silently half-works. http.endpoint overrides the ARM endpoint alone, for a private ARM proxy or a test server.

API versions. ARM requires an explicit api-version on every request, so both are pinned rather than omitted: compute_api_version (default 2024-07-01) for the VM list and network_api_version (default 2024-05-01) for interfaces and public addresses. Pinning means the response shape cannot change under Trickster when Microsoft ships a new version. Override either to move forward deliberately.

The join. An Azure VM carries no address. Addresses live on network interfaces, which the VM references by resource id, and a public address is a further reference from the interface to a publicIPAddresses resource. A refresh is therefore two list calls — three with address_type: public — joined in memory. Resource ids are matched case-insensitively, because Azure treats them that way and the casing genuinely differs between APIs; a VM’s interface reference commonly spells the resource group differently from the interface list. A VM whose references resolve to nothing is excluded with a message saying so specifically, since that is the symptom a broken join produces.

resource_group is worth setting on a large subscription: it narrows every list rather than enumerating thousands of machines to find a handful.

Hosts, not endpoints, exactly as for aws service: ec2 and gcp service: gce: address_type chooses the address and port/port_label supplies the port, with the tag winning per VM and the static port as fallback. VM tags are matched case-insensitively, as Azure treats them, so a machine tagged Role is found by a query for role.

Readiness. Off by default, ReadyUnknown for every member. The VM list carries provisioning state, not power state, and a provisioned VM may be stopped — reporting Ready on that basis would assert something Azure never said. Setting power_state: true requests the instance view, which makes running VMs Ready and removes stopped ones from membership entirely so they drain from pools. The cost is one extra list call per refresh, not one call per VM: the whole-subscription list accepts statusOnly=true and returns every machine’s instance view at once. Trickster’s own active health checks are the alternative, and cover the case either way.

statusOnly is a parameter of the subscription-wide list only. The resource-group-scoped list accepts it, returns 200, and silently ignores it, so Trickster always issues this one call at subscription scope — which is why power_state: true needs subscription-scope read even when resource_group is set. If the status list ever comes back with no instance view for any machine, the refresh fails and keeps the last-good membership rather than reading every VM as stopped and emptying the pool.

Labels. Members carry vm_name, vm_id, location, vm_size, resource_group, private_ip, public_ip, and power_state when requested. VM tags are carried as tag_<key>, prefixed so a user-defined tag cannot shadow a Trickster-assigned label.

The docker Provider

docker discovers containers through the Docker Engine API’s GET /containers/json, polled on http.interval.

discovery:
  containers:
    provider: docker
    docker: {}                 # api_version: v1.41 by default
    http:
      # endpoint: unix:///var/run/docker.sock   # the default
      # endpoint: tcp://dockerhost:2376         # remote daemon; add a tls block
      interval: 30s

backends:
  prom-alb:
    provider: alb
    alb:
      discovery:
        discoverer_name: containers
        template_backend: prom-template
        query:
          filters:
            label: [com.example.discover=yes]
          # network: backend      # required only when a container is on several
          # port: 9090            # or port_label, when the container exposes several
          # address_type: private # private (default), public, or ipv6
          # replica_group_label: com.example.shard

Endpoint. Taken from http.endpoint, defaulting to unix:///var/run/docker.sock. Both the unix:// and tcp:// forms DOCKER_HOST uses are accepted, so one can be pasted into the other. A tcp:// endpoint takes its client TLS from the shared http.tls block — that is how a remote daemon’s mutual TLS is configured — and becomes https on the wire when one is present. TLS on a unix:// endpoint is rejected rather than ignored.

Access. Trickster needs read access to the Docker socket, which on most hosts means membership of the docker group. That access is equivalent to root on the host, so prefer a read-only socket proxy in production over mounting the socket directly. Trickster only ever issues GET /containers/json.

API version. Pinned to v1.41 (Docker 20.10 and later) rather than left off. An unversioned request binds to whatever the daemon’s newest version happens to be, so the response shape would change under Trickster when the host upgrades Docker. Override with docker.api_version to use a newer one.

Endpoints, not hosts. Unlike the cloud providers, the Engine API reports ports, so port is not required. A container exposing exactly one TCP port resolves automatically; one exposing several is excluded with a reason naming the candidates, rather than having a guess made for it. UDP ports are never candidates, which is what makes the single-port case common in practice — a container exposing one TCP port beside two UDP ports is unambiguous. port_label reads the port from a container label and wins over a static port per container.

Addresses. address_type: private (the default) uses the container’s IP on its network, with the container-internal port. public uses the host binding instead — the address and port are taken from the same binding, so a container publishing one port on 127.0.0.1 and another on 0.0.0.0 does not produce a mismatched pair. A wildcard binding (0.0.0.0 or ::) is not dialable, so it resolves to 127.0.0.1. ipv6 uses the container’s global IPv6 address.

Networks. A container on exactly one network needs no network. One on several must name which, rather than having a map iteration pick — that would differ between polls and churn the pool. A container attached to no network is excluded.

Readiness. State: running is required; every other state leaves membership entirely, so containers drain from pools as they stop. Health comes from the container’s HEALTHCHECK: healthy is Ready, unhealthy and starting are NotReady, and a container with no healthcheck declared is ReadyUnknown, not Ready — the daemon knows the process started, not that it is serving, and Trickster’s own health checks cover that. Note that GET /containers/json reports health only inside its human-readable Status string; the per-container inspect that carries a structured Health object would cost one request per container per poll.

Selection. filters is passed to the Engine API as its own filter document and evaluated server-side, so the daemon does the narrowing: label, name, status, health, network, ancestor and the rest. A filter name the daemon does not recognize is a 400 and fails the refresh loudly, rather than being ignored into a silently empty pool. When the query sets no status filter, status: [running] is added, so a host with a long history of exited containers is not listed in full every poll.

Labels. Members carry container_id (short form), container_name, image, state, network and private_ip. Container labels are carried as label_<key>, prefixed so a user-defined label cannot shadow a Trickster-assigned one — Compose’s own labels arrive as label_com.docker.compose.service and the like.

Upstream Permissions

Every provider is read-only; Trickster never mutates a discovery source. The least privilege each needs:

Credentials are never required in config where the platform can supply them: aws uses the standard credential chain (IRSA, instance profile, environment), gcp uses Application Default Credentials (Workload Identity on GKE, the instance service account on GCE), and azure uses the managed identity of the VM or AKS pod. Prefer those over stored secrets wherever they are available — see each provider’s section. Secrets that are configured (aws.secret_key, azure.client_secret) redact themselves from config dumps and the management API.

Observing Discovered Members

Discovered members appear on the health status page alongside static pool members, under their generated backend names (<alb-name>-<member-name>), tagged with their provider, owning ALB, and discoverer (e.g., prometheus (prom-alb via in-cluster)).

Membership additions and removals are logged at info with the member name and origin (credentials embedded in origin URLs are masked), and the full membership is logged at debug on every change; all autodiscovery log events carry scope=discovery for easy filtering. Per-ALB member-count, member-change, snapshot-result, and refresh-staleness metrics — and per-discoverer refresh-error counters — are documented in metrics.md. When the ALB has a tracing_name configured, each membership reconcile cycle is traced as an alb.discovery.reconcile span via the standard tracing subsystem; the request hot path is never traced by discovery.