Listeners & TLS

• 1: TLS Support
• 2: HTTP/3
• 3: Apache Arrow Flight SQL Listeners

1 - TLS Support

Trickster supports TLS on both the frontend server and backend clients.

Basics

To enable the TLS server, specify the tls_port, and optionally, the tls_address in one of the listeners of your config file. For example:

listeners:
  # default is built-in and is the default listener for backends
  # it uses these values by default:
  default:
    port: 8480
    tls_port: 8483
    # listen on all interfaces
    tls_address: ''

Note, Trickster will only start listening on a TLS port if at least one origin mapped to the named listener has a valid certificate and key configured.

Each origin section of a Trickster config file can be augmented with the optional tls section to modify TLS behavior for front-end and back-end requests. For example:

backends:
  example: # example backend
    tls:   # TLS settings for example backend
      # frontend configs
      full_chain_cert_path: '/path/to/my/cert.pem'
      private_key_path: '/path/to/my/key.pem'
      # backend configs
      insecure_skip_verify: true
      certificate_authority_paths: [ '/path/to/ca1.pem', '/path/to/ca2.pem' ]
      certificate_authority_pem: |
        -----BEGIN CERTIFICATE-----
        ...an inline CA bundle, for material that arrives as configuration...
        -----END CERTIFICATE-----
      server_name: origin.internal.example.com
      exclude_system_roots: true
      client_cert_path: '/path/to/client/cert.pem'
      client_key_path: '/path/to/client/key.pem'

Server Configs - used when responding to clients

Each backend contributes up to 1 certificate and key pair, as configured in the TLS section of the backend config (demonstrated above). A listener serves the certificates of all backends mapped to it, selecting per-handshake by SNI (see Certificate Selection (SNI) below), so a single TLS listener can serve many hostnames.

If the path to any configured Certificate or Key file is unreachable or unparsable, Trickster will exit upon startup with an error providing reasonable context.

You may use the same TLS certificate and key for multiple backends, depending upon how your Trickster configurations are laid out. Any certificates configured by Trickster must match the hostname header of the inbound http request (exactly, or by wildcard interpolation), or clients will likely reject the certificate for security issues.

Certificate Selection (SNI)

When a listener has multiple certificates, Trickster selects the certificate for each TLS handshake in this order:

1. Exact match of the client’s SNI hostname against a certificate’s Subject Alternative Names
2. Wildcard match (e.g. a *.example.com certificate for foo.example.com)
3. A linear compatibility scan (covers clients that send no SNI, and IP SANs)
4. The listener’s first certificate, if nothing else matches

The exact and wildcard lookups are index-based, so per-handshake selection cost is independent of the number of certificates a listener serves.

Automatic Certificate Rotation Detection

Trickster automatically detects when a serving certificate is renewed in place on disk — same file paths, config untouched, as performed by tools like certbot or cert-manager — and hot-swaps the renewed certificate into the live listener. This is entirely independent of configuration reloads: no manual reload, auto_reload_interval, or config change is required.

• Detection is hybrid: filesystem events (via fsnotify) trigger a near-immediate check where the platform and filesystem support them, and a timer-based poll runs regardless — both as the fallback for deployments where filesystem events don’t work (e.g. some network and FUSE filesystems) and as a self-healing backstop for missed events. If event watching is unavailable, detection silently degrades to poll-only.
• Every check compares file contents (not modification times or event payloads), so detection works across platforms and through the atomic symlink swaps used by Kubernetes Secret and projected volumes.
• The certificate, key (and any associated CA bundle) files are watched and validated as one unit: if a poll observes a mid-rotation partial state (e.g. the cert file updated before the key file), the mismatched pair is never served; the last-good pair keeps serving and the change is retried on the next poll.
• Read errors and invalid content never disable detection: the watcher keeps retrying, logs a warning after sustained failures, and the last-good certificate keeps serving. Deleting the files is treated as a persistent failure, not as certificate removal.
• Startup behavior is unchanged: an unreadable or invalid configured certificate is still a fatal startup error. Only post-startup source failures are tolerated.

Rotation detection is on by default and is configured per listener. tls_watch_interval sets the fallback/backstop poll cadence (default 30s); filesystem events, where available, apply rotations within moments regardless of the interval. Setting the interval to 0 disables rotation detection entirely (events included):

listeners:
  default:
    tls_port: 8483
    # backstop poll interval for the cert/key files of mapped backends
    # (default 30s); filesystem events accelerate detection where supported.
    # set to 0 to disable automatic rotation detection.
    tls_watch_interval: 30s

Note: automatic rotation detection currently applies to HTTP(S) listeners. Native-protocol listeners (e.g. mysql) pick up rotated certificates on config reload.

Hot Swap and No-Close Semantics

Certificate swaps — whether from a config reload or automatic rotation detection — never restart, drain, or rebind the listener:

• The certificate is consulted only at handshake time, so established connections (including keep-alive connections and in-flight requests) are untouched by a swap; they continue on the certificate they were handshaken with until they close naturally.
• Only new handshakes see the new certificate.

Runtime Certificates

A listener normally serves TLS only when a mapped backend provides a certificate and key file. Setting tls_runtime_certs: true on a listener serves its tls_port even when no backend provides a file pair, so certificates can be supplied to the running process instead, for example by the Kubernetes Gateway/Ingress controller from TLS Secrets:

listeners:
  default:
    port: 8480
    tls_port: 8483
    tls_runtime_certs: true

Such a listener starts with an empty certificate store and fails handshakes until the first certificate is supplied. Certificates supplied at runtime are validated with the same pair-coherence checks as files, participate in SNI selection alongside any file-sourced certificates on the same listener, appear in the certificate inventory with source memory, and survive configuration reloads and file rotations: a reload replaces only the config-sourced certificates, and a rotation replaces only the file-sourced ones. tls_runtime_certs is valid only on http listeners.

Certificate Inventory (mgmt)

The mgmt listener exposes a read-only, per-listener certificate inventory at /trickster/certificates (configurable via mgmt.certificates_handler_path). Each entry reports the certificate’s id, source kind (file, memory or config), common name, subject alternative names, validity window and last-load time. The inventory never includes key material.

Observability

Certificate rotation and inventory are covered by the following metrics: trickster_tls_certificate_expiration_time_seconds, trickster_tls_certificate_last_load_time_seconds, trickster_tls_certificate_swaps_total, trickster_tls_certificate_validation_failures_total, trickster_tls_watcher_errors_total, and trickster_tls_certificate_store_size. See metrics.md and the example alerting rules in examples/alerting for cert-expiry, sustained-validation-failure and watcher-staleness alerts.

Client Configs - used when proxying to an origin

Each backend’s TLS configuration can also configure the https client used for making requests against the origin as demonstrated above.

insecure_skip_verify will instruct the http client to ignore hostname verification issues with the upstream origin’s certificate, and process the request anyway. This is analogous to -k | --insecure in curl.

certificate_authority_paths will provide the http client with a list of certificate authorities (used in addition to any OS-provided root CA’s) to use when determining the trust of an upstream origin’s TLS certificate. In all cases, the Root CA’s installed to the operating system on which Trickster is running are used for trust by the client.

certificate_authority_pem adds certificate authorities the same way, from a PEM bundle written inline in the configuration rather than read from a file, for material that arrives as configuration (a Kubernetes BackendTLSPolicy’s CA bundle, for instance). A bundle holding no parsable certificate fails validation.

exclude_system_roots: true makes the configured certificate authorities (certificate_authority_paths and certificate_authority_pem) the only ones trusted for the origin, rather than additions to the operating system’s. Use it when the origin must present a certificate from one specific private authority and a certificate from any public authority would be an error. It requires at least one authority to be configured, since excluding the system roots with none would trust nothing.

server_name sets the hostname sent as SNI and verified against the origin’s certificate, when it differs from the host in origin_url — an origin reached by IP or by an internal alias whose certificate names something else. It changes nothing about where the connection goes or which Host header is sent.

To us Mutual Authentication with an upstream origin server, configure Trickster with Client Certificates using client_cert_path and client_key_path parameters, as shown above. You will likely need to also configure a custom CA in certificate_authority_paths to represent your certificate signer, unless it has been added to the underlying Operating System’s CA list.

2 - HTTP/3

Trickster can serve any HTTP listener’s routes over HTTP/3 (RFC 9114) in addition to HTTP/1.1 and HTTP/2, using QUIC over UDP.

HTTP/3 helps most where Trickster acts as an edge cache: loss recovery without head-of-line blocking benefits parallel byte-range fetches, and connection migration keeps a client’s session alive as it changes networks.

Enabling

HTTP/3 attaches to an existing http listener that already serves TLS. QUIC has no cleartext mode, so a listener without a working TLS endpoint cannot serve it.

listeners:
  default:
    port: 8480
    tls_port: 8483
    http3:
      enabled: true

That is the whole configuration for the common case. The UDP endpoint defaults to the same address and port as the TLS endpoint, so clients find HTTP/3 where they already found HTTPS.

Options

listeners:
  default:
    tls_address: 0.0.0.0
    tls_port: 8483
    http3:
      enabled: true
      port: 8443          # bind UDP/8443
      advertised_port: 443 # but tell clients to use 443

TLS certificates come from the backends mapped to the listener, exactly as they do for the TLS/TCP endpoint — see TLS. HTTP/3 uses the same certificates; no separate configuration is needed.

How clients discover HTTP/3

Browsers and most clients do not attempt HTTP/3 first. They connect over TCP and look for an Alt-Svc response header naming an HTTP/3 endpoint:

Alt-Svc: h3=":8443"; ma=2592000

Trickster adds this header to every response from the listener’s TLS/TCP endpoint while HTTP/3 is enabled, so adoption is automatic and gradual. The TCP endpoint keeps serving HTTP/1.1 and HTTP/2 for clients that do not upgrade.

Protocol-upgrade requests (Connection: Upgrade) are not advertised, since such a request hijacks its connection before response headers matter.

Limitations

• No protocol upgrades. HTTP/3 has no equivalent of the HTTP/1.1 101 Switching Protocols handshake; RFC 9114 4.2 makes connection-specific headers malformed. WebSockets over HTTP/3 use Extended CONNECT (RFC 9220), which is not implemented. Send WebSocket traffic over the TCP endpoint, which supports it fully.
• Inbound only. Trickster speaks HTTP/1.1 or HTTP/2 to origins regardless of the protocol a client used to reach it. This matches Traefik; among major proxies only Envoy implements HTTP/3 to upstreams, and it needs Alt-Svc caching plus TCP fallback to do so safely.
• UDP must reach the listener. Some networks block or throttle UDP/443. Clients that cannot establish QUIC silently keep using TCP, so this degrades rather than fails.

Operating notes

UDP receive buffer

QUIC moves far more data through a single socket than typical UDP workloads, and the kernel default receive buffer is often too small. On Linux, raise it:

sysctl -w net.core.rmem_max=7500000
sysctl -w net.core.wmem_max=7500000

Without this, a warning is logged at startup. In containers that cannot change sysctls the warning is unactionable and can be silenced with QUIC_GO_DISABLE_RECEIVE_BUFFER_WARNING=1.

Reloads

The HTTP/3 listener participates in the same reload and drain lifecycle as every other listener. Route changes are hot-swapped without rebinding the socket; changing the bound or advertised port restarts just that endpoint.

Trying it locally

Most systems ship a curl built without HTTP/3 (curl --version | grep HTTP3 to check). Trickster includes a small client so this is not a prerequisite:

make dev-certs                     # self-signed cert for localhost
go run ./hack/h3-client -url https://127.0.0.1:8483/ -ca docs/developer/environment/certs/trickster-dev.crt

The client prints the negotiated protocol, so HTTP/3.0 200 OK confirms the request was served over QUIC.

3 - Apache Arrow Flight SQL Listeners

Trickster can serve Apache Arrow Flight SQL — the gRPC-based query protocol spoken by ADBC drivers, Grafana’s SQL-mode datasources, and the Arrow-native client SDKs — as a caching proxy in front of a Flight SQL-capable origin. The protocol support is vendor-neutral; each backend provider that adopts it supplies only its origin-specific wiring. InfluxDB 3.x is the first supported provider (see InfluxDB Support for its specifics and examples).

Enabling a Flight SQL listener

Define a listener with the flight-sql protocol and map exactly one compatible backend to it through the backend’s listener_names:

listeners:
  my-flight:
    protocol: flight-sql   # Apache Arrow Flight SQL over gRPC
    port: 8485
backends:
  influx3:
    provider: influxdb
    origin_url: 'http://influxdb3:8181/'
    listener_names: [ default, my-flight ]

Flight SQL listeners share Trickster’s standard listener lifecycle: connection limits (connections_limit), graceful drain on SIGTERM, and config reload (SIGHUP) — a reload with an unchanged backend configuration keeps serving on the existing socket, while a changed configuration drains the old server (closing its upstream connection) and rebinds.

TLS

The listener serves TLS when the mapped backend’s tls block presents a certificate and key, with in-place certificate rotation on config reload; without one it serves plaintext gRPC. The upstream dial is TLS-capable via provider options (for InfluxDB, influxdb.flight_upstream_tls).

Caching model

Every cache entry is scoped to the backend name plus a per-tenant namespace derived from the request metadata the provider declares (for InfluxDB: database, bucket-name, and a hash of authorization), mirroring the scope the upstream itself grants — tenants never share entries, and credentials never appear in cache keys.

Statement queries are served through three tiers:

1. Delta proxy cache — statements the provider’s SQL analyzer classifies as delta-cacheable are cached by time extent: repeat and overlapping queries fetch only missing sub-ranges from the origin, and responses are rebuilt into Arrow record batches conforming to the response’s original schema (types, column order, nullability, and schema metadata are preserved). A statement’s ORDER BY is carried through reconstruction, so a delta hit returns rows in the order the statement asked for; ordering terms that do not resolve to a select-list output fall to the next tier. Responses whose schemas the delta model cannot represent fall automatically to the next tier as well.
2. Object cache — everything else cacheable is stored as the verbatim Arrow IPC byte stream and returned byte-identically, with a short, provider-configured lifetime. Metadata RPCs use this tier.
3. Proxy — nondeterministic and non-SELECT statements are never cached.

Serving-cost characteristics of the tiers are recorded in Flight SQL Cache Tier Benchmarks.

Supported RPC surface

• Statement execution (GetFlightInfoStatement / DoGetStatement).
• Result-set schema requests (GetSchemaStatement / GetSchemaPreparedStatement), which ADBC drivers probe on connect. When the origin itself does not implement its schema RPC (InfluxDB 3 Core among them), the schema is derived by executing the statement through the object cache, so the probe’s cost folds into the execution the client performs next.
• Catalog metadata (GetTables, GetCatalogs, GetDbSchemas, GetTableTypes, GetSqlInfo), proxied and cached per tenant.
• Type and key discovery (GetXdbcTypeInfo, GetPrimaryKeys, GetImportedKeys, GetExportedKeys, GetCrossReference), which JDBC- and ODBC-shaped clients probe during metadata discovery, likewise proxied and cached per tenant and per request shape.
• The prepared-statement lifecycle (create, bind, execute, close). A parameterless prepared statement is served through the same three statement tiers — sharing cache entries with ad-hoc executions of the same text — while parameterized executions are cached whole, keyed by the bound parameter hash. Statements abandoned by disconnected clients are closed upstream after 15 minutes of inactivity.

A FlightInfo whose result the origin partitions across several endpoints is consumed in full: every endpoint’s ticket is resolved in order, and an endpoint that advertises its own location is retrieved from that location only when its exact host:port authority is explicitly allowlisted by the provider. This opt-in is required because request metadata, including credentials, is forwarded to the alternate host. A TLS primary is never downgraded to a plaintext alternate. Alternate connection caches are bounded (16 clients by default). Partitions that disagree on schema are refused rather than served partially.

Writes/updates/ingest, transactions, savepoints, Substrait plans, query cancellation, endpoint renewal, and session options return gRPC Unimplemented — they are outside a read-through cache’s contract. Clients requiring them should connect to the origin directly. Prepared-statement bindings are accepted as a single record batch; a client streaming several batches for one binding receives Unimplemented rather than a silently truncated binding.

Response size bound

Responses are buffered whole so they can be cached, so a single very large query would otherwise be able to exhaust the process heap. Each upstream response is bounded at 128MiB by default; a response exceeding the bound is refused with gRPC ResourceExhausted rather than being buffered. Providers expose the bound as a backend option (for InfluxDB, influxdb.flight_max_response_bytes; a negative value removes it). A separate aggregate budget bounds response bytes concurrently assembled or streamed (512MiB by default; influxdb.flight_max_buffered_bytes for InfluxDB).

Metrics

Statement executions record cache outcomes to the native SQL metrics under the flightsql dialect label: trickster_sql_query_cache_total (cache_mode x cache_status), the standard proxy request/points/duration metrics, trickster_sql_query_rewrite_failures_total for extent-rendering failures, and cache-error events on the shared cache metrics. See Metrics.

Architecture (for provider implementers)

The protocol lives in vendor-neutral packages:

• pkg/proxy/flightsql — the gRPC protocol server, caching statement/ metadata/prepared handling, and the upstream client. A provider supplies: the upstream address and TLS posture; ForwardMetadataKeys (which inbound gRPC metadata flows to the origin); a KeyScoper covering every forwarded key that affects results; and optionally a DeltaConfig carrying its SQL dialect analyzer to enable the delta tier.
• pkg/proxy/engines/nativedelta — the transport-agnostic delta-proxy-cache engine (shared with the MySQL native listener), driven by the frozen pkg/parsing/sqlanalyzer dialect contract.
• pkg/timeseries/dataset/arrow — dataset ⇄ Arrow record batch conversion used by the delta tier.

pkg/backends/influxdb/native_listener.go is the reference wiring: a provider integrates by implementing the native.Adapter interface (declaring Protocol() == "flight-sql" and its own BackendProvider()) and constructing the flightsql server from its backend options.