Observability

• 1: Trickster Metrics
• 2: Distributed Tracing via OpenTelemetry
• 3: Access and Error Logs
• 4: X-Trickster-Result Header

1 - Trickster Metrics

Trickster exposes a Prometheus /metrics endpoint with a customizable listener port number (default is 8481). For more information on customizing the metrics configuration, see configuring.md.

The following metrics are available for polling with any Trickster configuration:

• trickster_build_info (Gauge) - This gauge is always 1 when Trickster is running

  • labels:
    • goversion - the version of go under which the running Trickster binary was built
    • revision - the commit ID on which the running Trickster binary was built
    • version - semantic version of the running Trickster binary

• trickster_config_last_reload_successful (Gauge) - The value is 1 when true (the last config reload was successful) or 0 when false

• trickster_config_last_reload_success_time_seconds (Gauge) - Epoch timestamp of the last successful configuration reload

• trickster_frontend_requests_total (Counter) - Count of front end requests handled by Trickster

  • labels:
    • backend_name - the name of the configured backend handling the proxy request
    • provider - the type of the configured backend handling the proxy request
    • method - the HTTP Method of the proxied request
    • http_status - The HTTP response code provided by the backend
    • path - the Path portion of the requested URL

• trickster_frontend_requests_duration_seconds (Histogram) - Histogram of front end request durations handled by Trickster

  • labels:
    • backend_name - the name of the configured backend handling the proxy request
    • provider - the type of the configured backend handling the proxy request
    • method - the HTTP Method of the proxied request
    • http_status - The HTTP response code provided by the backend
    • path - the Path portion of the requested URL

• trickster_frontend_written_byte_total (Counter) - Count of bytes written in front end requests handled by Trickster

  • labels:
    • backend_name - the name of the configured backend handling the proxy request
    • provider - the type of the configured backend handling the proxy request
    • method - the HTTP Method of the proxied request
    • http_status - The HTTP response code provided by the backend
    • path - the Path portion of the requested URL

• trickster_proxy_requests_total (Counter) - The total number of requests Trickster has handled.

  • labels:
    • backend_name - the name of the configured backend handling the proxy request
    • provider - the type of the configured backend handling the proxy request
    • method - the HTTP Method of the proxied request
    • cache_status - status codes are described here
    • http_status - The HTTP response code provided by the backend
    • path - the Path portion of the requested URL

• trickster_proxy_points_total (Counter) - The total number of data points Trickster has handled.

  • labels:
    • backend_name - the name of the configured backend handling the proxy request
    • provider - the type of the configured backend handling the proxy request
    • cache_status - status codes are described here
    • path - the Path portion of the requested URL

• trickster_proxy_request_duration_seconds (Histogram) - Time required to proxy a given Prometheus query.

  • labels:
    • backend_name - the name of the configured backend handling the proxy request
    • provider - the type of the configured backend handling the proxy request
    • method - the HTTP Method of the proxied request
    • cache_status - status codes are described here
    • http_status - The HTTP response code provided by the backend
    • path - the Path portion of the requested URL

• trickster_proxy_max_connections (Gauge) - Trickster max number of allowed concurrent connections

• trickster_proxy_active_connections (Gauge) - Trickster number of concurrent connections

• trickster_proxy_requested_connections_total (Counter) - Trickster total number of connections requested by clients.

• trickster_proxy_accepted_connections_total (Counter) - Trickster total number of accepted client connections.

• trickster_proxy_closed_connections_total (Counter) - Trickster total number of administratively closed client connections.

• trickster_proxy_failed_connections_total (Counter) - Trickster total number of failed client connections.

• trickster_proxy_query_range_rejected_total (Counter) - Trickster total number of queries rejected due to exceeding the max_query_range limit.

  • labels:
    • backend - the name of the configured backend rejecting the query

• trickster_graphite_resolution_lookups_total (Counter) - Count of Graphite step-resolution lookups. Labels never include a metric path or target expression.

  • labels:
    • backend_name - the name of the configured Graphite backend
    • confidence - how the step was established: exact (read from an origin response for this leaf set and age), derived (computed from known leaf ladders), configured (from static_retentions, not yet probe-confirmed), or unknown (no usable step; the request is served unaccelerated)
    • source - where it came from: registry, response, probe, static, function, or none

• trickster_graphite_probes_total (Counter) - Count of synthetic requests issued to learn a metric’s archive ladder. Expect a spike at startup that collapses toward zero as ladders are learned.

  • labels:
    • backend_name - the name of the configured Graphite backend
    • kind - narrow (a one-second window that also discovers the retention edge), wide (what a real query at that age receives), or find (a /metrics/expand lookup)
    • result - step (a stepped series came back), empty (no series: beyond retention, or no such metric), or error

• trickster_graphite_ladders (Gauge) - Number of distinct archive ladders known to the resolution registry. Ladders come from storage-schemas.conf patterns, so this should flatten at a small number.

  • labels:
    • backend_name - the name of the configured Graphite backend

• trickster_graphite_registry_entries (Gauge) - Number of entries in each layer of the resolution registry.

  • labels:
    • backend_name - the name of the configured Graphite backend
    • layer - leaf (metric path to ladder), ladder (the ladders themselves), target (cached wildcard expansions), or negative (paths in resolution backoff)

• trickster_graphite_step_mispredictions_total (Counter) - Count of origin responses whose step differed from the predicted step. This should always be zero. A non-zero value means a cached ladder was wrong; Trickster discards the prediction, relearns and re-serves the request unaccelerated, so clients still receive correct data.

  • labels:
    • backend_name - the name of the configured Graphite backend

• trickster_graphite_fallbacks_total (Counter) - Count of render requests served without delta caching. Labels never include a target expression.

  • labels:
    • backend_name - the name of the configured Graphite backend
    • reason - parse_error, non_series_format, function_not_allowlisted, unknown_step, missing_target, multi_target_step_mismatch, passthrough_max_data_points, misprediction, client_identity, tz_unavailable, or resolution_identity

• trickster_sql_query_analysis_total (Counter) - Count of SQL query cache-eligibility classifications. Labels never include query text.

  • labels:
    • backend_name - the name of the configured backend analyzing the query
    • dialect - the SQL dialect of the analyzing backend (e.g., clickhouse)
    • cache_mode - the strongest cache mode supported by the query (delta, object, or none)
    • reason - the stable classification reason code (e.g., delta_cacheable, unsafe_predicate, unsupported_bucket)

• trickster_sql_query_rewrite_failures_total (Counter) - Count of SQL cache-miss extent rewrite failures. Labels never include query text.

  • labels:
    • backend_name - the name of the configured backend rendering the query
    • dialect - the SQL dialect of the rendering backend
    • reason - the fixed internal failure category

• trickster_cache_operation_objects_total (Counter) - The total number of objects upon which the Trickster cache has operated.

  • labels:
    • cache_name - the name of the configured cache performing the operation$
    • provider - the type of the configured cache performing the operation
    • operation - the name of the operation being performed (read, write, etc.)
    • status - the result of the operation being performed

• trickster_cache_operation_bytes_total (Counter) - The total number of bytes upon which the Trickster cache has operated.

  • labels:
    • cache_name - the name of the configured cache performing the operation$
    • provider - the type of the configured cache performing the operation
    • operation - the name of the operation being performed (read, write, etc.)
    • status - the result of the operation being performed

• trickster_alb_pool_admits_failing (Gauge) - 1 when an ALB pool’s healthy_floor admits members in the unavailable state, 0 otherwise. See alb.md for the recommended floor.

  • labels:
    • backend_name - the name of the configured ALB backend

• trickster_alb_pool_floor_reset (Gauge) - 1 when an ALB pool’s healthy_floor was reset to 0 at startup because pool members have no health check and could never reach the configured floor, 0 otherwise. See alb.md.

  • labels:
    • backend_name - the name of the configured ALB backend

The following metrics are available when ALB Autodiscovery is configured:

• trickster_alb_discovery_members (Gauge) - Current number of discovered ALB pool members

  • labels:
    • alb_name - the name of the discovery-backed ALB backend
    • discoverer - the name of the discoverer serving the ALB

• trickster_alb_discovery_member_changes_total (Counter) - Count of discovered pool member additions and removals

  • labels:
    • alb_name - the name of the discovery-backed ALB backend
    • discoverer - the name of the discoverer serving the ALB
    • event - add or remove

• trickster_alb_discovery_snapshots_total (Counter) - Count of membership snapshots processed, by result

  • labels:
    • alb_name - the name of the discovery-backed ALB backend
    • discoverer - the name of the discoverer serving the ALB
    • result - applied (membership updated), unchanged (no-op), rejected (guardrail-refused, e.g. min_members), or partial (applied with member instantiation failures)

• trickster_alb_discovery_last_refresh_success_time_seconds (Gauge) - Epoch timestamp of the last successfully processed snapshot, for staleness alerting

  • labels:
    • alb_name - the name of the discovery-backed ALB backend
    • discoverer - the name of the discoverer serving the ALB

• trickster_discovery_refresh_errors_total (Counter) - Count of provider-side refresh/watch errors (DNS resolution failures, Kubernetes list/sync failures, member-file read/parse failures)

  • labels:
    • discoverer - the name of the discoverer experiencing the error
    • provider - the discoverer’s provider type

• trickster_tls_certificate_expiration_time_seconds (Gauge) - NotAfter time of a serving TLS certificate, as unix seconds. See tls.md.

  • labels:
    • listener - the name of the listener serving the certificate
    • entry - the certificate’s source identity

• trickster_tls_certificate_last_load_time_seconds (Gauge) - Epoch timestamp a serving TLS certificate was last loaded from its source

  • labels:
    • listener - the name of the listener serving the certificate
    • entry - the certificate’s source identity

• trickster_tls_certificate_swaps_total (Counter) - Count of TLS certificates hot-swapped into a live listener by rotation detection

  • labels:
    • listener - the name of the listener serving the certificate
    • entry - the certificate’s source identity

• trickster_tls_certificate_validation_failures_total (Counter) - Count of detected TLS certificate source changes that failed pair validation (e.g. a mid-rotation partial write) and were not swapped in

  • labels:
    • entry - the certificate’s source identity

• trickster_tls_watcher_errors_total (Counter) - Count of errors reading watched TLS certificate source files

  • labels:
    • entry - the certificate’s source identity

• trickster_tls_certificate_store_size (Gauge) - Number of certificates in a listener’s TLS certificate store

  • labels:
    • listener - the name of the listener

The following metrics are available only for Caches Types whose object lifecycle Trickster manages internally (Memory, Filesystem and bbolt):

• trickster_cache_events_total (Counter) - The total number of events that change the Trickster cache, such as retention policy evictions.

  • labels:
    • cache_name - the name of the configured cache experiencing the event$
    • provider - the type of the configured cache experiencing the event
    • event - the name of the event being performed
    • reason - the reason the event occurred

• trickster_cache_usage_objects (Gauge) - The current count of objects in the Trickster cache.

  • labels:
    • cache_name - the name of the configured cache$
    • provider - the type of the configured cache$

• trickster_cache_usage_bytes (Gauge) - The current count of bytes in the Trickster cache.

  • labels:
    • cache_name - the name of the configured cache$
    • provider - the type of the configured cache$

• trickster_cache_max_usage_objects (Gauge) - The maximum allowed size of the Trickster cache in objects.

  • labels:
    • cache_name - the name of the configured cache$
    • provider - the type of the configured cache

• trickster_cache_max_usage_bytes (Gauge) - The maximum allowed size of the Trickster cache in bytes.

  • labels:
    • cache_name - the name of the configured cache$
    • provider - the type of the configured cache

In addition to these custom metrics, Trickster also exposes the standard Prometheus metrics that are part of the client_golang metrics instrumentation package, including memory and cpu utilization, etc.

2 - Distributed Tracing via OpenTelemetry

Trickster instruments Distributed Tracing with OpenTelemetry. We import the OpenTelemetry golang packages to instrument support for tracing.

As OpenTelemetry evolves to support additional exporter formats, we will work to extend Trickster to support those as quickly as possible. We also make a best effort to update our otel package imports to the latest releases, whenever we publish a new Trickster release. You can check the go.mod file to see which release of opentelemetry-go we are is using. In this view, to see which version of otel a specific Trickster release imports, use the branch selector dropdown to switch to the tag corresponding to that version of Trickster.

Supported Tracing Backends

• Jaeger (via OTLP)
• Console/Stdout (printed locally by the Trickster process)

Trickster’s OTLP exporter supports OTLP over HTTP and gRPC. HTTP is the default for existing OTLP configs. For Jaeger over OTLP/HTTP, configure Trickster with the collector’s HTTP endpoint, for example http://jaeger:4318/v1/traces. For Jaeger over OTLP/gRPC, set protocol: grpc and use the collector’s gRPC endpoint, for example http://jaeger:4317 from another Compose service or http://127.0.0.1:4317 from the host when using the developer Compose environment.

Configuration

Trickster allows the operator to configure multiple tracing configurations, which can be associated into each Backend configuration by name.

The example config has exhaustive examples of configuring Trickster for distributed tracing.

For OTLP tracing, protocol selects the exporter transport. Supported values are http and grpc; when omitted, Trickster uses http for backward compatibility with existing configs.

Context Propagation

When tracing is enabled for a Backend, Trickster uses the W3C Trace Context and Baggage propagators. It extracts incoming traceparent, tracestate, and baggage headers from client requests and injects the active outbound origin request span into the proxied request. This lets downstream Origins continue the same distributed trace instead of starting an unrelated trace.

Sampling

Trickster uses parent-based sampling for traced requests. The configured sample_rate controls root traces that Trickster starts when there is no sampled upstream trace context. When an incoming request already has a remote parent trace, Trickster follows the parent’s sampling decision so sampled upstream traces continue across the proxy boundary and unsampled upstream traces remain unsampled.

Span List

Trickster can insert several spans to the traces that it captures, depending upon the type and cacheability of the inbound client request, as described in the table below.

Tags / Attributes

Trickster supports adding custom tags to every span via the configuration. Depending upon your preferred tracing backend, these may be referred to as attributes. See the example config for examples of adding custom attributes.

Trickster also supports omitting any tags that Trickster inserts by default. The list of default tags are below. For example on the “request” span, an http.url tag is attached with the current full URL. In deployments where that tag may introduce too much cardinality in your backend trace storage system, you may wish to omit that tag and rely on the more concise path tag. Each tracer config can be provided a string list of tags to omit from traces.

Attributes added to request and core proxy/cache/fetch spans

• backend.name
• backend.provider
• cache.name
• cache.provider
• router.path - request path trimmed to the route match path for the request (e.g., /api/v1/query), good for aggregating when there are large variations in the full URL path
• router.handler

These resource attributes are attached when the corresponding backend, cache, or route configuration is available to the request flow.

Attributes added to top level (request) span

• http.url - the full HTTP request URL

Attributes added to cache/proxy spans

• cache.status - the lookup or proxy cache status where available. See the cache status reference for a description of the attribute values.

Attributes added to proxy/fetch spans

• http.status_code - the HTTP status returned by the Origin or generated proxy response, when available.

Attributes added to the FetchRevalidation span

• isRange - is true if the client request includes an HTTP Range header

Attributes added to the FetchObject span

• isPCF - is true if the origin is configured for Progressive Collapsed Forwarding

3 - Access and Error Logs

Trickster can write per-backend HTTP access logs and error logs, with customizable formats, rotation and retention. Both logs are off by default; each is enabled by configuring its filename.

Basic Configuration

backends:
  example1:
    provider: rp
    origin_url: http://example.com/
    access_log:
      filename: /var/log/trickster/example1.access.log
      error_filename: /var/log/trickster/example1.error.log

• The access log receives one line per request handled by the backend and is written only when filename is set.
• The error log receives one line per request whose response status is at or above error_threshold (default 400) and is written only when error_filename is set. An error-logged request also appears in the access log when both are configured.
• Two backends may share a filename; they will safely share the underlying file and its rotation.
• When instance_id is set in the main config, it is inserted into log filenames just as with the application log (e.g., example1.access.1.log).

Log Format

The format option accepts either a named preset or a custom format string using Apache-style % tokens, so the well-known conventions from Apache HTTP Server, Apache Traffic Server, Lighttpd, and similar servers apply directly.

Presets

Custom Formats

    access_log:
      filename: /var/log/trickster/example1.access.log
      format: '%h %u %t "%r" %>s %b %{ms}T %{cache-status}x'

Supported tokens:

Trickster-specific values use the %{key}x extension namespace:

Missing values render as -. Values derived from the request (like headers and usernames) are backslash-escaped so they cannot corrupt the log line structure. Unknown tokens fail validation at startup.

The json preset emits these fields per line: time, client_ip, user, method, path, query, proto, status, bytes, duration_ms, host, referer, user_agent, backend, provider, path_config, cache_status, engine.

Rotation and Retention

Access and error logs are rotated and pruned automatically, using nginx/logrotate-style numbered archives (example1.access.log.1.gz is the most recent archive, .2.gz the next, and so on).

    access_log:
      filename: /var/log/trickster/example1.access.log
      rotation:
        size: 256MB   # rotate when the live file would exceed this size (default 256MB)
        interval: 1d  # also rotate when the live file is older than this (default off)
      retention:
        count: 3      # keep at most 3 archives (default 80)
        age: 7d       # also prune archives older than this (default 7d)
      compress: true  # gzip archives (default true)

• size and interval may be combined; the log rotates when either threshold is reached. Setting both to 0 disables rotation.
• Sizes accept KB, MB, GB and TB suffixes (binary multiples), or a plain byte count.
• retention.count: 0 disables count-based pruning and keeps all archives.
• Writes are buffered for up to one second or 64 KiB. A process or machine crash can lose the buffered tail; an orderly shutdown flushes it.
• Interval rotation keeps its epoch in a <filename>.rotation sidecar so a restart does not reset the interval clock.

Archives created by older Trickster releases use timestamped lumberjack names and are not included in numbered-archive retention. They form a bounded legacy set and may be removed manually after upgrading.

When upgrading from the original logging implementation, note these filename and retention changes:

• retention.count: 0 now keeps all archives; configure a positive count to bound archive retention.
• With main.instance_id enabled, filenames without a .log suffix now also include the instance ID (trickster.out becomes trickster.2.out). Update log shippers that still follow the unsuffixed filename.

The same rotation, retention and compress options are also available in the main logging: config section to control rotation of the Trickster application log, with the same defaults.

Error Log Settings

Each error_* option inherits its value from the corresponding access log option when unset:

    access_log:
      filename: /var/log/trickster/example1.access.log
      format: combined
      error_filename: /var/log/trickster/example1.error.log
      error_format: ''      # default: inherits format
      error_threshold: 400  # log responses with status >= this (default 400)
      error_rotation:       # default: inherits rotation
        size: 64MB
      error_retention:      # default: inherits retention
        count: 7
      error_compress: true  # default: inherits compress

4 - X-Trickster-Result Header

Trickster adds the X-Trickster-Result response header to describe how it handled a request. The header is intended for debugging cache behavior, proxy fallbacks, and partial origin fetches.

Example:

X-Trickster-Result: engine=DeltaProxyCache; status=phit; fetched=[1612804980000-1612808580000]; ffstatus=hit

The header value is a semicolon-separated list of fields. Optional fields are included only when Trickster has a value for them.

Result Statuses

status uses the same values reported in metrics, logs, and tracing. Common examples are:

Proxy-Only Results

status=proxy-only means the response came from the origin through Trickster’s proxy path, without a cache read or cache write for that request. It does not necessarily mean the origin response was wrong.

Common reasons include:

When investigating proxy-only, check the backend provider, any proxy_only setting, request and response Cache-Control headers, and whether the request shape is supported by the configured backend provider.

Fetched And Failed Ranges

fetched and failed describe the ranges Trickster fetched or failed to fetch while serving the response.

Example:

X-Trickster-Result: engine=DeltaProxyCache; status=phit; fetched=[1612804980000-1612808580000;1612812180000-1612815780000]

For time series responses, range values are Unix timestamps in milliseconds. A phit result with fetched ranges usually means Trickster had some of the requested data cached and fetched the missing ranges from the origin.

failed ranges indicate the origin request for those ranges failed. Depending on the proxy engine and fanout behavior, Trickster may return an error response or a partial response with failure metadata.

Fast Forward Status

ffstatus appears on time series responses when the Delta Proxy Cache checks Fast Forward data:

Fast Forward is only relevant for supported time series backends and only when the request is eligible for the latest datapoint optimization.