1 - Prometheus Support

Trickster fully supports accelerating Prometheus, which we consider our First Class backend provider. They work great together, so you should give it a try!

Most configuration options that affect Prometheus reside in the main Backend config, since they generally apply to all TSDB providers alike.

Supported API Endpoints

Trickster supports the full Prometheus HTTP API (v1), including features introduced in Prometheus 3.x.

Cached Endpoints

EndpointCache StrategyScatter/Gather Merge
/api/v1/query_rangeDelta Proxy CacheYes
/api/v1/queryObject Proxy CacheYes
/api/v1/seriesObject Proxy CacheYes
/api/v1/labelsObject Proxy CacheYes
/api/v1/label/<name>/valuesObject Proxy CacheYes
/api/v1/alertsProxy + MergeYes
/api/v1/targetsObject Proxy CacheNo
/api/v1/targets/metadataObject Proxy CacheNo
/api/v1/rulesObject Proxy CacheNo
/api/v1/alertmanagersObject Proxy CacheNo
/api/v1/status/*Object Proxy CacheNo
/api/v1/query_exemplarsObject Proxy CacheNo
/api/v1/metadataObject Proxy CacheNo
/api/v1/format_queryObject Proxy CacheNo
/api/v1/parse_queryObject Proxy CacheNo
/api/v1/scrape_poolsObject Proxy CacheNo
/api/v1/featuresObject Proxy CacheNo

Range Query Notes

  • A step may be fractional seconds (for example 1.5 or 0.5), as Prometheus accepts; Trickster keeps the fraction and sends sub-second start and end values with millisecond precision.
  • A query_range whose expression uses the @ start() or @ end() modifier is proxied without caching. Those modifiers resolve against each request’s own range, so results fetched for part of a range could not be combined. A fixed @ <timestamp> is cached normally.

Step Alignment

Range queries default to the partial_end step alignment mode: points on the step grid from the grid point at or before start, plus Trickster’s Fast Forward point at the current time when the range reaches it. Fast Forward’s point is fetched through the object cache for partial_bucket_ttl, and skipped when the step is no longer than that.

  • truncate is the same without Fast Forward. It is the default with fast_forward_disable: true, which can’t be set alongside step_alignment.
  • drop starts at the grid point at or after start, so no point falls before the requested range.
  • off sends each range query as the client sent it, and caches the response as an object for one minute.
  • partial and partial_start aren’t supported: PromQL evaluates at instants, so there is no partial bucket at the start.

A query can choose its own mode, or turn Fast Forward off, with a comment: up # trickster-step-align:drop or up # trickster-fast-forward:off. See Per-Query Instructions.

Proxied Endpoints (not cached)

EndpointNotes
/api/v1/notifications/liveSSE streaming
/api/v1/writeRemote write (v1 and v2)
/api/v1/otlp/v1/metricsOTLP ingestion
/api/v1/admin/*Explicitly unsupported (returns error)

All other /api/v1/* paths are reverse-proxied to the origin without caching.

Prometheus 3.x Features

  • Native histograms are fully supported in query and query_range responses, including mixed series with both float samples and histogram samples.
  • UTF-8 metric and label names (e.g., {"metric.name"}) are supported in queries and cache keys.
  • Query stats (stats=all parameter) are cache-key differentiated, so responses with and without stats are cached separately.

Flavors

prometheus.flavor adapts a backend to a Prometheus-compatible managed service, applying its supported routes, limits and defaults.

flavorservice
cloudwatchAmazon CloudWatch PromQL
ampAmazon Managed Service for Prometheus
mimirGrafana Mimir
thanosThanos
backends:
  cloudwatch:
    provider: prometheus
    prometheus:
      flavor: cloudwatch
    sigv4:
      region: us-east-1

The cloudwatch and amp flavors limit the backend to the routes the service supports and answer every other path themselves. See AWS Integration for what each one sets. The mimir and thanos flavors change no routes; they select the series source that TSM global evaluation reads, and that evaluation reads their newest 5 minutes of samples again each time, since their replicas and ingesters may still be receiving them.

Series Sources

An ALB using TSM global evaluation reads raw series from its members rather than query results. series_api selects how a backend serves them, and series_endpoint where:

series_apiprotocoldefault for flavorseries_endpoint
remote_readPrometheus remote read over HTTP, as streamed chunks or sampled responsesmimiroptional; an absolute http or https URL. The default is the origin_url path plus /api/v1/read
storeapiThanos StoreAPI Series over gRPCthanosrequired; grpc://host:port for plaintext, or grpcs://host:port for TLS with the backend’s tls client settings

Any Prometheus backend can set series_api, since Prometheus itself serves remote read. A StoreAPI endpoint can be a Thanos Sidecar, Store Gateway, Receive or Querier. The cloudwatch and amp flavors serve no raw series, and reject both options.

backends:
  prom-a:
    provider: prometheus
    origin_url: http://prom-a:9090
    prometheus:
      series_api: remote_read
  mimir:
    provider: prometheus
    origin_url: http://mimir:8080/prometheus # remote read at /prometheus/api/v1/read
    prometheus:
      flavor: mimir
  thanos-a:
    provider: prometheus
    origin_url: http://prom-a:9090
    prometheus:
      flavor: thanos
      series_endpoint: grpc://prom-a-sidecar:10901

Remote read uses the backend’s HTTP client settings, and sends the client’s end-to-end request headers plus the request_headers the backend’s matching path sets, as a proxied request would; a tenant header such as X-Scope-OrgID, or an Authorization header, reaches the source this way. StoreAPI calls carry the same headers as gRPC metadata.

Both transports ask for compressed answers, which mostly shrinks the label sets the chunks travel with. Remote read asks for gzip, which Mimir answers with; Prometheus answers remote read uncompressed. StoreAPI calls ask for snappy, as Thanos components support; a store that rejects it is read again at once without compression. After that read succeeds, compression stays disabled until Trickster restarts or reloads.

A backend opens its StoreAPI connection on its first read, and closes it once a reload, shutdown, or autodiscovery replaces the backend and its requests have drained, as described for global evaluation.

Injecting Labels

Trickster can inject labels on a per-backend basis into Prometheus responses before returning them to the caller.

Here is the basic configuration for adding labels:

backends:
  prom-1a:
    provider: prometheus
    origin_url: http://prometheus-us-east-1a:9090
    prometheus:
      labels:
        datacenter: us-east-1a

  prom-1b:
    provider: prometheus
    origin_url: http://prometheus-us-east-1b:9090
    prometheus:
      labels:
        datacenter: us-east-1b

Interaction with ALB Merge Strategy

When using label injection with an ALB configured for Time Series Merge, injected labels are automatically stripped from responses before merging. This ensures that series from different backends are aggregated correctly, and the injected labels do not appear in the final response to the caller. See the ALB Merge Strategy documentation for details.

Max Query Range Limitation

Trickster supports enforcing a max_query_range limit on Prometheus backends. For details on how to configure and use query range limits, see the Query Range Limits documentation.

2 - ClickHouse Support

Trickster will accelerate ClickHouse queries that return time series data normally visualized on a dashboard. Acceleration works by using the Time Series Delta Proxy Cache to minimize the number and time range of queries to the upstream ClickHouse server.

Scope of Support

Trickster is tested with the official ClickHouse DataSource Plugin for Grafana v4.21.1 and supports acceleration of queries constructed by this plugin using its built-in time macros like $__fromTime and $__toTime. Delta caching with this an other plugins (e.g., Altinity/Vertamedia) depends on the supported query shapes below. Trickster also supports several other query formats that return “time series like” data.

Trickster also supports the ClickHouse Go SDK (clickhouse-go/v2) over its HTTP and Native protocols, including clickhouse.OpenDB.

Native Binary Protocol Support

Inbound and upstream protocols are configured independently:

  • A named listener with protocol: clickhouse accepts Native client connections.
  • A ClickHouse backend with protocol: native uses the Native protocol for its origin. An empty protocol or protocol: http uses HTTP.
listeners:
  default:
    protocol: http
    port: 8480
  clickhouse-native:
    protocol: clickhouse
    port: 8487

backends:
  click1:
    provider: clickhouse
    origin_url: http://clickhouse:8123
    listener_names: [default, clickhouse-native]
    cache_name: default

This exposes one HTTP route at /click1/ and one Native listener on port 8487, backed by the same cache and HTTP ClickHouse origin. A Native listener must map to exactly one backend; ALB and user-router multiplexing are not supported.

To use a Native origin instead, set the backend protocol and point origin_url at ClickHouse’s Native port:

backends:
  click1:
    provider: clickhouse
    origin_url: http://clickhouse:9000
    protocol: native

All four HTTP/Native ingress and HTTP/Native origin combinations are supported. Native credentials, database, query settings, parameters, and query IDs are forwarded to the origin. Native identity pools are bounded to 64 credential/database combinations per backend.

TLS

Native listener TLS uses the backend’s server certificate. Set require_tls: true and provide both certificate paths:

listeners:
  clickhouse-native-tls:
    protocol: clickhouse
    port: 9441
    tls_watch_interval: 30s

backends:
  click1-tls:
    provider: clickhouse
    origin_url: http://clickhouse:8123
    listener_names: [clickhouse-native-tls]
    require_tls: true
    tls:
      full_chain_cert_path: /etc/trickster/tls/server.crt
      private_key_path: /etc/trickster/tls/server.key

Certificate changes are hot-swapped. Since require_tls applies to every binding of a backend, use a separate backend entry when plaintext and TLS listeners must coexist.

For a TLS Native origin, use an https origin URL with the Native protocol. The common backend TLS options configure origin verification and optional mutual TLS:

backends:
  click1:
    provider: clickhouse
    origin_url: https://clickhouse:9440
    protocol: native
    tls:
      certificate_authority_paths: [/etc/trickster/tls/origin-ca.crt]
      client_cert_path: /etc/trickster/tls/client.crt
      client_key_path: /etc/trickster/tls/client.key

Native Limitations

The Native listener supports SELECT queries, ping/pong, revision 54460 framing, structured errors, and LZ4 compression. INSERT, external tables, and session-changing USE or SET statements are rejected; select a database and settings per request. ClientCancel is accepted as a no-op and does not asynchronously cancel an active upstream request.

Supported native protocol data types: all integer types (8–256 bit), Float32/64, String, FixedString(N), DateTime, DateTime64, Date, Date32, UUID, IPv4, IPv6, Enum8/16, Bool, Nullable(T), Array(T), Map(K,V), Tuple(T1,T2,…), LowCardinality(T), and Decimal.

Delta-cacheable query results are decoded into Trickster’s dataset model and re-encoded in the client’s requested format. Both the TSV and the FORMAT Native origin readers support every scalar type above, Nullable(T), LowCardinality(T), Array(T), Map(K, V) and Tuple(...) (including named elements and nesting); compound values are carried in ClickHouse’s text-literal form ([1,'a'], {'k':1}, ('a',1)) and parsed back when re-encoding to Native. Nested, Variant, Dynamic and JSON columns are rejected with an explicit error on the delta path rather than decoded incorrectly; such queries should use a non-delta-cacheable shape.

Native wire clients receive Native blocks. HTTP clients using a Native origin may request JSON, Native, CSV, or TSV-family output; compound columns in CSV/TSV and unsupported formats return an error. The modeler recognizes Native origin responses through the X-ClickHouse-Format response header, and HTTP Native framing honors client_protocol_version.

Trickster parses incoming ClickHouse statements into a full abstract syntax tree using the AfterShip ClickHouse SQL parser, then applies its own semantic analysis to determine whether a query is eligible for time series delta caching and, if so, its timestamp column, bucket cadence, time range, grouping tags, and cache identity. The cache key is derived from a canonical form of the query in which the requested time range is replaced with placeholders, so requests for different time ranges of the same logical series share one delta cache entry.

Trickster’s analysis fails closed: a valid query whose shape cannot be proven safe for delta caching is never rewritten approximately. It is instead served through the Object Proxy Cache (OPC) or proxied directly, and the classification reason is exported through the trickster_sql_query_analysis_total metric.

If you find query or response structures that are not yet supported, or providing inconsistent or unexpected results, we’d love for you to report those. We also always welcome any contributions around this functionality.

Delta-Cacheable Queries

To be eligible for the delta cache, a query must be a single SELECT statement containing a recognized time-bucketing expression in its select list, a supported time range in its WHERE or PREWHERE clause, and a GROUP BY clause that includes the time bucket. Each requirement is described below.

Time-Bucketing Expressions

Exactly one select-list expression must match a supported bucket form:

Grafana Plugin Format

SELECT intDiv(toUInt32(time_col), 60) * 60 [* 1000] [AS alias]

This is the approach used by the Grafana plugin. The argument to the ClickHouse intDiv function is the step value in seconds, since the toUInt32 function on a datetime column returns the Unix epoch seconds. An optional output multiplier of 1000, 1000000, or 1000000000 selects millisecond, microsecond, or nanosecond output timestamps.

ClickHouse Time Grouping Functions

SELECT toStartOfInterval(time_col, INTERVAL n unit) [AS alias]

with a positive constant n and a unit of millisecond, second, minute, hour, day, or week; or one of the fixed-period functions:

toStartOfNanosecond
toStartOfMicrosecond
toStartOfMillisecond
toStartOfSecond
toStartOfMinute
toStartOfFiveMinute
toStartOfTenMinutes
toStartOfFifteenMinutes
timeSlot
toStartOfHour
toStartOfDay
toStartOfWeek
toMonday

date_trunc('unit', time_col) and dateTrunc support the same fixed units. toStartOfWeek uses ClickHouse’s Sunday phase while toMonday and weekly date_trunc use Monday. Calendar-length month, quarter, and year buckets are served through the OPC.

The time column may be a plain, qualified (table.col), or quoted identifier, optionally wrapped in toDateTime, toInt32, or toUInt32. Integer constants defined in a scalar WITH clause may be used for the step value. Timezone-parameter variants of these functions (for example toStartOfHour(time_col, 'America/Denver')) are not eligible for delta caching and are served through the OPC. The current SQL parser does not accept INTERVAL MICROSECOND or INTERVAL NANOSECOND; use the corresponding fixed toStartOf... function.

Determining the Requested Time Range

Time range predicates must appear in a top-level AND conjunction of the WHERE or PREWHERE clause. Predicates joined by OR or negated with NOT make the query ineligible for delta caching.

Two predicate targets are supported, with different rules:

  • The raw time column (the column inside the bucket function): the lower bound must be inclusive (>=, or the lower end of BETWEEN); a strict > is served through the OPC. The upper bound may be exclusive (<) or inclusive (<=, or the upper end of BETWEEN). Values that do not fall on bucket boundaries — such as the live ranges produced by Grafana’s $__fromTime and $__toTime macros — leave partial buckets at the edges, which are never cached as complete aggregates; the backend’s step alignment mode decides what the response shows for them. Under the default, drop, they are left out, and an inclusive upper bound leaves out the bucket that contains it, because that bucket is only partly covered. A range with no complete bucket is sent to ClickHouse as written, and its response is cached as an object for partial_bucket_ttl.
  • The bucket alias (the output of the bucket expression): >, >=, <, <=, and BETWEEN are all supported, because bucket outputs are discrete; Trickster aligns each comparator to the first and last included bucket.

Bound values may be expressed as epoch integers, ClickHouse string dates in the form 2006-01-02 15:04:05 (or date-only, or RFC3339), toDateTime(n), toDateTime64(n, precision), or toDate(n) wrappers, WITH-clause constants, or now()/now64() with optional addition or subtraction of seconds. DateTime64 precision is retained. Floating epoch bounds and timezone-qualified conversions such as toDateTime(n, 'America/Denver') are not eligible.

If no upper bound is present, Trickster inserts a safe upper bound into origin requests automatically and caches every complete bucket up to the current time. The still-filling bucket is never cached: under the default step_alignment, drop, the response ends before it, and the partial and partial_end modes fetch it from the origin on each request.

Examples of delta-cacheable time range clauses (for a one-minute bucket cadence):

WHERE t >= '2020-10-15 00:00:00' AND t <= '2020-10-16 12:00:00'  -- bucket alias
WHERE t BETWEEN 1574686320 AND 1574689920                        -- bucket alias
WHERE time_col >= toDateTime(1574686320) AND time_col < toDateTime(1574689920)
WHERE t >= now() - 3600 AND t < now()                            -- bucket alias

Secondary date-range predicates whose values match the primary range — such as the Date-typed partition filters emitted by the Grafana plugin — are recognized and rewritten in step with the primary range.

Grouping and Result Shape

The GROUP BY clause must include the time bucket (by alias or by its full expression), and every non-aggregate column in the select list must also be grouped. Grouped columns become the series tags in the cached time series. Queries using GROUP BY ... WITH CUBE/ROLLUP, grouping on expressions that are not selected, or leaving a selected dimension ungrouped are served through the OPC.

Queries whose values in one time bucket can depend on other buckets, or on the whole result, are also served through the OPC:

  • LIMIT, LIMIT BY and TOP;
  • window functions (OVER (...) or a WINDOW clause) and cross-row functions such as neighbor, lagInFrame, leadInFrame and the running* family;
  • WITH TOTALS and ORDER BY ... WITH FILL (including INTERPOLATE);
  • a query-level SETTINGS clause;
  • subqueries, common table expressions and joins, which can carry time filters of their own (ARRAY JOIN is still delta-cacheable).

Ordering

Trickster rebuilds a delta-cached response from its cached buckets in ascending time order. A query is therefore delta-cacheable only with no ORDER BY, or with a single ascending term on the time bucket (by alias, by the bucket expression, or by its position, as in Grafana’s ORDER BY time). Any other ordering — descending time, additional terms, or other columns — is served through the OPC, where the origin’s row order is kept.

Output Formats

Delta-cacheable queries may specify FORMAT JSON, CSV, CSVWithNames, TabSeparated (TSV), TabSeparatedWithNames, or TabSeparatedWithNamesAndTypes, or omit the FORMAT clause. Trickster requests TSVWithNamesAndTypes from the origin and re-marshals cached data into the client’s requested format.

Trickster writes each format as ClickHouse writes it:

  • NULL is \N in TSV and CSV, and null in JSON. An empty string stays an empty string.
  • Numbers are bare JSON numbers, and floats use ClickHouse’s text (1e21, 1e-7, nan, inf). JSON writes NaN and infinities as null.
  • CSV quotes every text value. A FixedString is padded to its width.
  • Compound values: Array, Map and Tuple values are their ClickHouse literals in TSV and CSV, and JSON arrays and objects in JSON.

Responses honor output_format_json_quote_64bit_integers, output_format_json_quote_decimals, output_format_json_quote_denormals and date_time_output_format.

Time Zones

A DateTime without a zone in its type is written in the session’s time zone: the request’s session_timezone URL parameter, or else the server’s. Trickster caches UTC and writes each response in its client’s zone, so clients in different zones share cache entries.

The settings that change only how a response is written aren’t part of the cache key: date_time_output_format, the JSON quote settings, default_format, and client_protocol_version.

The zone is part of the key only when it changes which rows a query returns:

  • Buckets that start on the zone’s clock: a daily bucket starts at the zone’s midnight, and an hourly bucket in a zone offset by a half hour starts on the half hour. The zone is keyed unless every offset it has over the queried range is a whole number of buckets. So hourly and finer buckets in New York or UTC share entries; Kolkata hourly buckets, and daily buckets outside UTC, don’t.
  • Zone-sensitive expressions: another date or time function that reads the zone, such as toHour() or today(), or text compared as a time.

A query whose time bounds are text (ts >= '2026-09-29 00:00:00') or dates (toDate(…)) is read by ClickHouse in the session’s zone. Outside UTC it’s served through the OPC rather than delta-cached.

Trickster asks the origin for date_time_output_format=iso, which writes every DateTime as UTC. Its cache holds UTC, without the ambiguity of the hour a clock repeats when daylight saving time ends. It writes its DateTime64 range bounds with an explicit 'UTC' zone, so the session’s zone doesn’t move them.

Trickster learns the server’s zone from the X-ClickHouse-Timezone header of the origin’s responses. A backend that hasn’t seen one yet asks the server with SELECT timezone() before it caches a query. If that probe fails, for example because the origin requires credentials, Trickster proxies the request uncached to learn the zone from the response. If the zone is still unknown after that, Trickster assumes UTC and logs a warning.

Non-Time-Series Queries

Queries that are not cacheable as time series — such as LIMIT-based queries, queries with set operations (UNION, EXCEPT, INTERSECT), SELECT 1 health checks, or SDK handshake requests — are transparently proxied to the upstream ClickHouse server. These requests are cached using the Object Proxy Cache (OPC).

Cache Keys

Both the OPC and the delta proxy cache key a request on its SQL statement and on every other URL parameter, because query parameters (param_<name> values for {name:Type} placeholders) and settings can change the result. The native listener forwards its query parameters and settings the same way. Only transport parameters that cannot change the result are left out of the key: query_id, session_timeout, session_check, send_progress_in_http_headers, http_headers_progress_interval_ms, wait_end_of_query, buffer_size, log_comment, log_queries, quota_key, and add_http_cors_header.

Requests that carry a session_id are proxied without caching. A session’s SET statements change the results of later queries in that session, and that state is not part of any cache key.

Health and Ping Endpoint

Trickster exposes a /ping endpoint that returns a health check response, matching the endpoint provided by ClickHouse itself. This enables compatibility with clients and SDKs that probe /ping during connection initialization.

Step Alignment

ClickHouse supports every step alignment mode, and defaults to drop: responses hold complete buckets only, so the partial buckets at the edges of a live range, and the still-filling bucket at the present, are left out. truncate answers the whole first bucket instead.

partial, partial_start and partial_end add the partial buckets as ClickHouse computes them over the client’s own range. Each is a small query of its own, sent through the object cache and kept for partial_bucket_ttl, never in the time series cache, so these modes cost up to two extra origin queries per request. Fast Forward doesn’t apply to ClickHouse; partial_end shows the still-filling bucket instead.

Complete buckets inside the configured volatile_window are refetched until they settle. A query can choose its own mode or window with a comment directive, such as /* trickster-step-align:partial_end */; see Per-Query Instructions. Directives in # comments aren’t read, because Trickster’s ClickHouse SQL parser rejects them: a statement with one is served through the OPC, or proxied.

Observability

Query classification outcomes are exported through two low-cardinality metrics that never include query text:

  • trickster_sql_query_analysis_total — labeled by backend, dialect, cache mode (delta, object, none), and a stable reason code such as delta_cacheable, unsafe_predicate, or unsupported_bucket.
  • trickster_sql_query_rewrite_failures_total — counts failures to render an origin request from a cached query plan.

With debug logging enabled, classification decisions are also logged with the same structured reason codes.

Max Query Range Limitation

Trickster supports enforcing a max_query_range limit on ClickHouse backends. For details on how to configure and use query range limits, see the Query Range Limits documentation.

3 - InfluxDB Support

Trickster provides support for accelerating InfluxDB queries that return time series data normally visualized on a dashboard. Acceleration works by using the Time Series Delta Proxy Cache to minimize the number and time range of queries to the upstream InfluxDB server.

Scope of Support

Trickster is tested with the built-in InfluxDB DataSource Plugin for Grafana v5.0.0.

Trickster uses InfluxDB-provided packages to parse and normalize queries for caching and acceleration. If you find query or response structures that are not yet supported, or providing inconsistent or unexpected results, we’d love for you to report those so we can further improve our InfluxDB support.

Trickster supports integrations with InfluxDB 1.x, 2.x, and 3.x.

Prometheus Remote Read

For InfluxDB 1.x, Trickster accelerates Prometheus remote-read requests sent to POST /api/v1/prom/read. Point Prometheus at the same endpoint on Trickster and retain the InfluxDB query parameters, for example:

remote_read:
  - url: http://trickster:8480/api/v1/prom/read?db=metrics&rp=autogen

The InfluxDB endpoint supports one query per request and the Prometheus SAMPLES response type. Trickster delta-caches requests with that shape. Other request shapes continue through the normal proxy path so that InfluxDB remains responsible for its response and error behavior.

Raw remote-read samples do not advertise a guaranteed interval, so point-count sharding cannot split their extents without risking gaps. Requests use the normal proxy path when shard_max_size_points is enabled; time-based sharding remains supported. Cache retention (oldest and lru) and volatile_window_points use a positive hints.step_ms; a request without that hint uses the normal proxy path, including Prometheus instant queries with a zero step hint. The hint is retained in serialized cache entries and does not change the 1 ms precision used to locate missing raw samples.

Cacheable remote reads request Snappy directly from InfluxDB to avoid an extra HTTP compression layer. Reconstructed responses carry Content-Encoding: snappy on cache misses, partial hits, and full hits.

The db, rp, u, and p query parameters and the Authorization header are part of the cache identity. They are forwarded unchanged unless a path-level request rewrite or header configuration overrides them.

InfluxDB 3.x Support

Trickster supports InfluxDB 3.x via both the native v3 API endpoints and the v1/v2 compatibility endpoints.

Supported v3 Endpoints

  • GET/POST /api/v3/query_sql — SQL queries with delta-proxy caching
  • GET/POST /api/v3/query_influxql — InfluxQL queries with delta-proxy caching
  • POST /api/v3/write_lp — line protocol writes (proxied, not cached)

All other v3 API paths (including the /api/v3/configure/* management endpoints, with any HTTP method) are proxied through untouched.

Query requests on both query endpoints may arrive as URL parameters (GET), an application/json document ({"q": ..., "db": ..., "format": ..., "params": ...}), a form-encoded body, or a raw statement body — all four shapes are parsed and cached equivalently, and the db and params values participate in the cache identity.

InfluxQL over v3

/api/v3/query_influxql requests use the same delta-proxy caching as the v1 /query endpoint (queries with a GROUP BY time(...) interval, optionally with an offset such as time(1h, 15m), and a time-bounded WHERE clause; a non-UTC tz() clause shifts bucket boundaries by the zone offset, so those queries use the object proxy cache instead), but speak the v3 request/response shapes: the v3 request document above and the v3 tabular response formats, including the iox::measurement column, which Trickster treats as a series tag alongside any GROUP BY tags.

On both /query and /api/v3/query_influxql, statements whose values in one time bucket depend on other buckets or on the whole result use the object proxy cache instead of delta caching: fill(previous) and fill(linear); transformations such as derivative(), non_negative_derivative(), difference(), moving_average(), cumulative_sum(), elapsed(), integral() and the other technical analysis functions; LIMIT, OFFSET, SLIMIT and SOFFSET; and subqueries, whose own time ranges are not rewritten per fetch.

SQL Query Caching

SQL queries using date_bin() or date_trunc() for time binning are parsed for time range extraction and step detection, enabling delta-proxy caching. Trickster extracts time ranges from WHERE clauses and intervals from date_bin(INTERVAL '...', time) or date_trunc('unit', time) in SELECT. Live, unaligned time ranges (the shape dashboard tools emit) are rounded inward to complete buckets. Grouped queries (GROUP BY 1, host) are cached per tag series.

Example query that Trickster will accelerate:

SELECT date_bin(INTERVAL '1 hour', time) AS time, avg(temperature)
FROM weather
WHERE time >= '2024-01-01 00:00:00' AND time < '2024-01-02 00:00:00'
GROUP BY 1

SELECT queries that cannot be delta-cached (no fixed-cadence time bucket, joins, subqueries, window functions, compound selects such as UNION, LIMIT, variable-length buckets like '1 month', or unsafe time predicates) fall back to the object proxy cache, which caches the whole response briefly and passes results through unchanged. Non-SELECT statements and parameterized queries (a params field in the request) are proxied to the origin without delta caching.

The still-filling final bucket of a range that reaches the present is never cached. Under the SQL default step_alignment, drop, the response ends before it; the partial and partial_end modes, and InfluxQL’s default partial_end, fetch it from the origin on each request. volatile_window applies only to complete buckets.

Step Alignment

Each query language has its own step alignment support and default:

Query languageSupported modesDefault
InfluxQL, 1.x and over 3.xallpartial_end
SQL, over HTTP and Flight SQLalldrop
Fluxtruncate, offtruncate
Prometheus remote readtruncatetruncate

A backend’s step_alignment may name any of these modes; a query in a language that doesn’t support it runs in that language’s default, and the fallback is counted in trickster_step_alignment_fallbacks_total. The partial modes fetch each partial bucket as a small query of its own through the object cache, so they cost up to two extra origin queries per request. A query can choose its own mode with a comment: -- trickster-step-align:drop in InfluxQL and SQL, // trickster-step-align:off in Flux. See Per-Query Instructions.

Response Formats

Trickster supports the following v3 response formats, controlled by the format query parameter (in the URL or the request document) or, when no format is given, the Accept header (application/json, application/jsonl, text/csv, …):

  • json (default) — JSON array of objects
  • jsonl — JSON Lines (one JSON object per line)
  • csv — standard CSV with header row

The parquet and pretty formats are not supported for caching and will be proxied through.

InfluxDB 3 leaves a row’s null values out of JSON and JSON Lines, so a column can first appear on any row. Trickster reads every column a response names, on whichever row first names it.

v1/v2 Compatibility

InfluxDB 3.x ships with v1 and v2 compatibility endpoints. Trickster’s existing InfluxQL support works against these endpoints with no additional configuration — just point Trickster at the v3 instance and query via /query.

Flight SQL (gRPC)

InfluxDB 3.x exposes SQL via Apache Arrow Flight SQL on gRPC in addition to HTTP. Grafana’s InfluxDB datasource in SQL mode, the Python/Rust/Java SDKs, and ADBC all default to Flight SQL, so HTTP-only caching misses a significant fraction of real-world query traffic. The protocol-level (backend-agnostic) listener documentation lives in Apache Arrow Flight SQL Listeners; this section covers the InfluxDB specifics.

Trickster can expose a Flight SQL server that proxies to an upstream Flight SQL endpoint and caches the Arrow IPC byte stream. Enable it by defining a listener with the flight-sql protocol and mapping exactly one InfluxDB backend to it (alongside its usual HTTP listener):

listeners:
  influx3-flight:
    protocol: flight-sql   # Apache Arrow Flight SQL over gRPC
    port: 8485
backends:
  influx3:
    provider: influxdb
    origin_url: 'http://influxdb3:8181/'
    listener_names: [default, influx3-flight]
    influxdb:
      flight_upstream_address: 'influxdb3:8181'   # optional, defaults to origin_url host
      flight_max_response_bytes: 134217728        # optional, defaults to 128MiB
      flight_max_buffered_bytes: 536870912         # optional, aggregate in-flight budget; defaults to 512MiB
      flight_allowed_location_hosts:               # optional exact authorities for alternate endpoints
        - 'influxdb3-replica:8181'

The authorization and database headers are forwarded from the client through to the upstream, and every cache entry is keyed by the backend name, the database/bucket-name headers, and a hash of the authorization header — so tenants and databases never share cache entries. Requests that send no authorization header share one anonymous cache scope, mirroring the access the upstream would grant them.

Statement queries are served through a three-tier cache:

  1. Delta proxy cache — queries the SQL analyzer classifies as delta-cacheable (the same date_bin()/date_trunc() shapes as the HTTP path) are cached by time extent: repeat and overlapping queries fetch only the missing sub-ranges from the upstream, and responses are rebuilt into Arrow record batches conforming to the response’s original schema. A query’s ORDER BY is carried through that rebuild, so a cache hit returns rows in the requested order; ordering terms that do not resolve to a select-list output fall to the next tier. Entries use the backend’s timeseries_ttl and honor volatile_window. Under the default step_alignment, drop, a range holding at least one complete bucket returns only complete buckets: partial buckets at either end of the range, including the still-filling bucket of a range that reaches the present, are left out. The partial, partial_start and partial_end modes fetch those buckets from the upstream over the client’s own range, through the object cache for partial_bucket_ttl, and never delta-cache them. A range with no complete bucket gets the upstream’s own result for the statement, partial and still-filling buckets included, cached as an object for partial_bucket_ttl. Responses whose Arrow schemas the delta model cannot represent (nested types, non-string dictionaries, …) automatically fall to the next tier.
  2. Object cache — everything else cacheable is stored as the verbatim Arrow IPC byte stream, returned byte-identically, with a lifetime of influxdb.flight_cache_ttl (default 60s). Metadata RPCs and prepared statements always use this tier.
  3. Proxy — statements referencing nondeterministic functions (now(), current_timestamp, random(), …) and non-SELECT statements are never cached.

Flight SQL TLS

The Flight listener serves TLS when its mapped backend’s tls block presents a certificate and key (the same mechanism HTTP listeners use); certificate rotation applies on config reload without dropping the listener. Without a certificate the listener serves plaintext gRPC. The upstream dial uses TLS when influxdb.flight_upstream_tls is set, honoring the backend tls block’s insecure_skip_verify.

Unsupported Flight SQL RPCs

Trickster proxies queries, result-set schema requests (GetSchemaStatement/GetSchemaPreparedStatement, which ADBC drivers probe on connect), the metadata RPCs below, and the prepared-statement lifecycle. Other Flight SQL RPCs — writes/updates/ingest (ExecuteUpdate, DoPut ingestion), transactions, savepoints, Substrait plans, query cancellation, endpoint renewal, and session options — return gRPC Unimplemented; clients requiring them should connect to InfluxDB directly. A prepared-statement binding must arrive as a single record batch; a client streaming several batches for one binding receives Unimplemented rather than a binding taken from its first batch alone.

Flight SQL listeners share Trickster’s standard listener lifecycle: connection limits (connections_limit), graceful drain on SIGTERM (active streams are drained until the configured drain timeout, then closed), and config reload (SIGHUP) — a reload with an unchanged backend configuration keeps serving on the existing socket, while a changed configuration drains the old server and rebinds.

Metadata RPCs

ADBC clients (Grafana’s SQL datasource, the Python adbc_driver_flightsql, etc.) probe the following RPCs on connect to populate schema browsers and query editors. Trickster proxies these to the upstream and caches the Arrow IPC response:

  • GetFlightInfoTables / DoGetTables
  • GetFlightInfoCatalogs / DoGetCatalogs
  • GetFlightInfoSchemas / DoGetDBSchemas
  • GetFlightInfoTableTypes / DoGetTableTypes
  • GetFlightInfoSqlInfo / DoGetSqlInfo
  • GetFlightInfoXdbcTypeInfo / DoGetXdbcTypeInfo
  • GetFlightInfoPrimaryKeys / DoGetPrimaryKeys
  • GetFlightInfoImportedKeys / DoGetImportedKeys
  • GetFlightInfoExportedKeys / DoGetExportedKeys
  • GetFlightInfoCrossReference / DoGetCrossReference

Flight SQL response size

Flight responses are buffered whole so they can be cached. influxdb.flight_max_response_bytes bounds one upstream response (128MiB by default), while influxdb.flight_max_buffered_bytes bounds response bytes concurrently assembled or streamed (512MiB by default). Exceeding either bound returns gRPC ResourceExhausted; a negative value removes that bound.

A FlightInfo partitioned across several endpoints is consumed in full. Alternate endpoint locations are followed only when their exact host:port authority appears in influxdb.flight_allowed_location_hosts, because the request’s authorization and database metadata are forwarded to that host. A TLS upstream never follows a plaintext alternate location. influxdb.flight_max_location_clients bounds cached alternate connections and defaults to 16.

Prepared Statements

Trickster proxies the full prepared statement lifecycle: CreatePreparedStatement, DoPutPreparedStatementQuery (parameter binding), DoGetPreparedStatement, ClosePreparedStatement. A parameterless prepared statement is equivalent to executing its statement text, so it is served through the same three cache tiers as plain statement queries — delta caching included — and shares their cache entries. Executions with bound parameters are cached whole, keyed by the bound parameter hash so two clients running the same statement with different values don’t alias. Prepared statements abandoned by disconnected clients are closed upstream after 15 minutes of inactivity.

Note: InfluxDB 3 Core 3.10 reports a parameter schema at prepare time but does not resolve bound values during query planning (upstream limitation). Parameterless prepared statements work end-to-end; parameterized ones require a newer Core or Enterprise build.

Flux Language Support

Trickster supports the Flux Query Language for general/basic usage with InfluxDB 1.x and 2.x. Flux is not supported in InfluxDB 3.x.

The delta-proxy cache accepts now() as a range() bound and handles queries with aggregateWindow(every: ...) – the common Grafana shape. Multi-table Flux CSV responses (one table per series in the result set) are also read correctly.

InfluxDB answers Flux queries only in annotated CSV. Trickster can also answer them in JSON when the request’s Accept header asks for application/json. Each table’s records are written as objects, and a NaN or infinite value is written as null, as InfluxDB 3’s JSON writes it.

aggregateWindow may also set offset and timeSrc ("_stop", the default, or "_start"); Trickster aligns its fetches to the resulting window boundaries and labels. Queries whose windows cannot be mapped onto a fixed step grid are proxied without delta caching: a bare window() (its records keep their own _time), a period that differs from every, or a location.

Queries whose values in one window depend on other windows, or on the whole range, are also proxied without delta caching:

  • limit(), tail(), derivative(), difference(), cumulativeSum(), movingAverage() and the other moving-average and technical analysis functions, elapsed(), integral(), stateCount() and stateDuration();
  • fill(usePrevious: true);
  • timeShift(), which moves labels off the window grid;
  • reducers, selectors and sort() that follow aggregateWindow(), such as max(), last() or top(), which collapse or reorder every window;
  • more than one from(), as in union and join queries.

Trickster currently does not properly handle schema changes within a response CSV body (e.g., multiple CSVs in the same document with their own #annotation and header rows). We will fully support this use case in a future beta.

Max Query Range Limitation

Trickster supports enforcing a max_query_range limit on InfluxDB backends. For details on how to configure and use query range limits, see the Query Range Limits documentation.

4 - Apache Druid Support

Trickster accelerates eligible Apache Druid native JSON queries with the Time Series Delta Proxy Cache (DPC). Configure druid as the backend provider and point origin_url at a Druid Broker or Router.

backends:
  druid1:
    provider: druid
    origin_url: http://druid-router:8888
    cache_name: default
    volatile_window: 60s
    timeseries_retention_factor: 2048

Native query acceleration

POST /druid/v2 uses delta caching when all of these conditions hold:

  • queryType is timeseries, groupBy, or topN.
  • intervals contains exactly one ISO-8601 half-open interval.
  • granularity has a fixed width:
    • a simple granularity from second through day;
    • a positive duration granularity in milliseconds; or
    • a fixed ISO-8601 period granularity in UTC, such as PT15M or P1D.
  • The selected context does not request an alternate native response shape.

Trickster removes the interval from the logical cache identity and rewrites only missing extents into Druid’s [start,end) form. Druid’s end is exclusive, so the final cached bucket is rendered as extent.End + granularity.

An interval boundary inside a bucket is handled by the backend’s step_alignment, which defaults to partial for native queries: complete buckets come from the delta cache, and each edge bucket is Druid’s answer over the part of it the interval covers, fetched through the Object Proxy Cache for partial_bucket_ttl and never delta cached. A native query can choose its own mode, or its own volatile window, with keys in its context map, such as "trickster-step-align": "drop", which Trickster leaves out of the cache key and Druid ignores. See Step Alignment and Per-Query Instructions.

The response model preserves native timeseries, groupBy, and topN JSON shapes. Grouping dimensions become DataSet tags internally, and per-bucket positions keep groupBy and topN rows in Druid’s order. A descending: true query is cached in time order like any other and written newest first, as Druid writes it. See Response fidelity.

Object-cache fallback

A valid read query that is unsafe for delta merging automatically uses the Object Proxy Cache (OPC) with a one-minute fallback TTL when Druid does not provide explicit freshness headers. This includes:

  • other native query types such as scan, search, segmentMetadata, datasourceMetadata, and timeBoundary;
  • multiple intervals;
  • all, none, week, month, quarter, and year simple granularities;
  • calendar-width periods or period granularities in a non-UTC time zone;
  • timeseries limit, which keeps only the first rows of the whole result;
  • groupBy limits or dimension-first result ordering; and
  • response-changing contexts such as bySegment, serializeDateTimeAsLong, timeseries grandTotal, or groupBy resultAsArray.

The following native context keys are transport controls and are omitted from the cache identity: queryId, sqlQueryId, priority, timeout, and queryDeadline. They remain unchanged in the request sent to Druid. Semantic context keys, including skipEmptyBuckets, remain part of the cache identity.

Druid SQL acceleration

POST /druid/v2/sql uses the same delta cache for a deliberately conservative subset of Druid SQL. The request must be a JSON object using either the default or explicit resultFormat: "object", or resultFormat: "array" with header: true. The statement must be a single-table SELECT that has:

  • one TIME_FLOOR(__time, <fixed UTC period>) bucket expression with an explicit alias;
  • a GROUP BY containing that bucket and every selected dimension; and
  • a complete lower/upper time range on __time (unaligned edges follow the backend’s step_alignment, drop by default for SQL; partial edge buckets are never delta cached). A SQL comment such as -- trickster-step-align:partial chooses the query’s own mode, and wins over the same key in the request’s context.

The shared CockroachDB SQL analyzer canonicalizes the statement and renders each missing extent while preserving the original JSON context on the wire. Literal MILLIS_TO_TIMESTAMP(...) bounds generated by the Grafana Druid plugin are supported. Object rows and header-plus-array rows are converted to the standard Trickster DataSet internally and emitted in their requested shape after cache merging. Other valid SELECT statements and response formats remain safe OPC fallbacks; non-read statements, SQL task requests, and malformed requests are proxied.

Response fidelity

A response built from the cache is written the way Druid writes it:

  • Numbers, objects and arrays are written exactly as Druid wrote them, so a double keeps its form, such as 5319.0 or 9.999999999999999E22.
  • Strings and keys are escaped the way Druid’s JSON writer escapes them.
  • Each row’s members are in Druid’s order. For timeseries and groupBy results, that is the order of Druid’s Java hash maps.
  • Native responses end without a newline, and SQL responses end with one.
  • SQL rows without an ORDER BY follow Druid’s order: by the GROUP BY columns, with the time bucket first unless the GROUP BY ends with it.
  • Rows that tie on an ORDER BY also follow Druid’s order.
  • Nulls sort lowest, as in Druid: first when ascending and last when descending.

Two things can still differ from the response Druid builds for the same request, and both also vary between Druid’s own responses:

  • topN key order. Druid orders a topN row’s keys differently depending on whether its segment cache served the row, so one query can come back with either order. Trickster writes the dimension, then the aggregations, then the post-aggregations.
  • Floating-point sums. Druid adds a bucket’s values in an order that depends on the query’s interval. So the last digits of a doubleSum or floatSum can differ between two Druid queries that cover the same bucket. A cached bucket keeps the value from the query that fetched it.

Route policy

RouteMethodPolicy
/druid/v2POSTDPC when eligible, otherwise OPC or proxy
/druid/v2/sqlPOSTDPC for eligible SQL, otherwise OPC or proxy
/druid/v2/sql/taskPOSTProxy only
/druid/v2/datasources...GETOPC
/status/healthGETHealth probe; expects true
all other routesanyProxy only

SQL ingestion and management endpoints, ALB time-series merging, scan/search delta caching, and Fast Forward are not supported. Fast Forward is disabled for every Druid backend. A 60-second volatile_window is used when the backend does not configure one, so recently ingested buckets can be refreshed before segments settle.

Observability

  • trickster_druid_query_analysis_total counts classifications by backend, cache mode (delta, object, or proxy), and stable reason code.
  • trickster_druid_query_rewrite_failures_total counts failed missing-extent rewrites by backend and fixed failure category.

Neither metric includes query text, datasource names, or request IDs.

5 - Graphite Provider

Trickster accelerates Graphite’s render API with the Delta Proxy Cache: it caches each metric at the origin’s native resolution and, on subsequent requests, fetches only the time ranges it does not already hold.

Graphite makes that harder than most time series databases, because a render request does not say what resolution it will be answered at. Whisper chooses an archive based on how old the query’s from is, and no API exposes a metric’s archive ladder. Trickster therefore learns each ladder by probing the origin, caches what it learns, and accelerates only the requests whose resolution it can predict. Everything else is served correctly through ordinary object caching.

Specify graphite as the provider:

backends:
  graphite1:
    provider: graphite
    origin_url: 'http://graphite.example.com:80'
    cache_name: default

That is the whole minimum configuration. Graphite’s cache sizing differs from every other provider’s, so the provider supplies its own defaults rather than asking operators to discover them; see Sizing for what they are and when to change them.

Compatibility

Tested against graphite-web 1.1.10 with Whisper storage, and written against its source. It should work with any Graphite-protocol origin that serves /render and /metrics/expand, including go-carbon’s carbonserver, graphite-clickhouse and carbonapi, but only graphite-web is verified.

ClientStatus
Grafana’s Graphite data sourceVerified, GET and POST form bodies
Direct /render API clientsVerified
Graphite Composer / dashboard UIProxied, not accelerated (image formats)
Response formatBehavior
jsonAccelerated. Also what Trickster requests upstream.
raw, csv, msgpackAccelerated; rendered from the cached series
png, svg, pdf, pickle, dygraph, rickshawProxied and object-cached

Trickster always requests format=json from the origin regardless of what the client asked for, and renders the client’s format from the cached series. JSON is the only render format that carries the tags object, which Grafana relies on, and it is what makes one cached series able to serve every output format.

Configuration

Every common backend option applies (cache_name, timeseries_ttl, timeseries_retention_factor, volatile_window, max_object_size_bytes, timeout, healthcheck, paths, TLS, authenticators); two of them, max_object_size_bytes and timeseries_retention_factor, take Graphite-specific defaults, as Sizing explains.

Per-path request_headers and request_params apply to the synthetic requests resolution makes (probes, wildcard expansion) as well as to proxied traffic, so an origin behind a static API key or one that needs local=1 works for learning too. What cannot work is per-client upstream identity: resolution state (learned ladders, wildcard expansion tokens, the registry generation) is backend-wide, and synthetic requests carry only the backend’s configured identity. Trickster therefore declines acceleration — the request is served correctly through the object cache, which keys on the client’s Authorization and every declared result-affecting input — whenever a render request carries an Authorization header, a header named in the render path’s cache_key_headers, or a parameter or form field named in its cache_key_params/cache_key_form_fields (such as local, which selects the clustered origin view) that the path’s request_headers/request_params does not statically override, and whenever the render path or the backend itself has a request rewriter (req_rewriter_name) attached, since a rewriter can change the upstream host, path, headers, or parameters in ways synthetic requests do not see. The fallback is counted as trickster_graphite_fallbacks_total{reason="client_identity"}.

If every client shares one upstream identity — Grafana with a configured datasource credential is the common case — supply it in the graphite block to re-enable acceleration:

backends:
  graphite1:
    provider: graphite
    origin_url: 'http://graphite.example.com:80'
    graphite:
      origin_username: 'metrics'
      origin_password: '<the origin password>'

For origins using a non-Basic scheme, set origin_authorization to the verbatim header value (e.g., 'Bearer <token>') instead of the username/password pair.

The credential becomes the Authorization request header of every path — proxied client traffic, synthetic resolution requests (probes, wildcard expansion) and the default health check alike — so the origin sees one static identity, a client-supplied Authorization header is replaced rather than forwarded, and requests accelerate regardless of what the client sent. Cached objects are keyed on the configured credential, so rotating it invalidates previously cached responses and learned resolution state rather than serving the old tenant’s data. To authenticate Trickster’s own clients as well, attach an authenticator via authenticator_name: the validated client credential is stripped before proxying and the origin credential is applied in its place.

A path config that sets its own Authorization in request_headers (or removes it with -Authorization) overrides the origin credential for that path. Appending one with +Authorization alongside an origin credential is rejected at startup, since the effective header would be ambiguous. If per-path identities then diverge across the paths resolution uses — /render and /metrics/expand selecting different request_headers/request_params for synthetic GETs, or GET and POST /render pinned to different credentials — Trickster declines acceleration rather than mix upstream namespaces, counted as trickster_graphite_fallbacks_total{reason="resolution_identity"}.

The graphite block adds provider-specific options:

backends:
  graphite1:
    provider: graphite
    origin_url: 'http://graphite.example.com:80'
    graphite:
      time_zone: UTC
      passthrough_max_data_points: false
      find_cache_ttl: 1m
      resolution_registry:
        ttl: 24h
        negative_ttl: 30s
        max_entries: 100000
        persist: true
        probe_concurrency: 2
        probe_budget: 96
      static_retentions:
        - pattern: '^collectd\.'
          retentions: '10s:6h,1m:7d,10m:5y'
OptionDefaultPurpose
time_zoneUTCInterprets date-anchored from/until values (midnight, today, MM/DD/YY) when the request has no tz parameter. Set it to the origin’s graphite-web TIME_ZONE, or those queries resolve to different instants than the origin uses.
origin_username, origin_passwordunsetStatic HTTP Basic credential sent as the Authorization header of every upstream request — proxied, synthetic and health check alike. See Configuration above.
origin_authorizationunsetVerbatim Authorization header value for non-Basic schemes (e.g., Bearer <token>); mutually exclusive with origin_username/origin_password.
passthrough_max_data_pointsfalseSend the client’s maxDataPoints upstream instead of consolidating in Trickster. See maxDataPoints.
find_cache_ttl1mHow long a wildcard target’s expansion to concrete metric paths is reused. Lower it where metrics appear and disappear frequently.
max_targets_per_request128Renders carrying more targets than this are served through the object lane, untouched, before any parsing or per-target fan-out.
max_target_length16384A render with a longer single target expression (in bytes) is likewise served through the object lane untouched.
max_expanded_leaves4096A wildcard whose expansion matches more leaf paths than this is served through the object lane; the refusal is remembered for find_cache_ttl.
max_expansion_bytes2097152Bounds one expansion’s aggregate decoded leaf-name bytes the same way.
resolution_registry.ttl24hHow long a learned ladder is trusted. Whisper ladders change only when an operator runs whisper-resize.py, so this is deliberately long.
resolution_registry.negative_ttl30sInitial backoff after a failed resolution. Doubles per consecutive failure, capped at 10m.
resolution_registry.max_entries100000Bounds each registry layer; least-recently-used entries are evicted.
resolution_registry.persisttrueWrite learned ladders through to this backend’s cache so a restart does not relearn them.
resolution_registry.probe_concurrency2Simultaneous ladder-learning runs.
resolution_registry.probe_budget96Probes one learning run may issue before giving up.
static_retentionsemptySeed and override, from your storage-schemas.conf. See Static retentions.

Health checks

The default health check is GET /metrics/find?query=*, which every Graphite-protocol implementation serves. /version is not used because carbonapi and graphite-clickhouse answer it differently.

A configured origin credential rides on the probe even when the healthcheck block declares custom headers. To override it, set your own Authorization value there; to send the probe unauthenticated, set Authorization: ''.

Routed paths

PathHandling
/render (GET, POST)The render handler: delta cache, or object cache when the request is not accelerable
/metrics/find, /metrics/expand, /metrics/index.json, /tags, /tags/*Object-cached, 30s. These also back resolution’s own lookups, so caching them lowers probe cost.
/functions, /versionObject-cached, 1h
everything elseProxied

How resolution prediction works

Why it is necessary

Whisper picks an archive from the age of the query’s left edge, not from the width of the window:

diff = now - fromTime
for archive in header['archives']:
    if archive['retention'] >= diff:
        break
step = archive['secondsPerPoint']

So from=-7d&until=-6d and from=-7d&until=now are answered at the same step, and a query one second either side of a rung boundary is answered at different steps. The delta cache must know the step before it can compose a cache key or reason about which ranges it already holds — and the step is a property of the metric’s on-disk file, which nothing in the request reveals.

Static configuration is not sufficient on its own, because a Whisper file keeps whatever ladder it was created with. Editing storage-schemas.conf affects only newly created files, so in any long-lived installation the config and the files routinely disagree.

Probe and learn

On the first request for a metric Trickster does not know the ladder, so the request is served through the object cache — correct, just not delta-cached — and the response itself teaches the step at that age. In the background, Trickster then discovers the metric’s full ladder with a short series of synthetic /render probes: a geometric sweep outward in time to find where the step changes, then a binary search for each rung boundary, and a pair of probes to establish maxRetention. Each probe asks for a one-second window and pins the origin’s reference time with now, so it is cheap and exact.

Ladders are shared. They originate from storage-schemas.conf patterns, so a deployment with thousands of metrics typically has only a handful of distinct ladders; a new metric is first confirmed against the ladders already known (about 7 probes) and only discovered from scratch (about 40) if none matches. In the developer environment, 24 metrics across 4 ladders converge in about 380 probes total, after which probing stops entirely until the TTL expires.

Once a ladder is known, prediction is arithmetic and costs nothing.

Confidence levels

Every resolved target carries a confidence, exported as the confidence label on trickster_graphite_resolution_lookups_total:

ConfidenceMeaningBehavior
exactThe step was read from an origin response for this metric at this ageDelta cached
derivedComputed from known ladders — the shared step of a wildcard’s leaves, or a step-altering function Trickster understandsDelta cached
configuredFrom static_retentions only, not yet confirmed by probeDelta cached, and a confirming probe is scheduled
unknownNo usable stepObject cached

A request with several targets reports the weakest confidence among them.

Verification, and what happens when a prediction is wrong

The step is not merely predicted; it is checked. Every response Trickster models is compared against the predicted step. A mismatch means the cached ladder was wrong — a whisper-resize.py, a metric moved between schemas — and Trickster:

  1. discards the response rather than caching it under the predicted key,
  2. increments trickster_graphite_step_mispredictions_total and logs a warning,
  3. invalidates the registry generation, so every entry learned under the old assumption misses rather than colliding,
  4. relearns the affected ladders, and
  5. re-serves the request through the object cache, so the client still gets the correct answer.

One response shape cannot self-verify: JSON carries no explicit step, so a fetch that returns fewer than two points is consistent with any prediction. Trickster never trusts one. A delta fetch that would cover a single bucket — the tip fetch every steady-state refresh performs — is widened one bucket into the past, which cannot change the archive whisper selects (the pinned now keeps now - from identical) but makes the response two points, and therefore self-verifying, at no extra request cost. Any fetched response that still cannot prove its step — one point, or no series where the prediction promised data inside retention — is refused outright: nothing is cached, and the request is served through the object cache instead, while the background learner re-establishes the ladder.

trickster_graphite_step_mispredictions_total should be flat at zero. A non-zero value is not a client-visible error, but it does mean something the registry believed was wrong.

Static retentions

static_retentions mirrors storage-schemas.conf: an ordered list, first match wins, patterns matched anywhere in the metric path exactly as carbon applies them.

    graphite:
      static_retentions:
        - pattern: '^collectd\.'
          retentions: '10s:6h,1m:7d,10m:5y'
        - pattern: '\.count$'
          retentions: '1m:2d,5m:30d'
        - pattern: '.*'
          retentions: '1m:1d'

Retention syntax is Whisper’s: precision:retention pairs, where each part is a number with an optional unit (s, m for minutes, h, d, w, y), and a bare number in the second position is a point count. 1m:7d is a one-minute step kept for seven days.

This is a seed and an override, never the sole source of truth. A static match yields configured confidence and schedules a confirming probe; when the probe disagrees with the configuration, the probe wins and the ladder is relearned. That is deliberate: it means a stale static_retentions block degrades to a few extra probes rather than to incorrect data.

Use it to skip the warmup cost on a known-stable deployment, not as a substitute for learning.

Changing static_retentions invalidates the persisted registry on the next start, so a corrected block takes effect immediately rather than waiting out the TTL.

What is accelerated

A target is delta-cached only if every function in it satisfies two independent properties:

  1. Its step is predictable — the function does not change the resolution in a way Trickster cannot compute.
  2. It is range-decomposable — each output point is a function of the input points at that same timestamp, so the values for [t1,t2] are identical whether that window was fetched alone or as part of a wider one.

The second property is what delta caching actually requires, and it is stricter than it first appears. The v1 allowlist is deliberately small:

Cross-series aggregation — one output point per input timestamp: sumSeries (sum), averageSeries (avg), minSeries, maxSeries, diffSeries, multiplySeries, divideSeries, divideSeriesLists, stddevSeries, rangeOfSeries, countSeries, percentileOfSeries, aggregate, aggregateWithWildcards, aggregateSeriesLists, group, groupByNode, groupByNodes, asPercent (pct), weightedAverage, powSeries, unique, and the *SeriesLists and *WithWildcards variants.

Per-point transforms: scale, scaleToSeconds, offset, add, pow, exp, absolute, invert, squareRoot, sigmoid, logit, log, round, transformNull, isNonNull, removeAboveValue, removeBelowValue, consolidateBy, setXFilesFactor (xFilesFactor).

Naming, name-based filtering and cosmetics: alias, aliasSub, aliasByNode, aliasByMetric, aliasByTags, upper, lower, substr, exclude, grep, color, alpha, lineWidth, dashed, drawAsInfinite, secondYAxis, stacked, areaBetween.

Bare metric paths, wildcards (*, ?, [a-z], {a,b}) and pipe syntax are all accelerated.

What falls back, and why

Anything not on the list above is served through the object cache. The reason is recorded on trickster_graphite_fallbacks_total:

reasonCause
function_not_allowlistedA function in the target is not on the allowlist
unknown_stepThe step could not be resolved (metric not yet learned, probe failing, or the window is wholly beyond maxRetention)
missing_targetNo target parameter, or a wildcard that matches nothing
mixed_stepsA function combines series stored at different steps. graphite-web consolidates them to their least common multiple starting from each series’ first point, so the buckets move with from and cannot be stitched
parse_errorThe target expression, from/until, or now did not parse
non_series_formatAn image or pickle format, or graphType=pie
multi_target_step_mismatchTargets resolve to different steps and could not be split
passthrough_max_data_pointspassthrough_max_data_points is on and the request carries maxDataPoints
mispredictionA response contradicted the predicted step
tz_unavailableThe request names a tz whose validity could not be verified within the timezone cold-load budget (a burst of unique hostile tz values can spend it); the request is served unaccelerated with the original tz forwarded, rather than being reinterpreted in the configured zone
client_identityThe request carries a result-affecting identity or view selector — an Authorization header, a header named in the render path’s cache_key_headers, or a parameter/form field it names in cache_key_params/cache_key_form_fields (local above all) — that the path’s request_headers/request_params does not statically override, or the render path or backend has a request rewriter — see below
resolution_identityThe configured path identities are mixed: /render and /metrics/expand select different request_headers/request_params for synthetic GETs, or the request’s path config carries a different identity than the synthetic one (e.g., GET and POST /render pinned to different credentials)

Notable functions that are not accelerated, and why:

  • Time-windowed: movingAverage, movingSum, movingMin, movingMax, movingMedian, exponentialMovingAverage, derivative, nonNegativeDerivative, perSecond, integral, stdev, holtWinters*, linearRegression. Each output point depends on input points before it, so a cached range cannot be reused at its edges.
  • Whole-range selecting or ranking: highest*, lowest*, sortBy*, limit, mostDeviant, *Percentile, currentAbove/Below, averageAbove/Below, maximumAbove/Below, filterSeries. Which series come back depends on the entire requested range.
  • Bucketing: summarize, hitcount, smartSummarize. These look decomposable — with alignToFrom=false, summarize’s buckets align to absolute interval boundaries — but the bucket covering either edge of the requested window is summarized from only the points inside that window. Measured on graphite-web 1.1.10, the same absolute one-hour bucket reported 134885.899, 302921.880 and 307388.038 over three different windows. Caching such a value and reusing it for a different window would return data the origin never produced, so these take the object lane.
  • Time-shifting: timeShift, timeStack, timeSlice, delay.
  • Generators and tag queries: constantLine, threshold, timeFunction, randomWalk, sinFunction, identity, seriesByTag, groupByTags, events, and template().

The object lane is still a cache: the whole response is cached by request URL with a TTL, keyed on every render parameter. It is always correct because it makes no claim about the response’s internal structure.

Multiple targets

Grafana sends every target of a panel in one /render call. Trickster splits a multi-target request into one delta-cached fetch per target and merges the results in target order, so targets on different ladders are still accelerated. If any single target cannot be accelerated, the whole request is served unaccelerated, because consolidation at the origin spans all series in the response and must be applied consistently.

maxDataPoints and consolidation

Grafana sends maxDataPoints on every panel, sized to the panel’s pixel width. Two panels of different widths showing the same query would otherwise be two different cache entries.

Trickster strips maxDataPoints from upstream requests, caches at the origin’s native resolution, and applies consolidation when rendering the response to each client. One cached series therefore serves every panel width, every output format, and noNullPoints, jsonp, pretty and tz variations.

The consolidation reproduces graphite-web’s renderViewJson exactly, including its start “nudge”, its treatment of xFilesFactor, the consolidateBy function, and the maxDataPoints=1 special case — verified byte-for-byte against a live origin.

Set passthrough_max_data_points: true if you need the origin’s own consolidation byte-for-byte instead. Requests carrying maxDataPoints are then served unaccelerated, which for Grafana means most requests, so this largely disables acceleration. It exists for operators who must guarantee identical bytes to a pre-existing consumer.

Sizing

Two of the common backend settings default differently for Graphite, because Graphite is fetched at its native resolution: maxDataPoints is stripped upstream (see maxDataPoints), so Trickster buffers and caches every point Whisper holds for the window, not the few hundred a dashboard draws. The generic defaults are sized for origins that consolidate before responding.

SettingGeneric defaultGraphite default
max_object_size_bytes512 KB64 MB
timeseries_retention_factor1024 points524288 points

The Graphite values come from pkg/backends/graphite/options/defaults.go and are applied only where the configuration is silent, so setting either one in the file always wins.

max_object_size_bytes

The delta cache must buffer an entire response to model it. At native resolution a wide window on a fine archive is large: in the developer environment, a 120-day panel over two series on a 5-minute archive is 34,560 points per series and about 1.6 MB of JSON — three times the generic 512 KB limit.

A response over the limit cannot be delta-cached. Trickster logs upstream response exceeded MaxObjectSizeBytes and serves the request through the object lane instead, which streams rather than buffers — so clients still get their data, but that panel is never accelerated. Raise the limit if you serve panels wider than the 64 MB default holds:

    max_object_size_bytes: 134217728   # 128MB

timeseries_retention_factor

The cache keeps this many points per series, cropping older ones. The generic default of 1024 points is only about 3.5 days at a 5-minute step, so a 90-day dashboard panel would have almost its entire range cropped after each fetch and refetched on the next — a partial hit that re-fetches nearly everything. The Graphite default of 524288 covers 5 years at a 5-minute step, or 60 days at a 10-second one.

Raise it if you serve a wider window at a finer step than that, and lower it to cap what one series may occupy:

    timeseries_retention_factor: 1048576

Cache storage

Trickster does not roll up between rungs: a 10-second cached chunk is never reused to answer a query that resolves to 60 seconds, because the rollup Trickster computed would not match what Whisper’s lower archive holds (which depends on the metric’s aggregationMethod and xFilesFactor). A different step is a different cache key.

The practical implication is that a metric is cached once per rung it is queried at. A dashboard with a 1-hour, a 24-hour and a 30-day view of the same metric holds three cached series for it, not one. Size the cache for the distinct (metric, step) pairs your dashboards produce, not for the number of distinct metrics.

Metrics, logs, and tracing

Graphite-specific metrics, in addition to the standard trickster_proxy_requests_total{provider="graphite"} and trickster_cache_* families:

MetricTypeNotes
trickster_graphite_resolution_lookups_total{backend_name,confidence,source}counterOne per resolved request
trickster_graphite_probes_total{backend_name,kind,result}counterShould spike at start and fall to zero
trickster_graphite_ladders{backend_name}gaugeDistinct ladders known; should flatten at a small number
trickster_graphite_registry_entries{backend_name,layer}gaugeleaf, ladder, target, negative
trickster_graphite_step_mispredictions_total{backend_name}counterShould be flat at zero
trickster_graphite_fallbacks_total{backend_name,reason}counterWhy requests were not accelerated

No metric path, target expression or query text ever appears in a label; every label value comes from a closed set. See metrics.md.

Logs: a Debug line per resolution decision carrying the target, a bucketed age, the lane, confidence, source and step (or reason when declined); Warn on every misprediction and on every negative-cache entry.

Traces: GraphiteExpand (a child of the request span), GraphiteProbe (kind, result, step) and GraphiteLearnLadder (ladder, probe count).

Operations and troubleshooting

Everything is falling back

Check trickster_graphite_fallbacks_total by reason.

  • unknown_step dominating right after a start is normal; it should fall as ladders are learned. If it does not, resolution is failing — look for graphite ladder learning failed; negative-cached warnings.
  • function_not_allowlisted means the dashboard uses functions outside the allowlist. That is expected and correct, not a defect.
  • non_series_format means the client is asking for images; those panels cannot be delta-cached.

Probing never quiets down

trickster_graphite_probes_total should approach zero after warmup and trickster_graphite_ladders should flatten. If probing continues:

  • Metric names may be high-cardinality or ephemeral, so new leaves appear constantly. Raise resolution_registry.max_entries, or accept the cost.
  • The registry may be evicting under max_entries pressure and relearning. Compare trickster_graphite_registry_entries{layer="leaf"} against the configured maximum.
  • Learning may be failing and retrying under backoff. Check the warnings.

Step mispredictions are non-zero

Something changed a metric’s on-disk ladder — usually whisper-resize.py, a schema edit followed by new file creation, or a metric that moved between schema patterns. Trickster recovers on its own (it relearns and re-serves the request unaccelerated), so this is not a client-visible error, but the counter should return to flat once the new ladders are learned. Persistent mispredictions on one namespace suggest an origin whose resolution is not a function of now - from at all — a clustered graphite-web fronting stores with different schemas, for example.

A panel is never accelerated but should be

  • Look for upstream response exceeded MaxObjectSizeBytes in the logs and raise max_object_size_bytes.
  • Check whether the panel’s window is wholly beyond the metric’s maxRetention; those requests have nothing to cache.
  • A single non-allowlisted target in a multi-target panel makes the whole panel unaccelerated.

Repeated partial hits on wide panels

Raise timeseries_retention_factor. A partial hit that re-fetches nearly the whole range on every request is the signature of cropping.

Verifying correctness against the origin

Because the object lane and the delta lane must both be byte-identical to the origin, the simplest check is to configure two data sources in Grafana — one direct to Graphite, one through Trickster — and compare panels. The developer environment (docs/developer/environment/) is set up this way, with a dashboard whose panels deliberately exercise archive boundaries, the retention edge, schema drift, non-allowlisted functions and mixed-ladder multi-target requests.

Step alignment

Graphite supports the truncate step alignment mode, its default, and off. Whisper stores and returns whole buckets only, so there are no partial buckets for the other modes to fetch, and configuring one fails at load. off sends each render request to Graphite as the client sent it, through the fallback lane, cached as an object for one minute. A render target has no comment syntax, so Graphite queries can’t carry directives.

Known gaps

  • Fast Forward is not implemented. Graphite’s coarsest-rung behavior and its until > now clamp make the “one extra instant datapoint” approach ill-defined. fast_forward_disable is effectively always on.
  • No ALB / Time Series Merge support. Merging across replicas requires the replicas’ ladders to agree, an invariant not yet established. Graphite backends can still be members of non-TSM ALB mechanisms.
  • Non-decomposable functions are not accelerated. A future release may evaluate them inside Trickster over cached native-resolution leaves, which would move movingAverage, derivative, summarize, highest* and friends into the accelerated path.
  • No cross-resolution rollup, as described under Cache storage.
  • Tag-based queries (seriesByTag, groupByTags) are not accelerated. They are proxied and object-cached.
  • Non-ASCII metric names (graphite-web’s UTF8_METRICS) are not supported by the target parser; such targets fail to parse and take the object lane.

6 - PostgreSQL and TimescaleDB Provider

Trickster can accept native PostgreSQL client connections, relay each session to a PostgreSQL or TimescaleDB origin, and cache the results of eligible queries. Dashboards that bucket rows by time are served from Trickster’s delta cache: a refresh fetches only the buckets the cache does not hold yet.

This guide describes the supported contract. Anything not listed here is relayed to the origin without caching; cache eligibility always fails closed.

Compatibility

  • PostgreSQL 14 and later, with or without the TimescaleDB extension.
  • Grafana’s built-in PostgreSQL data source (Grafana 11 and later), with its TimescaleDB option on or off.
  • Any client that speaks protocol version 3.0 or 3.2 (psql, libpq, pgx, JDBC, …). Every client is relayed; which statements are cached is described under Cache classification.

Other engines that speak the PostgreSQL wire protocol are not part of this provider’s compatibility claim. The developer environment pins PostgreSQL 18 with TimescaleDB 2.30 and Grafana 13.

Provider names: postgres, and its alias timescaledb. Both select the same engine, and metrics report provider="postgres".

Direct backend configuration

A postgres listener accepts the connections and maps to exactly one backend (or one User Router) through the backend’s listener_names.

listeners:
  tsdb:
    protocol: postgres
    port: 5432

authenticators:
  tsdb-clients:
    provider: basic
    users:
      grafana_ro: ${GRAFANA_RO_PASSWORD}

backends:
  tsdb1:
    provider: timescaledb
    origin_url: postgres://trickster_ro:REPLACE_ME@tsdb.example:5432/metrics
    listener_names: [tsdb]
    authenticator_name: tsdb-clients
    cache_name: default
    healthcheck:
      interval: 5s
      timeout: 3s

A complete annotated example is in examples/conf/example.full.yaml.

Authentication modes

The backend’s authenticator decides who authenticates the client.

With authenticator_name (recommended). Trickster authenticates the client itself against the authenticator’s users and then logs in to the origin with the origin_url credentials. Clients are offered SCRAM-SHA-256, plus SCRAM-SHA-256-PLUS (channel binding) when the listener has a certificate. Authenticator entries may be plaintext passwords or PostgreSQL SCRAM-SHA-256$... verifiers copied from pg_authid. An entry stored as a PostgreSQL md5 hash or a crypt hash needs the client’s cleartext password, which is refused on a non-TLS connection unless the listener sets postgres.allow_cleartext_without_tls (or postgres.allow_md5 for the legacy md5 exchange). Every client shares the origin role named in origin_url, so give that role the least privilege the dashboards need. Replication connections are refused in this mode.

Without one (or with observe_only: true). The client’s own authentication exchange is relayed to the origin untouched and Trickster holds no credentials. One caveat: SCRAM-SHA-256-PLUS cannot succeed when both the listener and the origin use TLS, because the client binds to Trickster’s certificate and the origin expects its own. Clients that require channel binding must use the authenticator mode.

In both modes the cache is partitioned by the client’s user and database, so one user’s cached answers are never served to another.

TLS

TLS is negotiated in-band on the listener port, as PostgreSQL does it; there is no separate TLS port. Configure the backend’s tls.full_chain_cert_path and tls.private_key_path to let clients upgrade, and set require_tls: true to refuse plaintext clients.

TLS toward the origin is independent of the listener. Set the backend’s postgres.upstream_tls_mode:

ModeMeaning
disable (default)plaintext
requireTLS without certificate verification
verify-cathe origin’s chain must verify against tls.certificate_authority_paths
verify-fullas verify-ca, and the certificate must match the origin host name

tls.client_cert_path and tls.client_key_path add a client certificate.

Connections, limits, and lifecycle

A result that outgrows the result limits is streamed to the client through a fixed buffer, however large its rows are; the limits bound what Trickster holds in memory, not what a client may read.

Each client connection gets its own origin connection for its whole life, so session state never crosses clients, and Trickster adds no pooling. Put a pooler such as PgBouncer between Trickster and the origin if you need one.

SettingWhereDefaultEffect
max_concurrent_connsbackendunlimitedorigin connections; further clients get 53300 too many clients
connections_limitlistenerunlimitedaccepted client connections
postgres.handshake_timeoutlistener10sstartup and authentication
postgres.read_timeout / write_timeoutlistener30sone message in flight
postgres.idle_timeoutlistener5mwait for a client’s next message
postgres.max_query_size_byteslistener1 MiBlarger statements are relayed without being analyzed
postgres.max_result_rowsbackend100000larger results are relayed without being cached
postgres.max_result_size_bytesbackend64 MiBas above
timeoutbackend1morigin connect, and the wait for a cached statement’s result

Cancel requests work through the listener: clients receive a Trickster-issued cancellation key that maps to the origin’s, and a client that disconnects mid-query has its statement canceled at the origin. A configuration reload that changes a listener’s identity (origin, credentials, TLS, routes, limits) restarts that listener and drains its sessions; other reloads leave sessions alone.

Protocol behavior

Everything a client sends is relayed: the simple and extended query protocols, COPY, LISTEN/NOTIFY, transactions, multi-statement queries, notices and errors (with their SQLSTATE) all reach the client as the origin sent them.

Only one kind of message is ever answered from the cache: a simple-protocol Query holding a single read statement, sent while the session is idle (no open transaction, nothing in flight). That is what Grafana, psql and most reporting tools send. Statements sent with the extended protocol (Parse/Bind/Execute, i.e. prepared or parameterized statements) are relayed and never cached. For a driver that defaults to it, select its simple protocol mode if you want caching (pgx: default_query_exec_mode=simple_protocol; JDBC: preferQueryMode=simple).

Cached answers are the origin’s own bytes: the row description and rows are stored and replayed exactly as received, in text format. Trickster reads only the bucket column, to place rows on the time axis.

Cache classification

Each analyzed statement is counted in trickster_sql_query_analysis_total{backend_name,dialect,cache_mode,reason}.

cache_modeMeaning
deltacached per time bucket; only missing buckets are fetched
objectthe whole result is cached under the statement’s exact text
nonerelayed, never cached

Delta-cacheable statements

A statement uses the delta cache when it is a single-table SELECT that groups by one recognized time bucket and bounds the bucketed column with a literal (or now()-relative) lower limit and, optionally, an upper limit joined by AND. A range with no upper limit runs to the present.

BucketNotes
time_bucket(width, col)TimescaleDB; what Grafana’s $__timeGroup writes with the TimescaleDB option on
time_bucket(width, col, TIMESTAMPTZ '...')a typed origin
time_bucket(width, col, '...'::interval)a typed offset
time_bucket_gapfill(width, col)a single series, both bounds in WHERE, no locf or interpolate
floor(extract(epoch from col)/N)*N$__timeGroup with the TimescaleDB option off; also with date_part('epoch', col)
floor((col)/N)*N$__unixEpochGroup, over a column of epoch seconds with epoch-second bounds
date_bin(width, col, origin)PostgreSQL 14+
date_trunc('unit', col)second to week, and only while the session TimeZone is UTC

A width is a fixed-length interval: '300.000s', '5 minutes', '1h30m', INTERVAL '1 hour', '7 days'. Month and year widths never match. FILTER, ordered-set aggregates, TimescaleDB hyperfunctions such as first and last, extra GROUP BY columns, and ORDER BY on the bucket (either direction) are all fine.

Four behaviors are worth knowing:

  • Live ranges lose their partial edge buckets by default. Under the default step_alignment, drop, bounds that are not on the bucket grid, as Grafana’s now-relative ranges never are, are rounded inward, so the partial first and last buckets are left out of the answer rather than cached as if they were complete. The still-filling bucket that holds the present is partial too, so a range that reaches the present ends before it. col <= X keeps X’s bucket only when X is the last instant of it (...:59.999999). The partial, partial_start and partial_end modes instead fetch those buckets from the origin over the client’s own range, through the object cache for partial_bucket_ttl, and never cache them with the complete ones, at a cost of up to two small origin queries per request; truncate answers the whole first bucket. A query can choose its own mode with a comment, such as -- trickster-step-align:partial. See Step Alignment.
  • A range with no complete bucket is answered by the origin. When nothing complete remains, as for a range inside one bucket, across a single boundary, or starting inside the still-filling bucket, the client gets the origin’s own result for its statement, partial and still-filling buckets included, cached as an object for partial_bucket_ttl.
  • Zone-less literals need a UTC session. PostgreSQL reads '2026-09-17 00:00:00' and TIMESTAMP '...' in the session TimeZone, so such bounds (and origins) qualify only while that zone is UTC. Grafana’s Z-suffixed bounds always qualify.
  • Ordering within a bucket is the origin’s. Buckets are merged in time order and each keeps the row order the origin returned. ORDER BY must lead with the bucket or be absent; any other ordering makes the statement an object.

Statements cached as objects

Everything else that is a deterministic read: statements with no bucket, a LIMIT, a join, a subquery or CTE, DISTINCT, HAVING, window functions, set-returning functions, a gapfill outside the form above, month or year buckets, the time zone form of time_bucket, unsafe time predicates (OR, NOT, <>, a strict > lower bound on the raw column), and statements the SQL parser cannot write back faithfully (FROM ONLY, json casts, U&'...' strings) or cannot parse at all (named arguments such as origin => ..., GROUP BY ROLLUP). The reason label says which.

An object is keyed on the statement’s exact text. A dashboard panel whose range ends at now is therefore a new object on every refresh, and is served from cache only while its range is unchanged. Objects live for timeseries_ttl, so a repeated SELECT count(*) FROM t is answered from cache for that long; lower it, or set proxy_only: true, if that is too stale for your clients.

Statements never cached

  • Anything that is not a SELECT, and a SELECT that writes or locks (WITH x AS (DELETE ... RETURNING *) SELECT ..., FOR UPDATE, SELECT ... INTO).
  • A statement that calls a volatile or side-effecting function, or reads the clock anywhere but in a time bound: random(), now(), current_date, nextval(), pg_sleep(), pg_notify(), advisory locks and similar (reason="nondeterministic").
  • Multi-statement queries, statements inside a transaction, pipelined statements, and statements larger than postgres.max_query_size_bytes.
  • Everything in a session whose state Trickster can no longer vouch for; see Session state.

Trickster recognizes volatile functions by name. A user-defined volatile function, or one that changes a session setting as a side effect, is not detectable. Do not route such statements through a caching backend; use a proxy_only backend for them.

Continuous aggregates

Querying a continuous aggregate works like querying a table. To stay on the delta cache, re-bucket the aggregate’s bucket column and group by it:

SELECT time_bucket('15 minutes', bucket) AS "time", cab_type AS metric, sum(trips) AS trips
FROM trips_15m
WHERE bucket >= $__timeFrom() AND bucket < $__timeTo()
GROUP BY 1, 2 ORDER BY 1

A bare SELECT bucket, trips FROM trips_15m WHERE ... has no bucket function and no GROUP BY, so it is cached as an object. Real-time aggregates change their newest buckets as data arrives; volatile_window controls how much of the recent past Trickster re-fetches on each request.

Grafana macros and exact SQL shapes

These are the statements Grafana 13’s PostgreSQL data source sends, and how they are classified.

MacrosSent asMode
$__timeGroup(col,'5m'), TimescaleDB ontime_bucket('300.000s',col) AS "time"delta
$__timeGroup(col,'5m'), TimescaleDB offfloor(extract(epoch from col)/300)*300 AS "time"delta
$__timeGroupAlias(col,$__interval)as above, with the panel’s intervaldelta
$__timeFilter(col)col BETWEEN '...Z' AND '...Z'delta with a bucket
$__timeFrom(), $__timeTo()'...Z'delta with a bucket
$__unixEpochGroup(col,'10m')floor((col)/600)*600delta
$__unixEpochFilter(col)col >= N AND col <= Mdelta with a bucket
$__time(col), $__timeEpoch(col)a projection, no bucketobject
$__unixEpochNanoFilter(col)nanosecond integersobject

The minimum supported $__interval is 1 minute. Prefer col >= $__timeFrom() AND col < $__timeTo() over $__timeFilter(col): both are delta-cacheable, but the half-open form states exactly the range that is cached. Grafana’s pgx driver sends an empty -- ping query before each panel query; it is relayed and left out of the request metrics.

The executable form of this contract is pkg/backends/postgres/testdata/compatibility/v1.json.

Session state

Cached answers depend on session settings: TimeZone changes how every timestamptz is rendered, search_path changes which table a name means. Trickster therefore partitions the cache by a session identity: the user, the database, every setting the origin announces (TimeZone, DateStyle, IntervalStyle, client_encoding, search_path on PostgreSQL 18, …), and role, extra_float_digits and bytea_output from the client’s own SET statements. Two sessions share cached answers only when all of it matches.

A session that does something Trickster cannot follow stops being cached until it reconnects (reason="session_state"): any other SET (for example a custom app.tenant used by row-level security), set_config(), SELECT ... INTO, DDL, PREPARE/EXECUTE, DO, CALL, and fast-path function calls. A session with standard_conforming_strings off is relayed uncached as well. Transactions, DML, SHOW, EXPLAIN, and harmless settings such as statement_timeout and application_name do not end caching.

PostgreSQL never announces extra_float_digits or bytea_output, and a role or database default can change them. With an authenticator, Trickster reads the session’s effective values once at origin login, makes them part of the session identity, and returns to them on RESET. When the client’s own login is relayed, Trickster cannot ask, so it only knows a value the client set itself.

Results whose bucket column cannot be read safely are cached as objects instead: under a non-ISO DateStyle, and for a float8 epoch bucket ($__unixEpochGroup, or the date_part('epoch', ...) form) whenever extra_float_digits is negative or unknown, since PostgreSQL may then render 1788998400 as 2e+09. In relay-authentication mode such a panel needs the client to SET extra_float_digits (0 or higher) to use the delta cache.

Failure behavior

The cache always fails open to a plain relay of the client’s own statement: when the origin rejects a statement Trickster rewrote for a sub-range (trickster_sql_query_rewrite_failures_total{reason="origin_rejected"}), or when a result outgrows the result limits (reason="result_size"). Such a statement skips the cache until a marker expires, so it is not retried on every refresh. Origin errors reach the client verbatim and are never cached.

Protocol-aware User Router

A postgres listener can front a User Router ALB whose targets are postgres/timescaledb backends. Trickster authenticates the client against the router’s authenticator, picks the backend mapped to the startup user (or default_backend), and logs in to that backend’s origin with its origin_url credentials. The session stays on that backend until it ends.

  • The router requires an authenticator. Targets need origin credentials and no authenticator or listener of their own.
  • A user with no route, or whose backend is failing its health check, is refused with SQLSTATE 28000. Sessions never fail over to another backend.
  • to_user and to_credential are not supported.
  • Each target keeps its own cache, limits and metrics.

A complete example is in examples/conf/postgres-user-router.yaml.

Metrics, logs, and health

MetricLabels
trickster_sql_query_analysis_totalbackend_name, dialect, cache_mode, reason
trickster_sql_query_cache_totalbackend_name, dialect, cache_mode, cache_status
trickster_sql_query_rewrite_failures_totalbackend_name, dialect, reason
trickster_proxy_requests_total, ..._duration_seconds, trickster_proxy_points_totalthe common request labels, method="QUERY"
trickster_pgwire_connections_total, trickster_pgwire_active_connectionsbackend_name (and event)
trickster_pgwire_errors_totalbackend_name, class
trickster_pgwire_route_selections_totalrouter_name, backend_name, outcome

Query text, user names in labels, passwords and origin addresses are never logged or exported. The health check logs in to the origin on a fresh connection with the origin_url credentials and waits for ReadyForQuery; see Health Checks.

Operations and troubleshooting

A panel is never a hit. Look up its statement’s reason in the analysis metric. unsupported_limit, unsupported_bucket and unsupported_format mean it is an object, and an object whose range ends at now changes every refresh. session_state means the session did something unmodeled; find the SET the client sends at connect time. pipelined and in_transaction point at a driver that wraps reads in transactions or uses the extended protocol.

The first and last points are missing. Expected on unaligned ranges; see Delta-cacheable statements.

origin_rejected rewrite failures. Trickster re-spells the statement when it fetches a sub-range, and the origin refused the result. The client is unaffected, since its own statement is relayed, but the statement is not cached. Please file an issue with the statement.

Stale data after a bulk rewrite. Cached buckets older than the volatile window are not re-fetched until they expire (timeseries_ttl). After rewriting history, restart Trickster when using the memory cache, or clear the backend’s keys from a persistent one.

Rollout and rollback

Canary the listener with a limited client population and compare with the pre-release baseline. Roll back on a sustained increase of 5 percentage points in trickster_pgwire_errors_total or proxy-error outcomes, 1 percentage point in rewrite failures, or 20% in p95 request duration. Roll back by restoring the previous configuration and binary, reloading or restarting Trickster, and letting the listener drain; sessions are never moved between versions. Setting proxy_only: true on the backend is the quickest mitigation: it keeps the listener up and turns off all statement inspection and caching.

Environment variables and reload behavior

There are no TRK_* variables for postgres listener or backend options; configure them in YAML. Authenticator users values expand ${VARIABLE}; origin_url does not, so keep a configuration with an embedded password in a secret store. Validate before rollout:

trickster -validate-config -config /etc/trickster/trickster.yaml

Known compatibility gaps

  • Extended-protocol (prepared, parameterized, binary-format) statements are relayed but never cached.
  • Month, quarter and year buckets, the time zone overload of time_bucket, integer-time time_bucket(bigint, col), and explicit start/finish arguments to time_bucket_gapfill are objects, not deltas.
  • date_trunc and zone-less time literals are delta-cacheable only in UTC sessions.
  • User-defined volatile functions and functions that change session settings cannot be detected.
  • Downstream client-certificate authentication, GSSAPI encryption, and channel binding in relay-authentication mode are not supported.
  • No cross-session invalidation: a write through one session does not evict what another session cached.

7 - GreptimeDB Provider

Use provider: greptimedb for GreptimeDB’s HTTP, PostgreSQL and MySQL query surfaces. One backend may be mapped to all three listener protocols. The provider reuses the SQL and Prometheus cache engines, with GreptimeDB-specific parsing, session settings, result types and API paths.

Surfaces

SurfaceOrigin portBehavior
HTTP /v1/sql4000Eligible SELECT results use delta or object caching
PostgreSQL wire4003Eligible simple queries use delta or object caching; extended queries are relayed
MySQL wire4002Eligible text queries use delta or object caching
/v1/prometheus/api/v1/query_range4000Prometheus range caching and supported time-series merges
Prometheus instant and metadata APIs4000Object caching subject to HTTP cache policy
Ingest, TQL, /v1/promql, other HTTP APIs4000Proxied without SQL/PromQL delta rewriting
gRPC4001Not implemented by this provider

Wire compatibility does not make GreptimeDB a PostgreSQL or MySQL server. Use the greptimedb provider even on native listeners. For example, MySQL’s SHOW COUNT(*) WARNINGS is not a supported GreptimeDB statement, and its PostgreSQL session settings have different semantics.

The developer environment pins an official nightly that includes GreptimeDB #9295, the fix for Grafana’s comment-only PostgreSQL health query. See the developer environment for the exact image and reproducible checks. The tested Grafana plugin is the bundled PostgreSQL datasource, not a GreptimeDB-specific plugin.

Step Alignment

GreptimeDB’s surfaces follow the step alignment of the query language they carry:

SurfaceSupported modesDefault
Prometheus HTTP APItruncate, drop, partial_end, offpartial_end, which is Fast Forward (truncate with fast_forward_disable: true)
SQL, over HTTP, MySQL and PostgreSQLalldrop

A backend’s step_alignment may name any mode either supports; a PromQL query asked for partial or partial_start runs in its default instead, and the fallback is counted in trickster_step_alignment_fallbacks_total. The partial modes fetch each partial bucket as a small query of its own through the object cache, so they cost up to two extra origin queries per request. A query can choose its own mode with a comment, # ... in PromQL and -- ... or /* ... */ in SQL; see Per-Query Instructions.

Configuration

The complete example exposes HTTP on 8480, PostgreSQL on 8489 and MySQL on 8491. A minimal mixed backend is:

listeners:
  greptime-pg:
    protocol: postgres
    port: 8489
  greptime-mysql:
    protocol: mysql
    port: 8491

authenticators:
  greptime-readers:
    provider: basic
    proxy_preserve: true
    users:
      grafana_ro: ${GRAFANA_RO_PASSWORD}

backends:
  greptime:
    provider: greptimedb
    origin_url: http://greptime.example:4000
    listener_names: [default, greptime-pg, greptime-mysql]
    authenticator_name: greptime-readers
    postgres:
      upstream_url: postgres://trickster_ro:REPLACE_ME@greptime.example:4003/public
    mysql:
      upstream_url: mysql://trickster_ro:REPLACE_ME@greptime.example:4002/public

HTTP uses origin_url; postgres.upstream_url and mysql.upstream_url separately identify native endpoints and credentials. Never put the HTTP port in either native URL. Without a native override, an HTTP origin’s host is reused with the engine’s native default port, not its HTTP port or userinfo. Explicit URLs are recommended. URL credentials do not expand environment variables; authenticator user values do.

List only default for an HTTP-only backend, or just the chosen native listener for a native-only backend. A mixed backend probes HTTP /health; a native-only backend uses its native authenticated probe. Listener, origin, credential or protocol changes restart and drain affected native listeners.

Authentication And TLS

For HTTP, proxy_preserve: true forwards client credentials to GreptimeDB. Authenticated object responses are not shared unless the origin explicitly permits HTTP caching. SQL and PromQL cache identities retain their database, request parameters, effective timezone and authentication partitioning.

For PostgreSQL, an authenticator terminates client authentication at Trickster; the separate upstream URL supplies the origin role. Without an authenticator, authentication is passed through to GreptimeDB. MySQL requires a configured authenticator. Give every origin role only the permissions its clients need; terminating authentication does not preserve separate upstream roles for each client.

Native TLS and limits use the existing postgres and mysql option blocks; see PostgreSQL and MySQL. Support for an option in Trickster is not a promise that a particular GreptimeDB deployment supports the corresponding server feature. The loopback developer environment uses plaintext and published test credentials and must not be exposed publicly.

SQL Cache Classification

Delta caching requires one provably deterministic time-bucketed SELECT with supported grouping, ordering and time bounds. Other supported reads can use whole-result object caching. Writes, volatile queries and session-unsafe statements bypass caches; a parser rejection never makes a write cacheable.

PostgreSQL supports fixed-width date_bin intervals or compact widths such as 5m, UTC fixed-width date_trunc, and epoch buckets of the form floor(extract(epoch FROM ts)/300)*300 or date_part. Calendar widths, ambiguous time predicates, unknown timezone-dependent semantics, and submicrosecond PostgreSQL text buckets do not get delta plans. Extended protocol messages are always relayed.

HTTP SQL supports GET and form-encoded POST with the default greptimedb_v1 response format. A delta response retains typed schema, row ordering, NULLs and exact integer values. Like PostgreSQL, under the default drop step alignment, unaligned SQL bounds round inward to complete buckets before delta caching and partial edge buckets are omitted. Alternate formats, limit, unknown options and ranges with no complete bucket retain the original query. Rebuilt responses do not claim the origin’s execution duration or execution metrics. The default JSON response is decoded row by row into a DataSet, and its validated client serialization is reused.

MySQL supports DATE_BIN('1m', ts, FROM_UNIXTIME(0)) and fixed-width DATE_TRUNC buckets with verified UTC sessions, whole-second cadence and half-open bounds, which the default drop step alignment rounds inward to complete buckets. Other bucket origins, subsecond cadence, inclusive upper bounds and ranges with no complete bucket use the original query instead. Timestamp results keep up to nine fractional digits, text groups are compared case-sensitively, and NULL ordering follows GreptimeDB. Unsupported or failed session changes conservatively disable caching for that connection.

PromQL range endpoints round down to epoch-aligned steps, including 500ms steps, so equivalent aligned ranges share cached samples. Instant and metadata timestamps are unchanged. Route defaults do not grant permission to share authenticated object responses: only an origin’s explicit cache policy can authorize that sharing.

Grafana Macros

Use the built-in PostgreSQL datasource with TimescaleDB disabled and a minimum interval of one minute. Native epoch-second columns are integers, not timestamps cast to integers (GreptimeDB timestamp casts can yield nanoseconds).

Macro familyGreptimeDB behavior
$__time, $__timeEpochSupported projection; not a bucket by itself
$__timeFilter, $__timeFrom, $__timeToSupported timestamp bounds
$__timeGroup, $__timeGroupAliasFixed epoch-floor buckets with TimescaleDB mode off
$__unixEpochFilter, $__unixEpochFrom, $__unixEpochToNumeric epoch-second bounds
$__unixEpochGroup, $__unixEpochGroupAliasFixed buckets over an epoch-second column
$__unixEpochNanoFilterSupported SQL predicate; not an automatic delta plan
$__intervalUse a positive supported fixed duration

The compatibility corpus records real Grafana expansions, including unaligned windows and conservative fallbacks. Grafana’s MySQL macros are not interchangeable: UNIX_TIMESTAMP is unsupported in the tested GreptimeDB image. Use explicit GreptimeDB SQL when connecting a MySQL client.

Known Limits

  • RANGE ... ALIGN remains whole-result caching; TQL is passthrough. There is no advanced RANGE/FILL dependency rewriting.
  • PostgreSQL text timestamps carry microseconds even for a TIMESTAMP(9) column. MySQL and HTTP have separate precision contracts.
  • Transaction commands accepted by GreptimeDB are compatibility stubs, not proof of transactional isolation. PostgreSQL cancellation is not a supported upstream capability in the tested image.
  • SET LOCAL persists at session scope in GreptimeDB. Effective settings are probed and tracked; requested startup parameters alone are not proof of the resulting timezone. The development read-only account cannot change it.
  • PostgreSQL session load balancing is unsupported. MySQL listeners use the existing native MySQL routing rules.
  • PromQL count_values currently loses its grouping label upstream; merges reject that response rather than claim a correct aggregate.
  • Ingestion, mutations, enterprise-only features and gRPC caching are outside this provider’s accelerated contract.

Run the acceptance suites against an isolated seeded environment. Inspect cache counters as well as results: trickster_sql_query_analysis_total, trickster_sql_query_cache_total, and trickster_sql_query_rewrite_failures_total distinguish actual cache hits, fallbacks and failed rewrites. SQL dialect labels are greptimedb; transport metrics retain the native protocol name.

8 - QuestDB Provider

Use provider: questdb for a QuestDB backend that exposes both the HTTP SQL surface and the PostgreSQL wire protocol. Trickster keeps the HTTP surface as an authenticated passthrough and uses the native pgwire path for the verified SQL relay and cache contract.

Surfaces

SurfaceOriginBehavior
HTTP SQL, Web Console and other HTTP pathsQuestDB HTTP, normally 9000Passthrough; no HTTP SQL delta cache is applied
PostgreSQL wire protocolQuestDB pgwire, normally 8812Simple-query relay, object cache and verified fixed-width delta cache
Extended pgwire statementsQuestDB pgwireRelayed; extended statements are not analyzed for delta caching

The native listener uses the shared PostgreSQL wire adapter. A QuestDB backend can therefore be mapped to an HTTP listener and a protocol: postgres listener at the same time. The provider name, rather than postgres, selects QuestDB’s SQL analyzer and result semantics.

Configuration

The HTTP origin and native origin are separate. Put the HTTP endpoint in origin_url and the pgwire endpoint, including its credentials, in postgres.upstream_url:

listeners:
  questdb-pg:
    protocol: postgres
    port: 8490

authenticators:
  questdb-readers:
    provider: basic
    users:
      grafana_ro: ${QUESTDB_CLIENT_PASSWORD}

backends:
  questdb:
    provider: questdb
    origin_url: https://questdb.example:9000
    listener_names: [default, questdb-pg]
    authenticator_name: questdb-readers
    postgres:
      upstream_url: postgres://grafana_ro:REPLACE_ME@questdb.example:8812/qdb
      upstream_tls_mode: disable
    healthcheck:
      interval: 5s
      timeout: 3s

Use TLS and deployment-specific credentials in a real installation. The developer environment uses cleartext QuestDB pgwire because QuestDB OSS does not provide pgwire TLS in the tested image. proxy_preserve is only suitable when the HTTP origin is configured to accept the same client credential; the native upstream URL is the source of the pgwire origin credential.

If the backend is native-only, map only questdb-pg and use a PostgreSQL origin_url with an explicit user. Without an HTTP listener, the provider uses its authenticated pgwire health probe. With an HTTP listener, the health check uses authenticated QuestDB /execute and the native listener still uses the postgres.upstream_url.

SQL Cache Contract

The QuestDB analyzer has a deliberately narrow delta-cache surface:

Query shapeResult
SAMPLE BY 5m, 15m, and other positive fixed s/m/h/d widthsDelta cache
SAMPLE BY ... FILL(NULL)Object cache; fill rows depend on the selected range
timestamp_floor('5m', timestamp_column) with grouped outputDelta cache
floor(extract(epoch FROM ts)/N)*NDelta cache when the bounds and result are renderable
Other deterministic readsObject cache or relay, according to the normal pgwire policy
FILL(PREV), SAMPLE BY FROM ... TO ..., calendar M widths, ambiguous groupingObject/relay; no speculative rewrite
Writes, volatile statements, malformed or unknown time rangesRelay; never a delta plan

For SAMPLE BY, QuestDB applies the designated timestamp to the first plain selected column and implicitly groups the remaining plain selected columns. The adapter only accepts that unambiguous form. An explicit GROUP BY, a computed first item or another ambiguous shape fails closed rather than guessing the series identity.

The pgwire result path maps QuestDB’s verified PostgreSQL type OIDs and microsecond timestamp text to the shared time-series representation. Session settings that can affect results or rendering, including timezone, date style, interval style, float/bytea output and search path, remain part of the cache identity. Unknown startup settings also partition the identity until observed; unknown SET statements bypass caching conservatively.

Developer Environment

The reproducible developer environment adds a pinned QuestDB OSS image, a read-only pgwire role, an HTTP admin role for seeding, and the official questdb-questdb-datasource Grafana plugin. It reuses the shared generated trips fixture rather than downloading another dataset.

See QuestDB Details for ports, seed commands, dashboard datasource names and direct-versus-proxy checks. The versioned analyzer contract is in pkg/backends/questdb/testdata/compatibility.

The integration conformance target compares direct and proxied QuestDB results, protocol errors and session state. It also checks object and delta cache counters, so a successful passthrough response cannot be mistaken for a cache hit.

Known Limits

  • HTTP SQL is passthrough in this provider; native pgwire is the only path with QuestDB-specific delta caching in this change.
  • Extended pgwire messages are relayed and are not delta-cacheable.
  • QuestDB FILL(PREV), clause-owned FROM/TO ranges and calendar-month widths remain object/relay paths.
  • QuestDB OSS cleartext pgwire and the credentials in the developer compose file are for isolated local testing only.

9 - VictoriaMetrics Provider

Use provider: victoriametrics for VictoriaMetrics: single-node, or one tenant of a cluster’s vmselect. Trickster accelerates MetricsQL range queries through VictoriaMetrics’ Prometheus querying API, caches settled instant and metadata queries, and serves its Graphite render, find and tags APIs. VictoriaMetrics is its own provider rather than a flavor of prometheus: MetricsQL is a different language, and native VictoriaMetrics, not PromQL, decides every answer.

Surfaces

SurfaceRoutesBehavior
MetricsQL range queries/api/v1/query_rangeDelta proxy cache, or exact-request object cache for expressions that depend on the whole range
MetricsQL instant queries/api/v1/queryObject cache once the evaluation time has settled; relayed near the live edge
Metadata/api/v1/series, /api/v1/labels, /api/v1/label/<name>/valuesObject cache, 30 seconds
Other read APIs/api/v1/status/..., /api/v1/metadata and the rest of the Prometheus catalogObject cache, 30 seconds, keyed on every parameter
Graphite render/renderExact-request object cache for settled absolute ranges; relayed otherwise
Graphite discovery/metrics/find, /metrics/expand, /tags..., /functionsObject cache, 30 seconds
Graphite tag registration/tags/tagSeries, /tags/tagMultiSeries, /tags/delSeriesRelayed uncached
Everything elsewrites, imports, exports, /api/v1/admin/..., /internal/..., /federate, /vmuiRelayed uncached

Every Graphite route is also served under /graphite, as single-node VictoriaMetrics serves it.

Configuration

origin_url is the base of the Prometheus querying API: the server for single-node VictoriaMetrics, or a tenant’s /select/<tenant>/prometheus path on vmselect. The Graphite APIs are served from the same server, with a trailing /prometheus replaced by /graphite, as vmselect’s tenant paths require.

backends:
  vm:
    provider: victoriametrics
    origin_url: http://victoriametrics:8428

  vm-tenant:
    provider: victoriametrics
    # Graphite requests go to http://vmselect:8481/select/0/graphite
    origin_url: http://vmselect:8481/select/0/prometheus
    victoriametrics:
      # replaces the derived Graphite path, e.g. for a gateway with its own routes
      graphite_path: /select/0/graphite
      # set when VictoriaMetrics runs with -search.disableCache (see Range Queries)
      search_disable_cache: false
OptionDefaultDescription
victoriametrics.graphite_pathderivedThe upstream path of the Graphite APIs: an absolute path, without a query
victoriametrics.search_disable_cachefalseMirrors the origin’s -search.disableCache flag
volatile_window60sHow long recent points stay provisional; keep it longer than the origin’s -search.latencyOffset (30s) plus its ingestion delay

Authentication headers, TLS and request rewriting work as for any HTTP backend; requests that carry Authorization are cached under keys that include it. The default health check runs query=1 against the query API, so it also checks a tenant’s path.

Range Queries

Trickster answers a range query on the grid VictoriaMetrics evaluates it on. With its response cache on, VictoriaMetrics aligns a range of 50 or more points to start on a step boundary and keeps the number of points; a shorter range keeps the requested start. Trickster rewrites the range the same way before looking in its cache, so cached and native answers have the same timestamps. Grafana aligns its ranges itself, so its requests are unchanged. With search_disable_cache: true, every range keeps the requested start, as VictoriaMetrics then does.

Cache Eligibility

Each MetricsQL statement is parsed once with VictoriaMetrics’ own parser (metricsql, pinned to the version the provider was validated with); WITH templates are expanded before classification.

StatementPath
Selectors, rollups (rate, increase, *_over_time, rollup, with or without a window), aggregations, topk/bottomk, binary operators, label and math functions, offset, subqueries, @ with a timestampDelta proxy cache
range_*, running_*, keep_last_value, keep_next_value, interpolate, remove_resets, smooth_exponential, sort*, limit_offset, drop_common_labels, drop_empty_series, union, buckets_limit, start(), end(); topk_*/bottomk_* variants, limitk, any, outliers*; the limit modifier; @ with anything but a timestampExact-request object cache (step alignment off)
now(), rand(), rand_normal(), rand_exponential(), and statements that don’t parseRelayed uncached

A statement that doesn’t parse is relayed, so VictoriaMetrics returns its own error. A request is also relayed uncached when it:

  • sets nocache or trace;
  • has a parameter the route doesn’t read, or a parameter repeated, or the same parameter in both the URL and a POST form;
  • uses a time VictoriaMetrics would read differently than Trickster: anything but Unix seconds (with at most millisecond precision) or RFC 3339 with a zone, such as -1h, now, a zoneless time, or milliseconds;
  • is an instant query within the volatile window of now, or without a time;
  • is a POST that isn’t application/x-www-form-urlencoded.

round_digits, latency_offset, max_lookback, extra_label, extra_filters[], limit and optimize_repeated_binary_op_subexprs are part of a response’s cache identity; timeout and denyPartialResponse can only fail a request and are not. The classification of each request is counted in trickster_victoriametrics_query_analysis_total.

Fidelity

VictoriaMetrics evaluates a point from the samples it fetched for the whole request: an implicit window, and whether rate and similar functions use the sample before their window, depend on a scrape interval estimated from the last 20 sample intervals it read. Where a series’ scrape interval changes, a range assembled from separate fetches can differ slightly from one nocache=1 evaluation of it. VictoriaMetrics’ own response cache, which is on by default, assembles ranges from separate evaluations the same way, so the delta proxy cache’s answers match what VictoriaMetrics serves by default. For a single exact evaluation, set step_alignment: off on the backend, or write # trickster-step-align:off in the query.

Points within the volatile window of now, which VictoriaMetrics replaces within its -search.latencyOffset, are fetched on every request and never stored. A request’s own latency_offset widens that window.

Partial Responses

When some vmstorage nodes don’t answer, vmselect marks its response "isPartial":true. Trickster never caches such a response: a range query is relayed to the origin, and other API responses are marked Cache-Control: no-store. Responses served from cache for an origin that reports isPartial include "isPartial":false, as vmselect writes it. The stats object VictoriaMetrics adds to query responses describes one evaluation and is not kept in cached responses.

Graphite

The Graphite APIs use the Storage-Step request header, which sets the grid the render API returns; it is part of a render response’s cache identity, with every query parameter. A render is consolidated to maxDataPoints over its whole range, so renders are cached only as exact requests, never assembled from separate ranges. A render is cached only when from and until are Unix timestamps and until is outside the volatile window; Grafana sends both that way.

Time Series Merging

The tsm ALB mechanism doesn’t support VictoriaMetrics backends yet: the provider offers no merge planning, so MetricsQL is never merged with PromQL’s semantics, and requests through a tsm pool of VictoriaMetrics backends fail. Other ALB mechanisms work as for any backend.

Known Limits

  • Delta caching is not offered for the Graphite render API.
  • partial_end (Fast Forward) is not offered.
  • An unfiltered /tags or /tags/<tag> returns nothing for series VictoriaMetrics stored as labels; /tags/autoComplete/tags?expr=..., /tags/findSeries and seriesByTag find them. This is VictoriaMetrics’ behavior and is relayed as is.
  • Requests through the frontend’s /graphite alias are mapped to the Graphite path for vmselect origins too, where vmselect has no such alias.
  • Raw-series export (/api/v1/export) is relayed uncached.

Developer Environment

The developer environment runs single-node VictoriaMetrics v1.153.0 with seeded trips history and fixtures, the victoriametrics1 backend, and the Trips (VictoriaMetrics) and VictoriaMetrics Graphite dashboards. See VictoriaMetrics Details.

10 - MySQL Provider

Trickster can accept native MySQL client connections, proxy each authenticated session to a MySQL origin, and cache eligible text-protocol query results. This guide describes the initial supported contract. Behavior not listed here is rejected or proxied without caching; cache eligibility always fails closed.

Compatibility

The supported matrix is:

  • Oracle MySQL 8.4 LTS through 9.7 LTS;
  • Grafana 11.0 and above using Grafana’s built-in MySQL data source; and
  • native clients that use mysql_native_password and the supported commands below.

MariaDB, Percona Server, Aurora MySQL, other MySQL-compatible products, older Grafana releases, and third-party Grafana MySQL data sources are not part of the initial compatibility claim, but should work without problems. File an issue if you discover a compatibility issue.

The developer environment pins MySQL 8.4 and Grafana 13.

Direct backend configuration

The listener accepts the MySQL wire protocol on one TCP port. MySQL negotiates TLS in-band, so do not configure an HTTP-style tls_port.

listeners:
  mysql-native:
    protocol: mysql
    address: ""
    port: 8486
    connections_limit: 200
    mysql:
      handshake_timeout: 10s
      read_timeout: 30s
      write_timeout: 30s
      idle_timeout: 5m
      max_packet_size_bytes: 16777215
      max_query_size_bytes: 1048576

authenticators:
  mysql-clients:
    provider: basic
    users:
      grafana_reader: ${GRAFANA_MYSQL_PASSWORD}

caches:
  mysql-cache:
    provider: memory
    memory:
      max_size_bytes: 536870912

backends:
  mysql-primary:
    provider: mysql
    listener_names: [mysql-native]
    authenticator_name: mysql-clients
    origin_url: mysql://trickster_ro:REDACTED@mysql.example:3306/analytics
    cache_name: mysql-cache
    timeout: 30s
    max_concurrent_conns: 200
    max_object_size_bytes: 8388608
    mysql:
      max_result_rows: 100000
      max_result_size_bytes: 67108864
    healthcheck:
      interval: 5s
      timeout: 3s
      failure_threshold: 3
      recovery_threshold: 2

Exactly one direct MySQL backend, one supported MySQL User Router ALB, or one session-balancing ALB maps to a MySQL listener. The origin URL must use the mysql scheme and include an origin username. Percent-encode reserved username, password, and database characters. Configuration stringification and the sanitized management configuration redact an embedded origin password, but the source configuration must still be protected as a secret.

The authenticator is the only downstream credential source. Trickster terminates downstream authentication; it does not pass client credentials to the origin or silently fall back to origin credentials. Native-password verification requires the original password, so MySQL authenticator entries must be plaintext values, including environment-expanded values or a protected CSV users file. Htpasswd hashes are rejected for this listener. Do not expose the raw configuration management endpoint outside a trusted administrative network.

TLS

Downstream TLS

Add a server certificate and key to the listener-facing backend:

backends:
  mysql-primary:
    # ...direct backend settings above...
    require_tls: true
    tls:
      full_chain_cert_path: /etc/trickster/tls/server.crt
      private_key_path: /etc/trickster/tls/server.key

Without a server pair, downstream TLS is disabled. With the pair, TLS is optional unless require_tls: true rejects plaintext clients. TLS 1.2 is the minimum. Downstream client certificates are not supported in the initial release.

Upstream TLS

The upstream modes are:

  • no upstream TLS fields: plaintext;
  • insecure_skip_verify: true: encryption without server verification;
  • one certificate_authority_paths entry: CA and hostname verification; and
  • a complete client_cert_path/client_key_path pair: mutual TLS, combined with verified or explicitly unverified server mode.
backends:
  mysql-primary:
    # ...direct backend settings above...
    tls:
      certificate_authority_paths:
        - /etc/trickster/tls/origin-ca.crt
      client_cert_path: /etc/trickster/tls/origin-client.crt
      client_key_path: /etc/trickster/tls/origin-client.key

Certificate/key settings must be complete pairs. More than one upstream CA path, require_tls without a downstream server pair, or an incomplete pair is a configuration error. There is no CA-verification-without-hostname mode.

Connections, limits, and lifecycle

Each authenticated downstream connection owns exactly one upstream connection. There is no pooling or multiplexing, which prevents transaction, database, and session state from leaking between clients. Size connections_limit and max_concurrent_conns together and leave capacity for health probes. A useful starting point is one admitted origin connection per downstream connection, plus normal operational headroom.

Listener limits protect the downstream boundary:

  • connections_limit limits concurrent client connections (0 is unlimited);
  • handshake, command-read, response-write, and idle timeouts bound each phase;
  • max_packet_size_bytes is fixed at 16 MiB minus one; and
  • max_query_size_bytes limits SQL text within that packet ceiling.

Backend limits protect the origin and cache:

  • timeout bounds connect and query work;
  • max_concurrent_conns bounds upstream admission;
  • mysql.max_result_rows and mysql.max_result_size_bytes bound results; and
  • max_object_size_bytes skips caching an otherwise valid oversized object.

Handshake, packet, query, read, write, and idle failures close the client connection. Origin timeout or concurrency rejection closes the affected upstream connection and returns a bounded MySQL error. A row/byte overflow closes both sides because a partial result may already have been emitted.

On shutdown, Trickster stops accepting new sessions and drains existing ones within the configured management drain window. A reload that changes MySQL authentication, listener transport, certificates, origin transport, routing, or relevant limits restarts and drains the listener; an established session is never silently moved to new credentials, TLS policy, or a different origin.

Protocol behavior

Supported commands are:

  • COM_QUERY containing one text-protocol statement;
  • COM_INIT_DB;
  • COM_PING;
  • COM_QUIT; and
  • COM_RESET_CONNECTION.

Reset discards the upstream connection and tracked session state. The next command opens a fresh origin connection.

The initial release rejects binary prepared statements; SQL PREPARE, EXECUTE, and DEALLOCATE; server cursors; binary results; compression; LOCAL INFILE and all LOAD statements; multi-statements; stored-procedure CALL; HELP, XA, HANDLER, and CACHE INDEX; executable version comments; unclassified text response shapes; COM_CHANGE_USER; replica registration; and binlog commands. Connection attributes may be syntactically accepted but are ignored for authentication, routing, cache identity, and query semantics. Malformed, oversized, timed-out, or partially streamed commands close the connection. A fully consumed unsupported command normally leaves it usable.

Successful DML and DDL are proxied and never cached. They do not invalidate existing OPC or DPC entries, so another session may see cached data until TTL expiry or eviction. Use short TTLs or purge affected cache data when immediate read-after-write visibility across sessions is required.

Cache classification

Trickster uses three outcomes:

  • Delta Proxy Cache (DPC): a supported deterministic aggregate time-series SELECT with one unambiguous time output, literal positive cadence, safe result shape and ordering, and an inclusive-lower/exclusive-upper raw-time predicate;
  • Object Proxy Cache (OPC): a deterministic single-result SELECT that is safe to cache as one object but is not proven reusable by time extent; and
  • proxy-only: mutations, transactions, unsupported or unsafe session state, non-deterministic or rejected shapes, and any query whose safety cannot be established.

DPC supports the tested zero-phase integer/cast and FLOOR(UNIX_TIMESTAMP(column) / n) * n bucket forms when n is the same positive integer literal on both sides. Dynamic, mismatched, non-positive, overflowing, ambiguous, or non-zero-phase forms fail closed. Qualified and backtick-quoted identifiers, multiple numeric values, and string dimensions are supported only in the shapes recorded by the compatibility corpus.

The authenticated downstream username, selected terminal backend, normalized database, supported time-zone literal, and query identity participate in cache keys. Cache entries are never shared across authenticated users merely because their SQL is identical.

Grafana macros and exact SQL shapes

Only patterns represented by the versioned compatibility corpus are published as supported. The corpus covers $__time, $__timeEpoch, $__timeFilter, $__timeFrom, $__timeTo, $__timeGroup, $__timeGroupAlias, $__unixEpochFilter, $__unixEpochFrom, $__unixEpochTo, $__unixEpochGroup, and $__unixEpochGroupAlias. Its documented minimum $__interval is one minute.

Grafana’s normal inclusive $__timeFilter expansion is delta-cached (DPC):

SELECT
  CAST(CAST(UNIX_TIMESTAMP(ts)/(300) AS SIGNED)*300 AS SIGNED) AS time,
  COUNT(*) AS samples
FROM telemetry
WHERE ts BETWEEN FROM_UNIXTIME(1785542400)
             AND FROM_UNIXTIME(1785628800)
GROUP BY time
ORDER BY time

A half-open predicate composed from $__timeFrom() and $__timeTo() is also DPC:

SELECT
  CAST(CAST(UNIX_TIMESTAMP(ts)/(60) AS SIGNED)*60 AS SIGNED) AS time,
  AVG(value) AS mean_value
FROM telemetry
WHERE site = 'denver'
  AND ts >= FROM_UNIXTIME(1785542400)
  AND ts < FROM_UNIXTIME(1785628800)
GROUP BY time
ORDER BY time

The epoch-second equivalent is:

SELECT
  CAST(CAST(epoch_seconds/(300) AS SIGNED)*300 AS SIGNED) AS time,
  SUM(requests) AS requests
FROM service_rollups
WHERE region = 'us-west'
  AND epoch_seconds >= 1785542400
  AND epoch_seconds < 1785628800
GROUP BY time
ORDER BY time

Trickster caches only complete buckets. Under the default step_alignment, drop, it rounds the lower bound up and the upper bound down to the cadence, and an inclusive upper bound, such as the end of BETWEEN, also drops the bucket that contains it, because that bucket is only partly covered. The partial, partial_start and partial_end modes fetch those edge buckets from the origin over the client’s own bounds, through the object cache for partial_bucket_ttl, at a cost of up to two small origin queries per request, and truncate answers the whole first bucket. A range with no complete bucket is answered with the origin’s own result for the statement, cached as an object for partial_bucket_ttl. A query can choose its own mode with a comment, such as /* trickster-step-align:partial */. See Step Alignment. Grafana’s strict-lower $__unixEpochFilter expansion remains OPC, because a strict lower bound does not cover the first bucket completely. Native DATETIME/TIMESTAMP, epoch-second integer, and the corpus’s epoch-nanosecond adaptation are supported in their recorded shapes.

Session state

The only cache-safe tracked state is the selected default database and one session-scoped literal SET time_zone = '<value>'. USE and COM_INIT_DB update the database portion of cache identity.

Transactions and savepoints bypass cache while active. Other SET forms, character set or collation changes, sql_mode, lc_time_names, user variables, temporary objects, locks, session-local functions, mutations, and unclassified state changes make that connection cache-unsafe. Reconnect or a successful COM_RESET_CONNECTION is required to restore the configured baseline.

Protocol-aware User Router

A native MySQL User Router selects one direct terminal MySQL backend from the verified downstream username:

listeners:
  mysql-routed:
    protocol: mysql
    port: 8486
    connections_limit: 200

authenticators:
  tenant-clients:
    provider: basic
    users:
      tenant_a_reader: ${TENANT_A_CLIENT_PASSWORD}
      tenant_b_reader: ${TENANT_B_CLIENT_PASSWORD}

caches:
  tenant-a-cache:
    provider: memory
  tenant-b-cache:
    provider: filesystem

backends:
  tenant-a-mysql:
    provider: mysql
    authenticator_name: tenant-clients
    origin_url: mysql://tenant_a_ro:REDACTED@mysql-a.example:3306/analytics
    cache_name: tenant-a-cache
    healthcheck:
      interval: 5s
      timeout: 3s

  tenant-b-mysql:
    provider: mysql
    authenticator_name: tenant-clients
    origin_url: mysql://tenant_b_ro:REDACTED@mysql-b.example:3306/analytics
    cache_name: tenant-b-cache
    healthcheck:
      interval: 5s
      timeout: 3s

  mysql-by-tenant:
    provider: alb
    listener_names: [mysql-routed]
    authenticator_name: tenant-clients
    alb:
      mechanism: ur
      user_router:
        default_backend: tenant-a-mysql
        users:
          tenant_a_reader:
            to_backend: tenant-a-mysql
          tenant_b_reader:
            to_backend: tenant-b-mysql

The listener-facing router owns the one downstream authentication exchange, admission, and TLS. Direct terminal MySQL backend entries must reference the same named authenticator to satisfy backend validation, but do not perform a second client exchange. The terminal owns its origin credentials, upstream TLS, cache, health, limits, and query policy. Routing occurs once before opening the upstream connection and remains sticky for the whole downstream session. Transactions, database changes, and session state never trigger rerouting.

Only direct MySQL terminal backends are supported. Nested ALBs, Rules, cycles, mixed terminal providers, empty routes, and MySQL to_user/to_credential remapping are configuration errors. A router requires an authenticator. An unmapped user uses default_backend when configured; without one, Trickster returns a bounded native MySQL no-route error. HTTP no_route_status_code is not sent on the native connection. A selected unhealthy or unavailable terminal fails as a MySQL availability error; the session is not silently routed elsewhere.

The verified username and selected terminal remain in cache identity. Route metrics use configured router/backend names and bounded outcomes, never the username.

Balancing sessions across replicas

A MySQL listener can also map to an ALB that balances its sessions across a pool of direct MySQL backends, such as read replicas:

backends:
  replica-1:
    provider: mysql
    authenticator_name: app-clients
    origin_url: mysql://app_ro:REDACTED@replica-1.example:3306/analytics
    healthcheck:
      interval: 5s

  replica-2:
    provider: mysql
    authenticator_name: app-clients
    origin_url: mysql://app_ro:REDACTED@replica-2.example:3306/analytics
    healthcheck:
      interval: 5s

  replicas:
    provider: alb
    listener_names: [mysql-replicas]
    authenticator_name: app-clients
    alb:
      mechanism: lc # rr, p2c, lc or hrw
      pool:
        - replica-1
        - name: replica-2
          weight: 2

The ALB owns the downstream authentication exchange, admission, and TLS, as a User Router does, and each pool member owns its origin credentials, cache, health, and query policy. A session is committed to one member after it authenticates and stays there until it ends. lc and p2c compare members by their open sessions; hrw keeps a client address (hrw.key: client_ip, the default) or a user name (hrw.key: user) on one member. weight, backup members, and healthy_floor apply as they do for any ALB. lt, the fanout mechanisms, nested ALBs, mixed providers, and autodiscovery are configuration errors. See Load Balancing Native Protocol Sessions.

A change of mechanism or weight applies to new sessions on reload; a change to the set of pool members restarts the listener.

Metrics, logs, and health

Important metrics include:

  • trickster_sql_query_analysis_total{backend_name,dialect,cache_mode,reason};
  • trickster_sql_query_rewrite_failures_total{backend_name,dialect,reason};
  • trickster_sql_query_cache_total{backend_name,dialect,cache_mode,cache_status};
  • trickster_proxy_request_duration_seconds and trickster_proxy_points_total with provider="mysql";
  • trickster_mysql_connections_total, trickster_mysql_active_connections, trickster_mysql_errors_total, trickster_mysql_route_selections_total, and trickster_mysql_command_duration_seconds; and
  • cache operation, usage, limit, and event metrics labeled by cache_name.

Example PromQL, replacing names to match the deployment:

sum by (cache_mode, reason) (rate(trickster_sql_query_analysis_total{backend_name="mysql1"}[5m]))
sum by (reason) (rate(trickster_sql_query_rewrite_failures_total{backend_name="mysql1"}[5m]))
sum by (cache_mode, cache_status) (rate(trickster_sql_query_cache_total{backend_name="mysql1"}[5m]))
histogram_quantile(0.95, sum by (le) (rate(trickster_proxy_request_duration_seconds_bucket{backend_name="mysql1",provider="mysql"}[5m])))
sum by (cache_status) (rate(trickster_proxy_points_total{backend_name="mysql1",provider="mysql"}[5m]))
sum by (operation, status) (rate(trickster_cache_operation_objects_total{cache_name="mysql-cache"}[5m]))
trickster_cache_usage_bytes{cache_name="mysql-cache"}
sum by (reason) (rate(trickster_cache_events_total{cache_name="mysql-cache",event="eviction"}[5m]))

Analysis logs contain bounded backend, cache mode, reason, and statement-type fields. Rewrite and protocol failures use bounded categories; SQL text, passwords, and usernames are not metric labels. Keep debug logging temporary in production because classification logs can be high volume.

Native health checks open a bounded connection with the backend’s configured origin credentials and TLS policy, then issue COM_PING. They use the common interval, timeout, failure, and recovery thresholds. HTTP health request fields do not apply. Health output reports sanitized authentication, TLS, timeout, refused-connection, and server-error categories. See Health Checks for management endpoint details.

Kubernetes deployment

The base examples expose the optional mysql Service/container port on 8486. The safe routed example keeps the complete credential-bearing configuration in deploy/kube/mysql-user-router-secret.yaml and replaces the base ConfigMap volume with deploy/kube/mysql-deployment-patch.yaml. Replace every placeholder and use an encrypted/external secret controller in production. Do not put an origin DSN, downstream passwords, private keys, or client keys in deploy/kube/configmap.yaml.

For direct TLS or mTLS, add the server, CA, and client certificate material to a Kubernetes TLS or opaque Secret, mount it read-only, and reference the mount paths from the backend tls block. Rotating that Secret requires the configuration/certificate reload path described above; established sessions are drained rather than changing transport identity in place.

Operations and troubleshooting

Kubernetes readiness

The native health scheduler does not delay listener startup or make the general health endpoint return a non-200 status. /trickster/ping proves only that the process is running, and a TCP readiness probe on port 8486 proves only that the native listener is accepting connections. The companion manifest uses that TCP probe. Treat the scheduled MySQL status published in /trickster/health?json as the origin-readiness signal in an external controller or monitoring rule; do not assume the endpoint’s HTTP status reflects origin health. Allow at least interval * failure_threshold or interval * recovery_threshold for a stable transition. User Router terminals report separately, and a failing selected terminal returns a native availability error rather than rerouting the session.

Capacity planning

Start from peak concurrent dashboard/client sessions. Reserve approximately one origin connection per admitted client, then add health and rollout headroom. Bound result rows/bytes below the pod or process memory budget and set max_object_size_bytes below both the result-byte limit and a practical fraction of cache capacity. For an in-memory cache, budget retained objects, index overhead, concurrent result encoding, and normal process memory. Use finite query, command, and idle timeouts so abandoned clients cannot retain connections indefinitely.

Repeated misses or proxy-only outcomes

  1. Group trickster_sql_query_analysis_total by cache_mode and reason.
  2. Confirm the query is deterministic, single-result, and outside a transaction or unsafe session.
  3. For DPC, inspect the expanded SQL: require a literal cadence and >= lower, < upper raw-time predicates.
  4. Confirm the requested interval contains at least one complete cadence bucket and that cache TTL, volatile window, and retention are suitable.
  5. Check username, selected backend, database, and time zone; these intentionally isolate keys.
  6. Inspect cache operation status and eviction metrics for admission failures or churn.

Authentication and TLS

  • Authentication failure: verify the named authenticator, plaintext/CSV credential source, exact username, supported native-password plugin, and filesystem permissions on a mounted users file.
  • Downstream TLS failure: verify the server pair, require_tls, client TLS mode, hostname, and TLS 1.2+ support. Do not use tls_port.
  • Upstream TLS failure: verify the single CA path, hostname in the origin URL, complete client pair when using mTLS, file permissions, and whether insecure_skip_verify was intentionally selected.

Origin and cache failures

During an origin outage, health transitions and MySQL error metrics should rise; existing safe cache hits may continue, while misses fail. Restore origin health before increasing connection limits. A cache-provider failure should surface in cache operation/event metrics and logs; query handling fails closed to the provider’s normal proxy behavior rather than treating an unknown cache state as a hit.

Certificate rotation and relevant configuration reloads drain affected native listeners. Monitor active connections, authentication/TLS errors, route selection outcomes, health state, rewrite failures, cache status, and p95 latency until the old sessions have drained.

Rollout and rollback

Canary the listener/backend configuration with a limited client population. Compare against the pre-release baseline and roll back for a sustained increase of 5 percentage points in proxy/origin/protocol errors, 10 percentage points in proxy-only classifications, 1 percentage point in rewrite failures, or 20% in p95 proxy latency. Roll back by restoring the previous configuration and binary/image on the release branch, reloading or restarting Trickster, and allowing the old MySQL listener to drain. Do not move an established session between versions. Preserve metrics and sanitized logs for the incident record.

Environment variables and reload behavior

The global TRK_ORIGIN_URL and TRK_ORIGIN_TYPE variables map only to the default backend and are suitable only for a basic single-backend deployment. There are no one-to-one TRK_* variables for MySQL listener limits, backend limits, TLS, health, or User Router options; configure those in YAML. Authenticator users values support ${VARIABLE} expansion, and supported users files and secret volume mounts are preferred for production. The origin_url field does not expand ${VARIABLE} in structured YAML; store a configuration containing an embedded DSN as a protected secret, not a public ConfigMap.

Validate before rollout:

trickster -validate-config -config /etc/trickster/trickster.yaml

SIGHUP and the management reload endpoint reload a changed file. Authentication, TLS, transport, routing, and terminal runtime changes restart and drain the affected MySQL listener. Cache policy changes apply to new work; existing cache objects remain subject to their stored TTL and normal eviction.

Known compatibility gaps

The initial release does not claim support for:

  • MySQL-compatible server products outside Oracle MySQL 8.4-9.7;
  • downstream plugins other than mysql_native_password or downstream mTLS;
  • prepared/binary protocols, compression, local infile, multi-results, stored procedures, binlog/replication commands, or connection-attribute semantics;
  • general SQL parsing beyond the Vitess-supported and compatibility-corpus shapes;
  • window functions, grouping sets, rollups, ambiguous time axes or buckets, unsafe boolean predicates, non-deterministic outputs, dynamic/non-zero-phase buckets, or inclusive-range DPC;
  • arbitrary session-state caching or cross-session mutation invalidation; and
  • nested/mixed MySQL User Router topologies or origin credential remapping.

Streamed result metadata and FOUND_ROWS()

Vitess does not expose the terminal status flags and warning count of a streamed result set. To preserve those fields for the downstream client, Trickster issues SHOW COUNT(*) WARNINGS on the same upstream connection after the stream ends. That diagnostic statement can update connection-local MySQL diagnostics, so FOUND_ROWS() immediately after a streamed result may reflect the hidden warning-count query rather than the client’s previous statement.

This is a known compatibility gap in the Vitess streaming API, not a Trickster-specific SQL parsing defect. A Vitess API that exposes terminal result metadata is required to remove the workaround.

The executable SQL contract is pkg/backends/mysql/testdata/compatibility/v1.json. The maintainer-facing frozen release matrix is in docs/developer/mysql-release-contract.md.