This is the multi-page printable view of this section. Click here to print.
Providers
1 - Prometheus Support
Trickster fully supports accelerating Prometheus, which we consider our First Class backend provider. They work great together, so you should give it a try!
Most configuration options that affect Prometheus reside in the main Backend config, since they generally apply to all TSDB providers alike.
Supported API Endpoints
Trickster supports the full Prometheus HTTP API (v1), including features introduced in Prometheus 3.x.
Cached Endpoints
| Endpoint | Cache Strategy | Scatter/Gather Merge |
|---|---|---|
/api/v1/query_range | Delta Proxy Cache | Yes |
/api/v1/query | Object Proxy Cache | Yes |
/api/v1/series | Object Proxy Cache | Yes |
/api/v1/labels | Object Proxy Cache | Yes |
/api/v1/label/<name>/values | Object Proxy Cache | Yes |
/api/v1/alerts | Proxy + Merge | Yes |
/api/v1/targets | Object Proxy Cache | No |
/api/v1/targets/metadata | Object Proxy Cache | No |
/api/v1/rules | Object Proxy Cache | No |
/api/v1/alertmanagers | Object Proxy Cache | No |
/api/v1/status/* | Object Proxy Cache | No |
/api/v1/query_exemplars | Object Proxy Cache | No |
/api/v1/metadata | Object Proxy Cache | No |
/api/v1/format_query | Object Proxy Cache | No |
/api/v1/parse_query | Object Proxy Cache | No |
/api/v1/scrape_pools | Object Proxy Cache | No |
/api/v1/features | Object Proxy Cache | No |
Range Query Notes
- A
stepmay be fractional seconds (for example1.5or0.5), as Prometheus accepts; Trickster keeps the fraction and sends sub-secondstartandendvalues with millisecond precision. - A
query_rangewhose expression uses the@ start()or@ end()modifier is proxied without caching. Those modifiers resolve against each request’s own range, so results fetched for part of a range could not be combined. A fixed@ <timestamp>is cached normally.
Step Alignment
Range queries default to the partial_end step alignment mode: points on the step grid from the grid point at or before start, plus Trickster’s Fast Forward point at the current time when the range reaches it. Fast Forward’s point is fetched through the object cache for partial_bucket_ttl, and skipped when the step is no longer than that.
truncateis the same without Fast Forward. It is the default withfast_forward_disable: true, which can’t be set alongsidestep_alignment.dropstarts at the grid point at or afterstart, so no point falls before the requested range.offsends each range query as the client sent it, and caches the response as an object for one minute.partialandpartial_startaren’t supported: PromQL evaluates at instants, so there is no partial bucket at the start.
A query can choose its own mode, or turn Fast Forward off, with a comment: up # trickster-step-align:drop or up # trickster-fast-forward:off. See Per-Query Instructions.
Proxied Endpoints (not cached)
| Endpoint | Notes |
|---|---|
/api/v1/notifications/live | SSE streaming |
/api/v1/write | Remote write (v1 and v2) |
/api/v1/otlp/v1/metrics | OTLP ingestion |
/api/v1/admin/* | Explicitly unsupported (returns error) |
All other /api/v1/* paths are reverse-proxied to the origin without caching.
Prometheus 3.x Features
- Native histograms are fully supported in query and query_range responses, including mixed series with both float samples and histogram samples.
- UTF-8 metric and label names (e.g.,
{"metric.name"}) are supported in queries and cache keys. - Query stats (
stats=allparameter) are cache-key differentiated, so responses with and without stats are cached separately.
Flavors
prometheus.flavor adapts a backend to a Prometheus-compatible managed
service, applying its supported routes, limits and defaults.
| flavor | service |
|---|---|
cloudwatch | Amazon CloudWatch PromQL |
amp | Amazon Managed Service for Prometheus |
mimir | Grafana Mimir |
thanos | Thanos |
backends:
cloudwatch:
provider: prometheus
prometheus:
flavor: cloudwatch
sigv4:
region: us-east-1
The cloudwatch and amp flavors limit the backend to the routes the service supports and
answer every other path themselves. See AWS Integration for what
each one sets. The mimir and thanos flavors change no routes; they select
the series source that TSM global evaluation reads, and that
evaluation reads their newest 5 minutes of samples again each time, since
their replicas and ingesters may still be receiving them.
Series Sources
An ALB using TSM global evaluation reads raw
series from its members rather than query results. series_api selects how
a backend serves them, and series_endpoint where:
series_api | protocol | default for flavor | series_endpoint |
|---|---|---|---|
remote_read | Prometheus remote read over HTTP, as streamed chunks or sampled responses | mimir | optional; an absolute http or https URL. The default is the origin_url path plus /api/v1/read |
storeapi | Thanos StoreAPI Series over gRPC | thanos | required; grpc://host:port for plaintext, or grpcs://host:port for TLS with the backend’s tls client settings |
Any Prometheus backend can set series_api, since Prometheus itself serves
remote read. A StoreAPI endpoint can be a Thanos Sidecar, Store Gateway,
Receive or Querier. The cloudwatch and amp flavors serve no raw series, and
reject both options.
backends:
prom-a:
provider: prometheus
origin_url: http://prom-a:9090
prometheus:
series_api: remote_read
mimir:
provider: prometheus
origin_url: http://mimir:8080/prometheus # remote read at /prometheus/api/v1/read
prometheus:
flavor: mimir
thanos-a:
provider: prometheus
origin_url: http://prom-a:9090
prometheus:
flavor: thanos
series_endpoint: grpc://prom-a-sidecar:10901
Remote read uses the backend’s HTTP client settings, and sends the client’s
end-to-end request headers plus the request_headers the backend’s matching
path sets, as a proxied request would; a tenant header such as
X-Scope-OrgID, or an Authorization header, reaches the source this way.
StoreAPI calls carry the same headers as gRPC metadata.
Both transports ask for compressed answers, which mostly shrinks the label sets the chunks travel with. Remote read asks for gzip, which Mimir answers with; Prometheus answers remote read uncompressed. StoreAPI calls ask for snappy, as Thanos components support; a store that rejects it is read again at once without compression. After that read succeeds, compression stays disabled until Trickster restarts or reloads.
A backend opens its StoreAPI connection on its first read, and closes it once a reload, shutdown, or autodiscovery replaces the backend and its requests have drained, as described for global evaluation.
Injecting Labels
Trickster can inject labels on a per-backend basis into Prometheus responses before returning them to the caller.
Here is the basic configuration for adding labels:
backends:
prom-1a:
provider: prometheus
origin_url: http://prometheus-us-east-1a:9090
prometheus:
labels:
datacenter: us-east-1a
prom-1b:
provider: prometheus
origin_url: http://prometheus-us-east-1b:9090
prometheus:
labels:
datacenter: us-east-1b
Interaction with ALB Merge Strategy
When using label injection with an ALB configured for Time Series Merge, injected labels are automatically stripped from responses before merging. This ensures that series from different backends are aggregated correctly, and the injected labels do not appear in the final response to the caller. See the ALB Merge Strategy documentation for details.
Max Query Range Limitation
Trickster supports enforcing a max_query_range limit on Prometheus backends. For details on how to configure and use query range limits, see the Query Range Limits documentation.
2 - ClickHouse Support
Trickster will accelerate ClickHouse queries that return time series data normally visualized on a dashboard. Acceleration works by using the Time Series Delta Proxy Cache to minimize the number and time range of queries to the upstream ClickHouse server.
Scope of Support
Trickster is tested with the official ClickHouse DataSource Plugin for Grafana v4.21.1 and supports acceleration of queries constructed by this plugin using its built-in time macros like $__fromTime and $__toTime. Delta caching with this an other plugins (e.g., Altinity/Vertamedia) depends on the supported query shapes below. Trickster also supports several other query formats that return “time series like” data.
Trickster also supports the ClickHouse Go SDK (clickhouse-go/v2) over its HTTP and Native protocols, including clickhouse.OpenDB.
Native Binary Protocol Support
Inbound and upstream protocols are configured independently:
- A named listener with
protocol: clickhouseaccepts Native client connections. - A ClickHouse backend with
protocol: nativeuses the Native protocol for its origin. An empty protocol orprotocol: httpuses HTTP.
listeners:
default:
protocol: http
port: 8480
clickhouse-native:
protocol: clickhouse
port: 8487
backends:
click1:
provider: clickhouse
origin_url: http://clickhouse:8123
listener_names: [default, clickhouse-native]
cache_name: default
This exposes one HTTP route at /click1/ and one Native listener on port 8487, backed by the same cache and HTTP ClickHouse origin. A Native listener must map to exactly one backend; ALB and user-router multiplexing are not supported.
To use a Native origin instead, set the backend protocol and point origin_url at ClickHouse’s Native port:
backends:
click1:
provider: clickhouse
origin_url: http://clickhouse:9000
protocol: native
All four HTTP/Native ingress and HTTP/Native origin combinations are supported. Native credentials, database, query settings, parameters, and query IDs are forwarded to the origin. Native identity pools are bounded to 64 credential/database combinations per backend.
TLS
Native listener TLS uses the backend’s server certificate. Set require_tls: true and provide both certificate paths:
listeners:
clickhouse-native-tls:
protocol: clickhouse
port: 9441
tls_watch_interval: 30s
backends:
click1-tls:
provider: clickhouse
origin_url: http://clickhouse:8123
listener_names: [clickhouse-native-tls]
require_tls: true
tls:
full_chain_cert_path: /etc/trickster/tls/server.crt
private_key_path: /etc/trickster/tls/server.key
Certificate changes are hot-swapped. Since require_tls applies to every binding of a backend, use a separate backend entry when plaintext and TLS listeners must coexist.
For a TLS Native origin, use an https origin URL with the Native protocol. The common backend TLS options configure origin verification and optional mutual TLS:
backends:
click1:
provider: clickhouse
origin_url: https://clickhouse:9440
protocol: native
tls:
certificate_authority_paths: [/etc/trickster/tls/origin-ca.crt]
client_cert_path: /etc/trickster/tls/client.crt
client_key_path: /etc/trickster/tls/client.key
Native Limitations
The Native listener supports SELECT queries, ping/pong, revision 54460 framing, structured errors, and LZ4 compression. INSERT, external tables, and session-changing USE or SET statements are rejected; select a database and settings per request. ClientCancel is accepted as a no-op and does not asynchronously cancel an active upstream request.
Supported native protocol data types: all integer types (8–256 bit), Float32/64, String, FixedString(N), DateTime, DateTime64, Date, Date32, UUID, IPv4, IPv6, Enum8/16, Bool, Nullable(T), Array(T), Map(K,V), Tuple(T1,T2,…), LowCardinality(T), and Decimal.
Delta-cacheable query results are decoded into Trickster’s dataset model and re-encoded in the client’s requested format. Both the TSV and the FORMAT Native origin readers support every scalar type above, Nullable(T), LowCardinality(T), Array(T), Map(K, V) and Tuple(...) (including named elements and nesting); compound values are carried in ClickHouse’s text-literal form ([1,'a'], {'k':1}, ('a',1)) and parsed back when re-encoding to Native. Nested, Variant, Dynamic and JSON columns are rejected with an explicit error on the delta path rather than decoded incorrectly; such queries should use a non-delta-cacheable shape.
Native wire clients receive Native blocks. HTTP clients using a Native origin may request JSON, Native, CSV, or TSV-family output; compound columns in CSV/TSV and unsupported formats return an error. The modeler recognizes Native origin responses through the X-ClickHouse-Format response header, and HTTP Native framing honors client_protocol_version.
Trickster parses incoming ClickHouse statements into a full abstract syntax tree using the AfterShip ClickHouse SQL parser, then applies its own semantic analysis to determine whether a query is eligible for time series delta caching and, if so, its timestamp column, bucket cadence, time range, grouping tags, and cache identity. The cache key is derived from a canonical form of the query in which the requested time range is replaced with placeholders, so requests for different time ranges of the same logical series share one delta cache entry.
Trickster’s analysis fails closed: a valid query whose shape cannot be proven safe for delta caching is never rewritten approximately. It is instead served through the Object Proxy Cache (OPC) or proxied directly, and the classification reason is exported through the trickster_sql_query_analysis_total metric.
If you find query or response structures that are not yet supported, or providing inconsistent or unexpected results, we’d love for you to report those. We also always welcome any contributions around this functionality.
Delta-Cacheable Queries
To be eligible for the delta cache, a query must be a single SELECT statement containing a recognized time-bucketing expression in its select list, a supported time range in its WHERE or PREWHERE clause, and a GROUP BY clause that includes the time bucket. Each requirement is described below.
Time-Bucketing Expressions
Exactly one select-list expression must match a supported bucket form:
Grafana Plugin Format
SELECT intDiv(toUInt32(time_col), 60) * 60 [* 1000] [AS alias]
This is the approach used by the Grafana plugin. The argument to the ClickHouse intDiv function is the step value in seconds, since the toUInt32 function on a datetime column returns the Unix epoch seconds. An optional output multiplier of 1000, 1000000, or 1000000000 selects millisecond, microsecond, or nanosecond output timestamps.
ClickHouse Time Grouping Functions
SELECT toStartOfInterval(time_col, INTERVAL n unit) [AS alias]
with a positive constant n and a unit of millisecond, second, minute, hour, day, or week; or one of the fixed-period functions:
toStartOfNanosecond
toStartOfMicrosecond
toStartOfMillisecond
toStartOfSecond
toStartOfMinute
toStartOfFiveMinute
toStartOfTenMinutes
toStartOfFifteenMinutes
timeSlot
toStartOfHour
toStartOfDay
toStartOfWeek
toMonday
date_trunc('unit', time_col) and dateTrunc support the same fixed units. toStartOfWeek uses ClickHouse’s Sunday phase while toMonday and weekly date_trunc use Monday. Calendar-length month, quarter, and year buckets are served through the OPC.
The time column may be a plain, qualified (table.col), or quoted identifier, optionally wrapped in toDateTime, toInt32, or toUInt32. Integer constants defined in a scalar WITH clause may be used for the step value. Timezone-parameter variants of these functions (for example toStartOfHour(time_col, 'America/Denver')) are not eligible for delta caching and are served through the OPC. The current SQL parser does not accept INTERVAL MICROSECOND or INTERVAL NANOSECOND; use the corresponding fixed toStartOf... function.
Determining the Requested Time Range
Time range predicates must appear in a top-level AND conjunction of the WHERE or PREWHERE clause. Predicates joined by OR or negated with NOT make the query ineligible for delta caching.
Two predicate targets are supported, with different rules:
- The raw time column (the column inside the bucket function): the lower bound must be inclusive (
>=, or the lower end ofBETWEEN); a strict>is served through the OPC. The upper bound may be exclusive (<) or inclusive (<=, or the upper end ofBETWEEN). Values that do not fall on bucket boundaries — such as the live ranges produced by Grafana’s$__fromTimeand$__toTimemacros — leave partial buckets at the edges, which are never cached as complete aggregates; the backend’s step alignment mode decides what the response shows for them. Under the default,drop, they are left out, and an inclusive upper bound leaves out the bucket that contains it, because that bucket is only partly covered. A range with no complete bucket is sent to ClickHouse as written, and its response is cached as an object forpartial_bucket_ttl. - The bucket alias (the output of the bucket expression):
>,>=,<,<=, andBETWEENare all supported, because bucket outputs are discrete; Trickster aligns each comparator to the first and last included bucket.
Bound values may be expressed as epoch integers, ClickHouse string dates in the form 2006-01-02 15:04:05 (or date-only, or RFC3339), toDateTime(n), toDateTime64(n, precision), or toDate(n) wrappers, WITH-clause constants, or now()/now64() with optional addition or subtraction of seconds. DateTime64 precision is retained. Floating epoch bounds and timezone-qualified conversions such as toDateTime(n, 'America/Denver') are not eligible.
If no upper bound is present, Trickster inserts a safe upper bound into origin requests automatically and caches every complete bucket up to the current time. The still-filling bucket is never cached: under the default step_alignment, drop, the response ends before it, and the partial and partial_end modes fetch it from the origin on each request.
Examples of delta-cacheable time range clauses (for a one-minute bucket cadence):
WHERE t >= '2020-10-15 00:00:00' AND t <= '2020-10-16 12:00:00' -- bucket alias
WHERE t BETWEEN 1574686320 AND 1574689920 -- bucket alias
WHERE time_col >= toDateTime(1574686320) AND time_col < toDateTime(1574689920)
WHERE t >= now() - 3600 AND t < now() -- bucket alias
Secondary date-range predicates whose values match the primary range — such as the Date-typed partition filters emitted by the Grafana plugin — are recognized and rewritten in step with the primary range.
Grouping and Result Shape
The GROUP BY clause must include the time bucket (by alias or by its full expression), and every non-aggregate column in the select list must also be grouped. Grouped columns become the series tags in the cached time series. Queries using GROUP BY ... WITH CUBE/ROLLUP, grouping on expressions that are not selected, or leaving a selected dimension ungrouped are served through the OPC.
Queries whose values in one time bucket can depend on other buckets, or on the whole result, are also served through the OPC:
LIMIT,LIMIT BYandTOP;- window functions (
OVER (...)or aWINDOWclause) and cross-row functions such asneighbor,lagInFrame,leadInFrameand therunning*family; WITH TOTALSandORDER BY ... WITH FILL(includingINTERPOLATE);- a query-level
SETTINGSclause; - subqueries, common table expressions and joins, which can carry time filters of their own (
ARRAY JOINis still delta-cacheable).
Ordering
Trickster rebuilds a delta-cached response from its cached buckets in ascending time order. A query is therefore delta-cacheable only with no ORDER BY, or with a single ascending term on the time bucket (by alias, by the bucket expression, or by its position, as in Grafana’s ORDER BY time). Any other ordering — descending time, additional terms, or other columns — is served through the OPC, where the origin’s row order is kept.
Output Formats
Delta-cacheable queries may specify FORMAT JSON, CSV, CSVWithNames, TabSeparated (TSV), TabSeparatedWithNames, or TabSeparatedWithNamesAndTypes, or omit the FORMAT clause. Trickster requests TSVWithNamesAndTypes from the origin and re-marshals cached data into the client’s requested format.
Trickster writes each format as ClickHouse writes it:
- NULL is
\Nin TSV and CSV, andnullin JSON. An empty string stays an empty string. - Numbers are bare JSON numbers, and floats use ClickHouse’s text (
1e21,1e-7,nan,inf). JSON writes NaN and infinities asnull. - CSV quotes every text value. A
FixedStringis padded to its width. - Compound values:
Array,MapandTuplevalues are their ClickHouse literals in TSV and CSV, and JSON arrays and objects in JSON.
Responses honor output_format_json_quote_64bit_integers, output_format_json_quote_decimals, output_format_json_quote_denormals and date_time_output_format.
Time Zones
A DateTime without a zone in its type is written in the session’s time zone: the request’s session_timezone URL parameter, or else the server’s. Trickster caches UTC and writes each response in its client’s zone, so clients in different zones share cache entries.
The settings that change only how a response is written aren’t part of the cache key: date_time_output_format, the JSON quote settings, default_format, and client_protocol_version.
The zone is part of the key only when it changes which rows a query returns:
- Buckets that start on the zone’s clock: a daily bucket starts at the zone’s midnight, and an hourly bucket in a zone offset by a half hour starts on the half hour. The zone is keyed unless every offset it has over the queried range is a whole number of buckets. So hourly and finer buckets in New York or UTC share entries; Kolkata hourly buckets, and daily buckets outside UTC, don’t.
- Zone-sensitive expressions: another date or time function that reads the zone, such as
toHour()ortoday(), or text compared as a time.
A query whose time bounds are text (ts >= '2026-09-29 00:00:00') or dates (toDate(…)) is read by ClickHouse in the session’s zone. Outside UTC it’s served through the OPC rather than delta-cached.
Trickster asks the origin for date_time_output_format=iso, which writes every DateTime as UTC. Its cache holds UTC, without the ambiguity of the hour a clock repeats when daylight saving time ends. It writes its DateTime64 range bounds with an explicit 'UTC' zone, so the session’s zone doesn’t move them.
Trickster learns the server’s zone from the X-ClickHouse-Timezone header of the origin’s responses. A backend that hasn’t seen one yet asks the server with SELECT timezone() before it caches a query. If that probe fails, for example because the origin requires credentials, Trickster proxies the request uncached to learn the zone from the response. If the zone is still unknown after that, Trickster assumes UTC and logs a warning.
Non-Time-Series Queries
Queries that are not cacheable as time series — such as LIMIT-based queries, queries with set operations (UNION, EXCEPT, INTERSECT), SELECT 1 health checks, or SDK handshake requests — are transparently proxied to the upstream ClickHouse server. These requests are cached using the Object Proxy Cache (OPC).
Cache Keys
Both the OPC and the delta proxy cache key a request on its SQL statement and on every other URL parameter, because query parameters (param_<name> values for {name:Type} placeholders) and settings can change the result. The native listener forwards its query parameters and settings the same way. Only transport parameters that cannot change the result are left out of the key: query_id, session_timeout, session_check, send_progress_in_http_headers, http_headers_progress_interval_ms, wait_end_of_query, buffer_size, log_comment, log_queries, quota_key, and add_http_cors_header.
Requests that carry a session_id are proxied without caching. A session’s SET statements change the results of later queries in that session, and that state is not part of any cache key.
Health and Ping Endpoint
Trickster exposes a /ping endpoint that returns a health check response, matching the endpoint provided by ClickHouse itself. This enables compatibility with clients and SDKs that probe /ping during connection initialization.
Step Alignment
ClickHouse supports every step alignment mode, and defaults to drop: responses hold complete buckets only, so the partial buckets at the edges of a live range, and the still-filling bucket at the present, are left out. truncate answers the whole first bucket instead.
partial, partial_start and partial_end add the partial buckets as ClickHouse computes them over the client’s own range. Each is a small query of its own, sent through the object cache and kept for partial_bucket_ttl, never in the time series cache, so these modes cost up to two extra origin queries per request. Fast Forward doesn’t apply to ClickHouse; partial_end shows the still-filling bucket instead.
Complete buckets inside the configured volatile_window are refetched until they settle. A query can choose its own mode or window with a comment directive, such as /* trickster-step-align:partial_end */; see Per-Query Instructions. Directives in # comments aren’t read, because Trickster’s ClickHouse SQL parser rejects them: a statement with one is served through the OPC, or proxied.
Observability
Query classification outcomes are exported through two low-cardinality metrics that never include query text:
trickster_sql_query_analysis_total— labeled by backend, dialect, cache mode (delta,object,none), and a stable reason code such asdelta_cacheable,unsafe_predicate, orunsupported_bucket.trickster_sql_query_rewrite_failures_total— counts failures to render an origin request from a cached query plan.
With debug logging enabled, classification decisions are also logged with the same structured reason codes.
Max Query Range Limitation
Trickster supports enforcing a max_query_range limit on ClickHouse backends. For details on how to configure and use query range limits, see the Query Range Limits documentation.
3 - InfluxDB Support
Trickster provides support for accelerating InfluxDB queries that return time series data normally visualized on a dashboard. Acceleration works by using the Time Series Delta Proxy Cache to minimize the number and time range of queries to the upstream InfluxDB server.
Scope of Support
Trickster is tested with the built-in InfluxDB DataSource Plugin for Grafana v5.0.0.
Trickster uses InfluxDB-provided packages to parse and normalize queries for caching and acceleration. If you find query or response structures that are not yet supported, or providing inconsistent or unexpected results, we’d love for you to report those so we can further improve our InfluxDB support.
Trickster supports integrations with InfluxDB 1.x, 2.x, and 3.x.
Prometheus Remote Read
For InfluxDB 1.x, Trickster accelerates Prometheus remote-read requests sent to
POST /api/v1/prom/read. Point Prometheus at the same endpoint on Trickster and
retain the InfluxDB query parameters, for example:
remote_read:
- url: http://trickster:8480/api/v1/prom/read?db=metrics&rp=autogen
The InfluxDB endpoint supports one query per request and the Prometheus
SAMPLES response type. Trickster delta-caches requests with that shape. Other
request shapes continue through the normal proxy path so that InfluxDB remains
responsible for its response and error behavior.
Raw remote-read samples do not advertise a guaranteed interval, so point-count
sharding cannot split their extents without risking gaps. Requests use the
normal proxy path when shard_max_size_points is enabled; time-based sharding
remains supported. Cache retention (oldest and lru) and volatile_window_points
use a positive hints.step_ms; a request without that hint uses the
normal proxy path, including Prometheus instant queries with a zero step hint.
The hint is retained in serialized cache entries and does not change the 1 ms
precision used to locate missing raw samples.
Cacheable remote reads request Snappy directly from InfluxDB to avoid an extra
HTTP compression layer. Reconstructed responses carry Content-Encoding: snappy
on cache misses, partial hits, and full hits.
The db, rp, u, and p query parameters and the Authorization header are
part of the cache identity. They are forwarded unchanged unless a path-level
request rewrite or header configuration overrides them.
InfluxDB 3.x Support
Trickster supports InfluxDB 3.x via both the native v3 API endpoints and the v1/v2 compatibility endpoints.
Supported v3 Endpoints
GET/POST /api/v3/query_sql— SQL queries with delta-proxy cachingGET/POST /api/v3/query_influxql— InfluxQL queries with delta-proxy cachingPOST /api/v3/write_lp— line protocol writes (proxied, not cached)
All other v3 API paths (including the /api/v3/configure/* management endpoints, with any HTTP method) are proxied through untouched.
Query requests on both query endpoints may arrive as URL parameters (GET), an application/json document ({"q": ..., "db": ..., "format": ..., "params": ...}), a form-encoded body, or a raw statement body — all four shapes are parsed and cached equivalently, and the db and params values participate in the cache identity.
InfluxQL over v3
/api/v3/query_influxql requests use the same delta-proxy caching as the v1 /query endpoint (queries with a GROUP BY time(...) interval, optionally with an offset such as time(1h, 15m), and a time-bounded WHERE clause; a non-UTC tz() clause shifts bucket boundaries by the zone offset, so those queries use the object proxy cache instead), but speak the v3 request/response shapes: the v3 request document above and the v3 tabular response formats, including the iox::measurement column, which Trickster treats as a series tag alongside any GROUP BY tags.
On both /query and /api/v3/query_influxql, statements whose values in one time bucket depend on other buckets or on the whole result use the object proxy cache instead of delta caching: fill(previous) and fill(linear); transformations such as derivative(), non_negative_derivative(), difference(), moving_average(), cumulative_sum(), elapsed(), integral() and the other technical analysis functions; LIMIT, OFFSET, SLIMIT and SOFFSET; and subqueries, whose own time ranges are not rewritten per fetch.
SQL Query Caching
SQL queries using date_bin() or date_trunc() for time binning are parsed for time range extraction and step detection, enabling delta-proxy caching. Trickster extracts time ranges from WHERE clauses and intervals from date_bin(INTERVAL '...', time) or date_trunc('unit', time) in SELECT. Live, unaligned time ranges (the shape dashboard tools emit) are rounded inward to complete buckets. Grouped queries (GROUP BY 1, host) are cached per tag series.
Example query that Trickster will accelerate:
SELECT date_bin(INTERVAL '1 hour', time) AS time, avg(temperature)
FROM weather
WHERE time >= '2024-01-01 00:00:00' AND time < '2024-01-02 00:00:00'
GROUP BY 1
SELECT queries that cannot be delta-cached (no fixed-cadence time bucket, joins, subqueries, window functions, compound selects such as UNION, LIMIT, variable-length buckets like '1 month', or unsafe time predicates) fall back to the object proxy cache, which caches the whole response briefly and passes results through unchanged. Non-SELECT statements and parameterized queries (a params field in the request) are proxied to the origin without delta caching.
The still-filling final bucket of a range that reaches the present is never cached. Under the SQL default step_alignment, drop, the response ends before it; the partial and partial_end modes, and InfluxQL’s default partial_end, fetch it from the origin on each request. volatile_window applies only to complete buckets.
Step Alignment
Each query language has its own step alignment support and default:
| Query language | Supported modes | Default |
|---|---|---|
| InfluxQL, 1.x and over 3.x | all | partial_end |
| SQL, over HTTP and Flight SQL | all | drop |
| Flux | truncate, off | truncate |
| Prometheus remote read | truncate | truncate |
A backend’s step_alignment may name any of these modes; a query in a language that doesn’t support it runs in that language’s default, and the fallback is counted in trickster_step_alignment_fallbacks_total. The partial modes fetch each partial bucket as a small query of its own through the object cache, so they cost up to two extra origin queries per request. A query can choose its own mode with a comment: -- trickster-step-align:drop in InfluxQL and SQL, // trickster-step-align:off in Flux. See Per-Query Instructions.
Response Formats
Trickster supports the following v3 response formats, controlled by the format query parameter (in the URL or the request document) or, when no format is given, the Accept header (application/json, application/jsonl, text/csv, …):
json(default) — JSON array of objectsjsonl— JSON Lines (one JSON object per line)csv— standard CSV with header row
The parquet and pretty formats are not supported for caching and will be proxied through.
InfluxDB 3 leaves a row’s null values out of JSON and JSON Lines, so a column can first appear on any row. Trickster reads every column a response names, on whichever row first names it.
v1/v2 Compatibility
InfluxDB 3.x ships with v1 and v2 compatibility endpoints. Trickster’s existing InfluxQL support works against these endpoints with no additional configuration — just point Trickster at the v3 instance and query via /query.
Flight SQL (gRPC)
InfluxDB 3.x exposes SQL via Apache Arrow Flight SQL on gRPC in addition to HTTP. Grafana’s InfluxDB datasource in SQL mode, the Python/Rust/Java SDKs, and ADBC all default to Flight SQL, so HTTP-only caching misses a significant fraction of real-world query traffic. The protocol-level (backend-agnostic) listener documentation lives in Apache Arrow Flight SQL Listeners; this section covers the InfluxDB specifics.
Trickster can expose a Flight SQL server that proxies to an upstream Flight SQL endpoint and caches the Arrow IPC byte stream. Enable it by defining a listener with the flight-sql protocol and mapping exactly one InfluxDB backend to it (alongside its usual HTTP listener):
listeners:
influx3-flight:
protocol: flight-sql # Apache Arrow Flight SQL over gRPC
port: 8485
backends:
influx3:
provider: influxdb
origin_url: 'http://influxdb3:8181/'
listener_names: [default, influx3-flight]
influxdb:
flight_upstream_address: 'influxdb3:8181' # optional, defaults to origin_url host
flight_max_response_bytes: 134217728 # optional, defaults to 128MiB
flight_max_buffered_bytes: 536870912 # optional, aggregate in-flight budget; defaults to 512MiB
flight_allowed_location_hosts: # optional exact authorities for alternate endpoints
- 'influxdb3-replica:8181'
The authorization and database headers are forwarded from the client through to the upstream, and every cache entry is keyed by the backend name, the database/bucket-name headers, and a hash of the authorization header — so tenants and databases never share cache entries. Requests that send no authorization header share one anonymous cache scope, mirroring the access the upstream would grant them.
Statement queries are served through a three-tier cache:
- Delta proxy cache — queries the SQL analyzer classifies as delta-cacheable (the same
date_bin()/date_trunc()shapes as the HTTP path) are cached by time extent: repeat and overlapping queries fetch only the missing sub-ranges from the upstream, and responses are rebuilt into Arrow record batches conforming to the response’s original schema. A query’sORDER BYis carried through that rebuild, so a cache hit returns rows in the requested order; ordering terms that do not resolve to a select-list output fall to the next tier. Entries use the backend’stimeseries_ttland honorvolatile_window. Under the defaultstep_alignment,drop, a range holding at least one complete bucket returns only complete buckets: partial buckets at either end of the range, including the still-filling bucket of a range that reaches the present, are left out. Thepartial,partial_startandpartial_endmodes fetch those buckets from the upstream over the client’s own range, through the object cache forpartial_bucket_ttl, and never delta-cache them. A range with no complete bucket gets the upstream’s own result for the statement, partial and still-filling buckets included, cached as an object forpartial_bucket_ttl. Responses whose Arrow schemas the delta model cannot represent (nested types, non-string dictionaries, …) automatically fall to the next tier. - Object cache — everything else cacheable is stored as the verbatim Arrow IPC byte stream, returned byte-identically, with a lifetime of
influxdb.flight_cache_ttl(default 60s). Metadata RPCs and prepared statements always use this tier. - Proxy — statements referencing nondeterministic functions (
now(),current_timestamp,random(), …) and non-SELECT statements are never cached.
Flight SQL TLS
The Flight listener serves TLS when its mapped backend’s tls block presents a certificate and key (the same mechanism HTTP listeners use); certificate rotation applies on config reload without dropping the listener. Without a certificate the listener serves plaintext gRPC. The upstream dial uses TLS when influxdb.flight_upstream_tls is set, honoring the backend tls block’s insecure_skip_verify.
Unsupported Flight SQL RPCs
Trickster proxies queries, result-set schema requests (GetSchemaStatement/GetSchemaPreparedStatement, which ADBC drivers probe on connect), the metadata RPCs below, and the prepared-statement lifecycle. Other Flight SQL RPCs — writes/updates/ingest (ExecuteUpdate, DoPut ingestion), transactions, savepoints, Substrait plans, query cancellation, endpoint renewal, and session options — return gRPC Unimplemented; clients requiring them should connect to InfluxDB directly. A prepared-statement binding must arrive as a single record batch; a client streaming several batches for one binding receives Unimplemented rather than a binding taken from its first batch alone.
Flight SQL listeners share Trickster’s standard listener lifecycle: connection limits (connections_limit), graceful drain on SIGTERM (active streams are drained until the configured drain timeout, then closed), and config reload (SIGHUP) — a reload with an unchanged backend configuration keeps serving on the existing socket, while a changed configuration drains the old server and rebinds.
Metadata RPCs
ADBC clients (Grafana’s SQL datasource, the Python adbc_driver_flightsql, etc.) probe the following RPCs on connect to populate schema browsers and query editors. Trickster proxies these to the upstream and caches the Arrow IPC response:
GetFlightInfoTables/DoGetTablesGetFlightInfoCatalogs/DoGetCatalogsGetFlightInfoSchemas/DoGetDBSchemasGetFlightInfoTableTypes/DoGetTableTypesGetFlightInfoSqlInfo/DoGetSqlInfoGetFlightInfoXdbcTypeInfo/DoGetXdbcTypeInfoGetFlightInfoPrimaryKeys/DoGetPrimaryKeysGetFlightInfoImportedKeys/DoGetImportedKeysGetFlightInfoExportedKeys/DoGetExportedKeysGetFlightInfoCrossReference/DoGetCrossReference
Flight SQL response size
Flight responses are buffered whole so they can be cached. influxdb.flight_max_response_bytes bounds one upstream response (128MiB by default), while influxdb.flight_max_buffered_bytes bounds response bytes concurrently assembled or streamed (512MiB by default). Exceeding either bound returns gRPC ResourceExhausted; a negative value removes that bound.
A FlightInfo partitioned across several endpoints is consumed in full. Alternate endpoint locations are followed only when their exact host:port authority appears in influxdb.flight_allowed_location_hosts, because the request’s authorization and database metadata are forwarded to that host. A TLS upstream never follows a plaintext alternate location. influxdb.flight_max_location_clients bounds cached alternate connections and defaults to 16.
Prepared Statements
Trickster proxies the full prepared statement lifecycle: CreatePreparedStatement, DoPutPreparedStatementQuery (parameter binding), DoGetPreparedStatement, ClosePreparedStatement. A parameterless prepared statement is equivalent to executing its statement text, so it is served through the same three cache tiers as plain statement queries — delta caching included — and shares their cache entries. Executions with bound parameters are cached whole, keyed by the bound parameter hash so two clients running the same statement with different values don’t alias. Prepared statements abandoned by disconnected clients are closed upstream after 15 minutes of inactivity.
Note: InfluxDB 3 Core 3.10 reports a parameter schema at prepare time but does not resolve bound values during query planning (upstream limitation). Parameterless prepared statements work end-to-end; parameterized ones require a newer Core or Enterprise build.
Flux Language Support
Trickster supports the Flux Query Language for general/basic usage with InfluxDB 1.x and 2.x. Flux is not supported in InfluxDB 3.x.
The delta-proxy cache accepts now() as a range() bound and handles queries with aggregateWindow(every: ...) – the common Grafana shape. Multi-table Flux CSV responses (one table per series in the result set) are also read correctly.
InfluxDB answers Flux queries only in annotated CSV. Trickster can also answer them in JSON when the request’s Accept header asks for application/json. Each table’s records are written as objects, and a NaN or infinite value is written as null, as InfluxDB 3’s JSON writes it.
aggregateWindow may also set offset and timeSrc ("_stop", the default, or "_start"); Trickster aligns its fetches to the resulting window boundaries and labels. Queries whose windows cannot be mapped onto a fixed step grid are proxied without delta caching: a bare window() (its records keep their own _time), a period that differs from every, or a location.
Queries whose values in one window depend on other windows, or on the whole range, are also proxied without delta caching:
limit(),tail(),derivative(),difference(),cumulativeSum(),movingAverage()and the other moving-average and technical analysis functions,elapsed(),integral(),stateCount()andstateDuration();fill(usePrevious: true);timeShift(), which moves labels off the window grid;- reducers, selectors and
sort()that followaggregateWindow(), such asmax(),last()ortop(), which collapse or reorder every window; - more than one
from(), as in union and join queries.
Trickster currently does not properly handle schema changes within a response CSV body (e.g., multiple CSVs in the same document with their own #annotation and header rows). We will fully support this use case in a future beta.
Max Query Range Limitation
Trickster supports enforcing a max_query_range limit on InfluxDB backends. For details on how to configure and use query range limits, see the Query Range Limits documentation.
4 - Apache Druid Support
Trickster accelerates eligible Apache Druid native JSON queries with the Time
Series Delta Proxy Cache (DPC). Configure druid as the backend provider and
point origin_url at a Druid Broker or Router.
backends:
druid1:
provider: druid
origin_url: http://druid-router:8888
cache_name: default
volatile_window: 60s
timeseries_retention_factor: 2048
Native query acceleration
POST /druid/v2 uses delta caching when all of these conditions hold:
queryTypeistimeseries,groupBy, ortopN.intervalscontains exactly one ISO-8601 half-open interval.granularityhas a fixed width:- a simple granularity from
secondthroughday; - a positive
durationgranularity in milliseconds; or - a fixed ISO-8601
periodgranularity in UTC, such asPT15MorP1D.
- a simple granularity from
- The selected context does not request an alternate native response shape.
Trickster removes the interval from the logical cache identity and rewrites
only missing extents into Druid’s [start,end) form. Druid’s end is exclusive,
so the final cached bucket is rendered as extent.End + granularity.
An interval boundary inside a bucket is handled by the backend’s
step_alignment, which defaults to partial for native queries: complete
buckets come from the delta cache, and each edge bucket is Druid’s answer over
the part of it the interval covers, fetched through the Object Proxy Cache for
partial_bucket_ttl and never delta cached. A native query can choose its own
mode, or its own volatile window, with keys in its context map, such as
"trickster-step-align": "drop", which Trickster leaves out of the cache key
and Druid ignores. See Step Alignment and
Per-Query Instructions.
The response model preserves native timeseries, groupBy, and topN JSON
shapes. Grouping dimensions become DataSet tags internally, and per-bucket
positions keep groupBy and topN rows in Druid’s order. A descending: true
query is cached in time order like any other and written newest first, as Druid
writes it. See Response fidelity.
Object-cache fallback
A valid read query that is unsafe for delta merging automatically uses the Object Proxy Cache (OPC) with a one-minute fallback TTL when Druid does not provide explicit freshness headers. This includes:
- other native query types such as
scan,search,segmentMetadata,datasourceMetadata, andtimeBoundary; - multiple intervals;
all,none,week,month,quarter, andyearsimple granularities;- calendar-width periods or period granularities in a non-UTC time zone;
- timeseries
limit, which keeps only the first rows of the whole result; - groupBy limits or dimension-first result ordering; and
- response-changing contexts such as
bySegment,serializeDateTimeAsLong, timeseriesgrandTotal, or groupByresultAsArray.
The following native context keys are transport controls and are omitted from
the cache identity: queryId, sqlQueryId, priority, timeout, and
queryDeadline. They remain unchanged in the request sent to Druid. Semantic
context keys, including skipEmptyBuckets, remain part of the cache identity.
Druid SQL acceleration
POST /druid/v2/sql uses the same delta cache for a deliberately conservative
subset of Druid SQL. The request must be a JSON object using either the default
or explicit resultFormat: "object", or resultFormat: "array" with
header: true. The statement must be a single-table SELECT that has:
- one
TIME_FLOOR(__time, <fixed UTC period>)bucket expression with an explicit alias; - a
GROUP BYcontaining that bucket and every selected dimension; and - a complete lower/upper time range on
__time(unaligned edges follow the backend’sstep_alignment,dropby default for SQL; partial edge buckets are never delta cached). A SQL comment such as-- trickster-step-align:partialchooses the query’s own mode, and wins over the same key in the request’scontext.
The shared CockroachDB SQL analyzer canonicalizes the statement and renders
each missing extent while preserving the original JSON context on the wire.
Literal MILLIS_TO_TIMESTAMP(...) bounds generated by the Grafana Druid plugin
are supported. Object rows and header-plus-array rows are converted to the
standard Trickster DataSet internally and emitted in their requested shape
after cache merging. Other valid SELECT statements and response formats
remain safe OPC fallbacks; non-read statements, SQL task requests, and malformed
requests are proxied.
Response fidelity
A response built from the cache is written the way Druid writes it:
- Numbers, objects and arrays are written exactly as Druid wrote them, so a
double keeps its form, such as
5319.0or9.999999999999999E22. - Strings and keys are escaped the way Druid’s JSON writer escapes them.
- Each row’s members are in Druid’s order. For
timeseriesandgroupByresults, that is the order of Druid’s Java hash maps. - Native responses end without a newline, and SQL responses end with one.
- SQL rows without an
ORDER BYfollow Druid’s order: by theGROUP BYcolumns, with the time bucket first unless theGROUP BYends with it. - Rows that tie on an
ORDER BYalso follow Druid’s order. - Nulls sort lowest, as in Druid: first when ascending and last when descending.
Two things can still differ from the response Druid builds for the same request, and both also vary between Druid’s own responses:
- topN key order. Druid orders a topN row’s keys differently depending on whether its segment cache served the row, so one query can come back with either order. Trickster writes the dimension, then the aggregations, then the post-aggregations.
- Floating-point sums. Druid adds a bucket’s values in an order that
depends on the query’s interval. So the last digits of a
doubleSumorfloatSumcan differ between two Druid queries that cover the same bucket. A cached bucket keeps the value from the query that fetched it.
Route policy
| Route | Method | Policy |
|---|---|---|
/druid/v2 | POST | DPC when eligible, otherwise OPC or proxy |
/druid/v2/sql | POST | DPC for eligible SQL, otherwise OPC or proxy |
/druid/v2/sql/task | POST | Proxy only |
/druid/v2/datasources... | GET | OPC |
/status/health | GET | Health probe; expects true |
| all other routes | any | Proxy only |
SQL ingestion and management endpoints, ALB time-series merging,
scan/search delta caching, and Fast Forward are not supported. Fast Forward
is disabled for every Druid backend. A 60-second volatile_window is used when
the backend does not configure one, so recently ingested buckets can be
refreshed before segments settle.
Observability
trickster_druid_query_analysis_totalcounts classifications by backend, cache mode (delta,object, orproxy), and stable reason code.trickster_druid_query_rewrite_failures_totalcounts failed missing-extent rewrites by backend and fixed failure category.
Neither metric includes query text, datasource names, or request IDs.
5 - Graphite Provider
Trickster accelerates Graphite’s render API with the Delta Proxy Cache: it caches each metric at the origin’s native resolution and, on subsequent requests, fetches only the time ranges it does not already hold.
Graphite makes that harder than most time series databases, because a render
request does not say what resolution it will be answered at. Whisper chooses an
archive based on how old the query’s from is, and no API exposes a metric’s
archive ladder. Trickster therefore learns each ladder by probing the origin,
caches what it learns, and accelerates only the requests whose resolution it can
predict. Everything else is served correctly through ordinary object caching.
Specify graphite as the provider:
backends:
graphite1:
provider: graphite
origin_url: 'http://graphite.example.com:80'
cache_name: default
That is the whole minimum configuration. Graphite’s cache sizing differs from every other provider’s, so the provider supplies its own defaults rather than asking operators to discover them; see Sizing for what they are and when to change them.
Compatibility
Tested against graphite-web 1.1.10 with Whisper storage, and written against
its source. It should work with any Graphite-protocol origin that serves
/render and /metrics/expand, including go-carbon’s carbonserver,
graphite-clickhouse and carbonapi, but only graphite-web is verified.
| Client | Status |
|---|---|
| Grafana’s Graphite data source | Verified, GET and POST form bodies |
Direct /render API clients | Verified |
| Graphite Composer / dashboard UI | Proxied, not accelerated (image formats) |
| Response format | Behavior |
|---|---|
json | Accelerated. Also what Trickster requests upstream. |
raw, csv, msgpack | Accelerated; rendered from the cached series |
png, svg, pdf, pickle, dygraph, rickshaw | Proxied and object-cached |
Trickster always requests format=json from the origin regardless of what the
client asked for, and renders the client’s format from the cached series. JSON
is the only render format that carries the tags object, which Grafana relies
on, and it is what makes one cached series able to serve every output format.
Configuration
Every common backend option applies (cache_name, timeseries_ttl,
timeseries_retention_factor, volatile_window, max_object_size_bytes,
timeout, healthcheck, paths, TLS, authenticators); two of them,
max_object_size_bytes and timeseries_retention_factor, take
Graphite-specific defaults, as Sizing explains.
Per-path request_headers and request_params apply to the synthetic
requests resolution makes (probes, wildcard expansion) as well as to proxied
traffic, so an origin behind a static API key or one that needs local=1
works for learning too. What cannot work is per-client upstream identity:
resolution state (learned ladders, wildcard expansion tokens, the registry
generation) is backend-wide, and synthetic requests carry only the backend’s
configured identity. Trickster therefore declines acceleration — the
request is served correctly through the object cache, which keys on the
client’s Authorization and every declared result-affecting input —
whenever a render request carries an Authorization header, a header named
in the render path’s cache_key_headers, or a parameter or form field
named in its cache_key_params/cache_key_form_fields (such as local,
which selects the clustered origin view) that the path’s
request_headers/request_params does not statically override, and
whenever the render path or the backend itself has a request rewriter
(req_rewriter_name) attached, since a rewriter can change the upstream
host, path, headers, or parameters in ways synthetic requests do not see. The fallback is counted as
trickster_graphite_fallbacks_total{reason="client_identity"}.
If every client shares one upstream identity — Grafana with a configured
datasource credential is the common case — supply it in the graphite
block to re-enable acceleration:
backends:
graphite1:
provider: graphite
origin_url: 'http://graphite.example.com:80'
graphite:
origin_username: 'metrics'
origin_password: '<the origin password>'
For origins using a non-Basic scheme, set origin_authorization to the
verbatim header value (e.g., 'Bearer <token>') instead of the
username/password pair.
The credential becomes the Authorization request header of every path —
proxied client traffic, synthetic resolution requests (probes, wildcard
expansion) and the default health check alike — so the origin sees one
static identity, a client-supplied Authorization header is replaced
rather than forwarded, and requests accelerate regardless of what the
client sent. Cached objects are keyed on the configured credential, so
rotating it invalidates previously cached responses and learned resolution
state rather than serving the old tenant’s data. To authenticate
Trickster’s own clients as well, attach an
authenticator via authenticator_name: the
validated client credential is stripped before proxying and the origin
credential is applied in its place.
A path config that sets its own Authorization in request_headers (or
removes it with -Authorization) overrides the origin credential for that
path. Appending one with +Authorization alongside an origin credential is
rejected at startup, since the effective header would be ambiguous. If
per-path identities then diverge across the paths resolution
uses — /render and /metrics/expand selecting different
request_headers/request_params for synthetic GETs, or GET and POST
/render pinned to different credentials — Trickster declines
acceleration rather than mix upstream namespaces, counted as
trickster_graphite_fallbacks_total{reason="resolution_identity"}.
The graphite block adds provider-specific options:
backends:
graphite1:
provider: graphite
origin_url: 'http://graphite.example.com:80'
graphite:
time_zone: UTC
passthrough_max_data_points: false
find_cache_ttl: 1m
resolution_registry:
ttl: 24h
negative_ttl: 30s
max_entries: 100000
persist: true
probe_concurrency: 2
probe_budget: 96
static_retentions:
- pattern: '^collectd\.'
retentions: '10s:6h,1m:7d,10m:5y'
| Option | Default | Purpose |
|---|---|---|
time_zone | UTC | Interprets date-anchored from/until values (midnight, today, MM/DD/YY) when the request has no tz parameter. Set it to the origin’s graphite-web TIME_ZONE, or those queries resolve to different instants than the origin uses. |
origin_username, origin_password | unset | Static HTTP Basic credential sent as the Authorization header of every upstream request — proxied, synthetic and health check alike. See Configuration above. |
origin_authorization | unset | Verbatim Authorization header value for non-Basic schemes (e.g., Bearer <token>); mutually exclusive with origin_username/origin_password. |
passthrough_max_data_points | false | Send the client’s maxDataPoints upstream instead of consolidating in Trickster. See maxDataPoints. |
find_cache_ttl | 1m | How long a wildcard target’s expansion to concrete metric paths is reused. Lower it where metrics appear and disappear frequently. |
max_targets_per_request | 128 | Renders carrying more targets than this are served through the object lane, untouched, before any parsing or per-target fan-out. |
max_target_length | 16384 | A render with a longer single target expression (in bytes) is likewise served through the object lane untouched. |
max_expanded_leaves | 4096 | A wildcard whose expansion matches more leaf paths than this is served through the object lane; the refusal is remembered for find_cache_ttl. |
max_expansion_bytes | 2097152 | Bounds one expansion’s aggregate decoded leaf-name bytes the same way. |
resolution_registry.ttl | 24h | How long a learned ladder is trusted. Whisper ladders change only when an operator runs whisper-resize.py, so this is deliberately long. |
resolution_registry.negative_ttl | 30s | Initial backoff after a failed resolution. Doubles per consecutive failure, capped at 10m. |
resolution_registry.max_entries | 100000 | Bounds each registry layer; least-recently-used entries are evicted. |
resolution_registry.persist | true | Write learned ladders through to this backend’s cache so a restart does not relearn them. |
resolution_registry.probe_concurrency | 2 | Simultaneous ladder-learning runs. |
resolution_registry.probe_budget | 96 | Probes one learning run may issue before giving up. |
static_retentions | empty | Seed and override, from your storage-schemas.conf. See Static retentions. |
Health checks
The default health check is GET /metrics/find?query=*, which every
Graphite-protocol implementation serves. /version is not used because
carbonapi and graphite-clickhouse answer it differently.
A configured origin credential rides on the probe even when the
healthcheck block declares custom headers. To override it, set your own
Authorization value there; to send the probe unauthenticated, set
Authorization: ''.
Routed paths
| Path | Handling |
|---|---|
/render (GET, POST) | The render handler: delta cache, or object cache when the request is not accelerable |
/metrics/find, /metrics/expand, /metrics/index.json, /tags, /tags/* | Object-cached, 30s. These also back resolution’s own lookups, so caching them lowers probe cost. |
/functions, /version | Object-cached, 1h |
| everything else | Proxied |
How resolution prediction works
Why it is necessary
Whisper picks an archive from the age of the query’s left edge, not from the width of the window:
diff = now - fromTime
for archive in header['archives']:
if archive['retention'] >= diff:
break
step = archive['secondsPerPoint']
So from=-7d&until=-6d and from=-7d&until=now are answered at the same
step, and a query one second either side of a rung boundary is answered at
different steps. The delta cache must know the step before it can compose a
cache key or reason about which ranges it already holds — and the step is a
property of the metric’s on-disk file, which nothing in the request reveals.
Static configuration is not sufficient on its own, because a Whisper file keeps
whatever ladder it was created with. Editing storage-schemas.conf affects only
newly created files, so in any long-lived installation the config and the files
routinely disagree.
Probe and learn
On the first request for a metric Trickster does not know the ladder, so the
request is served through the object cache — correct, just not delta-cached —
and the response itself teaches the step at that age. In the background,
Trickster then discovers the metric’s full ladder with a short series of
synthetic /render probes: a geometric sweep outward in time to find where the
step changes, then a binary search for each rung boundary, and a pair of probes
to establish maxRetention. Each probe asks for a one-second window and pins
the origin’s reference time with now, so it is cheap and exact.
Ladders are shared. They originate from storage-schemas.conf patterns, so a
deployment with thousands of metrics typically has only a handful of distinct
ladders; a new metric is first confirmed against the ladders already known
(about 7 probes) and only discovered from scratch (about 40) if none matches.
In the developer environment, 24 metrics across 4 ladders converge in about 380
probes total, after which probing stops entirely until the TTL expires.
Once a ladder is known, prediction is arithmetic and costs nothing.
Confidence levels
Every resolved target carries a confidence, exported as the confidence label
on trickster_graphite_resolution_lookups_total:
| Confidence | Meaning | Behavior |
|---|---|---|
exact | The step was read from an origin response for this metric at this age | Delta cached |
derived | Computed from known ladders — the shared step of a wildcard’s leaves, or a step-altering function Trickster understands | Delta cached |
configured | From static_retentions only, not yet confirmed by probe | Delta cached, and a confirming probe is scheduled |
unknown | No usable step | Object cached |
A request with several targets reports the weakest confidence among them.
Verification, and what happens when a prediction is wrong
The step is not merely predicted; it is checked. Every response Trickster
models is compared against the predicted step. A mismatch means the cached
ladder was wrong — a whisper-resize.py, a metric moved between schemas — and
Trickster:
- discards the response rather than caching it under the predicted key,
- increments
trickster_graphite_step_mispredictions_totaland logs a warning, - invalidates the registry generation, so every entry learned under the old assumption misses rather than colliding,
- relearns the affected ladders, and
- re-serves the request through the object cache, so the client still gets the correct answer.
One response shape cannot self-verify: JSON carries no explicit step, so a
fetch that returns fewer than two points is consistent with any prediction.
Trickster never trusts one. A delta fetch that would cover a single bucket —
the tip fetch every steady-state refresh performs — is widened one bucket
into the past, which cannot change the archive whisper selects (the pinned
now keeps now - from identical) but makes the response two points, and
therefore self-verifying, at no extra request cost. Any fetched response
that still cannot prove its step — one point, or no series where the
prediction promised data inside retention — is refused outright: nothing is
cached, and the request is served through the object cache instead, while
the background learner re-establishes the ladder.
trickster_graphite_step_mispredictions_total should be flat at zero. A
non-zero value is not a client-visible error, but it does mean something the
registry believed was wrong.
Static retentions
static_retentions mirrors storage-schemas.conf: an ordered list, first
match wins, patterns matched anywhere in the metric path exactly as carbon
applies them.
graphite:
static_retentions:
- pattern: '^collectd\.'
retentions: '10s:6h,1m:7d,10m:5y'
- pattern: '\.count$'
retentions: '1m:2d,5m:30d'
- pattern: '.*'
retentions: '1m:1d'
Retention syntax is Whisper’s: precision:retention pairs, where each part is
a number with an optional unit (s, m for minutes, h, d, w, y), and a
bare number in the second position is a point count. 1m:7d is a one-minute
step kept for seven days.
This is a seed and an override, never the sole source of truth. A static
match yields configured confidence and schedules a confirming probe; when the
probe disagrees with the configuration, the probe wins and the ladder is
relearned. That is deliberate: it means a stale static_retentions block
degrades to a few extra probes rather than to incorrect data.
Use it to skip the warmup cost on a known-stable deployment, not as a substitute for learning.
Changing static_retentions invalidates the persisted registry on the next
start, so a corrected block takes effect immediately rather than waiting out
the TTL.
What is accelerated
A target is delta-cached only if every function in it satisfies two independent properties:
- Its step is predictable — the function does not change the resolution in a way Trickster cannot compute.
- It is range-decomposable — each output point is a function of the input
points at that same timestamp, so the values for
[t1,t2]are identical whether that window was fetched alone or as part of a wider one.
The second property is what delta caching actually requires, and it is stricter than it first appears. The v1 allowlist is deliberately small:
Cross-series aggregation — one output point per input timestamp:
sumSeries (sum), averageSeries (avg), minSeries, maxSeries,
diffSeries, multiplySeries, divideSeries, divideSeriesLists,
stddevSeries, rangeOfSeries, countSeries, percentileOfSeries,
aggregate, aggregateWithWildcards, aggregateSeriesLists, group,
groupByNode, groupByNodes, asPercent (pct), weightedAverage,
powSeries, unique, and the *SeriesLists and *WithWildcards variants.
Per-point transforms: scale, scaleToSeconds, offset, add, pow,
exp, absolute, invert, squareRoot, sigmoid, logit, log, round,
transformNull, isNonNull, removeAboveValue, removeBelowValue,
consolidateBy, setXFilesFactor (xFilesFactor).
Naming, name-based filtering and cosmetics: alias, aliasSub,
aliasByNode, aliasByMetric, aliasByTags, upper, lower, substr,
exclude, grep, color, alpha, lineWidth, dashed, drawAsInfinite,
secondYAxis, stacked, areaBetween.
Bare metric paths, wildcards (*, ?, [a-z], {a,b}) and pipe syntax are
all accelerated.
What falls back, and why
Anything not on the list above is served through the object cache. The reason
is recorded on trickster_graphite_fallbacks_total:
reason | Cause |
|---|---|
function_not_allowlisted | A function in the target is not on the allowlist |
unknown_step | The step could not be resolved (metric not yet learned, probe failing, or the window is wholly beyond maxRetention) |
missing_target | No target parameter, or a wildcard that matches nothing |
mixed_steps | A function combines series stored at different steps. graphite-web consolidates them to their least common multiple starting from each series’ first point, so the buckets move with from and cannot be stitched |
parse_error | The target expression, from/until, or now did not parse |
non_series_format | An image or pickle format, or graphType=pie |
multi_target_step_mismatch | Targets resolve to different steps and could not be split |
passthrough_max_data_points | passthrough_max_data_points is on and the request carries maxDataPoints |
misprediction | A response contradicted the predicted step |
tz_unavailable | The request names a tz whose validity could not be verified within the timezone cold-load budget (a burst of unique hostile tz values can spend it); the request is served unaccelerated with the original tz forwarded, rather than being reinterpreted in the configured zone |
client_identity | The request carries a result-affecting identity or view selector — an Authorization header, a header named in the render path’s cache_key_headers, or a parameter/form field it names in cache_key_params/cache_key_form_fields (local above all) — that the path’s request_headers/request_params does not statically override, or the render path or backend has a request rewriter — see below |
resolution_identity | The configured path identities are mixed: /render and /metrics/expand select different request_headers/request_params for synthetic GETs, or the request’s path config carries a different identity than the synthetic one (e.g., GET and POST /render pinned to different credentials) |
Notable functions that are not accelerated, and why:
- Time-windowed:
movingAverage,movingSum,movingMin,movingMax,movingMedian,exponentialMovingAverage,derivative,nonNegativeDerivative,perSecond,integral,stdev,holtWinters*,linearRegression. Each output point depends on input points before it, so a cached range cannot be reused at its edges. - Whole-range selecting or ranking:
highest*,lowest*,sortBy*,limit,mostDeviant,*Percentile,currentAbove/Below,averageAbove/Below,maximumAbove/Below,filterSeries. Which series come back depends on the entire requested range. - Bucketing:
summarize,hitcount,smartSummarize. These look decomposable — withalignToFrom=false,summarize’s buckets align to absolute interval boundaries — but the bucket covering either edge of the requested window is summarized from only the points inside that window. Measured on graphite-web 1.1.10, the same absolute one-hour bucket reported134885.899,302921.880and307388.038over three different windows. Caching such a value and reusing it for a different window would return data the origin never produced, so these take the object lane. - Time-shifting:
timeShift,timeStack,timeSlice,delay. - Generators and tag queries:
constantLine,threshold,timeFunction,randomWalk,sinFunction,identity,seriesByTag,groupByTags,events, andtemplate().
The object lane is still a cache: the whole response is cached by request URL with a TTL, keyed on every render parameter. It is always correct because it makes no claim about the response’s internal structure.
Multiple targets
Grafana sends every target of a panel in one /render call. Trickster splits a
multi-target request into one delta-cached fetch per target and merges the
results in target order, so targets on different ladders are still
accelerated. If any single target cannot be accelerated, the whole request is
served unaccelerated, because consolidation at the origin spans all series in
the response and must be applied consistently.
maxDataPoints and consolidation
Grafana sends maxDataPoints on every panel, sized to the panel’s pixel width.
Two panels of different widths showing the same query would otherwise be two
different cache entries.
Trickster strips maxDataPoints from upstream requests, caches at the
origin’s native resolution, and applies consolidation when rendering the
response to each client. One cached series therefore serves every panel width,
every output format, and noNullPoints, jsonp, pretty and tz variations.
The consolidation reproduces graphite-web’s renderViewJson exactly, including
its start “nudge”, its treatment of xFilesFactor, the consolidateBy
function, and the maxDataPoints=1 special case — verified byte-for-byte
against a live origin.
Set passthrough_max_data_points: true if you need the origin’s own
consolidation byte-for-byte instead. Requests carrying maxDataPoints are then
served unaccelerated, which for Grafana means most requests, so this largely
disables acceleration. It exists for operators who must guarantee identical
bytes to a pre-existing consumer.
Sizing
Two of the common backend settings default differently for Graphite, because
Graphite is fetched at its native resolution: maxDataPoints is stripped
upstream (see maxDataPoints), so Trickster
buffers and caches every point Whisper holds for the window, not the few
hundred a dashboard draws. The generic defaults are sized for origins that
consolidate before responding.
| Setting | Generic default | Graphite default |
|---|---|---|
max_object_size_bytes | 512 KB | 64 MB |
timeseries_retention_factor | 1024 points | 524288 points |
The Graphite values come from pkg/backends/graphite/options/defaults.go and are applied only where the configuration is silent, so setting either one in the file always wins.
max_object_size_bytes
The delta cache must buffer an entire response to model it. At native resolution a wide window on a fine archive is large: in the developer environment, a 120-day panel over two series on a 5-minute archive is 34,560 points per series and about 1.6 MB of JSON — three times the generic 512 KB limit.
A response over the limit cannot be delta-cached. Trickster logs
upstream response exceeded MaxObjectSizeBytes and serves the request through
the object lane instead, which streams rather than buffers — so clients still
get their data, but that panel is never accelerated. Raise the limit if you
serve panels wider than the 64 MB default holds:
max_object_size_bytes: 134217728 # 128MB
timeseries_retention_factor
The cache keeps this many points per series, cropping older ones. The generic default of 1024 points is only about 3.5 days at a 5-minute step, so a 90-day dashboard panel would have almost its entire range cropped after each fetch and refetched on the next — a partial hit that re-fetches nearly everything. The Graphite default of 524288 covers 5 years at a 5-minute step, or 60 days at a 10-second one.
Raise it if you serve a wider window at a finer step than that, and lower it to cap what one series may occupy:
timeseries_retention_factor: 1048576
Cache storage
Trickster does not roll up between rungs: a 10-second cached chunk is never
reused to answer a query that resolves to 60 seconds, because the rollup
Trickster computed would not match what Whisper’s lower archive holds (which
depends on the metric’s aggregationMethod and xFilesFactor). A different
step is a different cache key.
The practical implication is that a metric is cached once per rung it is
queried at. A dashboard with a 1-hour, a 24-hour and a 30-day view of the
same metric holds three cached series for it, not one. Size the cache for the
distinct (metric, step) pairs your dashboards produce, not for the number of
distinct metrics.
Metrics, logs, and tracing
Graphite-specific metrics, in addition to the standard
trickster_proxy_requests_total{provider="graphite"} and trickster_cache_*
families:
| Metric | Type | Notes |
|---|---|---|
trickster_graphite_resolution_lookups_total{backend_name,confidence,source} | counter | One per resolved request |
trickster_graphite_probes_total{backend_name,kind,result} | counter | Should spike at start and fall to zero |
trickster_graphite_ladders{backend_name} | gauge | Distinct ladders known; should flatten at a small number |
trickster_graphite_registry_entries{backend_name,layer} | gauge | leaf, ladder, target, negative |
trickster_graphite_step_mispredictions_total{backend_name} | counter | Should be flat at zero |
trickster_graphite_fallbacks_total{backend_name,reason} | counter | Why requests were not accelerated |
No metric path, target expression or query text ever appears in a label; every label value comes from a closed set. See metrics.md.
Logs: a Debug line per resolution decision carrying the target, a bucketed
age, the lane, confidence, source and step (or reason when declined); Warn on
every misprediction and on every negative-cache entry.
Traces: GraphiteExpand (a child of the request span), GraphiteProbe
(kind, result, step) and GraphiteLearnLadder (ladder, probe count).
Operations and troubleshooting
Everything is falling back
Check trickster_graphite_fallbacks_total by reason.
unknown_stepdominating right after a start is normal; it should fall as ladders are learned. If it does not, resolution is failing — look forgraphite ladder learning failed; negative-cachedwarnings.function_not_allowlistedmeans the dashboard uses functions outside the allowlist. That is expected and correct, not a defect.non_series_formatmeans the client is asking for images; those panels cannot be delta-cached.
Probing never quiets down
trickster_graphite_probes_total should approach zero after warmup and
trickster_graphite_ladders should flatten. If probing continues:
- Metric names may be high-cardinality or ephemeral, so new leaves appear
constantly. Raise
resolution_registry.max_entries, or accept the cost. - The registry may be evicting under
max_entriespressure and relearning. Comparetrickster_graphite_registry_entries{layer="leaf"}against the configured maximum. - Learning may be failing and retrying under backoff. Check the warnings.
Step mispredictions are non-zero
Something changed a metric’s on-disk ladder — usually whisper-resize.py, a
schema edit followed by new file creation, or a metric that moved between
schema patterns. Trickster recovers on its own (it relearns and re-serves the
request unaccelerated), so this is not a client-visible error, but the counter
should return to flat once the new ladders are learned. Persistent
mispredictions on one namespace suggest an origin whose resolution is not a
function of now - from at all — a clustered graphite-web fronting stores with
different schemas, for example.
A panel is never accelerated but should be
- Look for
upstream response exceeded MaxObjectSizeBytesin the logs and raisemax_object_size_bytes. - Check whether the panel’s window is wholly beyond the metric’s
maxRetention; those requests have nothing to cache. - A single non-allowlisted target in a multi-target panel makes the whole panel unaccelerated.
Repeated partial hits on wide panels
Raise timeseries_retention_factor. A partial hit that re-fetches nearly the
whole range on every request is the signature of cropping.
Verifying correctness against the origin
Because the object lane and the delta lane must both be byte-identical to the
origin, the simplest check is to configure two data sources in Grafana — one
direct to Graphite, one through Trickster — and compare panels. The developer
environment (docs/developer/environment/) is set up this way, with a
dashboard whose panels deliberately exercise archive boundaries, the retention
edge, schema drift, non-allowlisted functions and mixed-ladder multi-target
requests.
Step alignment
Graphite supports the truncate step alignment mode, its default, and off. Whisper stores and returns whole buckets only, so there are no partial buckets for the other modes to fetch, and configuring one fails at load. off sends each render request to Graphite as the client sent it, through the fallback lane, cached as an object for one minute. A render target has no comment syntax, so Graphite queries can’t carry directives.
Known gaps
- Fast Forward is not implemented. Graphite’s coarsest-rung behavior and
its
until > nowclamp make the “one extra instant datapoint” approach ill-defined.fast_forward_disableis effectively always on. - No ALB / Time Series Merge support. Merging across replicas requires the replicas’ ladders to agree, an invariant not yet established. Graphite backends can still be members of non-TSM ALB mechanisms.
- Non-decomposable functions are not accelerated. A future release may
evaluate them inside Trickster over cached native-resolution leaves, which
would move
movingAverage,derivative,summarize,highest*and friends into the accelerated path. - No cross-resolution rollup, as described under Cache storage.
- Tag-based queries (
seriesByTag,groupByTags) are not accelerated. They are proxied and object-cached. - Non-ASCII metric names (graphite-web’s
UTF8_METRICS) are not supported by the target parser; such targets fail to parse and take the object lane.
6 - PostgreSQL and TimescaleDB Provider
Trickster can accept native PostgreSQL client connections, relay each session to a PostgreSQL or TimescaleDB origin, and cache the results of eligible queries. Dashboards that bucket rows by time are served from Trickster’s delta cache: a refresh fetches only the buckets the cache does not hold yet.
This guide describes the supported contract. Anything not listed here is relayed to the origin without caching; cache eligibility always fails closed.
Compatibility
- PostgreSQL 14 and later, with or without the TimescaleDB extension.
- Grafana’s built-in PostgreSQL data source (Grafana 11 and later), with its TimescaleDB option on or off.
- Any client that speaks protocol version 3.0 or 3.2 (
psql, libpq, pgx, JDBC, …). Every client is relayed; which statements are cached is described under Cache classification.
Other engines that speak the PostgreSQL wire protocol are not part of this provider’s compatibility claim. The developer environment pins PostgreSQL 18 with TimescaleDB 2.30 and Grafana 13.
Provider names: postgres, and its alias timescaledb. Both select the same
engine, and metrics report provider="postgres".
Direct backend configuration
A postgres listener accepts the connections and maps to exactly one backend
(or one User Router) through the backend’s
listener_names.
listeners:
tsdb:
protocol: postgres
port: 5432
authenticators:
tsdb-clients:
provider: basic
users:
grafana_ro: ${GRAFANA_RO_PASSWORD}
backends:
tsdb1:
provider: timescaledb
origin_url: postgres://trickster_ro:REPLACE_ME@tsdb.example:5432/metrics
listener_names: [tsdb]
authenticator_name: tsdb-clients
cache_name: default
healthcheck:
interval: 5s
timeout: 3s
A complete annotated example is in
examples/conf/example.full.yaml.
Authentication modes
The backend’s authenticator decides who authenticates the client.
With authenticator_name (recommended). Trickster authenticates the
client itself against the authenticator’s users and then logs in to the origin
with the origin_url credentials. Clients are offered SCRAM-SHA-256, plus
SCRAM-SHA-256-PLUS (channel binding) when the listener has a certificate.
Authenticator entries may be plaintext passwords or PostgreSQL
SCRAM-SHA-256$... verifiers copied from pg_authid. An entry stored as a
PostgreSQL md5 hash or a crypt hash needs the client’s cleartext password,
which is refused on a non-TLS connection unless the listener sets
postgres.allow_cleartext_without_tls (or postgres.allow_md5 for the legacy
md5 exchange). Every client shares the origin role named in origin_url, so
give that role the least privilege the dashboards need. Replication
connections are refused in this mode.
Without one (or with observe_only: true). The client’s own
authentication exchange is relayed to the origin untouched and Trickster holds
no credentials. One caveat: SCRAM-SHA-256-PLUS cannot succeed when both the
listener and the origin use TLS, because the client binds to Trickster’s
certificate and the origin expects its own. Clients that require channel
binding must use the authenticator mode.
In both modes the cache is partitioned by the client’s user and database, so one user’s cached answers are never served to another.
TLS
TLS is negotiated in-band on the listener port, as PostgreSQL does it; there
is no separate TLS port. Configure the backend’s tls.full_chain_cert_path
and tls.private_key_path to let clients upgrade, and set require_tls: true
to refuse plaintext clients.
TLS toward the origin is independent of the listener. Set the backend’s
postgres.upstream_tls_mode:
| Mode | Meaning |
|---|---|
disable (default) | plaintext |
require | TLS without certificate verification |
verify-ca | the origin’s chain must verify against tls.certificate_authority_paths |
verify-full | as verify-ca, and the certificate must match the origin host name |
tls.client_cert_path and tls.client_key_path add a client certificate.
Connections, limits, and lifecycle
A result that outgrows the result limits is streamed to the client through a fixed buffer, however large its rows are; the limits bound what Trickster holds in memory, not what a client may read.
Each client connection gets its own origin connection for its whole life, so session state never crosses clients, and Trickster adds no pooling. Put a pooler such as PgBouncer between Trickster and the origin if you need one.
| Setting | Where | Default | Effect |
|---|---|---|---|
max_concurrent_conns | backend | unlimited | origin connections; further clients get 53300 too many clients |
connections_limit | listener | unlimited | accepted client connections |
postgres.handshake_timeout | listener | 10s | startup and authentication |
postgres.read_timeout / write_timeout | listener | 30s | one message in flight |
postgres.idle_timeout | listener | 5m | wait for a client’s next message |
postgres.max_query_size_bytes | listener | 1 MiB | larger statements are relayed without being analyzed |
postgres.max_result_rows | backend | 100000 | larger results are relayed without being cached |
postgres.max_result_size_bytes | backend | 64 MiB | as above |
timeout | backend | 1m | origin connect, and the wait for a cached statement’s result |
Cancel requests work through the listener: clients receive a Trickster-issued cancellation key that maps to the origin’s, and a client that disconnects mid-query has its statement canceled at the origin. A configuration reload that changes a listener’s identity (origin, credentials, TLS, routes, limits) restarts that listener and drains its sessions; other reloads leave sessions alone.
Protocol behavior
Everything a client sends is relayed: the simple and extended query protocols,
COPY, LISTEN/NOTIFY, transactions, multi-statement queries, notices and
errors (with their SQLSTATE) all reach the client as the origin sent them.
Only one kind of message is ever answered from the cache: a simple-protocol
Query holding a single read statement, sent while the session is idle (no
open transaction, nothing in flight). That is what Grafana, psql and most
reporting tools send. Statements sent with the extended protocol
(Parse/Bind/Execute, i.e. prepared or parameterized statements) are relayed
and never cached. For a driver that defaults to it, select its simple
protocol mode if you want caching (pgx: default_query_exec_mode=simple_protocol;
JDBC: preferQueryMode=simple).
Cached answers are the origin’s own bytes: the row description and rows are stored and replayed exactly as received, in text format. Trickster reads only the bucket column, to place rows on the time axis.
Cache classification
Each analyzed statement is counted in
trickster_sql_query_analysis_total{backend_name,dialect,cache_mode,reason}.
cache_mode | Meaning |
|---|---|
delta | cached per time bucket; only missing buckets are fetched |
object | the whole result is cached under the statement’s exact text |
none | relayed, never cached |
Delta-cacheable statements
A statement uses the delta cache when it is a single-table SELECT that
groups by one recognized time bucket and bounds the bucketed column with a
literal (or now()-relative) lower limit and, optionally, an upper limit
joined by AND. A range with no upper limit runs to the present.
| Bucket | Notes |
|---|---|
time_bucket(width, col) | TimescaleDB; what Grafana’s $__timeGroup writes with the TimescaleDB option on |
time_bucket(width, col, TIMESTAMPTZ '...') | a typed origin |
time_bucket(width, col, '...'::interval) | a typed offset |
time_bucket_gapfill(width, col) | a single series, both bounds in WHERE, no locf or interpolate |
floor(extract(epoch from col)/N)*N | $__timeGroup with the TimescaleDB option off; also with date_part('epoch', col) |
floor((col)/N)*N | $__unixEpochGroup, over a column of epoch seconds with epoch-second bounds |
date_bin(width, col, origin) | PostgreSQL 14+ |
date_trunc('unit', col) | second to week, and only while the session TimeZone is UTC |
A width is a fixed-length interval: '300.000s', '5 minutes', '1h30m',
INTERVAL '1 hour', '7 days'. Month and year widths never match.
FILTER, ordered-set aggregates, TimescaleDB hyperfunctions such as first
and last, extra GROUP BY columns, and ORDER BY on the bucket (either
direction) are all fine.
Four behaviors are worth knowing:
- Live ranges lose their partial edge buckets by default. Under the
default
step_alignment,drop, bounds that are not on the bucket grid, as Grafana’snow-relative ranges never are, are rounded inward, so the partial first and last buckets are left out of the answer rather than cached as if they were complete. The still-filling bucket that holds the present is partial too, so a range that reaches the present ends before it.col <= Xkeeps X’s bucket only when X is the last instant of it (...:59.999999). Thepartial,partial_startandpartial_endmodes instead fetch those buckets from the origin over the client’s own range, through the object cache forpartial_bucket_ttl, and never cache them with the complete ones, at a cost of up to two small origin queries per request;truncateanswers the whole first bucket. A query can choose its own mode with a comment, such as-- trickster-step-align:partial. See Step Alignment. - A range with no complete bucket is answered by the origin. When nothing
complete remains, as for a range inside one bucket, across a single
boundary, or starting inside the still-filling bucket, the client gets the
origin’s own result for its statement, partial and still-filling buckets
included, cached as an object for
partial_bucket_ttl. - Zone-less literals need a UTC session. PostgreSQL reads
'2026-09-17 00:00:00'andTIMESTAMP '...'in the sessionTimeZone, so such bounds (and origins) qualify only while that zone is UTC. Grafana’sZ-suffixed bounds always qualify. - Ordering within a bucket is the origin’s. Buckets are merged in time
order and each keeps the row order the origin returned.
ORDER BYmust lead with the bucket or be absent; any other ordering makes the statement an object.
Statements cached as objects
Everything else that is a deterministic read: statements with no bucket, a
LIMIT, a join, a subquery or CTE, DISTINCT, HAVING, window functions,
set-returning functions, a gapfill outside the form above, month or year
buckets, the time zone form of time_bucket, unsafe time predicates (OR,
NOT, <>, a strict > lower bound on the raw column), and statements the
SQL parser cannot write back faithfully (FROM ONLY, json casts,
U&'...' strings) or cannot parse at all (named arguments such as
origin => ..., GROUP BY ROLLUP). The reason label says which.
An object is keyed on the statement’s exact text. A dashboard panel whose
range ends at now is therefore a new object on every refresh, and is served
from cache only while its range is unchanged. Objects live for
timeseries_ttl, so a repeated SELECT count(*) FROM t is answered from
cache for that long; lower it, or set proxy_only: true, if that is too
stale for your clients.
Statements never cached
- Anything that is not a
SELECT, and aSELECTthat writes or locks (WITH x AS (DELETE ... RETURNING *) SELECT ...,FOR UPDATE,SELECT ... INTO). - A statement that calls a volatile or side-effecting function, or reads the
clock anywhere but in a time bound:
random(),now(),current_date,nextval(),pg_sleep(),pg_notify(), advisory locks and similar (reason="nondeterministic"). - Multi-statement queries, statements inside a transaction, pipelined
statements, and statements larger than
postgres.max_query_size_bytes. - Everything in a session whose state Trickster can no longer vouch for; see Session state.
Trickster recognizes volatile functions by name. A user-defined volatile
function, or one that changes a session setting as a side effect, is not
detectable. Do not route such statements through a caching backend; use a
proxy_only backend for them.
Continuous aggregates
Querying a continuous aggregate works like querying a table. To stay on the delta cache, re-bucket the aggregate’s bucket column and group by it:
SELECT time_bucket('15 minutes', bucket) AS "time", cab_type AS metric, sum(trips) AS trips
FROM trips_15m
WHERE bucket >= $__timeFrom() AND bucket < $__timeTo()
GROUP BY 1, 2 ORDER BY 1
A bare SELECT bucket, trips FROM trips_15m WHERE ... has no bucket function
and no GROUP BY, so it is cached as an object. Real-time aggregates change
their newest buckets as data arrives; volatile_window controls how much
of the recent past Trickster re-fetches on each request.
Grafana macros and exact SQL shapes
These are the statements Grafana 13’s PostgreSQL data source sends, and how they are classified.
| Macros | Sent as | Mode |
|---|---|---|
$__timeGroup(col,'5m'), TimescaleDB on | time_bucket('300.000s',col) AS "time" | delta |
$__timeGroup(col,'5m'), TimescaleDB off | floor(extract(epoch from col)/300)*300 AS "time" | delta |
$__timeGroupAlias(col,$__interval) | as above, with the panel’s interval | delta |
$__timeFilter(col) | col BETWEEN '...Z' AND '...Z' | delta with a bucket |
$__timeFrom(), $__timeTo() | '...Z' | delta with a bucket |
$__unixEpochGroup(col,'10m') | floor((col)/600)*600 | delta |
$__unixEpochFilter(col) | col >= N AND col <= M | delta with a bucket |
$__time(col), $__timeEpoch(col) | a projection, no bucket | object |
$__unixEpochNanoFilter(col) | nanosecond integers | object |
The minimum supported $__interval is 1 minute. Prefer
col >= $__timeFrom() AND col < $__timeTo() over $__timeFilter(col): both
are delta-cacheable, but the half-open form states exactly the range that is
cached. Grafana’s pgx driver sends an empty -- ping query before each panel
query; it is relayed and left out of the request metrics.
The executable form of this contract is
pkg/backends/postgres/testdata/compatibility/v1.json.
Session state
Cached answers depend on session settings: TimeZone changes how every
timestamptz is rendered, search_path changes which table a name means.
Trickster therefore partitions the cache by a session identity: the user, the
database, every setting the origin announces (TimeZone, DateStyle,
IntervalStyle, client_encoding, search_path on PostgreSQL 18, …), and
role, extra_float_digits and bytea_output from the client’s own SET
statements. Two sessions share cached answers only when all of it matches.
A session that does something Trickster cannot follow stops being cached
until it reconnects (reason="session_state"): any other SET (for example
a custom app.tenant used by row-level security), set_config(),
SELECT ... INTO, DDL, PREPARE/EXECUTE, DO, CALL, and fast-path
function calls. A session with standard_conforming_strings off is relayed
uncached as well. Transactions, DML, SHOW, EXPLAIN, and harmless settings
such as statement_timeout and application_name do not end caching.
PostgreSQL never announces extra_float_digits or bytea_output, and a role
or database default can change them. With an authenticator, Trickster reads
the session’s effective values once at origin login, makes them part of the
session identity, and returns to them on RESET. When the client’s own login
is relayed, Trickster cannot ask, so it only knows a value the client set
itself.
Results whose bucket column cannot be read safely are cached as objects
instead: under a non-ISO DateStyle, and for a float8 epoch bucket
($__unixEpochGroup, or the date_part('epoch', ...) form) whenever
extra_float_digits is negative or unknown, since PostgreSQL may then render
1788998400 as 2e+09. In relay-authentication mode such a panel needs the
client to SET extra_float_digits (0 or higher) to use the delta cache.
Failure behavior
The cache always fails open to a plain relay of the client’s own statement:
when the origin rejects a statement Trickster rewrote for a sub-range
(trickster_sql_query_rewrite_failures_total{reason="origin_rejected"}), or
when a result outgrows the result limits (reason="result_size"). Such a
statement skips the cache until a marker expires, so it is not retried on
every refresh. Origin errors reach the client verbatim and are never cached.
Protocol-aware User Router
A postgres listener can front a User Router ALB whose targets are
postgres/timescaledb backends. Trickster authenticates the client against
the router’s authenticator, picks the backend mapped to the startup user (or
default_backend), and logs in to that backend’s origin with its origin_url
credentials. The session stays on that backend until it ends.
- The router requires an authenticator. Targets need origin credentials and no authenticator or listener of their own.
- A user with no route, or whose backend is failing its health check, is
refused with SQLSTATE
28000. Sessions never fail over to another backend. to_userandto_credentialare not supported.- Each target keeps its own cache, limits and metrics.
A complete example is in
examples/conf/postgres-user-router.yaml.
Metrics, logs, and health
| Metric | Labels |
|---|---|
trickster_sql_query_analysis_total | backend_name, dialect, cache_mode, reason |
trickster_sql_query_cache_total | backend_name, dialect, cache_mode, cache_status |
trickster_sql_query_rewrite_failures_total | backend_name, dialect, reason |
trickster_proxy_requests_total, ..._duration_seconds, trickster_proxy_points_total | the common request labels, method="QUERY" |
trickster_pgwire_connections_total, trickster_pgwire_active_connections | backend_name (and event) |
trickster_pgwire_errors_total | backend_name, class |
trickster_pgwire_route_selections_total | router_name, backend_name, outcome |
Query text, user names in labels, passwords and origin addresses are never
logged or exported. The health check logs in to the origin on a fresh
connection with the origin_url credentials and waits for ReadyForQuery;
see Health Checks.
Operations and troubleshooting
A panel is never a hit. Look up its statement’s reason in the analysis
metric. unsupported_limit, unsupported_bucket and unsupported_format
mean it is an object, and an object whose range ends at now changes every
refresh. session_state means the session did something unmodeled; find the
SET the client sends at connect time. pipelined and in_transaction
point at a driver that wraps reads in transactions or uses the extended
protocol.
The first and last points are missing. Expected on unaligned ranges; see Delta-cacheable statements.
origin_rejected rewrite failures. Trickster re-spells the statement
when it fetches a sub-range, and the origin refused the result. The client is
unaffected, since its own statement is relayed, but the statement is not
cached. Please file an issue with the statement.
Stale data after a bulk rewrite. Cached buckets older than the volatile
window are not re-fetched until they expire (timeseries_ttl). After
rewriting history, restart Trickster when using the memory cache, or clear
the backend’s keys from a persistent one.
Rollout and rollback
Canary the listener with a limited client population and compare with the
pre-release baseline. Roll back on a sustained increase of 5 percentage points
in trickster_pgwire_errors_total or proxy-error outcomes, 1 percentage point
in rewrite failures, or 20% in p95 request duration. Roll back by restoring
the previous configuration and binary, reloading or restarting Trickster, and
letting the listener drain; sessions are never moved between versions.
Setting proxy_only: true on the backend is the quickest mitigation: it
keeps the listener up and turns off all statement inspection and caching.
Environment variables and reload behavior
There are no TRK_* variables for postgres listener or backend options;
configure them in YAML. Authenticator users values expand ${VARIABLE};
origin_url does not, so keep a configuration with an embedded password in a
secret store. Validate before rollout:
trickster -validate-config -config /etc/trickster/trickster.yaml
Known compatibility gaps
- Extended-protocol (prepared, parameterized, binary-format) statements are relayed but never cached.
- Month, quarter and year buckets, the time zone overload of
time_bucket, integer-timetime_bucket(bigint, col), and explicitstart/finisharguments totime_bucket_gapfillare objects, not deltas. date_truncand zone-less time literals are delta-cacheable only in UTC sessions.- User-defined volatile functions and functions that change session settings cannot be detected.
- Downstream client-certificate authentication, GSSAPI encryption, and channel binding in relay-authentication mode are not supported.
- No cross-session invalidation: a write through one session does not evict what another session cached.
7 - GreptimeDB Provider
Use provider: greptimedb for GreptimeDB’s HTTP, PostgreSQL and MySQL query
surfaces. One backend may be mapped to all three listener protocols. The
provider reuses the SQL and Prometheus cache engines, with GreptimeDB-specific
parsing, session settings, result types and API paths.
Surfaces
| Surface | Origin port | Behavior |
|---|---|---|
HTTP /v1/sql | 4000 | Eligible SELECT results use delta or object caching |
| PostgreSQL wire | 4003 | Eligible simple queries use delta or object caching; extended queries are relayed |
| MySQL wire | 4002 | Eligible text queries use delta or object caching |
/v1/prometheus/api/v1/query_range | 4000 | Prometheus range caching and supported time-series merges |
| Prometheus instant and metadata APIs | 4000 | Object caching subject to HTTP cache policy |
Ingest, TQL, /v1/promql, other HTTP APIs | 4000 | Proxied without SQL/PromQL delta rewriting |
| gRPC | 4001 | Not implemented by this provider |
Wire compatibility does not make GreptimeDB a PostgreSQL or MySQL server.
Use the greptimedb provider even on native listeners. For example, MySQL’s
SHOW COUNT(*) WARNINGS is not a supported GreptimeDB statement, and its
PostgreSQL session settings have different semantics.
The developer environment pins an official nightly that includes GreptimeDB #9295, the fix for Grafana’s comment-only PostgreSQL health query. See the developer environment for the exact image and reproducible checks. The tested Grafana plugin is the bundled PostgreSQL datasource, not a GreptimeDB-specific plugin.
Step Alignment
GreptimeDB’s surfaces follow the step alignment of the query language they carry:
| Surface | Supported modes | Default |
|---|---|---|
| Prometheus HTTP API | truncate, drop, partial_end, off | partial_end, which is Fast Forward (truncate with fast_forward_disable: true) |
| SQL, over HTTP, MySQL and PostgreSQL | all | drop |
A backend’s step_alignment may name any mode either supports; a PromQL query asked for partial or partial_start runs in its default instead, and the fallback is counted in trickster_step_alignment_fallbacks_total. The partial modes fetch each partial bucket as a small query of its own through the object cache, so they cost up to two extra origin queries per request. A query can choose its own mode with a comment, # ... in PromQL and -- ... or /* ... */ in SQL; see Per-Query Instructions.
Configuration
The complete example exposes HTTP on 8480, PostgreSQL on 8489 and MySQL on 8491. A minimal mixed backend is:
listeners:
greptime-pg:
protocol: postgres
port: 8489
greptime-mysql:
protocol: mysql
port: 8491
authenticators:
greptime-readers:
provider: basic
proxy_preserve: true
users:
grafana_ro: ${GRAFANA_RO_PASSWORD}
backends:
greptime:
provider: greptimedb
origin_url: http://greptime.example:4000
listener_names: [default, greptime-pg, greptime-mysql]
authenticator_name: greptime-readers
postgres:
upstream_url: postgres://trickster_ro:REPLACE_ME@greptime.example:4003/public
mysql:
upstream_url: mysql://trickster_ro:REPLACE_ME@greptime.example:4002/public
HTTP uses origin_url; postgres.upstream_url and mysql.upstream_url
separately identify native endpoints and credentials. Never put the HTTP
port in either native URL. Without a native override, an HTTP origin’s host
is reused with the engine’s native default port, not its HTTP port or userinfo.
Explicit URLs are recommended. URL credentials do not expand environment
variables; authenticator user values do.
List only default for an HTTP-only backend, or just the chosen native
listener for a native-only backend. A mixed backend probes HTTP /health;
a native-only backend uses its native authenticated probe. Listener, origin,
credential or protocol changes restart and drain affected native listeners.
Authentication And TLS
For HTTP, proxy_preserve: true forwards client credentials to GreptimeDB.
Authenticated object responses are not shared unless the origin explicitly
permits HTTP caching. SQL and PromQL cache identities retain their database,
request parameters, effective timezone and authentication partitioning.
For PostgreSQL, an authenticator terminates client authentication at Trickster; the separate upstream URL supplies the origin role. Without an authenticator, authentication is passed through to GreptimeDB. MySQL requires a configured authenticator. Give every origin role only the permissions its clients need; terminating authentication does not preserve separate upstream roles for each client.
Native TLS and limits use the existing postgres and mysql option blocks;
see PostgreSQL and MySQL. Support for an option in
Trickster is not a promise that a particular GreptimeDB deployment supports
the corresponding server feature. The loopback developer environment uses
plaintext and published test credentials and must not be exposed publicly.
SQL Cache Classification
Delta caching requires one provably deterministic time-bucketed SELECT with supported grouping, ordering and time bounds. Other supported reads can use whole-result object caching. Writes, volatile queries and session-unsafe statements bypass caches; a parser rejection never makes a write cacheable.
PostgreSQL supports fixed-width date_bin intervals or compact widths such
as 5m, UTC fixed-width date_trunc, and epoch buckets of the form
floor(extract(epoch FROM ts)/300)*300 or date_part. Calendar widths,
ambiguous time predicates, unknown timezone-dependent semantics, and
submicrosecond PostgreSQL text buckets do not get delta plans. Extended
protocol messages are always relayed.
HTTP SQL supports GET and form-encoded POST with the default greptimedb_v1
response format. A delta response retains typed schema, row ordering, NULLs
and exact integer values. Like PostgreSQL, under the default drop step
alignment, unaligned SQL bounds round inward to complete buckets before delta
caching and partial edge buckets are omitted.
Alternate formats, limit, unknown options and ranges with no complete bucket
retain the original query. Rebuilt responses do not claim the origin’s
execution duration or execution metrics. The default JSON response is decoded
row by row into a DataSet, and its validated client serialization is reused.
MySQL supports DATE_BIN('1m', ts, FROM_UNIXTIME(0)) and fixed-width
DATE_TRUNC buckets with verified UTC sessions, whole-second cadence and
half-open bounds, which the default drop step alignment
rounds inward to complete buckets. Other bucket origins,
subsecond cadence, inclusive upper bounds and ranges with no complete bucket
use the original query instead. Timestamp results keep up to nine
fractional digits, text groups are compared case-sensitively, and NULL ordering
follows GreptimeDB. Unsupported or failed session changes conservatively
disable caching for that connection.
PromQL range endpoints round down to epoch-aligned steps, including 500ms steps, so equivalent aligned ranges share cached samples. Instant and metadata timestamps are unchanged. Route defaults do not grant permission to share authenticated object responses: only an origin’s explicit cache policy can authorize that sharing.
Grafana Macros
Use the built-in PostgreSQL datasource with TimescaleDB disabled and a minimum interval of one minute. Native epoch-second columns are integers, not timestamps cast to integers (GreptimeDB timestamp casts can yield nanoseconds).
| Macro family | GreptimeDB behavior |
|---|---|
$__time, $__timeEpoch | Supported projection; not a bucket by itself |
$__timeFilter, $__timeFrom, $__timeTo | Supported timestamp bounds |
$__timeGroup, $__timeGroupAlias | Fixed epoch-floor buckets with TimescaleDB mode off |
$__unixEpochFilter, $__unixEpochFrom, $__unixEpochTo | Numeric epoch-second bounds |
$__unixEpochGroup, $__unixEpochGroupAlias | Fixed buckets over an epoch-second column |
$__unixEpochNanoFilter | Supported SQL predicate; not an automatic delta plan |
$__interval | Use a positive supported fixed duration |
The compatibility corpus
records real Grafana expansions, including unaligned windows and conservative
fallbacks. Grafana’s MySQL macros are not interchangeable: UNIX_TIMESTAMP
is unsupported in the tested GreptimeDB image. Use explicit GreptimeDB SQL
when connecting a MySQL client.
Known Limits
RANGE ... ALIGNremains whole-result caching; TQL is passthrough. There is no advanced RANGE/FILL dependency rewriting.- PostgreSQL text timestamps carry microseconds even for a TIMESTAMP(9) column. MySQL and HTTP have separate precision contracts.
- Transaction commands accepted by GreptimeDB are compatibility stubs, not proof of transactional isolation. PostgreSQL cancellation is not a supported upstream capability in the tested image.
SET LOCALpersists at session scope in GreptimeDB. Effective settings are probed and tracked; requested startup parameters alone are not proof of the resulting timezone. The development read-only account cannot change it.- PostgreSQL session load balancing is unsupported. MySQL listeners use the existing native MySQL routing rules.
- PromQL
count_valuescurrently loses its grouping label upstream; merges reject that response rather than claim a correct aggregate. - Ingestion, mutations, enterprise-only features and gRPC caching are outside this provider’s accelerated contract.
Run the acceptance suites against an
isolated seeded environment. Inspect cache counters as well as results:
trickster_sql_query_analysis_total, trickster_sql_query_cache_total, and
trickster_sql_query_rewrite_failures_total distinguish actual cache hits,
fallbacks and failed rewrites. SQL dialect labels are greptimedb; transport
metrics retain the native protocol name.
8 - QuestDB Provider
Use provider: questdb for a QuestDB backend that exposes both the HTTP SQL
surface and the PostgreSQL wire protocol. Trickster keeps the HTTP surface as
an authenticated passthrough and uses the native pgwire path for the verified
SQL relay and cache contract.
Surfaces
| Surface | Origin | Behavior |
|---|---|---|
| HTTP SQL, Web Console and other HTTP paths | QuestDB HTTP, normally 9000 | Passthrough; no HTTP SQL delta cache is applied |
| PostgreSQL wire protocol | QuestDB pgwire, normally 8812 | Simple-query relay, object cache and verified fixed-width delta cache |
| Extended pgwire statements | QuestDB pgwire | Relayed; extended statements are not analyzed for delta caching |
The native listener uses the shared PostgreSQL wire adapter. A QuestDB backend
can therefore be mapped to an HTTP listener and a protocol: postgres
listener at the same time. The provider name, rather than postgres, selects
QuestDB’s SQL analyzer and result semantics.
Configuration
The HTTP origin and native origin are separate. Put the HTTP endpoint in
origin_url and the pgwire endpoint, including its credentials, in
postgres.upstream_url:
listeners:
questdb-pg:
protocol: postgres
port: 8490
authenticators:
questdb-readers:
provider: basic
users:
grafana_ro: ${QUESTDB_CLIENT_PASSWORD}
backends:
questdb:
provider: questdb
origin_url: https://questdb.example:9000
listener_names: [default, questdb-pg]
authenticator_name: questdb-readers
postgres:
upstream_url: postgres://grafana_ro:REPLACE_ME@questdb.example:8812/qdb
upstream_tls_mode: disable
healthcheck:
interval: 5s
timeout: 3s
Use TLS and deployment-specific credentials in a real installation. The
developer environment uses cleartext QuestDB pgwire because QuestDB OSS does
not provide pgwire TLS in the tested image. proxy_preserve is only suitable
when the HTTP origin is configured to accept the same client credential; the
native upstream URL is the source of the pgwire origin credential.
If the backend is native-only, map only questdb-pg and use a PostgreSQL
origin_url with an explicit user. Without an HTTP listener, the provider
uses its authenticated pgwire health probe. With an HTTP listener, the health
check uses authenticated QuestDB /execute and the native listener still uses
the postgres.upstream_url.
SQL Cache Contract
The QuestDB analyzer has a deliberately narrow delta-cache surface:
| Query shape | Result |
|---|---|
SAMPLE BY 5m, 15m, and other positive fixed s/m/h/d widths | Delta cache |
SAMPLE BY ... FILL(NULL) | Object cache; fill rows depend on the selected range |
timestamp_floor('5m', timestamp_column) with grouped output | Delta cache |
floor(extract(epoch FROM ts)/N)*N | Delta cache when the bounds and result are renderable |
| Other deterministic reads | Object cache or relay, according to the normal pgwire policy |
FILL(PREV), SAMPLE BY FROM ... TO ..., calendar M widths, ambiguous grouping | Object/relay; no speculative rewrite |
| Writes, volatile statements, malformed or unknown time ranges | Relay; never a delta plan |
For SAMPLE BY, QuestDB applies the designated timestamp to the first plain
selected column and implicitly groups the remaining plain selected columns.
The adapter only accepts that unambiguous form. An explicit GROUP BY, a
computed first item or another ambiguous shape fails closed rather than
guessing the series identity.
The pgwire result path maps QuestDB’s verified PostgreSQL type OIDs and
microsecond timestamp text to the shared time-series representation. Session
settings that can affect results or rendering, including timezone, date style,
interval style, float/bytea output and search path, remain part of the cache
identity. Unknown startup settings also partition the identity until observed;
unknown SET statements bypass caching conservatively.
Developer Environment
The reproducible developer environment adds a pinned QuestDB OSS image, a
read-only pgwire role, an HTTP admin role for seeding, and the official
questdb-questdb-datasource Grafana plugin. It reuses the shared generated
trips fixture rather than downloading another dataset.
See QuestDB Details for
ports, seed commands, dashboard datasource names and direct-versus-proxy
checks. The versioned analyzer contract is in
pkg/backends/questdb/testdata/compatibility.
The integration conformance target compares direct and proxied QuestDB results, protocol errors and session state. It also checks object and delta cache counters, so a successful passthrough response cannot be mistaken for a cache hit.
Known Limits
- HTTP SQL is passthrough in this provider; native pgwire is the only path with QuestDB-specific delta caching in this change.
- Extended pgwire messages are relayed and are not delta-cacheable.
- QuestDB
FILL(PREV), clause-ownedFROM/TOranges and calendar-month widths remain object/relay paths. - QuestDB OSS cleartext pgwire and the credentials in the developer compose file are for isolated local testing only.
9 - VictoriaMetrics Provider
Use provider: victoriametrics for VictoriaMetrics: single-node, or one tenant of a cluster’s
vmselect. Trickster accelerates MetricsQL range queries through VictoriaMetrics’ Prometheus
querying API, caches settled instant and metadata queries, and serves its Graphite render, find
and tags APIs. VictoriaMetrics is its own provider rather than a flavor of prometheus:
MetricsQL is a different language, and native VictoriaMetrics, not PromQL, decides every answer.
Surfaces
| Surface | Routes | Behavior |
|---|---|---|
| MetricsQL range queries | /api/v1/query_range | Delta proxy cache, or exact-request object cache for expressions that depend on the whole range |
| MetricsQL instant queries | /api/v1/query | Object cache once the evaluation time has settled; relayed near the live edge |
| Metadata | /api/v1/series, /api/v1/labels, /api/v1/label/<name>/values | Object cache, 30 seconds |
| Other read APIs | /api/v1/status/..., /api/v1/metadata and the rest of the Prometheus catalog | Object cache, 30 seconds, keyed on every parameter |
| Graphite render | /render | Exact-request object cache for settled absolute ranges; relayed otherwise |
| Graphite discovery | /metrics/find, /metrics/expand, /tags..., /functions | Object cache, 30 seconds |
| Graphite tag registration | /tags/tagSeries, /tags/tagMultiSeries, /tags/delSeries | Relayed uncached |
| Everything else | writes, imports, exports, /api/v1/admin/..., /internal/..., /federate, /vmui | Relayed uncached |
Every Graphite route is also served under /graphite, as single-node VictoriaMetrics serves it.
Configuration
origin_url is the base of the Prometheus querying API: the server for single-node
VictoriaMetrics, or a tenant’s /select/<tenant>/prometheus path on vmselect. The Graphite
APIs are served from the same server, with a trailing /prometheus replaced by /graphite, as
vmselect’s tenant paths require.
backends:
vm:
provider: victoriametrics
origin_url: http://victoriametrics:8428
vm-tenant:
provider: victoriametrics
# Graphite requests go to http://vmselect:8481/select/0/graphite
origin_url: http://vmselect:8481/select/0/prometheus
victoriametrics:
# replaces the derived Graphite path, e.g. for a gateway with its own routes
graphite_path: /select/0/graphite
# set when VictoriaMetrics runs with -search.disableCache (see Range Queries)
search_disable_cache: false
| Option | Default | Description |
|---|---|---|
victoriametrics.graphite_path | derived | The upstream path of the Graphite APIs: an absolute path, without a query |
victoriametrics.search_disable_cache | false | Mirrors the origin’s -search.disableCache flag |
volatile_window | 60s | How long recent points stay provisional; keep it longer than the origin’s -search.latencyOffset (30s) plus its ingestion delay |
Authentication headers, TLS and request rewriting work as for any HTTP backend; requests that
carry Authorization are cached under keys that include it. The default health check runs
query=1 against the query API, so it also checks a tenant’s path.
Range Queries
Trickster answers a range query on the grid VictoriaMetrics evaluates it on. With its response
cache on, VictoriaMetrics aligns a range of 50 or more points to start on a step boundary and
keeps the number of points; a shorter range keeps the requested start. Trickster rewrites the
range the same way before looking in its cache, so cached and native answers have the same
timestamps. Grafana aligns its ranges itself, so its requests are unchanged. With
search_disable_cache: true, every range keeps the requested start, as VictoriaMetrics then does.
Cache Eligibility
Each MetricsQL statement is parsed once with VictoriaMetrics’ own parser
(metricsql, pinned to the version the provider
was validated with); WITH templates are expanded before classification.
| Statement | Path |
|---|---|
Selectors, rollups (rate, increase, *_over_time, rollup, with or without a window), aggregations, topk/bottomk, binary operators, label and math functions, offset, subqueries, @ with a timestamp | Delta proxy cache |
range_*, running_*, keep_last_value, keep_next_value, interpolate, remove_resets, smooth_exponential, sort*, limit_offset, drop_common_labels, drop_empty_series, union, buckets_limit, start(), end(); topk_*/bottomk_* variants, limitk, any, outliers*; the limit modifier; @ with anything but a timestamp | Exact-request object cache (step alignment off) |
now(), rand(), rand_normal(), rand_exponential(), and statements that don’t parse | Relayed uncached |
A statement that doesn’t parse is relayed, so VictoriaMetrics returns its own error. A request is also relayed uncached when it:
- sets
nocacheortrace; - has a parameter the route doesn’t read, or a parameter repeated, or the same parameter in both the URL and a POST form;
- uses a time VictoriaMetrics would read differently than Trickster: anything but Unix seconds
(with at most millisecond precision) or RFC 3339 with a zone, such as
-1h,now, a zoneless time, or milliseconds; - is an instant query within the volatile window of now, or without a
time; - is a POST that isn’t
application/x-www-form-urlencoded.
round_digits, latency_offset, max_lookback, extra_label, extra_filters[], limit and
optimize_repeated_binary_op_subexprs are part of a response’s cache identity; timeout and
denyPartialResponse can only fail a request and are not. The classification of each request
is counted in trickster_victoriametrics_query_analysis_total.
Fidelity
VictoriaMetrics evaluates a point from the samples it fetched for the whole request: an implicit
window, and whether rate and similar functions use the sample before their window, depend on a
scrape interval estimated from the last 20 sample intervals it read. Where a series’ scrape
interval changes, a range assembled from separate fetches can differ slightly from one
nocache=1 evaluation of it. VictoriaMetrics’ own response cache, which is on by default,
assembles ranges from separate evaluations the same way, so the delta proxy cache’s answers match
what VictoriaMetrics serves by default. For a single exact evaluation, set
step_alignment: off on the backend, or write # trickster-step-align:off in the query.
Points within the volatile window of now, which VictoriaMetrics replaces within its
-search.latencyOffset, are fetched on every request and never stored. A request’s own
latency_offset widens that window.
Partial Responses
When some vmstorage nodes don’t answer, vmselect marks its response "isPartial":true.
Trickster never caches such a response: a range query is relayed to the origin, and other API
responses are marked Cache-Control: no-store. Responses served from cache for an origin that
reports isPartial include "isPartial":false, as vmselect writes it. The stats object
VictoriaMetrics adds to query responses describes one evaluation and is not kept in cached
responses.
Graphite
The Graphite APIs use the Storage-Step request header, which sets the grid the render API
returns; it is part of a render response’s cache identity, with every query parameter. A render
is consolidated to maxDataPoints over its whole range, so renders are cached only as exact
requests, never assembled from separate ranges. A render is cached only when from and until
are Unix timestamps and until is outside the volatile window; Grafana sends both that way.
Time Series Merging
The tsm ALB mechanism doesn’t support VictoriaMetrics backends yet: the provider offers no merge
planning, so MetricsQL is never merged with PromQL’s semantics, and requests through a tsm pool
of VictoriaMetrics backends fail. Other ALB mechanisms work as for any backend.
Known Limits
- Delta caching is not offered for the Graphite render API.
partial_end(Fast Forward) is not offered.- An unfiltered
/tagsor/tags/<tag>returns nothing for series VictoriaMetrics stored as labels;/tags/autoComplete/tags?expr=...,/tags/findSeriesandseriesByTagfind them. This is VictoriaMetrics’ behavior and is relayed as is. - Requests through the frontend’s
/graphitealias are mapped to the Graphite path for vmselect origins too, where vmselect has no such alias. - Raw-series export (
/api/v1/export) is relayed uncached.
Developer Environment
The developer environment runs single-node VictoriaMetrics v1.153.0 with seeded trips history
and fixtures, the victoriametrics1 backend, and the Trips (VictoriaMetrics) and
VictoriaMetrics Graphite dashboards. See
VictoriaMetrics Details.
10 - MySQL Provider
Trickster can accept native MySQL client connections, proxy each authenticated session to a MySQL origin, and cache eligible text-protocol query results. This guide describes the initial supported contract. Behavior not listed here is rejected or proxied without caching; cache eligibility always fails closed.
Compatibility
The supported matrix is:
- Oracle MySQL 8.4 LTS through 9.7 LTS;
- Grafana 11.0 and above using Grafana’s built-in MySQL data source; and
- native clients that use
mysql_native_passwordand the supported commands below.
MariaDB, Percona Server, Aurora MySQL, other MySQL-compatible products, older Grafana releases, and third-party Grafana MySQL data sources are not part of the initial compatibility claim, but should work without problems. File an issue if you discover a compatibility issue.
The developer environment pins MySQL 8.4 and Grafana 13.
Direct backend configuration
The listener accepts the MySQL wire protocol on one TCP port. MySQL negotiates
TLS in-band, so do not configure an HTTP-style tls_port.
listeners:
mysql-native:
protocol: mysql
address: ""
port: 8486
connections_limit: 200
mysql:
handshake_timeout: 10s
read_timeout: 30s
write_timeout: 30s
idle_timeout: 5m
max_packet_size_bytes: 16777215
max_query_size_bytes: 1048576
authenticators:
mysql-clients:
provider: basic
users:
grafana_reader: ${GRAFANA_MYSQL_PASSWORD}
caches:
mysql-cache:
provider: memory
memory:
max_size_bytes: 536870912
backends:
mysql-primary:
provider: mysql
listener_names: [mysql-native]
authenticator_name: mysql-clients
origin_url: mysql://trickster_ro:REDACTED@mysql.example:3306/analytics
cache_name: mysql-cache
timeout: 30s
max_concurrent_conns: 200
max_object_size_bytes: 8388608
mysql:
max_result_rows: 100000
max_result_size_bytes: 67108864
healthcheck:
interval: 5s
timeout: 3s
failure_threshold: 3
recovery_threshold: 2
Exactly one direct MySQL backend, one supported MySQL User Router ALB, or one
session-balancing ALB maps to a MySQL
listener. The origin URL must use the mysql scheme and include an
origin username. Percent-encode reserved username, password, and database
characters. Configuration stringification and the sanitized management
configuration redact an embedded origin password, but the source configuration
must still be protected as a secret.
The authenticator is the only downstream credential source. Trickster terminates downstream authentication; it does not pass client credentials to the origin or silently fall back to origin credentials. Native-password verification requires the original password, so MySQL authenticator entries must be plaintext values, including environment-expanded values or a protected CSV users file. Htpasswd hashes are rejected for this listener. Do not expose the raw configuration management endpoint outside a trusted administrative network.
TLS
Downstream TLS
Add a server certificate and key to the listener-facing backend:
backends:
mysql-primary:
# ...direct backend settings above...
require_tls: true
tls:
full_chain_cert_path: /etc/trickster/tls/server.crt
private_key_path: /etc/trickster/tls/server.key
Without a server pair, downstream TLS is disabled. With the pair, TLS is
optional unless require_tls: true rejects plaintext clients. TLS 1.2 is the
minimum. Downstream client certificates are not supported in the initial
release.
Upstream TLS
The upstream modes are:
- no upstream TLS fields: plaintext;
insecure_skip_verify: true: encryption without server verification;- one
certificate_authority_pathsentry: CA and hostname verification; and - a complete
client_cert_path/client_key_pathpair: mutual TLS, combined with verified or explicitly unverified server mode.
backends:
mysql-primary:
# ...direct backend settings above...
tls:
certificate_authority_paths:
- /etc/trickster/tls/origin-ca.crt
client_cert_path: /etc/trickster/tls/origin-client.crt
client_key_path: /etc/trickster/tls/origin-client.key
Certificate/key settings must be complete pairs. More than one upstream CA
path, require_tls without a downstream server pair, or an incomplete pair is
a configuration error. There is no CA-verification-without-hostname mode.
Connections, limits, and lifecycle
Each authenticated downstream connection owns exactly one upstream connection.
There is no pooling or multiplexing, which prevents transaction, database, and
session state from leaking between clients. Size connections_limit and
max_concurrent_conns together and leave capacity for health probes. A useful
starting point is one admitted origin connection per downstream connection,
plus normal operational headroom.
Listener limits protect the downstream boundary:
connections_limitlimits concurrent client connections (0is unlimited);- handshake, command-read, response-write, and idle timeouts bound each phase;
max_packet_size_bytesis fixed at 16 MiB minus one; andmax_query_size_byteslimits SQL text within that packet ceiling.
Backend limits protect the origin and cache:
timeoutbounds connect and query work;max_concurrent_connsbounds upstream admission;mysql.max_result_rowsandmysql.max_result_size_bytesbound results; andmax_object_size_bytesskips caching an otherwise valid oversized object.
Handshake, packet, query, read, write, and idle failures close the client connection. Origin timeout or concurrency rejection closes the affected upstream connection and returns a bounded MySQL error. A row/byte overflow closes both sides because a partial result may already have been emitted.
On shutdown, Trickster stops accepting new sessions and drains existing ones within the configured management drain window. A reload that changes MySQL authentication, listener transport, certificates, origin transport, routing, or relevant limits restarts and drains the listener; an established session is never silently moved to new credentials, TLS policy, or a different origin.
Protocol behavior
Supported commands are:
COM_QUERYcontaining one text-protocol statement;COM_INIT_DB;COM_PING;COM_QUIT; andCOM_RESET_CONNECTION.
Reset discards the upstream connection and tracked session state. The next command opens a fresh origin connection.
The initial release rejects binary prepared statements; SQL PREPARE,
EXECUTE, and DEALLOCATE; server cursors; binary results; compression;
LOCAL INFILE and all LOAD statements; multi-statements; stored-procedure
CALL; HELP, XA, HANDLER, and CACHE INDEX; executable version comments;
unclassified text response shapes; COM_CHANGE_USER; replica registration; and
binlog commands. Connection attributes may be syntactically accepted but are
ignored for authentication, routing, cache identity, and query semantics.
Malformed, oversized, timed-out, or partially streamed commands close the
connection. A fully consumed unsupported command normally leaves it usable.
Successful DML and DDL are proxied and never cached. They do not invalidate existing OPC or DPC entries, so another session may see cached data until TTL expiry or eviction. Use short TTLs or purge affected cache data when immediate read-after-write visibility across sessions is required.
Cache classification
Trickster uses three outcomes:
- Delta Proxy Cache (DPC): a supported deterministic aggregate time-series
SELECTwith one unambiguous time output, literal positive cadence, safe result shape and ordering, and an inclusive-lower/exclusive-upper raw-time predicate; - Object Proxy Cache (OPC): a deterministic single-result
SELECTthat is safe to cache as one object but is not proven reusable by time extent; and - proxy-only: mutations, transactions, unsupported or unsafe session state, non-deterministic or rejected shapes, and any query whose safety cannot be established.
DPC supports the tested zero-phase integer/cast and
FLOOR(UNIX_TIMESTAMP(column) / n) * n bucket forms when n is the same
positive integer literal on both sides. Dynamic, mismatched, non-positive,
overflowing, ambiguous, or non-zero-phase forms fail closed. Qualified and
backtick-quoted identifiers, multiple numeric values, and string dimensions
are supported only in the shapes recorded by the compatibility corpus.
The authenticated downstream username, selected terminal backend, normalized database, supported time-zone literal, and query identity participate in cache keys. Cache entries are never shared across authenticated users merely because their SQL is identical.
Grafana macros and exact SQL shapes
Only patterns represented by the versioned compatibility corpus are published
as supported. The corpus covers $__time, $__timeEpoch, $__timeFilter,
$__timeFrom, $__timeTo, $__timeGroup, $__timeGroupAlias,
$__unixEpochFilter, $__unixEpochFrom, $__unixEpochTo,
$__unixEpochGroup, and $__unixEpochGroupAlias. Its documented minimum
$__interval is one minute.
Grafana’s normal inclusive $__timeFilter expansion is delta-cached (DPC):
SELECT
CAST(CAST(UNIX_TIMESTAMP(ts)/(300) AS SIGNED)*300 AS SIGNED) AS time,
COUNT(*) AS samples
FROM telemetry
WHERE ts BETWEEN FROM_UNIXTIME(1785542400)
AND FROM_UNIXTIME(1785628800)
GROUP BY time
ORDER BY time
A half-open predicate composed from $__timeFrom() and $__timeTo() is also DPC:
SELECT
CAST(CAST(UNIX_TIMESTAMP(ts)/(60) AS SIGNED)*60 AS SIGNED) AS time,
AVG(value) AS mean_value
FROM telemetry
WHERE site = 'denver'
AND ts >= FROM_UNIXTIME(1785542400)
AND ts < FROM_UNIXTIME(1785628800)
GROUP BY time
ORDER BY time
The epoch-second equivalent is:
SELECT
CAST(CAST(epoch_seconds/(300) AS SIGNED)*300 AS SIGNED) AS time,
SUM(requests) AS requests
FROM service_rollups
WHERE region = 'us-west'
AND epoch_seconds >= 1785542400
AND epoch_seconds < 1785628800
GROUP BY time
ORDER BY time
Trickster caches only complete buckets. Under the default step_alignment,
drop, it rounds the lower bound up and the upper bound down to the cadence,
and an inclusive upper bound, such as the end of BETWEEN, also drops the
bucket that contains it, because that bucket is only partly covered. The
partial, partial_start and partial_end modes fetch those edge buckets
from the origin over the client’s own bounds, through the object cache for
partial_bucket_ttl, at a cost of up to two small origin queries per request,
and truncate answers the whole first bucket. A range with no complete bucket
is answered with the origin’s own result for the statement, cached as an object
for partial_bucket_ttl. A query can choose its own mode with a comment, such
as /* trickster-step-align:partial */. See Step Alignment.
Grafana’s strict-lower $__unixEpochFilter expansion remains OPC, because a
strict lower bound does not cover the first bucket completely. Native DATETIME/TIMESTAMP, epoch-second integer,
and the corpus’s epoch-nanosecond adaptation are supported in their recorded
shapes.
Session state
The only cache-safe tracked state is the selected default database and one
session-scoped literal SET time_zone = '<value>'. USE and COM_INIT_DB
update the database portion of cache identity.
Transactions and savepoints bypass cache while active. Other SET forms,
character set or collation changes, sql_mode, lc_time_names, user
variables, temporary objects, locks, session-local functions, mutations, and
unclassified state changes make that connection cache-unsafe. Reconnect or a
successful COM_RESET_CONNECTION is required to restore the configured
baseline.
Protocol-aware User Router
A native MySQL User Router selects one direct terminal MySQL backend from the verified downstream username:
listeners:
mysql-routed:
protocol: mysql
port: 8486
connections_limit: 200
authenticators:
tenant-clients:
provider: basic
users:
tenant_a_reader: ${TENANT_A_CLIENT_PASSWORD}
tenant_b_reader: ${TENANT_B_CLIENT_PASSWORD}
caches:
tenant-a-cache:
provider: memory
tenant-b-cache:
provider: filesystem
backends:
tenant-a-mysql:
provider: mysql
authenticator_name: tenant-clients
origin_url: mysql://tenant_a_ro:REDACTED@mysql-a.example:3306/analytics
cache_name: tenant-a-cache
healthcheck:
interval: 5s
timeout: 3s
tenant-b-mysql:
provider: mysql
authenticator_name: tenant-clients
origin_url: mysql://tenant_b_ro:REDACTED@mysql-b.example:3306/analytics
cache_name: tenant-b-cache
healthcheck:
interval: 5s
timeout: 3s
mysql-by-tenant:
provider: alb
listener_names: [mysql-routed]
authenticator_name: tenant-clients
alb:
mechanism: ur
user_router:
default_backend: tenant-a-mysql
users:
tenant_a_reader:
to_backend: tenant-a-mysql
tenant_b_reader:
to_backend: tenant-b-mysql
The listener-facing router owns the one downstream authentication exchange, admission, and TLS. Direct terminal MySQL backend entries must reference the same named authenticator to satisfy backend validation, but do not perform a second client exchange. The terminal owns its origin credentials, upstream TLS, cache, health, limits, and query policy. Routing occurs once before opening the upstream connection and remains sticky for the whole downstream session. Transactions, database changes, and session state never trigger rerouting.
Only direct MySQL terminal backends are supported. Nested ALBs, Rules, cycles,
mixed terminal providers, empty routes, and MySQL to_user/to_credential
remapping are configuration errors. A router requires an authenticator. An
unmapped user uses default_backend when configured; without one, Trickster
returns a bounded native MySQL no-route error. HTTP no_route_status_code is
not sent on the native connection. A selected unhealthy or unavailable
terminal fails as a MySQL availability error; the session is not silently
routed elsewhere.
The verified username and selected terminal remain in cache identity. Route metrics use configured router/backend names and bounded outcomes, never the username.
Balancing sessions across replicas
A MySQL listener can also map to an ALB that balances its sessions across a pool of direct MySQL backends, such as read replicas:
backends:
replica-1:
provider: mysql
authenticator_name: app-clients
origin_url: mysql://app_ro:REDACTED@replica-1.example:3306/analytics
healthcheck:
interval: 5s
replica-2:
provider: mysql
authenticator_name: app-clients
origin_url: mysql://app_ro:REDACTED@replica-2.example:3306/analytics
healthcheck:
interval: 5s
replicas:
provider: alb
listener_names: [mysql-replicas]
authenticator_name: app-clients
alb:
mechanism: lc # rr, p2c, lc or hrw
pool:
- replica-1
- name: replica-2
weight: 2
The ALB owns the downstream authentication exchange, admission, and TLS, as a
User Router does, and each pool member owns its origin credentials, cache,
health, and query policy. A session is committed to one member after it
authenticates and stays there until it ends. lc and p2c compare members by
their open sessions; hrw keeps a client address (hrw.key: client_ip, the
default) or a user name (hrw.key: user) on one member. weight, backup
members, and healthy_floor apply as they do for any ALB. lt, the fanout
mechanisms, nested ALBs, mixed providers, and autodiscovery are configuration
errors. See
Load Balancing Native Protocol Sessions.
A change of mechanism or weight applies to new sessions on reload; a change to the set of pool members restarts the listener.
Metrics, logs, and health
Important metrics include:
trickster_sql_query_analysis_total{backend_name,dialect,cache_mode,reason};trickster_sql_query_rewrite_failures_total{backend_name,dialect,reason};trickster_sql_query_cache_total{backend_name,dialect,cache_mode,cache_status};trickster_proxy_request_duration_secondsandtrickster_proxy_points_totalwithprovider="mysql";trickster_mysql_connections_total,trickster_mysql_active_connections,trickster_mysql_errors_total,trickster_mysql_route_selections_total, andtrickster_mysql_command_duration_seconds; and- cache operation, usage, limit, and event metrics labeled by
cache_name.
Example PromQL, replacing names to match the deployment:
sum by (cache_mode, reason) (rate(trickster_sql_query_analysis_total{backend_name="mysql1"}[5m]))
sum by (reason) (rate(trickster_sql_query_rewrite_failures_total{backend_name="mysql1"}[5m]))
sum by (cache_mode, cache_status) (rate(trickster_sql_query_cache_total{backend_name="mysql1"}[5m]))
histogram_quantile(0.95, sum by (le) (rate(trickster_proxy_request_duration_seconds_bucket{backend_name="mysql1",provider="mysql"}[5m])))
sum by (cache_status) (rate(trickster_proxy_points_total{backend_name="mysql1",provider="mysql"}[5m]))
sum by (operation, status) (rate(trickster_cache_operation_objects_total{cache_name="mysql-cache"}[5m]))
trickster_cache_usage_bytes{cache_name="mysql-cache"}
sum by (reason) (rate(trickster_cache_events_total{cache_name="mysql-cache",event="eviction"}[5m]))
Analysis logs contain bounded backend, cache mode, reason, and statement-type fields. Rewrite and protocol failures use bounded categories; SQL text, passwords, and usernames are not metric labels. Keep debug logging temporary in production because classification logs can be high volume.
Native health checks open a bounded connection with the backend’s configured
origin credentials and TLS policy, then issue COM_PING. They use the common
interval, timeout, failure, and recovery thresholds. HTTP health request fields
do not apply. Health output reports sanitized authentication, TLS, timeout,
refused-connection, and server-error categories. See Health Checks
for management endpoint details.
Kubernetes deployment
The base examples expose the optional mysql Service/container port on 8486.
The safe routed example keeps the complete credential-bearing configuration in
deploy/kube/mysql-user-router-secret.yaml
and replaces the base ConfigMap volume with
deploy/kube/mysql-deployment-patch.yaml.
Replace every placeholder and use an encrypted/external secret controller in
production. Do not put an origin DSN, downstream passwords, private keys, or
client keys in deploy/kube/configmap.yaml.
For direct TLS or mTLS, add the server, CA, and client certificate material to
a Kubernetes TLS or opaque Secret, mount it read-only, and reference the mount
paths from the backend tls block. Rotating that Secret requires the
configuration/certificate reload path described above; established sessions
are drained rather than changing transport identity in place.
Operations and troubleshooting
Kubernetes readiness
The native health scheduler does not delay listener startup or make the general
health endpoint return a non-200 status. /trickster/ping proves only that the
process is running, and a TCP readiness probe on port 8486 proves only that the
native listener is accepting connections. The companion manifest uses that TCP
probe. Treat the scheduled MySQL status published in /trickster/health?json
as the origin-readiness signal in an external controller or monitoring rule;
do not assume the endpoint’s HTTP status reflects origin health. Allow at least
interval * failure_threshold or interval * recovery_threshold for a stable
transition. User Router terminals report separately, and a failing selected
terminal returns a native availability error rather than rerouting the session.
Capacity planning
Start from peak concurrent dashboard/client sessions. Reserve approximately
one origin connection per admitted client, then add health and rollout
headroom. Bound result rows/bytes below the pod or process memory budget and
set max_object_size_bytes below both the result-byte limit and a practical
fraction of cache capacity. For an in-memory cache, budget retained objects,
index overhead, concurrent result encoding, and normal process memory. Use
finite query, command, and idle timeouts so abandoned clients cannot retain
connections indefinitely.
Repeated misses or proxy-only outcomes
- Group
trickster_sql_query_analysis_totalbycache_modeandreason. - Confirm the query is deterministic, single-result, and outside a transaction or unsafe session.
- For DPC, inspect the expanded SQL: require a literal cadence and
>=lower,<upper raw-time predicates. - Confirm the requested interval contains at least one complete cadence bucket and that cache TTL, volatile window, and retention are suitable.
- Check username, selected backend, database, and time zone; these intentionally isolate keys.
- Inspect cache operation status and eviction metrics for admission failures or churn.
Authentication and TLS
- Authentication failure: verify the named authenticator, plaintext/CSV credential source, exact username, supported native-password plugin, and filesystem permissions on a mounted users file.
- Downstream TLS failure: verify the server pair,
require_tls, client TLS mode, hostname, and TLS 1.2+ support. Do not usetls_port. - Upstream TLS failure: verify the single CA path, hostname in the origin URL,
complete client pair when using mTLS, file permissions, and whether
insecure_skip_verifywas intentionally selected.
Origin and cache failures
During an origin outage, health transitions and MySQL error metrics should rise; existing safe cache hits may continue, while misses fail. Restore origin health before increasing connection limits. A cache-provider failure should surface in cache operation/event metrics and logs; query handling fails closed to the provider’s normal proxy behavior rather than treating an unknown cache state as a hit.
Certificate rotation and relevant configuration reloads drain affected native listeners. Monitor active connections, authentication/TLS errors, route selection outcomes, health state, rewrite failures, cache status, and p95 latency until the old sessions have drained.
Rollout and rollback
Canary the listener/backend configuration with a limited client population. Compare against the pre-release baseline and roll back for a sustained increase of 5 percentage points in proxy/origin/protocol errors, 10 percentage points in proxy-only classifications, 1 percentage point in rewrite failures, or 20% in p95 proxy latency. Roll back by restoring the previous configuration and binary/image on the release branch, reloading or restarting Trickster, and allowing the old MySQL listener to drain. Do not move an established session between versions. Preserve metrics and sanitized logs for the incident record.
Environment variables and reload behavior
The global TRK_ORIGIN_URL and TRK_ORIGIN_TYPE variables map only to the
default backend and are suitable only for a basic single-backend deployment.
There are no one-to-one TRK_* variables for MySQL listener limits, backend
limits, TLS, health, or User Router options; configure those in YAML.
Authenticator users values support ${VARIABLE} expansion, and supported
users files and secret volume mounts are preferred for production. The
origin_url field does not expand ${VARIABLE} in structured YAML; store a
configuration containing an embedded DSN as a protected secret, not a public
ConfigMap.
Validate before rollout:
trickster -validate-config -config /etc/trickster/trickster.yaml
SIGHUP and the management reload endpoint reload a changed file. Authentication, TLS, transport, routing, and terminal runtime changes restart and drain the affected MySQL listener. Cache policy changes apply to new work; existing cache objects remain subject to their stored TTL and normal eviction.
Known compatibility gaps
The initial release does not claim support for:
- MySQL-compatible server products outside Oracle MySQL 8.4-9.7;
- downstream plugins other than
mysql_native_passwordor downstream mTLS; - prepared/binary protocols, compression, local infile, multi-results, stored procedures, binlog/replication commands, or connection-attribute semantics;
- general SQL parsing beyond the Vitess-supported and compatibility-corpus shapes;
- window functions, grouping sets, rollups, ambiguous time axes or buckets, unsafe boolean predicates, non-deterministic outputs, dynamic/non-zero-phase buckets, or inclusive-range DPC;
- arbitrary session-state caching or cross-session mutation invalidation; and
- nested/mixed MySQL User Router topologies or origin credential remapping.
Streamed result metadata and FOUND_ROWS()
Vitess does not expose the terminal status flags and warning count of a
streamed result set. To preserve those fields for the downstream client,
Trickster issues SHOW COUNT(*) WARNINGS on the same upstream connection
after the stream ends. That diagnostic statement can update connection-local
MySQL diagnostics, so FOUND_ROWS() immediately after a streamed result may
reflect the hidden warning-count query rather than the client’s previous
statement.
This is a known compatibility gap in the Vitess streaming API, not a Trickster-specific SQL parsing defect. A Vitess API that exposes terminal result metadata is required to remove the workaround.
The executable SQL contract is
pkg/backends/mysql/testdata/compatibility/v1.json. The maintainer-facing frozen release matrix is in
docs/developer/mysql-release-contract.md.