This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Getting Started

How to get up and running with Trickster.

1 - Where to Place Trickster

Depending upon the size of your existing or planned deployment, there are several placement configurations available. These designs are suggestions based on common usage, and you may find alternative or hybrid placement configurations that make the most sense for your situation, based on the activity of your Dashboard and TSDB instance(s).

Single “Everything”

Single “Everything” is the most common placement model. In this configuration, you have one optional dashboard endpoint, one Trickster endpoint and one HTTP or TSDB endpoint. Behind each endpoint, you may have a single instance or a cluster. Each component is only aware of the other component’s endpoint exposure and not the underlying configuration. This configuration represents a one-for-one-for-one deployment of your Dashboard, Origin, and Trickster endpoints.

Multiple Backends

In a Multiple Backend placement, you have one dashboard endpoint, one Trickster endpoint, and multiple TSDB and/or HTTP endpoints. Trickster is aware of each upstream endpoint and treats each as a unique backend to which it proxies and caches data independently from the others. Trickster routes a request to a specific backend based on Host Header or URL Path in the client request.

This setup may benefit situations where you have one or more static file server origins serving HTML, CSS and JavaScript assets and/or one or more API endpoints, all supporting a common platform.

For Time Series Dashboard acceleration, this is a good configuration to use when you have a single dashboard that displays data about multiple redundant clusters (each with its own TSDB), or when you have a single dashboard representing information about many different kinds of systems. For example, if you operate a “Dashboard as a Service” solution under which many teams use your Dashboard system by designing their own dashboard screens and bringing their own databases, a single Trickster endpoint can be used to accelerate dashboards for all of your customers.

You will need to configure each Trickster-to-TSDB mapping separately in your dashboard application as a separately named TSDB data source. Refer to the multi-origin documentation for configuring multi-origin support in Trickster and Grafana.

In this configuration, be aware that the default ‘memory’ cache may be underpowered depending on the number of customers, as well as the size and number of queries that need to be cached by each customer. Refer to the caches document to select and configure the caching layers as needed to meet your specific situation.

Multi-Trickster

In a Multi-Trickster configuration, you have one dashboard endpoint, multiple Trickster endpoints, and multiple TSDB or HTTP endpoints, with each Trickster Endpoint having a one-to-one mapping to a TSDB/HTTP Endpoint as a pair. This is a good design if Multiple Backends is not performant enough for the amount of activity associated with your solution (e.g., you need more Tricksters). If the Dashboard system owner is different from the TSDB system owner, either party could own and operate the Trickster instance.

2 - Configuring Trickster

There are 3 ways to configure Trickster, listed here in the order of evaluation.

  • Configuration File
  • Environment Variables
  • Command Line Arguments

Note that while the Configuration file provides a very robust number of knobs you can adjust, the ENV and CLI Args options support only basic use cases.

Internal Defaults

Internal Defaults are set for all configuration values, and are overridden by the configuration methods described below. All Internal Defaults are described in examples/conf/example.full.yaml comments.

Configuration Files

Trickster accepts a -config /path/to/trickster.yaml command line argument to specify a custom configuration file. The path can also name a directory containing configuration files. If a provided path cannot be accessed by Trickster, it will exit with a fatal error.

When a -config parameter is not provided, Trickster will check for the presence of a config file at /etc/trickster/trickster.yaml and load it if present, or proceed with the Internal Defaults if not present.

Refer to examples/conf/example.full.yaml for full documentation on format of a configuration file.

Multiple Configuration Files

When -config names a file, Trickster loads that file first and then loads supported files from a sibling conf.d directory, if the directory exists. The primary file can select a different include directory:

main:
  config_include_directory: config-parts

A relative config_include_directory is resolved from the directory containing the primary file. An explicitly configured include directory must exist. Only the primary file can set this option; included files cannot redirect configuration discovery.

When -config names a directory, Trickster loads supported files directly from that directory. The directory must contain at least one supported file, and files in this mode cannot set main.config_include_directory.

In both modes, Trickster:

  • Loads only direct regular files whose names do not start with ., with .conf, .yaml, or .yml extensions matched case-insensitively.
  • Loads directory entries in ascending lexical filename order. The primary file, when present, always comes first.
  • Recursively merges mappings. A later scalar or sequence replaces the earlier value, while a later mapping adds to or overrides individual keys in the earlier mapping.
  • Requires each file participating in a multi-source configuration to have a mapping root, one YAML document, and no duplicate keys.

For example, a fragment containing only backends.prometheus.origin_url can change that field without removing the other fields under the prometheus backend. Use null, rather than an empty mapping, when a later file must clear an earlier mapping value.

Configuring Secrets or Sensitive Information

Trickster supports Environment variable substitution in its configuration file where sensitive information is expected.

  • Supported via the following fields:
    • caches[*].redis.password, backends[*].healthcheck.headers, backends[*].cors.headers, backends[*].paths[*].cors.headers, backends[*].paths[*].request_headers, backends[*].paths[*].request_params, backends[*].paths[*].response_headers

Usage ${ENV_VAR_NAME}, example:

caches:
  default:
    redis:
      password: "${MY_REDIS_PW}"

Environment Variables

Trickster will then check for and evaluate the following Environment Variables:

  • TRK_ORIGIN_URL=http://prometheus.example.com:9090 - The default origin URL for proxying all http requests
  • TRK_ORIGIN_TYPE=prometheus - The type of supported backend server
  • TRK_LOG_LEVEL=INFO - Level of Logging that Trickster will output
  • TRK_PROXY_PORT=8480 -Listener port for the HTTP Proxy Endpoint
  • TRK_METRICS_PORT=8481 - Listener port for the Metrics and pprof debugging HTTP Endpoint

Command Line Arguments

Finally, Trickster will check for and evaluate the following Command Line Arguments:

  • -log-level INFO - Level of Logging that Trickster will output
  • -config /path/to/trickster.yaml - See Configuration Files section above
  • -origin-url http://prometheus.example.com:9090 - The default origin URL for proxying all http requests
  • -provider prometheus - The type of supported backend server
  • -proxy-port 8480 - Listener port for the HTTP Proxy Endpoint
  • -metrics-port 8481 - Listener port for the Metrics and pprof debugging HTTP Endpoint

Inbound Listeners

The top-level listeners map configures inbound listeners. Trickster always auto-defines three entries using the existing defaults: default, metrics, and mgmt.

Native MySQL listeners have additional protocol, authentication, TLS, and session-lifecycle requirements. See the MySQL Provider Guide before configuring protocol: mysql. ClickHouse Native listeners use protocol: clickhouse; see the ClickHouse Support Guide for ingress, origin, and TLS options.

listeners:
  default:
    address: ""
    port: 8480
    tls_address: ""
    tls_port: 8483
    connections_limit: 0
    read_header_timeout: 10s
  private_api:
    protocol: http
    address: 127.0.0.1
    port: 9080

backends:
  default:
    listener_names: [default, private_api]
    provider: prometheus
    origin_url: http://prometheus:9090
  private:
    listener_names: [private_api]
    provider: reverseproxy
    origin_url: http://private-origin

listener_names binds a backend to one or more compatible listeners. An ordinary unbound backend uses default; internal routing targets remain unexposed. A backend cannot select the reserved mgmt or metrics listeners, and validation fails for undefined or provider-incompatible listeners.

Each native listener maps to exactly one backend. Multiple HTTP listeners can share a backend, and ClickHouse can bind the same backend to HTTP and ClickHouse Native listeners.

A user-defined listener with no mapped backend is not started and produces a warning. A configured TLS port is enabled only when at least one backend mapped to that listener provides a valid frontend certificate and key in its tls section; otherwise Trickster disables that TLS port and logs a warning.

The top-level frontend section and listener address/port fields under metrics and mgmt remain supported during the compatibility period. Trickster logs deprecation warnings when those legacy listener settings are used. When the same built-in listener is present in listeners, its new configuration takes precedence.

Configuration Validation

Trickster can validate configuration files by running trickster -validate-config -config /path/to/config. Trickster will load the file or directory and exit with the validation result, without running the configuration.

Reloading the Configuration

Trickster can gracefully reload its configuration sources from disk without impacting the uptime and responsiveness of the application.

Trickster supports manual reloads by requesting an HTTP endpoint or sending a SIGHUP (e.g., kill -1 $TRICKSTER_PID) to the Trickster process. It can also poll the effective configuration sources automatically. In all cases, at least one effective configuration source must have changed since the configuration was loaded.

Automatic Config Reload

Trickster can poll its effective configuration sources and reload after a change. This is disabled by default. Set mgmt.auto_reload_interval to a positive duration to enable it:

mgmt:
  auto_reload_interval: 10s

Polling uses the same validation and graceful reload path as SIGHUP and the management endpoint. The interval itself is reloadable, so a successful configuration update can change or disable automatic reloads. Polling is suitable for Kubernetes ConfigMap projected volumes, whose atomic symlink updates are not reliably represented as writes to the mounted file by filesystem notification APIs.

Config Reload via SIGHUP

Once you have made the desired modifications to your config file, send a SIGHUP to the Trickster process by running kill -1 $TRICKSTER_PID. The Trickster log will indicate whether the reload attempt was successful or not.

Config Reload via HTTP Endpoint

Trickster provides an HTTP Endpoint for viewing the running Configuration, as well as requesting a configuration reload.

The reload endpoint is configured by default to listen on address 127.0.0.1 and port 8484, at /trickster/config/reload. These values can be customized, as demonstrated in the example.full.yaml The examples in this section will assume the defaults. Set the port to -1 to disable the reload HTTP interface altogether.

To reload the config, simply make a GET request to the reload endpoint. If an underlying configuration source has changed, or a supported file has been added to or removed from a configured directory, the configuration will be reloaded and the caller will receive a success response. If the configuration sources have not changed, the caller will receive an unsuccessful response, and reloading will be disabled for the duration of the Reload Rate Limiter. By default, this is 3 seconds, but can be customized as demonstrated in the example config file.

If a listener address or port changes, Trickster drains the old listener before starting its replacement. Listeners whose network settings do not change retain their open sockets and receive the refreshed router in place. Removed or newly unused listeners are drained and stopped, while newly mapped listeners are started. The drain period is configurable and defaults to 30 seconds. The Drain Timeout also applies to old log files when a new log filename is provided.

View the Running Configuration

Trickster also provides a http://127.0.0.1:8484/trickster/config endpoint, which returns the yaml output of the currently-running Trickster configuration. The YAML-formatted configuration will include all defaults populated, overlaid with any configuration file settings, command-line arguments and or applicable environment variables. By default, this interface is available only on the management listener. Set mgmt.config_handler_listener to metrics, both, or off to change where it is exposed. This path is configurable as demonstrated in the example config file.

Trickster also provides a sanitized view of the running configuration at http://127.0.0.1:8484/trickster/config/sanitized. If the config_handler_path is customized, append /sanitized to the configured path. The sanitized output deep-copies the running configuration, renames cache, backend, listener, and tracing resources by provider and sequence number (for example, prom-1, prom-2, alb-1, memory-1, listener-1, otlp-1), renames authenticators as auth1, auth2, etc., updates references to those resources in backend, path, ALB, rule, cache, tracing, listener, and authenticator mappings, replaces backend origin_url, Redis endpoint and endpoints, tracing endpoint, and Host-related request rewriter values with example.com, redacts per-path request and response header values, and replaces embedded authenticator users with user1: redacted, user2: redacted, etc. This endpoint is intended for sharing running configuration details in support requests without exposing private infrastructure names, origin endpoints, or user credentials.

3 - Trickster 2.0

An All-New Bag of Tricks

Trickster 2.0 is a near-complete rewrite of the project with performance, durability and extensibility in mind. We’ve made major architectural improvements, added new features and improved performance. Here’s a comprehensive overview of it all:

Features

  • Application Load Balancer (ALB): We have a brand new Application Load Balancer available as a backend provider type, with unique and powerful options, including:
    • Fanout and merge data from multiple time series backends into a single response
      • currently supports Prometheus backends
    • Fanout and return the first response received from any backend pool member
    • Fanout and return the response with the newest ‘Last-Modified’ header from all backend pool members
    • Routing a request to a given backend based on Basic Auth username
    • General Round Robin
  • Enhanced Health Checking: Health checking now supports automated health check polling for any backend, and provides a global health status endpoint. Automated health checks for an ALB pool member backend determines whether the ALB will route requests to it or not.
  • Cache Object Chunking: We now support Cache Object Chunking. This optional configuration allows a Time Series dataset to be chunked into multiple cache entries based on a configurable chunk size
    • Also supported by standard Reverse Proxy Cache for cache-chunking objects by a Byte Range size
    • Only the chunked cache entries needed to span the request range are inspected, rather than the entire time series
    • Significantly improves Redis and Filesystem performance of large timeseries records
  • Time Series Backend Request Sharding: We now support Time Series Backend Request Sharding. This optional feature allows a client request destined for a Time Series Backend to be cloned into multiple concurrent requests with different time ranges constituting the full uncached range
    • Backend shard responses + already-cached data are merged into a single downstream response
    • Cache Chunking and TS Backend Request Sharding work together seamlessly and can be used in any combination (on/on, on/off, off/on, off/off, including different cache chunk and request shard sizes).
  • Authenticator: We’ve added a new Authenticator feature so you can guard backends with Basic Auth or ClickHouse Auth
  • Enhanced Rules Engine: The Rules Engine now supports rmatch operations to permit regular expression-based routing against any part of the HTTP request
  • Request Rewriters: You can now chain a collection of request rewriters for more robust request transformation possibilities
  • Cache Purging: We now support purging specific cache items by Key (on the public ports) or Path (on the mgmt port). Read more in the cache documentation
  • Simulated Latency: You can use Trickster for Simulated Latency in lab environments
  • We’ve added support for InfluxDB 2.0 and Flux Query Language, plus InfluxDB 3.x via the native v3 HTTP API (SQL and InfluxQL) and an optional Apache Arrow Flight SQL (gRPC) proxy. See the InfluxDB support doc for details.

Configuration & Security

  • YAML Configuration: Trickster 2.0 uses YAML for configuration instead of TOML
  • Environment Variable Substitution: Environment Variable substitution is now possible in configuration files where sensitive information is expected
    • Supported via the following fields:
      • caches[*].redis.password, backends[*].healthcheck.headers, backends[*].paths[*].request_headers, backends[*].paths[*].request_params, backends[*].paths[*].response_headers, authenticators[*].users
    • Usage: password: ${MY_SECRET_VAR}
  • Request Body Size Limit: A configurable Request Body Size limit has been added for POST, PUT and PATCH requests, with a default of 10MB. Requests with a body size exceeding the limit will receive a 413 Content Too Large response. See Request Body Handling Customizations for more info
  • Zipkin Deprecation: Trickster uses packages provided by the OpenTelemetry project for Distributed Tracing capabilities. These external packages recently deprecated support for exporting spans via Zipkin, therefore Zipkin support has been removed for Trickster 2.0. Trickster retains support for OTLP.

Under the Hood & Performance

  • Common Time Series Format: We now use a common time series format internally for caching all supported TSDB’s, rather than implementing delta + merge algorithms per-provider.
  • New HTTP Request Router: We’ve switched to an all-new, made-for-proxies HTTP Request Router, which is up to 10X faster than the previous one
  • SQL Parser: We’ve switched from Regular Expression matches for SQL-based Time Series Backends to an extensible lexer/parser solution, providing better performance and accuracy
    • ClickHouse backend providers now use the new SQL Parser
  • Race Condition Fixes: We’ve eliminated nearly 100 race conditions and random panics
  • Expanded Compression Support: Compression support now includes options for Brotli and Zstd

Developer & Environment Improvements

  • Package Reorganization: We’ve re-organized many packages in the codebase to be more easily importable by other projects. A future maintenance release will add documentation to the examples folder for using Trickster packages in your own projects, including caching, acceleration and load balancing
  • For Trickster contributors, we have a new Docker Compose for developer environments.
  • CI/CD Enhancements: We have added new CI tools, including better linters and race condition checkers to enforce and ensure ongoing project quality
  • Docker Automation: We’ve updated our Docker automation:
    • The trickster-docker-images repo is now retired and image publishing is handled in the trickster repo
    • All merges to main will now push an image to Docker Hub at trickstercache/trickster:main as well as to trickstercache/trickster:<COMMIT_ID>
      • We no longer push images to the legacy tricksterio and tricksterproxy orgs on DockerHub. Everything is now and only trickstercache 🎉!
    • Images are also now pushed to the GitHub Container Repository as ghcr.io/trickstercache/trickster
  • Helm Charts: The Helm Charts repository is now updated for Trickster 2.0
  • Vendor Directory: We no longer include the vendor directory in the project repository and vendor is now in .gitignore. vendor will continue to be included in Release source tarballs

Documentation & Examples

  • Example Configurations: Example configurations are relocated to the examples directory
  • Docker Compose Demo: The Trickster docker-compose demo has been relocated to the examples directory and updated to use the latest version tags. This is the easiest way to try out Trickster 2.0!

Still to Come

A future Trickster 2.0.x will include the following additional features that didn’t quite make it to the finish line:

  • incorporate ALB examples into the docker-compose demo

Installing

You can build the 2.0 binary from the main branch, download binaries from the Releases page, or use the trickstercache/trickster Docker image tag in containerized environments