Speed up your applications with Trickster’s HTTP reverse proxy caching and dashboard query accelerator for time series databases. Trickster’s two-pronged approach reduces unnecessary data transfer between the client, dashboard server, HTTP endpoints, and databases.

Customizable HTTP reverse proxy caching leverages features such as metrics, health checks, and distributed tracing to help you get the most out of each data request. And with two types of collapsed forwarding, you can ensure that your application is making only the necessary data requests.

By working between endpoints, databases, users and the dashboard server, Trickster can dramatically reduce time series database queries by checking for cached data and only asking for data that is outstanding.

Diagram of Trickster logo with arrows from icons representing the client and a dashboard server and arrows from the Trickster logo to icons for HTTP Endpoints and Time Series Databases

Diagram of Trickster working between the client, dashboard server, HTTP endpoints, and databases to reduce data requests.

Trickster Proxy Features

Trickster is a fully-featured HTTP reverse proxy cache for HTTP applications such as static file servers and web APIs. Trickster’s proxy features include the following:

Supports Transport Layer Security (TLS) Protocol and HTTP/2 for frontend termination and backend origination
Offers several options for a caching layer, including in-memory, filesystem, Redis and bbolt
Highly customizable, using minimal configuration settings, down to the HTTP Path
Built-in Prometheus metrics and customizable Health Check Endpoints for end-to-end monitoring
Negative Caching to prevent domino effect outages
High-performance Collapsed Forwarding
Best-in-class Byte Range Request caching and acceleration.
Distributed Tracing via OpenTelemetry, supporting Jaeger and Zipkin
Rules engine for custom request routing and rewriting

Time Series Database Accelerator

Trickster dramatically improves dashboard chart rendering times for end users by eliminating redundant computations on the time series databases, or TSDBs, that it fronts. In short, Trickster makes read-heavy Dashboard/TSDB environments, as well as those with highly-cardinalized datasets, significantly more performant and scalable.

Compatibility

Trickster works with virtually any Dashboard application that makes queries to any of these TSDBs:

Prometheus logo Prometheus

ClickHouse logo ClickHouse

InfluxDB logo InfluxDB

Circonus IRONdb logo Circonus IRONdb

See the Supported Providers documentation for details.

How Trickster Accelerates Time Series

1. Time Series Delta Proxy Cache

Most dashboards request from a time series database the entire time range of data they wish to present every time a user’s dashboard loads, as well as on every auto-refresh. Trickster’s Delta Proxy inspects the time range of a client query to determine what data points are already cached, and requests from the tsdb only the data points still needed to service the client request. This results in dramatically faster chart load times, since the tsdb is queried only for tiny incremental changes on each dashboard load, rather than several hundred data points of duplicative data.

Diagram of how Trickster determines cache range

2. Step Boundary Normalization

When Trickster requests data from a tsdb, it adjusts the client’s requested time range slightly to ensure that all data points returned are aligned to normalized step boundaries. For example, if the step is 300s, all data points will fall on the clock 0’s and 5’s. This ensures that the data is highly cacheable, is conveyed visually to users in a more familiar way, and that all dashboard users see identical data on their screens.

Diagram of Trickster adjusting the client's requested time range.

3. Fast Forward

Trickster’s Fast Forward feature ensures that even with step boundary normalization, real-time graphs still always show the most recent data, regardless of how far away the next step boundary is. For example, if your chart step is 300s, and the time is currently 1:21p, you would normally be waiting another four minutes for a new data point at 1:25p. Trickster will break the step interval for the most recent data point and always include it in the response to clients requesting real-time data.

Diagram of Trickster breaking the step interval for the most recent data point and including it in the response to clients requesting real-time data.

Next steps

Ready to try it out?

Quickstart: Try Trickster with Docker Compose

2 - Quickstart

Try Trickster with Docker Compose and minimal setup.

This composition creates service containers for Prometheus, Grafana, Jaeger, Zipkin, Redis, Trickster and Mockster that together demonstrate several basic end-to-end configurations for running Trickster in your environment with different cache and tracing provider options.

If you have any of these services running locally already, you may run into port conflicts and need to temporarily spin down the conflicting processes.

Prerequisites

You should already have the following installed:

Starting the demo

Clone the Trickster Github project.
In the trickster directory, change your working directory to ./examples/docker-compose with the following command:
```
cd examples/docker-compose
```
To run the demo from the demo directory, enter the following command:
```
docker-compose up -d
```
You can interact with each of the services on their exposed ports (as defined in Compose file), or by running docker logs $container_name, docker attach $container_name, etc.

Exploring Trickster

Grafana

Once the composition is running, we recommend exploring with Grafana, at http://127.0.0.1:3000/. Grafana is pre-configured with datasources and a sample dashboard that are ready-to-use for the demo.

Jaeger UI

Jaeger UI is available at http://127.0.0.1:16686, which provides visualization of traces shipped by Trickster and Grafana. The more you use trickster-based data sources in Grafana, the more traces you will see in Jaeger. This composition runs the Jaeger All-in-One container. Trickster ships some traces to the Agent and others directly to the Collector, so as to demonstrate both capabilities. The Trickster config determines which upstream origin ships which traces where.

For a variety of configurations and other bootstrap data, review the various files in the docker-compose-data folder. This might be useful for configuring and using Trickster (or any of these other fantastic projects) in your own deployments. Try adding, removing, or changing some of the trickster configurations in ./docker-compose-data/trickster-config/trickster.yaml and then docker exec docker-compose_trickster_1 kill -1 1 into the Trickster container to apply the changes, or restart the environment altogether with docker-compose restart. Just be sure to make a backup of the original config first, so you don’t have to download it again later.

Example Datasources

The sim-* datasources generate on-the-fly simulation data for any possible time range, so you can immediately use them after starting up the environment. Note, however, that the simulated data is not representative of reality in any way.

The non-sim Prometheus container that backs the prom-* datasources polls the newly-running environment to generate metrics that will then populate the dashboard. Since the Prometheus container only collects and stores metrics while the environment is running, you’ll need to wait a minute or two for those datasources to show any data on the dashoard in real-time.

Getting Real Dashboard Data

Using datasources backed by the real Prometheus and Trickster (the prom-trickster-* datasources), rather than the simulator, to explore the dashboard is more desirable for the demo. It better conveys the shape and nature of the Trickster-specific metrics that might be unfamiliar. However, since there is no historical data in the demo composition, that creates an upfront barrier.

Keeping the dashboard open and auto-refreshing against any trickster-labeled datasource will help to generate real metrics in Trickster, such as request rates, cache hit rates, etc. Prometheus will collect and store those metrics, and the Grafana dashboard will query and render those metrics. So by keeping the demo dashboard open and refreshing, you are helping to generate the very metrics that the dashboard presents, making the demo much more visually useful while being very meta.

In addition to generating metrics, using the trickster-labeled datasources generates traces that are viewable in Jaeger UI, as described above.

Stopping the Demo and Cleaning Up

To stop and remove the demo, run docker-compose down in the ./examples/docker-compose directory.

3 - Getting Started

The following articles cover getting up and running with Trickster.

3.1 - Installing Trickster

Install Trickster to get started.

Try Trickster

To try a demo of Trickster with Docker before installing, see our end-to-end Quickstart for a zero-configuration running environment.

Installing with Docker

Docker images are available on Docker Hub. To install Trickster with Docker, run the following command:

$ docker run --name trickster -d -v /path/to/trickster.conf:/etc/trickster/trickster.conf -p 0.0.0.0:8480:8480 tricksterproxy/trickster

See the Deployment documentation for more information about using or creating Trickster Docker images.

Installing with Kubernetes

To install Trickster with Kubernetes, see Deployment.

Helm

Trickster Helm Charts are located at https://helm.tricksterproxy.io for installation, and maintained at https://github.com/tricksterproxy/helm-charts. We welcome chart contributions.

Building from source

To build Trickster from the source code yourself you need to have a working Go environment with version 1.9 or greater installed.

You can directly use the go tool to download and install the trickster binary into your GOPATH:

$ go get github.com/tricksterproxy/trickster
$ trickster -origin-url http://prometheus.example.com:9090 -origin-type prometheus

You can also clone the repository yourself and build using make:

$ mkdir -p $GOPATH/src/github.com/tricksterproxy
$ cd $GOPATH/src/github.com/tricksterproxy
$ git clone https://github.com/tricksterproxy/trickster.git
$ cd trickster
$ make build
$ ./OPATH/trickster -origin-url http://prometheus.example.com:9090 -origin-type prometheus

The Makefile provides several targets, including:

build: build the trickster binary
docker: build a docker container for the current HEAD
clean: delete previously-built binaries and object files
test: runs unit tests
bench: runs benchmark tests
rpm: builds a Trickster RPM

3.2 - Configuring Trickster

Learn how to configure Trickster for your project.

There are 3 ways to configure Trickster, listed here in the order of evaluation.

Configuration File
Environment Variables
Command Line Arguments

Note that while the Configuration file provides a very robust number of knobs you can adjust, the ENV and CLI Args options support only basic use cases.

Internal Defaults

Internal Defaults are set for all configuration values, and are overridden by the configuration methods described below. All Internal Defaults are described in examples/conf/example.full.yaml comments.

Configuration File

Trickster accepts a -config /path/to/trickster.yaml command line argument to specify a custom path to a Trickster configuration file. If the provided path cannot be accessed by Trickster, it will exit with a fatal error.

When a -config parameter is not provided, Trickster will check for the presence of a config file at /etc/trickster/trickster.yaml and load it if present, or proceed with the Internal Defaults if not present.

Refer to examples/conf/example.full.yaml for full documentation on format of a configuration file.

Environment Variables

Trickster will then check for and evaluate the following Environment Variables:

TRK_ORIGIN_URL=http://prometheus.example.com:9090 - The default origin URL for proxying all http requests
TRK_ORIGIN_TYPE=prometheus - The type of supported backend server
TRK_LOG_LEVEL=INFO - Level of Logging that Trickster will output
TRK_PROXY_PORT=8480 -Listener port for the HTTP Proxy Endpoint
TRK_METRICS_PORT=8481 - Listener port for the Metrics and pprof debugging HTTP Endpoint

Command Line Arguments

Finally, Trickster will check for and evaluate the following Command Line Arguments:

-log-level INFO - Level of Logging that Trickster will output
-config /path/to/trickster.yaml - See Configuration File section above
-origin-url http://prometheus.example.com:9090 - The default origin URL for proxying all http requests
-provider prometheus - The type of supported backend server
-proxy-port 8480 - Listener port for the HTTP Proxy Endpoint
-metrics-port 8481 - Listener port for the Metrics and pprof debugging HTTP Endpoint

Configuration Validation

Trickster can validate a configuration file by running trickster -validate-config -config /path/to/config. Trickster will load the configuration and exit with the validation result, without running the configuration.

Reloading the Configuration

Trickster can gracefully reload the configuration file from disk without impacting the uptime and responsiveness of the the application.

Trickster provides 2 ways to reload the Trickster configuration: by requesting an HTTP endpoint, or by sending a SIGHUP (e.g., kill -1 $TRICKSTER_PID) to the Trickster process. In both cases, the underlying running Configuration File must have been modified such that the last modified time of the file is different than from when it was previously loaded.

Config Reload via SIGHUP

Once you have made the desired modifications to your config file, send a SIGHUP to the Trickster process by running kill -1 $TRICKSTER_PID. The Trickster log will indicate whether the reload attempt was successful or not.

Config Reload via HTTP Endpoint

Trickster provides an HTTP Endpoint for viewing the running Configuration, as well as requesting a configuration reload.

The reload endpoint is configured by default to listen on address 127.0.0.1 and port 8484, at /trickster/config/reload. These values can be customized, as demonstrated in the example.full.yaml The examples in this section will assume the defaults. Set the port to -1 to disable the reload HTTP interface altogether.

To reload the config, simply make a GET request to the reload endpoint. If the underlying configuration file has changed, the configuration will be reloaded, and the caller will receive a success response. If the underlying file has not changed, the caller will receive an unsuccessful response, and reloading will be disabled for the duration of the Reload Rate Limiter. By default, this is 3 seconds, but can be customized as demonstrated in the example config file. The Reload Rate Limiter applies to the HTTP interface only, and not SIGHUP.

If an HTTP listener must spin down (e.g., the listen port is changed in the refreshed config), the old listener will remain alive for a period of time to allow existing connections to organically finish. This period is called the Drain Timeout and is configurable. Trickster uses 30 seconds by default. The Drain Timeout also applies to old log files, in the event that a new log filename has been provided.

View the Running Configuration

Trickster also provides a http://127.0.0.1:8484/trickster/config endpoint, which returns the yaml output of the currently-running Trickster configuration. The YAML-formatted configuration will include all defaults populated, overlaid with any configuration file settings, command-line arguments and or applicable environment variables. This read-only interface is also available via the metrics endpoint, in the event that the reload endpoint has been disabled. This path is configurable as demonstrated in the example config file.

3.3 - Where to Place Trickster

Choose the best Trickster placement for your project.

Depending upon the size of your existing or planned deployment, there are several placement configurations available. These designs are suggestions based on common usage, and you may find alternative or hybrid placement configurations that make the most sense for your situation, based on the activity of your Dashboard and TSDB instance(s).

Single “Everything”

Diagram of one optional dashboard endpoint, one Trickster endpoint, and one HTTP or TSDB endpoint.

Single “Everything” is the most common placement model. In this configuration, you have one optional dashboard endpoint, one Trickster endpoint, and one HTTP or TSDB endpoint. Behind each endpoint, you may have a single instance or a cluster. Each component is only aware of the other component’s endpoint exposure and not the underlying configuration. This configuration represents a one-for-one-for-one deployment of your Dashboard, Origin, and Trickster endpoints.

Multiple Backends

Diagram of one dashboard endpoint, one Trickster endpoint, and multiple TSDB and/or HTTP endpoints.

In a Multiple Backend placement, you have one dashboard endpoint, one Trickster endpoint, and multiple TSDB and/or HTTP endpoints. Trickster is aware of each upstream endpoint and treats each as a unique backend to which it proxies and caches data independently from the others. Trickster routes a request to a specific backend based on Host Header or URL Path in the client request.

This setup may benefit situations where you have one ore more a static file server origins serving HTML, CSS and JavaScript assets and/or one or more API endpoints, all supporting a common platform.

For Time Series Dashboard acceleration, this is a good configuration to use when you have a single dashboard that displays data about multiple redundant clusters (each with its own TSDB), or when you have a single dashboard representing information about many different kinds of systems. For example, if you operate a “Dashboard as a Service” solution under which many teams use your Dashboard system by designing their own dashboard screens and bringing their own databases, a single Trickster endpoint can be used to accelerate dashboards for all of your customers.

You will need to configure each Trickster-to-TSDB mapping separately in your dashboard application as a separately named TSDB data source. Refer to the multi-origin documentation for configuring multi-origin support in Trickster and Grafana.

In this configuration, be aware that the default ‘memory’ cache may be underpowered depending on the number of customers, as well as the size and number of queries that need to be cached by each customer. Refer to the caches document to select and configure the caching layers as needed to meet your specific situation.

Multi-Trickster

Diagram of one dashboard endpoint, multiple Trickster endpoints, and multiple TSDB or HTTP endpoints, with each Trickster Endpoint having a one-to-one mapping to a TSDB/HTTP Endpoint as a pair.

In a Multi-Trickster configuration, you have one dashboard endpoint, multiple Trickster endpoints, and multiple TSDB or HTTP endpoints, with each Trickster Endpoint having a one-to-one mapping to a TSDB/HTTP Endpoint as a pair. This is a good design if Multiple Backends is not performant enough for the amount of activity associated with your solution (e.g., you need more Tricksters). If the Dashboard system owner is different from the TSDB system owner, either party could own and operate the Trickster instance.

3.4 - Deployment

Learn how to deploy Trickster for your project.

Docker

$ docker run --name trickster -d [-v /path/to/trickster.yaml:/etc/trickster/trickster.yaml] -p 0.0.0.0:9090:9090 trickstercache/trickster:latest

Kubernetes, Helm, RBAC

For helpful example files, see trickster/deploy.

If you want to use Helm and kubernetes rbac, use the following install steps in the deploy/helm directory.

Bootstrap Local Kubernetes-Helm Dev

Install Helm Client Version 2.9.1
```
brew install kubernetes-helm
```

Install kubectl client server 1.13.4, client version 1.13.4

curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.13.4/bin/darwin/amd64/kubectl && chmod +x kubectl && sudo mv kubectl /usr/local/bin/

Install minikube version 0.35.0

curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.23.2/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

Start minikube and enable RBAC make start-minikube or manually with --extra-config=apiserver.Authorization.Mode=RBAC --kubernetes-version=v1.8.0.
Install Tiller make bootstrap-peripherals
Wait until Tiller is running kubectl get po --namespace trickster -w
Deploy all K8 artifacts make bootstrap-trickster-dev

Deployment

Make any necessary configuration changes to deploy/helm/values.yaml or deploy/helm/template/configmap.yaml
Set your kubectl context to your target cluster kubectl config use-context <context>
Make sure Tiller is running kubectl get po --namespace trickster -w
Run deployment script ./deploy from within deploy/helm

Kubernetes

For helpful kubernetes deployment example files, see trickster/deploy/kube.

Bootstrap Local Kubernetes Dev

Install kubectl client server 1.8.0, client version 1.8.0

brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/e4b03ca8689987364852d645207be16a1ec1b349/Formula/kubernetes-cli.rb
brew pin kubernetes-cli

Install minikube version 0.25.0

brew cask install https://raw.githubusercontent.com/caskroom/homebrew-cask/903f1507e1aeea7fc826c6520a8403b4076ed6f4/Casks/minikube.rb

Start minikube make start-minikube or manually with minikube start.
Deploy all K8 artifacts make bootstrap-trickster-dev

Deployment

Make any necessary configuration changes to deploy/kube/configmap.yaml
Set your kubectl context to your target cluster kubectl config use-context <context>
Run deployment script ./deploy from within deploy/kube

Local Binary

Binary Dev

Use parent directory and run make, then ./trickster [-config <path>]

4 - Queries

The following articles cover queries with Trickster.

4.1 - ClickHouse Support

Accelerate ClickHouse queries.

Trickster will accelerate ClickHouse queries that return time series data normally visualized on a dashboard. Acceleration works by using the Time Series Delta Proxy Cache to minimize the number and time range of queries to the upstream ClickHouse server.

Scope of Support

Trickster is tested with the ClickHouse DataSource Plugin for Grafana v1.9.3 by Vertamedia, and supports acceleration of queries constructed by this plugin using the plugin’s built-in $timeSeries macro. Trickster also supports several other query formats that return “time series like” data.

Because ClickHouse does not provide a golang-based query parser, Trickster uses custom parsing code on the incoming ClickHouse query to deconstruct its components, determine if it is cacheable and, if so, what elements are factored into the cache key derivation. Trickster also determines the requested time range and step based on the provided absolute values, in order to normalize the query before hashing the cache key.

If you find query or response structures that are not yet supported, or providing inconsistent or unexpected results, we’d love for you to report those. We also always welcome any contributions around this functionality.

To constitute a cacheable query, the first column expression in the main or any subquery must be in one of two specific forms in order to determine the timestamp column and step:

Grafana Plugin Format

SELECT intDiv(toUInt32(time_col, 60) * 60) [* 1000] [as] [alias]

This is the approach used by the Grafana plugin. The time_col and/or alias is used to determine the requested time range from the WHERE or PREWHERE clause of the query. The argument to the ClickHouse intDiv function is the step value in seconds, since the toUInt32 function on a datetime column returns the Unix epoch seconds.

ClickHouse Time Grouping Function

SELECT toStartOf[Period](time_col) [as] [alias]

This is the approach that uses the following optimized ClickHouse functions to group timeseries queries:

toStartOfMinute
toStartOfFiveMinute
toStartOfTenMinutes
toStartOfFifteenMinutes
toStartOfHour
toDate

Again the time_col and/or alias is used to determine the request time range from the WHERE or PREWHERE clause, and the step is derived from the function name.

Determining the requested time range

Once the time column (or alias) and step are derived, Trickster parses each WHERE or PREWHERE clause to find comparison operations that mark the requested time range. To be cacheable, the WHERE clause must contain either a [timecol|alias] BETWEEN phrase or a [time_col|alias] >[=] phrase. The BETWEEN or >= arguments must be a parsable ClickHouse string date in the form 2006-01-02 15:04:05, a a ten digit integer representing epoch seconds, or the now() ClickHouse function with optional subtraction.

If a > phrase is used, a similar < phrase can be used to specify the end of the time period. If none is found, Trickster will still cache results up to the current time, but future queries must also have no end time phrase, or Trickster will be unable to find the correct cache key.

Examples of cacheable time range WHERE clauses:

WHERE t >= "2020-10-15 00:00:00" and t <= "2020-10-16 12:00:00"
WHERE t >= "2020-10-15 12:00:00" and t < now() - 60 * 60
WHERE datetime BETWEEN 1574686300 AND 1574689900

Note that these values can be wrapped in the ClickHouse toDateTime function, but ClickHouse will make that conversion implicitly and it is not required. All string times are assumed to be UTC.

Normalization and “Fast Forwarding”

Trickster will always normalize the calculated time range to fit the step size, so small variations in the time range will still result in actual queries for the entire time “bucket”. In addition, Trickster will not cache the results for the portion of the query that is still active – i.e., within the current bucket or within the configured backfill tolerance setting (whichever is greater)

4.2 - InfluxDB Support

Accelerate InfluxDB queries.

Trickster provides support for accelerating InfluxDB queries that return time series data normally visualized on a dashboard. Acceleration works by using the Time Series Delta Proxy Cache to minimize the number and time range of queries to the upstream InfluxDB server.

Scope of Support

Trickster is tested with the built-in InfluxDB DataSource Plugin for Grafana v5.0.0.

Trickster uses InfluxDB-provided packages to parse and normalize queries for caching and acceleration. If you find query or response structures that are not yet supported, or providing inconsistent or unexpected results, we’d love for you to report those so we can further improve our InfluxDB support.

Trickster supports integrations with InfluxDB 1.x and 2.0, however, the Flux language is not currently supported.

4.3 - Per-Query Time Series Instructions

Specify instructions for each query.

Beginning with Trickster v1.1, certain features like Fast Forward can be toggled a per-query basis, to assist with compatibility in your environment. This allows the drafters of a query to have some say over toggling these features on queries they find to have issues running through Trickster. This is done by adding directives via query comments. For example, in Prometheus, you can end any query with # any comment following a hashtag, so you can place the per-query instructions there.

Supported Per-Query Instructions

Fast Forward Disable

To disable fast forward, use the instruction trickster-fast-forward. You can only use this feature with Prometheus as other time series do not currently implement Fast Forward. Use as in the following example:

go_goroutines{job="trickster"}  # trickster-fast-forward:off

Note

You can only use trickster-fast-forward to disable fast forward. A value of on will have no effect.

Backfill Tolerance

To set the backfill tolerance, use the instruction trickster-backfill-tolerance. Trickster supports setting the backfill tolerance for all time series backends. Use as in the following example:

SELECT time, count(*) FROM table  # trickster-backfill-tolerance:120

Note

Notes: This overrides the backfill tolerance value for this query by the specified value (in seconds). Only integers are accepted.

4.4 - Prometheus Support

Trickster supports accelerating Prometheus.

Trickster fully supports accelerating Prometheus, which we consider our First Class backend provider. They work great together, so you should give it a try!

Most configuration options that affect Prometheus reside in the main Backend config, since they generally apply to all TSDB providers alike.

We offer one custom configuration for Prometheus, which is the ability to inject labels, on a per-backend basis to the Prometheus response before it is returned to the caller.

Injecting Labels

Here is the basic configuration for adding labels:

backends:
  prom-1a:
    provider: prometheus
    origin_url: http://prometheus-us-east-1a:9090
    prometheus:
      labels:
        datacenter: us-east-1a

  prom-1b:
    provider: prometheus
    origin_url: http://prometheus-us-east-1b:9090
    prometheus:
      labels:
        datacenter: us-east-1b

5 - Paths

What can your user do with your project?

5.1 - Customizing HTTP Path Behavior

Customize request and response behavior.

Trickster supports, via configuration, customizing the upstream request and downstream response behavior on a per-Path, per-Backend basis, by providing a paths configuration section for each backend configuration. Here are the basic capabilities for customizing Path behavior:

Modify client request headers prior to contacting the origin while proxying
Modify origin response headers prior to processing the response object in Trickster and delivering to the client
Modify the response code and body
Limit the scope of a path by HTTP Method
Select the HTTP Handler for the path (proxy, proxycache or a published provider-specific handler)
Select which HTTP Headers, URL Parameters and other client request characteristics will be used to derive the Cache Key under which Trickster stores the object.
Disable Metrics Reporting for the path

Path Matching Scope

Paths are matchable as exact or prefix

The default match is exact, meaning the client’s requested URL Path must be an exact match to the configured path in order to match and be handled by a given Path Config. For example a request to /foo/bar will not match an exact Path Config for /foo.

A prefix match will match any client-requested path to the Path Config with the longest prefix match. A prefix match Path Config to /foo will match /foo/bar as well as /foobar and /food. A basic string match is used to evaluate the incoming URL path, so it is recommended to consider finishing paths with a trailing /, like /foo/ in Path Configurations, if needed to avoid any unintentional matches.

Method Matching Scope

The methods section of a Path Config takes a string array of HTTP Methods that are routed through this Path Config. You can provide [ '*' ] to route all methods for this path.

Suggested Use Cases

Redirect a path by configuring Trickster to respond with a 302 response code and a Location header
Issue a blanket 401 Unauthorized code and custom response body to all requests for a given path.
Adjust Cache Control headers in either direction
Affix an Authorization header to requests proxied out by Trickster.
Control which paths are cached by Trickster, and which ones are simply proxied.

Request Rewriters

You can configure paths send inbound requests through a request rewriter that can modify any aspect of the inbound request (method, url, headers, etc.), before being processed by the path route. This means, when the path route inspects the request, it will have already been modified by the rewriter. Provide a rewriter with the req_rewriter_name config. It must map to a named/configured request rewriter (see request rewriters for more info). Note, you can also send requests through a rewriter at the backend level. If both are configured, backend-level rewriters are executed before path rewriters are.

request_rewriters:
  # this example request rewriter adds an additional header to the request
  # you can include as many instructions in rewriter as required
  example:
    instructions:
      - [ header, set, Example-Header-Name, Example Value ]

backends:
  default:
    provider: rpc
    origin_url: 'http://example.com'
    paths:
      root:
        path: /
        req_rewriter_name: example

Header and Query Parameter Behavior

In addition to running the request through a named rewriter, it is currently possible to make similar changes to the request with legacy path features that are described in this section. Note that these are likely to be deprecated in a future Trickster release, in favor of the more versatile named rewriters described above, which accomplish the same thing. Currently, if both a named rewriter and legacy path-based rewriting configs are defined for a given path, the named rewriter will be executed first.

Basics

You can specify request query parameters, as well as request and response headers, to be Set, Appended or Removed.

Setting

To Set a header or parameter means to insert if non-existent, or fully replace if pre-existing. To set a header, provide the header name and value you wish to set in the Path Config request_params, request_headers or response_headers sections, in the format of 'Header-or-Parameter-Name' = 'Value'.

As an example, if the client request provides a Cache-Control: no-store header, a Path Config with a header ‘set’ directive for 'Cache-Control' = 'no-transform' will replace the no-store entirely with a no-transform; client requests that have no Cache-Control header that are routed through this Path will have the Trickster-configured header injected outright. The same logic applies to query parameters.

Appending

Appending a means inserting the header or parameter if it doesn’t exist, or appending the configured value(s) into a pre-existing header with the given name. To indicate an append behavior (as opposed to set), prefix the header or parameter name with a ‘+’ in the Path Config.

Example: if the client request provides a token=SomeHash parameter and the Path Config includes the parameter '+token' = 'ProxyHash', the effective parameter when forwarding the request to the origin will be token=SomeHash&token=ProxyHash.

Removing

Removing a header or parameter means to strip it from the HTTP Request or Response when present. To do so, prefix the header/parameter name with ‘-’, for example, -Cache-control: none. When removing headers, a value is required to be provided in order to conform to YAML specification; this value, however, is ineffectual. Note that there is currently no ability to remove a specific header value from a specific header - only the entire removal header. Consider setting the header value outright as described above, to strip any unwanted values.

Response Header Timing

Response Header injections occur as the object is received from the origin and before Trickster handles the object, meaning any caching response headers injected by Trickster will also be used by Trickster immediately to handle caching policies internally. This allows users to override cache controls from upstream systems if necessary to alter the actual caching behavior inside of Trickster. For example, InfluxDB sends down a Cache-Control: No-Cache header, which is fine for the user’s browser, but Trickster needs to ignore this header in order to accelerate InfluxDB; so the default Path Configs for InfluxDB actually removes this header.

Cache Key Components

By default, Trickster will use the HTTP Method, URL Path and any Authorization header to derive its Cache Key. In a Path Config, you may specify any additional HTTP headers and URL Parameters to be used for cache key derivation, as well as information in the Request Body.

Using Request Body Fields in Cache Key Hashing

Trickster supports the parsing of the HTTP Request body for the purpose of deriving the Cache Key for a cacheable object. Note that body parsing requires reading the entire request body into memory and parsing it before operating on the object. This will result in slightly higher resource utilization and latency, depending upon the size of the client request body.

Body parsing is supported when the request’s HTTP method is POST, PUT or PATCH, and the request Content-Type is either application/x-www-form-urlencoded, multipart/form-data, or application/json.

In a Path Config, provide the cache_key_form_fields setting with a list of form field names to include when hashing the cache key.

Trickster supports parsing of the Request body as a JSON document, including documents that are multiple levels deep, using a basic pathing convention of forward slashes, to indicate the path to a field that should be included in the cache key. Take the following JSON document:

{
    "requestType": "query",
    "query": {
        "table": "movies",
        "fields": "eidr,title",
        "filter": "year=1979"
    }
}

To include the requestType, table, fields, and filter fields from this document when hashing the cache key, you can provide the following setting in a Path Configuration:

cache_key_form_fields = [ 'requestType', 'query/table', 'query/fields', 'query/filter' ]

Example Reverse Proxy Cache Config with Path Customizations

backends:
  default:
    provider: rpc
    paths:
      # root path '/'. Paths must be uniquely named but the
      # name is otherwise unimportant
      root:
        path: / # each path must be unique for the backend
        methods: [ '*' ] # All HTTP methods applicable to this config
        match_type: prefix # matches any path under '/'
        handler: proxy # proxy only, no caching (this is the default)
        # modify the query params en route to the origin; this adds authToken=secret_string
        request_params:
          authToken: secret string
        # When a user requests a path matching this route, Trickster will
        # inject these headers into the request before contacting the Origin
        request_headers:
          Cache-Control: No-Transform
        # inject these headers into the response from the Origin
        # before replying to the client
        response_headers:
          Expires: '-1'
      images:
        path: /images/
        methods:
          - GET
          - HEAD
        handler: proxycache # Trickster will cache the images directory
        match_type: prefix
        response_headers:
          Cache-Control: max-age=2592000 # cache for 30 days
      # but only cache this rotating image for 30 seconds
      images_rotating:
        path: /images/rotating.jpg
        methods:
          - GET
        handler: proxycache
        match_type: exact
        response_headers:
          Cache-Control: max-age=30
          '-Expires': ''
      # redirect this sunsetted feature to a discontinued message
      redirect:
        path: /blog
        methods:
          - '*'
        handler: localresponse
        match_type: prefix
        response_code: 302
        response_headers:
          Location: /discontinued
      # cache this API endpoint, keying on the query parameter
      api:
        path: /api/
        methods:
          - GET
          - HEAD
        handler: proxycache
        match_type: prefix
        cache_key_params:
          - query
      # same API endpoint, different HTTP methods to route against, which are denied
      api-deny:
        path: /api/
        methods:
          - POST
          - PUT
          - PATCH
          - DELETE
          - OPTIONS
          - CONNECT
        handler: localresponse
        match_type: prefix
        response_code: 401
        response_body: this is a read-only api endpoint
      # cache the query endpoint, permitting GET, HEAD, POST
      api-query:
        path: /api/query/
        methods:
          - GET
          - HEAD
          - POST
        handler: proxycache
        match_type: prefix
        cache_key_params:
          - query # for GET/HEAD
        cache_key_form_fields:
          - query # for POST

Modifying Behavior of Time Series Backend

Each of the Time Series Providers supported in Trickster comes with its own custom handlers and pre-defined Path Configs that are registered with the HTTP Router when Trickster starts up.

For example, when Trickster is configured to accelerate Prometheus, pre-defined Path Configs are registered to control how requests to /api/v1/query work differently from requests to /api/v1/query_range. For example, the /ap1/v1/query Path Config uses the query and time URL query parameters when creating the cache key, and is routed through the Object Proxy Cache; while the /api/v1/query_range Path Config uses the query, start, end and step parameters, and is routed through the Time Series Delta Proxy Cache.

In the Trickster config file, you can add your own Path Configs to your time series backend, as well override individual settings for any of the pre-defined Path Configs, and those custom settings will be applied at startup.

To know what configs you’d like to add or modify, take a look at the Trickster source code and examine the pre-definitions for the selected Backend Provider. Each supported Provider’s handlers and default Path Configs can be viewed under /pkg/backends/<provider>/routes.go. These files are in a standard format that are quite human-readable, even for a non-coder, so don’t be too intimidated. If you can understand Path Configs as YAML, you can understand them as Go code.

Examples of customizing Path Configs for Providers with Pre-Definitions:

backends:
  default:
    provider: prometheus
    paths:
      # route /api/v1/label* (including /labels/*)
      # through Proxy instead of ProxyCache as pre-defined
      label:
        path: /api/v1/label
        methods:
          - GET
        match_type: prefix
        handler: proxy
      # route fictional new /api/v1/coffee to ProxyCache
      series_range:
        path: /api/v1/coffee
        methods:
          - GET
        match_type: prefix
        handler: proxycache
        cache_key_params:
          - beans
      # block /api/v1/admin/ from being reachable via Trickster
      admin:
        path: /api/v1/admin/
        methods:
          - GET
          - POST
          - PUT
          - HEAD
          - DELETE
          - OPTIONS
        match_type: prefix
        handler: localresponse
        response_code: 401
        response_body: No soup for you!
        no_metrics: true

5.2 - Request Rewriters

Modify incoming HTTP requests.

A Request Rewriter is a named series of instructions that modifies any part of the incoming HTTP request. Request Rewriters are used in various parts of the Trickster configuration to make scoped changes. For example, a rewriter can modify the path, headers, parameters, etc. of a URL using mechanisms like search/replace, set and append.

In a configuration, request rewriters are represented as map of instructions, which themselves are represented as a list of string lists, in the following format:

request_rewriters:
  example_rewriter:
    instructions:
      - [ 'header', 'set', 'Cache-Control', 'max-age=60' ], # instruction 0
      - [ 'path', 'replace', '/cgi-bin/', '/' ],            # instruction 1
      - [ 'chain', 'exec', 'remove_accept_encoding' ]       # instruction 2

  remove_accept_encoding:
    instructions:
      - [ 'header', 'delete', 'Accept-Encoding' ] # instruction 0

In this case, any other configuration entity that supports mapping to a rewriter by name can do so with by referencing example_rewriter or remove_accept_encoding. Note that example_rewriter executes remove_accept_encoding using the chain instruction.

Where Rewriters Can Be Used

Rewriters are exposed as optional configurations for the following configuration constructs:

In a backend config, provide a req_rewriter_name to rewrite the Request using the named Request Rewriter, before it is handled by the Path route.

In a path config, provide a req_rewriter_name to rewrite the Request using the named Request Rewriter, before it is handled by the Path route.

In a rule config, provide ingress_req_rewriter_name, egress_req_rewriter_name and/or nomatch_req_rewriter_name configurations to rewrite the Request using the named Request Rewriter. The meaning of Ingress and Egress, in this case, are scoped to a Request’s traversal through the Rule in which these configuration values exist, and is unrelated to the wire traversal of the request. For ingress and egress, the rewriter is executed before or after, respectively, it is handled by the Rule (including any modifications made by a matching rule case). The No-Match Request Rewriter is only executed when the request does not match to any defined case.

In a Rule’s case configurations, provide req_rewriter_name. If there is a Rule Case match when executing the Rule against the incoming Request, the configured rewriter will execute on the Request before returning control back to the Rule to execute any configured egress request rewriter and hand the Request off to the next route.

In a Request Rewriter instruction using the chain instruction type. Provide the Rewriter Name as the third argument in the instruction as follows: [ 'chain', 'exec', '$rewriter_name']. See more information below.

Instruction Construction Guide

header rewriters modify a header with a specific name and support the following operations.

header set

header set will set a specific header to a specific value.

['header', 'set', 'Header-Name', 'header value']

header replace

header replace performs a search/replace function on the Header value of the provided Name

['header', 'replace', 'Header-Name', 'search value', 'replacement value']

header delete

header delete removes, if present, the Header of the provided Name

['header', 'delete', 'Header-Name']

header append

header append appends an additional value to Header in the format of value1[, value2=subvalue, ...]

['header', 'append', 'Header-Name', 'additional header value']

path

path rewriters modify the full or partial path and support the following operations.

path set

['path', 'set', '/new/path'] sets the entire request path to /new/path

['path', 'set', 'awesome', 0 ] sets the first part of the path (zero-indexed, split on /) to ‘awesome’. For example, /new/path => /awesome/path

path replace

['path', 'replace', 'search', 'replacement'] search replaces against the entire path scalar

['path', 'replace', 'search', 'replacement', 1] search replaces against the second part of the path; For example /my/example-search/path => /my/example-replacement/path

param

param rewriters modify the URL Query Parameter of the specified name, and support the following operations

param set

param set sets the URL Query Parameter of the provided name to the provided value

['param', 'set', 'paramName', 'new param value']

param replace

param replace performs a search/replace function on the URL Query Parameter value of the provided name

['param', 'replace', 'paramName', 'search value', 'replacement value']

param delete

param delete removes, if present, the URL Query Parameter of the provided name

['param', 'delete', 'paramName']

param append

param append appends the provided name and value to the URL Query Parameters regardless of whether a parameter name already exists with the same or different value.

['param', 'append', 'paramName', 'additional param value']

params

params rewriters update the entire URL parameter collection as a URL-encoded scalar string.

params set

params set will replace the Request’s entire URL Query Parameter encoded string with the provided value. The provided value is assumed to already be URL-encoded.

['params', 'set', 'param1=value1¶m2=value2']

To clear the URL parameters, use ['params', 'set', '']

params replace

params replace performs a search/replace operation on the url-encoded query string. The search and replacement values are assumed to already be URL-encoded.

['params', 'replace', 'search value', 'replacement value']

method

method rewriters update the HTTP request’s method

method set

method set sets the Request’s HTTP Method to the provided value. This value is not currently validated against known HTTP Method. The instruction should be configured to include a known and properly-formatted (all caps) HTTP Method.

['method', 'set', 'GET']

host

host rewriters update the HTTP Request’s host - defined as the hostname:port, as expressed in the Request’s Host header.

host set

host set sets the Request’s Host header to the provided value.

['host', 'set', 'my.new.hostname:9999']

['host', 'set', 'my.new.hostname'] Trickster will assume a standard source port 80/443 depending upon the URL scheme

host replace

host replace performs a search/replace operation on the Request’s Host Header.

['host', 'replace', ':8480', '']

['host', 'replace', ':443', ':8443']

['host', 'replace', 'example.com', 'trickstercache.io']

hostname

hostname rewriters update the HTTP Request’s hostname, without respect to the port.

hostname set

hostname set sets the Request’s hostname, without changing the port.

['hostname', 'set', 'my.new.hostname']

hostname replace

hostname replace performs a search/replace on the Request’s hostname, without respect to the port.

['hostname', 'replace', 'example.com', 'trickstercache.io']

port

port rewriters update the HTTP Request’s port, without respect to the hostname.

port set

port set sets the Request’s port.

['port', 'set', '8480']

port replace

port replace performs a search/replace on the port, as if it was a string. The search and replacement values must be integers and are not validated.

['port', 'replace', '8480', '']

port delete

port delete removes the port from the Request. This will cause the port to be assumed based on the URL scheme.

['port', 'delete']

scheme

scheme rewriters update the HTTP Request’s scheme (http or https).

scheme set

scheme set sets the scheme of the HTTP Request URL. This must be http or https in lowercase, and is not validated.

['scheme', 'set', 'https']

chain

chain rewriters do not directly rewrite the request, but execute other rewriters' instructions before proceeding with the current rewriter’s remaining instructions (if any). You can create a rewriter with some reusable functionality and include that in other rewriters with a chain exec. Or you can define a rewriter that is just a list of other chained rewriters. Note that there is currently no validation of the configuration to prevent infinite cyclic chained rewriter calls. There is, however, a hard limit of 32 chained rules before a request will stop rewriting and proceed with being served by the backend.

chain exec executes the supplied rewriter name. Trickster will error at startup if the rewriter name is invalid. An example is provided in the sample yaml config at the top of this article.

['chain', 'exec', 'example_rewriter']

6 - Caching

The following articles exlpain how Trickster leverages caching.

6.1 - Cache Options

Trickster supports a number of caches.

Supported Caches

Trickster supports several cache types:

In-Memory (default)
Filesystem
bbolt
BadgerDB
Redis (basic, cluster, and sentinel)

The sample configuration, trickster/examples/conf/example.full.yaml, demonstrates how to select and configure a particular cache type, as well as how to configure generic cache configurations such as Retention Policy.

In-Memory

In-Memory Cache is the default type that Trickster will implement if none of the other cache types are configured. The In-Memory cache utilizes a Golang sync.Map object for caching, which ensures atomic reads/writes against the cache with no possibility of data collisions. This option is good for both development environments and most smaller dashboard deployments.

When running Trickster in a Docker container, ensure your node hosting the container has enough memory available to accommodate the cache size of your footprint, or your container may be shut down by Docker with an Out of Memory error (#137). Similarly, when orchestrating with Kubernetes, set resource allocations accordingly.

Filesystem

The Filesystem Cache is a popular option when you have larger dashboard setup (e.g., many different dashboards with many varying queries, Dashboard as a Service for several teams running their own Prometheus instances, etc.) that requires more storage space than you wish to accommodate in RAM. A Filesystem Cache configuration keeps the Trickster RAM footprint small, and is generally comparable in performance to In-Memory. Trickster performance can be degraded when using the Filesystem Cache if disk i/o becomes a bottleneck (e.g., many concurrent dashboard users).

The default Filesystem Cache path is /tmp/trickster. The sample configuration demonstrates how to specify a custom cache path. Ensure that the user account running Trickster has read/write access to the custom directory or the application will exit on startup upon testing filesystem access. All users generally have access to /tmp so there is no concern about permissions in the default case.

bbolt

The BoltDB Cache is a popular key/value store, created by Ben Johnson. CoreOS’s bbolt fork is the version implemented in Trickster. A bbolt store is a filesystem-based solution that stores the entire database in a single file. Trickster, by default, creates the database at trickster.db and uses a bucket name of ‘trickster’ for storing key/value data. See the example config file for details on customizing this aspect of your Trickster deployment. The same guidance about filesystem permissions described in the Filesystem Cache section above apply to a bbolt Cache.

BadgerDB

BadgerDB works similarly to bbolt, in that it is a filesystem-based key/value datastore. BadgerDB provides its own native object lifecycle management (TTL) and other additional features that distinguish it from bbolt. See the configuration for more info on using BadgerDB with Trickster.

Redis

Note: Trickster does not come with a Redis server. You must provide a pre-existing Redis endpoint for Trickster to use.

Redis is a good option for larger dashboard setups that also have heavy user traffic, where you might see degraded performance with a Filesystem Cache. This allows Trickster to scale better than a Filesystem Cache, but you will need to provide your own Redis instance at which to point your Trickster instance. The default Redis endpoint is redis:6379, and should work for most docker and kube deployments with containers or services named redis. The sample configuration demonstrates how to customize the Redis endpoint. In addition to supporting TCP endpoints, Trickster supports Unix sockets for Trickster and Redis running on the same VM or bare-metal host.

Ensure that your Redis instance is located close to your Trickster instance in order to minimize additional roundtrip latency.

In addition to basic Redis, Trickster also supports Redis Cluster and Redis Sentinel. Refer to the sample configuration for customizing the Redis client type.

Purging the Cache

Cache purges should not be necessary, but in the event that you wish to do so, the following steps should be followed based upon your selected Cache Type.

A future release will provide a mechanism to fully purge the cache (regardless of the underlying cache type) without stopping a running Trickster instance.

Purging In-Memory Cache

Since this cache type runs inside the virtual memory allocated to the Trickster process, bouncing the Trickster process or container will effectively purge the cache.

Purging Filesystem Cache

To completely purge a Filesystem-based Cache, you will need to:

Docker/Kube: delete the Trickster container (or mounted volume) and run a new one
Metal/VM: Stop the Trickster process and manually run rm -rf /tmp/trickster (or your custom-configured directory).

Purging Redis Cache

Connect to your Redis instance and issue a FLUSH command. Note that if your Redis instance supports more applications than Trickster, a FLUSH will clear the cache for all dependent applications.

Purging bbolt Cache

Stop the Trickster process and delete the configured bbolt file.

Purging BadgerDB Cache

Stop the Trickster process and delete the configured BadgerDB path.

Cache Status

Trickster reports several cache statuses in metrics, logs, and tracing, which are listed and described in the table below.

Status	Description
kmiss	The requested object was not in cache and was fetched from the origin
rmiss	Object is in cache, but the specific data range requested (timestamps or byte ranges) was not
hit	The object was fully cached and served from cache to the client
phit	The object was cached for some of the data requested, but not all
nchit	The response was served from the Negative Cache
rhit	The object was served from cache to the client, after being revalidated for freshness against the origin
proxy-only	The request was proxied 1:1 to the origin and not cached
proxy-error	The upstream request needed to fulfill an associated client request returned an error

6.2 - Byte Range Request Support

Trickster’s HTTP Reverse Proxy Cache offers best-in-class acceleration and caching of Byte Range Requests.

Much like its Time Series Delta Proxy Cache, Trickster’s Reverse Proxy Cache will determine what ranges are cached, and only request from the origin any uncached ranges needed to service the client request, reconstituting the ranges within the cache object. This ensures minimal response time for all Range requests.

In addition to supporting requests with a single Range (Range: bytes=0-5) Trickster also supports Multipart Range Requests (Range: bytes=0-5, 10-20).

Fronting Origins That Do Not Support Multipart Range Requests

In the event that an upstream origin supports serving a single Range, but does not support serving Multipart Range Requests, which is quite common, Trickster can transparently enable that support on behalf of the origin. To do so, Trickster offers a unique feature called Upstream Range Dearticulation, that will separate any ranges needed from the origin into individual, parallel HTTP requests, which are reconstituted by Trickster. This behavior can be enabled for any origin that only supports serving a single Range, by setting the origin configuration value dearticulate_upstream_ranges = true, as in this example:

backends:
  default:
    provider: reverseproxycache
    origin_url: 'http://example.com/'
    dearticulate_upstream_ranges: true

If you know that your clients will be making Range requests (even if they are not Multipart), check to ensure the configured origin supports Multipart Range requests. Use curl to request any static object from the origin, for which you know the size, and include a Multipart Range request; like curl -v -H 'Range: bytes=0-1, 3-4' 'http://example.com/object.js'. If the origin returns 200 OK and the entire object body, instead of 206 Partial Content and a multipart body, enable Upstream Range Dearticulation to ensure optimal performance.

This is important because a partial hit could result in multiple ranges being needed from the origin - even for a single-Range client request, depending upon what ranges are already in cache. If Upstream Range Dearticulation is disabled in this case, full objects could be unnecessarily returned from the Origin to Trickster, instead of small delta ranges, irrespective of the object’s overall size. This may or may not impact your use case.

Rule of thumb: If the origin does not support Multipart requests, enable Upstream Range Dearticulation in Trickster to compensate. Conversely, if the origin does support Multipart requests, do not enable Upstream Range Dearticulation.

Disabling Multipart Ranges to Clients

One of the great benefits of using Upstream Range Dearticulation is that it transparently enables Multipart Range support for clients, when fronting any origin that already supports serving just a single Range.

There may, however, be cases where you do not want to enable Multipart Range support for clients (since its paired Origin does not), but need Upstream Range Dearticulation to optimize Partial Hit fulfillments. For those cases, Trickster offers a setting to disable Multipart Range support for clients, while Upstream Range Dearticulation is enabled. Set multipart_ranges_disabled = true, as in the below example, and Trickster will strip Multipart Range Request headers, which will result in a 200 OK response with the full body. Client single Range requests are unaffected by this setting. This should only be set if you have a specific use case where clients should not be able to make multipart Range requests.

backends:
  default:
    provider: reverseproxycache
    origin_url: 'http://example.com/'
    dearticulate_upstream_ranges: true
    multipart_ranges_disabled: true

Partial Hit with Object Revalidation

As explained above, whenever the client makes a Range request, and only part of the Range is in the Trickster cache, Trickster will fetch the uncached Ranges from the Origin, then reconstitute and cache all of the accumulated Ranges, while also replying to the client with its requested Ranges.

In the event that a cache object returns 1) a partial hit, 2) that is no longer fresh, 3) but can be revalidated, based on a) the Origin’s provided caching directives or b) overridden by the Trickster operator’s explicit path-based Header configs; Trickster will revalidate the client’s requested-but-cached range from the origin with the appropriate revalidation headers.

In a Partial Hit with Revalidation, the revalidation request is made as a separate, parallel request to the origin alongside the uncached range request(s). If the revalidation succeeds, the cached range is merged with the newly-fetched range as if it had never expired. If the revalidation fails, the Origin will return the range needed by the client that was previously cached, or potentially the entire object - either of which are used to complete the ranges needed by the client and update the cache and caching policy for the object.

Range Miss with Object Revalidation

Trickster recognizes when an object exists in cache, but has none of the client’s requested Ranges. This is a state that lies between Cache Miss and Partial Hit, and is known as “Range Miss.” Range Misses can happen frequently on Range-requested objects.

When a Range Miss occurs against an object that also requires revalidation, Trickster will not initiate a parallel revalidation request, since none of the client’s requested Ranges are actually eligible for revalidation. Instead, Trickster will use the Response Headers returned by the Range Miss Request to perform a local revalidation of the cache object. If the object is revalidated, the new Ranges are merged with the cached Ranges before writing to cache based on the newly received Caching Policy. If the object is not revalidated, the cache object is created anew solely from the Range Miss Response.

Multiple Parts Require Revalidation

A situation can arise where there is a partial cache hit has multiple ranges that require revalidation before they can be used to satisfy the client. In these cases, Trickster will check if Upstream Range Dearticulation is enabled for the origin to determine how to resolve this condition. If Upstream Range Dearticulation is not enabled, Trickster trusts that the upstream origin will support Multipart Range Requests, and will include just the client’s needed-and-cached-but-expired ranges in the revalidation request. If Upstream Range Dearticulation is enabled, Trickster will forward, without modification, the client’s requested Ranges to the revalidation request to the origin. This behavior means Trickster currently does not support multiple parallel revalidation requests. Whenever the cache object requires revalidation, there will be only 1 revalidation request upstream, and 0 to N additional parallel upstream range requests as required to fulfill a partial hit.

If-Range Not Yet Supported

Trickster currently does not support revalidation based on If-Range request headers, for use with partial download resumptions by clients. If-Range headers are simply ignored by Trickster and passed through to the origin, which can result in unexpected behavior with the Trickster cache for that object.

We plan to provide full support for If-Range as part of Trickster 1.1 or 2.0

Mockster Byte Range

For verification of Trickster’s compatibility with Byte Range Requests (as well as Time Series data), we created a golang library and accompanying standalone application dubbed Mockster. Mockster’s Byte Range library simply prints out the Lorem ipsum ... sample text, pared down to the requested range or multipart ranges, with a few bells and whistles that allow you to customize its response for unit testing purposes. We make extensive use of Mockster in unit testing to verify the integrity of Trickster’s output after performing operations like merging disparate range parts, extracting ranges from other ranges, or from a full body, compressing adjacent ranges into a single range in the cache, etc.

It is fairly straightforward to run or import Mockster into your own applications. For examples of using it for Unit Testing, check out /pkg/proxy/engines/objectproxycache_test.go.

6.3 - Collapsed Forwarding

Keep requests efficient with collapsed forwarding.

Collapsed Forwarding is feature common among Reverse Proxy Cache solutions like Squid, Varnish and Apache Traffic Server. It works by ensuring only a single request to the upstream origin is performed for any object on a cache miss or revalidation attempt, no matter how many users are requesting the object at the same time.

Trickster has support for two types of Collapsed Forwarding: Basic (default) and Progressive

Basic Collapsed Forwarding

Basic Collapsed Forwarding is the default functionality for Trickster, and works by waitlisting all requests for a cacheable object while a cache miss is being serviced for the object, and then serving the waitlisted requests once the cache has been populated.

The feature is further detailed in the following diagram:

Diagram of basic collapsed forwarding in Trickster

Progressive Collapsed Forwarding

Progressive Collapsed Forwarding (PCF) is an improvement upon the basic version, in that it eliminates the waitlist and serves all simultaneous requests concurrently while the object is still downloading from the server, similar to Apache Traffic Server’s “read-while-write” feature. This may be useful in low-latency applications such as DASH or HLS video delivery, since PCF minimizes Time to First Byte latency for extremely popular objects.

The feature is further detailed in the following diagram:

Diagram of progressive collapsed forwarding in Trickster

PCF for Proxy-Only Requests

Trickster provides a unique feature that implements PCF in Proxy-Only configurations, to bring the benefits of Collapsed Forwarding to HTTP Paths that are not configured to be routed through the Reverse Proxy Cache. See the Paths documentation for more info on routing.

The feature is further detailed in the following diagram:

Diagram of progressive collapsed forwarding for proxy-only requests in Trickster

How to enable Progressive Collapsed Forwarding

When configuring path configs as described in Paths Documentation you simply need to add progressive_collapsed_forwarding = true in any path config using the proxy or proxycache handlers.

Example:

origins:
  test:
    paths:
      thing1:
        path: /test_path1/
        match_type: prefix
        handler: proxycache
        progressive_collapsed_forwarding: true
      thing2:
        path: /test_path2/
        match_type: prefix
        handler: proxy
        progressive_collapsed_forwarding: true

See the example.full.yaml for more configuration examples.

How to test Progressive Collapsed Forwarding

An easy way to test PCF is to set up your favorite file server to host a large file(Lighttpd, Nginx, Apache WS, etc.), In Trickster turn on PCF for that path config and try make simultaneous requests. If the networking between your machine and Trickster has enough bandwidth you should see both streaming at the equivalent rate as the origin request.

Example:

Run a Lighttpd instance or docker container on your local machine and make a large file available to be served
Run Trickster locally
Make multiple curl requests of the same object

You should see the speed limited on the origin request by your disk IO, and your speed between Trickster limited by Memory/CPU

6.4 - Negative Caching

Cache responses to prevent system overload.

Negative Caching means to cache undesired HTTP responses for a very short period of time, in order to prevent overwhelming a system that would otherwise scale normally when desired, cacheable HTTP responses are being returned. For example, Trickster can be configured to cache 404 Not Found or 500 Internal Server Error responses for a short period of time, to ensure that a thundering herd of HTTP requests for a non-existent object, or unexpected downtime of a critical service, do not create an i/o bottleneck in your application pipeline.

Trickster supports negative caching of any status code >= 300 and < 600, on a per-Backend basis. In your Trickster configuration file, associate the desired Negative Cache Map to the desired Backend config. See the example.full.yaml, or refer to the snippet below for more information.

The Negative Cache Map must be an all-inclusive list of explicit status codes; there is currently no wildcard or status code range support for Negative Caching entries. By default, the Negative Cache Map is empty for all backend configs. The Negative Cache only applies to Cacheable Objects, and does not apply to Proxy-Only configurations.

For any response code handled by the Negative Cache, the response object’s effective cache TTL is explicitly overridden to the value of that code’s Negative Cache TTL, regardless of any response headers provided by the Backend concerning cacheability. All response headers are left in-tact and unmodified by Trickster’s Negative Cache, such that Negative Caching is transparent to the client. The X-Trickster-Result response header will indicate a response was served from the Negative Cache by providing a cache status of nchit.

Multiple negative cache configurations can be defined, and are referenced by name in the backend config. By default, a backend will use the ‘default’ Negative Cache config, which, by default is empty. The default can be easily populated in the config file, and additional configs can easily be added, as demonstrated below.

The format of a negative cache map entry is 'status_code': ttl_in_ms.

Example Negative Caching Config

negative_caches:
  default:
    '404': 3000 # cache 404 responses for 3 seconds
  foo:
    '404': 3000
    '500': 5000
    '502': 5000

backends:
  default:
    provider: rpc
    # by default will assume negative_cache_name = 'default'
  another:
    provider: rpc
    negative_cache_name: foo

6.5 - Trickster Caching Retention Policies

How long Trickster retains data

Basic HTTP Backends

Trickster will respect HTTP 1.0, 1.1 and 2.0 caching directives from both the downstream client and the upstream origin when determining object cacheability and TTL. You can override the TTL by setting a custom Cache-Control header on a per-Path Config basis.

Cache Object Evictions

If you use a Trickster-managed cache (Memory, Filesystem, bbolt), then a maximum cache size is maintained by Trickster. You can configure the maximum size in number of bytes, number of objects, or both. See the example configuration for more information.

Once the cache has reached its configured maximum size of objects or bytes, Trickster will undergo an eviction routine that removes cache objects until the size has fallen below the configured maximums. Trickster-managed caches maintain a last access time for each cache object, and utilizes a Least Recently Used (LRU) methodology when selecting objects for eviction.

Caches whose object lifetimes are not managed internally by Trickster (Redis, BadgerDB) will use their own policies and methodologies for evicting cache records.

Time Series Backends

For non-time series responses from a TSDB, Trickster will adhere to HTTP caching rules as directed by the downstream client and upstream origin.

For time series data responses, Trickster will cache as follows:

TTL Settings

TTL settings for each Backend configured in Trickster can be customized independently of each other, and separate TTL configurations are available for timeseries objects, and fast forward data. See examples/conf/example.full.yaml for more info on configuring default TTLs.

Time Series Data Retention

Separately from the TTL of a time series cache object, Trickster allows you to control the size of each timeseries object, represented as a count of maximum timestamps in the cache object, on a per origin basis. This configuration is known as the timeseries_retention_factor (TRF), and has a default of 1024. Most dashboards for most users request and display approximately 300-to-400 timestamps, so the default TRF allows users to still recall recently-displayed data from the Trickster cache for a period of time after the data has aged off of real-time views.

If you have users with a high-resolution dashboard configuration (e.g., a 24-hour view with a 1-minute step, amounting to 1440 data points per graph), then you may benefit from increasing the timeseries_retention_factor accordingly. If you use a managed cache (see caches) and increase the timeseries_retention_factor, the overall size of your cache will not change; the result will be fewer objects in cache, with the timeseries objects having a larger share of the overall cache size with more aged data.

Time Series Data Evictions

Once the TRF is reached for a time series cache object, Trickster will undergo a timestamp eviction process for the record in question. Unlike the Cache Object Eviction, which removes an object from cache completely, TRF evictions examine the data set contained in a cache object and remove timestamped data in order to reduce the object size down to the TRF.

Time Series Data Evictions apply to all cached time series data sets, regardless of whether or not the cache object lifecycle is managed by Trickster.

Trickster provides two eviction methodologies (timeseries_eviction_method) for time series data eviction: oldest (default) and lru, and is configurable per-origin.

When timeseries_eviction_method is set to oldest, Trickster maintains time series data by calculating the “oldest cacheable timestamp” value upon each request, using time.Now().Add(step * timeseries_retention_factor * -1). Any queries for data older than the oldest cacheable timestamp are intelligently offloaded to the proxy since they will never be cached, and no data that is older than the oldest cacheable timestamp will be stored in the query’s cache record.

When timeseries_eviction_method is set to lru, Trickster will not calculate an oldest cacheable timestamp, but rather maintain a last-accessed time for each timestamp in the cache object, and evict the Least-Recently-Used items in order to maintain the cache size.

The advantage of the oldest methodology better cache performance, at the cost of not caching very old data. Thus, Trickster will be more performant computationally while providing a slightly lower cache hit rate. The lru methodology, since it requires accessing the cache on every request and maintaining access times for every timestamp, is computationally more expensive, but can achieve a higher cache hit rate since it permits caching data of any age, so long as it is accessed frequently enough to avoid eviction.

Most users will find the oldest methodology to meet their needs, so it is recommended to use lru only if you have a specific use case (e.g., dashboards with data from a diverse set of time ranges, where caching only relatively young data does not suffice).

7 - Origins

The following articles exlpain how Trickster works with origins.

7.1 - Rule Backend

How to write rules for Trickster’s Rule Backend.

The Rule Backend is not really a true Backend; it only routes inbound requests to other configured Backends, based on how they match against the Rule’s cases.

A Rule is a single inspection operation performed against a single component of an inbound request, which determines the Next Backend to send the request to. The Next Backend can also be a rule Backend, so as to route requests through multiple Rules before arriving at a true Backend destination.

A rule can optionally rewrite multiple portions of the request before, during and after rule matching, by using request rewriters, which allows for powerful and limitless combinations of request rewriting and routing.

Rule Parts

A rule has several required parts, as follows:

Required Rule Parts

input_source - The part of the Request the Rule inspects
input_type - The source data type
operation - The operation taken on the input source
next_route - The Backend Name indicating the default next route for the Rule if no matching cases. Not required if redirect_url is provided.
redirect_url - The fully-qualified URL to issue as a 302 redirect to the client in the default case. Not required if next_route is provided.

Optional Rule Parts

input_key - case-sensitive lookup key; required when the source is header or URL param
input_encoding - the encoding of the input, which is decoded prior to performing the operation
input_index - when > -1, the source is split into parts and the input is extracted from parts[input_index]
input-delimiter - when input_index > -1, this delimiter is used to split the source into parts, and defaults to a standard space (' ‘)
ingress_req_rewriter name - provides the name of a Request Rewriter to operate on the Request before rule execution.
egress_req_rewriter name - provides the name of a Request Rewriter to operate on the Request after rule execution.
nomatch_req_rewriter name - provides the name of a Request Rewriter to operate on the Request after rule execution if the request did not match any cases.
max_rule_executions - limits the number of rules a Request is passed through, and aborts with a 400 status code when exceeded. Default is 16.

input_source permitted values

source name	example extracted value
url	https://example.com:8480/path1/path2?param1=value
url_no_params	https://example.com:8480/path1/path2
scheme	https
host	example.com:8480
hostname	example.com
port	8480 (inferred from scheme when no port is provided)
path	/path1/path2
params	?param1=value
param	(must be used with input_key as described below)
header	(must be used with input_key as described below)

input_type permitted values and operations

type name	permitted operations
string (default)	prefix, suffix, contains, eq, md5, sha1, modulo, rmatch
num	eq, le, ge, gt, lt, modulo
bool	eq

Rule Cases

Rule cases define the possible values are able to alter the Request and change the next route.

Case Parts

Required Case Parts

matches - A string list of values applicable to this case.
next_route - The Backend Name indicating the next route for the Rule when a request matches this Case. Not required if redirect_url is provided.
redirect_url - The fully-qualified URL to issue as a 302 redirect to the client when the Request matches this Case. Not required if next_route is provided.

Optional Case Parts

req_rewriter name - provides the name of a Request Rewriter to operate on the Request when this case is matched.

Example Rule - Route Request by Basic Auth Username

In this example config, requests routed through the /example path will be compared against the rules and routed to either the Reader cluster or the Writer cluster. Curling http://trickster-host/example/path would route to the reader or writer cluster based on a provided Authorization header.

rules:
  example-user-router:
    # default route is reader cluster
    next_route: example-reader-cluster

    input_source: header
    input_key: Authorization
    input_type: string
    input_encoding: base64 # Authorization: Basic <base64string>
    input_index: 1         # Field 1 is the <base64string>
    input_delimiter: ' '   # Authorization Header field is space-delimited
    operation: prefix      # Basic Auth credentials are formatted as user:pass,
                           # so we can check if it is prefixed with $user:
    cases:
      writers:
        matches: # route johndoe and janedoe users to writer cluster
          - 'johndoe:'
          - 'janedoe:'
        next_route: example-writer-cluster

backends:
  example:
    provider: rule
    rule_name: example-user-router

  example-reader-cluster:
    provider: rpc
    origin_url: 'http://reader-cluster.example.com'

  example-writer-cluster:
    provider: rpc
    origin_url: 'http://writer-cluster.example.com'
    path_routing_disabled: true  # restrict routing to this backend via rule only, so
                                 # users cannot directly access via /example-writer-cluster/

Example Rule - Route Request by Path Regex

In this example config, requests routed through the /example path will be compared against the rules and routed to either the Reader cluster or the Writer cluster. Curling http://trickster-host/example/reader and http://trickster-host/example/writer would route to the reader or writer cluster by matching the path.

rules:
  example-user-router:
    # default route is reader cluster
    next_route: example-reader-cluster

    input_source: path
    input_type: string
    operation: rmatch      # perform regex match against the path to see if it matches 'writer
    operation_arg: '^.*\/writer.*$'
    cases:
      rmatch-true:
        matches: [ 'true' ] # rmatch returns true when the input matches the regex; update next_route
        next_route: example-writer-cluster

backends:
  example:
    provider: rule
    rule_name: example-user-router

  example-reader-cluster:
    provider: rpc
    origin_url: 'http://reader-cluster.example.com'

  example-writer-cluster:
    provider: rpc
    origin_url: 'http://writer-cluster.example.com'
    path_routing_disabled: true  # restrict routing to this backend via rule only, so
                                 # users cannot directly access via /example-writer-cluster/

7.2 - Supported Providers

Find out which providers Trickster supports.

Trickster currently supports the following providers:

Generic HTTP Reverse Proxy Cache

Trickster operates as a fully-featured and highly-customizable reverse proxy cache, designed to accelerate and scale upstream endpoints like API services and other simple http services. Specify 'reverseproxycache' or just 'rpc' as the Provider when configuring Trickster.

Time Series Databases

Prometheus

Trickster fully supports the Prometheus HTTP API (v1). Specify 'prometheus' as the Provider when configuring Trickster. Trickster supports label injection for Prometheus.

InfluxDB

Trickster supports for InfluxDB. Specify 'influxdb' as the Provider when configuring Trickster.

See the InfluxDB Support Document for more information.

ClickHouse

Trickster supports accelerating ClickHouse time series. Specify 'clickhouse' as the Provider when configuring Trickster.

See the ClickHouse Support Document for more information.

Circonus IRONdb

Support has been included for the Circonus IRONdb time-series database. If Grafana is used for visualizations, the Circonus IRONdb data source plug-in for Grafana can be configured to use Trickster as its data source. All IRONdb data retrieval operations, including CAQL queries, are supported.

When configuring an IRONdb backend, specify 'irondb' as the provider in the Trickster configuration. The host value can be set directly to the address and port of an IRONdb node, but it is recommended to use the Circonus API proxy service. When using the proxy service, set the host value to the address and port of the proxy service, and set the api_path value to 'irondb'.

7.3 - Timeseries Request Sharding

This article covers the available sharding configurations.

Overview

A shard means “a small part of a whole,” and Trickster 2.0 supports the sharding of upstream HTTP requests when retrieving timeseries data. When configured for a given time series backend, Trickster will shard eligible requests by inspecting the time ranges needed from origin and subdividing them into smaller ranges that conform to the backend’s sharding configuration. Sharded requests are sent to the origin concurrently, and their responses are reconstituted back into a single dataset by Trickster after they’ve all been returned.

Mechanisms

Trickster support three main mechanisms for sharding:

Maximum Timestamps Per Shard: Trickster calculates the number of expected unique timestamps in the response by dividing the requested time range size by the step cadence, and then subdivides the time ranges so that each sharded request’s time range will return no more timestamps than the configured maximum.

Diagram of Trickster's subdivisions based on timestamps

Maximum Time Range Width Per Shard: Trickster inspects each needed time range, and subdivides them such that each sharded request’s time range duration is no larger than the configured maximum.

Diagram of Trickster's subdivisions based on time range duration

Epoch-Aligned Maximum Time Range Width Per Shard: Trickster inspects each needed time range, and subdivides them such that each sharded request’s time range duration is no larger than the configured maximum, while also ensuring that each shard’s time boundaries are aligned to the Epoch based on the configured shard step size.

Diagram of Trickster making sure that shard time boundaries are aligned to the shard step size

Configuring

Maximum Unique Timestamp Count Per Shard

In the Trickster configuration, use the shard_max_size_points configuration to shard requests by limiting the maximum number of unique timestamps in each sharded response.

backends:
  example:
    provider: prometheus
    origin_url: http://prometheus:9090
    shard_max_size_points: 10999

Maximum Time Range Width Per Shard

In the Trickster configuration, use the shard_max_size_ms configuration to shard requests by limiting the maximum width of each sharded request’s time range.

backends:
  example:
    provider: 'prometheus'
    origin_url: http://prometheus:9090
    shard_max_size_ms: 7200000

Epoch-Aligned Maximum Time Range Width Per Shard

In the Trickster configuration, use the shard_step_ms configuration to shard requests by limiting the maximum width of each sharded request’s time range, while ensuring shards align with the epoch on the configured cadence. This is useful for aligning shard boundaries with an upstream database’s partition boundaries, ensuring that sharded requests have as little partition overlap as possible.

backends:
  example:
    provider: 'prometheus'
    origin_url: http://prometheus:9090
    shard_step_ms: 7200000

shard_step_ms can be used in conjunction with shard_max_size_ms, so long as shard_max_size_ms is perfectly divisible by shard_step_ms. This combination configuration will align shards against the configured shard step, while sizing each shard’s time range to be multiple shard steps wide.

backends:
  example:
    provider: 'prometheus'
    origin_url: http://prometheus:9090
    shard_step_ms: 7200000
    shard_size_ms: 14400000

Neither shard_step_ms or shard_max_size_ms can be used in conjunction with shard_max_size_points.

7.4 - TLS Support

Trickster supports Transport Layer Security on both the frontend server and backend clients.

Basics

To enable the TLS server, you must specify the tls_listen_port, and optionally, the tls_listen_address in the frontend section of your config file. For example:

frontend:
  listen_port: 8480
  tls_listen_port: 8483

Note, Trickster only starts listening on the TLS port if at least one origin has a valid certificate and key configured.

Each origin section of a Trickster config file can be augmented with the optional tls section to modify TLS behavior for front-end and back-end requests. For example:

backends:
  example: # example backend
    tls:   # TLS settings for example backend
      # server configs
      full_chain_cert_path: '/path/to/my/cert.pem'
      private_key_path: '/path/to/my/key.pem'
      # back-end configs
      insecure_skip_verify: true
      certificate_authority_paths: [ '/path/to/ca1.pem', '/path/to/ca2.pem' ]
      client_cert_path: '/path/to/client/cert.pem'
      client_key_path: '/path/to/client/key.pem'

Server Configuration when responding to clients

Each backend can handle encryption with exactly 1 certificate and key pair, as configured in the TLS section of the backend config (demonstrated above).

If the path to any configured Certificate or Key file is unreachable or unparsable, Trickster will exit upon startup with an error providing reasonable context.

You may use the same TLS certificate and key for multiple backends, depending upon how your Trickster configurations are laid out. Any certificates configured by Trickster must match the hostname header of the inbound http request (exactly, or by wildcard interpolation), or clients will likely reject the certificate for security issues.

Client Configs when proxying to an origin

Each backend’s TLS configuration can also configure the https client used for making requests against the origin as demonstrated above.

insecure_skip_verify will instruct the http client to ignore hostname verification issues with the upstream origin’s certificate, and process the request anyway. This is analogous to -k | --insecure in curl.

certificate_authority_paths will provide the http client with a list of certificate authorities (used in addition to any OS-provided root CA’s) to use when determining the trust of an upstream origin’s TLS certificate. In all cases, the Root CA’s installed to the operating system on which Trickster is running are used for trust by the client.

To use Mutual Authentication with an upstream origin server, configure Trickster with Client Certificates using client_cert_path and client_key_path parameters, as shown above. You will likely need to also configure a custom CA in certificate_authority_paths to represent your certificate signer, unless it has been added to the underlying Operating System’s CA list.

7.5 - Using Multiple Backends

A single Trickster instance can use multiple backends.

Trickster supports proxying to multiple backends in a single Trickster instance, by examining the inbound request and using a multiplexer to direct the proxied request to the correct upstream origin, in the same way that web servers support virtual hosting.

There are 2 ways to route to multiple backends.

HTTP Pathing
DNS Aliasing

Basic Usage

To utilize multiple backends, you must craft a Trickster configuration file to be read when Trickster starts up - operating with environment variables or command line arguments only supports accelerating a single backend. The example.full.yaml provides good documentation and commented sections demonstrating multiple backends. The config file should be placed in /etc/trickster/trickster.yaml unless you specify a different path when starting Trickster with the -config command line argument.

Each backend that your Trickster instance supports must be explicitly enumerated in the configuration file. Trickster does not support open proxying.

Example backend configuration:

backends:
  my-backend-01: # routed via http://trickster:8480/my-backend-01/
    provider: prometheus
    origin_url: http://my-origin-01.example.com

  my-backend-02: # routed via http://trickster:8480/my-backend-02/
    hosts: [ my-fqdn-02.example.com ] # or http://my-fqdn-02.example.com:8480/
    provider: prometheus
    origin_url: http://my-origin-02.example.com

  my-backend-03:
    path_routing_disabled: true # only routable via Host header, an ALB, or a Rule
    hosts: [ my-fqdn-03.example.com ] # or http://my-fqdn-03.example.com:8480/
    provider: prometheus
    origin_url: http://my-origin-02.example.com

Default Backend

Whether proxying to one or more upstreams, Trickster has the concept of a “default” backend, which means it does not require a specific DNS hostname in the request, or a specific URL path, in order to proxy the request to a known backend. When a default backend is configured, if the inbound request does not match any mapped backends by path or FQDN, the request will automatically be routed through the default backend. You are probably familiar with this behavior from when you first tried out Trickster with the using command line arguments.

Here’s an example: if you have Trickster configured with an backend named foo that proxies to http://foo/ and is configured as the default backend, then requesting http://trickster/image.jpg will initiate a proxy request to http://foo/image.jpg, without requiring the path be prefixed with /foo. But requesting to http://trickster/foo/image.jpg would also work.

The default backend can be configured by setting is_default: true for the backend you have elected to make the default. Having a default backend is optional. In a single-backend configuration, Trickster will automatically set the sole backend as is_default: true unless you explicitly set is_default: false in the configuration file. If you have multiple backends, and don’t wish to have a default backend, you can just omit the value for all backends. If you set is_default: true for more than one backend, Trickster will exit with a fatal error on startup.

Path-based Routing Configurations

In this mode, Trickster will use a single FQDN but still map to multiple upstream backends by path. This is the simplest setup and requires the least amount of work. The client will indicate which backend is desired in URL Path for the request.

Example Path-based Configuration with Multiple Backends:

backends:
  # backend1 backend
  backend1:
    origin_url: 'http://prometheus.example.com:9090'
    provider: prometheus
    cache_name: default
    is_default: true
  # "foo" backend
  foo:
    origin_url: 'http://influxdb-foo.example.com:9090'
    provider: influxdb
    cache_name: default
  # "bar" backend
  bar:
    origin_url: 'http://prometheus-bar.example.com:9090'
    provider: prometheus
    cache_name: default

Using HTTP Path as the Backend Routing Indicator

The client prefixes the Trickster request path with the Backend Name.

This is the recommended method for integrating with applications like Grafana.

Example Client Request URLs:

To Request from Origin foo: http://trickster.example.com:8480/foo/query?query=xxx
To Request from Origin bar: http://trickster.example.com:8480/bar/query?query=xxx
To Request from Origin backend1 as default: http://trickster.example.com:8480/query?query=xxx
To Request from Origin backend1 (Method 2, with Origin Name): http://trickster.example.com:8480/backend1/query?query=xxx
Configuring Grafana to request from backend foo via Trickster:

DNS Alias Configuration

In this mode, multiple DNS records point to a single Trickster instance. The FQDN used by the client to reach Trickster is mapped to specific backend configurations using the hosts list. In this mode, the URL Path is not considered during Backend Selection.

Example DNS-based Backend Configuration:

backends:
  # backend1 backend
  backend1:
    hosts: # users can route to this backend via these FQDNs, or via `/backend1`
      - 1.example.com
      - 2.example.com
    origin_url: 'http://prometheus.example.com:9090'
    provider: prometheus
    cache_name: default
    is_default: true
  # "foo" backend
  foo:
    hosts: # users can route to this backend via these FQDNs, or via `/foo`
      - trickster-foo.example.com
    origin_url: 'http://prometheus-foo.example.com:9090'
    provider: prometheus
    cache_name: default
  # "bar" backend
  bar:
    hosts: # users can route to this backend via these FQDNs, or via `/bar`
      - trickster-bar.example.com
    origin_url: 'http://prometheus-bar.example.com:9090'
    provider: prometheus
    cache_name: default

Example Client Request URLs:

To Request from Origin foo: http://trickster-foo.example.com:8480/query?query=xxx
To Request from Origin bar: http://trickster-bar.example.com:8480/query?query=xxx
To Request from Origin backend1 as default: http://trickster.example.com:8480/query?query=xxx
To Request from Origin backend1 (Method 2, via FQDN): http://backend1.example.com:8480/query?query=xxx

Note: It is currently possible to specify the same FQDN in multiple backend configurations. You should not do this (obviously). A future enhancement will cause Trickster to exit fatally upon detection at startup.

Disabling Path-based Routing for a Backend

You may wish for a backend to be inaccessible via the /backend_name/ path, and only by Hostname or as the target of a rule or ALB. You can disable path routing by setting path_routing_disabled: true for the backend, as in this example, which requires the Request’s Host header match 1.example.com or 2.example.com in order to be routed to the backend:

backends:
  backend1:
    hosts:
      - 1.example.com
      - 2.example.com
    origin_url: 'http://prometheus.example.com:9090'
    provider: prometheus
    cache_name: default
    is_default: false
    path_routing_disabled: true # this will disable routing through /backend1

8 - Load Balancers

The following articles provide information on application load balancers and health checks.

8.1 - Application Load Balancers

Solve for scalabilty with Trickster’s state-of-the-art Application Load Balancer.

Trickster 2.0 provides an all-new Application Load Balancer that is easy to configure and provides unique features to aid with Scaling, High Availability and other applications. The ALB supports several balancing Mechanisms:

Mechanism	Config	Provides	Description
Round Robin	rr	Scaling	a basic, stateless round robin between healthy pool members
Time Series Merge	tsm	Federation	uses scatter/gather to collect and merge data from multiple replica tsdb sources
First Response	fr	Speed	fans a request out to multiple backends, and returns the first response received
First Good Response	fgr	Speed	fans a request out to multiple backends, and returns the first response received with a status code < 400
Newest Last‑Modified	nlm	Freshness	fans a request out to multiple backends, and returns the response with the newest Last-Modified header

Integration with Backends

The ALB works by applying a Mechanism to select one or more Backends from a list of Healthy Pool Members, through which to route a request. Pool member names represent Backend Configs (known in Trickster 0.x and 1.x as Origin Configs) that can be pre-existing or newly defined.

All settings and functions configured for a Backend are applicable to traffic routed via an ALB - caching, rewriters, rules, tracing, TLS, etc.

In Trickster configuration files, each ALB itself is a Backend, just like the pool members to which it routes.

Mechanisms Deep Dive

Each mechanism has its own use cases and pitfalls. Be sure to read about each one to understand how they might apply to your situation.

Basic Round Robin

A basic Round Robin rotates through a pool of healthy backends used to service client requests. Each time a client request is made to Trickster, the round robiner will identify the next healthy backend in the rotation schedule and route the request to it.

The Trickster ALB is intended to support stateless workloads, and does not support Sticky Sessions or other advanced ALB capabilities.

Weighted Round Robin

Trickster supports Weighted Round Robin by permitting repeated pool member names in the same pool list. In this way, an operator can craft a desired apportionment based on the number of times a given backend appears in the pool list. We’ve provided an example in the snippet below.

Trickster’s round robiner cycles through the pool in the order it is defined in the Configuration file. Thus, when using Weighted Round Robin, it is recommended to use a non-sorted, staggered ordering pattern in the pool list configuration, so as to prevent routing bursts of consecutive requests to the same backend.

More About Our Round Robin Mechanism

Trickster’s Round Robin Mechanism works by maintaining an atomic uint64 counter that increments each time a request is received by the ALB. The ALB then performs a modulo operation on the request’s counter value, with the denominator being the count of healthy backends in the pool. The resulting value, ranging from 0 to len(healthy_pool) - 1 indicates the assigned backend based on the counter and current pool size.

Example Round Robin Configuration

backends:

  # traditional Trickster backend configurations

  node01:
    provider: reverseproxycache # will cache responses to the default memory cache
    path_routing_disabled: true # disables frontend request routing via /node01 path
    origin_url: https://node01.example.com # make requests with TLS
    tls: # this backend might use mutual TLS Auth
      client_cert_path: ./cert.pem
      client_key_path: ./cert.key

  node02:
    provider: reverseproxy      # requests will be proxy-only with no caching
    path_routing_disabled: true # disables frontend request routing via /node02 path
    origin_url: http://node-02.example.com # make unsecured requests
    request_headers: # this backend might use basic auth headers
      Authoriziation: "basic jdoe:*****"

  # Trickster 2.0 ALB backend configuration, using above backends as pool members

  node-alb:
    provider: alb
    alb:
      mechanism: rr # round robin
      pool:
        - node01 # as named above
        - node02
        # - node02 # if this node were uncommented, weighting would change to 33/67
        # add backends multiple times to establish a weighting protocol.
        # when weighting, use a cycling list, rather than a sorted list.

Here is the visual representation of this configuration:

Diagram of Trickster round robin mechanism

Time Series Merge

The recommended application for using the Time Series Merge mechanism is as a High Availability solution. In this application, Trickster fans the client request out to multiple redundant tsdb endpoints and merges the responses back into a single document for the client. If any of the endpoints are down, or have gaps in their response (due to prior downtime), the Trickster cache along with the data from the healthy endpoints will ensure the client receives the most complete response possible. Instantaneous downtime of any Backend will result in a warning being injected in the client response.

Separate from an HA use case, it is possible to Time Series Merge as a Federation broker that merges responses from different, non-redundant tsdb endpoints; for example, to aggregate metrics from a solution running clusters in multiple regions, with separate, in-region-only tsdb deployments. In this use case, it is recommended to inject labels into the responses to protect against data collisions across series. Label injection is demonstrated in the snippet below.

Providers Supporting Time Series Merge

Trickster currently supports Time Series Merging for the following TSDB Providers:

Provider Name
Prometheus

We hope to support more TSDBs in the future and welcome any help!

Example TS Merge Configuration

backends:

  # prom01a and prom01b are redundant and poll the same targets
  prom01a:
    provider: prometheus
    origin_url: http://prom01a.example.com:9090
    prometheus:
      labels:
        region: us-east-1

  prom01b:
    provider: prometheus
    origin_url: http://prom01b.example.com:9090
      labels:
        region: us-east-1

  # prom-alb-01 scatter/gathers to prom01a and prom01b and merges responses for the caller
  prom-alb-01:
    provider: alb
    alb:
      mechanism: tsm # time series merge
      pool: 
        - prom01a
        - prom01b

  # prom02 and prom03 poll unique targets but produce the same metric names as prom01a/b
  prom02:
    provider: prometheus
    origin_url: http://prom02.example.com:9090
      labels:
        region: us-east-2

  prom03:
    provider: prometheus
    origin_url: http://prom03.example.com:9090
      labels:
        region: us-west-1

  # prom-alb-all scatter/gathers prom01a/b, prom02 and prom03 and merges their responses
  # for the caller. since a unique region label was applied to non-redundant backends,
  # collisions are avoided. each backend caches data independently of the aggregated response
  prom-alb-all:
    provider: alb
    alb:
      mechanism: tsm
      pool: 
        - prom01a
        - prom01b
        - prom02
        - prom03

Here is the visual representation of a basic TS Merge configuration:

Diagram of Trickster timeseries merge configuration

First Response

The First Response mechanism fans a request out to all healthy pool members, and returns the first response received back to the client. All other fanned out responses are cached (if applicable) but otherwise discarded. If one backend in the fanout has already cached the requested object, and the other backends do not, the cached response will return to the caller while the other backends in the fanout will cache their responses as well for subsequent requests through the ALB.

This mechanism works well when using Trickster as an HTTP object cache fronting multiple redundant origins, to ensure the fastest response possible is delivered to downstream clients - even if the HTTP Response Code indicates an error in the request or by the first backend to respond.

First Response Configuration Example

backends:
  node01:
    provider: reverseproxycache
    origin_url: http://node01.example.com

  node02:
    provider: reverseproxycache
    origin_url: http://node-02.example.com

  node-alb-fr:
    provider: alb
    alb:
      mechanism: fr # first response
      pool:
        - node01
        - node02

Here is the visual representation of this configuration:

Diagram of Trickster first response configuration

First Good Response

The First Good Response mechanism acts just as First Response does, except that waits to return the first response with an HTTP Status Code < 400. If no fanned out response codes are in the acceptable range once all responses are returned (or the timeout has been reached), then the healthiest response, based on min(all_responses_status_codes), is used.

This mechanism is useful in applications such as live internet television. Consider an operational condition where an object may have been written to Origin 1, but not yet written to redundant Origin 2, while users have already received references to and begin requesting the object in a separate manifest. Trickster, when used as an ALB+Cache in this scenario, will poll both backends for the object and cache the positive responses from Origin 1 for serving subsequent requests locally, while a negative cache configuration will avoid potential 404 storms on Origin 2 until the object can be written by the replication process.

First Good Response Configuration Example


negative-caches:
  default: # by default, backends use the 'default' negative cache
    "404": 500 # cache 404 responses for 500ms

backends:
  node01:
    provider: reverseproxycache
    origin_url: http://node-01.example.com

  node02:
    provider: reverseproxycache
    origin_url: http://node-02.example.com

  node-alb-fgr:
    provider: alb
    alb:
      mechanism: fgr # first good response
      pool:
        - node01
        - node02

Here is the visual representation of this configuration:

Diagram of Trickster first good response configuration

Newest Last-Modified

The Newest Last-Modified mechanism is focused on providing the user with the newest representation of the response, rather than responding as quickly as possible. It will fan the client request out to all backends, and wait for all responses to come back (or the ALB timeout to be reached) before determining which response is returned to the user.

If at least one fanout response has a Last-Modified header, then any response not containing the header is discarded. The remaining responses are sorted based on their Last Modified header value, and the newest value determines which response is chosen.

This mechanism is useful in applications where an object residing at the same path on multiple origins is updated frequently, such as a DASH or HLS manifest for a live video broadcast. When using Trickster as an ALB+Cache in this scenario, it will poll both backends for the object, and ensure the newest version between them is used as the client response.

Note that with NLM, the response to the user is only as fast as the slowest backend to respond.

Newest Last-Modified Configuration Example

backends:
  node01:
    provider: reverseproxycache
    origin_url: http://node01.example.com

  node02:
    provider: reverseproxycache
    origin_url: http://node-02.example.com

  node-alb-nlm:
    provider: alb
    alb:
      mechanism: nlm # newest last modified
      pool:
        - node01
        - node02

Here is the visual representation of this configuration:

Diagram of Trickster last-modified configuration

Maintaining Healthy Pools With Automated Health Check Integrations

Health Checks are configured per-Backend as described in the Health documentation. Each Backend’s health checker will notify all ALB pools of which it is a member when its health status changes, so long as it has been configured with a health check interval for automated checking. When an ALB is notified that the state of a pool member has changed, the ALB will reconstruct its list of healthy pool members before serving the next request.

Health Check States

A backend will report one of three possible health states to its ALBs: unavailable (-1), unknown (0), or available (1).

Health-Based Backend Selection

Each ALB has a configurable healthy_floor value, which is the threshold for determining which pool members are included in the healthy pool, based on their instantaneous health state. The healthy_floor represents the minimum acceptable health state value for inclusion in the healthy pool. The default healthy_floor value is 0, meaning Backends in a state >= 0 (unknown and available) are included in the healthy pool. Setting healthy_floor: 1 would include only available Backends, while a value of -1 will include all backends in the configured pool, including those marked as unavailable.

Backends that do not have a health check interval configured will remain in a permanent state of unknown. Backends will also be in an unknown state from the time Trickster starts until the first of any configured automated health check is completed. Note that if an ALB is configured with healthy_floor: 1, any pool members that are not configured with an automated health check interval will never be included in the ALB’s healthy pool, as their state is permanently 0.

Example ALB Configuration Routing Only To Known Healthy Backends

backends:
  prom01:
    provider: prometheus
    origin_url: http://prom01.example.com:9090
    healthcheck:
      interval_ms: 1000 # enables automatic health check polling for ALB pool reporting

  prom02:
    provider: prometheus
    origin_url: http://prom02.example.com:9090
    healthcheck:
      interval_ms: 1000

  prom-alb-tsm:
    provider: alb
    alb:
      mechanism: tsm   # times series merge healthy pool members
      healthy_floor: 1 # only include Backends reporting as 'available' in the healthy pool
      pool:
        - prom01
        - prom02

All-Backends Health Status Page

Trickster 2.0 provides a new global health status page available at http://trickster:metrics-port/trickster/health (or the configured health_handler_path).

The global status page will display the health state about all backends configured for automated health checking. Here is an example configuration and a possible corresponding status page output:

backends:
  proxy-01:
    provider: reverseproxy
    origin_url: http://server01.example.com
    # not configured for automated health check polling

  prom-01:
    provider: prometheus
    origin_url: http://prom01.example.com:9090
    healthcheck:
      interval_ms: 1000 # enables automatic health check polling every 1s

  flux-01:
    provider: inflxudb
    origin_url: http://flux01.example.com:8086
    healthcheck:
      interval_ms: 1000 # enables automatic health check polling every 1s

$ curl "http://${trickster-fqdn}:8481/trickster/health"

Trickster Backend Health Status            last change: 2020-01-01 00:00:00 UTC
-------------------------------------------------------------------------------

prom-01      prometheus   available

flux-01      influxdb     unavailable since 2020-01-01 00:00:00 UTC
                                    
proxy-01     proxy        not configured for automated health checks

-------------------------------------------------------------------------------
You can also provide a 'Accept: application/json' Header or query param ?json

JSON Health Status

As the table footer from the plaintext version of the health status page indicates, you may also request a JSON version of the health status for machine consumption. The JSON version includes additional detail about any Backends marked as unavailable, and is structured as follows:

$ curl "http://${trickster-fqdn}:8481/trickster/health?json" | jq

{
  "title": "Trickster Backend Health Status",
  "udpateTime": "2020-01-01 00:00:00 UTC",
  "available": [
    {
      "name": "flux-01",
      "provider": "influxdb"
    }
  ],
  "unavailable": [
    {
      "name": "prom-01",
      "provider": "prometheus",
      "downSince": "2020-01-01 00:00:00 UTC",
      "detail": "error probing target: dial tcp prometheus:9090: connect: connection refused"
    }
  ],
  "unchecked": [
    {
      "name": "proxy-01",
      "provider": "proxy"
    }
  ]
}

8.2 - Health Checks

Keep your Trickster instance running at peak performance.

Trickster Service Health - Ping Endpoint

Trickster provides a /trickster/ping endpoint that returns a response of 200 OK and the word pong if Trickster is up and running. The /trickster/ping endpoint does not check any proxy configurations or upstream origins. The path to the Ping endpoint is configurable, see the configuration documentation for more information.

Upstream Connection Health - Backend Health Endpoints

Trickster offers health endpoints for monitoring the health of the Trickster service with respect to its upstream connection to origin servers.

Each configured backend’s health check path is /trickster/health/BACKEND_NAME. For example, if your backend is named foo, you can perform a health check of the upstream server at http://<trickster_address:port>/trickster/health/foo.

The backend health path prefix /trickster/health/ is customizable. See the example.full.yaml for more info about setting the health_handler_path configuration, or refer to this example:

frontend:
  # this overrides the default '/trickster/health' to '/-/trickster/health'
  health_handler_path: /-/trickster/health

The behavior of a health request will vary based on the Backend provider, as each has their own health check protocol. For example, with Prometheus, Trickster makes a request to /query?query=up and (hopefully) receives a 200 OK, while for InfluxDB the request is to /ping which returns a 204 No Content.

Supported TSDB Providers are pre-configured in Trickster to perform a suitable health check operation, however these can be overridden in the configuration file.

For non-TSDB Backends, the default behavior is to make a GET request to http://origin_url:port/ and expect a 2xx response. However, all aspects of the Health Check request and expected response are configurable per-Backend.

Basic Health Check Configuration Example

backends:
  server1:
    provider: reverseproxycache
    origin_url: http://server1
    healthcheck: # all values below are optional
      verb: HEAD
      path: /health

Health Check With Exhaustive Request/Response Options

backends:
  server1:
    provider: reverseproxy
    origin_url: http://server1
    healthcheck: # all values below are optional
      # 
      ## customizing the health check request
      #
      verb: HEAD
      scheme: https
      host: alternate-hostname.example.com
      path: /health
      query: param1=value1&param2=value2
      headers:
        User-Agent: health-check-agent
      # if using a POST or PUT method, you can provide a string body
      # body: "my health check body"
      #
      ## customizing the expected response
      #
      # hc fails if a response takes longer than 1s
      timeout_ms: 1000
      # hc fails if the response code is not in the list
      expected_codes: [ 200, 204, 206, 301, 302, 304 ]
      #
      # hc fails if these response headers are not present and have the expected value
      expected_headers:
        X-Health-Check-Status: success
      # hc fails if the stringified response body does not match the expected value
      expected_body: "pass"

See more examples in examples/conf/example.full.yaml.

Health Check Integrations with Application Load Balancers

By default, a Backend will only initiate a health check on-demand, upon receiving a request to its health endpoint.

To facilitate integrations with the Trickster Application Load Balancer provider, additional options provide for 1) timed interval health checks and 2) thresholding for consecutive successful or unsuccessful health checks that determine the backend’s overall health status.

Example Health Check Configuration for use in ALB

backends:
  server1:
    provider: reverseproxy
    origin_url: http://server1
    healthcheck:
      path: /health
      timeout_ms: 1000      # timeout_ms should be <= interval_ms
      # for ALB integration:
      interval_ms: 1000     # auto-poll health every 1s
      failure_threshold: 3  # backend is unhealthy after 3 consecutive failures
      recovery_threshold: 3 # backend is healthy after 3 consecutive successes

Other Ways to Monitor Health

In addition to the out-of-the-box health checks to determine up-or-down status, you may want to setup alarms and thresholds based on the metrics instrumented by Trickster. See Metrics for collecting performance metrics about Trickster.

9 - Tracing and Metrics

The following articles cover tracing and metrics in Trickster.

9.1 - Metrics

Check your project performance with Trickster metrics.

Trickster exposes a Prometheus /metrics endpoint with a customizable listener port number (default is 8481). For more information on customizing the metrics configuration, see Configuring.

The following metrics are available for polling with any Trickster configuration:

trickster_build_info (Gauge) - This gauge is always 1 when Trickster is running
- labels:
  - goversion - the version of go under which the running Trickster binary was built
  - revision - the commit ID on which the running Trickster binary was built
  - version - semantic version of the running Trickster binary
trickster_config_last_reload_successful (Gauge) - The value is 1 when true (the last config reload was successful) or 0 when false
trickster_config_last_reload_success_time_seconds (Gauge) - Epoch timestamp of the last successful configuration reload
trickster_frontend_requests_total (Counter) - Count of front end requests handled by Trickster
- labels:
  - backend_name - the name of the configured backend handling the proxy request
  - provider - the type of the configured backend handling the proxy request
  - method - the HTTP Method of the proxied request
  - http_status - The HTTP response code provided by the backend
  - path - the Path portion of the requested URL
trickster_frontend_requests_duration_seconds (Histogram) - Histogram of front end request durations handled by Trickster
- labels:
  - backend_name - the name of the configured backend handling the proxy request
  - provider - the type of the configured backend handling the proxy request
  - method - the HTTP Method of the proxied request
  - http_status - The HTTP response code provided by the backend
  - path - the Path portion of the requested URL
trickster_frontend_written_byte_total (Counter) - Count of bytes written in front end requests handled by Trickster
- labels:
  - backend_name - the name of the configured backend handling the proxy request
  - provider - the type of the configured backend handling the proxy request
  - method - the HTTP Method of the proxied request
  - http_status - The HTTP response code provided by the backend
  - path - the Path portion of the requested URL
trickster_proxy_requests_total (Counter) - The total number of requests Trickster has handled.
- labels:
  - backend_name - the name of the configured backend handling the proxy request
  - provider - the type of the configured backend handling the proxy request
  - method - the HTTP Method of the proxied request
  - cache_status - status codes are described here
  - http_status - The HTTP response code provided by the backend
  - path - the Path portion of the requested URL
trickster_proxy_points_total (Counter) - The total number of data points Trickster has handled.
- labels:
  - backend_name - the name of the configured backend handling the proxy request
  - provider - the type of the configured backend handling the proxy request
  - cache_status - status codes are described here
  - path - the Path portion of the requested URL
trickster_proxy_request_duration_seconds (Histogram) - Time required to proxy a given Prometheus query.
- labels:
  - backend_name - the name of the configured backend handling the proxy request
  - provider - the type of the configured backend handling the proxy request
  - method - the HTTP Method of the proxied request
  - cache_status - status codes are described here
  - http_status - The HTTP response code provided by the backend
  - path - the Path portion of the requested URL
trickster_proxy_max_connections (Gauge) - Trickster max number of allowed concurrent connections
trickster_proxy_active_connections (Gauge) - Trickster number of concurrent connections
trickster_proxy_requested_connections_total (Counter) - Trickster total number of connections requested by clients.
trickster_proxy_accepted_connections_total (Counter) - Trickster total number of accepted client connections.
trickster_proxy_closed_connections_total (Counter) - Trickster total number of administratively closed client connections.
trickster_proxy_failed_connections_total (Counter) - Trickster total number of failed client connections.
trickster_cache_operation_objects_total (Counter) - The total number of objects upon which the Trickster cache has operated.
- labels:
  - cache_name - the name of the configured cache performing the operation$
  - provider - the type of the configured cache performing the operation
  - operation - the name of the operation being performed (read, write, etc.)
  - status - the result of the operation being performed
trickster_cache_operation_bytes_total (Counter) - The total number of bytes upon which the Trickster cache has operated.
- labels:
  - cache_name - the name of the configured cache performing the operation$
  - provider - the type of the configured cache performing the operation
  - operation - the name of the operation being performed (read, write, etc.)
  - status - the result of the operation being performed

The following metrics are available only for Caches Types whose object lifecycle Trickster manages internally (Memory, Filesystem and bbolt):

trickster_cache_events_total (Counter) - The total number of events that change the Trickster cache, such as retention policy evictions.
- labels:
  - cache_name - the name of the configured cache experiencing the event$
  - provider - the type of the configured cache experiencing the event
  - event - the name of the event being performed
  - reason - the reason the event occurred
trickster_cache_usage_objects (Gauge) - The current count of objects in the Trickster cache.
- labels:
  - cache_name - the name of the configured cache$
  - provider - the type of the configured cache$
trickster_cache_usage_bytes (Gauge) - The current count of bytes in the Trickster cache.
- labels:
  - cache_name - the name of the configured cache$
  - provider - the type of the configured cache$
trickster_cache_max_usage_objects (Gauge) - The maximum allowed size of the Trickster cache in objects.
- labels:
  - cache_name - the name of the configured cache$
  - provider - the type of the configured cache
trickster_cache_max_usage_bytes (Gauge) - The maximum allowed size of the Trickster cache in bytes.
- labels:
  - cache_name - the name of the configured cache$
  - provider - the type of the configured cache

In addition to these custom metrics, Trickster also exposes the standard Prometheus metrics that are part of the client_golang metrics instrumentation package, including memory and cpu utilization, etc.

9.2 - Distributed Tracing via OpenTelemetry

Trickster supports a variety of tracing backends.

Trickster instruments Distributed Tracing with OpenTelemetry, which is a currently emergent, comprehensive observability stack that is in Public Beta. We import the OpenTelemetry golang packages to instrument support for tracing.

As OpenTelemetry evolves to support additional exporter formats, we will work to extend Trickster to support those as quickly as possible. We also make a best effort to update our otel package imports to the latest releases, whenever we publish a new Trickster release. You can check the go.mod file to see which release of opentelemetry-go we are is using. In this view, to see which version of otel a specific Trickster release imports, use the branch selector dropdown to switch to the tag corresponding to that version of Trickster.

Supported Tracing Backends

Jaeger
Jaeger Agent
Zipkin
Console/Stdout (printed locally by the Trickster process)

Configuration

Trickster allows the operator to configure multiple tracing configurations, which can be associated into each Backend configuration by name.

The example config has exhaustive examples of configuring Trickster for distributed tracing.

Span List

Trickster can insert several spans to the traces that it captures, depending upon the type and cacheability of the inbound client request, as described in the table below.

Span Name	Observes when Trickster is:
request	initially handling the client request by a Backend
QueryCache	querying the cache for an object
WriteCache	writing an object to the cache
DeltaProxyCacheRequest	handling a Time Series-based client request
FastForward	making a Fast Forward request for time series data
ProxyRequest	communicating with an Origin server to fulfill a client request
PrepareFetchReader	preparing a client response from a cached or Origin response
CacheRevalidation	revalidating a stale cache object against its Origin
FetchObject	retrieving a non-time-series object from an Origin

Tags / Attributes

Trickster supports adding custom tags to every span via the configuration. Depending upon your preferred tracing backend, these may be referred to as attributes. See the example config for examples of adding custom attributes.

Trickster also supports omitting any tags that Trickster inserts by default. The list of default tags are below. For example on the “request” span, an http.url tag is attached with the current full URL. In deployments where that tag may introduce too much cardinality in your backend trace storage system, you may wish to omit that tag and rely on the more concise path tag. Each tracer config can be provided a string list of tags to omit from traces.

Attributes added to top level (request) span

http.url - the full HTTP request URL
backend.name
backend.provider
cache.name
cache.provider
router.path - request path trimmed to the route match path for the request (e.g., /api/v1/query), good for aggregating when there are large variations in the full URL path

Attributes added to QueryCache span

cache.status - the lookup status of cache query. See the cache status reference for a description of the attribute values.

Attributes added to the FetchRevalidation span

isRange - is true if the client request includes an HTTP Range header

Attributes added to the FetchObject span

isPCF - is true if the origin is configured for Progressive Collapsed Forwarding

10 - Customizing Trickster

The following articles cover customizing and extending Trickster.

10.1 - Adding a New Configuration Value

Find out where to add new configuration value references.

Trickster configurations are defined in options packages below each feature package (e.g., /backends/options, caches/options, etc) and are mapped to yaml struct tags.

When adding a configuration value, there are several places to add references, which are described below.

Configuration Code

Each new configuration value must be defined in an options package under an existing Configuration collection (Origins, Caches, Paths, etc.).

Make sure the YAML annotation uses a lowercase_no_spaces naming convention, while the configuration member name itself should be CamelCase. Follow the existing configs for guidance.

Feature Code

Once you have defined your configuration value(s), you must put them to work by referencing them elsewhere in the Trickster code, and used to determine or customize the application functionality. Exactly where this happens in the code depends upon the context and reach of your new configuration, and what features its state affects. Consult with a project maintainer if you have any questions.

Tests

All new values that you add should have accompanying unit tests to ensure the modifications the value makes to the application in the feature code work as designed. Unit Tests should include verification of: proper parsing of configuration value from test config files (in ./testdata), correct feature functionality enable/disable based on the configuration value, correct feature implementation, coverage of all executable lines of code.

Documentation

The feature should be documented under ./docs directory, in a suitable existing or new markdown file based on the nature of the feature. The documentation should show the key example configuration options and describe their expected results, and point to the example config file for more information.

The example config file should be updated to include the exhaustive description and options for the configuration value(s).

Deployment

The ./deply/kube/configmap.yaml must be updated to include the new configuration option(s). Generally this file contains a copy/paste of example.full.yaml.

The ./deploy/helm/trickster/values.yaml file must be updated to mirror the configuration option(s) in example.full.yaml, and ./deploy/helm/trickster/templates/configmap.yaml must be updated to map any new yamlCaseValues to their respective snake case values for config file generation via the template.

10.2 - Extending Trickster to Support a New Provider

Find context for supporting a new timeseries database.

Trickster 2.0 was written with extensibility in mind, and should be able to work with any time series database that has an HTTP-based API. In Trickster, we generically refer to our supported TSDB’s as Providers. Some Providers are easier to implement and maintain than others, depending upon a host of factors that are covered later in this document. This document is meant to help anyone wishing to extend Trickster to support a new Provider, particularly in gauging the level of effort, understanding what is involved, and implementing the required interfaces and rules.

Qualifications

Not every database server out there is a candidate for being fronted by Trickster. Trickster serves the specific purpose of accelerating the delivery of time series data sets, and will not benefit traditional relational databases, NoSQL, etc.

As mentioned, the database must be able to be queried for and return time series data via HTTP. Some databases that are not specifically TSDB’s actually do support querying for and returning data in a time series format, and Trickster will support those cases as detailed below, so long as they have an HTTP API.

Skills Needed

In addition to these requirements in the technology, there are also skills qualifications to consider.

Whether or not you’ve contributed to Open Source Software before, take a look at our Contributing guidelines so you know how the process works for the Trickster project. If you are unfamiliar with the Forking Workflow, read up on it so that you are able to contribute to the project through Pull Requests.

Trickster is a 100% Go project, so you will need to have experience writing in Go, and in particular, data marshaling/unmarshaling, HTTP 1.0 and 2 specifications, and manipulation of HTTP requests and responses. You will need to have a good understanding of the prospective Provider’s query language and response payload structure, so you can write the necessary parsing and modeling methods that allow Trickster to manipulate upstream HTTP requests and merge newly fetched data sets into the cache.

While this might sound daunting, it is actually much easier than it appears on the surface. Since Trickster’s DeltaProxyCache engine does a lot of the heavy lifting, you only have to write a series of interface functions before finding yourself near the finish line. And since a few Providers are already implemented, you can use their implementations for references, since the logic for your prospective Provider should be similar.

Interfacing

A Time Series Backend is used by Trickster to 1) manipulate HTTP requests and responses in order to accelerate the requests, 2) unmarshal data from origin databases into the Common Time Series Format, and 3) marshal from the CTSF into a format supported by the Provider as requested by the downstream Client.

Trickster provides 1 required interfaces for enabling a new Provider: Time Series Backend. Separately, you must implement io.Writer-based marshalers and unmarshalers that conform to Trickster’s Modeler specifications.

Once data is unmarshaled into the Common Time Series Format, Trickster’s other packages will handle operations like Delta Proxy Caching, etc. Thus, the implementer of a new Provider only needs to worry about wire protocols and formats. Specifically, you will need to know these things about the Backend:

What URL paths and methods must be supported, and which engine through which to route each path (Basic HTTP Proxy, Object Proxy Cache, or Time Series Delta Proxy Cache). The proxy engines will call your client implementation’s interface exports in order to service user requests.
What data inputs the origin expects (Path, URL parameters, POST Data, HTTP Headers, cookies, etc.), and how to manipulate the query’s time range when constructing those inputs to achieve a desired result.
The Content Type, format and structure of the Provider’s datasets when transmitted over the wire.

The Interface Methods you will need to implement are as follows:

RegisterHandlers registers the provided http.Handlers into the Router
DefaultPathConfigs returns the default PathConfigs for the given Provider
DefaultHealthCheckConfig returns the default HealthCheck Config for the Provider
SetExtents sets the list of the time ranges present in the cache
ParseTimeRangeQuery inspects the client request and returns a corresponding timeseries.TimeRangeQuery
FastForwardURL (optional) returns the URL to the origin to collect Fast Forward data points based on the provided HTTP Request.

Special Considerations

Query Language Complexity

One of the main areas of consideration is the complexity of parsing and manipulating an inbound query. You will need to (1) determine if it is indeed a request for a timeseries; and if so (2) extract the requested time range and step duration for the query; and in the event of a partial cache hit, (3) adjust the time range for the query to a provided range - all of which allows the DeltaProxyCache to fetch just the needed sections of data from the upstream origin. Requirements 1 and 2 are functionality in ParseTimeRangeQuery while requirement 3 is the functionality of SetExtent. The overall complexity of this process can significantly affect the level of effort required to implement a new Provider.

In the example of Prometheus, the process was extremely simple: since, in the Prometheus HTTP API, time range queries have a separate http endpoint path from instantaneous queries, and because the time range is provided as separate query parameters from the query itself, the range is easily modified without Trickster having any knowledge of the underlying query or having to even parse it at all.

In the example of ClickHouse, the process is much harder: since the query language is a variant of SQL standard, the requested time ranges are embedded into the query itself behind the WHERE clause. In cases such as this, Trickster must have some way, either through (1) importing a database package that can deserialize the query, allow manipulation, and serialize the modified query for you; or (2) new parsing and search/replacement logic introduced in your own package - which allows the Client to interpret the time range and modify it with time ranges provided by the DeltaProxyCache. With ClickHouse, since it is a C++ project and Trickster is Golang, we could not import any package to handle this work, so we crafted a regular expression to match against inbound queries. This regex extracts the timerange (and any ClickHouse time modifiers like startOfMinute) and step value as well as any other areas that include the timerange, and then use simple builtin string functions to inject tokens in place of specific ranges in the provided query. Then, when SetExtent is called, those tokens are search/replaced with the provided time values.

Data Model Considerations

Once you have the Client Interface implementation down and can interact with the upstream HTTP API, you will turn your attention to managing the response payload data and what to do with it, and that happens in your Timeseries Interface implementation. And like the Client interface, it will come with its own unique challenges.

The main consideration here is the format of the output and what challenges are presented by it. For example, does the payload include any required metadata (e.g., a count of total rows returned) that you will need to synthesize within your Timeseries after a Merge, etc. Going back to the ClickHouse example, since it is a columnar database that happens to have time aggregation functions, there are a million ways to formulate a query that yields time series results. That can have implications on the resulting dataset: which fields are the time and value fields, and what are the rest? Are all datapoints for all the series in a single large slice or have they been segregated into their own slices? Is the Timestamp in Epoch format, and if so, does it represent seconds or milliseconds? In order to support an upstream database, you may need to establish or adopt guidelines around these and other questions to ensure full compatibility. The ClickHouse plugin for Grafana requires that for each datapoint of the response, the first field is the timestamp and the second field is the numeric value - so we adopt and document the same guideline to conform to existing norms.

Getting More Help

On the Gophers Slack instance, you can find us on the #trickster channel for any help you may need.

10.3 - Spinning New Trickster Release

Instructions for maintainers and owners for spinning new Trickster releases.

Users with push access to trickstercache/trickster (maintainers and owners) can spin releases.

To spin a new Trickster release, clone the repo, checkout the commit ID for the release, tag it with a release in semantic version format (vX.Y.Z), and push the tag back to the GitHub repository.

GitHub actions will detect the publishing of the new tag, if it’s in the proper format, and cut a full release for the tag automatically.

The cut release will be published as a draft, giving the maintainer the opportunity to craft release notes in advance of the actual release.

Once a Trickster release is cut, the maintainer must follow the directions in the trickster-docker-images project to separately push the release to Docker Hub via automation.

11 - Release Info

Here you can find information on the latest Trickster updates and our plans going forward.

11.1 - Trickster 2.0

See the latest updates to Trickster.

What’s Improved

2.0 continues to improve the Trickster project, with a ton of new features, bug fixes, and optimizations. Here’s the quick rundown of what’s new and improved:

We now use YAML for configuration, and provide tooling to migrate a 1.x TOML configuration
Example configurations are relocated to the examples/conf directory
The Trickster docker-compose demo has been relocated to the examples directory and updated to use latest version tags
We now use a common time series format internally for caching all supported TSDB’s, rather than implementing each one separately
Health checking now uses a common package for all backend providers, rather than implementing separately in each backend, and we now support automated health check polling for any backend, and provide a global health status endpoint
We offer a brand new Application Load Balancer feature with unique and powerful options, like merging data from multiple backends into a single response.
We’ve updated to Go 1.16
We’ve re-organized many packages in the codebase to be more easily importable by other projects. Over the course of the beta, we’ll be publishing new patterns to the examples folder for using Trickster packages in your own projects, including caching, acceleration and load balancing.
InfluxDB and ClickHouse now support additional output formats like CSV. More documentation will be provided over the course of the beta.

New in Beta 2

For InfluxDB, we’ve improved our compatibility with InfluxQL and hope you’ll see improved cache hit rates. Flux support is not quite ready yet.
We’ve updated our “Backfill Tolerance” feature, which handles volatile data (e.g., real-time data that may not be immutable), to improve cache hit rates while ensuring volatile data is still refreshed. We’ve also added new options for controlling backfill tolerance thresholds. See the example config.
We’ve significantly revamped Trickster’s compression capabilities. Beta 2 makes the following enhancements:
- adds Brotli and Zstd compression support when serving responses (we already supported gzip and deflate), when included in an Accept-Encoding header
- changes the cache compression storage format from Snappy to Brotli. A future beta will allow customization via configuration
- passes through (or injects) an Accept-Encoding header to requests between Trickster and Origins, to support end-to-end content encoding. We previously were not accepting encodings from upstreams.
- Provides a generic HTTP Handler package, supporting Zstd, Brotli, Gzip and Deflate; which is importable by any golang application
We fixed up a memory leak and refined the Prometheus label injection feature, so it plays nicely with the TSMerge ALB
Our Rules Engine now supports rmatch operations to permit regular expression-based routing against any part of the HTTP request.
You can now chain a collection request rewriters for more robust possibilities.

Still to Come

Trickster 2.0 is not yet feature complete, and we anticipate including the following additional features before the GA Release:

support for InfluxData’s Flux query language, and for queries sourced by Chronograf
cache object purge via API
additional logging, metrics and tracing spans covering 2.0’s new features
an up-to-date Grafana dashboard template for monitoring Trickster
support for additional Time Series Database providers
incorporate ALB examples into the docker-compose demo

Known Issues With the Latest Beta

The current Trickster 2.0 beta has the following known issues:

the lru Time Series Eviction Method currently does not function, but will be added back in a future beta. This feature has not yet been ported into the Common Time Series format. Comment out this setting in your configuration to use the default eviction method.
Helm Charts are not yet available for Trickster 2.0 and will be provided in a subsequent beta release.

Installing

You can build the 2.0 binary from the main branch, download binaries from the Releases page, or use the trickstercache/trickster:2 Docker image tag in containerized environments.

Breaking Changes from 1.x

Metrics

In metrics related to Trickster’s operation, all label names of origin_name are changed to backend_name

Configuration

Using tricktool to migrate your configurations is the recommended approach. However, if you choose to convert your configuration by hand, here is what you need to know:

https://www.convertsimple.com/convert-toml-to-yaml/ is a good starting point
The [origins] section of the Trickster 1.x TOML config is named backends: in the 2.0 YAML config
All duration-based values are now represented in milliseconds. 1.x values ending in _secs are the same in 2.0 but end in _ms. Be sure to multiply by 1000
origin_type, cache_type and tracing_type are now called provider.
Health checking configurations now reside in their own healthcheck subsection under backends and use simplified config names like method, path, etc.

See the example configuration for more information.

11.2 - Trickster Roadmap

See what we’re working on for future versions of Trickster.

The roadmap for Trickster in 2021 focuses on delivering Trickster versions 2.0 and 2.1, as well as new supporting applications and cloud native integrations. Additional details for Q3 and Q4 will be provided as the year progresses.

Timeline

Q1 2021

Trickster v2.0 Beta Release
- Common Time Series Format used internally for all TSDBs
- Universal HTTP Health Checker Package
- ALB with features for high availability and scatter/gather timeseries merge
- YAML config support
- Extended support for ClickHouse
- Support for InfluxDB 2.0, Flux syntax and querying via Chronograf
- Purge object from cache by path or key
- Short-term caching of non-timeseries read-only queries (e.g., generic SELECT statements)
- Support Brotli encoding over the wire and as a cache compression format
Submit Trickster for CNCF Sandbox Consideration

Q2 2021

Trickster v2.0 GA Release
- Documentation overhaul using MkDocs with Release deployment automation
- Migrate integration tests infrastructure as needed to easily integrate with related CNCF projects.
Trickster v2.1 Beta Release
- Support for ElasticSearch
- Support operating as an adaptive, front-side cache for Grafana, including its UI, API’s, and accelerating any supported timeseries datasources.
- Better support for operating in front of Thanos
- Ability to parallelize large timerange queries by scatter/gathering smaller sections of the main timerange.
- Additional Rules Engine capabilities for more complex request routing
- Grafana-style environment variable support
- Subdirectory (e.g., /etc/trickster.conf.d/) support for chained config files
Register Official Docker Hub Repositories

Q3 2021

Trickster v2.1 GA Release

Q4 2021

Trickster v2.2 Beta Release

Get Involved

You can help by contributing to Trickster, or trying it out in your environment.

By giving Trickster a spin, you can help us identify and fix defects more quickly. Be sure to file issues if you find something wrong. If you can reliably reproduce the issue, provide detailed steps so that developers can more easily root-cause the issue.

If you want to contribute to Trickster, we’d love the help. Please take any issue that is not already assigned as per the contributing guidelines, or check with the maintainers to find out how best to get involved.

Thank You

We are so excited to share the Trickster with the community. This is only possible through our great community of contributors, users and supporters. Thank you for all you in making this project a success!