This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Caching

Cache data stores and behaviors common to all of Trickster’s caching modes.

1 - Cache Overview

Supported Caches

There are several cache types supported by Trickster

  • In-Memory (default)
  • Filesystem
  • bbolt
  • BadgerDB
  • Redis (basic, cluster, and sentinel)

The sample configuration (examples/conf/example.full.yaml) demonstrates how to select and configure a particular cache type, as well as how to configure generic cache configurations such as Retention Policy.

In-Memory

In-Memory Cache is the default type that Trickster will implement if none of the other cache types are configured. The In-Memory cache utilizes a Golang sync.Map object for caching, which ensures atomic reads/writes against the cache with no possibility of data collisions. This option is good for both development environments and most smaller dashboard deployments.

When running Trickster in a Docker container, ensure your node hosting the container has enough memory available to accommodate the cache size of your footprint, or your container may be shut down by Docker with an Out of Memory error (#137). Similarly, when orchestrating with Kubernetes, set resource allocations accordingly.

Filesystem

The Filesystem Cache is a popular option when you need more storage space than you wish to accommodate in RAM: a large dashboard setup (e.g., many different dashboards with many varying queries, Dashboard as a Service for several teams running their own Prometheus instances, etc.), or large objects like media and downloads. A Filesystem Cache configuration keeps the Trickster RAM footprint small. Trickster performance can be degraded when using the Filesystem Cache if disk i/o becomes a bottleneck.

The default Filesystem Cache path is /tmp/trickster. The sample configuration demonstrates how to specify a custom cache path. Ensure that the user account running Trickster has read/write access to the custom directory or the application will exit on startup upon testing filesystem access. All users generally have access to /tmp so there is no concern about permissions in the default case.

A local filesystem (e.g., ext4 or XFS) is recommended. Network filesystems are discouraged, as they are slow to create, rename and list the many files a cache is made of.

See Disk Caches for how the Filesystem Cache lays out, recovers and bounds what it stores.

bbolt

The BoltDB Cache is a popular key/value store, created by Ben Johnson. CoreOS’s bbolt fork is the version implemented in Trickster. A bbolt store is a filesystem-based solution that stores the entire database in a single file. Trickster, by default, creates the database at trickster.db and uses a bucket name of ’trickster’ for storing key/value data. See the example config file for details on customizing this aspect of your Trickster deployment. The same guidance about filesystem permissions described in the Filesystem Cache section above apply to a bbolt Cache.

bbolt commits each write to disk before it returns, so it is slower to store an object than the Filesystem Cache, and is as fast or faster to read one. It is a good fit for caches of many small objects, since it does not spend a file on each.

See Disk Caches for how the bbolt Cache recovers and bounds what it stores.

Disk Caches

The Filesystem and bbolt Caches keep whatever they are given, so Trickster manages the lifecycle of their objects with a Cache Index, which it holds in memory and persists to the cache.

Stored Objects

Each object is stored with a header that holds its cache key, its expiration and a checksum of its content. An object that is found to be expired, incomplete or damaged when it is read is treated as a cache miss and removed. The checksum of an object of 10 MiB or more is verified after the object is returned rather than before, so that a large read is not held for it; such an object, if damaged, is served once and then removed. The format of what is stored is private to Trickster and versioned: objects stored by a version that used another format are treated as cache misses and removed, so a disk cache can be cold after an upgrade. The release notes say when that is so.

The Filesystem Cache writes each object to a new file and renames it into place, so a reader never finds part of an object, even after a crash. Files are spread across two levels of directories under cache_path, named for the first characters of the file’s name, to keep any one directory small:

/tmp/trickster/9f/86/9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
/tmp/trickster/_meta/

A file is named for its cache key when the key is a lowercase hex digest of 32 or 64 characters, as the keys of HTTP objects are, and otherwise for a hash of the key. The _meta directory holds the Cache Index.

Serving Large Objects

An object proxied through the Reverse Proxy Cache is stored in two sections: what describes it (status, headers, caching policy) and its body. When the whole of an object is cached uncompressed, a GET that is a fresh cache hit reads the body from the disk cache as it is written to the client, and a Range request reads only the ranges that were asked for. Neither holds the object in memory, whatever its size. Objects that Trickster compresses in the cache (see compressible_types) are read whole, as are objects that must first be revalidated.

Cache Index

SettingDefaultDescription
reap_interval3show often expired objects are removed, and a cache that is over its size is brought back within it
flush_interval5show often changes to the index are persisted
index_expiry8760hhow old a persisted index can be, and still be used at startup
max_size_bytes536870912the size in bytes at which least-recently-accessed objects are evicted
max_size_backoff_bytes16777216how far under max_size_bytes an eviction takes the cache
max_size_objects0the count of objects at which least-recently-accessed objects are evicted; 0 for no limit
max_size_backoff_objects100how far under max_size_objects an eviction takes the cache
scan_interval24hhow often the cache is swept; 0 to sweep only when the index cannot be trusted
scan_batch_size512how many objects a sweep reads before it pauses
scan_batch_pause50mshow long a sweep pauses after each batch

The Filesystem Cache also accepts min_free_bytes, the space to keep free on the filesystem that holds cache_path. When less is free, the cache evicts as it does when it is over max_size_bytes. It is supported on Unix-like systems.

Persistence. The index is persisted as a snapshot and a journal of the changes made since. Each flush_interval, the changes are appended to the journal; once the journal has grown to half the size of the snapshot, a new snapshot replaces both. Objects that expire within a minute of being stored are not persisted.

Recovery. When Trickster starts with an index that was persisted by a clean shutdown, it is used as it is. When the index is missing, expired, damaged, or was not closed by a clean shutdown, Trickster uses what it can of it and sweeps the cache in the background, while serving requests. A sweep reads the header of every object, scan_batch_size at a time, and lists the objects the index did not know of, so that nothing in the cache is orphaned. It also removes what can’t be served: objects that are expired or damaged, files left by an interrupted write, and objects stored by a version of Trickster that used another format.

Eviction. Objects are evicted in the order they were least recently accessed. In a cache of more than 1024 objects, the object to evict is the least recently accessed of 32 taken at random, which costs the same however large the cache is, and closely follows the true order. A write that takes the cache over its size has the reaper act within a second, without waiting for reap_interval.

BadgerDB

BadgerDB works similarly to bbolt, in that it is a filesystem-based key/value datastore. BadgerDB provides its own native object lifecycle management (TTL) and other additional features that distinguish it from bbolt. See the configuration for more info on using BadgerDB with Trickster.

Redis

Note: Trickster does not come with a Redis server. You must provide a pre-existing Redis endpoint for Trickster to use.

Redis is a good option for larger dashboard setups that also have heavy user traffic, where you might see degraded performance with a Filesystem Cache. This allows Trickster to scale better than a Filesystem Cache, but you will need to provide your own Redis instance at which to point your Trickster instance. The default Redis endpoint is redis:6379, and should work for most docker and kube deployments with containers or services named redis. The sample configuration demonstrates how to customize the Redis endpoint. In addition to supporting TCP endpoints, Trickster supports Unix sockets for Trickster and Redis running on the same VM or bare-metal host.

Ensure that your Redis instance is located close to your Trickster instance in order to minimize additional roundtrip latency.

In addition to basic Redis, Trickster also supports Redis Cluster and Redis Sentinel. Refer to the sample configuration for customizing the Redis client type.

Trickster supports Redis servers that use TLS encryption by setting use_tls: true in the config. Refer to the sample configuration for more info.

Purging an Item from the Cache

You can purge an item from the cache by making a call to the purge endpoint, as follows:

http://${trickster-address}:${mgmt-port}/trickster/purge/path/${backendName}/${path/to/purge}

For example, if you want to purge /api/v1/labels from backend prom1, a curl might look like:

curl http://localhost:8484/trickster/purge/path/prom1/api/v1/labels

Purging the Full Cache

Full Cache purges should not be necessary, but in the event that you wish to do so, the following steps should be followed based upon your selected Cache Type.

A future release will provide a mechanism to fully purge the cache (regardless of the underlying cache type) without stopping a running Trickster instance.

Purging In-Memory Cache

Since this cache type runs inside the virtual memory allocated to the Trickster process, bouncing the Trickster process or container will effectively purge the cache.

Purging Filesystem Cache

To completely purge a Filesystem-based Cache, you will need to:

  • Docker/Kube: delete the Trickster container (or mounted volume) and run a new one
  • Metal/VM: Stop the Trickster process and manually run rm -rf /tmp/trickster (or your custom-configured directory).

Purging Redis Cache

Connect to your Redis instance and issue a FLUSH command. Note that if your Redis instance supports more applications than Trickster, a FLUSH will clear the cache for all dependent applications.

Purging bbolt Cache

Stop the Trickster process and delete the configured bbolt file.

Purging BadgerDB Cache

Stop the Trickster process and delete the configured BadgerDB path.

Cache Status

Trickster reports several cache statuses in metrics, logs, tracing, and the X-Trickster-Result response header, which are listed and described in the table below.

StatusDescription
kmissThe requested object was not in cache and was fetched from the origin
rmissObject is in cache, but the specific data range requested (timestamps or byte ranges) was not
hitThe object was fully cached and served from cache to the client
phitThe object was cached for some of the data requested, but not all
nchitThe response was served from the Negative Cache
rhitThe object was served from cache to the client, after being revalidated for freshness against the origin
purgeThe cache key was purged as directed by a request or response header
proxy-onlyThe request was proxied 1:1 to the origin and not cached
proxy-errorThe upstream request needed to fulfill an associated client request returned an error
errorTrickster encountered a cache lookup or cache handling error
proxy-hitThe request joined an existing in-flight origin fetch for the same cache key

2 - Trickster Caching Retention Policies

Basic HTTP Backends

Trickster will respect HTTP 1.0, 1.1 and 2.0 caching directives from both the downstream client and the upstream origin when determining object cacheability and TTL. You can override the TTL by setting a custom Cache-Control header on a per-Path Config basis.

Cache Object Evictions

If you use a Trickster-managed cache (Memory, Filesystem, bbolt), then a maximum cache size is maintained by Trickster. You can configure the maximum size in number of bytes, number of objects, or both. See the example configuration for more information.

Once the cache has reached its configured maximum size of objects or bytes, Trickster will undergo an eviction routine that removes cache objects until the size has fallen below the configured maximums. Trickster-managed caches maintain a last access time for each cache object, and utilizes a Least Recently Used (LRU) methodology when selecting objects for eviction.

Caches whose object lifetimes are not managed internally by Trickster (Redis, BadgerDB) will use their own policies and methodologies for evicting cache records.

Time Series Backends

For non-time series responses from a TSDB, Trickster will adhere to HTTP caching rules as directed by the downstream client and upstream origin.

For time series data responses, Trickster will cache as follows:

TTL Settings

TTL settings for each Backend configured in Trickster can be customized independently of each other, and separate TTL configurations are available for timeseries objects, and fast forward data. See examples/conf/example.full.yaml for more info on configuring default TTLs.

Time Series Data Retention

Separately from the TTL of a time series cache object, Trickster allows you to control the size of each timeseries object, represented as a count of maximum timestamps in the cache object, on a per origin basis. This configuration is known as the timeseries_retention_factor (TRF), and has a default of 1024. Most dashboards for most users request and display approximately 300-to-400 timestamps, so the default TRF allows users to still recall recently-displayed data from the Trickster cache for a period of time after the data has aged off of real-time views.

If you have users with a high-resolution dashboard configuration (e.g., a 24-hour view with a 1-minute step, amounting to 1440 data points per graph), then you may benefit from increasing the timeseries_retention_factor accordingly. If you use a managed cache (see caches) and increase the timeseries_retention_factor, the overall size of your cache will not change; the result will be fewer objects in cache, with the timeseries objects having a larger share of the overall cache size with more aged data.

Time Series Data Evictions

Once the TRF is reached for a time series cache object, Trickster will undergo a timestamp eviction process for the record in question. Unlike the Cache Object Eviction, which removes an object from cache completely, TRF evictions examine the data set contained in a cache object and remove timestamped data in order to reduce the object size down to the TRF.

Time Series Data Evictions apply to all cached time series data sets, regardless of whether or not the cache object lifecycle is managed by Trickster.

Trickster provides two eviction methodologies (timeseries_eviction_method) for time series data eviction: oldest (default) and lru, and is configurable per-origin.

When timeseries_eviction_method is set to oldest, Trickster maintains time series data by calculating the “oldest cacheable timestamp” value upon each request, using time.Now().Add(step * timeseries_retention_factor * -1). Any queries for data older than the oldest cacheable timestamp are intelligently offloaded to the proxy since they will never be cached, and no data that is older than the oldest cacheable timestamp will be stored in the query’s cache record.

When timeseries_eviction_method is set to lru, Trickster will not calculate an oldest cacheable timestamp, but rather maintain a last-accessed time for each timestamp in the cache object, and evict the Least-Recently-Used items in order to maintain the cache size.

The advantage of the oldest methodology better cache performance, at the cost of not caching very old data. Thus, Trickster will be more performant computationally while providing a slightly lower cache hit rate. The lru methodology, since it requires accessing the cache on every request and maintaining access times for every timestamp, is computationally more expensive, but can achieve a higher cache hit rate since it permits caching data of any age, so long as it is accessed frequently enough to avoid eviction.

Most users will find the oldest methodology to meet their needs, so it is recommended to use lru only if you have a specific use case (e.g., dashboards with data from a diverse set of time ranges, where caching only relatively young data does not suffice).

3 - Chunked Caching

Overview

In some caching setups, users may want to increase timeseries_retention_factor or timeseries_ttl forms a given backend to a very large size (e.g., a duration of days or weeks). This can cause issues if the cache provider is filesystem or redis, because the entire time series is loaded to extract even just a few data points. Eventually, this could negate the effects of caching altogether.

To mitigate this, Trickster supports chunking cache data, by splitting large datasets into subdivisions of a configurable maximum size. The chunks are reconstituted upon retrieval. Only the chunks needed to service a client request are accessed, rather than the entire time series cache object.

Chunking can be configured per-cache and applies to both timeseries and byterange data.

Configuration

Chunked caching can be enabled and disabled using use_cache_chunking:

fs1:
    provider: filesystem
    use_cache_chunking: true
    timeseries_chunk_factor: 420
    byterange_chunk_size: 4096

timeseries_chunk_factor determines the maximum extent of timerange chunks, and byterange_chunk_size determines the maximum size of byterange chunks. See Detail for more information.

Changes to any of these settings take effect on a configuration reload, without a restart.

Detail

Timeseries

Timeseries chunking splits the timeseries to be cached into parts with the same duration, but not necessarily the same literal size.

  • Determine a chunk duration by multiplying the timerange step by timeseries_chunk_factor (default 420)
  • Determine the smallest possible extent that is aligned to the epoch along the chunk duration, while containing the entire timeseries
  • To write: Write each chunk size subextent under a subkey
  • To read: Read each subkey and merge the timeseries results

Byterange

Byterange chunking splits the byterange into pieces with the same literal size. There are also some extra steps compared to the timeseries implementation to preserve the integrity of both full and partial responses being cached.

  • Determine a chunk size from byterange_chunk_size (default 4096)
  • Determine a range from the cache read/write request using the provided range or content length
  • Failure to determine a range on write results in an error
  • Failure to determine a range on read will read until the query fails
  • Determine a maximum range aligned along the chunk size that contains the entire byterange
  • To write: Write each chunk size range with RangeParts of all provided ranges cropped to that chunk range, under a subkey. A chunk that falls in a gap between cached ranges holds nothing and isn’t written.
  • To read: Read only the subkeys covering the requested range and reconstitute a body from RangeParts. Cached ranges outside those chunks stay in the cache and remain listed in the meta document, so a later write leaves them intact.
  • A chunk that was evicted or expired is treated as a gap: its bytes are fetched from the origin again and rewritten, rather than served from the remaining chunks.

Full Example

This example has one Prometheus backend with a memory cache that has chunking enabled. The memory cache uses 380 as its timeseries chunk factor, and doesn’t define a byterange chunk size, so the default of 4096 will be used.

caches:
  mem1:
    provider: memory
    index:
      max_size_objects: 512
      max_size_backoff_objects: 128
    use_cache_chunking: true
    timeseries_chunk_factor: 380

backends:
  prom1:
    latency_max: 150ms
    latency_min: 50ms
    provider: prometheus
    origin_url: 'http://127.0.0.1:9090'
    cache_name: mem1

logging:
  log_level: warn

4 - Negative Caching

Negative Caching means to cache undesired HTTP responses for a very short period of time, in order to prevent overwhelming a system that would otherwise scale normally when desired, cacheable HTTP responses are being returned. For example, Trickster can be configured to cache 404 Not Found or 500 Internal Server Error responses for a short period of time, to ensure that a thundering herd of HTTP requests for a non-existent object, or unexpected downtime of a critical service, do not create an i/o bottleneck in your application pipeline.

Trickster supports negative caching of any status code >= 300 and < 600, on a per-Backend basis. In your Trickster configuration file, associate the desired Negative Cache Map to the desired Backend config. See the example.full.yaml, or refer to the snippet below for more information.

A moq backend uses its negative cache’s 404 TTL for an origin’s DOES_NOT_EXIST refusal of a track, MoQ’s equivalent of a 404, and no other entry.

The Negative Cache Map must be an all-inclusive list of explicit status codes; there is currently no wildcard or status code range support for Negative Caching entries. By default, the Negative Cache Map is empty for all backend configs. The Negative Cache only applies to Cacheable Objects, and does not apply to Proxy-Only configurations.

For any response code handled by the Negative Cache, the response object’s effective cache TTL is explicitly overridden to the value of that code’s Negative Cache TTL, regardless of any response headers provided by the Backend concerning cacheability. All response headers are left in-tact and unmodified by Trickster’s Negative Cache, such that Negative Caching is transparent to the client. The X-Trickster-Result response header will indicate a response was served from the Negative Cache by providing a cache status of nchit.

Multiple negative cache configurations can be defined, and are referenced by name in the backend config. By default, a backend will use the ‘default’ Negative Cache config, which, by default is empty. The default can be easily populated in the config file, and additional configs can easily be added, as demonstrated below.

The format of a negative cache map entry is 'status_code': ttl.

Example Negative Caching Config

negative_caches:
  default:
    '404': 3s # cache 404 responses for 3 seconds
  foo:
    '404': 3s
    '500': 5s # caches 404 response for 3 seconds, and 500/502 for 5 seconds
    '502': 5s

backends:
  default:
    provider: rpc
    # by default will assume negative_cache_name = 'default'
  another:
    provider: rpc
    negative_cache_name: foo