Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

1.1 KB
Raw Blame
//! Cached tiktoken cl100k_base encoder.
//!
//! `tiktoken_rs::cl100k_base()` rebuilds the entire BPE encoder on every call
//! (decoding ~100k vocabulary entries and constructing several HashMaps), which
//! costs ~50-200ms of CPU per invocation. We call it on hot paths — every signal
//! emission and every stacktrace truncation check — so we cache it once for the
//! lifetime of the process.

use std::sync::LazyLock;
use tiktoken_rs::{cl100k_base, CoreBPE};

pub static CL100K_BPE: LazyLock<CoreBPE> =
    LazyLock::new(|| cl100k_base().expect("cl100k_base encoder must be constructible"));

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn cl100k_bpe_is_reused_across_calls() {
        // Two derefs should yield the same underlying instance (same address),
        // proving the encoder is initialized once and reused.
        let first: *const CoreBPE = &*CL100K_BPE;
        let second: *const CoreBPE = &*CL100K_BPE;
        assert_eq!(first, second);
    }

    #[test]
    fn cl100k_bpe_encodes_text() {
        let tokens = CL100K_BPE.encode_with_special_tokens("hello world");
        assert!(!tokens.is_empty());
    }
}