Skip to content

armgit / laya

Public Imported from GitHub
0

Non-autoregressive System 1 decision engine. Typed choice, score and yes/no decisions over any text in a single forward pass, in 100+ languages, with a router that picks the right checkpoint per request.

399 B
Raw Blame
//! The tokenizer abstraction [`crate::sequence_builder::SequenceBuilder::build`] is written
//! against, plus [`hf_tokenizer::HfTokenizer`], the real HuggingFace-backed implementation
//! (porting `HfTokenizer.cs`) used by [`crate::laya_engine::LayaEngine`].

pub mod hf_tokenizer;
pub mod laya_tokenizer;

pub use hf_tokenizer::HfTokenizer;
pub use laya_tokenizer::{LayaTokenizer, SpecialTokens};