This crate provides a subword tokenizer. A subword tokenizer splits a token into several pieces, so-called word pieces. Word pieces were popularized by and used in the BERT natural language encoder.
This repository was archived by the owner on Mar 30, 2026. It is now read-only.