hf_tokenizers 0.3.0 changelog

0.3.0 #

Add encodeWithOffsets, which returns each token as a TokenOffset carrying its id and the [start, end) UTF-8 byte span of the input it came from. This is what token-accurate chunking, span highlighting, and named-entity extraction need. Offsets are byte offsets (not UTF-16), so slice utf8.encode of the input rather than calling String.substring; special tokens carry an empty span. encode is unchanged for the id-only path.
This adds one native symbol (tk_encode_offsets), so the prebuilt binaries are rebuilt for this release. The build hook downloads the matching binaries automatically; nothing to install.

Add tokenToId and idToToken for single-token lookup in either direction. Both return null when the token or id is not in the vocabulary. idToToken keeps sub-word markers (such as WordPiece's ##), so it differs from a decode of one id.
This adds two native symbols (tk_token_to_id, tk_id_to_token), so the prebuilt binaries are rebuilt for this release. The build hook downloads the matching binaries automatically; nothing to install.

Rename the example to example/hf_tokenizers_example.dart so it is found under the package's published name.
Shorten the pubspec description so it fits pub.dev's 180-character guideline.

Initial release.
Tokenizer.fromFile / Tokenizer.fromBytes: load any HuggingFace tokenizer.json, backed by the Rust tokenizers crate over FFI.
encode / decode: byte-exact token ids and round-trip text.
vocabSize, prompt native cleanup via close() plus a finalizer.
Platform: macOS (arm64) in this release; other platforms to follow as the build hook gains prebuilt/CI coverage.