hf_tokenizers 0.1.0 copy "hf_tokenizers: ^0.1.0" to clipboard
hf_tokenizers: ^0.1.0 copied to clipboard

Published 25 days ago • verified publisherdeveloperyusuf.com• Latest: 1.1.0
[unknown platforms]
→

Metadata

HuggingFace tokenizers for Dart over FFI. Load any model's tokenizer.json and get byte-exact BPE, byte-level BPE, WordPiece, and Unigram tokenization. Encode text to token ids and decode them back, ba [...]

More...

0.1.0 #

  • Initial release.
  • Tokenizer.fromFile / Tokenizer.fromBytes: load any HuggingFace tokenizer.json, backed by the Rust tokenizers crate over FFI.
  • encode / decode: byte-exact token ids and round-trip text.
  • vocabSize, prompt native cleanup via close() plus a finalizer.
  • Platform: macOS (arm64) in this release; other platforms to follow as the build hook gains prebuilt/CI coverage.

← Metadata

0
likes
0
points
1.1k
downloads

Publisher

verified publisherdeveloperyusuf.com

Weekly Downloads

Metadata

HuggingFace tokenizers for Dart over FFI. Load any model's tokenizer.json and get byte-exact BPE, byte-level BPE, WordPiece, and Unigram tokenization. Encode text to token ids and decode them back, backed by the Rust tokenizers crate.

Repository (GitHub)
View/report issues

Topics

#llm #ai #tokenizer #ffi #nlp

License

unknown (license)

Dependencies

code_assets, ffi, hooks

More

Packages that depend on hf_tokenizers

Back