hf_tokenizers 0.1.1 copy "hf_tokenizers: ^0.1.1" to clipboard
hf_tokenizers: ^0.1.1 copied to clipboard

Published 24 days ago • verified publisherdeveloperyusuf.com• Latest: 1.1.0
[unknown platforms]
→

Metadata

HuggingFace tokenizers for Dart over FFI. Load any tokenizer.json and get byte-exact BPE, WordPiece, and Unigram encoding, backed by the Rust crate.

More...

0.1.1 #

  • Rename the example to example/hf_tokenizers_example.dart so it is found under the package's published name.
  • Shorten the pubspec description so it fits pub.dev's 180-character guideline.

0.1.0 #

  • Initial release.
  • Tokenizer.fromFile / Tokenizer.fromBytes: load any HuggingFace tokenizer.json, backed by the Rust tokenizers crate over FFI.
  • encode / decode: byte-exact token ids and round-trip text.
  • vocabSize, prompt native cleanup via close() plus a finalizer.
  • Platform: macOS (arm64) in this release; other platforms to follow as the build hook gains prebuilt/CI coverage.

← Metadata

0
likes
0
points
1.06k
downloads

Publisher

verified publisherdeveloperyusuf.com

Weekly Downloads

Metadata

HuggingFace tokenizers for Dart over FFI. Load any tokenizer.json and get byte-exact BPE, WordPiece, and Unigram encoding, backed by the Rust crate.

Repository (GitHub)
View/report issues

Topics

#llm #ai #tokenizer #ffi #nlp

License

unknown (license)

Dependencies

code_assets, ffi, hooks

More

Packages that depend on hf_tokenizers

Back