Chunker class

Divides text streams into manageable chunks for processing.

The Chunker class is responsible for breaking down large text streams into smaller, consistently sized chunks that can be processed by language models or embedding systems. It ensures chunks are properly sized for model context windows and can create overlapping chunks to preserve context across chunk boundaries.

Constructors

Chunker({int chunkSize = 300})
Creates a new Chunker with the specified chunk size.
const

Properties

chunkSize → int
The target size for each chunk in characters
final
hashCode → int
The hash code for this object.
no setterinherited
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited

Methods

noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
toString() → String
A string representation of this object.
inherited
transform(Stream<String> rawFeed) → Stream<Chunk>
Transforms a text stream into non-overlapping chunks of approximately equal size.
transformFile(File file) → Stream<Chunk>
Transforms a file into non-overlapping chunks using Chunky's file stringers.
transformFileWithOverlap(File file, {int overlap = 50}) → Stream<Chunk>
Transforms a file into overlapping chunks using Chunky's file stringers.
transformString(String input) → Stream<Chunk>
Transforms a raw string into non-overlapping chunks.
transformWithOverlap(Stream<String> rawFeed, {int overlap = 50}) → Stream<Chunk>
Transforms a text stream into chunks with overlap between consecutive chunks.

Operators

operator ==(Object other) → bool
The equality operator.
inherited