LlamaCppModelParams class

ModelParams for a GGUF model, which llama.cpp runs natively and on the Web through the WebGPU bridge.

It takes only the fields llama.cpp applies, so a LiteRT-LM option cannot be set by mistake, and a load of a LiteRT-LM model with it throws LlamaUnsupportedException. Each field means what it does on ModelParams. useMmap and useMlock apply natively only, and preferMemory64 and modelBytesHint on the Web only; a load elsewhere throws LlamaUnsupportedException naming them.

final engine = await LlamaEngine.load(
  LlamaModel(ModelSource.path('path/to/model.gguf')),
  params: const LlamaCppModelParams(contextSize: 8192, batchSize: 512),
);
Inheritance

Constructors

LlamaCppModelParams({int contextSize = 4096, ComputeDevice device = ComputeDevice.auto, int gpuLayers = maxGpuLayers, GpuBackend preferredBackend = GpuBackend.auto, ModelSplitMode splitMode = ModelSplitMode.layer, int mainGpu = 0, List<LoraAdapterConfig> loras = const [], String? chatTemplate, int numberOfThreads = 0, int numberOfThreadsBatch = 0, int batchSize = 0, int microBatchSize = 0, int maxParallelSequences = 1, int speculativeRollbackTokenMax = 0, bool useMmap = true, bool useMlock = false, bool loadMtp = false, FlashAttention flashAttention = FlashAttention.auto, KvCacheType cacheTypeK = KvCacheType.f16, KvCacheType cacheTypeV = KvCacheType.f16, bool? kvUnified, double? ropeFrequencyBase, double? ropeFrequencyScale, bool? preferMemory64, int? modelBytesHint})
Creates llama.cpp load parameters with ModelParams' defaults.
const

Properties

batchSize → int
Maximum prompt/eval tokens per decode call (n_batch).
finalinherited
cacheTypeK → KvCacheType
llama_context_params.type_k. Non-F16 requires flashAttention enabled.
finalinherited
cacheTypeV → KvCacheType
llama_context_params.type_v. Non-F16 requires flashAttention enabled.
finalinherited
chatTemplate → String?
Optional Jinja chat template that replaces the model's own template.
finalinherited
contextSize → int
Context size (n_ctx) in tokens.
finalinherited
device → ComputeDevice
Device to run the model on, for every runtime.
finalinherited
flashAttention → FlashAttention
llama_context_params.flash_attn_type. User-explicit values override the platform/backend heuristic.
finalinherited
gpuLayers → int
Number of model layers to offload to the GPU (n_gpu_layers).
finalinherited
hashCode → int
The hash code for this object.
no setterinherited
kvUnified → bool?
llama_context_params.kv_unified. null keeps the current heuristic (auto-enabled when maxParallelSequences > 1).
finalinherited
liteRtLmActivationDataType → LiteRtLmActivationDataType?
Native LiteRT-LM activation data type override.
finalinherited
liteRtLmBackend → LiteRtLmBackendPreference
Preferred LiteRT-LM runtime backend for .litertlm models.
finalinherited
liteRtLmCacheDir → String?
Native LiteRT-LM runtime cache directory.
finalinherited
liteRtLmDispatchLibDir → String?
Native LiteRT-LM dispatch library directory for Android NPU deployments.
finalinherited
liteRtLmEmbeddingEngine → bool
Selects LiteRT-LM's separate embedding engine for .litertlm bundles.
finalinherited
liteRtLmMaxProgramCacheBytes → int?
Size cap in bytes for the native LiteRT-LM GPU program cache files of one cache directory.
finalinherited
liteRtLmParallelFileSectionLoading → bool?
Native LiteRT-LM file-section loading override.
finalinherited
liteRtLmPrefillChunkSize → int?
Native LiteRT-LM prefill chunk size for CPU dynamic models.
finalinherited
loadMtp → bool
Whether llama.cpp should load bundled multi-token prediction tensors.
finalinherited
loras → List<LoraAdapterConfig>
Initial LoRA adapters to load along with the model.
finalinherited
mainGpu → int
Primary GPU device index for model loading.
finalinherited
maxParallelSequences → int
Maximum parallel sequence slots in context memory (n_seq_max).
finalinherited
microBatchSize → int
Micro-batch size used by backend schedulers (n_ubatch).
finalinherited
modelBytesHint → int?
Web/WebGPU only: approximate model size in bytes, used to decide whether to load the mem64 core up front (instead of waiting for an out-of-memory failure and retrying). null when unknown. Native llama.cpp and LiteRT-LM reject a non-null value.
finalinherited
numberOfThreads → int
Number of threads to use for generation (n_threads).
finalinherited
numberOfThreadsBatch → int
Number of threads to use for batch processing (n_threads_batch).
finalinherited
preferMemory64 → bool?
Web/WebGPU only: prefer the 64-bit (wasm64/mem64) bridge core.
finalinherited
preferredBackend → GpuBackend
Preferred GPU backend for inference.
finalinherited
ropeFrequencyBase → double?
llama_context_params.rope_freq_base. null keeps the model's trained value.
finalinherited
ropeFrequencyScale → double?
llama_context_params.rope_freq_scale. null keeps the model's trained value.
finalinherited
runtimeType → Type
A representation of the runtime type of the object.
no setterinherited
speculativeRollbackTokenMax → int
llama.cpp recurrent-state rollback snapshots per sequence (n_rs_seq).
finalinherited
splitMode → ModelSplitMode
Model tensor distribution strategy across GPU devices.
finalinherited
useMlock → bool
Whether llama.cpp should lock model weights in memory. Default false.
finalinherited
useMmap → bool
Whether llama.cpp should memory-map model weights. Default true.
finalinherited

Methods

copyWith({bool? liteRtLmEmbeddingEngine, int? contextSize, ComputeDevice? device, int? gpuLayers, GpuBackend? preferredBackend, LiteRtLmBackendPreference? liteRtLmBackend, LiteRtLmActivationDataType? liteRtLmActivationDataType, bool clearLiteRtLmActivationDataType = false, int? liteRtLmPrefillChunkSize, bool clearLiteRtLmPrefillChunkSize = false, bool? liteRtLmParallelFileSectionLoading, bool clearLiteRtLmParallelFileSectionLoading = false, String? liteRtLmDispatchLibDir, bool clearLiteRtLmDispatchLibDir = false, String? liteRtLmCacheDir, bool clearLiteRtLmCacheDir = false, int? liteRtLmMaxProgramCacheBytes, bool clearLiteRtLmMaxProgramCacheBytes = false, ModelSplitMode? splitMode, int? mainGpu, List<LoraAdapterConfig>? loras, String? chatTemplate, bool clearChatTemplate = false, int? numberOfThreads, int? numberOfThreadsBatch, int? batchSize, int? microBatchSize, int? maxParallelSequences, int? speculativeRollbackTokenMax, bool? useMmap, bool? useMlock, bool? loadMtp, FlashAttention? flashAttention, KvCacheType? cacheTypeK, KvCacheType? cacheTypeV, bool? kvUnified, bool clearKvUnified = false, double? ropeFrequencyBase, bool clearRopeFrequencyBase = false, double? ropeFrequencyScale, bool clearRopeFrequencyScale = false, bool? preferMemory64, bool clearPreferMemory64 = false, int? modelBytesHint, bool clearModelBytesHint = false}) → LlamaCppModelParams
Creates a copy with updated fields, as ModelParams.copyWith does.
override
noSuchMethod(Invocation invocation) → dynamic
Invoked when a nonexistent method or property is accessed.
inherited
toString() → String
A string representation of this object.
inherited
validate() → void
Validates the parameter combination. Throws LlamaArgumentException when a value is out of range or the combination is incompatible with llama.cpp (a non-F16 KV cache requires flashAttention != disabled), or when device contradicts another field:
inherited

Operators

operator ==(Object other) → bool
The equality operator.
inherited