distillFrom method

Implementation

Future<Chunk> distillFrom({
  required String header,
  required List<Chunk> chunks,
  int index = 0,
  bool summarize = false,
}) async => Chunk(
  index: index,
  lod: chunks.first.lod + 1,
  down: chunks.map((i) => i.index).toList(),
  content: await distill(
    input: chunks
        .mapIndexed(
          (i, index) =>
              index == chunks.length - 1 ? i.fullContent : i.content,
        )
        .join(""),
    header: header,
    prompt:
        summarize
            ? """
Your job is to summarize all of the information in the content provided into a clean concise representation of the information.
* Find the overall meaning and distill it into a few paragraphs or less.
* Only output the response, this is not a chat, do not include any extra commentary or formatting.
* Your output should be about $targetOutputSize characters.
        """.trim()
            : """
Your job is to distill all of the information in the content provided into a clean concise representation of the information without any loss of meaning.
* Strip out formatting, special characters, and other non-essential information. I.e. strip out json keys and just represent the information in written way.
* If the content has a lot of diacritics (OCR errors), do your best to make sense of them or correct them, if unsure, keep them as they are
* Only output the response, this is not a chat, do not include any extra commentary or formatting.
* Your output should be about $targetOutputSize characters.
        """.trim(),
  ),
);