Interface TokenTextSplitterParams

interface TokenTextSplitterParams {
    allowedSpecial: string[] | "all";
    chunkOverlap: number;
    chunkSize: number;
    disallowedSpecial: string[] | "all";
    encodingName: TiktokenEncoding;
    keepSeparator: boolean;
    lengthFunction?: ((text) => number) | ((text) => Promise<number>);
}

Hierarchy (view full)

Implemented by

Properties

allowedSpecial chunkOverlap chunkSize disallowedSpecial encodingName keepSeparator lengthFunction?

Properties

allowedSpecial

allowedSpecial: string[] | "all"

chunkOverlap

chunkOverlap: number

chunkSize

chunkSize: number

disallowedSpecial

disallowedSpecial: string[] | "all"

encodingName

encodingName: TiktokenEncoding

keepSeparator

keepSeparator: boolean

Optional lengthFunction

lengthFunction?: ((text) => number) | ((text) => Promise<number>)

Type declaration

    • (text): number

    • Parameters

      • text: string

      Returns number

Type declaration

    • (text): Promise<number>

    • Parameters

      • text: string

      Returns Promise<number>

Settings

Member Visibility

Theme

On This Page

Generated using TypeDoc