Swift Package Index

Packages for keyword “tokenizer”

These packages are available as a package collection, usable in Xcode or SwiftPM.

• swift-tokenizers

  High-performance tokenizers

  • DePasqualeOrg/swift-tokenizers
  • Active 27 days ago
  • 9 stars

• swift-casification

  Package for tokenizing and case swapping for free strings

  • CaptureContext/swift-casification
  • Active about 1 month ago
  • 0 stars
  • Has documentation

• SwiftHighlight

  Fast pure-Swift syntax highlighter: 33 languages, declarative Swift/JSON grammars, 13 themes plus VS Code/TextMate import, incremental highlighting, SwiftUI, UIKit, AppKit, HTML and ANSI output.

  • CreatureSurvive/SwiftHighlight
  • Active 12 days ago
  • 0 stars
  • Has documentation

• kaleidoscope-lexer

  A Lexer Named Kaleidoscope, powered by Swift Macro

  • FlickerSoul/kaleidoscope-lexer
  • Active 8 months ago
  • 7 stars
  • Has documentation

• sudachi-swift

  Swift bindings for the Sudachi Japanese morphological analyzer (sudachi.rs) — on-device tokenization for iOS and macOS via UniFFI

  • iasnezhkov/sudachi-swift
  • Active 14 days ago
  • 0 stars

• GPTEncoder

  Swift BPE Encoder/Decoder for OpenAI GPT Models. A programmatic interface for tokenizing text for OpenAI ChatGPT API.

  • alfianlosari/GPTEncoder
  • Active 3 years ago
  • 85 stars

• TiktokenSwift

  Swift bindings for OpenAI's tiktoken tokenizer using UniFFI. Count tokens, estimate costs, and manage context windows in your iOS and macOS apps

  • narner/TiktokenSwift
  • Active about 1 year ago
  • 27 stars

• swift-tiktoken

  A pure Swift implementation of OpenAI's tiktoken tokenizer

  • DePasqualeOrg/swift-tiktoken
  • Active 10 months ago
  • 6 stars

8 packages.