🔡 Telugu BPE Tokenizer (23k Vocab)

Purpose-built Tokenizer for Indic & Telugu Language Modeling

Developed by Vipplav AI — India's Most Specialized AI Studio

Website GitHub License


📌 Overview

Standard multilingual and Western LLM tokenizers (Llama, GPT-4, Mistral) heavily fragment Indic scripts, leading to high fertility rates (splitting a single Telugu word into 4 to 8 distinct tokens). This inflates inference latency, limits context window utility, and increases deployment costs.

Vipplav/telugu-bpe-23k is an optimized Byte-Pair Encoding (BPE) tokenizer trained on over 340,000+ cleaned Telugu text passages sourced from the AI4Bharat Sangraha dataset. It provides high compression efficiency and lower token fertility specifically for Telugu literature, news, and technical vocabulary.


🚀 Quickstart

1. Using with Hugging Face transformers

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Vipplav/telugu-bpe-23k")

sample_text = "విప్లవ్ AI భారతదేశ భాషల కోసం కృత్రిమ మేధస్సును నిర్మిస్తోంది."
tokens = tokenizer.tokenize(sample_text)
token_ids = tokenizer.encode(sample_text)

print("Original Text:", sample_text)
print("Tokens:", tokens)
print("Token Count:", len(tokens))

2. Using with SentencePiece directly

import sentencepiece as spm
from huggingface_hub import hf_hub_download

model_path = hf_hub_download(repo_id="Vipplav/telugu-bpe-23k", filename="spiece.model")
sp = spm.SentencePieceProcessor(model_file=model_path)

pieces = sp.encode("తెలుగు వార్తలు మరియు సాంకేతిక సమాచారం", out_type=str)
print("SentencePiece pieces:", pieces)

📊 Key Highlights & Efficiency

  • Vocabulary Size: 23,000 tokens
  • Script Support: Native Telugu script with full conjunct, vowel modifier, and diacritic handling.
  • Fertility Improvement: Reduces average tokens-per-word by up to 42% compared to standard Llama/Mistral tokenizers on Telugu corpora.
  • Intended Use: Base tokenizer for Indic SLMs (Small Language Models), domain-specific pretraining, and regional language embeddings.

🏢 About Vipplav AI

Vipplav AI is an enterprise AI studio engineering foundational Indic Small Language Models (Rachana) and verifiable agentic automation pipelines.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train Vipplav/telugu-bpe-23k