Byte-Pair Encoding & Special Token Injection
Learn how modern tokenizers (TikToken, SentencePiece) split tokens, compress bytes, and why prompt injection often hides in special tokens.
- •Understand Byte-Pair Encoding merge tables and compression ratios
- •Recognize how token boundary shifts break regexes and numerical logic
- •Safeguard prompt delimiters against control-character injections
The Subword Illusion & BPE Mechanics
LLMs do not read words, letters, or syllables. They operate strictly on sequences of discrete token IDs mapped to dense embedding tensors. The Byte-Pair Encoding (BPE) algorithm initializes a vocabulary with 256 raw byte tokens (0x00 through 0xFF), then iteratively computes pairwise frequencies and greedily merges the most frequent byte pair until reaching the target vocabulary capacity (e.g., 32,768 for Llama-2, 128,256 for Llama-3).
Byte-Pair Encoding Iterative Merge Pipeline
Convert Unicode characters into 256 fundamental byte representations
Segment text into words, contractions, and 1-3 digit numbers to prevent punctuation contamination
Count co-occurrence frequencies of adjacent symbol pairs across text corpus
Merge highest frequency pair (e.g. 'th' + 'e' -> 'the') and assign new token ID
Stop once vocabulary target is satisfied and serialize rank table
Why can't LLMs spell 'strawberry' reliably? Because the tokenizer represents 'strawberry' as ['str', 'aw', 'berry'] rather than individual characters.
Production Byte-Level Tokenizer Implementation Suite
Inspect the complete reference implementation of a Byte-Level BPE tokenizer: from raw byte dictionary training and Llama-3 regex pre-tokenization to special token sanitization.
from collections import Counter
from typing import Dict, List, Tuple
class BytePairEncoder:
"""Trains a byte-level BPE tokenizer matching OpenAI / Meta tokenizer specs."""
def __init__(self, target_vocab_size: int = 512):
self.target_vocab_size = target_vocab_size
self.vocab: Dict[int, bytes] = {i: bytes([i]) for i in range(256)}
self.merges: Dict[Tuple[int, int], int] = {}
def get_stats(self, token_ids_list: List[List[int]]) -> Counter:
pairs = Counter()
for tokens in token_ids_list:
for pair in zip(tokens, tokens[1:]):
pairs[pair] += 1
return pairs
def train(self, corpus: str):
# 1. Convert text to raw UTF-8 byte token IDs
words = corpus.split()
token_ids_list = [list(word.encode("utf-8")) for word in words]
next_token_id = 256
num_merges = self.target_vocab_size - 256
for step in range(num_merges):
stats = self.get_stats(token_ids_list)
if not stats:
break
best_pair, freq = stats.most_common(1)[0]
if freq < 2:
break
# Register new merged token
self.merges[best_pair] = next_token_id
self.vocab[next_token_id] = self.vocab[best_pair[0]] + self.vocab[best_pair[1]]
# Replace occurrences across corpus
new_token_ids_list = []
for tokens in token_ids_list:
i = 0
new_tokens = []
while i < len(tokens):
if i < len(tokens) - 1 and (tokens[i], tokens[i+1]) == best_pair:
new_tokens.append(next_token_id)
i += 2
else:
new_tokens.append(tokens[i])
i += 1
new_token_ids_list.append(new_tokens)
token_ids_list = new_token_ids_list
next_token_id += 1
print(f"BPE Training complete: {len(self.vocab)} tokens in vocabulary.")Run this lesson's Triton kernels and multi-file code directly in an ephemeral pre-configured PyTorch 2.5 container.