Byte-Pair Encoding Tokenizer

Every model you have used was trained on the output of an algorithm like this one, and almost nobody who uses tokenizers has implemented one.

Implement learn_bpe(corpus, num_merges):

• corpus is a list of words
• Each word starts as a tuple of its characters plus the end-of-word marker "</w>", so "low" becomes ("l", "o", "w", "</w>")
• Repeatedly find the most frequent adjacent symbol pair across the whole corpus (weighted by how often each word occurs) and merge it into one symbol
• Do this num_merges times, or stop early when no pair occurs more than once
• Break frequency ties by choosing the lexicographically smallest pair, so the result is deterministic

Return the ordered list of merges as (first, second) tuples.

The end-of-word marker is what lets the tokenizer distinguish "est" inside "estimate" from "est" ending "lowest": without it, subwords collapse across boundaries that matter.