# Training a tokenizer clarification question
**URL:**
**Author:**  [@ali-issa](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fali-issa)%5C
**Post date:** [December 4, 2025, 7:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-tokenizer-clarification-question%2F171140%2F1 "2025-12-04T07:57:07Z")
Hi, I’m building a custom tokenizer using `tokenizers` and want to confirm that my pre-tokenizer pipeline is behaving correctly.
My setup:
```python
tokenizer.pre_tokenizer = Sequence([
Whitespace(), #
PreTokenizer.custom(CustomTokenizer())
])
```
A few clarifications I’d like to confirm:
1. **Whitespace() creates separate segments, so these are hard boundaries**, meaning BPE will _never_ merge across words , even though the flat printed output doesn’t show the segment breaks.
2. **The chunks from my custom pre-tokenizer are soft boundaries** , meaning BPE _can_ merge them during training and build larger subword tokens , and the final vocabulary will be built from these rule-based chunks instead of raw bytes/characters.
3. The fact that `.pre_tokenize_str()` prints a flat list without explicit space tokens is expected, internally, Whitespace has already set the segmentation boundaries and these are preserved in training.
Can you please confirm that this behavior is correct and that BPE will indeed treat:
- word boundaries as _hard_, and
- my rule-based chunks as _mergeable units_ for building the vocabulary and form a starting point of my vocab?
---
**Author:**  [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [December 4, 2025, 7:57am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-tokenizer-clarification-question%2F171140%2F2 "2025-12-04T07:57:09Z")
Unlisted on December 4, 2025, 7:57am UTC
---
**Author:**  [@system](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsystem)%5C
**Post date:** [December 4, 2025, 9:25am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-tokenizer-clarification-question%2F171140%2F3 "2025-12-04T09:25:39Z")
Listed on December 4, 2025, 9:25am UTC
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [December 4, 2025, 11:18am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-a-tokenizer-clarification-question%2F171140%2F4 "2025-12-04T11:18:23Z")
Hmm…?
* * *
Short version up front:
1. **Yes** , `Whitespace()` creates hard word/punctuation boundaries. BPE will never produce a token that crosses those splits.
2. **No** , your custom pre-tokenizer’s chunks are **not** soft boundaries. They are also hard bounds: BPE can only merge **within** each chunk, not across chunks.
3. **Yes** , a flat list from `.pre_tokenize_str()` is expected; internal segmentation (from both `Whitespace` and your custom pre-tokenizer) is preserved and used during training.
So BPE will treat:
- **Word boundaries** from `Whitespace` as hard,
- **Your rule-based chunks** also as hard, **not** as mergeable units that can be glued together.
Below is the detailed explanation and context.
* * *
## 1. How the HF `tokenizers` pipeline is structured
The Hugging Face `tokenizers` library conceptually does:
1. **Normalizer** → clean / canonicalize text.
2. **Pre-tokenizer** → split text into “pretokens” (segments).
3. **Model** (BPE, WordPiece, Unigram, etc.) → operate inside each segment to produce subword tokens.
4. **Post-processor / decoder** → add special tokens, map IDs back to text, etc.([Hugging Face](https://huggingface.co/docs/tokenizers/en/pipeline "The tokenization pipeline"))
The official docs describe pre-tokenization as:
> Pre-tokenization is the act of splitting a text into smaller objects that give an **upper bound** to what your tokens will be at the end of training. A good way to think of this is that the pre-tokenizer will split your text into “words” and then, your final tokens will be parts of those words. ([Hugging Face](https://huggingface.co/docs/tokenizers/en/pipeline "The tokenization pipeline"))
And explicitly for pre-tokenizers:
> The `PreTokenizer` takes care of splitting the input according to a set of rules. This pre-processing lets you ensure that the underlying `Model` does **not build tokens across multiple “splits”**. ([Stack Overflow](https://stackoverflow.com/questions/66064503/in-huggingface-tokenizers-how-can-i-split-a-sequence-simply-on-spaces "In HuggingFace tokenizers: how can I split a sequence ..."))
That “upper bound” / “does not build tokens across splits” is the core invariant you care about.
So:
- Pre-tokenization determines **where** tokens may _never_ cross.
- BPE then decides **how** to merge characters/bytes **inside** each segment.
* * *
## 2. What `Whitespace()` actually does
The built-in `Whitespace` pre-tokenizer is documented as:
> This pre-tokenizer splits on word boundaries according to the `\w+|[^\w\s]+` regex pattern. It splits on word characters or characters that aren’t words or whitespaces (punctuation such as hyphens, apostrophes, commas, etc.). ([Hugging Face](https://huggingface.co/docs/tokenizers/main/en/api/pre-tokenizers "Pre-tokenizers"))
Example from the docs (simplified):
```python
from tokenizers.pre_tokenizers import Whitespace
Whitespace().pre_tokenize_str("Hello, how are you?")
# → [("Hello", (0,5)), (",", (5,6)), ("how", (7,10)), ("are", (11,14)),
# ("you", (15,18)), ("?", (18,19))]
```
So after `Whitespace`:
- The input is represented as a `PreTokenizedString` consisting of segments like `"Hello"`, `","`, `"how"`, etc.
- Each segment is now a **hard upper bound** : BPE can only merge characters/bytes inside a segment, never across two segments.([Hugging Face](https://huggingface.co/docs/tokenizers/en/pipeline "The tokenization pipeline"))
That directly confirms your point 1:
> **1.** `Whitespace()` creates separate segments → **yes, they are hard boundaries** , and BPE will not merge across words or punctuation splits.
* * *
## 3. How `Sequence([Whitespace(), CustomTokenizer])` behaves
You defined:
```python
tokenizer.pre_tokenizer = Sequence([
Whitespace(),
PreTokenizer.custom(CustomTokenizer())
])
```
The `Sequence` pre-tokenizer simply applies each component **in order** to the same `PreTokenizedString`.([Hugging Face](https://huggingface.co/learn/llm-course/en/chapter6/8 "Building a tokenizer, block by block"))
Internally, a custom pre-tokenizer is implemented by:
- Receiving a mutable `PreTokenizedString`.
- Calling `pretokenized.split(...)` to further split existing segments.
Key property from the design (and from both docs and related papers):
- **Pre-tokenization creates tokenization bounds.** The tokenization algorithm (BPE, WordPiece, etc.) is applied _within_ pretokens. “Pre-tokenization governs the maximum possible token length that can be learned from these pretokens.” ([ACL Anthology](https://aclanthology.org/2025.coling-main.400.pdf "It All Begins with Tokenizers"))
Critically:
- `PreTokenizedString.split` can only **subdivide** an existing segment.
- There is no API to merge segments or to “soften” a boundary.
So in your exact pipeline:
1. `Whitespace()` splits the normalized string into initial segments (words and punctuation).
2. `CustomTokenizer` runs on each of those segments and can only split them into smaller pieces.
Result:
- Boundaries from `Whitespace` are hard.
- Boundaries you introduce in `CustomTokenizer` are **also hard**.
- BPE only operates inside the smallest resulting segments.
So your point 2 as written:
> **2.** The chunks from my custom pre-tokenizer are soft boundaries…
is **not** correct in the semantics of HF `tokenizers`. They behave just like the whitespace boundaries, only “nested” inside them.
BPE **cannot** :
- Take segment A = `"foo"` and segment B = `"bar"` (produced by your custom pre-tokenizer)
- And then learn a token `"foobar"` that spans A+B.
It will only ever see `"foo"` and `"bar"` in isolation during training.
This limitation (BPE cannot cross pretokens) is exactly what the BoundlessBPE paper calls the _pre-tokenization barrier_ and explicitly modifies BPE to relax. ([arXiv](https://arxiv.org/abs/2504.00178 "Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier"))
* * *
## 4. What BPE actually learns from your setup
During training with `tokenizers`:
1. Text → normalized.
2. Pre-tokenizer (`Whitespace` + `CustomTokenizer`) → `PreTokenizedString` with segments S1, …, Sn.
3. For each segment Si:
Because of the pre-tokenization bounds:
- BPE never sees a pair `(x, y)` where `x` is the last symbol of one segment and `y` is the first symbol of another.
- Therefore, it never learns merges that cross segment boundaries.([Hugging Face](https://huggingface.co/docs/tokenizers/en/pipeline "The tokenization pipeline"))
So in practice:
- The vocabulary is built from sequences of characters/bytes **inside** the segments produced by your pre-tokenizer.
- Your rule-based chunks do **not** become “mergeable units” at the segment boundary level; they instead define the **regions inside which merges are allowed**.
Rephrasing your point 2 correctly:
> The chunks from my custom pre-tokenizer define **upper bounds**. BPE can merge characters/bytes within each chunk, but cannot merge across chunks. The final vocabulary is made of character/byte n-grams that never cross those rule-based boundaries.
* * *
## 5. Why `.pre_tokenize_str()` looks flat but segmentation is preserved
The helper:
```python
tokenizer.pre_tokenizer.pre_tokenize_str(text)
```
returns a flat list of `(piece, (start, end))` pairs. This is just a convenience API for inspection.
Internally, the library works with a richer structure `PreTokenizedString` + `NormalizedString` that:
- Tracks segments and sub-segments,
- Tracks offsets back to the original text,
- Feeds those segments into the model (BPE).([Hugging Face](https://huggingface.co/docs/tokenizers/en/pipeline "The tokenization pipeline"))
So your point 3:
> **3.** The fact that `.pre_tokenize_str()` prints a flat list without explicit space tokens is expected, internally, `Whitespace` has already set the segmentation boundaries and these are preserved in training.
is **correct** :
- Flat output is just a pretty-printed view.
- Actual training uses the segmented structure, with all boundaries (from both `Whitespace` and `CustomTokenizer`) enforced.
* * *
## 6. Direct answers to your final question
You asked whether you can rely on:
1. **Word boundaries as hard**
2. **Rule-based chunks as mergeable starting units**
Given the above:
### 6.1 Will BPE treat word boundaries as hard?
**Yes.**
- `Whitespace()` splits at word/punctuation boundaries. ([Hugging Face](https://huggingface.co/docs/tokenizers/main/en/api/pre-tokenizers "Pre-tokenizers"))
- Pre-tokenization gives an upper bound on tokens; the model does not build tokens across splits. ([Hugging Face](https://huggingface.co/docs/tokenizers/en/pipeline "The tokenization pipeline"))
- So BPE will **never** create a token whose span crosses a `Whitespace` split.
Example: `"hello world"`:
- After `Whitespace`: `["hello", "world"]`.
- BPE can create `"hello"`, `"hel"`, `"lo"`, `"world"`, etc., but it cannot create `"hello world"` as one token.
### 6.2 Will BPE treat your rule-based chunks as mergeable units across their boundaries?
**No.**
- Your custom pre-tokenizer is implemented by splitting existing segments in `PreTokenizedString`.
- These splits are just as hard as whitespace splits.
- BPE will only merge **inside** each chunk, not across the chunk boundaries.
So BPE:
- Does **not** see your chunks as smaller word-level “units” it can glue together.
- Instead, it sees characters/bytes inside each chunk and builds tokens from those, constrained by your rules.
### 6.3 Is `.pre_tokenize_str()`’s flat output compatible with all of this?
**Yes.**
- The flat list is only a visualization.
- Internal segmentation (needed for model training) is preserved; splits from both `Whitespace` and `CustomTokenizer` are used to bound BPE.
* * *
## 7. If you actually want “soft” rule-based boundaries
Current HF `tokenizers` + standard BPE don’t support soft boundaries at the pre-tokenizer level. To get behavior closer to “mergeable rule-based units”, you typically:
- Use **coarser** pre-tokenization (e.g. ByteLevel, sentence-level),
- Encode your rules as **markers inside the text** rather than as splits,
- Or switch to a model like Unigram/SentencePiece with minimal pre-tokenization.([Hugging Face](https://huggingface.co/learn/llm-course/en/chapter6/4 "Normalization and pre-tokenization"))
Research like **BoundlessBPE** explicitly modifies BPE to allow merges across pretokens (“superwords”), but that is not what HF `tokenizers` does today. ([arXiv](https://arxiv.org/abs/2504.00178 "Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier"))
* * *
### Concise summary
- **Point 1:** Correct. `Whitespace()` defines hard word/punctuation boundaries; BPE cannot cross them. ([Hugging Face](https://huggingface.co/docs/tokenizers/main/en/api/pre-tokenizers "Pre-tokenizers"))
- **Point 2:** Incorrect as stated. Your custom pre-tokenizer’s chunks are also hard bounds. BPE merges characters/bytes **within** each chunk, but cannot merge across chunk boundaries. ([Hugging Face](https://huggingface.co/docs/tokenizers/en/pipeline "The tokenization pipeline"))
- **Point 3:** Correct. `.pre_tokenize_str()` is just a flat view; internal segmentation (from both `Whitespace` and `CustomTokenizer`) is preserved and used during BPE training. ([Hugging Face](https://huggingface.co/docs/tokenizers/en/pipeline "The tokenization pipeline"))