# Using HuggingFace Tokenizers Without Special Characters
**URL:**
**Author:**  [@dotan1111](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdotan1111)%5C
**Post date:** [August 23, 2022, 7:20am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-huggingface-tokenizers-without-special-characters%2F21962%2F1 "2022-08-23T07:20:28Z")
Hey,
I would like to use HuggingFace Tokenizers for a unique dataset which doesnât require any special characters. Thus, the results vocabulary should consist only characters from the input file / files. For example, if my file contains the sentence:
âAAABBBCCCâ
The vocabulary should consist words with the letters: âAâ, âBâ and âCâ only.
My intentions are to run the following tokenizers: BPE, sentence piece and word piece. I looked at the base code but couldnât find the parameter to do so.
---
**Author:**  [@lianghsun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flianghsun)%5C
**Post date:** [October 27, 2022, 3:35am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-huggingface-tokenizers-without-special-characters%2F21962%2F2 "2022-10-27T03:35:05Z")
Hey @dotan1111, have you solved this question yet?
HF Tokenizers provide several `models` and `trainers` to learn how to tokenize the sentence to smaller piece. In modern way, it often split the sentence to _subwords_, and let tokenizer use certain model to learn how merge them to specific token. So, the question you ask is how to tokenize the sentence âAAABBBCCCâ to the letters: âAâ, âBâ and âCâ only, without any merges. However, the models in HF Tokenizers often learn to build a vocabulary with letters and merged pattern, and I believe you already know this. Here, Iâll provide several ways for you to build up a âchar-levelâ tokenizer using BPE.
1. BPE with defined minimum merged frequency
```auto
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
# Data
file = ['/path/to/AAABBBCCC.txt'] # AAABBBCCC only
# Use BPE model
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
# Setup BPE trainer
trainer = BpeTrainer(
min_frequency=1_000_000_000 # We set min_freq to large threshold here!
)
tokenizer.train(file, trainer)
# Show vocab list
tokernizer.get_vocab()
# > {'A': 0, 'B': 1, 'C': 2}
```
1. Pythonic way
```auto
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
# Context from Char.txt
# AAAABBB
# CCCCDDDD
# EEEFF
# GGHHHHH
# Get unique letters
letters = set()
with open('/path/to/Char.txt', 'r') as f:
for line in f.readlines():
for char in [*line.rstrip()]:
letters.add(char)
print(letters)
# Build your own vocab.json
import json
vocab = dict()
with open('/path/to/char_vocab.json', 'w') as f:
for idx, item in enumerate(letters):
vocab[item] = idx
json.dump(vocab, f, indent = 6)
# Build your own mergs.txt: please create an empty file and named it `char_merges.txt`
# Init a new tokenizer
new_tokenizer = Tokenizer(BPE.from_file('/path/to/char_vocab.json', '/path/to/char_merges.txt'))
new_tokenizer.get_vocab()
# > {'C': 7, 'F': 4, 'G': 3, 'E': 2, 'A': 0, 'B': 6, 'H': 5, 'D': 1}
```
I wouldnât say which method is the best way to implement char-level tokenizer đ Hope this solve your question. đ¤
---
**Author:**  [@dotan1111](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdotan1111)%5C
**Post date:** [November 2, 2022, 6:50am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fusing-huggingface-tokenizers-without-special-characters%2F21962%2F3 "2022-11-02T06:50:50Z")
Thanks! I have used a similar approach