# Byte Level Tokenizer While Training

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbyte-level-tokenizer-while-training%2F131009%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [December 14, 2024, 1:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbyte-level-tokenizer-while-training%2F131009 "2024-12-14T01:27:23Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![dinesh-bk](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dinesh-bk/32/37524_2.png) [@dinesh-bk](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdinesh-bk)%5C
**Post date:** [December 14, 2024, 1:27am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fbyte-level-tokenizer-while-training%2F131009%2F1 "2024-12-14T01:27:23Z")

</div>

Hi, I have trained the tokenizer using the model BPE and pre tokenizer as ByteLevel `tokenizer = Tokenizer(models.BPE(unk_token="[UNK]")) tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel() tokenizer.decoder = decoders.ByteLevel() `  
Now, my vocabulary is saved in bytes and tokenizer.tokenize give me output in bytes too which is obvious.  
`tokenizer.tokenize` output is

```auto
['Ġन',
 'à¥ĩ',
 'प',
 'ा',
 'ल',
 'à¥Ģ',
 'Ġà¤Ń',
 'ा',
 'ष',
 'ा',
 'म',
 'ा',
 'Ġय',
 'à¥ĭ',
 'Ġà¤ıà¤ķ',
 'Ġà¤īद',
 'ा',
 'हरण',
 'Ġह',
 'à¥ĭ।']

```

. Is there way to save my vocabulary in unicode character rather than bytes and show tokens in unicode characters too?