# Issues with BPE tokenizer

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissues-with-bpe-tokenizer%2F70381%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [January 24, 2024, 8:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissues-with-bpe-tokenizer%2F70381 "2024-01-24T08:49:57Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Imran1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/imran1/32/18779_2.png) [@Imran1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FImran1)%5C
**Post date:** [January 24, 2024, 8:49am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissues-with-bpe-tokenizer%2F70381%2F1 "2024-01-24T08:49:57Z")

</div>

I attempted to train the BPE Hugging Face tokenizer for Pashto from scratch, but it is not decoding words correctly.

---

<div class="post-metadata">

**Author:** ![malikarumi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/malikarumi/32/31844_2.png) [@malikarumi](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fmalikarumi)%5C
**Post date:** [January 24, 2024, 3:56pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissues-with-bpe-tokenizer%2F70381%2F2 "2024-01-24T15:56:05Z")

</div>

No one can help you with such limited information.

1. I am unfamiliar with Pashto. Does the official documentation say anything about BPE?
2. Is it possible that Pashto has already been pre trained with a different tokenizer?
3. Post your code so we can understand what the software is trying to tell you.

---

<div class="post-metadata">

**Author:** ![Imran1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/imran1/32/18779_2.png) [@Imran1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FImran1)%5C
**Post date:** [January 24, 2024, 4:09pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fissues-with-bpe-tokenizer%2F70381%2F3 "2024-01-24T16:09:38Z")

</div>

I solve the issues. 🤢
