# Text preprocessing for fitting Tokenizer model

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftext-preprocessing-for-fitting-tokenizer-model%2F24869%3E%5C
**Category:** 🤗Tokenizers\
**Created:** [October 24, 2022, 7:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftext-preprocessing-for-fitting-tokenizer-model%2F24869 "2022-10-24T07:15:42Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![antoine2323231](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@antoine2323231](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fantoine2323231)%5C
**Post date:** [October 24, 2022, 7:15am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftext-preprocessing-for-fitting-tokenizer-model%2F24869%2F1 "2022-10-24T07:15:42Z")

</div>

Hello 🙂 I have read that when preprocessing text it is best practice to remove stop words, remove special characters and punctuation, to end up only with list of words. My question is: If the original text I want my tokenizer to be fitted on is a text containing a lot of statistics (hence a lot of % = / etc…) but also some texts, then it makes sense to keep special characters and numbers as input to the tokenizer model? Or it should be removed in any case as a tokenizer can only understand words?  
Thanks a lot in advance 🙂

---

<div class="post-metadata">

**Author:** ![lianghsun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lianghsun/32/57774_2.png) [@lianghsun](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Flianghsun)%5C
**Post date:** [October 25, 2022, 7:47pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftext-preprocessing-for-fitting-tokenizer-model%2F24869%2F2 "2022-10-25T19:47:51Z")

</div>

I think this question may depends on what kind of task you might want to deal with. In briefly, as my opinion, I think you should keep those symbol in your vocab, because those symbol needs to be the _element_ in your context, no matter what kind of task you might to deal with.