# Reproduce RoBERTa Using Huggingface Transformers

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproduce-roberta-using-huggingface-transformers%2F48531%3E%5C
**Category:** 🤗Transformers\
**Created:** [July 28, 2023, 3:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproduce-roberta-using-huggingface-transformers%2F48531 "2023-07-28T15:30:01Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![JackBAI](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jackbai/32/17827_2.png) [@JackBAI](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJackBAI)%5C
**Post date:** [July 28, 2023, 3:30pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Freproduce-roberta-using-huggingface-transformers%2F48531%2F1 "2023-07-28T15:30:01Z")

</div>

I’m trying to reproduce RoBERTa pre-training results using `run_mlm.py` provided by huggingface `transformers`. However, I’m confused how exactly should the script be called. Here is my script:

```bash
python transformers/examples/pytorch/language-modeling/run_mlm.py \
    --config_name roberta-base \
    --tokenizer_name roberta-base \
    --dataset_name wikitext, bookcorpus, ccnews, openwebtext, stories \
    --dataset_config_name wikitext-2-raw-v1 \
    --per_device_train_batch_size 8 \
    --per_device_eval_batch_size 8 \
    --do_train \
    --do_eval \
    --output_dir ./crate/ckpt \
    --overwrite_output_dir

```

Is there anyone already successfully reproduced RoBERTa by this means? Looking forward to any possible help!