# Model's evaluation in DDP training is using only one GPU

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodels-evaluation-in-ddp-training-is-using-only-one-gpu%2F29295%3E%5C
**Category:** Beginners\
**Created:** [January 10, 2023, 12:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodels-evaluation-in-ddp-training-is-using-only-one-gpu%2F29295 "2023-01-10T12:49:31Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![sararb](https://avatars.discourse-cdn.com/v4/letter/s/258eb7/32.png) [@sararb](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsararb)%5C
**Post date:** [January 10, 2023, 12:49pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodels-evaluation-in-ddp-training-is-using-only-one-gpu%2F29295%2F1 "2023-01-10T12:49:31Z")

</div>

Hello! 😃  
I am training an HF model with torch DDP using the following command line:

`python -m torch.distributed.launch --nproc_per_node 2 my_script.py --{arguments}`

I noticed that while training was using the two available GPUs, the evaluation step was done only on a single GPU. After checking the source code, it seems that [\> here](https://github.com/huggingface/transformers/blob/a3c37825cc1e305dde63455b5f321586e6d29e07/src/transformers/trainer.py#L1360) the model is not wrapped inside the DDP when `training==False`.

Is it expected that only one GPU will be used during the evaluation step? If yes, could you explain why the DDP cannot be used for the evaluation as well?

---

<div class="post-metadata">

**Author:** ![zuzannad1](https://avatars.discourse-cdn.com/v4/letter/z/dc4da7/32.png) [@zuzannad1](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fzuzannad1)%5C
**Post date:** [September 14, 2023, 9:02am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fmodels-evaluation-in-ddp-training-is-using-only-one-gpu%2F29295%2F2 "2023-09-14T09:02:14Z")

</div>

Have you figured out how to use multiple GPUs for the eval loop during training? Am facing the same issue.