# Clarifying multi-GPU memory usage

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fclarifying-multi-gpu-memory-usage%2F1905%3E%5C
**Category:** Beginners\
**Created:** [November 5, 2020, 6:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fclarifying-multi-gpu-memory-usage%2F1905 "2020-11-05T18:20:01Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![deppen8](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/deppen8/32/516_2.png) [@deppen8](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fdeppen8)%5C
**Post date:** [November 5, 2020, 6:20pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fclarifying-multi-gpu-memory-usage%2F1905%2F1 "2020-11-05T18:20:01Z")

</div>

Am I reading this thread ([Training using multiple GPUs](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Ftraining-using-multiple-gpus%2F1279)) correctly? I interpret that to mean:

> Training a model with batch size 16 on one GPU is equivalent to running a model with batch size 4 on 4 GPUs

Is that correct? And does it differ between `DataParallel` and `DistributedDataParallel` modes?

---

<div class="post-metadata">

**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsgugger)%5C
**Post date:** [November 5, 2020, 10:22pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fclarifying-multi-gpu-memory-usage%2F1905%2F2 "2020-11-05T22:22:55Z")

</div>

It is correct. The difference between `DataParallel` and `DistributedDataParallel` is (in your current example):

- in `DataParallel` mode you have to set the batch size to 16 for your data loaders.
- in `DistributedDataParallel` you have to set the batch size to 4 for your data loaders.