# Parallelizing huggingface models

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparallelizing-huggingface-models%2F47923%3E%5C
**Category:** DeepSpeed\
**Created:** [July 24, 2023, 1:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparallelizing-huggingface-models%2F47923 "2023-07-24T13:04:18Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![akshatkatoch](https://avatars.discourse-cdn.com/v4/letter/a/ba8739/32.png) [@akshatkatoch](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fakshatkatoch)%5C
**Post date:** [July 24, 2023, 1:04pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fparallelizing-huggingface-models%2F47923%2F1 "2023-07-24T13:04:19Z")

</div>

Hi,

I have 4 gpus each with 24gb of memory and would like to parallelize the gpt neo models (2.7B and 1.3B) and train them on text data I have. I would also like to parallelize and train models like llama that have 7 billion parameters. I am unsure of how to do this in deepspeed since there is a lot of information and not a lot of straightforward implementations. Please let me know of how this can be done.

Thanks