# EncoderDecoderModel generate text for a ViT as encoder

**URL:** </static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fencoderdecodermodel-generate-text-for-a-vit-as-encoder%2F12332%3E%5C
**Category:** 🤗Transformers\
**Created:** [November 27, 2021, 9:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fencoderdecodermodel-generate-text-for-a-vit-as-encoder%2F12332 "2021-11-27T09:40:10Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![sachin](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sachin/32/26534_2.png) [@sachin](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2Fsachin)%5C
**Post date:** [November 27, 2021, 9:40am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fencoderdecodermodel-generate-text-for-a-vit-as-encoder%2F12332%2F1 "2021-11-27T09:40:10Z")

</div>

I was wondering if there was a way to do `generate` for a ViT to GPT2 `EncoderDecoderModel` . I managed to figure out how to get the loss by using the outputs of the ViT and pushing in the `encoder_outputs` into the model as shown below. However, it seems that for `generate` it is explicitly expecting `inputs_ids` . I’m fairly certain that somewhere under the hood all you need is just the `encoder_outputs` (and `inputs_ids` is unnecessary in that case). Is there a way to do this?

Also I realise that there is a `VisionEncoderDecoderModel` but I am trying to do this as a learning exercise.

vit2gpt2 = EncoderDecoderModel.from\_encoder\_decoder\_pretrained(VIT\_MODEL, DISTIL\_GPT2) tokenized\_captions = gpt2\_tokenizer\_fn(captions) labels = tokenized\_captions[“input\_ids”].clone() labels[tokenized\_captions[“attention\_mask”]==0] = LABEL\_MASK encoder\_outputs = vit2gpt2.encoder(pixel\_values=images) outputs = vit2gpt2( encoder\_outputs=encoder\_outputs, decoder\_input\_ids=tokenized\_captions[“input\_ids”], decoder\_attention\_mask=tokenized\_captions[“attention\_mask”], labels=labels, return\_dict=True, )

[Here](https://www.kaggle.com/sachin/vit-to-gpt2-encoder-decoder-model) is a kaggle kernel to a runnable version of above snippet.

## Update 1:

So I’ve managed to narrow down my necessary search down to `generation_utils.py` but I cannot see where in there it loops over the predicted values and feeds it back into the model. I’m hoping to replicate the process from there.