Prompt with <image> in the prompt leads to 'RuntimeError: CUDA error: device-side assert triggered'
Not an issue but good to know in case anyone runs into this issue. In other multimodal models an image token placeholder is needed in the prompt so it could come up when using those same prompts on this model as it was my case. Once I remove the <image> placeholder from the prompt, the issue when away.
I think this is because the processor for paligemma adds image tokens on its own. This is kind of inconvenient for when you want to input multiple images
Hi @vicaran ,
Sorry for late response, The issue arises when using a placeholder like <image> in the prompt for a multimodal model, but the model may not be handling the image input the way you expect. Kindly remove the <image> placeholder and passing the image through the expected input format should allow the model to function without triggering the CUDA error.
Thank you.