Request: Add vLLM Support for This Model
#12
by
kira
- opened
Hi team!
Thank you for creating and sharing this amazing model with the community.
I'm interested in using this model with vLLM for improved inference performance and efficiency. Would it be possible to add official vLLM support for this model? This would help users leverage vLLM's continuous batching and other optimizations.
If you're open to this, I'd be happy to help test or provide any additional information needed. Looking forward to your thoughts on this integration.
thanks
Thank you for your attention to our model. Actually, we had an implementation based on vLLM, but due to the outdated version of vLLM used, we cannot directly merge it. We will complete the code reorganization and submission as soon as possible.