Set tokenizer "model_max_length" property to 8192
#39
by
NohTow
- opened
Somehow composer exported model_max_length tokenizer property to a very huge value instead of 8192.
This breaks the tokenizer.model_max_length call that some pipelines rely on.
As we corrected max_pos_embeddings, I suggest we also fix this for consistency, although this is not an hard limit.
See this issue for more information.