ModernBioBERT

A modern variant of BioBERT based on ModernBERT. We continued the masked language modeling pre-training task for 1.000.000 steps on PubMed abstracts.

Pre-Training Details

Batchsize: 512
Learningrate: 1e-4
Warmupsteps: 500
Learning Rate Scheduler: Cosine Schedule
Max. Sequence Length: 512
Precision: bfloat16

---

datasets: - ncbi/pubmed language: - en base_model: - answerdotai/ModernBERT-base