metadata

language:
  - en
license: apache-2.0
tags:
  - generated_from_trainer
datasets:
  - postbot/multi-emails-hq
metrics:
  - accuracy
pipeline_tag: fill-mask
widget:
  - text: Can you please send me the [MASK] by the end of the day?
    example_title: end of day
  - text: >-
      I hope this email finds you well. I wanted to follow up on our [MASK]
      yesterday.
    example_title: follow-up
  - text: The meeting has been rescheduled to [MASK].
    example_title: reschedule
  - text: Please let me know if you need any further [MASK] regarding the project.
    example_title: further help
  - text: >-
      I appreciate your prompt response to my previous email. Can you provide an
      update on the [MASK] by tomorrow?
    example_title: provide update
  - text: Paris is the [MASK] of France.
    example_title: paris (default)
  - text: The goal of life is [MASK].
    example_title: goal of life (default)
base_model: google/bert_uncased_L-4_H-128_A-2
model-index:
  - name: bert_uncased_L-4_H-128_A-2-mlm-multi-emails-hq
    results: []

bert_uncased_L-4_H-128_A-2-mlm-multi-emails-hq

This model is a fine-tuned version of google/bert_uncased_L-4_H-128_A-2 on the None dataset. It achieves the following results on the evaluation set:

Loss: 2.8524
Accuracy: 0.5077

Model description

Double the layers of BERT-tiny, fine-tuned on email data for eight epochs.

Intended uses & limitations

This is primarily an example/test

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0003
train_batch_size: 8
eval_batch_size: 2
seed: 42
distributed_type: multi-GPU
gradient_accumulation_steps: 16
total_train_batch_size: 128
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.05
num_epochs: 8.0
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
3.5477	0.99	141	3.2637	0.4551
3.3307	1.99	282	3.0873	0.4785
3.252	2.99	423	2.9842	0.4911
3.1415	3.99	564	2.9230	0.4995
3.0903	4.99	705	2.8625	0.5070
3.0996	5.99	846	2.8615	0.5087
3.0641	6.99	987	2.8407	0.5120
3.0514	7.99	1128	2.8524	0.5077

Framework versions

Transformers 4.27.0.dev0
Pytorch 2.0.0.dev20230129+cu118
Datasets 2.8.0
Tokenizers 0.13.1