metadata

license: llama3.2
language:
  - zh
  - en
  - it
  - de
  - fr
  - ja
  - ko
base_model:
  - meta-llama/Llama-3.2-3B
  - lianghsun/Llama-3.2-Taiwan-3B
datasets:
  - lianghsun/tw-emergency-medicine-bench
  - lianghsun/tw-legal-nlp
  - lianghsun/tw-legal-synthetic-qa
  - lianghsun/tw-law-article-qa
  - lianghsun/tw-judgment-qa
  - lianghsun/tw-judgment-gist-chat
  - lianghsun/tw-bar-examination-2020-chat
  - lianghsun/tw-structured-law-article
  - lianghsun/tw-judgment-gist-chat
  - lianghsun/tw-contract-review-chat
  - lianghsun/reasoning-base-20k-chat
  - lianghsun/vulnerability-mitigation-qa-zh_tw
  - lianghsun/tw-instruct
  - rombodawg/Everything_Instruct_Multilingual
  - xzuyn/manythings-translations-alpaca
  - neural-bridge/rag-dataset-12000
  - minyichen/glaive_toolcall_zh_tw
pipeline_tag: text-generation
library_name: transformers
tags:
  - Taiwan
  - ROC
  - zh-tw
  - instruct
  - chat
  - llama3.2
  - SLM
model-index:
  - name: Llama-3.2-Taiwan-3B-Instruct
    results:
      - task:
          type: text-generation
          name: Single Choice Question
        dataset:
          type: lianghsun/tw-legal-benchmark-v1
          name: tw-legal-benchmark-v1
        metrics:
          - name: single choice
            type: accuracy
            value: 31.1
      - task:
          type: text-generation
          name: Single Choice Question
        dataset:
          type: lianghsun/Formosa-bench
          name: (Society) Formosa Taiwan Knowledge Bench
          config: society
          split: test
          revision: v2024.11.27
        metrics:
          - name: single choice
            type: accuracy
            value: 60.42
      - task:
          type: text-generation
          name: Single Choice Question
        dataset:
          type: lianghsun/Formosa-bench
          name: (Governmnt) Formosa Taiwan Knowledge Bench
          config: governmnt
          split: test
          revision: v2024.11.27
        metrics:
          - name: single choice
            type: accuracy
            value: 44.25
      - task:
          type: text-generation
          name: Single Choice Question
        dataset:
          type: lianghsun/Formosa-bench
          name: (Geography) Formosa Taiwan Knowledge Bench
          config: geography
          split: test
          revision: v2024.11.27
        metrics:
          - name: single choice
            type: accuracy
            value: 47.54
      - task:
          type: text-generation
          name: Single Choice Question
        dataset:
          type: lianghsun/Formosa-bench
          name: (History) Formosa Taiwan Knowledge Bench
          config: history
          split: test
          revision: v2024.11.27
        metrics:
          - name: single choice
            type: accuracy
            value: 60
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (geography_of_taiwan) tmmlu++
          config: geography_of_taiwan
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 36.2
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (dentistry) tmmlu++
          config: dentistry
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 33.83
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (technical) tmmlu++
          config: technical
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 35.07
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (statistics_and_machine_learning) tmmlu++
          config: statistics_and_machine_learning
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 28.57
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (clinical_psychology) tmmlu++
          config: clinical_psychology
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 29.6
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (tve_design) tmmlu++
          config: tve_design
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 38.54
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (three_principles_of_people) tmmlu++
          config: three_principles_of_people
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 48.2
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (introduction_to_law) tmmlu++
          config: introduction_to_law
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 29.96
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (linear_algebra) tmmlu++
          config: linear_algebra
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 21.43
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (agriculture) tmmlu++
          config: agriculture
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 24.5
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (jce_humanities) tmmlu++
          config: jce_humanities
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 38.89
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (music) tmmlu++
          config: music
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 25.9
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (secondary_physics) tmmlu++
          config: secondary_physics
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 33.04
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (physics) tmmlu++
          config: physics
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 27.84
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (advance_chemistry) tmmlu++
          config: advance_chemistry
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 27.64
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (junior_science_exam) tmmlu++
          config: junior_science_exam
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 30.05
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (veterinary_pathology) tmmlu++
          config: veterinary_pathology
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 25.09
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (financial_analysis) tmmlu++
          config: financial_analysis
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 25.13
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (national_protection) tmmlu++
          config: national_protection
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 42.65
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (macroeconomics) tmmlu++
          config: macroeconomics
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 26.76
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (politic_science) tmmlu++
          config: politic_science
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 27.44
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (ttqav2) tmmlu++
          config: ttqav2
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 61.06
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (junior_chinese_exam) tmmlu++
          config: junior_chinese_exam
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 30.86
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (traditional_chinese_medicine_clinical_medicine) tmmlu++
          config: traditional_chinese_medicine_clinical_medicine
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 25.9
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (junior_math_exam) tmmlu++
          config: junior_math_exam
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 21.71
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (auditing) tmmlu++
          config: auditing
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 21.82
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (anti_money_laundering) tmmlu++
          config: anti_money_laundering
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 37.31
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (pharmacology) tmmlu++
          config: pharmacology
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 30.68
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (trust_practice) tmmlu++
          config: trust_practice
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 28.18
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (tve_mathematics) tmmlu++
          config: tve_mathematics
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 18.67
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (human_behavior) tmmlu++
          config: human_behavior
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 32.04
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (pharmacy) tmmlu++
          config: pharmacy
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 22.76
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (tve_chinese_language) tmmlu++
          config: tve_chinese_language
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 36.65
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (optometry) tmmlu++
          config: optometry
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 25.11
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (physical_education) tmmlu++
          config: physical_education
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 30.73
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (organic_chemistry) tmmlu++
          config: organic_chemistry
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 35.78
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (tve_natural_sciences) tmmlu++
          config: tve_natural_sciences
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 33.73
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (education) tmmlu++
          config: education
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 37.9
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (mechanical) tmmlu++
          config: mechanical
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 42.37
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (taiwanese_hokkien) tmmlu++
          config: taiwanese_hokkien
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 14.73
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (nautical_science) tmmlu++
          config: nautical_science
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 30.49
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (business_management) tmmlu++
          config: business_management
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 39.57
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (logic_reasoning) tmmlu++
          config: logic_reasoning
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 27.34
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (marketing_management) tmmlu++
          config: marketing_management
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 39.78
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (economics) tmmlu++
          config: economics
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 25.95
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (basic_medical_science) tmmlu++
          config: basic_medical_science
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 28.41
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (occupational_therapy_for_psychological_disorders) tmmlu++
          config: occupational_therapy_for_psychological_disorders
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 35.73
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (general_principles_of_law) tmmlu++
          config: general_principles_of_law
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 31.13
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (junior_chemistry) tmmlu++
          config: junior_chemistry
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 24.88
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (veterinary_pharmacology) tmmlu++
          config: veterinary_pharmacology
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 36.3
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (educational_psychology) tmmlu++
          config: educational_psychology
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 33.52
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (finance_banking) tmmlu++
          config: finance_banking
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 32.59
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (official_document_management) tmmlu++
          config: official_document_management
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 32.43
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (fire_science) tmmlu++
          config: fire_science
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 30.65
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (junior_social_studies) tmmlu++
          config: junior_social_studies
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 47.62
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (accounting) tmmlu++
          config: accounting
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 20.94
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (engineering_math) tmmlu++
          config: engineering_math
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 27.18
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (education_(profession_level)) tmmlu++
          config: education_(profession_level)
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 24.07
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (chinese_language_and_literature) tmmlu++
          config: chinese_language_and_literature
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 27.64
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (management_accounting) tmmlu++
          config: management_accounting
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 24.19
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (culinary_skills) tmmlu++
          config: culinary_skills
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 39.38
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (administrative_law) tmmlu++
          config: administrative_law
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 25.71
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (insurance_studies) tmmlu++
          config: insurance_studies
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 33.42
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (real_estate) tmmlu++
          config: real_estate
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 22.83
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (computer_science) tmmlu++
          config: computer_science
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 31.61
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (taxation) tmmlu++
          config: taxation
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 27.47
      - task:
          type: question-answering
          name: Single Choice Question
        dataset:
          type: ikala/tmmluplus
          name: (trade) tmmlu++
          config: trade
          split: test
          revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c
        metrics:
          - name: single choice
            type: accuracy
            value: 20.32
widget:
  - text: 中華民國憲法第一條
metrics:
  - accuracy

Model Card for lianghsun/Llama-3.2-Taiwan-3B-Instruct

[Discord]

圖像生成來自 OpenArt：An anime-style 🦙 standing proudly atop the summit of Taiwan’s Yushan (Jade Mountain), gazing forward.

採用 lianghsun/Llama-3.2-Taiwan-3B 為基礎模型（foundation model），使用大量中華民國台灣的繁體中文對話集和多國語言對話集進行模型指令微調（instruction fine-tuning）和多輪迭代直接偏好優化（direct preference optimization, DPO），旨在訓練出具有中華民國台灣知識及風格的小語言模型（small langugae model, SLM）之對話模型。

Model Change Log

Update Date	Model Version	Key Changes
2025/01/01	v2025.01.01	Fine-tuning is based on the foundation model version v2024.12.28, and it uses self-prepared instruction datasets for this round of fine-tuning.
2024/12/13	v2024.12.13	Completed 1st round DPO training (10/10 epochs). Preparing for next round DPO training.
2024/11/27	v2024.11.27	Completed SFT training (5/5 epochs). Preparing for multi-round DPO training.
2024/11/25	v2024.11.25	Updated model version to v2024.11.25, training progressed to (3/5) epochs. Still in SFT stage, DPO training remains pending.
2024/11/22	v2024.11.22	Initial upload: Model version v2024.11.22, training completed up to (1/5) epochs. Currently trained only on SFT, DPO training not yet performed.

Model Details

Model Description

Developed by: Huang Liang Hsun
Model type: LlamaForCausalLM
Language(s) (NLP): Tranditional Chinese (zh-tw), English
License: llama3.2
Fine-tuned from model: lianghsun/Llama-3.2-Taiwan-3B

Model Sources

Repository: lianghsun/Llama-3.2-Taiwan-3B
Paper: (WIP, show me the time)
Playground: 🦉 Tawian SmolLM Chat 👈🏼 來玩看看 😻
Demo:

user: 請介紹台灣
assistant: 台灣，位於亞洲東部，地處太平洋與菲律賓海之間，面積約36,000平方公里，人口約2,300萬，是民主自由的國家，經濟實力強勁，擁有世界第10大經濟體。台灣以美食、文化、自然美景著稱，還有豐富的歷史與傳統，吸引全球遊客。台灣語為官方語言，但中文也廣為使用，英語也常用於國際交流。台灣政治多元，執政黨為民進黨，台灣是全球科技產業的重鎮，擁有先進的製造業與服務業。台灣氣候溫暖潮濕，四季分明，夏季炎熱，冬季涼爽，雨季則在5月至10月。台灣的美食以小吃為主，如滷肉飯、珍珠

Uses

Direct Use

本模型已經具備有繁體中文對話能力，使用者可以直接部署推論端點使用。

Downstream Use

若需強化模型在特定領域的知識，可透過微調進一步提升其性能與專業能力。

Out-of-Scope Use

本模型旨在提供資訊，不參與任何政治或法律問題的評斷或立場表達。

Bias, Risks, and Limitations

語言模型的生成內容可能因訓練集的多樣性而帶有偏見、特定立場，或包含與事實不符的言論，請使用者務必在使用過程中仔細確認內容的準確性與中立性。

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

How to Get Started with the Model

要使用 vLLM Docker image 來啟動此模型，您可以按照以下操作：

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model lianghsun/Llama-3.2-Taiwan-3B-Instruct

請注意，如果想要使用不同版本的 checkpoint，請加上 --revision <tag_name>

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HUGGING_FACE_HUB_TOKEN=<secret>" \
    -p 8000:8000 \
    --ipc=host \
    vllm/vllm-openai:latest \
    --model lianghsun/Llama-3.2-Taiwan-3B-Instruct --revision <tag_name>

Training Details

Training Data

繁體中文對話資料集

多國語系對話資料集

Training Procedure

Preprocessing

(WIP)

Training Hyperparameters

SFT stage for v2024.11.27

Note: 以下包含 v2024.11.22 和 v2025.11.25 的超參數設定

learning_rate: 5e-05
min_learning_rate: 5e-07
train_batch_size: 105
seed: 42
distributed_type: multi-GPU
num_devices: 4
gradient_accumulation_steps: 50
total_train_batch_size: 21,000
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_ratio: 0.01
num_epochs: 5.0
global_step: 590

Speeds, Sizes, Times

SFT stage for v2024.11.27

Note: 以下包含 v2024.11.22 和 v2025.11.25 的超參數設定

Duration: 5 days, 16:15:11.17
Train runtime: 490,511.1789
Train samples per second: 25.37
Train steps per second: 0.001
Total training FLOPs: 26,658,386,120,540,160
Train loss: 0.8533

Evaluation

Testing Data, Factors & Metrics

Formosa Taiwan Knowledge Bench

Testing Data

lianghsun/Formosa-bench

Factors

[More Information Needed]

Metrics

[More Information Needed]

Results

[More Information Needed]

Summary

lianghsun/tw-legal-benchmark-v1

Testing Data

Dataset: lianghsun/tw-legal-benchmark-v1
Revision: 66c3a5f3ff2298f6a1cf23201070b5317bdd1893

Factors

[More Information Needed]

Metrics

Accuracy

Results

Model Revision: v2024.11.27

Subset	Split	Score
lianghsun/tw-legal-benchmark-v1	train	31.1

Summary

tmmlu++

Testing Data

Dataset: ikala/tmmluplus
Revision: c0e8ae955997300d5dbf0e382bf0ba5115f85e8c

Factors

[More Information Needed]

Metrics

Accuracy

Results

Model Revision: v2024.11.27

Subset	Split	Score
geography_of_taiwan	test	36.2
dentistry	test	33.83
technical	test	35.07
statistics_and_machine_learning	test	28.57
clinical_psychology	test	29.6
tve_design	test	38.54
three_principles_of_people	test	48.2
introduction_to_law	test	29.96
linear_algebra	test	21.43
agriculture	test	24.5
jce_humanities	test	38.89
music	test	25.9
secondary_physics	test	33.04
physics	test	27.84
advance_chemistry	test	27.64
junior_science_exam	test	30.05
veterinary_pathology	test	25.09
financial_analysis	test	25.13
national_protection	test	42.65
macroeconomics	test	26.76
politic_science	test	27.44
ttqav2	test	61.06
junior_chinese_exam	test	30.86
traditional_chinese_medicine_clinical_medicine	test	25.9
junior_math_exam	test	21.71
auditing	test	21.82
anti_money_laundering	test	37.31
pharmacology	test	30.68
trust_practice	test	28.18
tve_mathematics	test	18.67
human_behavior	test	32.04
pharmacy	test	22.76
tve_chinese_language	test	36.65
optometry	test	25.11
physical_education	test	30.73
organic_chemistry	test	35.78
tve_natural_sciences	test	33.73
education	test	37.9
mechanical	test	42.37
taiwanese_hokkien	test	14.73
nautical_science	test	30.49
business_management	test	39.57
logic_reasoning	test	27.34
marketing_management	test	39.78
economics	test	25.95
basic_medical_science	test	28.41
occupational_therapy_for_psychological_disorders	test	35.73
general_principles_of_law	test	31.13
junior_chemistry	test	24.88
veterinary_pharmacology	test	36.3
educational_psychology	test	33.52
finance_banking	test	32.59
official_document_management	test	32.43
fire_science	test	30.65
junior_social_studies	test	47.62
accounting	test	20.94
engineering_math	test	27.18
education_(profession_level)	test	24.07
chinese_language_and_literature	test	27.64
management_accounting	test	24.19
culinary_skills	test	39.38
administrative_law	test	25.71
insurance_studies	test	33.42
real_estate	test	22.83
computer_science	test	31.61
taxation	test	27.47
trade	test	20.32

Summary

模型版號 v2024.11.27，無論是基礎模型（lianghsun/Llama-3.2-Taiwan-3B）還是指令微調模型（lianghsun/Llama-3.2-Taiwan-3B-Instruct），均未接受過 tmmlu++ 資料集的訓練，以確保測試的公平性。經測試，目前該模型在 tmmlu++ 上表現普遍不佳，未達及格分數，可能需要加入專業領域的資料集來強化基礎模型能力。

Model Examination [optional]

[More Information Needed]

Environmental Impact

Hardware Type: 🚀
Hours used: ⏳⏳⌛
Cloud Provider: 鴻鵠國際股份有限公司
Compute Region: 🇹🇼
Carbon Emitted: ♻️

Technical Specifications

Model Architecture and Objective

[More Information Needed]

Compute Infrastructure

[More Information Needed]

Hardware

CPU count: 32
Logical CPU count: 64
GPU count: 4
GPU type: NVIDIA H100 NVL

Software

OS version: Linux-5.15.0-124-generic-x86_64-with-glibc2.35
Python version: 3.12.7

Citation

@misc{lianghsun2024llama32taiwan3binstruct,
  author = {Huang, Liang Hsun},
  title = {Llama-3.2-Taiwan-3B-Instruct},
  year = {2024},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/lianghsun/Llama-3.2-Taiwan-3B-Instruct}},
  note = {Accessed: 2024-11-25}
}

Glossary [optional]

N/A

More Information

Acknowledge

在此致謝鴻鵠國際股份有限公司蔡長明先生無償地贊助算力，以及曾經幫忙過：廖振翔、chweng、Ben、kevin、Maxxchu、Lam 和陳林彥…等朋友們，才能讓這個模型得以訓練完成，提供算力者乃人生父母。

Usage

如果你基於此指令模型進行微調，希望能不吝嗇在 模型卡片(model card) 裡標註 基礎模型 為：

base_model: lianghsun/Llama-3.2-Taiwan-3B-Instruct

標註和 ❤️ 是給予我們最大的鼓勵，謝謝。😀

Model Card Authors

Huang Liang Hsun

Model Card Contact

Huang Liang Hsun

Framework versions

Transformers 4.45.2
Pytorch 2.4.1+cu121
Datasets 2.21.0
Tokenizers 0.20.0