madlad400 3b mt

---
license: apache-2.0
language:

• multilingual

• en

• ru

• es

• fr

• de

• it

• pt

• pl

• nl

• vi

• tr

• sv

• id

• ro

• cs

• zh

• hu

• ja

• th

• fi

• fa

• uk

• da

• el

• "no"

• bg

• sk

• ko

• ar

• lt

• ca

• sl

• he

• et

• lv

• hi

• sq

• ms

• az

• sr

• ta

• hr

• kk

• is

• ml

• mr

• te

• af

• gl

• fil

• be

• mk

• eu

• bn

• ka

• mn

• bs

• uz

• ur

• sw

• yue

• ne

• kn

• kaa

• gu

• si

• cy

• eo

• la

• hy

• ky

• tg

• ga

• mt

• my

• km

• tt

• so

• ku

• ps

• pa

• rw

• lo

• ha

• dv

• fy

• lb

• ckb

• mg

• gd

• am

• ug

• ht

• grc

• hmn

• sd

• jv

• mi

• tk

• ceb

• yi

• ba

• fo

• or

• xh

• su

• kl

• ny

• sm

• sn

• co

• zu

• ig

• yo

• pap

• st

• haw

• as

• oc

• cv

• lus

• tet

• gsw

• sah

• br

• rm

• sa

• bo

• om

• se

• ce

• cnh

• ilo

• hil

• udm

• os

• lg

• ti

• vec

• ts

• tyv

• kbd

• ee

• iba

• av

• kha

• to

• tn

• nso

• fj

• zza

• ak

• ada

• otq

• dz

• bua

• cfm

• ln

• chm

• gn

• krc

• wa

• hif

• yua

• srn

• war

• rom

• bik

• pam

• sg

• lu

• ady

• kbp

• syr

• ltg

• myv

• iso

• kac

• bho

• ay

• kum

• qu

• za

• pag

• ngu

• ve

• pck

• zap

• tyz

• hui

• bbc

• tzo

• tiv

• ksd

• gom

• min

• ang

• nhe

• bgp

• nzi

• nnb

• nv

• zxx

• bci

• kv

• new

• mps

• alt

• meu

• bew

• fon

• iu

• abt

• mgh

• mnw

• tvl

• dov

• tlh

• ho

• kw

• mrj

• meo

• crh

• mbt

• emp

• ace

• ium

• mam

• gym

• mai

• crs

• pon

• ubu

• fip

• quc

• gv

• kj

• btx

• ape

• chk

• rcf

• shn

• tzh

• mdf

• ppk

• ss

• gag

• cab

• kri

• seh

• ibb

• tbz

• bru

• enq

• ach

• cuk

• kmb

• wo

• kek

• qub

• tab

• bts

• kos

• rwo

• cak

• tuc

• bum

• cjk

• gil

• stq

• tsg

• quh

• mak

• arn

• ban

• jiv

• sja

• yap

• tcy

• toj

• twu

• xal

• amu

• rmc

• hus

• nia

• kjh

• bm

• guh

• mas

• acf

• dtp

• ksw

• bzj

• din

• zne

• mad

• msi

• mag

• mkn

• kg

• lhu

• ch

• qvi

• mh

• djk

• sus

• mfe

• srm

• dyu

• ctu

• gui

• pau

• inb

• bi

• mni

• guc

• jam

• wal

• jac

• bas

• gor

• skr

• nyu

• noa

• sda

• gub

• nog

• cni

• teo

• tdx

• sxn

• rki

• nr

• frp

• alz

• taj

• lrc

• cce

• rn

• jvn

• hvn

• nij

• dwr

• izz

• msm

• bus

• ktu

• chr

• maz

• tzj

• suz

• knj

• bim

• gvl

• bqc

• tca

• pis

• prk

• laj

• mel

• qxr

• niq

• ahk

• shp

• hne

• spp

• koi

• krj

• quf

• luz

• agr

• tsc

• mqy

• gof

• gbm

• miq

• dje

• awa

• bjj

• qvz

• sjp

• tll

• raj

• kjg

• bgz

• quy

• cbk

• akb

• oj

• ify

• mey

• ks

• cac

• brx

• qup

• syl

• jax

• ff

• ber

• tks

• trp

• mrw

• adh

• smt

• srr

• ffm

• qvc

• mtr

• ann

• kaa

• aa

• noe

• nut

• gyn

• kwi

• xmm

• msb

• text2text-generation

• text-generation-inference

• allenai/MADLAD-400

widget:

• text: "<2en> Como vai, amigo?"

• text: "<2de> Do you speak German?"

---

Model Card for MADLAD-400-3B-MT

Table of Contents

0. TL;DR
1. Model Details
2. Usage
3. Uses
4. Bias, Risks, and Limitations
5. Training Details
6. Evaluation
7. Environmental Impact
8. Citation

TL;DR

MADLAD-400-3B-MT is a multilingual machine translation model based on the T5 architecture that was
trained on 1 trillion tokens covering over 450 languages using publicly available data.
It is competitive with models that are significantly larger.

Disclaimer: Juarez Bochi, who was not involved in this research, converted
the original weights and wrote the contents of this model card based on the original paper and Flan-T5.

Model Details

Model Description

• Model type: Language model

• Language(s) (NLP): Multilingual (400+ languages)

• License: Apache 2.0

• Related Models: All MADLAD-400 Checkpoints

• Original Checkpoints: All Original MADLAD-400 Checkpoints

• Resources for more information:

Usage

Find below some example scripts on how to use the model:

Using the Pytorch model with transformers

Running the model on a CPU or GPU

<details>
<summary> Click to expand </summary>

First, install the Python packages that are required:

pip install transformers accelerate sentencepiece

from transformers import T5ForConditionalGeneration, T5Tokenizer
model_name = 'jbochi/madlad400-3b-mt'
model = T5ForConditionalGeneration.from_pretrained(model_name, device_map="auto")
tokenizer = T5Tokenizer.from_pretrained(model_name)
text = "<2pt> I love pizza!"
input_ids = tokenizer(text, return_tensors="pt").input_ids.to(model.device)
outputs = model.generate(input_ids=input_ids)
tokenizer.decode(outputs[0], skip_special_tokens=True)
Eu adoro pizza!

</details>

Running the model with Candle

<details>
<summary> Click to expand </summary>

Usage with candle:

$ cargo run --example t5 --release  -- \
  --model-id "jbochi/madlad400-3b-mt" \
  --prompt "<2de> How are you, my friend?" \
  --decode --temperature 0

We also provide a quantized model (1.65 GB vs the original 11.8 GB file):

cargo run --example quantized-t5 --release  -- \
  --model-id "jbochi/madlad400-3b-mt" --weight-file "model-q4k.gguf" \
  --prompt "<2de> How are you, my friend?" \
  --temperature 0
...
 Wie geht es dir, mein Freund?

</details>

Uses

Direct Use and Downstream Use

> Primary intended uses: Machine Translation and multilingual NLP tasks on over 400 languages.
> Primary intended users: Research community.

Out-of-Scope Use

> These models are trained on general domain data and are therefore not meant to
> work on domain-specific models out-of-the box. Moreover, these research models have not been assessed
> for production usecases.

Bias, Risks, and Limitations

> We note that we evaluate on only 204 of the languages supported by these models and on machine translation
> and few-shot machine translation tasks. Users must consider use of this model carefully for their own
> usecase.

Ethical considerations and risks

> We trained these models with MADLAD-400 and publicly available data to create baseline models that
> support NLP for over 400 languages, with a focus on languages underrepresented in large-scale corpora.
> Given that these models were trained with web-crawled datasets that may contain sensitive, offensive or
> otherwise low-quality content despite extensive preprocessing, it is still possible that these issues to the
> underlying training data may cause differences in model performance and toxic (or otherwise problematic)
> output for certain domains. Moreover, large models are dual use technologies that have specific risks
> associated with their use and development. We point the reader to surveys such as those written by
> Weidinger et al. or Bommasani et al. for a more detailed discussion of these risks, and to Liebling
> et al. for a thorough discussion of the risks of machine translation systems.

Known Limitations

More information needed

Sensitive Use:

More information needed

Training Details

> We train models of various sizes: a 3B, 32-layer parameter model,
> a 7.2B 48-layer parameter model and a 10.7B 32-layer parameter model.
> We share all parameters of the model across language pairs,
> and use a Sentence Piece Model with 256k tokens shared on both the encoder and decoder
> side. Each input sentence has a <2xx> token prepended to the source sentence to indicate the target
> language.

See the research paper for further details.

Training Data

> For both the machine translation and language model, M