Mixtral 8x7B

ProviderMistral AI
Categorytext-generation
Parameters47B
LicenseApache 2.0
Downloads5.0M
Stars7.2K

Overview

Mixtral 8x7B is a high-performance Sparse Mixture of Experts (SMoE) model that provides a compelling alternative to dense architectures. By utilizing a gated mechanism to activate only a fraction of its 47B parameters per token, it achieves a throughput and latency profile similar to much smaller models while maintaining the reasoning capabilities of larger ones. For developers, this means a significant reduction in compute overhead during inference without sacrificing quality in complex tasks like code generation or multilingual processing. It is released under the permissive Apache 2.0 license, making it ideal for production environments where data privacy and self-hosting are priorities. Compared to standard 7B or 13B models, Mixtral offers a substantial leap in logical coherence and context handling, bridging the gap between lightweight edge models and massive proprietary LLMs.

Highlights

  • Sparse MoE architecture optimizes inference speed and efficiency.
  • Strong performance in coding and multilingual text generation.
  • Apache 2.0 license allows flexible commercial deployment.
  • Competitive reasoning capabilities relative to larger dense models.

Usage

Install
# Install Hugging Face transformers
pip install transformers torch
SDK Usage
# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("mistralai/Mixtral-8x7B")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mixtral-8x7B")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance 
pip install -U huggingface_hub

CLI Download

Download the full repository

Download the full repository 
huggingface-cli download mistralai/Mixtral-8x7B

Download a single file to a local folder (e.g. config.json into ./dir)

Download a single file to a local folder (e.g. config.json into ./dir) 
huggingface-cli download mistralai/Mixtral-8x7B config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

SDK Download 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('mistralai/Mixtral-8x7B')

Git Download

Make sure git-lfs is installed first

Git Download 
git lfs install
git clone https://huggingface.co/mistralai/Mixtral-8x7B

To skip LFS large-file downloads, use:

Skip LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/mistralai/Mixtral-8x7B

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

Install Transformers 
pip install -U transformers torch

Load the model and run inference

Load the model and run inference 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('mistralai/Mixtral-8x7B')
tokenizer = AutoTokenizer.from_pretrained('mistralai/Mixtral-8x7B')

Full Documentation

来源: HuggingFace

---
library_name: vllm
license: apache-2.0
language:

  • fr

  • it

  • de

  • es

  • en

tags:
  • moe

  • mistral-common

extra_gated_description: >-
If you want to learn more about how we process your personal data, please read
our <a href="https://mistral.ai/fr/terms/">Privacy Policy</a>.
---

Model Card for Mixtral-8x7B


The Mixtral-8x7B Large Language Model (LLM) is a pretrained generative Sparse Mixture of Experts. The Mistral-8x7B outperforms Llama 2 70B on most benchmarks we tested.

For full details of this model please read our release blog post.

Warning

This repo contains weights that are compatible with vLLM serving of the model as well as Hugging Face transformers library. It is based on the original Mixtral torrent release, but the file format and parameter names are different. Please note that model cannot (yet) be instantiated with HF.

Run the model

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mistralai/Mixtral-8x7B-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_id)


model = AutoModelForCausalLM.from_pretrained(model_id)


text = "Hello my name is"
inputs = tokenizer(text, return_tensors="pt")


outputs = model.generate(inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

By default, transformers will load the model in full precision. Therefore you might be interested to further reduce down the memory requirements to run the model through the optimizations we offer in HF ecosystem:

In half-precision

Note float16 precision only works on GPU devices

<details>
<summary> Click to expand </summary>

diff
+ import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mistralai/Mixtral-8x7B-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_id)


+ model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16).to(0)


text = "Hello my name is"
+ inputs = tokenizer(text, return_tensors="pt").to(0)


outputs = model.generate(inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

</details>

Lower precision using (8-bit & 4-bit) using bitsandbytes

<details>
<summary> Click to expand </summary>

diff
+ import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mistralai/Mixtral-8x7B-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_id)


+ model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True)


text = "Hello my name is"
+ inputs = tokenizer(text, return_tensors="pt").to(0)


outputs = model.generate(inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

</details>

Load the model with Flash Attention 2

<details>
<summary> Click to expand </summary>

diff
+ import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "mistralai/Mixtral-8x7B-v0.1"
tokenizer = AutoTokenizer.from_pretrained(model_id)


+ model = AutoModelForCausalLM.from_pretrained(model_id, use_flash_attention_2=True)


text = "Hello my name is"
+ inputs = tokenizer(text, return_tensors="pt").to(0)


outputs = model.generate(inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

</details>

Notice

Mixtral-8x7B is a pretrained base model and therefore does not have any moderation mechanisms.

The Mistral AI Team

Albert Jiang, Alexandre Sablayrolles, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Louis Ternon, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, William El Sayed.
View official source
Join our Telegram