llava med v1.5 mistral 7b hf
Overview
Highlights
- Specialized in biomedical visual question answering tasks
- Powered by a high-performance Mistral 7B language backbone
- Seamless integration via Hugging Face Transformers library
- Apache-2.0 license allows for flexible commercial deployment
- Optimized for clinical image interpretation and medical reasoning
Usage
# Install Hugging Face transformers
pip install transformers torch# Load model with transformers
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("chaoyinshe/llava-med-v1.5-mistral-7b-hf")
tokenizer = AutoTokenizer.from_pretrained("chaoyinshe/llava-med-v1.5-mistral-7b-hf")Hugging Face Download
We recommend downloading the model via the Hugging Face CLI or Hub SDK.
Guidance：Before downloading, install huggingface_hub with:
pip install -U huggingface_hubCLI Download
Download the full repository
huggingface-cli download chaoyinshe/llava-med-v1.5-mistral-7b-hfDownload a single file to a local folder (e.g. config.json into ./dir)
huggingface-cli download chaoyinshe/llava-med-v1.5-mistral-7b-hf config.json --local-dir ./dirSee the official docs for more CLI options
SDK Download
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('chaoyinshe/llava-med-v1.5-mistral-7b-hf')Git Download
Make sure git-lfs is installed first
git lfs install
git clone https://huggingface.co/chaoyinshe/llava-med-v1.5-mistral-7b-hfTo skip LFS large-file downloads, use:
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/chaoyinshe/llava-med-v1.5-mistral-7b-hfModel files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.
PyTorch / Transformers Usage
Install Transformers
pip install -U transformers torchLoad the model and run inference
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained('chaoyinshe/llava-med-v1.5-mistral-7b-hf')
tokenizer = AutoTokenizer.from_pretrained('chaoyinshe/llava-med-v1.5-mistral-7b-hf')Full Documentation
---
license: apache-2.0
base_model:
- microsoft/llava-med-v1.5-mistral-7b
pipeline_tag: visual-question-answering
language:
- en
tags:
- medical
- biology
---
llava-med-v1.5-mistral-7b-hf
This repository contains a drop-in, Hugging Face–compatible checkpoint converted from
https://huggingface.co/microsoft/llava-med-v1.5-mistral-7b.
You can load it with the exact same code you use for the original model—no extra conversion steps required.
---
✅ Clarification on Vocab Size Expansion & Weight Integrity
You can refer to the original conversion code we provided to access the relevant implementation details and code snippets.
You may wonder if the vocab size mismatch (e.g., 32000 vs. 32064) breaks the original weights by causing dimension mismatches between embed_tokens/lm_head and the original weights.
The answer is no. Below are key clarifications:
1. Does expanding embed_tokens affect the attention mechanism?
No. The embed_tokens layer is simply an embedding lookup table (nn.Embedding) that maps token IDs to vectors. The attention mechanism (e.g., nn.MultiheadAttention or LlamaAttention) does not operate on this lookup table directly. It only cares if the dimension of input hidden states is consistent. Thus, even if you add more tokens, as long as the hidden_size remains unchanged, the weight shape of the attention layer is not affected at all.2. Which layers actually change after expansion?
| Module | Weight Shape Change | Affects Attention Calculation? | |----------------------|--------------------------------------|---------------------------------| | embed_tokens | [vocab_size+2, hidden_size] | ❌ No | | lm_head | [vocab_size+2, hidden_size] | ❌ No | | All attention layers | No shape change | ✅ Completely unchanged |#### Analogy for easier understanding
Think of the model as a dictionary:
- embed_tokens is the "new word list": You add two new words, but the length of each word’s explanation (hidden_size) stays the same.
- Attention is the "reading rule": It only focuses on how the vectors of each word in a sentence interact with each other, not how many words are in the dictionary.
#### The only "change" lies in input distribution
- The original model never encountered embeddings for tokens like
<image>or<pad>.
- Now these tokens are initialized, and their new vectors appear during the first forward pass.
- This is a data-level change, not damage to the model’s parameters.
#### Summary
Expanding the vocab only changes the "dictionary size", not the "reading rules". The weight shape and calculation logic of the attention mechanism remain completely unchanged. The output layer (lm_head) changes in shape but not in functional logic.
3. What exactly changes in the output layer (lm_head)?
The line below modifies the weight matrix of lm_head (the language modeling head):model.resize_token_embeddings(config.text_config.vocab_size + 2, pad_shape)Quick Start
from transformers import LlavaForConditionalGeneration, AutoProcessor
import torch
model_path = "chaoyinshe/llava-med-v1.5-mistral-7b-hf"
model = LlavaForConditionalGeneration.from_pretrained(
model_path,
torch_dtype=torch.bfloat16,
attn_implementation="flash_attention_2", # requires FA2
device_map="auto" # multi-GPU ready
)
processor = AutoProcessor.from_pretrained(model_path)
Example inference
messages = [
{
"role": "user",
"content": [
{"type": "image"},
{"type": "text", "text": "What is the main finding in this chest X-ray?"}
]
}
]
prompt = processor.tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True
)
inputs = processor(
images=[image], text=prompt, return_tensors="pt"
).to(model.device, torch.bfloat16)
with torch.inference_mode():
out = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(out[0], skip_special_tokens=True))
✅ Training Screenshot
(Note: The image below is for illustrative purposes only — actual training metrics may vary.)
🤗