vntl llama3 8b v2 gguf

Providerlmg-anon
Categorytranslation
Licensellama3
Downloads690.3K
Stars0

Overview

The vntl-llama3-8b-v2 is a specialized GGUF quantization of Meta's Llama 3 8B, fine-tuned specifically for high-fidelity translation tasks. For developers, this model offers a lightweight alternative to larger LLMs when building localization pipelines or real-time translation services. By utilizing the GGUF format, it is optimized for local deployment via llama.cpp or Ollama, allowing for efficient CPU/GPU inference with minimal VRAM overhead. Unlike the general-purpose Llama 3 base, this version is tuned to preserve nuance and technical accuracy across languages, making it ideal for integration into CI/CD documentation workflows or user-facing translation plugins where low latency and privacy are critical.

Highlights

  • Optimized GGUF format for efficient local hardware deployment
  • Fine-tuned specifically for high-accuracy translation workflows
  • Low VRAM footprint suitable for edge computing
  • Seamless integration with llama.cpp and Ollama ecosystems
  • Balanced performance between speed and linguistic precision

Usage

Install
# Install Hugging Face transformers
pip install transformers torch
SDK Usage
# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("lmg-anon/vntl-llama3-8b-v2-gguf")
tokenizer = AutoTokenizer.from_pretrained("lmg-anon/vntl-llama3-8b-v2-gguf")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance 
pip install -U huggingface_hub

CLI Download

Download the full repository

Download the full repository 
huggingface-cli download lmg-anon/vntl-llama3-8b-v2-gguf

Download a single file to a local folder (e.g. config.json into ./dir)

Download a single file to a local folder (e.g. config.json into ./dir) 
huggingface-cli download lmg-anon/vntl-llama3-8b-v2-gguf config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

SDK Download 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('lmg-anon/vntl-llama3-8b-v2-gguf')

Git Download

Make sure git-lfs is installed first

Git Download 
git lfs install
git clone https://huggingface.co/lmg-anon/vntl-llama3-8b-v2-gguf

To skip LFS large-file downloads, use:

Skip LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/lmg-anon/vntl-llama3-8b-v2-gguf

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

Install Transformers 
pip install -U transformers torch

Load the model and run inference

Load the model and run inference 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('lmg-anon/vntl-llama3-8b-v2-gguf')
tokenizer = AutoTokenizer.from_pretrained('lmg-anon/vntl-llama3-8b-v2-gguf')

Full Documentation

来源: HuggingFace

---
license: llama3
datasets:

  • lmg-anon/VNTL-v5-1k

language:
  • ja

  • en

base_model: rinna/llama-3-youko-8b
pipeline_tag: translation
---

Summary

This is a LLaMA 3 Youko qlora fine-tune, created using a new version of the VNTL dataset. The purpose of this fine-tune is to improve performance of LLMs at translating Japanese visual novels to English.

Unlike the previous version, this one doesn't includes the "chat mode".

Notes

For this new version of VNTL 8B, I've rebuilt and expanded VNTL's dataset from the groud up, and I'm happy to say it performs really well, outperforming the previous version when it comes to accuracy and stability, it makes far fewer mistakes than it even when running at high temperatures (though I still recommend temperature 0 for the best accuracy).

Some major changes in this version:

  • Switched to the default LLaMA3 prompt format since people had trouble with the custom one

  • Added proper support for multi-line translations (the old version only handled single lines)

  • Overall better translation accuracy

One thing to note: while the translations are more accurate, they tend to be more literal compared to the previous version.

Sampling Recommendations

For optimal results, it's highly recommended to use neutral sampling parameters (temperature 0 with no repetition penalty) when using this model.

Training Details

This fine-tune was done using similar hyperparameters as the previous version. The only difference is the dataset, which is a brand-new one.

  • Rank: 128
  • Alpha: 32
  • Effective Batch Size: 45
  • Warmup Ratio: 0.02
  • Learning Rate: 6e-5
  • Embedding Learning Rate: 1e-5
  • Optimizer: grokadamw
  • LR Schedule: cosine
  • Weight Decay: 0.01

Train Loss: 0.42

Translation Prompt

This fine-tune uses the LLaMA 3 prompt format, this is an prompt example for translation:

code
<|begin_of_text|><|start_header_id|>Metadata<|end_header_id|>


[character] Name: Uryuu Shingo (瓜生 新吾) | Gender: Male | Aliases: Onii-chan (お兄ちゃん)
[character] Name: Uryuu Sakuno (瓜生 桜乃) | Gender: Female<|eot_id|><|start_header_id|>Japanese<|end_header_id|>


[桜乃]: 『……ごめん』<|eot_id|><|start_header_id|>English<|end_header_id|>


[Sakuno]: 『... Sorry.』<|eot_id|><|start_header_id|>Japanese<|end_header_id|>


[新吾]: 「ううん、こう言っちゃなんだけど、迷子でよかったよ。桜乃は可愛いから、いろいろ心配しちゃってたんだぞ俺」<|eot_id|><|start_header_id|>English<|end_header_id|>


[Shingo]: "Nah, I know it’s weird to say this, but I’m glad you got lost. You’re so cute, Sakuno, so I was really worried about you."<|eot_id|>

The generated translation for that prompt, with temperature 0, is:

code
[Shingo]: "Nah, I know it’s weird to say this, but I’m glad you got lost. You’re so cute, Sakuno, so I was really worried about you."

Trivia

The Metadata section isn't limited to character information - you can also add trivia and teach the model the correct way to pronounce words it struggles with.

Here's an example:

code
<|begin_of_text|><|start_header_id|>Metadata<|end_header_id|>


[character] Name: Uryuu Shingo (瓜生 新吾) | Gender: Male | Aliases: Onii-chan (お兄ちゃん)
[character] Name: Uryuu Sakuno (瓜生 桜乃) | Gender: Female
[element] Name: Murasamemaru (叢雨丸) | Type: Quality<|eot_id|><|start_header_id|>Japanese<|end_header_id|>


[桜乃]: 『……ごめん』<|eot_id|><|start_header_id|>English<|end_header_id|>


[Sakuno]: 『... Sorry.』<|eot_id|><|start_header_id|>Japanese<|end_header_id|>


[新吾]: 「ううん、こう言っちゃなんだけど、迷子でよかったよ。桜乃は叢雨丸いから、いろいろ心配しちゃってたんだぞ俺」<|eot_id|><|start_header_id|>English<|end_header_id|>

The generated translation for that prompt, with temperature 0, is:

code
[Shingo]: "Nah, I know it’s not the best thing to say, but I’m glad you got lost. Sakuno’s Murasamemaru, so I was really worried about you, you know?"

View official source
Join our Telegram