ko sbert nli

Providerjhgan
Categorynatural-language-inference
LicenseApache-2.0
Downloads103.4K
Stars0

Overview

ko-sbert-nli is a specialized Sentence-BERT model fine-tuned for Natural Language Inference (NLI) specifically for the Korean language. Unlike standard BERT models, this architecture is optimized to map sentences into a dense vector space where semantic similarity is preserved, making it highly efficient for calculating cosine similarity between text pairs. For developers, this means it is an ideal tool for building semantic search engines, automated clustering, or duplicate question detection in Korean-centric applications. It integrates seamlessly with the sentence-transformers library, allowing for rapid deployment without the need for complex custom training loops. Compared to general multilingual models, this model provides higher precision for Korean linguistic nuances, reducing false positives in entailment and contradiction tasks.

Highlights

  • Optimized for Korean semantic similarity and NLI tasks.
  • Compatible with sentence-transformers for easy integration.
  • High-performance dense vector embeddings for semantic search.
  • Apache-2.0 license allows for flexible commercial use.

Usage

Install
# Install Hugging Face transformers
pip install transformers torch
SDK Usage
# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("jhgan/ko-sbert-nli")
tokenizer = AutoTokenizer.from_pretrained("jhgan/ko-sbert-nli")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance 
pip install -U huggingface_hub

CLI Download

Download the full repository

Download the full repository 
huggingface-cli download jhgan/ko-sbert-nli

Download a single file to a local folder (e.g. config.json into ./dir)

Download a single file to a local folder (e.g. config.json into ./dir) 
huggingface-cli download jhgan/ko-sbert-nli config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

SDK Download 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('jhgan/ko-sbert-nli')

Git Download

Make sure git-lfs is installed first

Git Download 
git lfs install
git clone https://huggingface.co/jhgan/ko-sbert-nli

To skip LFS large-file downloads, use:

Skip LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/jhgan/ko-sbert-nli

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

Install Transformers 
pip install -U transformers torch

Load the model and run inference

Load the model and run inference 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('jhgan/ko-sbert-nli')
tokenizer = AutoTokenizer.from_pretrained('jhgan/ko-sbert-nli')

Full Documentation

来源: HuggingFace

---
pipeline_tag: sentence-similarity
tags:

  • sentence-transformers

  • feature-extraction

  • sentence-similarity

  • transformers

---

ko-sbert-nli

This is a sentence-transformers model: It maps sentences & paragraphs to a 768 dimensional dense vector space and can be used for tasks like clustering or semantic search.

<!--- Describe your model here -->

Usage (Sentence-Transformers)

Using this model becomes easy when you have sentence-transformers installed:

code
pip install -U sentence-transformers

Then you can use the model like this:

python
from sentence_transformers import SentenceTransformer
sentences = ["안녕하세요?", "한국어 문장 임베딩을 위한 버트 모델입니다."]

model = SentenceTransformer('jhgan/ko-sbert-nli')
embeddings = model.encode(sentences)
print(embeddings)

Usage (HuggingFace Transformers)

Without sentence-transformers, you can use the model like this: First, you pass your input through the transformer model, then you have to apply the right pooling-operation on-top of the contextualized word embeddings.
python
from transformers import AutoTokenizer, AutoModel
import torch

#Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)


Sentences we want sentence embeddings for

sentences = ['This is an example sentence', 'Each sentence is converted']

Load model from HuggingFace Hub

tokenizer = AutoTokenizer.from_pretrained('jhgan/ko-sbert-nli')
model = AutoModel.from_pretrained('jhgan/ko-sbert-nli')

Tokenize sentences

encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')

Compute token embeddings

with torch.no_grad():
    model_output = model(encoded_input)

Perform pooling. In this case, mean pooling.

sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])

print("Sentence embeddings:")
print(sentence_embeddings)

Evaluation Results

<!--- Describe how your model was evaluated -->

KorNLI 학습 데이터셋으로 학습한 후 KorSTS 평가 데이터셋으로 평가한 결과입니다.

  • Cosine Pearson: 82.24
  • Cosine Spearman: 83.16
  • Euclidean Pearson: 82.19
  • Euclidean Spearman: 82.31
  • Manhattan Pearson: 82.18
  • Manhattan Spearman: 82.30
  • Dot Pearson: 79.30
  • Dot Spearman: 78.78

Training

The model was trained with the parameters:

DataLoader:

sentence_transformers.datasets.NoDuplicatesDataLoader.NoDuplicatesDataLoader of length 8885 with parameters:

code
{'batch_size': 64}

Loss**:

sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss with parameters:

code
{'scale': 20.0, 'similarity_fct': 'cos_sim'}

Parameters of the fit()-Method:

code
{
    "epochs": 1,
    "evaluation_steps": 1000,
    "evaluator": "sentence_transformers.evaluation.EmbeddingSimilarityEvaluator.EmbeddingSimilarityEvaluator",
    "max_grad_norm": 1,
    "optimizer_class": "<class 'transformers.optimization.AdamW'>",
    "optimizer_params": {
        "lr": 2e-05
    },
    "scheduler": "WarmupLinear",
    "steps_per_epoch": null,
    "warmup_steps": 889,
    "weight_decay": 0.01
}

Full Model Architecture

code
SentenceTransformer(
  (0): Transformer({'max_seq_length': 128, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False})
)

Citing & Authors

<!--- Describe where people can find more information -->

  • Ham, J., Choe, Y. J., Park, K., Choi, I., & Soh, H. (2020). Kornli and korsts: New benchmark datasets for korean natural language understanding. arXiv preprint arXiv:2004.03289

  • Reimers, Nils and Iryna Gurevych. “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.” ArXiv abs/1908.10084 (2019)

  • Reimers, Nils and Iryna Gurevych. “Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation.” EMNLP (2020).

View official source
Join our Telegram