Name: histlux ocr error denoising lrec
Author: impresso-project

简介

HistLux OCR Error Denoising LREC 是一款专注于历史文献数字化修复的专项模型。它并非通用 OCR，而是针对 OCR 识别后产生的噪声和错误进行“去噪”和校正。对于处理古籍、旧报纸等扫描件的开发者来说，该模型能有效补全误识字符，提升文本质量。其上手难度较低，可作为 OCR 识别流水线后的后处理插件，将原始识别结果转化为更精准的文本，极大地减轻了人工校对的工作量。

核心亮点

专注历史文献 OCR 识别结果的去噪与校正
有效降低古籍数字化过程中的字符误识率
可无缝集成至 OCR 文本后处理工作流
采用 AGPL-3.0 协议，适合开源项目集成

使用方法

安装依赖

# 安装 Hugging Face transformers
pip install transformers torch

SDK 使用

# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("impresso-project/histlux_ocr_error_denoising_lrec")
tokenizer = AutoTokenizer.from_pretrained("impresso-project/histlux_ocr_error_denoising_lrec")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引

pip install -U huggingface_hub

命令行下载

下载完整模型库

huggingface-cli download impresso-project/histlux_ocr_error_denoising_lrec

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

huggingface-cli download impresso-project/histlux_ocr_error_denoising_lrec config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('impresso-project/histlux_ocr_error_denoising_lrec')

Git 下载

请确保 lfs 已经被正确安装

Git 下载

git lfs install
git clone https://huggingface.co/impresso-project/histlux_ocr_error_denoising_lrec

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/impresso-project/histlux_ocr_error_denoising_lrec

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

pip install -U transformers torch

模型加载和推理

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('impresso-project/histlux_ocr_error_denoising_lrec')
tokenizer = AutoTokenizer.from_pretrained('impresso-project/histlux_ocr_error_denoising_lrec')

完整文档

来源: HuggingFace

---
tags:

sentence-transformers

sentence-similarity

dataset_size:120000

multilingual

base_model: Alibaba-NLP/gte-multilingual-base
widget:

source_sentence: Who is filming along?

sentences:
- Wién filmt mat?
- >-
Weider huet den Tatarescu drop higewisen, datt Rumänien durch seng
krichsbedélegong op de 6eite vun den allie'erten 110.000 mann verluer hätt.
- Brambilla 130.08.03 St.

source_sentence: 'Four potential scenarios could still play out: Jean Asselborn.'

sentences:
- >-
Dann ass nach eng Antenne hei um Kierchbierg virgesi Richtung RTL Gebai, do
gëtt jo een ganz neie Wunnquartier gebaut.
- >-
D'bedélegong un de wählen wir ganz stärk gewiéscht a munche ge'genden wor re
eso'gucr me' we' 90 prozent.
- Jean Asselborn gesäit 4 Méiglechkeeten, wéi et kéint virugoen.

source_sentence: >-

Non-profit organisation Passerell, which provides legal council to refugees
in Luxembourg, announced that it has to make four employees redundant in
August due to a lack of funding.
sentences:
- Oetringen nach Remich....8.20» 215»
- >-
D'ASBL Passerell, déi sech ëm d'Berodung vu Refugiéeën a Saache Rechtsfroe
këmmert, wäert am August mussen hir véier fix Salariéen entloossen.
- D'Regierung huet allerdéngs "just" 180.041 Doudeger verzeechent.

source_sentence: This regulation was temporarily lifted during the Covid pandemic.

sentences:
- Six Jours vu New-York si fir d’équipe Girgetti — Debacco
- Dës Reegelung gouf wärend der Covid-Pandemie ausgesat.
- ING-Marathon ouni gréisser Tëschefäll ofgelaf - 18 Leit hospitaliséiert.

source_sentence: The cross-border workers should also receive more wages.

sentences:
- D'grenzarbechetr missten och me' lo'n kre'en.
- >-
De Néckel: Firun! Dât ass jo ailes, wèll 't get dach neischt un der Bréck
gemâcht!
- >-
D'Grande-Duchesse Josephine Charlotte an hir Ministeren hunn d'Land
verlooss, et war den Optakt vun der Zäit am Exil.
pipeline_tag: sentence-similarity
library_name: sentence-transformers
model-index:

name: >-

SentenceTransformer based on
Alibaba-NLP/gte-multilingual-base
results:
- task:
type: contemporary-lb
name: Contemporary-lb
dataset:
name: Contemporary-lb
type: contemporary-lb
metrics:
- type: accuracy
value: 0.6216
name: SIB-200(LB) accuracy
- type: accuracy
value: 0.6282
name: ParaLUX accuracy
- task:
type: bitext-mining
name: LBHistoricalBitextMining
dataset:
name: LBHistoricalBitextMining
type: lb-en
metrics:
- type: accuracy
value: 0.9683
name: LB<->FR accuracy
- type: accuracy
value: 0.9715
name: LB<->EN accuracy
- type: mean_accuracy
value: 0.9793
name: LB<->DE accuracy
license: agpl-3.0
datasets:

impresso-project/HistLuxAlign

fredxlpy/LuxAlign

language:

lb

---

THIS IS A PREVIEW MODEL for the IMPRESSO HALLOWEEN WORKSHOP

This is a sentence-transformers model finetuned from Alibaba-NLP/gte-multilingual-base further adapted to support Historical and Contemporary Luxembourgish. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for (cross-lingual) semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

This model is specialised to perform cross-lingual semantic search to and from Historical/Contemporary Luxembourgish. This model would be particularly useful for libraries and archives that want to perform semantic search and longitudinal studies within their collections.

This is an Alibaba-NLP/gte-multilingual-base model that was further adapted by (Michail et al., 2025)

Limitations

We also release a model that performs better (18pp) on ParaLUX. If finding monolingual exact matches within adversarial collections is of at-most importance, please use histlux-paraphrase-multilingual-mpnet-base-v2

Model Description

Model Type: GTE-Multilingual-Base

Base model: Alibaba-NLP/gte-multilingual-base

Maximum Sequence Length: 8192 tokens

Output Dimensionality: 768 dimensions

Similarity Function: Cosine Similarity

Training Dataset: See below

Usage (Sentence-Transformers)

Using this model becomes easy when you have sentence-transformers installed:

code

pip install -U sentence-transformers

Then you can use the model like this:

python

from sentence_transformers import SentenceTransformer
sentences = ["This is an example sentence", "Each sentence is converted"]
model = SentenceTransformer('impresso-project/halloween_workshop_ocr_robust_with_lux_preview', trust_remote_code=True)
embeddings = model.encode(sentences)
print(embeddings)

Training Details

Training Dataset

The parallel sentences data mix is the following:

impresso-project/HistLuxAlign:
- LB-FR (x20,000)
- LB-EN (x20,000)
- LB-DE (x20,000)

fredxlpy/LuxAlign:
- LB-FR (x40,000)
- LB-EN (x20,000)

Total: 120 000 Sentence pairs in mixed batches of size 8

Contrastive Training

The model was trained with the parameters:

code

Loss:

sentence_transformers.losses.MultipleNegativesRankingLoss.MultipleNegativesRankingLoss with parameters:

{'scale': 20.0, 'similarity_fct': 'cos_sim'}

code

Parameters of the fit()-Method:

{
"epochs": 1,
"evaluation_steps": 520,
"max_grad_norm": 1,
"optimizer_class": "<class 'torch.optim.adamw.AdamW'>",
"optimizer_params": {
"lr": 2e-05
},
"scheduler": "WarmupLinear",
}

code

Citation

BibTeX

#### Adapting Multilingual Embedding Models to Historical Luxembourgish (introducing paper)

bibtex

@inproceedings{michail-etal-2025-adapting,
    title = "Adapting Multilingual Embedding Models to Historical {L}uxembourgish",
    author = "Michail, Andrianos  and
      Racl{\'e}, Corina  and
      Opitz, Juri  and
      Clematide, Simon",
    editor = "Kazantseva, Anna  and
      Szpakowicz, Stan  and
      Degaetano-Ortlieb, Stefania  and
      Bizzoni, Yuri  and
      Pagel, Janis",
    booktitle = "Proceedings of the 9th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2025)",
    month = may,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.latechclfl-1.26/",
    doi = "10.18653/v1/2025.latechclfl-1.26",
    pages = "291--298",
    ISBN = "979-8-89176-241-1"
}

#### Original Multilingual GTE Model

bibtex

@inproceedings{zhang2024mgte,
  title={mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval},
  author={Zhang, Xin and Zhang, Yanzhao and Long, Dingkun and Xie, Wen and Dai, Ziqi and Tang, Jialong and Lin, Huan and Yang, Baosong and Xie, Pengjun and Huang, Fei and others},
  booktitle={Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track},
  pages={1393--1412},
  year={2024}
}

About Impresso

Impresso project

Impresso - Media Monitoring of the Past is an interdisciplinary research project that aims to develop and consolidate tools for processing and exploring large collections of media archives across modalities, time, languages and national borders. The first project (2017-2021) was funded by the Swiss National Science Foundation under grant No. CRSII5_173719 and the second project (2023-2027) by the SNSF under grant No. [CRSII5_213

histlux ocr error denoising lrec

简介

核心亮点

使用方法

Hugging Face 下载

命令行下载

SDK 下载

Git 下载

PyTorch / Transformers 使用

完整文档

THIS IS A PREVIEW MODEL for the IMPRESSO HALLOWEEN WORKSHOP

Model Details

Limitations

Model Description

Usage (Sentence-Transformers)

Training Details

Training Dataset

Contrastive Training

Citation

BibTeX

About Impresso

Impresso project

扫码进交流群