clap htsat fused

Providerlaion
Categoryaudio-classification
Licenseapache-2.0
Downloads6.9K
Stars2

Overview

CLAP HTSAT Fused is a specialized audio representation model designed for high-accuracy audio classification and retrieval. By fusing the HTS-AT architecture with Contrastive Language-Audio Pretraining (CLAP), this model maps audio signals into a shared embedding space with textual descriptions. For developers, this means you can implement zero-shot audio classification or semantic audio search without needing extensive labeled datasets for every new category. It is particularly effective for environmental sound tagging, music genre identification, and event detection. Integration is straightforward for those familiar with the LAION ecosystem or PyTorch, offering a robust alternative to traditional spectrogram-based CNNs by leveraging transformer-based temporal modeling.

Highlights

  • Zero-shot audio classification via shared text-audio embeddings
  • Fused HTS-AT architecture for superior temporal feature extraction
  • Apache-2.0 license allows for flexible commercial integration
  • Optimized for semantic audio search and event tagging

Usage

Install
# Install Hugging Face transformers
pip install transformers torch
SDK Usage
# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("laion/clap-htsat-fused")
tokenizer = AutoTokenizer.from_pretrained("laion/clap-htsat-fused")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance 
pip install -U huggingface_hub

CLI Download

Download the full repository

Download the full repository 
huggingface-cli download laion/clap-htsat-fused

Download a single file to a local folder (e.g. config.json into ./dir)

Download a single file to a local folder (e.g. config.json into ./dir) 
huggingface-cli download laion/clap-htsat-fused config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

SDK Download 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('laion/clap-htsat-fused')

Git Download

Make sure git-lfs is installed first

Git Download 
git lfs install
git clone https://huggingface.co/laion/clap-htsat-fused

To skip LFS large-file downloads, use:

Skip LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/laion/clap-htsat-fused

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

Install Transformers 
pip install -U transformers torch

Load the model and run inference

Load the model and run inference 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('laion/clap-htsat-fused')
tokenizer = AutoTokenizer.from_pretrained('laion/clap-htsat-fused')

Model Download

We recommend downloading the model via the ModelScope CLI or SDK.

Guidance：Before downloading, install ModelScope with:

Guidance 
pip install modelscope

CLI Download

Download the full repository

Download the full repository 
modelscope download --model laion/clap-htsat-fused

Download a single file to a local folder (e.g. README.md into ./dir)

Download a single file to a local folder (e.g. README.md into ./dir) 
modelscope download --model laion/clap-htsat-fused README.md --local_dir ./dir

See the docs for more CLI options

SDK Download

SDK Download 
# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('laion/clap-htsat-fused')

Git Download

Make sure git-lfs is installed first

Git Download 
git lfs install
git clone https://www.modelscope.cn/laion/clap-htsat-fused.git

To skip LFS large-file downloads, use:

Skip LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/laion/clap-htsat-fused.git

ModelScope 模型页直接下载模型文件；无需将模型文件放在本站服务器。

Notebook Quickstart

Install the ModelScope library

Install the ModelScope library 
pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

Load the model and run inference

Load the model and run inference 
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

p = pipeline('text-generation', 'laion/clap-htsat-fused')

Full Documentation

来源: HuggingFace

---
license: apache-2.0
language:

  • en

pipeline_tag: audio-classification
tags:
  • zero-shot audio classification

  • zero-shot audio retrieval

---

Model card for CLAP

Model card for CLAP: Contrastive Language-Audio Pretraining

Dataset

LAION-CLAP was trained on LAION-audio-630k

Table of Contents

0. TL;DR
1. Model Details
2. Usage
3. Uses
4. Citation

TL;DR

The abstract of the paper states that:

> Contrastive learning has shown remarkable success in the field of multimodal representation learning. In this paper, we propose a pipeline of contrastive language-audio pretraining to develop an audio representation by combining audio data with natural language descriptions. To accomplish this target, we first release LAION-Audio-630K, a large collection of 633,526 audio-text pairs from different data sources. Second, we construct a contrastive language-audio pretraining model by considering different audio encoders and text encoders. We incorporate the feature fusion mechanism and keyword-to-caption augmentation into the model design to further enable the model to process audio inputs of variable lengths and enhance the performance. Third, we perform comprehensive experiments to evaluate our model across three tasks: text-to-audio retrieval, zero-shot audio classification, and supervised audio classification. The results demonstrate that our model achieves superior performance in text-to-audio retrieval task. In audio classification tasks, the model achieves state-of-the-art performance in the zero-shot setting and is able to obtain performance comparable to models' results in the non-zero-shot setting. LAION-Audio-630K and the proposed model are both available to the public.

Usage

You can use this model for zero shot audio classification or extracting audio and/or textual features.

Uses

Perform zero-shot audio classification

Using pipeline

python
from datasets import load_dataset
from transformers import pipeline

dataset = load_dataset("ashraq/esc50")
audio = dataset["train"]["audio"][-1]["array"]


audio_classifier = pipeline(task="zero-shot-audio-classification", model="laion/clap-htsat-fused")
output = audio_classifier(audio, candidate_labels=["Sound of a dog", "Sound of vaccum cleaner"])
print(output)
>>> [{"score": 0.999, "label": "Sound of a dog"}, {"score": 0.001, "label": "Sound of vaccum cleaner"}]

Run the model:

You can also get the audio and text embeddings using ClapModel

Run the model on CPU:

python
from datasets import load_dataset
from transformers import ClapModel, ClapProcessor

librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
audio_sample = librispeech_dummy[0]


model = ClapModel.from_pretrained("laion/clap-htsat-fused")
processor = ClapProcessor.from_pretrained("laion/clap-htsat-fused")


inputs = processor(audios=audio_sample["audio"]["array"], return_tensors="pt")
audio_embed = model.get_audio_features(inputs)

Run the model on GPU:

python
from datasets import load_dataset
from transformers import ClapModel, ClapProcessor

librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
audio_sample = librispeech_dummy[0]


model = ClapModel.from_pretrained("laion/clap-htsat-fused").to(0)
processor = ClapProcessor.from_pretrained("laion/clap-htsat-fused")


inputs = processor(audios=audio_sample["audio"]["array"], return_tensors="pt").to(0)
audio_embed = model.get_audio_features(inputs)

Citation

If you are using this model for your work, please consider citing the original paper:

code
@misc{https://doi.org/10.48550/arxiv.2211.06687,
  doi = {10.48550/ARXIV.2211.06687},
  
  url = {https://arxiv.org/abs/2211.06687},
  
  author = {Wu, Yusong and Chen, Ke and Zhang, Tianyu and Hui, Yuchen and Nezhurina, Marianna and Berg-Kirkpatrick, Taylor and Dubnov, Shlomo},
  
  keywords = {Sound (cs.SD), Audio and Speech Processing (eess.AS), FOS: Computer and information sciences, FOS: Computer and information sciences, FOS: Electrical engineering, electronic engineering, information engineering, FOS: Electrical engineering, electronic engineering, information engineering},
  
  title = {Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation},
  
  publisher = {arXiv},
  
  year = {2022},
  
  copyright = {Creative Commons Attribution 4.0 International}
}

View official source
Join our Telegram