audiobox aesthetics

---
license: cc-by-4.0
pipeline_tag: audio-classification
tags:

• model_hub_mixin

• pytorch_model_hub_mixin

This model has been pushed to the Hub using the PytorchModelHubMixin integration:

• Code: https://github.com/facebookresearch/audiobox-aesthetics

• Paper: https://huggingface.co/papers/2502.05139

--- README below copied from https://github.com/facebookresearch/audiobox-aesthetics

audiobox-aesthetics

 

Unified automatic quality assessment for speech, music, and sound.

• Paper arXiv / MetaAI.

• Blogpost ai.meta.com

<img src="assets/aes_model.png" alt="Model" height="400px">

Installation

1. Install via pip

pip install audiobox_aesthetics

2. Install directly from source

This repository requires Python 3.9 and Pytorch 2.2 or greater. To install, you can clone this repo and run:

pip install -e .

Pre-trained Models

Model | S3 | HuggingFace
|---|---|---|
All axes | checkpoint.pt | HF Repo

Usage

How to run prediction using CLI:

1. Create a jsonl files with the following format

{"path":"/path/to/a.wav"}
 {"path":"/path/to/b.flac"}
 ...
 {"path":"/path/to/z.wav"}

{"path":"/path/to/a.wav", "start_time":0, "end_time": 5}
 {"path":"/path/to/b.flac", "start_time":3, "end_time": 10}

2. Run following command

audio-aes input.jsonl --batch-size 100 > output.jsonl

If you have SLURM, run the following command

audio-aes input.jsonl --batch-size 100 --remote --array 5 --job-dir $HOME/slurm_logs/ --chunk 1000 > output.jsonl

3. Output file will contain the same number of rows as input.jsonl. Each row contains 4 axes of prediction with a JSON-formatted dictionary. Check the following table for more info:

Axes name | Full name
|---|---|
CE | Content Enjoyment
CU | Content Usefulness
PC | Production Complexity
PQ | Production Quality

Output line example:

{"CE": 5.146, "CU": 5.779, "PC": 2.148, "PQ": 7.220}

4. (Extra) If you want to extract only one axis (i.e. CE), post-process the output file with the following command using jq utility:

``jq '.CE' output.jsonl > output-aes_ce.txt

### How to run prediction from Python script or interpreter

1. Infer from file path

2. Infer from torch tensor

## Evaluation dataset
We released our evaluation dataset consisting of 4 axes of aesthetic annotation scores.

Here, we show an example of how to read and re-map each annotation to the actual audio file.

1. Recognize the dataset name from data_path. In the example, it is LibriTTS.
2. Replace "/your_path/" into your downloaded LibriTTS directory. 
3. Each axis contains 10 scores annotated by 10 different human annotators.

data_path | URL
|---|---|
LibriTTS |  https://openslr.org/60/
cv-corpus-13.0-2023-03-09 | https://commonvoice.mozilla.org/en/datasets
EARS | https://sp-uhh.github.io/ears_dataset/
MUSDB18 | https://sigsep.github.io/datasets/musdb.html
musiccaps | https://www.kaggle.com/datasets/googleai/musiccaps
(audioset) unbalanced_train_segments | https://research.google.com/audioset/dataset/index.html 
PAM | https://zenodo.org/records/10737388
License
The majority of audiobox-aesthetics is licensed under CC-BY 4.0, as found in the LICENSE file.
However, portions of the project are available under separate license terms: https://github.com/microsoft/unilm is licensed under MIT license.
Citation
If you found this repository useful, please cite the following BibTeX entry.

Acknowledgements