oneformer cityscapes swin large

Providershi-labs
Categoryimage-segmentation
Licensemit
Downloads107.4K
Stars0

Overview

OneFormer Cityscapes Swin-Large is a high-performance image segmentation model designed for precise semantic, instance, and panoptic segmentation within urban environments. Built on a Swin-Large transformer backbone, it leverages a universal architecture to handle multiple segmentation tasks through a single framework, eliminating the need for task-specific heads. For developers working on autonomous driving or urban planning tools, this model offers a robust solution for identifying complex street-level geometries and object boundaries. It integrates well into PyTorch-based pipelines and provides a significant accuracy boost over traditional CNN-based segmentation models when deployed on hardware capable of supporting large-scale transformer weights.

Highlights

  • Unified framework for semantic, instance, and panoptic segmentation
  • Powerful Swin-Large backbone for superior spatial feature extraction
  • Optimized for high-precision urban scene parsing and analysis
  • MIT licensed for flexible commercial and research integration

Usage

Install
# Install Hugging Face transformers
pip install transformers torch
SDK Usage
# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("shi-labs/oneformer_cityscapes_swin_large")
tokenizer = AutoTokenizer.from_pretrained("shi-labs/oneformer_cityscapes_swin_large")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance 
pip install -U huggingface_hub

CLI Download

Download the full repository

Download the full repository 
huggingface-cli download shi-labs/oneformer_cityscapes_swin_large

Download a single file to a local folder (e.g. config.json into ./dir)

Download a single file to a local folder (e.g. config.json into ./dir) 
huggingface-cli download shi-labs/oneformer_cityscapes_swin_large config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

SDK Download 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('shi-labs/oneformer_cityscapes_swin_large')

Git Download

Make sure git-lfs is installed first

Git Download 
git lfs install
git clone https://huggingface.co/shi-labs/oneformer_cityscapes_swin_large

To skip LFS large-file downloads, use:

Skip LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/shi-labs/oneformer_cityscapes_swin_large

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

Install Transformers 
pip install -U transformers torch

Load the model and run inference

Load the model and run inference 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('shi-labs/oneformer_cityscapes_swin_large')
tokenizer = AutoTokenizer.from_pretrained('shi-labs/oneformer_cityscapes_swin_large')

Full Documentation

来源: HuggingFace

---
license: mit
tags:

  • vision

  • image-segmentation

datasets:
  • huggan/cityscapes

widget:
  • src: https://huggingface.co/datasets/shi-labs/oneformer_demo/blob/main/cityscapes.png

example_title: Cityscapes
---

OneFormer

OneFormer model trained on the Cityscapes dataset (large-sized version, Swin backbone). It was introduced in the paper OneFormer: One Transformer to Rule Universal Image Segmentation by Jain et al. and first released in this repository.

!model image

Model description

OneFormer is the first multi-task universal image segmentation framework. It needs to be trained only once with a single universal architecture, a single model, and on a single dataset, to outperform existing specialized models across semantic, instance, and panoptic segmentation tasks. OneFormer uses a task token to condition the model on the task in focus, making the architecture task-guided for training, and task-dynamic for inference, all with a single model.

!model image

Intended uses & limitations

You can use this particular checkpoint for semantic, instance and panoptic segmentation. See the model hub to look for other fine-tuned versions on a different dataset.

How to use

Here is how to use this model:

python
from transformers import OneFormerProcessor, OneFormerForUniversalSegmentation
from PIL import Image
import requests
url = "https://huggingface.co/datasets/shi-labs/oneformer_demo/blob/main/cityscapes.png"
image = Image.open(requests.get(url, stream=True).raw)

Loading a single model for all three tasks

processor = OneFormerProcessor.from_pretrained("shi-labs/oneformer_cityscapes_swin_large")
model = OneFormerForUniversalSegmentation.from_pretrained("shi-labs/oneformer_cityscapes_swin_large")

Semantic Segmentation

semantic_inputs = processor(images=image, task_inputs=["semantic"], return_tensors="pt")
semantic_outputs = model(semantic_inputs)

pass through image_processor for postprocessing

predicted_semantic_map = processor.post_process_semantic_segmentation(outputs, target_sizes=[image.size[::-1]])[0]

Instance Segmentation

instance_inputs = processor(images=image, task_inputs=["instance"], return_tensors="pt")
instance_outputs = model(instance_inputs)

pass through image_processor for postprocessing

predicted_instance_map = processor.post_process_instance_segmentation(outputs, target_sizes=[image.size[::-1]])[0]["segmentation"]

Panoptic Segmentation

panoptic_inputs = processor(images=image, task_inputs=["panoptic"], return_tensors="pt")
panoptic_outputs = model(**panoptic_inputs)

pass through image_processor for postprocessing

predicted_semantic_map = processor.post_process_panoptic_segmentation(outputs, target_sizes=[image.size[::-1]])[0]["segmentation"]

For more examples, please refer to the documentation.

Citation

bibtex
@article{jain2022oneformer,
      title={{OneFormer: One Transformer to Rule Universal Image Segmentation}},
      author={Jitesh Jain and Jiachen Li and MangTik Chiu and Ali Hassani and Nikita Orlov and Humphrey Shi},
      journal={arXiv}, 
      year={2022}
    }
View official source
Join our Telegram