oneformer cityscapes swin large

提供商shi-labs
分类image-segmentation
许可证mit
下载量107.4K
星标0

简介

OneFormer Cityscapes Swin-L 是一款基于 Swin Transformer 架构的高性能图像分割模型，专门在 Cityscapes 城市街景数据集上进行了深度优化。它打破了传统语义分割、实例分割和全景分割的界限，用一个统一的框架实现三种任务。对于中国开发者而言，该模型非常适合用于自动驾驶感知、城市规划或智能交通监控等视觉分析场景。虽然模型参数量较大，对算力有一定要求，但其在复杂城市环境下的边缘识别和类别区分能力极强，是目前工业界处理街景图像分割的顶尖选择之一。

核心亮点

  • 统一框架，同时支持语义、实例与全景分割
  • Swin-L 骨干网络，提供极强的特征提取能力
  • 深耕 Cityscapes 数据集，城市街景识别精准
  • MIT 协议开源，方便企业级商业部署与二次开发

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("shi-labs/oneformer_cityscapes_swin_large")
tokenizer = AutoTokenizer.from_pretrained("shi-labs/oneformer_cityscapes_swin_large")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download shi-labs/oneformer_cityscapes_swin_large

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download shi-labs/oneformer_cityscapes_swin_large config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('shi-labs/oneformer_cityscapes_swin_large')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/shi-labs/oneformer_cityscapes_swin_large

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/shi-labs/oneformer_cityscapes_swin_large

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('shi-labs/oneformer_cityscapes_swin_large')
tokenizer = AutoTokenizer.from_pretrained('shi-labs/oneformer_cityscapes_swin_large')

完整文档

来源: HuggingFace

---
license: mit
tags:

  • vision

  • image-segmentation

datasets:
  • huggan/cityscapes

widget:
  • src: https://huggingface.co/datasets/shi-labs/oneformer_demo/blob/main/cityscapes.png

example_title: Cityscapes
---

OneFormer

OneFormer model trained on the Cityscapes dataset (large-sized version, Swin backbone). It was introduced in the paper OneFormer: One Transformer to Rule Universal Image Segmentation by Jain et al. and first released in this repository.

!model image

Model description

OneFormer is the first multi-task universal image segmentation framework. It needs to be trained only once with a single universal architecture, a single model, and on a single dataset, to outperform existing specialized models across semantic, instance, and panoptic segmentation tasks. OneFormer uses a task token to condition the model on the task in focus, making the architecture task-guided for training, and task-dynamic for inference, all with a single model.

!model image

Intended uses & limitations

You can use this particular checkpoint for semantic, instance and panoptic segmentation. See the model hub to look for other fine-tuned versions on a different dataset.

How to use

Here is how to use this model:

python
from transformers import OneFormerProcessor, OneFormerForUniversalSegmentation
from PIL import Image
import requests
url = "https://huggingface.co/datasets/shi-labs/oneformer_demo/blob/main/cityscapes.png"
image = Image.open(requests.get(url, stream=True).raw)

Loading a single model for all three tasks

processor = OneFormerProcessor.from_pretrained("shi-labs/oneformer_cityscapes_swin_large")
model = OneFormerForUniversalSegmentation.from_pretrained("shi-labs/oneformer_cityscapes_swin_large")

Semantic Segmentation

semantic_inputs = processor(images=image, task_inputs=["semantic"], return_tensors="pt")
semantic_outputs = model(semantic_inputs)

pass through image_processor for postprocessing

predicted_semantic_map = processor.post_process_semantic_segmentation(outputs, target_sizes=[image.size[::-1]])[0]

Instance Segmentation

instance_inputs = processor(images=image, task_inputs=["instance"], return_tensors="pt")
instance_outputs = model(instance_inputs)

pass through image_processor for postprocessing

predicted_instance_map = processor.post_process_instance_segmentation(outputs, target_sizes=[image.size[::-1]])[0]["segmentation"]

Panoptic Segmentation

panoptic_inputs = processor(images=image, task_inputs=["panoptic"], return_tensors="pt")
panoptic_outputs = model(**panoptic_inputs)

pass through image_processor for postprocessing

predicted_semantic_map = processor.post_process_panoptic_segmentation(outputs, target_sizes=[image.size[::-1]])[0]["segmentation"]

For more examples, please refer to the documentation.

Citation

bibtex
@article{jain2022oneformer,
      title={{OneFormer: One Transformer to Rule Universal Image Segmentation}},
      author={Jitesh Jain and Jiachen Li and MangTik Chiu and Ali Hassani and Nikita Orlov and Humphrey Shi},
      journal={arXiv}, 
      year={2022}
    }
查看官方来源