zoedepth nyu kitti

ProviderIntel
Categorydepth-estimation
Licensemit
Downloads642
Stars2

Overview

ZoeDepth is a zero-shot metric depth estimation model designed to bridge the gap between relative and absolute depth perception. Unlike traditional monocular depth models that output relative disparities, ZoeDepth leverages a combination of relative depth priors and metric fine-tuning on datasets like NYU Depth V2 and KITTI. For developers, this means the model can generalize across diverse indoor and outdoor scenes while providing actual distance measurements without requiring camera intrinsics for every new image. It is particularly useful for robotics, AR/VR spatial mapping, and 3D reconstruction pipelines where accurate scale is critical. Integration is straightforward via PyTorch, making it a robust alternative to MiDaS when absolute metric scale is a requirement rather than a luxury.

Highlights

  • Provides absolute metric depth instead of relative disparity
  • Zero-shot generalization across indoor and outdoor environments
  • Trained on NYU Depth V2 and KITTI datasets
  • Seamless integration with PyTorch-based vision pipelines
  • Permissive MIT license for commercial and research use

Usage

Install
# Install Hugging Face transformers
pip install transformers torch
SDK Usage
# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("Intel/zoedepth-nyu-kitti")
tokenizer = AutoTokenizer.from_pretrained("Intel/zoedepth-nyu-kitti")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance 
pip install -U huggingface_hub

CLI Download

Download the full repository

Download the full repository 
huggingface-cli download Intel/zoedepth-nyu-kitti

Download a single file to a local folder (e.g. config.json into ./dir)

Download a single file to a local folder (e.g. config.json into ./dir) 
huggingface-cli download Intel/zoedepth-nyu-kitti config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

SDK Download 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('Intel/zoedepth-nyu-kitti')

Git Download

Make sure git-lfs is installed first

Git Download 
git lfs install
git clone https://huggingface.co/Intel/zoedepth-nyu-kitti

To skip LFS large-file downloads, use:

Skip LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/Intel/zoedepth-nyu-kitti

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

Install Transformers 
pip install -U transformers torch

Load the model and run inference

Load the model and run inference 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('Intel/zoedepth-nyu-kitti')
tokenizer = AutoTokenizer.from_pretrained('Intel/zoedepth-nyu-kitti')

Model Download

We recommend downloading the model via the ModelScope CLI or SDK.

Guidance：Before downloading, install ModelScope with:

Guidance 
pip install modelscope

CLI Download

Download the full repository

Download the full repository 
modelscope download --model Intel/zoedepth-nyu-kitti

Download a single file to a local folder (e.g. README.md into ./dir)

Download a single file to a local folder (e.g. README.md into ./dir) 
modelscope download --model Intel/zoedepth-nyu-kitti README.md --local_dir ./dir

See the docs for more CLI options

SDK Download

SDK Download 
# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('Intel/zoedepth-nyu-kitti')

Git Download

Make sure git-lfs is installed first

Git Download 
git lfs install
git clone https://www.modelscope.cn/Intel/zoedepth-nyu-kitti.git

To skip LFS large-file downloads, use:

Skip LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/Intel/zoedepth-nyu-kitti.git

ModelScope 模型页直接下载模型文件；无需将模型文件放在本站服务器。

Notebook Quickstart

Install the ModelScope library

Install the ModelScope library 
pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

Load the model and run inference

Load the model and run inference 
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

p = pipeline('text-generation', 'Intel/zoedepth-nyu-kitti')

Full Documentation

来源: HuggingFace

---
license: mit
tags:

  • vision

pipeline_tag: depth-estimation
---

ZoeDepth (fine-tuned on NYU and KITTI)

ZoeDepth model fine-tuned on the NYU and KITTI datasets. It was introduced in the paper ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth by Shariq et al. and first released in this repository.

ZoeDepth extends the DPT framework for metric (also called absolute) depth estimation, obtaining state-of-the-art results.

Disclaimer: The team releasing ZoeDepth did not write a model card for this model so this model card has been written by the Hugging Face team.

Model description

ZoeDepth adapts DPT, a model for relative depth estimation, for so-called metric (also called absolute) depth estimation.

This means that the model is able to estimate depth in actual metric values.

<img src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/zoedepth_architecture_bis.png"
alt="drawing" width="600"/>

<small> ZoeDepth architecture. Taken from the <a href="https://arxiv.org/abs/2302.12288">original paper.</a> </small>

Intended uses & limitations

You can use the raw model for tasks like zero-shot monocular depth estimation. See the model hub to look for
other versions on a task that interests you.

How to use

The easiest is to leverage the pipeline API which abstracts away the complexity for the user:

python
from transformers import pipeline
from PIL import Image
import requests

load pipe

depth_estimator = pipeline(task="depth-estimation", model="Intel/zoedepth-nyu-kitti")

load image

url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

inference

outputs = depth_estimator(image)
depth = outputs.depth
For more code examples, we refer to the documentation.

BibTeX entry and citation info

bibtex
@misc{bhat2023zoedepth,
      title={ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth}, 
      author={Shariq Farooq Bhat and Reiner Birkl and Diana Wofk and Peter Wonka and Matthias Müller},
      year={2023},
      eprint={2302.12288},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
View official source
Join our Telegram