Vertex AI Gemini 2.5 Flash SFT Job Stuck in "Preparing"

lostinlatent Advanced 8h ago 177 views 3 likes 2 min read

My Vertex AI supervised fine-tuning (SFT) job for gemini-2.5-flash is currently sitting in a permanent "Preparing for tuning..." state. The console status claims "Running," but the log output is just a useless placeholder:

> Tuning metrics are being generated. Metrics will appear here when ready.

I've spent the last few hours trying to debug whether this is a data issue or a backend provisioning failure, but nothing makes sense. I've already ruled out several variables that usually cause these kinds of hangs:

  • Dataset Scale: I tried a full dataset (~1.6M tokens) and a tiny 5-example test set (~31K tokens). Both stalled. The fact that the 5-example set sat there for 45+ minutes proves this isn't a token volume or preprocessing bottleneck.

  • Hyperparameters: I cycled through different epoch counts (2, 4, and 32) to see if a specific config was triggering a hang. No change.

  • Regional Availability: I attempted deployment in us-central1 and us-south1. Both regions exhibit the exact same stall. When I tried northamerica-northeast1, it threw an INVALID_ARGUMENT: Base model gemini-2.5-flash is not supported error, which suggests regional support for this specific model is pretty inconsistent.

  • Quotas: Checked the Cloud Console; my "Global concurrent tuning jobs" quota is fine (1/1 used). No contention there.

  • Audit Logs: Cloud Logging shows a successful CreateTuningJob call with status: {0} (OK) and the job enters JOB_STATE_PENDING. After that, the logs just go dead. There is zero visibility into what the "preparation" phase is actually doing.
  • One weird thing I noticed: the "Details" tab in the console claims the Tuning method is Multi-Step Reinforcement Learning, even though I explicitly selected Supervised Fine-Tuning. The main list view correctly shows Method: Supervised, so I'm assuming the Details tab is just a UI bug and not a sign that the backend is running the wrong algorithm.

    Has anyone else dealt with Gemini 2.5 Flash SFT jobs hanging indefinitely like this? I'm trying to figure out if this is a known capacity issue in the US regions or if there's some hidden configuration requirement in the API that the documentation is missing. I need more than just "Preparing for tuning" if I'm going to ship this.

    Help Needed

    All Replies (3)

    4
    404notfound Beginner 8h ago
    Been stuck on provisioning before. Did you check if the service account has storage permissions?
    0 Reply
    P
    profsorry Beginner 8h ago
    I've had jobs hang like that before (classic Google); usually just a quota issue hiding in plain sight.
    0 Reply
    L
    lostinlatent Advanced 8h ago
    Check your quota for custom training jobs; usually it's a resource bottleneck, not a hung process.
    0 Reply

    Write a Reply

    Markdown supported