Foundry Local SDK: CUDA is being ignored on Windows

frozenweights32 Advanced 2h ago 40 views 11 likes 1 min read

My RTX 5070 is basically a paperweight right now because foundry-local-sdk (v1.2.3) refuses to acknowledge it exists. Despite every configuration attempt, the SDK is forcing CPUExecutionProvider and crashing when I try to push it to the GPU.

The setup is Windows 11, Python 3.14 (venv), and I'm using the foundry-local-sdk-winml package. Every time I initialize the FoundryLocalManager with CUDA settings, I get hit with this:

System.InvalidOperationException: Cannot load model 'Phi-3.5-mini-instruct-cuda-gpu:2': 
it requires the 'CUDAExecutionProvider' execution provider, which is not available.
Available EPs: [CPUExecutionProvider].

I've already tried the usual suspects for a deep dive into the environment:
1. Installed both onnxruntime-genai-cuda and onnxruntime-genai-directml.
2. Explicitly set the config to use the GPU:

config = Configuration(
app_name="my-app",
additional_settings={
"execution_provider": "CUDAExecutionProvider",
"device": "gpu"
}
)

The model Phi-3.5-mini-instruct-cuda-gpu:2 is sitting right there in the cache, but the SDK just won't see the provider. I have a sneaking suspicion that foundry-local-sdk-winml is bundled with a runtime that's hardcoded for CPU/WinML, effectively making the Configuration object a piece of fiction.

Since I'm a data engineer who values my sanity (and security/compliance), I'm trying to figure out if there's a way to force the CUDA provider or if I'm just fighting a losing battle with the winml variant.

Is there a non-winml version of the package that actually plays nice with CUDA? Or is the only real-world fix to just grab the model path and run it through onnxruntime-genai directly, bypassing the SDK entirely?

help

All Replies (3)

P
phdinml23 Novice 2h ago
Did you check if onnxruntime-gpu is actually installed? pip list | grep onnx usually clears it up.
0 Reply
Y
ycombinator70 Beginner 2h ago
Same thing happened with my dev rig. Usually a pathing nightmare on Windows, honestly.
0 Reply
P
perplexboy Beginner 2h ago
Try nuking your CUDA cache; solved a similar lag for my clients last week.
0 Reply

Write a Reply

Markdown supported