feat: custom IPEX-LLM Ollama Dockerfile, Intel GPU tuning, and VRAM/context docs

[eSlider]

Closes #37

Summary

Project Structure

Refactored into a clean <service>/Dockerfile + docker-compose.<service>.yml convention:

.
├── docker-compose.yml                # Main stack: Ollama (IPEX-LLM) + Open WebUI
├── docker-compose.sycl-ollama.yml    # SYCL-from-source Ollama + Open WebUI (alternative)
├── docker-compose.comfyui.yml        # ComfyUI image generation
├── docker-compose.sdnext.yml         # SD.Next image generation
├── docker-compose.whisper.yml        # OpenAI Whisper speech recognition
├── docker-compose.ramalama.yml       # RamaLama support
│
├── ipex-ollama/Dockerfile            # IPEX-LLM bundle build (Ollama v0.9.3, SYCL)
├── sycl-ollama/                      # SYCL-from-source build (Ollama v0.16.1)
│   ├── Dockerfile                    # Multi-stage: oneAPI build → minimal runtime
│   ├── patch-sycl.py                 # API compat patches (no-op since v0.16.1)
│   ├── start-ollama.sh               # Legacy entrypoint
│   └── test-glm-ocr.sh              # Vision model test script
│
├── docs/
│   ├── sycl-vs-vulkan.md             # SYCL vs Vulkan backend comparison
│   └── intel-arc-a770-context-limits.md  # VRAM & context length guide
└── ...

Custom Dockerfiles

• ipex-ollama/Dockerfile — IPEX-LLM bundle-based image (Ollama v0.9.3):

  • BuildKit # syntax=docker/dockerfile:1.4 with --mount=type=cache for apt and download caching
  • ARG version pins for all Intel GPU runtime components (bump in one place)
  • Latest runtimes: Level Zero v1.28.0, IGC v2.28.4, compute-runtime 26.05.37020.3, IPEX-LLM v2.3.0b20250725
  • Clean ENV section — no duplicates, no unverified vars, NPU disabled by default

• sycl-ollama/Dockerfile — SYCL-from-source image (Ollama v0.16.1):

  • Multi-stage build: Stage 1 compiles ggml-sycl with Intel oneAPI icpx, Stage 2 is minimal runtime
  • sycl-ollama/patch-sycl.py — backward-compatible API patching (no patches needed since v0.16.1 — APIs converged)
  • GGML commit ec98e200 (llama.cpp tag b7437) matching Ollama v0.16.1
  • Drops in libggml-sycl.so + stripped oneAPI runtime libs alongside official Ollama binary
  • Includes test-glm-ocr.sh vision model test script

Docker Compose

• docker-compose.yml (main stack):

  • All env vars configurable via ${VAR:-default} syntax (override with .env or shell)
  • shm_size: "16G" for SYCL/Level Zero shared memory (Docker defaults to 64 MB)
  • no_proxy / NO_PROXY on all services — prevents corporate/system HTTP proxies from intercepting container-to-container traffic
  • Intel GPU perf tuning (SYCL immediate command lists, SDP fusion, persistent cache, XeTLA)
  • Ollama context/memory management (context length, KV cache quantization, flash attention)
  • Detailed inline comments explaining each variable's impact
  • Restored full open-webui service with OLLAMA_BASE_URL, RAG web search, telemetry opt-out

• docker-compose.sycl-ollama.yml (alternative stack):

  • Self-contained alternative using SYCL-from-source build (Ollama v0.16.1)
  • Same env-var driven defaults, no_proxy, and Open WebUI config
  • Shares ollama-volume so models persist when switching between stacks

Documentation

• docs/sycl-vs-vulkan.md — SYCL vs Vulkan comparison:

  • Performance benchmarks (SYCL 40–100% faster on Intel Arc)
  • Three backend options: IPEX-LLM bundle, SYCL from source, upstream Vulkan
  • Full Dockerfile snippets showing the multi-stage SYCL build
  • How patch-sycl.py works (and why it's a no-op since v0.16.1)
  • Step-by-step guide for updating to new Ollama versions
  • Troubleshooting (device detection, ABI mismatch, OOM, kernel 6.18+ regression)

• docs/intel-arc-a770-context-limits.md — VRAM & context guide:

  • VRAM budget breakdown (weights + KV cache + overhead)
  • Context length vs VRAM trade-off tables (f16 / q8_0 / q4_0)
  • Recommended settings by model size (7B, 13B, 30B+) for 16 GB Arc
  • Environment variables reference for all Ollama and Intel tuning knobs
  • Building a custom image section with version pin table

• README.md:

  • Tested Hardware table, Documentation section, Project Structure tree
  • SYCL-from-source setup command and validate output in Setup section
  • Fixed broken Whisper markdown link

Build & test verification (2026-02-16)

IPEX-LLM stack (ipex-ollama)

• docker build -t ipex-ollama:latest ./ipex-ollama/ — all layers build successfully
• Ollama v0.9.3 starts and registers all API routes
• Intel GPU detected at startup (using Intel GPU)
• Server listens on 0.0.0.0:11434

SYCL-from-source stack (sycl-ollama)

• docker compose -f docker-compose.sycl-ollama.yml build — all stages pass (~87s)
• patch-sycl.py exits with code 0 — no patches needed (APIs converged in v0.16.1)
• ggml-sycl compiled successfully → libggml-sycl.so built and stripped
• Ollama v0.16.1 starts, discovers SYCL backend:

  name=SYCL0 description="Intel(R) Arc(TM) Graphics" type=discrete total="28.0 GiB"
  

• ollama list returns models from shared volume (7 models loaded)
• Inference test passed: ollama run llama3.2:1b responded correctly using SYCL0 compute buffer (1074 MiB allocated)
• Container cleans up properly with docker compose down

Test plan (remaining manual checks)

• Build SYCL-from-source stack: docker compose -f docker-compose.sycl-ollama.yml up --build
• Verify Ollama v0.16.1 responds
• Run a model and confirm correct inference output
• Confirm Intel GPU is used (SYCL0 device in logs)
• Verify patch-sycl.py exits with code 0 during build
• Run the IPEX-LLM stack: docker compose up -d
• Verify Open-WebUI loads at http://localhost:4040 and connects to Ollama
• Test with custom env overrides (e.g. OLLAMA_CONTEXT_LENGTH=8192 in .env)
• Verify no_proxy prevents proxy interference on corporate networks

[eSlider]

Test Patched Ollama 16.1(SYCL) on Intel Core Ultra 155H :

[larsblumberg]

Impressive work, thank you @eSlider!

I would love to test this PR on a Ultra 7 255H / Arc 140T.

How do I go best about testing all of these changes so that I can report back here?

[harpsychord]

This is incredible work @eSlider! I was able to use your branch as-is to use my Arc B580 (12GB VRAM) and have qwen2.5-coder:7b running very smoothly. I'm still quite new to all of this but I wanted to say thank you.

[eleiton]

Thanks for the PR @eSlider, happy to receive contributions to the codebase.
Can we split the PR into two: one for the new sycl-ollama proposal, and another for the ipex-ollama proposal? This should help keep things cleaner and make progress in smaller chunks.
Ideally we can keep this one for the sycl. I'll make some comments to the PR.

[eleiton]

Can this file be removed? I believe it's not needed for running the project.

[eleiton]

Can this file be removed? I believe it's not needed for running the project

[eleiton]

I believe this patch is needed still, and will probably be needed moving forward for a while.
I tested with Ollama 0.16.3 and GGML_COMMIT=ef83fb8601229ff650d952985be47e82d644bfaa

[eleiton]

This one on the other hand, I believe is not needed, as you suggest in the documentation.
So I think ideally this can either me removed, or the documentation updated to mention one is needed for all versions and the other only for older versions?

[eleiton]

What is this for loop used for?

[eleiton]

Any reason for this timezone?

[eSlider]

@eleiton, absolutely not.

[eleiton]

Ollama v0.16.3 requires the 'int batch_size' parameter that upstream llama.cpp at commit ef83fb86 doesn't have. The patch script detects this and applies the fix.

[eSlider]

Thank you all for your reviews, especially @eleiton!
I'll clean up the PR after work today!

[eSlider]

Thanks for the PR @eSlider, happy to receive contributions to the codebase. Can we split the PR into two: one for the new sycl-ollama proposal, and another for the ipex-ollama proposal? This should help keep things cleaner and make progress in smaller chunks. Ideally we can keep this one for the sycl. I'll make some comments to the PR.

The purpose of the research is to find the fastest calculation approach (inference method) by means of comparison. How would you suggest dividing up the competitors in order to then create a benchmark?

I'll leave PR as WIP for now.

[StevenIsaacs]

I would like to use an existing directory containing previously downloaded ollama models to avoid having to download models yet again. I'm running more than one machine and copy models between machines to avoid download delays on a 10Mb link (yeah, I'm in the boonies). To do so a new environment variable "OLLAMA_MODELS" (or something better) can be used which defaults to "{}". Then change the snippet from docker-compose-sycl-ollama.yml:

volumes:
  ollama-volume: {}
  open-webui-volume: {}

to:

volumes:
  ollama-volume: ${OLLAMA_MODELS}
  open-webui-volume: {}

I'm not a docker expert by any means so could be totally off base. But a suggestion. I hope you get the idea.

[rbrowning85]

Thank you for your hard work. I was able to get this up and running, but I am having some trouble with my dual GPU setup (Intel Arc a580 and a380).

I am trying to troubleshoot the error messages right now with Claude.

[rbrowning85]

Looking through the logs I found the following entries:

The program was built for 1 devices Build program log for 'Intel(R) Arc(TM) A380 Graphics': Exception caught at file:/ollama/ml/backend/ggml/ggml/src/ggml-sycl/ggml-sycl.cpp, line:3957 Error OP RMS_NORM

Is this a limitation of Ollama or a limitation of SYCL or just how this new project was built?

[rbrowning85]

I spent several hours with Claude troubleshooting the problem and rebuilding the dockerfile a least a dozen times. I am not a developer, so the process quickly went above my head, but I asked Claude to summarize our steps in the hopes that it would help others. Please let me know if you have any questions or any new builds that you would like me to try that support multiple GPUs.

`# Multi-GPU Issue: Intel Arc A580 + A380 with deepseek-r1:14b

Hardware

• Intel Arc A580 (8 GiB VRAM) — SYCL0
• Intel Arc A380 (6 GiB VRAM) — SYCL1
• Unraid host, Docker container

What Works

• Both GPUs are detected at startup (SYCL0, SYCL1)
• Small models (≤~5GB e.g. llama3.2:1b, llama3.2:3b) run successfully across both GPUs
• Layer distribution works correctly (layers split across SYCL0/SYCL1/CPU)
• 16k context works fine for smaller models

The Problem

Larger models (tested: deepseek-r1:14b at 8.37 GiB) crash at inference time with:

The program was built for 1 devices
Build program log for 'Intel(R) Arc(TM) A380 Graphics':
Exception caught at file:/ollama/ml/backend/ggml/ggml/src/ggml-sycl/ggml-sycl.cpp, line:3957
Error OP RMS_NORM

The model loads successfully — all layers are assigned and KV cache allocated across both GPUs — but crashes on the first inference call when the A380 tries to execute a RMS_NORM kernel.

Root Cause Analysis

The error "built for 1 devices" is a runtime SYCL program cache issue. ggml-sycl compiles GPU kernels (as OpenCL/SYCL programs) keyed to the first device it encounters (A580). When the A380 tries to execute those same cached programs, it fails because they were compiled in the context of a single device. The RMS_NORM operation is simply the first kernel that hits this issue.

Fix Attempts

Attempt 1: CMAKE_CXX_FLAGS with spir64_gen + AOT

Added AOT compilation targets for both GPU architectures at cmake configure time:

-DCMAKE_CXX_FLAGS="-fsycl-targets=spir64_gen,spir64 -Xs \"-device acm-g10,acm-g11\""

Result: Failed — spir64_gen requires ocloc which wasn't present in the build stage.

Attempt 2: Install ocloc in builder stage

Added Intel compute-runtime (intel-ocloc, intel-igc-*) to the sycl-builder stage so AOT compilation would have the required tools.

Result: Failed — ocloc version mismatch with oneAPI 2025.1.1:

Invalid option (arg 9): -ze-intel-greater-than-4GB-buffer-required

The oneAPI 2025.1.1 compiler passes a flag that compute-runtime 26.05.37020.3's ocloc does not support.

Attempt 3: Drop spir64_gen, use portable spir64 only

Removed ocloc requirement entirely, switched to portable SPIR-V IR:

-DCMAKE_CXX_FLAGS="-fsycl -fsycl-targets=spir64"

The idea: each GPU JITs from portable IR independently, so no shared compiled program.

Result: Build succeeded, but crash persisted. CMAKE_CXX_FLAGS is overridden by ggml's internal target_compile_options on the ggml-sycl target.

Attempt 4: Patch CMakeLists.txt directly

Used sed to inject -fsycl-targets=spir64 at the target level in ggml-sycl's CMakeLists.txt before the build:

sed -i \
  '/target_link_libraries(ggml-sycl/i target_compile_options(ggml-sycl PRIVATE -fsycl -fsycl-targets=spir64)\ntarget_link_options(ggml-sycl PRIVATE -fsycl -fsycl-targets=spir64)' \
  ml/backend/ggml/ggml/src/ggml-sycl/CMakeLists.txt

Result: Build succeeded with flags applied at the right level, but crash still persisted. Confirmed the "built for 1 devices" error is a runtime program cache issue, not a compile-time flag issue.

Attempt 5: Clear SYCL runtime cache

Searched for and cleared any persistent JIT kernel cache:

find /root -name '*.bin' -path '*/sycl*' -delete
find /tmp -name '*.bin' -path '*/sycl*' -delete

Also set SYCL_CACHE_PERSISTENT=0 to disable persistent caching entirely.

Result: No change. Cache was not the issue.

Conclusion

The "built for 1 devices" error originates inside ggml-sycl's runtime program building logic in ggml-sycl.cpp at line 3957. The fix likely requires a source-level patch to ensure SYCL program objects are built for all active devices rather than just the first one. This is beyond what Dockerfile build flags can address.

Workaround

Limit to a single GPU (the A580) via environment variables:

ONEAPI_DEVICE_SELECTOR=level_zero:0
ZE_AFFINITY_MASK=0

This allows larger models to run on the A580's 8 GiB alone. The A380 is unused in this configuration.

Environment

• Ollama: v0.16.1 (sycl-ollama build from this PR)
• oneAPI basekit: 2025.1.1
• Intel compute-runtime: 26.05.37020.3
• Level Zero: 1.28.0
• IGC: 2.28.4
• GGML commit: ec98e200 (llama.cpp tag b7437)`

[rbrowning85]

as a follow-up to my last post.

[SOLVED] Intel Arc SYCL Multi-GPU Deduplication (SameBackendDevice) & Tensor Splitting Fix

A community fix developed by a joint collaboration between Antigravity (Gemini 3.1 Pro), Claude (Sonnet 4.6), and Codex (GPT-5.4).

---

1. The Symptoms

If you are running Ollama with heterogeneous Intel Arc GPUs (for example, combining an A580 and an A380) natively over SYCL (Level Zero) via a Docker container, you may have encountered a silent multi-GPU failure:

1. Setting ONEAPI_DEVICE_SELECTOR=level_zero:0,1 is not respected.
2. The Ollama daemon logs detect BOTH cards initially (initial_count=2).
3. But the secondary card (the A380) vanishes immediately from the inference pool with zero error logs, leaving you with restricted VRAM and failing to split larger models.

2. The Root Cause (Silent Deduplication)

Through extensive source code tracing, our multi-agent AI framework located the root cause in the Go-layer abstractions, not the Intel drivers.

The flaw exists in ml/device.go inside the ml.DeviceInfo.Compare() deduplication function.
When the backend discover/runner.go collects hardware lists via RPC, the upstream ggml-sycl.cpp implementation (specifically inside ggml_backend_sycl_device_get_props) completely fails to assign the props->id and props->device_id values - returning empty strings ("").

The Go deduplicator compares multiple devices. Because SYCL0 and SYCL1 share the same library string but both report their identity as "", Go assumes they are clones of the same physical hardware wrapper. This triggers a silent deduction internally, erasing SYCL1 from the active array.

3. The Solution

To unify the VRAM (13.2 GiB on our A580+A380) and enable heterogeneous tensor splitting, we applied a surgical C++ identity injection hook and accompanying environment tweaks.

Part 1: The C++ Identity Hook (patch-sycl-v2.py)

When compiling the sycl-runner Docker image, we intercepted the build sequence with a custom Python script that manipulates ggml-sycl.cpp. By forcing dynamic struct definitions, we force the SYCL backend to generate unique "sycl:0" and "sycl:1" identity attributes.

This provides Compare() with enough identity data to return UniqueDevice status.

dev_ctx->id = "sycl:" + std::to_string(i);
dev_ctx->library = "SYCL";
dev_ctx->device_id = "sycl:" + std::to_string(i);
props->id = ctx->id.c_str();
props->device_id = ctx->device_id.c_str();
props->library = ctx->library.c_str();

We also included a diagnostic hook for ggml_sycl_has_mixed_device_topology(), which disables tensor-reorder tracking optimization to prevent secondary cross-device cache invalidations.

Part 2: OOM Safeguards & Context Isolation (.env)

While the identity patch enables the discovery of both cards, stability on disparate silicon requires environmental isolation to prevent queue stalls and out-of-memory (OOM) failures:

# JIT / Level Zero cache separation
SYCL_CACHE_PERSISTENT=0
SYCL_CACHE_IN_MEM=0
SYCL_ENABLE_DEFAULT_CONTEXTS=0
SYCL_RT_WARNING_LEVEL=1
ZE_ENABLE_PCI_ID_DEVICE_ORDER=1
ZE_FLAT_DEVICE_HIERARCHY=FLAT

# Conservative stability constraints (Working Production Config)
OLLAMA_NUM_PARALLEL=1
OLLAMA_MAX_LOADED_MODELS=1
OLLAMA_CONTEXT_LENGTH=8192

(Note: Flash Attention was logged as disabled during the successful run over this SYCL architecture).

4. The Results

Upon rebooting via docker compose up --build -d --force-recreate:

msg="inference compute" id=sycl:0 library=SYCL name=SYCL0 description="Intel(R) Arc(TM) A580 Graphics" available="7.5 GiB"
msg="inference compute" id=sycl:1 library=SYCL name=SYCL1 description="Intel(R) Arc(TM) A380 Graphics" available="5.6 GiB"
msg="vram-based default context" total_vram="13.2 GiB" 

The Go engine accepted BOTH indices as unique devices. We demonstrated a successful dual-GPU inference run using Qwen 2.5 14B, spilling 3.7GB onto the A580 and the remaining 2.3GB directly onto the A380. The KV layer map correctly assigned compute layers across both cards, achieving stable generation where it previously failed.

[eSlider]

Unfortunately, a few weeks ago I burned out the iGPU on my
Intel Core Ultra 7 155H, most likely due to overheating, and the laptop stopped turning on.

I’d appreciate it if someone could take on the task of adding or using these improvements in a PR.

My plans regarding whether to buy an Intel Arc GPU or an iGPU are still up in the air.

[rbrowning85]

I am currently working on trying to get it to work with the new Ollama releases. I am having trouble getting qwen3.5:9b to run.

I am happy to share my findings and files, but I've never managed a GitHub repository. What would be the best way to hand off?

[rbrowning85]

I dont want to admin how many hours and tokens I have spent across multiple providers trying to get this to work, but long story short, my AI finally sent me the follow:

The experiment is officially over. That deterministic gibberish is the final piece of proof we needed.

By stripping all the python memory patches back to native malloc_device and strictly filtering out the bounds errors, we tested exactly what the upstream llama.cpp developers compiled at that commit block. If it still hallucinates after that, it undeniably proves the underlying SYCL scheduler in version 0.20.6 is mathematically broken for dual-GPU topologies. The ggml event tracker is launching graph arrays across the two Intel Arc cards without properly synchronizing the memory streams, meaning the neural network is literally doing math on empty memory.

We can't fix a broken OpenCL scheduling pipe from outside the C++ compiler. You have officially outgrown the Intel Arc ecosystem.

It then went on to tell me to sell my Intel Arc GPUs and purchase new two Nvidia GPUs. LOL

I saw in a recent Unraid Uncast show that Spaceinvaderone created a new Unraid app specific to Ollama with Arc GPUs. I am going to give this a whirl, but based on the issues already posted in the gitbhub responsitory, it doesnt look promising.

https://github.com/SpaceinvaderOne/ollama-intel-gpu