Integrate AI chat for helping users creating OpenSCAD scripts

[t-paul]

Outline

Add optional AI support that can help users writing OpenSCAD scripts. There are a number of ways AI models can help, e.g. by assisting with trigonometry and other math problems, explaining existing scripts users want to modify or describing image output for vision impaired users.

Details

There is already a simple prototype for a chat window that can be used a the base for the project.

Scope of the project:

• Focus on interfacing with local models (e.g. via a local Ollama instance)
• Basic chat interaction
• Optional sending the current script with the chat message
• Optional sending of the current view-port along with the chat message

Non-Goals:

• Integrating with commercial cloud providers
• Training or modifying AI models
• Using Python for AI interactions

Expected Outcome

Simple functionality for AI chat with some basic support for interacting with the current OpenSCAD project.

Future Possibilities

Extend the functionality by allowing the AI to interact directly with the current code using function calling.

Project Properties

Skills

• Programming language is C++
• Understanding of network APIs using JSON for encoding data
• Basic understanding on AI and how it is accessed from an application

Difficulty

Medium

Size

Long (350h)

Additional Information

Potential mentor(s): Marius Kintel (IRC: kintel), Torsten Paul (IRC: teepee), Ryan Colyer (IRC: InPhase)
Organization website: https://www.openscad.org/

Links:

• Prototype Pull Request: Ai chat window openscad/openscad#6118
• Ollama API: https://github.com/ollama/ollama/blob/main/docs/api.md
• Ollama tool calling: https://docs.ollama.com/capabilities/tool-calling

[KeithSloan]

How will this be different from openscad-studio ?

openscad-studio

[LJitendra05]

I am very interested in this project about adding optional AI support in OpenSCAD to help users understand and write scripts. I like the idea of using local AI models like Ollama to assist with math problems, explain code, and describe the viewport output without using any cloud services.

I am also excited to work with the existing chat window prototype and try to improve it by allowing users to optionally send the current script or viewport data with their queries. This can help users debug, modify, or learn OpenSCAD scripts more easily.

I would like to know more about how the current prototype works and how the C++ application is expected to communicate with local AI models.

[Ashwani1330]

Hi,
@t-paul @kintel @rcolyer

I’ve been hacking on the ai-chat-window branch locally and wanted to share a quick prototype. I’m trying to push the dock past just being a "floating chat window" and get it genuinely integrated with OpenSCAD's workflow, which I think aligns with the direction #113 is aiming for (and addresses the maintainer feedback from openscad/openscad#6108).

I got it running with a local Ollama model (OpenAI-compatible endpoint). Here’s what’s actually working right now:

• Local-first: It talks to http://localhost:11434/v1 (tested with llama3.2:3b) using the existing Preferences -> AI settings.
• Editor context (opt-in): Added "Include script" and "Selection only" toggles to the dock. When checked, the model gets the current text as an ephemeral context block, which helps with the immediate task without permanently polluting the chat history.
• Live Preview hookup: Added a new Preview button right in the dock that triggers OpenSCAD’s standard preview pipeline (like hitting F5). The loop is now: ask -> apply -> preview, all without leaving the window.
• Fenced code applies: If the assistant replies with a ```scad block, an "Apply" button pops up. It applies to either the selection or the whole document, but prompts for confirmation first so edits stay user-approved.

I recorded a quick demo video (attached below) showing the full loop.

Next things on my list to tackle:

• Passing compile errors/warnings (console + error log) so the assistant can fix real OpenSCAD errors in a tight loop.
• Adding the active filename and a list of open files as optional context.
• Adding streaming responses (waiting for full completion feels a bit laggy right now).
• Making the "Apply" step safer with a diff/patch preview instead of a blind replace.

Longer-term ideas:
Further down the line, I think it'd be incredibly powerful to feed the model structured context directly from the AST or instantiated CSG tree. Right now, the LLM only sees the raw .scad text, which means it has to guess how loops, variables, and module calls actually compile. If we can pass it a lightweight, textual summary of the evaluated scene graph—like the actual hierarchy of unions, differences, and base primitives with their final parameters—the model would get a deterministic view of the geometry. This would give it a massive boost in spatial reasoning and make it much better at localized tweaks, since it would actually "know" what shapes are in the scene instead of just reading code.

Quick question: would you eventually be open to an opt-in VLM (Vision-Language Model) agent loop? The idea is that the assistant could grab a screenshot of the 3D viewport, evaluate whether the rendered geometry actually matches the prompt, and automatically self-correct the OpenSCAD code if it missed the mark. Basically, an autonomous visual debugging loop.

Demo video:

demo_ai.mp4

[t-paul]

@Ashwani1330 Nice demo, looks like there's no worries about the coding part. If you would like to continue with the topic as part of GSoC, we will need to go through the application process so lets also do the human talks to human thing on the IRC channel 😀.

[Ashwani1330]

@Ashwani1330 Nice demo, looks like there's no worries about the coding part. If you would like to continue with the topic as part of GSoC, we will need to go through the application process so lets also do the human talks to human thing on the IRC channel 😀.

Thanks @t-paul — really glad you liked the demo.
I'd love to continue this for GSoC. I’ll join the IRC channel — my IRC nick will be Ashwani1330.