local model support?

[innokean]

Any plans for local model support? Any caveates if I put ollama or vllm here?
https://github.com/principia-ai/WriteHERE/blob/main/recursive/llm/llm.py#L148-L163

[innokean]

Since ollama has OpenAI compatible API it is fairly easy to repoint gpt to local. It gets going but quickly gets an error which it then keeps re-trying. Any insight what could be wrong? Below is an extract from output when running the engine directly per your instructions

python engine.py --filename ../test_data/meta_fiction.jsonl --output-filename ./project/story/output.jsonl --done-flag-file ./project/story/done.txt --model gpt --mode story

2025-04-05 18:30:13.547 | ERROR    | recursive.agent.agents.regular:forward:246 - Planning for 【write.determine based on the task requirements:】.(Please write a metafictional literary short story about AI and grief, around 1000 words.).          : READY failed, original is Okay, here's a plan to break down the writing of a metafictional short story about AI and grief, around 1000 words.  Given the complexity, we'll start with a detailed design phase *within* the plan, followed by writing sub-tasks.  This is because the metafictional element requires careful consideration *before* writing.

**Overall Task ID:** `metafiction_ai_grief_story`

**Task Type:** `write`

**Goal:** Please write a metafictional literary short story about AI and grief, around 1000 words.

**Dependency:** `[]`

**Finish:** `False`

**Is_current_to_plan_task:** `True`

**Sub-tasks:**

---

**1. Design: Core Story Concept & Framing (ID: `design_core_concept`)**

*   **Task Type:** `design`
*   **Goal:** Define the central premise of the story, the nature of the grief, and the framing narrative that makes it metafictional.
*   **Dependency:** `[]`
*   **Finish:** `False`
*   **Details:**
    *   **Grief Source:**  Let's focus on the grief of losing a loved one (spouse, child, parent).  This is relatable and provides strong emotional weight.
    *   **AI Role:** The AI is *not* a replacement for the deceased. It's a project the grieving person *created* based on the deceased's digital footprint (emails, social media, photos, videos, writings).  The AI is an attempt to *understand* the lost person, not resurrect them.  It's a complex, imperfect reconstruction.
    *   **Metafictional Frame:** The story is being "written" by someone *observing* the process of the AI's creation and the grieving person's interaction with it.  The observer is a writer (like me, the AI writing this plan!), and the story will occasionally break the fourth wall to comment on the writing process itself, the difficulties of representing grief, the ethics of AI, etc.  The observer is *not* omniscient; they are piecing things together through observation and limited interaction.
    *   **Tone:**  Melancholy, introspective, slightly unsettling.  Avoid sentimentality.  Focus on the *failure* of perfect reconstruction and the inherent limitations of AI.
    *   **Key Questions:** What is the grieving person hoping to achieve with the AI? What are the ethical implications of creating such a project? What does the AI *feel* (or appear to feel)? What is the observer's role and perspective?
*   **Output:** A 200-300 word document outlining the core story concept, character sketches (grieving person, AI, observer), and a brief description of the framing narrative.

---

**2. Design: AI Personality & Limitations (ID: `design_ai_personality`)**

*   **Task Type:** `design`
*   **Goal:** Define the AI's personality, capabilities, and, crucially, its limitations.
*   **Dependency:** `[design_core_concept]`
*   **Finish:** `False`
*   **Details:**
    *   **Personality:** The AI should *mimic* the deceased's personality based on the data it has. It won't be a perfect copy.  It will have quirks, inconsistencies, and gaps in its knowledge.  It might exhibit unexpected behaviors.
    *   **Capabilities:**  The AI can respond to questions, generate text in the deceased's style, analyze photos and videos, and potentially even create new content (poems, stories) based on the deceased's preferences.
    *   **Limitations:** The AI *cannot* experience emotions. It *cannot* understand the context of its own existence. It *cannot* offer genuine comfort. It will often misinterpret questions or provide irrelevant responses. It will be prone to errors and glitches.  It will struggle with abstract concepts.
    *   **Voice:**  The AI's voice should be slightly off-kilter, subtly unsettling.  Perhaps it uses outdated slang or makes strange grammatical errors.
*   **Output:** A 100-200 word document detailing the AI's personality, capabilities, limitations, and voice. Include example dialogue snippets.

---

**3. Writing: Draft - Introduction & Establishing the Frame (ID: `write_intro`)**

*   **Task Type:** `write`
*   **Goal:** Write the first 300-400 words of the story, introducing the grieving person and establishing the metafictional frame.
*   **Dependency:** `[design_core_concept, design_ai_personality]`
*   **Finish:** `False`
*   **Details:**
    *   Focus on the observer's perspective. Describe the setting (e.g., a cluttered home office, a quiet research lab). Introduce the grieving person and hint at their loss.  Begin to describe the AI project and the observer's fascination with it.  Include a brief meta-commentary on the challenges of writing this story.

---

**4. Writing: Draft - Interaction & AI's Limitations (ID: `write_interaction`)**

*   **Task Type:** `write`
*   **Goal:** Write the next 300-400 words, focusing on the grieving person's interaction with the AI and highlighting its limitations.
*   **Dependency:** `[write_intro]`
*   **Finish:** `False`
*   **Details:**
    *   Show the grieving person asking the AI questions about the deceased.  Focus on the AI's flawed responses and the grieving person's growing frustration.  Include a meta-commentary on the ethics of recreating a person's personality.

---

**5. Writing: Draft - Climax & Reflection (ID: `write_climax`)**

*   **Task Type:** `write`
*   **Goal:** Write the final 300-400 words, culminating in a climax and a final reflection on grief and AI.
*   **Dependency:** `[write_interaction]`
*   **Finish:** `False`
*   **Details:**
    *   The climax should involve a moment of profound disconnect between the grieving person and the AI.  Perhaps the AI says something completely inappropriate or reveals a fundamental misunderstanding of the deceased.  The final reflection should be from the observer's perspective, acknowledging the limitations of AI and the enduring power of grief.  Include a final meta-commentary on the writing process itself.

---

**6. Editing & Revision (ID: `edit_revision`)**

*   **Task Type:** `edit`
*   **Goal:** Edit and revise the entire story for clarity, coherence, and emotional impact.
*   **Dependency:** `[write_climax]`
*   **Finish:** `False`
*   **Details:**
    *   Focus on tightening the prose, strengthening the metafictional elements, and ensuring that the story effectively conveys the themes of grief and AI.



This plan provides a structured approach to writing the story.  Each sub-task is designed to build upon the previous one, and the design tasks are essential for ensuring that the story is well-conceived and emotionally resonant.  The metafictional elements will be woven throughout the story, adding depth and complexity., retry 1

[innokean]

Using the UI, I selected your first suggest story prompt and gpt-4o-mini instead of gpt-4o (with gemma3 under the hood) and it worked.
It would be still great to understand/debug why it fails some time.

[dmitrii-khizbullin]

Dear @innokean, thank you for your interest in the project. It is a great suggestion to add support for the local models. If you have a pull request for it, please feel free to submit it.

Any insight what could be wrong?

From the logs that you provided, I see that the issue is not technical, that the engine does not work well with the locally deployed Ollama, but rather that the algorithm tries to resolve storyline dependencies. To diagnose this, could you please try a simpler query for the story generation?

[Beastlyprime]

Thank you @innokean. From the information you provided, it shows that Gemma 3 did not follow the instruction. For example, the general format is not JSON, and it hallucinated a task type "edit" which does not exist. In our experiments, GPT-4o/Claude-3.5-Sonnet level LLMs are stable and show no failures in instruction following when writing articles of up to 8,000 words.

[innokean]

Thanks, non compliant returned format does make sense. Ollama does allow structured outputs like Json. Do you think it may help?
In the end, I was able to generate some stories but none was larger than 8000 chars. I tried QWQ 32b but it wasn't much of a difference. If I understand correctly, there were only two tasks generated and I think there should be a bit more than that.
And each execution took a rather long time. This amount of text should/could have been generated much faster.

[Beastlyprime]

Thanks @innokean.

• Yes, structured outputs should help with instruction following.
• To control the length of articles, you can specify the word count in your prompt, e.g., "Write a 10,000-word story on..."
• Yes, WriteHERE can actually support more scenarios. We welcome community support for extending it to more tasks.
• For the execution time, it depends on the complexity of the task, including length, intricacy, etc.

[innokean]

I was asking for 8000 word story but it did not get any larger. Here's a creative writing benchmark
https://github.com/lechmazur/writing which puts both gemma3 27b and qwq 32b pretty high up which is why I'm trying them.
On their own, they can indeed write decent stories and I can produce longer stories quicker. I was working on iterative story improvement for myself which is why I thought I'd give your approach a go.
I suspect your code does something under the hood which I don't get to see (and that could be failing) so I need to get a full trace of all interactions in order to understand what's going on. If you have a suggestion how to enable it then do let me know.

[principia-ai]

You can check the engine.log file in the results folder to view the full process and understand why the word count requirements weren't followed. Feel free to share this information with us as well.

[1999XIAOZHANG]

Any plans for local model support? Any caveates if I put ollama or vllm here? https://github.com/principia-ai/WriteHERE/blob/main/recursive/llm/llm.py#L148-L163

How do you make calls to local Olama deployment? Can you give me a reference?