Update haskell.nix and use nixpkgs-2511

.github/workflows/aarch64-linux.yml

@@ -1,8 +1,7 @@
 name: DevX closures for aarch64-linux
 
 on:
-  # Disabled until we have a new aarch64-linux hydra builder
-  # push:
+  push:
 
 jobs:
   build:

.github/workflows/hello.yml

@@ -11,13 +11,11 @@ jobs:
       fail-fast: false
       matrix:
         platform:
-          - x86_64-darwin
           - x86_64-linux
-          # Skipping because we do not have runners for these set up.
-          #- aarch64-darwin
-          #- aarch64-linux
+          - x86_64-darwin
+          - aarch64-darwin
+          - aarch64-linux
         compiler-nix-name:
-          - ghc810
           - ghc96
           - ghc98
           - ghc910
@@ -34,9 +32,6 @@ jobs:
           - false
           - true
         exclude:
-          # Just cross compiling javascript with ghc 9.6 and above
-          - compiler-nix-name: ghc810
-            target-platform: "-js"
           # Windows cross compilation only works on x86_64 right now.
           - platform: aarch64-darwin
             target-platform: "-windows"

.github/workflows/lints.yml

@@ -42,7 +42,7 @@ jobs:
           nix_path: nixpkgs=channel:nixos-unstable
       - run: |
           nix profile install github:Kha/nixprof
-          nixprof record nix develop .#ghc8107 --accept-flake-config
+          nixprof record nix develop .#ghc96 --accept-flake-config
           nixprof report -p
           nixprof report -a
           nixprof report -s

CLAUDE.md

@@ -0,0 +1,246 @@
+# DevX Project - Claude Memory
+
+## Project Overview
+
+DevX provides pre-built Haskell development environments (devshells) for IOG projects, packaged and distributed via GitHub Container Registry (GHCR). It supports multiple GHC versions (9.6, 9.8, 9.10, 9.12), platforms (x86_64-linux, aarch64-linux, x86_64-darwin, aarch64-darwin), and target configurations (dynamic, static/musl, JavaScript/GHCJS, Windows/mingwW64).
+
+---
+
+## Current Issue: -env Job Generation Problem (PR #220)
+
+### Problem Statement
+
+The `-env` hydra jobs need to produce a wrapper script that contains all environment variables (as `declare -x VAR=value` lines) from a mkShell derivation. This wrapper script is then exported as a Nix store closure and uploaded to GHCR, allowing GitHub Actions runners to download and use pre-built development environments.
+
+### The Core Technical Challenge
+
+**`mkShell` does NOT produce a file with environment exports.** When you build a mkShell derivation, its output is a minimal placeholder file that documents build inputs - NOT a file containing `declare -x` lines.
+
+The `declare -x` environment lines are generated by `nix print-dev-env`, which:
+1. Takes a shell derivation as input
+2. Computes all environment variables that would be set
+3. Outputs them as `declare -x VAR=value` lines
+
+The challenge is running `nix print-dev-env` at build time without access to `recursive-nix`.
+
+### Evolution of Approaches (Last Two Commits)
+
+#### 1. Original Approach: recursive-nix (before HEAD~2)
+```nix
+pkgs.runCommand "${name}-env.sh" {
+  requiredSystemFeatures = [ "recursive-nix" ];
+  nativeBuildInputs = [ pkgs.nix ];
+} ''
+  nix --offline --extra-experimental-features "nix-command flakes" \
+    print-dev-env '${builtins.unsafeDiscardOutputDependency drv.drvPath}^*' >> $out
+''
+```
+
+**Problem:** `recursive-nix` is NOT supported on remote builders. Hydra's remote builder infrastructure cannot run nix commands inside derivations.
+
+Error: `"recursive-nix is not supported yet by external derivation builders"`
+
+#### 2. IFD Approach (commit ff75b34, HEAD~1)
+```nix
+let
+  rawEnvContent = builtins.readFile drv;  # IFD: triggers build at eval time
+  envContent = builtins.replaceStrings [...] rawEnvContent;
+in writeTextFile { text = envContent; }
+```
+
+**Problem:** This is fundamentally flawed because:
+1. `builtins.readFile drv` reads mkShell's output, which is NOT environment exports
+2. IFD forces cross-platform builds during evaluation (x86_64-linux evaluator would need to build aarch64-darwin shells)
+
+#### 3. Current Approach: Build-time read (commit 5c8b6c8, HEAD)
+```nix
+pkgs.runCommand "devx" {} ''
+  shellContent=$(cat ${drv})  # Read mkShell output at build time
+  filteredContent=$(echo "$shellContent" | grep -v 'declare -x out=')
+  # ... construct wrapper script
+''
+```
+
+**Problem:** This is ALSO fundamentally flawed for the same reason - `cat ${drv}` reads mkShell's placeholder output, NOT environment variables.
+
+### Hydra CI Status
+
+- **Jobset:** input-output-hk-devx:pullrequest-220
+- **URL:** https://ci.zw3rk.com/jobset/input-output-hk-devx/pullrequest-220
+
+### Latest Update (2026-02-14)
+
+**Eval 389** (commit c62060b): 568/643 succeeded, 59 dep-failed, 16 building.
+Root causes: postgresql-musl GC race (both arches), GCC-14.3.0 (infra), hadrian (stale cache).
+All code-level build failures (nghttp3, libev, tzdata, openssl, postgresql LTO) are FIXED.
+
+**Latest commit** (pending push) — disables pythonSupport/tclSupport for postgresql musl,
+reducing outputs from 7 (out,dev,doc,lib,man,plpython3,pltcl) to 5 (out,dev,doc,lib,man).
+
+**Summary of all fixes applied (in overlay order):**
+1. **happy disallowGhcReference** (build-fixes): Disable GHC reference check for
+   happy-2.1.7 which transitively references GHC through happy-lib.
+2. **tzdata mingw32** (mingw overlay): Extract to subdirectory — stdenv's
+   unconditional `chmod +x $sourceRoot` fails when sourceRoot="." on Determinate
+   Nix macOS sandbox (builder.json is read-only in build dir).
+3. **libev mingw32** (mingw overlay): `LDFLAGS=-no-undefined` + `LIBS=-lws2_32` —
+   libtool strips library flags from LDFLAGS, so -lws2_32 must go in LIBS.
+4. **nghttp3 mingw32** (mingw overlay): `ENABLE_LIB_ONLY=ON` — nghttp3 has no
+   separate ENABLE_EXAMPLES option; ENABLE_EXAMPLES is derived from ENABLE_LIB_ONLY.
+5. **OpenSSL musl test** (musl overlay): Skip flaky `82-test_ocsp_cert_chain.t`
+6. **PostgreSQL musl** (musl overlay): Multiple issues fixed:
+   - `jitSupport = false`: removes --with-llvm and LLVM build inputs
+   - `perlSupport = false`: musl perl lacks shared libperl
+   - `pythonSupport = false`: removes plpython3 output (unnecessary for musl cross)
+   - `tclSupport = false`: removes pltcl output (unnecessary for musl cross)
+   - `NIX_CFLAGS_COMPILE = "-fno-lto"`: GCC LTO + GNU ld is broken for postgresql
+     (.ltrans link failures). Can't use hardeningDisable=["lto"] because "lto" is
+     not a recognized hardening flag on musl cross stdenv.
+   - `llvmPackages_20` override to prevent LLVM stdenv switch (avoids LLVM dep)
+   - `outputChecks = {}`: removes unconditional LLVM disallowedRequisites refs
+   - `doCheck = false`: dict_snowball.so relocation fails in musl static
+   - `separateDebugInfo = false`: eliminates massive -debug output that causes
+     nix-copy transfer to race against builder min-free auto-GC
+
+**Remaining infra issues:**
+- **GCC 14.3.0** (34 Windows builds): `/usr/include` missing — darwin builders
+  claim x86_64-linux support but GCC's fixincludes expects real Linux host.
+  Fix: infra change to builder system type config, not a devx overlay.
+- **hadrian**: stale failure cache despite outputs existing in store.
+  Queue runner keeps re-caching after restart.
+
+**Key discovery:** nixpkgs-2511 `pkgsCross.musl64` doesn't set `isStatic = true`.
+This causes ALL optional features guarded by `!isStatic` to default to true,
+including jitSupport, perlSupport, pythonSupport, tclSupport, and the
+LTO stdenv switch in postgresql.
+
+The -env job generation using `devShellTools` is working correctly.
+
+---
+
+## CI/CD Pipeline Architecture
+
+### Flow
+
+1. **Hydra Evaluation:** Evaluates flake.nix on x86_64-linux for all platforms
+2. **Hydra Build:** Builds all devshells and `-env` jobs (using remote builders)
+3. **GitHub Actions:**
+   - `wait-for-hydra-eval`: Waits for Hydra `required` job
+   - `discover`: Queries GitHub API for all `*-env` check runs
+   - `process`: For each `-env` job:
+     - Downloads the derivation from binary cache (`nix-store -r`)
+     - Exports full closure (`nix-store --export`)
+     - Compresses with zstd
+     - Uploads to `ghcr.io/input-output-hk/devx:${DEV_SHELL}`
+
+### Key Files
+
+- `flake.nix`: Main configuration, defines devShells and hydraJobs
+- `.github/workflows/main.yml`: CI workflow for GHCR uploads
+- `extra/ghcr-upload.sh`: Script that exports and uploads closures
+- `dynamic.nix`, `static.nix`, `cross-js.nix`, `cross-windows.nix`: Shell definitions
+- `Dockerfile`: For building Docker images with pre-imported environments
+
+---
+
+## Research: Alternative Solutions
+
+### 1. nix-mk-shell-bin (Most Promising)
+
+**Repository:** https://github.com/rrbutani/nix-mk-shell-bin
+
+This project replicates the transformation that `nix develop` does on derivations, but at build time. It:
+- Doesn't require recursive-nix
+- Produces a self-contained script with all environment setup
+- Uses code extracted from Nix's develop functionality
+
+**Viability:** HIGH - This is exactly what devx needs. Would need to integrate this as a dependency or vendor the relevant code.
+
+### 2. Hydra Build Products
+
+Hydra build products (via `$out/nix-support/hydra-build-products`) can expose artifacts for download. However:
+- Products must be within the derivation's NAR (cannot point to external store paths)
+- Doesn't solve the environment extraction problem
+- More useful for exposing the closure tarball after it's built
+
+**Viability:** LOW for solving the core problem, but could be useful for alternative distribution.
+
+### 3. Pre-materialization
+
+Commit the generated environment scripts to the repository, similar to how haskell.nix materializes plan files.
+
+**Viability:** LOW - Would require regenerating on every flake update, complex to maintain for 200+ shell variants.
+
+### 4. Two-phase Build
+
+Split into two derivations:
+1. First derivation: Build the shell and generate environment script (would still need nix print-dev-env)
+2. Second derivation: Package the environment script
+
+**Viability:** LOW - Still requires running nix print-dev-env somewhere.
+
+### 5. Direct Shell Wrapper (Alternative Architecture)
+
+Instead of extracting environment variables, create a wrapper that directly invokes `nix develop`:
+
+```bash
+#!/usr/bin/env bash
+exec nix develop "github:input-output-hk/devx#$SHELL_NAME" --command "$@"
+```
+
+**Viability:** MEDIUM - Changes the usage model significantly. Would require nix on the target system and network access.
+
+### 6. Use stdenv.mkDerivation Instead of mkShell
+
+Create a derivation that explicitly outputs its environment in the format we need:
+
+```nix
+stdenv.mkDerivation {
+  name = "devx-env";
+  buildInputs = [ ... ];
+  buildPhase = ''
+    # Capture current environment as declare -x statements
+    export > $out
+  '';
+}
+```
+
+**Viability:** MEDIUM - Would need significant refactoring of shell definitions.
+
+---
+
+## Key Files for Implementation
+
+- `flake.nix:203-244` - The `-env` job generation code that needs fixing
+- `dynamic.nix` - Main devshell template (others are similar)
+- `extra/ghcr-upload.sh` - CI script that consumes the `-env` output
+
+---
+
+## Useful Commands
+
+```bash
+# Build a specific devshell
+nix build .#devShells.x86_64-linux.ghc96
+
+# Build a specific -env job
+nix build .#hydraJobs.x86_64-linux.ghc96-env
+
+# Check what mkShell actually outputs
+nix build .#devShells.x86_64-linux.ghc96 && cat result
+
+# Test nix print-dev-env (requires recursive-nix)
+nix print-dev-env .#devShells.x86_64-linux.ghc96
+
+# Check Hydra evaluation
+# Visit: https://ci.zw3rk.com/jobset/input-output-hk-devx/pullrequest-220
+```
+
+---
+
+## Notes
+
+- The `ccl` alias mentioned is unrelated to this project - it's a personal configuration management tool (`claude-config-link`)
+- Always use `nix develop -c make ...` for build commands per user preferences
+- Create meaningful commits with clear descriptions
+- License: Apache 2.0, Copyright: Moritz Angermann <[email protected]>, zw3rk pte. ltd.

extra/ghcr-upload.sh

@@ -9,5 +9,7 @@ set -euox pipefail
 nix-store -r "$SHELL_NIX_PATH" || nix-store -r "$SHELL_NIX_PATH" || nix-store -r "$SHELL_NIX_PATH"
 #nix build ".#hydraJobs.${DEV_SHELL}" --show-trace --accept-flake-config
 nix-store --export $(nix-store -qR "$SHELL_NIX_PATH") | tee store-paths.txt | zstd -z8T8 >${DEV_SHELL}
-if [[ ! $(tail -n 1 store-paths.txt) =~ "devx" ]]; then exit 1; fi
+# Verify the environment script is included in the closure
+# The script filename ends with -env.sh (e.g., ghc96-env.sh, ghc98-static-iog-env.sh)
+if [[ ! $(tail -n 1 store-paths.txt) =~ "-env.sh" ]]; then exit 1; fi
 oras push ghcr.io/input-output-hk/devx:${DEV_SHELL} ${DEV_SHELL}