Merge pull request #207 from MetaCell/feature/conversion_scripts

Conversion scripts from NRRD to precomputed

Author: ddelpiano

applications/virtual-fly-brain/scripts/README.md

@@ -0,0 +1,96 @@
+# NRRD to Neuroglancer Precomputed Workflow
+
+## Overview
+This guide shows how to convert NRRD segmentation files to Neuroglancer precomputed format with meshes.
+
+## Fixed Issues
+
+### Issue #1: Meshes not showing in Neuroglancer
+**Root cause**: Mesh vertices were in physical coordinates (microns) instead of voxel coordinates.
+**Fix**: Removed `spacing` parameter from `marching_cubes()` call. Neuroglancer applies resolution scaling separately.
+
+### Issue #2: Segmentation fragmentation
+**Clarification**: The NRRD files contain multiple anatomical structures as separate segments (this is correct).
+If you see unexpected fragmentation, use `merge_segments.py` to consolidate connected components.
+
+## Workflow
+
+### Step 1: Inspect your data (optional but recommended)
+```bash
+cd /home/ddelpiano/git/neuroglass/dataScripts/nrrd_to_precomputed
+python inspect_nrrd.py files/VFB_00101567.nrrd
+```
+
+### Step 2: (Optional) Merge fragmented segments
+Only use this if you want to merge disconnected pieces into single segments:
+```bash
+python merge_segments.py \
+  files/VFB_00101567.nrrd \
+  files/VFB_00101567_merged.nrrd \
+  --min-size 100 \
+  --verbose
+```
+
+### Step 3: Convert NRRD to precomputed format
+```bash
+python converter.py \
+  --input-dir files \
+  --output-path output \
+  --verbose
+```
+
+This will:
+- Find all `.nrrd` files in `files/` directory
+- Convert each to precomputed format in `output/` directory
+- Create a mapping file: `output/sources_to_dataset.json`
+
+### Step 4: Generate meshes
+```bash
+python meshes_generator.py \
+  --input-path output \
+  --dust-threshold 100 \
+  --verbose
+```
+
+This will:
+- Find all precomputed datasets in `output/` directory
+- Generate meshes for each segment (skipping segments < 100 voxels)
+- Create segment properties (labels and metadata)
+- Generate `output/neuroglancer_state.json` for easy loading
+
+### Step 5: Serve the data
+```bash
+cd output
+npx http-server . -p 8080 --cors
+```
+
+### Step 6: View in Neuroglancer
+1. Go to https://neuroglancer-demo.appspot.com/
+2. Click the `{}` (JSON) button in the top-right
+3. Copy the contents of `output/neuroglancer_state.json`
+4. Paste into the JSON editor
+5. Click outside the editor to load
+
+## Troubleshooting
+
+### Meshes still not showing?
+1. **Check the browser console** for errors
+2. **Verify mesh files exist**:
+   ```bash
+   ls -lh output/*/mesh/
+   ```
+3. **Check segment IDs match**:
+   ```bash
+   python inspect_nrrd.py files/VFB_00101567.nrrd | grep "Segment IDs"
+   ```
+4. **Try loading a single segment manually** in Neuroglancer UI
+
+### Segmentation looks wrong?
+- Use `inspect_nrrd.py` to see segment IDs and sizes
+- The NRRD files contain pre-labeled anatomical structures
+- If you see unexpected fragmentation, verify with the original data source
+
+### Performance issues?
+- Reduce `--dust-threshold` to skip more small segments
+- Generate meshes for fewer segments initially
+- Check your mesh file sizes

applications/virtual-fly-brain/scripts/check_mesh_coords.py

@@ -0,0 +1,84 @@
+#!/usr/bin/env python3
+"""Debug mesh and volume coordinate spaces"""
+import json
+import sys
+import os
+
+if len(sys.argv) < 2:
+    print("Usage: python check_mesh_coords.py <dataset_path>")
+    sys.exit(1)
+
+dataset_path = sys.argv[1]
+
+# Read main info
+with open(os.path.join(dataset_path, "info"), "r") as f:
+    info = json.load(f)
+
+print("=== Dataset Info ===")
+print(f"Type: {info['type']}")
+print(f"Data type: {info['data_type']}")
+print(f"Volume size (XYZ): {info['scales'][0]['size']}")
+print(f"Resolution (XYZ): {info['scales'][0]['resolution']}")
+print(f"Chunk sizes: {info['scales'][0]['chunk_sizes']}")
+
+# Calculate physical bounds
+size = info["scales"][0]["size"]
+resolution = info["scales"][0]["resolution"]
+physical_bounds = [s * r for s, r in zip(size, resolution)]
+print(f"Physical bounds: {physical_bounds}")
+
+# Read mesh info if exists
+mesh_info_path = os.path.join(dataset_path, "mesh", "info")
+if os.path.exists(mesh_info_path):
+    print("\n=== Mesh Info ===")
+    with open(mesh_info_path, "r") as f:
+        mesh_info = json.load(f)
+    print(json.dumps(mesh_info, indent=2))
+
+    # Check a sample mesh file
+    mesh_dir = os.path.join(dataset_path, "mesh")
+    mesh_files = [
+        f
+        for f in os.listdir(mesh_dir)
+        if f.endswith(".gz") or (f.isdigit() and not f.endswith(".gz"))
+    ]
+
+    if mesh_files:
+        sample_mesh = mesh_files[0]
+        print(f"\nSample mesh file: {sample_mesh}")
+
+        # Try to read and check coordinates
+        try:
+            from cloudvolume import CloudVolume
+            from cloudvolume.mesh import Mesh
+
+            vol = CloudVolume(f"file://{dataset_path}", mip=0)
+
+            # Get segment ID from filename
+            seg_id = int(sample_mesh.replace(".gz", "").replace(":", "_").split("_")[0])
+
+            try:
+                mesh = vol.mesh.get(seg_id)
+                if mesh and len(mesh.vertices) > 0:
+                    print(f"\nMesh {seg_id} vertices:")
+                    print(f"  Min: {mesh.vertices.min(axis=0)}")
+                    print(f"  Max: {mesh.vertices.max(axis=0)}")
+                    print(f"  Shape: {mesh.vertices.shape}")
+
+                    # Compare with volume bounds
+                    max_voxel = mesh.vertices.max(axis=0)
+                    print(f"\nComparison:")
+                    print(f"  Volume size (voxels): {size}")
+                    print(f"  Mesh max (should be <= volume size): {max_voxel}")
+
+                    if any(max_voxel > size):
+                        print(f"  ⚠ WARNING: Mesh vertices exceed volume bounds!")
+                    else:
+                        print(f"  ✓ Mesh vertices within volume bounds")
+            except Exception as e:
+                print(f"  Error loading mesh: {e}")
+
+        except Exception as e:
+            print(f"Error: {e}")
+else:
+    print("\n⚠ No mesh info found")