v0.5.1: Anchor & Refill Strategy, Precision Quantization & Operational Robustness

[0.5.1] - 2026-01-01 - Anchor & Refill Strategy, Precision Quantization & Operational Robustness

Added

• Anchor & Refill Strategy: Major architectural upgrade for partial order handling. Instead of moving partials, the bot now anchors them in place.
  • Case A: Merged Refill (Dust): Merges dust (< 5%) into the next geometric allocation and delays the opposite-side rotation until the dust portion is filled.
  • Case B: Full Anchor (Substantial): Upgrades partials (>= 5%) to 100% ideal size and places the leftover capital as a residual order at the spread.
• On-Chain Alignment for Refills: The bot now broadcasts limit_order_update for dust refills to ensure on-chain sizes perfectly match the merged internal allocation.
• Cumulative Fill Tracking: Added filledSinceRefill property to accurately trigger delayed rotations across multiple partial fills.
• Precision Quantization: Implemented size quantization to exact blockchain precision before order placement, eliminating float rounding errors.
• Pending-Aware Health Checks: Updated countOrdersByType and checkGridHealth to recognize intentional gaps created by delayed rotations, preventing false-positive corrections.
• Double-Aware Divergence Engine: Updated calculateGridSideDivergenceMetric to account for merged dust sizes, preventing unnecessary grid resets for anchored orders.
• Periodic Order Synchronization: Added readOpenOrders to the 4-hour periodic fetch to automatically reconcile the internal grid with the blockchain source of truth.
• Modernized Test Suite: Added comprehensive unit, integration, and E2E tests for the Anchor & Refill strategy and precision fixes.

Changed

• Pipeline-Aware Monitoring: checkGridHealth now only executes when the order pipeline is clear (no pending fills or corrections), increasing operational stability.
• Memory-Chain Alignment: Quantized order sizes are synchronized back to the internal memory state to ensure 1:1 parity with blockchain integers.
• State Persistence: Added full serialization for new strategy fields (isDoubleOrder, mergedDustSize, pendingRotation, filledSinceRefill).

Fixed

• Sync Reversion Protection: Prevented the bot from prematurely reverting merged sizes back to old on-chain sizes during synchronization gaps.
• Off-by-One Eradication: Fixed a recurring issue where small float remainders would block grid flow or cause spurious partial-state transitions.
• Race Condition Handling: Improved observability and lock management in dexbot_class.js to ensure sequential consistency during high-volume fill events.

v0.5.0 - Stability Milestone

Added

• Persistent General Settings: Implemented a new architecture using profiles/general.settings.json for untracked user overrides.
• Global Settings Manager: Added a new sub-menu to dexbot bots to manage global parameters (Log lvl, Grid, Timing).
• Grid Health Monitoring: New system to monitor structural grid integrity and log violations (e.g., ACTIVE orders further from market than VIRTUAL slots).
• Dual-Side Dust Recovery: Automatically refills small partial orders (< 5%) to ideal geometric sizes using cacheFunds when detected on both sides.
• Enhanced Spread Correction: Implemented proactive spread correction that pools both VIRTUAL and SPREAD slots to identify the best candidates for narrowing the market spread.
• Sequential Fill Queue: Implemented thread-safe sequential processing of fill events using AsyncLock to prevent accounting race conditions.
• Safe PM2 Lifecycle Management: Added pm2.js stop and pm2.js delete commands that safely filter for dexbot-specific processes.
• Robust Fill Detection: Implemented history mode for fill processing to reliably match orders from blockchain events.

Changed

• Global Terminology Migration: Renamed all occurrences of marketPrice to startPrice across codebase, CLI, and documentation to better reflect its role as the grid center.
• Menu-Driven Bot Editor: Refactored modules/account_bots.js into a sectional, menu-driven interface for faster configuration.
• Simplified Update Process: Removed fragile git stashing from update.sh and update-dev.sh; user settings are now preserved via untracked JSON.
• CLI Command Renaming: Renamed dexbot stop to dexbot disable for better alignment with its actual function (marking bots inactive in config).
• Price Calculation Accuracy: Updated buildUpdateOrderOp to use current sell amounts when deriving prices, fixing precision issues in small price moves.
• Default Log Level: Changed default LOG_LEVEL from debug to info.
• Architectural Cleanup: Consolidated core logic into pure utility functions to eliminate duplication and improve maintainability.

Fixed

• Fund Double-Counting: Fixed a critical bug in processFilledOrders where proceeds were incorrectly added to available funds twice.
• Startup Double-Initialization: Resolved a race condition that could cause corrupted virtual order sizes during bot startup.
• Reset Reliability: Fixed node dexbot reset command to ensure a true hard reset from blockchain state, including hot-reloading of bots.json.
• Stuck VIRTUAL Orders: Added error handling for rotation synchronization to prevent orders from being stuck in a virtual state.
• Logging Visibility: Ensured all cancellation operations provide explicit success/fail messages in logs.
• Offline Detection Fixes: Resolved edge cases in offline partial fill detection to ensure capital efficiency on startup.
• Update Script Robustness: Refactored update scripts to use git reset --hard to forcefully clear environment conflicts (e.g., in constants.js).
• Module Path Corrections: Fixed incorrect relative paths in startup_reconcile.js and streamlined operational logging.

v0.4.6 - CacheFunds Double-Counting Fix & Fill Deduplication

Release v0.4.6 - Critical Bug Fixes: CacheFunds Double-Counting, Fill Deduplication & Race Conditions

🔴 23 Critical Bugs Fixed

1. CRITICAL: CacheFunds Double-Counting in Partial Fills

• Location: modules/order/manager.js lines 570-596, 1618-1625
• Problem: Proceeds being counted twice in cacheFunds balance
  • When partial fill occurred, proceeds added to chainFree, then available recalculated from updated chainFree (which already included proceeds)
  • Both proceeds + available added to cacheFunds → double-counting
• Impact: User reported 649.72 BTS discrepancy in fund accounting
• Bug Timeline: Introduced in v0.4.0, present through v0.4.5
• Solution: Calculate available BEFORE updating chainFree, use pre-update value in processFilledOrders()

2. CRITICAL: Fee Double-Deduction After Bot Restart

• Location: modules/account_orders.js lines 427-551, modules/dexbot_class.js lines 42-48, 77-251, 652-660
• Problem: Permanent fund loss on bot restart during fill processing
  • When bot restarts, same fills detected again from blockchain history
  • processFilledOrders() called twice with identical fills
  • BTS fees double-deducted from cacheFunds
• Impact: Every bot restart during active trading could lose funds
• Solution: Persistent fill ID deduplication with multi-layer protection
  • In-Memory Layer (5 seconds): Prevents immediate reprocessing
  • Persistent Layer (1 hour): Saves fill IDs to disk, loads on startup
  • Automatic Cleanup: Removes entries older than 1 hour
  • AsyncLock Protection: Prevents race conditions during writes
• Defensive Impact: Protects entire fill pipeline (committed funds, fund cycling, grid rebalancing, order status)

3. 20+ Race Conditions: TOCTOU & Concurrent Access

AsyncLock Implementation: 7 lock instances protecting critical sections across the codebase

A. File Persistence Races - Prevent stale in-memory data overwrites

• Lock: _persistenceLock (account_orders.js)
• Protected: storeMasterGrid, updateCacheFunds, updateBtsFeesOwed, ensureBotEntries, processedFills methods
• Pattern: Reload-before-write to prevent Process-A-reads → Process-B-writes → Process-A-overwrites scenarios

B. Account Subscription Management Races - Prevent duplicate subscriptions

• Lock: _subscriptionLock (chain_orders.js)
• Protected: _ensureAccountSubscriber, listenForFills, unsubscribe
• Result: Atomic subscription creation and callback management

C. Account Resolution Cache Races - Atomic name/ID resolution

• Lock: _resolutionLock (chain_orders.js)
• Protected: resolveAccountName, resolveAccountId
• Result: Atomic cache check-and-set operations

D. Preferred Account State Races - Thread-safe global state

• Lock: _preferredAccountLock (chain_orders.js)
• Protected: setPreferredAccount, getPreferredAccount
• Pattern: All access through thread-safe getters/setters

E. Fill Processing Races - Serialized fill event handling

• Lock: _fillProcessingLock (dexbot_class.js)
• Protected: Fill callback, triggered resync, order manager loop
• Result: Prevents concurrent modifications during fill processing

F. Divergence Correction Races - Grid update serialization

• Lock: _divergenceLock (dexbot_class.js)
• Protected: Post-rotation divergence, timer-based divergence
• Result: Grid updates serialized, prevents concurrent conflicts

G. Order Corrections List Races - Foundation for serialized price correction

• Lock: _correctionsLock (manager.js)
• Status: Declared and prepared for active use
• Arrays affected: ordersNeedingPriceCorrection at 11 locations

📊 Files Modified

New:

• modules/order/async_lock.js (84 lines): FIFO queue-based synchronization utility

Modified (specific locations):

• modules/account_orders.js: Persistence lock, reload-before-write, processedFills tracking
• modules/chain_orders.js: 3 lock instances, thread-safe account management
• modules/dexbot_class.js: Fill dedup, lock declarations, persistent fill loading
• modules/order/manager.js: Cachefunds fix, pre-update available calculation

🔍 Technical Details

• Reload-Before-Write Pattern: Always reload from disk immediately before writing to prevent stale data overwrites
• Async/Await Consistency: All persistence methods properly awaited, no fire-and-forget promises
• Lock Nesting Prevention: No nested lock acquisition, prevents deadlocks
• Fill Dedup Storage Format: Persistent storage in profiles/orders/{botKey}.json with fill key: ${orderId}:${blockNum}:${historyId}

✓ Testing & Safety

• All 20 integration tests passing ✅
• Test coverage: ensureBotEntries, storeMasterGrid, cacheFunds persistence, fee deduction, fill dedup
• No changes to fill processing logic or output (only adds deduplication layer)
• Low-risk: Simple addition of locks to existing code paths, no core algorithm changes
• Backward compatible: No breaking changes, fully transparent to users

📈 Performance

• Minimal Overhead: Efficient FIFO queue, millisecond lock durations, ~5ms disk read negligible vs network latency
• Benefits: Eliminates fund loss, prevents duplicate processing, ensures consistent state recovery
• Cleanup Strategy: Runs ~10% of batches, not every batch (reduces I/O overhead)

🔒 Migration

• Backward Compatible: No API or configuration changes
• No Schema Changes: File format unchanged, existing bot data continues to work
• Automatic Initialization: processedFills field auto-initialized if missing

📋 Summary

Total Fixes: 23 critical bugs

• 1 cacheFunds double-counting fix
• 1 fee double-deduction fix
• 20+ race condition fixes (7 categories)
• 1 defensive fill deduplication system

Implementation: 7 AsyncLock instances, ~300 LOC, 5 files modified + 1 new, 20/20 tests ✅

Commits:

• f7cb80c - fix: prevent cacheFunds double-counting in partial fill processing
• 969f98b - feat: add persistent fill ID deduplication to prevent fee double-deduction
• cf39cd7 - docs: expand v0.4.6 CHANGELOG with comprehensive race condition and bug fix details

Release v0.4.5: Partial Order Fixes

Partial Order Counting & Grid Navigation Fix

Critical Fixes

Partial Orders Not Counted in Grid Targets

• Partial filled orders were excluded from order target counting
• Caused bot to create unnecessary orders even when at target capacity
• Now counts both ACTIVE and PARTIAL orders toward target
• Prevents erroneous order creation and grid position mixing

Grid Navigation Limited by ID Namespace

• preparePartialOrderMove() used ID-based navigation (sell-N/buy-N)
• Could not move partial orders across sell-/buy- namespace boundaries
• Example: sell-173 couldn't move to buy-0 (adjacent by price)
• Now uses price-sorted navigation for fluid grid movement
• Partial orders can move anywhere without artificial boundaries

Added

• countOrdersByType() helper function in utils.js
  • Counts both ACTIVE and PARTIAL orders by type
  • Used consistently across order target comparisons

Changed

• Order target checks include partial orders
• Spread calculation includes partial orders
• Grid treated as fluid with no artificial boundaries

Technical Details

• All 18 test suites pass
• Fixed test_crossed_rotation.js expectations
• Price-sorted navigation enables unrestricted partial order movement

Commits

• d7a23cd: fix: include partial orders in grid position counting and navigation
• 77fbba3: docs: release version 0.4.5

Release v0.4.4: Code Consolidation & BTS Fee Deduction Fix

Major Changes

🐛 Critical Bug Fix

• BTS Fee Deduction on Wrong Side: Fixed critical bug in grid resize operations
  • XRP/BTS pairs: BTS fees no longer incorrectly deducted from XRP (SELL side) funds
  • Buy side (assetB): Only deduct if assetB === 'BTS'
  • Sell side (assetA): Only deduct if assetA === 'BTS'
  • Fixes 70% order size reduction issue during grid resize

📈 Fee Multiplier Update

• Increased from 4x to 5x (1x creation + 4x rotation buffer, was 3x)
• Provides better buffer for multiple rotation cycles

🔧 Code Consolidation & Refactoring

• Moved 22 grid utility functions from grid.js to utils.js
• Eliminated duplicate code and consolidated inline requires
• Reduced grid.js by 46% (-555 lines)
• Added 15 new reusable utility functions
• Added manager helper methods for fund/chainFree tracking
• Removed debug console.log statements

✅ Quality Assurance

• All 18 test suites pass
• Rotation and divergence logic unchanged
• Net +166 lines justified by new utilities and JSDoc documentation

Files Changed

• modules/chain_orders.js: -31 lines (debug logs removed)
• modules/dexbot_class.js: Function imports updated
• modules/order/grid.js: -555 lines (refactored)
• modules/order/manager.js: +116 lines (new helpers)
• modules/order/utils.js: +599 lines (consolidated utilities)
• tests/test_order_grid.js: Updated imports

See CHANGELOG.md for complete details

v0.4.3 - Grid Edge Reconciliation & State Sync Improvements

🔧 Fixed

Grid Edge State Synchronization

• Fixed manager state sync after reducing largest order
• Search by blockchain orderId to find matching grid order in manager.orders
• Ensures manager's local grid state matches blockchain after order reduction

Grid Edge Order Reconciliation

• Refactored cancel+create approach (more efficient than reduce+restore)
  • N+1 operations instead of N+2 blockchain calls
  • Phase 1: Cancel largest order to free funds
  • Phase 2: Update remaining orders to targets
  • Phase 3: Create new order for cancelled slot
  • Simplified logic with proper index alignment

Previous Fixes (from commit 1390bb8)

• Asymmetric Rebalance Orders Logic: Corrected order matching in rebalanceOrders function
• Order Pairing Sorting & Startup Reconciliation: Optimized order matching algorithm
• Grid Data Corruption Prevention: Added validation for order sizes and IDs
• BTS Fee Reservation During Resize: Fixed target order selection
• 4x Blockchain Fee Buffer Enforcement: Corrected fee buffer application

📝 Changed

• Removed unused bot_instance.js module for code cleanup
• Enhanced startup_reconcile documentation in README
• Optimized grid edge reconciliation strategy for fewer blockchain operations

See full changelog at CHANGELOG.md

Release 0.4.2

0.4.2 - Grid Recalculation Fixes & Documentation Updates (2025-12-24)

Fixed

Grid Recalculation in Post-Rotation Divergence Flow

• Added missing grid recalculation call in post-rotation divergence flow
• Problem: Orders were losing size information during divergence correction
• Symptoms: "Skipping virtual X - no size defined" warnings, "Cannot read properties of undefined" batch errors
• Solution: Added Grid.updateGridFromBlockchainSnapshot() call to post-rotation flow, matching startup and timer divergence paths
• Impact: Prevents order size loss during divergence correction cycles

PARTIAL Order State Preservation at Startup

• Fixed state inconsistency during synchronization
• Problem: PARTIAL orders (those with remaining amounts being filled) were unconditionally converted to ACTIVE state at startup
• Symptoms: False divergence spikes (700%+), state mismatches between persistedGrid and calculatedGrid, unnecessary grid recalculations
• Solution: Preserve PARTIAL state across bot restarts if already set; only convert VIRTUAL orders to ACTIVE when matched on-chain
• Impact: Eliminates false divergence detection and maintains consistent order state across restarts

Redundant Grid Recalculation Removal

• Eliminated duplicate processing in divergence correction
• Problem: Grid was being recalculated twice when divergence was detected
• Symptoms: Double order size updates, unnecessary blockchain fetches, performance inefficiency
• Solution: Removed redundant recalculation from applyGridDivergenceCorrections()
• Impact: Single grid recalculation per divergence event, improved performance

BTS Fee Formula Documentation

• Updated outdated comments and logged output to accurately reflect the complete fee calculation formula
• Fixed modules/order/grid.js: Changed comment from "2x multiplier" to "4x multiplier"
• Updated formula in 5 files to show: available = max(0, chainFree - virtuel - cacheFunds - applicableBtsFeesOwed - btsFeesReservation)
• Fixed modules/order/logger.js: Console output now displays full formula
• Updated modules/order/manager.js: Changed variable references from "4xReservation" to "btsFeesReservation"
• Fixed modules/account_bots.js: Default targetSpreadPercent comment now correctly states 4x (not 3x)

Files Changed

• modules/order/grid.js
• modules/order/index.js
• modules/order/logger.js
• modules/order/manager.js
• modules/order/runner.js
• modules/order/utils.js
• modules/account_bots.js
• package.json
• CHANGELOG.md

Commits Included

• 21c3fb0 - docs: expand v0.4.2 changelog with comprehensive bug fix details
• 36add08 - release: version 0.4.2
• 3b65efb - fix: update outdated comments and formulas for BTS fee calculations
• 873a4f8 - fix: add missing grid recalculation in post-rotation divergence flow
• 07a2d11 - merge: bring main's BTS fee accounting fixes into dev
• 7e590f4 - fix: remove redundant grid recalculation in applyGridDivergenceCorrections
• de9763d - fix: preserve PARTIAL state during startup synchronization

v0.4.1 - Order Consolidation, Grid Edge Handling & Partial Order Fixes

Release v0.4.1

Key Features

• Code Consolidation: Eliminated ~1,000 lines of duplicate code

  • Extracted shared DEXBot class to modules/dexbot_class.js
  • Refactored bot.js and dexbot.js to thin wrappers

• Conditional Rotation: Smart order placement at grid boundaries

  • Creates new orders instead of rotating when below target count
  • Handles grid edge cases gracefully

• Partial Order Fixes: Enforced state machine invariants

  • PARTIAL orders guaranteed to have size > 0
  • Proper state transitions: ACTIVE → PARTIAL → SPREAD

• Repository Analytics: Interactive visualization of git history

  • Charts showing added/deleted lines over time
  • Daily and cumulative statistics

Files Changed

• 10 files changed, 2,101 insertions(+), 1,875 deletions(-)
• New: modules/dexbot_class.js, scripts/analyze-repo-stats.js, test files

See CHANGELOG.md for complete details.

v0.4.0: Fund Management Consolidation & Automatic Fund Cycling

Major Features

Automatic Fund Cycling

Available funds now automatically included in cacheFunds before rotation, making grids responsive to new deposits.

• Newly deposited funds immediately available for grid sizing
• Grid resizes when deposits arrive, not just after fills
• More responsive to market changes and new capital inflows

Unified Fund Management

Complete consolidation of pendingProceeds into cacheFunds for simplified fund tracking.

• Single cacheFunds field for all unallocated funds
• Cleaner codebase (272 line reduction)
• Backward compatible: legacy pendingProceeds automatically migrated

Fund Formula Updated:

OLD: available = max(0, chainFree - virtuel - cacheFunds - btsFeesOwed) + pendingProceeds
NEW: available = max(0, chainFree - virtuel - cacheFunds - btsFeesOwed)

Bot Metadata Initialization

Fixed new order files being created with null metadata (name, assetA, assetB).

• ensureBotEntries() called before any Grid initialization
• Applied to both bot.js and dexbot.js DEXBot classes

Bug Fixes

• Partial Order Precision: Fixed floating-point noise in partial fill detection using integer-based subtraction
• Logger Undefined Variables: Fixed references to removed pendingProceeds variables
• Grid Regeneration: Now includes available funds in threshold calculation

Breaking Changes

⚠️ pendingProceeds field removed from profiles/orders/.json

Migration Required:

# Backup first
cp -r profiles/orders profiles/orders.backup

# Run migration
node scripts/migrate_pending_proceeds.js

# Restart bots

Backward compatible: legacy pendingProceeds automatically merged into cacheFunds on load.

Configuration Changes

• GRID_REGENERATION_PERCENTAGE: 1% → 3% (more stable, reduces unnecessary regeneration)
• DIVERGENCE_THRESHOLD_PERCENTAGE: 1% (unchanged, remains strict for synchronization)

Testing

✅ Fund cycling trigger detection
✅ Crossed rotation with fund cycling
✅ Fee refinement with new fund system
✅ Partial order precision improvements
✅ Bot metadata initialization
✅ Migration script (5 test scenarios)

Resources

• 📖 CHANGELOG.md - Full change history
• 🔧 Migration Guide - Step-by-step upgrade instructions
• 📚 README.md - Full documentation

---

Always test with dryRun: true before enabling live trading.

Release 0.3.0

Promille to Percentage Threshold Conversion & Documentation Improvements

Major Features & Changes:

• ✅ Threshold Format: Convert divergence threshold from promille to percentage (0-100% scale)
• ✅ RMS Documentation: Add complete Root Mean Square definition and explanation
• ✅ Input Validation: Comprehensive framework for bot configuration validation
• ✅ Grid Divergence Detection: Automatic regeneration when divergence exceeds threshold
• ✅ Cache Funds Monitoring: Track and manage accumulated proceeds from fills
• ✅ Threshold Documentation: Reference tables and mathematical formulas

Configuration Changes:

• Grid Divergence Threshold: Now percentage-based (1% by default, configurable)
• Cache Funds Threshold: 2% by default (when cache reaches 2% of grid capital)
• Variable Naming: DIVERGENCE_THRESHOLD_PERCENTAGE (was DIVERGENCE_THRESHOLD_Promille)

Documentation Highlights:

• Percentage threshold reference table (0.01% to 10%)
• Explanation of 1/(1+n) distribution scaling for uneven errors
• RMS vs Simple Average comparison with examples
• Clear formula definitions and threshold trigger logic
• Configuration examples with current defaults

Technical Details:

• Grid.js: Updated threshold comparisons for percentage values
• Constants.js: Reorganized for better readability
• README: Comprehensive threshold documentation with tables
• Tests: Expanded test suite for grid comparison
• All metric calculations now use percentage scale (0.01 = 1% divergence)

Bug Fixes & Improvements:

• Fixed threshold formula clarity (Threshold vs Metric distinction)
• Improved RMS definition in brackets for clarity
• Updated cache funds example values to match defaults
• Removed misleading percentage conversion comments
• Consistent terminology throughout documentation