get 'er done

Author: marke000

README.md

@@ -6,57 +6,72 @@ A comprehensive Clojure library for numerical optimization, root-finding, interp
 
 ### Root-Finding
 
-- **`root-solvers`** — Univariate root-finding with multiple algorithms: Brent-Dekker, modified Newton-Raphson, Muller, plus Hipparchus methods (bisection, Brent, Illinois, Pegasus, Ridders, secant, Regula-Falsi). Includes quadratic equation solver.
+- **`roots.continuous`** — Univariate root-finding with multiple algorithms: Brent-Dekker, modified Newton-Raphson, Muller, plus Hipparchus methods (bisection, Brent, Illinois, Pegasus, Ridders, secant, Regula-Falsi). Includes quadratic equation solver.
 
-- **`integer-root-solvers`** — Bisection algorithm for strictly increasing discrete functions. Returns the minimum integer with function value ≥ 0.
+- **`roots.integer`** — Bisection algorithm for strictly increasing discrete functions. Returns the minimum integer with function value ≥ 0.
 
-- **`plateau-root-solvers`** — Root-finding for monotonic functions that return plateau values. Supports univariate and multivariate cases.
+- **`roots.plateau`** — Root-finding for monotonic functions that return plateau values. Supports univariate and multivariate cases.
+
+- **`roots.polynomial`** — Polynomial root-finding using Laguerre's method via Hipparchus.
 
 ### Optimization
 
-- **`optimize-univariate`** — Brent optimizer for 1D minimization/maximization over bounded intervals.
+- **`optimize.continuous-univariate`** — Brent optimizer for 1D minimization/maximization over bounded intervals.
+
+- **`optimize.integer-univariate`** — Integer optimizer using exponential search or ternary search. Handles unimodal functions over integer domains.
+
+- **`optimize.linear`** — Two-phase Simplex Method. Minimizes/maximizes linear objectives subject to linear constraints (equality, ≤, ≥).
+
+- **`optimize.mixed-integer`** — Mixed Integer Programming (MIP) using OjAlgo. Extends linear programming to support integer and binary variable constraints.
+
+- **`optimize.quadratic`** — Minimizes (1/2)x^T P x + q^T x subject to equality and inequality constraints using OjAlgo.
 
-- **`integer-optimize`** — Custom integer maximizer that exponentially focuses search around a guess. Handles functions with at most one sign change in derivative.
+- **`optimize.nlp-constrained`** — Constrained nonlinear optimization using COBYLA. Minimizes objectives subject to nonlinear inequality constraints (≥ 0).
 
-- **`linear-programming`** — Two-phase Simplex Method. Minimizes/maximizes linear objectives subject to linear constraints (equality, ≤, ≥).
+- **`optimize.nlp-unbounded`** — Unbounded nonlinear optimization with multiple pure-Clojure algorithms:
+  - *Powell*: derivative-free direction-set method
+  - *Nelder-Mead*: simplex-based derivative-free optimization
+  - *L-BFGS*: limited-memory quasi-Newton with optional gradient
+  - *Gauss-Newton*: nonlinear least-squares for residual minimization
+  - Auto-selecting orchestrator tries multiple algorithms
 
-- **`quadratic-programming`** — Minimizes (1/2)x^T P x + q^T x subject to equality and inequality constraints using OjAlgo.
+- **`optimize.nlp-bounded`** — Bounded nonlinear optimization with box constraints:
+  - *COBYLA*: linear-approximation constrained optimization
+  - *BOBYQA*: quadratic-model bound-constrained optimization
+  - *CMA-ES*: evolutionary strategy for non-smooth/non-convex objectives
 
-- **`nonlinear-programming`** — Multiple solvers for nonlinear optimization:
-  - *Constrained*: Cobyla for inequality constraints
-  - *Unbounded*: Powell, Nelder-Mead, Multi-directional Simplex, Conjugate Gradient (Polak-Ribière, Fletcher-Reeves)
-  - *Bounded*: BOBYQA, Cobyla, CMA-ES (evolutionary, handles non-smooth/non-convex objectives)
+### Systems (Equation Solving)
 
-### Constraint Satisfaction
+Solvers for systems of equations without an objective function to optimize:
 
-- **`nonlinear-constraints-without-objective`** — Finds variable values that satisfy nonlinear constraints:
+- **`systems.linear`** — Iterative solvers for symmetric linear systems (A × y = b): SYMMLQ and Conjugate Gradient methods. For small/dense systems, use direct least squares from `provisdom.math.linear-algebra` instead.
+
+- **`systems.nonlinear`** — Finds variable values that satisfy nonlinear constraints:
   - Nonlinear Least Squares (Levenberg-Marquardt, Gauss-Newton)
   - Nonlinear Ordered Constraints (prioritized constraint satisfaction)
 
 ### Interpolation & Curve Fitting
 
 Interpolation is organized by dimensionality:
 
-- **`interpolation-1d`** — 1D interpolation methods:
+- **`interpolation.univariate`** — 1D interpolation methods:
   - *Methods*: cubic, cubic-hermite, cubic-closed, Akima, PCHIP (monotone-preserving), barycentric-rational, linear, polynomial, step, LOESS (smoothing)
-  - *Specialized*: quadratic with slope, cubic-clamped with 2nd derivatives, B-spline with knots
+  - *Specialized*: quadratic with slope, cubic-clamped with 2nd derivatives, B-spline with knots, smoothing spline (with EDF control)
   - *Extras*: extrapolation modes (error/flat/linear), batch evaluation, spline integration, auto-selection
   - *Slope interpolation*: compute derivatives at query points
 
-- **`interpolation-2d`** — 2D grid-based interpolation (x-vals × y-vals → f-matrix):
+- **`interpolation.grid-2d`** — 2D grid-based interpolation (x-vals × y-vals → f-matrix):
   - *Methods*: polynomial, bicubic, bicubic-Hermite, bilinear
 
-- **`interpolation-3d`** — 3D grid-based interpolation (x-vals × y-vals × z-vals → f-tensor):
+- **`interpolation.grid-3d`** — 3D grid-based interpolation (x-vals × y-vals × z-vals → f-tensor):
   - *Methods*: tricubic
 
-- **`interpolation-nd`** — N-D scattered (non-grid) data interpolation:
+- **`interpolation.scatter-nd`** — N-D scattered (non-grid) data interpolation:
   - *Methods*: microsphere projection, RBF (radial basis functions), natural neighbor (2D only)
 
-- **`curve-fitting`** — Least-squares fitting to find best-fit parameters:
-  - Gaussian: `a * exp(-((x-b)/c)²)`
-  - Harmonic: `a * cos(ωx + φ)`
-  - Polynomial of specified degree
-  - Arbitrary parametric functions (with user-provided gradient)
+- **`fitting.curve`** — Least-squares fitting to find best-fit parameters:
+  - *Nonlinear* (Levenberg-Marquardt): Gaussian, Harmonic, Polynomial, arbitrary parametric
+  - *Linear basis* (closed-form): custom basis functions for univariate and multivariate data
 
 #### Interpolation vs Curve Fitting
 
@@ -71,41 +86,83 @@ Interpolation is organized by dimensionality:
 
 **Note**: LOESS is in the interpolation namespace (same API) but doesn't pass through points—it performs local regression for smoothing noisy data.
 
+#### Curve Fitting vs Regression
+
+| Aspect | Curve Fitting | Regression |
+|--------|---------------|------------|
+| **Goal** | Find parameters of a known functional form | Model statistical relationships between X and y |
+| **Functional form** | Arbitrary: `a·e^(-((x-b)/c)²)`, `a·cos(ωx+φ)` | Linear in parameters: `y = Xβ` (possibly transformed) |
+| **Optimization** | Nonlinear least squares (Levenberg-Marquardt) | Closed-form (OLS/Ridge) or IRLS (GLMs) |
+| **Assumptions** | Just minimize residuals | Error distribution, link function |
+| **Output** | Parameters of the specific function | Coefficients + diagnostics (R², MSE, precision) |
+
+**Curve fitting** answers: "Given this formula, what parameters fit best?"
+
+**Regression** answers: "What's the statistical relationship between predictors X and response y?"
+
+Logistic and beta regression aren't just curve fitting—they model the *distribution* of y given X (Bernoulli, Beta) with appropriate link functions, using Iteratively Reweighted Least Squares (IRLS).
+
 ### Regression
 
-- **`logistic-regression`** — Iteratively Reweighted Least Squares (IRLS) with optional ridge regularization.
+All regression methods are in the `provisdom.solvers.regression` namespace hierarchy:
+
+- **`regression.ordinary`** — Ordinary least squares with optional regularization:
+  - *OLS* — Standard least squares via QR decomposition
+  - *Ridge (L2)* — Closed-form solution with λI penalty
+  - *LASSO (L1)* — Coordinate descent for sparse solutions
+  - *Elastic Net* — Combined L1+L2 via coordinate descent
+
+- **`regression.logistic`** — Binary logistic regression using IRLS with optional ridge regularization.
+
+- **`regression.multinomial-logistic`** — Multi-class logistic regression (one-vs-all approach).
+
+- **`regression.beta`** — Beta regression for responses bounded in (0, 1) using IRLS.
+
+- **`regression.kernel-grnn`** — Generalized Regression Neural Network (Nadaraya-Watson kernel regression) with automatic spread optimization.
+
+- **`regression.stepwise`** — Stepwise variable selection:
+  - *Forward* — Start empty, iteratively add best predictor
+  - *Backward* — Start full, iteratively remove worst predictor
+  - *Both* — Combine forward and backward steps
+  - Supports AIC/BIC scoring with ordinary, logistic, or beta regression
+
+#### IRLS (Iteratively Reweighted Least Squares)
 
-- **`multinomial-logistic-regression`** — IRLS for multi-class classification.
+Logistic and beta regression use IRLS, which fits generalized linear models by iteratively solving weighted least squares problems:
 
-- **`generalized-regression-neural-network`** — GRNN for non-parametric regression with automatic spread parameter optimization.
+1. Compute working weights W based on current predictions
+2. Compute working response z (adjusted dependent variable)
+3. Solve weighted OLS: `β = (X'WX)⁻¹ X'Wz`
+4. Repeat until convergence
 
-### Linear Systems
+For logistic regression: `W = diag(μ(1-μ))`, `z = η + (y-μ)/(μ(1-μ))`
 
-- **`iterative-linear-least-squares`** — Iterative solvers for symmetric linear systems (A × y = b): SYMMLQ and Conjugate Gradient methods.
+For beta regression: `W = diag(φ·μ(1-μ))`, `z = η + (y-μ)/(μ(1-μ))`
 
 ## Usage
 
 ```clojure
-(require '[provisdom.solvers.root-solvers :as root])
-(require '[provisdom.solvers.nonlinear-programming :as nlp])
-(require '[provisdom.solvers.interpolation-1d :as interp-1d])
+(require '[provisdom.solvers.roots.continuous :as roots])
+(require '[provisdom.solvers.common :as common])
+(require '[provisdom.solvers.optimize.nlp-bounded :as nlp])
+(require '[provisdom.solvers.interpolation.univariate :as interp])
 
 ;; Find root of f(x) = x³ - 3x
-(root/root-solver
-  {::root/univariate-f (fn [x] (- (* x x x) (* 3 x)))
-   ::root/guess 2.0
-   ::root/interval [-5.0 5.0]})
+(roots/root-solver
+  {::roots/univariate-f (fn [x] (- (* x x x) (* 3 x)))
+   ::roots/guess 2.0
+   ::roots/interval [-5.0 5.0]})
 
 ;; Minimize f(x,y) = x² + y² subject to bounds
-(nlp/bounded-nonlinear-programming-without-evolutionary
-  {::nlp/objective (fn [da] (let [[x y] da] (+ (* x x) (* y y))))
-   ::nlp/vars-guess [1.0 1.0]
+(nlp/bounded-nlp-without-evolutionary
+  {::common/objective (fn [da] (let [[x y] da] (+ (* x x) (* y y))))
+   ::common/vars-guess [1.0 1.0]
    ::nlp/var-intervals [[-10.0 10.0] [-10.0 10.0]]})
 
 ;; Cubic spline interpolation
-(let [f (interp-1d/interpolation-1D
-          {::interp-1d/f-vals [0.0 1.0 4.0 9.0]
-           ::interp-1d/x-vals [0.0 1.0 2.0 3.0]})]
+(let [f (interp/interpolation-1D
+          {::interp/f-vals [0.0 1.0 4.0 9.0]
+           ::interp/x-vals [0.0 1.0 2.0 3.0]})]
   (f 1.5))
 ```
 

siderail/spline.clj

@@ -59,6 +59,7 @@
   :end)
 
 
+
 (comment
   (def rho 0.9)
   (def x (vec (sort (repeatedly 100 #(rand (* 10.0 2.0 Math/PI))))))
@@ -85,4 +86,4 @@
     #_(pprint data)
     (oz/v! (scatter data :log-variance :dof nil)))
 
-  :end)
+  :end)

siderail/stepwise.clj

@@ -10,7 +10,6 @@
 
 (defonce foo (ost/instrument))
 
-
 (update
   (stepwise/solve
     {::stepwise/x-mx               [[7.0 2.8] [8.0 9.3] [3.0 9.9]
@@ -39,4 +38,4 @@
      ::stepwise/prob-of-model-fn   (fn [component-group]
                                      1.0)})
   ::stepwise/component-group
-  keys)
+  keys)

src/provisdom/solvers/common.clj

@@ -14,11 +14,13 @@
 (s/def ::solver-category
   #{::bad-supplied-function   ; user-provided function has issues (wrong arity, throws, returns invalid values)
     ::diverged                ; algorithm diverged (rounding errors, etc.)
+    ::infeasible              ; constraints are contradictory or cannot be satisfied
     ::invalid-input           ; input violates solver requirements
     ::max-iterations-exceeded ; solver hit iteration limit without converging
     ::not-bracketed           ; root not bracketed (for bracketing root solvers)
     ::out-of-bounds           ; query point outside valid interpolation range
-    ::singular-matrix})       ; matrix is singular or ill-conditioned
+    ::singular-matrix         ; matrix is singular or ill-conditioned
+    ::unbounded})             ; objective can improve infinitely
 
 ;;;CONSTANTS
 (def accu-precision
@@ -191,3 +193,20 @@
   (s/with-gen (s/and ::m/int+ #(<= % 20))
     #(gen/choose 2 8)))
 
+;;; Smoothing Spline Specs
+(s/def ::smoothing-lambda
+  (s/with-gen ::m/non-
+    #(s/gen (s/double-in :min 0.0 :max 10.0 :NaN? false :infinite? false))))
+
+(s/def ::effective-df
+  (s/with-gen (s/and ::m/finite+ #(>= % 2.0))
+    #(s/gen (s/double-in :min 2.0 :max 10.0 :NaN? false :infinite? false))))
+
+;;;NLP (Nonlinear Programming)
+(s/def ::parallel? boolean?)
+(s/def ::objective ::array->number)
+
+(s/def ::cobyla-initial-change
+  (s/with-gen ::m/finite+
+    #(s/gen (s/double-in :infinite? false :NaN? false :min 1e-3 :max 1e3))))
+