Make SparseArrays a regular dependency of OrdinaryDiffEqCore.jl by JoshuaLampert · Pull Request #3031 · SciML/OrdinaryDiffEq.jl

JoshuaLampert · 2026-02-02T14:11:30Z

Checklist

Appropriate tests were added
Any code changes were done in a way that does not break public API
All documentation related to code changes were updated
The new code follows the
contributor guidelines, in particular the SciML Style Guide and
COLPRAC.
Any new documentation only uses public API

Additional context

This fixes #3025. See also #3011 (comment). SparseArrays.jl is a stdlib. So I think having it as a regular dependency should be fine.

ranocha · 2026-02-02T19:54:12Z

lib/OrdinaryDiffEqCore/src/misc_utils.jl

+# Efficient O(nnz) isdiag check for sparse matrices.
+# Standard isdiag is O(n²) which is prohibitively slow for large sparse matrices.
+"""
+    _isdiag(A::SparseMatrixCSC)
+
+Check if a sparse matrix is diagonal in O(nnz) time by traversing the CSC structure directly.
+Returns `true` if all non-zero elements are on the diagonal.
+"""
+function _isdiag(A::SparseArrays.SparseMatrixCSC)
+    m, n = size(A)
+    m != n && return false
+    @inbounds for j in 1:n
+        for k in A.colptr[j]:(A.colptr[j + 1] - 1)
+            A.rowval[k] != j && return false
+        end
+    end
+    return true
+end
+


Has this been upstreamed to SparseArrays?

no, but it should be.

Yes, I agree. So should this PR just revert #3011?

no. We should open this upstream, but keep this optimization in the meantime.

This is breaking CI in at least 5-10 our repos. This is pretty urgent for us.

Isn't the breaking part that this used to be in an extension? Shouldn't bringing it into OrdinaryDiffEqCore fix your issue?

Yes, if we can merge this PR as is that would also be fine as a hot fix. I thought you wanted to keep the status quo until this is upstreamed to SparseArrays.jl.

sorry for the confusion. I meant keeping the status quo of implimenting _isdiag ourselves, but changing where it lives to Core.

Yes, in other words: merging this PR because changing where _isdiag lives is exactly what this PR does. That's fine with me. Or do you mean something else than this PR currently does?

devmotion · 2026-02-02T22:08:14Z

SparseArrays.jl is a stdlib. So I think having it as a regular dependency should be fine.

SparseArrays is not part of the default system image anymore since Julia 1.10. This actually motivated a push towards SparseArrays extensions, e.g. JuliaArrays/FillArrays.jl#285 or JuliaStats/Distances.jl#251 or JuliaDiff/ChainRulesCore.jl#638

JoshuaLampert · 2026-02-03T08:54:42Z

SparseArrays.jl is a stdlib. So I think having it as a regular dependency should be fine.

SparseArrays is not part of the default system image anymore since Julia 1.10. This actually motivated a push towards SparseArrays extensions, e.g. JuliaArrays/FillArrays.jl#285 or JuliaStats/Distances.jl#251 or JuliaDiff/ChainRulesCore.jl#638

Ok, I see. Thanks for the clarification. But in this case this code should simply live in SparseArrays.jl. So I would propose to merge this PR as a hotfix to fix CI of Downstream packages using Julia v1.10 and then open an issue/a PR in SparseArrays.jl to include it there. Once that PR is merged and released the function can be removed here again. We need a fix for this as soon as possible because there are multiple PRs, which wait for this. Some I know of:

Even here the corresponding CI job gives an error, e.g., https://github.com/SciML/OrdinaryDiffEq.jl/actions/runs/21580707995/job/62177277916?pr=3024#step:5:1011, which for some reason doesn't let the job fail.

devmotion · 2026-02-03T09:00:16Z

Surely, the code should be moved to SparseArrays.

I haven't followed the bug reports closely, but to me it seems that the current problem might just be the known bug in Julia 1.10 with cycle detection that was fixed in subsequent Julia releases?

JoshuaLampert · 2026-02-03T09:08:23Z

Yes, that might be the case. But if OrdinaryDiffEq.jl officially supports julia v1.10, it should not give an error (or even a warning) on julia v1.10 when simply doing using OrdinaryDiffEqCore. This means it has to be fixed in OrdinaryDiffEq.jl even if it is a bug of Julia v1.10. Either OrdinaryDiffEq.jl should not support Julia v1.10 (I guess that's not an option) or it should work. This PR makes it work again under Julia v1.10. If there are no better solutions right now and no major issues with this fix, it should get merged soon. A better solution (which might take a bit more time than a hotfix) can be implemented afterwards.

devmotion · 2026-02-03T10:06:16Z

The only possible concern would be whether this just moves the problem downstream. How does this change affect downstream packages on Julia 1.10, that depend on OrdinaryDiffEqCore and have a SparseArrays extension? Will 1.10 start reporting a circular dependency for them?

JoshuaLampert · 2026-02-03T10:11:10Z

That could be the case. I don't know of any package, which has an extension for SparseArrays.jl and depends on OrdinaryDiffEqCore.jl to test it though.

devmotion · 2026-02-03T11:07:12Z

Maybe OrdinaryDiffEqDifferentiation?

JoshuaLampert · 2026-02-03T11:43:21Z

I tested it and I think the following tells us that OrdinaryDiffEqDifferentiation.jl is also fine with this PR:

lampert@nb1759:~/.julia/dev/OrdinaryDiffEq$ rm -r ~/.julia/compiled/v1.10/OrdinaryDiffEqCore/ ~/.julia/compiled/v1.10/OrdinaryDiffEqDifferentiation/ ~/.julia/compiled/v1.10/OrdinaryDiffEqDifferentiationSparseArraysExt/ ~/.julia/compiled/v1.10/EnzymeCore
lampert@nb1759:~/.julia/dev/OrdinaryDiffEq$ git checkout fix-circular-dep-v1.10 
Switched to branch 'fix-circular-dep-v1.10'
Your branch is up to date with 'origin/fix-circular-dep-v1.10'.
lampert@nb1759:~/.julia/dev/OrdinaryDiffEq$ julia +1.10 --project=run
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.10 (2025-06-27)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(run) pkg> dev OrdinaryDiffEqCore
Path `/home/lampert/.julia/dev/OrdinaryDiffEq` exists and looks like the correct repo. Using existing path.
   Resolving package versions...
    Updating `~/.julia/dev/OrdinaryDiffEq/run/Project.toml`
  [bbf590c4] + OrdinaryDiffEqCore v3.4.1 `~/.julia/dev/OrdinaryDiffEq/lib/OrdinaryDiffEqCore`
    Updating `~/.julia/dev/OrdinaryDiffEq/run/Manifest-v1.10.toml`
  [bbf590c4] ~ OrdinaryDiffEqCore v3.4.0 `~/.julia/dev/OrdinaryDiffEq/lib/OrdinaryDiffEqCore` ⇒ v3.4.1 `~/.julia/dev/OrdinaryDiffEq/lib/OrdinaryDiffEqCore`

(run) pkg> dev OrdinaryDiffEqDifferentiation
Path `/home/lampert/.julia/dev/OrdinaryDiffEq` exists and looks like the correct repo. Using existing path.
   Resolving package versions...
    Updating `~/.julia/dev/OrdinaryDiffEq/run/Project.toml`
  [4302a76b] + OrdinaryDiffEqDifferentiation v2.0.0 `~/.julia/dev/OrdinaryDiffEq/lib/OrdinaryDiffEqDifferentiation`
  No Changes to `~/.julia/dev/OrdinaryDiffEq/run/Manifest-v1.10.toml`

(run) pkg> add EnzymeCore
   Resolving package versions...
    Updating `~/.julia/dev/OrdinaryDiffEq/run/Project.toml`
  [f151be2c] + EnzymeCore v0.8.18
  No Changes to `~/.julia/dev/OrdinaryDiffEq/run/Manifest-v1.10.toml`
Precompiling packages finished.
  35 dependencies successfully precompiled in 85 seconds. 141 already precompiled.

(run) pkg> rm EnzymeCore
    Updating `~/.julia/dev/OrdinaryDiffEq/run/Project.toml`
  [f151be2c] - EnzymeCore v0.8.18
  No Changes to `~/.julia/dev/OrdinaryDiffEq/run/Manifest-v1.10.toml`

(run) pkg> rm OrdinaryDiffEqDifferentiation
    Updating `~/.julia/dev/OrdinaryDiffEq/run/Project.toml`
  [4302a76b] - OrdinaryDiffEqDifferentiation v2.0.0 `~/.julia/dev/OrdinaryDiffEq/lib/OrdinaryDiffEqDifferentiation`
  No Changes to `~/.julia/dev/OrdinaryDiffEq/run/Manifest-v1.10.toml`

julia> 
lampert@nb1759:~/.julia/dev/OrdinaryDiffEq$ rm -r ~/.julia/compiled/v1.10/OrdinaryDiffEqCore/ ~/.julia/compiled/v1.10/OrdinaryDiffEqDifferentiation/ ~/.julia/compiled/v1.10/OrdinaryDiffEqDifferentiationSparseArraysExt/ ~/.julia/compiled/v1.10/EnzymeCore
lampert@nb1759:~/.julia/dev/OrdinaryDiffEq$ git checkout master
Switched to branch 'master'
Your branch is up to date with 'origin/master'.
lampert@nb1759:~/.julia/dev/OrdinaryDiffEq$ julia +1.10 --project=run
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.10.10 (2025-06-27)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(run) pkg> dev OrdinaryDiffEqDifferentiation
Path `/home/lampert/.julia/dev/OrdinaryDiffEq` exists and looks like the correct repo. Using existing path.
   Resolving package versions...
    Updating `~/.julia/dev/OrdinaryDiffEq/run/Project.toml`
  [bbf590c4] ~ OrdinaryDiffEqCore v3.4.1 `~/.julia/dev/OrdinaryDiffEq/lib/OrdinaryDiffEqCore` ⇒ v3.4.0 `~/.julia/dev/OrdinaryDiffEq/lib/OrdinaryDiffEqCore`
  [4302a76b] + OrdinaryDiffEqDifferentiation v2.0.0 `~/.julia/dev/OrdinaryDiffEq/lib/OrdinaryDiffEqDifferentiation`
    Updating `~/.julia/dev/OrdinaryDiffEq/run/Manifest-v1.10.toml`
  [bbf590c4] ~ OrdinaryDiffEqCore v3.4.1 `~/.julia/dev/OrdinaryDiffEq/lib/OrdinaryDiffEqCore` ⇒ v3.4.0 `~/.julia/dev/OrdinaryDiffEq/lib/OrdinaryDiffEqCore`

(run) pkg> add EnzymeCore
   Resolving package versions...
    Updating `~/.julia/dev/OrdinaryDiffEq/run/Project.toml`
  [f151be2c] + EnzymeCore v0.8.18
  No Changes to `~/.julia/dev/OrdinaryDiffEq/run/Manifest-v1.10.toml`
┌ Warning: Circular dependency detected.
│ Precompilation will be skipped for dependencies in this cycle:
│  ┌ OrdinaryDiffEqCore → OrdinaryDiffEqCoreEnzymeCoreExt
│  └─ OrdinaryDiffEqCore → OrdinaryDiffEqCoreSparseArraysExt
│ Precompilation will also be skipped for the following, which depend on the above cycle:
│   OrdinaryDiffEqDifferentiation → OrdinaryDiffEqDifferentiationSparseArraysExt
│   OrdinaryDiffEqLowStorageRK
│   OrdinaryDiffEqLowOrderRK
│   OrdinaryDiffEqDifferentiation
│   OrdinaryDiffEqNonlinearSolve
│   OrdinaryDiffEqSDIRK
└ @ Pkg.API.Precompilation ~/.julia/juliaup/julia-1.10.10+0.x64.linux.gnu/share/julia/stdlib/v1.10/Pkg/src/precompilation.jl:583
Precompiling packages finished.
  28 dependencies successfully precompiled in 75 seconds. 141 already precompiled. 8 skipped due to circular dependency.

Please tell me if I should test something else.

devmotion · 2026-02-03T12:08:10Z

I was curious, and did a ] add OrdinaryDiffEqCore@3.4.0 on both Julia 1.10 and 1.12. In the former case, EnzymeCore and SparseArrays are installed as well whereas in the latter case only EnzymeCore is installed.

On Julia 1.10:

(jl_KPTG1m) pkg> why EnzymeCore
  OrdinaryDiffEqCore → DiffEqBase → BracketingNonlinearSolve → NonlinearSolveBase → EnzymeCore

(jl_KPTG1m) pkg> why SparseArrays
  OrdinaryDiffEqCore → DiffEqBase → BracketingNonlinearSolve → NonlinearSolveBase → SciMLBase → Statistics → SparseArrays
  OrdinaryDiffEqCore → DiffEqBase → BracketingNonlinearSolve → NonlinearSolveBase → SciMLJacobianOperators → SciMLBase → Statistics → SparseArrays
  OrdinaryDiffEqCore → DiffEqBase → BracketingNonlinearSolve → SciMLBase → Statistics → SparseArrays
  OrdinaryDiffEqCore → DiffEqBase → SciMLBase → Statistics → SparseArrays
  OrdinaryDiffEqCore → SciMLBase → Statistics → SparseArrays

On Julia 1.12:

(jl_IlqkUf) pkg> why EnzymeCore
  OrdinaryDiffEqCore → DiffEqBase → BracketingNonlinearSolve → NonlinearSolveBase → EnzymeCore

(jl_IlqkUf) pkg> why SparseArrays
ERROR: The following package names could not be resolved:
 * SparseArrays (not found in project or manifest)

On Julia 1.10, apparently SparseArrays is only pulled in via Statistics but the dependency of Statistics on SparseArrays was apparently removed in more recent Julia versions.

NonlinearSolveBase depends on EnzymeCore due to https://github.com/SciML/NonlinearSolve.jl/blob/ed96792bf00e5d42c7a01ea7da676d78ea37489a/lib/NonlinearSolveBase/src/autodiff.jl#L7 and https://github.com/SciML/NonlinearSolve.jl/blob/ed96792bf00e5d42c7a01ea7da676d78ea37489a/lib/NonlinearSolveBase/src/autodiff.jl#L17. It seems slightly unfortunate that both a dependency on EnzymeCore and a weak dependency on Enzyme are needed... I wonder if ADTypes could support specification of modes without having to depend on EnzymeCore (using Val?) or whether the list of default backends could be made dynamic and only list supported backends (e.g., the Enzyme extension would add AutoEnzyme to the list etc.).

JoshuaLampert · 2026-02-03T12:15:45Z

As long as this issue is fixed soon I'm fine with any fix. We only need it as soon as possible to be able to properly continue our work in the downstream packages.

devmotion · 2026-02-03T12:15:52Z

My take-away is that - regardless of the Julia version - as long as NonlinearSolveBase has a strong dependency on EnzymeCore, it doesn't make sense to make EnzymeCore a weak dependency of OrdinaryDiffEqCore. Maybe making EnzymeCore a strong dependency in OrdinaryDiffEqCore would be sufficient to break the "cycle" that Julia 1.10 (incorrectly IMO) complains about.

JoshuaLampert · 2026-02-03T12:33:36Z

My take-away is that - regardless of the Julia version - as long as NonlinearSolveBase has a strong dependency on EnzymeCore, it doesn't make sense to make EnzymeCore a weak dependency of OrdinaryDiffEqCore. Maybe making EnzymeCore a strong dependency in OrdinaryDiffEqCore would be sufficient to break the "cycle" that Julia 1.10 (incorrectly IMO) complains about.

Sounds good. I created #3032 for that.

ChrisRackauckas · 2026-02-03T14:02:57Z

The answer isn't to do that... just solve it at its core (pun intended) and remove the EnzymeCore from NonlinearSolveBase.

JoshuaLampert · 2026-02-03T14:11:29Z

If we can do that within the next couple of hours fine (but I will not create a PR for that). Otherwise I would prefer a hotfix, which can be improved later.

ChrisRackauckas · 2026-02-03T14:14:08Z

I don't think a Julia v1.10 hotfix release will come that quickly... but that doesn't mean we should put bugs in other places.

JoshuaLampert · 2026-02-03T14:15:22Z

I don't mean a Julia v1.10 hotfix release, but a hotfix release within the SciML framework.

ChrisRackauckas · 2026-02-03T14:19:44Z

The cleanest thing would be to just add Forward and Reverse structs to ADTypes.jl and allow those in there in a way that auto-converts or something.

JoshuaLampert · 2026-02-03T14:21:26Z

Great, but my answer from above is still valid:

If we can do that within the next couple of hours fine (but I will not create a PR for that). Otherwise I would prefer a hotfix, which can be improved later.

ChrisRackauckas · 2026-02-03T14:23:42Z

That would be literally just as fast to do.

JoshuaLampert · 2026-02-03T14:24:37Z

Sounds great! I'm looking forward to a corresponding PR.

ChrisRackauckas · 2026-02-03T14:27:01Z

Just waiting for @wsmoses to be up and should be done in like one chat.

ChrisRackauckas · 2026-02-04T16:16:46Z

I couldn't get in touch with @wsmoses so doing the hotfix for now.

JoshuaLampert · 2026-02-04T17:30:55Z

Thanks!

JoshuaLampert added 2 commits February 2, 2026 15:09

make SparseArrays a regular dependency of OrdinaryDiffEqCore.jl

9b156c1

remove SparseArrays from targets

84ee375

JoshuaLampert mentioned this pull request Feb 2, 2026

Upgrade Restart API to OrdinaryDiffEqCore 4 trixi-framework/Trixi.jl#2773

Draft

ranocha reviewed Feb 2, 2026

View reviewed changes

JoshuaLampert mentioned this pull request Feb 3, 2026

Make EnzymeCore.jl a strong dependency of OrdinaryDiffEqCore.jl #3032

Merged

5 tasks

ranocha mentioned this pull request Feb 4, 2026

Hotfix: restrict OrdinaryDiffEqCore to fix CI trixi-framework/Trixi.jl#2775

Merged

ChrisRackauckas closed this in #3032 Feb 4, 2026

JoshuaLampert deleted the fix-circular-dep-v1.10 branch February 4, 2026 17:30

Uh oh!

Conversation

JoshuaLampert commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Additional context

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devmotion commented Feb 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JoshuaLampert commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devmotion commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JoshuaLampert commented Feb 3, 2026

Uh oh!

devmotion commented Feb 3, 2026

Uh oh!

JoshuaLampert commented Feb 3, 2026

Uh oh!

devmotion commented Feb 3, 2026

Uh oh!

JoshuaLampert commented Feb 3, 2026

Uh oh!

devmotion commented Feb 3, 2026

Uh oh!

JoshuaLampert commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devmotion commented Feb 3, 2026

Uh oh!

JoshuaLampert commented Feb 3, 2026

Uh oh!

ChrisRackauckas commented Feb 3, 2026

Uh oh!

JoshuaLampert commented Feb 3, 2026

Uh oh!

ChrisRackauckas commented Feb 3, 2026

Uh oh!

JoshuaLampert commented Feb 3, 2026

Uh oh!

ChrisRackauckas commented Feb 3, 2026

Uh oh!

JoshuaLampert commented Feb 3, 2026

Uh oh!

ChrisRackauckas commented Feb 3, 2026

Uh oh!

JoshuaLampert commented Feb 3, 2026

Uh oh!

ChrisRackauckas commented Feb 3, 2026

Uh oh!

ChrisRackauckas commented Feb 4, 2026

Uh oh!

JoshuaLampert commented Feb 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

JoshuaLampert commented Feb 2, 2026 •

edited

Loading

devmotion commented Feb 2, 2026 •

edited

Loading

JoshuaLampert commented Feb 3, 2026 •

edited

Loading

devmotion commented Feb 3, 2026 •

edited

Loading

JoshuaLampert commented Feb 3, 2026 •

edited

Loading