HTTP errors during CI runs

[settwi]

I’ve seen some timeouts and 404 errors during CI runs. Is there a way we can trigger auto-reruns if an HTTP error is encountered?

@Cadair or @samaloney ?

[Cadair]

What's making the http requests? There's a bunch of ways to work around such issues, but they are all pretty context dependent.

[settwi]

One was getting the CHIANTI .sav files. Another was during a run on main, with a 404 issue. I don’t recall which step that was.

[Cadair]

There's no blanket "retry only on http error" solution, the best thing to do would be to mirror the chainti data somewhere for that.

There's also pytest-rerun we used in core for a while if we have a real problematic test, but it slows down the build if the test really fails.

[settwi]

@KriSun95 mentioned it could make sense to put the data files on the DIAS server as the primary mirror, we haven’t had issues with that location… @samaloney

as for other testing, the 404 error was during CI setup not during an actual test. maybe it was a weird one-off thing.

[Cadair]

I think github had some issues with cloning repos the other day, took down some builds elsewhere too

[Cadair]

If the files don't change that often we could also put them on data.sunpy.org (in the sunpy-data repo)

[samaloney]

Yea so I think this is primarily downloading the CHIANTI files from NASA with Sunpy DataManager. Could we use a cache at the actions level? Or as suggested add to sunpy data repo?

So subset of the file here https://soho.nascom.nasa.gov/solarsoft/packages/xray/dbase/chianti/

[Cadair]

Could we use a cache at the actions level? Or as suggested add to sunpy data repo?

Yes lol. We could do either or both.

So subset of the file here https://soho.nascom.nasa.gov/solarsoft/packages/xray/dbase/chianti/

How big of a subset and how often do they change?

[samaloney]

Currently I think we only use 3 files, I don't think they change too often. What's interesting is we have alternative urls set in the DataManger but they are't being used when the 1st one fail.

sunkit-spex/sunkit_spex/models/physical/io.py

Lines 16 to 24 in c9740cd

	@manager.require(
	"chianti_lines",
	[
	"https://soho.nascom.nasa.gov/solarsoft/packages/xray/dbase/chianti/chianti_lines_1_10_v71.sav",
	"https://lmsal.com/solarsoft/ssw/packages/xray/dbase/chianti/chianti_lines_1_10_v71.sav",
	],
	"2046d818efec207a83e9c5cc6ba4a5fa8574bf8c2bd8a6bb9801e4b8a2a0c677",
	)
	def load_chianti_lines_lite():