Skip to content

Commit 7a452c9

Browse files
cailmdaleycailmdaley
authored
Merge pull request #165 from CosmoStat/feature/ellipticity-suffix
Validated locally with test_catalog_paths_exist and manual CosmologyValidation runs on SP_v1.4.6.
2 parents 547ad61 + 99d24a3 commit 7a452c9

25 files changed

Lines changed: 3507 additions & 3014 deletions

‎.beads/issues.jsonl‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
{"id":"sp_validation-15j","title":"Review and commit uncommitted ellipticity-suffix work","description":"7 files with uncommitted changes: cosmo_val.py, rho_tau.py, cosmosis_fitting.py, cosmic shear paper configs. Appear to be bug fixes (bin construction, file naming, covariance).","status":"closed","priority":2,"issue_type":"task","created_at":"2025-12-16T17:31:10.777988207+01:00","updated_at":"2025-12-16T20:08:21.251378337+01:00","closed_at":"2025-12-16T20:08:21.251378337+01:00"}
2+
{"id":"sp_validation-1en","title":"Merge PR #165 (catalog path standardization)","description":"PR approved by Lisa and Sacha since Oct/Nov. Ready to merge for 6 weeks. Run: gh pr merge 165 --merge","status":"open","priority":2,"issue_type":"task","created_at":"2025-12-16T17:31:16.438050654+01:00","updated_at":"2025-12-16T17:31:16.438050654+01:00","dependencies":[{"issue_id":"sp_validation-1en","depends_on_id":"sp_validation-d4d","type":"blocks","created_at":"2025-12-16T17:31:37.693645344+01:00","created_by":"daemon"}]}
3+
{"id":"sp_validation-7ww","title":"PR #165: data_base_dir parameter removed (breaking)","description":"CosmologyValidation.__init__() no longer accepts data_base_dir. Paths now absolute in cat_config.yaml. May break external scripts. Approved by Lisa/Sacha - presumably validated for their use cases.","status":"open","priority":2,"issue_type":"task","created_at":"2025-12-16T20:19:19.480910265+01:00","updated_at":"2025-12-16T20:19:19.480910265+01:00"}
4+
{"id":"sp_validation-d4d","title":"Run test suite and validate","description":"After pytest fix, run full test suite to validate ellipticity-suffix changes.","status":"closed","priority":2,"issue_type":"task","created_at":"2025-12-16T17:31:13.621333151+01:00","updated_at":"2025-12-16T17:41:14.552278446+01:00","closed_at":"2025-12-16T17:41:14.552278446+01:00","dependencies":[{"issue_id":"sp_validation-d4d","depends_on_id":"sp_validation-es5","type":"blocks","created_at":"2025-12-16T17:31:37.617063707+01:00","created_by":"daemon"}]}
5+
{"id":"sp_validation-es5","title":"Fix pytest config (remove --pydocstyle)","description":"pyproject.toml line 81 has deprecated --pydocstyle flag. Tests won't run until fixed.","status":"closed","priority":2,"issue_type":"bug","created_at":"2025-12-16T17:31:07.933332484+01:00","updated_at":"2025-12-16T17:37:12.452426193+01:00","closed_at":"2025-12-16T17:37:12.452426193+01:00"}
6+
{"id":"sp_validation-gco","title":"Fix pyccl Cosmology.get API deprecation","description":"test_ccl_to_camb_basic and test_camb_backend fail due to pyccl API change. Cosmology object no longer has .get() method.","status":"open","priority":2,"issue_type":"bug","created_at":"2025-12-16T17:41:17.33553874+01:00","updated_at":"2025-12-16T17:41:17.33553874+01:00"}
7+
{"id":"sp_validation-o97","title":"Add nellbins suffix to pseudo-Cl output filenames","description":"For reproducibility, output files should include binning params. Deferred pending binning resolution in PR #168.","status":"open","priority":2,"issue_type":"task","created_at":"2025-12-16T18:40:03.290961433+01:00","updated_at":"2025-12-16T18:40:03.290961433+01:00"}
8+
{"id":"sp_validation-rq0","title":"Add SP_v1.4.5_glass_mock to test allow-list or fix path","description":"test_catalog_paths_exist fails: SP_v1.4.5_glass_mock shear file missing. Either add to allow-list or update path in cat_config.yaml","status":"open","priority":2,"issue_type":"bug","created_at":"2025-12-16T17:42:59.575066932+01:00","updated_at":"2025-12-16T17:42:59.575066932+01:00"}
9+
{"id":"sp_validation-xop","title":"Coordinate NmtBin linear binning with PR #168","description":"Deferred from ellipticity-suffix. Changes to NmtBin constructor, n_bands vs n_ell_bins, dynamic step. Sacha's PR #168 has 'To be implemented correctly' comment. Coordinate there.","status":"open","priority":2,"issue_type":"task","created_at":"2025-12-16T18:40:00.538355923+01:00","updated_at":"2025-12-16T18:40:00.538355923+01:00"}

‎.gitignore‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -171,5 +171,15 @@ notebooks/demo_calibrate_minimal_cat.ipynb
171171
notebooks/leakage_minimal.ipynb
172172
notebooks/demo_add_bands.ipynb
173173

174-
cosmo_inference/cosmosis_config/cosmosis_pipeline_S*.ini
175-
cosmo_inference/cosmosis_config/priors_S*.ini
174+
CLAUDE.md
175+
AGENTS.md
176+
cosmo_inference/cosmosis_config/*.ini
177+
!cosmo_inference/cosmosis_config/cosmosis_pipeline_A_ia_cell.ini
178+
!cosmo_inference/cosmosis_config/cosmosis_pipeline_A_ia.ini
179+
!cosmo_inference/cosmosis_config/cosmosis_pipeline_A_psf.ini
180+
!cosmo_inference/cosmosis_config/cosmosis_pipeline_A.ini
181+
!cosmo_inference/cosmosis_config/cosmosis_pipeline_glass_mock_00001_cell.ini
182+
!cosmo_inference/cosmosis_config/cosmosis_pipeline_glass_mock_00001.ini
183+
!cosmo_inference/cosmosis_config/cosmosis_pipeline_SP_v1.4.6_A_minsep=1.0_maxsep=250.0_nbins=20_npatch=1.ini
184+
!cosmo_inference/cosmosis_config/cosmosis_pipeline_SP_v1.4.6_A_minsep=1.0_maxsep=250.0_nbins=20_npatch=1_cell.ini
185+
!cosmo_inference/cosmosis_config/priors_mock.ini

‎cosmo_inference/README.md‎

Lines changed: 32 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -7,20 +7,42 @@ This folder contains the files neccessary to run the cosmological inference pipe
77
To run the pipeline, one would need to have installed [CosmoSIS](https://cosmosis.readthedocs.io/en/latest/) and [CosmoCov](https://github.com/CosmoLike/CosmoCov). To PSF leakage parameters, the fork of [cosmosis-standard-library](https://github.com/sachaguer/cosmosis-standard-library/) of Sacha Guerrini has to be used.
88

99
### To Run
10-
Run the bash script within this folder
10+
The inference pipeline is now orchestrated through Python. Run the main Snakemake workflow from the parent directory:
1111

12+
```bash
13+
snakemake -j<jobs> inference_fiducial
1214
```
13-
$ ./pipeline.sh
14-
```
15-
with one of the following flags:
16-
17-
`--pcf`: This step runs the `cosmo_val.py` script to calculate the various 2 point correlation functions. It will also write the $\xi_{pm}$ correlation functions in a fits file.
1815

19-
`--covmat`: The covariance matrix is calculated here using CosmoCov, by reading in the `./cosmocov_config/cosmocov_{output_root}.ini` file. **Hence make sure the `output_root` here corresponds to the one entered in the prompt**.
16+
This will automatically execute all steps:
17+
1. Calculate 2PCF ($\xi_{pm}$) via `cosmo_val.py`
18+
2. Compute covariance matrices using CosmoCov
19+
3. Prepare CosmoSIS data (FITS) via `cosmosis_fitting.py`
20+
4. Run CosmoSIS inference
21+
22+
For standalone FITS data preparation (real-space inputs plus optional pseudo-$C_\ell$ data), you can also use the Python script directly:
23+
24+
```bash
25+
python scripts/cosmosis_fitting.py \
26+
--cosmosis-root "catalog_version_config" \
27+
--data-dir "/path/to/output/chains" \
28+
--nz-file "/path/to/nz_file.txt" \
29+
--out-file "/path/to/output.fits" \
30+
--xi "/path/to/xi_plus.fits" "/path/to/xi_minus.fits" \
31+
--cov-xi "/path/to/covariance.txt" \
32+
--use-rho-tau \
33+
--rho-stats "/path/to/rho_stats.fits" \
34+
--tau-stats "/path/to/tau_stats.fits" \
35+
--cov-tau "/path/to/cov_tau.npy" \
36+
--cl-file "/path/to/pseudo_cl.fits" \
37+
--cov-cl "/path/to/pseudo_cl_cov.fits"
38+
```
2039

21-
`--inference`: This step writes out the relevant `./cosmosis_config/cosmosis_{output_root}.ini` file, in order to run CosmoSIS to conduct the cosmological inference. It also combines the data needed by CosmoSIS: the $\xi_{pm}$ fits files calculated in `cosmo_val.py`, the covariance matrix, and the nz catalogue, into a single `.fits` file. It also fetches the rho-statistics computed in `cosmo_val.py` to marginalize on PSF leakage parameters. To do so, specify the path to the `cosmo_val` outputs. Note that the Rho- and Tau-statistics have to be stored in a folder `rho_tau_stats` in this folder. If the 2PCF $\xi_{pm}$ are not found it will raise an `Error` to run `cosmo_val`. If the Rho- and Tau-statistics do not exist, the script will raise a `Warning` to run `cosmo_val` but will create a data vector without the Rho- and Tau-statistics.
40+
You can view all available options with:
41+
```bash
42+
python scripts/cosmosis_fitting.py --help
43+
```
2244

23-
`--mcmc_process`: You can finally analyse the chains with the `MCMC.ipynb` notebook.
45+
Ensure the pseudo-$C_\ell$ spectra (`pseudo_cl_*.fits`) and their covariance (`pseudo_cl_cov_*.fits`) produced by `cosmo_val.py` exist for the requested catalog version (or mock seed) before running the standalone command.
2446

2547

26-
This is the pipeline used to derive cosmological constraints with cosmic shear data from the UNIONS v1.4 catalogue.
48+
This is the pipeline used to derive cosmological constraints with cosmic shear data from the UNIONS v1.4 catalogue.

‎cosmo_inference/cosmosis_config/cosmosis_pipeline_SP_v1.4.5_A_minsep=1_maxsep=250_nbins=20_npatch=1.ini‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
#parameters used elsewhere in this file
2+
[DEFAULT]
3+
FITS_FILE = data/SP_v1.4.5_A_minsep=1_maxsep=250_nbins=20_npatch=1/cosmosis_SP_v1.4.5_A_minsep=1_maxsep=250_nbins=20_npatch=1.fits
4+
SCRATCH = /n09data/guerrini/output_chains/SP_v1.4.5_A_minsep=1_maxsep=250_nbins=20_npatch=1
5+
COSMOSIS_DIR = /home/guerrini/cosmosis-standard-library
6+
7+
8+
[pipeline]
9+
priors = cosmosis_config/priors_psf.ini
10+
values = cosmosis_config/values_psf.ini
11+
modules = consistency sample_S8 camb load_nz_fits photoz_bias linear_alignment projection add_intrinsic 2pt_shear add_xi_sys tau_from_rho 2pt_like
12+
likelihoods = 2pt_like
13+
extra_output = cosmological_parameters/omega_lambda cosmological_parameters/S_8 cosmological_parameters/sigma_8 cosmological_parameters/omega_m
14+
timing = T
15+
debug = T
16+
17+
[runtime]
18+
sampler = polychord
19+
verbosity = debug
20+
21+
[test]
22+
save_dir = %(SCRATCH)s/test_new_pipeline
23+
24+
[polychord]
25+
polychord_outfile_root = SP_v1.4.5_A_minsep=1_maxsep=250_nbins=20_npatch=1
26+
live_points = 192
27+
feedback = 3
28+
resume = T
29+
base_dir = %(SCRATCH)s/polychord
30+
31+
[output]
32+
filename = %(SCRATCH)s/SP_v1.4.5_A_minsep=1_maxsep=250_nbins=20_npatch=1/samples_SP_v1.4.5_A_minsep=1_maxsep=250_nbins=20_npatch=1.txt
33+
format = text
34+
lock = F
35+
36+
[consistency]
37+
file = %(COSMOSIS_DIR)s/utility/consistency/consistency_interface.py
38+
verbose = F
39+
40+
[sample_S8]
41+
file = %(COSMOSIS_DIR)s/utility/sample_sigma8/sample_S8.py
42+
43+
[camb]
44+
file = %(COSMOSIS_DIR)s/boltzmann/camb/camb_interface.py
45+
mode=power
46+
lmax=2508
47+
feedback=0
48+
do_reionization=F
49+
kmin=1e-5
50+
kmax=20.0
51+
nk=200
52+
zmax=5.0
53+
zmax_background=5.0
54+
nz_background=500
55+
halofit_version=mead2020_feedback
56+
nonlinear=pk
57+
neutrino_hierarchy=normal
58+
kmax_extrapolate = 500.0
59+
60+
[load_nz_fits]
61+
file = %(COSMOSIS_DIR)s/number_density/load_nz_fits/load_nz_fits.py
62+
nz_file =%(FITS_FILE)s
63+
data_sets = SOURCE
64+
65+
[photoz_bias]
66+
file = %(COSMOSIS_DIR)s/number_density/photoz_bias/photoz_bias.py
67+
mode = additive
68+
sample = nz_source
69+
bias_section = nofz_shifts
70+
interpolation = cubic
71+
output_deltaz_section_name = delta_z_out
72+
73+
[linear_alignment]
74+
file = %(COSMOSIS_DIR)s/intrinsic_alignments/la_model/linear_alignments_interface.py
75+
method = bk_corrected
76+
77+
[projection]
78+
file = %(COSMOSIS_DIR)s/structure/projection/project_2d.py
79+
ell_min_logspaced = 1.0
80+
ell_max_logspaced = 25000.0
81+
n_ell_logspaced = 400
82+
shear-shear = source-source
83+
shear-intrinsic = source-source
84+
intrinsic-intrinsic = source-source
85+
get_kernel_peaks = F
86+
verbose = F
87+
88+
[add_intrinsic]
89+
file = %(COSMOSIS_DIR)s/shear/add_intrinsic/add_intrinsic.py
90+
shear-shear=T
91+
position-shear=F
92+
perbin=F
93+
94+
[2pt_shear]
95+
file = %(COSMOSIS_DIR)s/shear/cl_to_xi_nicaea/nicaea_interface.so
96+
corr_type = 0 ; shear_cl -> shear_xi
97+
98+
[shear_m_bias]
99+
file = shear/shear_bias/shear_m_bias.py
100+
m_per_bin = True
101+
; Despite the parameter name, this can operate on xi as well as C_ell.
102+
cl_section = shear_xi_plus shear_xi_minus
103+
verbose = F
104+
105+
[add_xi_sys]
106+
file = %(COSMOSIS_DIR)s/shear/xi_sys/xi_sys_psf.py
107+
data_file=%(FITS_FILE)s
108+
rho_stats_name=RHO_STATS
109+
110+
[tau_from_rho]
111+
file = %(COSMOSIS_DIR)s/shear/xi_sys/tau_from_rho.py
112+
data_file=%(FITS_FILE)s
113+
114+
[2pt_like]
115+
add_xi_sys=T
116+
data_sets=XI_PLUS XI_MINUS TAU_0_PLUS TAU_2_PLUS
117+
file = %(COSMOSIS_DIR)s/likelihood/2pt/2pt_like_xi_sys.py
118+
data_file=%(FITS_FILE)s
119+
gaussian_covariance=F
120+
covmat_name=COVMAT
121+
cut_zeros=F
122+
like_name=2pt_like
123+
angle_range_XI_PLUS_1_1= 3.0 150.0
124+
angle_range_XI_MINUS_1_1= 10.0 200.0

‎cosmo_inference/cosmosis_config/cosmosis_pipeline_SP_v1.4.5_leak_corr_A_minsep=1_maxsep=250_nbins=20_npatch=1.ini‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
#parameters used elsewhere in this file
2+
[DEFAULT]
3+
FITS_FILE = data/SP_v1.4.5_leak_corr_A_minsep=1_maxsep=250_nbins=20_npatch=1/cosmosis_SP_v1.4.5_leak_corr_A_minsep=1_maxsep=250_nbins=20_npatch=1.fits
4+
SCRATCH = /n09data/guerrini/output_chains/SP_v1.4.5_leak_corr_A_minsep=1_maxsep=250_nbins=20_npatch=1
5+
COSMOSIS_DIR = /home/guerrini/cosmosis-standard-library
6+
7+
8+
[pipeline]
9+
priors = cosmosis_config/priors_psf.ini
10+
values = cosmosis_config/values_psf.ini
11+
modules = consistency sample_S8 camb load_nz_fits photoz_bias linear_alignment projection add_intrinsic 2pt_shear add_xi_sys tau_from_rho 2pt_like
12+
likelihoods = 2pt_like
13+
extra_output = cosmological_parameters/omega_lambda cosmological_parameters/S_8 cosmological_parameters/sigma_8 cosmological_parameters/omega_m
14+
timing = T
15+
debug = T
16+
17+
[runtime]
18+
sampler = polychord
19+
verbosity = debug
20+
21+
[test]
22+
save_dir = %(SCRATCH)s/test_new_pipeline
23+
24+
[polychord]
25+
polychord_outfile_root = SP_v1.4.5_leak_corr_A_minsep=1_maxsep=250_nbins=20_npatch=1
26+
live_points = 192
27+
feedback = 3
28+
resume = T
29+
base_dir = %(SCRATCH)s/polychord
30+
31+
[output]
32+
filename = %(SCRATCH)s/SP_v1.4.5_leak_corr_A_minsep=1_maxsep=250_nbins=20_npatch=1/samples_SP_v1.4.5_leak_corr_A_minsep=1_maxsep=250_nbins=20_npatch=1.txt
33+
format = text
34+
lock = F
35+
36+
[consistency]
37+
file = %(COSMOSIS_DIR)s/utility/consistency/consistency_interface.py
38+
verbose = F
39+
40+
[sample_S8]
41+
file = %(COSMOSIS_DIR)s/utility/sample_sigma8/sample_S8.py
42+
43+
[camb]
44+
file = %(COSMOSIS_DIR)s/boltzmann/camb/camb_interface.py
45+
mode=power
46+
lmax=2508
47+
feedback=0
48+
do_reionization=F
49+
kmin=1e-5
50+
kmax=20.0
51+
nk=200
52+
zmax=5.0
53+
zmax_background=5.0
54+
nz_background=500
55+
halofit_version=mead2020_feedback
56+
nonlinear=pk
57+
neutrino_hierarchy=normal
58+
kmax_extrapolate = 500.0
59+
60+
[load_nz_fits]
61+
file = %(COSMOSIS_DIR)s/number_density/load_nz_fits/load_nz_fits.py
62+
nz_file =%(FITS_FILE)s
63+
data_sets = SOURCE
64+
65+
[photoz_bias]
66+
file = %(COSMOSIS_DIR)s/number_density/photoz_bias/photoz_bias.py
67+
mode = additive
68+
sample = nz_source
69+
bias_section = nofz_shifts
70+
interpolation = cubic
71+
output_deltaz_section_name = delta_z_out
72+
73+
[linear_alignment]
74+
file = %(COSMOSIS_DIR)s/intrinsic_alignments/la_model/linear_alignments_interface.py
75+
method = bk_corrected
76+
77+
[projection]
78+
file = %(COSMOSIS_DIR)s/structure/projection/project_2d.py
79+
ell_min_logspaced = 1.0
80+
ell_max_logspaced = 25000.0
81+
n_ell_logspaced = 400
82+
shear-shear = source-source
83+
shear-intrinsic = source-source
84+
intrinsic-intrinsic = source-source
85+
get_kernel_peaks = F
86+
verbose = F
87+
88+
[add_intrinsic]
89+
file = %(COSMOSIS_DIR)s/shear/add_intrinsic/add_intrinsic.py
90+
shear-shear=T
91+
position-shear=F
92+
perbin=F
93+
94+
[2pt_shear]
95+
file = %(COSMOSIS_DIR)s/shear/cl_to_xi_nicaea/nicaea_interface.so
96+
corr_type = 0 ; shear_cl -> shear_xi
97+
98+
[shear_m_bias]
99+
file = shear/shear_bias/shear_m_bias.py
100+
m_per_bin = True
101+
; Despite the parameter name, this can operate on xi as well as C_ell.
102+
cl_section = shear_xi_plus shear_xi_minus
103+
verbose = F
104+
105+
[add_xi_sys]
106+
file = %(COSMOSIS_DIR)s/shear/xi_sys/xi_sys_psf.py
107+
data_file=%(FITS_FILE)s
108+
rho_stats_name=RHO_STATS
109+
110+
[tau_from_rho]
111+
file = %(COSMOSIS_DIR)s/shear/xi_sys/tau_from_rho.py
112+
data_file=%(FITS_FILE)s
113+
114+
[2pt_like]
115+
add_xi_sys=T
116+
data_sets=XI_PLUS XI_MINUS TAU_0_PLUS TAU_2_PLUS
117+
file = %(COSMOSIS_DIR)s/likelihood/2pt/2pt_like_xi_sys.py
118+
data_file=%(FITS_FILE)s
119+
gaussian_covariance=F
120+
covmat_name=COVMAT
121+
cut_zeros=F
122+
like_name=2pt_like
123+
angle_range_XI_PLUS_1_1= 3.0 150.0
124+
angle_range_XI_MINUS_1_1= 10.0 200.0

0 commit comments

Comments
 (0)