Skip to content

Commit 0bddaf6

Browse files
MarcelRosierMarcelRosier
authored
Merge pull request #61 from BrainLesion/60-update-panoptica-tutorials
60 update panoptica tutorials
2 parents 31b2546 + db33c67 commit 0bddaf6

File tree

7 files changed

+436
-411
lines changed

7 files changed

+436
-411
lines changed

‎panoptica/README.md‎

Lines changed: 35 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -1,61 +1,59 @@
11
# Panoptica Tutorials
22

3-
This folder contains several Jupyter notebooks to showcase different possible use cases of the [panoptica package](https://github.com/BrainLesion/panoptica).
4-
The package allows to compute instance-wise segmentation quality metrics for 2D and 3D semantic- and instance segmentation maps by providing 3 core modules:
3+
This folder contains several Jupyter notebooks to showcase different possible use cases of the [panoptica package](https://github.com/BrainLesion/panoptica).
4+
The package allows to compute instance-wise segmentation quality metrics for 2D and 3D semantic- and instance segmentation maps.
55

6-
**1. Instance Approximator:** instance approximation algorithms in panoptic segmentation evaluation. Available now: connected components algorithm.
6+
## Notebooks
7+
### Semantic: Spine example
78

8-
**2. Instance Matcher:** instance matching algorithm in panoptic segmentation evaluation, to align and compare predicted instances with reference instances.
9-
10-
**3. Instance Evaluator:** Evaluation of panoptic segmentation performance by evaluating matched instance pairs and calculating various metrics like true positives, Dice score, IoU, and ASSD for each instance.
9+
<img src="https://github.com/BrainLesion/panoptica/blob/main/examples/figures/semantic.png?raw=true" alt="semantic_figure" height="300"/>
1110

12-
![workflow_figure](https://github.com/BrainLesion/panoptica/blob/main/examples/figures/workflow.png)
11+
`example_spine_semantic.ipynb`
1312

14-
## Use Cases
13+
[![nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.jupyter.org/github/BrainLesion/tutorials/blob/main/panoptica/example_spine_semantic.ipynb)
14+
<a target="_blank" href="https://colab.research.google.com/github/BrainLesion/tutorials/blob/main/panoptica/example_spine_semantic.ipynb">
15+
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
16+
</a>
1517

16-
### Semantic Segmentation Input
18+
### Matched Instance: Spine example
1719

18-
<img src="https://github.com/BrainLesion/panoptica/blob/main/examples/figures/semantic.png?raw=true" alt="semantic_figure" height="300"/>
20+
<img src="https://github.com/BrainLesion/panoptica/blob/main/examples/figures/matched_instance.png?raw=true" alt="matched_instance_figure" height="300"/>
1921

20-
Although for many biomedical segmentation problems, an instance-wise evaluation is highly relevant and desirable, they are still addressed as semantic segmentation problems due to lack of appropriate instance labels.
2122

22-
**Modules [1-3]** can be used to obtain panoptic metrics of matched instances based on a semantic segmentation input.
23+
`example_spine_matched_instance.ipynb`
2324

24-
[Jupyter Notebook Example](https://github.com/BrainLesion/tutorials/tree/main/panoptica/example_spine_semantic.ipynb)
25+
[![nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.jupyter.org/github/BrainLesion/tutorials/blob/main/panoptica/example_spine_matched_instance.ipynb)
26+
<a target="_blank" href="https://colab.research.google.com/github/BrainLesion/tutorials/blob/main/panoptica/example_spine_matched_instance.ipynb">
27+
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
28+
</a>
2529

26-
### Unmatched Instances Input
30+
### Unmatched Instance: Spine example
2731

2832
<img src="https://github.com/BrainLesion/panoptica/blob/main/examples/figures/unmatched_instance.png?raw=true" alt="unmatched_instance_figure" height="300"/>
2933

30-
It is a common issue that instance segementation outputs have good segmentations with mismatched labels.
34+
`example_spine_unmatched_instance.ipynb`
3135

32-
For this case **modules [2-3]** can be utilized to match the instances and report panoptic metrics.
36+
[![nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.jupyter.org/github/BrainLesion/tutorials/blob/main/panoptica/example_spine_unmatched_instance.ipynb)
37+
<a target="_blank" href="https://colab.research.google.com/github/BrainLesion/tutorials/blob/main/panoptica/example_spine_unmatched_instance.ipynb">
38+
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
39+
</a>
3340

34-
[Jupyter Notebook Example](https://github.com/BrainLesion/tutorials/tree/main/panoptica/example_spine_unmatched_instance.ipynb)
35-
36-
### Matched Instances Input
37-
38-
<img src="https://github.com/BrainLesion/panoptica/blob/main/examples/figures/matched_instance.png?raw=true" alt="matched_instance_figure" height="300"/>
3941

40-
Ideally the input data already provides matched instances.
4142

42-
In this case **module 3** can be used to directly report panoptic metrics without requiring any internal preprocessing.
43+
### Matching: Spine example
44+
`example_spine_matching_algorithm.ipynb`
4345

44-
[Jupyter Notebook Example](https://github.com/BrainLesion/tutorials/tree/main/panoptica/example_spine_matched_instance.ipynb)
46+
[![nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.jupyter.org/github/BrainLesion/tutorials/blob/main/panoptica/example_spine_matching_algorithm.ipynb)
47+
<a target="_blank" href="https://colab.research.google.com/github/BrainLesion/tutorials/blob/main/panoptica/example_spine_matching_algorithm.ipynb">
48+
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
49+
</a>
4550

46-
## Citation
4751

48-
If you use panoptica in your research, please cite it to support the development!
52+
### Save/Load Configs example:
53+
`example_config.ipynb`
4954

50-
Kofler, F., Möller, H., Buchner, J. A., de la Rosa, E., Ezhov, I., Rosier, M., Mekki, I., Shit, S., Negwer, M., Al-Maskari, R., Ertürk, A., Vinayahalingam, S., Isensee, F., Pati, S., Rueckert, D., Kirschke, J. S., Ehrlich, S. K., Reinke, A., Menze, B., Wiestler, B., & Piraud, M. (2023). _Panoptica -- instance-wise evaluation of 3D semantic and instance segmentation maps._ [arXiv preprint arXiv:2312.02608](https://arxiv.org/abs/2312.02608).
55+
[![nbviewer](https://raw.githubusercontent.com/jupyter/design/master/logos/Badges/nbviewer_badge.svg)](https://nbviewer.jupyter.org/github/BrainLesion/tutorials/blob/main/panoptica/example_config.ipynb)
56+
<a target="_blank" href="https://colab.research.google.com/github/BrainLesion/tutorials/blob/main/panoptica/example_config.ipynb">
57+
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
58+
</a>
5159

52-
```
53-
@misc{kofler2023panoptica,
54-
title={Panoptica -- instance-wise evaluation of 3D semantic and instance segmentation maps},
55-
author={Florian Kofler and Hendrik Möller and Josef A. Buchner and Ezequiel de la Rosa and Ivan Ezhov and Marcel Rosier and Isra Mekki and Suprosanna Shit and Moritz Negwer and Rami Al-Maskari and Ali Ertürk and Shankeeth Vinayahalingam and Fabian Isensee and Sarthak Pati and Daniel Rueckert and Jan S. Kirschke and Stefan K. Ehrlich and Annika Reinke and Bjoern Menze and Benedikt Wiestler and Marie Piraud},
56-
year={2023},
57-
eprint={2312.02608},
58-
archivePrefix={arXiv},
59-
primaryClass={cs.CV}
60-
}
61-
```

‎panoptica/example_config.ipynb‎

Lines changed: 83 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -16,39 +16,69 @@
1616
},
1717
{
1818
"cell_type": "code",
19-
"execution_count": 9,
19+
"execution_count": 10,
20+
"metadata": {},
21+
"outputs": [],
22+
"source": [
23+
"!pip install panoptica auxiliary rich numpy > /dev/null"
24+
]
25+
},
26+
{
27+
"cell_type": "markdown",
2028
"metadata": {},
21-
"outputs": [
22-
{
23-
"name": "stdout",
24-
"output_type": "stream",
25-
"text": [
26-
"Requirement already satisfied: panoptica in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from -r requirements.txt (line 1)) (1.0.0.post2.dev0+2f7d01f)\n",
27-
"Requirement already satisfied: auxiliary in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from -r requirements.txt (line 2)) (0.0.42)\n",
28-
"Requirement already satisfied: rich in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from -r requirements.txt (line 3)) (13.6.0)\n",
29-
"Requirement already satisfied: numpy in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from -r requirements.txt (line 4)) (1.25.2)\n",
30-
"Requirement already satisfied: connected-components-3d<4.0.0,>=3.12.3 in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from panoptica->-r requirements.txt (line 1)) (3.12.3)\n",
31-
"Requirement already satisfied: ruamel.yaml<0.19.0,>=0.18.6 in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from panoptica->-r requirements.txt (line 1)) (0.18.6)\n",
32-
"Requirement already satisfied: scikit-image<0.23.0,>=0.22.0 in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from panoptica->-r requirements.txt (line 1)) (0.22.0)\n",
33-
"Requirement already satisfied: scipy<2.0.0,>=1.7.0 in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from panoptica->-r requirements.txt (line 1)) (1.11.2)\n",
34-
"Requirement already satisfied: nibabel>=3.0 in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from auxiliary->-r requirements.txt (line 2)) (5.1.0)\n",
35-
"Requirement already satisfied: path>=16.10.0 in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from auxiliary->-r requirements.txt (line 2)) (17.0.0)\n",
36-
"Requirement already satisfied: pathlib>=1.0 in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from auxiliary->-r requirements.txt (line 2)) (1.0.1)\n",
37-
"Requirement already satisfied: pillow>=10.0.0 in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from auxiliary->-r requirements.txt (line 2)) (10.0.0)\n",
38-
"Requirement already satisfied: tifffile>=2023.8.25 in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from auxiliary->-r requirements.txt (line 2)) (2023.8.30)\n",
39-
"Requirement already satisfied: markdown-it-py>=2.2.0 in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from rich->-r requirements.txt (line 3)) (3.0.0)\n",
40-
"Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from rich->-r requirements.txt (line 3)) (2.17.2)\n",
41-
"Requirement already satisfied: mdurl~=0.1 in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from markdown-it-py>=2.2.0->rich->-r requirements.txt (line 3)) (0.1.2)\n",
42-
"Requirement already satisfied: packaging>=17 in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from nibabel>=3.0->auxiliary->-r requirements.txt (line 2)) (23.1)\n",
43-
"Requirement already satisfied: ruamel.yaml.clib>=0.2.7 in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from ruamel.yaml<0.19.0,>=0.18.6->panoptica->-r requirements.txt (line 1)) (0.2.8)\n",
44-
"Requirement already satisfied: networkx>=2.8 in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from scikit-image<0.23.0,>=0.22.0->panoptica->-r requirements.txt (line 1)) (3.1)\n",
45-
"Requirement already satisfied: imageio>=2.27 in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from scikit-image<0.23.0,>=0.22.0->panoptica->-r requirements.txt (line 1)) (2.31.3)\n",
46-
"Requirement already satisfied: lazy_loader>=0.3 in /opt/anaconda3/envs/seg11panoptdev/lib/python3.11/site-packages (from scikit-image<0.23.0,>=0.22.0->panoptica->-r requirements.txt (line 1)) (0.3)\n"
47-
]
48-
}
49-
],
5029
"source": [
51-
"!pip install -r requirements.txt"
30+
"If you installed the packages and requirements on your own machine, you can skip this section and start from the import section.\n",
31+
"\n",
32+
"### Setup Colab environment (optional) \n",
33+
"Otherwise you can follow and execute the tutorial on your browser.\n",
34+
"In order to start working on the notebook, click on the following button, this will open this page in the Colab environment and you will be able to execute the code on your own (*Google account required*).\n",
35+
"\n",
36+
"<a target=\"_blank\" href=\"https://colab.research.google.com/github/BrainLesion/tutorials/blob/main/panoptica/example_config.ipynb\">\n",
37+
" <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
38+
"</a>\n",
39+
"\n",
40+
"Now that you are visualizing the notebook in Colab, run the next cell to install the packages we will use. There are few things you should follow in order to properly set the notebook up:\n",
41+
"1. Warning: This notebook was not authored by Google. Click on 'Run anyway'.\n",
42+
"1. When the installation commands are done, there might be \"Restart runtime\" button at the end of the output. Please, click it.\n",
43+
"If you run the next cell in a Google Colab environment, it will **clone the 'tutorials' repository** in your google drive. This will create a **new folder** called \"tutorials\" in **your Google Drive**.\n",
44+
"All generated file will be created/uploaded to your Google Drive respectively.\n",
45+
"\n",
46+
"After the first execution of the next cell, you might receive some warnings and notifications, please follow these instructions:\n",
47+
" - 'Permit this notebook to access your Google Drive files?' Click on 'Yes', and select your account.\n",
48+
" - Google Drive for desktop wants to access your Google Account. Click on 'Allow'.\n",
49+
"\n",
50+
"Afterwards the \"tutorials\" folder has been created. You can navigate it through the lefthand panel in Colab. You might also have received an email that informs you about the access on your Google Drive."
51+
]
52+
},
53+
{
54+
"cell_type": "code",
55+
"execution_count": 11,
56+
"metadata": {},
57+
"outputs": [],
58+
"source": [
59+
"import sys\n",
60+
"\n",
61+
"# Check if we are in google colab currently\n",
62+
"try:\n",
63+
" import google.colab\n",
64+
"\n",
65+
" colabFlag = True\n",
66+
"except ImportError as r:\n",
67+
" colabFlag = False\n",
68+
"\n",
69+
"# Execute certain steps only if we are in a colab environment\n",
70+
"if colabFlag:\n",
71+
" # Create a folder in your Google Drive\n",
72+
" from google.colab import drive\n",
73+
"\n",
74+
" drive.mount(\"/content/drive\")\n",
75+
" # clone repository and set path\n",
76+
" !git clone https://github.com/BrainLesion/tutorials.git /content/drive/MyDrive/tutorials\n",
77+
" BASE_PATH = \"/content/drive/MyDrive/tutorials/panoptica\"\n",
78+
" sys.path.insert(0, BASE_PATH)\n",
79+
"\n",
80+
"else: # normal jupyter notebook environment\n",
81+
" BASE_PATH = \".\" # current working directory would be BraTs-Toolkit anyways if you are not in colab"
5282
]
5383
},
5484
{
@@ -60,7 +90,7 @@
6090
},
6191
{
6292
"cell_type": "code",
63-
"execution_count": 10,
93+
"execution_count": 12,
6494
"metadata": {},
6595
"outputs": [],
6696
"source": [
@@ -90,7 +120,7 @@
90120
},
91121
{
92122
"cell_type": "code",
93-
"execution_count": 11,
123+
"execution_count": 13,
94124
"metadata": {},
95125
"outputs": [
96126
{
@@ -102,14 +132,14 @@
102132
" 106, 107, 108, 202, 203, 204, 205, 206, 207, 208], dtype=uint8))"
103133
]
104134
},
105-
"execution_count": 11,
135+
"execution_count": 13,
106136
"metadata": {},
107137
"output_type": "execute_result"
108138
}
109139
],
110140
"source": [
111-
"ref_masks = read_nifti(\"./spine_seg/matched_instance/ref.nii.gz\")\n",
112-
"pred_masks = read_nifti(\"./spine_seg/matched_instance/pred.nii.gz\")\n",
141+
"ref_masks = read_nifti(f\"{BASE_PATH}/spine_seg/matched_instance/ref.nii.gz\")\n",
142+
"pred_masks = read_nifti(f\"{BASE_PATH}/spine_seg/matched_instance/pred.nii.gz\")\n",
113143
"\n",
114144
"# labels are matching\n",
115145
"np.unique(ref_masks), np.unique(pred_masks)"
@@ -124,7 +154,7 @@
124154
},
125155
{
126156
"cell_type": "code",
127-
"execution_count": 12,
157+
"execution_count": 14,
128158
"metadata": {},
129159
"outputs": [
130160
{
@@ -156,7 +186,7 @@
156186
},
157187
{
158188
"cell_type": "code",
159-
"execution_count": 13,
189+
"execution_count": 15,
160190
"metadata": {},
161191
"outputs": [
162192
{
@@ -202,14 +232,14 @@
202232
},
203233
{
204234
"cell_type": "code",
205-
"execution_count": 14,
235+
"execution_count": 16,
206236
"metadata": {},
207237
"outputs": [
208238
{
209239
"name": "stdout",
210240
"output_type": "stream",
211241
"text": [
212-
"Saved config into /DATA/NAS/ongoing_projects/hendrik/panoptica/repo/panoptica/test_config.yaml\n"
242+
"Saved config into /home/marcelrosier/miniconda3/envs/tutorials/lib/python3.10/site-packages/panoptica/test_config.yaml\n"
213243
]
214244
}
215245
],
@@ -238,14 +268,14 @@
238268
},
239269
{
240270
"cell_type": "code",
241-
"execution_count": 15,
271+
"execution_count": 17,
242272
"metadata": {},
243273
"outputs": [
244274
{
245275
"name": "stdout",
246276
"output_type": "stream",
247277
"text": [
248-
"<panoptica.panoptica_evaluator.Panoptica_Evaluator object at 0x7f05ac7b80d0>\n"
278+
"<panoptica.panoptica_evaluator.Panoptica_Evaluator object at 0x7002f42ccd60>\n"
249279
]
250280
}
251281
],
@@ -256,7 +286,7 @@
256286
},
257287
{
258288
"cell_type": "code",
259-
"execution_count": 16,
289+
"execution_count": 18,
260290
"metadata": {},
261291
"outputs": [
262292
{
@@ -267,9 +297,9 @@
267297
"-- Got MatchedInstancePair, will evaluate instances\n",
268298
"\n",
269299
"+++ MATCHING +++\n",
270-
"Number of instances in reference (num_ref_instances): 1\n",
271-
"Number of instances in prediction (num_pred_instances): 1\n",
272-
"True Positives (tp): 1\n",
300+
"Number of instances in reference (num_ref_instances): 22\n",
301+
"Number of instances in prediction (num_pred_instances): 22\n",
302+
"True Positives (tp): 22\n",
273303
"False Positives (fp): 0\n",
274304
"False Negatives (fn): 0\n",
275305
"Recognition Quality / F1-Score (rq): 1.0\n",
@@ -278,12 +308,12 @@
278308
"Global Binary Dice (global_bin_dsc): 0.9744370224078394\n",
279309
"\n",
280310
"+++ INSTANCE +++\n",
281-
"Segmentation Quality IoU (sq): 0.9501484001456879 +- 0.0\n",
282-
"Panoptic Quality IoU (pq): 0.9501484001456879\n",
283-
"Segmentation Quality Dsc (sq_dsc): 0.9744370224078394 +- 0.0\n",
284-
"Panoptic Quality Dsc (pq_dsc): 0.9744370224078394\n",
285-
"Segmentation Quality ASSD (sq_assd): 0.1499152780072207 +- 0.0\n",
286-
"Segmentation Quality Relative Volume Difference (sq_rvd): -0.0006011763989268869 +- 0.0\n",
311+
"Segmentation Quality IoU (sq): 0.8328184295330796 +- 0.15186064004517466\n",
312+
"Panoptic Quality IoU (pq): 0.8328184295330796\n",
313+
"Segmentation Quality Dsc (sq_dsc): 0.900292616009954 +- 0.10253566174957332\n",
314+
"Panoptic Quality Dsc (pq_dsc): 0.900292616009954\n",
315+
"Segmentation Quality ASSD (sq_assd): 0.250331887879225 +- 0.07696680402317076\n",
316+
"Segmentation Quality Relative Volume Difference (sq_rvd): 0.0028133049062930553 +- 0.034518928495505724\n",
287317
"\n"
288318
]
289319
}
@@ -293,13 +323,6 @@
293323
"result, debug_data = evaluator.evaluate(pred_masks, ref_masks)[\"ungrouped\"]\n",
294324
"print(result) # yields same results as the evaluator object manually constructed"
295325
]
296-
},
297-
{
298-
"cell_type": "code",
299-
"execution_count": null,
300-
"metadata": {},
301-
"outputs": [],
302-
"source": []
303326
}
304327
],
305328
"metadata": {
@@ -318,7 +341,7 @@
318341
"name": "python",
319342
"nbconvert_exporter": "python",
320343
"pygments_lexer": "ipython3",
321-
"version": "3.11.6"
344+
"version": "3.10.14"
322345
}
323346
},
324347
"nbformat": 4,

0 commit comments

Comments
 (0)