Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 19 additions & 7 deletions docs/Cluster_basics/s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
- [How to access](#how-to-access)
- [Terminal from IRB cluster](#terminal-from-irb-cluster)
- [Open OnDemand](#open-ondemand)
- [Mount bucket on local](#mount-bucket-on-local)
- [Mount bucket on local](#mount-bucket-on-local-terminal-on-local)
- [How to use it](#how-to-use-it)
- [Seqera and Nextflow](#seqera-and-nextflow)
- [Python](#python)
Expand Down Expand Up @@ -50,8 +50,15 @@ files (e.g., metadata, images).~~

MinIO web access to S3 is now only available to users on paid plans. It is no longer accessible to IRB members.

From now on, access to S3 is only possible via the terminal (e.g., to navigate, run Python scripts, or execute Nextflow
pipelines outside Seqera). To use terminal access, you need to generate S3 credentials.
From now on, access to S3 **with write permissions is only possible** via the terminal (e.g., to navigate, run
Python scripts, or execute Nextflow pipelines outside Seqera). To use terminal access, you need to generate S3
credentials.

If you want to navigate S3 buckets — whether to check the file structure or read PDFs, Excel files, or text
files — you can do so using [Open OnDemand](#open-ondemand) or by
[mounting them](#mount-bucket-on-local-terminal-on-local).
You can create the credentials directly from Open OnDemand or with the local tool. Check the corresponding
section for details.

If you only want to use S3 files in your Nextflow pipelines through [seqera platform](https://cloud.seqera.io/), no
further setup is needed. Credentials are already configured in the pipelines; you only need to export them.
Expand Down Expand Up @@ -119,7 +126,8 @@ $ ls /home/mgrau/s3/bbg-scratch/
work
# Unmount
$ fusermount -u /home/mgrau/s3/bbg-scratch
[1]+ Done rclone mount irb-minio:bbg-scratch /home/mgrau/s3/bbg-scratch --vfs-cache-mode off --read-only
[1]+ Done rclone mount irb-minio:bbg-scratch /home/mgrau/s3/bbg-scratch
--vfs-cache-mode off --read-only
$ ls /home/mgrau/s3/bbg-scratch/
$
```
Expand All @@ -143,17 +151,21 @@ download files, but you cannot modify existing objects or upload new ones from t
1. Go to
[Open OnDemand Dashboard](https://ood.sc.irbbarcelona.org/pun/sys/dashboard/)
and log in with your LDAP credentials.
2. Click on "Home Directory System installed App".

2. Verify that your credentials are valid, or create new ones if they have expired.
![s3 open onDemand credentials](../assets/images/s3-ooo-credentials.png)

3. Once the credentials are valid, click on the "Home Directory System" section.

![Open OnDemand Home Directory System app screenshot](../assets/images/ooo-home-directory.png)

3. Select the S3 storage on the left panel and navigate through the folders.
4. Select the S3 storage on the left panel and navigate through the folders.
You can read PDFs and text files directly in the browser.
To view Excel or HTML files, you need to download them first.

![Open OnDemand S3 storage browser screenshot](../assets/images/ooo-s3.png)

### Mount bucket on local
### Mount bucket on local. Terminal on local

It is also possible to mount the S3 storage as a local partition (as we do with MacGyver and the workspace/).
This is a read-only option.
Expand Down
Binary file added docs/assets/images/s3-ooo-credentials.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading