Upload & download files with ada

[onnozweers]

Currently, we ask users to do uploads & downloads with Rclone, and use ada for everything else.

We can enable Ada to upload and download files, using curl.

Advantages:

• Users won't need Rclone. Rclone is absolutely great, but it is a different tool, and some users may not like it.
• Failing upload when data is corrupt. Ada could send the (MD5) checksum with an uploaded file, so that uploads are verified automatically. Corrupt files will not be accepted by dCache and will be cleaned up. Example use case: we have a project that does not have delete permissions because the users are afraid to accidentally delete data. Sometimes they receive uploads from external data providers. When an upload has failed and a corrupt file is left behind, they need to ask us admins to delete the file for them. In their case, sending files with their checksum will make sure that any file that is created will be correct. An additional benefit may be that the file in dCache will have an MD5 checksum. This checksum can then be listed with ada --checksum.
• It may be helpful for troubleshooting. In some cases, Rclone may run into problems, and then Ada may provide valuable information (like: token has expired).
• It would provide an extra example / reference of how to interact with dCache.

Challenges:

• This would need a WebDAV door and not the API. Getting a WebDAV door from the API is described in Ada: get API URL from a WebDAV door and vice versa #14.
• Some dCaches have not configured a redirect from API to WebDAV door. What to do in those cases?
• It would be best to not show the checksum on the curl command line, because that would allow other users on the same system to read file checksums with the ps command. Best would be to provide the checksum in a curl header file.

The dCache User Guide describes how to send the checksum with a file for integrity check. https://dcache.org/manuals/UserGuide-10.2/webdav.shtml#checksums (go to "Failing upload if data is corrupt")

[hailihu]

References:

• https://www.qed42.com/insights/using-curl-commands-with-webdav
• https://github.com/WebDAVDevs/webdav-request-samples/blob/master/webdav_curl.md

[onnozweers]

About "failing upload when data is corrupt": here's an example:

curl -H "Authorization: Bearer $BEARER_TOKEN" \
     -H "Content-MD5: $(md5sum $file | cut -d' ' -f1 | xxd -r -p | base64)" \
     -X PUT https://webdav.grid.surfsara.nl:2882/pnfs/grid.sara.nl/data/users/onno/disk/testfile-md5 \
     -L --post302 \
     -T $file \
     -v

Currently there seems to be an issue with it. When using a redirecting WebDAV door (where dCache redirects the client to the pool to do the file upload), a mismatch in the checksum will leave a (corrupt) file behind. When using a non-redirecting WebDAV door, a corrupt file will be cleaned up. I assume it's a bug and I have reported it. dCache/dcache#7862

[onnozweers]

Just an thought: set a trap for Ctrl+C to show debug information that the user can send to the service desk. Some ideas:

• The Ada command, ps -o args= $$ or $0 "$@"
• The curl command that the user interrupted
• The --whoami information
• The hostname
• The bash version
• The curl version
• The jq version
• The authentication method
• The decoded token (if used)
• The voms-proxy-info --all info, (if using an X509 proxy)
• The system's external IP addresses
• The available disk space (for a download)

The same info might be interesting for when a transfer fails.

[onnozweers]

Just a thought: for download of a file that is nearline and not online, we can add a warning that it may take a long time because the file needs to be staged first.

If there is also token authentication, we could check whether the token allows staging. If it doesn't, reading the file returns a 403 (forbidden) which has proven to be very confusing.

curl -H 'accept: application/json' --silent -X GET 'https://dcachetest.grid.surfsara.nl:20443/api/v1/namespace/%2Fusers%2Fonno%2Ftape%2Ftest-badger14-2?locality=true' \
| jq -r .fileLocality
NEARLINE

[onnozweers]

Another thought:

dCache WebDAV doors usually redirect clients to a pool for the actual file transfer. Some doors are configured to redirect to HTTPS on the pool, but some doors are configured to redirect to HTTP on the pool. Then there is no transport layer security, and a man in the middle could read the file that is being transferred.

Some dCache users have data that is private, such as genomic or other medical data. Ada should not blindly switch from HTTPS to HTTP; only if the user specificies this is allowed. If Ada could protect users from accidentally exposing their data, it would be a great feature!

Curl has flags to refuse a transfer over HTTP: curl --proto =https or a "downgrade" to HTTP: curl --proto-redir =https. The first one should be sufficient.

A small test:

[onno@mylaptop ~/git/Ada-upload-download] % curl --netrc --location --proto '=https' https://webdav.grid.surfsara.nl/pnfs/grid.sara.nl/data/users/onno/README.txt         
curl: (1) Protocol "http" disabled (in redirect)

I had to quote the '=https' to prevent the error "zsh: https not found". Bash may not need this.

[onnozweers]

I will add an optional argument --allow-insecure-redirects. With this, the --proto =https will not be used and users can still upload and download (over HTTP).