[FR]: Allow for directories and to "honor" structure

[Mercanuis]

What is the current behavior?

My team was working on a similar rule to allow uploading to S3. The rule here works for what it describes (it uses cp where we use sync). When run with bazel run ... the rule takes the given label and starts to cp each file it fines per the list.

Describe the feature

The feature request is to keep the original underlying structure. Our team's use case is that we have directories with specific structures that we build out via Bazel to make a consistent build. As an example here's a setup for a Techdocs documentation site and the underlying tree that is generated by the tool with Bazel:

 bazel build //ns/vault:vault_docs
INFO: Invocation ID: 29b8ac8b-f6a9-432d-9209-00be609e3076
INFO: Analyzed target //ns/vault:vault_docs (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //ns/vault:vault_docs up-to-date:
  bazel-bin/ns/vault/vault_docs_site
INFO: Elapsed time: 0.717s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

 tree bazel-bin/ns/vault/vault_docs_site 
bazel-bin/ns/vault/vault_docs_site
├── 404.html
├── about
│   └── index.html
├── application_setup
│   └── index.html
├── assets
│   ├── images
│   │   └── favicon.png
│   ├── javascripts
│   │   ├── bundle.ad660dcc.min.js
│   │   ├── bundle.ad660dcc.min.js.map
│   │   ├── lunr
│   │   │   ├── min
│   │   │   │   ├── lunr.ar.min.js
│   │   │   │   ├── lunr.da.min.js
│   │   │   │   ├── lunr.de.min.js
│   │   │   │   ├── lunr.du.min.js
│   │   │   │   ├── lunr.el.min.js
│   │   │   │   ├── lunr.es.min.js
│   │   │   │   ├── lunr.fi.min.js
│   │   │   │   ├── lunr.fr.min.js
│   │   │   │   ├── lunr.he.min.js
│   │   │   │   ├── lunr.hi.min.js
│   │   │   │   ├── lunr.hu.min.js
│   │   │   │   ├── lunr.hy.min.js
│   │   │   │   ├── lunr.it.min.js
│   │   │   │   ├── lunr.ja.min.js
│   │   │   │   ├── lunr.jp.min.js
│   │   │   │   ├── lunr.kn.min.js
│   │   │   │   ├── lunr.ko.min.js
│   │   │   │   ├── lunr.multi.min.js
│   │   │   │   ├── lunr.nl.min.js
│   │   │   │   ├── lunr.no.min.js
│   │   │   │   ├── lunr.pt.min.js
│   │   │   │   ├── lunr.ro.min.js
│   │   │   │   ├── lunr.ru.min.js
│   │   │   │   ├── lunr.sa.min.js
│   │   │   │   ├── lunr.stemmer.support.min.js
│   │   │   │   ├── lunr.sv.min.js
│   │   │   │   ├── lunr.ta.min.js
│   │   │   │   ├── lunr.te.min.js
│   │   │   │   ├── lunr.th.min.js
│   │   │   │   ├── lunr.tr.min.js
│   │   │   │   ├── lunr.vi.min.js
│   │   │   │   └── lunr.zh.min.js
│   │   │   ├── tinyseg.js
│   │   │   └── wordcut.js
│   │   └── workers
│   │       ├── search.b8dbb3d2.min.js
│   │       └── search.b8dbb3d2.min.js.map
│   ├── stylesheets
│   │   ├── main.6543a935.min.css
│   │   ├── main.6543a935.min.css.map
│   │   ├── palette.06af60db.min.css
│   │   └── palette.06af60db.min.css.map
│   ├── vault-agent-overview.jpg
│   ├── vault-icon.png
│   └── vault-request.png
├── best_practices
│   └── index.html
├── index.html
├── kubernetes-setup
│   └── index.html
├── search
│   ├── lunr.js
│   ├── main.js
│   ├── search_index.json
│   └── worker.js
├── sitemap.xml
├── sitemap.xml.gz
├── techdocs_metadata.json
├── troubleshooting
│   └── index.html
├── vault-agent
│   └── index.html
└── vault_setup
    └── index.html

The current target doesn't honor the tree structure, and instead flattens it and adds every file to the 'root' or the target bucket

  bazel run //ns/vault:vault_upload      
INFO: Invocation ID: b25630d5-b978-4452-87f2-e77da5b72f49
INFO: Analyzed target //ns/vault:vault_upload (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //ns/vault:vault_upload up-to-date:
  bazel-bin/ns/vault/vault_upload/s3_sync.sh
INFO: Elapsed time: 0.917s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/ns/vault/vault_upload/s3_sync.sh
Copying the following artifacts to s3://mybucket:
ns/vault/vault_docs_site

Copying ns/vault/vault_docs_site/404.html to s3://mybucket/404.html
upload: ns/vault/vault_docs_site/404.html to s3://mybucket/404.html
Copying ns/vault/vault_docs_site/about/index.html to s3://mybucketindex.html
upload: ns/vault/vault_docs_site/about/index.html to s3://mybucket/index.html
Copying ns/vault/vault_docs_site/application_setup/index.html to s3://mybucket/index.html
upload: ns/vault/vault_docs_site/application_setup/index.html to s3://mybucket/index.html
Copying ns/vault/vault_docs_site/assets/images/favicon.png to s3://mybucket/favicon.png
upload: ns/vault/vault_docs_site/assets/images/favicon.png to s3://mybucket/favicon.png
Copying ns/vault/vault_docs_site/assets/javascripts/bundle.ad660dcc.min.js to s3://mybucket/bundle.ad660dcc.min.js
upload: ns/vault/vault_docs_site/assets/javascripts/bundle.ad660dcc.min.js to s3://mybucket/bundle.ad660dcc.min.js
...

This limits the use of the rule to single files, or a bunch of files that are supposed to be under one directory at a given time. I think the rule becomes a lot more useful if it can copy the entire folder structure on top of the files itself, as s3 lets you do this (for example our team's own first try works with both single files and directories):

#!/usr/bin/env bash

set -eux

# Set the SOURCE location
readonly SOURCE="{{source}}"

# Set the Destination for SOURCE
readonly BUCKET_NAME="{{bucket_name}}"
readonly DESTINATION="{{destination}}"

# If the SOURCE is a directory we'll sync to avoid potential issues with copies
# A single file artifact we don't care as much so we'll stick to copying
if [ -d "$SOURCE" ]; then
  aws s3 sync "${SOURCE}" "${BUCKET_NAME}/${DESTINATION}"
else
  aws s3 cp "${SOURCE}" "${BUCKET_NAME}/${DESTINATION}"
fi

[Mercanuis]

I guess as a follow up question: why does the script want to always flatten directories [code]?

        # Always flatten directories
        for f in "${artifact}"/*; do
            cp_artifact "${f}" "${bucket}"
        done