Conversation
meta-cla
bot
added
the
CLA Signed
|
@cpuhrsch are you not wanting to put it into monarch? |
|
|
||
|
|
||
| ``` | ||
| remotemount(host_mesh, sourcepath, mntpoint=sourcepath) |
There was a problem hiding this comment.
would this be an API on host_mesh itself?
ie hm.remote_mount(sourcepath, mntpoint=sourcepath)?
|
@colin2328 - I'm not against that in principle, but I thought this could very well just be application on top of Monarch. For now I don't see a reason to merge it into the main package. I also think we'd still want to do a bit more work before we do so. Maybe we can keep it a separate package for now and then merge into core monarch later if needed? cc @zdevito on this as well. |
I agree. I think in general we should avoid building non-core things into Monarch, but instead encourage an ecosystem of components. |
Add support for RDMA transfers (ibverbs and EFA), symlink handling, configurable chunk sizes, and MAST/SLURM backend options. Replace Python-based file packing with C extension using mmap and parallel reads.
3 participants
This is just a straight transfer out of a separate, smaller repository into a top-level monarch_remotemount directory. Follow up work includes integrating it more directly with the monarch build system to be used as a module, if this is desired.
For now I'm including this to share the code more broadly.
See the readme or internal post for more detail.
EDIT: This also still needs a few more changes to Slurm to support forwarding a path to libfuse2 in case it doesn't exist on the remote hosts.