Using XetHub with containers
Running ML training on containers is the modern standard for ensuring reproducible runtime environments and results. A challenge, however, is the ability to efficiently access large data from containers without having download it all. Our mount feature is perfect for addressing this issue, lazily fetching files as needed to avoid spending expensive compute time on downloads, but making a mount system call in containerized environments like Docker and Kubernetes usually requires CAP_SYS_ADMIN capability — which provides far too much admin access to the container.
To get around this issue, we have created plugins to enable remote mounting via two alternate means:
By installing these plugins, users no longer need CAP_SYS_ADMIN to easily mount XetHub repositories to their container. Follow the doc links above to try it for yourself.