How We Optimized lakeFS Mount for Deep Learning

I am happy to share that lakeFS Mount is now available. lakeFS Mount allows for mounting a repository (or a specific path within one) as a local filesystem. 

A filesystem mount is the ability to present a local device or a remote location as a local directory. It is a basic feature provided by all operating systems and is widely used by system admins and developers.

Mounting an object store location as a POSIX directory isn’t a novel concept (see S3 Mountpoint, gcs-fuse and blobfuse). These tools provide an abstraction layer on top of the object store that allows it to appear as a directory on the machine. Reading and writing objects behaves exactly like reading and writing files from a local drive. There are several reasons why this is beneficial:

However, while these benefits are real, existing object storage mount solutions often fall short when it comes to performance and consistency, especially when used in the context of machine learning and deep learning environments. In this article we will review the reasons existing object storage mount solutions fall short, and why lakeFS, as a data version control system, is best positioned to provide a performant and consistent object storage mount. 

Leave a Comment