Filesystems#
Filesystems, also known as persistent storage, allow you to store your large datasets and the state of your instance, for example:
- Packages installed system-wide using
apt-get
. - Python packages installed using
pip
. - conda and Python venv virtual environments.
Lambda GPU Cloud filesystems have a capacity of 8 exabytes, or 8,000,000 terabytes, and you can have a total of 24 filesystems, except for filesystems created in the Texas, USA (us-south-1) region. The capacity of filesystems created in the Texas, USA (us-south-1) region is 10 terabytes.
How do I copy files to and from my filesystems?#
As of April 2025, Lambda has begun rolling out the Filesystem S3 Adapter to
select regions. This feature allows you to use S3-compatible tools like
rclone
and s5cmd
to copy files to and from your filesystems and to perform
common file management operations. For more information, see
Filesystem S3 Adapter.
If your filesystem's region doesn't support the Filesystem S3 Adapter yet, you
can use rsync
to copy files to and from Lambda instances and your computer,
as well as between instances in the same and different regions. To learn how to
use rsync
to copy files between filesystems, see
Importing and exporting data.
How are filesystems billed?#
Filesystems are billed per GB per month, in 1-hour increments. Billing continues for each filesystem as long as it exists, even if it's not attached to an instance.
For example, based on the price of $0.20 per GB used per month:
- If you use 1,000 GB of your filesystem capacity for an entire month (30 days, or 720 hours), you’ll be billed $200.00.
- If you use 1,000 GB of your filesystem capacity for a single day (24 hours), you’ll be billed $6.67.
Note
The actual price of persistent storage will be displayed when you create your filesystem.
Can filesystems be accessed without an instance?#
Accessing a filesystem from an On-Demand instance or 1-Click Cluster#
To access a filesystem from an On-Demand instance or 1-Click Cluster:
- Your filesystem must reside in the same region as the instance or cluster.
- You must attach the filesystem to your instance or cluster at the time that the instance or cluster is launched.
Note
Filesystems cannot currently be transferred between regions.
Accessing a filesystem remotely#
If your filesystem's region doesn't support the
Filesystem S3 Adapter
yet, Lambda recommends that you use rsync
to keep a local copy of the files
you have saved in your filesystems. For more information, see
Importing and exporting data.
Note
Filesystems can't be attached to running instances and can't be mounted remotely using NFS or similar protocols.
Can I manage my filesystems programmatically?#
You can use the Lambda Cloud API to manage filesystems programmatically. The API currently supports the following operations:
- Listing your existing filesystems
- Creating a new filesystem
- Deleting a filesystem
For details, see the Filesystems section in the Lambda Cloud API browser.
Can I set a limit (quota) on my filesystem usage?#
Currently, you can't set a limit (quota) on your persistent storage filesystem usage.
You can see the usage of a persistent storage filesystem from within an
instance by running df -h -BG
. This command will produce output similar to:
Filesystem 1G-blocks Used Available Use% Mounted on
udev 99G 0G 99G 0% /dev
tmpfs 20G 1G 20G 1% /run
/dev/vda1 1357G 23G 1335G 2% /
tmpfs 99G 0G 99G 0% /dev/shm
tmpfs 1G 0G 1G 0% /run/lock
tmpfs 99G 0G 99G 0% /sys/fs/cgroup
persistent-storage 8589934592G 0G 8589934592G 0% /home/ubuntu/persistent-storage
/dev/vda15 1G 1G 1G 6% /boot/efi
/dev/loop0 1G 1G 0G 100% /snap/core20/1822
/dev/loop1 1G 1G 0G 100% /snap/lxd/24061
/dev/loop2 1G 1G 0G 100% /snap/snapd/18357
tmpfs 20G 0G 20G 0% /run/user/1000
In the example output, above:
- The name of the filesystem is
persistent-storage
. - The size of the filesystem is
8589934592G
(8 exabytes). - The available capacity of the filesystem is
8589934592G
. - The used percentage of the filesystem is
0%
. - The filesystem is mounted on
/home/ubuntu/persistent-storage
.
Note
You can also use the Cloud API's /file-systems
endpoint to find out your
filesystem usage. For details, see
List filesystems
in the Cloud API browser.
Preserving the state of your system#
For saving the state of your system, including:
- Packages installed system-wide using
apt-get
- Python packages installed using
pip
- conda environments
We recommend creating containers using Docker or other software for creating containers.
You can also create a script that runs the commands needed to re-create your system state. For example:
Run the script each time you start an instance.
If you only need to preserve Python packages and not packages installed system-wide, you can create a Python virtual environment.
You can also create a conda environment.
Tip
For the highest performance when training, we recommend copying your dataset, containers, and virtual environments from persistent storage to your home directory. This can take some time but greatly increases the speed of training.