Create from a cluster

Upload from a cluster

Use these instructions to upload from:
    Corporate cluster.
    Academic cluster.
First, start screen on the jump node (to run jobs in the background).
1
screen -S upload
Copied!
If your jump node allows a memory-intensive process, then skip this step. Otherwise, request an interactive machine. Here's an example using SLURM.
1
srun --qos=batch --mem-per-cpu=10000 --ntasks=4 --time=12:00:00 --pty bash
Copied!
Once the job starts, install and log into Grid (get your username and ssh keys from the settings page).
1
# install
2
pip install lightning-grid --upgrade
3
4
# login
5
grid login --username YOUR_USERNAME --key YOUR_KEY
Copied!
Next, use the Datastores command to upload any folder:
1
grid datastores create --source imagenet_folder --name imagenet
Copied!
You can safely close your ssh connection to the cluster (the screen will keep things running in the background).
Last modified 6mo ago
Copy link