Create from a Session
Upload datasets from within a session into Datastore

Upload via Interactive Session

For huge datasets that need processing or a lot of manual work, we recommend this flow:
    launch an Interactive Session
    download the data
    process it
    upload

Screen

When you are in the interactive Session, use Screen to make sure you don't lose your progress.
1
# start screen (lets you close the tab without killing the process)
2
screen -S some_name
Copied!
now do whatever processing you need:
1
# download, etc...
2
curl http://a_dataset
3
unzip a_dataset
4
5
# process
6
do_something
7
something_else
8
bash process.sh
9
...
Copied!
when you're done, upload to Grid via the CLI (on the Interactive Session):
1
grid datastore create --source imagenet_folder --name imagenet
Copied!
Grid CLI is auto-installed on sessions and logged in under your credentials.

ImageNet Example

To see it in action, check out this tutorial
Last modified 1mo ago