Actions
Grid Actions give you the flexibility of adding steps to your training workflow without having to change your training script.
Example uses of actions:
    Installing libraries like Apex
    Downloading data
    Processing something prior to running a training script

Available Actions

on_image_build

These are executed as your project image is built. We cache these commands on every git SHA. This is useful for installing dependencies and other system setup tasks.
1
compute:
2
train:
3
4
actions:
5
on_image_build:
6
- apt-get install wget -y
7
- pip install tqdm
Copied!
Commands automatically run as sudo

on_before_training_start

Arbitrary commands that run just before your training process starts. This is useful for downloading and preparing data, etc.
1
compute:
2
train:
3
4
actions:
5
on_before_training_start:
6
- bash download_dataset.sh
Copied!

on_after_training_end

Runs after your script stops. This is useful for post-processing data, sending alerts and notifications to your systems, etc.
1
compute:
2
train:
3
4
actions:
5
on_after_training_end:
6
- apt-get install curl -y
7
- curl -X GET http://webhook.com?success=true
Copied!

Configuring Actions

You can configure Grid Actions by using a Grid config file (see details on Grid YML).
The Grid YAML spec supports three actions:
    on_image_build commands passed to the image builder which are interpreted as RUN commands in a Dockerfile.
    on_before_training_start which allows users to specify commands that need to be executed sequentially before the main experiment process starts.
    on_after_training_end same as above, but executed after the main process ends.
Here's a full example of a Grid configuration using actions:
1
compute:
2
provider:
3
credentials: cc-wv4l9
4
region: us-east-1
5
vendor: aws
6
7
train:
8
cpus: 1
9
gpus: 0
10
instance: t2.xlarge
11
12
# Actions need to be passed as one command
13
# per line.
14
actions:
15
on_image_build:
16
- apt-get install wget -y
17
- pip install tqdm
18
19
on_before_training_start:
20
- bash download_dataset.sh
21
22
on_after_training_end:
23
- apt-get install curl -y
24
- curl -X GET http://webhook.com?success=true
Copied!
As you can see, you can pass one command per line. You can pass as many commands as you'd like.

Environment variable substitution

Grid allows environment variable substitution for on_before_training_start and on_before_training_start actions. All declared environment variables for the run are available in the substitution (as well as some Grid predefined variables). Example config:
1
compute:
2
3
train:
4
cpus: 1
5
gpus: 0
6
instance: t2.medium
7
framework: lightning
8
9
environment:
10
WEBHOOK_URL: https://hooks.example.com/grid
11
12
# Actions need to be passed as one command
13
# per line.
14
actions:
15
on_image_build:
16
- apt-get update
17
- apt-get install curl -y
18
on_before_training_start:
19
- bash before.sh
20
21
on_after_training_end:
22
- |
23
curl -X POST -d '{"name": "${GRID_EXPERIMENT_ID}", "instance_type": "${GRID_INSTANCE_TYPE}", "status": "status", "step": "after"}' ${WEBHOOK_URL}
Copied!

Default environment variables

Grid sets several environment variables by default for all experiments:
    GRID_EXPERIMENT_ID - experiment's ID
    GRID_EXPERIMENT_NAME - experiment name
    GRID_USER_ID - ID of the user who created the experiment
    GRID_CLUSTER_ID - ID of the cluster where experiment is running
    GRID_INSTANCE_TYPE - machine type (t2.medium, g4dn.xlarge, etc.)

String operations

Grid provides partial emulation for bash string operations. This can be used to manipulate string values prior to substitution.
    Example variable substitution with substring:
1
on_after_training_end:
2
- |
3
echo ${GRID_EXPERIMENT_ID:0:4} # Getting first 4 symbols
Copied!
Grid emulates the below string operations:
1
${parameter^}
2
${parameter^^}
3
${parameter,}
4
${parameter,,}
5
${parameter:position}
6
${parameter:position:length}
7
${parameter#substring}
8
${parameter##substring}
9
${parameter%substring}
10
${parameter%%substring}
11
${parameter/substring/replacement}
12
${parameter//substring/replacement}
13
${parameter/#substring/replacement}
14
${parameter/%substring/replacement}
15
${#parameter}
16
${parameter=default}
17
${parameter:=default}
18
${parameter:-default}
Copied!
Last modified 1mo ago