Creating Runs from Config File

note

The following examples assume you have already installed and setup Grid. If you haven't, please visit the Getting Started page.

Using YAML

In addition to CLI parameters, Grid supports the use of YML files for passing configuration properties to a Grid Run. This eliminates the need to specify a long list of parameters at the command-line during frequent Run iterations. If compute parameters do not change often, we suggest you create a Grid config file and use that instead of specifying parameters at the command-line. Full yaml specification is available here.

Creating Runs With a Config File

You can create a Run with a config file instead of passing CLI arguments. For example:

grid run --config my_config.yml script.py

Config Files with Actions

Config files also allow the use of a special feature called Actions. Grid Actions give you the flexibility of adding steps to your training workflow without having to change your training script.

Example uses of Actions:

• Installing libraries like Apex
• Downloading data
• Processing something
• ...

Available Actions

on_build_start

These are executed as your Run's image is built. We cache these commands on every git SHA. This is useful for installing dependencies and other system setup tasks.

compute:
  train:

    actions:
      on_build_start:
        - apt-get install wget -y
        - pip install tqdm

on_build_end

These are executed as at the end of building Run's image.

compute:
  train:

    actions:
      on_build_end:
        - apt update

Commands automatically run as sudo

on_exeperiment_start

Arbitrary commands that run just before your training process starts. This is useful for downloading and preparing data, etc.

compute:
  train:

    actions:
      on_experiment_start:
        - bash download_dataset.sh

on_experiment_end

Runs after your script stops. This is useful for post-processing data, sending alerts and notifications to your systems, etc.

compute:
  train:

    actions:
      on_experiment_end:
        - apt-get install curl -y
        - curl -X GET http://webhook.com?success=true

Configuring Actions

You can configure Grid Actions by using a Grid config file (see details on Grid YML).

Here's a full example of a Grid configuration using actions:

compute:
  provider:
    credentials: cc-wv4l9
    region: us-east-1
    vendor: aws

  train:
    cpus: 1
    gpus: 0
    instance: t2.xlarge

    # Actions need to be passed as one command
    # per line.
    actions:
      on_build_start:
        - apt-get install wget -y
        - pip install tqdm

      on_experiment_start:
        - bash download_dataset.sh

      on_experiment_end:
        - apt-get install curl -y
        - curl -X GET http://webhook.com?success=true

As you can see, you can pass one command per line. You can pass as many commands as you'd like.

Environment variable substitution

Grid allows environment variable substitution for on_experiment_start and on_experiment_end actions. All declared environment variables for the run are available in the substitution (as well as some Grid predefined variables). Example config:

compute:

  train:
    cpus: 1
    gpus: 0
    instance: t2.medium
    framework: lightning

    environment:
      WEBHOOK_URL: https://hooks.example.com/grid

    # Actions need to be passed as one command
    # per line.
    actions:
      on_build_start:
        - apt-get update
        - apt-get install curl -y
      on_experiment_start:
        - bash before.sh

      on_experiment_end:
        - |
            curl -X POST -d '{"name": "${GRID_EXPERIMENT_ID}", "instance_type": "${GRID_INSTANCE_TYPE}", "status": "status", "step": "after"}' ${WEBHOOK_URL}

Default environment variables

Grid sets several environment variables by default for all experiments:

• GRID_EXPERIMENT_ID - experiment's ID
• GRID_EXPERIMENT_NAME - experiment name
• GRID_USER_ID - ID of the user who created the experiment
• GRID_CLUSTER_ID - ID of the cluster where experiment is running
• GRID_INSTANCE_TYPE - machine type (t2.medium, g4dn.xlarge, etc.)

String operations

Grid provides partial emulation for bash string operations. This can be used to manipulate string values prior to substitution.

• Example variable substitution with substring:

on_experiment_end:
- |
    echo ${GRID_EXPERIMENT_ID:0:4} # Getting first 4 symbols

Grid emulates the below string operations:

${parameter^}
${parameter^^}
${parameter,}
${parameter,,}
${parameter:position}
${parameter:position:length}
${parameter#substring}
${parameter##substring}
${parameter%substring}
${parameter%%substring}
${parameter/substring/replacement}
${parameter//substring/replacement}
${parameter/#substring/replacement}
${parameter/%substring/replacement}
${#parameter}
${parameter=default}
${parameter:=default}
${parameter:-default}

note

If you have additional questions about Runs, visit the FAQ. The section is periodically updated this with common questions from the Grid community.