Credentials are secret values that you can use to securely inject sensitive information into the Grid platform. Credentials are encrypted in the Grid-backend storage systems and are only decrypted immediately prior to use.
Access to Secrets (Teams Users)
Credentials can be created by any Grid user, with a limit of 50 Credentials per account. Only the user who created a credential (the owner) can delete that credential. All of the user's team members are able to utilize the credential in their own workflow (for example, while creating a Datastore from a private s3 bucket).
Sensitive credential type values can never be retrieved from Grid via the CLI. However,
the values of non-sensitive credential types (such as
--type s3) can be viewed by the user
who created the credential and any member of the user's team (for our
teams tier users.)
At the moment, the following credential types are supported:
s3(access for datastore creation from a private s3 bucket)
How to Create an
s3 Credential for Private S3 Access (BYOC users only)
Grid now supports the ability to create Datastores from private AWS s3 buckets by using
--no-copy mode via the CLI. In order to allow Grid to access your private buckets,
you'll need to create an authorized AWS Role using the
grid credential create --type s3
command (explained in detail below). After creating a role, you can run the
grid datastore create S3://<private-bucket-name-here> --no-copy command as usual - no
modifications needed. If any of your registered
s3 credentials can access the s3 bucket
path specified, then Grid will automatically use them when creating the Datastore (and
when using that Datastore in a
Prerequisite: Configure IAM Role in AWS
In order to provide Grid access to a private S3 bucket, you must first set up an AWS IAM Role configured with the appropriate permission policy. This is the absolute minimum permission Grid requires in order to list and retrieve files from an s3 bucket of your choice. The
grid credential create command will dynamically generate the IAM Role Trust Policy based on the cluster on which you are creating the Datastore. While all users will need to authorize the AWS account ID on which the Grid Platform controlplane runs, bring-your-own-cloud (
BYOC) users will also need to include the AWS account ID on which the
BYOC cluster runs.
Please refer to the AWS documentation on IAM Role Trust Policies and IAM Role Permission Policies for more detailed information.
We will illustrate the process of registering an IAM Role with Grid using the following example:
I want to run an experiment or session using sensitive data contained in a private s3 bucket. The bucket's name is
gridai-demo-bucketand I would like to create a Datastore from any file prefix (folder) in the bucket.
0. Change from deault cluster context
By default the cluster context is set to
Grid Cloud. Change this to your BYOC cluster you created.
grid user set-cluster-context <byoc cluster name>
1. Generate Trust and Permission Policiess
First: run the
grid credential create --type s3 command in order to find the required trust and
$ grid credential create --type s3
Please refer the the the documentation for how to create an AWS role and
The trust policy for the role should be:
The permission policy attached to the role should be:
Please be sure to change the <replace-with-bucket-name> field with the bucket
name you wish to grant access to. More information can be found on the docs.
Please Note: when creating the role name in the AWS console, the role name MUST
begin with the prefix: grid-s3-access- any valid characters can follow the
When complete, please enter the role ARN:
2. Add Trust and Permission Policies to AWS
grid credential create command will output the required trust and permission policies.
a. Naviagte to AWS WebUI -> IAM -> Role
b. Create Role -> Custom trust policy
c. Paste the output of the
trust policy from the CLI into the
json editor and click Next
d. In the
Add permissions page click Next
e. The role name must begin with
grid-s3-access- and click on Create role
f. You will be redirected to IAM -> Roles. Select your just created
g. Click on Add Permissions -> Create Inline Policy. Paste the output
permission policy from the CLI output after replacing your bucket name and click on Review Policy
h. Enter an appropriate name, for example: my-permission-policy-for-grid
4. Input role ARN in Grid
Within Roles -> Search for the role you created that began with
grid-s3-access- and click on it
Click on the icon next to the "role ARN" in order to copy the value to my clipboard:
I now go back to the terminal and paste my clipboard value into the input field I am being presented with:
When complete, please enter the role ARN: arn:aws:iam::880213070634:role/grid-s3-access-for-my-demo-bucket
The role has been successfully added, and I can see it in the list when I run
grid credentials list.
I can now run the regular command to create a Datastore and it will automatically pick the correct credential and create the Datastore from my private s3 bucket!
$ grid datastore create s3://gridai-demo-bucket/ --no-copy