Go to CrowdAI
All Collections
Technical Vault
Configure GCP to sync media to your account

Configure GCP to sync media to your account

Get set up to sync media from a location on Google Cloud Platform (GCP)
Paige Kassalen avatar
Written by Paige Kassalen. Updated over a week ago

If your media are stored in a location you can access on GCP Cloud Storage, you can link that cloud location to your CrowdAI account and sync media to use in the platform.

Please note: this is our "digest" version of how to manage access for your GCP account. For the most up-to-date information on how to set up and manage access controls for GCP, please visit the GCP documentation center.

Before you start

You will need:

  • A GCP account. Open console.cloud.google.com and either log in or create an account.

  • A GCP project.

  • One or more GCP Storage buckets containing data that you wish to upload to CrowdAI. You can browse these buckets via the top-level hamburger menu.

  • Adequate permissions/roles.

    • If you created the GCP account, you will likely have the “Owner” role, which provides more than enough permissions.

    • The “Editor” role also contains sufficient permissions.

Set up a CrowdAI service account via the GCP console

1. Create a CrowdAI role

A “role” is a collection of permissions. Here, we define a role that provides the permissions necessary for use by the CrowdAI platform (and nothing further).

This step only needs to be completed once per GCP project.

  1. Navigate to the roles page: Click on the top-left hamburger -> IAM & Admin -> Roles

  2. Click the + Create Role button in the top bar.

  3. Fill out the top part of the Create Role form

    1. A name like crowdai-credentials will help you to remember the context for this role.

    2. “Alpha” is fine for now.

  4. Click the + Add Permissions button

  5. In the permissions modal, select the following read-only permissions:

    1. storage.objects.list

    2. storage.objects.get

    3. Storage.buckets.list (candidate for removal)

    4. storage.buckets.get (candidate for removal)

  6. Click the Create button.

You should be redirected to the list of all roles. Our newly created crowdai-credentials role can be seen at the top:

2. Create a service account

We now need to create a service account, which is a sort of “machine user" that can access your GCP storage.

  1. Navigate to the service accounts page: Click on the top-left hamburger -> IAM & Admin -> Service accounts

  2. Click the + Create a new Service Account button in the top bar of the page.

  3. Fill in the fields. Click Create and Continue when ready.

  4. We need to assign the crowdai-credentials role from the previous step. Additionally, we will likely want to add “constraints.”

    1. The crowdai-credentials role can be found under the “custom” sub-menu. Select it.

    2. Now click the Add condition link directly next to the role selection.

  5. [Optional] Add a condition on the service key. Right now, our service key will be able to access all objects in all buckets of the GCP project. Oftentimes, though, you only want to expose one or a few buckets. This is where conditions come into play.

    1. Add a title and description.

    2. For copy+paste purposes, it will be easiest to use the “Condition Editor” tab.

    3. To expose only select buckets, use the following pattern:

      1. resource.name.startsWith("projects/_/buckets/crowdai-test-2") or resource.name.startsWith("projects/_/buckets/crowdai-test")

      2. The above example can be read as, “only objects from buckets crowdai-test’ OR crowdai-test-2 may be accessed.”

      3. We could further constrain the selection by adding a path. For example, projects/_/buckets/crowdai-test-2/inner only exposes files within the “inner” directory of bucket crowdai-test-2.

  6. Step 3 (“Grant users access to this service account”) can be skipped for now.

  7. Click the blue Done button at the bottom of the form.

You should be redirected to the service accounts list page. The newly created service account should appear in the list:

3. Create service key

Service keys contain the actual access information that we need to pass to the CrowdAI platform. Each service account may have several service keys attached to it. Moreover, these service keys may expire. These features are helpful in case a key is compromised: simply delete the compromised key and create a new one.

  1. Find the service account in the list of service accounts. Click the vertical dots in the far-right Action column. Select Manage keys.

  2. We are now in the “Keys” tab, which contains a list of keys.

  3. To create a new key, select Add key -> Create new key

  4. A Create key modal will appear. Select JSON (the default). Click Create.

  5. You will be prompted to download a JSON file. This file contains the “keys to the kingdom” (or, at least, part of it). So be careful:

    • If you intend to upload the JSON file to CrowdAI immediately, it would actually be preferable to “open” rather than “save” it. This will ensure that the information is deleted whenever you close your browser.

    • If you save the JSON file (e.g. because your browser automatically saves files), you should delete the file immediately after adding it to your CrowdAI account.

    • You should use this key to generate or update one set of credentials and nothing further. If, for example, you want to use GCP developer tools, then you should create a different service key.

With the JSON file in hand, you're ready to use it to create a set of Credentials in your CrowdAI account. Here's our article on how to do that.

Did this answer your question?