Authentication

This document describes how to setup server and client authentication for a EngFlow Remote Execution cluster.

Server Authentication

The only supported mechanism for server authentication is TLS. In order to use it, you have to set the --tls_certificate and --tls_key flags. You can either use an existing certificate authority, or you can use self-signed certificates.

For Bazel users: by default, Bazel trusts the set of root certificates that is shipped with the JDK. If you are using a non-standard certificate authority, you have to configure Bazel to accept its certificate using Bazel’s --tls_certificate flag.

Beware that Bazel only allows specifying one custom CA certificate. If your --remote_executor, --remote_cache, and --bes_backend endpoints are different, their TLS certificates must all be signed by this authority, or by a publicly trusted authority.

Client Authentication (gRPC)

The EngFlow Remote Execution Service supports several mechanisms to authenticate gRPC clients and authorize requests.

To configure authentication, use the --client_auth flag: none, basic, mtls, gcp_email, gcp_rbe, and external.

To configure authorization, check out the details of the specific authentication mechanisms below.

Which --client_auth to use?

If you already setup GCP RBE access roles, and every user has a Google account, then we recommend using the gcp_rbe mechanism, which allows continued use of your existing IAM setup. Alternatively you can use gcp_email too. Both methods rely on Google authenticating the clients, so schedulers need to be able to talk to Google’s servers.

If you want to use a custom authentication service, use external. Please contact us for details; you’ll need to implement a binary with a specific gRPC interface.

If you already have a TLS certificate authority and distribute client certificates to your clients, and you use a client that supports mTLS (e.g. Bazel 3.1 or newer), then we recommend using the mtls mechanism. This mechanism works even without internet access, because the client CA’s certificate is deployed to schedulers.

Otherwise you will have to decide whether to setup a certificate authority (which may require setting up additional infrastructure, such as HashiCorp Vault or smallstep), use a VPN (none) or use username-password pairs (basic). Like mtls, these mechanism don’t rely on external identity providers, so they work on clusters that have no internet access.

None

Usage: --client_auth=none

If you disable client authentication, anyone who can initiate a network connection to the cluster can use it. This must only be used in combination with network-based usage restrictions, e.g., over a VPN.

Authorization: The --principal_based_permissions flag controls permissions. With --client_auth=none, the principal name is always empty, so the only meaningful setting is --principal_based_permissions=*->ROLE (where ROLE is for example user or admin.)

Version support:

• v1.54 and older: The default value of --principal_based_permissions is ["*->user"], therefore every client has the user permissions (to read and write to the CAS and read and execute actions, but not directly write actions).
• v1.55 and newer: The default value of --principal_based_permissions is []: nobody can do anything unless you specify --principal_based_permissions values.

Deny

Usage: --client_auth=deny

Denies all gRPC requests.

Apart from testing, this could be used to cut off gRPC access to a cluster without shutting it down. The cluster could still serve HTTP requests with the right --http_auth flag.

Authorization: With --client_auth=deny, no principals can be authorized.

Basic

Usage: --client_auth=basic

Basic authentication uses user name / password pairs. These are transmitted as clear text over a potentially encrypted connection (if server-side TLS is enabled); however, this may still be susceptible to man-in-the-middle attacks, and we do not recommend basic authentication.

Bazel supports basic authentication as of version 4.0.0 using a .netrc file, but only for remote execution and caching, not for the Build Event Service.

To enable basic authentication: create a htpasswd file with the user-password pairs, copy it to every scheduler instance, then add its path to the config file with --basic_auth_htpasswd .

The password entries must be in apr1 format (Apache MD5). See https://httpd.apache.org/docs/2.4/misc/password_encryptions.html for details.

Authorization: The --principal_based_permissions flag controls permissions. With --client_auth=basic, the principal name is the user name in the Basic Auth credentials. An example setting:

--principal_based_permissions=alice->admin
--principal_based_permissions+=bob->user
--principal_based_permissions+=*->cache-reader
--principal_based_permissions+=chuck->none

Version support:

• v1.54 and older: The default value of --principal_based_permissions is ["*->user"], therefore every authenticated client has the user permissions (to read and write to the CAS and read and execute actions, but not directly write actions).
• v1.55 and newer: The default value of --principal_based_permissions is []: nobody can do anything unless you specify --principal_based_permissions values.

Example:

Generating a htpasswd file:

$ htpasswd -cm /etc/engflow/htpasswd alice

The generated entry will look like:

alice:$apr1$ILfUslW9$cXOjd6w4WoZrFKfD3Xhe91

By default, all authenticated users have the “user” role. You can specify more detailed permissions with --principal_based_permissions .

Example (/etc/engflow/config snippet):

--basic_auth_htpasswd=/etc/engflow/htpasswd
--principal_based_permissions=alice->admin
--principal_based_permissions+=bob->cache-reader

(Note the = and += operators: --principal_based_permissions is a list-type flag; = overrides the default flag value.)

If you are using Bazel, create a ~/.netrc file with an entry for the remote execution service address.

Example (~/.netrc):

machine demo.engflow.com
login alice
password foo

To use an alternative netrc file, set its path in the NETRC environment variable before running Bazel.

mTLS

Usage: --client_auth=mtls

mTLS or mutual TLS authentication requires each client to present a signed client TLS certificate whenever it establishes a connection to the cluster. Use the --tls_trusted_certificate flag to configure the certificate authority that is trusted to authenticate clients.

Remember to configure fine-grained permissions with --principal_based_permissions flag. The principal is the Common Name field in the client TLS certificate.

If you are using Bazel, you can configure it using the --tls_client_certificate and --tls_client_key flags.

Authorization: The --principal_based_permissions flag controls permissions. With --client_auth=mtls, the principal name is the Common Name (CN) of the client certificate. The CN can be any string supported by X.509 certificates, such as plain user names, email addresses, or FQDNs. If it’s an email address then you can use the *@example.com syntax to mean everyone under that email domain. Otherwise you must use the exact principal name, or * to mean “everyone”.

--principal_based_permissions=alice->admin
--principal_based_permissions+=bob@example.com->user
--principal_based_permissions+=*@example.com->cache-reader
--principal_based_permissions+=chuck@example.com->none

Version support:

• v1.54 and older: The default value of --principal_based_permissions is ["*->user"], therefore every authenticated client has the user permissions (to read and write to the CAS and read and execute actions, but not directly write actions).
• v1.55 and newer: The default value of --principal_based_permissions is []: nobody can do anything unless you specify --principal_based_permissions values.

GCP Email

Usage: --client_auth=gcp_email

GCP email-based authentication uses GCP OAuth 2.0 bearer tokens to prove ownership of an email address to the cluster. Remember to configure per-client permissions using the --principal_based_permissions flag.

If you are using Bazel, you can configure it using either --google_default_credentials or --google_credentials, and you also have to set --google_auth_scopes=email.

Authorization: The --principal_based_permissions flag controls permissions. With --client_auth=gcp_email, the principal name is the email address of the client. You can use exact principal names, or *@example.com to mean everyone under that email domain, or * to mean “everyone”.

--principal_based_permissions=alice@example.com->admin
--principal_based_permissions+=bob@example.com->user
--principal_based_permissions+=*@example.com->cache-reader
--principal_based_permissions+=chuck@example.com->none

Version support:

• v1.54 and older: The default value of --principal_based_permissions is ["*->user"], therefore every authenticated client has the user permissions (to read and write to the CAS and read and execute actions, but not directly write actions).
• v1.55 and newer: The default value of --principal_based_permissions is []: nobody can do anything unless you specify --principal_based_permissions values.

GCP RBE

Usage: --client_auth=gcp_rbe

GCP RBE-based authentication also uses GCP OAuth 2.0 bearer tokens. However, instead of relying on verified email addresses, it queries GCP’s IAM for the Google-defined Remote Build Executor permissions, and unlike other --client_auth methods it therefore ignores --principal_based_permissions .

In order to use this authentication mechanism, you must specify a GCP project using the --gcp_rbe_auth_project flag.

Authorization: Unlike other --client_auth mechanisms, the --principal_based_permissions flag is ignored with --client_auth=gcp_rbe. You have to specify permissions in GCP IAM, as described below.

The Google GCP permissions and roles are documented here: https://cloud.google.com/iam/docs/understanding-roles#other-roles

The mapping between GCP permissions and EngFlow permissions is incomplete: the EngFlow Remote Execution service supports only a subset of the RBE permissions and roles, and GCP defines only a subset of the permissions that EngFlow supports.

GCP RBE Permissions:

• remotebuildexecution.actions.create to run an action remotely
• remotebuildexecution.actions.delete to delete an action cache entry (this is not used by Bazel)
• remotebuildexecution.actions.get to read an action cache entry
• remotebuildexecution.actions.write to write an action cache entry from a client (remotely executed actions always have write access to the action cache)
• remotebuildexecution.blobs.create to write an entry to the CAS
• remotebuildexecution.blobs.get to read an entry from the CAS or query whether an entry is in the CAS based on its digest

These are the relevant roles, though note that you can create custom roles for different subsets of permissions:

• Remote Build Execution Artifact Creator aka roles/remotebuildexecution.artifactCreator

  Can run actions remotely. This is the most commonly used role, and maps to the user role defined in EngFlow RE.

• Remote Build Execution Artifact Admin aka roles/remotebuildexecution.artifactAdmin

  Can run actions remotely, and also delete actions. Cannot write actions to the cache.

• Remote Build Execution Action Cache Writer aka roles/remotebuildexecution.actionCacheWriter

  Can write CAS and action cache entries. This is primarily useful when using the system as a pure cache without remote execution. In this use case, the CI system should be allowed to read and write the cache (requires both this role and the Remote Build Execution Artifact Viewer), while individual engineers are only allowed to read the cache.

• Remote Build Execution Artifact Viewer aka roles/remotebuildexecution.artifactViewer

  Can read CAS and action cache entries. This is primarily useful when using the system as a pure cache without remote execution.

Build Event Streams with --client_auth=gcp_rbe

As of 2021-09-29, GCP IAM has no permissions to control Build Event Stream uploads that we could map to EngFlow Remote Execution permissions.

When using --client_auth=gcp_rbe, EngFlow RE versions support Build Event Stream uploads the following way:

v1.52 and older: don’t let clients upload Build Event Streams.

v1.53 and newer: allow clients with remotebuildexecution.blobs.create permission to also upload Build Event Streams.

External

Usage: --client_auth=external

Since v1.53 you can use an external service to authenticate and authorize gRPC client requests.

This must be a service running on localhost next to the scheduler, communicating over gRPC. Please contact us for details.

Authorization: Unlike other --client_auth mechanisms, the --principal_based_permissions flag is ignored with --client_auth=external. The external service is responsible for authorization too.

Client Authentication (HTTP)

The EngFlow Remote Execution service can be configured to store build events and serve a build results UI; see --enable_bes .

When enabled, you can visit the Remote Execution address in a browser and view build results. Client request authentication can be configured with the --http_auth flag: none, deny, basic, and google_login.

None

Usage: --http_auth=none

If you disable client authentication, anyone who can initiate a network connection to the cluster can use it. This must only be used in combination with network-based usage restrictions, e.g., over a VPN.

In order to change the default permissions, use the --principal_based_permissions flag, for example --principal_based_permissions=*->admin. As of v1.53.0, only admin can view the UI.

Authorization: same as --client_auth=none.

Deny

Usage: --http_auth=deny

Denies all HTTP requests.

This is useful when the Build Event Service is disabled on the cluster, or when you want to cut off HTTP access to a cluster without shutting it down. The cluster could still serve gRPC requests with the right --client_auth flag.

Authorization: same as --client_auth=deny.

Basic

Usage: --http_auth=basic

Same as --client_auth=basic but for HTTP requests, using the same --basic_auth_htpasswd file.

Authorization: same as --client_auth=basic.

Google login

Usage: --http_auth=google_login

Authenticates clients through a Google sign-in page. You need to create an OAuth 2.0 Client ID in GCP, then configure --principal_based_permissions for allowed clients.

Authorization: The --principal_based_permissions flag controls permissions. With --http_auth=google_login, the principal name is the email address of the client. You can use exact principal names, or *@example.com to mean everyone under that email domain, or * to mean “everyone”.

To view the UI, principals must have admin permission.

Version support:

• v1.54 and older: The default value of --principal_based_permissions is ["*->user"], therefore every client has the user permissions. However nobody can view the UI unless you specify --principal_based_permissions values with admin permission.
• v1.55 and newer: The default value of --principal_based_permissions is []: nobody can view the UI unless you specify --principal_based_permissions values with admin permission.

Setup:

1. Configure the OAuth Consent Screen in GCP.

   You can use the same GCP project as the one hosting the Remote Execution cluster, or use a different one.

   If all clients are in the same organization (e.g. everyone has an @your-company.com Google account), then you can set User Type to Internal.

   To support clients from multiple email domains (e.g. *@your-company.com, *@partner-company.com), you need to set User Type to External.

   Do not upload a Logo, and don’t configure any App Domains nor any Scopes. If you do, the app must be verified by Google. Once you upload a Logo it’s not possible to take it down, nor to cancel the verification process). So just leave every non-mandatory field blank.

   Once you’ve completed the steps, click on Publish. This will immediately change the status to In Production, without needing verification.

2. Create an OAuth 2.0 Client ID in GCP

   For Application Type, select Web application.

   For Authorized JavaScript origins, set https://YOUR-CLUSTER.engflow.com:443 or whatever is the cluster’s HTTP endpoint.

   For Authorized redirect URIs, set https://YOUR-CLUSTER.engflow.com:443/google-redirect (or whatever’s the endpoint).

   Once you’re done, note down the Client ID. You don’t need the Client secret.

3. Set the Client ID, and allowed principals in the EngFlow config.

   Example:

   --http_auth=google_login
   --google_client_id=123456789-ab12cd34.apps.googleusercontent.com
   --principal_based_permissions=*@example.com->admin
   --principal_based_permissions+=*@example-partner.com->admin
   

   (As of v1.53.0, only admin can view the UI.)

4. Optional: limit the expiry of client tokens.

   After clients sign in with their Google accounts and get authorized, the EngFlow server sets a cookie with a JSON Web Token (JWT) used for authentication.

   You can set the expiry of the JWTs using --experimental_web_login_expiration .