Vault usage in RKS Cloud Apps

This guide covers Vault Deployments on Rocket kubernetes Services (RKS) infrastructure. But is suitable for any standard Kubernetes deployment.

Vault Deployments

There are 3 vault deployments, one for each cluster

• Dev cluster → ct namespace
• Demo cluster → demo namespace
• Production cluster → prod namespace

Namespacing

Because there are a limitted number of vaults, multiple environments have to be able to share a vault without interferring with the other environments. If we had enterprise Vault we could use namespaces, but since we do not we implement our own adhoc namespacing. This section describes how that is done.

Secrets

Secrets in vault are heirarchical. We take advantage of this by having a «root» secret that is the same name as the environment (the kubernetes namespace). Consider ct as an example. The secret tree is as follows:

ct
|- encryption
|- attachments
|- tenants
|- ct_e2e
|- poma-encryption

Authentication

One can «mount» authentication endpoints on vault. Each «AuthMount» can provide a unique way to authenticate with Vault and can be tied to policies that restrict access to only specific secrets. Our «namespaces» with respect to Authentication has two parts.

• Policies. All policy names are of the form: /. For example: ct/encryption-attachments-read. Each policy will restrict access to certain secrets. The previous secret only allows read access to ct/data/encryption/attachments
• Authentication. All auth mounts endpoints has the form: / . For example: ct/kubernetes. This endpoint allows services in the ct namespace to authentication using the secret obtained from their kubernetes service account (see Kubernetes Authentication for details)

GCP / Kubernetes Authentication

At the moment using GCP IAM to authenticate with vault is broken. I have filed a bug report with the project: https://github.com/spring-projects/spring-vault/issues/463 but until this is sorted out we cannot use this authentication. As an alternative we are using Kubernetes Service Accounts to authenticate with Vault.

Kubernetes Authentication Concepts

• Service Account
  Each pod is associated with a service account.
  A single Service account can be associated with multiple pods
  A Service Account can be bound to multiple roles
  Each Service Account can have one or more secrets associated with it
  By default a pod is associated with the «default» service account
• Roles
  Declares what access a Service Account has
  For example, access to Kubernetes APIs
  Note: we don’t use these roles when accessing vault. But the vault service account does need a role that allows it to access the kubernetes auth API so that vault can verify that JWTs are valid.
• RoleBindings
  Represents the association between a Service Account and a Role
• Pod
  When a pod is started the a JWT token and a certificate and another file I have forgotten for the moment are automatically mounted on the filesystem of the pod at: /var/run/secrets/kubernetes.io/serviceaccount/
  If a pod wants to use internal kubernetes APIs (like listing pods in namespace) it can use Bearer authentication with the mounted JWT token and the token will be checked and depending on the service accounts roles, the system will allow to prevent access to the particular API requested.

Vault Authentication Concepts

• Authentication mechanism – There are multiple ways that vault can be configured to allow a service to authenticate with vault. The one we are considering in this document is the Kubernetes mechanism. This mechanism uses Kubernetes as a trusted source. When a Java Web Token is received it will query Kubernetes to verify if the token is valid.
• Auth Mounts/Endpoints – when configuring Vault, an authentication mechanism can be assigned an API endpoint (the process of mounting an authentication). In our vaults we mount the Kubernetes Auth mechanism on «/kuberenetes». To authenticate with this enpoint we would access the http://vault.ct/v1/auth/ct/kubernetes endpoint.
• Policies – Vault Policies restrict what access a caller has. The typical policy we use will restrict access to particular secrets and restrict the access to only read access (for example)
• Roles – each auth mount has configured roles. A role is an association to policies. Role for kubernetes auth define a «service account name» and «service account namespace» . When authenticating a service will indicate which role it wishes to use. Vault will check that the service’s service account has matching namespace and name. If it does then the role applies as the associated Policies are applied to the services further requests.

Consider the example of reading the permissions encryption secret from vault.

Vault

• • contains a secret ct/encryption/permissions
  • contains a policy ct/encryption-permissions-read
    • This allows read-only access to the path: ct/encryption/permissions
    • A service with this policy could only access that one secret and read it (not modify or delete it)
  • contains a authentication endpoint: ct/kubernetes
    • endpoint contains a role: encryption-permissions-read
      this role requires
      • service account name == permissions-serviceaccount
      • service account namespace == ct
    • if requirements are satisfied requestor can authenticate and obtain a token containing the policy ct/encryption-permissions-read

Example: Permissions Pod

Note: the actual paths are not correct. They are approximations to illustrate the concept.

• Has a file: /var/run/secrets/kubernetes.io/serviceaccount/token. This contains a JWT. The JWT indicates the service is associated with the service account permissions-serviceaccount
• Permissions will read file and make request: POST http://vault.ct/v1/auth/ct/kubernetes -H «Authentication: Bearer » -X
  • vault will accept request because requested role has criteria that matches.
  • vault will return a token with the policy information within the token
• Permissions will then perform a GET http://vault.ct/v1/ct/encryption/permissions -H «X-Vault-Token: <token from vault>
  • Vault will check the policy the user has and return the key

Spring Application Properties in Vault

We will begin to store application configuration in vault. Once the story RKSCLOUD-5513 – LEFTOVER: Improve Vault integration with our services CLOSED has been completed all RKS Cloud services will read configuration from vault, these values will have the highest priority. So if a database password is in vault, it will be used instead of the values present in the environment or in the application.properties.

The design has «common» application properties and environment specific properties. Priorities are 0 as lowest 4 for highest

• config/data/common/vault_config → contains application properties that can be read by all environments on the cluster and all services. For example the database password for the dev cluster can be put in this secret and all ci builds will load the password. Priority 0 (Lowest)
• config/data/common/ → contains application properties that can be read by all environments on the cluster but restricted to a specific service. Priority 1
• config/data/common/ → contains application properties that can be read only by any service but restricted to a specific namespace/environment. Priority 2
• config/data// → contains application properties that can be read only by a specific service in a specific namespace/environment. Priority 4 (Highest)

The magic happens in rks-kubernetes/

• shared/common/templates/_deployment.yaml
• services/shared/vault/templates/injector*.yaml
• misc/config-resolver

Why store application properties in Vault?

Currently we have 2 main methods of setting spring application properties for an application.

Inject an environment variable into the POD.
This is done either by hardcoding it in the helm charts or by creating a configmap (or secret) with the value and mapping it to a environment variable in the deployment definition.

Pros

• History of all changes are kept
• New properties are automatically added to all environments when deployed

Cons

• helm charts are stored with the bamboo release artifacts and to update a property (like a database configuration) we have to update helm charts and make a new release. Or we can directly edit the deployment/configmaps on the production system, but we must remember to update the helm charts in the repository and make a new release. If we deploy a version to production that is before the new released version we will get reverted to the old value and maybe the system will be broken.
• It is very annoying to add a new value (especially if it is a secret, so far only Bohdan has been able to successfully add new secrets)

Put value in Vault.
The values can be set in vault. The vault_configurer can be used to set the initial value. Or PPTerm can be used to set the values.

Pros

• Changing is easier, quicker and less likely to be accidentally reverted
  • Update vault value
  • Scale instances down to 0 then back up
• Values can be shared across all apps in cluster, namespaces, etc…
• PPTerm has commands for setting and reviewing values.

Cons

• Only last ~10 values are saved in history. Not comments to explain the change.

Database Configuration in Vault

All jdbc configuration has been migrated to Vault. Each environment will have a configuration configured in Vault using the precedence listed in «Application Properties in Vault» section. Configuration options that are common cluster wide are put in the
«config/data/common/vault_config» secret. Since all environments on the dev cluster use the same database most of the JDBC configuration is in this secret.

For properties like database name, the properties will be in the service specific (but namespace agnostic) property. «config/data/common/<helm_chart_id». For example tpms attachments database will be stored in the «config/data/common/attachments» secret.

Only when required will the secrets be placed in the most specific secret (config/data//). This will mostly be done for production but most other clusters do not require this level of security.

For minikube vault-configurer will configure these properties.

When setting up a new environment it is likely the «common» properties will apply to that new environment. For example, when starting a new CI build (and therefore environment) no changes need to be made to vault because all of the secrets and configuration required will already be available in the common configuration.

Configuring Data in Vault

There are 2 methods to configure data in vault. If the data is the same on all environments then the vault_configurer can be used to configure it. However this is normally not the case. If the data is consistent in all environments then it makes sense to put those values in the application.properties files directly. Rather it is more common to have to configure each environment with individual values. This section explains how to do this with the currently existing environments.

Currently there are 3 vaults, one for each cluster, dev, demo and prod. Vault on dev cluster has a UI for viewing and editing the vault data. Demo and dev do not have the UI enabled and thus a command line interface is required. To simplify this task the RKS CLI has vault commands for reading and writing secrets. The vault config … commands can be used to read and write the values to and from vault.

Encryption Keys in Vault

All encryption keys are stored in vault in the /encryption/ secret. For example the permissions encryption key is stored in the ct/encryption/permissions secret.

The one exception to this is the TPMS secrets for tenants are stored under: /encryption/tenants/ secrets. Each service provider has its own secret that contains its encryption key

Effect on Unit Tests

Each module that has spring tests (IE test that load the spring application context) must have a bootstrap.yaml (or bootstrap.properties) in the test resources directory that contains the line:

spring.profiles.include=unit_test