Sep 186 min read

Secrets Management: Why Design Matters

Secrets Management is a critical part of any DevOps environment, protecting credentials that enable orchestration and automation.

Many people have heard of or already understand the concept of secrets management. Few, however, have dived deep enough to understand how to design secrets management for large-scale deployments.

In this article, I'll focus on the key design principles for secrets management. While there is a lot of material available on vendor-specific deployments and implementations, there is very little on how to design for it.

Many products are now available for enabling secrets management. Although this area has been historically dominated by Hashicorp Vault, capabilities are now available from multiple other vendors and cloud providers.

Some notable examples (in no particular order) include

Commercial products such as Cyberark Conjur, Keeper Secrets Manager or AKeyLess
Cloud provider services such as AWS Secrets Manager, Azure Key Vault or GCP Secrets Manager

Regardless of the vendor, each product provides the following benefits for protecting static secrets. That is,

Reducing the points of exposure for clear text secrets in CICD pipelines and orchestration playbooks.
Providing secure mechanisms for referencing secrets within automation and orchestration workflows.
Securely injecting secrets during the deployment of applications or workloads
Centralising governance across how secrets are accessed and utilised within DevOps tooling and workflows.
Ensuring secrets are not inadvertently captured in audit logs generated by automation and orchestration tooling.

Before discussing the design principles, let's make an important distinction between static and dynamic secrets terminology.

Static secrets are typically long-lived and only rotate once a year, or potentially never at all. Examples of static secrets may include credentials used for Service Accounts, Private Keys for Digital Certificates, SSH Private Keys, or API Keys.

Dynamic secrets are the same secrets, but they are now rotated more frequently. So instead of having a digital certificate issued for one year or an API Key that never changes, those same secrets will be re-issued every few days, and the previous version of that secret will be revoked.

In this article we'll focus on the design principles for protecting static secrets, rather than short lived dynamic secrets. Dynamic secrets are important, but its solving a different problem space and with a different set of considerations for how they are utilised (we'll look to cover this in a separate article).

Here is a list of design principles when considering secrets management

Be opinionated on how secrets management will be integrated into DevOps workflows

The benefit of enabling Secrets Management is having tighter controls on how secrets are utilised within DevOps toolsets to reduce unnecessary exposure.

To do so, it needs to be opinionated on how consumers will utilise it and how it will be integrated into DevOps workflows to tighten those controls.

Establish standard patterns for integrating secrets management for each DevOps toolset.
Don't allow individual teams using bespoke DevOps toolsets to DIY the integration. Ensure any DevOps toolsets integrated into secrets management conform to an agreed-upon design pattern and set of minimum controls.
Reduce the ability for developers to deviate from how secrets will be referenced in their code. Provide code templates for how secrets are referenced in specific orchestration and automation tooling.

Keep consistency with access models for DevOps toolsets (where practical)

Review the onboarding model for new tenants for DevOps tools used for orchestration and automation.

Determining how to design the access controls and permission boundaries that are applied in secrets management requires an understanding on how DevOps toolsets are utilised within your organisation.

Being opinionated about the design for secrets management can create friction during onboarding, but that can be reduced by aligning with existing usage of DevOps toolsets.

Understand existing model for applied to onboarding new tenants, are there defined roles for users and what permission boundaries are applied.

Ensure onboarding into secrets management occurs through a templated setup that enforces a RBAC model for users from the start.
Define the boundaries for tenants based on how this will aligned to broader usage for DevOps services.
Where practical, keep consistency in defining user roles and associated permissions in DevOps to how consumers will be onboarding into Secrets Management. This will reduce friction in applying and enforcing RBAC and deviations.

Model the segregation of secrets around the use of DevOps workflows

Secrets should be segregated into three logical categories to support appropriate access restrictions.

There will primarily be three main categories for secrets held in Secrets Management

Secrets required to support the orchestrated deployment of an application
Secrets required to support automation of that deployment
Secrets required by that application to operate once deployed

Ensure secrets within Secrets Management are split between 'Automation', 'Orchestration and 'Workload' and apply corresponding access restrictions for DevOps tooling to have access to those different categories

Restrict access to 'Automation' secrets to CI/CD tooling that allows

Secrets required to trigger toolsets integrated as part of release pipeline stage gates (e.g. Service accounts to trigger testing)
Secrets required to trigger orchestration toolsets as part of deployment pipelines (e.g. Service accounts to deploy an application)

Restrict access to 'Orchestration' secrets to orchestration tooling to allow

Secrets required to package and deploy applications (e.g. SSH keys to deploy as application to a target host)
Secrets required to configure applications (e.g. SSH keys to update configuration on that host)

Restrict access to 'Workload' secrets required to operate that application

Inject secrets during application deployment into configuration files or environment variables.

Establish the boundaries for different environments

The segregation model for secrets must also incorporate boundaries for different environments (e.g., Development, Non-Production, and Production).

Maintaining segregation between environments means adding a secondary layer of segregation against the split between 'Automation', 'Orchestration and 'Workload'.

Ensure you secrets are split against Development, Non-Production and Production environments.
Don't split these environments too granular. Different group of applications are likely to have different definitions to the number of environments that they have as part of release. Keep consistency by maintaining overall segregation to just 3 main environment groups.
Ensure access to secrets incorporates the same security principles typically applied for segregation of environments (i.e. don't allow access to Production secrets for lower environments)

The metadata associated with a secret is just as important

Ensure that you have a clear standard for the metadata required and a set of conventions for how that metadata is defined. This is important not because it is required for the confidentiality of the secrets but because it is used to define authorisation policies and boundaries for access.

Keep in mind that different vendor products will have different approaches to how this data is captured.

For example, in Hashicorp Vault, you'll capture this metadata within the namespace or path name where secrets are being held.

vault kv put -mount=secret Orchestration/Production/MySecret password=secret

In AWS Secrets Manager, you'll capture this in the tagging attributes for those secrets.

aws secretsmanager tag-resource --secret-id MySecret --tags Key=Environment,Value=Production Key=Category,Value=Orchestration

At a minimum, make sure you've captured metadata regarding the previous two design principles.

Apply policies that enforce conformance to the standards that have been defined for that metadata

Determine what will represent 'Secrets Zero'

Lastly, when considering the different design principles required for defining access to secrets, it's important to note one of the main challenges—Determining 'secrets zero'.

Determining 'secret zero' is the chicken and the egg dilemma for all secrets management solutions. For any tool to authenticate to secrets management, it first needs a secret. That secret then needs to be securely stored and protected, which is the purpose of secrets management. Placing that secret into secrets management means that you then need a new secret to authenticate to secrets management to retrieve it.

This problem is easier to solve in cloud-hosted environments. Primarily because you can leverage the underlying native cloud IAM services to support that initial authentication. For example, in AWS, you can utilise IAM 'assume role' to initial authenticate the Secrets Management service in order to retrieve secrets.

In on-premise hosted environments, this is a much larger challenge to solve. Fundamentally, this will come back to having a trusted orchestration service that can asset identity (most likely a digital certificate) for that workload which can then later be used for authentication to secrets management.

Utilise the underlying IAM services for the hosting platform where available for solving 'secret zero'. This will be available for cloud hosting platforms or Kubernetes.
Where this is not provided, delegate the responsibility to the orchestration tool to provision the initial identity to that workload. This will likely be the case for on-premise hosted virtual machines and servers.
Use digital certificates (if required) to assert initial identity. Standards such as SPIFFE can assist with how to define that identity.

Want to learn more?

Reach out to us at Patterned Security Consulting to show how we can build security architecture design that matters.