Centralizing Secrets Management Across an AWS-Based Organization

Engagement

Organization-wide secrets management modernization.

Technologies

HashiCorp Vault · AWS IAM · Amazon EKS · Kubernetes Service Accounts · Vault Secrets Operator · Vault Agent · OIDC · Terraform · Helm · PKI · Dynamic Database Credentials · Audit Logging

Executive Summary

A confidential AWS-based organization was facing a familiar but risky problem: secrets were everywhere.

Application credentials lived in environment files, CI/CD variables, Kubernetes Secrets, VM configuration, deployment scripts, and team knowledge. TLS certificates were renewed manually. Database credentials were long-lived. Access patterns differed from team to team, and understanding who could access what often required searching through pipelines, manifests, local conventions, and undocumented operational history.

We designed and implemented a centralized secrets management platform for the entire organization using HashiCorp Vault.

The new platform established Vault as the central trust broker for applications, operators, Kubernetes workloads, AWS workloads, certificates, and dynamic credentials. Instead of distributing long-lived secrets to applications, workloads now authenticate using platform identity and receive only the secrets or certificates they are authorized to use.

The result was a measurable shift from static, scattered, manually managed secrets toward identity-based access, policy-driven authorization, automated rotation, and centralized auditability.

The Challenge

Before the project, the client’s secrets landscape had grown organically across teams and environments.

Secrets were stored in multiple places: application .env files, CI/CD variables, Kubernetes Secrets, VM configuration files, manually maintained certificate stores, and sometimes team memory. This created an operational model where secrets were available, but not consistently governed.

The core issue was not simply where secrets were stored. The deeper problem was that secret distribution was disconnected from identity, policy, rotation, and audit.

That created several business and technical risks:

• Credentials could remain valid long after they were copied, deployed, or no longer needed.
• Application teams had inconsistent patterns for storing and consuming secrets.
• Rotating a credential was risky because it was unclear which services depended on it.
• Certificate renewal required manual coordination.
• Human access and workload access were not always separated cleanly.
• Auditing access was difficult because secrets were spread across systems.
• Onboarding new applications required repeated custom work instead of following a standard platform pattern.

The client needed a centralized model that would work across teams, environments, Kubernetes workloads, AWS-hosted services, and human operators.

Project Objectives

The goal was to build a secrets platform that could serve the whole organization, not just one application team.

The target state was clear:

• No long-lived application credentials in source code, container images, VM disks, or deployment artifacts.
• Human operators authenticate through OIDC and receive policy-based access.
• AWS and Kubernetes workloads authenticate through platform identity instead of hardcoded credentials.
• Each application and environment receives its own policy boundary.
• Secrets and certificates are issued only to authorized workloads.
• TLS certificates can be issued automatically with short TTLs.
• Secret updates trigger application reloads or restarts where needed.
• All configuration is reproducible through Terraform, Helm, and code.
• Every secret access, certificate issuance, authentication attempt, renewal, and denial can be audited centrally.

The Solution

We implemented a centralized HashiCorp Vault platform for organization-wide secrets management.

Vault became the central control plane for secret storage, workload authentication, policy enforcement, certificate issuance, and audit logging.

The solution was designed around one principle: applications should not carry secrets forever. They should prove who they are, receive only what they need, and survive rotation.

Identity-Based Access

Human users authenticate through OIDC, allowing access to be mapped through identity provider groups and Vault policies.

Kubernetes workloads authenticate using Vault’s built-in Kubernetes authentication method. Each application’s service-account identity is validated directly by Vault, mapped to a Vault role, and exchanged for a short-lived token with tightly scoped policies.

EC2 instances and other AWS workloads authenticate using Vault’s built-in AWS authentication method. IAM roles, instance profiles, and task roles serve as the identity source — no static credentials required.

This moved the organization away from shared static credentials and toward identity-aware access control.

AWS Architecture Adaptation

The AWS-oriented design used the following model:

• Human operators authenticate through the organization’s OIDC provider.
• Amazon EKS workloads authenticate via Vault’s Kubernetes authentication method using service-account identity.
• EC2 instances and AWS workloads authenticate via Vault’s AWS authentication method using IAM identity.
• Vault policies define exactly which application, environment, and secret paths each workload can access.
• Vault Secrets Operator syncs approved Vault secrets into Kubernetes-native Secrets where required.
• Vault Agent is used for workloads that need rendered configuration files, local secret delivery, or automatic renewal.
• Vault PKI issues internal certificates through controlled roles.
• cert-manager integrates with Vault for automated Kubernetes TLS certificate issuance.
• Database secrets can be issued dynamically, allowing applications to receive short-lived database credentials instead of shared static passwords.

This created a consistent secrets access pattern across EKS, AWS-hosted workloads, operators, and infrastructure automation.

Because Vault is cloud-agnostic, this architecture also positions the organization to grow into hybrid or multi-cloud environments in the future without vendor lock-in. Authentication methods for Azure, GCP, or other platforms can be added alongside the existing AWS and Kubernetes methods as the organization evolves.

Implementation Approach

1. Secret Inventory and Access Model

The first phase focused on understanding the current state.

We reviewed existing secrets across application configuration, CI/CD systems, Kubernetes namespaces, VM configuration, database credentials, and TLS certificate handling.

From there, we defined a standard naming and path convention for Vault. The design separated access by application, environment, workload type, human versus machine access, and operational responsibility.

This was important because policy naming and path conventions become the foundation for long-term maintainability.

2. Vault Policy and Namespace Design

Each application and environment received a dedicated policy boundary.

A production workload could be allowed to read only its production secret path while being denied access to development, staging, or other application paths.

This made denied access an intentional security control, not an error condition. Instead of broad shared secrets, applications received narrowly scoped permissions tied to identity and environment.

3. Kubernetes Integration

For Kubernetes workloads, we configured Vault’s built-in Kubernetes authentication method alongside Vault Secrets Operator.

The flow works like this: an application Pod starts with a Kubernetes service account. That service-account identity is validated directly by Vault’s Kubernetes auth method, mapped to a Vault role, and exchanged for a short-lived Vault token with limited policies.

Vault Secrets Operator then syncs approved secrets into Kubernetes for application consumption where Kubernetes-native integration is required.

For certificate workflows, cert-manager integrates with Vault PKI so that Kubernetes TLS certificates can be issued and renewed automatically.

This gave application teams a familiar Kubernetes-native consumption pattern while keeping Vault as the source of truth.

4. AWS Workload Integration

For EC2 instances and other AWS-hosted workloads outside Kubernetes, we configured Vault’s built-in AWS authentication method.

Instead of provisioning static Vault tokens or embedding secret material on servers, AWS workloads authenticate to Vault using their IAM identity directly. Vault validates the identity against AWS and issues a scoped token.

For VM-style workloads, Vault Agent runs alongside the application and handles authentication, token renewal, and secret rendering. For services running with IAM roles, instance profiles, or task roles, the workload identity becomes the basis for Vault authentication and policy assignment.

This allowed AWS workloads to retrieve secrets without any manually distributed credentials.

5. Dynamic Database Credentials

Where possible, we replaced long-lived database passwords with dynamic database credentials.

With Vault’s database secrets engine, applications request credentials when needed. Vault creates a unique database user with limited privileges and a defined lease. When the lease expires or is revoked, Vault removes or revokes that database user.

This reduces the blast radius of a leaked credential and gives the organization a clear audit trail for database access.

6. Certificate Lifecycle Automation

We introduced Vault PKI as a centralized internal certificate authority pattern for workloads.

Certificates are issued through controlled PKI roles, with policies defining which application or environment can request which certificates.

This allowed the organization to automate certificate issuance, renewal, rotation, and revocation. The same design also supports short-lived internal certificates, reducing exposure if a certificate or private key is compromised.

7. Auditability and Governance

A major outcome of the project was centralized auditability.

Vault audit logs show authentication events, secret reads, certificate issuance, renewals, revocations, and denied access attempts.

The logs help answer operational and security questions such as: Who accessed this secret? Which workload requested this certificate? Was access allowed or denied? Which Vault policy was involved? When did the access happen?

The platform also benefits from Vault’s default behavior of masking sensitive values in audit logs, allowing access events to be reviewed without exposing the secret values themselves.

Results

Key outcomes included:

• Reduced secret sprawl by consolidating secrets into a central Vault-backed platform.
• Removed long-lived application credentials from code, images, and VM-based configuration patterns.
• Introduced identity-based access for both humans and workloads.
• Established app- and environment-specific policy boundaries.
• Standardized secret paths, naming conventions, and access models.
• Enabled Kubernetes workloads to consume centrally managed secrets using Vault Secrets Operator.
• Enabled AWS workloads to authenticate using Vault’s built-in AWS auth method rather than manually provisioned credentials.
• Automated internal certificate issuance and renewal through Vault PKI and cert-manager.
• Created a foundation for dynamic, short-lived database credentials.
• Improved auditability across authentication, reads, issuance, renewal, revocation, and denied access.
• Made the platform reproducible through Terraform, Helm, and code.
• Positioned the organization for hybrid or multi-cloud growth without vendor lock-in.

Most importantly, the organization moved from a model where applications “had secrets” to a model where applications prove identity and receive controlled access.

Business Impact

The new platform reduced operational risk and improved engineering velocity.

Security teams gained centralized visibility and policy enforcement. Platform teams gained a reusable onboarding model for new applications. Application teams gained a standard way to consume secrets and certificates without custom handling for every deployment. Operations teams gained safer rotation workflows because secret changes could be managed centrally and paired with reload or restart mechanisms.

The organization also gained a stronger foundation for compliance and incident response because Vault provided a central audit trail for sensitive access.

Lessons Learned

• Kubernetes Secrets are still Kubernetes Secrets. Vault Secrets Operator improves source of truth, synchronization, and rotation, but it does not eliminate every in-cluster threat by itself.
• Short TTLs are valuable only when reload and restart behavior is solved. A short-lived secret without a reliable refresh path creates instability instead of security.
• Denied access is a feature. Testing denied paths proves that app and environment boundaries are actually enforced.
• Policy naming and path conventions matter. A clean convention makes the platform easier to scale across teams.
• Human access and workload access should be modeled separately. Operators and applications have different access patterns and should not share the same assumptions.
• Certificates are secrets too. Certificate issuance, renewal, private key handling, and revocation need the same level of governance as passwords and API keys.
• No vendor lock-in. Vault’s cloud-agnostic design means the platform scales with the organization, whether that means more AWS accounts, hybrid cloud, or multi-cloud.

Client Perspective

“They helped us turn secrets management from a scattered operational burden into a centralized platform capability. We now have a consistent way for applications and teams to access secrets, rotate credentials, issue certificates, and audit sensitive access across our AWS environment.”