Secure HTTP Traffic with HashiCorp Vault as your PKI + Cert Manager in Kubernetes - Deep Dive!

Here's a deep dive technical guide with the steps to configure HashiCorp Vault as a Private Certificate Authority (PKI) and integrate it with cert-manager in Kubernetes to automate certificate management. I've configured it in production environments but for the purposes of this demo I am implementing it in my lab so that my internal apps can have HTTPS encryption in transit. Here are a few examples of internal apps with certs and as you can see the ping shows they are in a private network.

 

Main Benefits:

1. Centralized PKI Infrastructure: Vault provides a centralized solution for managing your entire certificate lifecycle. Instead of managing certificates across different applications and services, Vault acts as a single source of truth for all your PKI needs. This centralization simplifies management, improves security posture, and ensures consistent certificate policies across your organization.
2. Dynamic Certificate Issuance and Rotation: Vault can automatically issue short-lived certificates and rotate them before expiration. When integrated with cert-manager in Kubernetes, this automation eliminates the manual certificate renewal process that often leads to outages from expired certificates. The system can continuously issue, renew, and rotate certificates without human intervention.
3. Fine-grained Access Control: Vault's advanced policy system allows you to implement precise access controls around who can issue what types of certificates. You can limit which teams or services can request certificates for specific domains, restrict certificate lifetimes based on risk profiles, and implement comprehensive audit logging. This helps enforce the principle of least privilege across your certificate infrastructure.

An additional benefit is Vault's broader secret management capabilities – the same tool managing your certificates can also handle database credentials, API keys, and other sensitive information, giving you a unified approach to secrets management.

Prerequisites

• A DNS Server (I use my firewall)
• A running Kubernetes cluster (I am using microk8s)
• Vault server installed and initialized (vault 0.30.0 · hashicorp/hashicorp)
• cert-manager installed in your Kubernetes cluster (microk8s addon)
• Administrative access to both Vault and Kubernetes

See my homelab diagram in github: mdf-ido/mdf-ido: Config files for my GitHub profile.

1. Configure Vault as a PKI

1.1. Enable the PKI Secrets Engine

# Enable the PKI secrets engine
vault secrets enable pki

# Configure the PKI secrets engine with a longer max lease time (e.g., 1 year)
vault secrets tune -max-lease-ttl=8760h pki

1.2. Generate or Import Root CA

# Generate a new root CA
vault write -field=certificate pki/root/generate/internal \
    common_name="Root CA" \
    ttl=87600h > root_ca.crt

1.3. Configure PKI URLs

# Configure the CA and CRL URLs
vault write pki/config/urls \
    issuing_certificates="http://vault.example.com:8200/v1/pki/ca" \
    crl_distribution_points="http://vault.example.com:8200/v1/pki/crl"

1.4. Create an Intermediate CA

# Enable the intermediate PKI secrets engine
vault secrets enable -path=pki_int pki

# Set the maximum TTL for the intermediate CA
vault secrets tune -max-lease-ttl=43800h pki_int

# Generate a CSR for the intermediate CA
vault write -format=json pki_int/intermediate/generate/internal \
    common_name="Intermediate CA" \
    ttl=43800h > pki_intermediate.json

# Extract the CSR
cat pki_intermediate.json | jq -r '.data.csr' > pki_intermediate.csr

# Sign the intermediate CSR with the root CA
vault write -format=json pki/root/sign-intermediate \
    csr=@pki_intermediate.csr \
    format=pem_bundle \
    ttl=43800h > intermediate_cert.json

# Extract the signed certificate
cat intermediate_cert.json | jq -r '.data.certificate' > intermediate.cert.pem

# Import the signed certificate back into Vault
vault write pki_int/intermediate/set-signed \
    certificate=@intermediate.cert.pem

1.5. Create a Role for Certificate Issuance

# Create a role for issuing certificates
vault write pki_int/roles/your-domain-role \
    allowed_domains="yourdomain.com" \
    allow_subdomains=true \
    allow_bare_domains=true \
    allow_wildcard_certificates=true \
    max_ttl=720h

2. Configure Kubernetes Authentication in Vault

2.1. Enable Kubernetes Auth Method

# Enable the Kubernetes auth method
vault auth enable kubernetes

2.2. Configure Kubernetes Auth Method

# Get the Kubernetes API address
KUBE_API="https://kubernetes.default.svc.cluster.local"

# Get the CA certificate used by Kubernetes
KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)

# Get the JWT token for the Vault SA
KUBE_TOKEN=$(kubectl create token vault-auth)

# Configure the Kubernetes auth method in Vault
vault write auth/kubernetes/config \
    kubernetes_host="$KUBE_API" \
    kubernetes_ca_cert="$KUBE_CA_CERT" \
    token_reviewer_jwt="$KUBE_TOKEN" \
    issuer="https://kubernetes.default.svc.cluster.local"

2.3. Create Policy for Certificate Issuance

# Create a policy file
cat > pki-policy.hcl << EOF
# Read and list access to PKI endpoints
path "pki_int/*" {
  capabilities = ["read", "list"]
}

# Allow creating certificates
path "pki_int/sign/your-domain-role" {
  capabilities = ["create", "update"]
}

path "pki_int/issue/your-domain-role" {
  capabilities = ["create"]
}
EOF

# Create the policy in Vault
vault policy write pki-policy pki-policy.hcl

2.4. Create Kubernetes Auth Role

# Create a role that maps a Kubernetes service account to Vault policies (Created next)
vault write auth/kubernetes/role/cert-manager \
    bound_service_account_names="issuer" \
    bound_service_account_namespaces="default" \
    policies="pki-policy" \
    ttl=1h

3. Configure cert-manager to Use Vault

3.1. Create Service Account for cert-manager

# Create a file named cert-manager-vault-sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: issuer
  namespace: default

Apply the manifest:

kubectl apply -f cert-manager-vault-sa.yaml

3.2. Create Issuer Resource

# Create a file named vault-issuer.yaml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: vault-issuer
  namespace: default
spec:
  vault:
    server: http://vault.vault-system.svc.cluster.local:8200
    path: pki_int/sign/your-domain-role
    auth:
      kubernetes:
        mountPath: /v1/auth/kubernetes
        role: cert-manager
        serviceAccountRef:
          name: issuer

Apply the manifest:

kubectl apply -f vault-issuer.yaml

4. Request Certificates

4.1. Direct Certificate Request

# Create a file named certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-cert
  namespace: default
spec:
  secretName: example-tls
  issuerRef:
    name: vault-issuer
  commonName: app.yourdomain.com
  dnsNames:
  - app.yourdomain.com

Apply the manifest:

kubectl apply -f certificate.yaml

4.2. Using Ingress for Certificate Request

# Create a file named secure-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: secure-ingress
  annotations:
    cert-manager.io/issuer: "vault-issuer"
spec:
  tls:
  - hosts:
    - app.yourdomain.com
    secretName: example-tls
  rules:
  - host: app.yourdomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: example-app
            port:
              number: 80

Apply the manifest:

kubectl apply -f secure-ingress.yaml

5. Troubleshooting

5.1. Common Issues and Solutions

Cannot find cert issuer

The cert issuer was deployed to a specific namespace so if you are creating an ingress outside you might need to solve with a few things:

• Create a cluster issuer which is not restricted to a namespace
• Create a duplicate issuer in the specific namespace
• Create an externalName service and bridge the actual service.

Permission Denied

If you see permission denied errors:

• Check that your Vault policy includes the correct paths
• Verify that the role binding is correct in Vault
• Ensure the service account has the necessary permissions

# Check the Vault policy
vault policy read pki-policy

# Verify the role binding
vault read auth/kubernetes/role/cert-manager

Domain Not Allowed

If you see common name not allowed by this role errors:

• Update your Vault PKI role to allow the domain:

vault write pki_int/roles/your-domain-role \
    allowed_domains="yourdomain.com" \
    allow_subdomains=true \
    allow_bare_domains=true \
    allow_wildcard_certificates=true

Certificate Expiry Issues

If your certificate would expire after the CA certificate:

• Adjust the max TTL to be shorter than your CA expiration:

vault write pki_int/roles/your-domain-role \
    max_ttl="30d"

Issuer Annotation Issues

If multiple controllers are fighting for the certificate request:

• Check that you're using the correct annotation:
  • For namespaced Issuers: cert-manager.io/issuer
  • For ClusterIssuers: cert-manager.io/cluster-issuer

5.2. Checking Certificate Status

# Check certificate status
kubectl describe certificate example-cert

# Check certificate request status
kubectl get certificaterequest

# Check cert-manager logs
kubectl logs -n cert-manager deploy/cert-manager-controller

# Check if the secret was created
kubectl get secret example-tls

6. Best Practices

1. Certificate Rotation: Set appropriate TTLs and let cert-manager handle rotation
2. Secure Vault Access: Restrict access to Vault and use dedicated service accounts
3. Monitor Expirations: Set up alerts for certificate expirations
4. CA Renewals: Plan for CA certificate renewals well in advance
5. Backup: Regularly backup your Vault PKI configuration and CA certificates
6. Audit Logging: Enable audit logging in Vault to track certificate operations

7. Maintenance and Operations

7.1. Renewing the CA Certificate

Before your CA certificate expires, you'll need to renew it:

# Check when your CA certificate expires
vault read pki_int/cert/ca

# Plan and execute your CA renewal process well before expiration

7.2. Rotating Credentials

Periodically rotate your Kubernetes auth credentials:

# Update the JWT token used by Vault
KUBE_TOKEN=$(kubectl create token vault-auth)
vault write auth/kubernetes/config \
    token_reviewer_jwt="$KUBE_TOKEN"

Issues

1. Your ingresses need to be in the same namespace as the issuer
   1. Create an external service as bridge
2. You now have a fully functional PKI system using HashiCorp Vault integrated with cert-manager in Kubernetes. This setup automatically issues, manages, and renews TLS certificates for your applications, enhancing security and reducing operational overhead.

Conclusion

You now have a fully functional PKI system using HashiCorp Vault integrated with cert-manager in Kubernetes. This setup automatically issues, manages, and renews TLS certificates for your applications, enhancing security and reducing operational overhead.

---

by MDF