> ## Documentation Index
> Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Consider External Secret Storage

### More Info:

Kubernetes secrets at rest in etcd are not encrypted by default in OKE clusters. Use an external secrets manager such as OCI Vault or encrypt secrets at rest when creating the cluster.

### Risk Level

Medium

### Address

Security

### Compliance Standards

* CIS OKE

### Triage and Remediation

<Tabs>
  <Tab title="Remediation">
    ### Remediation

    <AccordionGroup>
      <Accordion title="Manual Steps" defaultOpen="true">
        1. **Determine how the cluster was created and whether Vault-based secret encryption was enabled**
           * On any machine with access to your IaC or console configuration, review the cluster definition:
             * If using OCI CLI/SDK/Terraform, inspect the cluster resource for fields related to **Vault / KMS key / secret encryption** for etcd.
             * In the OCI Console: go to **Developer Services → Kubernetes Clusters (OKE) → \[your cluster] → Details** and look for any section indicating **Secrets encryption / OCI Vault integration** at cluster creation.
           * If you find a setting that explicitly enables “Kubernetes secret encryption with OCI Vault” (or similar wording), document the Vault OCID/key used and proceed to step 6 (no further remediation needed, only periodic review).

        2. **Inventory Kubernetes Secrets currently in use**
           * On any machine with kubectl access:
             ```bash theme={null}
             kubectl get secrets --all-namespaces -o wide
             ```
           * Identify namespaces containing highly sensitive material (credentials, tokens, keys) and note which applications depend on them. This inventory is needed if you plan to migrate to external secret storage or recreate the cluster with encryption enabled.

        3. **Decide on your target posture: external secrets vs. encrypted secrets at rest in etcd**
           * Based on your organization’s security requirements and current cluster lifecycle practices, choose one (or both):
             * **External secrets manager pattern:** Store secrets in **OCI Vault** or another enterprise secrets manager; expose them to workloads using a secrets operator or sidecar (e.g., external-secrets controller, Vault agent, custom controller).
             * **Encrypted secrets at rest in etcd:** Plan to **recreate the OKE cluster** with the “secrets at rest encrypted by OCI Vault” option enabled at creation time, then migrate workloads and secrets.
           * Involve application owners, security, and platform teams since this affects deployment pipelines and runtime configuration.

        4. **Gather evidence and plan for external secrets integration (if chosen)**
           * On any machine with kubectl access, check if a secrets operator is already present:
             ```bash theme={null}
             kubectl get pods -A | grep -i secret
             ```
           * If you do not see a known operator (e.g., external-secrets, vault-agent-injector), assume you **do not** have external secrets in place.
           * Review how workloads currently consume secrets:
             ```bash theme={null}
             kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{" "}{.metadata.name}{"\t"}{.spec.containers[*].envFrom[*].secretRef.name}{"\t"}{.spec.volumes[*].secret.secretName}{"\n"}{end}' | grep -v '^\s*$'
             ```
           * Use this to estimate effort: which deployments will need to change to consume secrets from OCI Vault (or another manager) via an operator or sidecar.

        5. **Gather evidence and plan for cluster recreation with secrets encryption (if chosen)**
           * In the OCI Console or your IaC, confirm whether your **current cluster** allows toggling secrets encryption (in most OKE flows this is a **cluster-creation-time** choice and cannot be changed in place).
           * If it cannot be changed in-place, document a migration plan:
             * Create a **new OKE cluster** with “Encrypt Kubernetes secrets at rest in etcd using OCI Vault” enabled, referencing an existing or new Vault key.
             * Export and review manifests from the old cluster:
               ```bash theme={null}
               kubectl get all,configmap,secret,ingress -A -o yaml > current-cluster-export.yaml
               ```
             * Exclude or sanitize secrets from this export if your process requires that they be re-injected from OCI Vault or a separate secret pipeline rather than copied as plain Kubernetes Secrets.

        6. **Implement and validate the chosen approach**
           * After deploying an external-secrets solution or recreating the cluster with Vault-backed secret encryption:
             * On any machine with kubectl access, verify that workloads are running and retrieving their secrets successfully:
               ```bash theme={null}
               kubectl get pods -A
               kubectl describe pod <pod-name> -n <namespace>
               ```
             * For external secrets, confirm that the controller is reconciling without errors:
               ```bash theme={null}
               kubectl get pods -n <external-secrets-namespace>
               kubectl logs -n <external-secrets-namespace> <external-secrets-pod-name>
               ```
             * For a newly created, Vault-encrypted cluster, capture evidence from the OCI Console / IaC definition showing that **Kubernetes secrets at rest in etcd are encrypted with OCI Vault**, and retain it with your security documentation.
      </Accordion>

      <Accordion title="Using kubectl">
        ```bash theme={null}
        # 1) List all Secrets in all namespaces (surface how many native K8s secrets you rely on)
        # Run on: any machine with kubectl access
        kubectl get secrets --all-namespaces

        # 2) Show types of secrets in use (helps identify workload-managed vs manually created)
        kubectl get secrets --all-namespaces -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,TYPE:.type' | sort

        # 3) Inspect a sample of non-service-account secrets to understand sensitivity
        # Replace <namespace> and <secret-name> with values from the previous command
        kubectl get secret <secret-name> -n <namespace> -o yaml

        # 4) Decode specific data keys inside a secret to understand actual contents
        # List keys in a secret:
        kubectl get secret <secret-name> -n <namespace> -o jsonpath='{.data}' | jq

        # Decode one key (replace <key> with an entry from the previous output):
        kubectl get secret <secret-name> -n <namespace> -o jsonpath='{.data.<key>}' | base64 --decode; echo

        # 5) Identify Secrets that likely contain credentials (high-value for external storage)
        kubectl get secrets --all-namespaces \
          -o jsonpath='{range .items[?(@.type=="Opaque")]}{.metadata.namespace}{"\t"}{.metadata.name}{"\n"}{end}'

        # 6) Correlate workloads that mount or reference Secrets
        # a) Pods mounting secrets as volumes:
        kubectl get pods --all-namespaces -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{range .spec.volumes[?(@.secret)]}{.secret.secretName}{" "}{end}{"\n"}{end}' | sort

        # b) Pods using imagePullSecrets:
        kubectl get pods --all-namespaces -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{range .spec.imagePullSecrets[*]}{.name}{" "}{end}{"\n"}{end}' | sort

        # c) Pods using envFrom / env that reference Secrets:
        kubectl get pods --all-namespaces -o jsonpath='
        {range .items[*]}
        {.metadata.namespace}{"\t"}{.metadata.name}{"\t"}
        {range .spec.containers[*].envFrom[?(@.secretRef)]}{.secretRef.name}{" "}{end}
        {range .spec.containers[*].env[?(@.valueFrom.secretKeyRef)]}{.valueFrom.secretKeyRef.name}{" "}{end}
        {"\n"}
        {end}' | sort
        ```

        What to look for (indicating a potential problem for this control):

        * Many `Opaque` or other non-service-account secrets containing credentials, API keys, database passwords, or private keys when decoded in steps 3–4.
        * High-sensitivity data (e.g., root DB passwords, cloud provider credentials, long-lived tokens, signing keys) revealed in decoded values.
        * Workloads in steps 6a–6c relying heavily on such secrets directly, with no evidence in your IaC/docs that they are sourced from an external secrets manager (e.g., OCI Vault, external secrets operator).
        * If any of the above are true and your cluster was created without enabling etcd secret encryption using OCI Vault, that combination indicates you should consider moving to an external secrets solution and (for new clusters) enabling etcd secret encryption at creation time.
      </Accordion>

      <Accordion title="Automation">
        ```bash theme={null}
        #!/usr/bin/env bash
        # Check OKE CIS 4.4.2 – external / at-rest secret encryption usage
        # Run on: any machine with kubectl access
        # Prereqs: kubectl configured; optional: oci CLI configured for richer context

        set -euo pipefail

        echo "=== 1) Cluster info ==="
        kubectl cluster-info || {
          echo "ERROR: Unable to reach cluster with kubectl" >&2
          exit 1
        }

        echo
        echo "=== 2) Count of Kubernetes Secrets (by namespace and type) ==="
        kubectl get secrets --all-namespaces -o json \
          | jq -r '
            .items[]
            | [.metadata.namespace, .type] 
            | @tsv' \
          | sort \
          | awk '{
              key=$1"\t"$2;
              count[key]++
            }
            END {
              printf "NAMESPACE\tTYPE\tCOUNT\n";
              for (k in count) {
                split(k, a, "\t");
                printf "%s\t%s\t%d\n", a[1], a[2], count[k];
              }
            }'

        echo
        echo "=== 3) Identify secrets likely holding high-value credentials ==="
        echo "Heuristics: names containing password|token|key|secret|cert|credential"
        kubectl get secrets --all-namespaces -o json \
          | jq -r '
            .items[]
            | select(
                (.metadata.name|test("password|token|key|secret|cert|credential"; "i"))
                or (.type != "Opaque")
              )
            | [ .metadata.namespace
              , .metadata.name
              , .type
              ] | @tsv' \
          | sort \
          | awk 'BEGIN {
                   printf "NAMESPACE\tNAME\tTYPE\n"
                 } { print }'

        echo
        echo "=== 4) Sample of Secrets content metadata (sizes only, not values) ==="
        echo "This helps estimate how much sensitive data is in native Kubernetes secrets."
        echo "NOTE: This does NOT show secret values."
        kubectl get secrets --all-namespaces -o json \
          | jq -r '
            .items[]
            | {
                namespace: .metadata.namespace,
                name: .metadata.name,
                type: .type,
                keys: ( .data | keys ),
                size_bytes: ( .data
                              | to_entries
                              | map({k:.key, v: (.value|@base64d|length)})
                            )
              }' \
          | jq -r '
            . as $s
            | $s.size_bytes[]
            | [ $s.namespace
              , $s.name
              , $s.type
              , .k
              , .v
              ] | @tsv' \
          | sort \
          | awk 'BEGIN {
                   printf "NAMESPACE\tSECRET\tTYPE\tKEY\tLENGTH_BYTES\n"
                 } { print }'

        echo
        echo "=== 5) Detect common apps that could be migrated to external secrets ==="
        echo "These controllers commonly support OCI Vault / external secret managers."
        echo "Examples flagged: cert-manager, external-secrets.io, sealed-secrets, vault, external-dns, ingress controllers."
        kubectl get pods --all-namespaces -o json \
          | jq -r '
            .items[]
            | {
                ns: .metadata.namespace,
                pod: .metadata.name,
                containers: (.spec.containers[].image)
              }' \
          | jq -r '
            [ .ns, .pod, .containers ] | @tsv' \
          | grep -Ei 'cert-manager|external-secrets|sealed-secrets|vault|external-dns|nginx-ingress|traefik|haproxy' \
          | sort -u \
          | awk 'BEGIN {
                   printf "NAMESPACE\tPOD\tIMAGES_MATCHING_KNOWN_CONTROLLERS\n"
                 } { print }'

        echo
        echo "=== 6) OCI / OKE specific hints (MANUAL) ==="
        echo "kubectl cannot tell if etcd encryption with OCI Vault is enabled."
        echo "Review in OCI Console or IaC:"
        echo "- For each OKE cluster, check if 'Encrypt Kubernetes secrets in etcd using OCI Vault' (or equivalent) was enabled at cluster creation."
        echo "- If you use Terraform/other IaC, inspect the cluster resource for Vault/secret-encryption settings."
        echo
        echo "Script complete."
        ```

        **How to interpret the output**

        * Section 2:
          * Large numbers of `Opaque` secrets across many namespaces indicate widespread use of native Kubernetes secrets, which rely on etcd at-rest protection.
          * This is **not** inherently wrong, but all such secrets are in scope for at-rest encryption decisions.

        * Section 3:
          * Lines showing secrets whose names imply passwords, tokens, keys, certificates, or credentials (and non-`Opaque` types like `kubernetes.io/tls`, `kubernetes.io/dockerconfigjson`) highlight **high‑value secrets** stored in etcd.
          * If many critical credentials appear here, but the cluster was **not** created with OCI Vault-based etcd encryption and you are not using an external secrets manager, this is a **concern** for CIS 4.4.2.

        * Section 4:
          * Large `LENGTH_BYTES` values (big keys, blobs, cert bundles) for secrets related to production systems increase the impact if etcd is not encrypted with OCI Vault.

        * Section 5:
          * If you **do not** see controllers that can integrate with external secret stores (e.g., external-secrets, Vault, sealed-secrets) and you also know the cluster’s etcd is not Vault-encrypted, that suggests you are **fully dependent** on unencrypted-at-rest Kubernetes secrets in etcd.

        Use this report together with a **manual** review of each OKE cluster’s configuration in the OCI Console or IaC to decide whether you need to:

        * Recreate the cluster with “Kubernetes secrets in etcd encrypted using OCI Vault”, and/or
        * Introduce an external secrets manager pattern (e.g., syncing from OCI Vault into Kubernetes).
      </Accordion>
    </AccordionGroup>
  </Tab>
</Tabs>