> ## Documentation Index
> Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# OCI OKE Kubelet Configuration File Ownership Should Be root:root

### More Info:

The kubelet configuration file (typically /etc/kubernetes/kubelet.conf) should be owned by root:root. Incorrect ownership lets unprivileged users read or modify kubelet credentials and bootstrap state.

### Risk Level

High

### Address

Compliance, Security

### Compliance Standards

* CIS OKE

### Triage and Remediation

<Tabs>
  <Tab title="Remediation">
    ### Remediation

    <AccordionGroup>
      <Accordion title="Using Console" defaultOpen="true">
        In OKE you can’t directly “click a button” in the console to change file ownership inside worker nodes; you have to use the console to push a startup script (cloud‑init) via the node pool configuration so each node fixes it on boot.

        Below are step‑by‑step instructions to remediate this using only the OCI Console (no manual SSH per node).

        ***

        ### 1. Identify the kubelet config file path on OKE nodes

        On standard OKE managed node images, the kubelet config is typically at:

        * `/var/lib/kubelet/config.yaml`\
          (and sometimes a secondary config at `/etc/kubernetes/kubelet.conf`)

        You should fix ownership for both, to be safe.

        ***

        ### 2. Prepare a cloud‑init script (user data)

        You’ll use a cloud‑init script that runs on each node at startup and enforces ownership and permissions.

        Example script:

        ```bash theme={null}
        #!/bin/bash
        # Fix kubelet config file ownership/permissions

        for f in /var/lib/kubelet/config.yaml /etc/kubernetes/kubelet.conf; do
          if [ -f "$f" ]; then
            chown root:root "$f"
            chmod 600 "$f"
          fi
        done

        # Restart kubelet so it picks up correct permissions if needed
        systemctl restart kubelet || true
        ```

        You’ll paste this into the node pool’s “User data” section in the console.

        ***

        ### 3. Update an existing Node Pool to apply the script

        1. In OCI Console, go to:
           * **Developer Services** → **Kubernetes Clusters (OKE)**.
        2. Click your **cluster**.
        3. In the cluster details page, go to **Node Pools** tab.
        4. Click the **node pool** you want to fix.
        5. Click **Edit** (or **Update Node Pool Configuration**, depending on UI).
        6. Find the **Node Metadata / Cloud-Init / User Data** (or “Boot volume, Metadata” or similar) section.
        7. In **User data**, select **Plain text** and paste the script from step 2.
        8. Save the changes.

        This ensures any new/replaced nodes in this pool will automatically set the kubelet config file to `root:root` and `600` on boot.

        > Note: Existing running nodes won’t automatically re‑run cloud‑init just because you updated user data. You need to cycle them.

        ***

        ### 4. Recycle nodes so the script runs

        To ensure all current worker nodes pick up the fix:

        1. In the same node pool page, note the **Number of nodes**.
        2. Option A – Rolling replacement by scaling:
           * Temporarily **increase** the node count (e.g., from 3 to 4).
           * Wait for the new node(s) to be **Active** and **Ready** in the cluster.
           * Then **decrease** the node count back (e.g., from 4 to 3), which terminates old nodes.
        3. Option B – Delete and let the pool replace nodes (if “Autoscale” or node draining is configured appropriately):
           * From **Compute → Instances**, filter by the node pool’s compartment and name.
           * Terminate nodes one by one; the node pool will create new ones using the updated user data.

        New nodes created after the update will run the script at first boot and fix file ownership.

        ***

        ### 5. (Optional) Verify via Kubernetes / SSH

        If you can SSH (or via bastion) to a node:

        ```bash theme={null}
        stat -c '%U:%G %a %n' /var/lib/kubelet/config.yaml /etc/kubernetes/kubelet.conf
        ```

        Expected:

        * Owner: `root`
        * Group: `root`
        * Mode: `600` (or at least not group/world writable or readable)

        Example output:

        ```text theme={null}
        root:root 600 /var/lib/kubelet/config.yaml
        root:root 600 /etc/kubernetes/kubelet.conf
        ```

        ***

        ### 6. Apply this pattern to all Node Pools

        Repeat steps 3–4 for every node pool in the cluster (and across clusters) so every worker node is remediated consistently.

        ***

        Summary via OCI Console:

        * Use **Node Pool → Edit → User data** to add a small boot script (`chown root:root` + `chmod 600` + `systemctl restart kubelet`).
        * Recycle/replace nodes through node pool scaling so the new config is enforced cluster‑wide.
      </Accordion>

      <Accordion title="Using CLI">
        You fix this on the worker node OS; the OCI CLI is used to target the instances and run a remote command that corrects file ownership.

        Below is a minimal, end‑to‑end way using **Instance Agent Commands** via OCI CLI.

        ***

        ### 1. Identify the OKE node pool and nodes

        If you don’t already have the node pool OCID:

        ```bash theme={null}
        oci ce node-pool list \
          --compartment-id <COMPARTMENT_OCID> \
          --cluster-id <OKE_CLUSTER_OCID> \
          --all
        ```

        Then get the node pool details to see the worker nodes:

        ```bash theme={null}
        oci ce node-pool get \
          --node-pool-id <NODEPOOL_OCID> \
          --query "data.nodes[].id" \
          --raw-output
        ```

        Each node has an associated **Compute instance**. Get them:

        ```bash theme={null}
        oci ce node get \
          --node-id <NODE_OCID> \
          --query "data.instance-id" \
          --raw-output
        ```

        Repeat or use `--all` and a jmespath query to list all instance OCIDs.

        ***

        ### 2. Ensure Instance Agent is enabled

        On each instance OCID, the **Oracle Cloud Agent** must be enabled and running (this is the default for OKE images):

        ```bash theme={null}
        oci compute instance get \
          --instance-id <INSTANCE_OCID> \
          --query "data.\"agent-config\""
        ```

        `"is-management-disabled": false` and `"is-monitoring-disabled": false` is what you want.

        ***

        ### 3. Run the remediation command via OCI CLI

        On OKE worker nodes, kubelet config is typically at:

        * `/var/lib/kubelet/config.yaml` (most OKE images)
        * or `/etc/kubernetes/kubelet.conf` (older/custom images)

        You can set ownership to `root:root` for both safely.

        For **each** instance:

        ```bash theme={null}
        oci compute instance-agent command create \
          --instance-id <INSTANCE_OCID> \
          --display-name "Fix kubelet config ownership" \
          --content "$(cat << 'EOF'
        {
          "contentType": "TEXT",
          "scriptContent": "#!/bin/bash\n\
        set -e\n\
        if [ -f /var/lib/kubelet/config.yaml ]; then\n\
          chown root:root /var/lib/kubelet/config.yaml\n\
        fi\n\
        if [ -f /etc/kubernetes/kubelet.conf ]; then\n\
          chown root:root /etc/kubernetes/kubelet.conf\n\
        fi\n"
        }
        EOF
        )"
        ```

        Notes:

        * The script is run as `root` by the agent.
        * You can add permissions hardening if desired, e.g.:

          ```bash theme={null}
          chmod 600 /var/lib/kubelet/config.yaml /etc/kubernetes/kubelet.conf 2>/dev/null || true
          ```

        ***

        ### 4. Verify ownership on each node

        Option A – via another instance-agent command:

        ```bash theme={null}
        oci compute instance-agent command create \
          --instance-id <INSTANCE_OCID> \
          --display-name "Check kubelet ownership" \
          --content "$(cat << 'EOF'
        {
          "contentType": "TEXT",
          "scriptContent": "#!/bin/bash\n\
        ls -l /var/lib/kubelet/config.yaml 2>/dev/null || true\n\
        ls -l /etc/kubernetes/kubelet.conf 2>/dev/null || true\n"
        }
        EOF
        )"
        ```

        Retrieve output:

        ```bash theme={null}
        oci compute instance-agent command get \
          --instance-agent-command-id <COMMAND_OCID> \
          --query "data.results[].output" \
          --raw-output
        ```

        You should see `root root` as owner and group.

        Option B – SSH into a node and:

        ```bash theme={null}
        ls -l /var/lib/kubelet/config.yaml /etc/kubernetes/kubelet.conf 2>/dev/null
        ```

        ***

        ### 5. Make it persistent for new nodes

        For future OKE nodes (new node pools or scale‑out), ensure this is applied automatically, e.g.:

        * Use a **custom node image** with correct ownership baked in, or
        * Add an **OKE node pool cloud‑init** script that fixes ownership on boot:

        ```bash theme={null}
        #cloud-config
        runcmd:
          - chown root:root /var/lib/kubelet/config.yaml 2>/dev/null || true
          - chown root:root /etc/kubernetes/kubelet.conf 2>/dev/null || true
        ```

        Then attach this via `--node-config-details` / `--node-metadata` (cloud-init user-data) when creating the node pool.
      </Accordion>

      <Accordion title="Using Python">
        In OKE you can’t SSH into every node manually at scale, so the standard pattern is:

        1. Run a privileged DaemonSet that:
           * Mounts the host’s `/var/lib/kubelet/config.yaml` (or the kubelet config directory).
           * Runs `chown root:root` and `chmod 600` (or whatever you require) on the host file.
        2. Create/apply that DaemonSet using Python (either via `kubectl` or the Kubernetes Python client).

        Below is a minimal, end‑to‑end example using Python and the Kubernetes Python client.

        ***

        ### 1. Identify kubelet config path on OKE nodes

        On OKE node pools today it is typically:

        ```bash theme={null}
        /var/lib/kubelet/config.yaml
        ```

        Confirm on one node (via Cloud Shell + `ssh` into the node, or via an existing debug DaemonSet):

        ```bash theme={null}
        sudo ls -l /var/lib/kubelet/config.yaml
        ```

        Use that path in the script below.

        ***

        ### 2. Python script to deploy a DaemonSet that fixes ownership

        This script:

        * Connects to your OKE cluster using local kubeconfig.
        * Creates a `DaemonSet` in `kube-system` that:
          * Runs privileged.
          * Mounts the host’s `/var/lib/kubelet/config.yaml`.
          * Executes `chown root:root /host/kubelet/config.yaml && chmod 600 /host/kubelet/config.yaml`.

        After it runs on all nodes once, you can optionally delete the DaemonSet.

        ```python theme={null}
        from kubernetes import client, config

        NAMESPACE = "kube-system"
        DAEMONSET_NAME = "fix-kubelet-config-perms"
        KUBELET_CONFIG_HOST_PATH = "/var/lib/kubelet/config.yaml"
        CONTAINER_IMAGE = "busybox:1.36"   # any tiny image with chown/chmod/sh

        def main():
            # Load kubeconfig (e.g., ~/.kube/config configured for your OKE cluster)
            config.load_kube_config()

            apps_v1 = client.AppsV1Api()

            # Define Pod spec
            container = client.V1Container(
                name="fix-kubelet-config",
                image=CONTAINER_IMAGE,
                security_context=client.V1SecurityContext(
                    privileged=True,
                ),
                command=[
                    "sh",
                    "-c",
                    (
                        "if [ -f /host/kubelet/config.yaml ]; then "
                        "  chown root:root /host/kubelet/config.yaml && "
                        "  chmod 600 /host/kubelet/config.yaml; "
                        "else "
                        "  echo 'config.yaml not found'; "
                        "fi; "
                        "sleep 3600"
                    ),
                ],
                volume_mounts=[
                    client.V1VolumeMount(
                        name="kubelet-config",
                        mount_path="/host/kubelet/config.yaml",
                        read_only=False,
                    )
                ],
            )

            # HostPath volume pointing to kubelet config on host
            volume = client.V1Volume(
                name="kubelet-config",
                host_path=client.V1HostPathVolumeSource(
                    path=KUBELET_CONFIG_HOST_PATH,
                    type="File"
                ),
            )

            pod_spec = client.V1PodSpec(
                containers=[container],
                restart_policy="Always",
                host_network=True,  # not strictly required but common for node utilities
                tolerations=[
                    client.V1Toleration(
                        operator="Exists"
                    )
                ],
                volumes=[volume],
            )

            # Pod template
            template = client.V1PodTemplateSpec(
                metadata=client.V1ObjectMeta(labels={"app": DAEMONSET_NAME}),
                spec=pod_spec,
            )

            # DaemonSet spec
            ds_spec = client.V1DaemonSetSpec(
                selector=client.V1LabelSelector(
                    match_labels={"app": DAEMONSET_NAME}
                ),
                template=template,
            )

            daemonset = client.V1DaemonSet(
                api_version="apps/v1",
                kind="DaemonSet",
                metadata=client.V1ObjectMeta(
                    name=DAEMONSET_NAME,
                    namespace=NAMESPACE,
                ),
                spec=ds_spec,
            )

            # Create or replace DaemonSet
            try:
                existing = apps_v1.read_namespaced_daemon_set(
                    name=DAEMONSET_NAME,
                    namespace=NAMESPACE,
                )
                print(f"DaemonSet {DAEMONSET_NAME} exists, replacing...")
                daemonset.metadata.resource_version = existing.metadata.resource_version
                apps_v1.replace_namespaced_daemon_set(
                    name=DAEMONSET_NAME,
                    namespace=NAMESPACE,
                    body=daemonset,
                )
            except client.exceptions.ApiException as e:
                if e.status == 404:
                    print(f"Creating DaemonSet {DAEMONSET_NAME}...")
                    apps_v1.create_namespaced_daemon_set(
                        namespace=NAMESPACE,
                        body=daemonset,
                    )
                else:
                    raise

            print("DaemonSet applied. Wait until all pods are Running, then verify and optionally delete it.")

        if __name__ == "__main__":
            main()
        ```

        ***

        ### 3. Verify the remediation

        After the DaemonSet pods are `Running` on all nodes:

        ```bash theme={null}
        # On each node (or via another debug DaemonSet):
        sudo ls -l /var/lib/kubelet/config.yaml
        # Should show: -rw------- 1 root root ...
        ```

        When you’re satisfied:

        ```bash theme={null}
        kubectl -n kube-system delete daemonset fix-kubelet-config-perms
        ```

        ***

        ### Notes

        * Adjust `KUBELET_CONFIG_HOST_PATH` if your OKE version uses a different location.
        * If your policy requires different permissions (e.g., `640`), change the `chmod` value in the command.
        * If you prefer, replace the `sleep 3600` with `sleep infinity` or keep the DaemonSet only long enough to fix the files, then delete it.
      </Accordion>

      <Accordion title="Using Terraform">
        ```hcl theme={null}
        # There is no Terraform argument on oci_containerengine_node_pool (or any other
        # OCI Container Engine resource) that controls ownership of /etc/kubernetes/kubelet.conf.
        # This setting is inside the node OS and cannot be remediated at the node pool API level.

        # You must enforce root:root ownership using:
        # - A custom node image where kubelet.conf permissions are pre-hardened, or
        # - Cloud-init / bootstrap scripts baked into that image or into your instance config.

        # Example (conceptual) node pool definition; note there is no way to set kubelet.conf ownership here.

        resource "oci_containerengine_node_pool" "OKE_NODEPOOL" {
          compartment_id = "OCID_OF_COMPARTMENT"
          cluster_id     = "OCID_OF_CLUSTER"
          name           = "NODEPOOL_NAME"
          kubernetes_version = "K8S_VERSION"

          node_shape = "VM.Standard3.Flex"

          node_source_details {
            source_type = "IMAGE"
            image_id    = "OCID_OF_HARDENED_CUSTOM_IMAGE" # Image where kubelet.conf is already owned by root:root
          }

          node_config_details {
            size = 3
            placement_configs {
              availability_domain = "AVAILABILITY_DOMAIN"
              subnet_id           = "OCID_OF_SUBNET"
            }
          }
        }

        # This finding cannot be fixed directly by Terraform on the oci_containerengine_node_pool resource.
        # Use the Console or custom image build pipeline to ensure /etc/kubernetes/kubelet.conf is owned by root:root
        # in the node OS. Then recreate/roll the node pool to pick up the hardened image.

        # Verification: `terraform plan` will show no changes specifically about kubelet.conf ownership,
        # because that configuration is outside Terraform’s OCI provider surface.
        ```
      </Accordion>
    </AccordionGroup>
  </Tab>
</Tabs>