> ## Documentation Index
> Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# OCI OKE Kubelet Configuration File Permissions Should Be 644

### More Info:

The kubelet configuration file should have permissions of 644 or more restrictive. World-writable kubelet config files can be tampered with by any user on the node and lead to credential or configuration theft.

### Risk Level

High

### Address

Compliance, Security

### Compliance Standards

* CIS OKE

### Triage and Remediation

<Tabs>
  <Tab title="Remediation">
    ### Remediation

    <AccordionGroup>
      <Accordion title="Using Console" defaultOpen="true">
        To fix this in OKE, you need to ensure every worker node sets the kubelet config file permissions at boot. In OKE this is done via the node pool configuration (cloud‑init) from the OCI Console, then by recycling nodes.

        Below are the steps using only the OCI Console.

        ***

        ### 1. Identify the kubelet config file path in your OKE version

        For recent OKE versions, kubelet config is typically at one of:

        * `/var/lib/kubelet/config.yaml`\
          or
        * `/etc/kubernetes/kubelet/kubelet-config.json`

        If you’re unsure:

        1. SSH to one worker node (from Bastion or similar).
        2. Run:
           ```bash theme={null}
           sudo find / -maxdepth 6 -name "config.yaml" -path "*kubelet*" 2>/dev/null
           sudo find / -maxdepth 6 -name "kubelet-config.json" 2>/dev/null
           ```
        3. Note the correct path (use that in the script in step 3).

        Assume below the file is `/var/lib/kubelet/config.yaml`. Replace with your actual path if different.

        ***

        ### 2. Open your OKE Cluster and Node Pool in the Console

        1. In OCI Console, go to **Developer Services** → **Kubernetes Clusters (OKE)**.
        2. Click your **cluster**.
        3. Go to the **Node Pools** tab.
        4. Click the **node pool** you want to fix (repeat for each pool type if needed).

        ***

        ### 3. Add a cloud-init script to enforce 644 permissions

        You will inject a small script that runs on each node at boot and fixes the permissions.

        1. In the Node Pool details page, click **Edit** (or **Edit Node Pool**).

        2. Find the section for **Node Configuration**, **Custom cloud-init**, or **Additional configuration** (name varies slightly by UI version, but it is where “User data” or “Cloud-init script” goes).

        3. In the **Cloud-init script** (YAML) box, add/append the following:

           ```yaml theme={null}
           #cloud-config
           runcmd:
             - |
               KUBELET_CONFIG_FILE="/var/lib/kubelet/config.yaml"
               if [ -f "$KUBELET_CONFIG_FILE" ]; then
                 chown root:root "$KUBELET_CONFIG_FILE"
                 chmod 644 "$KUBELET_CONFIG_FILE"
               fi
           ```

           * If you already have a `#cloud-config` block, just append the `runcmd` section or add these commands to the existing `runcmd` list.
           * If your kubelet config file path is different, change `KUBELET_CONFIG_FILE` accordingly.

        4. Click **Save changes** (or **Update**).

        This ensures all *new* nodes from this pool will apply the correct permissions automatically.

        ***

        ### 4. Recycle/replace existing worker nodes

        Existing VMs won’t retroactively run the new cloud-init. You need to cycle them:

        **Option A – Rolling node termination (recommended)**

        1. Still in the **Node Pool** details page, go to the **Nodes** list.
        2. For each node (one or a few at a time to avoid downtime):
           * Select the node.
           * Click **Terminate**.
        3. Ensure the node pool is set to maintain its configured node count (it usually is by default).
        4. OKE will automatically create replacement nodes, which will run the new cloud-init and set kubelet config to `644`.

        **Option B – Scale down then up (more disruptive)**

        1. Edit the Node Pool and temporarily reduce **Number of nodes** to 0 and save.
        2. After all nodes terminate, edit again and set **Number of nodes** back to the desired value.
        3. All new nodes will have the correct permissions.

        ***

        ### 5. Verify permissions

        After new nodes are up:

        1. SSH to a new worker node.
        2. Run:
           ```bash theme={null}
           ls -l /var/lib/kubelet/config.yaml
           ```
           (or your actual path).

        You should see:

        ```bash theme={null}
        -rw-r--r-- 1 root root ... /var/lib/kubelet/config.yaml
        ```

        i.e., mode `644`, owner `root:root`.

        Repeat for all node pools if you have multiple pools.
      </Accordion>

      <Accordion title="Using CLI">
        For OKE you can’t change kubelet file permissions directly *from* the control plane; you must fix them on the worker nodes (node pool instances). Using OCI CLI, the cleanest way is via the Oracle Cloud Agent “Run Command” (or via SSH if you prefer). Below are step‑by‑step instructions using OCI CLI.

        Assumptions (adjust paths as needed for your OKE image/version):

        * Kubelet config file path: `/var/lib/kubelet/config.yaml`
        * Desired permissions: `644`
        * Ownership should remain `root:root`

        ***

        ### 1. Identify your node pool and nodes

        ```bash theme={null}
        # 1.1 Get node pools in a given cluster
        oci ce node-pool list \
          --compartment-id <COMPARTMENT_OCID> \
          --cluster-id <CLUSTER_OCID> \
          --query 'data[].{"Name": "name", "Id":"id"}' \
          --output table

        # 1.2 List nodes in the target node pool
        oci ce node-pool get \
          --node-pool-id <NODEPOOL_OCID> \
          --query 'data.nodes[].{"NodeName":"hostname", "InstanceId":"id"}' \
          --output table
        ```

        You now have the **instance OCIDs** of the worker nodes.

        ***

        ### 2. Make sure Oracle Cloud Agent “Run Command” is enabled

        On each instance, the **OS Management / Oracle Cloud Agent Management** plugin must have the “Run Command” capability enabled.

        Check per instance (example for one instance):

        ```bash theme={null}
        oci compute instance get \
          --instance-id <INSTANCE_OCID> \
          --query 'data."agent-config"."plugins-config"' \
          --output json
        ```

        If `run-command` (or similar) plugin is disabled, update:

        ```bash theme={null}
        oci compute instance update \
          --instance-id <INSTANCE_OCID> \
          --agent-config '{"pluginsConfig":[{"name":"OS Management","desiredState":"ENABLED"},{"name":"Run Command","desiredState":"ENABLED"}]}'
        ```

        (Names may vary slightly by region/image; enable the Run Command-related plugin used in your tenancy.)

        ***

        ### 3. Prepare the remediation command

        The command you need on each node:

        ```bash theme={null}
        sudo chmod 644 /var/lib/kubelet/config.yaml
        sudo chown root:root /var/lib/kubelet/config.yaml
        ```

        You’ll run this through the “Run Command” feature via OCI CLI.

        ***

        ### 4. Run the command on each worker node via OCI CLI

        OCI provides a run‑shell‑script style API under Oracle Cloud Agent. In many tenancies this is exposed as:

        ```bash theme={null}
        oci compute instance run-command \
          --instance-id <INSTANCE_OCID> \
          --mode DETACHED \
          --content '#!/bin/bash
        set -e
        if [ -f /var/lib/kubelet/config.yaml ]; then
          chmod 644 /var/lib/kubelet/config.yaml
          chown root:root /var/lib/kubelet/config.yaml
        fi
        ' \
          --timeout-in-seconds 300
        ```

        If your tenancy exposes it via the *Os Management / Work Requests* interface, the pattern is similar:

        ```bash theme={null}
        oci os-management-hub managed-instance run-command \
          --managed-instance-id <INSTANCE_OCID> \
          --run-command-details '{
            "commandType": "RUN_SHELL_SCRIPT",
            "timeoutInSeconds": 300,
            "scriptContent": "IyEvYmluL2Jhc2gKc2V0IC1lCmlmIFsgLWYgL3Zhci9saWIva3ViZWxldC9jb25maWcueWFtbCBdOyB0aGVuCiAgY2htb2QgNjQ0IC92YXIvbGliL2t1YmVsZXQvY29uZmlnLnlhbWwKICBjaG93biByb290OnJvb3QgL3Zhci9saWIva3ViZWxldC9jb25maWcueWFtbAplZgo="
          }'
        ```

        `scriptContent` above is base64 of the same bash script; you can create this on your own system:

        ```bash theme={null}
        cat << 'EOF' > fix_kubelet_perm.sh
        #!/bin/bash
        set -e
        if [ -f /var/lib/kubelet/config.yaml ]; then
          chmod 644 /var/lib/kubelet/config.yaml
          chown root:root /var/lib/kubelet/config.yaml
        fi
        EOF

        base64 -w0 fix_kubelet_perm.sh
        ```

        Use that base64 output in `scriptContent`.

        Run this for each worker instance in the node pool.

        ***

        ### 5. Verify permissions

        Use either:

        * “Run Command” again with a verification script, or
        * SSH (via Bastion / public IP) and check manually.

        Example via SSH:

        ```bash theme={null}
        ssh -i <KEY> opc@<NODE_PUBLIC_IP> \
          'ls -l /var/lib/kubelet/config.yaml'
        ```

        Expected:

        ```text theme={null}
        -rw-r--r-- 1 root root ... /var/lib/kubelet/config.yaml
        ```

        ***

        ### 6. Make it persistent for *new* nodes in the node pool

        To avoid this reappearing when nodes are replaced or scaled:

        1. Edit the node pool metadata (cloud-init) via OCI CLI to add a small post‑boot script:

        ```bash theme={null}
        oci ce node-pool update \
          --node-pool-id <NODEPOOL_OCID> \
          --node-metadata '{"user_data":"<BASE64_OF_CLOUD_INIT_SCRIPT>"}'
        ```

        Where `user_data` is a base64-encoded cloud‑init file, for example:

        ```yaml theme={null}
        #cloud-config
        runcmd:
          - [ bash, -c, "if [ -f /var/lib/kubelet/config.yaml ]; then chmod 644 /var/lib/kubelet/config.yaml && chown root:root /var/lib/kubelet/config.yaml; fi" ]
        ```

        Encode and supply in `user_data`.

        2. For existing nodes you already fixed, nothing more is needed; for any *new* or *replaced* nodes, the cloud‑init will enforce 644 automatically.

        ***

        If you share your exact OKE worker image/version, I can give the precise kubelet config path and an exact `oci` command variant for your environment.
      </Accordion>

      <Accordion title="Using Python">
        Below is a practical way to remediate this in Oracle Container Engine for Kubernetes (OKE) using Python:

        **Goal:** Ensure kubelet config file(s) on every worker node have permission `644` (rw-r--r--).

        Typical kubelet config paths on OKE nodes (you may have one or both):

        * `/etc/kubernetes/kubelet.conf`
        * `/var/lib/kubelet/config.yaml`

        ***

        ## Approach 1: Python + Kubernetes API (recommended; no direct SSH)

        We will:

        1. Use the Kubernetes Python client.
        2. Create and run a privileged DaemonSet that:
           * Mounts the host `/` filesystem.
           * Executes `chmod 644` on kubelet config files on each node.
        3. Delete the DaemonSet after it finishes.

        ### 1. Install dependencies

        ```bash theme={null}
        pip install kubernetes
        ```

        Make sure your kubeconfig for OKE is set (e.g. `KUBECONFIG=~/.kube/config` or default path).

        ### 2. Python script to apply a DaemonSet that fixes permissions

        ```python theme={null}
        from kubernetes import client, config

        NAMESPACE = "kube-system"
        DAEMONSET_NAME = "fix-kubelet-perms"

        daemonset_manifest = {
            "apiVersion": "apps/v1",
            "kind": "DaemonSet",
            "metadata": {
                "name": DAEMONSET_NAME,
                "namespace": NAMESPACE,
                "labels": {"app": DAEMONSET_NAME},
            },
            "spec": {
                "selector": {"matchLabels": {"app": DAEMONSET_NAME}},
                "template": {
                    "metadata": {"labels": {"app": DAEMONSET_NAME}},
                    "spec": {
                        "hostPID": True,
                        "hostNetwork": True,
                        "tolerations": [
                            {"operator": "Exists"}  # run on all nodes, including masters if any
                        ],
                        "containers": [
                            {
                                "name": "fix-kubelet-perms",
                                "image": "oraclelinux:8",
                                "securityContext": {
                                    "privileged": True
                                },
                                "command": [
                                    "bash",
                                    "-c",
                                    # Adjust file paths if your environment differs
                                    """
                                    set -e
                                    FILES=(
                                      /etc/kubernetes/kubelet.conf
                                      /var/lib/kubelet/config.yaml
                                    )
                                    for f in "${FILES[@]}"; do
                                      if [ -f "$f" ]; then
                                        chmod 644 "$f"
                                        echo "Set 644 on $f"
                                      else
                                        echo "File $f not found, skipping"
                                      fi
                                    done
                                    # sleep briefly so logs can be read, then exit
                                    sleep 10
                                    """,
                                ],
                                "volumeMounts": [
                                    {
                                        "name": "rootfs",
                                        "mountPath": "/",
                                        "readOnly": False
                                    }
                                ]
                            }
                        ],
                        "volumes": [
                            {
                                "name": "rootfs",
                                "hostPath": {"path": "/", "type": "Directory"}
                            }
                        ]
                    }
                },
                "updateStrategy": {"type": "RollingUpdate"},
            }
        }


        def main():
            # Load kubeconfig
            config.load_kube_config()

            apps_v1 = client.AppsV1Api()

            # Create DaemonSet
            try:
                apps_v1.create_namespaced_daemon_set(
                    namespace=NAMESPACE,
                    body=daemonset_manifest
                )
                print(f"DaemonSet {DAEMONSET_NAME} created in namespace {NAMESPACE}.")
            except client.exceptions.ApiException as e:
                if e.status == 409:
                    print("DaemonSet already exists, continuing...")
                else:
                    raise

            # Optionally wait until at least one pod runs on each node and exits successfully,
            # then delete the DaemonSet. Minimal example: delete after N seconds.
            import time
            time.sleep(120)  # wait ~2 minutes for all nodes

            # Delete DaemonSet (cleanup)
            apps_v1.delete_namespaced_daemon_set(
                name=DAEMONSET_NAME,
                namespace=NAMESPACE,
                body=client.V1DeleteOptions(
                    propagation_policy="Foreground"
                )
            )
            print(f"DaemonSet {DAEMONSET_NAME} deleted.")


        if __name__ == "__main__":
            main()
        ```

        **What this does:**

        * Runs a privileged pod on every node.
        * Each pod `chmod 644` on kubelet config files if they exist.
        * Then the script deletes the DaemonSet.

        ***

        ## Approach 2: Python + SSH (Paramiko) to each worker node

        Use this if you prefer to log in to OKE worker nodes directly (bastion or VCN access required).

        ### 1. Install Paramiko

        ```bash theme={null}
        pip install paramiko
        ```

        ### 2. Python script to fix permissions via SSH

        ```python theme={null}
        import paramiko

        # Replace with your worker node IPs or hostnames
        NODES = [
            "10.0.1.10",
            "10.0.1.11",
            # ...
        ]

        SSH_USER = "opc"            # typically 'opc' for Oracle Linux
        SSH_KEY_PATH = "/path/to/your/private_key"  # e.g. ~/.ssh/id_rsa

        COMMAND = """
        FILES=(
          /etc/kubernetes/kubelet.conf
          /var/lib/kubelet/config.yaml
        )
        for f in "${FILES[@]}"; do
          if [ -f "$f" ]; then
            chmod 644 "$f"
            echo "Set 644 on $f"
          else:
            echo "File $f not found, skipping"
          fi
        done
        """

        def run_command(host, user, key_path, cmd):
            key = paramiko.RSAKey.from_private_key_file(key_path)
            client = paramiko.SSHClient()
            client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
            client.connect(hostname=host, username=user, pkey=key)
            stdin, stdout, stderr = client.exec_command(cmd)
            print(f"=== {host} STDOUT ===")
            print(stdout.read().decode())
            print(f"=== {host} STDERR ===")
            print(stderr.read().decode())
            client.close()

        def main():
            for node in NODES:
                print(f"Fixing kubelet permissions on {node}")
                run_command(node, SSH_USER, SSH_KEY_PATH, COMMAND)

        if __name__ == "__main__":
            main()
        ```

        ***

        ## Verification

        On any worker node (via SSH or via a debug pod with `hostPath`), run:

        ```bash theme={null}
        stat -c "%a %n" /etc/kubernetes/kubelet.conf 2>/dev/null
        stat -c "%a %n" /var/lib/kubelet/config.yaml 2>/dev/null
        ```

        You should see permissions `644` where the files exist.

        ***

        If you share:

        * exact OKE version,
        * OS image (Oracle Linux version),
          I can adjust the file paths and DaemonSet manifest precisely to your environment.
      </Accordion>

      <Accordion title="Using Terraform">
        ```hcl theme={null}
        resource "oci_containerengine_node_pool" "oke_nodepool" {
          # Existing config for your node pool
          compartment_id = "COMPARTMENT_OCID"
          cluster_id     = "OKE_CLUSTER_OCID"
          name           = "OKE_NODE_POOL_NAME"
          kubernetes_version = "K8S_VERSION"
          node_shape         = "NODE_SHAPE"

          node_config_details {
            size = 3

            placement_configs {
              availability_domain = "AD_NAME"
              subnet_id           = "SUBNET_OCID"
            }

            # Ensure this map exists; add/merge the user_data key if you already use node_metadata
            node_metadata = {
              # Cloud-init script to set kubelet config permissions to 0644
              user_data = base64encode(<<-EOF
                #cloud-config
                runcmd:
                  - [ chmod, "0644", "/var/lib/kubelet/config.yaml" ]
              EOF
              )
            }
          }

          node_source_details {
            source_type = "image"
            image_id    = "NODE_IMAGE_OCID"
          }

          ssh_public_key = "SSH_PUBLIC_KEY"
        }
        ```

        Changing `node_metadata.user_data` will cause existing worker nodes in this node pool to be replaced (a rolling recycle), which can briefly disrupt workloads if you lack sufficient pod disruption budgets or capacity.

        Verification with `terraform plan` should show an in-place update to `oci_containerengine_node_pool.oke_nodepool` with a change to `node_config_details[0].node_metadata.user_data`, and no other resources changed.
      </Accordion>
    </AccordionGroup>
  </Tab>
</Tabs>