OCI OKE Kubelet Configuration File Permissions Should Be 644

Using Console

To fix this in OKE, you need to ensure every worker node sets the kubelet config file permissions at boot. In OKE this is done via the node pool configuration (cloud‑init) from the OCI Console, then by recycling nodes.Below are the steps using only the OCI Console.

1. Identify the kubelet config file path in your OKE version

For recent OKE versions, kubelet config is typically at one of:

/var/lib/kubelet/config.yaml
or
/etc/kubernetes/kubelet/kubelet-config.json

If you’re unsure:

SSH to one worker node (from Bastion or similar).

Run:

sudo find / -maxdepth 6 -name "config.yaml" -path "*kubelet*" 2>/dev/null
sudo find / -maxdepth 6 -name "kubelet-config.json" 2>/dev/null

Note the correct path (use that in the script in step 3).

Assume below the file is /var/lib/kubelet/config.yaml. Replace with your actual path if different.

2. Open your OKE Cluster and Node Pool in the Console

In OCI Console, go to Developer Services → Kubernetes Clusters (OKE).
Click your cluster.
Go to the Node Pools tab.
Click the node pool you want to fix (repeat for each pool type if needed).

3. Add a cloud-init script to enforce 644 permissions

You will inject a small script that runs on each node at boot and fixes the permissions.

In the Node Pool details page, click Edit (or Edit Node Pool).
Find the section for Node Configuration, Custom cloud-init, or Additional configuration (name varies slightly by UI version, but it is where “User data” or “Cloud-init script” goes).
In the Cloud-init script (YAML) box, add/append the following:
#cloud-config runcmd: - | KUBELET_CONFIG_FILE="/var/lib/kubelet/config.yaml" if [ -f "$KUBELET_CONFIG_FILE" ]; then chown root:root "$KUBELET_CONFIG_FILE" chmod 644 "$KUBELET_CONFIG_FILE" fi
- If you already have a #cloud-config block, just append the runcmd section or add these commands to the existing runcmd list.
- If your kubelet config file path is different, change KUBELET_CONFIG_FILE accordingly.
Click Save changes (or Update).

This ensures all new nodes from this pool will apply the correct permissions automatically.

4. Recycle/replace existing worker nodes

Existing VMs won’t retroactively run the new cloud-init. You need to cycle them:Option A – Rolling node termination (recommended)

Still in the Node Pool details page, go to the Nodes list.
For each node (one or a few at a time to avoid downtime):
- Select the node.
- Click Terminate.
Ensure the node pool is set to maintain its configured node count (it usually is by default).
OKE will automatically create replacement nodes, which will run the new cloud-init and set kubelet config to 644.

Option B – Scale down then up (more disruptive)

Edit the Node Pool and temporarily reduce Number of nodes to 0 and save.
After all nodes terminate, edit again and set Number of nodes back to the desired value.
All new nodes will have the correct permissions.

5. Verify permissions

After new nodes are up:

SSH to a new worker node.
Run:
ls -l /var/lib/kubelet/config.yaml
(or your actual path).

You should see:

-rw-r--r-- 1 root root ... /var/lib/kubelet/config.yaml

i.e., mode 644, owner root:root.Repeat for all node pools if you have multiple pools.

Using CLI

For OKE you can’t change kubelet file permissions directly from the control plane; you must fix them on the worker nodes (node pool instances). Using OCI CLI, the cleanest way is via the Oracle Cloud Agent “Run Command” (or via SSH if you prefer). Below are step‑by‑step instructions using OCI CLI.Assumptions (adjust paths as needed for your OKE image/version):

Kubelet config file path: /var/lib/kubelet/config.yaml
Desired permissions: 644
Ownership should remain root:root

1. Identify your node pool and nodes

# 1.1 Get node pools in a given cluster
oci ce node-pool list \
  --compartment-id <COMPARTMENT_OCID> \
  --cluster-id <CLUSTER_OCID> \
  --query 'data[].{"Name": "name", "Id":"id"}' \
  --output table

# 1.2 List nodes in the target node pool
oci ce node-pool get \
  --node-pool-id <NODEPOOL_OCID> \
  --query 'data.nodes[].{"NodeName":"hostname", "InstanceId":"id"}' \
  --output table

You now have the instance OCIDs of the worker nodes.

2. Make sure Oracle Cloud Agent “Run Command” is enabled

On each instance, the OS Management / Oracle Cloud Agent Management plugin must have the “Run Command” capability enabled.Check per instance (example for one instance):

oci compute instance get \
  --instance-id <INSTANCE_OCID> \
  --query 'data."agent-config"."plugins-config"' \
  --output json

If run-command (or similar) plugin is disabled, update:

oci compute instance update \
  --instance-id <INSTANCE_OCID> \
  --agent-config '{"pluginsConfig":[{"name":"OS Management","desiredState":"ENABLED"},{"name":"Run Command","desiredState":"ENABLED"}]}'

(Names may vary slightly by region/image; enable the Run Command-related plugin used in your tenancy.)

3. Prepare the remediation command

The command you need on each node:

sudo chmod 644 /var/lib/kubelet/config.yaml
sudo chown root:root /var/lib/kubelet/config.yaml

You’ll run this through the “Run Command” feature via OCI CLI.

4. Run the command on each worker node via OCI CLI

OCI provides a run‑shell‑script style API under Oracle Cloud Agent. In many tenancies this is exposed as:

oci compute instance run-command \
  --instance-id <INSTANCE_OCID> \
  --mode DETACHED \
  --content '#!/bin/bash
set -e
if [ -f /var/lib/kubelet/config.yaml ]; then
  chmod 644 /var/lib/kubelet/config.yaml
  chown root:root /var/lib/kubelet/config.yaml
fi
' \
  --timeout-in-seconds 300

If your tenancy exposes it via the Os Management / Work Requests interface, the pattern is similar:

oci os-management-hub managed-instance run-command \
  --managed-instance-id <INSTANCE_OCID> \
  --run-command-details '{
    "commandType": "RUN_SHELL_SCRIPT",
    "timeoutInSeconds": 300,
    "scriptContent": "IyEvYmluL2Jhc2gKc2V0IC1lCmlmIFsgLWYgL3Zhci9saWIva3ViZWxldC9jb25maWcueWFtbCBdOyB0aGVuCiAgY2htb2QgNjQ0IC92YXIvbGliL2t1YmVsZXQvY29uZmlnLnlhbWwKICBjaG93biByb290OnJvb3QgL3Zhci9saWIva3ViZWxldC9jb25maWcueWFtbAplZgo="
  }'

scriptContent above is base64 of the same bash script; you can create this on your own system:

cat << 'EOF' > fix_kubelet_perm.sh
#!/bin/bash
set -e
if [ -f /var/lib/kubelet/config.yaml ]; then
  chmod 644 /var/lib/kubelet/config.yaml
  chown root:root /var/lib/kubelet/config.yaml
fi
EOF

base64 -w0 fix_kubelet_perm.sh

Use that base64 output in scriptContent.Run this for each worker instance in the node pool.

5. Verify permissions

Use either:

“Run Command” again with a verification script, or
SSH (via Bastion / public IP) and check manually.

Example via SSH:

ssh -i <KEY> opc@<NODE_PUBLIC_IP> \
  'ls -l /var/lib/kubelet/config.yaml'

Expected:

-rw-r--r-- 1 root root ... /var/lib/kubelet/config.yaml

6. Make it persistent for new nodes in the node pool

To avoid this reappearing when nodes are replaced or scaled:

Edit the node pool metadata (cloud-init) via OCI CLI to add a small post‑boot script:

oci ce node-pool update \
  --node-pool-id <NODEPOOL_OCID> \
  --node-metadata '{"user_data":"<BASE64_OF_CLOUD_INIT_SCRIPT>"}'

Where user_data is a base64-encoded cloud‑init file, for example:

#cloud-config
runcmd:
  - [ bash, -c, "if [ -f /var/lib/kubelet/config.yaml ]; then chmod 644 /var/lib/kubelet/config.yaml && chown root:root /var/lib/kubelet/config.yaml; fi" ]

Encode and supply in user_data.

For existing nodes you already fixed, nothing more is needed; for any new or replaced nodes, the cloud‑init will enforce 644 automatically.

If you share your exact OKE worker image/version, I can give the precise kubelet config path and an exact oci command variant for your environment.

Using Python

Below is a practical way to remediate this in Oracle Container Engine for Kubernetes (OKE) using Python:Goal: Ensure kubelet config file(s) on every worker node have permission 644 (rw-r—r—).Typical kubelet config paths on OKE nodes (you may have one or both):

/etc/kubernetes/kubelet.conf
/var/lib/kubelet/config.yaml

Approach 1: Python + Kubernetes API (recommended; no direct SSH)

We will:

Use the Kubernetes Python client.
Create and run a privileged DaemonSet that:
- Mounts the host / filesystem.
- Executes chmod 644 on kubelet config files on each node.
Delete the DaemonSet after it finishes.

1. Install dependencies

pip install kubernetes

Make sure your kubeconfig for OKE is set (e.g. KUBECONFIG=~/.kube/config or default path).

2. Python script to apply a DaemonSet that fixes permissions

from kubernetes import client, config

NAMESPACE = "kube-system"
DAEMONSET_NAME = "fix-kubelet-perms"

daemonset_manifest = {
    "apiVersion": "apps/v1",
    "kind": "DaemonSet",
    "metadata": {
        "name": DAEMONSET_NAME,
        "namespace": NAMESPACE,
        "labels": {"app": DAEMONSET_NAME},
    },
    "spec": {
        "selector": {"matchLabels": {"app": DAEMONSET_NAME}},
        "template": {
            "metadata": {"labels": {"app": DAEMONSET_NAME}},
            "spec": {
                "hostPID": True,
                "hostNetwork": True,
                "tolerations": [
                    {"operator": "Exists"}  # run on all nodes, including masters if any
                ],
                "containers": [
                    {
                        "name": "fix-kubelet-perms",
                        "image": "oraclelinux:8",
                        "securityContext": {
                            "privileged": True
                        },
                        "command": [
                            "bash",
                            "-c",
                            # Adjust file paths if your environment differs
                            """
                            set -e
                            FILES=(
                              /etc/kubernetes/kubelet.conf
                              /var/lib/kubelet/config.yaml
                            )
                            for f in "${FILES[@]}"; do
                              if [ -f "$f" ]; then
                                chmod 644 "$f"
                                echo "Set 644 on $f"
                              else
                                echo "File $f not found, skipping"
                              fi
                            done
                            # sleep briefly so logs can be read, then exit
                            sleep 10
                            """,
                        ],
                        "volumeMounts": [
                            {
                                "name": "rootfs",
                                "mountPath": "/",
                                "readOnly": False
                            }
                        ]
                    }
                ],
                "volumes": [
                    {
                        "name": "rootfs",
                        "hostPath": {"path": "/", "type": "Directory"}
                    }
                ]
            }
        },
        "updateStrategy": {"type": "RollingUpdate"},
    }
}


def main():
    # Load kubeconfig
    config.load_kube_config()

    apps_v1 = client.AppsV1Api()

    # Create DaemonSet
    try:
        apps_v1.create_namespaced_daemon_set(
            namespace=NAMESPACE,
            body=daemonset_manifest
        )
        print(f"DaemonSet {DAEMONSET_NAME} created in namespace {NAMESPACE}.")
    except client.exceptions.ApiException as e:
        if e.status == 409:
            print("DaemonSet already exists, continuing...")
        else:
            raise

    # Optionally wait until at least one pod runs on each node and exits successfully,
    # then delete the DaemonSet. Minimal example: delete after N seconds.
    import time
    time.sleep(120)  # wait ~2 minutes for all nodes

    # Delete DaemonSet (cleanup)
    apps_v1.delete_namespaced_daemon_set(
        name=DAEMONSET_NAME,
        namespace=NAMESPACE,
        body=client.V1DeleteOptions(
            propagation_policy="Foreground"
        )
    )
    print(f"DaemonSet {DAEMONSET_NAME} deleted.")


if __name__ == "__main__":
    main()

What this does:

Runs a privileged pod on every node.
Each pod chmod 644 on kubelet config files if they exist.
Then the script deletes the DaemonSet.

Approach 2: Python + SSH (Paramiko) to each worker node

Use this if you prefer to log in to OKE worker nodes directly (bastion or VCN access required).

1. Install Paramiko

pip install paramiko

2. Python script to fix permissions via SSH

import paramiko

# Replace with your worker node IPs or hostnames
NODES = [
    "10.0.1.10",
    "10.0.1.11",
    # ...
]

SSH_USER = "opc"            # typically 'opc' for Oracle Linux
SSH_KEY_PATH = "/path/to/your/private_key"  # e.g. ~/.ssh/id_rsa

COMMAND = """
FILES=(
  /etc/kubernetes/kubelet.conf
  /var/lib/kubelet/config.yaml
)
for f in "${FILES[@]}"; do
  if [ -f "$f" ]; then
    chmod 644 "$f"
    echo "Set 644 on $f"
  else:
    echo "File $f not found, skipping"
  fi
done
"""

def run_command(host, user, key_path, cmd):
    key = paramiko.RSAKey.from_private_key_file(key_path)
    client = paramiko.SSHClient()
    client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    client.connect(hostname=host, username=user, pkey=key)
    stdin, stdout, stderr = client.exec_command(cmd)
    print(f"=== {host} STDOUT ===")
    print(stdout.read().decode())
    print(f"=== {host} STDERR ===")
    print(stderr.read().decode())
    client.close()

def main():
    for node in NODES:
        print(f"Fixing kubelet permissions on {node}")
        run_command(node, SSH_USER, SSH_KEY_PATH, COMMAND)

if __name__ == "__main__":
    main()

Verification

On any worker node (via SSH or via a debug pod with hostPath), run:

stat -c "%a %n" /etc/kubernetes/kubelet.conf 2>/dev/null
stat -c "%a %n" /var/lib/kubelet/config.yaml 2>/dev/null

You should see permissions 644 where the files exist.

If you share:

exact OKE version,
OS image (Oracle Linux version), I can adjust the file paths and DaemonSet manifest precisely to your environment.

Using Terraform

resource "oci_containerengine_node_pool" "oke_nodepool" {
  # Existing config for your node pool
  compartment_id = "COMPARTMENT_OCID"
  cluster_id     = "OKE_CLUSTER_OCID"
  name           = "OKE_NODE_POOL_NAME"
  kubernetes_version = "K8S_VERSION"
  node_shape         = "NODE_SHAPE"

  node_config_details {
    size = 3

    placement_configs {
      availability_domain = "AD_NAME"
      subnet_id           = "SUBNET_OCID"
    }

    # Ensure this map exists; add/merge the user_data key if you already use node_metadata
    node_metadata = {
      # Cloud-init script to set kubelet config permissions to 0644
      user_data = base64encode(<<-EOF
        #cloud-config
        runcmd:
          - [ chmod, "0644", "/var/lib/kubelet/config.yaml" ]
      EOF
      )
    }
  }

  node_source_details {
    source_type = "image"
    image_id    = "NODE_IMAGE_OCID"
  }

  ssh_public_key = "SSH_PUBLIC_KEY"
}

Changing node_metadata.user_data will cause existing worker nodes in this node pool to be replaced (a rolling recycle), which can briefly disrupt workloads if you lack sufficient pod disruption budgets or capacity.Verification with terraform plan should show an in-place update to oci_containerengine_node_pool.oke_nodepool with a change to node_config_details[0].node_metadata.user_data, and no other resources changed.

OCI Misconfigurations

OCI OKE Kubelet Configuration File Permissions Should Be 644

More Info:

Risk Level

Address

Compliance Standards

Triage and Remediation

Remediation

1. Identify the kubelet config file path in your OKE version

2. Open your OKE Cluster and Node Pool in the Console

3. Add a cloud-init script to enforce 644 permissions

4. Recycle/replace existing worker nodes

5. Verify permissions

1. Identify your node pool and nodes

2. Make sure Oracle Cloud Agent “Run Command” is enabled

3. Prepare the remediation command

4. Run the command on each worker node via OCI CLI

5. Verify permissions

6. Make it persistent for new nodes in the node pool

Approach 1: Python + Kubernetes API (recommended; no direct SSH)

1. Install dependencies

2. Python script to apply a DaemonSet that fixes permissions

Approach 2: Python + SSH (Paramiko) to each worker node

1. Install Paramiko

2. Python script to fix permissions via SSH

Verification

​More Info:

​Risk Level

​Address

​Compliance Standards

​Triage and Remediation

​Remediation

​1. Identify the kubelet config file path in your OKE version

​2. Open your OKE Cluster and Node Pool in the Console

​3. Add a cloud-init script to enforce 644 permissions

​4. Recycle/replace existing worker nodes

​5. Verify permissions

​1. Identify your node pool and nodes

​2. Make sure Oracle Cloud Agent “Run Command” is enabled

​3. Prepare the remediation command

​4. Run the command on each worker node via OCI CLI

​5. Verify permissions

​6. Make it persistent for new nodes in the node pool

​Approach 1: Python + Kubernetes API (recommended; no direct SSH)

​1. Install dependencies

​2. Python script to apply a DaemonSet that fixes permissions

​Approach 2: Python + SSH (Paramiko) to each worker node

​1. Install Paramiko

​2. Python script to fix permissions via SSH

​Verification

More Info:

Risk Level

Address

Compliance Standards

Triage and Remediation

Remediation

1. Identify the kubelet config file path in your OKE version

2. Open your OKE Cluster and Node Pool in the Console

3. Add a cloud-init script to enforce 644 permissions

4. Recycle/replace existing worker nodes

5. Verify permissions

1. Identify your node pool and nodes

2. Make sure Oracle Cloud Agent “Run Command” is enabled

3. Prepare the remediation command

4. Run the command on each worker node via OCI CLI

5. Verify permissions

6. Make it persistent for new nodes in the node pool

Approach 1: Python + Kubernetes API (recommended; no direct SSH)

1. Install dependencies

2. Python script to apply a DaemonSet that fixes permissions

Approach 2: Python + SSH (Paramiko) to each worker node

1. Install Paramiko

2. Python script to fix permissions via SSH

Verification