> ## Documentation Index
> Fetch the complete documentation index at: https://cloudanix.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# OCI OKE Node Pools Should Use Private Nodes

### More Info:

Worker nodes should be deployed in private subnets without public IPs so they are not directly reachable from the internet. Combine with a NAT/Service Gateway for outbound traffic.

### Risk Level

High

### Address

Compliance, Security

### Compliance Standards

* CIS OKE

### Triage and Remediation

<Tabs>
  <Tab title="Remediation">
    ### Remediation

    <AccordionGroup>
      <Accordion title="Using Console" defaultOpen="true">
        Below are the remediation steps in the **OCI Console** to ensure OKE node pools use private nodes (no public IPs, only private subnets).

        ***

        ## 1. Prerequisites: Ensure You Have Private Subnets

        1. In the OCI Console, go to **Networking → Virtual Cloud Networks**.
        2. Select the **VCN** used by your OKE cluster.
        3. Under **Subnets**, verify you have:
           * One or more **private subnets** (no Internet Gateway in their route table, typically using a **NAT Gateway** for outbound access).
           * Proper **security lists / NSGs** to allow:
             * Node-to-node traffic
             * Node-to-control-plane traffic (per OKE docs)
        4. If you don’t have a private subnet:
           1. Click **Create Subnet**.
           2. Choose:
              * **Private Subnet**: Yes
              * Associate with the **VCN** used by the OKE cluster
              * Attach route table that sends `0.0.0.0/0` to a **NAT Gateway** (not Internet Gateway).
           3. Save the subnet.

        ***

        ## 2. Confirm Current Node Pool Configuration

        1. Go to **Developer Services → Kubernetes Clusters (OKE)**.
        2. Select your **Cluster**.
        3. Click **Node Pools**.
        4. For each node pool:
           * Check the **Subnets** column.
           * Open the node pool, and in the **Details** panel verify:
             * Whether it is using **public subnets**.
             * Whether nodes have **public IP addresses** (this is typically controlled by the subnet type and advanced node settings).

        You cannot “flip” an existing pool from public to private; you must create a new private node pool and migrate workloads.

        ***

        ## 3. Create a New Private Node Pool

        1. On the **Node Pools** tab of the cluster, click **Create node pool**.
        2. Fill in:
           * **Name**: e.g., `private-node-pool`.
           * **Kubernetes version**: same as the cluster or as required.
           * **Node shape** and **Image**: match or improve upon current node pool.
        3. Under **Placement**:
           * Choose the same **Availability Domains** / **Fault Domains** as required.
           * In **Subnets**, select **only private subnets** (created in step 1).
        4. Under **Node configuration** / **Node metadata & networking** (wording may vary slightly):
           * Ensure **Assign public IP addresses** (or similar option) is **unchecked/disabled**.
           * Confirm that **Network security groups** or **Security lists** allow needed traffic.
        5. Set the desired **Node count**.
        6. Click **Create** and wait for the node pool status to become **Active** and nodes to be **Ready**.

        ***

        ## 4. Migrate Workloads to the Private Node Pool

        Use `kubectl` (or OKE Cloud Shell) for this part.

        1. Label the new node pool’s nodes (optional but helpful):
           ```bash theme={null}
           kubectl get nodes -o wide
           # Identify new nodes (by name / subnet)
           kubectl label node <new-node-name> nodepool=private
           ```
        2. If your workloads use node selectors, affinities, or taints:
           * Update **Deployments/StatefulSets/DaemonSets** to schedule to the new nodes (e.g., `nodeSelector: { nodepool: private }`).
        3. Gradually cordon and drain old (public) nodes:
           ```bash theme={null}
           # For each old node
           kubectl cordon <old-node-name>
           kubectl drain <old-node-name> --ignore-daemonsets --delete-emptydir-data
           ```
        4. Verify all Pods are running on the new private nodes:
           ```bash theme={null}
           kubectl get pods -A -o wide
           ```

        ***

        ## 5. Delete the Old Public Node Pool

        1. In OCI Console, return to the **Node Pools** tab for your cluster.
        2. Confirm:
           * No workloads are running on the old node pool’s nodes.
        3. Click on the old (public) node pool.
        4. Click **Delete** and confirm.

        ***

        ## 6. Validate Private-Only Access

        1. In **Compute → Instances**, locate instances from the new node pool:
           * Confirm each instance has **no public IP**.
           * Confirm they are placed in your **private subnet**.
        2. Confirm cluster behavior:
           * Pods and services function normally.
           * Outbound access (e.g., to Docker Hub, OCI Registry) works via the **NAT Gateway**, not Internet Gateway.
        3. If you use a **bastion host** or **VPN / FastConnect**, make sure admins can still reach nodes if required (SSH via private IP only).

        ***

        **Result:** Your OKE node pools now use private nodes (in private subnets, no public IPs) as required.
      </Accordion>

      <Accordion title="Using CLI">
        To use **private nodes** in an OKE node pool via **OCI CLI**, you must:

        1. **Disable public IP assignment on the node pool**, and
        2. **Ensure the node pool uses private subnets** (no internet gateway attached).

        Below are the key steps and example commands.

        ***

        ## 1. Prerequisites

        * OCI CLI configured with appropriate privileges.
        * OKE Cluster OCID.
        * Private subnet OCIDs in the cluster VCN (subnets should not be associated to an Internet Gateway; use NAT Gateway for outbound if needed).

        ***

        ## 2. Identify the Node Pool

        ```bash theme={null}
        # List node pools for a cluster
        oci ce node-pool list \
          --compartment-id <COMPARTMENT_OCID> \
          --cluster-id <CLUSTER_OCID> \
          --all

        # Get details of a specific node pool
        oci ce node-pool get \
          --node-pool-id <NODE_POOL_OCID> \
          --query 'data'
        ```

        Check:

        * `nodeConfigDetails.isPublicIpEnabled`
        * `nodeConfigDetails.placementConfigs[].subnetId`

        ***

        ## 3. Update Node Pool to Disable Public IPs

        `isPublicIpEnabled` must be `false`.

        Example:

        ```bash theme={null}
        oci ce node-pool update \
          --node-pool-id <NODE_POOL_OCID> \
          --node-config-details '{
            "size": <DESIRED_SIZE>,
            "isPublicIpEnabled": false,
            "placementConfigs": [
              {
                "availabilityDomain": "<AD_NAME>",
                "subnetId": "<PRIVATE_SUBNET_OCID>"
              }
            ]
          }' \
          --force \
          --wait-for-state SUCCEEDED
        ```

        Notes:

        * `size` must match or be set to your desired node count.
        * Ensure `subnetId` references a **private subnet**.
        * This setting applies to new/replaced nodes; existing ones keep their current IPs until cycled.

        ***

        ## 4. Move Nodes Fully to Private Subnets (If Needed)

        If your current nodes are in public subnets or already have public IPs, you must replace them after updating the pool:

        ```bash theme={null}
        # List nodes in the node pool
        oci ce node-pool get \
          --node-pool-id <NODE_POOL_OCID> \
          --query 'data.nodes[*].id' \
          --output table

        # Option 1: Replace all nodes in the pool
        oci ce node-pool replace-nodes \
          --node-pool-id <NODE_POOL_OCID> \
          --nodes '[{"id": "<NODE_OCID_1>"},{"id":"<NODE_OCID_2>"}]' \
          --wait-for-state SUCCEEDED
        ```

        Before replacing, you should cordon & drain via `kubectl` to avoid workload disruption.

        ***

        ## 5. (Alternative) Create a New Private Node Pool

        If you prefer not to modify the existing pool:

        ```bash theme={null}
        oci ce node-pool create \
          --compartment-id <COMPARTMENT_OCID> \
          --cluster-id <CLUSTER_OCID> \
          --name "private-node-pool" \
          --kubernetes-version <K8S_VERSION> \
          --node-image-id <IMAGE_OCID> \
          --node-shape <SHAPE> \
          --node-config-details '{
            "size": <DESIRED_SIZE>,
            "isPublicIpEnabled": false,
            "placementConfigs": [
              {
                "availabilityDomain": "<AD_NAME>",
                "subnetId": "<PRIVATE_SUBNET_OCID>"
              }
            ]
          }' \
          --wait-for-state SUCCEEDED
        ```

        Then:

        1. Drain workloads from the old node pool.
        2. Delete the old (public) node pool:

        ```bash theme={null}
        oci ce node-pool delete \
          --node-pool-id <OLD_NODE_POOL_OCID> \
          --force \
          --wait-for-state SUCCEEDED
        ```

        ***

        **Compliance check:**\
        After changes, confirm:

        ```bash theme={null}
        oci ce node-pool get \
          --node-pool-id <NODE_POOL_OCID> \
          --query 'data.nodeConfigDetails.{isPublicIpEnabled:isPublicIpEnabled,subnets:placementConfigs[*].subnetId}'
        ```

        `isPublicIpEnabled` should be `false`, and `subnets` should be private subnets only.
      </Accordion>

      <Accordion title="Using Python">
        To use **private nodes** in an OCI OKE node pool, you must ensure that:

        1. The node pool’s worker nodes are placed in **private subnets** (no Internet Gateway route).
        2. Existing node pools that use public subnets are **migrated** to new node pools in private subnets.

        Below is a step‑by‑step outline, followed by a Python example using the OCI SDK.

        ***

        ## 1. Prerequisites

        * Python 3.x
        * `oci` SDK installed:
          ```bash theme={null}
          pip install oci
          ```
        * OCI config file set up (`~/.oci/config`) and a profile (e.g., `DEFAULT`) with permissions for:
          * `ContainerEngineClient` (OKE)
          * `VirtualNetworkClient` (networking)

        ***

        ## 2. Network Setup (ensure private subnets)

        1. Identify / create a **VCN**.
        2. Create or identify a **private subnet**:
           * Subnet is in the same VCN and AD as the cluster.
           * Subnet uses a **route table without an Internet Gateway** (use a NAT gateway or Service Gateway for outbound if needed).
           * Proper security lists / NSGs are applied.

        Python (optional) to verify that a subnet is private (no route to Internet Gateway):

        ```python theme={null}
        import oci

        config = oci.config.from_file("~/.oci/config", "DEFAULT")
        vn_client = oci.core.VirtualNetworkClient(config)

        def is_private_subnet(subnet_id):
            subnet = vn_client.get_subnet(subnet_id).data
            rt = vn_client.get_route_table(subnet.route_table_id).data
            for rule in rt.route_rules:
                if rule.network_entity_id and 'internetgateway' in rule.network_entity_id.lower():
                    return False
            return True
        ```

        Create a **new private subnet** if needed and use its `subnet_id` for the node pool.

        ***

        ## 3. Create a New Private Node Pool with Python

        You cannot “flip” a public node pool to private directly; you typically:

        * Create a new node pool using **private subnets**.
        * Cordon & drain workloads from the old node pool.
        * Delete the old node pool.

        ```python theme={null}
        import oci
        from oci.container_engine import ContainerEngineClient
        from oci.container_engine.models import (
            CreateNodePoolDetails,
            NodePoolNodeConfigDetails,
            NodePoolPlacementConfigDetails,
            KeyValue
        )

        # --- CONFIG ---
        config = oci.config.from_file("~/.oci/config", "DEFAULT")
        compartment_id     = "<COMPARTMENT_OCID>"
        cluster_id         = "<OKE_CLUSTER_OCID>"
        node_pool_name     = "private-node-pool"
        k8s_version        = "<KUBERNETES_VERSION>"  # e.g. "v1.29.1"
        node_shape         = "VM.Standard.E4.Flex"
        ocpus              = 2
        memory_gbs         = 16
        node_image_id      = "<NODE_IMAGE_OCID>"    # use an OKE node image
        private_subnet_id  = "<PRIVATE_SUBNET_OCID>"  # ensure this is a *private* subnet
        node_pool_size     = 3
        ssh_public_key     = "<YOUR_SSH_PUBLIC_KEY>"  # for node access if needed

        ce_client = ContainerEngineClient(config)

        # Optional: GPU/shape config, placement config, etc.
        placement_configs = [
            NodePoolPlacementConfigDetails(
                availability_domain="<AD_NAME>",  # e.g. "kIdk:US-ASHBURN-AD-1"
                subnet_id=private_subnet_id
            )
        ]

        node_config = NodePoolNodeConfigDetails(
            size=node_pool_size,
            placement_configs=placement_configs,
            # If your tenancy uses subnet_ids directly instead:
            # subnet_ids=[private_subnet_id],
            # nsg_ids=[...],  # optional
        )

        create_details = CreateNodePoolDetails(
            name=node_pool_name,
            compartment_id=compartment_id,
            cluster_id=cluster_id,
            kubernetes_version=k8s_version,
            node_shape=node_shape,
            node_source_details={
                "imageId": node_image_id,
                "sourceType": "IMAGE"
            },
            node_config_details=node_config,
            node_shape_config={
                "ocpus": ocpus,
                "memoryInGBs": memory_gbs
            },
            ssh_public_key=ssh_public_key,
            initial_node_labels=[
                KeyValue(key="environment", value="prod")
            ]
        )

        response = ce_client.create_node_pool(create_details)
        new_node_pool_id = response.data.id
        print("Created private node pool:", new_node_pool_id)
        ```

        **Key point:** Using a **private subnet** (no Internet Gateway in its route table) ensures nodes get **only private IPs**. That is what makes the node pool “private” in OKE.

        ***

        ## 4. Migrate Workloads and Remove Old Public Node Pool

        Once the new private node pool is active:

        1. Update your cluster’s node selectors / taints so workloads can run on the new nodes.
        2. Cordon and drain nodes in the old (public) node pool from `kubectl`:
           ```bash theme={null}
           kubectl cordon <old-node>
           kubectl drain <old-node> --ignore-daemonsets --delete-emptydir-data
           ```
        3. Delete the old node pool via Python:

        ```python theme={null}
        old_node_pool_id = "<OLD_NODE_POOL_OCID>"

        delete_resp = ce_client.delete_node_pool(old_node_pool_id)
        print("Deleting old node pool:", old_node_pool_id)
        ```

        ***

        ## 5. Optional: Enforce Private-Only Node Pools (Programmatically)

        If you want to ensure that new node pools are always private, you can add a check in your automation that:

        * Validates `subnet_id` is private (no IGW route) using the helper in step 2.
        * Rejects or fails creation if any subnet is public.

        This completes the Python-based remediation: all OKE node pools use **private subnets**, so nodes are private-only.
      </Accordion>

      <Accordion title="Using Terraform">
        ```hcl theme={null}
        # Private subnet for OKE worker nodes (no public IPs)
        resource "oci_core_subnet" "OKE_PRIVATE_SUBNET" {
          compartment_id = var.COMPartment_OCID          # replace with your compartment OCID variable or value
          vcn_id         = oci_core_vcn.OKE_VCN.id       # replace with your VCN resource or ID
          cidr_block     = "10.0.1.0/24"                 # replace with your desired CIDR

          display_name   = "oke-private-subnet"

          # Key setting: do not allow public IPs on VNICs in this subnet
          prohibit_public_ip_on_vnic = true

          # Ensure the route table only has NAT/Service Gateway for outbound,
          # and no Internet Gateway route.
          route_table_id = oci_core_route_table.OKE_PRIVATE_RT.id

          dns_label = "okepriv"                          # adjust as needed
        }

        # Example route table for the private subnet (NAT / Service Gateway only)
        resource "oci_core_route_table" "OKE_PRIVATE_RT" {
          compartment_id = var.COMPartment_OCID
          vcn_id         = oci_core_vcn.OKE_VCN.id
          display_name   = "oke-private-rt"

          # NAT Gateway for outbound internet access from private nodes
          route_rules {
            destination       = "0.0.0.0/0"
            destination_type  = "CIDR_BLOCK"
            network_entity_id = oci_core_nat_gateway.NAT_GATEWAY.id
          }

          # Optional: Service Gateway for OCI services
          # route_rules {
          #   destination       = "all-services-in-oracle-services-network"
          #   destination_type  = "SERVICE_CIDR_BLOCK"
          #   network_entity_id = oci_core_service_gateway.SERVICE_GATEWAY.id
          # }
        }

        # OKE Node Pool using the private subnet
        resource "oci_containerengine_node_pool" "OKE_NODE_POOL" {
          compartment_id = var.COMPartment_OCID
          cluster_id     = oci_containerengine_cluster.OKE_CLUSTER.id

          name          = "oke-node-pool-private"
          kubernetes_version = "v1.29.1"                # replace with your desired version
          node_shape         = "VM.Standard3.Flex"      # replace with your shape

          node_config_details {
            size = 3                                    # replace with your desired node count

            placement_configs {
              availability_domain = data.oci_identity_availability_domains.ADs.availability_domains[0].name
              subnet_id           = oci_core_subnet.OKE_PRIVATE_SUBNET.id
            }
          }

          node_source_details {
            source_type = "IMAGE"
            image_id    = data.oci_core_images.OKE_NODE_IMAGE.images[0].id
          }

          node_shape_config {
            ocpus         = 2                           # adjust as needed
            memory_in_gbs = 16
          }
        }
        ```

        Changing an existing node pool to use a different (private) subnet will force replacement of the node pool (nodes are recreated), and changing `prohibit_public_ip_on_vnic` on an existing subnet forces subnet replacement; plan this to avoid outages.

        After updating, `terraform plan` should show:

        * `prohibit_public_ip_on_vnic = true` being set on the worker subnet (or a new private subnet created), and
        * the `oci_containerengine_node_pool` using that private subnet ID in its `placement_configs`.
      </Accordion>
    </AccordionGroup>
  </Tab>
</Tabs>