K8s on Oracle Cloud III - Dual Terraform configuration - Infra and App deployments

As an alternative to the approach followed in K8S on Oracle Cloud II - Single Terraform Config for Cluster and Datadog Observability, we'll follow the two Terraform state deployment here because it brings different advantages, one of which is you do not have to resort to workarounds like null resources.

We'll have a clean experience setting up the cluster, generate kubeconfig, then move to the second deployment module where we deploy Datadog, the load balancer, and future apps.

We will share in this article the general full layout. For the load balancer and cert manager, check upcoming articles.

Tree

This is the tree of the project. We will be using two Terraform modules, one for the cluster and one for the deployments.

.
├── README.md
├── cluster
│   ├── cert_manager.tf
│   ├── k8s.tf
│   ├── main.tf
│   ├── terraform.tfstate
│   ├── terraform.tfstate.backup
│   └── variables.tf
├── deployments
│   ├── cert.tf
│   ├── dd.tf
│   ├── helm_releases.tf
│   ├── lb.tf
│   ├── main.tf
│   ├── terraform.tfstate
│   ├── terraform.tfstate.backup
│   └── variables.tf
└── helm
    ├── archive
    │   └── datadog-agent.yaml
    ├── bye-world
    │   ├── Chart.lock
    │   ├── Chart.yaml
    │   ├── charts
    │   │   └── common-templates-0.1.1.tgz
    │   ├── templates
    │   │   ├── NOTES.txt
    │   │   ├── deployment.yml
    │   │   ├── hpa.yaml
    │   │   ├── ingress.yaml
    │   │   ├── service.yaml
    │   │   └── serviceaccount.yaml
    │   └── values.yaml
    ├── charts
    │   ├── cluster-issuer
    │   │   ├── Chart.yaml
    │   │   ├── templates
    │   │   │   └── cluster-issuer.yaml
    │   │   └── values.yaml
    │   └── common-templates
    │       ├── Chart.yaml
    │       └── templates
    │           ├── _deployment.yaml
    │           ├── _helpers.tpl
    │           ├── _hpa.yaml
    │           ├── _ingress.yaml
    │           ├── _service.yaml
    │           └── _serviceaccount.yaml
    └── hello-world
        ├── Chart.lock
        ├── Chart.yaml
        ├── charts
        │   └── common-templates-0.1.2.tgz
        ├── templates
        │   ├── NOTES.txt
        │   ├── deployment.yml
        │   ├── hpa.yaml
        │   ├── ingress.yaml
        │   ├── service.yaml
        │   └── serviceaccount.yaml
        └── values.yaml

1. Cluster

The cluster module is the one that will create the cluster. As I progress on this, you can see the cert manager already deployed to grab Let's Encrypt certificates for the load balancer.

main.tf - Cluster

terraform {
  required_providers {
    oci = {
      source  = "oracle/oci"
      version = "~> 7.0.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.36.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.17.0"
    }

  }
}

provider "oci" {
}

variables.tf

This file contains the variables for the cluster module, including the OCI provider configuration.

// Vars
variable "cluster_name" {
  type        = string
  default     = "k8s-001"
  description = "The name of the OKE cluster."
}
// OCI provider configuration
variable "TENANCY_OCID" {
  type        = string
  description = "The OCID of the tenancy."
  sensitive   = true // Mark as sensitive
}

variable "USER_OCID" {
  type        = string
  description = "The OCID of the user."
  sensitive   = true // Mark as sensitive
}

variable "FINGERPRINT" {
  type        = string
  description = "The fingerprint of the API key."
  sensitive   = true // Mark as sensitive
}

variable "PRIVATE_KEY_PATH" {
  type        = string
  description = "The path to the OCI API private key file."
  sensitive   = true // Mark as sensitive
}

variable "REGION" {
  type        = string
  description = "The OCI region to operate in."
}

K8s.tf

Most of the code here is generated from the OCI Resource Manager Stack if you followed K8S on Oracle Cloud - I.

locals {
  tenancy_ocid                              = var.TENANCY_OCID
  kubernetes_version                        = "v1.32.1"
  cluster_name                              = "k8s-001" # Used for general naming, tags
  vcn_dns_label                             = "k8s001"  # Specific for VCN dns_label to avoid replacement
  node_pool_name                            = "pool1"
  node_subnet_cidr_block                    = "10.0.10.0/24"
  kubernetes_api_endpoint_subnet_cidr_block = "10.0.0.0/28"
  service_gateway_service_id                = "ocid1.service.oc1.example-region-1.replaceuniqueserviceid"
  node_pool_image_id                        = "ocid1.image.oc1.example-region-1.replaceuniqueimageid"
  availability_domain                       = "EXAMPLE:REGION-1-AD-1"
  node_shape                                = "VM.Standard.A1.Flex"
  node_memory_in_gbs                        = "6"
  node_ocpus                                = "1"
  display_name_suffix                       = "examplesuffix"
  resource_name_prefix                      = "oke"
}

resource "oci_core_vcn" "generated_oci_core_vcn" {
  cidr_block     = "10.0.0.0/16"
  compartment_id = local.tenancy_ocid
  display_name   = "${local.resource_name_prefix}-vcn-quick-${local.cluster_name}-${local.display_name_suffix}"
  dns_label      = local.vcn_dns_label # Using specific variable to keep original value
}

resource "oci_core_internet_gateway" "generated_oci_core_internet_gateway" {
  compartment_id = local.tenancy_ocid
  display_name   = "${local.resource_name_prefix}-igw-quick-${local.cluster_name}-${local.display_name_suffix}"
  enabled        = "true"
  vcn_id         = oci_core_vcn.generated_oci_core_vcn.id
}

resource "oci_core_nat_gateway" "generated_oci_core_nat_gateway" {
  compartment_id = local.tenancy_ocid
  display_name   = "${local.resource_name_prefix}-ngw-quick-${local.cluster_name}-${local.display_name_suffix}"
  vcn_id         = oci_core_vcn.generated_oci_core_vcn.id
}

resource "oci_core_service_gateway" "generated_oci_core_service_gateway" {
  compartment_id = local.tenancy_ocid
  display_name   = "${local.resource_name_prefix}-sgw-quick-${local.cluster_name}-${local.display_name_suffix}"
  services {
    service_id = local.service_gateway_service_id
  }
  vcn_id = oci_core_vcn.generated_oci_core_vcn.id
}

resource "oci_core_route_table" "generated_oci_core_route_table" {
  compartment_id = local.tenancy_ocid
  display_name   = "${local.resource_name_prefix}-private-routetable-${local.cluster_name}-${local.display_name_suffix}"
  route_rules {
    description       = "traffic to the internet"
    destination       = "0.0.0.0/0"
    destination_type  = "CIDR_BLOCK"
    network_entity_id = oci_core_nat_gateway.generated_oci_core_nat_gateway.id
  }
  route_rules {
    description       = "traffic to OCI services"
    destination       = "all-example-region-1-services-in-oracle-services-network"
    destination_type  = "SERVICE_CIDR_BLOCK"
    network_entity_id = oci_core_service_gateway.generated_oci_core_service_gateway.id
  }
  vcn_id = oci_core_vcn.generated_oci_core_vcn.id
}

resource "oci_core_subnet" "service_lb_subnet" {
  cidr_block                 = "10.0.20.0/24"
  compartment_id             = local.tenancy_ocid
  display_name               = "${local.resource_name_prefix}-svclbsubnet-quick-${local.cluster_name}-${local.display_name_suffix}-regional"
  dns_label                  = "dnslabel7777" 
  prohibit_public_ip_on_vnic = "false"
  route_table_id             = oci_core_default_route_table.generated_oci_core_default_route_table.id
  security_list_ids          = [oci_core_vcn.generated_oci_core_vcn.default_security_list_id]
  vcn_id                     = oci_core_vcn.generated_oci_core_vcn.id
}

resource "oci_core_subnet" "node_subnet" {
  cidr_block                 = local.node_subnet_cidr_block
  compartment_id             = local.tenancy_ocid
  display_name               = "${local.resource_name_prefix}-nodesubnet-quick-${local.cluster_name}-${local.display_name_suffix}-regional"
  dns_label                  = "dnslabel8888" 
  prohibit_public_ip_on_vnic = "true"
  route_table_id             = oci_core_route_table.generated_oci_core_route_table.id
  security_list_ids          = [oci_core_security_list.node_sec_list.id]
  vcn_id                     = oci_core_vcn.generated_oci_core_vcn.id
}

resource "oci_core_subnet" "kubernetes_api_endpoint_subnet" {
  cidr_block                 = local.kubernetes_api_endpoint_subnet_cidr_block
  compartment_id             = local.tenancy_ocid
  display_name               = "${local.resource_name_prefix}-k8sApiEndpoint-subnet-quick-${local.cluster_name}-${local.display_name_suffix}-regional"
  dns_label                  = "dnslabel9999" // This seems unique, keeping as is
  prohibit_public_ip_on_vnic = "false"
  route_table_id             = oci_core_default_route_table.generated_oci_core_default_route_table.id
  security_list_ids          = [oci_core_security_list.kubernetes_api_endpoint_sec_list.id]
  vcn_id                     = oci_core_vcn.generated_oci_core_vcn.id
}

resource "oci_core_default_route_table" "generated_oci_core_default_route_table" {
  display_name = "${local.resource_name_prefix}-public-routetable-${local.cluster_name}-${local.display_name_suffix}"
  route_rules {
    description       = "traffic to/from internet"
    destination       = "0.0.0.0/0"
    destination_type  = "CIDR_BLOCK"
    network_entity_id = oci_core_internet_gateway.generated_oci_core_internet_gateway.id
  }
  manage_default_resource_id = oci_core_vcn.generated_oci_core_vcn.default_route_table_id
}

resource "oci_core_security_list" "service_lb_sec_list" {
  compartment_id = local.tenancy_ocid
  display_name   = "${local.resource_name_prefix}-svclbseclist-quick-${local.cluster_name}-${local.display_name_suffix}"
  vcn_id         = oci_core_vcn.generated_oci_core_vcn.id
}

resource "oci_core_security_list" "node_sec_list" {
  compartment_id = local.tenancy_ocid
  display_name   = "${local.resource_name_prefix}-nodeseclist-quick-${local.cluster_name}-${local.display_name_suffix}"
  egress_security_rules {
    description      = "Allow pods on one worker node to communicate with pods on other worker nodes"
    destination      = local.node_subnet_cidr_block
    destination_type = "CIDR_BLOCK"
    protocol         = "all"
    stateless        = "false"
  }
  egress_security_rules {
    description      = "Access to Kubernetes API Endpoint"
    destination      = local.kubernetes_api_endpoint_subnet_cidr_block
    destination_type = "CIDR_BLOCK"
    protocol         = "6"
    stateless        = "false"
  }
  egress_security_rules {
    description      = "Kubernetes worker to control plane communication"
    destination      = local.kubernetes_api_endpoint_subnet_cidr_block
    destination_type = "CIDR_BLOCK"
    protocol         = "6"
    stateless        = "false"
  }
  egress_security_rules {
    description      = "Path discovery"
    destination      = local.kubernetes_api_endpoint_subnet_cidr_block
    destination_type = "CIDR_BLOCK"
    icmp_options {
      code = "4"
      type = "3"
    }
    protocol  = "1"
    stateless = "false"
  }
  egress_security_rules {
    description      = "Allow nodes to communicate with OKE to ensure correct start-up and continued functioning"
    destination      = "all-example-region-1-services-in-oracle-services-network"
    destination_type = "SERVICE_CIDR_BLOCK"
    protocol         = "6"
    stateless        = "false"
  }
  egress_security_rules {
    description      = "ICMP Access from Kubernetes Control Plane"
    destination      = "0.0.0.0/0"
    destination_type = "CIDR_BLOCK"
    icmp_options {
      code = "4"
      type = "3"
    }
    protocol  = "1"
    stateless = "false"
  }
  egress_security_rules {
    description      = "Worker Nodes access to Internet"
    destination      = "0.0.0.0/0"
    destination_type = "CIDR_BLOCK"
    protocol         = "all"
    stateless        = "false"
  }
  ingress_security_rules {
    description = "Allow pods on one worker node to communicate with pods on other worker nodes"
    protocol    = "all"
    source      = local.node_subnet_cidr_block
    stateless   = "false"
  }
  ingress_security_rules {
    description = "Path discovery"
    icmp_options {
      code = "4"
      type = "3"
    }
    protocol  = "1"
    source    = local.kubernetes_api_endpoint_subnet_cidr_block
    stateless = "false"
  }
  ingress_security_rules {
    description = "TCP access from Kubernetes Control Plane"
    protocol    = "6"
    source      = local.kubernetes_api_endpoint_subnet_cidr_block
    stateless   = "false"
  }
  ingress_security_rules {
    description = "Inbound SSH traffic to worker nodes"
    protocol    = "6"
    source      = "0.0.0.0/0"
    stateless   = "false"
  }
  vcn_id = oci_core_vcn.generated_oci_core_vcn.id
}

resource "oci_core_security_list" "kubernetes_api_endpoint_sec_list" {
  compartment_id = local.tenancy_ocid
  display_name   = "${local.resource_name_prefix}-k8sApiEndpoint-quick-${local.cluster_name}-${local.display_name_suffix}"
  egress_security_rules {
    description      = "Allow Kubernetes Control Plane to communicate with OKE"
    destination      = "all-example-region-1-services-in-oracle-services-network"
    destination_type = "SERVICE_CIDR_BLOCK"
    protocol         = "6"
    stateless        = "false"
  }
  egress_security_rules {
    description      = "All traffic to worker nodes"
    destination      = local.node_subnet_cidr_block
    destination_type = "CIDR_BLOCK"
    protocol         = "6"
    stateless        = "false"
  }
  egress_security_rules {
    description      = "Path discovery"
    destination      = local.node_subnet_cidr_block
    destination_type = "CIDR_BLOCK"
    icmp_options {
      code = "4"
      type = "3"
    }
    protocol  = "1"
    stateless = "false"
  }
  ingress_security_rules {
    description = "External access to Kubernetes API endpoint"
    protocol    = "6"
    source      = "0.0.0.0/0"
    stateless   = "false"
  }
  ingress_security_rules {
    description = "Kubernetes worker to Kubernetes API endpoint communication"
    protocol    = "6"
    source      = local.node_subnet_cidr_block
    stateless   = "false"
  }
  ingress_security_rules {
    description = "Kubernetes worker to control plane communication"
    protocol    = "6"
    source      = local.node_subnet_cidr_block
    stateless   = "false"
  }
  ingress_security_rules {
    description = "Path discovery"
    icmp_options {
      code = "4"
      type = "3"
    }
    protocol  = "1"
    source    = local.node_subnet_cidr_block
    stateless = "false"
  }
  vcn_id = oci_core_vcn.generated_oci_core_vcn.id
}

resource "oci_containerengine_cluster" "generated_oci_containerengine_cluster" {
  cluster_pod_network_options {
    cni_type = "OCI_VCN_IP_NATIVE"
  }
  compartment_id = local.tenancy_ocid
  endpoint_config {
    is_public_ip_enabled = "true"
    subnet_id            = oci_core_subnet.kubernetes_api_endpoint_subnet.id
  }
  freeform_tags = {
    "OKEclusterName" = local.cluster_name
  }
  kubernetes_version = local.kubernetes_version
  name               = local.cluster_name
  options {
    admission_controller_options {
      is_pod_security_policy_enabled = "false"
    }
    persistent_volume_config {
      freeform_tags = {
        "OKEclusterName" = local.cluster_name
      }
    }
    service_lb_config {
      freeform_tags = {
        "OKEclusterName" = local.cluster_name
      }
    }
    service_lb_subnet_ids = [oci_core_subnet.service_lb_subnet.id]
  }
  type   = "BASIC_CLUSTER"
  vcn_id = oci_core_vcn.generated_oci_core_vcn.id
}

resource "oci_containerengine_node_pool" "create_node_pool_details0" {
  cluster_id     = oci_containerengine_cluster.generated_oci_containerengine_cluster.id
  compartment_id = local.tenancy_ocid
  freeform_tags = {
    "OKEnodePoolName" = local.node_pool_name
  }
  initial_node_labels {
    key   = "name"
    value = local.cluster_name # Assuming node pool label relates to cluster name
  }
  kubernetes_version = local.kubernetes_version
  name               = local.node_pool_name
  node_config_details {
    freeform_tags = {
      "OKEnodePoolName" = local.node_pool_name
    }
    node_pool_pod_network_option_details {
      cni_type       = "OCI_VCN_IP_NATIVE"
      pod_subnet_ids = [oci_core_subnet.node_subnet.id] // Reference to the new pod subnet
    }
    placement_configs {
      availability_domain = local.availability_domain
      subnet_id           = oci_core_subnet.node_subnet.id
    }
    size = "4" // Keeping size as is, can be variabilized if needed
  }
  node_eviction_node_pool_settings {
    eviction_grace_duration = "PT1H" // Keeping as is
  }
  node_shape = local.node_shape
  node_shape_config {
    memory_in_gbs = local.node_memory_in_gbs
    ocpus         = local.node_ocpus
  }
  node_source_details {
    image_id    = local.node_pool_image_id
    source_type = "IMAGE"
  }
}

2. Generate KubeConfig

Generate via OCI CLI.

oci ce cluster create-kubeconfig \
  --cluster-id ocid1.cluster.oc1.example-region-1.replaceuniqueclusterid \
  --file $HOME/.kube/config \
  --region example-region-1 \
  --token-version 2.0.0 \
  --kube-endpoint PUBLIC_ENDPOINT

3. Deployments

This brings us to the second Terraform configuration meant to track application deployments within the cluster.

Here we'll already have access to kubeconfig, which eases up setting up the providers for Kubernetes and Helm, without the chicken and egg problem where the kubeconfig is not available yet. Terraform loads providers with the dummy kubeconfig file early on before their creation that we covered in K8S on Oracle Cloud II - Single Terraform Config for Cluster and Datadog Observability.

main.tf - deployments

This file contains the main configuration for the deployments module, including the OCI, K8s, and Helm providers configuration.

terraform {
  required_providers {
    oci = {
      source  = "oracle/oci"
      version = "~> 7.0.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.36.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.17.0"
    }

  }
}

provider "oci" {
}

provider "kubernetes" {
  config_path = pathexpand("~/.kube/config") # auto‑detect current context
}

provider "helm" {
  kubernetes {
    config_path = pathexpand("~/.kube/config")
  }
}

variables.tf - deployments

This file contains the variables for the deployments module.

variable "datadog_secret_name" {
  description = "Name of the secret for Datadog API key"
  type        = string
  default     = "datadog-secret"
  sensitive   = true
}

variable "datadog_api_key" {
  description = "Datadog API key used for agent authentication"
  type        = string
  sensitive   = true
}

variable "kubernetes_namespace_apps" {
  description = "Apps Kubernetes namespace for Helm releases"
  type        = string
  default     = "apps"
}

variable "cluster_name" {
  type        = string
  default     = "cluster-001"
  description = "The name of the Kubernetes cluster."
}

variable "TENANCY_ID" {
  type        = string
  description = "The ID of the tenancy. Ex: ocid1.tenancy.oc1..exampleuniqueid"
  sensitive   = true
}

variable "USER_ID" {
  type        = string
  description = "The ID of the user. Ex: ocid1.user.oc1..exampleuniqueid"
  sensitive   = true
}

variable "API_KEY_FINGERPRINT" {
  type        = string
  description = "The fingerprint of the API key. Ex: xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx"
  sensitive   = true
}

variable "API_PRIVATE_KEY_PATH" {
  type        = string
  description = "The path to the API private key file. Ex: /home/user/.oci/oci_api_key.pem"
  sensitive   = true
}

variable "REGION" {
  type        = string
  description = "The cloud provider region to operate in. Ex: us-ashburn-1"
}

dd.tf - Datadog

Datadog is the first app deployment. We will follow up later with cert management and setting up the load balancer.

#######################################
#  datadog.tf  – only the essentials
#######################################

# (1) namespace for isolation
resource "kubernetes_namespace" "datadog" {
  metadata { name = "datadog" }

}

# (2) API‑key secret the Operator will reuse
resource "kubernetes_secret" "datadog_api" {
  metadata {
    name      = var.datadog_secret_name
    namespace = kubernetes_namespace.datadog.metadata[0].name
  }
  data = { api-key = var.datadog_api_key } # keep the key in TF Cloud / *.auto.tfvars
  type = "Opaque"

  depends_on = [kubernetes_namespace.datadog]
}

# (3) Operator **and** Agent in one Helm release
resource "helm_release" "datadog" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog-operator"
  namespace  = kubernetes_namespace.datadog.metadata[0].name
  depends_on = [
    kubernetes_secret.datadog_api
  ]
}

resource "helm_release" "datadog_agent" {
  name       = "datadog-agent"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = kubernetes_namespace.datadog.metadata[0].name

  values = [<<-YAML
    datadog:
      apiKeyExistingSecret: ${var.datadog_secret_name}
      site: datadoghq.eu
      clusterName: ${var.cluster_name}

    agents:
      enabled: true
  YAML
  ]

  depends_on = [
    kubernetes_secret.datadog_api, helm_release.datadog
  ]
}

Source Code

The Terraform code for the cluster module (infrastructure part) discussed in this article, which sets up the OCI OKE cluster, can be found in the following public repository.

The Terraform code for the deployments module (applications and services part) can be found in the same repository.

13 July 2025