K8S on Oracle Cloud II - Single Terraform Config for Cluster and Datadog Observability

This is a follow-up to K8S on Oracle Cloud - I. The objective here is to get some visibility on the cluster that we created previously via Terraform.

It's practical to use Terraform as it saves you the headache of manual infrastructure management and resource tracking. What we will be doing in this article is to install the DD agent via Terraform and Helm.

Keep in mind that this is for your own exploration and learning, this is obviously not a production setup, as we will be facing a chicken and egg problem, which is classic with Terraform:

The goal for us is to create the cluster and install the DD agents in one go via a script, in an enterprise setup, you will most likely have multiple Terraform repos with different states, one to initialize the cluster and one to manage the deployments, but for our purpose, we will keep it simple, at least to deploy the building blocks which are observability and provisioning a load balancer later on.

For deployments of workloads, we might settle for a Helm only approach later on.

What we need to do is to create a Terraform plan that will do the following:

Acceptance criteria:

a working Terraform plan before any K8S resources are created
a working Terraform apply in a single go that will create the cluster and install the DD agent
a working Terraform destroy that will remove the cluster and the DD agent and clean up the kubeconfig

terraform plan

Setup tailored to local dev environment

We'll follow a simple plan that relies on generating the kubeconfig file via the OCI CLI instead of inferring it from the OCI provider. This is a bit hacky, but it works for our local dev environment.

So the execution plan will be as follows:

Create a dummy kubeconfig file from a template to allow the first Terraform plan to run
Run the first Terraform plan
Create the cluster via the first Terraform apply
Generate the kubeconfig file via terraform_data calling OCI CLI and overwrite the dummy kubeconfig file
The first Terraform will be able to create the cluster but not the Datadog agent, as the Terraform providers will be initialized with the dummy kubeconfig file still and won't pick up the new kubeconfig file
Run a second Terraform plan to create the Datadog agent

Helm and Kubernetes Providers:

provider "kubernetes" {
  config_path = pathexpand(var.kubeconfig_path) # auto-detect current context
}

provider "helm" {
  kubernetes {
    config_path = pathexpand(var.kubeconfig_path)
  }
}

OCI Generate Kubeconfig After Cluster Creation

Add a clear dependency on the cluster creation to ensure that the kubeconfig is generated after the cluster is created.

resource "terraform_data" "wait_for_kubeconfig" {
  provisioner "local-exec" {
    command = "oci ce cluster create-kubeconfig --cluster-id ${oci_containerengine_cluster.generated_oci_containerengine_cluster.id} --file ${var.kubeconfig_path}"
  }
  depends_on = [oci_containerengine_cluster.generated_oci_containerengine_cluster]
}

Tree

Overview of the files in the project:

├── dd.tf
├── init.sh
├── deinit.sh
├── k8s.tf
├── main.tf
├── oke_config.yaml
├── oke_config.yaml.template.yaml
├── terraform.tfstate
├── terraform.tfstate.backup
└── variables.tf

1. Initial kubeconfig files

Assuming you already created two files to represent the kubeconfig and a template, initialize them both to the template file at the beginning:

oke_config.yaml.template.yaml & oke_config.yaml in the beginning of the script

apiVersion: v1
clusters:
contexts:
current-context:
kind: Config
preferences: {}
users:

1. init.sh

The script that will initialize the cluster, overwrite the kubeconfig file and run the Datadog agent Helm charts afterwards via a dual Terraform apply

#!/bin/bash

if ! cmp -s oke_config.yaml.template.yaml oke_config.yaml; then
  echo "Files are different. Copying template to oke_config.yaml..."
  cp oke_config.yaml.template.yaml oke_config.yaml
else
  echo "Files are identical. No copy needed."
fi

terraform init
terraform plan
terraform apply -auto-approve
terraform apply -auto-approve

2. deinit.sh

The script that will destroy all resources and overwrite the kubeconfig file back to the template values

#!/bin/bash

terraform destroy -auto-approve
cp oke_config.yaml.template.yaml oke_config.yaml

Datadog Resources (dd.tf)

With the setup above, we can now create the Datadog resources via helm_release and the Kubernetes resources.

This will involve creating a namespace for Datadog, a Kubernetes secret for the Datadog API key, installing the Datadog operator and the Datadog agent via Helm.

dd.tf

#######################################
#  datadog.tf  – only the essentials
#######################################

# (1) namespace for isolation
resource "kubernetes_namespace" "datadog" {
  metadata { name = "datadog" }

  depends_on = [terraform_data.wait_for_kubeconfig] # ensure kubeconfig is created
}

# (2) API-key secret the Operator will reuse
resource "kubernetes_secret" "datadog_api" {
  metadata {
    name      = var.datadog_secret_name
    namespace = kubernetes_namespace.datadog.metadata[0].name
  }
  data = { api-key = var.datadog_api_key } # keep the key in TF Cloud / *.auto.tfvars
  type = "Opaque"

  depends_on = [kubernetes_namespace.datadog, terraform_data.wait_for_kubeconfig]
}

# (3) Operator **and** Agent in one Helm release
resource "helm_release" "datadog" {
  name       = "datadog"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog-operator"
  namespace  = kubernetes_namespace.datadog.metadata[0].name
  depends_on = [
    kubernetes_secret.datadog_api, terraform_data.wait_for_kubeconfig
  ]
}

resource "helm_release" "datadog_agent" {
  name       = "datadog-agent"
  repository = "https://helm.datadoghq.com"
  chart      = "datadog"
  namespace  = kubernetes_namespace.datadog.metadata[0].name

  values = [<<-YAML
    datadog:
      apiKeyExistingSecret: ${var.datadog_secret_name}
      site: datadoghq.eu
      clusterName: ${var.cluster_name}

    agents:
      enabled: true
  YAML
  ]

  depends_on = [
    kubernetes_secret.datadog_api, helm_release.datadog, terraform_data.wait_for_kubeconfig
  ]
}

Conclusion

This low-friction approach allows us to get up and running quickly with a working Terraform plan and apply that will create the cluster and have observability via Datadog in one go.

Just by running ./init.sh and ./deinit.sh we can create and destroy the cluster resources and maintain a single Terraform state if you are willing to accept this minimal scripting and less than perfect dual Terraform apply.

13 July 2025