Deploy A Private Elastic Cloud Kubernetes Cluster On Azure DevOps Pipelines For CI/CD

ElasticSearch has developed a great Operator for Kubernetes to orchestrate tasks to make things easier to deploy and prevent cowboy engineers like me forcing changes that end up breaking stuff 😀 . In this article I will go over deploying ECK on AKS via ADO and I will share some FYIs. (Too many acronyms, get ready for more.)

I. DNS Configuration to leverage PrivateLink and ExpressRoute

My client has an Express Route configured which works well for servers where you can hard code an IP, for PaaS there is a little more to do. When you tell AKS to build a private cluster the API endpoint is created with a privatelink as seen below so to get traffic working you must set a conditional forwarder in your internal DNS Zone.

For this example we are creating an cluster in Azure’s US East Region so a conditional forwarder was created for eastus.azmk8s.io and point it to Azure’s Platform IP to use the private DNS Zones in Azure. DNS Servers in Azure can be added as well.

Sources:
Private Endpoint DNS: https://docs.microsoft.com/en-us/azure/private-link/private-endpoint-dns
Azure Platform IP: https://docs.microsoft.com/en-us/azure/virtual-network/what-is-ip-address-168-63-129-16

II. Deploy AKS with Terraform from a hosted Agent

Since I am deploying a private cluster all traffic will stay inside the network so the next parts of the deployments will take place on a hosted Azure DevOps Agent that lives in a VLAN with firewall access to Azure’s VNETs.

3 Things you will need:

1. Add the record for privatelink in Azure’s private DNS Zone
2. Add the parameters for private AKS cluster:
   1. private_cluster_enabled = true
   2. private_dns_zone_id = Link the resource created previously so that the AKS endpoint can have an internal name.
   3. private_cluster_public_fqdn_enabled = false so that step 2 can take place.
3. Service Principal = We pass the credentials from Azure DevOps pipeline variables.

III. Create the service connection

I complete this step from the hosted Azure DevOps agent since it already has the Azure CLI installed.

I am using the kubeconfig method to login so to get it I use the following command:

az aks get-credentials --resource-group RGNAME --name aksclustername

This configures my kubeconfig file which I then copy in Azure DevOps service connection. YOU WILL NOT BE ABLE TO VALIDATE. Since this is a private endpoint but it will work from the Azure DevOps hosted agent.

IV. Deploy ECK Pre Requisites

To keep all internal I am using the client company’s SSL certificate so that we can use the internal name later. I use Azure DevOps secure file to store and download the cert and keys in a secure way since this is encrypted and results are not posted in the console.

There are different file types and there is a distinction made below:

Nginx uses:

kubectl create secret tls ${CERT_NAME} --key ${KEY_FILE} --cert ${CERT_FILE}

ECK uses:

kubectl create secret generic my-cert --from-file=ca.crt --from-file=tls.crt --from-file=tls.key

Don’t forget to set the reference to your secure files so you can easily consume them in later steps.

V. Deploy ECK

We are doing the 2 steps recommended from ECK’s documentation link below but in our pipelines but we are using this stage to also apply our license file to unlock enterprise features and for support with the 2 lines below in tasks inside the pipeline.

kubectl create secret generic eck-license --from-file=my-license-file.json -n elastic-system

kubectl label secret eck-license "license.k8s.elastic.co/scope"=operator -n elastic-system

Sources:

Deploy ECK: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-deploy-eck.html

VI. Deploy ES

Same deal here, we create a kubectl task with our service connection and we push the ES Version, # Of Nodes and the init containers for installing our plugins for Azure Storage, virtual memory and JVM size settings but the key here is to add the internal cert, annotation to keep the LoadBalancer internal and the IP.

Sources: https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-node-configuration.html

VII. Deploy NGINX to LB ElasticSearch

The last piece is to install NGINX with the annotation for internal load balancer and static IP

loadBalancerIP: 10.240.0.42
    annotations:
      service.beta.kubernetes.io/azure-load-balancer-internal: "true"

Add an Ingress Rule for ES and create a local DNS entry to hit the endpoint with internal name, internal cert and traffic.

Source: https://docs.microsoft.com/en-us/azure/aks/ingress-internal-ip?tabs=azure-cli