How to use the Azure Private Link with uncommon or new PaaS offerings. You need the subresource names!
Azure, like other clouds, has a private link feature that allows connectivity to stay “inside” the network if you have an Express Route or a P2P. The one advantage is that you don’t have to have an internet facing endpoint, you don’t have to whitelist domains or insane ranges of IPs and you can also use your internal DNS.
I like to use Terraform to build the different PaaS offerings and in the same templates I can add the private endpoints to the services. The one thing that took me a while to find is the sub resource names. See below:
resource "azurerm_private_endpoint" "keyvault" {
name = "key_vault-terraform-endpoint"
location = azurerm_resource_group.rg.location
resource_group_name = azurerm_resource_group.rg.name
subnet_id = "${data.azurerm_subnet.rg.id}"
private_service_connection {
name = “key_vault-terraform-privateserviceconnection”
private_connection_resource_id = azurerm_key_vault.main.id
subresource_names = [ “vault” ]
is_manual_connection = false
}
A private-link resource is the destination target of a specified private endpoint.
Some Benefits
The benefits to most common private endpoints I’ve used are for the following services are
- Azure Container Registry
- The benefit here is that I can have a Docker Hub like container registry and I can push/pull containers to my local dev without having to go out to the internet
- Another benefit is that I can hook up security scans as well
- Azure SQL DBs
- The benefit is that again you can connect from a local server to this DB using internal IPs and DNS
- Azure Key Vault
- The benefit here is that your services and vault are not in the internet. Even in the internet they will need accounts to login but I like to know that the service can only be used inside the network.
If all your services are inside then there is no need to allow public networks. You can disable access and only allow trusted Microsoft Services (Log Analytics, Defender, etc.)
Disable public access to Azure Container Registry
Kubernetes Secret for Fleet Elasticsearch Agents
If you are having issues with your Elasticsearch Agents not getting the correct Fleet server enrollment token then this fix is for you. Elasticsearch support docs recommend adding the token as clear text in the Elastic agent daemonset manifest but here I show how to add it in the daemonset but as a kubernetes secret!Read more
Self-hosted Postgresql crashed and no backup! How to restore you DB from raw files?
Here's a great story on why it's good to automate builds, backups and restores, I had the privilege of working on an issue that spelled trouble from the beginning. A long long time ago there was a POC and lots of hype around the product so this POC, which was done manually, turned into PROD. Couple months later the single container running the postgres database crashes and there was no backup. Luckily the container was running in kubernetes and it had a persistent volume with the pgdata directory in it. The container crashed and was not able to come back because of an upgrade done to the DB so the MOST IMPORTANT thing you can do here is to protect the raw files by copying them into another directory. In a previous issue I worked on I have seen deployments that even when you use the existing claim they can wipe the data in it and start fresh, so backup the files you have into a compressed folder and create another copy to restore, this way you can at least have a working base.
Don't be a part of the naughty list of the internet exposed Kubernetes clusters!
A finding by the Shadow Foundation uncovered close to half a million k8s endpoints on the internet which can be targets to exploits. One factor is that by default these clusters are built with public IPs since cloud providers are outside your network and not all companies can have ExpressRoutes or dedicated point to point connectivity. To increase the security and have easier routing of your kubernetes cluster you can create a private cluster. In Azure Kubernetes Service the private cluster assigns an internal IP to your k8s API but NGINX defaults to external IP so in this article I walk through configuring NGINX to have internal IPs as well to keep it all inside the network.
Self-Healing I.T. Orchestration with Jenkins, Powershell, ServiceNow and Azure DevOps!
Most I.T. ticketing systems have an incident module to wait for users to submit issues so that the team can triage and react to solve the issue. In some cases you can spot repetitive issues and automate a fix or workaround to, at least, proactively bring services back online faster than waiting for an user to get an issue and then report the incident. Another benefit on automating fixes or work arounds is that they work 24/7 and they respond faster then the on call person. In this article I will show how to use Jenkins, Powershell, ServiceNow and Azure DevOps to orchestrate a server reboot after detecting a specific issue in the event log of a Windows server.
Easiest Way to Deploy Ubuntu 20.04 with NVIDIA Drivers and the Latest CUDA toolkit via Packer.
I am building an analytics system that deploys containers on top of the Azure NCasT4_v3-series virtual machines which are powered by Nvidia Tesla T4 GPUs and AMD EPYC 7V12(Rome) CPUs. I am deploying the VM from an Azure DevOps pipeline using Hashicorp Packer and after trying a few ways I found a very easy way to deploy the VM, Driver and Cuda Toolkit which I will share in this article.
Avoid Self-Monitoring on your PROD ElasticSearch Cluster! Ship logs to a separate deployment.
It is highly recommended to disable self-monitoring on your production Elasticsearch deployment for performance and built-in support in Elastic Cloud on Kubernetes. In this article we go over configuring the monitoring cluster.
Deploy A Private Elastic Cloud Kubernetes Cluster On Azure DevOps Pipelines For CI/CD
ElasticSearch has developed a great Operator for Kubernetes to orchestrate tasks to make things easier to deploy and prevent cowboy engineers like me forcing changes that end up breaking stuff :D . In this article I will go over deploying ECK on AKS via ADO and I will share some FYIs. (Too many acronyms, get ready for more.)
How to create an Azure DevOps Service Connection to my private AKS Cluster?
Ok, so we can deploy AKS and restrict the IPs that can consume the K8S API but this is still "out in the internet" which might not sit right with management or security. So.... we can deploy a private AKS (article coming soon..) and keep all traffic internal. The trade off is that a private AKS cluster's api endpoint cannot be restricted to specific subnets from the AKS console but you can deploy NSGs if needed. In this article we go trough setting up a service connection so you can deploy from Azure DevOps
Configuring snapshots for Elastic Cloud on Kubernetes with AKS and Azure Storage Accounts
We had the opportunity to migrate Elastic Cloud Enterprise running on bare metal to Azure. Initially we were going to spin up 3-5 VMs and install ECE with the Ansible role but then we found ECK and made our lives way easier. Migrating was very easy following Elastic's documentation and video posted here but we had a little trouble configuring the snapshots with an Azure Storage account that had a private link so in this article we share how we got it done.