How to use the Azure Private Link with uncommon or new PaaS offerings. You need the subresource names!
Azure, like other clouds, has a private link feature that allows connectivity to stay “inside” the network if you have an Express Route or a P2P. The one advantage is that you don’t have to have an internet facing endpoint, you don’t have to whitelist domains or insane ranges of IPs and you can also use your internal DNS.
I like to use Terraform to build the different PaaS offerings and in the same templates I can add the private endpoints to the services. The one thing that took me a while to find is the sub resource names. See below:
resource "azurerm_private_endpoint" "keyvault" {
name = "key_vault-terraform-endpoint"
location = azurerm_resource_group.rg.location
resource_group_name = azurerm_resource_group.rg.name
subnet_id = "${data.azurerm_subnet.rg.id}"
private_service_connection {
name = “key_vault-terraform-privateserviceconnection”
private_connection_resource_id = azurerm_key_vault.main.id
subresource_names = [ “vault” ]
is_manual_connection = false
}
A private-link resource is the destination target of a specified private endpoint.
Some Benefits
The benefits to most common private endpoints I’ve used are for the following services are
- Azure Container Registry
- The benefit here is that I can have a Docker Hub like container registry and I can push/pull containers to my local dev without having to go out to the internet
- Another benefit is that I can hook up security scans as well
- Azure SQL DBs
- The benefit is that again you can connect from a local server to this DB using internal IPs and DNS
- Azure Key Vault
- The benefit here is that your services and vault are not in the internet. Even in the internet they will need accounts to login but I like to know that the service can only be used inside the network.
If all your services are inside then there is no need to allow public networks. You can disable access and only allow trusted Microsoft Services (Log Analytics, Defender, etc.)
Disable public access to Azure Container Registry
Using Github Actions to Build a Kasm Workspace for XChat IRC Client
I really like building and customizing my own Kasm images to use containers to run my applications instead of installing them directly in my computer. Here’s how I built the xchat client Kasm Workspace.
Dockerfile
Most and I mean most of the work is done by the Kasm team since the base images are loaded with all the dependencies needed for Kasm Workspaces and all you have to do is install your app and customize it.
Github Actions for Docker:
- Create a new secret named
DOCKER_HUB_USERNAMEand your Docker ID as value. - Create a new Personal Access Token (PAT) for Docker Hub.
- Add the PAT as a second secret in your GitHub repository, with the name
DOCKER_HUB_ACCESS_TOKEN.
The Github action will run to login, build and push your container to your DockerHub account. Once that’s ready you can proceed to configure kasm to use your container image.
Configuration for Kasm
Once the container is available in the DockerHub repo or other container registries it can be pulled to the Kasm server.
Once the container is in my pulled images I can setup the Kasm Image
Now it is ready for me to use and further customize.
Kubernetes Secret for Fleet Elasticsearch Agents
If you are having issues with your Elasticsearch Agents not getting the correct Fleet server enrollment token then this fix is for you. Elasticsearch support docs recommend adding the token as clear text in the Elastic agent daemonset manifest but here I show how to add it in the daemonset but as a kubernetes secret!Read more
Self-hosted Postgresql crashed and no backup! How to restore you DB from raw files?
Here's a great story on why it's good to automate builds, backups and restores, I had the privilege of working on an issue that spelled trouble from the beginning. A long long time ago there was a POC and lots of hype around the product so this POC, which was done manually, turned into PROD. Couple months later the single container running the postgres database crashes and there was no backup. Luckily the container was running in kubernetes and it had a persistent volume with the pgdata directory in it. The container crashed and was not able to come back because of an upgrade done to the DB so the MOST IMPORTANT thing you can do here is to protect the raw files by copying them into another directory. In a previous issue I worked on I have seen deployments that even when you use the existing claim they can wipe the data in it and start fresh, so backup the files you have into a compressed folder and create another copy to restore, this way you can at least have a working base.
Using Bitwarden as a secret source and Powershell vault to keep secrets hidden! - Part 1: Retrieving Secrets
SHHHH! It's a secret! If you are using Bitwarden to store your secrets you are awesome since I am doing the same as it is my favorite vault for personal use. My main use case is for the browser plugin but with the CLI I've been using it in my powershell and python scripts. In this article I will cover a function I created to retrieve secrets from Bitwarden and pass them in Powershell.
In upcoming articles I will show how to use it in Python, Azure DevOps or Jenkins depending on demand for increased security and automation.
The NodeSelector Is My Mixed-CPU-Architecture Kubernetes Cluster's Best Friend!
I have an always growing Kubernetes cluster. I currently have a cluster made of 2 Raspberry Pis and 1 PC but HOW, isn't that frowned upon? Well, you can use the NodeSelector attribute to make the containers stick to specific nodes. The specifics we are covering in this article are the CPU architectures of the nodes since the Raspberry Pis run ARM and my PCs run AMD64.
Don't be a part of the naughty list of the internet exposed Kubernetes clusters!
A finding by the Shadow Foundation uncovered close to half a million k8s endpoints on the internet which can be targets to exploits. One factor is that by default these clusters are built with public IPs since cloud providers are outside your network and not all companies can have ExpressRoutes or dedicated point to point connectivity. To increase the security and have easier routing of your kubernetes cluster you can create a private cluster. In Azure Kubernetes Service the private cluster assigns an internal IP to your k8s API but NGINX defaults to external IP so in this article I walk through configuring NGINX to have internal IPs as well to keep it all inside the network.
5 Quick but powerful tips for Dev#$%!Ops Success
There are a ton of variations out there like DevSecOps, MLOps, GitOps (My Favorite), NetOps, DataOps, BizOps, even NoOps, etc. In my opinion, it all comes back to the basic definition which says that Dev<whatever in the middle>OPS is like a soup recipe with 3 main ingredients that are easy to find in your organization's pantry which are people, processes and automation tools. Add the right amount of each ingredient to taste and turn on the heat (Do not cook people)!
The end goal is to serve your customers the best bowl of software soup they've had?!
Self-Healing I.T. Orchestration with Jenkins, Powershell, ServiceNow and Azure DevOps!
Most I.T. ticketing systems have an incident module to wait for users to submit issues so that the team can triage and react to solve the issue. In some cases you can spot repetitive issues and automate a fix or workaround to, at least, proactively bring services back online faster than waiting for an user to get an issue and then report the incident. Another benefit on automating fixes or work arounds is that they work 24/7 and they respond faster then the on call person. In this article I will show how to use Jenkins, Powershell, ServiceNow and Azure DevOps to orchestrate a server reboot after detecting a specific issue in the event log of a Windows server.
Scale Down a specific node in Azure Kubernetes Service
In the past I've scaled up a cluster to test a new deployment or to provide extra compute during an upgrade but when it comes to scaling down AKS won't pick the node with less resources or even the drained one so I've seen it briefly disrupt a deployment.
To have more control on what AKS scales down you can use virtual machine scale set protection policies to specify which node to remove. Here's how to do so...