Most I.T. ticketing systems have an incident module to wait for users to submit issues so that the team can triage and react to solve the issue. In some cases you can spot repetitive issues and automate a fix or workaround to, at least, proactively bring services back online faster than waiting for an user to get an issue and then report the incident. Another benefit on automating fixes or work arounds is that they work 24/7 and they respond faster then the on call person. In this article I will show how to use Jenkins, Powershell, ServiceNow and Azure DevOps to orchestrate a server reboot after detecting a specific issue in the event log of a Windows server.
I. Powershell Script to detect for issues:
First we need a way to check the windows event log for a specific issues so a simple powershell script can help. The example below will remote in to a server, collect logs, for the last 30 minutes, from a specific source and specific error message and save the output into a variable that we can use to parse and take action.
If we find an error with “Specific error message” then send an email to ServiceNow so it can trigger and start a flow.
If we don’t find any errors then send a message to a Microsoft Teams channel to alert that the check was successful and no errors were found
Invoke-Command -ComputerName RemoteServer.domain.com -Credential $Creds -ScriptBlock {
$date = Get-Date
$checklog = Get-EventLog -LogName Application -Source XXAppService -Message "Specific Error Message" -After $date.AddMinutes(-30) -Before $date
return $checklog
}
if ($null -ne $checklog){
Send-MailMessage -From Automation@domain.com -To company@service-now.com -Body "Restart Server" -Subject "Automation trigger message" -SmtpServer smtp.company.com -Port 25
}
else {
Send-MailMessage -From Automation@domain.com -To itchannel@amer.teams.ms -Body "No errors found." -Subject "Automation Server Check" -SmtpServer smtp.company.com -Port 25
}II. Jenkins
Now that the script detects the specific error message it needs to run constantly to check and evaluate the conditions. For me Jenkins and the powershell plugin (https://plugins.jenkins.io/powershell/) are the best solution to get this done. I can set a schedule and run our powershell script right from the console or from my source code repository. Since the powershell script is set to look at the logs for the past 30 minutes then the schedule in Jenkins will be to run every 30 minutes to narrow down the log check.
Since our powershell script is set connect to a remote server to detect errors in logs we need to pass credentials with rights to do so. Here’s where the integration starts to happen. We can add and store credentials in Jenkins (with the credentials plugin that comes OOB) and then consume them in our powershell scripts in so that at least we don’t hard code the secret values in our scripts.
From the Jenkins dashboard go to Manage Jenkins –> Credentials and go to your store to add Credentials
Now that we have the credentials saved and encrypted in the Jenkins store we can use it in our powershell script by binding the secret we created earlier in the build steps and creating variables to hold the username and password to pass to the script.
$SrvPassword = ConvertTo-SecureString "$($ENV:SrvPassword)" -AsPlainText -Force
$Credential = New-Object System.Management.Automation.PSCredential ("$ENV:SrvUser", $SrvPassword)
Invoke-Command -ComputerName RemoteServer.domain.com -Credential $Creds -ScriptBlock {}After this step you will have a job that will run every 30 minutes and check for logs. If logs are found an email to ServiceNow will be sent to create a task and trigger the start of a Flow. If no errors are found in the logs then send a message to the IT Teams channel to let them know that the script ran successfully and no errors were found.
You could stop here since the steps above will proactively detect an error and create a ticket for your helpdesk to act but we can do more and automate the reboot with a ServiceNow flow that triggers an Azure DevOps pipeline.
Go to the next page to see how to automate the rest of the tasks.