Automatically scaling an application cluster

https://gallery.technet.microsoft.com/CPU-Disk-and-Memory-a8a123bc First thing to do in a 'auto-scaling' script is to get the health of your cluster. There are a few ways you might wish to do this:

• Create a request to the website and check its timelyness
• Correlate the requests/sec from WMI or SNMP to the number or servers e.g. each server can deal with 300req/sec

Example 1 - Checking the performance of a endpoint

while (1 -eq 1){
    $request = Measure-Command { Invoke-WebRequest http://mywebsitecluster.com/ }
    if ($request.TotalSeconds -gt 4){
        Write-Warning "Website is running slowly"
        Scale-Up
    } 
    if ($request.TotalSeconds -lt 1){
        Write-Warning "Website is a bit too fast"
        Scale-Down
    }
     else {
        Write-Host "Website is happy"
        Start-Sleep -Seconds 30 
    }
}

Example 2 - Checking an WMI performance counter

For this example, you can use a safer metric. How many requests per second is each web server doing and what is the average. I've assumed we're looking at the Default Web Site in IIS and you have WMI allowed to the VPN connected clients.
I've also used a script that automates the Cisco VPN client to hop onto the network first. https://github.com/DimensionDataCBUSydney/DimensionData.ComputeClient/blob/master/PowerShell%20Examples/Connect%20to%20Cisco%20VPN%20endpoint.ps1 call this vpnConnect.ps1

# Connect to the Cisco VPN so we can read WMI data.
.\vpnConnect.ps1 -VPNHost "au1.cloud-vpn.net" -username "myUser" -password "myPassord"

while (1 -eq 1){
    $webNetwork = Get-CaasNetworks | Where name -like "Web Cluster"
    $servers = Get-CaasDeployedServer | Where networkId -eq $webNetwork.Id

    $totalAvg=0
    $totalTotal=0
    foreach ($server in $servers){
        # Get the number of requests per second.
        $counters = @("\Web Service(Default Web Site)\Get Requests/sec")
        $metrics = Get-Counter -ComputerName $server.privateIp -Counter $counters -SampleInterval 5 -MaxSamples 5

        $total = 0
        foreach($metric in $metrics)            
        {            
          $total += $metric.CookedValue            
        } 
        $avg = $total/5
        Write-Host "Server $($server.name) is doing $avg requests per second"
        $totalTotal += $avg
    }
    $totalAvg = $totalTotal/$servers.Count

    if ($totalAvg -gt 500){
        Write-Warning "Cluster is heavily loaded"
        Scale-Up
    } 
    if ($totalAvg -lt 100){
        Write-Warning "Website is quiet enough"
        Scale-Down
    } else {
        Write-Host "Cluster is happy"
        
    }
    Start-Sleep -Seconds 30 
}

Scale Up Action

Next up you need a way to quickly add a new server to a farm, the best method is to have a web template saved in your customer images then use a devops tool like puppet,chef,octopus or salt to configure the runtime settings.

function Scale-Up {
    $maxServers = 10
    $myTestNetwork  = Get-CaasNetworks | Where name -like "Web Cluster"

    # Get the template for the web server from our custom images
    $webServerTemplate = Get-CaasCustomerImages -NetworkWithLocations $myTestNetwork | Where name -eq "Web Server Template"

    #Get all the servers deployed using the web server template
    $myTestServers = Get-CaasDeployedServer | Where sourceImageId -eq $webServerTemplate.Id

    if ($myTestServers.Count -lt $maxServers){
        $newid = [guid]::NewGuid()
        $newServerDetails = New-CaasServerDetails -Name "WebServer $($newid)" -Description "Web server" -AdminPassword "MyPassword123!" -IsStarted $true -OsServerImage $webServerTemplate -Network $myTestNetwork
        New-CaasVM -ServerDetails $newServerDetails
        Write-Host "Created WebServer $($newid)"
    }
}

Scale down action

This is the same as the scale up but you have a minimum number of servers and you kill the first one over that amount.

function Scale-Down {
    $minServers = 3
    $myTestNetwork  = Get-CaasNetworks | Where name -like "Web Cluster"

    # Get the template for the web server from our custom images
    $webServerTemplate = Get-CaasCustomerImages -NetworkWithLocations $myTestNetwork | Where name -eq "Web Server Template"

    #Get all the servers deployed using the web server template
    $myTestServers = Get-CaasDeployedServer | Where sourceImageId -eq $webServerTemplate.Id

    if ($myTestServers.Count -gt $minServers){
        # Remove the first server in the cluster
        Remove-CaasVM -Server $myTestServers[0]
    }
}

Learnings - - You would need to consider here how you measure the health. VarnishCache has a requests per node metric, but no SNMP support. Apache Httpd can be scripted for SNMP but not out of the box. - You need to consider how you want each server to 'spin-up', I would recommend a configuration agent such as puppet agent, octopus tentacle or a SCCM agent. To setup the server. You could use PSExec or Putty.exe to remotely configure some basic commands e.g. service httpd start - You have to consider the load balancing here and how that works, this does not add the new server to the farm for you. You would need a software load balancer in this configuration. This is provided as an illustrative example, not a fully tested product. Enjoy!