Remote Desktop Session Host Servers getting hung regularly?

In this post, we will discuss on an issue that may be faced by quite many folks who have a remote desktop farm in their environment. The RDSH (remote desktop session host) servers might be getting hung quite often, especially during peak business hours. Out of all the servers in the farm, this situation could happen with any of the server, with no particular order.

Althought there could be many reasons behind it, but in some cases, it could become more problematic because event viewer might not be revealing any definitve clue. There could be few logs which we can associate with the server hung state, but none of them would be pointing to the right direction, like below.

1> The winlogon notification subscriber is taking long time to handle the notification event.

2> A timeout (120000 milliseconds) was reached while waiting for a transaction response from the SessionEnv service.

3> The server did not register with DCOM within the required timeout.

Remote Desktop Session Host server hangs/locks up (2008 R2 in vSphere 4.1) Here is a blog post bearing similarity to this case, where this issue is discussed.

Diagnosing the problem

To find the right direction to search for the solution, we can run a DCS (Data Collector Set) in Performance monitor for CPU utilization and Memory consumption, in any or all of the RDSH servers. Thec onfiguration steps are explained in this link Create a Data Collector Set to Monitor Performance Counters. This way we can know the picture of resource utilization, just moments before the server dives into hung state, the next time.

If the RDSH servers are VMware based, we can analyze the VM level logsand look for following.

1> GuestMsg: Too many channels opened

2> GuestRpc: Channel 6, unable to send the reset rpc.

Also the ESXI level logs related to performance could tell about the resource utilization during the time of VM’s hung state.

Another blog post of vmware community might shed some light into the topic VM of Esxi 6 crash after too many TSE connexion 

Here we must discuss, the TCP Chimney Offload, Receive Side Scaling (RSS), and Network Direct Memory Access (NetDMA) features that are available for the TCP/IP protocol in Windows Server 2008 onwards.

Disabling these three features, in many cases resovles the issue Information about the TCP Chimney Offload, Receive Side Scaling, and Network Direct Memory Access features in Windows Server 2008

1> Receive Side Scaling (RSS).

RSS enables network adapters to distribute the kernel-mode network processing load across multiple processor cores in multi-core computers.

2> Network Direct Memory Access (NetDMA)

NetDMA provides operating system support for direct memory access (DMA) offload. TCP/IP uses NetDMA to relieve the CPU from copying received data into application buffers, reducing CPU load.

3> TCP Chimney Offload

TCP Chimney Offload is a networking technology that helps transfer the workload from the CPU to a network adapter during network data transfer.

If this also has not resolved the issue, we can proceed further with another solution.

Windows System Resource Manager (WSRM)

Windows System Resource Manager (WSRM) can be used to allocate processor and memory resources to applications, users, Remote Desktop Services sessions, and Internet Information Services (IIS) application pools.

With Windows System Resource Manager for the Windows Server® 2012 operating system, you can manage server processor and memory usage with standard or custom resource policies.

Equal per session

Out of all poicies, this is the one which might solve our purpose. When the Equal_Per_Session resource allocation policy is managing the system, resources are allocated on an equal basis for each session connected to the system. This policy is for use with RD Session Host servers.

This used to work nicely prior to Windows Server 2012 R2, but for some really disappointing reason, Microsoft has removed this feature, beginning with Windows Server 2012 R2 and left us with no alternatives.

There seems to be a workaround for this too. WINDOWS SYSTEM RESOURCE MANAGER AND WINDOWS SERVER 2012 R2 In this blog, author has described a way to still be able to use WSRM in Windows Server 2012 R2 and later editions. But this requires a presence of Windows Server 2012, which is not possible many times.

There is also a third party tool Process Lasso Server Edition, which claims to perform this task, but haven’t tested it.

So now we head to the final section, which is more of a preventive step.

Microsoft Remote Desktop Services 2012 Management Pack for System Center 2012

The Remote Desktop Services Management Pack helps you manage your computers that are running Remote Desktop Services on Windows Server 2008 R2 by monitoring the health of the following Remote Desktop Services role services.

When there is problem with the availability or performance of one of these components, Microsoft System Center Operations Manager 2007 uses the Remote Desktop Services Management Pack to detect the issue and alert you so that you can diagnose the problem and fix it.

To set this up, this link can be really helpful Monitoring RDS 2012 with System Center Operations Manager 2012 (Part 1)

I will post more here, if I find any more definitive solution to this issue.

Advertisements

Exporting Network Printers’ information using Powershell

The printers in a windows domain can be managed by Print Server role of Windows Server. The network printers can also be published in Active Directory from Print Management console. Therefore we can extract information on printers that are published in AD and the ones which are present in Print Servers. It could be possible that not all of the printers in Print Servers are published. Hence a need arises to have a report on all the printers for proper classification and management.

Exporting Information from Active Directory

Using Powershell command Get-ADObject

From a powershell console, we can execute below command to list all the printers published in the domain. The scope of the search can be limited using -searchscope switch.

get-adobject -filter {ObjectClass -eq "printQueue"} -SearchScope subtree -Properties location | select Name,Location >Printers.csv

Using CSV Directory Exchange utility

Also we can make use of CSVDE to export the list of all printers published in AD to a CSV file. This command can be executed both from CMD and PowerShell prompt and search scope can be limited using -p switch.

csvde -r "(objectClass=printQueue)" -p subtree -l "name,location" -f "C:\printers.csv"

Exporting information from remote Print Servers

Now we will try to export the printers from print servers and compare the same to the report we exported earier using CSVDE/Get-ADObject. Following script is very effective in this regard which I have borrowed from the link below and modified a small bit. The usage and description is included in the parenthesis below.

<#
.SYNOPSIS
 Script to create a Excel spreadsheet with detailed information about
 the printers installed on the server
.DESCRIPTION 
 Script was designed to give you a good description of how your print
 server(s) are installed and configured.

 * Requires Microsoft Excel be installed on the workstation you are running
 the script from.
.PARAMETER PrintServers
 Name of the server you wish to run the script again. Can also be an
 array of servers.
.OUTPUTS
 Excel spreadsheet
.EXAMPLE
 .\Export-PrinterInfo.ps1 -PrintServers "MyPrintServer"
.EXAMPLE
 .\Export-PrinterInfo.ps1 -PrintServers (Get-Content c:\scripts\myprintserverlist.txt)
.NOTES
 Author: Martin Pugh
 Twitter: @thesurlyadm1n
 Spiceworks: Martin9700
 Blog: www.thesurlyadmin.com

 Changelog:
 1.0 Initial Release
.LINK
 http://community.spiceworks.com/scripts/show/2186-export-printer-information-to-spreadsheet
#>
[CmdletBinding()]
Param (
 [Parameter(Mandatory=$true)]
 [string[]]$PrintServers = "yourPrintServer"
)

# Create new Excel workbook
Write-Verbose "$(Get-Date): Script begins!"
Write-Verbose "$(Get-Date): Opening Excel..."
[threading.thread]::CurrentThread.CurrentCulture = 'en-US'
$Excel = New-Object -ComObject Excel.Application
$Excel.Visible = $True
$WorkBook = $Excel.Workbooks.Add()
$WorkBook.worksheets.add()
$Sheet = $WorkBook.Worksheets.Item(1)
$Sheet.Name = "Printer Inventory"
#======================================================
$Sheet.Cells.Item(1,1) = "Print Server"
$Sheet.Cells.Item(1,2) = "Printer Name"
$Sheet.Cells.Item(1,3) = "Location"
$Sheet.Cells.Item(1,4) = "Comment"
$Sheet.Cells.Item(1,5) = "IP Address"
$Sheet.Cells.Item(1,6) = "Driver Name"
$Sheet.Cells.Item(1,7) = "Driver Version"
$Sheet.Cells.Item(1,8) = "Driver"
$Sheet.Cells.Item(1,9) = "Shared"
$Sheet.Cells.Item(1,10) = "Share Name"
$Sheet.Cells.Item(1,11) = "AD - Published"
#=======================================================
$intRow = 2
$range = $Sheet.UsedRange
$range.Interior.ColorIndex = 40
$range.Font.ColorIndex = 11
$range.Font.Bold = $True
#=======================================================

# Get printer information
ForEach ($PrintServer in $PrintServers)
{ Write-Verbose "$(Get-Date): Working on $PrintServer..."
 $Printers = Get-WmiObject Win32_Printer -ComputerName $PrintServer
 ForEach ($Printer in $Printers)
 {
 If ($Printer.Name -notlike "Microsoft XPS*")
 { $Sheet.Cells.Item($intRow, 1) = $PrintServer
 $Sheet.Cells.Item($intRow, 2) = $Printer.Name
 $Sheet.Cells.Item($intRow, 3) = $Printer.Location
 $Sheet.Cells.Item($intRow, 4) = $Printer.Comment

 If ($Printer.PortName -notlike "*\*")
 { $Ports = Get-WmiObject Win32_TcpIpPrinterPort -Filter "name = '$($Printer.Portname)'" -ComputerName $Printserver
 ForEach ($Port in $Ports)
 {
 $Sheet.Cells.Item($intRow, 5) = $Port.HostAddress
 }
 }

 #################### 
 $Drivers = Get-WmiObject Win32_PrinterDriver -Filter "__path like '%$($Printer.DriverName)%'" -ComputerName $Printserver
 ForEach ($Driver in $Drivers)
 { $Drive = $Driver.DriverPath.Substring(0,1)
 $Sheet.Cells.Item($intRow, 7) = (Get-ItemProperty ($Driver.DriverPath.Replace("$Drive`:","\\$PrintServer\$Drive`$"))).VersionInfo.ProductVersion
 $Sheet.Cells.Item($intRow,8) = Split-Path $Driver.DriverPath -Leaf
 }
 #################### 
 $Sheet.Cells.Item($intRow, 6) = $Printer.DriverName
 $Sheet.Cells.Item($intRow, 9) = $Printer.Shared
 $Sheet.Cells.Item($intRow, 10) = $Printer.ShareName
 $Sheet.Cells.Item($intRow, 11) = $Printer.Published
 $intRow ++
 }
 }
}

$intRow ++ 
$Sheet.Cells.Item($intRow,1) = "Printer inventory completed"
$Sheet.Cells.Item($intRow,1).Font.Bold = $True
$Sheet.Cells.Item($intRow,1).Interior.ColorIndex = 40
$Sheet.Cells.Item($intRow,2).Interior.ColorIndex = 40
Write-Verbose "$(Get-Date): Completed!"

Therefore with the reports extracted by means which are discussed in this article, we can have a detailed report at our hand, regarding the complete printers information in our domain.

Booting from ISO file on remote computer via ILO console on HP Proliant servers

Accessing an ISO file for new OS installation a new HP Proliant server can be tricky if we don’t have manual access to the server or if we intend to do this remotely.

In page number: 237 of the ILO 4 user guide available at HPE iLO 4 User Guide, I found below paragraph, which can be an option to access the ISO image from new physical server.

===============================================================

Using an image file through a URL (IIS/Apache) You can connect scripted media by using the .NET IRC or Java IRC.

Scripted media supports only 1.44 MB floppy disk images (.img) and CD/DVD-ROM images (.iso).

The image must be on a web server on the same network as iLO.

  1. Start the .NET IRC or Java IRC.
  2. Depending on the image type you will use, select Virtual Drives→URL Removable Media (.img) or Virtual Drives→URL CD-ROM/DVD (.iso). The Image file at URL dialog box opens.
  3. Enter the URL for the image file that you want to mount as a virtual drive, and then click Connect. The virtual drive activity LED does not show drive activity for URL-mounted virtual media.

================================================================

Following is the process flow which has been mentioned in the HP blog ILO 4 Scripted Media URL

  1. Use an existing web server, or install a new web server for the purpose of delivering the ISO files

2. Create a folder to hold the ISO images

3. Add ISO file(s)

4. Enable directory browsing in Web Services. You can do this with the IIS manager if          its a Windows web server. If you created a custom folder for the files, enable        directory browsing on that folder.

5. You must add a MIME type for the ISO extension. In Server 2008 IIS, you can do this          from the HTTP Headers selection in IIS Manager.

a> .ISO application/octet-stream

b> .IMG application/octet-stream

6. Login to the ILO target server, and open the remote console

7. At the top of the window, click on Virtual Drives, and then select URL DVD ROM

8. Input the HTTP path to the image file, including the file name. Click connect and it              will mount the drive. Path will resemble “http://hostname or IP/folder/filename.ISO

Now the problem with this approach is network latency, especially when the infrastructure is globally spread.

So what can be solution here. Virtual folders?

lets read below two paragraphs taken from the user guide.

==============================================

Operating system considerations: Virtual Folder

  • Windows—A Virtual Folder appears automatically after Windows recognizes the mounting of the virtual USB device. You can use the folder the same way that you use a locally attached device. Virtual Folders are nonbootable. Attempting to boot from the Virtual Folder might prevent the server from starting.

===============================================

Virtual folders enable you to access, browse to, and transfer files from a client to a managed server. You can mount and dismount a local or networked directory that is accessible through the client. After you create a virtual image of a folder or directory, the server connects to the image as a USB storage device, enabling you to browse to the server and transfer the files from the virtual image to the server.

Using iLO Virtual Media The Virtual Folder is nonbootable and read-only; the mounted folder is static. Changes to the client folder are not replicated in the mounted folder.

=================================================

Apart from this virtual folders also have a size limit of 2 GB.

So we come down to our next option -> Image File CD/DVD-ROM

For this option we first need to copy the ISO file to a temporary server in the network where we are installing the new physical server. From this temporary server, we should access the ILO of new server. And select VIrtual Drives Option as Image File CD/DVD-ROM. We then have to browse to ISO file stored locally on temporary server and mount it. As this time the ISO file is in the same network as the new server, the network latency would be eliminated and access speed would be greatly enhanced.

Powershell script for obtaining DHCP scope usage statistics for all the DHCP servers in the domain

I am presenting in this post, a solution to a common task of a Windows admin, where one has to pull report of DHCP scope usage statistics and the following script will do just the job at hand.

Algorithm:

1> Get the list of all DHCP servers in the domain from Get-DhcpServerInDC command.

2> Store the DHCP servers in a variable $machines.

3> Get the DHCP scopes recursively for each DHCP server stored in $machines, from Get-DhcpServerv4Scope command.

4> Store the DHCP scopes in a variable $AllScopes.

5> Get usage statistics for each DHCP scope recursively, from Get-DhcpServerv4ScopeStatistics command.

6> Store the statistics for a particular scope in a variable $ScopeStats.

7> Append the data collected to a csv file

Script:

$CurrentDate = Get-Date -format dddd
$CSVFile = "C:\Dhcp_$CurrentDate.csv"
$machines = Get-DhcpServerInDC

"DHCPServer,ScopeName,ScopeID,FreeAddresses,AddressesInUse, `
PercentInUse" |out-file -FilePath $CSVFile -append

foreach ($machine in $machines)
{
 $DHCPName = $machine.DnsName
 $AllScopes = Get-DhcpServerv4Scope -ComputerName $DHCPName
 foreach ($scope in $AllScopes) 
    {
     $ScopeName = $scope.Name
     $ScopeId = $scope.ScopeId
     $ScopeStats = Get-DhcpServerv4ScopeStatistics`
     -ScopeId $ScopeId -ComputerName $DHCPName
     $ScopePercentInUse = $ScopeStats.PercentageInUse
     $Addfree = $ScopeStats.AddressesFree
     $AddUse = $ScopeStats.AddressesInUse
     $OutInfo = $DHCPName + "," + $ScopeName + "," + $ScopeId`
     + "," + $Addfree + "," + $AddUse + "," + $ScopePercentInUse
     Write-Host $OutInfo
     Add-Content -Value $OutInfo -Path $CSVFile
    }
}
We can extend this solution to a scenario where we have to obtain average DHCP scope usage statistics for a whole week. I agree that this solution is not the most elegant way to approach the requirement, but we can have the job completed with this script.

Algorithm:

1> Import CSVs containing per day DHCP scope data in array format. 2> Combine the arrays produced, to form a single array of data and output as CSV file. 3> Remove Duplicate columns from the CSV produces in previous step, by assigning headers to each column and selecting the desired columns and output as CSV file. 4> Calculate average of each scope’s statistics from all seven week days and add this data as columns in CSV generated in previous step and output as final CSV file.

Script:

<#---Creating arrays of csv files for each weekday's 
and storing in seperate variables--#>

$csv1 = @(Get-Content "C:\Dhcp_Monday.csv")
$csv2 = @(Get-Content "C:\Dhcp_Tuesday.csv")
$csv3 = @(Get-Content "C:\Dhcp_Wednesday.csv")
$csv4 = @(Get-Content "C:\Dhcp_Thursday.csv")
$csv5 = @(Get-Content "C:\Dhcp_Friday.csv")
$csv6 = @(Get-Content "C:\Dhcp_Saturday.csv")
$csv7 = @(Get-Content "C:\Dhcp_Sunday.csv")

<#------Creating an array with combined csvs 
and giving output in csv-------------#>

$csv = @()
for ($i=0; $i -lt $csv1.Count; $i++) 
{
 $csv += $csv1[$i] + ', ' + $csv2[$i] + ', ' + $csv3[$i]+ ', ' + `
 $csv4[$i] + ', ' + $csv5[$i]+ ', ' + $csv6[$i] + ', ' + $csv7[$i]
}
$csv | Out-File "C:\consolidated.csv" -encoding default

<#---Removing duplicate or unwanted columns 
from combined csv from previous step----#>

$source = "C:\consolidated.csv"
$destination = "C:\consolidated2.csv"
$finaldestination = "C:\consolidatedfinal.csv"

(Import-CSV $source -Header 1,2,3,4,5,6,7,8,9,10,11,12,
13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,`
31,32,33,34,35,36,37,38,39,40,41,42 |
Select "1","2","3","4","5","6","10","11","12","16","17",`
"18","22","23","24","28","29","30","34","35","36","40",`
"41","42" |
 ConvertTo-Csv -NoTypeInformation |
 Select-Object -Skip 2) -replace '"' | Set-Content $destination

<#-----Performing averaging operation on 
selected columns and removing unwanted columns 
and giving final output in csv------#>

(Import-csv $destination -header "DHCP Server","Scope Name",`
"Scope ID",4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,`
21,22,23,24 | select "DHCP Server","Scope Name","Scope ID",`
@{Name="(Average of week) No. of IP Addresses free";`
Expression={[math]::Round(([Decimal]$_.4 + [Decimal]$_.7` 
+ [Decimal]$_.10 + [Decimal]$_.13 + [Decimal]$_.16 +`
[Decimal]$_.19 + [Decimal]$_.22)/7,2)}},`
@{Name="(Average of week) No. of IP Addresses in Use";`
Expression={[math]::Round(([Decimal]$_.5 + [Decimal]$_.8`
+ [Decimal]$_.11 + [Decimal]$_.14 + [Decimal]$_.17 +`
[Decimal]$_.20 + [Decimal]$_.23)/7,2)}},`
@{Name="(Average of week) Percent IP Addresses in Use";`
Expression={[math]::Round(([Decimal]$_.6 + [Decimal]$_.9`
+ [Decimal]$_.12 + [Decimal]$_.15 + [Decimal]$_.18 +`
[Decimal]$_.21 + [Decimal]$_.24)/7,2)}}|
ConvertTo-Csv -NoTypeInformation) -replace '"'|
Set-Content $finaldestination

<#-------Removing unwanted files---------#>

Remove-Item "C:\consolidated.csv"
Remove-Item "C:\consolidated2.csv"

#-------Creating new folder with 
current month's and week's name----------#>

$week = (Get-WmiObject Win32_LocalTime).weekinmonth
$month = get-date -Format MMMM

New-Item -ItemType directory -Path "C:\$month Week$week"

<#-----Moving the dhcp reports corresponding 
to each weekday's to new folder created----#>     

Move-item "C:\Dhcp_Monday.csv" "C:\$month Week$week"
Move-item "C:\Dhcp_Tuesday.csv" "C:\$month Week$week"
Move-item "C:\Dhcp_Wednesday.csv" "C:\$month Week$week"
Move-item "C:\Dhcp_Thursday.csv" "C:$month Week$week"
Move-item "C:\Dhcp_Friday.csv" "C:$month Week$week"
Move-item "C:\Dhcp_Saturday.csv" "C:$month Week$week"
Move-item "C:\Dhcp_Sunday.csv" "C:$month Week$week"
Move-item "C:\consolidatedfinal.csv" "C:$month Week$week"

All thanks to Adam Dimech’s Coding Blog https://code.adonline.id.au/csv-to-xlsx-powershell/,  we can have the final csv report converted to a more productive xlsx file, as follows.

<#----Assign the desired path for 
storing final xlsx file to a variable----#>     

$week = (Get-WmiObject Win32_LocalTime).weekinmonth
$month = get-date -Format MMMM
$csv = "C:\consolidatedfinal.csv" #Location of the source file
$xlsx = "C:\$month Week$week\consolidated.xlsx"  
$delimiter = "," 

<#------Create a new Excel workbook 
with one empty sheet-------------#>

[threading.thread]::CurrentThread.CurrentCulture = 'en-US'
$excel = New-Object -ComObject excel.application 
$excel.DisplayAlerts=$false
$workbook = $excel.Workbooks.Add(1)
$worksheet = $workbook.worksheets.Item(1)

<#-----Build the QueryTables.Add command 
and reformat the data--------------#>

$TxtConnector = ("TEXT;" + $csv)
$Connector = $worksheet.QueryTables.`
add($TxtConnector,$worksheet.Range("A1"))
$query = $worksheet.QueryTables.item($Connector.name)
$query.TextFileOtherDelimiter = $delimiter
$query.TextFileParseType  = 1
$query.TextFileColumnDataTypes = ,1 * $worksheet.Cells.Columns.Count
$query.AdjustColumnWidth = 1

<#------Execute & delete the import query------#>

$query.Refresh()
$query.Delete()

<#------Save & close the Workbook as XLSX-------#>

$workbook.SaveAs($xlsx)
$excel.Quit() 
Thanks to a brilliant module for PowerShell by Warren F http://ramblingcookiemonster.github.io/PSExcel-Intro/ , we can remove inactive dhcp servers’ data from the final xlsx generated. as follows.
Import-XLSX "C:\$month Week$week\consolidated.xlsx"|

Where-Object {($_.'DHCP Server' -notin @("server1.abc.corp",`
"server2.abc.corp"))}|Export-XLSX -Path`
"C:\$month Week$week\$month Week$week dhcp scope report.xlsx" 

<#---Remove unwanted consolidated csv file---#>

Remove-Item "C:\$month Week$week\consolidated.xlsx"