XenApp 6.5 on Server 2008 R2 does not allow RDP Connections after a Hard Reboot

I recently observed some strange behavior on a Citrix XenApp 6.5 application server. After some piece of software crashed on this server (still working on this, perhaps csrss.exe contention), I was not able to establish an RDP session to this box. While looking at this server from the Citrix AppCenter, I saw that all 35 sessions were listed as DISCONNECTED. I could not connect from the console or RDP, just the command line. Instead of trying to kill these existing disconnected sessions from the command line, I hard-rebooted this server.

This was a mistake. When the server came back up I could now connect at the console and with SCCM, however I could not RDP to the server. Every time I started an RDP session to the server, the mstsc client would authenticate then immediately close out. It would flash a correctly-sized RDP window momentarily than just disappear and close out.

After troubleshooting for a few hours, I noticed the output of the qwinsta command:

C:\Users\msp>qwinsta /counter
 SESSIONNAME       USERNAME                 ID  STATE   TYPE        DEVICE
 services                                    0  Disc
 console                                     1  Conn
 ica-tcp                                 65536  Listen
 rdp-tcp                                 65537  Listen
Total sessions created: 1
Total sessions disconnected: 6
Total sessions reconnected: 0

The “Total sessions disconnected” counter would increment every time I attempted to connect. At least now I knew the RDS services were not crashing, and were actually servicing my requests. I was also seeing various eventlog errors about Citrix HealthMon not allowing this server to accept new connections:

windows event vwr log showing citrix health mon errorAfter doing some more digging, I found this excellent article here. It looks like similar issues, which lead to me finding the Citrix ICA Session registry key (HKLM\Software\Citrix\ICA\Session):

regedit opened to citrx ICA Session keyFrom here, I could see all the disconnected sessions that existed on the server before it was hard-rebooted. Citrix did not have a chance to clean this up. I deleted and recreated the Session key. Make sure to re-add permissions for the NT SERVICE\TermService account to read and write to this key. After doing this, I was immediately able to RDP back into this server. It seems as though something in Citrix was still looking at this key to determine either some kind of load balancing or licensing and causing the sessions to disconnect. I believe that if I was able to manually force a high session number I would’ve been able to create a new RDP session.

Powershell for Parsing Logon and Logoff events from Windows Security Logs

$outputfile = @()
$ID = 4624,4634
Get-ChildItem -include *Security*.evtx, Archive*.evt, Archive*.evtx -Path C:\Windows\System32\winevt\Logs -recurse |
ForEach-Object {
"Parsing $($_.fullname)`r`n" >> .\Logging.txt
Try
{
Get-WinEvent -FilterHashtable @{path=$_.fullname;logname='Security';ID=$ID; data='S-1-5-21-XXXXXXXXXX-XXXXXXXXXX-XXXXXXXXXX-XXXXX'; ProviderName='Microsoft-Windows-Security-Auditing';} -EA Stop >> .\Logging.txt
}
Catch [System.Exception]
{
"No logon events in current log" >> .\Logging.txt
}
}

The code above will run through Windows Log directory (C:\Windows\System32\WinEvt\Logs\) and search Security logs for any Logon and Logoff Event IDs. This can easily be modified to search other evtx logs for any ID, just edit as necessary.