Recently, I saw the issue when after Exchange servers were rebooted, users were not able to access OWA, ECP, Outlook and powershell was not started (error 500, server is unavailable). According to information from IT administrator no changes were made (at least, he’s aware of).

Resolution to this case is described in internet already, though not Exchange related and there’s one step that we needed to perform was missing. Also, would be useful do describe troubleshooting steps applicable to Exchange Server.

🛠️First troubleshooting steps

So, first of all we verified IIS and protocol logs to see, if we are able to reach Backend Web Site, what the error is and get any additional information.

We have 4 common steps for Outlook (as example) protocol flow (mapi \ http) that are reflected in logs below:

1. C:\inetpub\logs\LogFiles\W3SVC1 – IIS logs on Default Web Site (default path). First, client connect to some Front End server from load balancing pool
2. C:\Program Files\Microsoft\Exchange Server\V15\Logging\HttpProxy\Mapi – next protocol logs on Front End for mapi\http (default location)
3. C:\inetpub\logs\LogFiles\W3SVC2 – IIS logs on Backend Web Site (default path). This is mailbox server where user’s mailbox is located (default location)
4. C:\Program Files\Microsoft\Exchange Server\V15\Logging\MapiHttp\Mailbox – protocol logs on Back End for mapi\http (default location)

First, we need to say that all logs contain records about connectivity for problematic user. So, it looks like, connections were successfully proxied to Back End. IIS logs contain errors with code 500, but no useful information were found in this one.

However, logs at step 4 contained some interesting information:

2023-04-18T19:42:30.713Z,70a73eaf-046a-4fa8-b478-d10a65d31bfc,{A5DFBDC1-6410-4BE5-B511-E1C761133DA0}:1,{705649C0-475D-4FBB-8B93-A5C52A73DADD},Connect,500,,,,5,Unknown,15,2,1118,9,,,,,8d40c8cb-69ce-4e78-9fef-b52dbad55b72@contoso.com,36d52cd0-dabf-4def-b39b-2ded3587be40,,,EX01.contoso.com,<null>,,,,{705B594A-2234-4DBD-A603-5E539C0D119E}:125480049,,,,Negotiate,,,,,,,,,Contoso\User1,,,cpn=M_ABR/RUM_ABR/RUM_ABRC/M_APAR/M_AER/;cpv=0/1/1/4/4/;,AMSI Scan HttpRequest=Microsoft.Exchange.HttpRequestFiltering.AmsiClientDotNet.AmsiException: Failed to AmsiInterop.AmsiScanBuffer with HResult -2147024875 —> System.IO.FileNotFoundException: The device is not ready. (Exception from HRESULT: 0x80070015)    at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode  IntPtr errorInfo)    at Microsoft.Exchange.HttpRequestFiltering.AmsiClientDotNet.AmsiHResultValidator.Validate(String operation  Int32 hresult)    — End of inner exception stack trace —    at Microsoft.Exchange.HttpRequestFiltering.AmsiClientDotNet.AmsiHResultValidator.Validate(String operation  Int32 hresult)    at Microsoft.Exchange.HttpRequestFiltering.AmsiClientDotNet.AmsiScan.ScanBuffer(Byte[] content  String contentName)    at Microsoft.Exchange.HttpRequestFiltering.HttpRequestFilteringModule.AmsiScanRequest(AmsiScan amsiScan  HttpRequest httpRequest)    at Microsoft.Exchange.HttpRequestFiltering.HttpRequestFilteringModule.OnBeginRequestInternal(HttpApplication httpApplication  IRequestDetailsLogger logger);

Although, we didn’t head about issues with AMSI integration recently, we decided to disable this feature temporarily to see, if it works. More details in that here 🔗More about AMSI integration with Exchange Server

It didn’t resolved issue with connectivity, but overall behavior changed a little bit and we’ve got some hints.

First, while trying to log in to OWA we’ve got some more meaningful description of the issue:

Also, in IIS Backend logs we saw records with same status codes.

#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken 

2023-04-18 19:13:56 10.0.0.4 POST /Microsoft-Server-ActiveSync/Proxy/default.eas User=testuser&DeviceId=AK8M8OC7T92939T917CESUP1KC&DeviceType=iPhone&Cmd=Ping 444 Contoso\testuser 10.0.0.5 Apple-iPhone14C2/2005.252 – 500 19 126 17

Description of error code 126 means, that we’re trying to load some module that cannot be found.

The description of this error is quite clear and described in here: 🔗HTTP Error 500.19 on Internet Information Services (IIS) webpages – Internet Information Services | Microsoft Learn

We reviewed content of IIS config file located here C:\Windows\System32\inetsrv\config\ApplicationHost.config

And found that, it has references to WSUS library that doesn’t exist anymore.

scheme name=”xpress” doStaticCompression=”false” doDynamicCompression=”true” dll=”C:\Program Files\Update Services\WebServices\suscomp.dll” staticCompressionLevel=”10 ” dynamicCompressionLevel=”0″ />

The XPress compression schema module (suscomp.dll), which gets installed with WSUS, is defined globally and loads in every Application Pool. As I mentioned, this is described in many blogs already. But for us it doesn’t help to remove the above section only.

We also needed to remove section related to express compression filters:

<key path=”LM/W3SVC/FILTERS/COMPRESSION/xpress”>

                <property id=”2243″ dataType=”DWord” userType=”1″ attributes=”None” value=”0″ />

                <property id=”2215″ dataType=”DWord” userType=”1″ attributes=”None” value=”0″ />

                <property id=”2238″ dataType=”MultiSZ” userType=”1″ attributes=”None” value=”” />

                <property id=”2240″ dataType=”DWord” userType=”1″ attributes=”None” value=”1″ />

                <property id=”2244″ dataType=”MultiSZ” userType=”1″ attributes=”None” value=”asmx ” />

            </key>

After IIS was restarted, issue was resolved.

So, in the end, we figured out that contractor, with whom customer worked with, installed WSUS on Exchange Servers by mistake. Than, WSUS was uninstalled, but Exchange servers were rebooted after some time. WSUS doesn’t unroll all his changes correctly. Compression settings, dedicated pool remain intact (maybe something else, who knows).

Remember, it’s mostly BAD idea to install some additional services  \  software on Exchange Server.

End.