Shadow redundancy was introduced in Exchange 2010 to provide redundant copies of messages before they’re delivered to mailboxes. It’s not a purpose if this post to describe, how this functionality works. Instead, we will look at real life practical example on how to troubleshoot issues with Shadow Redundancy.

If you’re not familiar with this one, review official tech doc here: Shadow redundancy in Exchange Server

🧩Specific issue description

Assume, that our Exchange environment experience issues with duplicate email deliveries. This happens from time to time with different senders and recipients. We should expect, that we cannot trust any statistics. In Exchange Server we have duplicate delivery prevention option, i.e. message won’t be delivered twice, if it was delivered to mailbox already. Thus, theoretically this issue can happen more often, than noticeable by users.

But anyway, let’s try to troubleshoot this issue.

🔄Troubleshooting steps

First thing I would ask for – examples of headers for duplicated messages. And message tracking logs for corresponding message id’s.

Let’s look at some useful information from headers.

This message was delivered first and overall looks normal. EndToEndLatency is small, AuthMechanism equals to 07, that means, that message was sent via smtp with authentication.

X-MS-Exchange-Organization-Network-Message-Id: 25cd2baf-60c1-4de7-8dc0-08db7048beb7
X-MS-Exchange-Organization-AuthSource: tr-ex1
X-MS-Exchange-Organization-AuthAs: Internal
X-MS-Exchange-Organization-AuthMechanism: 07
X-MS-Exchange-Transport-EndToEndLatency: 00:00:00.2310237

Second example – message was delivered after some time. EndToEndLatency is about 1 hour, we can see reference that it was sent with ShadowRedundancy, that confirmed by AuthMechanism that equals to 0c.

Received: from ex-tr2 (10.0.0.2) by ex-tr1
(10.0.0.1) with ShadowRedundancy id 15.2.922.19; Wed, 28 Jun 2023
23:07:57 +0000
X-MS-Exchange-Organization-Network-Message-Id: 7c5ed59c-60c1-4de7-8dc0-dbeb14920b77
X-MS-Exchange-Organization-AuthAs: Internal
X-MS-Exchange-Organization-AuthMechanism: 0c
X-MS-Exchange-Transport-EndToEndLatency: 00:58:07.1796301

From here, we can confirm, that message was resubmitted by Shadow Redundancy. What’s interesting, Network-Message-Id is different for these messages, that’s why message was delivered twice against duplicate delivery. I wasn’t able to reproduce this issue in my test environment, but Exchange version in this case is really specific and was released 2 years ago.

Let’s look at message tracking logs from test environment.

HARECEIVE and HAREDIRECT events mean, that shadow copy was created successfully. But no HADISCARD events were registered for this message. That means, that message is still in shadow queue.

In real life example, message was resubmitted from shadow queue after some time. Let’s look at this process in more detail.

Server TR-EX2 created shadow copy of message on server TR-EX1. Corresponding queue state on TR-EX1:

If we dig into message tracking log, we can also notice, that Exchange is not able to create shadow copies on TR-EX2.

“Service not available” usually means, that we have issues with network connectivity, appropriate connector doesn’t exist, or some settings don’t correspond to what we expect.

Even more, we can investigate connectivity logs to see, what’s happening where.

HUB connectivity logs show the same error message (C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\Hub\Connectivity):

2023-06-28T11:40:28.843Z,08DB745BC9DF51B3,SMTP,tr-ex2.contoso.com,>,Established connection to 10.0.0.102
2023-06-28T11:40:28.843Z,08DB745BC9DF51B3,SMTP,tr-ex2.contoso.com,-,Messages: 0 Bytes: 0 (Retry : Service not available)

We can also expect different events in OS Application logs. But that depends from version \ issue.

For mail submission from database and shadow redundancy we use Default receive connector created upon Exchange Server installation. Not all internal communications are logged by default. To enable this for transport you should at least complete those configuration steps (for this specific issue):

Set-ReceiveConnector “TR-EX1\Default TR-EX1” -ProtocolLoggingLevel verbose

Set-TransportService TR-EX1 -IntraOrgConnectorProtocolLoggingLevel Verbose

So, the same issue we can see from SmtpSend logs (C:\Program Files\Microsoft\Exchange Server\V15\TransportRoles\Logs\Hub\ProtocolLog\SmtpSend) while tr-ex1 is trying to create shadow copy on tr-ex2.

Fields: date-time,connector-id,session-id,sequence-number,local-endpoint,remote-endpoint,event,data,context

2023-06-28T11:00:28.578Z,Intra-Organization SMTP Send Connector,08DB745BC9DF5177,0,,10.0.0.102:2525,,None,Set Session Permissions 2023-06-28T11:00:28.578Z,Intra-Organization SMTP Send Connector,08DB745BC9DF5177,1,,10.0.0.102:2525,,,attempting to connect
2023-06-28T11:00:28.579Z,Intra-Organization SMTP Send Connector,08DB745BC9DF5177,2,10.0.0.101:33826,10.0.0.102:2525,+,,
2023-06-28T11:00:28.579Z,Intra-Organization SMTP Send Connector,08DB745BC9DF5177,3,10.0.0.101:33826,10.0.0.102:2525,<,421 4.3.2 Service not available,
2023-06-28T11:00:28.579Z,Intra-Organization SMTP Send Connector,08DB745BC9DF5177,4,10.0.0.101:33826,10.0.0.102:2525,*,,Dropping connection because server is not accepting mail. Server response :421 4.3.2 Service not available

Let’s verify settings for appropriate connector on TR-EX2. Of cause, there can be another settings that can break all the staff. But these ones causing issues more often.

Everything looks OK despite of RemoteIPRanges. That property enables connections only from one IP address and this is not Exchange Server.

💡Note. Changes on Default connectors are not recommended and not desired. In this configuration unwanted changes broke Shadow Redundancy and Submission. That actually created a single point of Failure.

Let’s fix connector properties.

Now we can see corresponding events from smtp send logs:

2023-06-28T12:06:29.024Z,Intra-Organization SMTP Send Connector,08DB745BC9DF51DA,18,10.0.0.101:38607,10.0.0.102:2525,<,250 8HRSl3D3P0m2abo/1qRlww==, 2023-06-28T12:06:29.024Z,Intra-Organization SMTP Send Connector,08DB745BC9DF51DA,19,10.0.0.101:38607,10.0.0.102:2525,>,XQDISCARD 50,

In this case, services wasn’t restored after we fixed connector settings. For troubleshooting purposes we can use Get-ExchangeDiagnosticInfo -Server tr-ex2 -Process EdgeTransport -Component ShadowRedundancy -Argument Verbose

That cmdlets show configuration settings for specific component in one place and also provides some diagnostics information.

In our case LastHeartbeatTime showed that last attempt was before we fixed issue with configuration. Transport service and server were not restarted since than. So, we decided to restart service and issue was resolved.

In last Cu’s there are attempts to reset heartbeat state. So probably, this is not the case for latest versions.

End.