Exchange Email Delivery Year 2022 bug resolved by Microsoft

On-premise Exchange 2016 and 2019 servers encountered a bug as we moved in 2022. The bug is being called the Year 2022 bug. The format of the date on the Exchange server could not accommodate the Year 2022 dates in the pre-existing 32-bit integer variable. Due to the inability of the Exchange servers to use the Year 2022 dates, the email delivery got affected. Transport queues on the Microsoft Exchange servers got full, as the server was unable to process the delivery in the absence of a proper and recognizable format of the date.

This issue manifested on 1st January 2022. Thankfully, most businesses were closed on account of the new year holidays. Microsoft has released a fix to ensure that the Year 2022 dates can be recognized by the Microsoft Exchange servers. The resolution involves running PowerShell scripts on the Exchange servers, to resolve the antimalware engine’s issue. It has also shared the two alternative approaches to fixing the Year 2022 bug on the Microsoft Exchange servers. Remember, this affects the on-premium Exchange 2016 and Exchange 2019 servers only.

Also remember, that this is not a security vulnerability or deficiency in any of the core server services on the Exchange 2016 and Exchange 2019 servers.

How do I know if I am affected by the Exchange 2016 and Exchange 2019 the Year 2022 bug?

For a start, a stalled transport queue on the Exchange servers is the first sign that your on-premise Exchange server is affected. To validate the issue, check the application log of the Event viewer on your Exchange Server. If you see entries with an error event code of 5300 or 1106 on FIPFS, you are impacted by the Year 2022 bug.

Another way to confirm is by checking the error description in the application logs. Here are two entries for the description on the application log:

Description: The FIP-FS "Microsoft" Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can't convert "2201010001"

Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.

If you see these description entries on the application log of the Exchange server 2016 and 2019, it is a confirmation of the Year 2022 bug affecting the Exchange servers 2016 and 2019.

How can I fix the Year 2022 bug on Exchange 2016 and Exchange 2019 servers?

The first thing to note is that the Year 2022 bug only affects the Exchange 2016 and Exchange 2019 servers that are hosted on-premise. If you are on Microsoft 365, there is nothing that needs to be done on your part. If your on-premise Exchange server is affected by the Year 2022 bug, you can apply a patch released by Microsoft. You could choose to deploy the patch automatically by executing a script. Or, you can take a manual approach to clean the transport queues and apply the patch manually. We discuss both ways to patch the Exchange servers 2016 or Exchange servers 2019 against the Year 2022 bug. This seems like the Y2K bug that hit servers when the century changed to 2000. It seems that we are reliving the Y2K days again.

Microsoft has stated on its blog that the issue is with the inability of the antimalware engine on the Exchange server 2016 and Exchange server 2019 to work with the signature file. The corrupt signature file cannot be validated by the antimalware engine. This causes the antimalware engine to fail, and stall the email delivery process on the Exchange servers.

The impact of the Year 2022 bug is on Exchange Servers 2016 and 2019 that download antimalware updates. Edge servers on the Exchange network are not affected by the bug. Automated or manual fixes have to be applied on the core exchange servers that perform the antimalware checks on the email delivery pipeline.

Resolve Year 2022 bug on Exchange automatically

The automated approach involves downloading the PowerShell script and executing it on the Exchange servers that download antimalware updates. The key is applying the PowerShell script ONLY on servers that download the antimalware updates. You may have to enable the execution of remote signed PowerShell scripts on the target Exchange servers.

The PowerShell script is available for download from https://aka.ms/ResetScanEngineVersion.

So, the bottom line is that for the automated solution, you will still need to download and execute the PowerShell script offered by Microsoft.

Manual resolution of the Year 2022 bug on Exchange Server 2016 and 2019

You can resolve the Year 2022 bug on Exchange Servers 2016 and 2019 through a manual process. As part of the process, you need to update the UpdateEngine on the Exchange server to version 2112330001 or higher.

As part of the process, you need to:

• Validate if your exchange engine is affected by the bug. Run Get-EngineUpdateInformation and check the update engine version. If it starts with 21, you remain unaffected and there is no further action that needs to be taken. If it starts with 22, you will need to remove the data engine definitions and update the Exchange engine.
• Stop the Microsoft Filtering Management service.
• Stop the Transport Services.
• Do make sure that updateservice.exe is not running. You can verify it through the task manager.
• Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
• Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.
• Start the Microsoft Filtering Management Service and the Transport Services.
• From the Scripts folder on the Exchange Server – %ProgramFiles%\Microsoft\Exchange Server\V15\Scripts – Update-MalwareFilteringServer.ps1 <server FQDN>. This will update the malware definitions on the affected Exchange Servers.

You can verify if the bug has been resolved on the Exchange Servers by validating the version of the UpdateEngine. If the version number is 2112330001 or higher, then the issue is resolved. You can check the current version of the UpdateEngine using the following command:

• In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
• Run Get-EngineUpdateInformation.

Please do make sure that the UpdateEngine is running on version 2112330001 or higher.

You can also verify from the application logs. If the issue is resolved, the application logs will be clean in respect of the error events 5300 and 1106 on FIPFS.

Between the automated and manual approaches, we prefer using the manual approach as we are in better control of the entire process of deleting the antimalware definitions and then updating the engine to download the latest antimalware definitions that are compatible with the 2022 Year date.

You may read more about the bug and its fixes on the Microsoft blog.

Summary

The issue with delays in email delivery or stalled email queues on Exchange Servers 2016 and 2019 due to the Year 2022 bug can be resolved by updating the engine of the antimalware services on the core exchange servers that download the antimalware updates. Bear in mind that the resolution needs to be applied only on the Exchange servers that handle antimalware updates and antimalware definitions.