 |  | | Dec 14, 2024
(updated Jan 8, 2025)
By Lance Zakin, HCL CASA, CAAD
Notes and Domino
NotesMail - HCL BP | The Notes app tool below is a workaround fix which automatically resolves the mail routing issue in HCL knowledge article KB0118192, but not the Mail Rules issue. However, refer to step 5 below to semi-automatically or possibly automatically fix the Mail Rules. It supports Domino 14 and lower including End of Life (EOL) versions 10.0.x, 9.0.x, 8.5.x, 7.0.x, as well as, single and multiple mail router mailboxes.  | |
|
The critical bug was dormant for 35 years and activated at the worst possible time during the holiday season. If you are an unlicensed Domino customer you most likely are feeling frustrated. The tool is a fully supported workaround fix by NotesMail - HCL Business Partner. If you don't have access to the HCL fix packs and interim fixes (hotfixes), then you should use this tool.
Open a service case with NotesMail if you have any questions regarding this tool, otherwise, you can download it below. After clicking the download link you must provide your Domino gateway mail server names in the "Domino Server" field. i.e. */US/Company, Server1/US/Company, etc. Note: We require Domino server names (or wildcards names) since we need to register them in the software due to our 2025 security policy to prevent software piracy. "The Business Software Alliance (BSA) estimates that the commercial value of unlicensed software is nearly $46 billion per year."
Workaround Fix
1. Install "Auto Router Bug Fix" (ARF) Notes app tool on each Domino gateway mail server...
Download: Auto Router Bug Fix (ARF) for Domino
 
A. Create a non-replica copy of ARF on each gateway mail server in the root or a custom folder.
B. Open ARF, then click the "Enable scheduler" button using a Domino administrator or developer Notes ID which has (1) ACL Access to the mail router mailboxes on the gateway servers [i.e. mail.box, mail1.box, mail2.box, mail3.box] and (2) Access to run LotusScript/Java agents in the Domino Directory on the gateway mail server records as seen below.
i.e. Domino Directory server record on gateway mail servers
Programmability Restrictions
Sign or run restricted LotusScript/Java agents: LocalDominoAdmins, Domino Administrators, Domino Designers
C. Open your gateway mail server logs to verify ARF is running. If it starts and finishes as seen below, then proceeed to step 2. Note: You might need to wait up to 10 minutes. And to verify it is scheduled to run you can issue server console command: tell amgr sched
i.e. Domino server log.nsf
12/20/2024 02:39:37 AM AMgr: Agent ('Auto Router Bug Fix' in 'admin\router-bugfix.nsf') printing: Auto Router Bug Fix: © 1997-2025 IVE Technologies LLC (dba NotesMail), All Rights Reserved. www.NotesMail.com
12/20/2024 02:39:37 AM AMgr: Agent ('Auto Router Bug Fix' in 'admin\router-bugfix.nsf') printing: Auto Router Bug Fix: Started.
12/20/2024 02:39:37 AM AMgr: Agent ('Auto Router Bug Fix' in 'admin\router-bugfix.nsf') printing: Auto Router Bug Fix: Reminder - Held messages are processed on schedule every 10 minutes
12/20/2024 02:39:37 AM AMgr: Agent ('Auto Router Bug Fix' in 'admin\router-bugfix.nsf') printing: Auto Router Bug Fix: Processing held messages in mail router mailbox mail1.box ...
12/20/2024 02:39:37 AM AMgr: Agent ('Auto Router Bug Fix' in 'admin\router-bugfix.nsf') printing: Auto Router Bug Fix: Scanning 8 messages in mail1.box ...
12/20/2024 02:39:37 AM AMgr: Agent ('Auto Router Bug Fix' in 'admin\router-bugfix.nsf') printing: Auto Router Bug Fix: Processed 1 held messages in mail1.box .
12/20/2024 02:39:37 AM AMgr: Agent ('Auto Router Bug Fix' in 'admin\router-bugfix.nsf') printing: Auto Router Bug Fix: Processing held messages in mail router mailbox mail2.box ...
12/20/2024 02:39:37 AM AMgr: Agent ('Auto Router Bug Fix' in 'admin\router-bugfix.nsf') printing: Auto Router Bug Fix: Scanning 8 messages in mail2.box ...
12/20/2024 02:39:37 AM AMgr: Agent ('Auto Router Bug Fix' in 'admin\router-bugfix.nsf') printing: Auto Router Bug Fix: Held messages not found in mail2.box
12/20/2024 02:39:37 AM AMgr: Agent ('Auto Router Bug Fix' in 'admin\router-bugfix.nsf') printing: Auto Router Bug Fix: Finished.
2. Configure your Domino gateway mail servers (last hop before relaying to SMTP server) to hold undeliverable mail...
A. Open the Domino Directory on your admin server using your Notes client, then click Configurations - Servers - Configurations in the left frame menu as seen below.
B. Edit each gateway mail server configuration record. Click Router/SMTP tab - Advanced tab - Controls tab a seen below, then in the "Undeliverable Mail "section enable the "Hold undeliverable mail" field and click Save & Close button. Note: Verify this change replicated from the admin server to the gateway mail servers.
C. Issue console command on each gateway mail server: restart task router
 
3. The bug causes Domino to only use a single mail.box regardless of settings in the Domino server configuration record, so you must change it on the unpatched gateway mail servers (last hop before relaying to SMTP server) to reflect that it is only using a single mail.box since ARF checks this setting to determine which mail router boxes to process...
Note: In the scenario where your gateway mail servers are patched, but you did not patch your other Domino 10 or lower servers with the HCL interfix fix (available with extended support), then you can skip this step.
A. Open the Domino Directory on the admin server using your Notes client, then click Configurations - Servers - Configurations in the left frame menu as seen below.
B. Edit each gateway mail server configuration record which has not been patched by the HCL interim fix. Click Router/SMTP tab - Basics tab as seen below, then in the "Router/SMTP Basics" section verify the "Number of mailboxes" field value is set to 1, otherwise, change it to 1 and click Save & Close button. Note: Verify this change replicated from the admin server to the gateway mail servers.
C. If you changed the mailbox number in the step 3B above, then issue console command on each gateway mail server: restart server
D. If the number of mailboxes was originally higher than 1 in step 3B above, then you will will need to copy/paste the messages from the router mailboxes (i.e. mail1.box, mail2.box, mail3.box) to mail.box after the server restarts.
 
4. Restarting the unpatched Domino servers...
A. If you did not restart your unpatched servers after Dec 12, 2024, then optionally restart them by issuing console command: restart server
B. If you already restarted your unpatched servers after Dec 12, 2024 before installing this tool... Open the mail router mailboxes on the gateway mail servers, then click "Resend All Dead Messages To Original Intended Recipient" as seen below. Note: Alternatively, if you have many dead messages prior to day you rebooted the server after Dec 12, you can select all dead messages starting from the reboot date, then click "Resend Selected Dead Messages To Original Intended Recipient".
5. If Mail Rules are not executing, then you can replace your mailbox designs with the Notes 9.0.1 FP9 or higher mail template and users can periodically click More - Mail Rules - Run Rules from the Inbox. However, the user must be upgraded to Notes 9.0.1 FP9 (Standard) or higher. The Domino server version most likely can be running any version, for example, 8.5 or higher.
A. Verify the users have upgraded to Notes 9.0.1 FP9.
B. Open the Notes (Standard) client, then click Mail - Enable Run Rules on Existing Mail as seen below. Note: This setting can be automatically pushed out to all users via a Domino Mail Settings policy on a Domino Server 9.0.1 FP9 or higher.
C. Close all mail tabs or restart Notes.
D. The users can open the Inbox as seen below, then click: More - Mail Rules - Run Rules
Question: Is it possible to automatically run Mail Rules for all users? If you are running Domino 9.0.1 FP9 server with Notes 9.0.1 FP9 mailbox designs it might be possible to create a scheduled agent in your user's mail template to automatically run the user's Mail Rules by simulating step 5D above. The agent will use the "Before new email arrives trigger" which is run by the Router server task and not the Agent Manager. This is still under investigation.
Reference Sources
CRITICAL ALERT: Mail not routing after Domino restarts beginning December 13, 2024
The HCL Domino Development team has identified an issue which will affect *ALL Domino server versions* as of December 13, 2024. If you restart your server, a router error will occur which results in delivery failures due to a routing loop. Also mail rules will also start failing. In addition, Domino will use only a single mail.box regardless of what the configuration document calls for.
Dec 13th Defect Fix Guidance for IBM Domino v9.0.x and v10.0.x Customers
Customers using IBM Domino 9.0.1 or 10.0.x or earlier... HCL has created an interim fix which can be installed on top of the latest Fix Packs, for Domino 9.0.1 FP10 and Domino 10.0.1FP8. Interim fixes for other releases in the IBM Domino v9.0.1 and 10.0.x code stream will be made based on demand and whether we can build them. Please open a support ticket with your request. Only customers with a valid entitlement and active Support and Extended Support for v9.0.x and/or v10.0.x can receive and apply the fix. Any HCL Domino fixes or patches are the Intellectual Property of HCLSoftware, any unauthorized use, reproduction, or distribution is strictly prohibited and would infringe on HCLSoftware’s IP rights. For customers with prior releases, e.g., 8.5.3 and earlier, the only option is to upgrade to at least v9.0.1 Fix Pack 10, preferably to Domino v12.0.2 FP5 or later as this release is in standard support.
Mail rules on existing emails in HCL Notes Client
Applies to HCL Notes 9.0.1 FP9 and later versions... Can Notes Client mail rules run on existing emails? Using the 'Enable Run Rules On Existing Mail' setting in the Mail Settings policy, you can run mail rules on messages already stored in folders. For this feature to work, you must have an upgraded mail file that uses the Mail9.ntf template included with 9.0.1 Feature Pack 9. Whenever you create a new Rule, you will see Additional Options (Run Now on Inbox & Run Now on All documents).
tags: domino, download, critical, alert, bug, mail, routing, stuck, loop, restart, issue, fix pack, interim fix, hotfix, patch, dec 13, kb0118192, ibm, lotus, notes | |