HCL got a big Friday the 13:th problem, due to a bug created back in early days of the Domino server when no one saw anything beyond 5 years ahead as a potential problem. Simplified what I’ve read is that a limit in an calculation formula for epoch dates created this problem that made the function return -1 instead of an incremental number, a complete writeup will come later from HCL in the technote.
This makes the mail routing to stop working if you have more than one mailbox, routing between servers to get to the internet. Read the technote here it’s updated continuously and if you can apply the hotfixes do so.
https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0118192
HCL is as always quick on the problem and did within 24 hours identify the problem and did release fixes for the domino versions that is active in maintenance now. A big shoutout to all developers that has worked to getting this out so fast for version 11,12 and 14 (Fixes is uploaded where you can download other fixes), if you still are on version 9 or 10 you could get a fix if you are on an extended support plan by opening a support ticket.
So for most companies that has servers that is on any earlier version than 11 might have potential problems with mail routing and server mail rules.
My initial findings is that if you only have one server, one mail.box no mail rules you are not affected by mail deliver problems, it could be other things that stops working other than mail rules because the problem makes settings not being loaded from the config.
NO PANIC FOR ADMINS with the above config regarding mail delivery
(comment if you have found something)
I usually like to try checking out if there is an workaround, I started to check if there was a notes.ini parameter to set Max mail hops but I couldn’t find that one.
Then I switch over to check what the problem was and found that removing all extra mail.box databases so you only have one (mail.box). This makes the emails being sent out if you run the send agent on that server, that fixes some of the problems.
Emails being routed from other servers only routes one server from the sent server and then gets stuck. But if I use release held email they are being routed. Hmmm, so I created an scheduled agent in another database and check for emails in the mail.box that has the specific attributes and the field $hops set to 0.
That actually releases and sends the emails in my environment, It doesn’t solve problems with mail rules and potential everything but some is my findings. It needs to be on all servers that get emails from other servers sent to them to be routed.
Of course the hotfixes from HCL is always the best to apply but if you don’t have the possibility this might help you temporary to get this working with mail routing without the hotfixes for a while, until you can upgrade the server.
Adam wrote a good post about ways to get the patches with / without your agreement with HCL. http://ab1osborne.blogspot.com/2024/12/your-support-options-for-friday-13th.html?m=1
If you are located in Sweden and have licensing questions feel free to reach out to me thru this page or thru my company QNOVA if you are located in some other part of the world check out the partner locator
Another this that has come up is that because the problem makes the server go back to 1 mailbox and the fix is using the correct number again. Remember that emails can then be stuck in mail.box that you need to copy over to one of the active boxes after the patch.
Also you need to have “Hold undeliverable mail” set to enable
Older version of the nab template
If the interim installer is failing with the error message “Notes/Domino related process is still running” – see Installing FixPack/Hotfix on Windows fails due to Windows Management Instrumentation Service KB0038497
https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0038497
This is the code I have in the extra database with the scheduled agent. ( You can’t add it to mail.box )
This code is used a your own risk, without any HCL support and it will NOT fix all problems related to this bug.
Dim session As New NotesSession
Dim mailboxdb As New NotesDatabase(session.Currentdatabase.server,"mail.box")
Dim v As NotesView
Dim doc As notesdocument
If mailboxdb.Isopen Then
Set v=mailboxdb.Getview("mail")
Call v.Refresh()
v.Autoupdate=false
Set doc=v.Getfirstdocument()
Do While Not doc Is Nothing
If Not doc.hasitem("hopsretry") Then
If doc.Getitemvalue("RoutingState")(0)="HOLD" Then
If CStr(doc.Getitemvalue("$hops")(0))="0" then
Call doc.Replaceitemvalue("RoutingState","")
Call doc.Removeitem("SaveOptions")
If doc.Hasitem("RoutingState") And
Not doc.getitemvalue("RoutingState")(0)="" Then
'Keep Form
Else
If doc.Hasitem("MailSavedForm") And
Not doc.getitemvalue("MailSavedForm")(0)="" Then
Call doc.Replaceitemvalue("Form",doc.Getitemvalue("MailSavedForm"))
ElseIf doc.Hasitem("AppointmentType") Then
Call doc.Replaceitemvalue("Form","Notice")
Else
Call doc.Replaceitemvalue("Form","Memo")
End If
End If
Call doc.Replaceitemvalue("$Hops",25)
Call doc.Removeitem("RoutingStateBy")
Call doc.Replaceitemvalue("hopsretry",1)
Call doc.save(True,False)
End if
End If
End If
Set doc=v.Getnextdocument(doc)
Loop
End if
Thank you for posting, Fredrik. I’m sure you have helped many people in a very difficult situation.
Thank you for your post about what options everyone have to get patches.
Hi
Does this address
Maximum hop count exceeded. Message probably in a routing loop. We are continuously receiving these mails in the mail server log and mail not received when we have this issue
If you check the $hops field on the emails that is stuck. Or try to release them, if that works then you could use my code in an agent.
But best is always patching.
@Fredrik, many thanks for your post!
regarding multiple mail.box dbs: should the agent also treat those, or are they becoming irrelevant once the bug strikes?
The problem makes Domino to only use mail.box and this agent will only look in mail.box.
understood. My question was: should I extend the scope of the agent to also handle other mail*.box dbs on ther same server. But I think your answer is “no” 😀
Thx once again
You are correct, without the patch only mail.box is used
I do not have access on “My HCL Software Portal.” my domino server stopped working.
Is there another place where i can download the patch for V12.02 ?
Contact HCL Support to get access again
I am not a subscriber of HCL , my subscription have expired last year
They put a time bomb in their software , they block all users in the world, and they cannot give the patch for free ?
They prefer to make ton of money on their issue by forcing everyone which forgot to pay them to pay now. Is there more problems to come in the futur ?
There is still a few users which didnt migrated to exchange, I think they regret it now.
Then the time bomb was created in the mid 90:s according to what I’ve heard. Of course someone should have seen this coming but mistakes happens.
It must have been lots of developers during the Lotus, IBM and now HCL that didn’t anticipated this.
But lots of users has got it to work with my code above and changing their setup some, try it out if you haven’t.
Congratulation for your code, it is nice to help us
Great piece of code !!
Note : I am disapointed so much by HCL, they just asked me to pay, they could send the fix for free
A shame to ask money for their own mistake
Thanks for posting
Any idea when a fix for Domino 9.0.1 Windows would be available?
It is available now if you open a support ticket but only for customers with an active extended support package.
Or a customer that purchase one, check with your HCL partner.
If you do not have one send me an message thru contact at the top of the page.
I actually work for a company that is a HCL Domino Business Partner
When HCL moved downloads to My HCLSoftware portal https://my.hcltechsw.com/downloads I was surprised to find out (and also confirmed by HCL Partner Support) that in order to access the EOL Domino 9.0.1/10.x downloads, business partners too would need to purchase the 9.0.1/10.x extended support and service optio.
I found that policy wise inconsistent considering we already had an active business partner pack agreement in place.
Well, the partner pack includes versions in active maintenance so not that weird that the 9.01 fix is not included.
We BP:s should use supported versions of the software 😉
Oh, I see it’s already available if you purcahse the 12 month extended S&S for 10k USD.
https://support.hcl-software.com/csm?id=kb_article&sysparm_article=KB0118252
LOL
A shame to ask money for their own mistake
I hope this is not going to give them idea for the futur
Hi Fredrik.
In case of different domains inside one firm, the code is not enough to hop between internal domain.
Otherwise brilliant, as usual !
The instructions in this link contain required steps including a Notes app tool you can download called “Auto Router Bug Fix” (ARF) to temporarily resolve the issue mentioned above in HCL knowledge article KB0118192.
https://www.notesmail.com/home.nsf/tip20241213?openpage
No it does not, that is another solution created by someone else. My solution is the code at the bottom of this post.
Hello Fredrik,
thanks for the code you wrote in attempt to fix the very-late-millennium-bug of Domino.
I’ve modified the script (some problem identified in your original script… starting from the missing/truncated “end if” and then I found some other problems in my Domino infrastructure, and I rearranged it to my needs)
I’m not a developer, so my code is horrible and feel free to use it or just trow it away. Obviously i don’t guarantee nothing at all, but you gave me some ideas to work on and it seems right to me to give back the code with the modification I made.
Best regards and Merry Christmas
%REM
Agent Fix_mailbox_localserver
Created Dec 19, 2024 by Luca B.
Description: Comments for Agent
%END REM
Option Public
Option Declare
Sub Initialize
Dim session As New NotesSession
Dim mailboxdb As New NotesDatabase(session.Currentdatabase.server,”mail.box”)
Dim v As NotesView
Dim doc As NotesDocument
If mailboxdb.Isopen Then
Print “start”
Set v=mailboxdb.Getview(“Mail”)
Call v.Refresh()
v.Autoupdate=False
Set doc=v.Getfirstdocument()
Do While Not doc Is Nothing
If Not doc.hasitem(“hopsretry”) Then
‘Print “hopretry”
‘Print doc.Getitemvalue(“IntendedRecipient”)(0)
If doc.Getitemvalue(“RoutingState”)(0)=”HOLD” Or doc.Getitemvalue(“RoutingState”)(0)=”DEAD” Then
If CStr(doc.Getitemvalue(“$hops”)(0))=”0″ Then
Call doc.Replaceitemvalue(“RoutingState”,””)
If doc.hasitem(“IntendedRecipient”) Then
Print doc.Getitemvalue(“IntendedRecipient”)(0)
Call doc.Replaceitemvalue(“Recipients”,doc.Getitemvalue(“IntendedRecipient”)(0))
End if
Call doc.Removeitem(“SaveOptions”)
If doc.Hasitem(“RoutingState”) And Not doc.getitemvalue(“RoutingState”)(0)=”” Then
Print “Keep Form”
‘Keep Form
Else
If doc.Hasitem(“MailSavedForm”) And Not doc.getitemvalue(“MailSavedForm”)(0)=”” Then
Call doc.Replaceitemvalue(“Form”,doc.Getitemvalue(“MailSavedForm”))
Print “replace Form”
ElseIf doc.Hasitem(“AppointmentType”) Then
Call doc.Replaceitemvalue(“Form”,”Notice”)
‘Print “replace Form 2”
Else
Call doc.Replaceitemvalue(“Form”,”Memo”)
‘Print “replace Form 3”
End If
End If
Call doc.Replaceitemvalue(“$Hops”,25)
Call doc.Removeitem(“RoutingStateBy”)
Call doc.Replaceitemvalue(“hopsretry”,1)
Call doc.save(True,False)
‘Print “remove ops”
End If
End If
End If
Set doc=v.Getnextdocument(doc)
‘Print “next”
Loop
End if
End Sub
Thanks for sharing 🙂
Hello Fredrik, Luca B,
thanks for your code, i would like to know the code is worked for old version of domino ? for e.g. version 5/6/8/9?
wait for your reply and many thanks again.
Yes, I think it works for all versions of domino.
Hello Ivy,
I used the code on Domino version 8.5 and 9. It probably will work also on older versions. I’m still not sure how much effective is, but surely has helped.
Thx a lot for the code,but how i implement this in a server running V9? sorry my ignorance but how i program this agent ? please let me know thx again!!!
You add the code in a scheduled agent in a separate database on the server.
Or you can use this https://www.notesmail.com/home.nsf/tip20241213?openpage
I am clarifying my post above. Yes, “Auto Router Buf Fix (ARF)” as a workaround fix works fine and running in production at several companies! However, it does not resolve the Mail Rules issue, but includes a tech note for a semi-automated Mail Rules workaround fix. Basically, ARF is running in production at least 15-20 companies including environments running Domino 7.0.1, 8.5, 9.0.1, 12 and 14. Each company confirmed the mail started routing properly that was stuck in a mail loop. It works! Note: After following instructions they released their dead mail as well which started routing if dead mail was stuck in a mail loop.
You can download the ARF Notes app tool by clicking link below. Please specify your Domino gateway mail server names on the form. Happy New Year!
FYI
CRITICAL BUG: Download Workaround Fix – Domino mail not routing starting Dec 13, 2024 ⚠️❗❗
https://www.notesmail.com/home.nsf/tip20241213?openpage
The Notes app tool in link above is a workaround which automatically fixes the mail routing issue in HCL knowledge article KB0118192, but not the Mail Rules issue. However, refer to step 5 to semi-automatically fix the Mail Rules. It supports Domino versions 14 and lower including 10.0, 9.0, 8.5 and 7.0.1 which are End of Life (EOL), as well as, single and multiple mail router mailboxes.
The critical bug was dormant for 35 years and activated at the worst possible time during the holiday season. If you are an unlicensed Domino customer you most likely are feeling frustrated. The tool is a fully supported workaround fix by NotesMail – HCL Business Partner. If you don’t have access to the HCL fix packs and interim fixes (hotfixes), then you should use this tool.
This also causes domino cluster replication not working – and with no other symptoms than documents nor being replicated (ie. You don’t see any errors).
For a small environment a nasty fix is to turn back time when restarting Domino… however You need to cut smtp to the server so mails do not arrive before the date is set again.
Buenas disculpa en que parte ejecuto ese agente que enviaste
Usted crea una base de datos con un agente programado que se ejecuta en el intervalo que usted elija y pega el código en ese agente Lotusscript. Traducido con deepl.com
Hi there i do some fix for Lotus Server 9, i added a external smtp (open relay or with id) Configuration -Messaging Settings – General , Host relay ENable and enable use of smtp inside of domain of internet with option in all messages. And the magic is force Domino server to use an “Internet Address” from the Person document in Domino Directory as a sender’s adress with this Domino server configuration to remove “INetFrom” item from email header, url with this steps:
https://ypastov.blogspot.com/2013/01/how-to-force-outbound-email-to-use.html
How can help to others without patching. With this workaround the rules for send to another email works.
Regards and sorry for my bad english 😉
Great thanks for sharing
Thanks a lot for the useful tips and code, you saved my day!
Hello, I’m trying to set up a code for an agent that would forward me HOLD emails but there are errors in the code. Does anyone have the correct code please?
Thank you
Hi
The code should be correct but there is 2 lines starting with Not they should be added up to the previous line.
/Fredrik
Hi,
I’m not the developer, and my company is using Dominio 8.5.3. Does it create a new database and the scheduled agent with the provided code on the server? Are there any other setups or configurations needed?
Thanks
/Ben
No that is manual task you need to do.
the code needs to go in a scheduled agent.
Check out this more ready made solution.
https://www.notesmail.com/home.nsf/tip20241213?openpage
Thanks