TC forum mails are taking to long
Moderators: white, Hacker, petermad, Stefan2
TC forum mails are taking to long
The forum mails for watched threads are taking a very long time to deliver.
Its definitely not a problem with my mail provider which is web.de.
I get mails from Google groups, Sourceforge or other sources almost immediately
but the TC mailer sometimes even takes hours to deliver thread updates.
Its definitely not a problem with my mail provider which is web.de.
I get mails from Google groups, Sourceforge or other sources almost immediately
but the TC mailer sometimes even takes hours to deliver thread updates.
Windows 11 Home x64 Version 23H2 (OS Build 22631.3527)
TC 11.03 x64 / x86
Everything 1.5.0.1375a (x64), Everything Toolbar 1.3.3, Listary Pro 6.3.0.78
QAP 11.6.3.3 x64
TC 11.03 x64 / x86
Everything 1.5.0.1375a (x64), Everything Toolbar 1.3.3, Listary Pro 6.3.0.78
QAP 11.6.3.3 x64
- ghisler(Author)
- Site Admin
- Posts: 48124
- Joined: 2003-02-04, 09:46 UTC
- Location: Switzerland
- Contact:
Re: TC forum mails are taking to long
The mailer sends mails immediately when someone posts a reply. You are probably seing so-called greylisting: Here a mail server refuses the inial delivery and asks the server to re-send later. Since many spam servers are "fire and forget", they do not re-send mails. But regular mails get delayed due to this. You can check on the web.de mail interface whether you could whitelist ghisler.ch somehow.
Author of Total Commander
https://www.ghisler.com
https://www.ghisler.com
Re: TC forum mails are taking to long
Web.de has no such optionghisler(Author) wrote: ↑2019-01-07, 15:20 UTC The mailer sends mails immediately when someone posts a reply. You are probably seing so-called greylisting: Here a mail server refuses the inial delivery and asks the server to re-send later. Since many spam servers are "fire and forget", they do not re-send mails. But regular mails get delayed due to this. You can check on the web.de mail interface whether you could whitelist ghisler.ch somehow.
but as I said delivery from other boards are in seconds or minutes and not delayed at all.
So I don't believe that could be the reason for the delays.
Windows 11 Home x64 Version 23H2 (OS Build 22631.3527)
TC 11.03 x64 / x86
Everything 1.5.0.1375a (x64), Everything Toolbar 1.3.3, Listary Pro 6.3.0.78
QAP 11.6.3.3 x64
TC 11.03 x64 / x86
Everything 1.5.0.1375a (x64), Everything Toolbar 1.3.3, Listary Pro 6.3.0.78
QAP 11.6.3.3 x64
Re: TC forum mails are taking to long
2Horst.Epp
I'm afraid that you can't see options for greylisting in any existing webmail. You should add the email address (or the whole domain) of the sender to the white list instead.
I'm afraid that you can't see options for greylisting in any existing webmail. You should add the email address (or the whole domain) of the sender to the white list instead.
Andrzej P. Wozniak
Polish subforum moderator
Polish subforum moderator
Re: TC forum mails are taking to long
Sorry, but why should web.de handle mails from TC in another way than for all other senders in the world ?
If also use no webmail but Pop3 and this is checked every 15 minutes for new mail from a tool.
The info about your reply was coming fast this time.
Windows 11 Home x64 Version 23H2 (OS Build 22631.3527)
TC 11.03 x64 / x86
Everything 1.5.0.1375a (x64), Everything Toolbar 1.3.3, Listary Pro 6.3.0.78
QAP 11.6.3.3 x64
TC 11.03 x64 / x86
Everything 1.5.0.1375a (x64), Everything Toolbar 1.3.3, Listary Pro 6.3.0.78
QAP 11.6.3.3 x64
Re: TC forum mails are taking to long
2Horst.Epp
There may be many many reasons. It's a long story without happy end.
In this case the email server may greylist all messages sent by PHPmailer, see explanation here:
https://github.com/PHPMailer/PHPMailer/wiki/Sending-to-lists#maximising-performance
I may check notification receiving on some email servers (at least one of them uses greylisting for sure). It may take about 24 hours or more in the worst case… If you don't want to wait, you can start reading about greylisting on your own, f.e. on Wikipedia: https://en.wikipedia.org/wiki/Greylisting
There may be many many reasons. It's a long story without happy end.
In this case the email server may greylist all messages sent by PHPmailer, see explanation here:
https://github.com/PHPMailer/PHPMailer/wiki/Sending-to-lists#maximising-performance
I may check notification receiving on some email servers (at least one of them uses greylisting for sure). It may take about 24 hours or more in the worst case… If you don't want to wait, you can start reading about greylisting on your own, f.e. on Wikipedia: https://en.wikipedia.org/wiki/Greylisting
Andrzej P. Wozniak
Polish subforum moderator
Polish subforum moderator
Re: TC forum mails are taking to long
I'm surprised to see a wiki article and a standard term for this method. I thought it was an obscure hack used by some some small providers like mine. The initial delay I see is about 20-30 minutes, but messages come through if more than one are sent. It seems my server has a relaxed detection of resend.
Maybe the still present SpamAssassin detection has an influence over greylisting.
E-mail in general has reached a sorry state unfortunately. It is mostly bloated webmail now and controlled by big providers that are able set up the DNS configuration and cerificates. No longer possible to send legitimate mail from own computer. And I was informed that my mail server with POP access will be closed soon, and they recommend going to Google.
Maybe the still present SpamAssassin detection has an influence over greylisting.
Apparently the X-Priority: 3 field is tripping SpamAssassin.X-Spam-Flag: YES
X-Spam-Level: ****
X-Spam-Status: Yes, score=4.2 required=4.0 tests=BAYES_50,FROM_EXCESS_BASE64, XPRIO autolearn=no autolearn_force=no version=3.4.0
X-Spam-Report: * 2.0 BAYES_50 BODY: Bayes spam probability is 40 to 60% * [score: 0.5000] * 0.0 FROM_EXCESS_BASE64 From: base64 encoded unnecessarily * 2.2 XPRIO Has X-Priority header
E-mail in general has reached a sorry state unfortunately. It is mostly bloated webmail now and controlled by big providers that are able set up the DNS configuration and cerificates. No longer possible to send legitimate mail from own computer. And I was informed that my mail server with POP access will be closed soon, and they recommend going to Google.
#148174 Personal license
Running Total Commander v8.52a
Running Total Commander v8.52a
- ghisler(Author)
- Site Admin
- Posts: 48124
- Joined: 2003-02-04, 09:46 UTC
- Location: Switzerland
- Contact:
Re: TC forum mails are taking to long
I have removed the X-Priority header from the mailer functions in functions_messenger.php now. Maybe it helps?
Author of Total Commander
https://www.ghisler.com
https://www.ghisler.com
Re: TC forum mails are taking to long
Mail is no longer marked as spam! I can't say if that impacts the delay, as I've now received several messages in short time, and they arrive instantly.
(I use Email Topic button below any thread to test.)
(I use Email Topic button below any thread to test.)
#148174 Personal license
Running Total Commander v8.52a
Running Total Commander v8.52a
Re: TC forum mails are taking to long
2ghisler(Author)
It's a litle bit better now with greylisting as well. The first notification (with X-Priority) was delayed by 3 hours, the second one - by 1 hour only.
You cannot remove links from the message body, but you may do some more fixes in the header:
It's a litle bit better now with greylisting as well. The first notification (with X-Priority) was delayed by 3 hours, the second one - by 1 hour only.
You cannot remove links from the message body, but you may do some more fixes in the header:
- You can remove "X-MSMail-Priority" and "X-MimeOLE" fields from the message header - they are specific for MS software (Outlook Express, Windows Live Mail etc.)
- "X-Mailer" is another non-standard (undocumented) field, used mainly by MS software. Standard provides "User-Agent" field instead.
- You can also remove "Reply-To" field - it's required only when using different email address in "From" field. Now it's just unneeded duplicate.
- For header encoding Quoted Printable (QP) should be used rather than Base64. QP works better with Latin based languages available on the forum. Base64 is abused by spammers to fool simplest spam filters.
Andrzej P. Wozniak
Polish subforum moderator
Polish subforum moderator
- ghisler(Author)
- Site Admin
- Posts: 48124
- Joined: 2003-02-04, 09:46 UTC
- Location: Switzerland
- Contact:
Re: TC forum mails are taking to long
OK, I have now also removed "X-MSMail-Priority", "X-MimeOLE" and "X-Mailer".
I have also added code to only add "Reply-To" if it differs from the "From:" field.
Where does the mail use base64? Isn't that standard for MIME encoding?
I have also added code to only add "Reply-To" if it differs from the "From:" field.
Where does the mail use base64? Isn't that standard for MIME encoding?
Author of Total Commander
https://www.ghisler.com
https://www.ghisler.com
Re: TC forum mails are taking to long
Well, it looks like they are cosmetic changes for my provider's greylisting. Now the notification was delayed by 1.5 hour. Just one remark - it's better to replace "X-Mailer" name with "User-Agent" rather then delete that field.ghisler(Author) wrote: ↑2019-01-09, 16:47 UTCOK, I have now also removed "X-MSMail-Priority", "X-MimeOLE" and "X-Mailer".
I have also added code to only add "Reply-To" if it differs from the "From:" field.
It seems to be the most important change to avoid greylisting or false spam positive detection now.ghisler(Author) wrote: ↑2019-01-09, 16:47 UTCWhere does the mail use base64? Isn't that standard for MIME encoding?
MIME requires to use transport encoding in header if there are characters outside of us-ascii charset: accented letters in Latin based alphabets, other alphabets or writing systems, many special characters (mathematical, emoji etc.).
English text or text without accented characters is 7-bit and don't require encoding at all.
Quoted Printable encoding is preferred for header fields using Latin based alphabets, with a few accented characters - it's better readable and gives less overhead.
Base64 should be used for other cases, when (almost) all characters use more than 1 byte in their UTF-8 presentation.
Now compare current encoding of some header fields:
Code: Select all
To: =?UTF-8?B?VXNoZXI=?= <my@address.here.invalid>
Subject: =?UTF-8?B?VG9waWMgcmVwbHkgbm90aWZpY2F0aW9uIC0gIlRDIGZvcnVtIG1haWxzIGFy?=
=?UTF-8?B?ZSB0YWtpbmcgdG8gbG9uZyI=?=
From: "=?UTF-8?B?Q2hyaXN0aWFuIEdoaXNsZXI=?=" <forumadmin@forum.address>
Code: Select all
To: "Usher" <my@address.here.invalid>
Subject: Topic reply notification - "TC forum mails are taking to long"
From: "Christian Ghisler" <forumadmin@forum.address>
Code: Select all
To: =?utf-8?Q?Andrzej_P._Wo=C5=BAniak?= <my@address.here>
Mail body is 8-bit and sent in UTF-8 according to charset declaration in header, only binary parts (f.e. inline pictures or attachments) should be always encoded in Base64.
Andrzej P. Wozniak
Polish subforum moderator
Polish subforum moderator
- ghisler(Author)
- Site Admin
- Posts: 48124
- Joined: 2003-02-04, 09:46 UTC
- Location: Switzerland
- Contact:
Re: TC forum mails are taking to long
OK, I have used code from http://php.net/manual/en/function.quoted-printable-encode.php
to support quoted printable. I can't use quoted-printable-encode directly because the lines are too long.
Here is the code I use now:
Please let me know if it can be improved in any way.
to support quoted printable. I can't use quoted-printable-encode directly because the lines are too long.
Here is the code I use now:
Code: Select all
function mail_encode($string, $eol = "\r\n")
{
if (strlen($string)<=60 && strpos($string,"?")==false && imap_8bit($string) == $string)
return $string;
$encoding='utf-8';
$string = str_replace(" ", "_", trim($string)) ;
// We need to delete "=\r\n" produced by imap_8bit() and replace '?'
$string = str_replace("?", "=3F", str_replace("=\r\n", "", imap_8bit($string))) ;
$string = '=?'.$encoding.'?Q?'. $string; // Ghisler: First line shorter, for header name
$string = chunk_split($string, 60);
// We also have to remove last unneeded \r\n :
$string = substr($string, 0, strlen($string)-2);
// replace newlines with encoding text "=?UTF ..."
$string = str_replace("\r\n", "?=".$eol." =?".$encoding."?Q?", $string) ;
return $string.'?=';
}
Author of Total Commander
https://www.ghisler.com
https://www.ghisler.com
Re: TC forum mails are taking to long
Lines in header may be even 998 bytes long (including field name), it's not a problem. Now the header looks pretty good, but…
Unfortunately, email was delayed by 3.5 hour because removed fields went back:
There should be only one:
That's all that I can help for now after tests with my email provider. There's a very popular email software on the mail server there. It's qmail with simscan add-on (with clamAV and SpamAssassin plugins, inactive or hidden), with default settings it probably delays all subscription messages for a few minutes at least.
Unfortunately, email was delayed by 3.5 hour because removed fields went back:
Code: Select all
X-MSMail-Priority: Normal
X-Mailer: phpBB3
X-MimeOLE: phpBB3
Code: Select all
User-Agent: phpBB3
Andrzej P. Wozniak
Polish subforum moderator
Polish subforum moderator
- ghisler(Author)
- Site Admin
- Posts: 48124
- Joined: 2003-02-04, 09:46 UTC
- Location: Switzerland
- Contact:
Re: TC forum mails are taking to long
Ah, sorry, I tested this in the test forum and then copied it to the main forum, that's why that last change was lost. Please try again now.
Author of Total Commander
https://www.ghisler.com
https://www.ghisler.com