TC forum mails are taking to long

English support forum

Moderators: sheep, Hacker, Stefan2, white

User avatar
Horst.Epp
Power Member
Power Member
Posts: 3430
Joined: 2003-02-06, 17:36 UTC
Location: Germany

TC forum mails are taking to long

Post by *Horst.Epp » 2019-01-03, 16:58 UTC

The forum mails for watched threads are taking a very long time to deliver.
Its definitely not a problem with my mail provider which is web.de.
I get mails from Google groups, Sourceforge or other sources almost immediately
but the TC mailer sometimes even takes hours to deliver thread updates.
Windows 10 Home x64 November 2019 Update, Version 1909 (OS Build 18363.476)
Intel(R) Core(TM) i7-4770 CPU @ 3.40GH, 16GB RAM
TC 9.50ß5 x64 / x86, Everything 1.4.1.955 (x64)

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 38074
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: TC forum mails are taking to long

Post by *ghisler(Author) » 2019-01-07, 15:20 UTC

The mailer sends mails immediately when someone posts a reply. You are probably seing so-called greylisting: Here a mail server refuses the inial delivery and asks the server to re-send later. Since many spam servers are "fire and forget", they do not re-send mails. But regular mails get delayed due to this. You can check on the web.de mail interface whether you could whitelist ghisler.ch somehow.
Author of Total Commander
http://www.ghisler.com

User avatar
Horst.Epp
Power Member
Power Member
Posts: 3430
Joined: 2003-02-06, 17:36 UTC
Location: Germany

Re: TC forum mails are taking to long

Post by *Horst.Epp » 2019-01-07, 17:10 UTC

ghisler(Author) wrote:
2019-01-07, 15:20 UTC
The mailer sends mails immediately when someone posts a reply. You are probably seing so-called greylisting: Here a mail server refuses the inial delivery and asks the server to re-send later. Since many spam servers are "fire and forget", they do not re-send mails. But regular mails get delayed due to this. You can check on the web.de mail interface whether you could whitelist ghisler.ch somehow.
Web.de has no such option
but as I said delivery from other boards are in seconds or minutes and not delayed at all.
So I don't believe that could be the reason for the delays.
Windows 10 Home x64 November 2019 Update, Version 1909 (OS Build 18363.476)
Intel(R) Core(TM) i7-4770 CPU @ 3.40GH, 16GB RAM
TC 9.50ß5 x64 / x86, Everything 1.4.1.955 (x64)

User avatar
Usher
Power Member
Power Member
Posts: 569
Joined: 2011-03-11, 10:11 UTC

Re: TC forum mails are taking to long

Post by *Usher » 2019-01-07, 17:16 UTC

2Horst.Epp
I'm afraid that you can't see options for greylisting in any existing webmail. You should add the email address (or the whole domain) of the sender to the white list instead.
Regards from Poland
Andrzej P. Wozniak

User avatar
Horst.Epp
Power Member
Power Member
Posts: 3430
Joined: 2003-02-06, 17:36 UTC
Location: Germany

Re: TC forum mails are taking to long

Post by *Horst.Epp » 2019-01-07, 17:28 UTC

Usher wrote:
2019-01-07, 17:16 UTC
2Horst.Epp
I'm afraid that you can't see options for greylisting in any existing webmail. You should add the email address (or the whole domain) of the sender to the white list instead.
Sorry, but why should web.de handle mails from TC in another way than for all other senders in the world ?
If also use no webmail but Pop3 and this is checked every 15 minutes for new mail from a tool.
The info about your reply was coming fast this time.
Windows 10 Home x64 November 2019 Update, Version 1909 (OS Build 18363.476)
Intel(R) Core(TM) i7-4770 CPU @ 3.40GH, 16GB RAM
TC 9.50ß5 x64 / x86, Everything 1.4.1.955 (x64)

User avatar
Usher
Power Member
Power Member
Posts: 569
Joined: 2011-03-11, 10:11 UTC

Re: TC forum mails are taking to long

Post by *Usher » 2019-01-07, 18:09 UTC

2Horst.Epp
There may be many many reasons. It's a long story without happy end.

In this case the email server may greylist all messages sent by PHPmailer, see explanation here:
https://github.com/PHPMailer/PHPMailer/wiki/Sending-to-lists#maximising-performance

I may check notification receiving on some email servers (at least one of them uses greylisting for sure). It may take about 24 hours or more in the worst case… If you don't want to wait, you can start reading about greylisting on your own, f.e. on Wikipedia: https://en.wikipedia.org/wiki/Greylisting
Regards from Poland
Andrzej P. Wozniak

User avatar
j7n
Member
Member
Posts: 149
Joined: 2005-08-07, 21:56 UTC

Re: TC forum mails are taking to long

Post by *j7n » 2019-01-09, 00:05 UTC

I'm surprised to see a wiki article and a standard term for this method. I thought it was an obscure hack used by some some small providers like mine. The initial delay I see is about 20-30 minutes, but messages come through if more than one are sent. It seems my server has a relaxed detection of resend.

Maybe the still present SpamAssassin detection has an influence over greylisting.
X-Spam-Flag: YES
X-Spam-Level: ****
X-Spam-Status: Yes, score=4.2 required=4.0 tests=BAYES_50,FROM_EXCESS_BASE64, XPRIO autolearn=no autolearn_force=no version=3.4.0
X-Spam-Report: * 2.0 BAYES_50 BODY: Bayes spam probability is 40 to 60% * [score: 0.5000] * 0.0 FROM_EXCESS_BASE64 From: base64 encoded unnecessarily * 2.2 XPRIO Has X-Priority header
Apparently the X-Priority: 3 field is tripping SpamAssassin.

E-mail in general has reached a sorry state unfortunately. It is mostly bloated webmail now and controlled by big providers that are able set up the DNS configuration and cerificates. No longer possible to send legitimate mail from own computer. And I was informed that my mail server with POP access will be closed soon, and they recommend going to Google.
#148174 Personal license
Running Total Commander v8.52a

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 38074
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: TC forum mails are taking to long

Post by *ghisler(Author) » 2019-01-09, 11:15 UTC

I have removed the X-Priority header from the mailer functions in functions_messenger.php now. Maybe it helps?
Author of Total Commander
http://www.ghisler.com

User avatar
j7n
Member
Member
Posts: 149
Joined: 2005-08-07, 21:56 UTC

Re: TC forum mails are taking to long

Post by *j7n » 2019-01-09, 13:16 UTC

Mail is no longer marked as spam! I can't say if that impacts the delay, as I've now received several messages in short time, and they arrive instantly.

(I use Email Topic button below any thread to test.)
#148174 Personal license
Running Total Commander v8.52a

User avatar
Usher
Power Member
Power Member
Posts: 569
Joined: 2011-03-11, 10:11 UTC

Re: TC forum mails are taking to long

Post by *Usher » 2019-01-09, 14:22 UTC

2ghisler(Author)
It's a litle bit better now with greylisting as well. The first notification (with X-Priority) was delayed by 3 hours, the second one - by 1 hour only.

You cannot remove links from the message body, but you may do some more fixes in the header:
  • You can remove "X-MSMail-Priority" and "X-MimeOLE" fields from the message header - they are specific for MS software (Outlook Express, Windows Live Mail etc.)
  • "X-Mailer" is another non-standard (undocumented) field, used mainly by MS software. Standard provides "User-Agent" field instead.
  • You can also remove "Reply-To" field - it's required only when using different email address in "From" field. Now it's just unneeded duplicate.
  • For header encoding Quoted Printable (QP) should be used rather than Base64. QP works better with Latin based languages available on the forum. Base64 is abused by spammers to fool simplest spam filters.
Regards from Poland
Andrzej P. Wozniak

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 38074
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: TC forum mails are taking to long

Post by *ghisler(Author) » 2019-01-09, 16:47 UTC

OK, I have now also removed "X-MSMail-Priority", "X-MimeOLE" and "X-Mailer".
I have also added code to only add "Reply-To" if it differs from the "From:" field.
Where does the mail use base64? Isn't that standard for MIME encoding?
Author of Total Commander
http://www.ghisler.com

User avatar
Usher
Power Member
Power Member
Posts: 569
Joined: 2011-03-11, 10:11 UTC

Re: TC forum mails are taking to long

Post by *Usher » 2019-01-10, 02:52 UTC

ghisler(Author) wrote:
2019-01-09, 16:47 UTC
OK, I have now also removed "X-MSMail-Priority", "X-MimeOLE" and "X-Mailer".
I have also added code to only add "Reply-To" if it differs from the "From:" field.
Well, it looks like they are cosmetic changes for my provider's greylisting. Now the notification was delayed by 1.5 hour. Just one remark - it's better to replace "X-Mailer" name with "User-Agent" rather then delete that field.
ghisler(Author) wrote:
2019-01-09, 16:47 UTC
Where does the mail use base64? Isn't that standard for MIME encoding?
It seems to be the most important change to avoid greylisting or false spam positive detection now.

MIME requires to use transport encoding in header if there are characters outside of us-ascii charset: accented letters in Latin based alphabets, other alphabets or writing systems, many special characters (mathematical, emoji etc.).
English text or text without accented characters is 7-bit and don't require encoding at all.
Quoted Printable encoding is preferred for header fields using Latin based alphabets, with a few accented characters - it's better readable and gives less overhead.
Base64 should be used for other cases, when (almost) all characters use more than 1 byte in their UTF-8 presentation.

Now compare current encoding of some header fields:

Code: Select all

To: =?UTF-8?B?VXNoZXI=?= <my@address.here.invalid>
Subject: =?UTF-8?B?VG9waWMgcmVwbHkgbm90aWZpY2F0aW9uIC0gIlRDIGZvcnVtIG1haWxzIGFy?=
 =?UTF-8?B?ZSB0YWtpbmcgdG8gbG9uZyI=?=
From: "=?UTF-8?B?Q2hyaXN0aWFuIEdoaXNsZXI=?=" <forumadmin@forum.address>
…and proper, unencoded version:

Code: Select all

To: "Usher" <my@address.here.invalid>
Subject: Topic reply notification - "TC forum mails are taking to long"
From: "Christian Ghisler" <forumadmin@forum.address>
Finally, see QP in header:

Code: Select all

To: =?utf-8?Q?Andrzej_P._Wo=C5=BAniak?= <my@address.here>
and compare with my signature…

Mail body is 8-bit and sent in UTF-8 according to charset declaration in header, only binary parts (f.e. inline pictures or attachments) should be always encoded in Base64.
Regards from Poland
Andrzej P. Wozniak

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 38074
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: TC forum mails are taking to long

Post by *ghisler(Author) » 2019-01-11, 10:49 UTC

OK, I have used code from http://php.net/manual/en/function.quoted-printable-encode.php
to support quoted printable. I can't use quoted-printable-encode directly because the lines are too long.
Here is the code I use now:

Code: Select all

function mail_encode($string, $eol = "\r\n")
{
    if (strlen($string)<=60 && strpos($string,"?")==false && imap_8bit($string) == $string)
       return $string;

    $encoding='utf-8';
    $string = str_replace(" ", "_", trim($string)) ;
    // We need to delete "=\r\n" produced by imap_8bit() and replace '?'
    $string = str_replace("?", "=3F", str_replace("=\r\n", "", imap_8bit($string))) ;

    $string = '=?'.$encoding.'?Q?'. $string;  // Ghisler: First line shorter, for header name
    $string = chunk_split($string, 60);
    // We also have to remove last unneeded \r\n :
    $string = substr($string, 0, strlen($string)-2);
    // replace newlines with encoding text "=?UTF ..."
    $string = str_replace("\r\n", "?=".$eol." =?".$encoding."?Q?", $string) ;
   
    return $string.'?=';
}
Please let me know if it can be improved in any way.
Author of Total Commander
http://www.ghisler.com

User avatar
Usher
Power Member
Power Member
Posts: 569
Joined: 2011-03-11, 10:11 UTC

Re: TC forum mails are taking to long

Post by *Usher » 2019-01-11, 20:18 UTC

Lines in header may be even 998 bytes long (including field name), it's not a problem. Now the header looks pretty good, but…

Unfortunately, email was delayed by 3.5 hour because removed fields went back:

Code: Select all

X-MSMail-Priority: Normal
X-Mailer: phpBB3
X-MimeOLE: phpBB3
There should be only one:

Code: Select all

User-Agent: phpBB3
That's all that I can help for now after tests with my email provider. There's a very popular email software on the mail server there. It's qmail with simscan add-on (with clamAV and SpamAssassin plugins, inactive or hidden), with default settings it probably delays all subscription messages for a few minutes at least.
Regards from Poland
Andrzej P. Wozniak

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 38074
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: TC forum mails are taking to long

Post by *ghisler(Author) » 2019-01-15, 20:23 UTC

Ah, sorry, I tested this in the test forum and then copied it to the main forum, that's why that last change was lost. Please try again now.
Author of Total Commander
http://www.ghisler.com

Post Reply