Cannot get it to support Cyrillic

[vesper8]

Hi there,

I have a client that receives emails in either English or Cyrillic

As it stands I am not able to parse the Cyrillic ones because the subject/body gets all garbled up

For example this subject:

Резервация от Ски Училище

Gets converted to

Резе�ва�и� о� Ски У�или�е

Am happy to share headers and whatever you need to help me resolve this issue

this is the output of var_dump($message->header)

object(Webklex\PHPIMAP\Header)#2064 (4) {
  ["raw"]=>
  string(1142) "Return-Path: <redacted@hulk.icnhost.net>
Delivered-To: office+redacted@redacted.com
Received: from hulk.icnhost.net
    by hulk.icnhost.net with LMTP
    id fF3sC6zH8F+MQwAAAsqQFQ
    (envelope-from <redacted@hulk.icnhost.net>)
    for <office+redacted@redacted.com>; Sat, 02 Jan 2021 21:21:16 +0200
Return-path: <redacted@hulk.icnhost.net>
Envelope-to: office@redacted.com
Delivery-date: Sat, 02 Jan 2021 21:21:16 +0200
Received: from localhost ([127.0.0.1]:34100 helo=hulk.icnhost.net)
    by hulk.icnhost.net with smtp (Exim 4.92)
    (envelope-from <redacted@hulk.icnhost.net>)
    id 1kvmSi-0004Uj-2S
    for office@redacted.com; Sat, 02 Jan 2021 21:21:16 +0200
Date: Sat, 02 Jan 2021 21:21:16 +0200
X-SG-User: redacted
X-SG-Opt:  SCRIPT_FILENAME=/home/redacted/public_html/index.php  REQUEST_URI=/bg/redacted/  PWD=/home/redacted/public_html  REMOTE_ADDR=213.222.49.150
To: office@redacted.com
Subject: Резервация от Ски Училище
From:Redacted <office@redacted.com>
Reply-to:redacted@gmail.com
MIME-Version: 1.0
Content-type: text/html; charset=UTF-8

"
  ["attributes":protected]=>
  array(18) {
    ["from"]=>
    array(1) {
      [0]=>
      object(stdClass)#2066 (5) {
        ["personal"]=>
        string(15) "Redacted"
        ["mailbox"]=>
        string(6) "office"
        ["host"]=>
        string(18) "redacted.com"
        ["mail"]=>
        string(25) "office@redacted.com"
        ["full"]=>
        string(43) "Redacted <office@redacted.com>"
      }
    }
    ["to"]=>
    array(1) {
      [0]=>
      object(stdClass)#2065 (5) {
        ["personal"]=>
        string(0) ""
        ["mailbox"]=>
        string(6) "office"
        ["host"]=>
        string(18) "redacted.com"
        ["mail"]=>
        string(25) "office@redacted.com"
        ["full"]=>
        string(25) "office@redacted.com"
      }
    }
    ["reply_to"]=>
    array(1) {
      [0]=>
      object(stdClass)#2067 (5) {
        ["personal"]=>
        string(0) ""
        ["mailbox"]=>
        string(11) "redacted"
        ["host"]=>
        string(9) "gmail.com"
        ["mail"]=>
        string(21) "redacted@gmail.com"
        ["full"]=>
        string(21) "redacted@gmail.com"
      }
    }
    ["subject"]=>
    string(107) "Резе�ва�и� о� Ски У�или�е"
    ["date"]=>
    object(Carbon\Carbon)#2069 (16) {
      ["constructedObjectId":protected]=>
      string(32) "00000000441a3eec000000002479e6a7"
      ["localMonthsOverflow":protected]=>
      NULL
      ["localYearsOverflow":protected]=>
      NULL
      ["localStrictModeEnabled":protected]=>
      NULL
      ["localHumanDiffOptions":protected]=>
      NULL
      ["localToStringFormat":protected]=>
      NULL
      ["localSerializer":protected]=>
      NULL
      ["localMacros":protected]=>
      NULL
      ["localGenericMacros":protected]=>
      NULL
      ["localFormatFunction":protected]=>
      NULL
      ["localTranslator":protected]=>
      NULL
      ["dumpProperties":protected]=>
      array(3) {
        [0]=>
        string(4) "date"
        [1]=>
        string(13) "timezone_type"
        [2]=>
        string(8) "timezone"
      }
      ["dumpLocale":protected]=>
      NULL
      ["date"]=>
      string(26) "2021-01-02 21:21:16.000000"
      ["timezone_type"]=>
      int(1)
      ["timezone"]=>
      string(6) "+02:00"
    }
    ["return_path"]=>
    string(53) "<redacted@hulk.icnhost.net> <redacted@hulk.icnhost.net>"
    ["delivered_to"]=>
    string(34) "office+redacted@redacted.com"
    ["received"]=>
    array(10) {
      [0]=>
      string(21) "from hulk.icnhost.net"
      [1]=>
      string(29) "by hulk.icnhost.net with LMTP"
      [2]=>
      string(25) "id fF3sC6zH8F+MQwAAAsqQFQ"
      [3]=>
      string(42) "(envelope-from <redacted@hulk.icnhost.net>)"
      [4]=>
      string(73) "for <office+redacted@redacted.com>; Sat, 02 Jan 2021 21:21:16 +0200"
      [5]=>
      string(56) "from localhost ([127.0.0.1]:34100 helo=hulk.icnhost.net)"
      [6]=>
      string(41) "by hulk.icnhost.net with smtp (Exim 4.92)"
      [7]=>
      string(42) "(envelope-from <redacted@hulk.icnhost.net>)"
      [8]=>
      string(19) "id 1kvmSi-0004Uj-2S"
      [9]=>
      string(62) "for office@redacted.com; Sat, 02 Jan 2021 21:21:16 +0200"
    }
    ["envelope_to"]=>
    string(25) "office@redacted.com"
    ["delivery_date"]=>
    string(31) "Sat, 02 Jan 2021 21:21:16 +0200"
    ["x_sg_user"]=>
    string(7) "redacted"
    ["x_sg_opt"]=>
    string(143) "SCRIPT_FILENAME=/home/redacted/public_html/index.php  REQUEST_URI=/bg/redacted/  PWD=/home/redacted/public_html  REMOTE_ADDR=213.222.49.150"
    ["mime_version"]=>
    string(3) "1.0"
    ["content_type"]=>
    string(9) "text/html"
    ["toaddress"]=>
    string(25) "office@redacted.com"
    ["fromaddress"]=>
    string(43) "Redacted <office@redacted.com>"
    ["reply_toaddress"]=>
    string(21) "redacted@gmail.com"
    ["charset"]=>
    string(5) "UTF-8"
  }
  ["config":protected]=>
  array(11) {
    ["delimiter"]=>
    string(1) "/"
    ["fetch"]=>
    int(2)
    ["sequence"]=>
    int(3)
    ["fetch_body"]=>
    bool(true)
    ["fetch_flags"]=>
    bool(true)
    ["message_key"]=>
    string(4) "list"
    ["fetch_order"]=>
    string(3) "asc"
    ["dispositions"]=>
    array(2) {
      [0]=>
      string(10) "attachment"
      [1]=>
      string(6) "inline"
    }
    ["common_folders"]=>
    array(5) {
      ["root"]=>
      string(5) "INBOX"
      ["junk"]=>
      string(10) "INBOX/Junk"
      ["draft"]=>
      string(12) "INBOX/Drafts"
      ["sent"]=>
      string(10) "INBOX/Sent"
      ["trash"]=>
      string(11) "INBOX/Trash"
    }
    ["decoder"]=>
    array(2) {
      ["message"]=>
      string(5) "utf-8"
      ["attachment"]=>
      string(5) "utf-8"
    }
    ["open"]=>
    array(0) {
    }
  }
  ["fallback_encoding"]=>
  string(5) "UTF-8"
}

[Webklex]

Hi @vesper8 ,
many thanks for your report.

The problem most likely is hidden within this function:

php-imap/src/Header.php

Lines 415 to 452 in 6120e70

	private function decode($value) {
	if (is_array($value)) {
	return $this->decodeArray($value);
	}
	$original_value = $value;
	$decoder = $this->config['decoder']['message'];
	if ($value !== null) {
	if($decoder === 'utf-8' && extension_loaded('imap')) {
	$value = \imap_utf8($value);
	if (strpos(strtolower($value), '=?utf-8?') === 0) {
	$value = mb_decode_mimeheader($value);
	}
	if ($this->notDecoded($original_value, $value)) {
	$decoded_value = $this->mime_header_decode($value);
	if (count($decoded_value) > 0) {
	if(property_exists($decoded_value[0], "text")) {
	$value = $decoded_value[0]->text;
	}
	}
	}
	}elseif($decoder === 'iconv') {
	$value = iconv_mime_decode($value);
	}else{
	$value = mb_decode_mimeheader($value);
	}
	if (strpos(strtolower($value), '=?utf-8?') === 0) {
	$value = mb_decode_mimeheader($value);
	}
	if ($this->notDecoded($original_value, $value)) {
	$value = $this->convertEncoding($original_value, $this->getEncoding($original_value));
	}
	}
	return $value;
	}

I'm a bit scared to touch this function and make changes to it on the fly - if you feel like it, feel free to play around with it :)

Best regards,

[Webklex]

I've just tested your header with the latest release:

$raw = "Return-Path: <redacted@hulk.icnhost.net>
Delivered-To: office+redacted@redacted.com
Received: from hulk.icnhost.net
    by hulk.icnhost.net with LMTP
    id fF3sC6zH8F+MQwAAAsqQFQ
    (envelope-from <redacted@hulk.icnhost.net>)
    for <office+redacted@redacted.com>; Sat, 02 Jan 2021 21:21:16 +0200
Return-path: <redacted@hulk.icnhost.net>
Envelope-to: office@redacted.com
Delivery-date: Sat, 02 Jan 2021 21:21:16 +0200
Received: from localhost ([127.0.0.1]:34100 helo=hulk.icnhost.net)
    by hulk.icnhost.net with smtp (Exim 4.92)
    (envelope-from <redacted@hulk.icnhost.net>)
    id 1kvmSi-0004Uj-2S
    for office@redacted.com; Sat, 02 Jan 2021 21:21:16 +0200
Date: Sat, 02 Jan 2021 21:21:16 +0200
X-SG-User: redacted
X-SG-Opt:  SCRIPT_FILENAME=/home/redacted/public_html/index.php  REQUEST_URI=/bg/redacted/  PWD=/home/redacted/public_html  REMOTE_ADDR=213.222.49.150
To: office@redacted.com
Subject: Резервация от Ски Училище
From:Redacted <office@redacted.com>
Reply-to:redacted@gmail.com
MIME-Version: 1.0
Content-type: text/html; charset=UTF-8";

$header = new \Webklex\PHPIMAP\Header($raw);

var_dump($header->subject);

This results in the following output:

object(Webklex\PHPIMAP\Attribute)#519 (2) {
  ["name":protected]=>
  string(7) "subject"
  ["values":protected]=>
  array(1) {
    [0]=>
    string(47) "Резервация от Ски Училище"
  }
}

The output looks fine to me. Please go ahead and update to v2.4.0 and try again. If you are still getting the same result, please consider "donating" a test mail to me (github@webklex.com) so we can both test based on the same mail.

One last thing. Are you having the native php-imap module enabled / installed? If not, please try to enable it - this might help with this decoding issue as well.

sudo apt-get install php*-imap
sudo apache2ctl graceful

Best regards,

[vesper8]

Hey @Webklex!

Thanks for the quick and awesome support as usual.. much appreciated!

That's a big update you just released with quite a lot of changes. First I just want to mention two breaking changes that I needed to change small things for in case they were unexpected

First I had to change

$message->getTo()[0]->mail

to

$message->getTo()->first()->mail

This one was perhaps a little hacky on my part but I couldn't figure out a better way to get the actual "to" email. All good

Then I had to change

$message->date->format('Y-m-d H:i:s')

to

$message->date->first()->format('Y-m-d H:i:s')

This one feels wrong.. messages should only ever have one date no?

Lastly.. unfortunately it does not seem to have solved my Cyrillic problem. Doing a var_dump on the header results in the same as before. I have forwarded you the offending mail in question to your email you provided so that you may help me further with this issue.

I have not tried to deal with this problem on a Ubuntu server yet, right now I'm testing on OSX so am unsure how to install php-imap there. I can look into that but first let's see if the email decodes properly now that I've forwarded it to you

Many thanks!

[Webklex]

Hi @vesper8 ,
many thanks for testing!

I just received your mail:

Subject: =?UTF-8?B?0KDQtdC30LXRgNCy0LDRhtC40Y8g0L7RgiDQodC60Lgg0KPRh9C40Ls=?=
 =?UTF-8?B?0LjRidC1?=

This time the subject is base64 and utf-8 encoded which should work perfectly fine? Maybe the used mail client changed the original mail? Btw does the content get decoded correctly or is it broken as well?

Yes you are right - the date is an attribute I've missed to test in detail. Maybe treating it as a string works smoother?
(string)$message->date

A message can heave as many duplicate headers as you may which. Yes there are official RFCs but most mail clients don't care and do what ever they want...
I implemented the Attribute class to have a base value to work with. Unlike before, you can now choose if you want to work with strings, arrays or a mix of both if you want to do so :)

Btw: $message->getTo()[0]->mail will work in the future.

Best regards,

[vesper8]

Hrm I guess you must be right and my client from which I forwarded the message (Thunderbird) must have fixed the encoding so you're not getting the exact same email as I am getting from the imap server

I did some further debugging inside the function you referenced and since I have neither the imap extension or iconv installed, it is ending up inside the last else $value = mb_decode_mimeheader($value);

Which fails to turn the Cyrillic into anything legible

However I noticed that further down inside that function, it fails to enter this conditional:

            if ($this->notDecoded($original_value, $value)) {
                $value = $this->convertEncoding($original_value, $this->getEncoding($original_value));
            }

That is, $this->notDecoded($original_value, $value) returns false

But if I comment out the conditional and just output the result of $value = $this->convertEncoding($original_value, $this->getEncoding($original_value));

Then in this case it does correctly return the Cyrillic value

[vesper8]

It should be noted that this line $value = mb_decode_mimeheader($value);

Is actually what garbles up the Cyrillic

It is, as far as I can tell, doing the opposite of what it should do.. but I'm not well versed at all in the secret art of encoding/decoding

[vesper8]

Btw does the content get decoded correctly or is it broken as well?

I wasn't getting that far because I was doing a strpos check on the subject before trying to process the body. But I just checked now while $message->getSubject() garbles up the subject, $message->getRawBody() does not garble up the content

perhaps it would be good to implement a $message->getRawSubject() or another means of exposting the raw header values before they get garbled up by private function decode($value)

It so happens that I am feeding each message into https://github.com/beyondcode/laravel-mailbox and that library does somehow manage to properly decode the subject

So what I'm doing now is

$inboundEmail = InboundEmail::fromMessage($message->getHeader()->raw . $message->getRawBody());

$subject = $inboundEmail->message()->getHeaderValue('Subject');

And then the subject is properly decoded/encoded and I can handle the Cyrillic that way

So my problem seems to be solved, although it's thanks to laravel-mailbox so it may certainly still be worth seeing if your character encoding logic can be improved further

[Webklex]

Hi @vesper8 ,
thanks for pinpointing to the code section in question. I'll look into it once I have some spare time :)

[Webklex]

Hi @vesper8 ,
please update to the latest release v2.4.4.

Best regards,

[alaevka]

Hi! You need do this and for Windows-1251 encodig. Similar situations as a UTF-8:
`Subject: =?Windows-1251?B?..xxxxxxxx...