Import csv with special chars

[Fero Gachulinec]

I am from Slovakia and Slovak language includes special chars as Ščťšľ. Could somebody please advice me how i can fix this problem? This problem appears when I want to import a csv file of my contacts with special chars.
Thanks

 

[frm.mwz]

I am from Slovakia and Slovak language includes special chars as Ščťšľ. Could somebody please advice me how i can fix this problem? This problem appears when I want to import a csv file of my contacts with special chars.
Thanks

As such samples from non-English character sets may cause many problems down the road, one way of mitigating these upfront would be to transcribe them before import.
Just as ä->ae, ö->oe, ß->ss, ü->ue, you can do Š->sh, č->ch, etc.
As far as I could observe so far, this is how Asian languages also deal with the character set problem in official int'l documents.
Email was conceived with ASCII128 in mind, and some of its conventions can be difficult to overcome (see RFCs).
Hope this helps a little bit

some background info, in case anyone is interested:
https://en.wikipedia.org/wiki/International_email
https://en.wikipedia.org/wiki/Unicode#Email

 

[Fero Gachulinec]

As such samples from non-English character sets may cause many problems down the road, one way of mitigating these upfront would be to transcribe them before import.
Just as ä->ae, ö->oe, ß->ss, ü->ue, you can do Š->sh, č->ch, etc.
As far as I could observe so far, this is how Asian languages also deal with the character set problem in official int'l documents.
Email was conceived with ASCII128 in mind, and some of its conventions can be difficult to overcome (see RFCs).
Hope this helps a little bit

some background info, in case anyone is interested:
https://en.wikipedia.org/wiki/International_email
https://en.wikipedia.org/wiki/Unicode#Email

Thank you for your prompt reply, email field is ok, but first name, last name, city/town, ... fields include these special chars. To transcribe them before import would be difficult task for customers who have many contacts. I think it is needed to change something in /apps/customer/controllers/List_importController.php.

 

[Fero Gachulinec]

It is needed to use utf8_encode($string) instead of encode($string), I think

 

[twisted1919]

@Fero Gachulinec - can you PM me with a csv file of contacts that cause issues so that i can try to import it and see if it does the same on my end?

 

[Fero Gachulinec]

@Fero Gachulinec - can you PM me with a csv file of contacts that cause issues so that i can try to import it and see if it does the same on my end?

Yes, please try to upload test.txt (rename it to test.csv). I will have to look at Yii framework. It seem to me it will be needed to add some code (convert uploaded csv file to utf-8) here: /apps/customer/controllers/List_importController.php. I can see also this code in /apps/common/models/ListImportAbstract.php:
if (!StringHelper::fixFileEncoding($uploadPath . $this->file_name)) {
@unlink($uploadPath . $this->file_name);
$this->addError('file', Yii::t('list_import', 'Your uploaded file is not using the UTF-8 charset. Please save it in UTF-8 then upload it again.'));
$this->file_name = null;
return false;
}

return true;
}
I think the code above does not work properly. It is unpleasent for customers they should convert imported csv.
Thanks

 

[twisted1919]

have you tried removing that code and see if it does any difference?

 

[Fero Gachulinec]

No, I would have to know more about Yii framework and about this software structure. It would be better if Cristian Sebran would say something. Simply, we would need the following procedure/algoritm:
procedure will find out whether imported csv is in utf8 charset or not. If not, then imported csv is AUTOMATICALLY converted to utf-8 and these values are stored in DB table. This software, Mailwizz, is really fantastic, it does not deserve to have such minus.

 

[twisted1919]

Well thing is that we try to fix the file encoding but handling utf-8 characters in php is not the easiest thing one could do.
This is why i asked to remove that block of code and see if it fixes the problem, that way i could get some hints.

 

[Fero Gachulinec]

Understood, it is really a difficult problem. But a customer usually begins with contact import and at the start he will find out there is a problem with csv import for example in Central Europe countries as Slovakia, Czech Republic, ... .