Pcntl reliable?

Niko

Niko

Active Member
Hello
I have the last version 1.3.9 but I had some issue with delivery using pcntl enabled
In many campaigns some user had received till 10 times the same email..so the system send duplicates emails..Pls you can look at this serious bug? Thx
 
Hello Niko,

Disable pcntl and then try to run a small test campaign and see it is sending more than 1 time.
 
Hi Vpul,
yes I tried and without the system send correctly..without double sends..
So i think PCNTL is not reliable
 
Has anyone noticed the opposite - i.e. MailWizz reporting that all mail has been sent, but your Postfix/maillog will only show a fraction of the mail?
 
I am facing the same problem also using PCNTl my clients are also recieveing 4 emails same time from a single campaign and after 100%sent campaign still shows processing and i have to manually change it to sent. Any solution is appreciated.
 
I am running CentOS Linux release 7.2.1511 (Core) with 4GB RAM and 4 cores

Code: 
Server version: Apache/2.4.6 (CentOS)
Server built:   Nov 19 2015 21:43:13
Server's Module Magic Number: 20120211:24
Server loaded:  APR 1.4.8, APR-UTIL 1.5.2
Compiled using: APR 1.4.8, APR-UTIL 1.5.2
Architecture:   64-bit
Server MPM:     prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/etc/httpd"
-D SUEXEC_BIN="/usr/sbin/suexec"
-D DEFAULT_PIDLOG="/run/httpd/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="conf/mime.types"
-D SERVER_CONFIG_FILE="conf/httpd.conf"

PHP is 5.6.17:
upload_2016-2-4_7-35-38.png
 
FWIW - I have been running tests where even without PCNTL mails are dropped. I have a ticket with @twisted1919 and we will run some debug campaigns as my subscriber list is small (<3000 in one list) and we run it every day - so easy to trace and hopefully determine where the problem is.
 
with PCNTL i was having issue that single campaign was sending 3 times it self in one shot. without PCNTL it works fine but quite slow
 
From my own experiments on a 2CPU/2GB instance at Vultr, I can send 10,000 emails in a little under six minutes.

To get this optimal setting, I changed "Subscriber batches in parallel" to one under Settings/Cron. I used the default of 500 in "Subscribers at once" as a baseline. Then, I ran a dummy campaign from the command line using this command "php apps/console/console.php send-campaigns --campaigns_type=regular --verbose=1" to gauge how many emails I could deliver from MW to my MTA within a minute. Once I found my sweet spot of 600 to where the cronjob completed in under a minute, I change the "Subscribers in parallel" back to its defaults. In order to keep the PHP processes from stumbling over each other fighting for CPU, I reduced "Subscribers in parallel" to four which left resources available for other systems services.

I'm guessing when running multiple campaigns, depending on your CPU, you'll need to adjust "Subscriber batches in parallel". For example, if I were running four different campaigns consistently, I'd reduce "Subscriber batches in parallel" to one to allow four PHP processes instead it possible running sixteen (twisted1919, correct me if this is wrong).

Apache Compile Settings:

Code: 
Server version: Apache/2.4.18 (Unix)
Server built:   Feb  9 2016 12:07:40
Server's Module Magic Number: 20120211:52
Server loaded:  APR 1.5.2, APR-UTIL 1.5.4
Compiled using: APR 1.5.2, APR-UTIL 1.5.4
Architecture:   64-bit
Server MPM:     event
  threaded:     yes (fixed thread count)
    forked:     yes (variable process count)
Server compiled with....
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/usr"
-D SUEXEC_BIN="/usr/bin/suexec"
-D DEFAULT_PIDLOG="/var/run/httpd.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="/etc/apache2/mime.types"
-D SERVER_CONFIG_FILE="/etc/apache2/httpd.conf"

PHP Compile Settings:
Code: 
PHP Version => 5.6.17-pl0-gentoo

System => Linux ajuntapall 4.1.15-gentoo-r1 #2 SMP Thu Feb 4 10:34:04 EST 2016 x86_64
Build Date => Feb  4 2016 06:40:11
Configure Command =>  './configure'  '--prefix=/usr' '--build=x86_64-pc-linux-gnu' '--host=x86_64-pc-linux-gnu' '--mandir=/usr/share/man' '-
-infodir=/usr/share/info' '--datadir=/usr/share' '--sysconfdir=/etc' '--localstatedir=/var/lib' '--prefix=/usr/lib64/php5.6' '--mandir=/usr/
lib64/php5.6/man' '--infodir=/usr/lib64/php5.6/info' '--libdir=/usr/lib64/php5.6/lib' '--with-libdir=lib64' '--without-pear' '--enable-maint
ainer-zts' '--disable-bcmath' '--with-bz2=/usr' '--disable-calendar' '--enable-ctype' '--with-curl=/usr' '--enable-dom' '--without-enchant'
'--disable-exif' '--enable-fileinfo' '--enable-filter' '--disable-ftp' '--with-gettext=/usr' '--without-gmp' '--enable-hash' '--without-mhas
h' '--with-iconv' '--enable-intl' '--enable-ipv6' '--enable-json' '--without-kerberos' '--enable-libxml' '--with-libxml-dir=/usr' '--enable-
mbstring' '--with-mcrypt=/usr' '--without-mssql' '--with-onig=/usr' '--with-openssl=/usr' '--with-openssl-dir=/usr' '--enable-pcntl' '--enab
le-phar' '--enable-pdo' '--enable-opcache' '--without-pgsql' '--enable-posix' '--without-pspell' '--without-recode' '--enable-simplexml' '--
disable-shmop' '--without-snmp' '--disable-soap' '--disable-sockets' '--without-sqlite3' '--without-sybase-ct' '--disable-sysvmsg' '--disabl
e-sysvsem' '--disable-sysvshm' '--without-fpm-systemd' '--without-tidy' '--enable-tokenizer' '--disable-wddx' '--enable-xml' '--enable-xmlre
ader' '--disable-xmlwriter' '--without-xmlrpc' '--without-xsl' '--enable-zip' '--with-zlib=/usr' '--disable-debug' '--without-cdb' '--withou
t-db4' '--disable-flatfile' '--without-gdbm' '--disable-inifile' '--without-qdbm' '--with-freetype-dir=/usr' '--with-t1lib=/usr' '--disable-
gd-jis-conv' '--with-jpeg-dir=/usr' '--with-png-dir=/usr' '--without-xpm-dir' '--without-vpx-dir' '--with-gd' '--with-imap=/usr' '--with-ima
p-ssl=/usr' '--without-interbase' '--with-mysql=mysqlnd' '--with-mysqli=mysqlnd' '--with-mysql-sock=/var/run/mysqld/mysqld.sock' '--without-
unixODBC' '--without-iodbc' '--without-oci8' '--without-pdo-dblib' '--with-pdo-mysql=mysqlnd' '--without-pdo-pgsql' '--without-pdo-sqlite' '
--without-pdo-odbc' '--without-pdo-oci' '--without-readline' '--without-libedit' '--without-mm' '--with-pic' '--with-pcre-regex=/usr' '--wit
h-pcre-dir=/usr' '--with-config-file-path=/etc/php/cli-php5.6' '--with-config-file-scan-dir=/etc/php/cli-php5.6/ext-active' '--disable-embed
' '--enable-cli' '--disable-cgi' '--disable-fpm' '--without-apxs2' 'build_alias=x86_64-pc-linux-gnu' 'host_alias=x86_64-pc-linux-gnu' 'CFLAG
S=-march=native '-O2' '-pipe'' 'LDFLAGS=-Wl,-O1 '-Wl,--as-needed'' 'CPPFLAGS=' 'CXXFLAGS=-march=native '-O2' '-pipe' '-fvisibility-inlines-h
idden''
Server API => Command Line Interface
Virtual Directory Support => enabled
Configuration File (php.ini) Path => /etc/php/cli-php5.6
Loaded Configuration File => /etc/php/cli-php5.6/php.ini
Scan this dir for additional .ini files => /etc/php/cli-php5.6/ext-active
Additional .ini files parsed => /etc/php/cli-php5.6/ext-active/apcu.ini,
/etc/php/cli-php5.6/ext-active/geoip.ini,
/etc/php/cli-php5.6/ext-active/ioncube.ini,
/etc/php/cli-php5.6/ext-active/opcache.ini,
/etc/php/cli-php5.6/ext-active/redis.ini,
/etc/php/cli-php5.6/ext-active/suhosin.ini


PHP API => 20131106
PHP Extension => 20131226
Zend Extension => 220131226
Zend Extension Build => API220131226,TS
PHP Extension Build => API20131226,TS
Debug Build => no
Thread Safety => enabled
Zend Signal Handling => disabled
Zend Memory Manager => enabled
Zend Multibyte Support => provided by mbstring
IPv6 Support => enabled
DTrace Support => disabled

Registered PHP Streams => https, ftps, compress.zlib, compress.bzip2, php, file, glob, data, http, ftp, phar, zip
Registered Stream Socket Transports => tcp, udp, unix, udg, ssl, sslv3, sslv2, tls, tlsv1.0, tlsv1.1, tlsv1.2
Registered Stream Filters => zlib.*, bzip2.*, convert.iconv.*, mcrypt.*, mdecrypt.*, string.rot13, string.toupper, string.tolower, string.strip_tags, convert.*, consumed, dechunk

This program makes use of the Zend Scripting Language Engine:
Zend Engine v2.6.0, Copyright (c) 1998-2015 Zend Technologies
  with the ionCube PHP Loader (enabled) + Intrusion Protection from ioncube24.com (unconfigured) v5.0.18, Copyright (c) 2002-2015, by ionCube Ltd.
  with Zend OPcache v7.0.6-dev, Copyright (c) 1999-2015, by Zend Technologies
  with Suhosin v0.9.38, Copyright (c) 2007-2015, by SektionEins GmbH
 
Last edited:
Just an update - this morning we switched on PCNTL (all the other settings remained the same) which resulted in duplicate mails being sent:
  • Recipients in campaign: 2,851
  • Mails processed according to MailWizz: 4,751
  • Mails transmitted via Port25: 8798
While SwiftMailer upgrade seems to fix that too few emails are being sent, switching PCNTL results in duplicate mails being transmitted.

When analysing the MailWizz output, it is obvious that there must be some threading issue when using PCNTL. When going through log files some recipients will receive the same mail up to 6 times, others will receive it twice.

The MailWizz log reports "Done for [EMAIL-ADDRESS], logging delivery..." 9607 times. Of which 6763 emails are duplicates - this is a serious issue and I doubt this is isolated to just my installation (I guess many MailWizz users have not done a full cross-check between MailWizz, their MTA and the actual recipient).

A ticket is logged with @twisted1919 and I hope we can isolate the issue.

Our PHP config is almost identical to @Mxyzptlk FishStix :
upload_2016-2-25_10-24-53.png

I am switching PCNTL off, but with out parallel sending MailWizz will not be suitable to handle larger campaigns.
 
@bidorbuy - attached is the latest send-campaigns command that we have tested yesterday against around 10 campaigns, each for 100k subscribers, in various ways, and we didn't get any issue, so it seems that fixes it for good.
Please use it and let me know how it goes. The file(after unzip) goes into apps/console/commands folder.
Let me know how it goes.
 

Attachments

twisted1919 said:
@bidorbuy - attached is the latest send-campaigns command that we have tested yesterday against around 10 campaigns, each for 100k subscribers, in various ways, and we didn't get any issue, so it seems that fixes it for good.
Please use it and let me know how it goes. The file(after unzip) goes into apps/console/commands folder.
Let me know how it goes.
Click to expand...

Thank you. I have deployed the change, but since we run a 80K campaign tomorrow, I will rather leave PCNTL off. I will turn PCNTL on for the smaller campaigns from Monday onwards.
 
twisted1919 said:
@bidorbuy - Sure, just please update me via PM on the way things goes so that i know if we're done with this or not :)
Click to expand...

Thank you so much! This is really looking good. This morning we sent out two campaigns (3057 subs - took 7 mins / 86100 subs - took 45 mins) - and I am happy to say that your changes have fixed the duplication issue.

Server utilisation (4 cores / 4 GB RAM with MySQL on the same box) was not a problem:
upload_2016-2-26_9-50-56.png

I am not quite happy with the throughput - my current setup (500 subscribers at once / 5 batches):
upload_2016-2-26_9-48-40.png

Do you think it will be better to increase the batches (to let's say 7-8) and leave subscribers at once the way it is or should I drop subscribers at once to 300 and then have batches raised to 10?

I would tend to think that increasing the number of batches would improve throughput (obviously it would raise CPU, but we have only 60% utilisation).
 
Finally good news after so many try/fail trials :D
bidorbuy said:
I would tend to think that increasing the number of batches would improve throughput (obviously it would raise CPU, but we have only 60% utilisation).
Click to expand...
Yes, i agree with you, go up to 10 subscriber batches in parallel with 300/500 subscribers at once ;)
 
You must log in or register to reply here.
Back
Top