Mail throughput performance comparison

bidorbuy

bidorbuy

Member
We currently have the following settings:
  • PCNTL enabled, 5 campaigns in parallel, 10 subscriber batches in parallel
  • Subscribers at once: 10,000
and we just finished a campaign to 275K subscribers in 1 hour 46 min on a 4core server with 4GB of RAM. During the send the CPU was around 50% utilised.

In our environment we have MailWizz queue into a local postfix which then relays into Port25. When tailing the local maillog, there are certainly pauses and I do feel that I should be able to push the campaign out much faster into the local postfix. Our Port25 manages the QoS and sending limits/backoffs across a number of VMTAs and I just want to have the campaign pushed out from the local MailWizz server as fast as possible.

If anyone has some stats/config suggestions, this would be appreciated - especially throughput examples. In comparison, our current Interspire manages the same campaign in under 1 hour.

Correction: The above settings did not result in 275K mails being sent. MailWizz reported this, but when looking at the postfix maillog, only 38K mails were processed. I have logged a support ticket, but we have run out of ideas - in my opinion the issue is possibly related to PCNTL.

In another test with PCNTL, the MailWizz log shows 52593 mails logged, Postfix has only received and processed 3988.

When turning PCNTL off and using default settings, it takes 5 hours for 73,000 mails into local Postfix.
 
Last edited:
The pauses are because the cron job, which might run at second 1 and have nothing to pick then at second 2 the previous cron job finishes processing and leaves a campaign ready for sending. This way you waste 59 seconds where you'd be able to throw few thousand emails.
You can create a shell script that calls send-campaigns command in a loop, this way making sure there is no second wasted.
Such script would look like:
Code: 
#!/bin/bash
while :
do
    /usr/bin/php -q /home/domain/public_html/apps/console/console.php send-campaigns --verbose=1
    sleep 1
done
you'd save it as sendcampaigns.sh maybe, then you'd chmod +x sendcampaigns.sh to make it executable.
Next you'd disable the cron job that calls the send-campaigns command so that you don't break processing and you'd run the new command from command line as:
Code: 
./sendcampaigns.sh
Which will produce lots of output so that you know what happens.
If you need to supress that output, then /dev/null it:
Code: 
./sendcampaigns.sh >/dev/null
CTRL + C kills the running script.

At some point you might like what you have accomplished and want to stick to it but you see that if you close your ssh connection, this little script dies too. At that point, nohup it to send it in background:
Code: 
nohup ./sendcampaigns.sh >/dev/null 2>&1


Please note that i haven't tested all the above code, but 99% should work.
If you ask me, 275K in ~2 hours is good enough.


Have fun ;)
 
Thanks for the explanation of the cronjob. Could you perhaps explain how raising the limit of "Subscribers at once" and "Subscriber batches in parallel" work - my thought was that if it is 10,000 subscribers at once and batches is 10, does this mean that 10 batch processes run processing 1,000 subscribers each or is it 10 batches of 10,000 subscribers?

If the cronjob runs every 2 minutes and the current job is still busy, will it spawn another job and if so how does it affect the batches?

Since I have not seen much CPU issues, I am wondering how high I could go with batches and subscribers at once and if there is any bottleneck I should look out for (i.e. result set returned by MySQL could cause memory issues if subscribers at once is too big?)
 
bidorbuy said:
my thought was that if it is 10,000 subscribers at once and batches is 10, does this mean that 10 batch processes run processing 1,000 subscribers each or is it 10 batches of 10,000 subscribers?
Click to expand...
This is correct. This means that many will be processed, but doesn't have a timestamp to know exactly how long will take to process them.
Basically it spawns the needed processes to process that much. Those processes can end in 1 minute, or 10, depending on your settings.

bidorbuy said:
If the cronjob runs every 2 minutes and the current job is still busy, will it spawn another job and if so how does it affect the batches?
Click to expand...
It doesn't. The cron job is smart enough to only pick campaigns that are not yet in process mode.
 
bidorbuy said:
Since I have not seen much CPU issues, I am wondering how high I could go with batches and subscribers at once and if there is any bottleneck I should look out for (i.e. result set returned by MySQL could cause memory issues if subscribers at once is too big?)
Click to expand...
usually mysql will be the main problem not the cpu. This is all a try and fail type of process where you need to try a lot and see what settings fits you best.
 
Anyone else able to provide their throughput and settings. With PCNTL on, mails seem to drop. With PCNTL off, throughput is horrible.
 
Mxyzptlk FishStix said:
Post you postfix config (postconf -n) and a screenshot of your delivery server settings.
Click to expand...

Please also have a look at this thread - https://forum.mailwizz.com/threads/...end-1-000-000-mails-at-24-hour.198/#post-8838

I don't think there is anything wrong with our postfix conf as we are able to send several million transactional mails from our Java servers via local postfix relaying into Port25.

Here the config:

Code: 
alias_maps = hash:/etc/aliases
command_directory = /usr/sbin
config_directory = /etc/postfix
daemon_directory = /usr/libexec/postfix
inet_interfaces = 10.0.0.83, 127.0.0.1
mail_owner = postfix
masquerade_domains = DOMAIN.COM
mydestination = $myhostname localhost.$mydomain localhost $mydomain
mydomain =  SUB.DOMAIN.COM
myhostname = $mydomain
mynetworks = 127.0.0.0/8
mynetworks_style = host
myorigin = $mydomain
queue_directory = /var/spool/postfix
relay_domains =
relayhost = [###.###.###.###]
unknown_local_recipient_reject_code = 550
virtual_alias_domains = mailwizz.pvt.lan
virtual_alias_maps = hash:/etc/postfix/virtual

upload_2016-2-12_15-56-20.png
 
You must log in or register to reply here.
Back
Top