Sending Speed (Need Walkthrough)

[Mark Lewis]

I am having some trouble with sending speeds; about 7k emails over 2 hours. We have a quad core machine, with sql on SSD. We are also using MailGun via SMTP. We have used IEM in past on this same server and can accomplish about 750k per hour thru MailGun.

Using SMTP option for MailWizz right now. I have looked into parallel processing but I can't seem to get it to work and neither can our server guy; it does not seem to make any difference nor can we tell if it is running. I just don't understand how it could be a connection issue with our IEM installation works just fine.

We are running slightly old version of MailWizz and don't have the PCNTL options in the settings/cron area. How do we adjust threads and settings in cli? How do we know if threads are sending? How do we check for bottlenecks? What do you need from us to help us troubleshoot this?

I have checked user groups, cron settings, for delivery settings... and here (/settings/customers/campaigns) all 3 places I have set:

Mem limit, 4gb, camps at once, 100, subs at once, 1,000, send at once, 0, pause, 1, emails per minute, 0, change server at 0. These settings are set both in each user group and cron settings tab.

 

[twisted1919]

Hi Mark,

What version of mailwizz are you actually using, is it that old? Why not upgrading?
Have you actually tried to run the send-campaigns command in verbose mode as shown here?

Let me know about the above and we can continue from there.
Thanks.

 

[Mark Lewis]

We will run verbrose tonight with your server engineer. Our MailWizz version is 1.3.5.7. We have not upgrade because our development team had modified direct files to give us some new features (like allowing user to use an admin email database, etc). The team that did the development was not very thoughtful in that they made the changes directly in the code and not via extensions. So if we were to upgrade, we would likely need our current development team (much more thorough) to rebuild everything via extensions.

 

[Mark Lewis]

Here is verbose results -

[2016-09-21 19:23:34] Current progress: 207 out of 710
[2016-09-21 19:23:34] Checking if the delivery server is allowed to send to the subscriber email address domain...OK, took 0 seconds.
[2016-09-21 19:23:34] Checking the subscriber email address into the blacklist...OK, took 0 seconds.
[2016-09-21 19:23:34] Checking server sending quota...OK, took 0 seconds.
[2016-09-21 19:23:34] Checking customer sending quota...OK, took 0 seconds.
[2016-09-21 19:23:34] Preparing email...done, took 0.001 seconds.
[2016-09-21 19:23:34] -> Sending the email for redacted@email.com by using direct method...done, took 0.878 seconds.
[2016-09-21 19:23:35] The email has been sent successfully!
[2016-09-21 19:23:35] Logging delivery...done, took 0.001 seconds.

[2016-09-21 19:23:35] Current progress: 208 out of 710
[2016-09-21 19:23:35] Checking if the delivery server is allowed to send to the subscriber email address domain...OK, took 0 seconds.
[2016-09-21 19:23:35] Checking the subscriber email address into the blacklist...OK, took 0 seconds.
[2016-09-21 19:23:35] Checking server sending quota...OK, took 0 seconds.
[2016-09-21 19:23:35] Checking customer sending quota...OK, took 0.009 seconds.
[2016-09-21 19:23:35] Preparing email...done, took 0.001 seconds.
[2016-09-21 19:23:35] -> Sending the email for redacted@email.com by using direct method...done, took 0.852 seconds.
[2016-09-21 19:23:36] The email has been sent successfully!
[2016-09-21 19:23:36] Logging delivery...done, took 0.001 seconds.

[2016-09-21 19:23:36] Current progress: 209 out of 710
[2016-09-21 19:23:36] Checking if the delivery server is allowed to send to the subscriber email address domain...OK, took 0 seconds.
[2016-09-21 19:23:36] Checking the subscriber email address into the blacklist...OK, took 0 seconds.
[2016-09-21 19:23:36] Checking server sending quota...OK, took 0 seconds.

Delay is sending mail to MailGun but we use MailGun with IEM very quickly but IEM batches emails in groups of 1500 to send over to Mailgun on 1 connection... with 8 threads, 8 connections processing 1500 mails each in IEM.

This log looks like it is processing 1 by 1 and not batching emails. Do I have settings wrong?

 

[Mark Lewis]

Attached are our settings at all three places (they are all the same). Does MW not batch emails per connection like IEM? Like by subscribers at once, I am expecting MW to send 1,000 emails per connection to MailGun in 1 instance. Right now it looks to be doing 1 email per instance.

 

[twisted1919]

We send a single email per connection by default when using smtp. Newer version of mailwizz allows you to set the number of emails per connections though, when you define a new smtp server you have this option.


Now, mailwizz also supports web api integration and the communication is done via their own PHP library which does not have multiple connections support, which means the emails are sent one by one when using the web api.
For both of the above options you can enable and use pcntl.

Bad news is that version 1.3.5.7 that you use does not have any of these options so your only chance is to try a separate install of 1.3.7.2 and see how that goes.

If i remember correctly, 1.3.5.7 still has the send-parallel-campaigns shell command in so maybe you can try that.
There's an obsolete guide here which can help.

Good luck.

 

[Mark Lewis]

Parallel only helps if more than 1 campaign going at once, right? Like if we have 1 camp to 750k, parallel sending won't help, right? Is there something we could do to get us by until upgrade? What about piping emails locally and having an exim or postfix engine send the emails over to mailgun? That way exim could be threaded and MW would be writing locally much faster. Another question, can your scripts (or newest version) run multiple connections simultaneously?

 

[frm.mwz]

Parallel only helps if more than 1 campaign going at once, right? Like if we have 1 camp to 750k, parallel sending won't help, right?

It will spawn multiple parallel processes of the "send-campaigns" command, hence, it will help, I tested it.
It is advisable to read the article.

What about piping emails locally and having an exim or postfix engine send the emails over to mailgun? That way exim could be threaded and MW would be writing locally much faster.

You can do that, but parallel sending might be the quicker solution.

Another question, can your scripts (or newest version) run multiple connections simultaneously?

That is exactly the point of both, the pcntl and the parallel daemon.

For example, without pcntl/parallel sending, one of our servers (in connection with one of our smtp DS) can send out 500/53s with full authentication,
but w/ pcntl it can do 5k/42s (that is over one and the same smtp with multiple concurrent connections).

 

[twisted1919]

What about piping emails locally and having an exim or postfix engine send the emails over to mailgun? That way exim could be threaded and MW would be writing locally much faster

YES, this is a viable option as well. In mailwizz we have delivery servers of type Pickup Directory which you can configure accordingly, then you can use Mailwizz Directory Pickup daemon to pick those emails from the pickup folder and handle them to your mta
See https://github.com/twisted1919/mdp-go and read the readme

Another question, can your scripts (or newest version) run multiple connections simultaneously?

Yes, with OCNTL enabled you can do that.

 

[Mark Lewis]

We tried this here (http://www.mailwizz.com/article/increase-delivery-speed-recursive-or-parallel-processing?rand=123)
How do we know if multiple processes are actually running? Because it looks like it is not working. Also how do we adjust # of threads with our version (no option in backend)? Since it is going 1x1, would like to do 100 processes so it is 100 processes x 1 email (per second) for 100/sec rate of 6k per minute, 360k per hour. There is literally no load on the server when it is sending but looks to be just 1 thread even though we have done this link exactly numerous times.. Please advise.

FYI, we are in process of upgrading our software and building our enhancements as plugins. We just need to get by until everything is recoded as plugins

 

[frm.mwz]

How do we know if multiple processes are actually running?

Look into any process manager (control panel, cli).

Also how do we adjust # of threads with our version (no option in backend)?

Default is 3, see article, which can be adjusted in backend, just as for pcntl in new versions. If it is not there (never saw such a version, but perhaps such exists) then it is in the code
Don't forget to switch off the 'normal' cron when you run the daemon

 

[Mark Lewis]

Attached are the settings we have under cron.. I have checked demo of mailwizz and I know what you are talking about... but I don't have it. Which file would contain this. @twisted1919 Any suggestions?

 

[frm.mwz]

Attached are the settings we have under cron.. I have checked demo of mailwizz and I know what you are talking about... but I don't have it. Which file would contain this. @twisted1919 Any suggestions?

For testing perhaps set
campaigns 10
subs at once 100
send at once 100
pause 0
You can find max throughput by experimenting with the parameters, but higher numbers do not always translate into higher speed.

The file that contains "processes_count=15" is the one from the daemon command
\apps\console\commands\shell\send-parallel-campaigns
Ideally run it from cli to be able to see/adjust what happens.

 

[twisted1919]

@Mark Lewis - the send-parallel-campaigns command comes with a help command, so if you run ./send-parallel-campaigns -h 1 then you get a small help menu.
To view how many processes are running, you do from command line: ps aux | grep send-campaigns

Attached are the settings we have under cron.. I have checked demo of mailwizz and I know what you are talking about... but I don't have it. Which file would contain this. @twisted1919 Any suggestions?

I think he was reffering to latest app version not to the app version you have.

 

[Mark Lewis]

@twisted1919 In command prompt, how do I adjust # of threads? It is mentioned that we can adjust threads in backend but our version does not allow us to change threads in backend. So what is the procedure for adjusting threads in our version?

 

[frm.mwz]

See above in my post, you can hard-code the number of processes to reset the default of 15 to a value of your liking. And you can also set the number of processes in the command line.
You can also find the help info in that file (unchanged from v1357 to v1372):

.../apps/console/commands/shell/send-parallel-campaigns -h 1
Waiting 2 seconds to init...
---------------------------------------------------------------
| Usage: ./send-parallel-campaigns -s 5 -c 100 -p /usr/bin/php-cli  
| To force PHP CLI binary :
| ./send-parallel-campaigns -p /path/to/php-cli/binary              
| To force the number of processes :
| ./send-parallel-campaigns -c 100                                  
| To force the number of sleep seconds between processes :
| ./send-parallel-campaigns -s 5                                    
| To force sending only regular campaigns :
| ./send-parallel-campaigns -t regular                              
| To force sending only autoresponder campaigns :
| ./send-parallel-campaigns -t autoresponder                        
---------------------------------------------------------------