Debugging send-campaign command

rritz

rritz

Member
Maybe someone now can help me interpret this:

This is verbose output
[2017-08-19 07:03:36] - xxxxx@gmail.com - 19/400
[2017-08-19 07:03:36] - Checking if we can send to domain of xxxxx@gmail.com...
[2017-08-19 07:03:36] - Checking if xxxxx@gmail.com is blacklisted...
[2017-08-19 07:03:36] - Checking if the server is over quota...
[2017-08-19 07:03:36] - Preparing the entire email...
[2017-08-19 07:03:36] - Using delivery server: xxxxx (ID: 16).
[2017-08-19 07:03:38] - Sending response is: OK
[2017-08-19 07:03:38] - Sending OK.
[2017-08-19 07:03:38] - Done for xxxxx@gmail.com, logging delivery...

There is a gap of 2 seconds between Using the delivery server and sending response OK
Why does the sending server take so long to reply?
Sometimes it's one second, sometimes five or six seconds. On the average 2 sec means only 30 mails per minute going out ...

The server is my own smtp, but hosted on a different server
What can I do to speed things up?

Oh and just for understanding things better, when I run the verbose command, then 1 cron job gets processed, and when it's finished it will stop sending.

I real, the cron job starts a process sending a batch of 400 subs (in my case) but after a minute the cron job runs again and starts another process sending 400 subs?
So in real time, several processes should be running parallel, to be exact, about 15 or more should be running in parallel, because it takes about 15 minutes for 400 subs to complete in this scenario here.

And in real this should 15x my sending per minute. But it doesn't
My real sending per minute is 30 emails or even less. Could it be that the sending batches do not run parallel for some reason? How will I know?

Can I run the command in verbose every minute and start parallel processes?

Ok, and now after some time, also between one sub and the next I get gaps:
[2017-08-19 08:55:35] - Sending OK.
[2017-08-19 08:55:35] - Done for xxxxxx@xxxxxx.com, logging delivery...

[2017-08-19 08:55:40] - xxxxxx@hotmail.com - 331/400
[2017-08-19 08:55:40] - Checking if we can send to domain of xxxxxx@hotmail.com...
[2017-08-19 08:55:40] - Checking if xxxxxx@hotmail.com is blacklisted...
[2017-08-19 08:55:40] - Checking if the server is over quota...
[2017-08-19 08:55:40] - Preparing the entire email...
[2017-08-19 08:55:40] - Using delivery server: xxxxxx.com (ID: 16).
[2017-08-19 08:55:52] - Sending response is: OK
[2017-08-19 08:55:52] - Sending OK.
[2017-08-19 08:55:52] - Done for xxxxxx@hotmail.com, logging delivery...

5 sec gap between logging and the next sub start

And may I add that the server load is still very low: about 500MB of RAM used only, and about 1%-2% CPU usage
 
Last edited:
the servers are all located in the US, same hosting company
anyway, the latency between the mw server and the smtp server doesn't really bother me now. I did a test with the smtp that's on the same server as mw. Sending is faster, like 1- 2 mails per second
still, it doesn't go over 120 mail per minute.

So I have to assume that at all times only one send is running.

In the kb I read though that the next cron job after 1 minute should trigger a second send command and a second batch should get loaded and sent. It seems to me this just doesn't happen in my mw installation. I wonder why?
 
rritz said:
I wonder why?
Click to expand...
You're using pcntl sending, which means we you;re already sending in parallel, and to be able to do this without any issues, when sending starts, we put a lock on the command so that successive calls to it don't succeed. This makes sure your campaigns runs in parallel without them being loaded by other processes and get corrupted data.
You can increase the number of subscriber batches from settings and decrease the number of subscribers at once, this way you force mailwizz send more in parallel thus reaching more connections in a minute ;)
 
I don't understand your answer. sorry.
First of all, pcntl is disabled
And second, are you talking about the debug command or the regular way mw is running?

Maybe I am daft, but this is what I thought is happening when MW is running regularly - without pcntl:

Cron triggers send campaigns command at minute 0
campaigns 1 - 5 are found, 1st campaign loads 400 subs (cause that are my settings)
The 400 subs are sent to, all is logged
next campaign is loaded, 400 subs for next campaign
etc

While this is happening, cron has triggered next send campaign command at minute 1
campaigns 1 - 5 are found, 1st campaign loads 400 subs (cause that are my settings)
The 400 subs are sent to, all is logged
next campaign is loaded, 400 subs for next campaign
etc

and so on and so on

At some point a lot of send commands are running at once
If one send command sends 30 Emails per minute then by the time they are all running at once - say 20 are running at once - I will be sending 600 emails per minute. If my server has enough memory and cpu, and if the sql queries are fast enough

Is this right or wrong?

Now what I see: in regular running, my mw installation does not run the send campaigns commands at once, but queues them up one after the other.
So, instead of the second batch being loaded at minute 1, the second batch gets loaded at minute 20, because it is waiting for the send command that started at minute 0 to FINISH

An now I am sending not 600 mails per minute, but only 30 mails per minute, and my campaign takes 3 days and 19 hours to finish

I could as well run this mw installation on a shared hosting and let the cron job run every 20 minutes - the speed would be just the same.

There's something here I obviously don't understand. Do I have to have pcntl enabled too run batches in parallel? I didn't think so.
 
rritz said:
I don't understand your answer. sorry.
First of all, pcntl is disabled
And second, are you talking about the debug command or the regular way mw is running?

Maybe I am daft, but this is what I thought is happening when MW is running regularly - without pcntl:

Cron triggers send campaigns command at minute 0
campaigns 1 - 5 are found, 1st campaign loads 400 subs (cause that are my settings)
The 400 subs are sent to, all is logged
next campaign is loaded, 400 subs for next campaign
etc

While this is happening, cron has triggered next send campaign command at minute 1
campaigns 1 - 5 are found, 1st campaign loads 400 subs (cause that are my settings)
The 400 subs are sent to, all is logged
next campaign is loaded, 400 subs for next campaign
etc

and so on and so on

At some point a lot of send commands are running at once
If one send command sends 30 Emails per minute then by the time they are all running at once - say 20 are running at once - I will be sending 600 emails per minute. If my server has enough memory and cpu, and if the sql queries are fast enough

Is this right or wrong?

Now what I see: in regular running, my mw installation does not run the send campaigns commands at once, but queues them up one after the other.
So, instead of the second batch being loaded at minute 1, the second batch gets loaded at minute 20, because it is waiting for the send command that started at minute 0 to FINISH

An now I am sending not 600 mails per minute, but only 30 mails per minute, and my campaign takes 3 days and 19 hours to finish

I could as well run this mw installation on a shared hosting and let the cron job run every 20 minutes - the speed would be just the same.

There's something here I obviously don't understand. Do I have to have pcntl enabled too run batches in parallel? I didn't think so.
Click to expand...

MailWizz is not running any cron over another one, i.e. if one cron is still being worked off, no new one is allowed, this is to avoid data corruption (I think this is somewhere explained, either above or in the KB).

To nicely see what is happening, you could set camps at once to 1 and pcntl both settings to 1 or off, and use only 10 subs in the batch, and see exactly what the output is.
Then next manual run, you can change one setting and see again.
This way, you get a nice view into the workings!

;)
 
frm.mwz said:
MailWizz is not running any cron over another one, i.e. if one cron is still being worked off, no new one is allowed, this is to avoid data corruption (I think this is somewhere explained, either above or in the KB).
Click to expand...
This is correct, but it only applies when pcntl is active for sending, this is very important to note. When pcntl is off this does not apply anymore.

rritz said:
Is this right or wrong?
Click to expand...
Your understanding is correct, this is exactly how things work when pcntl is disabled.
There was an older version of mailwizz where we had a global lock(maybe @frm.mwz is confused because of that) but we have removed that lock, so if you are using version 1.4.3 you should see the above behavior like you have explained it.
 
twisted1919 said:
This is correct, but it only applies when pcntl is active for sending, this is very important to note. When pcntl is off this does not apply anymore.


Your understanding is correct, this is exactly how things work when pcntl is disabled.
There was an older version of mailwizz where we had a global lock(maybe @frm.mwz is confused because of that) but we have removed that lock, so if you are using version 1.4.3 you should see the above behavior like you have explained it.
Click to expand...

Ok, so if my mw installation is behaving odd - ie it is not sending in parallel but waiting for the first send to finish before the next one starts - what can I do to find out what's happening? And find out how to fix it?

I am running 1.4.3
 
screenshot here: https://ibb.co/iD9t1k

you can see that the send campaigns command runs for 1583 sec and uses 22MB - this one has sent 400 emails
the next commands run for a few sec or less, use 2MB and send no emails
Any suggestion?

Maybe MW is thinking pcntl in on and locking the sends?
 
Try to disable the cron job for send-campaigns, then open two ssh shells and run the send-campaigns command in each of them, in verbose mode, run the second one at a 5 seconds interval to be sure you give time to the first one to start.
Make sure you have at least 2 campaigns for sending.
Make sure pcntl is disabled from backend > settings > cron. Double check and save those settings once again, regardless of their values.
 
I did that, @twisted1919 but in the second instance the output is:
[2017-08-23 06:42:08] - Loading 5 campaigns, starting with offset 0...
[2017-08-23 06:42:08] - Found 1 campaigns and now starting processing them...
[2017-08-23 06:42:09] - This cycle completed in 0.31 seconds and used 4 MB of memory!
[2017-08-23 06:42:09] - CPU usage in last minute: 1.63, in last 5 minutes: 1.40, in last 15 minutes: 1.25!
[root@xxxxxxxxxxxx ~]# /usr/bin/php -q /var/www/mw/apps/console/console.php send-campaigns --verbose=1
[2017-08-23 06:42:26] - Loading 5 campaigns, starting with offset 0...
[2017-08-23 06:42:26] - Found 1 campaigns and now starting processing them...
[2017-08-23 06:42:26] - This cycle completed in 0.31 seconds and used 4 MB of memory!
[2017-08-23 06:42:26] - CPU usage in last minute: 1.63, in last 5 minutes: 1.42, in last 15 minutes: 1.26!

No emails get sent, and here even the second campaign is not found
 
@twisted1919 I found a file on my server in apps/console/commands/shell called send-parallel-campaigns

I looked through the mw folders for version 1.4.3 and 1.4.1 and neither in latest nor in update this file exists
Could it be a leftover from an earlier version? Could it cause the issue?
 
@rritz - that file is not used anymore, we removed it a long time ago from the app.

From the output i see in your above post, it seems it finds a campaign in each case and it starts processing it, what happens next ?
 
@twisted1919 ok thxs, I have removed the file
to answer your question: What you see is all that ever happens in the second shell instance.
When I run the send campaign command this is the result:

[2017-08-23 06:42:08] - Loading 5 campaigns, starting with offset 0...
[2017-08-23 06:42:08] - Found 1 campaigns and now starting processing them...
[2017-08-23 06:42:09] - This cycle completed in 0.31 seconds and used 4 MB of memory!
[2017-08-23 06:42:09] - CPU usage in last minute: 1.63, in last 5 minutes: 1.40, in last 15 minutes: 1.25!

then it goes into idle
so a - it finds only one campaign out of two
b - it doesn't load any subs nor send any email

The process in the first shell instance - the one that got started first - runs off the subs 1-400 for camp 1 and then 1-400 for camp 2
and then reports a completed cycle
 
@rritz - Let me look into this for you, open a ticket with backend url and login to your app and ssh access. Please also prepare 2 campaigns for sending so that i can test.
 
Hi @twisted1919 thank you - I don't wanna waste your time though.
I was thinking maybe the developer who helped me set up my server messed around in mw code too
otherwise, I wouldn't understand this going wrong?
Let me try one more thing - just need to know if it will break my mw installation
Can I delete the apps folder and replace it with the apps folder of a fresh 1.4.3 download? Or are there any files with settings I need to keep?
Maybe just the console folder?
To make sure I don't have any code in there that the developer messed around with, without having to do a complete reinstall. Or is this idea no good?

If this doesn't help, I will buy support pack and open a ticket to let you look into it - Thxs a lot!
 
You must log in or register to reply here.
Back
Top