Process campaigns using Redis Queue.

twisted1919

twisted1919

Administrator
Staff member
NOTE: Please do not rely on this functionality anymore. Starting with mailwizz 1.3.5.9 the send-campaigns command has been improved so much that there is no need for Redis or the like anymore so in the future we will remove redis from mailwizz entirely.


Since version 1.3.5, MailWizz adds experimental support for using queues with Redis.
Using regular cron job sending, on a linode with 2 cores and 2GB of ram(~512 available), sending a campaign with 10k subscribers at once took as shown below:
Code: 
[root@work-01 ~]# time /usr/bin/php -q /home/domain/public_html/apps/console/console.php send-campaigns

real    17m40.299s
user    8m1.417s
sys     2m53.203s

When redis queue has been enabled, sending same amount of emails, 10k, took as shown below:
Code: 
[root@work-01 ~]# time /usr/bin/php -q /home/domain/public_html/apps/console/console.php send-campaigns

real    4m6.028s
user    2m26.167s
sys     0m53.307s
As we can see, sending with redis queue is considerably faster, ~4 times faster on a considerably small machine.

In order to use this sending method, your server needs to meet the following requirements:
1. Redis server
2. PHP's pcntl's extension installed and enabled with no function disabled.
3. If possible, install php's native client for redis.
4. command line/ssh access so that you can run various commands.

See https://forum.mailwizz.com/threads/redis-install.205/ for a guide on how to install #1 and #3.

In order to enable sending via this method, firstly we need to enable and configure the Redis Queue from mailwizz backend, so go to Backend -> Settings -> Queue and do the needed adjustments and enable it, then save the changes.

Once the redis queue is ready for use, we need to decide which delivery servers are going to use the queue. If you try to edit a delivery server now after the queue feature has been enabled, you will notice a new field, called "Use queue". You need to set this field to YES for the servers that you want to use this sending method. Do that and then save your delivery server.
(P.S: you can have servers using queue and servers doing regular delivery without queue, both work just fine in the same time)

Now the system is ready to queue all your emails in the redis server when campaigns are being sent via the delivery servers you have choosen to use the queue on.
If you send a test campaign, you will notice that sending it is going to be blazing fast, like really fast that you'll think something is wrong.
Well, there's nothing wrong, it's so fast because we just queue the emails in the redis server, we are not sending them at all.
In order to do the actual sending, we need some kind of daemon that will run permanently and read the redis queue, pick the campaign emails, send them and then empty the queue.
For this exact reason, a new mailwizz command line command is available.
If from command line you run:
Code: 
/usr/bin/php -q /absolute/path/to/apps/console/console.php
you will see something like:
Code: 
[root@work-01 ~]# /usr/bin/php -q /home/domain/public_html/apps/console/console.php
Yii command runner (based on Yii v1.1.16)
Usage: /home/domain/public_html/apps/console/console.php <command-name> [parameter                                                                    s...]

The following commands are available:
- archive-campaigns-delivery-logs
- archivecampaignsdeliverylogs
- bounce-handler
- bouncehandler
- daily
- feedback-loop-handler
- feedbackloophandler
- hello
- migrate
- option
- process-delivery-and-bounce-log
- process-subscribers
- processdeliveryandbouncelog
- processsubscribers
- queue
- redisqueue
- send-campaigns
- send-transactional-emails
- sendcampaigns
- sendtransactionalemails
- update

To see individual command help, use the following:
   /home/domain/public_html/apps/console/console.php help <command-name>
So the command that interests us is the queue command.
Please note that this is not a command that should be placed in the cron jobs to be called once at x minutes. This is a daemon that once started it will run permanently and you should not have more than one instance of it running in the same time.

It can be called from command line like:
Code: 
/usr/bin/php -q /absolute/path/to/apps/console/console.php queue --workers=10 --interval=5 --verbose=0
The --workers param means how many separate processes will be started to check your queue. Please note that all these processes will run in parallel and might eact a considerable amount of ram and cpu if the number of workers is too high. This is PHP, not the best tool for this kind of stuff.
The --interval param, is the number of seconds between the queue checks. In the above case, the queue will be checked once at 5 seconds by each worker.
The --verbose param is a debug flag, if it's set to 1, you can see how and what is processing.

Once you call this command (you should call it with --verbose=1) you will see how it starts processing all the emails from the redis queue.
Once you are sure this works as expected you can kill the queue process so that you can later start it again in background.
To do this, simply run:
Code: 
ps aux | grep queue
Which will return something like:
Code: 
[root@work-01 ~]# ps aux | grep queue
root     21641  0.0  0.5 550952 11116 ?        Ss   Feb04   2:18 /usr/bin/php...
Where 21641 is the process PID and we can kill it with:
Code: 
kill -9 21641

Then we can start the process again and send it in background with following command:
Code: 
nohup /usr/bin/php -q /absolute/path/to/apps/console/console.php queue --workers=10 --interval=5 --verbose=0 >/dev/null 2>&1 &
After you do this, the process will run till you kill it, till the server reboots or till it dies because of some fatal error, like insufficient memory or the like.

To recap, when using redis queue, the cron job that runs once a minute, the one that calls send-campaigns will load your campaigns from database and instead of sending the emails directly, it will store those emails in the redis queue from where they will be later picked up by the queue daemon that you have started and it runs in background and this daemon is the one that does the actual delivery. Each worker will process 100 emails at once, so when you have 10 workers, you'll process 1k emails at once. Keep this in mind so that you don't use all your server memory because of this.

That's it.
If any questions related to this, please ask but keep in mind, this is still experimental stuff.
For me it worked fine while i tested it but I recommend you to test this enough before you rely on it.

Thanks.
 
Last edited:
Hi

I've been playing around with this and I have redis installed on a separate server, the daemon obviously won't find the queue in this instance, how I can place the daemon on the redis server? Or is that not going to be possible?

Cheers

Rob
 
Hey,

I think it doesn't matter where redis is installed, why would it?
I mean, as long as you give the right details into the configuration, it should connect(remotely) and put/get data from redis without issues.

Did you try it? Do you get any specific issue?
 
Hi

OK, probably I'm not understanding where the "queue daemon" is firing instructions to and where the email is actually sitting, if I can get my head around the workflow/process then it shouldn't be too difficult.

I setup a delivery server and set it to use redis (another server), set the daemon running using verbose=1 and it fires as it should. Campaigns are queued but emails aren't sent.

Using "redis-cli monitor" on the redis server it shows that there is an "email:queue" so redis is receiving the data.

A little more investigation is needed it would seem :)

Cheers

Rob
 
Hey,

Okay, that's good, if emails are sent into redis you are half way there.
Now you need to process them from redis and empty the queue, and you do that by running:
Code: 
/usr/bin/php -q /absolute/path/to/apps/console/console.php queue --workers=10 --interval=5 --verbose=1
Once you run the above, mailwizz will connect to Redis and get all queued emails and process them.
Please note that the above command is a daemon that once started runs forever, it's not something that should be added as a cron job, but started as described above, with nohup when you are done testing.

Thanks.
 
Hi

Got this sorted in the end. I noticed in another post you use PHP 5.6 for dev so I upgraded to 5.6.7 and it started working fine. I was on 5.5.9.

Now to run some simulations.

Cheers

Rob
 
Is it possible to automate the following command, so the daemon starts automatically upon reboot or if it dies:

nohup /usr/bin/php -q /absolute/path/to/apps/console/console.php queue --workers=10 --interval=5 --verbose=0 >/dev/null 2>&1 &​
 
Not to my knowledge, funnily enough I've been working on a monitoring check today and some way to reinstate it if it stops working. I'll let you know what I work out.

Cheers

Rob
 
Couldn't a simple scipr
Rob said:
Not to my knowledge, funnily enough I've been working on a monitoring check today and some way to reinstate it if it stops working. I'll let you know what I work out.

Cheers

Rob
Click to expand...
Couldn't a simple script that search for "emails-queue" process and starts a new process if it isn't found work?
 
Question. Does this Redis queue adhere to the limits in settings -> cron? So if I set the limit to 5,000 emails per minute, we wouldn't go over that?

I'm trying to find the best settings for the Redis queue process so that we hit our limit yet dont continually eat CPU and RAM when there are no emails to send.
 
Is there a way to track the rate at which Redis queue sends the email? I presume the "Estimated completion rate" for each "Campaign overview" is the rate at which emails are being queued into Redis, not the actual rate of emails being sent. I'd be interested to know how fast Redis is sending emails.
 
Question. Does Redis need to be configured (i.e. change any Redif settings in redis.conf) to make it properly work for MailWizz or is default fine?

Furthermore, how does MailWizz make use of Redis? Does it clear out Redis when it is done with all queues? Are there specific keys created? I ask because I'm thinking of writing a script that checks Redis for specific keys related to MailWizz; if it finds some, it checks to see if the Redis queue agent is running. If not running, it runs it. If it finds no keys, it kills all Redis queue agents that are running. Throw this script into a cron and that way the Redis queue agent is run / killed on an as needed basis.
 
A few answers for you guys:
Lakjin said:
Couldn't a simple script that search for "emails-queue" process and starts a new process if it isn't found work?
Click to expand...
That could work as well, something like:
Code: 
#!/bin/bash
ps cax | grep  queue >/dev/null
if[ $? -eq 0];then
     echo "Process is running."
else
     echo "Process is not running, start it..."
fi
ref: http://stackoverflow.com/questions/9117507/linux-unix-command-to-determine-if-process-is-running

Lakjin said:
Is there a way to track the rate at which Redis queue sends the email? I presume the "Estimated completion rate" for each "Campaign overview" is the rate at which emails are being queued into Redis, not the actual rate of emails being sent. I'd be interested to know how fast Redis is sending emails.
Click to expand...
Actually if you run:
Code: 
php -q /the/path/to/console.php queue --show_size=1
You'll get back the number of emails in queue.

Lakjin said:
Question. Does this Redis queue adhere to the limits in settings -> cron? So if I set the limit to 5,000 emails per minute, we wouldn't go over that?
Click to expand...
Yes it does.

Lakjin said:
Furthermore, how does MailWizz make use of Redis? Does it clear out Redis when it is done with all queues? Are there specific keys created? I ask because I'm thinking of writing a script that checks Redis for specific keys related to MailWizz; if it finds some, it checks to see if the Redis queue agent is running. If not running, it runs it. If it finds no keys, it kills all Redis queue agents that are running. Throw this script into a cron and that way the Redis queue agent is run / killed on an as needed basis.
Click to expand...
Mailwizz uses resque behind the scenes, so have a look at https://github.com/resque/resque for techincal stuff.
 
You must log in or register to reply here.
Back
Top