Improving ElasticEmail API

[Lakjin]

in /apps/common/models/DeliveryServerElasticemailWebApi.php change this...

            if ($response['status'] != 'success' || strpos($response['message'], '-') !== false) {
                throw new Exception(Yii::app()->ioFilter->stripClean($response['message']));
            }

To this...

            if ($response['status'] != 'success' || substr_count($response['message'], '-') != 4 || strlen(trim($response['message'])) != 36 || stripos($response['message'], 'error') !== false) {
                throw new Exception(Yii::app()->ioFilter->stripClean($response['message']));
            }

This helps better identify that the response is the job ID (the unique ID ElasticEmail returns upon successful send) and identifies if ElasticEmail returns an error (I've had an issue with their API returning 403 error but MailWizz not properly detecting that).

 

[twisted1919]

Why does the $response['message'] must be 36 chars in length ?

 

[Lakjin]

Why does the $response['message'] must be 36 chars in length ?

Because in their v1 API (which MailWizz uses; you may want to consider upgrading to v2) returns a GUID on successful delivery, which is 36 characters in length with 4 dashes.

 

[Lakjin]

Why does the $response['message'] must be 36 chars in length ?

What happens when delivery fails -- where is the failed delivery recorded? Does MailWizz attempt redelivery? After adding the above code, I no longer see failed deliveries (the one that threw 403 errors) in delivery_log but they also have not been delivered.

 

[twisted1919]

@Lakjin - We don't retry sending failed emails. We used to, but dozen version ago we removed this because of the overhead.

I no longer see failed deliveries (the one that threw 403 errors) in delivery_log but they also have not been delivered

Most likely it needs more attention, i'm looking into it today.

 

[frm.mwz]

@Lakjin - We don't retry sending failed emails. We used to, but dozen version ago we removed this because of the overhead.

Most likely it needs more attention, i'm looking into it today.

Definitely a good idea to bring this back, with a blast/improvement (and allow those who don't want it to switch it off).
Especially now, that there is @AHK's extension!
Thx for looking into it

 

[Lakjin]

@Lakjin - We don't retry sending failed emails. We used to, but dozen version ago we removed this because of the overhead.

Would you consider adding it back? It seems like a critical feature for deliverability reliability, as delivery can fail for numerous reasons. What about a simple approach of simply keeping a delivery_failed database and reprocessing the emails in that database at the end of a campaign?

EDIT: Another approach would be simply attempting redelivery right after failure.

 

[twisted1919]

@Lakjin - Sure i am considering to bring it back, but not in the way it was till now.
I was thinking to do a check right before the campaign must be put to sent status, and remove all the emails with giveup status, do this 3 times, and then mark the campaign as sent. This way we know we tried.
I just need to make sure and weight the impact this might have on performance though and more than this, how this will work with the temporary tables feature, which will be pretty tricky since it implies repopulating those tables.

 

[frm.mwz]

remove all the emails with giveup status

Do you mean remove them so that one cannot even find that htey were GiveUps? This would make @AHK's extension unusuable, and it is so good to have it!

weight the impact this might have on performance

If it is optional, then folks decide if they want to focus on delivery or speed.

how this will work with the temporary tables feature

Can you remind if the temp tables work with pcntl, and if they are enabled by default now (so far it was in beta and not enabled, correct)?

Maybe combine options:
# temp tables and not much error handling (and no pcntl)
# no temp tables, full error handling, pcntl
?

 

[twisted1919]

This would make @AHK's extension unusuable, and it is so good to have it!

His extension does exactly this, it removes giveups to send to them again

If it is optional, then folks decide if they want to focus on delivery or speed.

Yeah, most likely we will put an option in admin, makes sense this way.

Can you remind if the temp tables work with pcntl, and if they are enabled by default now (so far it was in beta and not enabled, correct)?

They do work just fine with pcntl, but they are not enabled by default just yet.

 

[frm.mwz]

His extension does exactly this, it removes giveups to send to them again

Ah, thanks for clarifying what "remove giveups" means. I like this kinda 'remove', and hence pledge to remove all errors (ie take care of them)!


Maybe a notification email with all errors (ie all stati except 'success') could be sent (optionally) after each camp/cron? This way one would have a record of (hopefully) decreasing delivery errors (if any), and be able to handle manually if need be?

Thanks for working into this direction, it is one the great strengths of MailWizz to take care of things far more thoroughly than other mailers!

 

[Lakjin]

@Lakjin - Sure i am considering to bring it back, but not in the way it was till now.
I was thinking to do a check right before the campaign must be put to sent status, and remove all the emails with giveup status, do this 3 times, and then mark the campaign as sent. This way we know we tried.
I just need to make sure and weight the impact this might have on performance though and more than this, how this will work with the temporary tables feature, which will be pretty tricky since it implies repopulating those tables.

How about an easy solution something along the lines of:

-Create a new "delivery_failed" table
-When delivery fails of an email, instead of putting that email into "delivery_log", put it into "delivery_failed"
-Every time send-campaigns cron runs, check "delivery_failed" table for emails that need to be resent. Emails that are resent successfully, or have been retried 3 times (or whatever limit you set), can be put in "delivery_log" table

This way, no impact on performance and it will work just fine with temporary queue tables.

I think this feature is critical for normal uses. In my situation, it is even more so critical due to issues I'm having with 403 delivery errors. Apparently ElasticEmail's API limits 30 concurrent connections per IP, so about 10-15% of my emails are returning with 403 errors due to having too many connections to their API and MailWizz is not resending the emails later on. I've added temporary code to attempt retry, but my temporary code retries immediately after failure, which isn't a solution given the connections limitation.

 

[Lakjin]

@Lakjin - We don't retry sending failed emails. We used to, but dozen version ago we removed this because of the overhead.

Most likely it needs more attention, i'm looking into it today.

Were you able to figure out why failed emails aren't in delivery_log?

 

[frm.mwz]

ElasticEmail's API limits 30 concurrent connections per IP

Have you tried with no more than 30 pcntl?

 

[Lakjin]

Have you tried with no more than 30 pcntl?

That would work, of course, but that means our sending is slower than I want. ElasticEmail has helped me find a temporary solution to the problem, as the problem is at their end of 30 concurrent connections limitation, while they work on a more permanent solution.

 

[twisted1919]

An update here, i added back the ability to resend to failed emails, you have to enable it from Backend > Settings > Cron:


The way it works is just i explained above, right before closing a campaign and marking it as sent, we do a lookup for all the failed emails and we mark them for sending once again. This works just fine for regular campaigns and autoresponders, and also when using queue tables.

 

[Lakjin]

An update here, i added back the ability to resend to failed emails, you have to enable it from Backend > Settings > Cron:
View attachment 3728

The way it works is just i explained above, right before closing a campaign and marking it as sent, we do a lookup for all the failed emails and we mark them for sending once again. This works just fine for regular campaigns and autoresponders, and also when using queue tables.

Awesome, thank you. When will the version with this be released?

 

[twisted1919]

I don't have an ETA right now, maybe we can make a release over the weekend.

 

[Lakjin]

I don't have an ETA right now, maybe we can make a release over the weekend.

Got it, thanks!

 

[frm.mwz]

An update here, i added back the ability to resend to failed emails, you have to enable it from Backend > Settings > Cron:
View attachment 3728

The way it works is just i explained above, right before closing a campaign and marking it as sent, we do a lookup for all the failed emails and we mark them for sending once again. This works just fine for regular campaigns and autoresponders, and also when using queue tables.

That's great.
Would this mean that, once the 3x retry is done, any failed ones (are they still/again marked/findable?) can still be attempted with AHK's extension (e.g. one would have time to prepare other DS)?