Is MailWizz double sending emails?

Lakjin · Nov 13, 2016

I have a problem ever since updating to the latest MailWizz. Emails send fine, but campaigns appear to be stuck at 100% processing for hours. The campaigns eventually finish, but I'm a bit worried about why it is taking literally hours at 100% processing. It takes the regular few hours to get to 100% processing, but once i it is there, a campaign can be at 100% processing for 12 hours or more. Is MailWizz sending out double emails? What is going on? Why is it taking so long to finish campaigns?

Attached is an example screenshot of a campaign at 100% processing that is still sending out emails (I verified via MySQL requests that it is still processing emails).

Vpul Shah · Nov 14, 2016

You can enable debug mode and see whats going on.

twisted1919 · Nov 14, 2016

@Lakjin - I doubt the app sends double emails. When the app gives you the impression it hangs it means it actually reprocessing emails that failed delivery, and it does this 3 times till it gives up so that can take a while to complete.
The Campaign Delivery Logs from backend > misc should give all the info.

Lakjin · Nov 14, 2016

twisted1919 said:
@Lakjin - I doubt the app sends double emails. When the app gives you the impression it hangs it means it actually reprocessing emails that failed delivery, and it does this 3 times till it gives up so that can take a while to complete.
The Campaign Delivery Logs from backend > misc should give all the info.

This is probably what is happening. Thank you. Will take a look.

frm.mwz · Nov 15, 2016

Lakjin said:
I have a problem ever since updating to the latest MailWizz. Emails send fine, but campaigns appear to be stuck at 100% processing for hours. The campaigns eventually finish, but I'm a bit worried about why it is taking literally hours at 100% processing. It takes the regular few hours to get to 100% processing, but once i it is there, a campaign can be at 100% processing for 12 hours or more. Is MailWizz sending out double emails? What is going on? Why is it taking so long to finish campaigns?

Attached is an example screenshot of a campaign at 100% processing that is still sending out emails (I verified via MySQL requests that it is still processing emails).

besides the above
considering the miniscule difference of 0.1% (182 of 160927 emails) this could be also unsubscribes diminishing the 'out of {total}'

Lakjin · Nov 19, 2016

twisted1919 said:
@Lakjin - I doubt the app sends double emails. When the app gives you the impression it hangs it means it actually reprocessing emails that failed delivery, and it does this 3 times till it gives up so that can take a while to complete.
The Campaign Delivery Logs from backend > misc should give all the info.

This is the query being run, can you tell me what MailWizz is doing?

SELECT `t`.`subscriber_id` AS `t0_c0`, `t`.`subscriber_uid` AS `t0_c1`, `t`.`email` AS `t0_c3`, `t`.`list_id` AS `t0_c2`, `t`.`ip_address` AS `t0_c4`, `t`.`source` AS `t0_c5`, `t`.`date_added` AS `t0_c7` FROM `mw_list_subscriber` `t` LEFT JOIN `mw_list_field_value` `fieldValues40` ON (`fieldValues40`.`subscriber_id`=`t`.`subscriber_id`) LEFT JOIN `mw_list_field_value` `fieldValues18` ON (`fieldValues18`.`subscriber_id`=`t`.`subscriber_id`) LEFT JOIN `mw_list_field_value` `fieldValues19` ON (`fieldValues19`.`subscriber_id`=`t`.`subscriber_id`) LEFT OUTER JOIN `mw_campaign_delivery_log` `deliveryLogs` ON (`deliveryLogs`.`subscriber_id`=`t`.`subscriber_id`) AND (deliveryLogs.campaign_id = '3882') WHERE (((t.list_id='3') AND (t.status='confirmed') AND (`fieldValues40`.`field_id` = 40 AND (`fieldValues40`.`value` = 'yes') ) AND (`fieldValues18`.`field_id` = 18 AND (`fieldValues18`.`value` = 'yes') ) AND (`fieldValues19`.`field_id` = 19 AND (`fieldValues19`.`value` = 'no') )) AND ((t.list_id='3') AND (t.status='confirmed'))) AND (deliveryLogs.subscriber_id IS NULL) GROUP BY t.subscriber_id ORDER BY t.subscriber_id DESC LIMIT 2000 OFFSET 14000

And this

SELECT COUNT(DISTINCT t.subscriber_id) as counter FROM `mw_list_subscriber` `t` LEFT JOIN `mw_list_field_value` `fieldValues40` ON (`fieldValues40`.`subscriber_id`=`t`.`subscriber_id`) LEFT JOIN `mw_list_field_value` `fieldValues18` ON (`fieldValues18`.`subscriber_id`=`t`.`subscriber_id`) LEFT JOIN `mw_list_field_value` `fieldValues19` ON (`fieldValues19`.`subscriber_id`=`t`.`subscriber_id`) LEFT JOIN `mw_list_field_value` `fieldValues46` ON (`fieldValues46`.`subscriber_id`=`t`.`subscriber_id`) LEFT OUTER JOIN `mw_campaign_delivery_log` `deliveryLogs` ON (`deliveryLogs`.`subscriber_id`=`t`.`subscriber_id`) AND (deliveryLogs.campaign_id = '3882') WHERE (((t.list_id='3') AND (t.status='confirmed') AND (`fieldValues40`.`field_id` = 40 AND (`fieldValues40`.`value` = 'yes') ) AND (`fieldValues18`.`field_id` = 18 AND (`fieldValues18`.`value` = 'yes') ) AND (`fieldValues19`.`field_id` = 19 AND (`fieldValues19`.`value` = 'no') ) AND (`fieldValues46`.`field_id` = 46 AND (`fieldValues46`.`value` != 'no') )) AND ((t.list_id='3') AND (t.status='confirmed'))) AND (deliveryLogs.subscriber_id IS NULL)

Lakjin · Nov 20, 2016

Lakjin said:
This is the query being run, can you tell me what MailWizz is doing?

SELECT `t`.`subscriber_id` AS `t0_c0`, `t`.`subscriber_uid` AS `t0_c1`, `t`.`email` AS `t0_c3`, `t`.`list_id` AS `t0_c2`, `t`.`ip_address` AS `t0_c4`, `t`.`source` AS `t0_c5`, `t`.`date_added` AS `t0_c7` FROM `mw_list_subscriber` `t` LEFT JOIN `mw_list_field_value` `fieldValues40` ON (`fieldValues40`.`subscriber_id`=`t`.`subscriber_id`) LEFT JOIN `mw_list_field_value` `fieldValues18` ON (`fieldValues18`.`subscriber_id`=`t`.`subscriber_id`) LEFT JOIN `mw_list_field_value` `fieldValues19` ON (`fieldValues19`.`subscriber_id`=`t`.`subscriber_id`) LEFT OUTER JOIN `mw_campaign_delivery_log` `deliveryLogs` ON (`deliveryLogs`.`subscriber_id`=`t`.`subscriber_id`) AND (deliveryLogs.campaign_id = '3882') WHERE (((t.list_id='3') AND (t.status='confirmed') AND (`fieldValues40`.`field_id` = 40 AND (`fieldValues40`.`value` = 'yes') ) AND (`fieldValues18`.`field_id` = 18 AND (`fieldValues18`.`value` = 'yes') ) AND (`fieldValues19`.`field_id` = 19 AND (`fieldValues19`.`value` = 'no') )) AND ((t.list_id='3') AND (t.status='confirmed'))) AND (deliveryLogs.subscriber_id IS NULL) GROUP BY t.subscriber_id ORDER BY t.subscriber_id DESC LIMIT 2000 OFFSET 14000

And this

SELECT COUNT(DISTINCT t.subscriber_id) as counter FROM `mw_list_subscriber` `t` LEFT JOIN `mw_list_field_value` `fieldValues40` ON (`fieldValues40`.`subscriber_id`=`t`.`subscriber_id`) LEFT JOIN `mw_list_field_value` `fieldValues18` ON (`fieldValues18`.`subscriber_id`=`t`.`subscriber_id`) LEFT JOIN `mw_list_field_value` `fieldValues19` ON (`fieldValues19`.`subscriber_id`=`t`.`subscriber_id`) LEFT JOIN `mw_list_field_value` `fieldValues46` ON (`fieldValues46`.`subscriber_id`=`t`.`subscriber_id`) LEFT OUTER JOIN `mw_campaign_delivery_log` `deliveryLogs` ON (`deliveryLogs`.`subscriber_id`=`t`.`subscriber_id`) AND (deliveryLogs.campaign_id = '3882') WHERE (((t.list_id='3') AND (t.status='confirmed') AND (`fieldValues40`.`field_id` = 40 AND (`fieldValues40`.`value` = 'yes') ) AND (`fieldValues18`.`field_id` = 18 AND (`fieldValues18`.`value` = 'yes') ) AND (`fieldValues19`.`field_id` = 19 AND (`fieldValues19`.`value` = 'no') ) AND (`fieldValues46`.`field_id` = 46 AND (`fieldValues46`.`value` != 'no') )) AND ((t.list_id='3') AND (t.status='confirmed'))) AND (deliveryLogs.subscriber_id IS NULL)

@twisted1919 I think I may have figured out what is going on. The two queries above are from findSubscribers and countSubscribers functions. I think what is happening, new subscribers are added to our list as MailWizz is processing campaigns sent to the list. When MailWizz gets to the end of a campaign, it keeps picking up new subscribers which is why it stays at 100% processing for so long. Any fix for this behavior? For example, only send campaigns to subscribers on the list at the time the campaign has started? An easy way to implement this is to modify the queries to only look for subscribers that where on the list on or before the time the campaign started.

Lakjin · Nov 20, 2016

Lakjin said:
@twisted1919 I think I may have figured out what is going on. The two queries above are from findSubscribers and countSubscribers functions. I think what is happening, new subscribers are added to our list as MailWizz is processing campaigns sent to the list. When MailWizz gets to the end of a campaign, it keeps picking up new subscribers which is why it stays at 100% processing for so long. Any fix for this behavior? For example, only send campaigns to subscribers on the list at the time the campaign has started? An easy way to implement this is to modify the queries to only look for subscribers that where on the list on or before the time the campaign started.

@twisted1919 Specifically, this would work: make sure `date_added` in `mw_list_subscriber` is less than or equal to `started_at` in `mw_campaign` when processing subscribers. I assume both are recorded in same timezone, right? How would I add this to the send campaign command? I really need this.

twisted1919 · Nov 21, 2016

@Lakjin - open apps/common/models/Campaign.php and look for:

PHP:

protected function addRegularCriteria(CDbCriteria $criteria)
{
    if (!empty($this->option->regular_open_unopen_campaign_id) && !empty($this->option->regular_open_unopen_action)) {
        if ($this->option->regular_open_unopen_action == CampaignOption::REGULAR_OPEN_UNOPEN_ACTION_OPEN) {
            $criteria->with['trackOpens'] = array(
                'select'    => false,
                'together'  => true,
                'joinType'  => 'INNER JOIN',
                'on'        => 'trackOpens.campaign_id = :opensUnopensCid',
                'condition' => 'trackOpens.id = (SELECT id FROM {{campaign_track_open}} WHERE campaign_id = :opensUnopensCid AND subscriber_id = t.subscriber_id ORDER BY id ASC LIMIT 1)',
                'params'    => array(':opensUnopensCid' => $this->option->regular_open_unopen_campaign_id),
            );
            return true;
        } elseif ($this->option->regular_open_unopen_action == CampaignOption::REGULAR_OPEN_UNOPEN_ACTION_UNOPEN) {
            $criteria->with['trackOpens'] = array(
                'select'    => false,
                'together'  => true,
                'joinType'  => 'LEFT OUTER JOIN',
                'on'        => 'trackOpens.campaign_id = :opensUnopensCid',
                'condition' => 'trackOpens.subscriber_id IS NULL',
                'params'    => array(':opensUnopensCid' => $this->option->regular_open_unopen_campaign_id),
            );
            return true;
        }
        return false;
    }

    return true;
}

and make it:

PHP:

protected function addRegularCriteria(CDbCriteria $criteria)
{
    // ADD THESE 2 LINES
    $criteria->addCondition('t.date_added <= :cdate');
    $criteria->params[':cdate'] = $this->date_added;
    
    if (!empty($this->option->regular_open_unopen_campaign_id) && !empty($this->option->regular_open_unopen_action)) {
        if ($this->option->regular_open_unopen_action == CampaignOption::REGULAR_OPEN_UNOPEN_ACTION_OPEN) {
            $criteria->with['trackOpens'] = array(
                'select'    => false,
                'together'  => true,
                'joinType'  => 'INNER JOIN',
                'on'        => 'trackOpens.campaign_id = :opensUnopensCid',
                'condition' => 'trackOpens.id = (SELECT id FROM {{campaign_track_open}} WHERE campaign_id = :opensUnopensCid AND subscriber_id = t.subscriber_id ORDER BY id ASC LIMIT 1)',
                'params'    => array(':opensUnopensCid' => $this->option->regular_open_unopen_campaign_id),
            );
            return true;
        } elseif ($this->option->regular_open_unopen_action == CampaignOption::REGULAR_OPEN_UNOPEN_ACTION_UNOPEN) {
            $criteria->with['trackOpens'] = array(
                'select'    => false,
                'together'  => true,
                'joinType'  => 'LEFT OUTER JOIN',
                'on'        => 'trackOpens.campaign_id = :opensUnopensCid',
                'condition' => 'trackOpens.subscriber_id IS NULL',
                'params'    => array(':opensUnopensCid' => $this->option->regular_open_unopen_campaign_id),
            );
            return true;
        }
        return false;
    }

    return true;
}

frm.mwz · Nov 21, 2016

twisted1919 said:

@Lakjin - open apps/common/models/Campaign.php and look for:

PHP:

protected function addRegularCriteria(CDbCriteria $criteria)
{
    if (!empty($this->option->regular_open_unopen_campaign_id) && !empty($this->option->regular_open_unopen_action)) {
        if ($this->option->regular_open_unopen_action == CampaignOption::REGULAR_OPEN_UNOPEN_ACTION_OPEN) {
            $criteria->with['trackOpens'] = array(
                'select'    => false,
                'together'  => true,
                'joinType'  => 'INNER JOIN',
                'on'        => 'trackOpens.campaign_id = :opensUnopensCid',
               'condition' => 'trackOpens.id = (SELECT id FROM {{campaign_track_open}} WHERE campaign_id = :opensUnopensCid AND subscriber_id = t.subscriber_id ORDER BY id ASC LIMIT 1)',
                'params'    => array(':opensUnopensCid' => $this->option->regular_open_unopen_campaign_id),
            );
            return true;
        } elseif ($this->option->regular_open_unopen_action == CampaignOption::REGULAR_OPEN_UNOPEN_ACTION_UNOPEN) {
            $criteria->with['trackOpens'] = array(
                'select'    => false,
                'together'  => true,
                'joinType'  => 'LEFT OUTER JOIN',
                'on'        => 'trackOpens.campaign_id = :opensUnopensCid',
                'condition' => 'trackOpens.subscriber_id IS NULL',
                'params'    => array(':opensUnopensCid' => $this->option->regular_open_unopen_campaign_id),
            );
            return true;
        }
        return false;
    }

    return true;
}

and make it:

PHP:

protected function addRegularCriteria(CDbCriteria $criteria)
{
    // ADD THESE 2 LINES
    $criteria->addCondition('t.date_added <= :cdate');
    $criteria->params[':cdate'] = $this->date_added;
   
    if (!empty($this->option->regular_open_unopen_campaign_id) && !empty($this->option->regular_open_unopen_action)) {
        if ($this->option->regular_open_unopen_action == CampaignOption::REGULAR_OPEN_UNOPEN_ACTION_OPEN) {
            $criteria->with['trackOpens'] = array(
                'select'    => false,
                'together'  => true,
                'joinType'  => 'INNER JOIN',
                'on'        => 'trackOpens.campaign_id = :opensUnopensCid',
               'condition' => 'trackOpens.id = (SELECT id FROM {{campaign_track_open}} WHERE campaign_id = :opensUnopensCid AND subscriber_id = t.subscriber_id ORDER BY id ASC LIMIT 1)',
                'params'    => array(':opensUnopensCid' => $this->option->regular_open_unopen_campaign_id),
            );
            return true;
        } elseif ($this->option->regular_open_unopen_action == CampaignOption::REGULAR_OPEN_UNOPEN_ACTION_UNOPEN) {
            $criteria->with['trackOpens'] = array(
                'select'    => false,
                'together'  => true,
                'joinType'  => 'LEFT OUTER JOIN',
                'on'        => 'trackOpens.campaign_id = :opensUnopensCid',
                'condition' => 'trackOpens.subscriber_id IS NULL',
                'params'    => array(':opensUnopensCid' => $this->option->regular_open_unopen_campaign_id),
            );
            return true;
        }
        return false;
    }

    return true;
}

perhaps to consider as a (for some) meaningful option (as they could chose 'send to all subs that were subd at the beginning of the campaign' or 'send to all subs that were subd til end of the campaign' (then what about unsubs during the campaign, but perhaps that is not necessary)
what do you think @twisted1919

Lakjin · Nov 21, 2016

twisted1919 said:

@Lakjin - open apps/common/models/Campaign.php and look for:

PHP:

protected function addRegularCriteria(CDbCriteria $criteria)
{
    if (!empty($this->option->regular_open_unopen_campaign_id) && !empty($this->option->regular_open_unopen_action)) {
        if ($this->option->regular_open_unopen_action == CampaignOption::REGULAR_OPEN_UNOPEN_ACTION_OPEN) {
            $criteria->with['trackOpens'] = array(
                'select'    => false,
                'together'  => true,
                'joinType'  => 'INNER JOIN',
                'on'        => 'trackOpens.campaign_id = :opensUnopensCid',
                'condition' => 'trackOpens.id = (SELECT id FROM {{campaign_track_open}} WHERE campaign_id = :opensUnopensCid AND subscriber_id = t.subscriber_id ORDER BY id ASC LIMIT 1)',
                'params'    => array(':opensUnopensCid' => $this->option->regular_open_unopen_campaign_id),
            );
            return true;
        } elseif ($this->option->regular_open_unopen_action == CampaignOption::REGULAR_OPEN_UNOPEN_ACTION_UNOPEN) {
            $criteria->with['trackOpens'] = array(
                'select'    => false,
                'together'  => true,
                'joinType'  => 'LEFT OUTER JOIN',
                'on'        => 'trackOpens.campaign_id = :opensUnopensCid',
                'condition' => 'trackOpens.subscriber_id IS NULL',
                'params'    => array(':opensUnopensCid' => $this->option->regular_open_unopen_campaign_id),
            );
            return true;
        }
        return false;
    }

    return true;
}

and make it:

PHP:

protected function addRegularCriteria(CDbCriteria $criteria)
{
    // ADD THESE 2 LINES
    $criteria->addCondition('t.date_added <= :cdate');
    $criteria->params[':cdate'] = $this->date_added;
   
    if (!empty($this->option->regular_open_unopen_campaign_id) && !empty($this->option->regular_open_unopen_action)) {
        if ($this->option->regular_open_unopen_action == CampaignOption::REGULAR_OPEN_UNOPEN_ACTION_OPEN) {
            $criteria->with['trackOpens'] = array(
                'select'    => false,
                'together'  => true,
                'joinType'  => 'INNER JOIN',
                'on'        => 'trackOpens.campaign_id = :opensUnopensCid',
                'condition' => 'trackOpens.id = (SELECT id FROM {{campaign_track_open}} WHERE campaign_id = :opensUnopensCid AND subscriber_id = t.subscriber_id ORDER BY id ASC LIMIT 1)',
                'params'    => array(':opensUnopensCid' => $this->option->regular_open_unopen_campaign_id),
            );
            return true;
        } elseif ($this->option->regular_open_unopen_action == CampaignOption::REGULAR_OPEN_UNOPEN_ACTION_UNOPEN) {
            $criteria->with['trackOpens'] = array(
                'select'    => false,
                'together'  => true,
                'joinType'  => 'LEFT OUTER JOIN',
                'on'        => 'trackOpens.campaign_id = :opensUnopensCid',
                'condition' => 'trackOpens.subscriber_id IS NULL',
                'params'    => array(':opensUnopensCid' => $this->option->regular_open_unopen_campaign_id),
            );
            return true;
        }
        return false;
    }

    return true;
}

Thank you good sir. I tried figuring this out on my own but I wasnt even close.

Question. '$this->date_added' is the date / time the campaign was added? Can we do it so we use the date / time that the campaign was started? Because a campaign may be dated a few days before a campaign is actually started, and the campaign would then miss all those newer subs.

twisted1919 · Nov 22, 2016

@Lakjin - makes sense, this should do it:

Code:

$criteria->addCondition('t.date_added <= :cdate');
$criteria->params[':cdate'] = date('Y-m-d', strtotime($this->send_at));

twisted1919 · Nov 22, 2016

Sorry, it;s:

Code:

$criteria->addCondition('t.date_added <= :cdate');
$criteria->params[':cdate'] = $this->send_at;

frm.mwz · Nov 22, 2016

twisted1919 said:
Sorry, it;s:

Code:

$criteria->addCondition('t.date_added <= :cdate'); $criteria->params[':cdate'] = $this->send_at;

can you make this an option (or standard) in the app pls?

Lakjin · Nov 22, 2016

twisted1919 said:
Sorry, it;s:

Code:

$criteria->addCondition('t.date_added <= :cdate'); $criteria->params[':cdate'] = $this->send_at;

Thank you!

Lakjin · Nov 22, 2016

frm.mwz said:
can you make this an option (or standard) in the app pls?

This would be best, agreed.

twisted1919 · Nov 23, 2016

Already did

frm.mwz · Nov 23, 2016

twisted1919 said:
Already did

awesome! that's why we love you!

Is MailWizz double sending emails?

Lakjin

Active Member

Vpul Shah

Well-Known Member

twisted1919

Administrator

Lakjin

Active Member

frm.mwz

Well-Known Member

Lakjin

Active Member

Lakjin

Active Member

Lakjin

Active Member

twisted1919

Administrator

frm.mwz

Well-Known Member

Lakjin

Active Member

twisted1919

Administrator

twisted1919

Administrator

frm.mwz

Well-Known Member

Lakjin

Active Member

Lakjin

Active Member

twisted1919

Administrator

frm.mwz

Well-Known Member