Where is the regex blacklist stored

droidman

droidman

Member
Hi, i can't find where the Blacklist (the one in regex) is stored.
I also wanted to know how big can it be.
i have a regex list with more than 600k suppressed domains and i added like 200k and since the page takes time to load i searched for it on the database and could only find the email_blacklist table, not the other one with the regular expressions.
is it in a file ?

thanks
 
@droidman - It's in the mw_option table. 200k is huge, it hasn't been designed for such high usage, but rather in the order of "a few records". For blacklisting entire domains we don't have good solutions right now since we're more concerned in blacklisted emails not domains.
 
yes i know that it is very big and heavy but this list blocks the ability for most of the users to hit common traps like hitamail.com gmil.com and all of those typos that might get into the system. i know that there are companies out there for data validation but that was my last line of defense that i used in Interspire. Big but works.
I'm going to try to figure out some different angles to work this out.
Thanks
 
droidman said:
I'm going to try to figure out some different angles to work this out.
Click to expand...
If you know your way with PHP you could hook into the app when the check against the blacklist is made and compare the email against another database or against a file of emails and take an action against that.
PHP: 
Yii::app()->hooks->addFilter('email_blacklist_is_email_blacklisted', function ($isBlacklisted, $email, ListSubscriber $subscriber = null, Customer $customer = null, array $params = array()) {
    // if already blacklisted we stop
    if ($isBlacklisted !== false) {
        return $isBlacklisted;
    }

    // without customer we stop
    if (empty($customer)) {
        return $isBlacklisted;
    }

    if ( /* whatever code here to check if $email is blacklisted decides the email should be blacklisted */ ) {
        return $isBlacklisted = true;
    }

    return $isBlacklisted = false;
});
 
im not familiar with the app yet and my php skill is very limited but i'm probably going to invest some time into this.
maybe someone wants to make an addon for that :D
 
@droidman - so basically you have a txt file with one regex per line, i.e:
Code: 
[...]
/(.*)@0-grvity\.co\.uk/i
/(.*)@0g\.com/i
/(.*)@0gvhptvmch\.1c\.pf/i
/(.*)@0gyzd95w0s\.nl/i
[...]

Create a file called init-custom.php in /apps/ folder ( you already have a init.php file there ) with following contents:
PHP: 
<?php

Yii::app()->hooks->addFilter('email_blacklist_is_email_blacklisted', function ($isBlacklisted, $email, ListSubscriber $subscriber = null, Customer $customer = null, array $params = array()) {
    // if already blacklisted we stop
    if ($isBlacklisted !== false) {
        return $isBlacklisted;
    }

    // without customer we stop
    if (empty($customer)) {
        return $isBlacklisted;
    }

    $filePath = Yii::getPathOfAlias('common.data') . '/suppression-regexes.txt';
    foreach(new SplFileObject($filePath) as $regex) {
        if (!preg_match($regex, $email)) {
            continue;
        }
        return $isBlacklisted = true; 
    }       
    return $isBlacklisted = false;
});

If you look in the code, you will see it looks for a file called suppression-regexes.txt which should be located in apps/common/data folder. so make sure you name your file like so and you upload it there.

keep in mind that your regexes should be correct otherwise errors will show-up.
Also please note that going over 600k regexes might not be fast.
 
Wow that was fast.
Your support is amazing.
So i have placed all the files on the correct folders, renamed the regex and uploaded it and changed chmod and chown to match the rest of the files.
So, when does this run ? when i import data?
also, if you make another mailwizz update will this work ?
thanks !!!!! :)
 
i have created all those files and imported a list under a dummy client account and imported only blacklisted domains and all went in so i think that something did not work. any way for me to debug this ? or do i need to configure this init-custom.php file anywhere on the admin pannel ?
 
Can you remove this part from the code and try again:
Code: 
// without customer we stop
    if (empty($customer)) {
        return $isBlacklisted;
    }
 
Ok i did that and nothing changed
it said this:
From a total of 4 subscribers, so far 4 have been processed, 4 successfully and 0 with errors. 100% completed.
2 of them had to fail
 
This is something I would absolutely love to have. Any chance you can share that list with me (or everyone for that matter.)

@twisted1919 this could make a good extension: “Process large regex block list”
 
@Gregg - the implementation is really simple as you can see above, so it wouldn't really make an extension as it's just loading and looping through a file, simple stuff :P
 
You must log in or register to reply here.
Back
Top