Import Subscriber is slow

[chrisj383]

Hello Forum,

How to get the import faster? I now already have a 21gb SQL with about 50M subscribers.

It takes about 4 days now to import 1M subs :'(

Use 32gb RAM and 5 delivery server.

 

[twisted1919]

So you're doing SQL import, from another database? Where is the other database located?
For CSV/TXT files, try to enable the command line import, see /backend/index.php/settings/import-export and click the info icon for more info.

 

[chrisj383]

No i import TXT or CSV. At the beginning I did 500k lists in a few minutes. Now 100k txt files take like 1 hour.

I use browser import. Because I'm a Linux n00b

 

[twisted1919]

I see, try the command line import in this case, it might help in this case.

 

[twisted1919]

Also, please make sure you're not having any list cleaning extension enabled to verify subscribers at import, that might slow down things a lot.

 

[Gregg]

Also, please make sure you're not having any list cleaning extension enabled to verify subscribers at import, that might slow down things a lot.

What if you need to clean your lists though? In some other tools they give you a way to verify your list, comes up with a progress bar and goes through. I use TheChecker.co for that. I was on that other tool, but it had so many problems that I moved to Mailwizz.

My import is going extremely slow. 1000 takes 5 minutes.

I see parallel processing for other things, is there no parallel processing for importing?

 

[twisted1919]

@Gregg - Importing when a checking extension is enabled will always be slow, no matter what you do, and this is because you need to make network calls to the checking service and these might take a few ms or a few seconds, depends on the service.
This is why my advice is to clean your list first, the entire list, then import it in mailwizz and use the checking extension only for the subscribe forms.

 

[Gregg]

That all makes sense. I am just wondering about some sort of parallel processing for large imports.

If there is no way to do that automatically. What if I just split the list up into 10 separate smaller lists, open 10 tabs and do the import 10 times at once? Can Mailwizz handle that?

 

[twisted1919]

What if I just split the list up into 10 separate smaller lists, open 10 tabs and do the import 10 times at once? Can Mailwizz handle that?

Yup, you can try it like so.

 

[Gregg]

Perfect. Off to the command line I go. If it wasn’t for split, shuf, grep, awk and sed I don’t know how I would get along.

 

[twisted1919]

If it wasn’t for split, shuf, grep, awk and sed I don’t know how I would get along.

I hear you. Though mailwizz does offer commands to help you with the import process, see my other reply