Re: Need advice on large mailing lists

[Alex Nishri <nishri@utcc.utoronto.ca>]

> > I am setting up a very large mailing list on a Linux(2.0.0) machine
> > and I am looking for any advice on what the best way to do this might
> > be.  The list has approx 50,000 recipients and will be used once
> > a month to send out a newsletter.  The current solution uses 
> > majordomo and sendmail.  When a mail is sent out, it kills the machine
> > (to put it lightly) for a few days which is not accpetable.
> 
> 	For that rare posting I suggest running some batch-sending
> 	script which picks circa 100 addresses a time from the list,
> 	and sends them in one message, then next 100, and so on.

We use bulk_mailer (reference ftp://ftp.greatcircle.com/pub/majordomo/FAQ).
Bulk_mailer sorts the addresses belonging to the list by reversed domain and
sorts them into groups containing no more than N domains; we trivially
modified it to also limit the number of recipients. So with 50,000 recipients
we actually hand Zmailer 1,000 separate messages with 50 recipients each.
(This is called chunking.) Our /local/share/mail/db/aliases contains a line
like:

list-outgoing: "|bulk_mailer owner-address /local/file.with.list.of.addresses"

You may also want to keep mailing list explosions in a separate
/var/spool/postoffice to prevent delaying your non-mailing list email.

Then there is L-Soft's LSMTP (http://www.lsoft.com), a commercial SMTP
product especially designed for sending to large mailing lists. L-Soft claims
100,000 deliveries an hour running on a PC. I don't have experience with it,
but I understand that one key difference between Zmailer/sendmail and LSMTP
is that the latter will accept a single message with 50,000 recipients (ie no
chunking) and work on the addresses concurrently. At bottlenecks like DNS
resolution (which takes a long time for a signficant number of addresses on a
large international mailing list) LSMTP can have many more concurrent DNS
lookups outstanding. To gain that sort of concurrency with Zmailer you would
have to break up the 50,000 recipients into 50,000 messages and run a huge
number of routers and smtp delivery channels--but then the overhead of
dealing with 50,000 messages would be a factor.

Alex Nishri
University of Toronto