Re: Need advice on large mailing lists

[Matti Aarnio <mea@nic.funet.fi>]

> I am setting up a very large mailing list on a Linux(2.0.0) machine
> and I am looking for any advice on what the best way to do this might
> be.  The list has approx 50,000 recipients and will be used once
> a month to send out a newsletter.  The current solution uses 
> majordomo and sendmail.  When a mail is sent out, it kills the machine
> (to put it lightly) for a few days which is not accpetable.

	For that rare posting I suggest running some batch-sending
	script which picks circa 100 addresses a time from the list,
	and sends them in one message, then next 100, and so on.

	Sending 50 000 individual letters is also an option, but I
	don't suggest it -- though it would not kill ZMailer, while
	sendmail (and system) would die a horrible death..

	Right now the ZMailer is not very good at expanding lists with
	many thousands of addresses. The router processes producing
	route analysis for such a bulks will easily grow into humongous
	sizes (tens of megabytes.  It used to grow in N^2 manner, but
	I was fortunately able to contain that a bit..)

	A datapoint on that process growth:  VGER.RUTGERS.EDU has its
	router processes grown to about 10 MB, which happens when it
	expands largest list it has: 1830 addresses.  The growth is
	fairly linear, and the zero size is about 4 MB.  With circa
	3.3kB/recipient it gives circa 170 MB router process size,
	if it has to expand the list of 50 000 addresses in core...
	(The router process does not shrink, because the malloc()'s
	 free() does not deallocate memory pages..)

> The solutions I am looking at include replacing sendmail with zmailer
> and useing ProcList (free versions) or SmartList.  We want people to be
> able to subscribe and unsubscribe to the list (we will probably also
> develop some type of web interface for this).

	Yes, handling subscriptions, and doing postings can be done
	entirely separately from postings, and I suggest it warmly.

	Usually postings are done in a sort of list-expand operations,
	however that too easily kills your mailers...

	However, nothing prevents you from running the posting script
	on a pipe thru your mailer -- you have some alias, which expands
	to a pipe running proper scripts, and takeing the incoming
	message into some temporary storage, and then sending copies
	of that.


> Any thoughts, bits of wisdom, good/bad experiences would be 
> appreciated.


	In the ZMailer source package there is an example of a posting
	script written in PERL  (utils/lightpost.pl), which posts INDIVIDUAL
	messages.  You can follow its lead, and write a posting program
	that posts the message in bundles of a few-dozen-to-few-hundred
	recipient addresses a time.

	The purpose of that sample utility is to show HOW to do it in your
	own code, and while it is a "mostly functional" replacement for
	the /usr/lib/sendmail, it is not intended to be run for each
	recipient (It being PERL should make that obvious :) )

	If possible, have your posting program to sort the recipient
	email domains so that (if possible) you pool recipients to some
	target system to be submitted at the same time.  In the SMTP
	transport all those recipients will be delivered on one session
	with SINGLE message block associated with all of the addresses.

> thanks
> andrew
> .....................................................................
> Andrew Wyllie
> wyllie@realbeer.com

	/Matti Aarnio <mea@nic.funet.fi>