Re: "large customer" cases

[Matti Aarnio <mea@nic.funet.fi>]

On Wed, Oct 11, 2000 at 11:54:05AM -0300, Mariano Absatz wrote:
> Hi,
> 
> We would need info about really large ISP's installation using ZMailer.
...
> The point is that a large customer of us is asking us to consider other 
> software (payware) with large marketing behind them (software.com, 
> critical path, etc).
...
> We are targeting large ISP's with virtual domain hosting (from a few 
> hundreds domains to tens of thousands domains and from tens of thousands 
> users to hundreds of thousends users).
> 
> We are interesting in the following info (but would settle for less, or 
> even different info).
> 
> Don't forget to tag the info that you want us to keep confidential. That 
> info will only be used in aggregate statistics so it can not be 
> individually inferred.
> 
> Name of provider: 
>         (provider is the organization providing mail services, it
>         could be you, your organization, or your customer)

		Sonera Corporation, several installations around
		the world. (Also using payware systems, and not
		always happy with them.)

> Category: 
>         (ISP / ASP / Internet Portal / etc.)

		ISP (and many other things)

> Target profile of provider's customers/users:
>         (i.e. what kind of customers/users does the provider have: 
>         individual home users, individual business users, small
>         businesses, medium business, large corporations, vertical
>         markets, etc.; or combinations of the above)

		Retail internet users, also small and medium business
		(it all really depends on account management system,
		 which is external to MTA/MS system, plus a bit of
		 service collection, e.g. is IMAP4 supported or not.)

> Number of virtual domains hosted:

		Over 30 000 per cluster.
		(Indeed no technical limit.)

> Number of mailboxes hosted:

		About 300 000 per message store machine.
		(CPU/filesystem IOPS limited depending on how frequently
		 users poll their mailbox, and how much they keep email
		 in the mailboxes.)
		(Less with UFS filesystem, more with VxFS filesystem.)

> Average number of messages/day:

		MRTG data suggests: 211/min average over 24h --> 300 000/d
		(per busiest system in the set.)

		Roughly one message per user per day.

> Peak number of messages/hour:

		500 msg/min -> 30 000/h
		(top load runs daily for 2-4 hours.)

> Total number of servers running ZMailer:
>         (either as inbound relay, outbound relay, local delivery, etc.)

		A  L4(-aware) switch  fronting services to the real
		backend servers.  Does load-balance on users, although
		if a storage service node is down, that subset of users
		is out.

		A crash-resistant high-available message store is
		very difficult, possibly impossible, in traditional
		UNIX filesystem based solutions.

		That is separate issue from Crash Surviving systems,
		which are apparently fairly easy in UNIXes.


		Having a crash (runaway overload usually) every 2-3 months
		might not necessarily be sufficient reason for spending
		$10-$30 per user on extra pricy hardware, plus software
		licenses.


> Describe succintly (if you can/wish) the server distribution:
>         (e.g: 
>                 2 sun 250 with 512M RAM / 36G HD for inbound relay and
>                         mailbox remote access
>                 2 sun netra t1 with 256M RAM / 18G HD for outbound relay
>         )


		A set of Sun Ultra2 dual-cpu boxes with SSA disks.
		512 MB RAM, VxFS filesystems on a stripped/mirrored
		diskset.
		(With current hardware models, propably E220/netra + A1000)


		$POSTOFFICE/ is some 4-6 GB per machine
		$LOGDIR/ is 8-20 GB per machine
		$MAILDIR/ is, say 1 MB * 300 000 users = 300 GB gross
			  (at $MAILDIR the more is the merrier..)


		The $MAILDIR/ subsystem is split into two layers of
		A-thru-Z subdirs giving 676 subdirs using double -X
		option at mailbox program for CRC32 hash of username.
		(That is the most even of various hash functions available.)


		The $POSTOFFICE/ has several of its dirs divided into one
		or two layers of subdir hashes.  Especially the scheduler
		is run with two -H options.


		Usage of the L4-aware switch allows easy separation of
		input SMTP servers from message store servers, although
		setups have also been run with SMTP input, output,
		redirection, pop3-proxy function, imap4 proxy function,
		and actual pop3/imap4 servers in same set of machines.
		Users were distributed among them with having multiple
		A-records for site specific  'mail.xxxx.com'  domain.


		user-visible pop3/imap4 proxies don't need ZMailer at
		their machines (if separate at all)


> Means of mailbox access (in order of usage):
>         (POP3, IMAP4, WebMail, telnet, etc.)

		POP3, IMAP4

		Webmail is an external entity accessing message-
		store via IMAP4

> Server software used for the following services:
>         local delivery:
		ZMailer

>         POP3:
		pop-proxy (see zmailer sources), plus seriously hacked
		UCDAVIS pop3 server.
		(Access only mailbox file per a: login username, b: hash
		 of admins choice -- use same as 'mailbox'.)

>         IMAP4:

		somewhat hacked UW-IMAP with similar features as pop3 server.

>         WebMail:
>         LDAP:
>         Whatever else that you think applies:
> 
> Add any info that you think is relevant:

		Proprietary set of libraries contacting backend
		databases via load-balance/failover arrangements
		in the libraries, plus external interface appearing
		as  'getpwnam()'.

		Add there 'ldap-like' interface to same backend DB
		to be used to map   user.name@virtual.dom.ain   to
		login_id@msgstore-node.dom.ain  for delivery to the
		appropriate message store node.


> We thank you in advance for any info you can provide.
> --
> Mariano Absatz
> mailto:baby@pert.com.ar
> PERT Consultores
> http://www.pert.com.ar

-- 
/Matti Aarnio	<mea@nic.funet.fi>