Re: [ZMailer] Reorder Queue/Channel, Delete Single Message

[Matti Aarnio <mea@nic.funet.fi>]

On Tue, May 18, 2010 at 08:40:19AM +0100, Darryl Miles wrote:
> >It depends.
> >How much traffic supports mail server?
> >
> >Exim is more user friendly regards some fast config changes (like queues) 
> >and has more users (more sites, more help).
> >Exim has more tools in most of standard linux distributions.
> >Exim works not on root id.

For most parts ZMailer works fine on non-root id.
Only part needing root is when smtpserver is binding on port 25 et.al.
and the scheduler on the mailq.
(Even those can be hacked in platform dependent ways - like SE Linux
supplying special privileges to smtpserver binary to bind to allocated
ports without root capability.)


> >But exim is not so fast.
> 
> Those that do use Exim (or anything else for that matter) do they have:
>  * A top 10 list of the features most in demand.
>  * A top 10 gripe list of things Zmailer does badly.
> 
> Also what platforms do you care about ?
> 
> One factor with Zmailer in order to make progress forward is that it 
> needs to expunge all circa 1990-2002s cruft.  These days there isn't 
> that must diversity between platforms, the inefficiencies/bugs of 
> platform provided libraries isn't so significant as CPUs have become 
> faster, etc...

The IO to persistent storage is always "slow".
Present ZMailer's queue file handling is reasonably fast by minimizing
operations that generate physical IO, and if really necessary, one can
spread the postoffice around multiple spindles.  However these days
you probably pick an SSD for a postoffice spool disk instead of old
style spinning electromechanical devices.  Mailbox storage is entirely
different animal.

The modern ultra-fast CPUs make physical IO painfully obvious.

An extreme example of what the email load is being served by ZMailer
can be seen here:

  http://vger.kernel.org/z/

However its loading is radically different from your average mailbox server.
It is email-list system installation - one message in, 1000+ out.


> The major platforms I see today are Linux/xBSD/Solaris.  Who runs a mail 
> service on anything else and why (i.e. price/performance/maintenance) ? 
>  I'm sure people do but I'm also sure it becomes hard to justify 
> continuing to do so as the years roll by.

The GNU Autoconfig framework helps a lot on keeping around the support for
less used platforms.  Of course platforms without testing are prone to get
affected by bitrot..


> Matti has done a fine job of being gatekeeper for zmailer over the past 
> decade it would be kind of nice to hear his point of view on development 
> and usage also.

Queue scheduler management interface is missing.
Mainly a management _tool_ is missing - kick this message from that queue,
start that queue ("SMTP ETRN" hack exists, though.) etc.

Improving the hacky interface layer at smtpserver to run message content
scanners needs some work.

Improving the documentation.

The default queue modelling is also a bit difficult at times.  Namely sticking
on arrival order can result in stuck queue, as only first one is tried.
Original model was to randomize the queue order before delivery attempts, but
some sort of hybride might be better - prefer arrival order, if a retry is
ordered then put the troubled message on tail of the queue and try new queue
leader, but do not try too many... ( 500 messages, stuck because of 'connection
timed out', no point in retrying them all ad-infinitum.  Give the transport
agent channel to some other target thread for a while.)

Also these days I might brave the universe, and declare pthreads as a fundamental
resource available everywhere.  It would result in a radically different software
with tons of spinlocks to protect the queue data models.  That would affect
scheduler's construction.

At work I usually do Java programs using existing java.util.* classes to
do most of my heavy lifting with data structures.   (I use also Perl, and
ANSI-C, case depending..)

Rewriting entire system in Java is out of the question.  Rewriting major
parts of it even in C++ is very great challenge, and I don't have that much
free time.  I do not see that I would get immediate maintenance benefits
and/or improved features from doing such a thing.

Furthermore, I don't do email for a job anymore.  Just for a hobby.

> Darryl

-- 
/Matti Aarnio	<mea@nic.funet.fi>
--
To unsubscribe from this list: send the line "unsubscribe zmailer" in
the body of a message to majordomo@nic.funet.fi