Scheduling algorithm?

[Marek Kowal <marek.kowal@portal.onet.pl>]

Dear all,
 
Recently we observed some problems with the delivery of the messages
directed to the SMTP channel. The problem appears on the machines that have
about 100 000+ messages in the queue (on the scheduler side in smtp/*), but
actually appears to affect also systems with smaller numbers of messages in
queues. It turns out, that - probably - the problem is also linked to the
large number of threads with destinations that time out, not only to the
large number of messages waiting to be delivered. 
 
Anyway, as a result, some customers complain that their messages have been
kept on the server for a very long time before being pushed out by the
system to the outside world. And THEIR destinations are responding well, it
is just scheduler that schedules the thread to be processed not often
enough. The systems in question process about 20-40 mails per second,
depending on the daytime.
 
Now, we managed to speed the delivery to specific destinations by creation
of separate thread groups in scheduler.conf. This helped things a little -
for those channels, which are explicitly mentioned in configuration. But the
others still suffer. And I cannot afford mantaining few hundred entries in
scheduler.conf - just the maintanance would take ages. At the moment, I
observe on our system about 7-10 k threads. The number of TAs is set to 800.
Increasing the number of threads probably will help, but the machine limits
will be hit soon, if I'll continue in this direction.
 
So, in order to really solve the problem, I need to know the exact
scheduling algorithm and understand the real reason of the delays. But this
is not very well described in the docs and leaves a lot of room for
questions. The only sentence that gives some hint about the real internals
of the scheduler is:
 
"Also in normal cases, when messages in given thread are all delivered, or
otherwise determined that nothing can be done, the transport agent can be
switched to another thread within the same thread group."
 
So, let us consider our setup. For simplicity, let us assume that all
threads are in one thread group smtp/*, and that the MaxTHR is 1. The
scheduler probably assigns the 800 TAs to the first 800 threads. Now, in
accordance with the sentence from the docs the TAs will consume the messages
until all are sent out, and then switch to the next 800 threads. And so on.
Is that correct? (this is first problem: how are threads within thread group
ordered, when deciding whom to process next)? What if one of the threads
contains 10k messages? Will it attempt deliver all 10k at one go? How does
scheduler treat the messages, that appeared in actually processed thread
while it was already being processed? Are they sent out in the same
iteration, or are they left for the next iteration?
 
Also, will scheduler return to the first thread only after all other 7k-799
threads have been scanned (sort of FIFO?). That would explain, why I am
seeing such delays - it probably takes several minutes to scan the whole set
of threads and return to the first one.
 
Also, under such heavy loads, what is (if there is any) the meaning of the
interval configuration parameter? Also, what happens if the destination of
the thread is not responding, and thread has many messages waiting, and new
appear at constant rate? Will scheduler give up the whole thread after few
first tries on first few messages and switch to the next thread, or will it
keep on trying each of the messages in the queue in accordance with
retries(....) specification? If so, when will it give up and for how long?
This is partly covered in the docs
http://zmailer.org/zman/zadm-scheduler.shtml, but it is not clear to me. If
4000 of the destinations are not responding (this is what basically hapens,
they usually time out, and as a result more than 100k messages cannot be
delivered and after few days they will be returned to the senders), what
will the scheduler do? Retry each message?
 
And now two questions related to such setups:
 
1. how do you handle such heavy loads? I mean, some of you must have
experienced the same problems. 
 
2. the startup phase. it takes about one hour to scan the control files
after the restart of the scheduler - it opens all those control files and
there are lots of them. any ideas how to speed things up? (apart from buying
better hardware, that is ;-)))
 
Cheers,
Marek
 
-
To unsubscribe from this list: send the line "unsubscribe zmailer" in
the body of a message to majordomo@nic.funet.fi