I. Tutorial

Chapter 1. Introduction to Email

1.1. Messaging

This chapter is quite different from the rest of this document. Here we build a foundation for understanding messaging, instead of focusing on how ZMailer behaves.

This chapter may feel a bit theoretical and abstract, being detached from practical life.

In reality, however, experience shows that most problems with messaging are a result of not understanding the underlying messaging model, or of not respecting said model.

The terminology used here may seem a bit X.400 oriented. It is, because folks from what was then known as CCITT (now known as ITU-T) adapted the model originally developed by IFIP. Of course, CCITT added a lot of things of its own invention (like ADMDs and PRMDs), that we don't need to bother ourselves with.

Although the terminology comes from X.400, it is in no way restricted to it. Our presentation here is a generic messaging presentation not restricted to any type of protocol.

Messaging, as the name says, is all about exchanging messages, short (or sometimes long) pieces of information. Messaging is always directional (which means that there is always a sender and one or more recipients), targeted (the list of recipients is fixed) and store-and-forward based.

There are a few messaging-like applications in which the message is broadcast to a wide, unspecified audience. A prime example of this latter application is Usenet News. News is not messaging, as it is not targeted.

So what about mailing lists that are linked to News, are they messaging or not? As long as the message is transported as a mail message, it is messaging. One of the recipients of the message may well be a Usenet News newsgroup. Similarly, a sender of a message might be the News system, or the author who initiated the submission by using News. Messaging is not necessarily interpersonal.

It is also quite normal for different applications to communicate by using messaging methods. A prime example of this would be EDI traffic. It is clearly messaging, but not interpersonal.

1.1.1. The Messaging Model

In addition to users, the basic building blocks of messaging are User Agents (UAs) and Message Submission Agent (MSA) is newer term for specific sub-task of Internet email, namely authenticated message submission to first MTA of MTS system. Message Transfer Agents (MTAs). User agents are the interface through which a human user interacts with the messaging system. On non-interpersonal messaging user agents may be built directly into applications. MTAs are used to transport messages from one computer system to another. An example of a good MTA is ZMailer. Access Units (AUs) can be used for accessing telematic services, for example telefax. (Or in general act on user behalf somehow, e.g. automated scripts.) Message Stores (MSs) can be used between MTAs and UAs. They are used for storing messages before and while UAs are used to access and manipulate them. Message Delivery Agent (MDA) is newer term intended to cover specific sub-task of moving the message from MTA system to the care of MS, UA or AU. Some new MTA suites even do parts of the UA functionality (.forward -processing) in the MDA. Gateways (GWs) are used in between two different types of messaging systems, for example between SMTP and X.400 mail. Gateways are conceptually made of two MTAs and a UA that interconnects them.

There are two more acronyms worth looking at in the messaging model, namely MTS (Message Transport System) and MHS (Message Handling System). MTS is the collection of MTAs (and GWs and MSAs and MDAs), while MHS includes MTS and UA functionality (UAs, MSs and AUs).

All in all, X.400 terms are not a complete match on how things are done in Internet email, nor should they be considered as normative, merely giving you a reasonable frame of reference that isn't very wrong.

A graphical example of the messaging model is shown in figure Figure 1-1. It shows the relationships between different elements of the model.

Figure 1-1. A graphical example of the messaging model.

As can be seen, a user may use more than one UA, and a UA can be connected to more than one MTA.

Although it is important to understand the relationships between different entities in the messaging model, it is even more important to understand the nature of a message and the way UAs and MTAs deal with it.

A message consists of a body and headers. In case of messages with more than one body-part (for example some MIME messages) the different body parts are all part of the outermost body-part.

Figure 1-2. How a message looks normally.

    From:    The Manager
    To:      One Bright Employee
    CC:      secretary
    Subject: Salary raise
    Date:    17 May 1997

    Dear Employee,
    The Manager

Figure 1-3. A possible, more complex message structure.


As can be seen, there is always just one outermost body that contains all other body-parts. In some cases, for example X.400 the protocol seems to violate this by leaving out the outermost body-part. However, even on those cases we must assume, at the abstract level, that the outermost body-part is there.

However, this is not all that there is to the structure of a message. When a message is in transit, being handled by MTAs it is put inside an envelope, just like a normal letter is inside an envelope while the postal service is carrying it. Just like the postal service is not permitted to look inside the envelope, neither are MTAs permitted to look inside. Whenever there is a need to look inside the envelope, it is always a UA function, and done on behalf, and on the authority of, a UA.

There are some violations of this. When using the SMTP protocol, the Received: lines are put in the headers by MTAs. This is bad engineering, but as the process of adding a new header line is fairly straightforward, it doesn't cause too much pain. In some cases MTAs, and especially the GWs modify the header even more, and sometimes they even mess with the body. This is a sure recipe for trouble.

Graphically, the way a message should be dealt with is shown in figure Figure 1-4.

Figure 1-4. How a message should be handled.

The user creates the message with the help of a UA. How the interaction is arranged is a local matter. Once the message has been prepared, it is passed to a nearby MTA together with necessary envelope information and put into an envelope. The MTA puts its stamp on the envelope to show that it has received the message. The first MTA passes the message to the second MTA. The second MTA puts its stamp to the message and passes it to the third MTA, and so on. The final MTA passes the message to a UA, and the envelope is removed.

There are at least three ways to pass the message from an MTA to a UA. The message may be pushed to a (running) UA, a UA may pull it from an MTA, or an MTA may pass the message to an MS from which a UA will receive it at a convenient time.

The normal UNIX way of delivering mail (/usr/spool/mail/user) can be seen as any of the above three mechanisms, but should normally be seen as a UA pulling a message. The reason for this confusion is with the de-facto SMTP standard MTA, Sendmail.

Although Sendmail is in many ways a very clever piece of software, it is also the reason for many problems, as it has blurred the line between MTA and UA. Sendmail is clearly an MTA, but it also performs many of the UA level functions, like handling of .forward files. This behaviour has become the de-facto standard way for UNIX MTAs to behave, making it necessary for any MTA, including ZMailer, to behave the same way.

1.1.2. Routing And Delivering Messages

As MTAs only deal with envelopes, all routing and delivery decisions have to be based on information available on the envelope. It follows from this that the envelope and headers may contain conflicting information. This is normal, and is not a cause for worry.

MTAs may, and often do modify addresses present in the envelope. This might include changing addresses to a format more suitable for mail delivery and alias expansion.

It is important to make a distinction between aliasing and forwarding mail. Aliasing is an MTA function, in which an MTA effectively knows that to reach a seemingly local user, mail should be sent to a different address. To accomplish this, the MTA changes the recipient information on the envelope. Forwarding is a UA function. When forwarding, the mail message is received by the original, intended recipient, and re-sent to another address. Although forwarding is a UA function, it doesn't have to result in a change to body or headers, but on the envelope, both sender and recipient should be changed. Sendmail bluntly violates this, and makes most other MTAs violate it as well.

Most mailing lists today are just alias expansions, on which one recipient address on the envelope is replaced with multiple addresses. In many cases this is a reasonable approach. However, all major mailing lists should be set up as a UA function. This involves changing not only the recipient address but also the sender address in the envelope. In this way, undeliverable messages are sent to the owner of the list, who can deal with the problem, and not to the sender of the message, who can do nothing to remedy the situation. (The ZMailer has some built-in facilities for this, see about Mailing Lists… on section Section 13.2.2.)

Error messages must be sent to the envelope sender address, as this is a MTA function.

Replies to messages should be sent to header addresses, because replying is a UA function.

Chapter 2. ZMailer's Features, and Facilities

2.1. Introduction

ZMailer is a mailer subsystem for the UNIX operating systems. It is in charge of handling all mail messages that are created on a system, from their creation until final disposition locally or by transfer to another system.

As such, the mailer subsystem (the Message Transfer Agent) must interface to local mail reading and composing programs (User Agents), to the various transport methods that can be used to reach other mailers, and to a variety of databases describing the mailer's environment.

ZMailer provides this functionality in a package and with a philosophy that has benefitted from experiences with earlier mailers.

ZMailer provides a capable, robust, efficient subsystem to do the job, which will excel in demanding environments, but is technologically simple enough to fit easily everywhere. (In UNIX-like environments.)

However ZMailer is not smallest memory footprint MTA there is, nor it tries to be. What it has and does, are ways to limit resource expenditure, while still providing high-performance services. One can't very easily overwhelm a machine where ZMailer MTA runs by just feeding it too much work in parallel. (Message Queues can grow beyond any reason, but even there are safety limiters.) Limits are available for example:

  • Number of parallel incoming SMTP sessions overall

  • Number of parallel incoming SMTP sessions from any given single IP address

  • Number of messages per source IP address in timeframe

  • Number of parallel internal processing programs

  • Number of parallel SMTP sessions feeding messages outside

For ZMailer's incoming SMTP interaction there are ways to define, that the usual anonymous user from given address space can send only so many messages with only so many recipients per time interval. Such limits help keeping Zombie botnet Windows machines from causing too much trouble. Technology aimed for keeping such service abusers under control is ever evolving, and can probably never be 100% capable. Also the abusers do learn rather quickly what is bad in their behaviour, and they do modify their programs to get past any filters thrown at them. At the same time, legitimate users are hadly ever evolving their behaviour, and are always behaving rather foolishly, or rather their used UA softwares behaves more like hijacked Zombies, or perhaps there is just a NAT box, and several users beyond it...

In the longer run, service providers will need bigger and bigger servers (or clusters of smaller servers) to make their inbound and outbound SMTP services. They will also need ways to track user behaviour online, and if necessary, modify reaction algorithms in timescales of few hours. Further complication is, that the service providers need different behaviour models according to "this is Consumer" vs. "this is Corporate" customer. Corporate clients usually have some MTA of their own, and thus their legitimate network behaviour is a lot more like what spammers do these days, and separating spammer hijacked system there is really difficult.

2.1.1. Design Summary

Figure 2-1. ZMailer's processes.

ZMailer is a multi-process mailer, using three daemon processes to manipulate messages. Used technologies are as simple as possible, e.g. while networking stuff is as advanced as possible (with fallbacks to simplest basic behaviour), some of other modern things (like threads) are not used in favor of simpler approaches

  • Message arrive in via sendmail program "API" for internally originated messages, or via Smtpserver subsystem, which is system front-door for messages coming in from the external network. The Smtpserver is in reality a cluster of auxiliary programs providing efficient low-overhead support for various analysis things needed while messages are coming in. Second one of these processes is a Router, and makes all decisions about what should happen to a message; routing and possibly message header visible things rewriting. The third daemon is a message queue manager, Scheduler, used to schedule the delivery of messages. The Router uses a configuration file that closely follows Bourne shell script syntax and semantics, with minimal magic. Message files are moved around in a series of directories, and the Scheduler and its Transport Agents run off of control files created by the Router.

  • The Sendmail is very simple "plug-compatible" message submission agent for system internal message submissions into the ZMailer MTA, and does all its things without any sort of set-uid privilege escalation needs.

  • The Smtpserver has evolved into rather complicated animal, as it aims to really efficiently support things that in the early days required truly heavy-weight auxiliary program startups for every incoming SMTP connection -- startup of such auxiliaries has now been shared over a large number of arriving connection and messages by means of having them as Smtpserver's permant auxiliary helpers:

    • First is a slightly less burdened Router for determining if source or destination address domains are possibly known in the system, and primarily being able to reject messages at front door that are destined to nonexistent addresses. (This instance is separate from main Router, and thus is somewhat duplicating main Router's task, but this doesn't e.g. do list expansions and other such expensive things.)

    • There is also Content-Filter that can be used up to how ever complicated message content analysis syncronously with incoming message feed (there is also a possible "input" subdirectory for offline non-synchronous content analysis, however synchronous processing has certain appeal in itself, not the least the ability to tell message sender to go and stuff their SPAMs into...)

    • And third is the Rate-Tracker subsystem, that can keep track of such things as "Non-authenticated customer at IP address N.N.N.N has sent more than 60 messages in past hour, stop that sending until the sliding window allows more to be sent."

    These auxiliary servers do operate so that when smtpserver-subsystem shutdown has been ordered, and last client needing support goes away, they drop away themselves.

  • The main Router subsystem will process messages one at a time (per Router instance), as it finds them in a directory where User Agents submit their outgoing messages. Message Envelope and Message Header information is all kept in the same message file along with the message body, and this file is never modified by any ZMailer program. After parsing the envelope and RFC822 header information, the Router validates the information extracted, and calls functions defined in the configuration file to decide exactly, how to deliver the message and how to transform the embedded addresses. The algorithms that do this are easily re-configurable, since the control flow and address manipulation is specified by familiar(ish) shell script statements. When the Router is finished on a message, it will produce a message control file for use by the delivery processing stage of ZMailer, and move the original message file to another location.

  • Once the main Router subsystem has decided what to do with each of the addresses in a message, the Scheduler builds a summary of this information by reading the control file created by the Router. This knowledge is merged with a data structure it maintains that stores which messages are supposed to be sent where, and how. According to a pre-arranged agenda, the Scheduler will execute delivery programs to properly move the message envelope, header, and body, to the immediate destination. These delivery programs are called Transport Agents, and communicate with the Scheduler using a simple protocol that tells them which messages to process and returns status reports to the Scheduler. The Scheduler also manages status reports, taking appropriate action on delivery errors and when all delivery instructions for a message have been processed.

  • There are several standard Transport Agents included with the ZMailer distribution. The collection currently includes a local delivery program, an SMTP client implementation, and a Transport Agent that can run Sendmail-M-line-compatible delivery programs.

  • A separate mailq utility allows querying the Scheduler for the state of its queues. For existing Sendmail installations, a replacement program is included that simulates most of the Sendmail functionality in the ZMailer environment. This allows ZMailer to replace a Sendmail installation without requiring changes in standard User Agents.

  • Several other tools and utilities exist for other specific purposes.

  • There are also facilities that allow loosely coupled cluster creation. Each member machine of the cluster is separate entity, but they can keep track of rate-tracking data of all of their neighbours, as well as activate on demand ETRN queue flush in cluster wide setup. None of these is guaranteed in any strict way, which makes their implementation technology considerably simpler, and also occasional limits that they implement are not considered to be upheld rigorously under every possible conditions. A tripple-sigma reliability on e.g. limit enforcement is considered "a plenty good enough".

    In loosely coupled clusters, delivery to system internal (UNIX-style) message store is not clusterized, unless the store in question is some sort of e.g. LMTP connected external Message Store. (Even UNIX-style mailbox files may work, if underneath there is some sort of cluster-wide filesystem, which has working FCNTL locks.) One such way was once to use front-line processing to map arrived message to actual recipient and node into which it was expected to go, and to proxy POP and IMAP services so that users didn't need to know on which node their mailboxes did actually reside.

Figure 2-2. Directories that ZMailer uses for message processing.

2.2. Running ZMailer

ZMailer is fairly simple to run, once the setups are completed it can be left to run on its own with very little supervision.

Things that might need supervision are things like:

  • Timely cycling of log files, which otherwise will grow until they fill all of the available disk space (One need not log everything possible, about the only thing this system does not allow you to log is the message body content.)

  • Keeping watchful eye on $POSTOFFICE/freezer/, and $POSTOFFICE/postman/ directories. Former for processing SPAM email, latter for pathological problem cases.

    “Logging and Statistic for the Administrator:” Chapter 16, “Checking the Log Files:” Section 4.6, “Trim-down of Logging:” Section 4.8, “Postmaster Analysis Area:” Section 11.4.

We look closer into these issues at latter parts of this document, but now it is sufficient to tell, that the principal tool for active monitoring of the system health is command:

$ mailq -ss
which does tell, if router, or scheduler are up and about, or not, and also does tell about the sizes of the different sub-spools.

The general management interface for starting and stopping different subsystems is command

# zmailer
which the system installs into $MAILBIN/ directory, and which command usually needs a symlink to itself from some more common location for administrative convenience ( /usr/sbin/zmailer -> $MAILBIN/zmailer ) so that the administrator does not need to add $MAILBIN/ directory into his or her PATH. On overall, it is intention that not even admin user should need to run directly the programs located at the $MAILBIN/ directory.

Basically the administration is as follows:

  • At system start-up (to start all subsystems):

    # zmailer

  • At system shutdown (to kill all subsystems):

    # zmailer kill

There is also a way to make sure the system will not let the ZMailer to start at the system start-up, because you have some massive work going on, and the system is not in condition to accept email for a while:

# zmailer freeze
and the antidote for the “freeze” is, naturally:
# zmailer thaw

Normal operations can not be started at “frozen” system without “thawing” it at first.

The user-visible component of the ZMailer is (for de-facto interface)

$ /usr/lib/sendmail
(a.k.a. /usr/sbin/sendmail) which is “simple” message submission program that mimics sendmail commands behaviour, but of course many details of sendmail are not really implemented at all, mostly because they do not have equivalents in the ZMailer system.

There are also functional equivalents (or near equivalents) of other sendmail/system utilities: mailq, newaliases, and vacation.

2.3. Factors Affecting Overall System Performance

  • Speed of $POSTOFFICE/ directory filesystem. Specifically directory metadata operation speed (e.g. fully synchronous directory metadata update is way slower, and safer, than fully asynchronous.)

  • Possible separate filesystem spindle on $POSTOFFICE/transport// directory.

  • Amount of memory, and thus filesystem buffering, and (lack of) swapping by component processes.

  • DNS server quality

  • “Staticness” of routing data, e.g. the less there is DNS lookups involved for email delivery, the better it works.