Re: No MIME

[Dave Rhodes <Barry.Bouwsma@tuke.sk>]

[    This message is being posted in discussion groups for Pine (Usenet    ]
[ comp.mail.pine or by mail, pine-info@cac.washington.edu), as well as     ]
[ zmailer (mail only, zmailer@cs.toronto.edu) and MIME (comp.mail.mime or  ]
[ mail info-mime@cs.utk.edu), in the event some reader of the latter would ]
[ volunteer to help zmailer with this problem.                             ]
[    The real question appears near the end.  All that leads up to it is   ]
[ purely background fluff.                                                 ]

On Thu, 13 Apr 1995, David L Miller wrote:

> > If the 8th bit is delivered correct and not cut off by some stupid gateways,
> > there is no need to put an ordinary ISO-8859-X text into 'quoted-printable'.
> > I'd like to turn that off, too.
> 
> The next release of Pine will be able to negotiate 8bit c-t-e for text if
> your MTA understands ESMTP 8BITMIME. 
> 
> |\ |  |\/|  David L. Miller    dlm@cac.washington.edu  (206) 685-6240

    Given that Pine3.92 will be supporting the 8BITMIME ESMTP option, it
then becomes the responsibility of the MTA to guarantee proper delivery
of 8BITMIME mail to the recipient, performing any conversions as are
needed to get the message through any intermediate MTAs without losing
any of the information originally contained in the message.
    (For a description of the 8BITMIME ESMTP option, refer to RFC 1652.
In this RFC, an MTA is given two options -- either convert the message,
or bounce it as undeliverable.  It is to be hoped that few, if any, MTAs
would choose the latter course, so instead we concentrate on the conversion
of the 8-bit message to some 7-bit-safe equivalent.)

    At present, there are relatively few MTAs which claim to speak the
8BITMIME needed to properly handle the message.  The ones I know of are:
PMDF
certain versions of the Mercury NLM for Novell LANs
newer versions of zmailer from Matti Aarnio
and, not yet released, BSD Sendmail v8.7.

    From what I can tell, PMDF is conformant to the RFC, as will be the
default configuration of sendmail v8.7.  However, bouncing a variety of
experimental messages off the other MTAs revealed that they don't quite
match the performance expected by the RFC.  (To their credit, they don't
bounce the message, which I would consider a hindrance to the widespread
use of 8BITMIME in preference to the present widespread practice of
JustSend8 mail with the possibility of loss of data or even bounced mail.)

    My bounce tests were performed by composing a MIME message with an
8-bit text/plain part, which I then sent with an 8-bit-clean version of
BSD Sendmail v8 (does not convert from 8-bit to 7-bit equivalent) to an
8BITMIME mailer to get the conversion needed to conform to the RFC in the
ESMTP dialog.  The resulting 8BITMIME message was then delivered to the
mailer in question, with a destination address of a non-8BITMIME MTA,
often a version of IDA sendmail which is decidedly *not* 8-bit clean.
The proper behavior of the MTA under attack in this fashion should be to
convert the message to a 7-bit form by either BASE64-ing it, or converting
it to a QUOTED-PRINTABLE-ized form.  The RFC states that in no circumstances
should the suspect MTA try to deliver 8-bit data.

    The JustSend8 to 8BITMIME conversion was usually performed by a version
of zmailer, with 8BITMIME support added by Matti Aarnio.  The log files on
the host indicate that the MAIL command of the ESMTP dialog conformed to
the RFC, even though the original incoming message arrived with SMTP as
JustSend8 8-bit data.

    The version of PMDF running at Innosoft performed conversions of MIME
text/plain messages with no problems.  I also generated a multipart/mixed
message with two body parts of text/plain with 8-bit data, which PMDF was
able to handle properly.  (At least, in a way which would not affect the
content of the multipart messages which Pine creates, which consist of one
text/plain part, and any number of attachments [text, image, application]
already made mail-safe in BASE64 form.)

    At the time of my play^H^H^H^Hexperimentation, half a year or more ago,
Eric Allman was only mentioning incorporating 8BITMIME support into BSD
sendmail.  Even today, no 8.7.1 release has made it out, so there is no
way for me to test the compliance of it, though that will be one of the
first things I test when it does get released, Real Soon Now.

    However, the behavior of Mercury showed that, while the banner it
provided would lead one into believing it supported 8BITMIME messages
and all that this implies, my tests showed otherwise.  The version that I
pounded on was 1.13, which was configured to pass all mail it received with
a destination outside its domain to BSD sendmail v8.6 for delivery.  As
this version of sendmail makes no claim to handle 8BITMIME data, the
proper response of Mercury should have been to make the conversion, which
it did not do.  As a result, I only saw the stripped 7-bit equivalents of
the message I was sending in the IDA destination mailbox.
    Perhaps this announcement of 8BITMIME was intended for incoming mail
which would get delivered locally and never make it out.  But as soon as
one would make use of Mercury for handling outgoing mail, or use it as
SMTP server, or if a recipient on the other side of Mercury were to forward
mail elsewhere, mail which might be intended as 8BITMIME data suddenly
becomes JustSend8 data.  When I checked the system I had tortured before,
I discovered that the version of Mercury running had been upgraded to 1.20,
and it no longer laid claim to being able to speak 8BITMIME to the world.

    The moral behind this part of my research is that one should not point
Pine3.92 at a Mercury 1.13 SMTP server and attempt to speak 8BITMIME, and
that sites running 1.13 should upgrade their version to avoid advertising
the false claim of conformance to RFC 1652.  This should be no problem for
configuration of Pine, since one must select whether one wants Pine (or
PC-Pine) to try to send mail with 8BITMIME if the SMTP server claims to
support it -- it's not the default, from what I've been told.

    The problem of zmailer is a bit more complicated.  I was using it to
generate the ESMTP 8BITMIME dialog, for which it performed admirably when
speaking to an 8BITMIME ESMTP server regardless of top-level message type
(text/plain or multipart/mixed).  If I were to send JustSend8 mail to it
for delivery to a non-8BITMIME MTA, it would add MIME headers and convert
to QUOTED-PRINTABLE encoding.  If I were to send JustSend8 mail with MIME
C-T-E: 8BIT headers, it would convert the text/plain message to Q-P for
delivery.  So it was able to handle simple single-part messages well.

    However, my tests half a year ago revealed problems handling multipart
messages.  The mea version of zmailer did not recognize them, and would
convert the entire message to Q-P, adding its own MIME headers for the
unknown charset it believed it was being presented with.  In other words,
it simply did not understand multipart messages when any part of them had
an 8BIT content.  The resulting mail that was sent out ended up garbled
so that a MIME-aware MUA could not make sense of the attachments, but would
display the entire message.  In other words, zmailer, while claiming support
for 8BITMIME, could only handle single-part messages properly.

    That was then, this is several months later.  Luckily, zmailer is one
of those programs that is actively being worked on thanks to Matti.  So,
again, I dragged out my multipart 8-bit mail and repeated my tests, since
there was a new version, different from the one I had tested earlier.  This
time my tests were not as thorough -- I simply handed the JustSend8 mail
directly to the zmailer in question, rather than running it through another
zmailer to perform the 8BITMIME dialog according to the RFC (my guess was
that if zmailer treated JustSend8 mail as 8BITMIME incoming, it would still
do so regardless of content-type).

    I didn't try the untagged JustSend8 mail with this pass, and I don't
think that I tried JustSend8 mail with MIME headers, as I had already made
my tests with those earlier and didn't think zmailer's behavior would be
any different.  However, there was a distinct change in the way my multipart
message was handled.  Luckily, the top-level headers were left intact.  This
means that any MIME-aware MUA would have no difficulty recognizing an
attachment from Pine (I understand that Pine will still use BASE64 encoding
for attachments as it does now).  This is quite an improvement over the
previous behavior.  On the other hand, no attempt was made to convert the
message parts with C-T-E of 8BIT.  This means that my IDA mailbox received
the 7-bit stipped equivalents of the characters I was testing.  In other
words, the zmailer I tested handles text/plain messages as 8BITMIME
messages, but it handles multipart messages as JustSend8 messages.

    Thus, we come to the reason behind this excessively long post.  I've
posted this to the MIME group in case some generous soul there happens to
want to take a look at zmailer and offer the code needed for it to perform
the complete 8BITMIME conversion for all parts of a message.  My guess is
that it shouldn't be too difficult, as top-level conversion is implemented
adequately.  But then, I don't write code.  With this, then zmailer's
claim to be 8BITMIME-compliant will be much less likely to cause problems.

    I'm also including this in both the zmailer and Pine discussion lists,
as any SysAdmin who runs both should be aware of this incompatibility of
zmailer with the upcoming Pine3.92.  Again, maybe someone who likes to hack
on zmailer code would volunteer to contribute the changes needed.

    And I'm asking this of the Pine developers:  Given that zmailer claims
8BITMIME support, and indeed is able to handle text/plain messages, which
in my opinion would be the bulk of the messages with 8-bit data it would
receive from Pine3.92, but has difficulty with multipart messages, would
it be possible to include in Pine, in addition to the option to speak
8BITMIME to the SMTP server, an option to disable 8BITMIMEspeak when
sending multipart messages and the first option is set, for peaceful
co-existence with zmailer?

    My reasoning behind this is as follows...  If Pine will be sending
its data in Q-P or BASE64 form anyway if the ESMTP dialog fails to reveal
an 8BITMIME server, it should be simple to add code to detect if the message
is multipart and act similarly.  This should be a temporary thing, as I am
guessing that eventually zmailer will have full 8BITMIME compliance.  And,
as zmailer is one of a small number of MTAs which attempt to conform to
8BITMIME, helping to advance its acceptance in a world where JustSend8 is
preferred by many over QUOTED-PRINTABLE, and is seeing use in environments
where 8-bit data is commonly used for text mail, I'd hope that people would
be willing to accommodate it to work around its present shortcomings.  (I
don't know how many sites use zmailer.)  Furthermore, MIME multipart messages
are much less useful to a recipient without a MIME MUA, while text/plain
messages delivered with 8BITMIME are readable by the majority of mailreaders,
certainly those in use where JustSend8 non-English text prevails, and so
the loss of sending multipart messages with Q-P is rather insignificant.

    I am guessing that with the upcoming release of BSD Sendmail v8.7,
and the ability of Pine3.92 to make use of 8BITMIME ESMTP servers, the
use of 8BITMIME in sending mail will increase such that eventually zmailer
will have to fully conform, or drop its claim, in order to co-exist with
sites that still run 7-bit MTAs like my IDA site.  Then, instead of people
complaining about the MUAs like Pine using Q-P, they can focus on upgrading
the MTAs to speak 8BITMIME so that a good proportion of mail can be
delivered with 8-bit data intact, and in easily readable form.


Barry Bouwsma
<barryb@tuke.sk>